Add sustainable Go parser in prometheus/common

[bwplotka]

See discussions around prometheus/common#808 cc @ywwg

Some Slack discussion

[bwplotka]

Ideally based on ABNF one similar to Python CI https://github.com/prometheus/docs/actions/runs/17431855980/job/49491877093?pr=2634

[bwplotka]

I iterated a bit on the most promising Go libs that might help here (looking for something that would help with both lexing and parser generation, similar to our lex+yacc PromQL use), but the state is not ideal.

I tried:

• https://github.com/pandatix/go-abnf but it fails to parse our grammar without any meaningful error ("failed to parse grammar").
• https://github.com/ghettovoice/abnf/ (probably the most promising ABNF lexer I could find as it has a solid CLI to generate lexing) , but it needs a bit more work (e.g. CLI support for =/ (incremental alternative) ghettovoice/abnf#46 and definitely error handling). It also could be more typed (needed for better parsing later).

We could also try some helpers for handwriting parsers e.g. https://github.com/oleiade/gomme

(1) I also noticed we already use do use "lex" via golex + manual (similar to how yacc would do this but tailored?) parser on textparse side.

Then we use Yacc language for PromQL 🙈

Then for common decoder (text) we hand write it (following https://go.dev/talks/2011/lex.slide idea)

This space is interesting, looks like all those tools are somewhat discourages for high performance implementations and error parsing: (see https://tomassetti.me/why-you-should-not-use-flex-yacc-and-bison/ +discussion, https://www.reddit.com/r/ProgrammingLanguages/comments/1bd215t/why_all_the_hate_for_lexparse_generators/ and ofc Russ Cox presentation.

So far.... ABNF is amazing for readability on spec, it's doable to have a common Go code (we have Python code already) for validation. But for parsing, relying on ABNF needs a bit more work (which might be worth it if we want to maintain 3+ versions of OM (1.0, 1.1, 2.0)), but it likely won't be as fast as handwritten one. Also for genericity we would need to repeat code for each OM version (which could be fine).

However, we already use Lex in scrape textparse and we likely need to transform this lex for new versions (or create new ones). It would be ideal if we somehow reuse similar technique or code with that approach. Two challenges remain as for scrape text parse:

• (dep hell) We use prometheus/prometheus model types (and our custom parser interface)

	"github.com/cespare/xxhash/v2"
	"github.com/prometheus/common/model"

	"github.com/prometheus/prometheus/model/exemplar"
	"github.com/prometheus/prometheus/model/histogram"
	"github.com/prometheus/prometheus/model/labels"
	"github.com/prometheus/prometheus/model/value"
	"github.com/prometheus/prometheus/schema"
)

• (velocity) we can't make it really stable as we want to prioritize Prometheus use for this hot path

For common / standalone expfmt parsing ideally we stick to dto "github.com/prometheus/client_model/go" and "github.com/prometheus/common/model" deps.

[bwplotka]

Switching from ABNF to lex/flex that is used by all our parsers could be useful, but ABNF feels more readable. Maybe conversion makes sense?

[bwplotka]

Another idea is to switch to https://github.com/antlr/antlr4/blob/4.13.2/doc/lexicon.md which seems to be the newest way of doing things (e.g. https://blog.gopheracademy.com/advent-2017/parsing-with-antlr4-and-go/, https://github.com/antlr/antlr4)