Conversation
PixiBixi
force-pushed
the
bug/prometheus-crypt-hash-validation
branch
from
August 26, 2026 22:28
bc36a67 to
acef066
Compare
…ashes The prometheus endpoint reads its basic auth users from the secret named by the prometheus-endpoint-auth-secret annotation. For each user it split the stored value on '$' and read fields 1 and 2 to rebuild the crypt salt. Nothing checked that these fields existed. A value that is not a crypt hash, which is what an operator writes when they skip the mkpasswd step described in documentation/prometheus.md, produced a slice of one element. Reading field 1 panicked. The handler runs in the sync goroutine, so the panic took down the controller, and it came back at every sync as long as the secret stayed in place. Any user allowed to write a secret in a watched namespace could stop the controller. This change reads the salt through cryptSalt, which refuses anything that does not carry the SHA-256 identifier. A rejected user is skipped with an error in the logs and the endpoint keeps answering 401 for them, which is the outcome they already had: a value that is not a hash never matches what crypt computes. The other users of the secret keep their access. Requiring the identifier, and not only a leading '$', also separates a hash from a plaintext password that contains '$'. A value like "$ecret$pass$word" is shaped like a hash. Kept, it would lock the user out with nothing in the logs to say why. cryptSalt also cuts the hash at its last '$' instead of at a fixed field index. A hash carrying a rounds= parameter, which mkpasswd writes with -R, used to yield "$5$rounds=N$" as the salt, losing the salt itself. That user could never authenticate, in silence. The salt now keeps the parameter, where crypt needs it. Backport this fix to 3.2 and 3.1.
These tests drive PrometheusEndpoint.Update with a store holding a main configmap and the secret it names. The first test gives a plaintext password. Without the fix it panics with an index out of range, which is what the controller did in production. The second test walks the shapes a purely syntactic check lets through, among them a plaintext password carrying '$' and a hash from another algorithm. The third test reads back the registered user and replays the comparison that prometheusHandler performs on every request: it hashes the password with the stored salt and expects the stored hash. It runs once with a plain salt and once with a rounds= salt. A salt cut at the wrong '$' passes an equality assertion on its own but fails this one. The last test puts a good user and a bad one in the same secret, and checks that the good one keeps its access.
oktalz
force-pushed
the
bug/prometheus-crypt-hash-validation
branch
from
September 3, 2026 12:01
acef066 to
d68605d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The prometheus endpoint panics when the secret named by
prometheus-endpoint-auth-secretholds a password that is not a SHA-256 crypt hash.pkg/handler/prometheus.gosplit the stored value on$and read fields 1 and 2 torebuild the crypt salt, without checking that these fields existed:
A plaintext password, which is what an operator writes when they skip the
mkpasswdstep of
documentation/prometheus.md, gives a slice of one element. Reading field 1panics. The handler runs in the sync goroutine, so the panic takes down the controller,
and it comes back at every sync as long as the secret stays in place. Any user allowed
to write a secret in a watched namespace can stop the controller.
Reproduce
With
--prometheuson:Fix
The salt is read through
cryptSalt, which refuses anything that does not carry theSHA-256 identifier. A rejected user is skipped with an error in the logs and the endpoint
keeps answering 401 for them, which is the outcome they already had: a value that is not
a hash never matches what crypt computes. The other users of the secret keep their access.
Requiring the identifier, and not only a leading
$, also separates a hash from aplaintext password that contains
$.$ecret$pass$wordis shaped like a hash; kept, itwould lock the user out with nothing in the logs to say why.
Second bug, same expression
cryptSaltcuts the hash at its last$instead of at a fixed field index. A hashcarrying a
rounds=parameter, whichmkpasswdwrites with-R, used to give$5$rounds=N$as the salt, losing the salt itself. That user could never authenticate,in silence.
Going through
crypt.Verify(hash, password)instead was tried and does not work here:common.Salt.Decodedoesbytes.SplitN(raw, '$', 4), so on$5$rounds=N$salt$digestthe fourth field absorbs
salt$digestas a whole and verification always fails. Thesalt has to be extracted before the call.
Affected versions
The faulty expression was introduced by 279d16d and is present from v3.1.11 and
v3.2.0 onwards. v3.1.10 and earlier build a HAProxy userlist instead and are not
affected.
Tests
pkg/handler/prometheus_test.go, 20 subcases. The one that matters replays thecomparison
prometheusHandlerperforms on every request: it hashes the password withthe stored salt and expects the stored hash, with and without
rounds=. A salt cut atthe wrong
$passes an equality assertion on its own but fails this one.Checked that the tests fail against the previous implementation: the first one panics
again, and the
rounds=case fails.