Skip to content

Commit ef26df3

Browse files
committed
feat(adopt): confirm the installed node's fingerprint before trusting it
After a node installs and reboots during adoption it presents a new management certificate that nothing links to the maintenance certificate the operator confirmed. Adoption no longer pins it on first use. It captures the certificate, streams the awaiting-fingerprint-confirmation phase with its SHA-256 and the adoption ID, and waits for ConfirmAdoptionFingerprint. A matching fingerprint (case, colons and spaces ignored) resumes the adoption, and only then is the certificate pinned and the node dialed against it. A mismatch, no confirmation within 15 minutes, or the client going away fails the adoption with nothing pinned, recorded or registered. Confirmations and refusals are audited with the operator. The new RPC is excluded from the MCP catalog. Closes #144 Refs #135 Signed-off-by: Bugs5382 <12115015+Bugs5382@users.noreply.github.com>
1 parent cd3a52f commit ef26df3

11 files changed

Lines changed: 732 additions & 88 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@
1818

1919
A node is linked to the FM via **mutual consent**: the operator declares the FM's URL + trust anchor in the node's machine config and reboots; the node calls the FM's `EnrollNode` with a TPM EK attestation; the FM operator approves in the UI; the FM issues the node a 90-day peer cert (EKU `clientAuth` only). Either side can revoke or unlink.
2020

21-
Adopting a fresh node from its maintenance endpoint is safe to retry. The manager mints a bootstrap admin credential per node name and stores it under `MANAGER_NODE_CREDS_DIR` (default `/var/lib/cryptos-manager/node-creds`). A retry reuses it, because a node that got as far as installing trusts only that credential. When the node reports it has already booted its installed system, the retry skips the install, runs the first-boot ceremony only if a Root has not finished it, and registers the node. If the stored credential is gone, the node refuses the handshake and the adoption fails with a hint to reset the node from its console and adopt it again.
21+
Adopting a fresh node from its maintenance endpoint is safe to retry. The manager mints a bootstrap admin credential per node name and stores it under `MANAGER_NODE_CREDS_DIR` (default `/var/lib/cryptos-manager/node-creds`). A retry reuses it, because a node that got as far as installing trusts only that credential. When the node reports it has already booted its installed system, the retry skips the install, runs the first-boot ceremony only if a Root has not finished it, and registers the node. If the stored credential is gone, the node refuses the handshake and the adoption fails with a hint to reset the node from its console and adopt it again. After the node reboots into its installed system, the adoption pauses until an admin confirms the certificate fingerprint it presents against the `Mgmt SHA-256` line on its console; see [docs/node-trust.md](docs/node-trust.md#-what-adoption-records).
2222

2323
The node always checks the manager's admin credential, and the manager always checks the node's server certificate: against the node's recorded CA chain once the node signs its management certificate with its CA, or against a pinned `server.crt` before that. A node it can't verify is refused. `manager -check-node-trust` lists how each node is verified. See [`docs/node-trust.md`](docs/node-trust.md).
2424

‎docs/mcp.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -286,6 +286,7 @@ Not registered as tools at all:
286286

287287
- `ExportCAKey` and `ImportCAKey` (CA key material); `DecommissionNode`; `AdoptNode`,
288288
`PreviewAdoption` and `ListInstallDisks` (disk wipe and provisioning);
289+
`ConfirmAdoptionFingerprint` (a person confirms a new node's identity from its console);
289290
`ApplyNodeConfig` and `SetNodeProtocol` (a node's configuration and its enrolment
290291
endpoints); `RekeyNode`; `CreateEnrollment` and `ApproveEnrollment` (node admin
291292
credentials, subordinate signing); `RenameNode`; `RevokeOperatorCredential`, the

‎docs/node-trust.md‎

Lines changed: 36 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -102,28 +102,53 @@ Adoption records what the manager needs to verify the node afterwards:
102102

103103
1. The fingerprint you confirm in the adopt wizard pins the node's
104104
maintenance-mode certificate for the install steps.
105-
2. The installed node boots with a new management certificate. When it comes
106-
back, the manager saves that certificate as the node's `server.crt` and
107-
shows its fingerprint in the adoption progress:
105+
2. The installed node boots with a new management certificate. Nothing links
106+
it to the maintenance certificate you confirmed, so adoption pauses on the
107+
`awaiting-fingerprint-confirmation` phase and shows the fingerprint the
108+
node presents:
108109

109110
```text
110-
node is back in running mode; pinned its management certificate sha256 5a0e...c3. Check it against the Mgmt SHA-256 line on the node's console
111+
the node is back in running mode and presents certificate sha256 5a0e...c3. Compare it with the Mgmt SHA-256 line on the node's console and confirm it to continue
111112
```
112113

113114
> [!CAUTION]
114-
> This pin is trust on first use: nothing links the installed node's new
115-
> certificate to the maintenance certificate you confirmed. Compare the
116-
> fingerprint with the `Mgmt SHA-256` line on the node's console. If they
117-
> differ, something other than your node answered on its address; stop and
118-
> find out what before you use the node.
115+
> Compare the fingerprint with the `Mgmt SHA-256` line on the node's
116+
> console before you confirm it. If they differ, something other than your
117+
> node answered on its address: choose "Does not match" in the wizard and
118+
> find out what answered before you adopt again.
119+
120+
Confirming (the adopt wizard, or `ConfirmAdoptionFingerprint` with the
121+
stream's `adoption_id`) resumes the adoption. Only then does the manager
122+
save that certificate as the node's `server.crt` and dial the node,
123+
verified against it. The comparison ignores case, colons and spaces.
124+
125+
The adoption stops, and nothing is pinned, recorded or registered, when:
126+
127+
- the confirmed fingerprint differs from the presented one (the call and
128+
the adoption fail with `InvalidArgument`);
129+
- nobody confirms within 15 minutes (`DeadlineExceeded`);
130+
- the wizard cancels or the adoption stream is closed.
131+
132+
The manager admin credential minted for the adoption stays, so adopting
133+
the node again resumes from here. Both a confirmation and a refused
134+
fingerprint are audited (`node-adoption-fingerprint-confirmed`,
135+
`node-adoption-fingerprint-rejected`) with the operator who sent it.
136+
137+
> [!WARNING]
138+
> The waiting adoption lives in the manager replica that runs the adoption
139+
> stream. With `replicaCount` above 1, a confirmation that reaches another
140+
> replica returns `NotFound`; send it again so it reaches the replica that
141+
> holds the stream, or adopt with a single replica.
119142
120143
3. A root's ceremony gives it its CA. The manager reads the CA chain from the
121144
node and saves it as `ca.crt` next to the admin certificate. An intermediate
122145
or issuing node gets its CA when its subordinate enrollment is approved, and
123146
the manager saves the signed chain the same way.
124147

125-
A retried adoption replaces the node's old `server.crt` with the certificate
126-
the re-installed node presents.
148+
A retried adoption asks you to confirm the certificate the node presents
149+
again, then replaces the node's old `server.crt` with it. A root that has
150+
already run its ceremony presents a CA-signed certificate by then; confirm that
151+
one the same way.
127152

128153
## 🔄 When the node gets its CA
129154

‎go.mod‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ require (
3737
)
3838

3939
require (
40-
github.com/CryptOS-PKI/api v0.0.0-20260930170156-1de16da02cdf
40+
github.com/CryptOS-PKI/api v0.0.0-20260930184803-29b56ecd3e46
4141
golang.org/x/net v0.58.0 // indirect
4242
golang.org/x/sys v0.47.0 // indirect
4343
golang.org/x/text v0.41.0 // indirect

‎go.sum‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,8 @@ github.com/Bugs5382/go-apperr v1.0.0 h1:YfepdvZm+ctIt5DDJ9XVbwPRk+lo9aE95btYqaSf
44
github.com/Bugs5382/go-apperr v1.0.0/go.mod h1:4kzGmhaC5tcPDskwAi8HIRDKSkMrJdvOUxik9Uuij1k=
55
github.com/Bugs5382/go-log v1.3.0 h1:lJv2iv6yL9l1Eqii62ArcJ1eoKbpLFdrY7L9gPNSqcI=
66
github.com/Bugs5382/go-log v1.3.0/go.mod h1:OGgIWWKKNg2AXVJlbVJZb/s8dnqqsrKwWznYXik547E=
7-
github.com/CryptOS-PKI/api v0.0.0-20260930170156-1de16da02cdf h1:ZnVAluaEprpxQA8AXLafPoip1J4swQloeBOhJFnHcw4=
8-
github.com/CryptOS-PKI/api v0.0.0-20260930170156-1de16da02cdf/go.mod h1:tmeaYhNoUblqPYYNtTbGBGIUsZ7xLATgLdKq4S33rmE=
7+
github.com/CryptOS-PKI/api v0.0.0-20260930184803-29b56ecd3e46 h1:vLGhyS1TdPo3CVVZsOHXrD0TxICVhtoHW3x5XlKEwX8=
8+
github.com/CryptOS-PKI/api v0.0.0-20260930184803-29b56ecd3e46/go.mod h1:/x//qWOzXOQ3lQDa3xy4ncsVZWv1KHEgvV+IcxAGSAQ=
99
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
1010
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
1111
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=

‎internal/fleet/adopt.go‎

Lines changed: 85 additions & 65 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ limitations under the License.
1919
import (
2020
"context"
2121
"crypto/sha256"
22+
"crypto/x509"
2223
"encoding/hex"
2324
"encoding/json"
2425
"errors"
@@ -143,10 +144,8 @@ func (s *Service) AdoptNode(ctx context.Context, req *connect.Request[fleetv1.Ad
143144
if err := requireAdmin(ctx); err != nil {
144145
return err
145146
}
146-
sink := func(phase, detail string, done bool) error {
147-
return stream.Send(s.adoptResponse(req.Msg, phase, detail, done))
148-
}
149-
return s.runAdoption(ctx, req.Msg, sink)
147+
id := newAdoptionID()
148+
return s.runAdoptionAs(ctx, id, req.Msg, s.adoptSink(id, req.Msg, stream.Send))
150149
}
151150

152151
// adoptResponse builds one streamed adoption message. The final message of a
@@ -161,11 +160,17 @@ func (s *Service) adoptResponse(msg *fleetv1.AdoptNodeRequest, phase, detail str
161160
return resp
162161
}
163162

164-
// runAdoption is the transport-independent adoption orchestration, driving the
165-
// phaseSink so the Connect handler and tests share one code path. A step that
166-
// fails streams an error phase (done) and returns a Connect error; it never
167-
// hangs — the reboot wait is bounded.
163+
// runAdoption runs an adoption under a new adoption ID.
168164
func (s *Service) runAdoption(ctx context.Context, msg *fleetv1.AdoptNodeRequest, send phaseSink) error {
165+
return s.runAdoptionAs(ctx, newAdoptionID(), msg, send)
166+
}
167+
168+
// runAdoptionAs is the transport-independent adoption orchestration, driving
169+
// the phaseSink so the Connect handler and tests share one code path. id names
170+
// the adoption for ConfirmAdoptionFingerprint. A step that fails streams an
171+
// error phase (done) and returns a Connect error; it never hangs — the reboot
172+
// and confirmation waits are bounded.
173+
func (s *Service) runAdoptionAs(ctx context.Context, id string, msg *fleetv1.AdoptNodeRequest, send phaseSink) error {
169174
if s.dialMaintenance == nil {
170175
return connect.NewError(connect.CodeUnimplemented, errors.New("fleet: adoption not configured"))
171176
}
@@ -244,20 +249,33 @@ func (s *Service) runAdoption(ctx context.Context, msg *fleetv1.AdoptNodeRequest
244249
}
245250
}
246251

247-
// Wait for the running node, dialing with the admin cert it trusts (a
248-
// managed dial, no maintenance pin: the running server cert differs). The
249-
// certificate it comes back with is pinned first, so the dial verifies it.
250-
conn, pinnedSHA256, err := s.awaitRunningNode(ctx, nodeName, endpoint, adminCertPath, adminKeyPath)
252+
// Wait for the running node. The certificate it comes back with has no
253+
// link to the maintenance certificate the operator confirmed, so the
254+
// adoption pauses until the operator confirms it from the node's console;
255+
// only then is it pinned and the node dialed with the admin cert it trusts
256+
// (a managed dial, verified against the pin).
257+
node := store.Node{Name: nodeName, Endpoint: endpoint, AdminCert: adminCertPath, AdminKey: adminKeyPath}
258+
running, err := s.awaitRunningCert(ctx, node)
251259
if err != nil {
252260
return s.adoptFail(send, connect.CodeDeadlineExceeded, err)
253261
}
254-
defer func() { _ = conn.Close() }()
255-
if pinnedSHA256 != "" {
256-
if err := send(phaseAwaitingReboot, fmt.Sprintf(
257-
"node is back in running mode; pinned its management certificate sha256 %s. Check it against the Mgmt SHA-256 line on the node's console", pinnedSHA256), false); err != nil {
258-
return err
262+
if running != nil {
263+
sum := sha256.Sum256(running.Raw)
264+
presented := hex.EncodeToString(sum[:])
265+
if err := s.awaitFingerprintConfirm(ctx, id, nodeName, endpoint, presented, send); err != nil {
266+
return s.adoptFailErr(send, err)
259267
}
268+
path, err := nodeclient.WritePin(node, running)
269+
if err != nil {
270+
return s.adoptFail(send, connect.CodeInternal, err)
271+
}
272+
log.Printf("fleet: adopt %s: pinned the confirmed management certificate sha256 %s at %s", nodeName, presented, path)
260273
}
274+
conn, err = s.awaitRunningNode(ctx, node)
275+
if err != nil {
276+
return s.adoptFail(send, connect.CodeDeadlineExceeded, err)
277+
}
278+
defer func() { _ = conn.Close() }()
261279

262280
// A root self-signs its CA via the first-boot ceremony. A subordinate
263281
// (intermediate/issuing) node cannot: the node has already staged its own
@@ -332,71 +350,62 @@ func (s *Service) runAdoption(ctx context.Context, msg *fleetv1.AdoptNodeRequest
332350
return send(phaseEstablished, "node adopted and established", true)
333351
}
334352

335-
// awaitRunningNode waits for the node to finish installing, self-reboot, and
336-
// come back in RUNNING mode: serving mTLS with the bootstrap admin trust,
337-
// awaiting the first-boot ceremony. It dials with a managed admin-cert dial
338-
// (no maintenance pin — the running server cert differs from the maintenance
339-
// one) and confirms a cheap RPC before returning, bounded by rebootWait so a
340-
// node that never returns streams an error rather than hanging. A grace period
341-
// lets the node begin rebooting before the first poll.
342-
//
343-
// With a capture seam, every poll first reads the certificate the node
344-
// presents and pins it (replacing any pin from an earlier adoption), so the
345-
// dial verifies the node against it. The node's new certificate has no link
346-
// to the maintenance certificate the operator confirmed, so this is trust on
347-
// first use; the returned SHA-256 is streamed for the operator to check
348-
// against the node's console.
349-
func (s *Service) awaitRunningNode(ctx context.Context, name, endpoint, adminCertPath, adminKeyPath string) (NodeConn, string, error) {
353+
// awaitRunningCert waits for the node to finish installing, self-reboot and
354+
// come back in RUNNING mode, and returns the certificate it presents there.
355+
// Nothing is pinned: the caller has the operator confirm it first. Without a
356+
// capture seam it returns nil and the caller dials as before. Bounded by
357+
// rebootWait after a grace period that lets the node begin rebooting.
358+
func (s *Service) awaitRunningCert(ctx context.Context, node store.Node) (*x509.Certificate, error) {
350359
select {
351360
case <-ctx.Done():
352-
return nil, "", ctx.Err()
361+
return nil, ctx.Err()
353362
case <-time.After(rebootPollGrace):
354363
}
355-
356-
node := store.Node{Name: name, Endpoint: endpoint, AdminCert: adminCertPath, AdminKey: adminKeyPath}
364+
if s.captureServerCert == nil {
365+
return nil, nil
366+
}
357367
deadline := time.Now().Add(rebootWait)
358368
for {
359-
pinned, perr := s.pinRunningCert(node)
360-
if perr == nil {
361-
conn, err := s.dial(node)
362-
if err == nil {
363-
// A lazy gRPC client dial can succeed before the server is up;
364-
// confirm the node answers a cheap RPC before proceeding.
365-
if _, serr := conn.GetStatus(ctx); serr == nil {
366-
return conn, pinned, nil
367-
}
368-
_ = conn.Close()
369-
}
369+
cert, err := s.captureServerCert(node)
370+
if err == nil {
371+
return cert, nil
370372
}
371373
if time.Now().After(deadline) {
372-
return nil, "", fmt.Errorf("fleet: node did not come back in running mode on %s within %s of install", endpoint, rebootWait)
374+
return nil, fmt.Errorf("fleet: node did not come back in running mode on %s within %s of install: %w", node.Endpoint, rebootWait, err)
373375
}
374376
select {
375377
case <-ctx.Done():
376-
return nil, "", ctx.Err()
378+
return nil, ctx.Err()
377379
case <-time.After(rebootPollGap):
378380
}
379381
}
380382
}
381383

382-
// pinRunningCert captures the certificate node presents and pins it,
383-
// returning its SHA-256. Without a capture seam it pins nothing.
384-
func (s *Service) pinRunningCert(node store.Node) (string, error) {
385-
if s.captureServerCert == nil {
386-
return "", nil
387-
}
388-
cert, err := s.captureServerCert(node)
389-
if err != nil {
390-
return "", err
391-
}
392-
path, err := nodeclient.WritePin(node, cert)
393-
if err != nil {
394-
return "", err
384+
// awaitRunningNode dials the running node with a managed admin-cert dial
385+
// (verified against the confirmed pin) and confirms a cheap RPC before
386+
// returning, bounded by rebootWait so a node that never answers streams an
387+
// error rather than hanging.
388+
func (s *Service) awaitRunningNode(ctx context.Context, node store.Node) (NodeConn, error) {
389+
deadline := time.Now().Add(rebootWait)
390+
for {
391+
conn, err := s.dial(node)
392+
if err == nil {
393+
// A lazy gRPC client dial can succeed before the server is up;
394+
// confirm the node answers a cheap RPC before proceeding.
395+
if _, serr := conn.GetStatus(ctx); serr == nil {
396+
return conn, nil
397+
}
398+
_ = conn.Close()
399+
}
400+
if time.Now().After(deadline) {
401+
return nil, fmt.Errorf("fleet: node did not come back in running mode on %s within %s of install", node.Endpoint, rebootWait)
402+
}
403+
select {
404+
case <-ctx.Done():
405+
return nil, ctx.Err()
406+
case <-time.After(rebootPollGap):
407+
}
395408
}
396-
sum := sha256.Sum256(cert.Raw)
397-
fp := hex.EncodeToString(sum[:])
398-
log.Printf("fleet: adopt %s: pinned the running node's management certificate sha256 %s at %s (trust on first use; check it against the node's console)", node.Name, fp, path)
399-
return fp, nil
400409
}
401410

402411
// adoptFail streams a terminal error phase and returns the Connect error, so a
@@ -407,6 +416,17 @@ func (s *Service) adoptFail(send phaseSink, code connect.Code, cause error) erro
407416
return connect.NewError(code, cause)
408417
}
409418

419+
// adoptFailErr is adoptFail for an error that already carries its Connect
420+
// code.
421+
func (s *Service) adoptFailErr(send phaseSink, err error) error {
422+
var cerr *connect.Error
423+
if errors.As(err, &cerr) {
424+
_ = send(phaseError, cerr.Message(), true)
425+
return err
426+
}
427+
return s.adoptFail(send, connect.CodeInternal, err)
428+
}
429+
410430
// relayCeremony forwards every ceremony event as a ceremony phase and reports
411431
// whether the stream reached COMPLETE. io.EOF ends the stream cleanly.
412432
func relayCeremony(stream interface {

0 commit comments

Comments
 (0)