Skip to content

test(dgw): cover RDP credential injection end to end - #1938

Open
irvingouj@Devolutions (irvingoujAtDevolution) wants to merge 24 commits into
masterfrom
test/cred-injection-reconnect-e2e
Open

irvingouj@Devolutions (irvingoujAtDevolution) wants to merge 24 commits into
masterfrom
test/cred-injection-reconnect-e2e

Conversation

@irvingoujAtDevolution

@irvingoujAtDevolution irvingouj@Devolutions (irvingoujAtDevolution) commented Aug 20, 2026 •

Copy link
Copy Markdown
Contributor

No test ran an RDP client through a live Gateway with injected credentials, so the path from preflight to the RDP server had no end-to-end check. These tests start Gateway, provision credentials through /jet/preflight, and connect over TCP and over RDCleanPath to a fake RDP server, with a mock KDC for Kerberos.

They cover the scenarios in testsuite/src/rdp_injection/INTENT.md: NTLM and Kerberos injection where the RDP server receives the target credentials while the client sends only the proxy credentials, a target username without a domain staying on NTLM, reconnects with the same association token, and ordinary forwarding when provisioning is missing or expired. A wrong target password or an unreachable KDC fails without falling back to ordinary forwarding, and missing KDC information fails before Gateway connects to the RDP server.

The RDCleanPath tests run the published ironrdp-agent 0.1.0 CLI, which CI now installs and caches. The testsuite gains direct IronRDP, kdc, picky-krb, picky-asn1-der and x509-cert dependencies, all already in its build graph through Gateway.

Issue: DVLS-14697

@irvingoujAtDevolution

irvingouj@Devolutions (irvingoujAtDevolution) commented Aug 20, 2026 •

Copy link
Copy Markdown
Contributor Author

Implementation notes (current as of 007357720):

  • testsuite/tests/cli/dgw/cred_injection.rs covers the TCP path against a fake target that only records the X.224 cookie Gateway forwards.
  • testsuite/tests/cli/dgw/cred_injection_kdc.rs completes CredSSP on both legs against MockRdp (an sspi CredSspServer) and MockKdc (the kdc crate Gateway also embeds), and runs the RDCleanPath cases through the published ironrdp-agent 0.1.0 CLI.
  • The harness in testsuite/src/rdp_injection/ starts Gateway as a child process with disable_token_validation and debug logs, and provisions credentials through /jet/preflight (provision-credentials, plus provision-connection-options with krb_kdc for Kerberos).
  • Each test cites its scenario in testsuite/src/rdp_injection/INTENT.md with an // oracle: intent … §n marker. Gateway logs show which path Gateway took; the intent decides which path is right. The X.224 cookie asserts carry oracle: mutant markers, because no intent sentence says what the cookie must be.
  • MockRdp and FakeClosedTarget are reached as localhost, the host in the TERMSRV/localhost SPN, so they listen on both 127.0.0.1 and ::1. Windows tries ::1 first, and a refused connect there takes about 2 seconds, which hid a dial from the missing-krb_kdc test.

Note

LLM-assisted content (no human feedback).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds process-level coverage for Gateway RDP credential injection.

Changes:

  • Tests NTLM injection, reconnect attempts, expiration, and fail-closed routing.
  • Adds mock KDC and CredSSP integration coverage.
  • Extends test configuration and dependencies.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
Cargo.lock Locks new test dependencies.
testsuite/Cargo.toml Adds Kerberos and IronRDP test dependencies.
testsuite/src/dgw_config.rs Exposes Kerberos injection test configuration.
testsuite/tests/cli/dgw/mod.rs Registers new test modules.
testsuite/tests/cli/dgw/cred_injection.rs Adds credential injection lifecycle tests.
testsuite/tests/cli/dgw/cred_injection_kdc.rs Adds mock KDC/CredSSP integration testing.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread testsuite/Cargo.toml Resolved
@irvingoujAtDevolution irvingouj@Devolutions (irvingoujAtDevolution) changed the title test(dgw): cover credential injection reconnect test(dgw): cover credential injection reconnect hops Aug 21, 2026
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) marked this pull request as ready for review August 25, 2026 16:31
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) changed the base branch from refactor/rdp-proxy-credssp-extract to feat/enable-kerberos-injection August 25, 2026 16:41
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) force-pushed the test/cred-injection-reconnect-e2e branch 2 times, most recently from 3e5f5ba to 7a6fd9a Compare August 25, 2026 22:18
irvingouj@Devolutions (irvingoujAtDevolution) added a commit that referenced this pull request Aug 26, 2026
Keep provisioned injection mappings alive for as long as Gateway still
accepts the association token, so native RDP reconnects reuse the same
JWT and get injected again. If injection was provisioned but the
material is gone, checkout fails closed before we dial the target.

Generic RDP and RDCleanPath now share one CredSSP session path. Kerberos
uses the token `dst_hst` for the client-facing TERMSRV SPN.

I wrote the lifecycle contract down in
[src/credential/INTENT.md](https://github.com/Devolutions/devolutions-gateway/blob/85545af64a3f911d0a7e3d3d68ecb8a105a1121f/devolutions-gateway/src/credential/INTENT.md)
(following
[intent-files.md](https://github.com/Devolutions/devolutions-gateway/blob/master/docs/intent-files.md))
— that's what this PR implements.

### Why this design

Credentials live in two phases. Before the first connect, the preflight
`time_to_live` bounds how long staged secrets wait in memory. After the
first checkout, retention switches to the token acceptance deadline
(`exp` + leeway). If Gateway accepts the token for a reconnect, the
mapping has to still be there — the old one-shot `take_mapping` broke
exactly that. `jet_reuse` starts at disconnect and the TTL starts at
preflight, so neither can cap the other; the token deadline is the one
clock every connection attempt shares.

### Where to look hardest

- [Generation-guarded KDC
publication](https://github.com/Devolutions/devolutions-gateway/blob/85545af64a3f911d0a7e3d3d68ecb8a105a1121f/devolutions-gateway/src/credential_injection.rs#L667-L698):
re-provisioning the same JTI bumps a generation, and an older in-flight
lease can never overwrite a newer published KDC, so the newest
provisioning wins deterministically.
- [Deadline-driven
cleanup](https://github.com/Devolutions/devolutions-gateway/blob/85545af64a3f911d0a7e3d3d68ecb8a105a1121f/devolutions-gateway/src/credential_injection.rs#L774-L835):
expired session KDC material is removed by a background task, not lazily
on the next checkout, and secrets are zeroized on drop.
- [Release after
CredSSP](85545af):
the per-connection mapping and KDC lease drop as soon as both CredSSP
legs finish, so plain forwarding never holds secret material.

Tests cover staging expiry, reconnect reuse, generation replacement, and
physical removal without a later checkout. Manual run: 22-case DVLS
matrix with live NTLM and Kerberos CredSSP through Gateway. E2E hops are
stacked on #1938.

Issue: DVLS-14697

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@CBenoit Benoît Cortier (CBenoit) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, I really appreciate the e2e tests here.

Architecturally-wise, I think the "helpers" are getting big enough that it may warrant some of the code to be moved somewhere under the lib.rs of the testsuite crate. One example could be FakeRdpTarget. If the "TLS fixtures" are meant to be shared, then I think the library side could be the best place.

It’s already pretty close, but make things as self-contained as possible so it’s possible to use the INTENT.md / .intent.md convention for providing some human documentation about the most meaningful abstractions and important properties under test. I would appreciate such files before we merge this PR.

Comment thread testsuite/Cargo.toml Resolved
Base automatically changed from feat/enable-kerberos-injection to master August 26, 2026 17:48
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) marked this pull request as ready for review August 28, 2026 19:02
@irvingoujAtDevolution

Copy link
Copy Markdown
Contributor Author

Lines changed

git diff --numstat origin/master...007357720 (merge base e8a4b082b):

Part Files Added Removed
Production code (ships to users) 0 0 0
Tests: harness (testsuite/src/, without the intent file) 12 1,777 0
Tests: cases (testsuite/tests/) 4 800 50
Other: CI, manifest, lockfile, intent 4 66 4
Total 20 2,643 54

Other is .github/workflows/ci.yml (+16, installs and caches ironrdp-agent 0.1.0), testsuite/Cargo.toml (+12 −4), Cargo.lock (+8, no new package) and testsuite/src/rdp_injection/INTENT.md (+30).

The 50 removed lines are the self-signed localhost certificate and key, moved from testsuite/tests/cli/dgw/tls_anchoring.rs to testsuite/src/tls_fixtures.rs so the new tests share them.

Note

LLM-assisted content (no human feedback).

@irvingoujAtDevolution

Copy link
Copy Markdown
Contributor Author

Dependency pruner

Scope: the testsuite/Cargo.toml and Cargo.lock diff against origin/master.

Result: nothing removed. Every package was already in the testsuite's build graph through the devolutions-gateway dev-dependency:

  • Cargo.lock adds no package, only 11 edges from testsuite.
  • cargo tree -p testsuite -e normal,dev,build: 618 packages with the master manifest and with this PR's manifest, the same set.
  • cargo tree -d over the same graph: the same duplicate-version list (178 lines) before and after.
Dependency Change Used in (testsuite/src/) What it does there Why it stays
ironrdp-pdu 0.9 (std) new rdp_injection/rdp.rs, credssp.rs, mock_rdp.rs PCB v2, X.224 Connection Request and Confirm RDP wire format (MS-RDPBCGR, MS-RDPEPS), not something to hand-roll
ironrdp-core 0.2 (std) new same files encode_vec and decode for those PDUs ironrdp-pdu re-exports it only as a hidden, deprecated path
ironrdp-tokio 0.10 new rdp.rs, mock_rdp.rs, credssp.rs TPKT and TSRequest framing on async streams frame-length parsing
ironrdp-connector 0.10 new credssp.rs, mock_rdp.rs, mock_kdc.rs only its sspi re-export: CredSSP client and server, NTLM, Kerberos see below
kdc 0.1 new mock_kdc.rs a real KDC answering AS-REQ and TGS-REQ Kerberos KDC; the same crate Gateway embeds
picky-krb 0.12 new mock_kdc.rs, credssp.rs KDC proxy messages and AS/TGS types, to record principals Kerberos ASN.1 types
picky-asn1-der 0.5 new mock_kdc.rs DER decoding of those types DER parser; picky-krb does not re-export it
x509-cert 0.3 (std) new rdp_injection/tls.rs reads the subject public key that CredSSP binds to X.509 parsing; same version and features as Gateway
rustls 0.23 dev to normal tls.rs, credssp.rs, mock_rdp.rs TLS for the fake server and the test client moved because the helpers now live in the library
tokio-rustls 0.26 dev to normal same files async TLS same
base64 0.23 dev to normal rdp_injection/tokens.rs base64url for unsigned JWTs encoding; same
tokio feature io-util widened gateway.rs, preflight.rs, rdp.rs, credssp.rs, mock_kdc.rs async read/write helpers, BufReader the library needs it when built on its own

ironrdp-connector is the one with an alternative: a direct sspi dependency would do the same job. It stays because Gateway imports sspi the same way (devolutions-gateway/src/credential_injection.rs:20, use ironrdp_connector::sspi;). Going through the re-export keeps the test's CredSSP on the sspi version Gateway runs, so an IronRDP bump cannot leave two sspi versions in the tree.

On running ironrdp-agent instead of linking IronRDP: all four IronRDP crates are used by the fakes that play the RDP server. FakeRdpTarget (rdp.rs) decodes the X.224 request Gateway forwards, and MockRdp (mock_rdp.rs) runs the target-side CredSSP server. ironrdp-agent is a client, so it cannot stand in for either. Moving the TCP-path client to ironrdp-agent would not drop any crate.

Net change: 8 new direct dependencies, 3 moved from dev to normal, 1 widened tokio feature, 0 new packages.

Test pruner

Scope: the 14 tests in testsuite/tests/cli/dgw/cred_injection.rs and cred_injection_kdc.rs, run as integration_tests cli::dgw::cred_injection against a Gateway binary rebuilt for each mutant.

Result: no deletion and no consolidation. One coverage gap is closed by 007357720, and one stays open (dm-16, below).

Time: about 55 minutes of the one-hour budget, 11:25 to 12:17 EDT, mutation runs included.

Controls

  • Clean runs: 14 of 14 tests pass in four runs on the base and one run on 007357720, so no test is flaky.
  • Positive control pc-01 (Kerberos is never selected) fails 7 tests on the base and on 007357720, so the tests run the patched Gateway.
  • The base is cd0d38451, an earlier local build of this branch. Its Gateway code is the same as at 007357720; only two test helpers differ. MockRdp moved its 127.0.0.1 + ::1 listener into LocalhostListener without changing behavior, and FakeClosedTarget now listens on ::1 too. Only kerberos_missing_krb_kdc_fails_closed uses FakeClosedTarget, so pc-01, dm-01, dm-02 and dm-12 were rerun on 007357720: every mutant that test caught, plus the gap it now closes.

Oracle classification

Every test cites testsuite/src/rdp_injection/INTENT.md (and some devolutions-gateway/src/credential/INTENT.md §Decisions). I checked each citation against the sentence it names. Gateway logs are the observation channel for which path Gateway took (RDP-TLS forwarding with credential injection, Upstream forwarding, kerberos=); the intent decides the expected path, not the log wording.

Oracle table (14 tests: 14 independent, 0 unknown)
Test Class Source that decides it
first_rdp_connection_injects_ntlm independent; cookie assert self-derived (mutant) §1.c, NTLM for a target without a domain
reconnect_same_association_token_still_injects independent; cookie assert self-derived §3; credential Decision 1
expired_staging_uses_ordinary_forward independent §4.a; credential Decision 4. The client cookie is an input the test chose, and ordinary forwarding passes it through
unprovisioned_rdp_uses_ordinary_forward independent §4.a
kerberos_reconnect_reuses_generation_until_reprovision independent; cookie assert self-derived §3; credential Decisions 2 and 5 (one synthetic KDC per JTI, replaced on re-provision)
domainless_target_stays_ntlm_even_with_krb_kdc independent §1.g
kerberos_injection_completes_credssp_against_mock_kdc independent; cookie assert self-derived §2.f (AS-REQ for the target user, TGS-REQ for TERMSRV/<host>), §2.g (target account finishes CredSSP)
kerberos_client_and_target_legs_complete_credssp independent; cookie assert self-derived §2.d (client reaches the KDC proxy, gets AS-REP and TGS-REP for the proxy user), §2.f, §2.g
kerberos_wrong_target_password_fails_closed independent; cookie assert self-derived §4.b; no TGS-REQ without a TGT (RFC 4120 §3.3)
kerberos_kdc_down_fails_closed independent; cookie assert self-derived §4.b
kerberos_missing_krb_kdc_fails_closed independent §4.c (no forwarding, no injection, target never dialled)
ntlm_injection_completes_credssp_both_legs independent; cookie assert self-derived §1.c, §1.f
ironrdp_agent_rdcleanpath_ntlm_injection independent; cookie assert self-derived §1.c, §1.f over RDCleanPath
ironrdp_agent_rdcleanpath_kerberos_injection independent; cookie assert self-derived §2.f, §2.g over RDCleanPath

The only self-derived assertions are the X.224 cookie checks, already marked oracle: mutant.

Mutants

A fresh-context agent wrote dm-01 to dm-16 from the two intent files and the Gateway code, without reading the tests. I wrote pc-01, and ck-01 and rc-01 to check the faults the cookie markers name.

Mutant Fault Tests that fail
pc-01 Kerberos is never selected (positive control) 7
ck-01 TCP path forwards the client X.224 cookie instead of the target username 8
rc-01 RDCleanPath rewrites the X.224 cookie to the target username 2
dm-01 TCP path looks up the mapping by jet_aid instead of jti 10
dm-02 A failed checkout falls back to ordinary forwarding and dials the target 1 (0 on the base)
dm-03 RDCleanPath looks up the mapping by jet_aid instead of jti 2
dm-04 Staging expiry uses the token deadline instead of the provisioning TTL 1
dm-05 The first checkout removes the mapping, so a reconnect finds none 2
dm-06 The first checkout no longer extends retention to the token deadline 0
dm-07 Re-provisioning keeps the old entry 1
dm-08 Re-provisioning keeps the generation, so Kerberos reuses the old synthetic KDC 2
dm-09 Gateway sends the proxy credential to the RDP server 5
dm-10 The synthetic KDC is built from the target credential 1
dm-11 The domain check is dropped; any username selects Kerberos 5
dm-12 A Kerberos target without krb_kdc switches to NTLM 1
dm-13 The target KDC port is forced to 88 4
dm-14 /jet/KdcProxy looks up the KDC by the KDC token's own jti 1
dm-15 Inverted expiry check drops the cached session KDC on every lookup 0
dm-16 A failed target-side CredSSP is logged and ignored 0

Unique kills (a mutant only one test catches):

Test Mutants caught Unique kills
expired_staging_uses_ordinary_forward 1 dm-04
kerberos_reconnect_reuses_generation_until_reprovision 6 dm-07
kerberos_client_and_target_legs_complete_credssp 7 dm-10, dm-14
kerberos_missing_krb_kdc_fails_closed 4 dm-02, dm-12
unprovisioned_rdp_uses_ordinary_forward 0 none
the other 9 tests 3 to 5 each none
Kill matrix: each failing test and the first check that failed (53 kills)
Mutant Test First failing check
pc-01 kerberos_reconnect_reuses_generation_until_reprovision assert at cred_injection.rs:175
pc-01 ironrdp_agent_rdcleanpath_kerberos_injection RDCleanPath Kerberos target CredSSP
pc-01 kerberos_client_and_target_legs_complete_credssp client-leg Kerberos CredSSP
pc-01 kerberos_injection_completes_credssp_against_mock_kdc Gateway-facing NTLM CredSSP
pc-01 kerberos_kdc_down_fails_closed KDC down must still start Kerberos injection
pc-01 kerberos_missing_krb_kdc_fails_closed timed out waiting for 1 occurrence(s) of "missing or expired; re-provision to retry"
pc-01 kerberos_wrong_target_password_fails_closed wrong password must still start Kerberos injection
ck-01 first_rdp_connection_injects_ntlm assert at cred_injection.rs:42
ck-01 kerberos_reconnect_reuses_generation_until_reprovision assert at cred_injection.rs:223
ck-01 reconnect_same_association_token_still_injects assert at cred_injection.rs:83
ck-01 kerberos_client_and_target_legs_complete_credssp RDP X.224 cookie must be administrator@example.invalid; cookies=["client-cookie-user"]
ck-01 kerberos_injection_completes_credssp_against_mock_kdc RDP X.224 cookie must be administrator@example.invalid; cookies=["client-cookie-user"]
ck-01 kerberos_kdc_down_fails_closed KDC down still rewrites the X.224 cookie; cookies=["client-cookie-user"]
ck-01 kerberos_wrong_target_password_fails_closed wrong password still rewrites the X.224 cookie; cookies=["client-cookie-user"]
ck-01 ntlm_injection_completes_credssp_both_legs RDP X.224 cookie must be injected-target-user; cookies=["client-cookie-user"]
rc-01 ironrdp_agent_rdcleanpath_kerberos_injection RDCleanPath forwards the client X.224 cookie; cookies=["administrator@example.invalid"]
rc-01 ironrdp_agent_rdcleanpath_ntlm_injection RDCleanPath forwards the client X.224 cookie; cookies=["injected-target-user"]
dm-01 domainless_target_stays_ntlm_even_with_krb_kdc timed out waiting for 1 occurrence(s) of "RDP-TLS forwarding with credential injection"
dm-01 first_rdp_connection_injects_ntlm timed out waiting for 1 occurrence(s) of "RDP-TLS forwarding with credential injection"
dm-01 kerberos_reconnect_reuses_generation_until_reprovision timed out waiting for 1 occurrence(s) of "RDP-TLS forwarding with credential injection"
dm-01 reconnect_same_association_token_still_injects timed out waiting for 1 occurrence(s) of "RDP-TLS forwarding with credential injection"
dm-01 kerberos_client_and_target_legs_complete_credssp client-leg Kerberos CredSSP
dm-01 kerberos_injection_completes_credssp_against_mock_kdc Gateway-facing NTLM CredSSP
dm-01 kerberos_kdc_down_fails_closed timed out waiting for 1 occurrence(s) of "RDP-TLS forwarding with credential injection"
dm-01 kerberos_missing_krb_kdc_fails_closed timed out waiting for 1 occurrence(s) of "missing or expired; re-provision to retry"
dm-01 kerberos_wrong_target_password_fails_closed timed out waiting for 1 occurrence(s) of "RDP-TLS forwarding with credential injection"
dm-01 ntlm_injection_completes_credssp_both_legs client-leg NTLM CredSSP
dm-02 kerberos_missing_krb_kdc_fails_closed missing krb_kdc must fail closed
dm-03 ironrdp_agent_rdcleanpath_kerberos_injection RDCleanPath Kerberos target CredSSP
dm-03 ironrdp_agent_rdcleanpath_ntlm_injection RDCleanPath NTLM target CredSSP
dm-04 expired_staging_uses_ordinary_forward timed out waiting for 1 occurrence(s) of "Upstream forwarding"
dm-05 kerberos_reconnect_reuses_generation_until_reprovision timed out waiting for 2 occurrence(s) of "RDP-TLS forwarding with credential injection"
dm-05 reconnect_same_association_token_still_injects timed out waiting for 2 occurrence(s) of "RDP-TLS forwarding with credential injection"
dm-07 kerberos_reconnect_reuses_generation_until_reprovision timed out waiting for 2 occurrence(s) of "Published synthetic KDC"
dm-08 domainless_target_stays_ntlm_even_with_krb_kdc preflight HTTP status was ""
dm-08 kerberos_reconnect_reuses_generation_until_reprovision timed out waiting for 2 occurrence(s) of "Published synthetic KDC"
dm-09 ironrdp_agent_rdcleanpath_kerberos_injection RDCleanPath Kerberos target CredSSP
dm-09 ironrdp_agent_rdcleanpath_ntlm_injection RDCleanPath NTLM target CredSSP
dm-09 kerberos_client_and_target_legs_complete_credssp target-leg Kerberos CredSSP
dm-09 kerberos_injection_completes_credssp_against_mock_kdc timed out waiting for Kerberos CredSSP on mock RDP
dm-09 ntlm_injection_completes_credssp_both_legs target-leg NTLM CredSSP
dm-10 kerberos_client_and_target_legs_complete_credssp client-leg Kerberos CredSSP
dm-11 domainless_target_stays_ntlm_even_with_krb_kdc assert at cred_injection.rs:257
dm-11 first_rdp_connection_injects_ntlm timed out waiting for 1 occurrence(s) of "RDP-TLS forwarding with credential injection"
dm-11 reconnect_same_association_token_still_injects timed out waiting for 1 occurrence(s) of "RDP-TLS forwarding with credential injection"
dm-11 ironrdp_agent_rdcleanpath_ntlm_injection RDCleanPath NTLM target CredSSP
dm-11 ntlm_injection_completes_credssp_both_legs read X.224 CC
dm-12 kerberos_missing_krb_kdc_fails_closed timed out waiting for 1 occurrence(s) of "missing or expired; re-provision to retry"
dm-13 ironrdp_agent_rdcleanpath_kerberos_injection RDCleanPath Kerberos target CredSSP
dm-13 kerberos_client_and_target_legs_complete_credssp target-leg Kerberos CredSSP
dm-13 kerberos_injection_completes_credssp_against_mock_kdc timed out waiting for Kerberos CredSSP on mock RDP
dm-13 kerberos_kdc_down_fails_closed Gateway never TCP-connected the provisioned KDC
dm-14 kerberos_client_and_target_legs_complete_credssp client-leg Kerberos CredSSP

Survivors

Mutant Triage Evidence
dm-02 coverage gap, closed by 007357720 §4.c: "must not connect to the RDP server". The test reaches the target as localhost. utils::tcp_connect tries the resolved addresses in order, and Windows resolves localhost to ::1 first. The fake target listened only on 127.0.0.1, and a refused ::1 connect takes about 2 seconds on Windows (2057 ms measured), longer than the test's 250 ms window. With ::1 also listening, dm-02 fails kerberos_missing_krb_kdc_fails_closed at "missing krb_kdc must fail closed".
dm-06 caught outside this scope The unit test provisioning::tests::first_get_extends_expiry_to_token_deadline fails at provisioning.rs:583 with the mutant and passes without it.
dm-15 caught outside this scope The unit tests credential_injection::tests::checkout_reuses_synthetic_kdc_for_the_same_generation (credential_injection.rs:1018) and checkout_reuses_kerberos_session_after_connection_options_expire (:1060) fail with the mutant and pass without it. The process-level test counts Published synthetic KDC, which only the live registry logs, so it cannot see the session cache being rebuilt.
dm-16 coverage gap, open §4.b: a wrong target password or an unreachable KDC "fails at 2.e". With the mutant, Gateway logs the failed target leg and carries on with the connection. All 14 tests and all 275 Gateway unit tests still pass. The two §4.b tests discard the client result (let _ = complete_ntlm_credssp(tls).await), and their Upstream forwarding check only covers the other path, since generic_client.rs writes that log only when it does not inject. So nothing observes what Gateway does after the target leg fails.

dm-16 is not closed because the budget ran out. A gap test needs a signal from outside Gateway that it stopped at 2.e, for example the client connection closing soon after the target leg fails. I have not checked whether that signal differs under the mutant.

Decisions

  • No deletion. All 14 tests are independent, and an independent test is never deleted on mutant evidence. That includes unprovisioned_rdp_uses_ordinary_forward, which no mutant of this pass reaches (§4.a).
  • The cookie assertions are self-derived and keep their oracle: mutant markers. They fail under ck-01 in 8 TCP-path tests and under rc-01 in both RDCleanPath tests, so none is a deletion candidate.
  • No consolidation. The two RDCleanPath tests and the two §4.b tests differ in setup and in what they check, not only in data. For example, the Kerberos RDCleanPath test starts a mock KDC and checks its AS and TGS requests. The two §4.a tests (expired_staging_uses_ordinary_forward and unprovisioned_rdp_uses_ordinary_forward) check the same outcome and differ only in whether credentials were provisioned and allowed to expire, so they could become two cases of one test. I did not merge them before the budget ran out.
  • Committed: 007357720 (dm-02 gap). The fake target that proves Gateway never dials now listens on ::1 as well.

Not covered

  • cargo-mutants did not run. It lists 227 mutants for the seven Gateway files under test. Each one rebuilds Gateway (28 to 68 s here), so a --jobs 1 run would take about 3 hours.
  • The dm-02, dm-06, dm-15 and dm-16 unit-suite checks ran on the base. The Gateway code is the same at 007357720.

Note

LLM-assisted content (no human feedback).

@irvingoujAtDevolution

Copy link
Copy Markdown
Contributor Author

Intent drafts: handoff

testsuite/src/rdp_injection/INTENT.md as of c47e05c44; the file is unchanged after it. The questions behind it were asked against the tree at 02e370526, the parent of c47e05c44.

Provenance

Level 1 rests on git history: 3ed859808 added the file empty as a placeholder "to be filled by hand", 09e58fb9e filled it with no co-author trailer, and its text kept the original typos. c47e05c44 changed only the headings and added 1.g.

# Commitment as written Human's words (if tidied) Level
1.a Preflight send a NTLM credential injection request to the Gateway 1
1.b RDP client will send association token with matching JTI in the credential injection request to Gateway. 1
1.c Gateway will correctly recognize this connetion is intended for credential injection 1
1.d RDP client will proceed with TLS + CredSSP 1
1.e Gateway will intercept the CredSSP and also create CredSSP request as client to RDP server. 1
1.f We assert on that the RDP server receives the target credentials while the RDP client only sends the proxy credentials 1
1.g A target username without a domain uses NTLM, even when the preflight request includes KDC information. 3 (Irving picked this sentence from options the agent wrote)
2.a Preflight send a Kerberos credential injection request to the Gateway including the KDC information. 1
2.b RDP client will send association token with matching JTI in the credential injection request to Gateway. 1
2.c Gateway will correctly recognize this connetion is intended for credential injection 1
2.d RDP client will proceed with TLS + CredSSP, here it will also call KDC Proxy (which is also Gateway) 1
2.e Gateway will intercept the CredSSP and also create CredSSP request as client to RDP server. 1
2.f Gateway will forward the Kerberos request to the KDC with state driven by the target credentials, and we will assert on that the KDC receives the correct request. 1
2.g We assert on that the RDP server receives the target credentials while the RDP client only sends the proxy credentials 1
3 Reconnect stability, make sure we could reconnect with the same association token. 1
4.a Missing or expired provisioning fails at 1.c or 2.c and uses ordinary forwarding. 1
4.b An incorrect target password or unavailable KDC fails at 2.e and must not fall back to ordinary forwarding. 1
4.c Missing KDC information fails before 2.e and must not connect to the RDP server. 1

Left out on purpose

  • The X.224 mstshash cookie: no sentence says what the cookie must be on either path. The cookie asserts carry oracle: mutant markers instead.
  • Kerberos injection without debug flags: no sentence. Irving dropped the test that checked it, since the other Kerberos tests already cover the path.

Placement

  • Everything is in testsuite/src/rdp_injection/INTENT.md. No other directory shares a part file.
  • Irving chose to turn the four scenarios into ### headings under ## Scenarios and to fix the heading typo Senarios, so test markers can cite §1 to §4. The scenario text itself did not change.

Pending experiments

None.

Note

LLM-assisted content (no human feedback).

Comment thread testsuite/Cargo.toml Resolved
Add process-level tests for DVLS-like preflight, first inject,
jet_reuse reconnect, fail-closed missing mappings, and synthetic
KDC generation reuse.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Clippy separated_literal_suffix failed CI lints on the stacked
reconnect tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Drive CredSSP through a TCP kdc crate and IronRDP acceptor so
target-leg Kerberos injection is proven, not just log-matched.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Prove both CredSSP hops against mock KDC/RDP, NTLM CredSSP both
legs, and Kerberos fail-closed when the password, KDC, or krb_kdc
is wrong. Token-cache jet_reuse still needs signed JWTs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Decode AS-REQ/TGS-REQ on the mock KDC (cname, realm, TERMSRV/localhost),
record CredSSP Finished account names and X.224 cookies, and require
/jet/KdcProxy AS-REP plus TGS-REP.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Routing tests now decode X.224 Cookie instead of raw-byte search.
Client-leg Kerberos asserts synthetic-KDC AS-REQ cname. Fail-closed
Kerberos paths require inject-started then no Finished identity.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Decode X.224 cookies, KdcProxy AS/TGS principals, and attribute
KDC-down TCP to Gateway via a separate refusing listener.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Checkout-before-connect on the stack below lets the fail-closed path
prove accepted==0.

Issue: DGW-1900

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pin the public CLI release and complete NTLM and Kerberos target
CredSSP over ws://127.0.0.1/jet/rdp.

Issue: DGW-1900

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Read framed X.224 cookies, bound CredSSP handshakes, and share
the localhost TLS fixture.

Issue: DVLS-14697

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep credential-injection tests focused on public behavior instead of
serializing the removed Kerberos debug option.

Issue: DVLS-14697

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Treat an expired staging mapping as absent because association tokens do
not carry an injection requirement. Verify ordinary forwarding preserves
the client X.224 cookie.

Issue: DVLS-14697

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use the direct rustls dependency added on master so the credential-injection
E2E compiles under the workspace unused-qualifications lint.

Issue: DVLS-14697

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move GatewayProc, provisioning, fake RDP target, mock KDC/RDP, CredSSP
client leg, agent harness, and TLS fixtures from the test files into
testsuite/src so the abstractions are self-contained, per review.

Add empty INTENT.md placeholders to be filled by hand.

Issue: DVLS-14697
Record the intended NTLM, Kerberos, reconnect, and failure scenarios.

Name test tokens and identifiers by their protocol roles.
The behavioral contract already lives in
devolutions-gateway/src/credential/INTENT.md; the suite-level
INTENT.md covers the test scenarios, so the per-file placeholders
are unnecessary.

Issue: DVLS-14697
The RDCleanPath injection tests are the only coverage of that path;
skipping them when the binary is missing silently drops it. Missing
binary is now a failure, with IRONRDP_AGENT_SKIP=1 as explicit
opt-out. CI installs a pinned ironrdp-agent (cached per version).

Issue: DVLS-14697
Any Gateway test can drive a real RDP client, not just credential
injection, so the harness lives at testsuite::ironrdp_agent and
connect_rdcleanpath takes the credentials as parameters.

Issue: DVLS-14697
Issue: DVLS-14697

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cite the injection intent for each test so the oracle marker check
passes. Split the intent's scenarios into headings so markers can
cite them, and state that a domainless target user stays on NTLM.

Mark the X.224 cookie asserts as mutants, since no intent decides
the cookie. Drop the debug-flag test and the KDC registration count,
which repeat what the other Kerberos asserts already check.

Issue: DVLS-14697

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A wildcard listener makes Windows Defender Firewall ask for elevation
once per test binary path. Listen on 127.0.0.1 and ::1 at the same
port instead, so `localhost` still reaches the server on Windows,
where it resolves to ::1 first.

Issue: DVLS-14697

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The missing `krb_kdc` test checks that Gateway never dials the target,
which it reaches as `localhost`. On Windows, Gateway tries ::1 first,
so a dial to a target listening only on 127.0.0.1 waits for the refusal
and lands after the check. A regression that fell back to ordinary
forwarding passed. The target now listens on both loopback addresses.

Issue: DVLS-14697

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fake RDP target, the CredSSP server, the KDCs and the CredSSP client
now live in crates/credential-injection-mock, one role per process.
Each role binds loopback and prints JSON event lines, and the testsuite
spawns it and reads those events.

This keeps the RDP and Kerberos crates out of the testsuite manifest.
ironrdp-agent cannot replace the client side: its direct TCP path sends
no preconnection blob, which the Gateway TCP listener requires.

Issue: DVLS-14697

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Record why the mock is its own program, why it also holds the RDP
client, and that sharing sspi and the kdc crate with Gateway is
accepted.

Issue: DVLS-14697

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants