Skip to content

fix: retry LookupAccountName with exponential backoff to prevent ERROR_NONE_MAPPED on fresh EC2 instances - #262

Merged
leongdl merged 5 commits into
OpenJobDescription:mainlinefrom
leon-li-inspire:fix/named-pipe-system-sid-lookup
Jun 12, 2026
Merged

leongdl merged 5 commits into
OpenJobDescription:mainlinefrom
leon-li-inspire:fix/named-pipe-system-sid-lookup

Conversation

@leon-li-inspire

@leon-li-inspire leon-li-inspire commented Jun 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • create_security_attributes() calls win32security.LookupAccountName("", username) to resolve the current user's SID for named pipe access control
  • This fails intermittently with pywintypes.error: (1332, 'LookupAccountName', 'No mapping between account names and security IDs was done.') on freshly started EC2 instances where the LSA (Local Security Authority) service has not finished initializing
  • The failure is intermittent — re-runs pass because LSA is warm by then
  • Add retry with exponential backoff (up to 3 attempts, sleeping 1s then 2s) on error 1332, then re-raise if all attempts are exhausted

Affected repos

Fixes intermittent Windows integration test failures in all DCC repos that use LOGNAME=SYSTEM in their CodeBuild Windows environment: Cinema 4D, Blender, Houdini, Maya, 3ds Max, VRED, and Unreal Engine.

Related ticket: https://t.corp.amazon.com/P447504169

Test plan

  • test_create_security_attributes_uses_lookup: verifies LookupAccountName is called with the current username
  • test_create_security_attributes_retries_on_lsa_not_ready: verifies retry succeeds after a transient 1332 error, with correct backoff sleep
  • test_create_security_attributes_raises_after_max_retries: verifies the original error is re-raised after exhausting all retries

@leon-li-inspire
leon-li-inspire requested a review from a team as a code owner June 9, 2026 05:18
…ttributes

When CodeBuild Windows integration tests run with LOGNAME=SYSTEM, getpass.getuser()
returns "SYSTEM" and the subsequent LookupAccountName("", "SYSTEM") call fails with
ERROR_NONE_MAPPED (1332) on freshly started reserved-fleet EC2 instances because the
LSA service hasn't finished initializing.

SYSTEM (S-1-5-18) is a well-known SID constant that never varies across Windows
machines. Use ConvertStringSidToSid("S-1-5-18") to bypass the LSA lookup entirely,
consistent with how the network SID (S-1-5-2) is already handled in the same method.

Fixes intermittent Windows integration test failures in deadline-cloud-for-cinema-4d
and 6 other DCC repos that share the LOGNAME=SYSTEM CodeBuild workaround.

Signed-off-by: leon-li-inspire <2182521+leon-li-inspire@users.noreply.github.com>
@leon-li-inspire
leon-li-inspire force-pushed the fix/named-pipe-system-sid-lookup branch from bc7b020 to 1dbca63 Compare June 9, 2026 05:24
Comment thread CHANGELOG.md Resolved
crowecawcaw
crowecawcaw previously approved these changes Jun 10, 2026
@crowecawcaw
crowecawcaw enabled auto-merge (squash) June 10, 2026 19:29
@crowecawcaw
crowecawcaw disabled auto-merge June 10, 2026 19:29
@crowecawcaw
crowecawcaw enabled auto-merge (squash) June 10, 2026 19:30
Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
auto-merge was automatically disabled June 10, 2026 20:32

Head branch was pushed to by a user without write access

@leon-li-inspire

Copy link
Copy Markdown
Contributor Author

Pushed a formatting fix (commit 1735af2) — black was flagging 3 long lines in the test file. The new workflow runs need maintainer approval to execute. Could someone please approve the pending CI runs?

LookupAccountName mock must return a real PySID object (not a string)
because downstream code passes it to dacl.AddAccessAllowedAce() which
validates the type on Windows.

Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
@leon-li-inspire

Copy link
Copy Markdown
Contributor Author

Pushed another fix (commit 617a7c9) — the Windows tests were failing because the mock returned a string "fake_sid" instead of a real PySID object. The code downstream calls dacl.AddAccessAllowedAce(..., user_sid) which validates the type. Fixed by using win32security.ConvertStringSidToSid("S-1-1-0") to produce a real PySID.

The CI runs need maintainer approval again — could someone please approve the pending workflow runs?

crowecawcaw
crowecawcaw previously approved these changes Jun 10, 2026
@crowecawcaw
crowecawcaw enabled auto-merge (squash) June 10, 2026 21:17
Comment thread CHANGELOG.md Resolved
Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
auto-merge was automatically disabled June 10, 2026 21:55

Head branch was pushed to by a user without write access

@leon-li-inspire
leon-li-inspire requested a review from leongdl June 10, 2026 22:14
):
named_pipe_helper.NamedPipeHelper.generate_pipe_name("AdaptorTest")

@patch("getpass.getuser", return_value="regularuser")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we will need to port this to openjd-rs too!

@leongdl
leongdl merged commit 0c85c36 into OpenJobDescription:mainline Jun 12, 2026
43 checks passed
leon-li-inspire added a commit to leon-li-inspire/openjd-rs that referenced this pull request Jun 21, 2026
…MAPPED

LookupAccountNameW fails intermittently with ERROR_NONE_MAPPED (1332) on
freshly started EC2 instances where the LSA service hasn't finished
initializing. Add retry with exponential backoff (up to 3 attempts,
sleeping 1s then 2s) to the lookup_sid() function, then re-raise if all
attempts are exhausted.

This is the Rust equivalent of the Python fix in
OpenJobDescription/openjd-adaptor-runtime-for-python#262 which addresses
the same intermittent failure in Windows integration tests across all DCC
repos that use LOGNAME=SYSTEM in their CodeBuild Windows environment.

Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
mwiebe pushed a commit to leon-li-inspire/openjd-rs that referenced this pull request Jun 30, 2026
…MAPPED

LookupAccountNameW fails intermittently with ERROR_NONE_MAPPED (1332) on
freshly started EC2 instances where the LSA service hasn't finished
initializing. Add retry with exponential backoff (up to 3 attempts,
sleeping 1s then 2s) to the lookup_sid() function, then re-raise if all
attempts are exhausted.

This is the Rust equivalent of the Python fix in
OpenJobDescription/openjd-adaptor-runtime-for-python#262 which addresses
the same intermittent failure in Windows integration tests across all DCC
repos that use LOGNAME=SYSTEM in their CodeBuild Windows environment.

Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
mwiebe pushed a commit to leon-li-inspire/openjd-rs that referenced this pull request Jul 2, 2026
…MAPPED

LookupAccountNameW fails intermittently with ERROR_NONE_MAPPED (1332) on
freshly started EC2 instances where the LSA service hasn't finished
initializing. Add retry with exponential backoff (up to 3 attempts,
sleeping 1s then 2s) to the lookup_sid() function, then re-raise if all
attempts are exhausted.

This is the Rust equivalent of the Python fix in
OpenJobDescription/openjd-adaptor-runtime-for-python#262 which addresses
the same intermittent failure in Windows integration tests across all DCC
repos that use LOGNAME=SYSTEM in their CodeBuild Windows environment.

Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
jericht pushed a commit to leon-li-inspire/openjd-rs that referenced this pull request Jul 2, 2026
…MAPPED

LookupAccountNameW fails intermittently with ERROR_NONE_MAPPED (1332) on
freshly started EC2 instances where the LSA service hasn't finished
initializing. Add retry with exponential backoff (up to 3 attempts,
sleeping 1s then 2s) to the lookup_sid() function, then re-raise if all
attempts are exhausted.

This is the Rust equivalent of the Python fix in
OpenJobDescription/openjd-adaptor-runtime-for-python#262 which addresses
the same intermittent failure in Windows integration tests across all DCC
repos that use LOGNAME=SYSTEM in their CodeBuild Windows environment.

Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
jericht pushed a commit to OpenJobDescription/openjd-rs that referenced this pull request Jul 2, 2026
…MAPPED (#221)

* fix: retry LookupAccountNameW with exponential backoff on ERROR_NONE_MAPPED

LookupAccountNameW fails intermittently with ERROR_NONE_MAPPED (1332) on
freshly started EC2 instances where the LSA service hasn't finished
initializing. Add retry with exponential backoff (up to 3 attempts,
sleeping 1s then 2s) to the lookup_sid() function, then re-raise if all
attempts are exhausted.

This is the Rust equivalent of the Python fix in
OpenJobDescription/openjd-adaptor-runtime-for-python#262 which addresses
the same intermittent failure in Windows integration tests across all DCC
repos that use LOGNAME=SYSTEM in their CodeBuild Windows environment.

Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>

* fix: gate LookupAccountNameW retry on LSA-not-ready to avoid penalizing unmapped names

ERROR_NONE_MAPPED (1332) is overloaded: Windows returns it both when the LSA
service is still initializing (transient, only at instance startup) and when a
name genuinely does not map to an account (permanent). The previous retry fired
on every 1332, so steady-state lookups of a non-existent principal — notably
the WindowsSessionUser::is_process_user fallback path — blocked for ~3s before
returning, a regression for a path that previously returned immediately.

Gate the backoff on a process-global LSA-availability check: once any lookup
succeeds (or a probe of the current process user resolves), the LSA is known to
be up and a subsequent 1332 must mean the name is unmapped, so it fails fast.
The LSA never restarts within a process lifetime, so the result is cached.

Update test_lookup_sid_fails_for_nonexistent_user accordingly: it now asserts
the unmappable-name lookup fails fast (<1s) once LSA is confirmed up, instead of
asserting the old >=3s backoff.

Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>

---------

Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
leongdl added a commit that referenced this pull request Nov 4, 2026
* fix: take the named pipe user SID from the process token

NamedPipeHelper.create_security_attributes() resolved the DACL's user SID
with LookupAccountName("", getpass.getuser()). getpass.getuser() only reads
the LOGNAME, USER, LNAME and USERNAME environment variables, and the name
lookup goes through LSA, which can fail with ERROR_NONE_MAPPED (1332). A
Windows CI host running as SYSTEM hit 1332 on both the server path
(create_named_pipe_server) and the client path
(establish_named_pipe_connection), and the 3-attempt retry from #262 ran
out.

- Read the SID from the process token: OpenProcessToken(GetCurrentProcess(),
  TOKEN_QUERY) and GetTokenInformation(token, TokenUser)[0], closing the
  token in a finally. No name lookup, so 1332 cannot occur. The #262 retry
  and the getpass import are removed.
- The client no longer builds a DACL. CreateFile ignores the security
  descriptor when opening an existing pipe and reads only bInheritHandle. It
  passes SECURITY_ATTRIBUTES with SECURITY_DESCRIPTOR = None and
  bInheritHandle = True. pywin32's SECURITY_ATTRIBUTES() defaults
  bInheritHandle to TRUE (PySECURITY_ATTRIBUTES.cpp), so the client handle
  stays inheritable, as before. Passing None would have made it
  non-inheritable.
- The server DACL is unchanged: GENERIC_READ|GENERIC_WRITE allowed to the
  user SID, then GENERIC_ALL denied to S-1-5-2 (NETWORK).

Tests:
- Unit: the token SID ends up in the DACL; with the env user name set to
  an unmapped name, LookupAccountName is not called; the token is closed
  when GetTokenInformation raises; the client passes no descriptor, an
  inheritable handle and OPEN_EXISTING, and does not call
  create_security_attributes. The three LookupAccountName/retry tests
  pinned the removed behaviour and are replaced.
- Integ: test_pipe_security_does_not_depend_on_user_name runs a real pipe
  round-trip with LOGNAME, USER, LNAME and USERNAME set to
  no-such-user-1332, once for server and client and once for the client
  only, plus an ambient-environment control. It asserts the precondition
  that LookupAccountName("", "no-such-user-1332") raises 1332, and reads the
  DACL back with GetSecurityInfo. The old code fails this test
  deterministically.
- CI: code_quality.yml runs windows-latest x Python 3.9-3.14 through
  OpenJobDescription/.github reusable_python_build.yml, whose last step
  (line 72) is `hatch run test`, which is `pytest ... test -vv`. That
  collects test/openjd/adaptor_runtime/integ/_named_pipe/. No marker gates
  the new test.

Ran on macOS (Python 3.13, hatch 1.14.1):
- hatch run lint, before and after: ruff "All checks passed!", black
  "146 files would be left unchanged", mypy "no issues found in 146
  source files".
- hatch run mypy --platform win32 src test, before and after: no issues
  in 146 source files.
- hatch run test: before (mainline d9044ea) 482 passed, 26 skipped,
  coverage 93.39%. After: 482 passed, 29 skipped, coverage 93.39%. The 3
  new skips are the 3 new integ cases. The Windows-only unit module is
  one module-level skip before and after.
- Windows tests skip on macOS. To run them anyway, a fake pywin32 plugin
  (not committed) models the APIs these tests touch, using the pywin32 C
  sources. Fidelity control: mainline code with mainline tests passes on
  it (11 passed). Branch: 15 passed, 1 skipped (the GITHUB_ACTIONS-only
  different-user test).

Mutation check, EXECUTED on the fake (sha256 restore, __pycache__
cleared between mutants), all caught:
- M1 restore LookupAccountName(getpass.getuser()) plus retry: fails
  test_create_security_attributes_uses_process_token_user,
  ..._does_not_look_up_user_name, ..._closes_token_when_query_fails and
  integ [unmapped-user-name-for-server-and-client].
- M2 restore the client DACL: fails
  test_establish_named_pipe_connection_builds_no_dacl.
- M3 original code (M1+M2): also fails integ
  [unmapped-user-name-for-client].
- M4 no finally, M5 no close, M6 client not inheritable, M7 client None,
  M8 client keeps blank descriptor, M9 allow Everyone, M10 drop network
  deny: each caught. M4-M8 by the token-close or client unit test, M9
  and M10 also by all three integ cases.
ARGUED for real Windows, not run: with the old code, LookupAccountName
raises 1332 on every attempt (the test asserts this as a precondition),
so both unmapped integ cases fail. M2 alone is equivalent at the OS level,
because CreateFile ignores the descriptor. Only the unit test catches it.

Auditors (three, independent; each returned PASS):
- Pin/mutation: 27 of their own mutants, 26 killed. The survivor removes
  the bInheritHandle = True line, which is equivalent because pywin32
  defaults it to TRUE. Kept the explicit line for the reader. The
  CloseHandle-call assertion rejects an equivalent token.Close(). Kept,
  since it matches the repo's mock style.
- Gap analysis: 9 mutants, all killed. Pre-existing:
  test_fail_to_connect_to_named_pipe_with_different_user has its assert
  inside pytest.raises, so the assert never runs. Not in this diff, and
  tightening it to winerror 5 would gate CI on a claim nobody has run.
  Recorded as a follow-up. _secure_open.py has the same LookupAccountName
  pattern; follow-up.
- Craft/cross-platform: lint, win32 mypy and collection green; pywin32
  API shapes checked against source. Fixed: the client-only case now sets
  the unmapped name inside the try, so a failed precondition still closes
  the server handle. Declined: cross-checking the SID with an LSA name
  lookup. It would bring back the lookup this fix removes. The test
  helper hard-codes TokenUser independently, and a TokenOwner mutant is
  killed.

Follow-up, out of scope: openjd-sessions-for-python _get_process_user_sid()
uses the same name-lookup pattern.

Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>

* test: expect the deny-first ACE order on the named pipe DACL

Windows CI showed the pipe DACL reads back with the NETWORK deny ACE
before the user allow ACE, both in the in-memory security attributes
and on the created pipe. That is canonical order, and the order that
makes the deny apply to the user's own network logons. The new tests
asserted allow-first and failed on all six Windows jobs; production
code was not involved, since mainline builds the same DACL.

Assert the canonical order exactly, and find the allowed SID by ACE
type rather than by index.

Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>

---------

Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants