Repository navigation
fix: retry LookupAccountName with exponential backoff to prevent ERROR_NONE_MAPPED on fresh EC2 instances - #262
Conversation
…ttributes
When CodeBuild Windows integration tests run with LOGNAME=SYSTEM, getpass.getuser()
returns "SYSTEM" and the subsequent LookupAccountName("", "SYSTEM") call fails with
ERROR_NONE_MAPPED (1332) on freshly started reserved-fleet EC2 instances because the
LSA service hasn't finished initializing.
SYSTEM (S-1-5-18) is a well-known SID constant that never varies across Windows
machines. Use ConvertStringSidToSid("S-1-5-18") to bypass the LSA lookup entirely,
consistent with how the network SID (S-1-5-2) is already handled in the same method.
Fixes intermittent Windows integration test failures in deadline-cloud-for-cinema-4d
and 6 other DCC repos that share the LOGNAME=SYSTEM CodeBuild workaround.
Signed-off-by: leon-li-inspire <2182521+leon-li-inspire@users.noreply.github.com>
bc7b020 to
1dbca63
Compare
Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
Head branch was pushed to by a user without write access
|
Pushed a formatting fix (commit 1735af2) — |
LookupAccountName mock must return a real PySID object (not a string) because downstream code passes it to dacl.AddAccessAllowedAce() which validates the type on Windows. Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
|
Pushed another fix (commit 617a7c9) — the Windows tests were failing because the mock returned a string The CI runs need maintainer approval again — could someone please approve the pending workflow runs? |
Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
Head branch was pushed to by a user without write access
| ): | ||
| named_pipe_helper.NamedPipeHelper.generate_pipe_name("AdaptorTest") | ||
|
|
||
| @patch("getpass.getuser", return_value="regularuser") |
There was a problem hiding this comment.
we will need to port this to openjd-rs too!
…MAPPED LookupAccountNameW fails intermittently with ERROR_NONE_MAPPED (1332) on freshly started EC2 instances where the LSA service hasn't finished initializing. Add retry with exponential backoff (up to 3 attempts, sleeping 1s then 2s) to the lookup_sid() function, then re-raise if all attempts are exhausted. This is the Rust equivalent of the Python fix in OpenJobDescription/openjd-adaptor-runtime-for-python#262 which addresses the same intermittent failure in Windows integration tests across all DCC repos that use LOGNAME=SYSTEM in their CodeBuild Windows environment. Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
…MAPPED LookupAccountNameW fails intermittently with ERROR_NONE_MAPPED (1332) on freshly started EC2 instances where the LSA service hasn't finished initializing. Add retry with exponential backoff (up to 3 attempts, sleeping 1s then 2s) to the lookup_sid() function, then re-raise if all attempts are exhausted. This is the Rust equivalent of the Python fix in OpenJobDescription/openjd-adaptor-runtime-for-python#262 which addresses the same intermittent failure in Windows integration tests across all DCC repos that use LOGNAME=SYSTEM in their CodeBuild Windows environment. Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
…MAPPED LookupAccountNameW fails intermittently with ERROR_NONE_MAPPED (1332) on freshly started EC2 instances where the LSA service hasn't finished initializing. Add retry with exponential backoff (up to 3 attempts, sleeping 1s then 2s) to the lookup_sid() function, then re-raise if all attempts are exhausted. This is the Rust equivalent of the Python fix in OpenJobDescription/openjd-adaptor-runtime-for-python#262 which addresses the same intermittent failure in Windows integration tests across all DCC repos that use LOGNAME=SYSTEM in their CodeBuild Windows environment. Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
…MAPPED LookupAccountNameW fails intermittently with ERROR_NONE_MAPPED (1332) on freshly started EC2 instances where the LSA service hasn't finished initializing. Add retry with exponential backoff (up to 3 attempts, sleeping 1s then 2s) to the lookup_sid() function, then re-raise if all attempts are exhausted. This is the Rust equivalent of the Python fix in OpenJobDescription/openjd-adaptor-runtime-for-python#262 which addresses the same intermittent failure in Windows integration tests across all DCC repos that use LOGNAME=SYSTEM in their CodeBuild Windows environment. Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
…MAPPED (#221) * fix: retry LookupAccountNameW with exponential backoff on ERROR_NONE_MAPPED LookupAccountNameW fails intermittently with ERROR_NONE_MAPPED (1332) on freshly started EC2 instances where the LSA service hasn't finished initializing. Add retry with exponential backoff (up to 3 attempts, sleeping 1s then 2s) to the lookup_sid() function, then re-raise if all attempts are exhausted. This is the Rust equivalent of the Python fix in OpenJobDescription/openjd-adaptor-runtime-for-python#262 which addresses the same intermittent failure in Windows integration tests across all DCC repos that use LOGNAME=SYSTEM in their CodeBuild Windows environment. Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com> * fix: gate LookupAccountNameW retry on LSA-not-ready to avoid penalizing unmapped names ERROR_NONE_MAPPED (1332) is overloaded: Windows returns it both when the LSA service is still initializing (transient, only at instance startup) and when a name genuinely does not map to an account (permanent). The previous retry fired on every 1332, so steady-state lookups of a non-existent principal — notably the WindowsSessionUser::is_process_user fallback path — blocked for ~3s before returning, a regression for a path that previously returned immediately. Gate the backoff on a process-global LSA-availability check: once any lookup succeeds (or a probe of the current process user resolves), the LSA is known to be up and a subsequent 1332 must mean the name is unmapped, so it fails fast. The LSA never restarts within a process lifetime, so the result is cached. Update test_lookup_sid_fails_for_nonexistent_user accordingly: it now asserts the unmappable-name lookup fails fast (<1s) once LSA is confirmed up, instead of asserting the old >=3s backoff. Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com> --------- Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
* fix: take the named pipe user SID from the process token
NamedPipeHelper.create_security_attributes() resolved the DACL's user SID
with LookupAccountName("", getpass.getuser()). getpass.getuser() only reads
the LOGNAME, USER, LNAME and USERNAME environment variables, and the name
lookup goes through LSA, which can fail with ERROR_NONE_MAPPED (1332). A
Windows CI host running as SYSTEM hit 1332 on both the server path
(create_named_pipe_server) and the client path
(establish_named_pipe_connection), and the 3-attempt retry from #262 ran
out.
- Read the SID from the process token: OpenProcessToken(GetCurrentProcess(),
TOKEN_QUERY) and GetTokenInformation(token, TokenUser)[0], closing the
token in a finally. No name lookup, so 1332 cannot occur. The #262 retry
and the getpass import are removed.
- The client no longer builds a DACL. CreateFile ignores the security
descriptor when opening an existing pipe and reads only bInheritHandle. It
passes SECURITY_ATTRIBUTES with SECURITY_DESCRIPTOR = None and
bInheritHandle = True. pywin32's SECURITY_ATTRIBUTES() defaults
bInheritHandle to TRUE (PySECURITY_ATTRIBUTES.cpp), so the client handle
stays inheritable, as before. Passing None would have made it
non-inheritable.
- The server DACL is unchanged: GENERIC_READ|GENERIC_WRITE allowed to the
user SID, then GENERIC_ALL denied to S-1-5-2 (NETWORK).
Tests:
- Unit: the token SID ends up in the DACL; with the env user name set to
an unmapped name, LookupAccountName is not called; the token is closed
when GetTokenInformation raises; the client passes no descriptor, an
inheritable handle and OPEN_EXISTING, and does not call
create_security_attributes. The three LookupAccountName/retry tests
pinned the removed behaviour and are replaced.
- Integ: test_pipe_security_does_not_depend_on_user_name runs a real pipe
round-trip with LOGNAME, USER, LNAME and USERNAME set to
no-such-user-1332, once for server and client and once for the client
only, plus an ambient-environment control. It asserts the precondition
that LookupAccountName("", "no-such-user-1332") raises 1332, and reads the
DACL back with GetSecurityInfo. The old code fails this test
deterministically.
- CI: code_quality.yml runs windows-latest x Python 3.9-3.14 through
OpenJobDescription/.github reusable_python_build.yml, whose last step
(line 72) is `hatch run test`, which is `pytest ... test -vv`. That
collects test/openjd/adaptor_runtime/integ/_named_pipe/. No marker gates
the new test.
Ran on macOS (Python 3.13, hatch 1.14.1):
- hatch run lint, before and after: ruff "All checks passed!", black
"146 files would be left unchanged", mypy "no issues found in 146
source files".
- hatch run mypy --platform win32 src test, before and after: no issues
in 146 source files.
- hatch run test: before (mainline d9044ea) 482 passed, 26 skipped,
coverage 93.39%. After: 482 passed, 29 skipped, coverage 93.39%. The 3
new skips are the 3 new integ cases. The Windows-only unit module is
one module-level skip before and after.
- Windows tests skip on macOS. To run them anyway, a fake pywin32 plugin
(not committed) models the APIs these tests touch, using the pywin32 C
sources. Fidelity control: mainline code with mainline tests passes on
it (11 passed). Branch: 15 passed, 1 skipped (the GITHUB_ACTIONS-only
different-user test).
Mutation check, EXECUTED on the fake (sha256 restore, __pycache__
cleared between mutants), all caught:
- M1 restore LookupAccountName(getpass.getuser()) plus retry: fails
test_create_security_attributes_uses_process_token_user,
..._does_not_look_up_user_name, ..._closes_token_when_query_fails and
integ [unmapped-user-name-for-server-and-client].
- M2 restore the client DACL: fails
test_establish_named_pipe_connection_builds_no_dacl.
- M3 original code (M1+M2): also fails integ
[unmapped-user-name-for-client].
- M4 no finally, M5 no close, M6 client not inheritable, M7 client None,
M8 client keeps blank descriptor, M9 allow Everyone, M10 drop network
deny: each caught. M4-M8 by the token-close or client unit test, M9
and M10 also by all three integ cases.
ARGUED for real Windows, not run: with the old code, LookupAccountName
raises 1332 on every attempt (the test asserts this as a precondition),
so both unmapped integ cases fail. M2 alone is equivalent at the OS level,
because CreateFile ignores the descriptor. Only the unit test catches it.
Auditors (three, independent; each returned PASS):
- Pin/mutation: 27 of their own mutants, 26 killed. The survivor removes
the bInheritHandle = True line, which is equivalent because pywin32
defaults it to TRUE. Kept the explicit line for the reader. The
CloseHandle-call assertion rejects an equivalent token.Close(). Kept,
since it matches the repo's mock style.
- Gap analysis: 9 mutants, all killed. Pre-existing:
test_fail_to_connect_to_named_pipe_with_different_user has its assert
inside pytest.raises, so the assert never runs. Not in this diff, and
tightening it to winerror 5 would gate CI on a claim nobody has run.
Recorded as a follow-up. _secure_open.py has the same LookupAccountName
pattern; follow-up.
- Craft/cross-platform: lint, win32 mypy and collection green; pywin32
API shapes checked against source. Fixed: the client-only case now sets
the unmapped name inside the try, so a failed precondition still closes
the server handle. Declined: cross-checking the SID with an LSA name
lookup. It would bring back the lookup this fix removes. The test
helper hard-codes TokenUser independently, and a TokenOwner mutant is
killed.
Follow-up, out of scope: openjd-sessions-for-python _get_process_user_sid()
uses the same name-lookup pattern.
Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
* test: expect the deny-first ACE order on the named pipe DACL
Windows CI showed the pipe DACL reads back with the NETWORK deny ACE
before the user allow ACE, both in the in-memory security attributes
and on the created pipe. That is canonical order, and the order that
makes the deny apply to the user's own network logons. The new tests
asserted allow-first and failed on all six Windows jobs; production
code was not involved, since mainline builds the same DACL.
Assert the canonical order exactly, and find the allowed SID by ACE
type rather than by index.
Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
---------
Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
Summary
create_security_attributes()callswin32security.LookupAccountName("", username)to resolve the current user's SID for named pipe access controlpywintypes.error: (1332, 'LookupAccountName', 'No mapping between account names and security IDs was done.')on freshly started EC2 instances where the LSA (Local Security Authority) service has not finished initializingAffected repos
Fixes intermittent Windows integration test failures in all DCC repos that use
LOGNAME=SYSTEMin their CodeBuild Windows environment: Cinema 4D, Blender, Houdini, Maya, 3ds Max, VRED, and Unreal Engine.Related ticket: https://t.corp.amazon.com/P447504169
Test plan
test_create_security_attributes_uses_lookup: verifiesLookupAccountNameis called with the current usernametest_create_security_attributes_retries_on_lsa_not_ready: verifies retry succeeds after a transient 1332 error, with correct backoff sleeptest_create_security_attributes_raises_after_max_retries: verifies the original error is re-raised after exhausting all retries