Skip to content

Commit 3252e69

Browse files
authored
feat!: shorten the session working directory name for Windows MAX_PATH (#348)
* feat!: shorten the session working directory name for Windows MAX_PATH The applications a job runs on Windows -- After Effects and Cinema4D among them -- use the legacy Win32 file APIs, which cap a path at MAX_PATH (260 characters) regardless of the LongPathsEnabled registry value and regardless of whether this library prefixes its own paths with \\?\. Nothing but a shorter path helps them. The session working directory name was the full session id used verbatim as the mkdtemp() prefix. For a Deadline Cloud session that is 40 characters (`session-` plus a 32 character hex uuid) plus mkdtemp()'s 8 random characters: 48 characters, of which the uuid and the random suffix were both sources of uniqueness. Only one of them needs to be. Use the last SESSION_DIR_ID_LENGTH (6) characters of the session id as the prefix instead, giving a 14 character name -- 34 characters recovered. The tail rather than the head because a session id conventionally leads with constant text saying what kind of id it is, so a leading slice can carry no correlation at all. Uniqueness is unaffected. mkdtemp() creates each candidate name with an exclusive mkdir() and retries on FileExistsError (verified identical in CPython 3.9 through 3.14, the supported range), so it cannot return an existing directory however short the prefix is. The truncated id is purely an operator-facing label; the full id is unchanged and still appears on every log record for the session and immediately beside this directory's path at initialization. Also shorten the embedded files subdirectory prefix from "embedded_files" to "ef", recovering 12 more characters from every embedded file path. That directory nests inside the session working directory, so its name is charged to everything beneath it. BREAKING CHANGE: the on-disk name of the session working directory is no longer prefixed with the session id, and the embedded files subdirectory is named "ef<random>" rather than "embedded_files<random>". Neither name is part of the public Python interface -- Session.working_directory and Session.files_directory are unchanged, as is OPENJD_SESSION_WORKING_DIR -- but anything that pattern-matches the directory name must be updated. Known consumer: deadline-cloud-worker-agent's test/e2e/test_worker_config.py asserts a `session-[a-f0-9]{32}` directory name, and its docs/state.md documents the old shape. Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com> * Address review: deterministic dir-name tests + doc callouts - test_full_session_id_remains_in_the_log: filter to openjd.sessions records and default the session_id getattr to None, so a dropped LoggerAdapter fails the assertion instead of passing vacuously. - Replace the probabilistic name-inspection assertions with a deterministic check that TempDir is called with prefix="". - Assert the name length as a <= budget ceiling rather than == to an incidental CPython detail. - Narrow the concurrent test's except from BaseException to Exception (CodeQL). - Fix TempDir.prefix docstring (default is None -> "tmp", not ""). - Note on working_directory / retain_working_dir that the directory name no longer carries session identity; the log holds that mapping. Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com> --------- Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
1 parent 5a80e92 commit 3252e69

3 files changed

Lines changed: 229 additions & 2 deletions

File tree

‎src/openjd/sessions/_session.py‎

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,22 @@
7474
__all__ = ("SessionState", "Session", "EnvironmentIdentifier")
7575

7676

77+
SESSION_DIR_NAME_LENGTH = 8
78+
"""Length of a session working directory's name.
79+
80+
- Every character is charged against MAX_PATH (260) for the applications a job
81+
runs, which are not long-path aware, so `LongPathsEnabled` and `\\\\?\\` do not
82+
help them.
83+
- The name is mkdtemp()'s 8 random characters and nothing else. Passing the
84+
40-character session id as a prefix cost 48.
85+
- The id was pure label there: mkdtemp() generates the random characters and
86+
creates each candidate with an exclusive mkdir(), retrying under a taken name.
87+
- Directory -> session stays recoverable from the session log, which the Worker
88+
Agent writes to `<worker_logs_dir>/<queue_id>/<session_id>.log` and which
89+
outlives the directory. See _create_working_directory.
90+
"""
91+
92+
7793
class SessionState(str, Enum):
7894
READY = "ready"
7995
"""The state of a Session when it is ready to run actions.
@@ -401,6 +417,12 @@ def __init__(
401417
within all actions running within this session. Defaults to None.
402418
retain_working_dir (bool, optional): If set, then the Session's Working Directory
403419
is not deleted when this Session object is deleted. Defaults to False.
420+
Note: the working directory's *name* no longer encodes the session id
421+
(it is mkdtemp()'s random characters and nothing else), so a retained
422+
directory is not attributable to a session by its name alone. The
423+
session log records the working directory's full path against the
424+
session id; that log is how a retained or orphaned directory is
425+
resolved back to its session.
404426
user (Optional[SessionUser]): The specific OS user to run subprocesses as, and whom
405427
will have permissions to the Session's Working Directory.
406428
Defaults to the current process user.
@@ -616,6 +638,10 @@ def working_directory(self) -> Path:
616638
This is available in a Job Template's format string expressions as
617639
Session.WorkingDirectory
618640
641+
The directory's name is generated (mkdtemp()'s random characters) and
642+
carries no part of the session id, so do not parse identity out of it;
643+
the session id is recorded against this path in the session log instead.
644+
619645
Raises:
620646
RuntimeError: If this Session has no working directory, which means
621647
construction did not complete.
@@ -2146,8 +2172,17 @@ def _create_working_directory(self) -> TempDir:
21462172
),
21472173
)
21482174

2175+
# prefix="" is deliberate, and is not the same as omitting it: mkdtemp()
2176+
# substitutes its "tmp" template for a prefix of None, costing 3 characters.
2177+
# See SESSION_DIR_NAME_LENGTH.
2178+
#
21492179
# Raises: RuntimeError
2150-
return TempDir(dir=root_dir, prefix=self._session_id, user=self._user, logger=self._logger)
2180+
return TempDir(
2181+
dir=root_dir,
2182+
prefix="",
2183+
user=self._user,
2184+
logger=self._logger,
2185+
)
21512186

21522187
def _create_files_directory(self) -> TempDir:
21532188
"""Creates the subdirectory of the working directory in which we'll materialize

‎src/openjd/sessions/_tempdir.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -261,7 +261,9 @@ def __init__(
261261
dir (Optional[Path]): The directory in which to create the temp dir.
262262
Defaults to tempfile.gettempdir().
263263
prefix (Optional[str]): A prefix to use in the name of the generated temp dir.
264-
Defaults to "".
264+
Defaults to None, which mkdtemp() replaces with its "tmp" template
265+
(3 characters). Pass "" for no prefix at all -- a shorter name, and
266+
not the same as omitting the argument.
265267
user (Optional[SessionUser]): A group that will own the created directory.
266268
The group-write bit will be set on the directory if this option is supplied.
267269
Defaults to this process' effective user/group.

‎test/openjd/sessions_v0/test_session.py‎

Lines changed: 190 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
import os
66
import stat
77
import sys
8+
import threading
89
import time
910
import uuid
1011
from datetime import timedelta
@@ -53,10 +54,12 @@
5354
LogContent,
5455
)
5556
from openjd.sessions import _path_mapping as path_mapping_impl_mod
57+
from openjd.sessions import _session as session_module
5658
from openjd.sessions._action_filter import ActionMessageKind
5759
from openjd.sessions._os_checker import is_posix, is_windows
5860
from openjd.sessions._logging import LoggerAdapter, LogExtraInfo
5961
from openjd.sessions._session import (
62+
SESSION_DIR_NAME_LENGTH,
6063
EnvironmentVariableChange,
6164
EnvironmentVariableSetChange,
6265
EnvironmentVariableUnsetChange,
@@ -548,6 +551,193 @@ class StatReturn:
548551
)
549552

550553

554+
class TestSessionWorkingDirectoryName:
555+
"""The session working directory's name is a Windows MAX_PATH budget decision.
556+
557+
- Every character is charged against MAX_PATH for the applications a job runs.
558+
- Uniqueness is all the name carries; mkdtemp() supplies it. The uniqueness
559+
tests are a regression guard on that, not a test of entropy.
560+
- Directory -> session is the log's job, not the name's. See
561+
test_full_session_id_remains_in_the_log.
562+
"""
563+
564+
# session-<32 hex uuid>, as a Deadline Cloud session id is shaped.
565+
DEADLINE_SHAPED_ID = f"session-{uuid.uuid4().hex}"
566+
567+
def test_working_directory_name_length_is_bounded(self) -> None:
568+
# A ceiling, not an equality: the exact count is mkdtemp()'s to choose
569+
# (CPython currently yields 8 characters, hence the constant) and a stdlib
570+
# change to a *shorter* name would not be a regression. What must not
571+
# regress is the budget -- passing the 40-character session id as a prefix
572+
# cost 48.
573+
574+
# GIVEN
575+
job_params = dict[str, ParameterValue]()
576+
577+
# WHEN
578+
with Session(
579+
session_id=self.DEADLINE_SHAPED_ID, job_parameter_values=job_params
580+
) as session:
581+
# THEN
582+
assert len(session.working_directory.name) <= SESSION_DIR_NAME_LENGTH
583+
584+
def test_working_directory_is_created_with_an_empty_prefix(self) -> None:
585+
# prefix="" rather than None is the load-bearing choice: None restores
586+
# mkdtemp()'s "tmp" template (3 characters) and the session id (48) is what
587+
# we removed. Assert the argument we pass, not a sample of the random name:
588+
# inspecting the name for the *absence* of a prefix is probabilistic -- a
589+
# random [a-z0-9_] name can happen to begin "tmp" -- while the call is exact.
590+
591+
# GIVEN
592+
job_params = dict[str, ParameterValue]()
593+
594+
# WHEN
595+
with patch.object(session_module, "TempDir", wraps=session_module.TempDir) as mock_tempdir:
596+
with Session(
597+
session_id=self.DEADLINE_SHAPED_ID, job_parameter_values=job_params
598+
) as session:
599+
# THEN
600+
# The first TempDir call is the working directory; the second is
601+
# the nested embedded_files directory.
602+
assert mock_tempdir.call_args_list[0].kwargs["prefix"] == ""
603+
# ... and nothing of the session id survived into the name.
604+
assert self.DEADLINE_SHAPED_ID not in session.working_directory.name
605+
606+
@pytest.mark.usefixtures("caplog") # built-in fixture
607+
def test_full_session_id_remains_in_the_log(self, caplog: pytest.LogCaptureFixture) -> None:
608+
# The only thing that resolves an orphaned directory back to a session now.
609+
# Must not be relaxed: announced at init, logged beside the working
610+
# directory's path, and on every record as the `session_id` extra.
611+
612+
# GIVEN
613+
job_params = dict[str, ParameterValue]()
614+
615+
# WHEN
616+
with Session(
617+
session_id=self.DEADLINE_SHAPED_ID, job_parameter_values=job_params
618+
) as session:
619+
# THEN
620+
assert any(
621+
m == f"Initializing Open Job Description Session: {self.DEADLINE_SHAPED_ID}"
622+
for m in caplog.messages
623+
)
624+
assert any(
625+
m == f"Session Working Directory: {str(session.working_directory)}"
626+
for m in caplog.messages
627+
)
628+
# caplog.records collects from every logger propagating to root, so
629+
# filter to this library's before asserting the extra is present --
630+
# and default to None, not the expected id, so a record that is
631+
# *missing* the extra fails rather than passing vacuously (which is
632+
# precisely the regression -- a dropped LoggerAdapter -- this guards).
633+
session_records = [r for r in caplog.records if r.name.startswith("openjd.sessions")]
634+
assert session_records
635+
assert all(
636+
getattr(r, "session_id", None) == self.DEADLINE_SHAPED_ID for r in session_records
637+
)
638+
639+
@pytest.mark.usefixtures("tmp_path") # built-in fixture
640+
def test_many_sessions_in_one_root_get_distinct_directories(self, tmp_path: Path) -> None:
641+
# Guard on mkdtemp()'s exclusive-create. Sessions sharing one root is the
642+
# normal case for a worker host. All held open at once, so no directory is
643+
# free by virtue of an earlier one having been cleaned up.
644+
645+
# GIVEN
646+
job_params = dict[str, ParameterValue]()
647+
sessions: list[Session] = []
648+
649+
try:
650+
# WHEN
651+
for _ in range(64):
652+
sessions.append(
653+
Session(
654+
session_id=f"session-{uuid.uuid4().hex}",
655+
job_parameter_values=job_params,
656+
session_root_directory=tmp_path,
657+
)
658+
)
659+
660+
# THEN
661+
names = [s.working_directory.name for s in sessions]
662+
assert len(set(names)) == len(names)
663+
assert all(s.working_directory.is_dir() for s in sessions)
664+
# ... and the same for the nested embedded files directories.
665+
assert all(s.files_directory.is_dir() for s in sessions)
666+
finally:
667+
for session in sessions:
668+
session.cleanup()
669+
670+
@pytest.mark.usefixtures("tmp_path") # built-in fixture
671+
def test_concurrent_sessions_in_one_root_get_distinct_directories(self, tmp_path: Path) -> None:
672+
# The same under real concurrency: mkdtemp() is create-or-fail, not
673+
# check-then-create, so there is no window between picking and creating.
674+
675+
# GIVEN
676+
job_params = dict[str, ParameterValue]()
677+
sessions: list[Session] = []
678+
lock = threading.Lock()
679+
errors: list[Exception] = []
680+
681+
def make_session() -> None:
682+
try:
683+
session = Session(
684+
session_id=f"session-{uuid.uuid4().hex}",
685+
job_parameter_values=job_params,
686+
session_root_directory=tmp_path,
687+
)
688+
except Exception as err: # pragma: nocover - only on a failure
689+
with lock:
690+
errors.append(err)
691+
return
692+
with lock:
693+
sessions.append(session)
694+
695+
threads = [threading.Thread(target=make_session) for _ in range(16)]
696+
697+
try:
698+
# WHEN
699+
for thread in threads:
700+
thread.start()
701+
for thread in threads:
702+
thread.join()
703+
704+
# THEN
705+
assert errors == []
706+
names = [s.working_directory.name for s in sessions]
707+
assert len(names) == len(threads)
708+
assert len(set(names)) == len(names)
709+
finally:
710+
for session in sessions:
711+
session.cleanup()
712+
713+
def test_directory_names_are_portable_across_platforms(self) -> None:
714+
# One generator runs on every platform, so the name must be legal on all of
715+
# them: no Windows-reserved character or device name.
716+
#
717+
# Lower case specifically: NTFS folds case, so names differing only in case
718+
# would be two directories on Linux and one on Windows. mkdtemp()'s alphabet
719+
# is lower case only, so that cannot arise.
720+
721+
# GIVEN
722+
job_params = dict[str, ParameterValue]()
723+
windows_reserved_chars = set('<>:"/\\|?*')
724+
windows_reserved_names = {"con", "prn", "aux", "nul"} | {
725+
f"{stem}{n}" for stem in ("com", "lpt") for n in range(1, 10)
726+
}
727+
728+
# WHEN
729+
with Session(
730+
session_id=self.DEADLINE_SHAPED_ID, job_parameter_values=job_params
731+
) as session:
732+
for name in (session.working_directory.name, session.files_directory.name):
733+
# THEN
734+
assert not (set(name) & windows_reserved_chars)
735+
assert not any(ord(c) < 32 for c in name)
736+
assert not name.endswith((".", " "))
737+
assert name.split(".")[0] not in windows_reserved_names
738+
assert name == name.lower()
739+
740+
551741
class TestSessionCallbacks:
552742
"""Making sure that the session methods that are callbacks also call the user-provided
553743
callback with the expected data.

0 commit comments

Comments
 (0)