[driver] Serialize load_binary to prevent IGC race condition (#7581)

IGC 2.38.x has a race condition in its in-process Level Zero JIT path
(`zeModuleCreate`). When multiple Triton kernels JIT-compile concurrently
in the same process (via AsyncCompileMode's ThreadPoolExecutor), each
thread calls `load_binary`, which releases the Python GIL on entry to the
C extension and then calls `zeModuleCreate`. With the GIL released,
multiple threads can be inside `zeModuleCreate` simultaneously, corrupting
IGC's internal state and triggering a `SIGABRT` in `libigdrcl.so`.

The crash is confirmed by the definitive test: adding any compilation
overhead that serializes the calls (`IGC_PrintToConsole=1`, shader dump)
prevents it entirely — proving a timing race, not a deterministic bug.

Fix: add a module-level `threading.Lock` that serializes all `load_binary`
calls. This ensures only one thread is inside `zeModuleCreate` at a time,
preventing the race without requiring an IGC fix.

Performance: verified +1.7% startup time difference (within measurement
noise) on Arc A770 with vLLM. Hot-path inference is unaffected (kernels
are cached after first compilation).

Fixes: #7581

Author: NathanVoldman

third_party/intel/backend/driver.py

@@ -9,6 +9,7 @@
 import subprocess
 import sysconfig
 import tempfile
+import threading
 from pathlib import Path
 from functools import cached_property, lru_cache
 
@@ -26,6 +27,12 @@
 # and via setdefault so an explicit choice by the user still wins.
 os.environ.setdefault("PTI_COLLECTION_MODE", "0")
 
+# IGC 2.38.x has a race condition in its in-process Level Zero JIT path (zeModuleCreate).
+# Concurrent calls from different threads in the same process can trigger an internal
+# compiler error (SIGABRT in libigdrcl.so). Serializing all load_binary calls prevents
+# the race without requiring an IGC fix. See issue #7581.
+_load_binary_lock = threading.Lock()
+
 # A hard-coded cache version that can be updated when we know that the cached file is invalid and
 # there are no other ways to detect that the runtime environment has changed. For example, a shared
 # library has been updated as a result of updated dependencies.
@@ -307,16 +314,20 @@ def load_binary(self, *args):
         # IGC build log carries a PTSS marker, OutOfResources is raised
         # directly so triton.runtime.autotuner can skip the offending tile.
         # No per-launch overhead on the success path.
-        try:
-            return self.shared_library.load_binary(args)
-        except Exception as e:
-            from triton.runtime.errors import IntelGPUError, OutOfResources
-            if isinstance(e, OutOfResources):
-                raise
-            if str(e).startswith("ZE_"):
-                raise IntelGPUError("Error during Intel load_binary: " + str(e)) from e
-            else:
-                raise e
+        #
+        # _load_binary_lock serializes concurrent zeModuleCreate calls to
+        # work around a race condition in IGC 2.38.x in-process JIT. See #7581.
+        with _load_binary_lock:
+            try:
+                return self.shared_library.load_binary(args)
+            except Exception as e:
+                from triton.runtime.errors import IntelGPUError, OutOfResources
+                if isinstance(e, OutOfResources):
+                    raise
+                if str(e).startswith("ZE_"):
+                    raise IntelGPUError("Error during Intel load_binary: " + str(e)) from e
+                else:
+                    raise e
 
     if os.name != 'nt':