Skip to content

Commit db720a5

Browse files
rustyconoverclaude
andcommitted
storage: serialize every operation on the shared-cache :memory: store
`vgi-fixture-http` failed 69 cases of the upstream integration suite with `sqlite3.OperationalError: database table is locked`, direct to the worker. The shared-cache in-memory store serialized only its writers. Shared-cache locks are per table and conflict both ways -- a writer cannot modify a table another connection is reading, a reader cannot read one with another connection's uncommitted write -- and either is SQLITE_LOCKED, which the busy handler never retries. So a `state_scan` overlapping an `execution_clear` failed exactly as two pops had before a550612. It stayed hidden because the fixture never really ran on `:memory:`: until 94df3e3, `import vgi` resolved `Function.storage` before the fixture's `main()` could set `VGI_WORKER_SQLITE_PATH=:memory:`, so it used the shared file store. Making resolution lazy (0.34.1) let the setting apply and exposed the bug. - Every public FunctionStorageSqlite operation now goes through the lock (renamed `_op_guard`). Derived from the class rather than listed: the hand- written writer list is how the four readers were missed. `close` is the only exclusion. Reads fetch eagerly, so no statement outlives the guard. - `_DefaultStorageDescriptor` resolves once under a lock. With resolution lazy, first use is a request, and concurrent first requests each built a store -- for `:memory:`, separate databases, so state written through one was invisible through the other. Tests (all three failed before): 8 threads of mixed reads and writes on one scope; every public operation is guarded; 8 concurrent first uses build one store. Measured: the upstream suite takes 150s on the fixed in-memory store vs 152s on a file store, and passes in full (10,700 assertions) on both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 5b5193f commit db720a5

3 files changed

Lines changed: 178 additions & 33 deletions

File tree

‎tests/test_function_storage.py‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -554,3 +554,122 @@ def test_non_sharding_backend_without_attach_gets_empty_key(self) -> None:
554554

555555
bs = BoundStorage(FunctionStorageSqlite(db_path=":memory:"), b"exec1")
556556
assert bs._shard_key == ""
557+
558+
559+
class TestSharedCacheMemoryConcurrency:
560+
"""The ``:memory:`` store under concurrent threads — what ``vgi-fixture-http`` runs.
561+
562+
A shared-cache in-memory database locks per *table*, in both directions: a
563+
connection cannot write a table another connection is reading, nor read one
564+
another connection has written and not committed. Either collision is
565+
``SQLITE_LOCKED`` ("database table is locked"), which the busy handler never
566+
retries. Serializing only the writers therefore still failed the moment a
567+
reader overlapped a writer — the shape of every catalog transaction, which
568+
reads state, then clears it on commit.
569+
"""
570+
571+
def test_concurrent_readers_and_writers_do_not_lock(self) -> None:
572+
"""Mixed readers and writers on one scope, every thread at once.
573+
574+
The same mix as a concurrent integration run: put, get, scan, append,
575+
log-scan, counter add/get, and a periodic ``execution_clear``. Before
576+
readers were serialized too, every thread failed within its first few
577+
operations.
578+
"""
579+
import threading
580+
581+
storage = FunctionStorageSqlite(db_path=":memory:")
582+
threads_n = 8
583+
iters = 200
584+
barrier = threading.Barrier(threads_n)
585+
errors: list[BaseException] = []
586+
scope = b"shared-scope" # one scope for all: the contention a transaction produces
587+
588+
def worker(i: int) -> None:
589+
barrier.wait() # maximize contention
590+
try:
591+
for n in range(iters):
592+
storage.state_put_many(scope, b"ns", [(b"k%d" % i, b"v%d" % n)])
593+
storage.state_get_many(scope, b"ns", [b"k0", b"k1"])
594+
storage.state_scan(scope, b"ns")
595+
storage.state_append(scope, b"log", b"k", b"item")
596+
storage.state_log_scan(scope, b"log", b"k")
597+
storage.state_counter_add(scope, b"counters", b"c", 1)
598+
storage.state_counter_get(scope, b"counters", b"c")
599+
if n % 50 == 49:
600+
storage.execution_clear(scope)
601+
except BaseException as exc: # noqa: BLE001 — collected and asserted below
602+
errors.append(exc)
603+
604+
threads = [threading.Thread(target=worker, args=(i,)) for i in range(threads_n)]
605+
for t in threads:
606+
t.start()
607+
for t in threads:
608+
t.join()
609+
610+
assert errors == [], f"{len(errors)} thread(s) failed: {sorted({str(e) for e in errors})}"
611+
612+
def test_every_public_operation_is_serialized(self) -> None:
613+
"""No operation can be added, or left out, without the guard.
614+
615+
The writer-only guard named its methods by hand, and the four readers
616+
were simply not on the list. Deriving the set from the class makes that
617+
omission impossible; this pins it, so a new public operation that somehow
618+
escaped the wrapper fails here rather than as a flaky lock error under load.
619+
"""
620+
import inspect
621+
622+
ops = [
623+
name
624+
for name, member in vars(FunctionStorageSqlite).items()
625+
if not name.startswith("_") and inspect.isfunction(member) and name != "close"
626+
]
627+
assert "state_scan" in ops and "state_put_many" in ops, "the derivation found nothing"
628+
unguarded = [name for name in ops if not hasattr(getattr(FunctionStorageSqlite, name), "__wrapped__")]
629+
assert unguarded == []
630+
631+
632+
class TestDefaultStorageResolution:
633+
"""``Function.storage`` resolves lazily — and exactly once, even when raced.
634+
635+
Resolution moved from import time to first use in 0.34.1, which means the
636+
first use is now a request, and requests arrive concurrently. An unguarded
637+
check-then-set let two first callers each build a store; with ``:memory:``
638+
that is two unrelated databases, so state written through one is invisible
639+
through the other.
640+
"""
641+
642+
def test_concurrent_first_use_resolves_once(self, monkeypatch: pytest.MonkeyPatch) -> None:
643+
"""Eight simultaneous first uses build one store, and all eight get it."""
644+
import threading
645+
import time
646+
647+
import vgi.function as function_module
648+
from vgi.function import _DefaultStorageDescriptor
649+
650+
built: list[object] = []
651+
652+
def slow_resolve() -> object:
653+
time.sleep(0.05) # widen the window between the check and the set
654+
store = object()
655+
built.append(store)
656+
return store
657+
658+
monkeypatch.setattr(function_module, "_resolve_storage", slow_resolve)
659+
descriptor = _DefaultStorageDescriptor()
660+
threads_n = 8
661+
barrier = threading.Barrier(threads_n)
662+
seen: list[object] = []
663+
664+
def first_use() -> None:
665+
barrier.wait()
666+
seen.append(descriptor.__get__(None, object))
667+
668+
threads = [threading.Thread(target=first_use) for _ in range(threads_n)]
669+
for t in threads:
670+
t.start()
671+
for t in threads:
672+
t.join()
673+
674+
assert len(built) == 1, f"the default store was built {len(built)} times"
675+
assert all(s is built[0] for s in seen)

‎vgi/function.py‎

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121

2222
import logging
2323
import os
24+
import threading
2425
from abc import ABC
2526
from typing import (
2627
Annotated,
@@ -73,19 +74,32 @@ def _resolve_storage() -> FunctionStorage:
7374

7475

7576
class _DefaultStorageDescriptor:
76-
"""Resolve `FunctionStorage` lazily on first attribute access.
77+
"""Resolve `FunctionStorage` lazily on first attribute access — exactly once.
7778
7879
This avoids evaluating environment variables at import time. When a
7980
subclass explicitly sets ``storage = SomeStorage(...)``, the plain
8081
attribute shadows this descriptor — no interference.
82+
83+
Resolution is locked because first use is a *request* now, and requests
84+
arrive concurrently. Until 0.34.1 the store was resolved as a side effect of
85+
``import vgi``, single-threaded; once that was made lazy, an unguarded
86+
check-then-set let two first callers each build a store — and with the
87+
``:memory:`` backend that is two unrelated databases, so state one thread
88+
wrote was invisible to the other. Double-checked, so every later access
89+
stays lock-free.
8190
"""
8291

8392
_resolved: FunctionStorage | None = None
93+
_lock = threading.Lock()
8494

8595
def __get__(self, obj: object | None, objtype: type | None = None) -> FunctionStorage:
86-
if self._resolved is None:
87-
self._resolved = _resolve_storage()
88-
return self._resolved
96+
resolved = self._resolved
97+
if resolved is None:
98+
with self._lock:
99+
if self._resolved is None:
100+
self._resolved = _resolve_storage()
101+
resolved = self._resolved
102+
return resolved
89103

90104

91105
# Default max_workers when not explicitly specified (effectively unlimited)

‎vgi/function_storage.py‎

Lines changed: 41 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
import contextlib
2121
import enum
2222
import functools
23+
import inspect
2324
import logging
2425
import os
2526
import sqlite3
@@ -944,9 +945,9 @@ def __init__(self, db_path: str | None = None) -> None:
944945
self._memory_uri = None
945946
self._anchor_conn = None
946947
self.db_path = db_path if db_path is not None else _get_default_db_path()
947-
# Shared-cache in-memory DBs need a process-local write lock; file DBs
948-
# do not. See `_write_guard`.
949-
self._write_lock: threading.Lock | None = threading.Lock() if self._memory_uri is not None else None
948+
# Shared-cache in-memory DBs need a process-local lock around every
949+
# operation; file DBs do not. See `_op_guard`.
950+
self._op_lock: threading.Lock | None = threading.Lock() if self._memory_uri is not None else None
950951
self._tls = threading.local()
951952
self._ensure_tables()
952953

@@ -986,8 +987,8 @@ def _conn(self) -> sqlite3.Connection:
986987
return conn
987988

988989
@contextlib.contextmanager
989-
def _write_guard(self) -> Iterator[None]:
990-
"""Serialize writers on a shared-cache in-memory DB; no-op for a file DB.
990+
def _op_guard(self) -> Iterator[None]:
991+
"""Serialize every operation on a shared-cache in-memory DB; no-op for a file DB.
991992
992993
`_conn` gives each thread its own connection so SQLite's own locking
993994
serializes writers "without a Python-level lock and without forfeiting
@@ -1008,11 +1009,25 @@ def _write_guard(self) -> Iterator[None]:
10081009
Serializing writes costs nothing here. Measured: pops are 0.007 ms and
10091010
concurrency already made them *slower* (128k/s at 1 thread → 79k/s at
10101011
16), so there was never any write parallelism to lose — only the error.
1012+
1013+
**Readers too.** This first serialized writers only, which is not
1014+
enough: shared-cache locks are per *table* and conflict in both
1015+
directions. A writer cannot modify a table another connection is
1016+
reading, and a reader cannot read one another connection has written
1017+
and not yet committed — either is the same unretried `SQLITE_LOCKED`.
1018+
So a `state_scan` overlapping an `execution_clear` failed exactly as
1019+
two pops had. It stayed hidden because `vgi-fixture-http`, the one
1020+
in-tree user of `:memory:`, silently ran on the file DB until 0.34.1:
1021+
`import vgi` resolved `Function.storage` before the fixture's `main()`
1022+
could ask for memory. Once it really ran here, 69 cases of the
1023+
integration suite failed with "database table is locked". Every read
1024+
fetches eagerly (`fetchall`/`fetchone`), so no statement outlives the
1025+
guard, and — as above — there was no concurrency to give up.
10111026
"""
1012-
if self._write_lock is None:
1027+
if self._op_lock is None:
10131028
yield
10141029
return
1015-
with self._write_lock:
1030+
with self._op_lock:
10161031
yield
10171032

10181033
def close(self) -> None:
@@ -1477,40 +1492,37 @@ def state_counter_delete(self, scope_id: bytes, ns: bytes, key: bytes, *, shard_
14771492
conn.commit()
14781493

14791494

1480-
def _guard_sqlite_writes() -> None:
1481-
"""Route every FunctionStorageSqlite writer through `_write_guard`.
1495+
#: Public methods that run no statement against the shared cache, so they are
1496+
#: left outside `_op_guard`. ``close`` only drops the calling thread's own
1497+
#: connection.
1498+
_UNGUARDED_OPS = frozenset({"close"})
1499+
14821500

1483-
Applied here rather than by editing each method so no writer can be added
1484-
later and silently miss the guard — a partial fix would be worse than none,
1485-
since one unguarded writer is enough to re-introduce `SQLITE_LOCKED` for
1486-
every other one sharing the cache.
1501+
def _guard_sqlite_ops() -> None:
1502+
"""Route every public FunctionStorageSqlite operation through `_op_guard`.
1503+
1504+
Derived from the class rather than listed: the previous version named its
1505+
methods by hand, the four readers were never on the list, and one unguarded
1506+
operation is enough to re-introduce `SQLITE_LOCKED` for every other one
1507+
sharing the cache. A new public method is now guarded by default, and
1508+
`_UNGUARDED_OPS` is the only way out of it.
14871509
"""
14881510

14891511
def guarded(fn: Any) -> Any:
14901512
@functools.wraps(fn)
14911513
def wrapper(self: "FunctionStorageSqlite", *a: Any, **k: Any) -> Any:
1492-
with self._write_guard():
1514+
with self._op_guard():
14931515
return fn(self, *a, **k)
14941516

14951517
return wrapper
14961518

1497-
for name in (
1498-
"queue_push",
1499-
"queue_pop",
1500-
"queue_clear",
1501-
"state_put_many",
1502-
"state_drain",
1503-
"state_delete",
1504-
"state_append",
1505-
"execution_clear",
1506-
"state_counter_add",
1507-
"state_counter_set",
1508-
"state_counter_delete",
1509-
):
1510-
setattr(FunctionStorageSqlite, name, guarded(getattr(FunctionStorageSqlite, name)))
1519+
for name, member in list(vars(FunctionStorageSqlite).items()):
1520+
if name.startswith("_") or name in _UNGUARDED_OPS or not inspect.isfunction(member):
1521+
continue
1522+
setattr(FunctionStorageSqlite, name, guarded(member))
15111523

15121524

1513-
_guard_sqlite_writes()
1525+
_guard_sqlite_ops()
15141526

15151527

15161528
class ShardedSqliteStorage:

0 commit comments

Comments
 (0)