runtime: lock the timer queue when resetTimer mutates timer fields - #5623
Open
JacksonBopp wants to merge 1 commit into
Open
JacksonBopp wants to merge 1 commit into
JacksonBopp wants to merge 1 commit into
Conversation
resetTimer's fast path for a periodic timer still firing on another goroutine allocates a fresh timerNode that points at the same shared *timer as the node currently running its callback. Every place that reads or writes that timer's when/period fields while it can be firing concurrently takes the scheduler's timer lock first: reAddTimer does, and the two reads in the threads scheduler's timerRunner do too. resetTimer's own writes to t.timer.when and t.timer.period were the one place that didn't, an unsynchronized store racing against those lock-protected accesses. Update tinygo-org#5469 That issue is a nil pointer dereference during a fuzz-style repeat of the context tests, and tinygo-org#5470 already fixed one instance of this same "accessed outside the lock" pattern in timerRunner. This closes another surviving instance of it: resetTimer's writes now go through the same lock via two new functions, lockTimerQueue/unlockTimerQueue, implemented per scheduler (the real lock for threads and cores, interrupt disable/restore for the cooperative scheduler, and a no-op for scheduler.none, where timers are unsupported entirely). I wasn't able to reproduce the original crash directly. It's rare enough that the issue itself needed a repeated loop to hit it, and a targeted stress test hammering Ticker.Reset concurrently with active firing (16 tickers, 3 goroutines each, several minutes total across both the original and patched runtime) didn't trigger it either way. What I can confirm: the data race is real and reproducible by inspection (the write in resetTimer has no lock while every other writer and reader of the same fields has one), and the fix doesn't introduce any deadlock or regression under that same stress test.
Member
|
Should this call to |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
resetTimer's fast path for a periodic timer still firing on another goroutine allocates a freshtimerNodethat points at the same shared*timeras the node currently running its callback. Every place that reads or writes that timer'swhen/periodfields while it can be firing concurrently takes the scheduler's timer lock first:reAddTimerdoes, and the two reads in the threads scheduler'stimerRunnerdo too.resetTimer's own writes tot.timer.whenandt.timer.periodwere the one place that didn't, an unsynchronized store racing against those lock-protected accesses.Update #5469
That issue is a nil pointer dereference during a fuzz-style repeat of the context tests, and #5470 already fixed one instance of this same "accessed outside the lock" pattern in
timerRunner. This closes another surviving instance of it:resetTimer's writes now go through the same lock via two new functions,lockTimerQueue/unlockTimerQueue, implemented per scheduler (the real lock for threads and cores, interrupt disable/restore for the cooperative scheduler, and a no-op forscheduler.none, where timers are unsupported entirely).Test plan
I wasn't able to reproduce the original crash directly. It's rare enough that the issue itself needed a repeated loop to hit it, and a targeted stress test hammering
Ticker.Resetconcurrently with active firing (16 tickers, 3 goroutines each, several minutes total across both the original and patched runtime) didn't trigger it either way.What I can confirm:
resetTimerhas no lock while every other writer and reader of the same fields has one.gofmtis clean on all changed files.contextpackage tests pass, and the same targeted stress test ran clean (no hangs, no crashes) against the patched runtime.Happy to add a regression test if there's a preferred pattern for timing-dependent runtime races in this codebase, since a deterministic one isn't straightforward given how narrow the window is.