Repository navigation
Commit 5cdaa09
authored
Several fixes to worker logic (#57)
* Fix worker stop, timestamp handling, and imports
Preserve explicit timestamp values and strengthen worker shutdown behavior. Key changes:
- recording_manager: use timestamp if timestamp is not None else time.time() so falsy timestamps (e.g. 0) are preserved instead of being overridden.
- services/dlc_processor: import DLCLive at module top and enable profiling; make _stop_worker return a boolean and have reset() check that return value to avoid resetting the DLCLive instance while the worker thread is still alive (with warnings/logging). Adjusted comments about thread termination.
- gui/main_window: use a truthiness check for dlc_cam_id to avoid treating empty values as valid IDs.
These updates prevent spurious timestamp overrides, avoid unsafe DLCLive resets when threads are still running, and clarify import/profiling intent.
* Add WorkerState enum for worker lifecycle
Introduce a WorkerState enum and import Enum/auto in dlc_processor.py to represent worker lifecycle states (STOPPED, STARTING, RUNNING, STOPPING, FAULTED). This centralizes state representation for processing workers and improves clarity for future state management.
* Improve DLCLive processor lifecycle and safety
Add explicit worker lifecycle management and synchronization to DLCLive processor: introduce WorkerState, a lifecycle Lock, and stop/start state transitions. Make _start/_stop behavior safer by snapshotting the queue, returning boolean on stop, joining the thread with fault handling, and avoiding races by using a local queue reference in the worker loop. Harden enqueue_frame to respect lifecycle/state/stop flags and reduce copies by reusing a single frame copy and enqueuing enqueue timestamp. Improve DLCLive initialization error handling (set FAULTED on failure) and ensure caller-visible errors when DLCLive is not initialized. Prevent configuring the processor while running and add logging/warnings for shutdown/termination issues. Minor cleanup: adjust where initialized/state flags are set and remove redundant checks.
* Restart worker if dead and clear stop event
When queuing work, ensure the background worker is started if the worker thread is None or not alive by calling _start_worker_locked(frame_c, timestamp). Also clear the _stop_event in both the early-return (no thread) and final cleanup paths when stopping so a stale stop flag doesn't prevent future restarts.
* Mark worker FAULTED on queue errors
Improve error handling in the worker loop when the queue is missing or queue.get raises unexpected exceptions. When q is None the code now logs a clearer message, sets the worker state to WorkerState.FAULTED under the lifecycle lock, and emits an error signal. Also adds a broad except around queue.get to log the exception, set FAULTED state, emit the error message, and break the loop to avoid silent failures.
* Force-terminate frozen camera threads on stop
Make multi-camera shutdown more robust by iterating threads with their camera IDs, skipping non-running threads, and quitting each thread. If a thread fails to stop within 5000ms, log an error and call terminate() then wait an additional 1000ms to ensure it stops. Also simplify the start() docstring.
* Count unique cameras when starting multi-camera
Previously _expected_cameras was set to len(active_settings), which could overcount when multiple CameraSettings referred to the same physical camera. This change computes a set of unique camera IDs (via get_camera_id) and sets _expected_cameras to its length, ensuring duplicates aren't double-counted.
* Improve VideoRecorder shutdown and writer loop
Add robust shutdown and error handling for the video recorder. Prevent writes after stop by checking the stop event early; record and raise encoding errors as before. Refactor stop() to use local vars, signal the writer via a sentinel, join the writer thread safely, and avoid saving timestamps prematurely. Rework _writer_loop to handle a missing queue, broaden exception handling when pulling/writing items, ensure q.task_done() is always called, properly handle the sentinel, update encoding stats and logging, and perform final cleanup (save timestamps, clear queue and thread) when exiting.
* Refactor writer thread error handling and stats
Consolidate the writer-thread loop and tighten up exception handling. Removed the conditional _save_timestamps path during teardown. In the writer loop sentinel handling and frame processing are unified; on write errors the exception is stored under _stats_lock, logged, and stop_event/stop_now are set. The stats updates (frames_written, total_latency, last_latency, written_times, frame_timestamps and periodic FPS computation) were moved into the same control flow to ensure consistent accounting and termination behavior.
* Revert DLCLive imports due to unguarded torch
* Improve camera thread cleanup and recorder finalization
Add cooperative shutdown timeouts and robust cleanup for camera threads: introduce QUIT_WAIT_MS and TERMINATE_WAIT_MS, connect thread.finished to a new _cleanup_camera that removes frame data and deletes worker/thread QObjects, and enhance stop() to wait, terminate, and log/retain references if threads refuse to terminate (avoids use-after-free/segfaults). Adjust per-camera shutdown checks and ensure frames/timestamps are cleared only after safe termination. In video_recorder, remove premature clearing of the queue and _writer_thread during finalization to avoid unsafe cleanup while writer thread may still be exiting.
* Update dlc_processor.py
* Add lifecycle management & robust writer shutdown
Introduce lifecycle locking and an "abandoned" state to VideoRecorder to prevent unsafe restarts and improve shutdown handling. start() now acquires a lifecycle lock, refuses to restart when a leftover thread is marked abandoned, and initializes/reset writer state inside the lock. stop() now marks the recorder as abandoned and records an encode error if the writer thread fails to terminate within the timeout, and ensures timestamps are saved. The writer loop was simplified to break directly on sentinel or encode errors and now closes WriteGear during finalization with exception handling. Miscellaneous reordering and small bookkeeping fixes were made to improve safety and logging during shutdown.
* Improve lifecycle handling and stale-writer cleanup
dlclivegui/services/dlc_processor.py: Move the frame copy and enqueue timestamp into the lifecycle lock to avoid unnecessary copying when the processor is stopped; on initialization errors set WorkerState.FAULTED under the lifecycle lock before emitting the error.
dlclivegui/services/multi_camera_controller.py: Reformat the running-cameras shutdown check into a multi-line condition (keeps the same intent of detecting normal shutdown when no cameras are started and all threads are stopped).
dlclivegui/services/video_recorder.py: Add best-effort cleanup for a stale video writer in start() by attempting to call its close() and logging any errors before clearing writer/queue/thread references. Wrap stop() logic with the lifecycle lock to avoid races when stopping, preserve timeout/abandon behavior for a non-terminating writer thread, and ensure writer closure and timestamp saving are handled with proper error logging.
* Cleanup stale camera workers on start/shutdown
Detect and remove stale camera worker/thread state when starting cameras and during shutdown. _start_camera now checks the existing thread for a cam_id and calls _cleanup_camera if the worker exists but the thread is missing or not running, and updates the log message to indicate the camera is already running. The shutdown loop likewise invokes _cleanup_camera for None or not-running threads and after successful quit/terminate waits to ensure worker/thread entries are cleared. These changes prevent stale entries from blocking restarts and ensure cleaner shutdown behavior.
* Make stop join timeout configurable and add test
Introduce STOP_JOIN_TIMEOUT (5s) and use it for the writer thread join timeout instead of a hardcoded value. Reorder lifecycle checks in VideoRecorder.start to raise if the recorder is marked _abandoned before short-circuiting on is_running, preventing accidental restarts of abandoned recorders. Add a BlockingWriteGear fake, fixture, and a test (test_stop_timeout_marks_abandoned_and_prevents_restart) that forces a stop timeout, verifies the recorder is marked abandoned, prevents restart, and performs cleanup of the blocked writer thread.
* Fix locking, camera check, and writer finalization
Minimize time holding the DLCLiveProcessor lifecycle lock by moving expensive operations (frame.copy and timestamp handling) outside the lock and re-checking state before starting the worker to avoid races. Correct a duplicated condition in MultiCameraController so normal shutdown is detected using self._running. Move writer cleanup into a dedicated _finalize_writer(), add a warning when Queue.task_done() is called too many times, and centralize writer close/exception handling.
* Add background reaper for stalled DLC worker
Introduce STOP_WORKER_TIMEOUT and a background reaper to handle worker threads that don't terminate within the stop timeout. Increase join timeout to STOP_WORKER_TIMEOUT, set _pending_reset when a reset is requested while the worker is alive, and schedule _schedule_reap to join the thread in the background instead of immediately marking the processor as FAULTED. The reaper performs final cleanup (clearing thread, queue, state, stop event) and applies a pending DLCLive reset once the thread is reaped, with guards to ensure only one reaper runs at a time.
* Cache _queue to local variable in VideoRecorder
Cache self._queue to a local variable (q) and use it for subsequent operations to avoid races and repeated attribute access. Remove the redundant assert and simplify qsize lookup by relying on the cached reference. This makes queue checks and put/qsize calls consistent in a multithreaded context and prevents potential None/attribute changes between checks and use.
* Prevent configure while processor running
Add a lifecycle lock and guard in DLCLiveProcessor.configure to ensure the processor is stopped before reconfiguration. The method now acquires self._lifecycle_lock and raises a RuntimeError if the processor state is not WorkerState.STOPPED, preventing concurrent/race-condition updates to settings or processor while running.
* Cleanup abandoned recorder and set pending reset
Ensure proper cleanup and restartability when writer threads time out: VideoRecorder.stop now clears abandoned state and associated resources if the writer thread has already exited, allowing the recorder to be fully stopped and restarted. DLCLiveProcessor: moved settings/processor assignment under the lifecycle lock, and mark _pending_reset when shutdown cannot stop the worker thread so the instance can be reaped later. Minor comment tweak for STOP_WORKER_TIMEOUT semantics. Tests updated to assert abandoned is cleared and that recorder can be restarted after the writer exits.
* Refactor VideoRecorder.stop lifecycle handling
Detect and early-return when recorder is already stopped or recovered from an abandoned state, and reduce the time holding the lifecycle lock by moving queue/thread/writer shutdown work outside the lock. Ensure sentinel is enqueued, writer thread is joined with timeout (marking recorder as abandoned and recording an encode_error if it remains alive), close the writer cleanly, save timestamps, and finally clear lifecycle fields under the lock. Improves concurrency, cleanup correctness, and prevents restarting a recorder that was marked abandoned.
* Disable profiling for now
* Log late DLC cleanup; save recorder timestamps
Add a warning when the DLC worker thread is cleaned up after a timeout to make late reaping visible in logs. When VideoRecorder fails to stop within the timeout, ensure the recorder is marked abandoned under the lifecycle lock and save timestamps before returning; expand the critical log to note that timestamps were saved (but may be incomplete). These changes improve observability and help preserve timing data for debugging.
* Handle force-stop with leftover camera threads
When a forced stop finds camera threads still running, also clear failed camera state, buffered frames and timestamps (under the frame lock), and reset expected cameras to 0. Add a critical log warning that background camera threads may still be active and emit all_stopped so the GUI can leave the running state; warn that an application restart may be required. This prevents stale state from blocking future starts while making the risk visible in logs.
* Remove direct writer.close() call
Stop capturing and closing the WriteGear writer from the shutdown path. The change removes the local `writer` assignment and the try/except block that called `writer.close()`, avoiding potential double-close/race-condition exceptions during shutdown. The recorder closure is handled by the finally block calling _finalize_writer
* Clarify timeout comment and camera log message
Tweak wording for clarity in two places: update STOP_WORKER_TIMEOUT comment to better describe waiting in the STOPPING state before scheduling background reaping, and reflow the MultiCameraController critical log message into a single, spaced string to avoid awkward concatenation. These are documentation/logging refinements only and do not change runtime behavior.
Files modified:
- dlclivegui/services/dlc_processor.py: adjust timeout comment
- dlclivegui/services/multi_camera_controller.py: reformat and clarify critical log message1 parent 5b5d04c commit 5cdaa09
6 files changed
Lines changed: 508 additions & 139 deletions
File tree
- dlclivegui
- gui
- services
- tests/services
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1376 | 1376 | | |
1377 | 1377 | | |
1378 | 1378 | | |
1379 | | - | |
| 1379 | + | |
1380 | 1380 | | |
1381 | 1381 | | |
1382 | 1382 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
139 | 139 | | |
140 | 140 | | |
141 | 141 | | |
142 | | - | |
| 142 | + | |
143 | 143 | | |
144 | 144 | | |
145 | 145 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
24 | | - | |
25 | | - | |
26 | | - | |
| 24 | + | |
27 | 25 | | |
28 | 26 | | |
29 | 27 | | |
| |||
34 | 32 | | |
35 | 33 | | |
36 | 34 | | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
37 | 39 | | |
38 | 40 | | |
39 | 41 | | |
40 | 42 | | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
41 | 51 | | |
42 | 52 | | |
43 | 53 | | |
| |||
140 | 150 | | |
141 | 151 | | |
142 | 152 | | |
| 153 | + | |
143 | 154 | | |
144 | 155 | | |
145 | 156 | | |
| 157 | + | |
146 | 158 | | |
147 | 159 | | |
| 160 | + | |
| 161 | + | |
148 | 162 | | |
149 | 163 | | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
150 | 167 | | |
151 | 168 | | |
152 | 169 | | |
| |||
169 | 186 | | |
170 | 187 | | |
171 | 188 | | |
172 | | - | |
173 | | - | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
174 | 194 | | |
175 | 195 | | |
176 | 196 | | |
177 | | - | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
178 | 205 | | |
179 | 206 | | |
180 | 207 | | |
| |||
191 | 218 | | |
192 | 219 | | |
193 | 220 | | |
194 | | - | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
195 | 229 | | |
196 | 230 | | |
197 | 231 | | |
198 | 232 | | |
199 | | - | |
200 | | - | |
201 | | - | |
202 | | - | |
203 | | - | |
204 | | - | |
205 | | - | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
206 | 259 | | |
207 | 260 | | |
208 | 261 | | |
209 | | - | |
| 262 | + | |
210 | 263 | | |
211 | 264 | | |
212 | 265 | | |
| |||
264 | 317 | | |
265 | 318 | | |
266 | 319 | | |
267 | | - | |
| 320 | + | |
| 321 | + | |
268 | 322 | | |
269 | 323 | | |
270 | | - | |
271 | 324 | | |
272 | 325 | | |
| 326 | + | |
273 | 327 | | |
274 | 328 | | |
275 | 329 | | |
| |||
278 | 332 | | |
279 | 333 | | |
280 | 334 | | |
281 | | - | |
282 | | - | |
283 | | - | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
284 | 363 | | |
285 | | - | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
286 | 369 | | |
287 | | - | |
288 | | - | |
289 | | - | |
290 | | - | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
291 | 388 | | |
292 | | - | |
293 | | - | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
294 | 396 | | |
295 | 397 | | |
296 | 398 | | |
| |||
333 | 435 | | |
334 | 436 | | |
335 | 437 | | |
| 438 | + | |
| 439 | + | |
336 | 440 | | |
337 | 441 | | |
338 | 442 | | |
| |||
382 | 486 | | |
383 | 487 | | |
384 | 488 | | |
385 | | - | |
386 | | - | |
387 | 489 | | |
388 | 490 | | |
389 | 491 | | |
| |||
408 | 510 | | |
409 | 511 | | |
410 | 512 | | |
411 | | - | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
412 | 525 | | |
413 | 526 | | |
414 | 527 | | |
| |||
421 | 534 | | |
422 | 535 | | |
423 | 536 | | |
| 537 | + | |
| 538 | + | |
424 | 539 | | |
425 | 540 | | |
426 | 541 | | |
| |||
436 | 551 | | |
437 | 552 | | |
438 | 553 | | |
| 554 | + | |
| 555 | + | |
439 | 556 | | |
440 | 557 | | |
441 | 558 | | |
442 | 559 | | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
443 | 570 | | |
444 | 571 | | |
445 | 572 | | |
446 | 573 | | |
447 | 574 | | |
448 | | - | |
| 575 | + | |
449 | 576 | | |
450 | | - | |
| 577 | + | |
451 | 578 | | |
452 | 579 | | |
453 | 580 | | |
| |||
460 | 587 | | |
461 | 588 | | |
462 | 589 | | |
463 | | - | |
| 590 | + | |
464 | 591 | | |
465 | 592 | | |
466 | 593 | | |
467 | 594 | | |
468 | 595 | | |
469 | 596 | | |
470 | 597 | | |
471 | | - | |
| 598 | + | |
472 | 599 | | |
473 | 600 | | |
474 | 601 | | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
475 | 608 | | |
476 | 609 | | |
477 | 610 | | |
| |||
481 | 614 | | |
482 | 615 | | |
483 | 616 | | |
484 | | - | |
| 617 | + | |
485 | 618 | | |
486 | 619 | | |
487 | 620 | | |
| |||
518 | 651 | | |
519 | 652 | | |
520 | 653 | | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
521 | 658 | | |
522 | 659 | | |
523 | 660 | | |
| |||
531 | 668 | | |
532 | 669 | | |
533 | 670 | | |
534 | | - | |
535 | 671 | | |
536 | 672 | | |
537 | 673 | | |
538 | | - | |
539 | 674 | | |
540 | 675 | | |
541 | 676 | | |
| |||
0 commit comments