feat: Add auto-duplicate resolution with task cancellation and fix critical deadlocks

Major Features:
- Auto-duplicate resolution: 6 strategies (newest, oldest, merge, highest_quality_format, most_metadata, largest_file_size)
- Automatic cancellation of pending tasks and scheduled jobs when books deleted by resolution
- Settings UI for enabling/configuring auto-resolution with cooldown periods
- Enhanced Duplicates Manager UI with clickable book covers, titles, and Edit/Archive buttons

Performance Fixes:
- Fixed critical application hang: Pass pre-scanned duplicate groups to auto_resolve_duplicates() to avoid expensive re-scan
- Fixed deadlock in cancel_tasks_for_book(): Access queue/dequeued directly instead of using .tasks property to prevent recursive lock
- Optimized incremental scan to include last scanned book (>= instead of >)

Implementation Details:
- cps/duplicates.py: auto_resolve_duplicates() with dry-run preview, backup, deletion, and audit logging
- cps/tasks/duplicate_scan.py: Pass found_duplicate_groups to resolution, added comprehensive debug logging
- cps/services/worker.py: cancel_tasks_for_book() method with deadlock prevention
- scripts/cwa_db.py: scheduled_cancel_for_book() to cancel pending auto-send/scheduled jobs
- cps/templates/duplicates.html: Fixed blueprint endpoints, added clickable UI elements
- cps/templates/cwa_settings.html: Uncommented and fixed auto-resolution settings section

Bug Fixes:
- Fixed template crash from wrong blueprint endpoint ('editbook' vs 'edit-book')
- Fixed settings page overwriting format lists with duplicate_auto_resolve_cooldown_minutes
- Fixed permission errors by bypassing user context check for automatic deletions
- Fixed SQL query debugging output for hybrid prefilter

Author: crocodilestick

CONTRIBUTORS

@@ -314,7 +314,7 @@ Copyright (C) 2024-2026 Calibre-Web Automated contributors
 - zhiyue (1 commits)
 # Fork Contributors (crocodilestick/calibre-web-automated)
 
-- crocodilestick (946 commits)
+- crocodilestick (948 commits)
 - jmarmstrong1207 (73 commits)
 - demitrix (30 commits)
 - sirwolfgang (29 commits)

cps/cwa_functions.py

@@ -619,12 +619,13 @@ def set_cwa_settings():
     boolean_settings = []
     string_settings = []
     list_settings = []
-    integer_settings = ['ingest_timeout_minutes', 'auto_send_delay_minutes', 'hardcover_auto_fetch_batch_size', 'hardcover_auto_fetch_schedule_hour', 'duplicate_scan_hour', 'duplicate_scan_chunk_size', 'duplicate_scan_debounce_seconds']  # Special handling for integer settings
+    integer_settings = ['ingest_timeout_minutes', 'auto_send_delay_minutes', 'hardcover_auto_fetch_batch_size', 'hardcover_auto_fetch_schedule_hour', 'duplicate_scan_hour', 'duplicate_scan_chunk_size', 'duplicate_scan_debounce_seconds', 'duplicate_auto_resolve_cooldown_minutes']  # Special handling for integer settings
     float_settings = ['hardcover_auto_fetch_min_confidence', 'hardcover_auto_fetch_rate_limit']  # Special handling for float settings
     json_settings = ['metadata_provider_hierarchy', 'metadata_providers_enabled', 'duplicate_format_priority']  # Special handling for JSON settings
+    skip_settings = ['auto_convert_ignored_formats', 'auto_ingest_ignored_formats', 'auto_convert_retained_formats']  # Handled through individual format checkboxes
 
     for setting in cwa_default_settings:
-        if setting in integer_settings or setting in float_settings or setting in json_settings:
+        if setting in integer_settings or setting in float_settings or setting in json_settings or setting in skip_settings:
             continue  # Handle separately
         elif isinstance(cwa_default_settings[setting], int):
             boolean_settings.append(setting)

cps/duplicates.py

@@ -149,6 +149,52 @@ def end_task(self, task_id):
             if str(task.id) == str(task_id) and task.is_cancellable:
                 task.stat = STAT_CANCELLED if task.stat == STAT_WAITING else STAT_ENDED
 
+    def cancel_tasks_for_book(self, book_id):
+        """Cancel all pending tasks associated with a specific book ID
+        
+        Args:
+            book_id: The book ID whose tasks should be cancelled
+            
+        Returns:
+            int: Number of tasks cancelled
+        """
+        cancelled_count = 0
+        ins = self.get_instance()
+        
+        try:
+            with ins.doLock:
+                # Access queue and dequeued directly to avoid recursive lock from .tasks property
+                tasks_snapshot = list(ins.queue.to_list() + ins.dequeued)
+        except Exception as e:
+            log.warning("[worker] Could not get tasks snapshot: %s", str(e))
+            return 0
+        
+        # Process outside the lock to avoid deadlock
+        tasks_to_cancel = []
+        for queued_task in tasks_snapshot:
+            task = queued_task.task
+            # Check if task has a book_id attribute and it matches
+            if hasattr(task, 'book_id') and task.book_id == book_id:
+                # Only cancel if task is waiting or scheduled
+                if task.stat in (STAT_WAITING,) and task.is_cancellable:
+                    tasks_to_cancel.append((task, 'book_id'))
+            # Also check for scheduled tasks with bookId attribute (some tasks use different naming)
+            elif hasattr(task, 'bookId') and task.bookId == book_id:
+                if task.stat in (STAT_WAITING,) and task.is_cancellable:
+                    tasks_to_cancel.append((task, 'bookId'))
+        
+        # Cancel tasks without holding the main lock
+        for task, attr_name in tasks_to_cancel:
+            try:
+                task.stat = STAT_CANCELLED
+                task.error = f"Cancelled: Book {book_id} was removed from library"
+                log.info("[worker] Cancelled task %s for book %s", task.name, book_id)
+                cancelled_count += 1
+            except Exception as e:
+                log.warning("[worker] Failed to cancel task %s: %s", task.name, str(e))
+        
+        return cancelled_count
+
 
 class CalibreTask:
     __metaclass__ = abc.ABCMeta

cps/services/worker.py

@@ -74,6 +74,9 @@ def run(self, worker_thread):
             if self.full_scan:
                 duplicate_groups = find_duplicate_books(include_dismissed=False, user_id=self.user_id)
                 self.result_count = len(duplicate_groups)
+                
+                # Store the duplicate groups for passing to auto-resolution
+                self.found_duplicate_groups = duplicate_groups
 
                 if self.stat in (STAT_CANCELLED, STAT_ENDED):
                     return
@@ -104,6 +107,11 @@ def run(self, worker_thread):
                 last_scanned_book_id = int(cache_data.get('last_scanned_book_id') or 0)
                 candidate_ids = find_duplicate_candidate_ids_sql(use_title, use_author, user_id=self.user_id,
                                                                 min_book_id=last_scanned_book_id)
+                
+                log.debug("[cwa-duplicates] Incremental scan: last_scanned_book_id=%s, candidate_ids=%s",
+                         last_scanned_book_id, len(candidate_ids) if candidate_ids else 0)
+                print(f"[cwa-duplicates] Incremental scan: last_scanned_book_id={last_scanned_book_id}, "
+                      f"candidates={len(candidate_ids) if candidate_ids else 0}", flush=True)
 
                 max_book_id = 0
                 try:
@@ -175,17 +183,113 @@ def run(self, worker_thread):
                         log.warning("[cwa-duplicates] Failed to update incremental cache: %s", str(ex))
 
                     # Result count is unresolved duplicates for this run (exclude dismissed)
-                    self.result_count = len(find_duplicate_books_python(
+                    unresolved_in_candidates = find_duplicate_books_python(
                         use_title, use_author, use_language, use_series, use_publisher, use_format,
                         include_dismissed=False, user_id=self.user_id, candidate_ids=candidate_ids
-                    ))
+                    )
+                    self.result_count = len(unresolved_in_candidates)
+                    
+                    # Store the duplicate groups for passing to auto-resolution
+                    self.found_duplicate_groups = unresolved_in_candidates
+                    
+                    log.debug("[cwa-duplicates] Incremental scan result: %s unresolved groups among candidates",
+                             self.result_count)
+                    print(f"[cwa-duplicates] Incremental scan result: {self.result_count} unresolved duplicate groups", 
+                          flush=True)
 
             self.progress = 1
             if self.full_scan:
                 self.message = N_('Duplicate scan completed: %(count)s groups', count=self.result_count)
             else:
                 self.message = N_('Duplicate scan completed: %(count)s new groups', count=self.result_count)
             self._handleSuccess()
+            
+            # Check if auto-resolution is enabled
+            log.debug("[cwa-duplicates] Scan complete. result_count=%s, trigger_type=%s", 
+                     self.result_count, self.trigger_type)
+            print(f"[cwa-duplicates] Scan complete: {self.result_count} groups found, trigger_type={self.trigger_type}", 
+                  flush=True)
+            
+            if self.result_count > 0:  # Only if duplicates were found
+                try:
+                    auto_resolve_enabled = cwa_db.cwa_settings.get('duplicate_auto_resolve_enabled', 0)
+                    auto_resolve_strategy = cwa_db.cwa_settings.get('duplicate_auto_resolve_strategy', 'newest')
+                    cooldown_minutes = int(cwa_db.cwa_settings.get('duplicate_auto_resolve_cooldown_minutes', 0))
+                    
+                    log.debug("[cwa-duplicates] Auto-resolution settings: enabled=%s, strategy=%s, cooldown=%s min",
+                             auto_resolve_enabled, auto_resolve_strategy, cooldown_minutes)
+                    print(f"[cwa-duplicates] Auto-resolution settings: enabled={auto_resolve_enabled}, "
+                          f"strategy={auto_resolve_strategy}, cooldown={cooldown_minutes} min", flush=True)
+                    
+                    if auto_resolve_enabled:
+                        # Check cooldown period
+                        if cooldown_minutes > 0:
+                            try:
+                                last_resolution = cwa_db.cur.execute("""
+                                    SELECT MAX(timestamp) FROM cwa_duplicate_resolutions 
+                                    WHERE trigger_type='automatic'
+                                """).fetchone()[0]
+                                
+                                if last_resolution:
+                                    from datetime import datetime, timedelta
+                                    last_time = datetime.fromisoformat(last_resolution)
+                                    now = datetime.now()
+                                    elapsed = (now - last_time).total_seconds() / 60
+                                    
+                                    if elapsed < cooldown_minutes:
+                                        remaining = cooldown_minutes - elapsed
+                                        log.info("[cwa-duplicates] Auto-resolution skipped due to cooldown: %.1f minutes remaining", 
+                                                remaining)
+                                        print(f"[cwa-duplicates] Auto-resolution on cooldown ({remaining:.1f} min remaining)", 
+                                              flush=True)
+                                        return
+                            except Exception as e:
+                                log.warning("[cwa-duplicates] Cooldown check failed: %s", str(e))
+                        
+                        log.info("[cwa-duplicates] Auto-resolution enabled, triggering resolution with strategy: %s", 
+                                auto_resolve_strategy)
+                        print(f"[cwa-duplicates] Auto-resolution enabled, triggering with strategy: {auto_resolve_strategy}", 
+                              flush=True)
+                        
+                        from cps.duplicates import auto_resolve_duplicates
+                        
+                        # Pass the pre-scanned duplicate groups to avoid re-scanning
+                        groups_to_pass = getattr(self, 'found_duplicate_groups', None)
+                        log.debug("[cwa-duplicates] Passing %s groups to auto_resolve (type: %s)", 
+                                 len(groups_to_pass) if groups_to_pass else 'None', type(groups_to_pass).__name__)
+                        print(f"[cwa-duplicates] Passing {len(groups_to_pass) if groups_to_pass else 'None'} pre-scanned groups to auto_resolve", 
+                              flush=True)
+                        
+                        result = auto_resolve_duplicates(
+                            strategy=auto_resolve_strategy,
+                            dry_run=False,
+                            user_id=None,
+                            trigger_type='automatic',
+                            duplicate_groups=groups_to_pass
+                        )
+                        
+                        if result['success']:
+                            log.info("[cwa-duplicates] Auto-resolution completed: resolved=%s, kept=%s, deleted=%s",
+                                    result['resolved_count'], result['kept_count'], result['deleted_count'])
+                            print(f"[cwa-duplicates] Auto-resolution completed: {result['resolved_count']} groups resolved, "
+                                  f"{result['deleted_count']} books deleted, {result['kept_count']} books kept", flush=True)
+                            
+                            self.message = N_('Duplicate scan completed: %(count)s groups auto-resolved', 
+                                            count=result['resolved_count'])
+                        else:
+                            log.warning("[cwa-duplicates] Auto-resolution completed with errors: %s", 
+                                       result.get('errors', []))
+                            print(f"[cwa-duplicates] Auto-resolution errors: {result.get('errors', [])}", flush=True)
+                    else:
+                        log.debug("[cwa-duplicates] Auto-resolution disabled in settings")
+                        print("[cwa-duplicates] Auto-resolution disabled in settings", flush=True)
+                except Exception as ex:
+                    log.error("[cwa-duplicates] Exception during auto-resolution check: %s", str(ex))
+                    print(f"[cwa-duplicates] Exception during auto-resolution check: {str(ex)}", flush=True)
+            else:
+                log.debug("[cwa-duplicates] No duplicates found, skipping auto-resolution")
+                print("[cwa-duplicates] No duplicates found, skipping auto-resolution", flush=True)
+                    
         except Exception as ex:
             log.error("[cwa-duplicates] Duplicate scan task failed: %s", str(ex))
             self._handleError(str(ex))

cps/tasks/duplicate_scan.py

@@ -952,7 +952,6 @@ <h4 class="settings-section-header">{{_('Duplicate Format Priority Ranking')}}</
 
     <br>
 
-<!--
     <div class="settings-container">
       <h4 class="settings-section-header">{{_('Duplicate Notifications & Auto-Resolution')}}</h4>
 
@@ -980,46 +979,42 @@ <h4 for="duplicate_auto_resolve_strategy" class="settings-section-header">{{_('A
           <option value="newest" {% if cwa_settings.get('duplicate_auto_resolve_strategy', 'newest') == 'newest' %} selected {% endif %}>
             {{_('Keep Newest')}} - {{_('Delete older copies, keep the most recently added book')}}
           </option>
+          <option value="oldest" {% if cwa_settings.get('duplicate_auto_resolve_strategy') == 'oldest' %} selected {% endif %}>
+            {{_('Keep Oldest')}} - {{_('Delete newer copies, keep the earliest added')}}
+          </option>
+          <option value="merge" {% if cwa_settings.get('duplicate_auto_resolve_strategy') == 'merge' %} selected {% endif %}>
+            {{_('Merge Formats')}} - {{_('Merge formats into the newest book, delete the rest')}}
+          </option>
           <option value="highest_quality_format" {% if cwa_settings.get('duplicate_auto_resolve_strategy') == 'highest_quality_format' %} selected {% endif %}>
-            {{_('Keep Highest Quality Format')}} - {{_('Keep EPUB over MOBI, AZW3 over AZW, etc.')}}
+            {{_('Keep Highest Quality Format')}} - {{_('Prefer EPUB > KEPUB > AZW3 > MOBI > PDF')}}
           </option>
           <option value="most_metadata" {% if cwa_settings.get('duplicate_auto_resolve_strategy') == 'most_metadata' %} selected {% endif %}>
-            {{_('Keep Most Complete Metadata')}} - {{_('Keep the book with most tags, series info, etc.')}}
+            {{_('Keep Most Metadata')}} - {{_('Keep book with most complete information')}}
           </option>
           <option value="largest_file_size" {% if cwa_settings.get('duplicate_auto_resolve_strategy') == 'largest_file_size' %} selected {% endif %}>
-            {{_('Keep Largest File Size')}} - {{_('Keep the book with the largest file size')}}
+            {{_('Keep Largest File Size')}} - {{_('Keep book with largest total file size')}}
           </option>
         </select>
         <p class="cwa-settings-explanation settings-explanation" style="margin-top: 2rem !important;">{{_('Choose which book to keep when duplicates are automatically resolved.')}}</p>
       </div>
 
+      <div class="form-group" style="padding: 2rem; background: #151e2680;">
+        <h4 class="settings-section-header">{{_('Rate Limiting')}}</h4>
+        <label for="duplicate_auto_resolve_cooldown_minutes">{{_('Cooldown Period (minutes)')}}</label>
+        <input type="number" class="form-control" id="duplicate_auto_resolve_cooldown_minutes" 
+               name="duplicate_auto_resolve_cooldown_minutes" min="0" max="1440"
+               value="{{ cwa_settings.get('duplicate_auto_resolve_cooldown_minutes', 0) }}">
+        <p class="cwa-settings-explanation settings-explanation" style="margin-top: 1rem;">
+          {{_('Minimum time between automatic resolutions (0 to disable). Prevents rapid-fire deletions during batch imports.')}}
+        </p>
+      </div>
+
       <div class="cwa-settings-tip">
         <small class="settings-explanation">
-          <strong>{{_('Note:')}}</strong> {{_('Auto-resolution runs when the duplicates page is manually triggered. Dismissed duplicate groups are never auto-resolved.')}}
+          <strong>{{_('Note:')}}</strong> {{_('Auto-resolution runs after duplicate scans detect new duplicates. Dismissed duplicate groups are never auto-resolved.')}}
         </small>
       </div>
     </div>
--->
-
-    <!-- Duplicate Notifications -->
-    <div class="settings-container">
-      <h4 class="settings-section-header">{{_('Duplicate Notifications')}}</h4>
-
-      <p class="cwa-settings-explanation settings-explanation">
-        {{_('Configure how you are notified about duplicate books.')}}
-      </p>
-
-      <!-- Notifications Toggle -->
-      <div class="checkbox-wrapper">
-        {% if cwa_settings['duplicate_notifications_enabled'] %}
-        <input type="checkbox" id="duplicate_notifications_enabled" name="duplicate_notifications_enabled" value="True" checked style="accent-color: var(--color-secondary);">
-        {% else %}
-        <input type="checkbox" id="duplicate_notifications_enabled" name="duplicate_notifications_enabled" value="True" style="accent-color: var(--color-secondary);">
-        {% endif %}
-        <label for="duplicate_notifications_enabled">{{_('Enable Duplicate Notifications')}}</label>
-      </div>
-      <p class="cwa-settings-tip" style="font-size: small;">{{_('Show popup notifications when unresolved duplicates are detected. Admins and users with edit rights will see a badge on the Duplicates sidebar button and a notification popup when they login.')}}</p>
-    </div>
 
     <br>
 

cps/templates/cwa_settings.html

@@ -635,6 +635,29 @@ def scheduled_mark_cancelled(self, row_id: int) -> None:
         except Exception as e:
             print(f"[cwa-db] ERROR marking scheduled job cancelled: {e}")
 
+    def scheduled_cancel_for_book(self, book_id: int) -> int:
+        """Cancel all scheduled jobs (auto-send, etc.) for a specific book
+        
+        Args:
+            book_id: The book ID whose scheduled jobs should be cancelled
+            
+        Returns:
+            int: Number of jobs cancelled
+        """
+        try:
+            self.cur.execute(
+                "UPDATE cwa_scheduled_jobs SET state='cancelled' WHERE book_id=? AND state='scheduled'",
+                (int(book_id),)
+            )
+            self.con.commit()
+            cancelled_count = self.cur.rowcount
+            if cancelled_count > 0:
+                print(f"[cwa-db] Cancelled {cancelled_count} scheduled job(s) for book {book_id}", flush=True)
+            return cancelled_count
+        except Exception as e:
+            print(f"[cwa-db] ERROR cancelling scheduled jobs for book {book_id}: {e}", flush=True)
+            return 0
+
     def scheduled_update_job_id(self, row_id: int, scheduler_job_id: str) -> None:
         try:
             self.cur.execute("UPDATE cwa_scheduled_jobs SET scheduler_job_id=? WHERE id=?", (scheduler_job_id, int(row_id)))