Merge pull request #760 from sirwolfgang/kosync

KOReader Progress Syncing with Book Identification & Kobo Integration

Author: crocodilestick

.github/copilot-instructions.md

@@ -186,6 +186,7 @@ Pluggable providers in `cps/metadata_provider/`:
 - `CWA_WATCH_MODE`: Force polling watcher (`poll`) or inotify (default)
 - `HARDCOVER_TOKEN`: API key for Hardcover metadata provider
 - `COOKIE_PREFIX`: Custom prefix for session cookies
+- `TRUSTED_PROXY_COUNT`: Number of proxies to trust for X-Forwarded-* headers (default: 1, use 2+ for CF Tunnel + reverse proxy)
 
 ## Common Pitfalls
 1. **Don't import SQLite on main thread**: Always use `init_db_thread()` in background tasks
@@ -195,6 +196,7 @@ Pluggable providers in `cps/metadata_provider/`:
 5. **WAL mode errors**: Usually means network share deployment without `NETWORK_SHARE_MODE=true`
 6. **Port binding**: Ports below 1024 need `cap_add: [NET_BIND_SERVICE]` in docker-compose
 7. **Calibre plugins**: Requires `customize.py.json` in `/config/.config/calibre/` to register
+8. **Session protection errors**: Behind multiple proxies? Set `TRUSTED_PROXY_COUNT` to match your proxy chain depth
 
 ## Version Management
 - **Installed version**: `/app/CWA_RELEASE` (baked at build time)

.gitignore

@@ -166,6 +166,7 @@ venv/
 ENV/
 env.bak/
 venv.bak/
+tmp/
 
 # Spyder project settings
 .spyderproject

CONTRIBUTORS

@@ -303,7 +303,7 @@ Copyright (C) 2024-2025 Calibre-Web Automated contributors
 - zhiyue (1 commits)
 # Fork Contributors (crocodilestick/calibre-web-automated)
 
-- crocodilestick (772 commits)
+- crocodilestick (776 commits)
 - jmarmstrong1207 (73 commits)
 - demitrix (30 commits)
 - sirwolfgang (22 commits)
@@ -357,6 +357,7 @@ Copyright (C) 2024-2025 Calibre-Web Automated contributors
 - jspiers (1 commits)
 - kevpam (1 commits)
 - lazyusername (1 commits)
+- marauder37 (1 commits)
 - Marodeur80 (anon) (1 commits)
 - Matteo Benaroyo (anon) (1 commits)
 - morpheus65535 (1 commits)
@@ -371,6 +372,7 @@ Copyright (C) 2024-2025 Calibre-Web Automated contributors
 - spezzino (1 commits)
 - stadler-pascal (1 commits)
 - stefanop1 (1 commits)
+- tomried (1 commits)
 - Turmaxx (1 commits)
 - ugyes (1 commits)
 - werdeil (1 commits)

README.md

@@ -100,6 +100,16 @@ This tells CWA to avoid enabling WAL on the Calibre `metadata.db` and the `app.d
 - On Docker Desktop (Windows/macOS), the container runs on a LinuxKit/WSL2 VM and host-mounted paths may not propagate `inotify` events reliably. CWA auto-detects Docker Desktop at startup and prefers the same polling watcher for reliability.
 - Advanced: You can also force polling regardless of share mode by setting `CWA_WATCH_MODE=poll`.
 
+### Running behind multiple proxies (Cloudflare Tunnel, reverse proxy)
+
+- CWA uses Werkzeug's ProxyFix middleware to properly handle `X-Forwarded-For`, `X-Forwarded-Proto`, and other proxy headers.
+- By default, it trusts **1 proxy** in the chain. If you have multiple proxies (e.g., Cloudflare Tunnel → nginx → CWA), set:
+
+  - `TRUSTED_PROXY_COUNT=2` (or the total number of proxies in your chain)
+
+- **Why this matters**: Session protection validates requests based on the client's IP address. If ProxyFix doesn't trust enough proxies, it may see different IPs between requests, causing "Session protection triggered" warnings and forcing re-login.
+- **Troubleshooting**: If you see frequent session protection warnings in logs, check your proxy chain depth and adjust this variable accordingly.
+
 ## **_Features:_**
 
 ### CWA supports all Stock CW Features:
@@ -184,11 +194,12 @@ This tells CWA to avoid enabling WAL on the Calibre `metadata.db` and the `app.d
 - [Hardcover](https://hardcover.app/) is also currently in the process of being added to CWA as a Metadata Provider
 
 #### **KOReader Syncing (KOSync)** 📖⚡
-- CWA now includes built-in KOReader syncing functionality, providing a modern alternative to traditional KOReader sync servers
-- **Universal KOReader Syncer:** Works across all KOReader-compatible devices, storing sync data in a readable format for future CWA features
-- **Modern Authentication:** Uses RFC 7617 compliant header-based authentication instead of legacy MD5 hashing for enhanced security
-- **CWA Integration:** Leverages your existing CWA user accounts and permissions - no additional server setup required
-- **Easy Installation:** Plugin and setup instructions are available directly from your CWA instance at `/kosync`
+Built-in KOReader progress sync with automatic book identification:
+- **Book Identification:** Auto-generates KOReader-compatible partial MD5 checksums for all books
+- **Unified Progress:** Syncs KOReader → CWA reading status → Kobo devices
+- **Zero Config:** Checksums generated on startup and import, no manual setup
+- **Modern Auth:** RFC 7617 HTTP Basic Auth with existing CWA accounts
+- **Plugin Available:** Download from `/kosync` endpoint on your CWA instance
 
 #### **Enhanced OAuth 2.0/OIDC Authentication** 🔐
 - **Auto-Discovery:** Automatic endpoint configuration via OIDC metadata URLs for seamless setup with providers like Keycloak, Authentik, Google, and Azure AD
@@ -304,7 +315,7 @@ services:
       - CWA_PORT_OVERRIDE=8083
     volumes:
       # CW users migrating should stop their existing CW instance, make a copy of the config folder, and bind that here to carry over all of their user settings ect.
-      - /path/to/config/folder:/config 
+      - /path/to/config/folder:/config
       # This is an ingest dir, NOT a library one. Anything added here will be automatically added to your library according to the settings you have configured in CWA Settings page. All files placed here are REMOVED AFTER PROCESSING
       - /path/to/the/folder/you/want/to/use/for/book/ingest:/cwa-book-ingest
       # If you don't have an existing library, CWA will automatically create one at the bind provided here
@@ -395,23 +406,23 @@ CWA now includes built-in KOReader syncing functionality, allowing you to sync y
 
 ## Local Development Setup
 
-1. **Build the image**  
+1. **Build the image**
    Edit and run [`build.sh`](https://github.com/crocodilestick/Calibre-Web-Automated/blob/main/build.sh) to build a local Docker image of Calibre-Web-Automated.  See the script itself for usage details.
 
-2. **Edit [`docker-compose.yml.dev`](https://github.com/crocodilestick/Calibre-Web-Automated/blob/main/docker-compose.yml.dev)**  
-   Update at minimum:  
-   - `image:` → your image tag from step 1  
-   - `volumes mounts` → paths for config, ingest, library, plugins  
-  
+2. **Edit [`docker-compose.yml.dev`](https://github.com/crocodilestick/Calibre-Web-Automated/blob/main/docker-compose.yml.dev)**
+   Update at minimum:
+   - `image:` → your image tag from step 1
+   - `volumes mounts` → paths for config, ingest, library, plugins
+
  To have the app refresh dynamically in response to code changes, see comments in the  [`docker-compose.yml.dev`](https://github.com/crocodilestick/Calibre-Web-Automated/blob/main/docker-compose.yml.dev)** for details and examples on "live-edit" mounts.
 
-3. **Start the service**  
+3. **Start the service**
 ```
 $ docker compose -f docker-compose.yml.dev up -d
 ```
 
-4. **Log in & configure**  
-   - Sign in with the [default admin login](https://github.com/crocodilestick/Calibre-Web-Automated?tab=readme-ov-file#default-admin-login)  
+4. **Log in & configure**
+   - Sign in with the [default admin login](https://github.com/crocodilestick/Calibre-Web-Automated?tab=readme-ov-file#default-admin-login)
    - Optionally follow [Post-Install Tasks](https://github.com/crocodilestick/Calibre-Web-Automated?tab=readme-ov-file#post-install-tasks)-
 
 ---

cps/__init__.py

@@ -83,7 +83,11 @@
 
 # Fix for running behind reverse proxy (e.g. nginx, apache, caddy, ...)
 # Without it, url_for will generate http:// urls even if https:// is used
-app.wsgi_app = ProxyFix(app.wsgi_app, x_for=1, x_proto=1)
+# Set TRUSTED_PROXY_COUNT to the number of proxies in your chain (default: 1)
+# For CF Tunnel + reverse proxy, use TRUSTED_PROXY_COUNT=2
+num_proxies = int(os.environ.get('TRUSTED_PROXY_COUNT', '1'))
+app.wsgi_app = ProxyFix(app.wsgi_app, x_for=num_proxies, x_proto=num_proxies, x_host=num_proxies, x_prefix=num_proxies)
+log.info(f'ProxyFix configured to trust {num_proxies} proxy(ies) for X-Forwarded-* headers')
 
 lm = MyLoginManager()
 
@@ -120,7 +124,7 @@ def create_app():
 
     config_sql.load_configuration(ub.session, encrypt_key)
     config.init_config(ub.session, encrypt_key, cli_param)
-    
+
     # Set OAuth redirect host consistency
     if hasattr(config, 'config_oauth_redirect_host') and config.config_oauth_redirect_host:
         from urllib.parse import urlparse
@@ -220,7 +224,7 @@ def _cwa_ensure_db_session():
         except Exception:
             # Failsafe: let route-level code handle specific DB errors
             pass
-    
+
     # Load user from reverse proxy header early in request lifecycle
     # This ensures current_user resolves correctly before any code accesses user settings
     @app.before_request
@@ -229,11 +233,11 @@ def _load_reverse_proxy_user():
         Load user from reverse proxy authentication header if configured.
         Sets g.flask_httpauth_user early so that current_user proxy resolves correctly
         for user-specific settings like theme preferences.
-        
+
         This must run before any blueprint before_request handlers that access current_user.
         """
         from flask import g, request
-        
+
         if config.config_allow_reverse_proxy_header_login:
             from . import usermanagement
             user = usermanagement.load_user_from_reverse_proxy_header(request)
@@ -242,7 +246,7 @@ def _load_reverse_proxy_user():
             else:
                 # Explicitly set to None to indicate we checked but found nothing
                 g.flask_httpauth_user = None
-    
+
     from .schedule import register_scheduled_tasks, register_startup_tasks
     register_scheduled_tasks(config.schedule_reconnect)
     register_startup_tasks()

cps/db.py

@@ -376,6 +376,13 @@ def __init__(self, book):
         self.book = book
 
 
+# Import BookFormatChecksum from progress_syncing.models to keep model definition centralized
+# Import directly from models module to avoid triggering progress_syncing/__init__.py chain
+# which would cause circular import (models needs Base from db, but progress_syncing imports kosync)
+from .progress_syncing import models as _progress_models  # noqa: E402
+BookFormatChecksum = _progress_models.BookFormatChecksum
+
+
 class Books(Base):
     __tablename__ = 'books'
 
@@ -728,6 +735,10 @@ def setup_db(cls, config_calibre_dir, app_db_path):
         for inst in cls.instances:
             inst.init_session()
 
+        # Ensure progress syncing tables exist in metadata.db (book checksums)
+        from .progress_syncing.models import ensure_calibre_db_tables
+        ensure_calibre_db_tables(conn)
+
         cls._init = True
 
     def get_book(self, book_id):
@@ -1183,4 +1194,4 @@ def __init__(self, name, cat_id, rating=None):
         self.name = name
         self.id = cat_id
         self.rating = rating
-        self.count = 1
+        self.count = 1

cps/embed_helper.py

@@ -33,6 +33,32 @@ def do_calibre_export(book_id, book_format):
         _, err = p.communicate()
         if err:
             log.error('Metadata embedder encountered an error: %s', err)
+
+        # calibredb export with --template may create either:
+        # 1. A subdirectory with the template name containing the file
+        # 2. A file directly with a modified name
+
+        # First check if a subdirectory was created
+        export_dir = os.path.join(tmp_dir, temp_file_name)
+        if os.path.isdir(export_dir):
+            # Look for the book file with the specified format
+            for filename in os.listdir(export_dir):
+                if filename.lower().endswith('.' + book_format.lower()):
+                    # Found the exported file - return the directory and the filename without extension
+                    actual_filename = os.path.splitext(filename)[0]
+                    return export_dir, actual_filename
+
+            log.warning(f'No {book_format} file found in export directory: {export_dir}')
+        else:
+            # No subdirectory - look for files directly in tmp_dir
+            for filename in os.listdir(tmp_dir):
+                if filename.lower().endswith('.' + book_format.lower()):
+                    actual_filename = os.path.splitext(filename)[0]
+                    return tmp_dir, actual_filename
+
+            log.warning(f'No {book_format} file found in {tmp_dir}')
+
+        # Fallback to original behavior
         return tmp_dir, temp_file_name
     except OSError as ex:
         # ToDo real error handling

cps/helper.py

@@ -1048,6 +1048,8 @@ def save_cover_with_thumbnail_update(img, book_path, book_id=None):
 def do_download_file(book, book_format, client, data, headers):
     book_name = data.name
     download_name = filename = None
+    metadata_was_embedded = False  # Track if we embedded metadata
+
     if config.config_use_google_drive:
         # startTime = time.time()
         df = gd.getFileFromEbooksFolder(book.path, data.name + "." + book_format)
@@ -1063,8 +1065,10 @@ def do_download_file(book, book_format, client, data, headers):
                 gd.downloadFile(book.path, book_name + "." + book_format, output)
                 if book_format == "kepub" and config.config_kepubifypath:
                     filename, download_name = do_kepubify_metadata_replace(book, output)
+                    metadata_was_embedded = True
                 elif book_format != "kepub" and config.config_binariesdir:
                     filename, download_name = do_calibre_export(book.id, book_format)
+                    metadata_was_embedded = True
             else:
                 return gd.do_gdrive_download(df, headers)
         else:
@@ -1081,16 +1085,54 @@ def do_download_file(book, book_format, client, data, headers):
         if book_format == "kepub" and config.config_kepubifypath and config.config_embed_metadata:
             filename, download_name = do_kepubify_metadata_replace(book, os.path.join(filename,
                                                                                       book_name + "." + book_format))
+            metadata_was_embedded = True
         elif book_format != "kepub" and config.config_binariesdir and config.config_embed_metadata:
             filename, download_name = do_calibre_export(book.id, book_format)
+            metadata_was_embedded = True
+
+            # Rename the exported file to match the expected download name (from Content-Disposition)
+            # This ensures KOReader calculates the checksum on the same file we calculated it on
+            if filename and download_name:
+                uuid_file = os.path.join(filename, download_name + "." + book_format)
+                expected_file = os.path.join(filename, book_name + "." + book_format)
+
+                if os.path.exists(uuid_file) and uuid_file != expected_file:
+                    try:
+                        # Remove the target file if it already exists
+                        if os.path.exists(expected_file):
+                            os.remove(expected_file)
+                        # Rename UUID file to expected name
+                        os.rename(uuid_file, expected_file)
+                        download_name = book_name
+                        log.info(f'Renamed exported file to match expected name: {book_name}.{book_format}')
+                    except Exception as e:
+                        log.error(f'Failed to rename exported file: {e}')
         else:
             download_name = book_name
 
+    # Calculate and store checksum if metadata was embedded
+    if metadata_was_embedded and filename and download_name:
+        try:
+            from .progress_syncing import calculate_and_store_checksum
+
+            # Calculate checksum on the EXPORTED file (with embedded metadata)
+            # This is what KOReader actually downloads and calculates the checksum for
+            exported_file = os.path.join(filename, download_name + "." + book_format)
+
+            if os.path.exists(exported_file):
+                calculate_and_store_checksum(
+                    book_id=book.id,
+                    book_format=book_format,
+                    file_path=exported_file  # Use exported file with embedded metadata!
+                )
+        except Exception as e:
+            log.error(f"Failed to calculate/store checksum for book {book.id}: {e}")
+            # Don't fail the download if checksum calculation fails
+
     response = make_response(send_from_directory(filename, download_name + "." + book_format))
     # ToDo Check headers parameter
     for element in headers:
         response.headers[element[0]] = element[1]
-    log.info('Downloading file: {}'.format(os.path.join(filename, book_name + "." + book_format)))
     return response