prep v0.5.9 release docs (#75)

* prep v0.5.9 release docs

* update base image

* adjust mapping update debug info

Author: bbilly1

Dockerfile

@@ -1,6 +1,6 @@
 # build the docs and load static files into nginx
 
-FROM python:3.11.13-slim-bookworm AS builder
+FROM python:3.13.11-slim-trixie AS builder
 ENV PATH=/root/.local/bin:$PATH
 
 RUN apt-get update -y && apt-get install -y libcairo2

mkdocs/docs/advanced.md

@@ -88,11 +88,13 @@ ES will not start, if the data is corrupted. So, stop all containers, delete eve
 In general the mapping update process is as follows:
 
 1. Compare existing mapping with predefined expected mapping
-	1. If that is identical, there is nothing to do
-1. Else create a `_backup` of the existing index
-1. Delete the original index and create a new empty one with the new mapping in place
-1. Copy over the previously created `_backup` index to apply the new mappings
-1. Delete the now leftover `_backup` index.
+	- If that is identical, there is nothing to do
+2. If the difference is just adding a new field, that is simply added in place
+3. If the difference is a change in how an existing field is indexed, that needs a index rebuild:
+	- Reindex into a new index by appending a version to the index nummer, e.g. `ta_video_v2`.
+	- That will also remove any no longer needed fields.
+	- Delete the old index
+	- Create an alias to point all new requests to the new version of the index
 
 If you are not sure if anything is happening, you can monitor your index and `docs.count` value for each index. Those values should change over time during the process and you should get an indicator of progress happening:
 
@@ -102,23 +104,6 @@ From within the ES container:
 curl -u elastic:$ELASTIC_PASSWORD "localhost:9200/_cat/indices?v&s=index"
 ```
 
-If that process gets interrupted before deleting the `_backup` index and you try to run this again, you will see an error like `resource_already_exists_exception`, for example `index [ta_comment_backup/...] already exists` indicating in this case that your migration previously failed for the `ta_comment` index.
-
-First, make sure you still have the original index with the command above. After verifying, stop the TA container, then you can delete the `_backup` index, e.g. in the case of `ta_comment_backup`.
-
-```bash
-curl -XDELETE -u elastic:$ELASTIC_PASSWORD "localhost:9200/ta_comment_backup?pretty"
-```
-
-and you should get:
-```json
-{
-  "acknowledged" : true
-}
-```
-
-Then you can restart the container and the migration will run again. If your error persists, the ES and TA logs should give additional debug info.
-
 ## Manual yt-dlp update
 !!! warning 
 	Doing this is **very likely** going to break things for you. You will want to try this out on a testing instance first. Regularly there have been subtle changes in the yt-dlp API, so only do this if you know how to debug this project by yourself, but obviously share your fixes so any problems can be dealt with before release.

mkdocs/docs/settings/actions.md

@@ -9,22 +9,32 @@ Accessible at `/settings/actions/` of your **Tube Archivist** instance, this pag
 !!! warning
     This is inherently error prone, as there are many variables, some outside of the control of this project. Read this carefully and use at your own risk.
 
-!!! note
-    The importer will *try* to fetch metadata from YouTube after identifying the video, as it is assumed this is newer, even if you add the info json file as described below.
-
 Add the files you'd like to import to the `/cache/import` folder. Only add files, don't add subdirectories. All files you are adding need to have the same *base name* as the media file. Then, start the process from the settings page with the *Manual Media Files Import* button.
 
 Valid media extensions are *.mp4*, *.mkv* or *.webm*. If you have other file extensions or incompatible codecs, convert them first to mp4. **Tube Archivist** can identify the videos with one of the following methods:
 
-### Method 1:
+### Prefer embedded metadata
+If you activate that, the import process will prioritize the [embedded metadata](application.md/#embed-metadata) and will use that if available and will not try to fetch from YT directly. That can be advantageous if you know that data is fresh and trusted.  
+
+Otherwise TA will try to prioritize remote metadata from YT.
+
+### Ignore missing metadata errors
+The import task will stop with an error if there is no metadata available:
+
+- No matching info json file
+- No embedded metadata
+- Not available on YT anymore
+
+That is a core failure of the task. If you still want to continue, activate this option. You'll probably want to remove the file from `/cache/import` after the task finishes.
+
+### Identify by info.json file
 Add a matching *.info.json* file with the media file. Both files need to have the same base name, for example:
 
 - For the media file: `<base-name>.mp4`
 - For the JSON file: `<base-name>.info.json`
 
 The import process then looks for the 'id' key within the JSON file to identify the video.
 
-
 Sometimes you may need to create this file manually. The following are the absolute minimum required tags as keys for manual importing.
 
 !!! note
@@ -55,21 +65,28 @@ However, you may fill out additional tags if they are known for a more complete
 }
 ```
 
-### Method 2:
+### Filename Based Detection:
 Detect the YouTube ID from filename. This accepts the default yt-dlp naming convention for file names like:
 
 - `<base-name>[<youtube-id>].mp4`
 
 !!! success "Required Naming Convention"
     The YouTube ID in square brackets at the end of the filename is the crucial part.
 
+### Import from other TA instances:
+!!! warning
+    Make sure you trust the source from where you import files from. The importer makes a best effort to validate the metadata and will stop if there is an error in validation.
+
+You can import media files from other **Tube Archivist** instances by putting them here as well. The nameing convention is identical, simply `<youtube-id>.mp4`. Just add the file, without the channel folder.
+
 ### Offline import:
 If the video you are trying to import is not available on YouTube any more, **Tube Archivist** can import the required metadata:
 
-- The file `<base-name>.info.json` is required to extract the required information.
+- From the `<base-name>.info.json` file.
+- Or the embedded metadata, make sure you have activated that on the settings page.
 - Add the thumbnail as `<base-name>.<ext>`, where valid file extensions are *.jpg*, *.png* or *.webp*. If there is no thumbnail file, **Tube Archivist** will try to extract the embedded cover from the media file or will fallback to a default thumbnail.
 - Add subtitles as `<base-name>.<lang>.vtt` where *lang* is the appropriate two letter ISO 639 language code. This will archive all subtitle files you add to the import folder, independent from your configurations. Subtitles can be archived and used in the player, but they can't be indexed or made searchable due to the fact, that they have a very different structure than the subtitles as **Tube Archivist** needs them.
-- For videos, where the whole channel is not available any more, TA will extract as much info as possible from the info json file.
+- For videos, where the whole channel is not available any more, TA will extract as much info as possible from the info json file and create the channel dynamically.
 
 ### Some notes:
 
@@ -78,14 +95,11 @@ If the video you are trying to import is not available on YouTube any more, **Tu
 - A notification box will show with progress, follow the docker logs to monitor for errors.
 
 !!! tip "Start Small"
-    Starting with a small subset of the files to import to test and confirm that your settings, configurations and files will work as recommended.
-
-## Embed thumbnails into media file
-This will write or overwrite all thumbnails in the media file using the downloaded thumbnail. This is only necessary if you didn't download the files with the option *Embed Thumbnail* enabled or you want to make sure all media files get the newest thumbnail.
+    Starting with a small subset of the files to import to test and confirm that your settings, configurations and files will work as expected.
 
 ## Embed metadata into media file
 
-This starts a background task adding metadata into the media file. Is is meant to update your existing mediafiles. Also see [application/embed-metadata](application.md/#embed-metadata).
+This starts a background task adding metadata and artwork into the media file. Is is meant to update your existing mediafiles. Also see [application/embed-metadata](application.md/#embed-metadata).
 
 !!! tip "This is slow"
 
@@ -95,7 +109,7 @@ This starts a background task adding metadata into the media file. Is is meant t
     This task will fail if there are any corrupted media files. That can happen due to various reasons, like filesystem, hardware or interrupted transfers. Or specifically, if yt-dlp and ffmpeg didn't download and merge a valid mp4 file.
 
     - This is not a bug, but a happy accident, as that informs you of any problems in the media file that might have gone unnoticed otherwise.
-    - Changes are, a redownload will fix that. Then start the task again.
+    - Chances are, a redownload will fix that. Then start the task again.
 
 ## ZIP file index backup
 This will backup your metadata into a zip file. The file will get stored at `/cache/backup` and will contain the necessary files to restore the Elasticsearch index formatted **nd-json** files. For data consistency, make sure there aren't any other tasks running that will change the index during the backup process. This is very slow, particularly for large archives.
@@ -120,8 +134,13 @@ This action will go through all your media files and looks at the whole index to
 - Should the filename not match with the indexed media url, this will rename the video files correctly and update the index with the new link.
 - When you delete media files from the filesystem outside of the **Tube Archivist** interface, this will delete leftover metadata from the index.
 - When you have media files that are not indexed yet, this will grab the metadata from YouTube as if it was a newly downloaded video. This can be useful when restoring from an older backup file with missing metadata but already downloaded mediafiles. NOTE: This only works if the media files are named in the same convention as **Tube Archivist** expects, alternatively see above for *Manual Media Files Import*.
-- The task will stop when adding a video fails, for example if the video is no longer available on YouTube.
 - This will also check all of your thumbnails and download any that are missing.
 
 !!! danger "BE AWARE"
     There is no undo. Deleted references and metadata are removed and cannot be brought back without a restore operation.
+
+### Prefer embedded metadata
+This this trigger indexing a new video, if you enable this option, [embedded metadata](application.md/#embed-metadata) will be prefered and, if available, the task will not be fetched again from YT.
+
+### Ignore missing metadata errors
+The task will stop when adding a video fails, for example if the video is no longer available on YouTube and no embedded metadata was found so you can take appropriate actions. Activate this option to still continue the task, monitor the logs for the details.

mkdocs/docs/settings/application.md

@@ -71,6 +71,7 @@ This saves:
 - Title
 - Artist (Channel Name)
 - Description
+- Artwork: Cover art, channel art and playlist art
 - `ta`: That's the TA metadata as indexed.
 
 The `ta` tag is a json object and contains the complete metadata as indexed in TA. That can be advantageous to embed directly in the file, e.g. for data recovery, portability or reusing of the media files.
@@ -81,6 +82,7 @@ This includes, based on your configurations:
 - **Comments**: All comments
 - **Subtitles**: These are the full text segments as indexed and optimized for searching
 - **Playlists**: Full playlist metadata if the video is part of any Playlist/s
+- **Version**: TA version number of the embedded metadata format
 
 ??? "Examples accessing `ta` metadata"
     Using ffprobe:
@@ -108,9 +110,6 @@ This includes, based on your configurations:
 
 Also see [settings/embed-metadata-into-media-file](actions.md/#embed-metadata-into-media-file) to update metadata into existing files.
 
-### Embed Thumbnail
-This saves the thumbnail into the media file by passing `--embed-thumbnail` to yt-dlp.
-
 ## Subtitles
 
 ### Subtitle Language
@@ -129,15 +128,15 @@ Set your configuration for downloading and indexing comments. This takes the sam
 
 Example:  
 
-- `all,100,all,30`: Get 100 max-parents and 30 max-replies-per-thread.
-- `1000,all,all,50`: Get a total of 1000 comments over all, 50 replies per thread.
+- `all,100,all,30,all`: Get 100 max-parents and 30 max-replies-per-thread at any depth.
+- `1000,all,all,50,2`: Get a total of 1000 comments over all, 50 replies per thread only 2 levels of depth.
 
 ### Comment sort method
 Change sort method between *top* or *new*. The default is *top*, as decided by YouTube.
 
 - The [Refresh Metadata](scheduling.md#refresh-metadata) background task will get comments from your already archived videos, spreading the requests out over time.
 
-Archiving comments is slow as only a few comments get returned per request with yt-dlp. Choose your configuration above wisely. Tube Archivist will download comments after the download queue finishes. Your videos will already be available while the comments are getting downloaded.
+Archiving comments is slow as only a few comments get returned per request with yt-dlp. Choose your configuration above wisely. Tube Archivist will download comments after the download queue finishes during post processing. Your videos will already be available while the comments are getting downloaded.
 
 ## Cookie
 

requirements.txt

@@ -1,5 +1,5 @@
 cairosvg==2.8.2
-mkdocs-material==9.6.23
+mkdocs-material==9.7.1
 mkdocs-swagger-ui-tag==0.7.2
 mkdocs==1.6.1
-pillow==12.0.0
+pillow==12.1.0