Skip to content

Commit 3b13281

Browse files
authored
refactor!: rebuild the "refresh" system into expanded "sync" system (#1501)
* chore: bump version to v9.7.0 * feat: store and update file created and modified dates * refactor!: rework and expand refresh (sync) system * fix: don't lint st_birthtime * docs: update library sync documentation * fix(ui): fix banner not always updating label position * fix: fix edge cases and rework tests based on docs case table * refactor: remove legacy 'search and relink' system, fold into auto-relink * refactor(ui): make 'fix unlined entries' panel refresh button use sync banner * fix: use correct rg path in actual ripgrep process call * fix: have ripgrep ignore .gitignore files it finds * fix: block cyclical symlink traversal in internal scanner * fix: fix wcmatch not matching root directories, fix '**' to match 'zero or more' * tests: add ignore matching tests * fix: fix wcmatch matching '*' to '/' * fix: fix whitespace stripping inconsistencies in wcmatch * perf: don't check is_dir() in ripgrep with --files * perf: increase sync batch sizes * perf: skip NFD normalization for ASCII paths * fix(tests): don't test for case sensitivity on windows * refactor(ui): allow for multiple different banners at once using a stack * fix: prevent edge case where syncs would carry over between libraries
1 parent bd8810e commit 3b13281

69 files changed

Lines changed: 3322 additions & 831 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
9.15 KB
Loading

‎docs/ignore.md‎

Lines changed: 7 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -8,12 +8,6 @@ icon: material/file-document-remove
88

99
# :material-file-document-remove: Ignoring Files & Directories
1010

11-
<!-- prettier-ignore -->
12-
!!! warning "Legacy File Extension Ignoring"
13-
TagStudio versions prior to v9.5.4 use a different, more limited method to exclude or include file extensions from your library and subsequent searches. Opening a pre-exiting library in v9.5.4 or later will non-destructively convert this to the newer, more extensive `.ts_ignore` format.
14-
15-
If you're still running an older version of TagStudio in the meantime, you can access the legacy system by going to "Edit -> Manage File Extensions" in the menubar.
16-
1711
TagStudio offers the ability to ignore specific files and directories via a `.ts_ignore` file located inside your [library's](libraries.md) `.TagStudio` folder. This file is designed to use very similar [glob](<https://en.wikipedia.org/wiki/Glob_(programming)>)-style pattern matching as the [`.gitignore`](https://git-scm.com/docs/gitignore) file used by Git™[^1]. It can be edited within TagStudio or opened to edit with an external program by going to the "Edit -> Ignore Files" option in the menubar.
1812

1913
This file is only referenced when scanning directories for new files to add to your library, and does not apply to files that have already been added to your library.
@@ -131,14 +125,15 @@ The forward slash "`/`" is used as the directory separator. Separators may occur
131125
A `!` prefix before a pattern negates the pattern, allowing any files matched matched by previous patterns to be un-matched.
132126

133127
- Any matching file excluded by a previous pattern will become included again.
134-
- **It is not possible to re-include a file if a parent directory of that file is excluded.**
128+
- **An excluded parent directory can not have files within it reincluded!** (e.g. `Photos/`)
129+
- Alternatively, if one or more files within a directory is excluded (e.g. `!Photos/*.jpg`) then files within can be reincluded from there (e.g. `Photos/a.jpg`).
135130

136131
<!-- prettier-ignore-start -->
137132
=== "Example negation"
138133
```toml
139134
# All .jpg files will be ignored, except any located in the 'Photos' folder.
140135
*.jpg
141-
Photos/!*.jpg
136+
!Photos/*.jpg
142137
```
143138
=== "Escape a ! Symbol"
144139
```toml
@@ -147,6 +142,10 @@ A `!` prefix before a pattern negates the pattern, allowing any files matched ma
147142
```
148143
<!-- prettier-ignore-end -->
149144

145+
<!-- prettier-ignore -->
146+
!!! bug "Directory Exclusion Negation"
147+
TagStudio attempts to match the behavior of a `.gitignore` file 1:1, however if you don't have `ripgrep` installed on your system and TagStudio falls back to its internal pattern matcher, excluded directories can be overwritten by further negations, unlike `.gitignore` behavior. Be wary that this is **not officially supported**, and this behavior may be removed at any time.
148+
150149
---
151150

152151
### Wildcards

‎docs/libraries.md‎

Lines changed: 59 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -33,13 +33,68 @@ To create or open a [library](libraries.md), go to **File -> Open/Create Library
3333
!!! info "Legacy Library Migration"
3434
If you open a library created with TagStudio **v9.4.2 or earlier** in **[v9.5.0](changelog.md#950-march-3rd-2025) or later**, you'll be walked through a migration process that converts the old `ts_library.json` save file to the new `ts_library.sqlite` format. The original JSON file is preserved and can be easily deleted from the **View -> Library Information** panel once you're satisfied with the migration.
3535

36-
## :material-database-refresh: Refreshing Directories
36+
## :material-database-sync: Library Syncing
37+
38+
A TagStudio library gets synced with the files found in your chosen content folders and certain metadata attributes (e.g. stats) found on those files. This is a **non-destructive, read-only** process and none of your content files are moved, modified, or deleted. Syncing is indicated by a temporary progress bar, and you can continue to use TagStudio normally while syncing occurs.
39+
40+
Syncing automatically occurs when you open a library by default, and you can manually sync a library at any time by going to **File -> Sync Library** in the menubar to by pressing <kbd>Ctrl</kbd>+<kbd>R</kbd> (<kbd>⌘ Command </kbd>+<kbd>R</kbd> on macOS). If you do not wish for your library to be synced when opened, you can disable this behavior in the settings.
41+
42+
<figure markdown="span">
43+
![Settings -> Sync Library on Open](assets/settings_refresh_library_on_open.png)
44+
<figcaption>
45+
Settings -> Sync Library on Open
46+
</figcaption>
47+
</figure>
48+
49+
### :material-link-variant: Automatic Relinking
50+
51+
Unlinked entries are file entries in your TagStudio library that have become "unlinked" from their original file on disk, likely as a result of the original file being renamed, moved, or deleted. TagStudio attempts to automatically relink any of these entries as a part of the syncing process, but there are some scenarios where automatic relinking is not possible or too ambiguous and requires a manual review. Below is a complete table of every scenario in which file entries can become unlinked, and whether or not TagStudio can auto-relink them:
52+
53+
| Case | File Moved? | File Renamed? | File Modified? | Unlinked Entries | Matched Files | Auto-Relink |
54+
| -------: | :---------: | :-----------: | :------------: | :--------------: | :-----------: | :---------------------------------: |
55+
| **\#0** | _No_ | _No_ | **Yes** | 0 | — | :material-minus-circle:{.lg .gray} |
56+
| **\#1** | — | — | — | 1 | 0 | :material-close-circle:{.lg .red} |
57+
| **\#2** | **Yes** | **Yes** | **Yes** | 1 | 0 | :material-close-circle:{.lg .red} |
58+
| **\#3** | **Yes** | **Yes** | _No_ | 1 | 1 | :material-check-circle:{.lg .green} |
59+
| **\#4** | **Yes** | **Yes** | _No_ | 1 | 2+ | :material-close-circle:{.lg .red} |
60+
| **\#5** | **Yes** | **Yes** | _No_ | 2+ | Any | :material-close-circle:{.lg .red} |
61+
| **\#6** | **Yes** | _No_ | **Yes** | 1 | 1 | :material-check-circle:{.lg .green} |
62+
| **\#7** | **Yes** | _No_ | _No_ | 1 | 1 | :material-check-circle:{.lg .green} |
63+
| **\#8** | **Yes** | _No_ | _No_ | 1 | 2+ | :material-close-circle:{.lg .red} |
64+
| **\#9** | **Yes** | _No_ | _No_ | 2+ | Any | :material-close-circle:{.lg .red} |
65+
| **\#10** | _No_ | **Yes** | **Yes** | 1 | 0 | :material-close-circle:{.lg .red} |
66+
| **\#11** | _No_ | **Yes** | _No_ | 1 | 1 | :material-check-circle:{.lg .green} |
67+
| **\#12** | _No_ | **Yes** | _No_ | 1 | 2+ | :material-close-circle:{.lg .red} |
68+
| **\#13** | _No_ | **Yes** | _No_ | 2+ | Any | :material-close-circle:{.lg .red} |
69+
70+
#### Explanations
71+
72+
- **Case \#0**: _Modifying file content alone does not create unlinked entries._
73+
- **Case \#1**: If the original file was deleted, no matches will be found. TagStudio leaves the decision to delete entries up to the user.
74+
- **Case \#2**: If the original file bears no similarities to the unlinked entry anymore, it is indistinguishable from a deleted or new file.
75+
- **Case \#3**: The file has been moved and renamed with a high degree of confidence.
76+
- **Case \#4**: If more than one file is matched with the same stats, the case is too ambiguous.
77+
- **Case \#5**: If two or more entries share the same stats, it's not clear which entry a matched file belongs to.
78+
- **Case \#6**: The file has been moved and modified, but since no other file shares its filename, it is assumed to be the same file with a decent degree of confidence.
79+
- **Case \#7**: The original file has been moved with a high degree of confidence.
80+
- **Case \#8**: If multiple copies of the same moved file are matched in different locations, the case is ambiguous.
81+
- **Case \#9**: _Similar to **\#5**._ If two or more entries share the same filename and stats, it's not clear which entry a matched file belongs to.
82+
- **Case \#10**: _Same as **\#2**._
83+
- **Case \#11**: The file has been renamed with a high degree of confidence.
84+
- **Case \#12**: _Same as **\#4**._
85+
- **Case \#13**: _Same as **\#5**._
86+
87+
Every numbered case above assumes the entry has saved file metadata attributes to help match against (added in **v9.7**), in which case the **file modification date** combined with the **file size** is used as a soft file signature to aid in scenarios such as renames or moves. If no file metadata is stored with the file entry, or if this soft file signature doesn't lead to a confident match, TagStudio falls back to matching by filename alone: a single file found with that name is automatically relinked, while zero or multiple filename matches leave the entry unlinked for manual review.
3788

38-
TagStudio automatically scans for new or updated files when opening a library by default. This behavior can be toggled in the settings if your library is very large and/or located on a slow drive.
89+
<!-- prettier-ignore -->
90+
!!! warning
91+
There's currently no way to manually specify which remaining unlinked entries should be linked with which files, only to delete the entries from the library. Manual relinking is a high priority feature for future releases.
3992

40-
![Settings -> Automatically Load New Files](assets/settings_refresh_library_on_open.png)
93+
<!-- prettier-ignore -->
94+
!!! warning "Switching from a Case-Sensitive to Case-Insensitive Filesystem (i.e. Windows)"
95+
If you switch from using TagStudio on a computer with a case-sensitive filesystem to one with a *case-insensitive* one, TagStudio will treat any entries added up to this point with the same filepath + name under case-insensitivity as duplicate entries and merge them. The automatic relinking process will also take case-insensitivity into account when relinking entries.
4196

42-
To manually refresh your library at any time, use **File -> Refresh Directories** from the menu or by using <kbd>Ctrl</kbd>+<kbd>R</kbd> (<kbd>⌘ Command </kbd>+<kbd>R</kbd> on macOS).
97+
Currently, TagStudio only uses this case-insensitivity mode when running on Windows. Future versions will be more precise about this distinction, with the aim of determining the case sensitivity on a per-drive basis.
4398

4499
## :material-database-cog: Library Information Panel
45100

‎docs/roadmap.md‎

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,7 @@ A detailed specification written for the TagStudio tag and/or library format. In
9696
- [x] Delete Old Backups **[[v9.5.4](changelog.md#954-september-1st-2025)]**
9797
- [x] Delete Legacy JSON File **[[v9.5.4](changelog.md#954-september-1st-2025)]**
9898
- [x] Translations
99-
- [ ] Search Bar Rework :material-chevron-triple-up:{ .priority-high title="High Priority" } **[v9.6.x]**
99+
- [ ] Search Bar Rework :material-chevron-triple-up:{ .priority-high title="High Priority" } **[v9.7.x]**
100100
- [ ] Improved Tag Autocomplete :material-chevron-triple-up:{ .priority-high title="High Priority" }
101101
- [ ] Tags appear as widgets in search bar _(similar to new tag search/create bar)_ :material-chevron-triple-up:{ .priority-high title="High Priority" }
102102
- [x] Unified Media Player
@@ -136,7 +136,7 @@ A detailed specification written for the TagStudio tag and/or library format. In
136136
- [x] Theme
137137
- [x] Thumbnail Generation **[[v9.5.4](changelog.md#954-september-1st-2025)]**
138138
- [x] Configurable Page Size
139-
- [ ] Library Settings :material-chevron-triple-up:{ .priority-high title="High Priority" } **[v9.6.x]**
139+
- [ ] Library Settings :material-chevron-triple-up:{ .priority-high title="High Priority" } **[v9.7.x]**
140140
- [ ] Stored in `.TagStudio` folder :material-chevron-triple-up:{ .priority-high title="High Priority" }
141141
- [ ] Toggle File Extension Label :material-chevron-double-up:{ .priority-med title="Medium Priority" }
142142
- [ ] Toggle Duration Label :material-chevron-double-up:{ .priority-med title="Medium Priority" }
@@ -149,13 +149,13 @@ A detailed specification written for the TagStudio tag and/or library format. In
149149

150150
- [x] Per-Library Tags
151151
- [ ] Global Tags :material-chevron-double-up:{ .priority-med title="Medium Priority" } **[v9.8.x]**
152-
- [ ] Multiple Root Directories :material-chevron-triple-up:{ .priority-high title="High Priority" } **[v9.6.x]**
153-
- [ ] Ability to store TagStudio data folder separate from library content folder(s) :material-chevron-triple-up:{ .priority-high title="High Priority" } **[v9.6.x]**
154-
- [ ] Automatic Entry Relinking :material-chevron-triple-up:{ .priority-high title="High Priority" } **[v9.8.x]**
155-
- [ ] Detect Renames :material-chevron-triple-up:{ .priority-high title="High Priority" }
156-
- [ ] Detect Moves :material-chevron-triple-up:{ .priority-high title="High Priority" }
157-
- [ ] Detect Deletions :material-chevron-triple-up:{ .priority-high title="High Priority" }
158-
- [ ] Performant :material-chevron-triple-up:{ .priority-high title="High Priority" }
152+
- [ ] Multiple Root Directories :material-chevron-triple-up:{ .priority-high title="High Priority" } **[v9.7.x]**
153+
- [ ] Ability to store TagStudio data folder separate from library content folder(s) :material-chevron-triple-up:{ .priority-high title="High Priority" } **[v9.7.x]**
154+
- [x] Automatic Entry Relinking :material-chevron-triple-up:{ .priority-high title="High Priority" } **[v9.7.0]**
155+
- [x] Detect Renames :material-chevron-triple-up:{ .priority-high title="High Priority" }
156+
- [x] Detect Moves :material-chevron-triple-up:{ .priority-high title="High Priority" }
157+
- [ ] Some way to safely handle unlinked entries presumed to be from deleted files automatically (deleted after X days?) :material-chevron-double-up:{ .priority-med title="Medium Priority" }
158+
- [x] Performant :material-chevron-triple-up:{ .priority-high title="High Priority" }
159159
- [ ] Background File Scanning :material-chevron-triple-up:{ .priority-high title="High Priority" }
160160
- [x] Thumbnail Caching **[[v9.5.0](changelog.md#950-march-3rd-2025)]**
161161
- [ ] Audio Waveform Caching :material-chevron-double-up:{ .priority-med title="Medium Priority" } **[v9.7.x]**
@@ -166,7 +166,7 @@ A detailed specification written for the TagStudio tag and/or library format. In
166166
File or file-like [entries](entries.md) stored in the library.
167167

168168
- [x] File Entries **[v1.0.0]**
169-
- [ ] URL Entries / Bookmarks :material-chevron-up:{ .priority-low title="Low Priority" } **[v9.6.x]**
169+
- [ ] URL Entries / Bookmarks :material-chevron-up:{ .priority-low title="Low Priority" } **[v9.8.x]**
170170
- [x] Fields
171171
- [x] Text Lines
172172
- [x] Text Boxes
@@ -267,7 +267,7 @@ Discrete library objects representing [attributes](<https://en.wikipedia.org/wik
267267
Sharable TagStudio library data in the form of data packs (tags, colors, etc.) or other formats.
268268
Packs are intended as an easy way to import and export specific data between libraries and users, while export-only formats are intended to be imported by other programs.
269269

270-
- [ ] Color Packs :material-chevron-triple-up:{ .priority-high title="High Priority" } **[v9.6.x]**
270+
- [ ] Color Packs :material-chevron-triple-up:{ .priority-high title="High Priority" } **[v9.7.x]**
271271
- [ ] Importable
272272
- [ ] Exportable
273273
- [x] UUIDs + Namespaces :material-chevron-triple-up:{ .priority-high title="High Priority" }

‎docs/stylesheets/extra.css‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,7 @@ img {
8080
th,
8181
td {
8282
padding: 0.5em 1em 0.5em 1em !important;
83+
min-width: 0 !important;
8384
}
8485

8586
hr {

‎docs/usage.md‎

Lines changed: 6 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,13 @@ icon: material/mouse
1212

1313
To create or open a [library](libraries.md), go to **File -> Open/Create Library** in the menu bar or use <kbd>Ctrl</kbd>+<kbd>O</kbd> (<kbd>⌘ Command </kbd>+<kbd>O</kbd> on macOS) and chose a folder with file contents you'd like to use as a TagStudio library. If a `.TagStudio` folder doesn't already exist inside the directory, TagStudio will create one and automatically scan the folder for files to include. Otherwise, the pre-existing library is opened.
1414

15-
### :material-database-refresh: Refreshing Directories
15+
### :material-database-sync: Library Syncing
1616

17-
TagStudio automatically scans for new or updated files when opening a library by default. Manually refresh by going to **File -> Refresh Directories** in the menu or by using <kbd>Ctrl</kbd>+<kbd>R</kbd> (<kbd>⌘ Command </kbd>+<kbd>R</kbd> on macOS).
17+
A TagStudio library gets synced with the files found in your content folders. This is a **non-destructive, read-only** process and none of your files are moved, modified, or deleted. Syncing is indicated by a temporary progress bar, and you can continue to use TagStudio normally while syncing occurs.
18+
19+
### :material-link-variant: Automatic Relinking
20+
21+
Files that become moved, renamed, or modified will try to be automatically relinked during syncing. If a file cannot be found or a match cannot be safely made, it will stay unlinked until the entries are manually deleted under "Fix Unlinked Entries". _Manual relinking is a high priority feature for future versions._
1822

1923
<!-- prettier-ignore -->
2024
!!! abstract "TagStudio Libraries"
@@ -115,20 +119,6 @@ Creating and adding fields to entries is extremely similar to how [tagging](#tag
115119

116120
---
117121

118-
## Relinking Moved Files
119-
120-
Inevitably some of the files inside your library will be renamed, moved, or deleted. If a file has been renamed or moved, TagStudio will display the thumbnail as a red broken chain link. To relink moved files or delete these entries, select the "Manage Unlinked Entries" option under the Tools menu. Click the "Refresh" button to scan your library for unlinked entries. Once complete, you can attempt to "Search & Relink" any unlinked file entries to their respective files, or "Delete Unlinked Entries" in the event the original files have been deleted and you no longer wish to keep their entries inside your library.
121-
122-
<!-- prettier-ignore -->
123-
!!! warning
124-
There is currently no method to relink entries to files that have been renamed - only moved or deleted. This is a high priority for future releases.
125-
126-
<!-- prettier-ignore -->
127-
!!! warning
128-
If multiple matches for a moved file are found (matches are currently defined as files with a matching filename as the original), TagStudio will currently ignore the match groups. Adding a GUI for manual selection, as well as smarter automated relinking, are high priorities for future versions.
129-
130-
---
131-
132122
## Launch Arguments
133123

134124
There are a handful of launch arguments you can pass to TagStudio via the command line or a desktop shortcut.

‎src/tagstudio/core/library/alchemy/constants.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414

1515
DB_VERSION_CURRENT_KEY: str = "CURRENT"
1616
DB_VERSION_INITIAL_KEY: str = "INITIAL"
17-
DB_VERSION: int = 400
17+
DB_VERSION: int = 500
1818

1919
TAG_CHILDREN_QUERY = text("""
2020
WITH RECURSIVE ChildTags AS (

‎src/tagstudio/core/library/alchemy/db.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,8 @@
99
from sqlalchemy import Dialect, String, TypeDecorator
1010
from sqlalchemy.orm import DeclarativeBase
1111

12+
from tagstudio.core.utils.normalization import norm_path
13+
1214
logger = structlog.getLogger(__name__)
1315

1416

@@ -19,7 +21,7 @@ class PathType(TypeDecorator):
1921
@override
2022
def process_bind_param(self, value: Path | None, dialect: Dialect):
2123
if value is not None:
22-
return Path(value).as_posix()
24+
return norm_path(Path(value), case_sensitive=True).as_posix()
2325
return None
2426

2527
@override

‎src/tagstudio/core/library/alchemy/enums.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,10 @@ class ItemType(enum.Enum):
7272

7373
class SortingModeEnum(enum.Enum):
7474
DATE_ADDED = "file.date_added"
75+
DATE_CREATED = "file.date_created"
76+
DATE_MODIFIED = "file.date_modified"
7577
FILE_NAME = "generic.filename"
78+
FILE_SIZE = "file.size"
7679
PATH = "file.path"
7780
RANDOM = "sorting.mode.random"
7881

0 commit comments

Comments
 (0)