fix(user-sync): align with stage and the tools docs

- list the page in the existing Tools nav group, describe scripts there
- page through all User items and adopt pages at the deterministic title
- skip blocked accounts by default, never propose placeholder names
- prune via a second jsondata edit; oold 2.3 ignores __iris__ edits
- never write employment_contract_status; add store and prune tests

Author: LukasGold

docs/tools/index.md

@@ -2,8 +2,10 @@
 
 Besides the Python API, osw ships two adapters that talk to a live instance:
 the `osw` command line client, and an MCP server for agent clients such as
-Claude Code. Both run the same operations from one shared, SDK-free core
-(`osw.service`), so a command and its matching tool behave identically. They
+Claude Code. This section also covers standalone scripts built on the Python
+API. The current one is the [user sync](user-sync.md). The two adapters run
+the same operations from one shared, SDK-free core (`osw.service`), so a
+command and its matching tool behave identically. They
 differ in exactly one way: only the CLI accepts filesystem paths.
 
 ## Setup
@@ -49,4 +51,5 @@ extras are listed in the [Get Started guide](../get-started.md#optional-extras).
 | ---- | -------- |
 | [CLI](cli.md) | The `.env` quick start, the full command reference, and the global flags |
 | [MCP server](mcp.md) | The tool surface, the no-filesystem-access and one-server-per-instance rules, and how to register the server with a client such as Claude Code |
+| [User Sync (script)](user-sync.md) | A Python script that creates or updates `User` items from the MediaWiki accounts of an instance, with the options, the data it writes and how it reconciles existing items |
 | [Configuration](configuration.md) | What both adapters share: where credentials and settings come from, and the full environment-variable reference |

docs/tools/user-sync.md

@@ -4,6 +4,10 @@ Create or update OSW `User` items from the MediaWiki accounts of an OSL
 instance, enriching ORCID users from the public ORCID API. The run is
 idempotent and previews every change before writing.
 
+This is a Python script built on the osw library. It is not a command of the
+`osw` CLI and not an MCP tool, see [Tools](index.md). It runs from a source
+checkout, because `examples/` is not part of the installed package.
+
 ## When to use it
 
 OSL instances gain accounts from manual creation and from whitelisted ORCID
@@ -16,12 +20,20 @@ semantic `User` item until this tool reconciles them.
 uv run python examples/user_sync.py --domain your-instance.example.org --dry-run
 ```
 
-Credentials are read from `accounts.pwd.yaml` (or `--cred-filepath`), the same
-mechanism as `OswExpress`. Start with `--dry-run` to preview, then run without
-it to write after confirming at the prompt.
-
-To also remove data no longer wanted (disabled optional fields and
-`employment_contract_status`) from existing items, add `--prune`:
+The script reads the credential file given by `--cred-filepath`, otherwise
+`accounts.pwd.yaml` in the working directory, otherwise the file named by the
+`OSW_CRED_FILEPATH` environment variable. It does not read a `.env` file.
+`--domain` falls back to the `OSW_DOMAIN` environment variable. See
+[Credentials](configuration.md#credentials) for the file format. Start with
+`--dry-run` to preview, then run without it to write after confirming at the
+prompt.
+
+To also clean up existing items, add `--prune`. Without it, the sync never
+writes `employment_contract_status` and never removes anything. With it,
+`employment_contract_status` on existing items is overwritten with an empty
+value. So are `website` and `organization`, unless `--with-websites` or
+`--with-organizations` enables them. This happens in a second edit of the
+page's jsondata after the store:
 
 ```bash
 uv run python examples/user_sync.py --domain your-instance.example.org --prune --dry-run
@@ -49,11 +61,14 @@ only core identity plus email.
 | `--with-websites` | Also store ORCID researcher URLs (opt-in). |
 | `--with-organizations` | Also create and link Organization items from ORCID affiliations (opt-in). |
 | `--with-extras` | Enable both websites and organizations. |
-| `--prune` | Remove disabled optional fields and `employment_contract_status` from existing items (default off; a normal run only adds). |
+| `--include-blocked` | Do not skip blocked MediaWiki accounts. |
+| `--prune` | Overwrite `employment_contract_status`, and `website` or `organization` when their opt-in flag is not given, on existing items with an empty value, in a second edit of the page's jsondata after the store (default off; a normal run never writes `employment_contract_status` and never removes anything). |
 
 ## What it does
 
-1. Enumerate MediaWiki accounts, skipping bots and privileged groups
+1. Enumerate MediaWiki accounts, skipping blocked accounts (unless
+   `--include-blocked` is set; the User item of a skipped blocked account is
+   left unchanged), bots and privileged groups
    (`bot`, `sysop`, `bureaucrat`, `interface-admin`) and MediaWiki reserved
    system usernames. Accounts whose username is an exact ORCID iD are exempt
    from the group filter (so a person's own admin account is synced); other
@@ -62,10 +77,14 @@ only core identity plus email.
    always; researcher URLs and affiliations only when their flag is enabled.
 3. Map each account to a proposed `User` item with a deterministic id
    (uuid5 on the ORCID iD, else the username), so re-runs are idempotent.
-4. Reconcile against existing items by `username` into NEW, GAP_FILL, CONFLICT,
-   REMOVE or UNCHANGED, then preview; gap-fills apply automatically and conflicts
-   are resolved interactively. Removals only appear with `--prune`; a normal run
-   never removes anything.
+4. Reconcile against existing items. Existing items are found by `username`
+   across all `User` items. An item whose deterministic page already exists is
+   updated and not created again. Each user gets one of four categories: NEW,
+   GAP_FILL, CONFLICT or UNCHANGED. Each changed field has one of three
+   statuses: GAP_FILL, CONFLICT or REMOVE. The preview shows a removed field as
+   `-field`. A user with only gap-fills or removals is GAP_FILL. Gap-fills apply
+   automatically and conflicts are resolved interactively. Removals appear only
+   with `--prune`; a normal run never removes anything.
 5. Store items, create `User:<username>` redirects to each item, ensure any
    linked Organization items, verify by reloading, and warn about users with no
    email.
@@ -76,14 +95,16 @@ only core identity plus email.
 | --- | --- |
 | `username`, `first_name`, `surname`, `label`, `orcid` | Always (core identity). |
 | `email` | Standard: always attempted; missing email is warned, not fatal; never removed. |
-| `website` | Opt-in (`--with-websites`); removed from existing only with `--prune`. |
-| `organization` | Opt-in (`--with-organizations`); removed from existing only with `--prune`. |
-| `employment_contract_status` | Never written; removed from existing only with `--prune`. |
+| `website` | Opt-in (`--with-websites`); without that flag, on existing items overwritten with an empty value only with `--prune`, in a second jsondata edit after the store. |
+| `organization` | Opt-in (`--with-organizations`); without that flag, on existing items overwritten with an empty value only with `--prune`, in a second jsondata edit after the store. |
+| `employment_contract_status` | Never written without `--prune`; with `--prune`, overwritten with an empty value on existing items in a second jsondata edit after the store. |
 
 ## Notes
 
 - MediaWiki does not expose other users' email or real name, so ORCID is the
   only source of rich data. Non-ORCID accounts get username-based placeholder
-  names, flagged in the preview.
-- On a first accepted run the library autofetches schemas and may regenerate
-  `src/osw/model/entity.py`; restore it if you do not want that change.
+  names, flagged in the preview. Placeholder names are written only for new
+  items and are never proposed for an existing item.
+- In a source checkout (for developers), the first accepted run autofetches
+  schemas and may regenerate `src/osw/model/entity.py`. Restore it if you do not
+  want that change.

src/osw/tools/user_sync/build.py

@@ -1,9 +1,13 @@
 """Build pydantic User/Organization entities from proposals.
 
 Must run within an open OSW connection (the models resolve linked entities on
-construction). Organization references are passed as a list of page-title IRIs,
-which oold stores in the entity's ``__iris__`` map; the reference attribute is
-never read back here, as that would trigger a network resolve.
+construction). Organization references are passed as a list of page-title IRIs;
+the reference attribute is never read back here, as that would trigger a
+network resolve.
+
+An empty field is set to None, so its key is absent from the serialized item.
+The store merges with the stored page, so an absent key keeps the stored value.
+Clearing a stored value is done by ``prune.clear_jsondata_keys`` instead.
 """
 
 from __future__ import annotations
@@ -24,17 +28,13 @@ def _labels(text: str) -> List[Any]:
 
 
 def _strip_protected(entity: Any) -> None:
-    """Clear protected relations (e.g. employment_contract_status).
+    """Unset protected relations (e.g. employment_contract_status).
 
-    Set to an empty list rather than removed: the model re-applies its default
-    when the field is absent, and the store only overwrites a relation that is
-    present as an explicit empty value (with ``remove_empty=False``).
+    The attribute is set to None, so the key is absent from the serialized item
+    and the model default is not written. A stored value is left unchanged.
     """
-    iris = getattr(entity, "__iris__", None)
-    if isinstance(iris, dict):
-        for name in PROTECTED_FIELDS:
-            if name in iris:
-                iris[name] = []
+    for name in PROTECTED_FIELDS:
+        setattr(entity, name, None)
 
 
 def build_organization(proposed: ProposedOrganization) -> Organization:
@@ -66,9 +66,11 @@ def build_user(proposed: ProposedUser) -> User:
 def apply_update(entity: Any, proposed: ProposedUser, apply_fields: Set[str]) -> Any:
     """Apply the accepted fields onto a loaded existing entity, in place.
 
-    Fields whose proposal is empty (REMOVE) are cleared; protected fields in
-    ``apply_fields`` are stripped from the entity's ``__iris__`` map.
+    An empty organization or website value is set to None, which leaves the
+    stored value unchanged. Protected fields are always unset.
     """
+    if "username" in apply_fields:
+        entity.username = proposed.username
     if "label" in apply_fields:
         entity.label = _labels(proposed.label)
     if "first_name" in apply_fields:
@@ -80,10 +82,10 @@ def apply_update(entity: Any, proposed: ProposedUser, apply_fields: Set[str]) ->
     if "emails" in apply_fields:
         entity.email = set(proposed.emails)
     if "websites" in apply_fields:
-        entity.website = set(proposed.websites)
+        entity.website = set(proposed.websites) if proposed.websites else None
     if "organizations" in apply_fields:
-        # Empty list both links (non-empty) and clears (empty, on removal).
-        entity.__iris__["organization"] = list(proposed.organizations)
-    if any(name in apply_fields for name in PROTECTED_FIELDS):
-        _strip_protected(entity)
+        entity.organization = (
+            list(proposed.organizations) if proposed.organizations else None
+        )
+    _strip_protected(entity)
     return entity

src/osw/tools/user_sync/config.py

@@ -43,6 +43,8 @@ class SyncConfig:
     link_organizations: bool = False
     # Remove disabled optional fields and protected fields from existing items.
     prune: bool = False
+    # Blocked MediaWiki accounts are skipped unless this is set.
+    include_blocked: bool = False
     orcid_api_base: str = ORCID_API_BASE_DEFAULT
     excluded_groups: List[str] = field(default_factory=lambda: list(SYSTEM_GROUPS))
 
@@ -111,8 +113,9 @@ def build_arg_parser() -> argparse.ArgumentParser:
         "--prune",
         dest="prune",
         action="store_true",
-        help="Remove disabled optional fields and employment_contract_status from "
-        "existing items (data cleanup). Off by default; a normal run only adds.",
+        help="Remove disabled optional fields from existing items and overwrite "
+        "employment_contract_status with an empty value (data cleanup). Off by "
+        "default; a normal run only adds.",
     )
     account_scope = parser.add_mutually_exclusive_group()
     account_scope.add_argument(
@@ -133,6 +136,12 @@ def build_arg_parser() -> argparse.ArgumentParser:
         action="store_false",
         help="Do not skip MediaWiki reserved system accounts (Maintenance script etc.).",
     )
+    parser.add_argument(
+        "--include-blocked",
+        dest="include_blocked",
+        action="store_true",
+        help="Do not skip blocked MediaWiki accounts.",
+    )
     return parser
 
 
@@ -152,4 +161,5 @@ def config_from_args(argv: Optional[List[str]] = None) -> SyncConfig:
         include_websites=args.include_websites or args.with_extras,
         link_organizations=args.link_organizations or args.with_extras,
         prune=args.prune,
+        include_blocked=args.include_blocked,
     )

src/osw/tools/user_sync/existing.py

@@ -9,8 +9,13 @@
 from dataclasses import dataclass, field
 from typing import Any, Dict, List, Optional
 
+from osw.wiki_tools import SearchParam
+
 from .config import USER_CATEGORY
 
+# Results per semantic search request.
+PAGE_SIZE = 1000
+
 
 @dataclass
 class ExistingUsers:
@@ -34,6 +39,36 @@ def _as_entity_list(result: Any) -> List[Any]:
     return [result]
 
 
+def _query_user_titles(
+    osw: Any, category: str, limit: Optional[int] = None
+) -> List[str]:
+    """Return the titles of all items of a category, paging through the results.
+
+    The next offset is the one the wiki reports (``query-continue-offset``), not
+    the offset plus the page size, because the wiki can cut a page shorter.
+    """
+    titles: List[str] = []
+    offset = 0
+    while True:
+        query = SearchParam(
+            query=f"[[HasType::{category}]]|offset={offset}",
+            limit=PAGE_SIZE,
+            return_meta=True,
+        )
+        result = osw.site.semantic_search(query)[0]
+        titles.extend(result.titles)
+        if limit is not None and len(titles) >= limit:
+            return titles[:limit]
+        if not result.truncated:
+            return titles
+        if result.next_offset is None or result.next_offset <= offset:
+            raise RuntimeError(
+                f"The wiki reported the next offset {result.next_offset} after "
+                f"offset {offset} for '{category}'. Paging cannot continue."
+            )
+        offset = result.next_offset
+
+
 def load_existing_users(
     osw: Any,
     category: str = USER_CATEGORY,
@@ -42,14 +77,15 @@ def load_existing_users(
     """Query and load all existing User items, indexed by username.
 
     Args:
-        osw: An authenticated OSW connection exposing ``query_instances`` and
-            ``load_entity``.
+        osw: An authenticated OSW connection exposing ``site.semantic_search``
+            and ``load_entity``.
         category: The User category page title.
         limit: Load at most this many items (for testing).
+
+    Raises:
+        RuntimeError: If the wiki reports a next offset that does not advance.
     """
-    titles = osw.query_instances(category=category)
-    if limit is not None:
-        titles = titles[:limit]
+    titles = _query_user_titles(osw, category, limit)
     if not titles:
         return ExistingUsers()
     entities = _as_entity_list(osw.load_entity(titles))
@@ -59,3 +95,35 @@ def load_existing_users(
         if username:
             by_username[username] = entity
     return ExistingUsers(by_username=by_username)
+
+
+def adopt_existing_pages(
+    osw: Any, existing: ExistingUsers, proposed_users: List[Any]
+) -> List[str]:
+    """Add items that exist at a proposed user's deterministic page title.
+
+    A user without a match by ``username`` would be created at its deterministic
+    page title. If that page already exists (for example, it has no username or
+    is not typed as a User), it is loaded and indexed under the proposed
+    username, so the sync reconciles against it instead of creating it again.
+
+    Returns:
+        The page titles that were adopted.
+    """
+    candidates = [p for p in proposed_users if existing.get(p.username) is None]
+    if not candidates:
+        return []
+    titles = [p.full_page_title for p in candidates]
+    missing = set(osw._get_missing_page_titles(titles, confirm_delay_s=0))
+    present = [p for p in candidates if p.full_page_title not in missing]
+    if not present:
+        return []
+    entities = _as_entity_list(osw.load_entity([p.full_page_title for p in present]))
+    by_uuid = {str(getattr(e, "uuid", None)): e for e in entities if e is not None}
+    adopted: List[str] = []
+    for proposed in present:
+        entity = by_uuid.get(str(proposed.uuid))
+        if entity is not None:
+            existing.by_username[proposed.username] = entity
+            adopted.append(proposed.full_page_title)
+    return adopted

src/osw/tools/user_sync/prune.py

@@ -0,0 +1,38 @@
+"""Clear stored relations of existing user items (``--prune``).
+
+The store merges with the stored page, so an absent key keeps the stored value.
+The keys are therefore overwritten with an empty list in a second edit of the
+page's jsondata slot.
+"""
+
+from __future__ import annotations
+
+import json
+from typing import Any, Iterable
+
+from osw.wtsite import WtPage
+
+
+def clear_jsondata_keys(osw: Any, title: str, keys: Iterable[str]) -> bool:
+    """Set each present, non-empty key in the page's jsondata slot to ``[]``.
+
+    The page is read with a new WtPage, which downloads the current revision and
+    bypasses the WtSite page cache. Other keys are written back unchanged.
+    Returns True if the page was written, False if nothing changed.
+    """
+    page = WtPage(wtSite=osw.site, title=title)
+    content = page.get_slot_content("jsondata")
+    if isinstance(content, str):
+        content = json.loads(content)
+    if not isinstance(content, dict):
+        return False
+    changed = False
+    for key in keys:
+        if key in content and content[key] not in (None, [], "", {}):
+            content[key] = []
+            changed = True
+    if not changed:
+        return False
+    page.set_slot_content("jsondata", content)
+    page.edit(comment="user-sync: prune")
+    return True