Skip to content

Commit 8a059eb

Browse files
LukasGoldLukasGold
authored
Merge pull request #174 from OpenSemanticLab/user-item-upd
feat(user-sync): sync OSW User items from MediaWiki and ORCID
2 parents 477f4ca + ebda514 commit 8a059eb

28 files changed

Lines changed: 4184 additions & 2 deletions

‎docs/tools/index.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,10 @@
22

33
Besides the Python API, osw ships two adapters that talk to a live instance:
44
the `osw` command line client, and an MCP server for agent clients such as
5-
Claude Code. Both run the same operations from one shared, SDK-free core
6-
(`osw.service`), so a command and its matching tool behave identically. They
5+
Claude Code. This section also covers standalone scripts built on the Python
6+
API. The current one is the [user sync](user-sync.md). The two adapters run
7+
the same operations from one shared, SDK-free core (`osw.service`), so a
8+
command and its matching tool behave identically. They
79
differ in exactly one way: only the CLI accepts filesystem paths.
810

911
## Setup
@@ -49,4 +51,5 @@ extras are listed in the [Get Started guide](../get-started.md#optional-extras).
4951
| ---- | -------- |
5052
| [CLI](cli.md) | The `.env` quick start, the full command reference, and the global flags |
5153
| [MCP server](mcp.md) | The tool surface, the no-filesystem-access and one-server-per-instance rules, and how to register the server with a client such as Claude Code |
54+
| [User Sync (script)](user-sync.md) | A Python script that creates or updates `User` items from the MediaWiki accounts of an instance, with the options, the data it writes and how it reconciles existing items |
5255
| [Configuration](configuration.md) | What both adapters share: where credentials and settings come from, and the full environment-variable reference |

‎docs/tools/user-sync.md‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
# User Item Sync
2+
3+
Create or update OSW `User` items from the MediaWiki accounts of an OSL
4+
instance, enriching ORCID users from the public ORCID API. The run is
5+
idempotent and previews every change before writing.
6+
7+
This is a Python script built on the osw library. It is not a command of the
8+
`osw` CLI and not an MCP tool, see [Tools](index.md). It runs from a source
9+
checkout, because `examples/` is not part of the installed package.
10+
11+
## When to use it
12+
13+
OSL instances gain accounts from manual creation and from whitelisted ORCID
14+
single sign-on (whose username is the ORCID iD). Those accounts have no
15+
semantic `User` item until this tool reconciles them.
16+
17+
## Run it
18+
19+
```bash
20+
uv run python examples/user_sync.py --domain your-instance.example.org --dry-run
21+
```
22+
23+
The script reads the credential file given by `--cred-filepath`, otherwise
24+
`accounts.pwd.yaml` in the working directory, otherwise the file named by the
25+
`OSW_CRED_FILEPATH` environment variable. It does not read a `.env` file.
26+
`--domain` falls back to the `OSW_DOMAIN` environment variable. See
27+
[Credentials](configuration.md#credentials) for the file format. Start with
28+
`--dry-run` to preview, then run without it to write after confirming at the
29+
prompt.
30+
31+
To also clean up existing items, add `--prune`. Without it, the sync never
32+
writes `employment_contract_status` and never removes anything. With it,
33+
`employment_contract_status` is removed from the stored jsondata of existing
34+
items. `website` and `organization` are overwritten with an empty list, unless
35+
`--with-websites` or `--with-organizations` enables them. This happens in a
36+
second edit of the page's jsondata after the store. The sync decides from the
37+
stored jsondata, not from the loaded item, which shows the model default for
38+
`employment_contract_status`. A second `--prune` run therefore changes nothing:
39+
40+
```bash
41+
uv run python examples/user_sync.py --domain your-instance.example.org --prune --dry-run
42+
```
43+
44+
Users whose ORCID record exposes no email are listed in a warning at the end of
45+
the run; a missing email never fails the sync.
46+
47+
### Options
48+
49+
By default the tool syncs both ORCID and MediaWiki-native accounts and writes
50+
only core identity plus email.
51+
52+
| Flag | Effect |
53+
| --- | --- |
54+
| `--domain` | Target OSL domain. |
55+
| `--cred-filepath` | Path to `accounts.pwd.yaml`. |
56+
| `--dry-run` | Preview only; never writes. |
57+
| `--auto-apply` | Non-interactive: apply creates, gap-fills and removals; keep existing on conflicts. |
58+
| `--limit N` | Process at most N accounts (testing). |
59+
| `--orcid-only` | Only ORCID-username accounts (excludes `--mw-only`). |
60+
| `--mw-only` | Only non-ORCID (MediaWiki-native) accounts (excludes `--orcid-only`). |
61+
| `--include-system` | Do not skip MediaWiki reserved system accounts. |
62+
| `--no-redirects` | Do not create `User:` redirect pages. |
63+
| `--with-websites` | Also store ORCID researcher URLs (opt-in). |
64+
| `--with-organizations` | Also create and link Organization items from ORCID affiliations (opt-in). |
65+
| `--with-extras` | Enable both websites and organizations. |
66+
| `--include-blocked` | Do not skip blocked MediaWiki accounts. |
67+
| `--prune` | Remove `employment_contract_status` from the stored jsondata of existing items, and overwrite `website` or `organization` with an empty list when their opt-in flag is not given, in a second edit of the page's jsondata after the store (default off; a normal run never writes `employment_contract_status` and never removes anything). |
68+
69+
## What it does
70+
71+
1. Enumerate MediaWiki accounts, skipping blocked accounts (unless
72+
`--include-blocked` is set; the User item of a skipped blocked account is
73+
left unchanged), bots and privileged groups
74+
(`bot`, `sysop`, `bureaucrat`, `interface-admin`) and MediaWiki reserved
75+
system usernames. Accounts whose username is an exact ORCID iD are exempt
76+
from the group filter (so a person's own admin account is synced); other
77+
accounts, such as `<orcid>@bot`, are still excluded by their groups.
78+
2. Enrich ORCID users from `https://pub.orcid.org/v3.0/<id>`: names and email
79+
always; researcher URLs and affiliations only when their flag is enabled.
80+
3. Map each account to a proposed `User` item with a deterministic id
81+
(uuid5 on the ORCID iD, else the username), so re-runs are idempotent.
82+
4. Reconcile against existing items. Existing items are found by `username`
83+
across all `User` items. An item whose deterministic page already exists is
84+
updated and not created again. Each user gets one of four categories: NEW,
85+
GAP_FILL, CONFLICT or UNCHANGED. Each changed field has one of three
86+
statuses: GAP_FILL, CONFLICT or REMOVE. The preview shows a removed field as
87+
`-field`. A user with only gap-fills or removals is GAP_FILL. Gap-fills apply
88+
automatically and conflicts are resolved interactively. Removals appear only
89+
with `--prune`; a normal run never removes anything.
90+
5. Store items, create `User:<username>` redirects to each item, ensure any
91+
linked Organization items, verify by reloading, and warn about users with no
92+
email.
93+
94+
## Data written
95+
96+
| Field | Policy |
97+
| --- | --- |
98+
| `username`, `first_name`, `surname`, `label`, `orcid` | Always (core identity). |
99+
| `email` | Standard: always attempted; missing email is warned, not fatal; never removed. |
100+
| `website` | Opt-in (`--with-websites`); without that flag, on existing items overwritten with an empty list only with `--prune`, in a second jsondata edit after the store. |
101+
| `organization` | Opt-in (`--with-organizations`); without that flag, on existing items overwritten with an empty list only with `--prune`, in a second jsondata edit after the store. |
102+
| `employment_contract_status` | Never written without `--prune`; with `--prune`, removed from the stored jsondata of existing items in a second jsondata edit after the store. |
103+
104+
## Notes
105+
106+
- MediaWiki does not expose other users' email or real name, so ORCID is the
107+
only source of rich data. Non-ORCID accounts get username-based placeholder
108+
names, flagged in the preview. Placeholder names are written only for new
109+
items and are never proposed for an existing item.
110+
- In a source checkout (for developers), the first accepted run autofetches
111+
schemas and may regenerate `src/osw/model/entity.py`. Restore it if you do not
112+
want that change.

‎examples/user_sync.py‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
"""Create or update OSW User items from MediaWiki accounts and ORCID.
2+
3+
Usage:
4+
uv run python examples/user_sync.py --domain llm4eln.semos.dev --dry-run
5+
"""
6+
7+
from osw.defaults import paths
8+
from osw.express import OswExpress
9+
from osw.tools.user_sync import config_from_args, run_user_sync
10+
11+
12+
def main() -> None:
13+
config = config_from_args()
14+
cred_filepath = config.cred_filepath
15+
if cred_filepath is None and paths.cred_filepath.is_file():
16+
# OswExpress only reads OSW_CRED_FILEPATH; use the default accounts file.
17+
cred_filepath = paths.cred_filepath
18+
osw = OswExpress(domain=config.domain, cred_filepath=cred_filepath)
19+
report = run_user_sync(config, osw=osw)
20+
print(report.summary())
21+
for title in report.created:
22+
print(f" created: {title}")
23+
for title in report.updated:
24+
print(f" updated: {title}")
25+
for key, error in report.failed.items():
26+
print(f" FAILED {key}: {error}")
27+
28+
29+
if __name__ == "__main__":
30+
main()

‎src/osw/tools/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
"""Maintenance and automation tools built on top of the osw core library."""

‎src/osw/tools/user_sync/__init__.py‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
"""Sync OSW User items from MediaWiki accounts and public ORCID data."""
2+
3+
from __future__ import annotations
4+
5+
from .config import (
6+
ORGANIZATION_CATEGORY,
7+
USER_CATEGORY,
8+
SyncConfig,
9+
build_arg_parser,
10+
config_from_args,
11+
)
12+
from .sync import SyncReport, run_user_sync
13+
14+
__all__ = [
15+
"ORGANIZATION_CATEGORY",
16+
"USER_CATEGORY",
17+
"SyncConfig",
18+
"SyncReport",
19+
"build_arg_parser",
20+
"config_from_args",
21+
"run_user_sync",
22+
]

‎src/osw/tools/user_sync/build.py‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
"""Build pydantic User/Organization entities from proposals.
2+
3+
Must run within an open OSW connection (the models resolve linked entities on
4+
construction). Organization references are passed as a list of page-title IRIs;
5+
the reference attribute is never read back here, as that would trigger a
6+
network resolve.
7+
8+
An empty field is set to None, so its key is absent from the serialized item.
9+
The store merges with the stored page, so an absent key keeps the stored value.
10+
Clearing a stored value is done by ``prune.clear_jsondata_keys`` instead.
11+
"""
12+
13+
from __future__ import annotations
14+
15+
from typing import Any, List, Set
16+
17+
from opensemantic.base.v1 import Organization, User
18+
19+
from .config import PROTECTED_FIELDS
20+
from .mapping import ProposedOrganization, ProposedUser
21+
22+
# The concrete DisplayName type used by the label field.
23+
_DISPLAY_NAME = User.__fields__["label"].type_
24+
25+
26+
def _labels(text: str) -> List[Any]:
27+
return [_DISPLAY_NAME(text=text)]
28+
29+
30+
def _strip_protected(entity: Any) -> None:
31+
"""Unset protected relations (e.g. employment_contract_status).
32+
33+
The attribute is set to None, so the key is absent from the serialized item
34+
and the model default is not written. A stored value is left unchanged.
35+
"""
36+
for name in PROTECTED_FIELDS:
37+
setattr(entity, name, None)
38+
39+
40+
def build_organization(proposed: ProposedOrganization) -> Organization:
41+
return Organization(uuid=proposed.uuid, label=_labels(proposed.name))
42+
43+
44+
def build_user(proposed: ProposedUser) -> User:
45+
"""Build a full User entity for creation (never carries protected fields)."""
46+
args: dict = {
47+
"uuid": proposed.uuid,
48+
"username": proposed.username,
49+
"first_name": proposed.first_name,
50+
"surname": proposed.surname,
51+
"label": _labels(proposed.label),
52+
}
53+
if proposed.orcid:
54+
args["orcid"] = proposed.orcid
55+
if proposed.emails:
56+
args["email"] = set(proposed.emails)
57+
if proposed.websites:
58+
args["website"] = set(proposed.websites)
59+
if proposed.organizations:
60+
args["organization"] = list(proposed.organizations)
61+
entity = User(**args)
62+
_strip_protected(entity)
63+
return entity
64+
65+
66+
def apply_update(entity: Any, proposed: ProposedUser, apply_fields: Set[str]) -> Any:
67+
"""Apply the accepted fields onto a loaded existing entity, in place.
68+
69+
An empty organization or website value is set to None, which leaves the
70+
stored value unchanged. Protected fields are always unset.
71+
"""
72+
if "username" in apply_fields:
73+
entity.username = proposed.username
74+
if "label" in apply_fields:
75+
entity.label = _labels(proposed.label)
76+
if "first_name" in apply_fields:
77+
entity.first_name = proposed.first_name
78+
if "surname" in apply_fields:
79+
entity.surname = proposed.surname
80+
if "orcid" in apply_fields:
81+
entity.orcid = proposed.orcid
82+
if "emails" in apply_fields:
83+
entity.email = set(proposed.emails)
84+
if "websites" in apply_fields:
85+
entity.website = set(proposed.websites) if proposed.websites else None
86+
if "organizations" in apply_fields:
87+
entity.organization = (
88+
list(proposed.organizations) if proposed.organizations else None
89+
)
90+
_strip_protected(entity)
91+
return entity

0 commit comments

Comments
 (0)