Support excluding personal data fields when syncing OpenSanctions datasets

[angpao]

Hi team,

First of all, thank you for maintaining Yente.

We are running a self-hosted Yente instance and synchronizing data from OpenSanctions datasets. For compliance and data minimization reasons, we would like to exclude certain personal data attributes from being stored and indexed during the synchronization process.

Examples of fields we would like to exclude:

Email addresses
Telephone numbers
Regional / geographic details
Physical addresses
Other personally identifiable information (PII)

Our use case primarily requires sanctions screening and entity matching, but we do not need all personal contact information available in the source datasets.

Questions:

Is there currently a supported way to configure Yente to ignore or remove specific properties during ingestion?
Can field-level filtering be applied before data is written to the database and search index?
If this is not currently supported, would you consider adding a configuration option (e.g., allowlist/blocklist of properties) for self-hosted deployments?

Example:

ingest:
exclude_properties:
- email
- phone
- address
- region

This would help organizations comply with internal privacy requirements while still benefiting from OpenSanctions data.

Thank you for your guidance.

[pudo]

Hi @angpao, thanks for the kind words and for the detailed write-up.

Short answer: there's no per-property allowlist/blocklist in yente today, and I'd lean against adding one. Doing this would expand yente's scope of responsibility — the joy of the tool right now is that it does very little, and tries to do that well. Field-level redaction config sits more naturally in a step in front of the indexer than inside it. The good news is that your use case is very achievable with a small pre-processing step, and there's also a hosted option that may sidestep the problem entirely. A few thoughts:

1. Pre-process the data before yente ingests it.

yente already supports indexing a custom dataset from a local file on disk rather than fetching from OpenSanctions directly. So the clean separation is: you control a step that downloads the FtM entity feed, strips the properties you don't want, writes the result to disk, and yente indexes that file.

FtM data is a line-based JSON feed where each line is one entity with a properties dict, so the redaction step is trivial — e.g.:

# Strip selected properties from a FollowTheMoney entity feed
jq -c 'del(.properties.email, .properties.phone, .properties.address, .properties.addressEntity)' \
  entities.ftm.json > entities.clean.ftm.json

Then point a manifest datasets entry at the cleaned file:

datasets:
  - name: sanctions_clean
    title: OpenSanctions (PII-reduced)
    path: /data/entities.clean.ftm.json
    version: "20260613001"   # bump to force re-index

This keeps the data-minimization policy fully in your hands and auditable, without coupling it to yente's release cycle.

2. Worth weighing: this will reduce match quality.

The properties that read as "the most PII" are often exactly the ones that make screening work — the whole point is to confidently identify a specific person. Names, dates of birth, nationalities, and identifiers are the strongest matching signals; dropping them increases both false positives and false negatives. Contact details like email/phone are lower-value for matching, so those are the safer ones to strip if you do go this route.

One clarification on addresses specifically: OpenSanctions data is built from public government watchlists and registers, and generally does not contain individuals' home addresses — addresses tend to be business or official addresses already published by the issuing authority. So the privacy exposure there may be smaller than you're assuming.

3. If the concern is storing the data in your infrastructure at all, consider the hosted API.

If the goal is to avoid persisting this PII in your own systems, our hosted matching API is an alternative: the data lives on our side, and you only receive entity details in the response to a match. You're then free to store only the fields you need (or none at all) from each result.

Happy to help if you hit snags wiring up the pre-processing approach.

[angpao]

Thank you so much for your support.
Hi team,

Thank you for the earlier clarification regarding field-level redaction. Based on your recommendation, I'm planning to preprocess the OpenSanctions dataset before Yente ingests it.

My intended workflow is:

Cron Job
  ↓
Download OpenSanctions dataset
  ↓
Remove selected properties
  ↓
Generate entities.clean.ftm.json
  ↓
Configure Yente to index the sanitized file
  ↓
Trigger re-index

Example preprocessing script:

import requests
import json

DOWNLOAD_URL = "https://data.opensanctions.org/datasets/latest/default/entities.ftm.json"

REMOVE_FIELDS = {
    "email",
    "phone",
    "telephone",
    "address",
    "addressEntity",
    "region"
}

r = requests.get(DOWNLOAD_URL, stream=True)
r.raise_for_status()

with open("entities.clean.ftm.json", "w", encoding="utf-8") as outfile:
    for line in r.iter_lines(decode_unicode=True):
        if not line:
            continue

        entity = json.loads(line)

        props = entity.get("properties", {})

        for field in REMOVE_FIELDS:
            props.pop(field, None)

        entity["properties"] = props

        outfile.write(json.dumps(entity, ensure_ascii=False) + "\n")

Before implementing this approach, I would appreciate confirmation on a few points:

1. FtM compatibility

Will the generated entities.clean.ftm.json still be considered a valid FtM entity feed for Yente, assuming I only remove selected properties and keep the rest of the entity structure unchanged?

2. Properties safe to remove

Are the following properties safe to remove without causing indexing issues?

• email
• phone
• telephone
• address
• addressEntity
• region

Are there any properties that Yente depends on internally and therefore should never be removed?

3. Manifest configuration

The default manifest contains:

catalogs:
  - url: "https://data.opensanctions.org/datasets/latest/index.json"
    scope: default
    resource_name: entities.ftm.json

datasets: []

If I want Yente to index only the sanitized local dataset, is the recommended approach to replace the catalog configuration with a local dataset entry such as:

catalogs: []

datasets:
  - name: sanctions_clean
    title: OpenSanctions (PII-reduced)
    path: /data/entities.clean.ftm.json
    version: "20260615001"

Or is there a better/recommended way to configure this workflow?

4. Version handling

For local datasets, does Yente use the version field only to determine whether re-indexing is required?

If so, would it be reasonable to generate the version automatically (e.g. timestamp or SHA256 hash of the sanitized file)?

5. Re-indexing behavior

After generating a new sanitized file and updating the version, what is the recommended way to trigger a re-index?

Specifically:

• Does Yente automatically detect dataset/version changes?
• Is there a CLI command that should be executed?
• Is there an HTTP API endpoint to trigger a re-index?
• Or is restarting the Yente service/container the expected approach?

My goal is to fully automate the workflow using a cron job without manual intervention.

6. Recommended automation workflow

Could you please confirm whether the following overall approach aligns with Yente's intended usage?

Cron Job
  ↓
Download OpenSanctions dataset
  ↓
Sanitize selected properties
  ↓
Generate entities.clean.ftm.json
  ↓
Update dataset version
  ↓
Trigger Yente re-index

My goal is to minimize storage of selected PII fields while keeping indexing and matching functionality intact.

Thanks again for your guidance and for maintaining Yente.

[sophymarine]

Yente will index whatever fields arrive inside the FollowTheMoney entity payload. There isn’t a server-side config layer that strips fields like email, phone, or address before they hit the index. So if those appear in the FtM object, they end up searchable.

Most teams that need GDPR-style minimisation handle it earlier. Right at ingestion.

OpenSanctions data is just an FtM graph, which makes this fairly easy. Insert a small transform in the ingest step and drop the properties you don’t want from each EntityProxy before it lands in PostgreSQL or the search index. I’d lean on an allowlist rather than a blocklist. FtM schemas get extended, and a blocklist tends to miss new fields. That way the matching keys stay in place while contact data never enters the index.