Skip to content

Commit 43a6681

Browse files
Clare72Clare72
committed
Update README.md
1 parent 330d830 commit 43a6681

1 file changed

Lines changed: 156 additions & 4 deletions

File tree

‎README.md‎

Lines changed: 156 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,161 @@
11
# VFB Pipeline 2: Preparing datadumps (vfb-pipeline-dumps)
22

3-
Developer notes:
4-
- Make sure `ENV SPARQL_ENDPOINT` is set
3+
**Developer notes:**
4+
- Make sure `ENV SPARQL_ENDPOINT` is set (it is **not** verified by the `checkenv` target).
55

6-
Dockerhub:
6+
Docker Hub: https://hub.docker.com/repository/docker/virtualflybrain/vfb-pipeline-dumps
77

8-
https://hub.docker.com/repository/docker/virtualflybrain/vfb-pipeline-dumps
8+
---
99

10+
The **data-dump stage** of the [Virtual Fly Brain](https://virtualflybrain.org) (VFB) pipeline.
11+
12+
This repository is a Dockerised batch job that **queries VFB's assembled triplestore and produces the
13+
final data artifacts that serve the website**: a Solr search index, an OWLery reasoning ontology, and
14+
the Neo4j "PDB" production graph (plus side-loaded connectome/expression data).
15+
16+
It runs *after* [`vfb-pipeline-collectdata`](https://github.com/VirtualFlyBrain/vfb-pipeline-collectdata),
17+
which assembles the ontologies and knowledge-base export and loads them into the triplestore. The
18+
overall chain is:
19+
20+
```
21+
collectdata → (triplestore) → dumps → Solr + OWLery + Neo4j PDB → live site
22+
```
23+
24+
## How it works
25+
26+
- [`process.sh`](process.sh) is the container entrypoint. It creates the output directories, symlinks
27+
any `connectome_*.owl` side-load files into `raw/`, then runs the pipeline with
28+
`make -j${CORES} all`.
29+
- [`dumps.Makefile`](dumps.Makefile) (copied into the image as `Makefile`) is the actual engine — a
30+
dependency graph of SPARQL + ROBOT steps. Running the stages as Make targets gives parallelism
31+
(`-j`) and incremental rebuilds for free.
32+
- The data source is the SPARQL endpoint set by `SPARQL_ENDPOINT`
33+
(default `http://ts.p2.virtualflybrain.org/rdf4j-server/repositories/vfb`), i.e. the RDF4J
34+
triplestore populated by `collectdata`.
35+
36+
Output is written to the container's `/out` volume:
37+
38+
| Directory | Contents |
39+
|---|---|
40+
| `/out/raw` | Intermediate TTL/OWL files produced by each query and reasoning step |
41+
| `/out/dumps` | Final artifacts (`solr.json`, `owlery.owl`, `pdb.owl`) and `csv_imports/` |
42+
| `/out/dumps.log` | Full build log |
43+
44+
## Pipeline steps
45+
46+
1. **Remove embargoed data.** The `remove_embargoed_data` target runs every
47+
[`sparql/delete_*.sparql`](sparql) as a SPARQL `UPDATE` against `${SPARQL_ENDPOINT}/statements`,
48+
deleting embargoed data directly in the triplestore before anything is dumped.
49+
50+
2. **Construct dumps** (one per query). The core Make pattern rules turn each
51+
`sparql/construct_<name>.sparql` query into an OWL file:
52+
- `raw/<name>.ttl` — run the CONSTRUCT query against the endpoint (`curl`), save as Turtle.
53+
- `raw/construct_<name>.owl` — `robot merge`, stamp an ontology IRI, convert to OWL.
54+
- `raw/construct_all.owl` is special: after merging it **reasons** with ELK
55+
(`SubClass EquivalentClass ClassAssertion`), then `relax` / `reduce` to materialise the class
56+
hierarchy.
57+
58+
There are ~25 construct queries — the core graph (`all`), display/search helpers
59+
(`has_image`, `image_names`, `preferred_roots`, `deprecation_label`), connectivity, GO
60+
cross-references, and per-dataset connectome queries (`hemibrain`, `flywire`, `manc`, `fafb`,
61+
`l1em`, `opticlobe`, …).
62+
63+
3. **Enrich.** Two external JARs (downloaded in the `Dockerfile`) add inferred data:
64+
- `infer-annotate.jar` ([vfb_expression_annotator](https://github.com/VirtualFlyBrain/vfb_expression_annotator))
65+
→ `raw/inferred_annotation.owl` and `raw/unique_facets.owl`.
66+
- Reasoned sub-class queries (`constructReasoned_has_subClass.sparql`) run over the merged data and
67+
the side-load ontologies.
68+
69+
4. **Build the three final products** (the `all` target), each a different merge of the constructed
70+
pieces:
71+
72+
| Output | Consumer | Built from |
73+
|---|---|---|
74+
| `dumps/solr.json` | **Solr** search / autocomplete index | `DUMPS_SOLR` (`all preferred_roots deprecation_label image_names has_image`) → `obographs.json` (ROBOT `convert -f json`) → `obographs-solr.py` |
75+
| `dumps/owlery.owl` | **OWLery** reasoning endpoint (class-expression queries) | `DUMPS_OWLERY` (`all`) + reasoned + side-loads, filtered to logical axioms only |
76+
| `dumps/pdb.owl` + `dumps/csv_imports/` | **PDB** — the production Neo4j graph the site queries | `DUMPS_PDB` (`all preferred_roots deprecation_label has_image`) + reasoned + inferred; `owl2neo4jcsv.jar` converts `pdb.owl` to Neo4j bulk-import CSVs |
77+
78+
5. **Side-loads.** Large connectivity / expression datasets
79+
(`PDB_EXTERNAL_ONTS = connectome_*.owl VFB_scRNAseq_exp_*.owl VFB_EPseq_exp_*.owl`) are **not**
80+
merged into the main graph. The `pdb_sideloads` target converts each into **edge-only** Neo4j CSVs
81+
(`owl2neo4jcsv.jar … only_edges`), keeping the huge connectome/expression data out of the
82+
reasoning and merge steps.
83+
84+
## Configuration
85+
86+
Defaults are set in the [`Dockerfile`](Dockerfile); override via `docker run --env`.
87+
88+
| Variable | Default | Purpose |
89+
|---|---|---|
90+
| `SPARQL_ENDPOINT` | `http://ts.p2.virtualflybrain.org/rdf4j-server/repositories/vfb` | Source triplestore (RDF4J) — **must be set** |
91+
| `VFB_CONFIG` | `http://virtualflybrain.org/config/neo4j2owl-config.yaml` | neo4j2owl config (CURIE map, filters) used by the Solr and CSV steps |
92+
| `CORES` | `4` | Parallel `make` jobs |
93+
| `ROBOT_ARGS` | `-Xmx20G -D…parallelism=1` | JVM args applied to ROBOT and the JARs (`ROBOT_JAVA_ARGS` / `JAVA_OPTS`) |
94+
| `INFER_ANNOTATE_RELATION` | `http://n2o.neo/property/nodeLabel` | Relation used by the annotation inference / CSV export |
95+
| `UNIQUE_FACETS_ANNOTATION` | `http://n2o.neo/property/uniqueFacets` | Annotation for unique-facet inference |
96+
| `STDOUT_FILTER` | grep filter | Appended to ROBOT commands to suppress noisy parser warnings |
97+
98+
`process.sh` sets the directory variables (`OUTDIR`, `RAW_DUMPS_DIR`, `FINAL_DUMPS_DIR`,
99+
`SPARQL_DIR`, `SCRIPTS_DIR`); the `checkenv` target fails fast if any required variable is missing.
100+
101+
### Adding a new dump
102+
103+
1. Pick a name and add it to the relevant list in `dumps.Makefile` (`DUMPS_SOLR`, `DUMPS_PDB`,
104+
and/or `DUMPS_OWLERY`).
105+
2. Create `sparql/construct_<name>.sparql`.
106+
107+
Non-SPARQL goals (e.g. `inferred_annotation`) need their own Make target.
108+
109+
## Repository layout
110+
111+
| Path | Purpose |
112+
|---|---|
113+
| [`process.sh`](process.sh) | Container entrypoint; sets up dirs and runs `make all` |
114+
| [`dumps.Makefile`](dumps.Makefile) | The pipeline definition (copied to `Makefile` in the image) |
115+
| [`Makefile`](Makefile) | Local Docker build/run/publish helpers (**not** the pipeline) |
116+
| [`sparql/`](sparql) | `construct_*.sparql` (dumps), `constructReasoned_*.sparql`, `delete_*.sparql` (embargo) |
117+
| [`scripts/obographs-solr.py`](scripts/obographs-solr.py) | Converts OBOGraphs JSON → VFB Solr documents |
118+
| [`scripts/lib.py`](scripts/lib.py) | Helpers (ID/CURIE variants, JSON I/O) |
119+
| [`scripts/clean_owl.py`](scripts/clean_owl.py) | OWL cleanup helper |
120+
| [`scripts/run_sparql_in_chunks.sh`](scripts/run_sparql_in_chunks.sh) | Runs a SPARQL query in paged chunks for large result sets |
121+
122+
The `owl2neo4jcsv.jar` ([neo4j2owl](https://github.com/VirtualFlyBrain/neo4j2owl)) and
123+
`infer-annotate.jar` are downloaded into `scripts/` at image-build time (see `Dockerfile`).
124+
125+
## Building and running
126+
127+
The [`Makefile`](Makefile) wraps the Docker commands:
128+
129+
```bash
130+
# Build the image
131+
make docker-build # with cache
132+
make docker-build-no-cache
133+
134+
# Run the pipeline (writes to OUTDIR, default ~/data/vfb)
135+
make docker-run
136+
137+
# Publish to Docker Hub
138+
make docker-publish
139+
```
140+
141+
Override variables inline, e.g.:
142+
143+
```bash
144+
make docker-run OUTDIR=/path/to/out SPARQL_ENDPOINT=http://my-triplestore/repositories/vfb
145+
```
146+
147+
The image mounts two volumes: `/out` (output data) and `/logs`.
148+
149+
## Continuous integration
150+
151+
[`.github/workflows/docker-image.yml`](.github/workflows/docker-image.yml) builds the image on every
152+
push and on PRs against `master`. Pushes are published to Docker Hub as
153+
`virtualflybrain/vfb-pipeline-dumps` tagged with the branch name; PR builds are built but not pushed.
154+
155+
## Tools used
156+
157+
- [ROBOT](http://robot.obolibrary.org/) — OWL manipulation, ELK reasoning, format conversion
158+
- [ELK](https://github.com/liveontologies/elk-reasoner) — OWL EL reasoner (via ROBOT)
159+
- [neo4j2owl / owl2neo4jcsv](https://github.com/VirtualFlyBrain/neo4j2owl) — OWL → Neo4j CSV conversion
160+
- [vfb_expression_annotator](https://github.com/VirtualFlyBrain/vfb_expression_annotator) — annotation / unique-facet inference
161+
- [OBOGraphs](https://github.com/geneontology/obographs) JSON — intermediate format for the Solr index

0 commit comments

Comments
 (0)