Skip to content

Commit 715f323

Browse files
MathiasVDAMathiasVDA
committed
feat: inspect SPARQL Anything sources for autocompletion, preview and a query builder
A chip on each closed SERVICE <x-sparql-anything:...> block loads, on request, the structure of its source with one GROUP BY query to the endpoint of the tab, sending the user's SERVICE IRI and fx:properties options unchanged. The result feeds: - autocompletion: the columns, keys and XML element types come first, under "From <file>" (new yasqe contextTerms option) - a Source view in the side panel, only listed while the query has such a block: the structure tree (click to insert), a Turtle preview of the first 100 triples, and a query builder that writes fx:anySlot paths with OPTIONAL for optional values Loaded structures are cached per endpoint and options for a day. Turn it off with sparqlAnything: false. The side panel gets setViewVisible(). dev/sparql-anything has a Docker Compose test environment (facadex/sparql-anything 1.3.0, a file server only the endpoint can reach, sample CSV/TSV/JSON/XML files, and a curl script for the structure query).
1 parent 8bc9572 commit 715f323

34 files changed

Lines changed: 3393 additions & 24 deletions

‎.changeset/sparql-anything-sources.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
---
2+
"@matdata/yasgui": minor
3+
"@matdata/yasqe": minor
4+
---
5+
6+
Help with SPARQL Anything queries. A chip after the opening brace of each `SERVICE <x-sparql-anything:...> { }` block loads, on request, the structure of its source with one query to the endpoint of the tab (the SERVICE IRI and `fx:properties` options are sent unchanged, so files that only the endpoint can reach work too). Once loaded, the columns, keys and XML elements (`xyz:email`, `xyz:book`) are suggested first in autocompletion, and a new **Source** view of the side panel shows the structure (click to insert), a preview of the first 100 triples, and a query builder that writes the `fx:anySlot` patterns for the values you tick, with OPTIONAL for optional values. The Source view is only listed while the query has such a block, and nothing is sent for other queries. Loaded structures are remembered per endpoint and options for a day. Switch it off with `sparqlAnything: false`.
7+
8+
YASQE gets `getSparqlAnythingSources()`, `getSparqlAnythingSourceAt()`, `Yasqe.SparqlAnything.findSparqlAnythingSources()`, and the `contextTerms` and `sparqlAnythingChip` config options. The side panel gets `setViewVisible(view, visible)`. `dev/sparql-anything` has a Docker Compose test environment with a SPARQL Anything server and sample CSV, TSV, JSON and XML files.

‎dev/sparql-anything/README.md‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
# SPARQL Anything test environment
2+
3+
A [SPARQL Anything](https://sparql-anything.cc) server with a few sample files, to try the SPARQL Anything support
4+
of MatGUI: the chip on `SERVICE <x-sparql-anything:...>` blocks, autocompletion of columns, keys and elements, and
5+
the **Source** view of the side panel (structure, preview, query builder).
6+
7+
```bash
8+
cd dev/sparql-anything
9+
docker compose up -d
10+
```
11+
12+
- Endpoint: `http://localhost:3000/sparql.anything` (it also has its own Yasgui at `http://localhost:3000/sparql`).
13+
- The files in `data/` are mounted in the server at `/data/...` and served over HTTP at `http://files/...` inside the
14+
compose network only. Neither is reachable from the browser, as with files that only the endpoint can read.
15+
16+
Use it with MatGUI in one of these ways:
17+
18+
- `npm run dev` in the repository root and open `http://localhost:4000/yasgui.html`, then switch the endpoint to
19+
`http://localhost:3000/sparql.anything`.
20+
- `docker compose --profile matgui up -d --build` builds MatGUI from this checkout and serves it on
21+
`http://localhost:8080`, with the SPARQL Anything endpoint as the default.
22+
23+
Stop with `docker compose down`.
24+
25+
`./check-structure.sh` sends the structure query of the Source view for each sample file with curl and prints the
26+
rows, which is handy to see what MatGUI gets back without a browser.
27+
28+
## Example queries
29+
30+
Paste a query, wait for the chip after the opening brace of the SERVICE block, and click **Load structure**. Then
31+
type `xyz:` inside the block to see the columns, or click **Inspect** to open the Source view.
32+
33+
CSV with a header row:
34+
35+
```sparql
36+
PREFIX fx: <http://sparql.xyz/facade-x/ns/>
37+
PREFIX xyz: <http://sparql.xyz/facade-x/data/>
38+
SELECT * WHERE {
39+
SERVICE <x-sparql-anything:location=/data/people.csv,csv.headers=true> {
40+
?root a fx:root ; fx:anySlot ?row .
41+
?row xyz:name ?name .
42+
}
43+
}
44+
```
45+
46+
The same file over HTTP, with the options written as triples:
47+
48+
```sparql
49+
PREFIX fx: <http://sparql.xyz/facade-x/ns/>
50+
PREFIX xyz: <http://sparql.xyz/facade-x/data/>
51+
SELECT * WHERE {
52+
SERVICE <x-sparql-anything:> {
53+
fx:properties fx:location "http://files/people.csv" ;
54+
fx:csv.headers true .
55+
?row xyz:email ?email .
56+
}
57+
}
58+
```
59+
60+
Tab separated values:
61+
62+
```sparql
63+
SELECT * WHERE {
64+
SERVICE <x-sparql-anything:location=/data/stations.tsv,csv.headers=true,csv.format=TDF> {
65+
?s ?p ?o
66+
}
67+
}
68+
```
69+
70+
Nested JSON (objects, arrays, arrays of strings):
71+
72+
```sparql
73+
PREFIX fx: <http://sparql.xyz/facade-x/ns/>
74+
PREFIX xyz: <http://sparql.xyz/facade-x/data/>
75+
SELECT * WHERE {
76+
SERVICE <x-sparql-anything:location=http://files/books.json> {
77+
?s ?p ?o
78+
}
79+
}
80+
```
81+
82+
XML (elements become types, attributes properties, text in slots):
83+
84+
```sparql
85+
PREFIX fx: <http://sparql.xyz/facade-x/ns/>
86+
PREFIX xyz: <http://sparql.xyz/facade-x/data/>
87+
SELECT * WHERE {
88+
SERVICE <x-sparql-anything:location=/data/library.xml> {
89+
?book a xyz:book .
90+
}
91+
}
92+
```
93+
94+
A location that is a variable can't be inspected before the query runs; the chip says so:
95+
96+
```sparql
97+
PREFIX fx: <http://sparql.xyz/facade-x/ns/>
98+
SELECT * WHERE {
99+
VALUES ?file { "/data/people.csv" }
100+
SERVICE <x-sparql-anything:> {
101+
fx:properties fx:location ?file .
102+
?s ?p ?o
103+
}
104+
}
105+
```

‎dev/sparql-anything/check-structure.sh‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
#!/bin/sh
2+
# Sends the structure query of the MatGUI Source view for each sample file to the local SPARQL Anything server
3+
# and prints the result as CSV. Run after `docker compose up -d`.
4+
ENDPOINT="${ENDPOINT:-http://localhost:3000/sparql.anything}"
5+
6+
structure_query() {
7+
cat <<QUERY | sed "s|__OPTIONS__|$1|"
8+
SELECT ?gstep ?ptype ?pstep ?type ?step ?kind (COUNT(*) AS ?n) (COUNT(DISTINCT ?s) AS ?subjects) (SAMPLE(?o) AS ?example)
9+
WHERE {
10+
SERVICE <x-sparql-anything:__OPTIONS__> {
11+
?s ?p ?o .
12+
OPTIONAL { ?s <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> ?t }
13+
OPTIONAL { ?parent ?pp ?s . OPTIONAL { ?parent <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> ?pt } OPTIONAL { ?grand ?gp ?parent } }
14+
}
15+
FILTER(?p != <http://www.w3.org/1999/02/22-rdf-syntax-ns#type>)
16+
BIND(IF(STRSTARTS(STR(?p), "http://www.w3.org/1999/02/22-rdf-syntax-ns#_") || ?p = <http://www.w3.org/2000/01/rdf-schema#member>, <urn:matgui:sparql-anything:slot>, ?p) AS ?step)
17+
BIND(IF(STRSTARTS(STR(?pp), "http://www.w3.org/1999/02/22-rdf-syntax-ns#_") || ?pp = <http://www.w3.org/2000/01/rdf-schema#member>, <urn:matgui:sparql-anything:slot>, ?pp) AS ?pstep)
18+
BIND(IF(STRSTARTS(STR(?gp), "http://www.w3.org/1999/02/22-rdf-syntax-ns#_") || ?gp = <http://www.w3.org/2000/01/rdf-schema#member>, <urn:matgui:sparql-anything:slot>, ?gp) AS ?gstep)
19+
BIND(?t AS ?type)
20+
BIND(?pt AS ?ptype)
21+
BIND(IF(isLiteral(?o), "literal", "node") AS ?kind)
22+
}
23+
GROUP BY ?gstep ?ptype ?pstep ?type ?step ?kind
24+
LIMIT 5000
25+
QUERY
26+
}
27+
28+
for options in \
29+
"location=/data/people.csv,csv.headers=true" \
30+
"location=http://files/people.csv,csv.headers=true" \
31+
"location=/data/stations.tsv,csv.headers=true,csv.format=TDF" \
32+
"location=http://files/books.json" \
33+
"location=/data/library.xml"; do
34+
echo "== $options"
35+
curl -sS -X POST "$ENDPOINT" -H "Accept: text/csv" --data-urlencode "query=$(structure_query "$options")"
36+
echo
37+
done
38+
39+
echo "== preview (first 10 triples of books.json)"
40+
curl -sS -X POST "$ENDPOINT" -H "Accept: text/turtle" --data-urlencode "query=PREFIX fx: <http://sparql.xyz/facade-x/ns/>
41+
PREFIX xyz: <http://sparql.xyz/facade-x/data/>
42+
CONSTRUCT { ?s ?p ?o } WHERE { SERVICE <x-sparql-anything:location=http://files/books.json> { ?s ?p ?o } } LIMIT 10"

‎dev/sparql-anything/data/books.json‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
{
2+
"library": "Matdata reading room",
3+
"books": [
4+
{
5+
"title": "Dune",
6+
"year": 1965,
7+
"isbn": "978-0441013593",
8+
"author": { "name": "Frank Herbert", "born": 1920 },
9+
"tags": ["science fiction", "desert"]
10+
},
11+
{
12+
"title": "The Left Hand of Darkness",
13+
"year": 1969,
14+
"author": { "name": "Ursula K. Le Guin", "born": 1929 },
15+
"tags": ["science fiction"]
16+
},
17+
{
18+
"title": "Neuromancer",
19+
"year": 1984,
20+
"isbn": "978-0441569595",
21+
"author": { "name": "William Gibson" },
22+
"tags": ["cyberpunk"]
23+
}
24+
]
25+
}

‎dev/sparql-anything/data/library.xml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<library name="Matdata reading room">
3+
<book id="b1" lang="en">
4+
<title>Dune</title>
5+
<author born="1920">Frank Herbert</author>
6+
<year>1965</year>
7+
</book>
8+
<book id="b2" lang="en">
9+
<title>The Left Hand of Darkness</title>
10+
<author born="1929">Ursula K. Le Guin</author>
11+
<year>1969</year>
12+
</book>
13+
<magazine id="m1">
14+
<title>Railway Gazette</title>
15+
<issue>2026-10</issue>
16+
</magazine>
17+
</library>

‎dev/sparql-anything/data/people.csv‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
id,name,email,employer,city,birth_date,active
2+
P-0001,Laura Grey,laura@example.com,Acme NV,Ghent,1987-03-12,true
3+
P-0002,Craig Johnson,craig@example.com,Infrabel,Brussels,1979-11-02,false
4+
P-0003,Ana Lopez,ana@example.com,Acme NV,Antwerp,1992-06-21,true
5+
P-0004,Tom De Smet,,Matdata,Leuven,1985-01-30,true
6+
P-0005,Mia Peeters,mia@example.com,Infrabel,Ghent,1990-09-14,true

‎dev/sparql-anything/data/stations.tsv‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
code name country lat lon
2+
FBMZ Brussel-Zuid BE 50.8357 4.3361
3+
FGSP Gent-Sint-Pieters BE 51.0362 3.7104
4+
FN Antwerpen-Centraal BE 51.2172 4.4211
5+
NLASC Amsterdam Centraal NL 52.3791 4.9003

‎dev/sparql-anything/docker-compose.yml‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
# Test environment for the SPARQL Anything support of MatGUI (Source view, chips, autocompletion).
2+
#
3+
# docker compose up -d # SPARQL Anything + a file server only reachable by it
4+
# docker compose --profile matgui up -d # also a MatGUI built from this checkout, on http://localhost:8080
5+
#
6+
# Endpoint: http://localhost:3000/sparql.anything (CORS enabled). See README.md for example queries.
7+
name: matgui-sparql-anything
8+
9+
services:
10+
sparql-anything:
11+
image: facadex/sparql-anything:v1.3.0
12+
ports:
13+
- "3000:3000"
14+
volumes:
15+
# Files the server reads from disk: location=/data/people.csv
16+
- ./data:/data:ro
17+
restart: unless-stopped
18+
19+
# Serves the same files over HTTP inside the compose network only (http://files/people.csv), like a file that
20+
# only the endpoint can reach: the browser can't fetch it, so MatGUI has to ask the endpoint.
21+
files:
22+
image: nginx:1.27-alpine
23+
volumes:
24+
- ./data:/usr/share/nginx/html:ro
25+
restart: unless-stopped
26+
27+
matgui:
28+
profiles: ["matgui"]
29+
build:
30+
context: ../..
31+
dockerfile: docker/Dockerfile
32+
environment:
33+
MATGUI_DEFAULT_ENDPOINT: http://localhost:3000/sparql.anything
34+
ports:
35+
- "8080:8080"
36+
depends_on:
37+
- sparql-anything

‎docs/developer-guide.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -723,6 +723,11 @@ interface Config {
723723

724724
// AI assistant (see AI Assistant Configuration); { enabled: false } removes it
725725
ai: Partial<AiConfig>;
726+
/**
727+
* SPARQL Anything support (default: true): a chip on each `SERVICE <x-sparql-anything:...>` block loads the
728+
* structure of its source from the endpoint, for autocompletion and the "Source" view of the side panel.
729+
*/
730+
sparqlAnything?: boolean;
726731
}
727732
```
728733

@@ -893,6 +898,11 @@ interface YasqeConfig {
893898
// show its label, description and type). Return nothing to show no tooltip.
894899
iriHover: ((yasqe: Yasqe, iri: IriAtPosition) => HTMLElement | null | undefined) | undefined;
895900
iriHoverDelay: number; // default: 500 (ms)
901+
// Terms for the place in the query, suggested before the endpoint terms, each group under its own heading
902+
// (Yasgui: the columns, keys and elements of a SPARQL Anything source)
903+
contextTerms: ((yasqe: Yasqe, type: "class" | "property") => Promise<ContextTerms[] | undefined> | ContextTerms[] | undefined) | undefined;
904+
// Chip after the opening brace of each SERVICE <x-sparql-anything:...> block (undefined: no chips)
905+
sparqlAnythingChip: ((yasqe: Yasqe, source: SparqlAnythingSource) => HTMLElement | null | undefined) | undefined;
896906
}
897907
```
898908

‎docs/user-guide.md‎

Lines changed: 37 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@
4848
- [IRI Hover Info](#iri-hover-info)
4949
- [Endpoint Overview](#endpoint-overview)
5050
- [AI Assistant](#ai-assistant)
51+
- [SPARQL Anything Sources](#sparql-anything-sources)
5152
- [Query Tabs](#query-tabs)
5253
- [Settings Modal](#settings-modal)
5354
- [SPARQL Endpoints Management](#sparql-endpoints-management)
@@ -419,10 +420,11 @@ You can also find a snippet with the [search](#search) (`Ctrl+K`). The snippets
419420

420421
### Side Panel
421422

422-
The side panel on the left has three views, selected with **Queries**, **Endpoint** and **AI** at its top:
423+
The side panel on the left has up to four views, selected with **Queries**, **Endpoint**, **Source** and **AI** at its top:
423424

424425
- **Queries**: the query browser for your workspaces. Pick a workspace, search, open a saved query in a new tab, or save the current tab with **Save current tab**
425426
- **Endpoint**: the endpoint overview of the current endpoint (see [Endpoint Overview](#endpoint-overview))
427+
- **Source**: the structure, a preview and a query builder for the file of a SPARQL Anything `SERVICE` block (see [SPARQL Anything Sources](#sparql-anything-sources)). This view is only listed while the query of the tab has such a block
426428
- **AI**: the AI assistant, which writes, explains and fixes queries (see [AI Assistant](#ai-assistant)). The **AI** button in the editor toolbar opens it too. A site can switch it off, and then this view is not shown
427429

428430
While the panel is closed, a narrow bar of icons stays at the left edge: **Open side panel**, then one button per view. Click a view to open the panel on it. You can also press `Ctrl+B` (`⌘B` on a Mac) to show or hide the panel, `F8` to toggle the Endpoint view, or use **Show saved queries** and **Explore endpoint** in the [search](#search).
@@ -1027,6 +1029,40 @@ To switch them off, untick **Let the assistant run check queries on the endpoint
10271029

10281030
The class and property names come from the same place as the [autocomplete](#class-and-property-autocomplete): the endpoint's VoID description, the [Endpoint Overview](#endpoint-overview) results, or one limited background query. When you switched autocomplete off for an endpoint, the assistant gets only your prefixes for it. Telling the assistant which vocabulary to use ("use the ERA ontology") also helps.
10291031

1032+
### SPARQL Anything Sources
1033+
1034+
[SPARQL Anything](https://sparql-anything.cc) queries CSV, JSON, XML and other files as if they were RDF, with a `SERVICE <x-sparql-anything:...>` block. The column names, keys and element names of the file become properties and types in the `xyz:` namespace (`http://sparql.xyz/facade-x/data/`), for example `xyz:email` for an `email` column. MatGUI can find these for you, so you don't have to open the file to know what to write.
1035+
1036+
```sparql
1037+
PREFIX fx: <http://sparql.xyz/facade-x/ns/>
1038+
PREFIX xyz: <http://sparql.xyz/facade-x/data/>
1039+
SELECT * WHERE {
1040+
SERVICE <x-sparql-anything:location=https://example.org/people.csv,csv.headers=true> {
1041+
?root a fx:root ; fx:anySlot ?row .
1042+
?row xyz:email ?email .
1043+
}
1044+
}
1045+
```
1046+
1047+
**Loading the structure**
1048+
1049+
As soon as you close the block with `}`, a chip appears after its opening brace, with the file name and **Load structure**. Clicking it sends one query to the endpoint of the tab, with your SERVICE IRI and `fx:properties` options unchanged. The endpoint reads the file, so this also works for files that only the endpoint can reach (a path on the server, a URL behind a firewall, or a URL that needs authentication). Nothing is sent before you click: SPARQL Anything reads the whole file to answer, which can take a while for big files. While it runs the chip offers **Cancel**; when it fails it shows **Retry**, with the endpoint's error when you hover it.
1050+
1051+
When the structure is loaded, the chip shows the number of properties and **Inspect**, and:
1052+
1053+
- **Autocompletion** suggests the properties of the file (`Ctrl+Space` after `xyz:` or in a property position) first, under **From people.csv**, with how often they occur. After `a`, the XML element types are suggested. Picking one adds `PREFIX xyz:` when it is missing. Inside a SERVICE block only that block's source is used; elsewhere in the query, all loaded sources.
1054+
- The **Source** view of the side panel (**Inspect** on the chip, or **Inspect people.csv** in the [search](#search)) shows the source of the block the cursor is in. With several blocks, a picker at the top switches between them.
1055+
1056+
The structure is remembered per endpoint and set of options for a day; the reload button in the Source view loads it again. A block whose location is a variable (`fx:location ?file`) can't be inspected before the query runs, and its chip says so.
1057+
1058+
**The Source view**
1059+
1060+
- **Structure**: the tree of the file as SPARQL Anything sees it: the document (`fx:root`), its rows, array items or elements, and their properties, each with a count and an example value. Click a property or element to insert it at the cursor.
1061+
- **Preview**: the first 100 triples of the converted file, in Turtle, fetched with a `CONSTRUCT ... LIMIT 100` query. **Open as query** puts that query in a new tab.
1062+
- **Build query**: tick the values you need. MatGUI writes the patterns from the root to each value: `fx:anySlot` for rows and array items, `a xyz:<element>` for XML elements, and `OPTIONAL` for values you mark as optional (the first value you tick is required, later ones optional; click the label to switch). **Replace block** puts the patterns in the SERVICE block (keeping its `fx:properties` options; `Ctrl+Z` undoes it), **Insert at cursor** inserts them, and **New tab** opens a complete SELECT query.
1063+
1064+
CSV, JSON and XML sources are shown best; other formats SPARQL Anything supports work too, with a more generic tree. A site can switch all of this off with `sparqlAnything: false`. To try it, the repository has a Docker Compose setup with a SPARQL Anything server and sample files in `dev/sparql-anything`.
1065+
10301066
### Query Tabs
10311067

10321068
Manage multiple queries simultaneously with tabs.

0 commit comments

Comments
 (0)