feat(yasqe): always include the SPARQL Anything vocabularies

Adds the SPARQL Anything namespaces xyz (http://sparql.xyz/facade-x/data/),
fx (http://sparql.xyz/facade-x/ns/) and fxe (http://sparql.xyz/facade-x/engine/)
to ALWAYS_INCLUDED. SPARQL Anything publishes no RDF files, so the fx magic
property, class and functions and the fxe options are listed there.

- fetchPrefixData.js adds the prefixes of ALWAYS_INCLUDED to the prefix.cc
  data, so xyz, fx, fxe (and railml) survive the refresh at build time.
- The bundled terms get an optional fifth list of functions, suggested by a
  new function autocompleter where an expression can start.

Author: MathiasVDA

.changeset/sparql-anything-vocabularies.md

@@ -0,0 +1,5 @@
+---
+"@matdata/yasqe": minor
+---
+
+Always include the SPARQL Anything vocabularies: the `xyz:`, `fx:` and `fxe:` prefixes are always in the bundled prefix list (also after a prefix.cc refresh), the `fx:anySlot` magic property, the `fx:Root` class and the `fxe:` options (`fxe:location`, `fxe:csv.headers`...) are suggested as properties and classes, and a new `function` autocompleter suggests the `fx:` functions (`fx:entity`, `fx:String.substring`...) where an expression can start.

docs/user-guide.md

@@ -714,6 +714,8 @@ When typing a prefix declaration, Matgui queries [prefix.cc](https://prefix.cc)
 PREFIX foaf: <  # Auto-suggests http://xmlns.com/foaf/0.1/
 ```
 
+MatGUI ships the prefix.cc list, so this works offline too. A few prefixes are always in it, with these namespaces, whatever prefix.cc says: `era:` (ERA ontology), `railml:` (railML 3), `rtm:` (RailTopoModel) and the three [SPARQL Anything](https://sparql-anything.cc) namespaces `xyz:` (`http://sparql.xyz/facade-x/data/`), `fx:` (`http://sparql.xyz/facade-x/ns/`) and `fxe:` (`http://sparql.xyz/facade-x/engine/`). Typing one of them in a query, such as `fxe:location`, adds its `PREFIX` declaration.
+
 ### Endpoint Quick Switch
 
 Quick switch endpoints let you change the SPARQL endpoint of a tab with a single click. They are listed in the **Quick switch** group at the top of the endpoint dropdown (see [Endpoint Picker](#endpoint-picker)).
@@ -862,7 +864,30 @@ Ctrl+clicking on `http://dbpedia.org/resource/European_Union` automatically quer
 Press `Ctrl+Space` after `a` (or `rdf:type`) to get class suggestions, or in the position of a property to get property suggestions. MatGUI suggests:
 
 1. **Used in this endpoint**: the classes and properties the endpoint of the tab actually uses, most used first, with the number of instances (classes) or triples (properties) next to them. This also covers vocabularies that are not registered anywhere, such as railML 3, the ERA ontology or your own ontologies.
-2. **Linked Open Vocabularies**: matching classes and properties of the most used vocabularies in [LOV](https://lov.linkeddata.es/dataset/lov/). MatGUI ships this list, so no request is made to LOV while you type. Terms that the endpoint already suggested are not repeated, and a term is inserted as a prefixed name with its `PREFIX` when LOV knows the vocabulary's preferred prefix. The ERA ontology (`era:`), railML 3 (`railml:`) and RailTopoModel (`rtm:`) are always included, even though they are not in LOV.
+2. **Linked Open Vocabularies**: matching classes and properties of the most used vocabularies in [LOV](https://lov.linkeddata.es/dataset/lov/). MatGUI ships this list, so no request is made to LOV while you type. Terms that the endpoint already suggested are not repeated, and a term is inserted as a prefixed name with its `PREFIX` when LOV knows the vocabulary's preferred prefix. The ERA ontology (`era:`), railML 3 (`railml:`), RailTopoModel (`rtm:`) and the SPARQL Anything vocabularies (`fx:`, `fxe:`, see below) are always included, even though they are not in LOV.
+
+**SPARQL Anything**
+
+The [SPARQL Anything](https://sparql-anything.cc) terms are always suggested, so you can write its queries without looking up the documentation:
+
+- `fx:` (`http://sparql.xyz/facade-x/ns/`): the `fx:anySlot` magic property and the `fx:Root` class as properties and classes, and the functions, such as `fx:entity`, `fx:literal`, `fx:String.substring` or `fx:LevenshteinDistance`, wherever an expression can start (in `BIND`, `FILTER`, a `SELECT` expression...). A function also matches on its local name, so `substr` finds `fx:String.substring`.
+- `fxe:` (`http://sparql.xyz/facade-x/engine/`): the options of a SPARQL Anything `SERVICE` clause, such as `fxe:location`, `fxe:media-type`, `fxe:csv.headers` or `fxe:json.path`.
+- `xyz:` (`http://sparql.xyz/facade-x/data/`): the namespace of the generated data. Its terms come from your data (`xyz:name`), so only the prefix is known, unless the endpoint uses them.
+
+```sparql
+PREFIX xyz: <http://sparql.xyz/facade-x/data/>
+PREFIX fx: <http://sparql.xyz/facade-x/ns/>
+PREFIX fxe: <http://sparql.xyz/facade-x/engine/>
+
+SELECT ?name WHERE {
+  SERVICE <x-sparql-anything:> {
+    fxe:properties fxe:location "https://sparql-anything.cc/example1.json" .
+    ?show xyz:name ?name ; xyz:stars ?star .
+    ?star fx:anySlot "Courteney Cox" .
+  }
+  BIND(fx:entity("https://example.org/show/", fx:String.toLowerCase(?name)) AS ?iri)
+}
+```
 
 Without typing anything, the endpoint's most used terms are listed. While you type, a suggestion matches on its full IRI, its prefixed name (`era:Op`) or, when you type a plain word (`Operat`), on any part of its local name.
 

fetchLovData.js

@@ -29,8 +29,11 @@ const MAX_BYTES = 600_000;
 
 /**
  * Namespaces that are always bundled, first, regardless of their popularity and of the size budget.
- * `prefix` overrides the prefix from LOV. `sources` are RDF files (Turtle, N-Triples...) of the vocabulary whose
+ * `prefix` overrides the prefix from LOV; fetchPrefixData.js also adds it to the bundled prefixes, so it is always
+ * suggested and declared when used. `sources` are RDF files (Turtle, N-Triples...) of the vocabulary whose
  * classes and properties in the namespace are added to those of the dump, e.g. when LOV lacks the vocabulary.
+ * `terms` lists local names of classes, properties and functions, for vocabularies without RDF files.
+ * A namespace without any terms is left out of the bundle, but its prefix is still bundled.
  */
 export const ALWAYS_INCLUDED = [
   {
@@ -53,6 +56,150 @@ export const ALWAYS_INCLUDED = [
     prefix: "rtm",
     sources: ["https://ontology.railml.org/ontology.ttl"],
   },
+  // SPARQL Anything (https://sparql-anything.cc), which publishes no RDF files for its vocabularies, so the terms are
+  // listed here, from https://github.com/SPARQL-Anything/sparql.anything/tree/HEAD/docs and the Façade-X specs.
+  {
+    // Façade-X: the terms of the generated data, the magic properties and the functions
+    namespace: "http://sparql.xyz/facade-x/ns/",
+    prefix: "fx",
+    terms: {
+      classes: ["Root"],
+      properties: ["anySlot", "slot-key", "cachedGraph", "cachedGraphCreation", "sparqlAlgebra"],
+      functions: [
+        "cardinal",
+        "isContainerMembershipProperty",
+        "before",
+        "after",
+        "previous",
+        "next",
+        "forward",
+        "backward",
+        "String.startsWith",
+        "String.endsWith",
+        "String.indexOf",
+        "String.lastIndexOf",
+        "String.substring",
+        "String.toLowerCase",
+        "String.toUpperCase",
+        "String.trim",
+        "String.replace",
+        "String.stripLeading",
+        "String.stripTrailing",
+        "String.removeTags",
+        "WordUtils.capitalize",
+        "WordUtils.capitalizeFully",
+        "WordUtils.initials",
+        "WordUtils.swapCase",
+        "WordUtils.uncapitalize",
+        "LevenshteinDistance",
+        "CosineDistance",
+        "JaccardDistance",
+        "JaroWinklerDistance",
+        "LongestCommonSubsequenceDistance",
+        "HammingDistance",
+        "QGramDistance",
+        "DigestUtils.md2Hex",
+        "DigestUtils.md5Hex",
+        "DigestUtils.sha1Hex",
+        "DigestUtils.sha256Hex",
+        "DigestUtils.sha384Hex",
+        "DigestUtils.sha512Hex",
+        "URLEncoder.encode",
+        "URLDecoder.decode",
+        "serial",
+        "entity",
+        "literal",
+        "bnode",
+      ],
+    },
+  },
+  {
+    // Façade-X Engine: the options of a SPARQL Anything SERVICE clause (fxe:properties fxe:location "...")
+    namespace: "http://sparql.xyz/facade-x/engine/",
+    prefix: "fxe",
+    terms: {
+      classes: ["Configuration", "Option"],
+      properties: [
+        "properties",
+        "location",
+        "content",
+        "command",
+        "read-from-std-in",
+        "query",
+        "from-archive",
+        "archive-format",
+        "archive.matches",
+        "root",
+        "media-type",
+        "namespace",
+        "blank-nodes",
+        "trim-strings",
+        "null-string",
+        "triplifier",
+        "charset",
+        "metadata",
+        "audit",
+        "ondisk",
+        "ondisk.reuse",
+        "strategy",
+        "slice",
+        "slice.size",
+        "use-rdfs-member",
+        "annotate-triples-with-slot-keys",
+        "generate-predicate-labels",
+        "http.client.useSystemProperties",
+        "http.method",
+        "http.payload",
+        "http.protocol",
+        "http.auth.user",
+        "http.auth.password",
+        "http.redirect",
+        "http.sleep",
+        "s3.endpoint",
+        "s3.bucket-name",
+        "s3.key",
+        "s3.access-key",
+        "s3.secret-key",
+        "s3.region",
+        "bin.encoding",
+        "csv.delimiter",
+        "csv.format",
+        "csv.headers",
+        "csv.headers-row",
+        "csv.headers.sanitize",
+        "csv.ignore-columns-with-no-header",
+        "csv.null-string",
+        "csv.quote-char",
+        "html.browser",
+        "html.browser.screenshot",
+        "html.browser.timeout",
+        "html.browser.wait",
+        "html.metadata",
+        "html.parser",
+        "html.selector",
+        "json.include-null-values",
+        "json.literalize",
+        "json.path",
+        "slides.extract-sections",
+        "spreadsheet.composite-values",
+        "spreadsheet.evaluate-formulas",
+        "spreadsheet.headers",
+        "spreadsheet.headers-row",
+        "spreadsheet.ignore-columns-with-no-header",
+        "txt.group",
+        "txt.regex",
+        "txt.split",
+        "xml.lang-tags",
+        "xml.path",
+        "yaml.allow-duplicate-keys",
+      ],
+    },
+  },
+  {
+    // Façade-X data: the default namespace of the generated data, whose terms come from the source (xyz:name...)
+    namespace: "http://sparql.xyz/facade-x/data/",
+    prefix: "xyz",
+  },
 ];
 
 const RDF_TYPE = "http://www.w3.org/1999/02/22-rdf-syntax-ns#type";
@@ -167,26 +314,29 @@ export async function collectTerms(quads) {
 }
 
 /**
- * Adds the terms of the always included namespaces to the namespaces of the dump: the configured prefix, and
- * the classes and properties of extraTerms (namespaces collected from their sources).
+ * Adds the terms of the always included namespaces to the namespaces of the dump: the configured prefix, the
+ * configured terms, and the classes and properties of extraTerms (namespaces collected from their sources).
  */
 export function addAlwaysIncluded(namespaces, alwaysIncluded = ALWAYS_INCLUDED, extraTerms = []) {
   const result = namespaces.map((ns) => ({ ...ns }));
-  for (const { namespace, prefix } of alwaysIncluded) {
+  const merge = (current = [], added = []) => Array.from(new Set([...current, ...added])).sort();
+  for (const { namespace, prefix, terms } of alwaysIncluded) {
     let ns = result.find((ns) => ns.namespace === namespace);
     if (!ns) result.push((ns = { namespace, classes: [], properties: [], usedBy: 0 }));
     if (prefix) ns.prefix = prefix;
-    for (const extra of extraTerms.filter((extra) => extra.namespace === namespace)) {
-      ns.classes = Array.from(new Set([...ns.classes, ...extra.classes])).sort();
-      ns.properties = Array.from(new Set([...ns.properties, ...extra.properties])).sort();
+    for (const extra of [...(terms ? [terms] : []), ...extraTerms.filter((extra) => extra.namespace === namespace)]) {
+      ns.classes = merge(ns.classes, extra.classes);
+      ns.properties = merge(ns.properties, extra.properties);
+      if (extra.functions?.length) ns.functions = merge(ns.functions, extra.functions);
     }
   }
-  return result.filter((ns) => ns.classes.length || ns.properties.length);
+  return result.filter((ns) => ns.classes.length || ns.properties.length || ns.functions?.length);
 }
 
 /**
  * The bundled format: the always included namespaces, then the others most used first, as
- * [namespace, prefix, classes, properties]. Namespaces are added until the serialized file would exceed maxBytes,
+ * [namespace, prefix, classes, properties] or, for namespaces with functions, [namespace, prefix, classes,
+ * properties, functions]. Namespaces are added until the serialized file would exceed maxBytes,
  * except the always included ones, which are added regardless.
  */
 export function toBundle(namespaces, maxBytes = MAX_BYTES, source = DUMP_URL, alwaysIncluded = ALWAYS_INCLUDED) {
@@ -199,6 +349,7 @@ export function toBundle(namespaces, maxBytes = MAX_BYTES, source = DUMP_URL, al
   ];
   for (const ns of ordered) {
     const entry = [ns.namespace, ns.prefix || "", ns.classes.join(" "), ns.properties.join(" ")];
+    if (ns.functions?.length) entry.push(ns.functions.join(" "));
     const entrySize = JSON.stringify(entry).length + 1;
     if (size + entrySize > maxBytes && always.indexOf(ns.namespace) < 0) continue;
     bundle.namespaces.push(entry);
@@ -268,8 +419,7 @@ async function main() {
   if (stats.skipped) console.log(`Skipped ${stats.skipped} invalid lines`);
   const bundle = toBundle(addAlwaysIncluded(namespaces, ALWAYS_INCLUDED, await collectExtraTerms()));
   const terms = bundle.namespaces.reduce(
-    (count, [, , classes, properties]) =>
-      count + (classes ? classes.split(" ").length : 0) + (properties ? properties.split(" ").length : 0),
+    (count, [, , ...lists]) => count + lists.reduce((sum, list) => sum + (list ? list.split(" ").length : 0), 0),
     0,
   );
   fs.writeFileSync(OUTPUT_FILE, JSON.stringify(bundle) + "\n");

fetchPrefixData.js

@@ -10,12 +10,16 @@
  *
  * When a successful fetch occurs, the committed fallback file is also updated so
  * future offline builds stay reasonably fresh.
+ *
+ * The prefixes of the always included vocabularies of fetchLovData.js (ERA, railML, SPARQL Anything...) are
+ * added to the prefix.cc data, replacing prefix.cc's namespace for the same prefix, so they are always available.
  */
 
 import https from "https";
 import http from "http";
 import fs from "fs";
 import path from "path";
+import { ALWAYS_INCLUDED } from "./fetchLovData.js";
 
 const PREFIXES_URL_HTTPS = "https://prefix.cc/popular/all.file.json";
 const PREFIXES_URL_HTTP = "http://prefix.cc/popular/all.file.json";
@@ -43,6 +47,15 @@ function fetchUrl(url, options = {}) {
   });
 }
 
+/** The prefix data with the prefixes of the always included vocabularies, serialized */
+function withAlwaysIncluded(prefixes, alwaysIncluded = ALWAYS_INCLUDED) {
+  const result = { ...prefixes };
+  for (const { prefix, namespace } of alwaysIncluded) {
+    if (prefix) result[prefix] = namespace;
+  }
+  return JSON.stringify(result, null, 2) + "\n";
+}
+
 async function main() {
   let data = null;
 
@@ -71,7 +84,7 @@ async function main() {
   // Validate the fetched JSON
   if (data) {
     try {
-      JSON.parse(data);
+      data = withAlwaysIncluded(JSON.parse(data));
     } catch (e) {
       console.warn("Fetched data is not valid JSON, ignoring and using committed fallback.");
       data = null;
@@ -85,7 +98,7 @@ async function main() {
   } else {
     // Use the committed fallback
     if (fs.existsSync(FALLBACK_FILE)) {
-      data = fs.readFileSync(FALLBACK_FILE, "utf-8");
+      data = withAlwaysIncluded(JSON.parse(fs.readFileSync(FALLBACK_FILE, "utf-8")));
       console.log(`Using committed fallback: ${FALLBACK_FILE}`);
     } else {
       console.error("No fallback file found and all fetches failed. Cannot provide prefix data.");

packages/yasqe/src/autocompleters/functions.ts

@@ -0,0 +1,47 @@
+import * as Autocompleter from "./";
+import { bundledLovTerms, searchLov } from "./lov";
+import type { Token } from "../";
+
+/**
+ * The complete token can span the expression before the function name, e.g. "BIND(fx:ent": keeps the part after the
+ * last opening parenthesis, comma or operator.
+ */
+function trimToFunctionName(token: Token): Autocompleter.AutocompletionToken {
+  const match = /[\s(),!=&|*+]([^\s(),!=&|*+]*)$/.exec(token.string);
+  if (!match) return token;
+  const start = token.end - match[1].length;
+  // `from` makes the suggestion replace the function name only
+  return { ...token, start, string: match[1], from: { ch: start } };
+}
+
+/**
+ * Suggests the functions of the bundled vocabularies (see the `lovTerms` config option), such as the SPARQL Anything
+ * functions fx:entity or fx:String.substring, where an expression can start: in FILTER, BIND, SELECT expressions...
+ */
+var conf: Autocompleter.CompleterConfig = {
+  get: function (yasqe, token) {
+    if (!token) return [];
+    const lovTerms = yasqe.config.lovTerms === undefined ? bundledLovTerms : yasqe.config.lovTerms;
+    if (!lovTerms) return [];
+    return searchLov(lovTerms, "function", token).map((term) =>
+      Autocompleter.endpointTermToSuggestion(yasqe, token, term, Autocompleter.FUNCTION_SECTION),
+    );
+  },
+  isValidCompletionPosition: function (yasqe) {
+    const token = trimToFunctionName(yasqe.getCompleteToken());
+    if (token.string.length == 0) return false; //we want -something- to autocomplete
+    if (token.string[0] === "?" || token.string[0] === "$") return false; // we are typing a var
+    // An expression can start here: a built-in call such as STR(...) is allowed
+    return token.state.possibleCurrent.indexOf("STR") >= 0;
+  },
+  preProcessToken: function (yasqe, token) {
+    return Autocompleter.preprocessIriForCompletion(yasqe, trimToFunctionName(token));
+  },
+  postProcessSuggestion: function (yasqe, token, suggestedString) {
+    return Autocompleter.postprocessIriCompletion(yasqe, token, suggestedString);
+  },
+  bulk: false,
+  name: "function",
+};
+
+export default conf;

packages/yasqe/src/autocompleters/index.ts

@@ -365,6 +365,7 @@ export const fetchFromLov = (
 const ENDPOINT_SUGGESTIONS_LIMIT = 50;
 export const ENDPOINT_SECTION = { name: "Used in this endpoint", rank: 0 };
 export const LOV_SECTION = { name: "Linked Open Vocabularies", rank: 1 };
+export const FUNCTION_SECTION = { name: "Functions", rank: 2 };
 
 function localName(iri: string): string {
   return iri.substring(Math.max(iri.lastIndexOf("#"), iri.lastIndexOf("/")) + 1);
@@ -458,9 +459,16 @@ export const fetchIriSuggestions = async (
 };
 
 import { bundledLovTerms, searchLov } from "./lov";
-export type { LovTerms } from "./lov";
+export type { LovTerms, LovTermType } from "./lov";
 import variableCompleter from "./variables";
 import prefixCompleter from "./prefixes";
 import propertyCompleter from "./properties";
 import classCompleter from "./classes";
-export var completers: CompleterConfig[] = [variableCompleter, prefixCompleter, propertyCompleter, classCompleter];
+import functionCompleter from "./functions";
+export var completers: CompleterConfig[] = [
+  variableCompleter,
+  prefixCompleter,
+  propertyCompleter,
+  classCompleter,
+  functionCompleter,
+];