feat(yasqe): suggest classes and properties used in the endpoint (#235)

* feat(yasqe): suggest classes and properties used in the endpoint

Class and property autocompletion lists the terms the endpoint of the tab
uses, most used first with their usage count, before the Linked Open
Vocabularies suggestions. MatGUI takes them from the Endpoint overview
(VoID partitions, complete Classes/Properties results, or one limited
background query cached per endpoint) and inserts prefixed names with the
missing PREFIX. The suggestions can be switched off per endpoint in the
overview.

Closes #195

* fix(yasqe): use the new LOV term API for class and property suggestions

LOV moved its API from /dataset/lov/api/v2/autocomplete/terms (now 404)
to /dataset/api/v2/term/autocomplete and /dataset/api/v2/term/search.
IRIs and prefixed names use autocomplete, plain words the full text
search. The uri of a result may be a string or a one-item list.

* feat(yasqe): bundle LOV terms instead of calling the LOV API

The LOV API is broken (404 on the old path, a search backend error on the
new one). Class and property suggestions now search a bundled list of the
classes and properties of the most used LOV vocabularies, generated from
the LOV N-Quads dump by fetchLovData.js (npm run util:fetchLov) and
refreshed monthly by a workflow that opens a pull request. LOV terms are
inserted as prefixed names with their PREFIX. The lovTerms option replaces
the list or, set to null, leaves out LOV suggestions.

* fix: run fetchLovData.js on Windows too

* fix: skip invalid lines of the LOV dump instead of aborting

* chore: update package-lock

* feat(yasqe): add lov terms

* test: format the expected endpoint term count in the browser's locale

* feat(yasqe): always bundle the ERA ontology terms

The LOV dump has no graph for the ERA ontology (http://data.europa.eu/949/), so
fetchLovData.js now reads its classes and properties from the ontology on
GitLab. Namespaces in ALWAYS_INCLUDED are bundled first, regardless of their
popularity and the size budget.

Author: MathiasVDA

.changeset/endpoint-autocomplete.md

@@ -0,0 +1,8 @@
+---
+"@matdata/yasgui": minor
+"@matdata/yasqe": minor
+---
+
+Suggest the classes and properties used in the current endpoint in autocompletion, most used first with their usage count, before the Linked Open Vocabularies suggestions (#195).
+
+The terms come from the Endpoint overview: VoID class/property partitions, complete Classes/Properties results, or otherwise one limited background query (with the tab's authentication) whose result is cached per endpoint. Picking a suggestion inserts a prefixed name and declares the missing prefix. The suggestions can be switched off per endpoint in the Endpoint overview, or for all endpoints with `endpointDescribe.autocomplete: false`. YASQE gets an `endpointTerms` option to provide such terms, and hints can now carry a detail, a section and prefixes to declare.

.changeset/lov-api-v2-term.md

@@ -0,0 +1,5 @@
+---
+"@matdata/yasqe": minor
+---
+
+Class and property suggestions from Linked Open Vocabularies no longer call the LOV API, which is broken (HTTP 404 on the old path, a search backend error on the new one). Yasqe bundles the classes and properties of the most used LOV vocabularies (`packages/yasqe/src/lov-terms.json`, about 600 KB at most, refreshed with `npm run util:fetchLov` or the "Refresh LOV terms" workflow) and searches them in the browser. LOV suggestions are now inserted as prefixed names with their `PREFIX`. The new `lovTerms` option replaces the list, or leaves out LOV suggestions when set to `null`.

.github/workflows/refresh-lov-terms.yml

@@ -0,0 +1,40 @@
+name: Refresh LOV terms
+
+# Regenerates the bundled Linked Open Vocabularies term list (packages/yasqe/src/lov-terms.json)
+# from the LOV dump and opens a pull request when it changed.
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 4 1 * *"
+
+permissions:
+  contents: write
+  pull-requests: write
+
+jobs:
+  refresh:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v5
+
+      - name: Set up Node.js
+        uses: actions/setup-node@v6
+        with:
+          node-version: "lts/*"
+          cache: npm
+
+      - name: Install dependencies
+        run: npm ci
+
+      - name: Build the LOV term list
+        run: npm run util:fetchLov
+
+      - name: Open a pull request
+        uses: peter-evans/create-pull-request@v7
+        with:
+          branch: chore/refresh-lov-terms
+          commit-message: "chore(yasqe): refresh the bundled LOV terms"
+          title: "chore(yasqe): refresh the bundled LOV terms"
+          body: Regenerated `packages/yasqe/src/lov-terms.json` from the LOV dump with `npm run util:fetchLov`.
+          add-paths: packages/yasqe/src/lov-terms.json

docs/developer-guide.md

@@ -1019,6 +1019,38 @@ const yasqe = new Yasqe(document.getElementById('yasqe'), {
 });
 ```
 
+#### Endpoint Terms for Autocompletion
+
+The class and property autocompleters list terms used in the endpoint before the Linked Open Vocabularies suggestions. MatGUI fills them from its Endpoint overview. A standalone Yasqe (or an embedding page that wants its own source) can provide them with `endpointTerms`:
+
+```typescript
+interface EndpointTerm {
+  iri: string;
+  count?: number;      // shown next to the suggestion
+  prefix?: string;     // with namespace: insert a prefixed name and declare the prefix
+  namespace?: string;
+}
+
+const yasqe = new Yasqe(element, {
+  endpointTerms: async (yasqe, type /* "class" | "property" */) => {
+    return [{ iri: "http://data.europa.eu/949/OperationalPoint", count: 52311, prefix: "era", namespace: "http://data.europa.eu/949/" }];
+  },
+});
+```
+
+Return the terms most used first; Yasqe filters them on what the user typed and shows at most 50. Setting `yasqe.endpointTerms` in the MatGUI config replaces the Endpoint overview as the source.
+
+#### Linked Open Vocabularies Terms
+
+The LOV suggestions come from `packages/yasqe/src/lov-terms.json`, which is bundled with Yasqe: no request is made to the LOV API at runtime. The file holds the classes and properties of the most used LOV vocabularies (ranked by the number of vocabularies that use them, up to about 600 KB) and is generated from the LOV N-Quads dump:
+
+```bash
+npm run util:fetchLov                      # downloads https://lov.linkeddata.es/lov.nq.gz
+node fetchLovData.js path/to/lov.nq.gz     # or use a downloaded dump
+```
+
+The **Refresh LOV terms** GitHub workflow runs the same script monthly (or on demand) and opens a pull request when the list changed. Pass your own list with the `lovTerms` option (`{ namespaces: [[namespace, prefix, "ClassA ClassB", "propA propB"], ...] }`), or set it to `null` to leave out LOV suggestions.
+
 #### Share Configuration
 
 Configure how users can share their SPARQL queries with multiple output formats including URLs, cURL, PowerShell, and wget commands.
@@ -2018,6 +2050,10 @@ interface EndpointDescribeConfig {
 
   // Fetch the service description and VoID when the panel opens
   fetchMetadata: boolean;  // default: true
+
+  // Suggest the classes and properties used in the endpoint in the editor's autocomplete.
+  // Users can still switch this off per endpoint in the panel.
+  autocomplete: boolean;  // default: true
 }
 
 interface DescribeQuery {

docs/user-guide.md

@@ -44,6 +44,7 @@
   - [Endpoint Quick Switch](#endpoint-quick-switch)
   - [Configuration Import/Export](#configuration-importexport)
   - [URI Explorer](#uri-explorer)
+  - [Class and Property Autocomplete](#class-and-property-autocomplete)
   - [IRI Hover Info](#iri-hover-info)
   - [Endpoint Overview](#endpoint-overview)
   - [Query Tabs](#query-tabs)
@@ -303,7 +304,7 @@ The query editor where you write and edit SPARQL queries.
 **Features:**
 
 - Syntax highlighting for SPARQL
-- Auto-completion for keywords, prefixes, and URIs
+- Auto-completion for keywords, prefixes, and URIs (classes and properties used in the endpoint come first, see [Class and Property Autocomplete](#class-and-property-autocomplete))
 - Line numbers and code folding
 - Query validation with error highlighting
 - Find and replace functionality
@@ -852,6 +853,29 @@ Alternatively, use `Ctrl`+`Shift` and click on the URI to executes a CONSTRUCT q
 **Example:**
 Ctrl+clicking on `http://dbpedia.org/resource/European_Union` automatically queries for all triples where the European Union is the subject of.
 
+### Class and Property Autocomplete
+
+Press `Ctrl+Space` after `a` (or `rdf:type`) to get class suggestions, or in the position of a property to get property suggestions. MatGUI suggests:
+
+1. **Used in this endpoint**: the classes and properties the endpoint of the tab actually uses, most used first, with the number of instances (classes) or triples (properties) next to them. This also covers vocabularies that are not registered anywhere, such as railML 3, the ERA ontology or your own ontologies.
+2. **Linked Open Vocabularies**: matching classes and properties of the most used vocabularies in [LOV](https://lov.linkeddata.es/dataset/lov/). MatGUI ships this list, so no request is made to LOV while you type. Terms that the endpoint already suggested are not repeated, and a term is inserted as a prefixed name with its `PREFIX` when LOV knows the vocabulary's preferred prefix.
+
+Without typing anything, the endpoint's most used terms are listed. While you type, a suggestion matches on its full IRI, its prefixed name (`era:Op`) or, when you type a plain word (`Operat`), on any part of its local name.
+
+Picking a suggestion inserts a prefixed name when MatGUI knows a prefix for it, and adds the missing `PREFIX` declaration to your query, just like clicking an IRI in the [Endpoint Overview](#endpoint-overview). When you are typing with a prefix that your query declares, that prefix is used.
+
+**Where the suggestions come from**
+
+MatGUI reuses what the endpoint overview already knows about the endpoint, so in most cases no extra query is needed:
+
+- The class and property partitions of the endpoint's VoID description.
+- The results of the **Classes** and **Properties** overview queries, when they hold the complete list.
+- Otherwise MatGUI runs one limited query in the background (the 200 most used classes or properties) the first time you ask for suggestions, with the endpoint and authentication settings of the tab. The result is remembered per endpoint like the other overview results; use **Run again** on the overview's Classes or Properties query to refresh what is suggested. When this query fails or times out, it is not retried for an hour.
+
+**Switching it off for an endpoint**
+
+Counting all properties of a very large public endpoint is expensive. Open the [Endpoint Overview](#endpoint-overview) for that endpoint and clear **Suggest this endpoint's classes and properties in autocomplete**: MatGUI then neither queries the endpoint nor shows its terms, and only the Linked Open Vocabularies suggestions remain. The choice is remembered per endpoint.
+
 ### IRI Hover Info
 
 Rest the mouse on an IRI or prefixed name in the query editor (for example `era:solNature`) to see what it means, without running anything or losing your results. After a short moment a tooltip shows:
@@ -901,6 +925,7 @@ The panel contains predefined queries, grouped in categories:
 - Public endpoints limit how many (expensive) queries you can run. When an endpoint answers that you are sending too many requests, wait a moment before running more queries. Errors of the endpoint are shown below the query (e.g. "estimated execution time exceeds the limit").
 - Describe queries run in the background with the endpoint and authentication settings of the current tab. They do not change your query or the results view. The default and named graphs configured for the tab are not applied, so the whole endpoint is described.
 - Click an IRI in the results to insert it into your query. MatGUI uses a prefixed name when it knows the prefix and adds the missing `PREFIX` declaration.
+- The classes and properties the overview knows are also suggested in the editor's autocomplete. The **Suggest this endpoint's classes and properties in autocomplete** checkbox at the top of the panel switches this off for the current endpoint (see [Class and Property Autocomplete](#class-and-property-autocomplete)).
 - Click the **Open query in a new tab** button next to a query to open it in a new tab, to adapt it further.
 
 **Remembered results**
@@ -2405,7 +2430,7 @@ The proxy solves this by:
 2. **Wait for Typing Pause**: Autocomplete appears after a brief delay
 3. **Position Cursor Correctly**: Autocomplete context-dependent (prefixes, keywords, etc.)
 4. **Manually Trigger**: Press `Ctrl+Space` to force autocomplete
-5. **LOV ontology description**: The implementation uses https://lov.linkeddata.es/dataset/lov/ to retrieve classes and predicates from ontologies. If the ontology that you use cannot be found here, then the autocomplete will not work.
+5. **Classes and properties**: Suggestions come from the endpoint of the tab and from a list of the most used vocabularies in LOV that ships with MatGUI. Rarely used vocabularies are not in that list. If your ontology is not in it, the endpoint suggestions still cover the terms the endpoint uses, unless they are switched off for that endpoint in the Endpoint overview, or the background query failed (it is retried after an hour; running the Classes or Properties overview query also fills the suggestions). See [Class and Property Autocomplete](#class-and-property-autocomplete).
 
 #### Lost Queries