Skip to content

Commit 2271666

Browse files
hohonuulihohonuuli
committed
doc: add doc about building queries
1 parent 39ec27d commit 2271666

5 files changed

Lines changed: 384 additions & 4 deletions

File tree

‎README.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ Once running, access the Swagger API documentation at `http://localhost:8080/doc
5757

5858
## Overview
5959

60-
Annosaurus is one component of MBARI's [Video Annotation and Reference System](https://github.com/mbari-org/m3-quickstart). It is a REST-based web service that stores and retrieves annotations for videos and images, providing a language-agnostic API so developers and scientists can build their own annotation tools against any tech stack.
60+
Annosaurus is one component of MBARI's Video Annotation and Reference System (VARS). It is a REST-based web service that stores and retrieves annotations for videos and images, providing a language-agnostic API so developers and scientists can build their own annotation tools against any tech stack.
6161

6262
The service is self-contained and requires only a database — either PostgreSQL or SQL Server.
6363

@@ -315,10 +315,11 @@ Refer to [DEPLOYMENT.md](annosaurus/src/site/docs/DEPLOYMENT.md) for production
315315

316316
## Related Projects
317317

318-
This service is part of MBARI's [Video Annotation and Reference System](https://github.com/mbari-org/m3-quickstart):
318+
This service is part of MBARI's Video Annotation and Reference System (VARS):
319319

320320
- [vampire-squid](https://github.com/mbari-org/vampire-squid) - Video asset management service
321-
- [m3-quickstart](https://github.com/mbari-org/m3-quickstart) - Complete system deployment guide
321+
- [vars-quickstart-public](https://github.com/mbari-org/vars-quickstart-public) - Complete system deployment guide for use outside MBARI
322+
- [vars-quickstart-mbari](https://github.com/mbari-org/vars-quickstart-mbari) - Complete system deployment guide for MBARI deployments
322323

323324
## Contributing
324325

‎annosaurus/src/site/docs/howto/advanced-queries.md‎

Lines changed: 279 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,279 @@
1+
# Advanced Queries
2+
3+
The `/v1/query` endpoints let you query annotations the way you would query a table: pick the columns you want, filter on any of them, and get a tab-separated file back. They are the right tool for assembling a dataset for analysis, and they can express filters that the [fast annotation endpoints](fetch_annotations.md) cannot.
4+
5+
The typical workflow is two steps:
6+
7+
1. `GET /v1/query/columns` — discover what you can select and filter on.
8+
2. `POST /v1/query/download` — submit a query and download the results as TSV.
9+
10+
`POST /v1/query/count` is a useful third step between the two: it tells you how many rows a query will return before you commit to downloading them.
11+
12+
None of these endpoints require authentication — they are read-only.
13+
14+
> **There is also `/v1/query/run`.** It accepts the same request body and returns TSV in the response body instead of as a file attachment. It builds the entire result set in memory before responding, so it is slower and far more memory-hungry than `/v1/query/download`, which streams rows to a file as it reads them. Prefer `/v1/query/download`.
15+
16+
## Everything queries a single view
17+
18+
These endpoints do not query the annotation tables directly. They query one flat database view — named `annotations` by default, configurable with `DATABASE_QUERY_VIEW` — that joins imaged moments, observations, associations, image references, ancillary data, and video reference info into a single wide, denormalized row set.
19+
20+
Two consequences matter:
21+
22+
- **One row is not one annotation.** Because the view is a join across several one-to-many relationships, an observation with three associations and two framegrabs appears as six rows. Use `distinct`, or select only the columns you actually need, to control this.
23+
- **The available columns depend on the deployment.** The view is created by a migration and can be customized per site, so the column list is not part of the API contract. Always ask `/v1/query/columns` rather than hard-coding column names.
24+
25+
## Step 1: discover the columns
26+
27+
```text
28+
GET http://myserver.org/anno/v1/query/columns
29+
```
30+
31+
The response is a JSON array of column descriptors, one per column in the view:
32+
33+
```json
34+
[
35+
{
36+
"columnName": "index_recorded_timestamp",
37+
"columnType": "timestamptz",
38+
"columnSize": 35,
39+
"columnLabel": "index_recorded_timestamp",
40+
"columnClassName": "java.sql.Timestamp"
41+
},
42+
{
43+
"columnName": "concept",
44+
"columnType": "varchar",
45+
"columnSize": 256,
46+
"columnLabel": "concept",
47+
"columnClassName": "java.lang.String"
48+
},
49+
{
50+
"columnName": "depth_meters",
51+
"columnType": "float4",
52+
"columnSize": 15,
53+
"columnLabel": "depth_meters",
54+
"columnClassName": "java.lang.Float"
55+
}
56+
]
57+
```
58+
59+
`columnName` is what you put in `select`, `where`, and `orderBy`. `columnClassName` is the useful field for building a request, because **which constraints are legal depends on the column's type** — see [Choosing the right constraint](#choosing-the-right-constraint).
60+
61+
The default view provides these columns, grouped by where they come from:
62+
63+
| Source | Columns |
64+
| --- | --- |
65+
| Imaged moment | `imaged_moment_uuid`, `index_recorded_timestamp`, `index_elapsed_time_millis`, `index_timecode` |
66+
| Observation | `observation_uuid`, `concept`, `observer`, `activity`, `observation_group`, `observation_timestamp`, `duration_millis` |
67+
| Association | `link_name`, `to_concept`, `link_value`, `association_mime_type`, `associations` |
68+
| Image reference | `image_reference_uuid`, `image_url`, `image_format`, `image_width`, `image_height`, `image_description` |
69+
| Ancillary data | `latitude`, `longitude`, `depth_meters`, `altitude`, `salinity`, `temperature_celsius`, `oxygen_ml_per_l`, `pressure_dbar`, `light_transmission`, `coordinate_reference_system`, `x`, `y`, `z`, `phi`, `theta`, `psi`, `xyz_position_units` |
70+
| Video reference info | `camera_platform`, `dive_number`, `chief_scientist` |
71+
72+
The column list above is the PostgreSQL view. The SQL Server view is nearly identical but names the concatenated association column `association` rather than `associations`, and adds an `association_uuid` column — another reason to read the column list from the server instead of assuming it.
73+
74+
Note that `video_reference_uuid` is *not* in the default view. Filter by `dive_number` or `camera_platform` instead, or use the [fast endpoints](fetch_annotations.md) when you already have a video reference UUID.
75+
76+
## Step 2: build the request
77+
78+
The request body is the same for `download`, `count`, and `run`. Field names are camelCase; column names are the snake_case names from `/v1/query/columns`.
79+
80+
```json
81+
{
82+
"select": ["concept", "index_recorded_timestamp", "depth_meters"],
83+
"distinct": false,
84+
"where": [
85+
{ "column": "concept", "equals": "Nanomia bijuga" },
86+
{ "column": "depth_meters", "minmax": [200, 800] }
87+
],
88+
"orderBy": ["index_recorded_timestamp"],
89+
"limit": 5000,
90+
"offset": 0,
91+
"concurrentObservations": false,
92+
"relatedAssociations": false,
93+
"strict": true
94+
}
95+
```
96+
97+
| Field | Meaning |
98+
| --- | --- |
99+
| `select` | Columns to return. **Required** for `download` and `run`; ignored by `count`. |
100+
| `distinct` | Emit `SELECT DISTINCT`. Default `false`. |
101+
| `where` | Constraints, combined with `AND`. **Required** by `count`, and by `download`/`run` when you explicitly set `"strict": true`. |
102+
| `orderBy` | Columns to sort by, always ascending. Defaults to the first column in `select`. |
103+
| `limit` / `offset` | Page through results. |
104+
| `concurrentObservations` | Also return the other observations recorded at the same imaged moment. Default `false`. |
105+
| `relatedAssociations` | Also return the other associations belonging to matched observations. Default `false`. |
106+
| `strict` | Whether to return *only* the columns you selected. Default `true`. |
107+
108+
All constraints in `where` are ANDed together. There is no `OR` and no nesting; for alternatives on a single column use `in`.
109+
110+
### Choosing the right constraint
111+
112+
Each object in `where` names a `column` plus **exactly one** constraint. If you supply more than one, only the first is used, in the order listed below — the rest are silently ignored. Split them into separate `where` entries instead.
113+
114+
| Constraint | JSON type | SQL | Use on |
115+
| --- | --- | --- | --- |
116+
| `between` | array of 2 ISO-8601 instants | `col BETWEEN ? AND ?` | timestamp columns |
117+
| `contains` | string | `col LIKE '%value%'` | text columns |
118+
| `equals` | string | `col = ?` | text and UUID columns |
119+
| `in` | array of strings | `col IN (?, …)` | text and UUID columns |
120+
| `isnull` | boolean | `col IS NULL` / `IS NOT NULL` | any column |
121+
| `like` | string | `col LIKE ?` | text columns |
122+
| `notlike` | string | `col NOT LIKE ?` | text columns |
123+
| `max` | number | `col <= ?` | numeric columns |
124+
| `min` | number | `col >= ?` | numeric columns |
125+
| `minmax` | array of 2 numbers | `col BETWEEN ? AND ?` | numeric columns |
126+
127+
Matching the constraint to the column type matters, because a mismatch reaches the database as a type error rather than an empty result:
128+
129+
- **Numeric columns** (`columnClassName` of `java.lang.Float`, `java.lang.Double`, `java.lang.Integer`, `java.math.BigDecimal`) take `min`, `max`, and `minmax`. Using `equals` or `like` on one fails — the value is sent as a string, and the database rejects `integer = character varying`.
130+
- **Text columns** (`java.lang.String`) take `equals`, `in`, `like`, `notlike`, and `contains`. Using `min`/`max`/`minmax` on one fails the same way, in reverse.
131+
- **Timestamp columns** (`java.sql.Timestamp`) take `between`, with exactly two ISO-8601 values. `between` is timestamps only — use `minmax` for numeric ranges.
132+
- **UUID columns** (`java.util.UUID`) work with `equals` and `in`, passing the UUID as a string.
133+
- `isnull` works on any column.
134+
135+
With `like` and `notlike` you supply the `%` wildcards yourself; `contains` wraps the value in `%` for you.
136+
137+
### `strict`, and the two expansion flags
138+
139+
`strict` controls whether the query returns exactly the columns you asked for:
140+
141+
- `"strict": true` (the default) returns precisely your `select` list, ordered by `select`'s first column unless you set `orderBy`.
142+
- `"strict": false` prepends `observation_uuid` and `index_recorded_timestamp` to your `select` list and orders by those two. The extra columns let a client regroup flat rows back into annotations.
143+
144+
Setting `concurrentObservations` or `relatedAssociations` to `true` **forces `strict` to `false`**, whatever you passed in, because those results are meaningless without the grouping columns.
145+
146+
The two flags widen the result set rather than narrowing it. Your `where` clause becomes a subquery that selects matching rows, and then:
147+
148+
- `concurrentObservations` returns every row whose `imaged_moment_uuid` matched — all observations made at the same instant in the video, not just the ones matching your filter.
149+
- `relatedAssociations` returns every row whose `observation_uuid` matched — all associations on the matched observations.
150+
- Both together return all observations at the matched moments, along with all of their associations.
151+
152+
```json
153+
{
154+
"select": ["concept", "link_name", "link_value", "index_recorded_timestamp"],
155+
"where": [{ "column": "concept", "equals": "Sebastes" }],
156+
"relatedAssociations": true
157+
}
158+
```
159+
160+
That query returns every association on every *Sebastes* observation, not only the rows where the association itself matched.
161+
162+
> **Note:** `distinct` is rarely useful with either flag set, since the forced `observation_uuid` column makes nearly every row unique.
163+
164+
## Step 3: check the size, then download
165+
166+
`count` runs your `where` clause and returns just the row count. It requires a `where` clause and ignores `select`:
167+
168+
```text
169+
POST http://myserver.org/anno/v1/query/count
170+
Content-Type: application/json
171+
172+
{
173+
"where": [
174+
{ "column": "concept", "equals": "Nanomia bijuga" },
175+
{ "column": "depth_meters", "minmax": [200, 800] }
176+
]
177+
}
178+
```
179+
180+
```json
181+
{ "count": 14203 }
182+
```
183+
184+
Then download the rows:
185+
186+
```bash
187+
curl -X POST 'http://myserver.org/anno/v1/query/download' \
188+
-H 'Content-Type: application/json' \
189+
-o nanomia.tsv \
190+
-d '{
191+
"select": ["concept", "index_recorded_timestamp", "depth_meters", "latitude", "longitude"],
192+
"where": [
193+
{ "column": "concept", "equals": "Nanomia bijuga" },
194+
{ "column": "depth_meters", "minmax": [200, 800] }
195+
],
196+
"orderBy": ["index_recorded_timestamp"]
197+
}'
198+
```
199+
200+
The response is a file attachment with `Content-Type: text/tab-separated-values` and `Content-Disposition: attachment; filename=results.tsv`. Give `curl` an `-o` of your own, as above, or use `-OJ` to accept the server's name — every download is named `results.tsv`, so `-OJ` will collide if you run more than one.
201+
202+
Notes on the output:
203+
204+
- The first line is a header of column names.
205+
- Rows are ordered by `orderBy`, or by the defaults described under [`strict`](#strict-and-the-two-expansion-flags).
206+
- `NULL` values are written as empty fields.
207+
- Timestamps are rendered in UTC.
208+
- Fields are **not** quoted or escaped. A `link_value` containing a tab or newline — free-text comments are the usual suspect — will break column alignment for that row. Exclude such columns, or post-process, when that matters.
209+
- The server writes the file to temporary storage and deletes it about two minutes after responding. Read the response body to completion; there is no URL to come back to later.
210+
211+
## Worked examples
212+
213+
**All annotations for one dive, grouped back into annotations by the client**
214+
215+
```json
216+
{
217+
"select": ["concept", "observer", "index_recorded_timestamp", "associations", "image_url"],
218+
"where": [{ "column": "dive_number", "equals": "Doc Ricketts 1373" }],
219+
"strict": false
220+
}
221+
```
222+
223+
**Concept counts by platform over a date range** — download and aggregate locally
224+
225+
```json
226+
{
227+
"select": ["camera_platform", "concept"],
228+
"where": [
229+
{ "column": "index_recorded_timestamp", "between": ["2023-01-01T00:00:00Z", "2024-01-01T00:00:00Z"] },
230+
{ "column": "concept", "isnull": false }
231+
],
232+
"orderBy": ["camera_platform", "concept"]
233+
}
234+
```
235+
236+
**Distinct concepts observed below 1000 m**
237+
238+
```json
239+
{
240+
"select": ["concept"],
241+
"distinct": true,
242+
"where": [{ "column": "depth_meters", "min": 1000 }]
243+
}
244+
```
245+
246+
**Everything observed at the same moment as a squid** — the flag turns a filter for squid into a filter for moments containing squid
247+
248+
```json
249+
{
250+
"select": ["concept", "index_recorded_timestamp", "depth_meters"],
251+
"where": [{ "column": "concept", "contains": "squid" }],
252+
"concurrentObservations": true
253+
}
254+
```
255+
256+
**Measurement associations only, paged**
257+
258+
```json
259+
{
260+
"select": ["concept", "link_name", "to_concept", "link_value"],
261+
"where": [{ "column": "link_name", "like": "%length%" }],
262+
"orderBy": ["concept"],
263+
"limit": 1000,
264+
"offset": 0
265+
}
266+
```
267+
268+
## Troubleshooting
269+
270+
| Symptom | Likely cause |
271+
| --- | --- |
272+
| `select clause is required` | `download` and `run` need a non-empty `select`. |
273+
| `where clause is required` | `count` always needs `where`; so do `download`/`run` when you pass `"strict": true`. |
274+
| `operator does not exist: … = character varying` | A string constraint (`equals`, `in`, `like`, `contains`, `notlike`) on a numeric column. Use `min`/`max`/`minmax`. |
275+
| `operator does not exist: character varying >= double precision` | A numeric constraint on a text column. Use `equals`/`in`/`like`. |
276+
| A syntax error mentioning `ASC` | `orderBy` values are plain column names. `ASC` is appended for you, so `"concept DESC"` produces invalid SQL — descending order is not supported. |
277+
| Server error on a `between` or `minmax` | Both need exactly two array elements. |
278+
| Far more rows than expected | The view is a join; one observation can produce many rows. Narrow `select`, or set `distinct`. |
279+
| A column you expected is missing | The view is deployment-specific. Confirm with `/v1/query/columns`. |

‎annosaurus/src/site/docs/index.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
## Overview
66

7-
Annosaurus is a REST API for creating and managing video and image annotations. It is a core service of MBARI's [Video Annotation and Reference System](https://github.com/mbari-org/m3-quickstart), providing a language-agnostic interface so annotations can be created and retrieved from any programming environment.
7+
Annosaurus is a REST API for creating and managing video and image annotations. It is a core service of MBARI's Video Annotation and Reference System (VARS), providing a language-agnostic interface so annotations can be created and retrieved from any programming environment.
88

99
The service requires only a database (PostgreSQL or SQL Server) and exposes a Swagger UI at `/docs` once running.
1010

0 commit comments

Comments
 (0)