Skip to content

Commit d1c34c6

Browse files
committed
sqlite: add Internal helpers for connection, schema, and shared collections
Six modules under Internal/ that the per-entity public API will compose: - Sql: connection open with PRAGMA foreign_keys=ON, transaction wrapper, named-parameter helpers (execNonQuery, queryAll, tryQueryOne). - Schema: loads the embedded insdc_schema.sql resource and applies it. - Identifiers: write/read the unified per-owner identifier table, generic over owner table+accession column; covers all five kinds (PRIMARY, SECONDARY, EXTERNAL, SUBMITTER, UUID) with label/namespace preserved. - Attributes: write/read the tag/value/units triples, ordinal-preserving. - Links: write/read the Link DU (URL/XREF/ENTREZ) via the link_kind discriminator; handles EntrezLink's nullable int64 Id. - References: write/read a RefObject + its nested identifier table; the read path is generic over the concrete subclass so callers get a typed ExperimentStudyRef / BioSampleDescriptor / RunExperimentRef directly. Verified via fsi smoke test: 42 tables load; identifier, attribute, and link round trips preserve every populated field including NULLs.
1 parent b47b62a commit d1c34c6

8 files changed

Lines changed: 552 additions & 8 deletions

File tree

‎plans/sqlite-store.md‎

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -332,14 +332,16 @@ O(entity_count) queries per reconstruction; with FK indexes this stays cheap.
332332

333333
**Result:** Empty F# library compiles; full solution build is clean. Pre-existing NU1903 warnings on FAKE build tooling are unchanged.
334334

335-
### Chunk 3 — Internal helpers
336-
337-
- [ ] `Internal/Sql.fs` — `openConnection : string -> SqliteConnection` (issues `PRAGMA foreign_keys = ON` after open); `withTransaction` wrapper; small param-binding helpers
338-
- [ ] `Internal/Schema.fs` — loads `insdc_schema.sql` from embedded resources and executes against a connection
339-
- [ ] `Internal/Identifiers.fs` — `IdentifierOwner` record + `writeIdentifiers` / `readIdentifiers` (generic over owner table, accession column, accession value)
340-
- [ ] `Internal/Attributes.fs` — `writeAttributes` / `readAttributes` (same shape)
341-
- [ ] `Internal/Links.fs` — `writeLinks` / `readLinks` with `link_kind` resolution against the `Link` DU (UrlLink / XrefLink / EntrezLink)
342-
- [ ] `Internal/References.fs` — `writeRefObject` / `readRefObject` for any `RefObject` subtype, including its sibling identifier table
335+
### Chunk 3 — Internal helpers ✅
336+
337+
- [x] `Internal/Sql.fs` — `openConnection : string -> SqliteConnection` (issues `PRAGMA foreign_keys = ON` after open); `withTransaction` wrapper; small param-binding helpers (`execNonQuery`, `queryAll`, `tryQueryOne`, `readStringOrNull`)
338+
- [x] `Internal/Schema.fs` — loads `insdc_schema.sql` from embedded resources and executes against a connection
339+
- [x] `Internal/Identifiers.fs` — `IdentifierOwner` record + `write` / `read` (generic over owner table, accession column, accession value)
340+
- [x] `Internal/Attributes.fs` — `write` / `read` (same shape)
341+
- [x] `Internal/Links.fs` — `write` / `read` with `link_kind` resolution against the `Link` DU (UrlLink / XrefLink / EntrezLink)
342+
- [x] `Internal/References.fs` — `write` / `read` for any `RefObject` subtype, including its sibling identifier table (new()-constrained generic so the right concrete subclass — `ExperimentStudyRef`, `BioSampleDescriptor`, `RunExperimentRef` — is returned)
343+
344+
**Result:** Clean build, zero warnings. Smoke test through `dotnet fsi`: schema loads (42 tables), all 5 identifier kinds round-trip (PRIMARY/SECONDARY/EXTERNAL/SUBMITTER/UUID) with label/namespace preserved, attributes round-trip with NULL value/units preserved, all 3 Link DU cases (URL/XREF/ENTREZ with nullable int64 Id) round-trip cleanly.
343345

344346
### Chunk 4 — Per-entity public modules
345347

‎src/BioFSharp.INSDC.SQLite/BioFSharp.INSDC.SQLite.fsproj‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,12 @@
88
</PropertyGroup>
99

1010
<ItemGroup>
11+
<Compile Include="Internal\Sql.fs" />
12+
<Compile Include="Internal\Schema.fs" />
13+
<Compile Include="Internal\Identifiers.fs" />
14+
<Compile Include="Internal\Attributes.fs" />
15+
<Compile Include="Internal\Links.fs" />
16+
<Compile Include="Internal\References.fs" />
1117
<EmbeddedResource Include="schema\insdc_schema.sql" />
1218
<None Include="..\..\docs\img\icon.png" Pack="true" PackagePath="" />
1319
<None Include="..\..\README.md" Pack="true" PackagePath="" />
Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
namespace BioFSharp.INSDC.SQLite.Internal
2+
3+
open Microsoft.Data.Sqlite
4+
open BioFSharp.FileFormats.INSDC
5+
6+
/// Identity of the `{entity}_attributes` table for one owning entity.
7+
type AttributeOwner = {
8+
Table: string
9+
AccessionColumn: string
10+
Accession: string
11+
}
12+
13+
/// Persists / hydrates the tag/value/units attribute collection that every
14+
/// INSDC entity carries. The schema uses one row per attribute keyed by
15+
/// `(accession, ordinal)`; ordinal preserves the original XML document order
16+
/// so XML → SQLite → XML round trips don't reshuffle.
17+
module Attributes =
18+
19+
/// Inserts each `attribute` into the owner's attributes table, assigning
20+
/// ordinals in iteration order. Null or empty input writes nothing.
21+
let write
22+
(connection: SqliteConnection)
23+
(owner: AttributeOwner)
24+
(attributes: System.Collections.Generic.IEnumerable<Attribute>)
25+
: unit =
26+
if isNull attributes then () else
27+
let sql =
28+
sprintf
29+
"INSERT INTO %s (%s, ordinal, tag, value, units) VALUES (@acc, @ordinal, @tag, @value, @units);"
30+
owner.Table
31+
owner.AccessionColumn
32+
attributes
33+
|> Seq.iteri (fun i attr ->
34+
Sql.execNonQuery
35+
connection
36+
sql
37+
[
38+
"@acc", box owner.Accession
39+
"@ordinal", box i
40+
"@tag", box attr.Tag
41+
"@value", box attr.Value
42+
"@units", box attr.Units
43+
]
44+
|> ignore)
45+
46+
/// Reads all rows in the owner's attributes table in ordinal order and
47+
/// builds matching `Attribute` instances. Returns an empty list when none
48+
/// exist; callers populate their parent's attribute collection with `Add`.
49+
let read (connection: SqliteConnection) (owner: AttributeOwner) : Attribute list =
50+
let sql =
51+
sprintf
52+
"SELECT tag, value, units FROM %s WHERE %s = @acc ORDER BY ordinal;"
53+
owner.Table
54+
owner.AccessionColumn
55+
Sql.queryAll
56+
connection
57+
sql
58+
[ "@acc", box owner.Accession ]
59+
(fun reader ->
60+
Attribute(
61+
Tag = reader.GetString(0),
62+
Value = Sql.readStringOrNull reader 1,
63+
Units = Sql.readStringOrNull reader 2))
Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
namespace BioFSharp.INSDC.SQLite.Internal
2+
3+
open Microsoft.Data.Sqlite
4+
open BioFSharp.FileFormats.INSDC
5+
6+
/// Identity of the row family that holds an `Identifier` composite — five
7+
/// entity tables (`{entity}_identifiers`) and three reference-nested tables
8+
/// (`{owner}_{ref}_identifiers`) all share the same column shape, so the read
9+
/// and write paths are parameterized over the table to avoid eight near-
10+
/// duplicate modules.
11+
type IdentifierOwner = {
12+
/// Fully-qualified SQLite table name, e.g. `"bioproject_identifiers"`.
13+
Table: string
14+
/// FK column referencing the owning entity, e.g. `"bioproject_accession"`.
15+
AccessionColumn: string
16+
/// Accession value of the owning row that all written identifiers belong to.
17+
Accession: string
18+
}
19+
20+
/// Persists / hydrates the `Identifier` composite (PrimaryId, SecondaryId[],
21+
/// ExternalId[], SubmitterId, Uuid[]) against a unified per-owner identifier
22+
/// table. The DB shape is one row per identifier kind+ordinal pair; the kind
23+
/// discriminator restores which slot in the F# composite a row populates.
24+
module Identifiers =
25+
26+
[<Literal>]
27+
let private kindPrimary = "PRIMARY"
28+
29+
[<Literal>]
30+
let private kindSecondary = "SECONDARY"
31+
32+
[<Literal>]
33+
let private kindExternal = "EXTERNAL"
34+
35+
[<Literal>]
36+
let private kindSubmitter = "SUBMITTER"
37+
38+
[<Literal>]
39+
let private kindUuid = "UUID"
40+
41+
let private insertRow
42+
(connection: SqliteConnection)
43+
(owner: IdentifierOwner)
44+
(kind: string)
45+
(ordinal: int)
46+
(value: string)
47+
(label: string)
48+
(ns: string)
49+
: unit =
50+
let sql =
51+
sprintf
52+
"INSERT INTO %s (%s, kind, ordinal, value, label, namespace) VALUES (@acc, @kind, @ordinal, @value, @label, @namespace);"
53+
owner.Table
54+
owner.AccessionColumn
55+
Sql.execNonQuery
56+
connection
57+
sql
58+
[
59+
"@acc", box owner.Accession
60+
"@kind", box kind
61+
"@ordinal", box ordinal
62+
"@value", box value
63+
"@label", box label
64+
"@namespace", box ns
65+
]
66+
|> ignore
67+
68+
/// Writes every populated identifier on `identifiers` into the owner's
69+
/// identifier table. Null `identifiers` is treated as "nothing to write",
70+
/// matching the XSD where the whole IDENTIFIERS element is optional.
71+
let write (connection: SqliteConnection) (owner: IdentifierOwner) (identifiers: Identifier) : unit =
72+
if isNull identifiers then () else
73+
74+
match identifiers.PrimaryId with
75+
| null -> ()
76+
| n -> insertRow connection owner kindPrimary 0 n.Value n.Label null
77+
78+
if not (isNull identifiers.SecondaryId) then
79+
identifiers.SecondaryId
80+
|> Seq.iteri (fun i n ->
81+
insertRow connection owner kindSecondary i n.Value n.Label null)
82+
83+
if not (isNull identifiers.ExternalId) then
84+
identifiers.ExternalId
85+
|> Seq.iteri (fun i qn ->
86+
insertRow connection owner kindExternal i qn.Value qn.Label qn.Namespace)
87+
88+
match identifiers.SubmitterId with
89+
| null -> ()
90+
| qn -> insertRow connection owner kindSubmitter 0 qn.Value qn.Label qn.Namespace
91+
92+
if not (isNull identifiers.Uuid) then
93+
identifiers.Uuid
94+
|> Seq.iteri (fun i n ->
95+
insertRow connection owner kindUuid i n.Value n.Label null)
96+
97+
/// Reads every row in the owner's identifier table and reassembles the
98+
/// F# `Identifier` composite. Returns `null` when no rows exist — the
99+
/// generated INSDC types keep the IDENTIFIERS element as an optional
100+
/// reference, and CLR-`null` is what the XML serializer round-trips to.
101+
let read (connection: SqliteConnection) (owner: IdentifierOwner) : Identifier =
102+
let sql =
103+
sprintf
104+
"SELECT kind, ordinal, value, label, namespace FROM %s WHERE %s = @acc ORDER BY kind, ordinal;"
105+
owner.Table
106+
owner.AccessionColumn
107+
let rows =
108+
Sql.queryAll
109+
connection
110+
sql
111+
[ "@acc", box owner.Accession ]
112+
(fun reader ->
113+
let kind = reader.GetString(0)
114+
let value = reader.GetString(2)
115+
let label = Sql.readStringOrNull reader 3
116+
let ns = Sql.readStringOrNull reader 4
117+
kind, value, label, ns)
118+
119+
if List.isEmpty rows then null else
120+
121+
let identifiers = Identifier()
122+
for (kind, value, label, ns) in rows do
123+
match kind with
124+
| k when k = kindPrimary ->
125+
identifiers.PrimaryId <- Name(Value = value, Label = label)
126+
| k when k = kindSecondary ->
127+
identifiers.SecondaryId.Add(Name(Value = value, Label = label))
128+
| k when k = kindExternal ->
129+
identifiers.ExternalId.Add(QualifiedName(Value = value, Label = label, Namespace = ns))
130+
| k when k = kindSubmitter ->
131+
identifiers.SubmitterId <- QualifiedName(Value = value, Label = label, Namespace = ns)
132+
| k when k = kindUuid ->
133+
identifiers.Uuid.Add(Name(Value = value, Label = label))
134+
| other ->
135+
failwithf "Unexpected identifier kind '%s' in %s for %s='%s'" other owner.Table owner.AccessionColumn owner.Accession
136+
identifiers
Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
namespace BioFSharp.INSDC.SQLite.Internal
2+
3+
open System
4+
open Microsoft.Data.Sqlite
5+
open BioFSharp.FileFormats.INSDC
6+
7+
/// Identity of the `{entity}_links` table for one owning entity.
8+
type LinkOwner = {
9+
Table: string
10+
AccessionColumn: string
11+
Accession: string
12+
}
13+
14+
/// Persists / hydrates the `Link` collection that every INSDC entity carries.
15+
/// `Link` is a DU in spirit (exactly one of `UrlLink` / `XrefLink` / `EntrezLink`
16+
/// is set per row); the schema mirrors that with a `link_kind` discriminator
17+
/// column and only populates the subset of payload columns relevant to that kind.
18+
module Links =
19+
20+
[<Literal>]
21+
let private kindUrl = "URL"
22+
23+
[<Literal>]
24+
let private kindXref = "XREF"
25+
26+
[<Literal>]
27+
let private kindEntrez = "ENTREZ"
28+
29+
let private insertRow
30+
(connection: SqliteConnection)
31+
(owner: LinkOwner)
32+
(ordinal: int)
33+
(kind: string)
34+
(label: string)
35+
(url: string)
36+
(db: string)
37+
(id: string)
38+
(query: string)
39+
: unit =
40+
let sql =
41+
sprintf
42+
"INSERT INTO %s (%s, ordinal, link_kind, label, url, db, id, query) VALUES (@acc, @ordinal, @kind, @label, @url, @db, @id, @query);"
43+
owner.Table
44+
owner.AccessionColumn
45+
Sql.execNonQuery
46+
connection
47+
sql
48+
[
49+
"@acc", box owner.Accession
50+
"@ordinal", box ordinal
51+
"@kind", box kind
52+
"@label", box label
53+
"@url", box url
54+
"@db", box db
55+
"@id", box id
56+
"@query", box query
57+
]
58+
|> ignore
59+
60+
/// Writes each `Link`'s populated DU case to the owner's links table.
61+
/// The kind is derived from which sub-element is non-null on the Link
62+
/// instance; a Link with multiple sub-elements set would be invalid by
63+
/// the XSD but is still handled deterministically (URL → XREF → ENTREZ).
64+
let write
65+
(connection: SqliteConnection)
66+
(owner: LinkOwner)
67+
(links: System.Collections.Generic.IEnumerable<Link>)
68+
: unit =
69+
if isNull links then () else
70+
links
71+
|> Seq.iteri (fun i link ->
72+
if not (isNull link.UrlLink) then
73+
let u = link.UrlLink
74+
insertRow connection owner i kindUrl u.Label u.UrlProperty null null null
75+
elif not (isNull link.XrefLink) then
76+
let x = link.XrefLink
77+
insertRow connection owner i kindXref x.Label null x.Db x.Id null
78+
elif not (isNull link.EntrezLink) then
79+
let e = link.EntrezLink
80+
let idText = if e.Id.HasValue then string e.Id.Value else null
81+
insertRow connection owner i kindEntrez e.Label null e.Db idText e.Query
82+
else
83+
failwithf
84+
"Link at ordinal %d in %s for %s='%s' has no URL_LINK / XREF_LINK / ENTREZ_LINK set"
85+
i owner.Table owner.AccessionColumn owner.Accession)
86+
87+
/// Reads all link rows in ordinal order and reconstructs `Link` instances
88+
/// with the appropriate DU case populated. Returns an empty list when
89+
/// none exist.
90+
let read (connection: SqliteConnection) (owner: LinkOwner) : Link list =
91+
let sql =
92+
sprintf
93+
"SELECT link_kind, label, url, db, id, query FROM %s WHERE %s = @acc ORDER BY ordinal;"
94+
owner.Table
95+
owner.AccessionColumn
96+
Sql.queryAll
97+
connection
98+
sql
99+
[ "@acc", box owner.Accession ]
100+
(fun reader ->
101+
let kind = reader.GetString(0)
102+
let label = Sql.readStringOrNull reader 1
103+
let url = Sql.readStringOrNull reader 2
104+
let db = Sql.readStringOrNull reader 3
105+
let id = Sql.readStringOrNull reader 4
106+
let query = Sql.readStringOrNull reader 5
107+
let link = Link()
108+
match kind with
109+
| k when k = kindUrl ->
110+
link.UrlLink <- Url(Label = label, UrlProperty = url)
111+
| k when k = kindXref ->
112+
link.XrefLink <- XRef(Label = label, Db = db, Id = id)
113+
| k when k = kindEntrez ->
114+
let entrez = LinkEntrezLink(Label = label, Db = db, Query = query)
115+
if not (isNull id) then
116+
entrez.Id <- Nullable<int64>(Int64.Parse(id))
117+
link.EntrezLink <- entrez
118+
| other ->
119+
failwithf
120+
"Unexpected link_kind '%s' in %s for %s='%s'"
121+
other owner.Table owner.AccessionColumn owner.Accession
122+
link)

0 commit comments

Comments
 (0)