i.hyper.lib_relab/i.hyper.lib_relab.html at main · YannChemin/i.hyper.lib_relab · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
<!DOCTYPE html>
<html>
<head>
<title>i.hyper.lib_relab - GRASS GIS manual</title>
<meta charset="utf-8">
</head>
<body>

<h2>DESCRIPTION</h2>

<p><em>i.hyper.lib_relab</em> is the third <b>i.hyper.lib_*</b> harvester
(after <em>i.hyper.lib_ecosis</em> and <em>i.hyper.lib_usgs</em>): it
parses a local download of the RELAB (Reflectance Experiment Laboratory)
spectral database, run by Brown University for NASA's Keck/RELAB program,
and writes it into the <b>same shared, source-tagged database</b> at
<code>$HOME/grassdata/hyperspeclib</code>, using the exact same row schema
as the other two harvesters &mdash; one <code>pyarrow.dataset</code> scan
spans EcoSIS field spectra, USGS lab reference spectra, and RELAB lab
reference spectra together.</p>

<p>RELAB's data release, like USGS's, has no live query API &mdash; it is a
static downloaded archive. <b>input_dir</b> must point at the root of that
download (e.g. <code>~/DBDATA/RelabDatabase2025Dec31</code>), containing
its <code>catalogues/</code> and <code>data/</code> subdirectories as
distributed.</p>

<h3>Downloading the data</h3>

<ol>
  <li>Go to the RELAB Spectral Database page:
    <a href="https://sites.brown.edu/relab/relab-spectral-database/">https://sites.brown.edu/relab/relab-spectral-database/</a>
    (Reflectance Experiment Laboratory, Department of Earth, Environmental
    and Planetary Sciences, Brown University, run for NASA's Keck/RELAB
    program).</li>
  <li>The page offers the two catalogue files (<code>Sample_Catalogue</code>
    / <code>Spectra_Catalogue</code>, needed to look up individual spectra
    by sample) and, per this module's own <code>ReadMeFirst.txt</code>,
    "a compressed zip file containing all the catalogues and data" as a
    single combined download &mdash; that single zip is the simplest
    option and is what this harvester expects.</li>
  <li>Extract the zip into a directory of your choice, e.g.
    <code>~/DBDATA/RelabDatabase&lt;date&gt;/</code> &mdash; it unpacks
    directly into <code>catalogues/</code> (the TSV catalogue files) and
    <code>data/</code> (one <code>.asc</code>/<code>.txt</code> pair per
    spectrum, bucketed into <code>data/&lt;pi_code&gt;/&lt;sample_id
    prefix&gt;/</code>) as siblings. This extracted root is what
    <b>input_dir</b> must point at.</li>
  <li>The full archive is several GB (34,564 spectra in the 2025-12-31
    release used to build and verify this harvester); ingestion reads
    only the <code>.asc</code> files (not the equivalent
    <code>.txt</code> files, which carry the same data in a different
    unit/format) plus the small catalogue TSVs.</li>
</ol>

<h3>Why this needed its own parser (not shared with the other two)</h3>

<p>RELAB's on-disk layout was studied via the reference parser in
<code>$HOME/dev/spectral/spectral/database/relab.py</code> (the
<code>spectral</code> Python package) and via the archive's own
<code>ReadMeFirst.txt</code> &mdash; not used as a dependency,
reimplemented here because RELAB's real local download turned out to be
<b>richer and more structured</b> than either the reference parser or
<em>i.hyper.lib_usgs</em>'s per-sample-HTML approach: it ships two proper
relational catalogue tables (tab-separated, one header row each) instead
of requiring metadata to be scraped out of free text per spectrum.</p>

<ul>
  <li><b><code>catalogues/Sample_Catalogue.txt</code></b> (~16,800 rows):
    one row per physical sample &mdash; name, PI code, mineralogy
    classification (<code>GeneralType1</code>/<code>Type1</code>/<code>Type2</code>/<code>SubType</code>),
    particle size, texture, origin/locality, free-text description.</li>
  <li><b><code>catalogues/Spectra_Catalogue.txt</code></b> (~33,800 rows):
    one row per measured spectrum, foreign-keyed to
    <code>Sample_Catalogue</code> by <code>SampleID</code> &mdash;
    measurement date, instrument code (<code>SpecCode</code>), spectral
    range, geometry angles, temperature, atmosphere, research context.</li>
  <li><b><code>catalogues/PI_Address.txt</code></b> and
    <b><code>catalogues/MeasurementTypes.txt</code></b> are small lookup
    tables (PI code -&gt; institution/full name; SpecCode -&gt;
    human-readable spectrometer name and measurement mode) joined in to
    enrich <b>organization</b> and <code>extra_metadata</code>
    respectively, rather than leaving raw codes as the only trace.</li>
  <li><b>The spectrum data itself</b> lives in
    <code>data/&lt;pi_code&gt;/&lt;sample_id prefix&gt;/&lt;SpectrumID&gt;.asc</code>,
    lowercased (e.g. sample <code>BF-BFJ-004</code>, spectrum
    <code>C1BF04</code> -&gt; <code>data/bfj/bf/c1bf04.asc</code>) &mdash; a
    first line giving the number of data points, then that many
    <code>wavelength(nm) reflectance [stddev]</code> lines
    (space-delimited; the optional third stddev column, present in roughly
    40% of files, is not carried into the common
    <code>wavelengths</code>/<code>values</code> row schema), then
    trailing free-text ignored by this module. Unlike USGS's
    <code>.txt</code> files, RELAB's <code>.asc</code> wavelengths are
    already in nanometers &mdash; no unit conversion needed.</li>
</ul>

<p>This catalogue-driven approach means <b>no per-file HTML scraping</b> is
needed at all: <code>Spectra_Catalogue.txt</code> is scanned once (a cheap
metadata-only pass, joined against the in-memory <code>Sample_Catalogue</code>
dict) to group every spectrum by category <i>before</i> touching a single
<code>.asc</code> file, then each category's actual spectra are parsed and
written in the same incremental, batched way as the other two
harvesters.</p>

<h3>Dataset grouping</h3>

<p>RELAB has no chapter-like folder structure at all (its <code>data/</code>
directory is bucketed by PI code + sample-ID prefix, an organizational
convenience with no semantic meaning) &mdash; so, analogous to
<em>i.hyper.lib_usgs</em>'s use of USGS's chapters, this harvester groups
by <b><code>Sample_Catalogue</code>'s <code>GeneralType1</code></b> field
(Mineral, Rock, Sediment, Glass, Biological, Mixture, Organic, Regolith,
... &mdash; about 26 categories observed in practice, including a few
small/idiosyncratic ones and an <code>Unclassified</code> bucket for
samples with no <code>GeneralType1</code> recorded), sanitized into a
filesystem-safe <code>dataset_id</code> (e.g. <code>Chemical Compound</code>
-&gt; <code>Chemical_Compound</code>) while keeping the original text as
<code>dataset_title</code>. <b>general_types=</b> restricts ingestion to
specific categories (matched against the sanitized form; default:
all).</p>

<h3>No geolocation, no live per-record web page</h3>

<p>Same as <em>i.hyper.lib_usgs</em>: RELAB lab samples have no coordinates
(a free-text <code>Location</code>/<code>Origin</code> field exists for
some, not structured lat/lon), so <code>longitude</code>/<code>latitude</code>
are always null for this source. And unlike EcoSIS, there is no confirmed
single per-record web page or DOI to cite in the local download itself, so
<b>source_url</b>/<b>source_api_url</b> are left null rather than guessed
&mdash; per-record traceability instead comes entirely from
<code>extra_metadata</code>'s <code>local_spectrum_file</code> path plus
the <code>SpectrumID</code>/<code>SampleID</code> identifiers, which any
RELAB catalogue search can resolve back to the original record.</p>

<h3>Incremental re-runs and streaming</h3>

<p>Same conventions as the other two harvesters: a shared
<code>_manifest.json</code> at the database root (keyed
<code>relab:&lt;category&gt;</code>) tracks each category's <i>source</i>
spectrum count (<code>expected_total</code>) to decide whether a re-run
can skip it (<b>-f</b> to force), and records are batched
(<b>stream_batch_size</b>, default 2000) and written incrementally via the
same <code>ParquetWriter</code>/<code>executemany()</code> per-batch
pattern.</p>

<h2>NOTES</h2>

<p>Use <b>-k</b> to keep going past a spectrum that raises a parse error
instead of aborting the whole run. A spectrum whose catalogued
<code>.asc</code> file is simply <i>missing on disk</i> (a real, if
uncommon, gap in the archive itself &mdash; confirmed present in this
session's actual download: 170 of 33,786 catalogued spectra) is always
skipped silently regardless of <b>-k</b>, since that is an expected, not
exceptional, condition.</p>

<h2>EXAMPLES</h2>

<h3>Ingest a single (small) category, GeoParquet backend</h3>

<div class="code"><pre>
i.hyper.lib_relab input_dir=$HOME/DBDATA/RelabDatabase2025Dec31 \
    output=/tmp/test_relab_lib format=parquet general_types=Coating
</pre></div>

<div class="code"><pre>
Loading RELAB catalogues (Sample_Catalogue, PI_Address, MeasurementTypes)&hellip;
16786 samples in Sample_Catalogue.
Scanning Spectra_Catalogue&hellip;
Found 10 spectra across 1 category/categories.
Wrote 10 record(s) for 'Coating' &rarr;
/tmp/test_relab_lib/source_database=relab/dataset_Coating.parquet
Done: 1 categor(y/ies) ingested (10 spectra), 0 already up to date, library
at /tmp/test_relab_lib (format=parquet).
</pre></div>

<p>One of those 10 rows, inspected directly:</p>

<div class="code"><pre>
record_id:      BIR1LH053
dataset_title:  Coating
organization:   Department of Geological Sciences, Brown University
n_bands:        3527
wavelengths[:3]: [1428.4, 1428.8, 1429.2]   (nm -- already native, no conversion)
extra_metadata:
  sample_id:          LH-JFM-053
  sample_name:        SB-01 chips
  general_type1:      Coating
  type1:              Iron Oxide
  sub_type:           Jarosite
  texture:            Chip
  origin:             Rio Tinto, Spain
  spectrometer_name:  Thermo Nexus 870 with PIKE AutoDiff
  measurement_mode:   Reflectance (Off-axis biconical FT-IR)
  spectra_catalogue_speccode:      BCF-FTIR2
  spectra_catalogue_researchtype:  Astrobiology, Student
  local_spectrum_file: .../data/jfm/lh/bir1lh053.asc
</pre></div>

<h3>Full library: all categories into the shared database</h3>

<div class="code"><pre>
i.hyper.lib_relab input_dir=$HOME/DBDATA/RelabDatabase2025Dec31
</pre></div>

<div class="code"><pre>
Loading RELAB catalogues (Sample_Catalogue, PI_Address, MeasurementTypes)&hellip;
16786 samples in Sample_Catalogue.
Scanning Spectra_Catalogue&hellip;
Found 33786 spectra across 26 category/categories.
Wrote 609 record(s) for 'Biological' &rarr; .../dataset_Biological.parquet
Wrote 507 record(s) for 'Glass' &rarr; .../dataset_Glass.parquet
...
Wrote 5016 record(s) for 'Sediment' &rarr; .../dataset_Sediment.parquet
Wrote 97 record(s) for 'Unclassified' &rarr; .../dataset_Unclassified.parquet
Done: 26 categor(y/ies) ingested (33616 spectra), 0 already up to date,
library at /home/yann/grassdata/hyperspeclib (format=parquet).
</pre></div>

<p>170 of the 33,786 catalogued spectra were skipped (their
<code>.asc</code> file is not actually present in this download &mdash; a
real, pre-existing gap in the archive, confirmed by checking each missing
path directly, not a parsing failure). Re-running the same command
afterward is fully idempotent (<code>0 categor(y/ies) ingested ... 26
already up to date</code>).</p>

<h3>Querying across all three harvested sources at once</h3>

<p>With <em>i.hyper.lib_ecosis</em> and <em>i.hyper.lib_usgs</em> having
already populated the same library root, all three sources now live side
by side under their own <code>source_database=</code> partition,
queryable together in one scan:</p>

<div class="code"><pre>
import pyarrow.dataset as ds

dataset = ds.dataset("/home/yann/grassdata/hyperspeclib", format="parquet", partitioning="hive")
table = dataset.to_table(columns=["source_database"])
print(table.to_pandas()["source_database"].value_counts())
</pre></div>

<div class="code"><pre>
source_database
ecosis          229893
relab            33616
usgs_splib07      2457
</pre></div>

<p>228 total Parquet files, 2.5 GB, 265,966 spectra from three
independently harvested sources, one unified schema.</p>

<h2>SEE ALSO</h2>

<em>
<a href="i.hyper.lib_ecosis.html">i.hyper.lib_ecosis</a>,
<a href="i.hyper.lib_usgs.html">i.hyper.lib_usgs</a>,
<a href="i.hyper.spectroscopy.html">i.hyper.spectroscopy</a>,
<a href="i.hyper.endmembers.html">i.hyper.endmembers</a>
</em>

<p>RELAB (Reflectance Experiment Laboratory), Brown University, operated
for NASA's Keck/RELAB program.</p>

<p>GeoParquet specification: <a href="https://geoparquet.org">https://geoparquet.org</a></p>

<h2>AUTHOR</h2>

Spectral Feature Extraction and Interpretation Engine

</body>
</html>