Skip to content

Commit 22d48c9

Browse files
jmaeng72jmaeng72
committed
Merge branch 'master' of https://github.com/nasa/Common-Metadata-Repository into CMR-10636-main
2 parents 2a111a0 + 81be257 commit 22d48c9

20 files changed

Lines changed: 767 additions & 107 deletions

File tree

‎Generics.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,30 @@ Generic documents are documents which conform to the "Generic" API in CMR. These
55

66
## Configuration
77

8+
### Generic Configuration File
9+
10+
Within the Generic config.json file there is a section called `IndexConfiguration`. This contains two settings:
11+
12+
* AllowAppending: When set to true, then multiple Indexes with the same `Name` value will be appended together to create one larger value. Otherwise the last one in the config file will be the setting used.
13+
* AdditionalKeywords: List of simple fields to be added to the `keyword` field for general searching. By default CMR uses: LongName, Version, Description, RelatedURLs.
14+
15+
Indexes:
16+
* Description: Human readable description of field. Shows up in some logs
17+
* Field: [jq](https://jqlang.org) like path to field data
18+
* Name: Field Name
19+
* Mapping: The Elastic field type
20+
* token: text-field-mapping
21+
* string: string-field-mapping
22+
* int: int-field-mapping
23+
* date: date-field-mapping
24+
* Indexer:
25+
* default (none): direct, one-to-one mapping
26+
* simple-array-field: index a sub field of an array element
27+
* complex-fields-only: Complex indexer field that handles both single objects and arrays, and formats using field values only (not field names)
28+
* complex-field: takes a list of sub fields and combines them
29+
30+
### CMR settings
31+
832
If adding a new document, you will need to update the defconf variable by either setting an ENV for global change, or by updating the default value in [/common-lib/src/cmr/common/config.clj](/common-lib/src/cmr/common/config.clj). The format for this value is either JSON for an ENV variable or a clojure map if setting directly in the default attribute of the defconfig like this:
933

1034
(defconfig approved-pipeline-documents

‎elastic-utils-lib/project.clj‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
(def elastic-version "7.17.14")
1+
(def elastic-version "7.17.25")
22

33
(defproject nasa-cmr/cmr-elastic-utils-lib "0.1.0-SNAPSHOT"
44
:description "A library containing utilities for dealing with Elasticsearch."

‎elastic-utils-lib/src/cmr/elastic_utils/generics.clj‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
(ns cmr.elastic-utils.generics
2+
"A set of functions for dealing with Generics and Elastic")
3+
4+
(defn only-elastic-preferences
5+
"Go through all the index configurations and return only the ones related to
6+
generating elastic values. If an index does not specify what type it is for,
7+
then assume elastic"
8+
[list-of-indexes]
9+
(keep #(if (not (nil? %)) %)
10+
(map
11+
(fn [x] (when (or (nil? (:Type x)) (= "elastic" (:Type x))) x))
12+
list-of-indexes)))

‎es-spatial-plugin/project.clj‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,8 @@
2323
(def es-deps-target-path
2424
"es-deps")
2525

26+
(def elastic-version "7.17.25")
27+
2628
(defproject nasa-cmr/cmr-es-spatial-plugin "0.1.0-SNAPSHOT"
2729
:description "A Elastic Search plugin that enables spatial search entirely within elastic."
2830
:url "https://github.com/nasa/Common-Metadata-Repository/tree/master/es-spatial-plugin"
@@ -42,7 +44,7 @@
4244
[com.fasterxml.jackson.dataformat/jackson-dataformat-cbor]
4345
[com.fasterxml.jackson.dataformat/jackson-dataformat-smile]
4446
[com.fasterxml.jackson.dataformat/jackson-dataformat-yaml]]]
45-
[org.elasticsearch/elasticsearch "7.17.14"]
47+
[org.elasticsearch/elasticsearch ~elastic-version]
4648
[org.clojure/tools.reader "1.3.2"]
4749
[org.yaml/snakeyaml "1.31"]]}
4850
:es-deps {:dependencies [[nasa-cmr/cmr-spatial-lib "0.1.0-SNAPSHOT"
@@ -77,7 +79,7 @@
7779
[org.clojure/tools.reader "1.3.2"]
7880
[nasa-cmr/cmr-common-lib "0.1.1-SNAPSHOT"]
7981
[nasa-cmr/cmr-spatial-lib "0.1.0-SNAPSHOT"]
80-
[org.elasticsearch/elasticsearch "7.17.14"]
82+
[org.elasticsearch/elasticsearch ~elastic-version]
8183
[org.clojars.gjahad/debug-repl "0.3.3"]
8284
[org.clojure/tools.nrepl "0.2.13"]
8385
[org.clojure/tools.namespace "0.2.11"]

‎indexer-app/src/cmr/indexer/data/concepts/generic.clj‎

Lines changed: 177 additions & 60 deletions
Original file line numberDiff line numberDiff line change
@@ -10,15 +10,17 @@
1010
[clojure.string :as string]
1111
[cmr.common.concepts :as concepts]
1212
[cmr.common.generics :as generics]
13+
[cmr.common.time-keeper :as time-keeper]
1314
[cmr.common.util :as util]
15+
[cmr.common-app.config :as app-config]
1416
[cmr.indexer.data.concepts.association-util :as assoc-util]
1517
[cmr.indexer.data.concept-parser :as c-parser]
1618
[cmr.indexer.data.concepts.generic-util :as gen-util]
1719
[cmr.indexer.data.concepts.keyword-util :as keyword-util]
1820
[cmr.indexer.data.elasticsearch :as esearch]
1921
[cmr.transmit.metadata-db :as meta-db]))
2022

21-
(defn field->index-complex-field
23+
(defn- field->index-complex-field
2224
"This is an example of a complex indexer which takes a list of sub fields and
2325
combines them into one field"
2426
[settings data]
@@ -40,37 +42,88 @@
4042
{(keyword field-name) field-value
4143
(keyword field-name-lower) field-value-lower}))
4244

43-
(defn field->index-simple-array-field
44-
"The gets the a sub field of an array element and puts those values
45-
into a list so that each value can be searched on one of two indexes:
46-
the literal case, another is all lower case."
45+
(defn- field->index-complex-field-with-values-only
46+
"Complex indexer field that handles both single objects and arrays,
47+
and formats using field values only (not field names)"
48+
[settings data]
49+
(let [field-list (get settings :Field ".")
50+
field-data (get-in data (generics/jq->list field-list keyword) {})
51+
config (get settings :Configuration {})
52+
sub-fields (get config :sub-fields {})
53+
layout (get config :format "%s=%s")
54+
field-name (util/safe-lowercase (:Name settings))
55+
field-name-lower (str field-name "-lowercase")
56+
field-value (if (vector? field-data)
57+
;; Handle array case
58+
(mapv (fn [element]
59+
(let [values (map #(get element (keyword %)) sub-fields)]
60+
(apply format layout values)))
61+
field-data)
62+
;; Handle single object case
63+
(let [values (map #(get field-data (keyword %)) sub-fields)]
64+
(apply format layout values)))
65+
66+
field-value-lower (if (vector? field-value)
67+
(mapv #(util/safe-lowercase %) field-value)
68+
(util/safe-lowercase field-value))]
69+
{(keyword field-name) field-value
70+
(keyword field-name-lower) field-value-lower}))
71+
72+
(defn- field->index-simple-array-field
73+
"The gets the a sub field of an array element and puts those values into a list so that each value
74+
can be searched on one of two indexes: the literal case, another is all lower case.
75+
Example usage:
76+
(field->index-simple-array-field
77+
;; settings
78+
{:Field \".ConceptIds\"
79+
:Name \"Concept-Ids\"
80+
:Configuration {:sub-fields [\"Value\", \"ShortName\"]}}
81+
;; data
82+
{:ConceptIds [{:Value \"V-One\" :ShortName \"Short-One\" :ignore \"I-one\"}
83+
{:Value \"V-Two\" :ShortName \"Short-Two\" :ignore \"I-two\"}]})
84+
Result:
85+
{:concept-ids \"Short-Two Short-One V-Two V-One\"
86+
:concept-ids-lowercase \"short-two short-one v-two v-one\"}
87+
"
4788
[settings data]
4889
(let [field-list (get settings :Field ".")
4990
field-data (get-in data (generics/jq->list field-list keyword) {})
5091
field-name (util/safe-lowercase (:Name settings))
5192
field-name-lower (str field-name "-lowercase")
5293
config (get settings :Configuration {})
5394
sub-fields (get config :sub-fields {})
54-
value (reduce (fn [data, key-name]
55-
(into data (map (keyword key-name) field-data)))
56-
(sequence nil)
57-
sub-fields)
58-
value-lower (map #(util/safe-lowercase %) value)]
95+
value-raw (reduce (fn [data, key-name]
96+
(into data (map (keyword key-name) field-data)))
97+
(sequence nil)
98+
sub-fields)
99+
value (cond
100+
(string? value-raw) value-raw
101+
(coll? value-raw) (clojure.string/join " " (map str value-raw))
102+
:else (str value-raw))
103+
value-lower (util/safe-lowercase value)]
59104
{(keyword field-name) value
60105
(keyword field-name-lower) value-lower}))
61106

62-
(defn field->index-default-field
63-
"The default indexer which will map one metadata field to two indexes. One is
64-
with the literal case, another is all lower case"
107+
(defn- field->index-default-field
108+
"The default indexer which will map one metadata field to two indexes. One is with the literal
109+
case, another is all lower case. If no field is found, then don't return anything
110+
Example Usage:
111+
(field->index-default-field
112+
{:Field \".VisualizationType\" :Name \"VisualizationType\"}
113+
{:VisualizationType \"Default\"})
114+
Result:
115+
{:visualizationtype \"Default\", :visualizationtype-lowercase \"default\"}
116+
"
65117
[settings data]
66118
(let [field-name (util/safe-lowercase (:Name settings))
67119
field-name-lower (str field-name "-lowercase")
68120
value (get-in data (generics/jq->list (:Field settings) keyword))
69121
value-lower (util/safe-lowercase value)]
70-
{(keyword field-name) value
71-
(keyword field-name-lower) value-lower}))
122+
(when value
123+
{(keyword field-name) value
124+
(keyword field-name-lower) value-lower})))
72125

73-
(defn field->index
126+
(defn- field->index
74127
"Functions which convert a part of metadata to a name-value which can be added
75128
to an index document. This function is directed by looking for an :Indexer
76129
value in settings and assuming :default if it is not set.
@@ -86,54 +139,115 @@
86139
[settings data]
87140
(case (:Indexer settings)
88141
"complex-field" (field->index-complex-field settings data)
142+
"complex-fields-only" (field->index-complex-field-with-values-only settings data)
89143
"simple-array-field" (field->index-simple-array-field settings data)
90144
(field->index-default-field settings data)))
91145

146+
(defn- merge-or-concat
147+
"Facilitate the behavior of the :AllowAppending configuration by concatinating two values"
148+
([existing new-value]
149+
(merge-or-concat existing new-value " "))
150+
([existing new-value separator]
151+
(string/trim (if (nil? existing)
152+
(str new-value)
153+
(if (or (empty? existing) (empty? new-value))
154+
(str existing new-value)
155+
(str existing separator new-value))))))
156+
157+
(defn- post-process-keyword
158+
"Go through the :keyword and :keyword-lowercase fields in the Elastic Document and make sure they
159+
are tokenized and sorted in the 'CMR' way."
160+
[doc]
161+
(-> doc
162+
(assoc :keyword
163+
(keyword-util/field-values->keyword-text [(:keyword doc)]))
164+
(assoc :keyword-lowercase
165+
(keyword-util/field-values->keyword-text [(:keyword-lowercase doc)]))))
166+
167+
(defn- release-number
168+
"Format the software release number for using in the index document. Use a date if in development."
169+
[]
170+
(let [rel-ver (app-config/release-version)]
171+
(if (= rel-ver "dev")
172+
(format "%s-%s", rel-ver, (time-keeper/now))
173+
(app-config/release-version))))
174+
92175
(defn- parsed-concept->elastic-doc
93176
"Generate elastic document"
94-
[context concept parsed-concept]
95-
(let [{:keys [concept-id revision-id deleted provider-id user-id
96-
revision-date native-id]} concept
97-
parsed-concept (if (:deleted concept)
98-
(c-parser/parse-concept context concept)
99-
parsed-concept)
100-
generic-associations (esearch/parse-non-tombstone-associations
101-
context
102-
(meta-db/get-generic-associations-for-concept context concept))
103-
gen-name (csk/->kebab-case (get-in parsed-concept [:MetadataSpecification :Name] ""))
104-
gen-ver (get-in parsed-concept [:MetadataSpecification :Version])
105-
index-data-file (format "schemas/%s/v%s/config.json" gen-name gen-ver)
106-
index-file-raw (slurp (io/resource index-data-file))
107-
index-data (json/parse-string index-file-raw true)
108-
schema-keys [:LongName
109-
:Version
110-
:Description
111-
:RelatedURLs]
112-
keyword-values (keyword-util/concept-keys->keyword-text
113-
parsed-concept schema-keys)
114-
common-doc ;; fields common to all generic documents
115-
{:concept-id concept-id
116-
:revision-id revision-id
117-
:deleted deleted
118-
:gen-name gen-name
119-
:gen-name-lowercase (util/safe-lowercase gen-name)
120-
:gen-version gen-ver
121-
:generic-type (str gen-name " " gen-ver)
122-
:provider-id provider-id
123-
:provider-id-lowercase (util/safe-lowercase provider-id)
124-
:keyword keyword-values
125-
:user-id user-id
126-
:revision-date revision-date
127-
:native-id native-id
128-
:native-id-lowercase (string/lower-case native-id)
129-
:associations-gzip-b64 (assoc-util/associations->gzip-base64-str generic-associations concept-id)}
130-
configs (gen-util/only-elastic-preferences (:Indexes index-data))
131-
;; now add the configured indexes
132-
doc (reduce
133-
(fn [data, config] (into data (field->index config parsed-concept)))
134-
common-doc
135-
configs)]
136-
doc))
177+
([context concept parsed-concept]
178+
;; Abstract away the need to have context or find files in the jar with this form of the function
179+
;; so that the next interface can be called in tests.
180+
(let [parsed-concept (if (:deleted concept)
181+
(c-parser/parse-concept context concept)
182+
parsed-concept)
183+
generic-associations (esearch/parse-non-tombstone-associations
184+
context
185+
(meta-db/get-generic-associations-for-concept context concept))
186+
gen-name (csk/->kebab-case (get-in parsed-concept [:MetadataSpecification :Name] ""))
187+
gen-ver (get-in parsed-concept [:MetadataSpecification :Version])
188+
index-data (-> "schemas/%s/v%s/config.json"
189+
(format gen-name gen-ver)
190+
(io/resource)
191+
(slurp)
192+
(json/parse-string true))]
193+
(parsed-concept->elastic-doc concept
194+
parsed-concept
195+
generic-associations
196+
gen-name
197+
gen-ver
198+
index-data)))
199+
200+
;; Creating a new overload of the function so that the core functionality can be tested without a
201+
;; context
202+
([concept parsed-concept generic-associations gen-name gen-ver index-data]
203+
(let [{:keys [concept-id revision-id deleted provider-id user-id
204+
revision-date native-id]} concept
205+
allow-appending (get-in index-data [:IndexConfiguration :AllowAppending])
206+
additional-keywords (get-in index-data [:IndexConfiguration :AdditionalKeywords] {})
207+
;; Allow Generic Document Configurations to extend the list of simple top level fields that
208+
;; can be added to the keyword field. This process does not preclude the use of more
209+
;; complicated fields also being added when using the "AllowAppending" setting.
210+
schema-keys (into [:LongName
211+
:Version
212+
:Description
213+
:RelatedURLs]
214+
(map keyword additional-keywords))
215+
keyword-values (keyword-util/concept-keys->keyword-text
216+
parsed-concept schema-keys)
217+
common-doc ;; fields common to all generic documents
218+
{:cmr-version (release-number)
219+
:concept-id concept-id
220+
:revision-id revision-id
221+
:deleted deleted
222+
:gen-name gen-name
223+
:gen-name-lowercase (util/safe-lowercase gen-name)
224+
:gen-version gen-ver
225+
:generic-type (str gen-name " " gen-ver)
226+
:provider-id provider-id
227+
:provider-id-lowercase (util/safe-lowercase provider-id)
228+
:keyword keyword-values
229+
:keyword-lowercase keyword-values
230+
:user-id user-id
231+
:revision-date revision-date
232+
:native-id native-id
233+
:native-id-lowercase (string/lower-case native-id)
234+
:associations-gzip-b64 (assoc-util/associations->gzip-base64-str
235+
generic-associations concept-id)}
236+
configs (gen-util/only-elastic-preferences (:Indexes index-data))
237+
;; now add the configured indexes
238+
doc (reduce
239+
(fn [data config]
240+
;; Allow Appending from the configuration file allows for multiple rules to be
241+
;; defined using the same :Name. When this is set, then these indexes are added
242+
;; together to form one larger value made up of different parts of the Generic
243+
;; document. Otherwise, the legacy behavier is that the last rule 'wins'.
244+
(if allow-appending
245+
(merge-with merge-or-concat data (field->index config parsed-concept))
246+
(into data (field->index config parsed-concept))))
247+
common-doc
248+
configs)]
249+
;; Finally clean up the keyword indexs which may have been appended
250+
(post-process-keyword doc))))
137251

138252
(defn field->keyword-fields
139253
"Gets the name of the fields that are used for keyword searches.
@@ -170,15 +284,18 @@
170284
parsed-concept (if (:deleted concept)
171285
(c-parser/parse-concept context concept)
172286
parsed-concept)
287+
;; No associations in drafts
173288
concept-type (concepts/concept-id->type (:concept-id concept))
174289
version (generics/current-generic-version concept-type)
175290
gen-name (csk/->kebab-case (get-in parsed-concept [:MetadataSpecification :Name] ""))
176291
gen-ver (get-in parsed-concept [:MetadataSpecification :Version])
177292
index-data-file (format "schemas/%s/v%s/config.json" (name concept-type) version)
178293
index-file-raw (slurp (io/resource index-data-file))
179294
index-data (json/parse-string index-file-raw true)
295+
;; No default keyword processing till latter
180296
common-doc ;; fields common to all generic documents
181-
{:concept-id concept-id
297+
{:cmr-version (release-number)
298+
:concept-id concept-id
182299
:revision-id revision-id
183300
:deleted deleted
184301
:gen-name gen-name

‎indexer-app/src/cmr/indexer/data/concepts/generic_util.clj‎

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,14 @@
11
(ns cmr.indexer.data.concepts.generic-util
22
"Contains functions to parse and convert Generic Documents (that is a document
33
complying to a schema supported by the Generic Document system) to and object
4-
that can be indexed in lucine."
5-
(:require
6-
[clojure.string :as string]))
4+
that can be indexed in lucine.")
75

86
(defn only-elastic-preferences
97
"Go through all the index configurations and return only the ones related to
108
generating elastic values. If an index does not specify what type it is for,
119
then assume elastic"
1210
[list-of-indexes]
13-
(keep #(if (not (nil? %)) %)
11+
(keep #(when (not (nil? %)) %)
1412
(map
1513
(fn [x] (when (or (nil? (:Type x)) (= "elastic" (:Type x))) x))
1614
list-of-indexes)))

0 commit comments

Comments
 (0)