Feat: pixml loader for rfc forecasts by msweier · Pull Request #254 · HydrologicEngineeringCenter/cwms-cli

msweier · 2026-08-07T20:07:03Z

Use cwms-cli nws pixml to load an NWS/RFC Delft-FEWS PI-XML forecast product into a CWMS database. Parameter mapping, timeseries group overrides, versioning, and issued-time tracking is driven by a JSON config file or a config blob stored in CDA.

nws pixml supports:

loading a single PI-XML file or URL (.gz/.zip auto-unzipped)
config-driven NWS to CWMS parameter mapping
TSID resolution via timeseries group alias override with optional built fallback
run selection by filename pattern (e.g. base / auto / CRF)
per-run versioning control (versioned or unversioned)
issued-time tracking via a consolidated JSON blob

This was based on MVP and MVM's needs and tested on the MVP and MVM RFC products with the hope that it may be configurable for other district's needs.

Developed with Claude Opus.

krowvin

A lot of acronyms. I'm not familiar with pixml. Might consider spelling the first ones out or linking to external docs more too.

Commenting for now, give Eric some time to peak this if he can.

krowvin · 2026-08-10T14:45:12Z

+The loader resolves its JSON config from the first source that matches:
+
+1. ``--config`` — a local JSON file path
+2. ``--config-blob`` — a blob ID to fetch from CDA


Suggested change

2. ``--config-blob`` — a blob ID to fetch from CDA

2. ``--config-blob-id`` — a blob ID to fetch from CDA

Without ID or the description one might think it's the actual blob data

krowvin · 2026-08-10T14:46:41Z

+----------------
+
+The config is a JSON object.  See the example configs shipped in
+``docs/nws/mvp.example.json`` and ``docs/nws/mvm.example.json``.


Will this link to these? Might see if you can

krowvin · 2026-08-10T14:48:00Z

+The config is a JSON object.  See the example configs shipped in
+``docs/nws/mvp.example.json`` and ``docs/nws/mvm.example.json``.
+
+Top-level keys:


Do we have a reference for external docs we can give for these as well?

krowvin · 2026-08-10T14:49:03Z

+   warned and skipped.  If ``build_missing_timeseries`` is false or omitted,
+   unmatched series are skipped instead of being built from the config.
+
+If two series in one product resolve to the same TSID, only the first is


Can we make sure to toss a warning log when this happens too. We can hope people do read this though.

krowvin · 2026-08-10T14:50:02Z

+- ``CDA_API_ROOT`` — default value for ``-a/--api-root``
+- ``CDA_API_KEY`` — default value for ``-k/--api-key``
+
+Example: MVP setup


Suggested change

Example: MVP setup

Example: MVP (St. Paul District) setup

krowvin

The only thing I really want to confirm before approval is our use of options, flags, and sub commands.

Typically something like
-i is a short option
--input is a long option
cwms-cli nws is a sub command of cwms-cli

In the source it is reading like
cwms-cli nws input ...etc is a valid option/arg.

I'd say the rest is largely up to you to decide if you wish to make changes, just let me know your thoughts in the replies.

That and if Mike has any opinions on that massively negative number. To me that looks like it's a 128bit signed number. Suggesting if you go to the bottom max of a 128bit signed value that's pretty much zero?

krowvin · 2026-08-10T14:54:00Z

+logger = logging.getLogger(__name__)
+
+# CWMS sentinel for a missing value and its quality code.
+CWMS_MISSING_VALUE = -340282346638528859811704183484516925440


@MikeNeilson

krowvin · 2026-08-10T14:54:44Z

+
+# Regular CWMS interval names keyed by their length in seconds. Used to derive the
+# interval segment of a built timeseries id from a PI-XML <timeStep> multiplier.
+_SECONDS_TO_INTERVAL = {


We should expose this via a util/global. Could get reused.

krowvin · 2026-08-10T14:56:13Z

+# --------------------------------------------------------------------------- #
+# Parse
+# --------------------------------------------------------------------------- #
+def _document_timezone(root, ns) -> timezone:


Consider adding tests for all these private methods?

krowvin · 2026-09-04T19:01:15Z

+@click.option(
+    "-i",
+    "--input",
+    "input_",


Unless we have a reason for this option I would take it out, I typically don't see somearg_ with a trailing slash and the other two should cover it

The trailing _ was one thing, but then I realized this ends up reading like new commands.

If we do want to go this route we'll have to document it because not every command does this

krowvin · 2026-09-04T19:03:09Z

+    "input_",
+    required=True,
+    type=str,
+    help="Path or URL to the PI-XML product. URLs ending in .gz or .zip are unzipped automatically.",


Do we have set options for this input? You might consider a method that reads the ending formats and throws an error if it's not one of those formats, normalized?

Options may be able to support extensions for files directly as well. We use Path in other places but there very well maybe a Url too.

While this isn't a blocker it could help with the user experience!

krowvin · 2026-09-04T19:22:58Z

+- ``OFFICE`` — default value for ``-o/--office``
+- ``CDA_API_ROOT`` — default value for ``-a/--api-root``
+- ``CDA_API_KEY`` — default value for ``-k/--api-key``


Possibly link to and/or have it display these from the global doc location on this?

krowvin · 2026-09-04T19:24:39Z

+  "location_alias_groups": [
+    {
+      "group_id": "NWS Handbook 5 ID",
+      "category_id": "Agency Aliases",
+      "group_office_id": "CWMS",
+      "category_office_id": "CWMS"
+    },


If the location group object ever changes we might have to come back in here and change this.

The way it was handled in GWW was to refrence what we need and then point to the generic cwmsjs/etc page that would define this.

In our case with this being python, the type doc pages do not exist... yet

krowvin · 2026-09-04T19:25:43Z

If possible can we rendered this config file somewhere? Part of the reason i chose RST was so we could dynamically render existing files into the docs themselves.

And that way you only have one place to change it, see above comment linking to config doc

krowvin · 2026-09-04T19:26:29Z

@@ -0,0 +1,100 @@
+{
+  "office": "MVP",
+  "pi_namespace": "http://www.wldelft.nl/fews/PI",


Should pixml also default to this if this namespace is not provided?

I also notice this is http? Do they not have an SSL varient/HTTPS?

krowvin · 2026-09-04T19:26:59Z

+    "SQIN": "Flow-Sim",
+    "QINE": "Flow",
+    "PELV": "Elev",
+    "SPEL": "Elev",
+    "RAIM": "Precip-RainAndMelt",
+    "MAP": "Precip-Rain",
+    "SSTG": "Stage"


If these params and other items are documented on their site lets be sure to add links to that. It also helps show that we are not responsible if these change.

msweier added 7 commits July 24, 2026 15:47

intial commit

d07deba

add tsid collision warning, add doc timezone, prevent overwriting blob

311937c

change order in doc tree

d2f3059

change to opt in for building timeseries, update error reporting

cef0a54

fix mvp parsing

f71577d

Merge remote-tracking branch 'origin/main' into feat/pixml

9c213d6

update maintainers

29d1d32

msweier requested review from Enovotny and krowvin as code owners August 7, 2026 20:07

msweier marked this pull request as draft August 7, 2026 20:08

msweier added 2 commits August 10, 2026 08:23

Merge branch 'main' into feat/pixml

af2adb9

Merge branch 'main' into feat/pixml

967c923

krowvin reviewed Aug 10, 2026

View reviewed changes

Merge branch 'main' into feat/pixml

a62c823

msweier marked this pull request as ready for review August 26, 2026 17:56

msweier requested review from krowvin August 26, 2026 19:07

msweier added 3 commits September 4, 2026 13:13

fix update blob w/ partial ts store failure

2251aa9

fix missing stores

d5e98b0

date issued blob store fallback

69f1298

krowvin requested changes Sep 4, 2026

View reviewed changes

	2. ``--config-blob`` — a blob ID to fetch from CDA
	2. ``--config-blob-id`` — a blob ID to fetch from CDA

Uh oh!

Conversation

msweier commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

krowvin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

krowvin Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

krowvin left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

msweier commented Aug 7, 2026 •

edited

Loading

krowvin Aug 10, 2026 •

edited

Loading

krowvin left a comment •

edited

Loading