Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
705f820
Add yaml for config files
krowvin Sep 16, 2025
d0684f3
Add initial templates
krowvin Sep 16, 2025
e66c7a7
Update report yaml to extend to columns/rows
krowvin Sep 16, 2025
d3ba838
Add initial reporting cmd/attempt render/scaffold design
krowvin Sep 16, 2025
661142e
Change to a default template given a config, update internal daily te…
krowvin Sep 17, 2025
3b0b54a
Update to allow for levels and per column href with specific begin/end
krowvin Sep 17, 2025
84820cc
Make the header dynamic
krowvin Sep 18, 2025
d724fc1
Add reporting diagram
krowvin Sep 18, 2025
713e14c
Add additional location placeholders for report, test target_time
krowvin Sep 22, 2025
2874fc1
Add dateparser/dateutil for extensive date options/strings in cwms-cli
krowvin Sep 23, 2025
c745f25
Restructure from single file
krowvin Sep 23, 2025
39fe299
Create reporting docs dir
krowvin Sep 23, 2025
e56dc6a
Add per column item begin/end times
krowvin Sep 23, 2025
7a24ade
Remove time delta idea
krowvin Sep 30, 2025
84f30f2
20 setup read the docs documentation (#31)
krowvin Sep 19, 2025
b1da09f
29 blob script needs lazy import for imghdr (#30)
krowvin Sep 19, 2025
e0b5160
18 overhaul csv2cwms (#34)
krowvin Sep 19, 2025
de5140f
Bump actions/checkout from 4 to 5 (#36)
dependabot[bot] Sep 23, 2025
bc61b0d
Bump actions/setup-python from 5 to 6 (#37)
dependabot[bot] Sep 23, 2025
c5cc51d
Combine USGS Scripts into Group (#40)
krowvin Sep 23, 2025
f337d3f
Update .readthedocs.yaml (#43)
krowvin Sep 23, 2025
d5441e4
revert back to 3.9 (#42)
Enovotny Sep 24, 2025
1ba1a9c
Fix global imports for lazy loading, fixes #47
krowvin Oct 6, 2025
41b596f
Add requirement for yaml library for configs
krowvin Dec 12, 2025
21dde5a
Add all locations in
krowvin Jan 13, 2026
e668447
Merge branch 'main' of https://github.com/HydrologicEngineeringCenter…
krowvin Mar 19, 2026
cd3b90a
- Add title/footer options
krowvin Mar 19, 2026
8d43589
Update model and design to accept a more generic community provided t…
krowvin Mar 19, 2026
2b4f4b1
- cli options for loc/dates
krowvin Mar 19, 2026
6cd6cc1
Ignore report outputs
krowvin Mar 20, 2026
a5cba81
Add a yearly report to core and update the lock
krowvin Mar 20, 2026
8c19c35
Add example configs
krowvin Mar 20, 2026
49cc69d
Add initial reporting docs
krowvin Mar 20, 2026
6233400
Add report testing
krowvin Mar 20, 2026
1076060
Merge branch 'main' of https://github.com/HydrologicEngineeringCenter…
krowvin Mar 20, 2026
977a8fb
refactor: narrow reporting mvp
krowvin Jun 21, 2026
7aa8451
feat: add SWT monthly lake report
krowvin Jun 21, 2026
cb7a226
refactor: make text reports template driven
krowvin Jun 21, 2026
76570f5
feat: allow report config overrides
krowvin Jun 21, 2026
bd61007
fix: support monthly report variants
krowvin Jun 21, 2026
e54c5da
fix: align monthly report fixed-width fields
krowvin Jun 21, 2026
46be33f
fix: handle monthly report boundary dates
krowvin Jun 21, 2026
6e63158
fix: align monthly summary fields
krowvin Jun 21, 2026
7630d26
fix: align legacy monthly template layout
krowvin Jun 21, 2026
b121278
feat: support monthly template display overrides
krowvin Jun 21, 2026
d710de3
fix: match legacy monthly text formatting
krowvin Jun 21, 2026
d766978
feat: allow monthly summary value overrides
krowvin Jun 21, 2026
29333ba
feat: load reports from package directories
krowvin Jun 21, 2026
2329cce
fix: retry transient report time series requests
krowvin Jun 21, 2026
3e36f9b
feat: configure report request retries
krowvin Jun 21, 2026
647cdc8
feat: add report series arithmetic helpers
krowvin Jun 21, 2026
0fd2f38
feat: select named reports from packages
krowvin Jun 21, 2026
8d4f4e7
feat: inspect report packages
krowvin Jun 21, 2026
0401ec9
fix: handle missing report values in templates
krowvin Jun 22, 2026
8538b6a
fix: support prior monthly report overrides
krowvin Jun 22, 2026
652411b
test: cover package text templates
krowvin Jun 22, 2026
4954637
test: cover report min max helpers
krowvin Jun 22, 2026
75b793c
test: use generic report fixtures
krowvin Jun 22, 2026
d8fe77a
fix: parse shell-safe nested report overrides
krowvin Jun 22, 2026
7b71f33
test: cover package yml manifests
krowvin Jun 22, 2026
b76bd6b
test: cover package custom report fields
krowvin Jun 22, 2026
8e688a9
test: cover report helper ranges and ties
krowvin Jun 22, 2026
81e4a87
test: cover named package report helpers
krowvin Jun 22, 2026
9085a73
feat: expose report builder layout context
krowvin Jul 30, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,14 @@ cython_debug/
# Ruff stuff:
.ruff_cache/

# Rendered report outputs from local `cwms-cli report generate` runs.
/*.html
/*.pdf
/report*.txt
/wm_daily_*.txt
/KEYS*.txt
/.tmp_reports/

# PyPI configuration file
.pypirc

Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ A collection of scripts to create, read, update, list, and delete data through C

[![Docs](https://readthedocs.org/projects/cwms-cli/badge/?version=latest)](https://cwms-cli.readthedocs.io/en/latest/cli.html#cwms-cli) - 📖 Read the docs: https://cwms-cli.readthedocs.io/en/latest/

[![Docs](https://readthedocs.org/projects/cwms-cli/badge/?version=latest)](https://cwms-cli.readthedocs.io/en/latest/)

## Install

```sh
Expand Down
2 changes: 2 additions & 0 deletions cwmscli/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@

from cwmscli.commands import commands_cwms
from cwmscli.load import __main__ as load
from cwmscli.reporting import report_cli
from cwmscli.usgs import usgs_group
from cwmscli.utils.click_help import add_version_to_help_tree
from cwmscli.utils.logging import LoggingConfig, setup_logging
Expand Down Expand Up @@ -54,6 +55,7 @@ def cli(log_file: Optional[str], no_color: bool, log_level: str) -> None:
cli.add_command(commands_cwms.csv2cwms_cmd)
cli.add_command(commands_cwms.update_cli_cmd)
cli.add_command(commands_cwms.blob_group)
cli.add_command(report_cli)
cli.add_command(load.load_group)
add_version_to_help_tree(cli)

Expand Down
212 changes: 107 additions & 105 deletions cwmscli/commands/blob.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@
import sys
from typing import Optional, Sequence

import cwms
import pandas as pd
import requests

from cwmscli.utils import get_api_key
from cwmscli.utils.deps import requires

Expand All @@ -23,21 +27,22 @@
"link": "https://docs.python.org/3/library/imghdr.html",
}
)
def _determine_ext(data: bytes) -> str:
def _determine_ext(data: bytes | str, write_type: str) -> str:
"""
Attempt to determine the file extension from the data itself.
Requires the imghdr module (lazy import) to inspect the bytes for image types.
If not an image, defaults to .bin

Args:
data: The binary data to inspect.
data: The binary data or base64 string to inspect.
write_type: The mode in which the data will be written ('wb' for binary, 'w' for text).

Returns:
The determined file extension, including the leading dot (e.g., '.png', '.jpg').
"""
import imghdr

kind: Optional[str] = imghdr.what(None, data)
kind = imghdr.what(None, data)
if kind == "jpeg":
kind = "jpg"
return f".{kind}" if kind else ".bin"
Expand All @@ -46,7 +51,7 @@ def _determine_ext(data: bytes) -> str:
def _save_base64(
b64_or_dataurl: str,
dest: str,
media_type_hint: Optional[str] = None,
media_type_hint: str | None = None,
) -> str:
m = DATA_URL_RE.match(b64_or_dataurl.strip())
if m:
Expand Down Expand Up @@ -75,8 +80,8 @@ def _save_base64(
ext = ".jpg"
# last resort, try to determine from the data itself
# requires imghdr to dig into the bytes to determine image type
if not ext and isinstance(data, bytes):
ext = _determine_ext(data)
if not ext:
ext = _determine_ext(data, write_type)
dest = base + ext

os.makedirs(os.path.dirname(dest) or ".", exist_ok=True)
Expand All @@ -86,11 +91,8 @@ def _save_base64(


def store_blob(**kwargs):
import cwms
import requests

file_data = kwargs.get("file_data")
blob_id = kwargs.get("blob_id", "").upper()
blob_id = kwargs.get("blob_id").upper()
# Attempt to determine what media type should be used for the mime-type if one is not presented based on the file extension
media = kwargs.get("media_type") or get_media_type(kwargs.get("input_file"))

Expand Down Expand Up @@ -144,15 +146,13 @@ def store_blob(**kwargs):


def retrieve_blob(**kwargs):
import cwms
import requests

blob_id = kwargs.get("blob_id", "").upper()
blob_id = kwargs.get("blob_id", "")
if not blob_id:
logging.warning(
"Valid blob_id required to download a blob. cwms-cli blob download --blob-id=myid. Run the list directive to see options for your office."
)
sys.exit(0)
blob_id = blob_id.upper()
logging.debug(f"Office: {kwargs.get('office')} Blob ID: {blob_id}")
try:
blob = cwms.get_blob(
Expand All @@ -174,17 +174,14 @@ def retrieve_blob(**kwargs):


def delete_blob(**kwargs):
import cwms
import requests

blob_id = kwargs.get("blob_id").upper()
logging.debug(f"Office: {kwargs.get('office')} Blob ID: {blob_id}")

try:
cwms.delete_blob(
office_id=kwargs.get("office"),
blob_id=kwargs.get("blob_id").upper(),
)
# cwms.delete_blob(
# office_id=kwargs.get("office"),
# blob_id=kwargs.get("blob_id").upper(),
# )
logging.info(f"Successfully deleted blob with ID: {blob_id}")
except requests.HTTPError as e:
details = getattr(e.response, "text", "") or str(e)
Expand All @@ -202,11 +199,8 @@ def list_blobs(
sort_by: Optional[Sequence[str]] = None,
ascending: bool = True,
limit: Optional[int] = None,
):
) -> pd.DataFrame:
logging.info(f"Listing blobs for office: {office!r}...")
import cwms
import pandas as pd

result = cwms.get_blobs(office_id=office, blob_id_like=blob_id_like)

# Accept either a DataFrame or a JSON/dict-like response
Expand Down Expand Up @@ -235,7 +229,7 @@ def list_blobs(
if limit is not None:
df = df.head(limit)

logging.info(f"Found {len(df):,} blob(s)")
logging.info(f"Found {len(df):,} blobs")
# List the blobs in the logger
for _, row in df.iterrows():
logging.info(f"Blob ID: {row['id']}, Description: {row.get('description')}")
Expand All @@ -247,6 +241,77 @@ def get_media_type(file_path: str) -> str:
return mime_type or "application/octet-stream"


def main(
directive: str,
input_file: str,
blob_id: str,
description: Optional[str],
media_type: Optional[str],
office: str,
api_root: str,
api_key: str,
overwrite: Optional[bool] = True,
dry_run: Optional[bool] = False,
):
"""
Upload, Download, Delete, or Update blob data in CWMS.

DIRECTIVE is the action to perform (upload, download, delete, update).
INPUT_FILE is the path to the file on disk.
BLOB_ID is the blob ID to store under.
"""

cwms.api.init_session(api_root=api_root, api_key=api_key)
file_data = None
if directive in ["upload", "update"]:
if not input_file or not os.path.isfile(input_file):
logging.warning(
"Valid input_file required for upload/update. Use --input-file to specify."
)
sys.exit(0)
try:
file_size = os.path.getsize(input_file)
with open(input_file, "rb") as f:
file_data = f.read()
logging.info(f"Read file: {input_file} ({file_size} bytes)")
except Exception as e:
logging.error(f"Failed to read file: {e}")
sys.exit(1)

# Determine what should be done based on directive
if directive == "upload":
store_blob(
office=office,
api_root=api_root,
input_file=input_file,
blob_id=blob_id,
description=description,
media_type=media_type,
file_data=file_data,
overwrite=overwrite,
dry_run=dry_run,
)
elif directive == "list":
list_blobs(office=office, blob_id_like=blob_id, sort_by="blob_id")
elif directive == "download":
retrieve_blob(
office=office,
blob_id=blob_id,
)
elif directive == "delete":
# TODO: Delete endpoint does not exist in cwms-python yet
logging.warning(
"[NOT IMPLEMENTED] Delete Blob is not supported yet!\n\thttps://github.com/HydrologicEngineeringCenter/cwms-python/issues/192"
)
pass
elif directive == "update":
# TODO: Patch endpoint does not exist in cwms-python yet
logging.warning(
"[NOT IMPLEMENTED] Update Blob is not supported yet! Consider overwriting instead if a rename is not needed.\n\thttps://github.com/HydrologicEngineeringCenter/cwms-python/issues/192"
)
pass


def upload_cmd(
input_file: str,
blob_id: str,
Expand All @@ -258,10 +323,7 @@ def upload_cmd(
api_root: str,
api_key: str,
):
import cwms
import requests

cwms.init_session(api_root=api_root, api_key=get_api_key(api_key, ""))
cwms.api.init_session(api_root=api_root, api_key=get_api_key(api_key, ""))
try:
file_size = os.path.getsize(input_file)
with open(input_file, "rb") as f:
Expand Down Expand Up @@ -289,7 +351,7 @@ def upload_cmd(
params = {"fail-if-exists": not overwrite}

if dry_run:
logging.info(f"DRY RUN: would POST {api_root}blobs with params={params}")
logging.info(f"--dry-run: would POST {api_root}blobs with params={params}")
logging.info(
json.dumps(
{
Expand All @@ -303,7 +365,7 @@ def upload_cmd(
return

try:
cwms.store_blobs(blob, fail_if_exists=not overwrite)
cwms.store_blobs(blob, fail_if_exists=overwrite)
logging.info(f"Uploaded blob: {blob_id_up}")
logging.info(f"View: {api_root}blobs/{blob_id_up}?office={office}")
except requests.HTTPError as e:
Expand All @@ -315,18 +377,8 @@ def upload_cmd(
sys.exit(1)


def download_cmd(
blob_id: str, dest: str, office: str, api_root: str, api_key: str, dry_run: bool
):
import cwms
import requests

if dry_run:
logging.info(
f"DRY RUN: would GET {api_root} blob with blob-id={blob_id} office={office}."
)
return
cwms.init_session(api_root=api_root)
def download_cmd(blob_id: str, dest: str, office: str, api_root: str, api_key: str):
cwms.api.init_session(api_root=api_root, api_key=get_api_key(api_key, ""))
bid = blob_id.upper()
logging.debug(f"Office={office} BlobID={bid}")

Expand All @@ -344,65 +396,18 @@ def download_cmd(
sys.exit(1)


def delete_cmd(blob_id: str, office: str, api_root: str, api_key: str, dry_run: bool):
import cwms

if dry_run:
logging.info(
f"DRY RUN: would DELETE {api_root} blob with blob-id={blob_id} office={office}"
)
return
cwms.init_session(api_root=api_root, api_key=api_key)
cwms.delete_blob(office_id=office, blob_id=blob_id)
logging.info(f"Deleted blob: {blob_id} for office: {office}")


def update_cmd(
input_file: str,
blob_id: str,
description: str,
media_type: str,
overwrite: bool,
dry_run: bool,
office: str,
api_root: str,
api_key: str,
):
import cwms
def delete_cmd(blob_id: str, office: str, api_root: str, api_key: str):
logging.warning(
"[NOT IMPLEMENTED] Delete Blob is not supported yet.\n"
"See: https://github.com/HydrologicEngineeringCenter/cwms-python/issues/192"
)

if dry_run:
logging.info(
f"DRY RUN: would PATCH {api_root} blob with blob-id={blob_id} office={office}"
)
return
file_data = None
if input_file:
try:
file_size = os.path.getsize(input_file)
with open(input_file, "rb") as f:
file_data = f.read()
logging.info(f"Read file: {input_file} ({file_size} bytes)")
except Exception as e:
logging.error(f"Failed to read file: {e}")
sys.exit(1)
# Setup minimum required payload
blob = {"office-id": office, "id": blob_id.upper()}
if description:
blob["description"] = description
if media_type:
blob["media-type-id"] = media_type
else:
logging.info("Media type not specified; Retrieving existing media type...")
blob_metadata = cwms.get_blobs(office_id=office, blob_id_like=blob_id)
blob["media-type-id"] = blob_metadata.df.get(
"media-type-id", "application/octet-stream"
)[0]
logging.info(f"Using existing media type: {blob['media-type-id']}")

if file_data:
blob["value"] = base64.b64encode(file_data).decode("utf-8")
cwms.init_session(api_root=api_root, api_key=api_key)
cwms.update_blob(blob, fail_if_not_exists=not overwrite)
def update_cmd(blob_id: str, input_file: str, office: str, api_root: str, api_key: str):
logging.warning(
"[NOT IMPLEMENTED] Update Blob is not supported yet. Consider --overwrite with upload.\n"
"See: https://github.com/HydrologicEngineeringCenter/cwms-python/issues/192"
)


def list_cmd(
Expand All @@ -416,10 +421,7 @@ def list_cmd(
api_root: str,
api_key: str,
):
import cwms
import pandas as pd

cwms.init_session(api_root=api_root)
cwms.api.init_session(api_root=api_root, api_key=get_api_key(api_key, None))
df = list_blobs(
office=office,
blob_id_like=blob_id_like,
Expand Down
Loading
Loading