Conversation
Fixes CI failures: black --check was failing on three files, and actions/checkout, actions/cache, setup-miniconda, setup-node, and codecov-action were pinned to old majors that target the now deprecated Node 20 runtime. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Mambaforge installers are no longer published upstream, causing a 404 on download. Miniforge3 (the action's default) now bundles mamba by default and use-mamba: true is already set, so this is a no-op behaviorally. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
0018_remove_zentra_add_open_meteo_fields depended on the old 0017_remove_aggregateddepthprediction_tile_size_and_more migration, which had already been squashed and superseded by 0003_riverchannel via 0001_squashed_0017/0002_remove_depthprediction_bounding_box_and_more. On a fresh database Django rewires that stale dependency onto the squash node directly, creating two leaf migrations and failing CI with "Conflicting migrations detected". Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Migrates ManyFEWS’ weather-data ingestion away from Zentra Cloud + NOAA GEFS to the Open-Meteo API (historical archive + ensemble forecast), updating the model pipeline, schema, scheduled tasks, and documentation accordingly. Also tightens some security-related settings defaults and fixes Twilio verification semantics.
Changes:
- Replace Zentra/GEFS ingestion with new
calculations/open_meteo.pyand update task flow to run per Open-Meteo ensemble member. - Update DB schema (remove Zentra models, add Open-Meteo forecast fields + aggregated historical model) and adjust downstream flood-risk processing to aggregate across ensemble members.
- Refresh docs/CI/config to remove Zentra credentials and align with the Open-Meteo-based workflow; add Twilio verification status tests and behavior fix.
Reviewed changes
Copilot reviewed 27 out of 29 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Updates project description to Open-Meteo and adjusts project-team table. |
| manyfews/webapp/tests.py | Adds unit test ensuring Twilio verification only passes on "approved". |
| manyfews/webapp/load.py | Removes ZentraDevice dependency from test-data loader script. |
| manyfews/webapp/fixtures/initial_data.json | Removes Zentra periodic task + device fixture data. |
| manyfews/webapp/alerts.py | Fixes Twilio verification to return boolean “approved-only”. |
| manyfews/manyfews/settings.py | Removes insecure defaults for secrets; adds Open-Meteo settings. |
| manyfews/manyfews/.env.CI | Updates CI env vars for SECRET_KEY and Open-Meteo settings. |
| manyfews/calculations/zentra.py | Removes Zentra ingestion implementation. |
| manyfews/calculations/zentra_devices.py | Removes Zentra device discovery/import implementation. |
| manyfews/calculations/tests.py | Reworks tests to mock Open-Meteo and validate ensemble member behavior. |
| manyfews/calculations/tasks.py | Replaces Zentra/GEFS tasks with Open-Meteo historical + ensemble forecast flow. |
| manyfews/calculations/open_meteo.py | Adds Open-Meteo archive + ensemble ingestion and 6-hour bucketing. |
| manyfews/calculations/models.py | Removes Zentra models; adds NoaaForecast.issue_date and ensemble_member; adds AggregatedWeatherReading. |
| manyfews/calculations/migrations/0018_remove_zentra_add_open_meteo_fields.py | Introduces migration to drop Zentra tables and add Open-Meteo-related schema. |
| manyfews/calculations/generate_river_flows.py | Generalizes river-flow generation to work with “weather data” (forecast/historical) and supports ensemble-member selection. |
| manyfews/calculations/gefs.py | Removes GEFS download implementation. |
| manyfews/calculations/flood_risk.py | Dedupes scheduling by forecast_time and aggregates flows across ensemble outputs. |
| manyfews/calculations/fixtures/ZentraDevice.json | Removes Zentra device fixture. |
| manyfews/calculations/admin.py | Removes Zentra device admin registration. |
| docs/DEVELOPMENT.md | Removes Zentra account setup steps; clarifies Open-Meteo is keyless. |
| Dockerfile | Updates base images and replaces Zentra dummy vars with SECRET_KEY/DB_PASSWORD for build stages. |
| config/manyFEWS.base.yml | Updates dependencies: drops pygrib/xlrd/Zentra API; adds tenacity/requests; bumps Django/celery constraints. |
| .pre-commit-config.yaml | Bumps Black hook revision. |
| .github/workflows/run_unitTest_GenerateRiverFlows.yml | Updates actions versions; removes Zentra secrets; aligns with Open-Meteo settings. |
| .github/workflows/black.yml | Updates checkout action version. |
| .github/azure/docker-compose.backend.yml | Removes Zentra env block; adds Open-Meteo-related env vars. |
| .github/azure/azure-pipelines.yml | Removes Zentra env references; adds Open-Meteo env references. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
10
to
+14
| | Project Member | Contact address | Role | Unit | | ||
| |--------------------------------------------------|----------------------------------------------------------------------|----------------------------------|-------------------------------------------------------------------------------------| | ||
|
|
||
|
|
||
| | Prof. Simon Mathias | [simon.mathias@durham.ac.uk](mailto:simon.mathias@durham.ac.uk) | Project Lead (PI) | [Department of Engineering](https://www.durham.ac.uk/departments/academic/engineering/) | |
|
|
||
|
|
||
| | Prof. Simon Mathias | [simon.mathias@durham.ac.uk](mailto:simon.mathias@durham.ac.uk) | Project Lead (PI) | [Department of Engineering](https://www.durham.ac.uk/departments/academic/engineering/) | | ||
| | Prof. [Sim Reaney](https://github.com/simreaney) | [sim.reaney@durham.ac.uk](mailto:sim.reaney@durham.ac.uk) | Java Flood One Co-Investigator | [Department of Geography](https://www.durham.ac.uk/departments/academic/geography/) | |
Comment on lines
+19
to
+21
| dependencies = [ | ||
| ("calculations", "0003_riverchannel"), | ||
| ] |
Comment on lines
+49
to
+53
| forecast_times = ( | ||
| RiverFlowCalculationOutput.objects.filter( | ||
| prediction_date=latest_prediction_date, | ||
| forecast_time__lte=today + timedelta(days=16), | ||
| ) |
Comment on lines
+76
to
+80
| outputs = RiverFlowCalculationOutput.objects.filter( | ||
| prediction_date=prediction_date, forecast_time=forecast_time | ||
| ).first() | ||
| flow_values_iter = output.riverflowprediction_set.values_list( | ||
| "river_flow", flat=True | ||
| ) | ||
| flow_values_iter = RiverFlowPrediction.objects.filter( | ||
| calculation_output__in=outputs |
| logger.info(f"Loading parameters from {filename}") | ||
|
|
||
| total_rows = sum(1 for _ in open(filename)) | ||
| total_rows = sum(1 for _ in open(filename, encoding="utf-8-sig")) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
merge