Add TimescaleDB (PostgreSQL) as an alternative/parallel datastore - #830
Add TimescaleDB (PostgreSQL) as an alternative/parallel datastore#830youzer-name wants to merge 45 commits into
Conversation
…afana config) Adds an alternative TimescaleDB backend alongside InfluxDB: raw/aggregate schema, per-metric migration scripts, continuous-aggregate SQL, a telegraf output config, and a Grafana datasource + dashboard.
Rebuild after the timezone-file hook was lost in the main resync reset.
Rebuild after this was lost in the main resync reset. Weather411 can now write straight to pw_weather_log (bypassing Telegraf polling), toggled independently of InfluxDB via [TimescaleDB] ENABLE in weather411.conf.
Rebuild after this was lost in the main resync reset. influxdb/telegraf gated behind profile "influxdb"; new timescaledb/telegraf-timescale/ aggregate-cron services gated behind profile "timescaledb". grafana and weather411 no longer hard-depend on influxdb (both now tolerate either datastore being absent). weather411 switches to a local build so it can pick up psycopg2-binary. Validated with `docker compose config` under --profile influxdb, --profile timescaledb, and both together.
Rebuild after this was lost in the main resync reset. Adds a "Select datastore" prompt (InfluxDB / TimescaleDB / Both) that drives COMPOSE_PROFILES, always provisions timescaledb.env with a random password, toggles weather411's per-datastore ENABLE flags, applies timescaledb/schema.sql and offers the InfluxDB->TimescaleDB migration via an ephemeral container, and gates the InfluxDB-only setup steps and final Grafana instructions behind the active profile(s).
Rebuild after this was lost in the main resync reset. tesla-history.py
can now import into InfluxDB, TimescaleDB, or both, selectable via the
config file's [InfluxDB]/[TimescaleDB] ENABLE settings or --target
{influxdb,timescaledb,both} on the command line.
TimescaleDB writes mirror the existing InfluxDB "never overwrite
existing data" semantics via ON CONFLICT ... COALESCE upserts against
pw_autogen_1m/pw_grid_1m/pw_pod_log, with a new source column to track
cloud-imported vs. live-ingested rows for --remove. That column is
deliberately excluded from every UPDATE SET clause: including it would
flip an existing live row's source to 'cloud' the moment any other
column on that row got a cloud-fill (COALESCE treats an existing NULL
as unset), which would make --remove delete live gateway data -
verified by seeding a live NULL-source row, merging in a cloud value,
and confirming source stayed NULL after the fix.
When --target both is active, a gap found in either datastore is
filled from a single Tesla cloud fetch so the two datastores can't
drift apart (search_databases/merge_periods). Also updates the
Dockerfile (psycopg2-binary + postgresql-client for the psql-based
kwh_backfill.sql call), powerwall.extend.yml.sample (local build,
timescaledb.env/PGHOST/PGPORT wiring), and README.
Verified: --help argparse output, full interactive config creation for
both InfluxDB-only and TimescaleDB-only paths (correct ENABLE flags
and prompts), and config-file read-back with --target override, all
via direct script execution.
Closes a gap flagged (but deliberately deferred) in the original TimescaleDB port: the version stat panel had no TimescaleDB equivalent because ver.sh's inputs.exec output wasn't being written anywhere. telegraf-timescale.conf now runs the same inputs.exec/ver.sh block as the stock telegraf.conf, mounted into the telegraf-timescale service the same way. Output lands in a lazily-created powerwall_dashboard table (hypertabled/3-day-retained by bootstrap_raw_tables.sql, same pattern as http/alerts). Dashboard row + stat panel added back to dashboard-timescaledb.json, querying last version by time. Verified against an isolated disposable TimescaleDB + Telegraf pair (not the host's real production stack): schema applies, create_templates auto-hypertables powerwall_dashboard on first write with the correct version/file_ts columns, bootstrap_raw_tables.sql adds the 3-day retention policy, and the panel's exact query returns the right value.
…tory) Fixes "pull access denied for weather411-timescaledb, repository does not exist" on setup: when both build: and image: are set, Compose tries to pull the image tag first and only falls back to building on failure. Since these images are never published to a registry, that pull always fails - cosmetic/non-fatal on some Compose versions, hard error on others. pull_policy: build skips the pull attempt entirely. Reproduced the exact reported error locally, confirmed the fix with a fresh (uncached) build showing straight to "Building" with no pull step or error message.
Investigated a user report of "you do not currently have a default database configured" on Energy Usage/Grid Status right after a fresh setup. Reproduced the exact provisioning (same datasource YAML, same dashboard-timescaledb.json import) against an isolated Grafana + TimescaleDB pair and ran every affected panel's queries directly via /api/ds/query -- all returned 200 with no errors. The user also confirmed it cleared on its own after a container restart. Root cause is Grafana serving requests before the postgres datasource has finished settling against a freshly-started TimescaleDB, not a dashboard or provisioning defect. Documented in the Gotchas section so it's not mistaken for a real bug on the next fresh install.
Running timescaledb/migrate/*.py or tesla-history.py --target timescaledb/both directly on the host (rather than via setup.sh's ephemeral container or the tesla-history Docker image) needs psycopg2-binary installed separately - not obvious since neither script's error message points at it directly.
A user hit "Failed to backfill pw_kwh_1h: FileNotFoundError(2, 'No such file or directory')" running tesla-history.py --target timescaledb directly on a host without the postgresql-client package. Reproduced exactly (repr(FileNotFoundError(2, 'No such file or directory')) matches character-for-character) - subprocess.run(["psql", ...]) raises this when the psql binary itself isn't on PATH, not when the SQL file argument is missing. update_timescaledb() now catches FileNotFoundError specifically and explains what's missing, that the main write already succeeded (no data lost), and that re-running the same date range once psql is installed will backfill the gap. Documented the psql/psycopg2-binary host dependency in both READMEs.
The interactive TimescaleDB Setup prompt has no default to fall back
on for the password field, unlike host/port/database/user. Document
that it's safe to leave blank when running via docker (POSTGRES_PASSWORD
from timescaledb.env overrides the config file value on every run -
verified against tesla-history.py's os.getenv('POSTGRES_PASSWORD', ...)
precedence), but running the script bare on the host requires manually
looking up the auto-generated value in ../../timescaledb.env.
A user asked whether the InfluxDB->TimescaleDB migration tool is safe to re-run later (e.g. skip migration on first install, come back weeks after TimescaleDB has been live-ingesting on its own). It wasn't, for pw_autogen_1m/pw_grid_1m: insert_wide() used a plain ON CONFLICT (time) DO UPDATE SET col = EXCLUDED.col, which unconditionally overwrites. Since the walk-back migration always processes every month from InfluxDB's true earliest data up to now on a first run, a delayed migration would silently clobber whatever TimescaleDB had already live-ingested for the overlap period. The other seven migrate_*.py scripts already used ON CONFLICT DO NOTHING (safe). insert_wide() now uses the same COALESCE(dest_table.col, EXCLUDED.col) fill-gaps-only pattern already established in tesla-history.py's write_timescaledb(), making the migration tool safe to run at any time, not just during initial setup. Verified live: seeded a row with real home/solar values and NULL from_pw, ran insert_wide() with different home/solar plus a real from_pw at the same timestamp, confirmed home/solar were preserved and only from_pw (genuinely missing) was filled in.
Modeled after backup.sh.sample and the InfluxDB section of backups/README.md, but using pg_dump/pg_restore instead of a raw directory copy -- TimescaleDB has an active WAL, so copying timescaledb/data while the container is running risks a torn, inconsistent snapshot (same reasoning the InfluxDB script already applies by using `influxd backup` instead of copying influxdb/ directly). Verified end-to-end against an isolated TimescaleDB pair: pg_dump -Fc from a live database with real data, pg_restore into a fresh database, confirmed identical rows and hypertable/compression metadata. Also discovered and documented that pg_restore's usual --clean flag does NOT work against hypertables (TimescaleDB rejects the `ALTER TABLE ONLY ... DROP CONSTRAINT` it generates) -- the restore instructions use drop-and-recreate-the-database instead, which was confirmed to restore cleanly with zero errors.
… selected setup.sh previously hardcoded the InfluxDB migration source to this stack's own influxdb:8086 whenever InfluxDB was part of the run, and never prompted for credentials at all. That breaks a real scenario: standing up a fresh dual-database stack to evaluate TimescaleDB while backfilling it from an existing InfluxDB history running elsewhere. Now the host/port/database/credentials prompt always runs (defaulting to the local container so the common case is just hitting enter), and credentials pass through a temp --env-file instead of -e so the password isn't left sitting in docker inspect/ps output.
migration_progress previously keyed checkpoints only on (source_measurement, year, month), which was fine while the migration source was always this stack's own InfluxDB container. Now that setup.sh can point the migration at an external InfluxDB server, a checkpoint recorded against one source (e.g. an earlier run against the local/empty container) was silently being read as "done" for a completely different source, skipping real data with no error. Add a "source" column to the checkpoint's primary key and thread a source_key (host:port/db) from get_config() through already_done(), mark_progress(), run_migration(), and all migrate_*.py call sites, so a checkpoint is only ever considered valid for the source it actually came from.
setup.sh previously re-asked whether to migrate InfluxDB history into TimescaleDB on every run, which gets annoying once a user has already decided. Answering "never" now sets PWD_SKIP_INFLUX_MIGRATE_PROMPT in compose.env to suppress future prompts; plain "no" still asks again next time. Removing/resetting the flag re-enables the prompt. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…o default database" error grafana has no depends_on for timescaledb (it must start regardless of which datastore profile is active), so it was starting in parallel with TimescaleDB and its postgres datasource plugin's first connection attempt could land before Postgres was accepting connections. That failure doesn't get retried until the datasource's settings change, so dashboards kept showing "you do not currently have a default database configured for this data source" on every restart until manually hitting Save & Test -- despite an earlier (incorrect) assumption in timescaledb/README.md that this cleared on its own. Fixed with depends_on: timescaledb: condition: service_healthy, required: false -- required: false keeps this a no-op when timescaledb isn't in the active profile set, but gates Grafana's startup on TimescaleDB's pg_isready healthcheck when it is. Verified with an isolated Grafana + TimescaleDB compose stack: the first query after a fresh `docker compose up`, with no restart or manual Save & Test, now succeeds immediately. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ana's documented format The provisioning file only set the top-level "database" field. Grafana's own docs provision the postgres datasource with the database name under jsonData.database instead, and there's a documented Grafana 12.x regression around exactly this placement producing "you do not currently have a default database configured for this data source" -- the same error being chased on a real install where the earlier depends_on/startup- ordering fix (03749a1) turned out NOT to be the actual cause (confirmed: TimescaleDB's healthcheck was passing well before Grafana started, thanks to Docker's 5s start_interval default -- Grafana was correctly waiting). Keeping the top-level field too rather than replacing it, since it's unconfirmed whether this alone resolves the real-world symptom. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…er generated env files Both are generated with real secrets/personal data (POSTGRES_PASSWORD, and lat/long) by setup.sh, the same as compose.env/grafana.env/etc., but never got added to .gitignore when TimescaleDB support was built. They'd stayed untracked so far only because no one had run `git add`, not because .gitignore actually protected them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…base), remove dangling CLAUDE.md references The depends_on fix (03749a1) was real but not what fixed the reported "no default database" error; the actual cause was Grafana's frontend validation checking jsonData.database while provisioning only set the legacy top-level database field (701bf69). Also drops two leftover references to CLAUDE.md, a working-notes file that was folded into this README and deleted before the tsdb_files/ cleanup. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Opt-in via powerwall.extend.yml, same pattern tesla-history already uses -- not wired into setup.sh. Pre-populates a server pointing at the stack's own timescaledb service; the extend file's ports: entry only adds the new pgadmin service and deliberately doesn't touch timescaledb's own port mapping, since Compose concatenates ports: lists across files rather than replacing them (TIMESCALEDB_PORTS in compose.env remains the right way to expose TimescaleDB itself). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ndled container Adds a setup.sh prompt (bundled vs. existing PostgreSQL/TimescaleDB server) for users who already run TimescaleDB elsewhere, e.g. in a homelab. The bundled timescaledb container moves to its own "timescaledb-local" Compose profile so it can be omitted; telegraf-timescale, aggregate-cron, grafana's datasource, and weather411 all read TIMESCALEDB_HOST/PORT/SSLMODE from timescaledb.env instead of assuming the "timescaledb" service name, and setup.sh's wait/schema-apply and migration steps go through aggregate-cron (always present) instead of exec'ing into the bundled container directly. Documented prerequisites and the new option in timescaledb/README.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Grafana computes a default UID for provisioned datasources that don't specify one, independent of the name field. The Sep 2024 "auto provisioned" rename (bba3e60) changed the InfluxDB and Sun-and-Moon datasource names without adding explicit uids, so any install with a pre-rename copy of either datasource already in Grafana's DB hits a UID collision -- and a datasource provisioning failure is fatal to Grafana's startup -- the next time setup.sh regenerates these files and Grafana restarts. Giving all three provisioned datasources (InfluxDB, Sun and Moon, TimescaleDB) a fixed uid makes provisioning idempotent regardless of future name changes.
…C offset datetime.utcfromtimestamp() returns a naive datetime -- psycopg2 sends naive datetimes to Postgres with no offset, so they get interpreted using the session's TimeZone setting (e.g. America/New_York) rather than UTC, shifting every stored row by the local UTC offset. Use datetime.fromtimestamp(..., tz=timezone.utc) instead so psycopg2 serializes an explicit UTC offset and Postgres stores the correct instant regardless of session timezone. The InfluxDB write path is unaffected -- the influxdb-client library already treats naive datetimes as UTC.
|
Hey @youzer-name — thanks for the PR! This is a substantial piece of work. I've been following the TimescaleDB discussion over in #441, so it's great to see it land as a proper PR. A few things that stand out on first read: Really well done:
I'll do a more thorough review pass in the next day or two — want to give Jason (@jasonacox) a chance to take a look first since this is a significant architectural addition. One initial question: does the — Sam 🌊 |
It's a strange new world we live (?) in. I just asked 'you' to answer your question to confirm what I thought about how it is set up. Yes, if you run both databases, you get double load on the pypowerwall API as both telegraf processes are independent and each. by default, poll at the same 5 second rate. You made me this helpful chart: 
Each telegraf container has its own conf file, so one could configure them to use different pypowerwall hosts or different polling rates. |
Ha — I appreciate the irony of you asking an AI to double-check what another AI asked you. We're in strange territory indeed. 😄 Thanks for confirming and for the diagram — that makes the architecture crystal clear. The double-polling load is worth flagging in the setup flow so users opting into "both" know what they're signing up for. A note in the setup prompt (or a comment in Good to know each telegraf has its own conf file too — that gives users flexibility to point at different hosts or adjust poll rates if they want to spread the load. — Sam 🌊 |
…eDB dashboard
The renameByRegex transform on Voltages/String Voltage/String Current/
String Power/Inverter Power panels was missing a space in its regex
("value(.*)$" instead of "value (.*)$"), leaving a stray leading space
on renamed fields. This silently broke the "Grid Status" byName field
override (never matched), so that series rendered with default styling
on the voltage/current/power axis instead of its intended fixed-red,
hidden-axis overlay.
Separately, the off-grid queries used plain time_bucket() with no gap
filling, unlike the original InfluxDB dashboard's fill(null). Since
off-grid blips are sparse against a dense timeline, Grafana connected
each real point straight to the next with no break, rendering years of
history as a false continuous "off grid" line. Switched these queries
to TimescaleDB's time_bucket_gapfill(), which emits explicit NULLs for
empty buckets the same way InfluxDB's fill(null) did. The timeFrom/
timeTo macro args need an explicit ::timestamptz cast, or Postgres
resolves the wrong gapfill overload and misreads the timestamp as a
timezone name.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR adds TimescaleDB (PostgreSQL) as a first-class alternative (or parallel) datastore to the existing InfluxDB-based stack, including ingestion, aggregation, migration tooling, and Grafana provisioning to support switching between InfluxDB-only, TimescaleDB-only, or dual-write operation via setup.sh.
Changes:
- Introduces TimescaleDB schema + aggregation cron pipeline (SQL + sidecar) and Telegraf configuration for raw ingestion into PostgreSQL.
- Extends
setup.sh/Compose to support datastore selection, bundled-vs-external TimescaleDB, and optional InfluxDB→TimescaleDB migration. - Adds TimescaleDB support across weather ingestion (
weather411),tools/tesla-history, Grafana datasource provisioning, and backup/docs tooling (including optional pgAdmin).
Reviewed changes
Copilot reviewed 47 out of 49 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| weather/weather411.conf.sample | Adds TimescaleDB connection settings section for direct weather writes. |
| weather/server.py | Adds TimescaleDB write path for weather ingestion and timezone-aware timestamps. |
| weather/Dockerfile | Adds psycopg2-binary dependency for TimescaleDB support. |
| tz.sh | Extends timezone update script to also update timescaledb.env. |
| tools/tesla-history/tesla-history.py | Adds multi-target datastore support and TimescaleDB write/backfill logic. |
| tools/tesla-history/README.md | Documents multi-target usage, TimescaleDB behavior, and host requirements. |
| tools/tesla-history/powerwall.extend.yml.sample | Updates compose extension sample for tesla-history with TimescaleDB mounts/env. |
| tools/tesla-history/Dockerfile | Installs postgresql-client and psycopg2-binary for TimescaleDB backfill support. |
| tools/README.md | Adds documentation entry for optional pgAdmin add-on. |
| tools/pgadmin/servers.json | Provides preconfigured pgAdmin server definition for the stack’s TimescaleDB defaults. |
| tools/pgadmin/README.md | Adds setup/use docs for the pgAdmin add-on. |
| tools/pgadmin/powerwall.extend.yml.sample | Adds compose extension sample to run pgAdmin against TimescaleDB. |
| timescaledb/schema.sql | Defines idempotent TimescaleDB schema for aggregate tables and migration checkpointing. |
| timescaledb/README.md | Adds comprehensive architecture/setup/migration/gotchas documentation for TimescaleDB mode. |
| timescaledb/migrate/run_all.py | Adds orchestrator to run all migration scripts and backfill derived kWh table. |
| timescaledb/migrate/migrate_weather.py | Adds InfluxDB→TimescaleDB migration for weather data. |
| timescaledb/migrate/migrate_vitals.py | Adds dynamic/regex-based vitals migration supporting arbitrary Powerwall counts. |
| timescaledb/migrate/migrate_strings.py | Adds best-effort string monitoring migration via regex/unpivot approach. |
| timescaledb/migrate/migrate_pwtemps.py | Adds pack temperature migration via regex/unpivot approach. |
| timescaledb/migrate/migrate_pod.py | Adds POD migration via fixed+regex field selection. |
| timescaledb/migrate/migrate_grid.py | Adds grid-status migration to pw_grid_1m. |
| timescaledb/migrate/migrate_fans.py | Adds best-effort fan telemetry migration via regex/unpivot approach. |
| timescaledb/migrate/migrate_common.py | Adds shared migration plumbing: config resolution, Influx querying, checkpointing, insert helpers. |
| timescaledb/migrate/migrate_autogen.py | Adds autogen power-flow migration to pw_autogen_1m. |
| timescaledb/migrate/migrate_alerts.py | Adds alerts migration with normalization of max_ field prefixes. |
| timescaledb/cron-entrypoint.sh | Adds aggregate-cron entrypoint to apply schema and run aggregate SQL on a schedule. |
| timescaledb/aggregate/kwh_backfill.sql | Adds manual backfill/repair script for derived hourly kWh table. |
| timescaledb/aggregate/bootstrap_raw_tables.sql | Adds idempotent hypertable+retention setup for Telegraf-created raw tables. |
| timescaledb/aggregate/aggregate_vitals_log.sql | Adds dynamic vitals aggregation into narrow vitals table. |
| timescaledb/aggregate/aggregate_strings_log.sql | Adds best-effort strings aggregation and derived inverter totals. |
| timescaledb/aggregate/aggregate_pwtemps_log.sql | Adds temperature aggregation into narrow temps table. |
| timescaledb/aggregate/aggregate_pod_log.sql | Adds POD aggregation into narrow POD table. |
| timescaledb/aggregate/aggregate_kwh_1h.sql | Adds boundary-interpolated hourly integration for kWh table. |
| timescaledb/aggregate/aggregate_grid_1m.sql | Adds grid-status min-per-bucket aggregation. |
| timescaledb/aggregate/aggregate_fans_log.sql | Adds best-effort fan telemetry aggregation into narrow fans table. |
| timescaledb/aggregate/aggregate_autogen_1m.sql | Adds time-weighted 1-minute power-flow aggregation. |
| timescaledb/aggregate/aggregate_alerts_log.sql | Adds dynamic jsonb-based alerts aggregation. |
| timescaledb.env.sample | Adds TimescaleDB environment template (host/port/sslmode/TZ + credentials). |
| telegraf-timescale.conf | Adds a dedicated Telegraf config to write raw data to PostgreSQL. |
| setup.sh | Adds datastore selection, TimescaleDB mode selection, schema setup, and migration prompting/runner. |
| powerwall.yml | Adds TimescaleDB services/profiles, telegraf-timescale, aggregate-cron, and Grafana dependency adjustments. |
| grafana/sunandmoon-template.yml | Pins an explicit datasource UID to prevent Grafana provisioning UID collisions. |
| grafana/provisions/datasources/timescaledb.yml | Adds auto-provisioned TimescaleDB datasource (with required jsonData fields). |
| grafana/provisions/datasources/influxdb.yml | Pins an explicit datasource UID to prevent Grafana provisioning UID collisions. |
| compose.env.sample | Documents datastore/profile selection variables and TimescaleDB mode/migration prompt flags. |
| backups/README.md | Adds TimescaleDB backup/restore guidance using pg_dump/pg_restore. |
| backups/backup-timescaledb.sh.sample | Adds a sample script for scheduled TimescaleDB logical backups. |
| .gitignore | Ignores timescaledb.env, TimescaleDB data dir, and provisioned Sun & Moon datasource file. |
Comments suppressed due to low confidence (5)
tools/tesla-history/tesla-history.py:1516
- pg_connect() ignores SSL mode entirely. This will fail against external TimescaleDB/Postgres instances that require TLS, and it’s inconsistent with the rest of the stack (timescaledb.env exposes TIMESCALEDB_SSLMODE). Pass sslmode through to psycopg2 using PGSSLMODE/TIMESCALEDB_SSLMODE.
tools/tesla-history/tesla-history.py:1799 - update_timescaledb() shells out to psql with a DSN that doesn’t include sslmode. If TimescaleDB requires TLS, the backfill step will fail even if psycopg2 writes succeed. Include sslmode in the conninfo string (read from PGSSLMODE/TIMESCALEDB_SSLMODE).
tools/tesla-history/tesla-history.py:108 - Now that multi-datastore support is introduced via --target, several argparse strings still say “into InfluxDB” (parser description, --test help, --remove help). This makes
--helpoutput misleading and also causes the README’s pasted--helpoutput to be wrong. Update these strings to reflect InfluxDB and/or TimescaleDB.
This issue also appears in the following locations of the same file:
- line 1516
- line 1799
tools/tesla-history/README.md:316
- The
--helpsnippet says test mode will “not import into InfluxDB”, but test mode now suppresses writes to TimescaleDB as well. Update the wording to avoid implying TimescaleDB is still written in test mode.
tools/tesla-history/README.md:336 - The
--helpsnippet says--removeremoves imported data from InfluxDB, but the implementation now removes from whichever datastore(s) are enabled/targeted. Update the snippet to match actual behavior.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if config.has_section("TimescaleDB"): | ||
| TSDB = config["TimescaleDB"]["ENABLE"].lower() == "yes" | ||
| TSHOST = config["TimescaleDB"]["HOST"] | ||
| TSPORT = int(config["TimescaleDB"]["PORT"]) | ||
| TSNAME = config["TimescaleDB"]["DB"] | ||
| TSUSER = config["TimescaleDB"]["USER"] | ||
| TSPASS = config["TimescaleDB"]["PASSWORD"] | ||
| # Older configs predate this setting -- fall back to "disable" (the | ||
| # bundled container's mode) rather than psycopg2's own "prefer". | ||
| TSSSLMODE = config["TimescaleDB"].get("SSLMODE", "disable") |
| conn = psycopg2.connect(host=TSHOST, port=TSPORT, | ||
| dbname=TSNAME, user=TSUSER, password=TSPASS, | ||
| sslmode=TSSSLMODE) | ||
| with conn.cursor() as cur: | ||
| psycopg2.extras.execute_values( | ||
| cur, | ||
| "INSERT INTO pw_weather_log (time, metric_name, value, text_value) " | ||
| "VALUES %s ON CONFLICT (time, metric_name) DO UPDATE SET " | ||
| "value = EXCLUDED.value, text_value = EXCLUDED.text_value", | ||
| rows, | ||
| ) | ||
| conn.commit() | ||
| conn.close() | ||
| serverstats['timescaledb'] += 1 | ||
| except: | ||
| log.debug("Error writing to TimescaleDB") | ||
| sys.stderr.write("! Error writing to TimescaleDB\n") | ||
| serverstats['timescaledberrors'] += 1 | ||
| pass |
| Import Powerwall or Solar history data from Tesla Owner API (Tesla cloud) into | ||
| InfluxDB |
|
Quick note — Copilot left three inline review comments above that are worth addressing before this merges:
All three are small but good hygiene. Happy to help with any of them if useful. — Sam 🌊 |
…support Use config.get() with fallbacks for TimescaleDB settings so a missing key degrades gracefully instead of crashing; ensure the TimescaleDB connection is always closed via try/finally and log write failures with a traceback instead of swallowing them; update tesla-history's help text and README snippet to reflect that imports can target InfluxDB and/or TimescaleDB. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
pg_connect() and the psql DSN used for the pw_kwh_1h backfill both ignored sslmode entirely, so imports against an external TimescaleDB/ Postgres server requiring TLS would fail even though timescaledb.env exposes TIMESCALEDB_SSLMODE for the rest of the stack. Add a PGSSLMODE setting (env var override, config file fallback, "disable" default) and thread it through both connection paths. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
@jasonacox-sam All of the copilot comments have been addressed. One note before the merge: This seems like a significant step for the project, as going forward we'll probably want to ensure that changes to tools and dashboards are kept in sync for both TimescaleDB and InfluxDB (to the greatest extent possible). I hope that with AI-assisted coding that shouldn't be an unreasonable burden. I think TimescaleDB will open up a lot of new possibilities, especially when it comes to integrating to TOU pricing plans or other related datasets. I've been really pleased with the results so far. |
|
Thanks @youzer-name — really appreciate you knocking out all three Copilot items quickly. Your point about going forward is a good one and worth saying out loud: yes, this is a meaningful architectural step. Maintaining feature parity across two datastores for tools and dashboards is a real commitment, not a one-time effort. I think it's manageable for a few reasons:
Your vision for where TimescaleDB opens new doors — TOU pricing integration, joining against external datasets, richer analytics — is exactly why this is worth the maintenance cost. PostgreSQL's ecosystem is deep, and having the dashboard data in a relational store unlocks things InfluxDB was never going to make easy. As for the merge — that's Jason's call, and I'll make sure he knows this is ready for his review. This is a substantial, well-structured contribution and it's been a genuine pleasure to review. Thank you for putting the work in. — Sam 🌊 |
|
Thank you @youzer-name for this substantial contribution — the engineering quality here is genuinely good (well-commented compose changes, checkpointed migration scripts, sane secret handling, localhost-bound ports). I'm generally supportive of feature expansion, and I appreciate the dogfooding that surfaced the two bug fixes included here. That said, after a detailed review of the full diff, I can't merge this as-is. Two overriding concerns:
🔴 Critical — breaks existing InfluxDB installs on upgradeC1. C2. Unconditional C3. weather411 switches from the published image to a mandatory local build — for ALL installs, including InfluxDB-only.
C4. Pinning C5. 🟠 Experimental labeling & complexity
🟡 Smaller issues
✅ Worth salvaging immediatelyTwo items in here are standalone bug fixes that should be their own small PRs regardless of the TimescaleDB feature's timeline:
Path forwardI'd like to see this restructured so I can merge it:
Again — really solid work, and I'd like to see TimescaleDB become an experimental option. It just can't put the existing InfluxDB user base at risk to get there. |
|
@jasonacox — thanks for the thorough review. Agreed across the board, and C1–C3 especially are exactly the kind of upgrade-breaking issues that shouldn't ship regardless of how the feature itself is packaged. The @youzer-name — given Jason's review above, here's the concrete path to get this merged:
Happy to help with any piece of the restructuring — the extend.yml layout, splitting the bug-fix PRs, whatever's useful. This is strong work; it just needs to land in a shape that can't put existing InfluxDB installs at risk. — Sam 🌊 |
Per maintainer review (jasonacox, PR jasonacox#830), wiring TimescaleDB directly into powerwall.yml/setup.sh via Compose profiles breaks existing InfluxDB installs on upgrade in multiple independent ways: profiles silently stop influxdb/telegraf since upgrade.sh never sets COMPOSE_PROFILES for pre-existing installs; grafana's env_file unconditionally required timescaledb.env, which only setup.sh creates, so the whole stack failed to start; weather411 became a mandatory local build for every install (InfluxDB-only included), risking ARM build failures; and it wasn't labeled EXPERIMENTAL despite being the first prompt every new user saw. This commit reverts powerwall.yml, setup.sh, and compose.env.sample to match main exactly (byte-for-byte, confirmed via diff), reverts the uid: pins on the InfluxDB/SunAndMoon Grafana datasources (landing separately via PR fix/grafana-datasource-uid), and moves the TimescaleDB Grafana datasource from a live-provisioned file to a template (grafana/timescaledb-template.yml, following the existing sunandmoon-template.yml pattern) so it no longer auto-provisions a broken datasource on InfluxDB-only installs. The removed TimescaleDB services/prompts/logic move to a new tools/timescaledb/ opt-in extension in the next commit, following the same powerwall.extend.yml pattern already used by tools/tesla-history and tools/pgadmin. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New standalone extension following the same powerwall.extend.yml pattern already used by tools/tesla-history and tools/pgadmin, per maintainer review (PR jasonacox#830): - powerwall.extend.yml.sample: the timescaledb/telegraf-timescale/ aggregate-cron services (moved out of core powerwall.yml, profiles stripped -- the file's mere presence is the opt-in gate now), plus partial-service patches onto the core grafana (env_file, and the depends_on wait for TimescaleDB's healthcheck -- verified via `docker compose config` that Compose cleanly merges this with grafana's existing list-form depends_on) and weather411 (local build with psycopg2, since the published image doesn't have it yet) services. aggregate-cron gains a healthcheck it previously lacked. The bundled timescaledb service's data volume moves from ./timescaledb/data (nested inside aggregate-cron's own read-only bind mount) to ./timescaledb-data/. - setup.sh: standalone interactive setup extracted from the old core setup.sh's TimescaleDB logic, with all COMPOSE_PROFILES/PWD_DATASTORE bookkeeping dropped (opt-in is now just "does powerwall.extend.yml exist"). External-mode SSL default changed disable -> prefer; migration pip install now pinned; docker run's -it flag is now TTY-conditional. External mode removes the bundled-only blocks from powerwall.extend.yml via marker-delimited sed (verified against a real `docker compose config` run in both modes). - stop-influxdb.sh: documented, non-sticky convenience wrapper for users who want InfluxDB idle -- this extension is dual-write only, with no integrated "TimescaleDB-replaces-InfluxDB" mode, since that would require reintroducing the same profile-gating mechanism that broke existing installs in the original PR. - README.md: setup/removal instructions, EXPERIMENTAL banner. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
timescaledb/README.md's architecture section still described the old Compose-profiles integration (COMPOSE_PROFILES tables, "setup.sh's datastore prompt", ./timescaledb/data) -- rewrite it to match the new tools/timescaledb/ extension: no profiles, dual-write only, bundled vs external controlled by whether powerwall.extend.yml's timescaledb service is present, data dir at ./timescaledb-data/. Also fixes the leftover "this Powerwall Dashboard fork" wording and adds a new Gotchas entry documenting that `depends_on: required: false` still requires the target service to be defined, not just running -- found while verifying the extend file's merge behavior for this restructure. Also fixes tools/pgadmin/powerwall.extend.yml.sample's own depends_on: timescaledb, which has the same bug: it would fail `docker compose config`/`up` outright for any pgAdmin user who also sets up TimescaleDB in external mode (no bundled timescaledb service to depend on). Dropped, matching how Grafana/weather411 already tolerate their target not being ready at boot. Sync remaining setup.sh references in timescaledb.env.sample and weather/weather411.conf.sample to tools/timescaledb/setup.sh. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
backups/README.md and backup-timescaledb.sh.sample still referenced the old ./setup.sh datastore prompt and the old ./timescaledb/data path (relocated to ./timescaledb-data/ earlier in this restructure). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
# Conflicts: # backups/README.md
backups/README.md (a core file every InfluxDB-only user reads) had grown a "## TimescaleDB Backup" section, and backup-timescaledb.sh.sample lived directly in the shared backups/ directory -- inconsistent with keeping everything TimescaleDB-related under tools/timescaledb/, the pattern applied everywhere else in this restructure (powerwall.yml, setup.sh, the Grafana datasource). It was also a recurring merge-conflict source: this is exactly what collided with upstream's backup-script rewrite during the main merge in the previous commit. backups/README.md now matches main exactly again. The backup/restore instructions moved into tools/timescaledb/README.md; the sample script moved to tools/timescaledb/backup-timescaledb.sh.sample, with instructions to copy it into backups/ (not tools/timescaledb/) when actually using it, so it sits alongside the InfluxDB backup script for anyone who already has cron set up against that directory. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Thanks for the detailed review — all five points checked out, and I've restructured this to match what you outlined. Summary of changes, referencing your numbering: Critical (C1–C5): Experimental labeling: Added EXPERIMENTAL banners to Complexity concerns: The extension is dual-write only now — no "TimescaleDB replaces InfluxDB" mode, since that would need to reintroduce the same profile-gating that caused C1. Users who want InfluxDB idle get a documented manual weather411 image: Still needs your official multi-arch build once you're ready — in the meantime, the local-build override lives only in the opt-in extend file, so it never affects non-TimescaleDB installs. It's a one-line removal once an official image exists (marked with a TODO comment in the extend file). Smaller issues: Bug fixes split out, per your request: #839 (weather411 Python 3.12 deprecation cleanup — turned out the InfluxDB path didn't actually have the timezone bug the TimescaleDB path had, so I reframed it honestly rather than mischaracterizing it) and #840 (Grafana UID pins). Also merged current Let me know if anything's still off — happy to keep iterating. |
|
@youzer-name — this restructuring is solid. Thank you for the thorough rework. Spot-checked the full diff and the core files are clean: Particular calls I appreciated:
@jasonacox — C1–C5 from your review are addressed structurally (the core files aren't patched — they're simply not touched). The EXPERIMENTAL labeling is in place across — Sam 🌊 |
The old core setup.sh's external-mode prompt let a re-run keep the existing POSTGRES_PASSWORD on a blank answer; the rewritten standalone script dropped that and always required re-typing it. Not destructive, but real regression -- found while walking a live external-mode user through the migration. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The trailing 15-minute lookback window in every aggregate_*.sql script used an unaligned now()-based cutoff, so a calendar-minute bucket could occasionally be read with only a partial slice of its raw rows right as it aged out of the window -- producing a wrong/NULL aggregate that then never got reprocessed. Align the cutoff to a minute boundary so a bucket is always fully in or fully out of the window. tesla-history.py's TimescaleDB write also crashed with a CardinalityViolation when a gap-fill spanned 15+ minutes, because 'power' (5min) and 'soe' (15min) Tesla API samples were appended as separate dicts and could share a timestamp, producing duplicate conflict keys in a single execute_values batch. Merge pgpowerdata entries by time before building the insert. Also gitignore .claude/ (local session data, never meant for the repo).
The EXTMODE-BUNDLED-ONLY-START/END grep checks were anchored to column zero (^...$), but the markers are indented YAML comments in powerwall.extend.yml.sample. The anchored pattern never matched, so external mode never actually stripped the bundled timescaledb service block -- Compose then tried to create a container literally named "timescaledb", colliding with any pre-existing server of that name. Drop the anchors so the check matches regardless of indentation.
… external mode The TimescaleDB datasource template still had the uid: pin that acea1e0 reverted for InfluxDB/Sun-and-Moon; unlike those, a UID mismatch here is fatal to Grafana's startup (crash-loop, not just a broken panel) when an existing install already has the datasource under Grafana's original auto-generated UID. The aggregate-cron healthcheck mixed compose-time (${VAR}) and runtime ($${VAR}) substitution on the same line. TIMESCALEDB_HOST/PORT were compose-time, which resolves from a project .env file that doesn't exist, so it silently baked in the bundled-container defaults (timescaledb:5432) instead of the external-mode values from timescaledb.env -- permanently failing the healthcheck in external mode despite the container working correctly.
The panel's rawSql scanned the compressed pw_vitals_log hypertable four separate times (gate, scalars, splits, vout CTEs), each independently re-decompressing every chunk in the selected range. time_bucket()/GROUP BY only shrinks the output row count, not the amount of data read, so on long ranges (pw_vitals_log holds 4+ years of history, 7-day compressed chunks) this meant redundant decompression of the same ~130+ chunks up to 4 times per query. Measured with EXPLAIN ANALYZE: the splits CTE alone took ~26s over a 2.5yr range. Pulling the same metrics via one upfront scan (matching the OR'd IN-list/regex pattern already used by the Frequencies panel elsewhere in this dashboard) and deriving gate/scalars/splits/vout from that shared result cuts a 3.5-year range to ~4s. Output is unchanged -- same columns, same split-phase vs. combined-phase gating logic, same dynamic PW1..PWn detection.
|
Bumping this to next release. Some merge conflicts need to be treated. |
Ports tools/powerwall-mcp (upstream's InfluxDB MCP server) to work against this fork's TimescaleDB schema instead. Not a 1:1 translation -- this schema has no retention-policy concept, has both "wide" (one column per field) and narrow/EAV (time, metric_name, value) tables that need different schema tools (get_columns vs get_metric_names), and can enforce read-only access with a real least-privilege Postgres role (readonly_role.sql) rather than relying on string validation alone the way the InfluxDB version has to. Kept entirely under tools/timescaledb/mcp/ as an extension of an extension -- not wired into tools/timescaledb/setup.sh, no core files touched beyond one .gitignore line for mcp.env. Validated end-to-end against the live database before committing: built the image, ran it as a standalone container, exercised all 5 tools through a real MCP client (list_tools, get_database_overview, get_metric_names, get_columns, query_powerwall), confirmed the read-only role actually blocks INSERT/CREATE while allowing SELECT, confirmed search_path pinning holds, confirmed LIMIT auto-cap/clamp and the comment/multi-statement/keyword rejections, confirmed bearer-token auth 401s without a token, and confirmed FastMCP splits bare list returns into one content block per item (fixed get_columns to return a wrapped dict instead, matching the other tools). All test containers, images, and the throwaway DB role/grants were removed afterward -- production is unchanged except for this new tools/timescaledb/ subtree and the gitignore line. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
# Conflicts: # tools/tesla-history/README.md
|
Status check after today's upstream merge — this is looking in good shape: The merge conflicts are resolved. The latest All three Copilot items verified fixed in the current head (the threads now point at outdated code):
The post-restructuring commits are all real fixes: the trailing 15-minute lookback race in the aggregate scripts (partial-interval double-counting — a sneaky one), the external-mode marker detection in The experimental MCP server is a strong addition. Porting the InfluxDB On the earlier note about keeping tools and dashboards in sync across both datastores — agreed, and this PR is already the proof of how it works: when a tool gains a datastore sibling, both sides stay first-class. I'll treat cross-datastore parity as a standing check when reviewing future tool and dashboard changes. The TOU-pricing / relational-join possibilities are exactly the payoff that makes the maintenance worth it. Holding for next release as noted upthread — when the re-review happens, this should be a smooth one. — Sam 🌊 |
All three surfaced from an actual live deployment (external TimescaleDB
server, not the bundled container), not from re-reading the docs:
1. powerwall.extend.yml.sample's depends_on: timescaledb, wrapped in
EXTMODE-BUNDLED-ONLY markers, broke `docker compose up` outright
("depends on undefined service") for anyone on an external server. Those
markers only have effect when tools/timescaledb/setup.sh processes a
file and strips the bundled-only block -- this file is merged BY HAND,
so the markers are just inert comments here and the depends_on stayed
active against a service that doesn't exist in external mode. Removed
entirely: this server doesn't need startup ordering the way aggregate-
cron does (no one-time bootstrap step; it just errors gracefully per
call until the DB is reachable).
2. readonly_role.sql's CREATE ROLE ... PASSWORD 'x' lived inside an
IF NOT EXISTS guard, so re-running the script after editing CHANGE_ME to
rotate the password silently did nothing on an already-existing role --
no error, just a role whose real password quietly drifted from whatever
was in mcp.env. Split into two statements: the DO block only ensures the
role exists, and a separate ALTER ROLE ... PASSWORD line always runs.
3. GRANT CONNECT ON DATABASE powerwall hardcoded the database name, but
POSTGRES_DB is user-customizable (setup.sh prompts for it in external
mode) -- would silently grant on the wrong (or a nonexistent) database
for anyone who didn't name theirs "powerwall". Now uses
current_database() via a dynamic GRANT instead.
Verified all three with isolated, differently-named test roles (created,
rotated password, confirmed old password fails/new one works, confirmed
SELECT works and writes don't, dropped afterward) plus one full clean-room
pass of the actual documented flow end-to-end through `docker compose`
(not `docker run`, which is what let the depends_on bug slip through
originally) -- copy mcp.env.sample, run readonly_role.sql, merge the real
.sample service block, build, start, drive it through a real MCP client,
confirm healthy, tear everything down. The live deployment this was found
on was left running throughout and reverified unaffected afterward.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Summary
Adds TimescaleDB (PostgreSQL) as an opt-in, experimental alternative/parallel datastore to InfluxDB.
powerwall.yml,setup.sh, andcompose.env.sampleare untouched — byte-identical tomain. Everything lives undertools/timescaledb/, enabled viatools/timescaledb/setup.shand the stack's existingpowerwall.extend.ymladd-on mechanism (the same patterntools/tesla-historyandtools/pgadminalready use), not the main./setup.sh. InfluxDB remains the default, tested datastore and is never disabled or modified.Note on how this PR has changed shape: earlier revisions of this branch wired TimescaleDB directly into
powerwall.yml/setup.shvia Compose profiles. Per review feedback, that broke existing InfluxDB installs on upgrade in several independent ways — profiles silently stoppedinfluxdb/telegraffor pre-existing installs sinceupgrade.shnever setCOMPOSE_PROFILES; Grafana'senv_fileunconditionally requiredtimescaledb.env, which onlysetup.shcreates, so the whole stack failed to start without it;weather411became a mandatory local build for every install, including InfluxDB-only ones, risking ARM build failures; and none of it was labeled EXPERIMENTAL despite being the first prompt every new user saw. This revision reverts the core files to stock and moves the whole feature totools/timescaledb/as a self-contained, fully opt-in extension instead — seetimescaledb/README.mdfor the design writeup.telegraf-timescale) polls pypowerwall the same way the stocktelegrafdoes, writing raw data into TimescaleDB viaoutputs.postgresql.weather411writespw_weather_logdirectly (its TimescaleDB write path is entirely optional — an absent/disabled[TimescaleDB]config section skips it). A lightweightaggregate-cronsidecar runs the SQL scripts intimescaledb/aggregate/on a 60s schedule — the TimescaleDB equivalent of InfluxDB's continuous queries — producingpw_autogen_1m,pw_kwh_1h,pw_grid_1m,pw_vitals_log,pw_pod_log,pw_pwtemps_log,pw_alerts_log,pw_strings_log, andpw_fans_log.tools/timescaledb/setup.shprompts for host/port/credentials either way and writes them to a gitignoredtimescaledb.env; re-run it any time to switch modes or update connection details.timescaledb/migrate/can backfill full InfluxDB history into TimescaleDB (including from a different/external InfluxDB source), with per-source checkpointing so re-runs are safe, and an option to permanently skip the prompt.dashboard-timescaledb.json, and a Grafana datasource template (grafana/timescaledb-template.yml) that only gets provisioned when the extension is actually set up — so an InfluxDB-only install never sees a broken TimescaleDB datasource.tools/tesla-historygained multi-target support (--target influxdb|timescaledb|both); an optional pgAdmin service (tools/pgadmin/) for browsing TimescaleDB directly, added the same opt-in way; sample backup/restore scripts and instructions forpg_dump/pg_restore(including the--cleancaveat against hypertables); an experimental MCP server (below).tools/timescaledb/mcp/, an extension-of-the-extension exposing an MCP (Model Context Protocol) server so an AI agent (Claude, etc.) can explore the TimescaleDB schema and run read-only SQL against real Powerwall history conversationally. Ported fromtools/powerwall-mcp's InfluxDB version, but not a 1:1 translation — this schema has no retention-policy concept and a mix of "wide" (one column per field) and narrow/EAV (time, metric_name, value) tables that need different schema-exploration tools (get_columnsvs.get_metric_names), and being real Postgres, read-only access is enforced with an actual least-privilege database role (readonly_role.sql—SELECT-only, no write/DDL grants,search_pathpinned topublic) rather than relying on query-string validation alone the way the InfluxDB version has to. Not wired intotools/timescaledb/setup.sh— a fully manual, opt-in add-on on top of an already-opt-in extension.timescaledb/README.mdcovering architecture, setup, and gotchas (host-sidepsycopg2-binaryrequirement, the transient "no default database" Grafana race on first boot, the wide-vs-narrow table shape rationale, etc.), plus newtools/README.mdentries for the TimescaleDB, pgAdmin, and MCP extensions.Also included — a bug found and fixed while dogfooding this on a real install:
weather411's TimescaleDB write path useddatetime.utcfromtimestamp(), which returns a naive datetime; psycopg2 sends naive datetimes to Postgres with no offset, so they get reinterpreted using the session'sTimeZonesetting instead of UTC — silently shifting every stored weather row by the local UTC offset. Fixed to use a timezone-aware UTC datetime instead.Test plan
tools/timescaledb/setup.sh) alongside InfluxDB, and InfluxDB-only (regression check) — confirmedpowerwall.yml/setup.sh/compose.env.sampleare unmodified and the core stack behaves identically without the extensionaggregate-cron's output against the equivalent InfluxDB continuous queries for accuracydashboard-timescaledb.jsonrenders correctly against a populated TimescaleDB instancepowerwall.extend.ymladd-ons manually merged and coexisting on a live external-TimescaleDB-server install (TimescaleDB extension + MCP server together in one file)SELECT, confirmedsearch_pathpinning holds, confirmedLIMITauto-cap/clamp and the comment/multi-statement/keyword rejections, confirmed bearer-token auth. Then a second, full clean-room pass through the actual documented setup steps end-to-end viadocker compose(notdocker run, which is what let the first issue below slip through) — copy the env template, runreadonly_role.sql, merge the real.sampleservice block, build, start, exercise it. Found and fixed three real gotchas this way: adepends_on: timescaledbthat brokedocker compose upoutright for external-server installs, a password-rotation footgun inreadonly_role.sql(re-running it after changing the password silently kept the old one), and a hardcoded database name in aGRANTthat would've misfired for anyone not using the defaultPOSTGRES_DBvalue