Skip to content

Commit 83ab56c

Browse files
atulmguptaCopilot
andcommitted
fix(mqtt): preserve telemetry event time
Ship a pinned Fleet Telemetry producer that publishes Tesla source timestamps, quarantine timestamp-less payloads, and propagate event time through live state, FSMs, and drive/charging sessions. Continuously reconcile the durable MQTT subscription with generation-safe SUBACK leases so silent subscription loss self-heals without deleting the offline queue. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5fe6d-5e00-4d43-a379-96a3491480e3
1 parent a1249d9 commit 83ab56c

54 files changed

Lines changed: 1472 additions & 175 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.gitattributes

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@ cmd/protogen-tesla/testdata/golden/*_gen.go text eol=lf
4444
*.sh text eol=lf
4545
*.bash text eol=lf
4646
*.py text eol=lf
47+
*.patch text eol=lf -whitespace
4748
*.css text eol=lf
4849
*.scss text eol=lf
4950
*.html text eol=lf
@@ -86,4 +87,3 @@ Dockerfile* text eol=lf
8687
# marked above (top of file) for clarity; the rule below covers the
8788
# generic case for any other proto we may vendor.
8889
*.proto binary
89-

.github/ARCHITECTURE.md

Lines changed: 13 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -662,7 +662,7 @@ Tesla Vehicle ── mTLS stream ──▶ Tesla Fleet Telemetry ──▶│ Mo
662662

663663
**Five-line summary:**
664664

665-
1. **Vehicle → Mosquitto:** Tesla streams Fleet Telemetry over mTLS; with `transmit_decoded_records: true` (helm-pinned) the upstream emits ONE signal per topic of the form `telemetry/{VIN}/v/{Field}` with the bare JSON value as body.
665+
1. **Vehicle → Mosquitto:** Tesla streams Fleet Telemetry over mTLS; TeslaSync's pinned Fleet Telemetry build emits ONE signal per topic of the form `telemetry/{VIN}/v/{Field}` with `{"value":...,"ts":"<Payload.CreatedAt>"}` as the body.
666666
2. **Decode → Normalize:** `PipelineSubscriber` filters `{base}/+/v/+`, codec `DecodeJSONField` translates the per-field body to a `[]codec.Atomic` keyed by canonical proto field name, and `normalize.Pipeline.ProcessAtomics` converts each value to SI using per-vehicle `Setting*Unit` history.
667667
3. **Route → Persist:** `routing.yaml` (static, no per-vehicle logic) routes each field to a destination table writer + optional `signal_log` history. **Codec failures route to the DLQ via `ErrPayloadDrop`; writer failures only log+counter** (never redeliver — a poisoned per-VIN topic would otherwise pin redelivery forever).
668668
4. **Live state, three tiers:** L1 = in-process `signal.Store` (FSM, sessions hot path) · L2 = Redis `vehicle:{id}:signals` HSET + Pub/Sub (cross-pod, restart recovery) · durable = `signal_log` hypertable (charts, replay, point-in-time snapshots).
@@ -998,12 +998,13 @@ amendment.
998998
period.
999999
2. **Codec entry point.** `internal/tesla/codec.DecodeJSONField(field,
10001000
body, vin, fallbackTs)` is the SINGLE per-field MQTT translation
1001-
point. Body shape is per `protomodel.SignalsByName[field].ValueKind`:
1002-
bare JSON value for atomic kinds, JSON object for compound kinds, the
1003-
proto-prefixed string form (`"ShiftStateD"`) for typed enums (codec
1004-
strips `EnumStringPrefix`). An optional envelope `{"value":<bare>,
1005-
"ts":"<RFC3339>"}` carries replay event-time; production traffic uses
1006-
bare bodies.
1001+
point. The production body is an envelope `{"value":<bare>,
1002+
"ts":"<RFC3339Nano>"}` where `ts` is Tesla's original
1003+
`Payload.CreatedAt`; the inner value shape follows
1004+
`protomodel.SignalsByName[field].ValueKind`. Bare values remain
1005+
decoder-compatible for controlled tests and emergency tooling, but the
1006+
production MQTT boundary rejects and quarantines them because replay
1007+
receipt time is not valid event time.
10071008
3. **Pipeline interface narrowed.** `mqtt.Pipeline` exposes a single
10081009
method `ProcessAtomics(ctx, []codec.Atomic, vehicleID)`
10091010
`Process(ctx, []byte, vehicleID)` is removed. This re-affirms ADR-004
@@ -1052,6 +1053,11 @@ amendment.
10521053
config.json is REQUIRED. A future fleet-telemetry release that
10531054
removes this knob would break the wire shape; the helm template
10541055
pins the value explicitly.
1056+
- The TeslaSync Fleet Telemetry derivative image is REQUIRED until upstream
1057+
preserves `Payload.CreatedAt` in its per-field MQTT output. Deploy the
1058+
timestamp-producing image before or with the strict API consumer. Legacy
1059+
queued bare messages are intentionally quarantined because their original
1060+
event time cannot be reconstructed from MQTT 3.1.1.
10551061
- DLQ depth is now a per-field, per-VIN concern — a single vehicle's
10561062
malformed `Soc` body no longer delays neighbour vehicles. This is a
10571063
net improvement, but operators must sweep the DLQ topic regularly to

.github/instructions/helm-docker.instructions.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,9 @@ FLEET_TELEMETRY_HOST=
8585

8686
Enable with: `docker compose --profile telemetry up -d`
8787

88-
- Uses official `tesla/fleet-telemetry:latest` image
88+
- Uses TeslaSync's pinned `ghcr.io/ev-dev-labs/teslasync-fleet-telemetry`
89+
image, which preserves the upstream `Payload.CreatedAt` timestamp in every
90+
MQTT field envelope
8991
- Config mounted from `fleet-telemetry-config.json`
9092
- TLS cert/key mounted from paths in `.env`
9193
- Dispatches vehicle data to `http://teslasync:8080/api/v1/telemetry`

.github/workflows/ci.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -305,6 +305,8 @@ jobs:
305305
file: Dockerfile.notification
306306
- image: export-worker
307307
file: Dockerfile.export-worker
308+
- image: fleet-telemetry
309+
file: Dockerfile.fleet-telemetry
308310
- image: web
309311
file: Dockerfile.web
310312
steps:

.github/workflows/release.yml

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -163,6 +163,8 @@ jobs:
163163
file: Dockerfile.automation
164164
build-args: |
165165
VERSION=${{ needs.version.outputs.new_tag }}
166+
- image: fleet-telemetry
167+
file: Dockerfile.fleet-telemetry
166168
steps:
167169
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
168170
- uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3.12.0
@@ -402,7 +404,7 @@ jobs:
402404
fail-fast: false
403405
matrix:
404406
# No `file` / `build-args` here: this stage does NOT build.
405-
image: [api, web, notification-worker, export-worker, automation-worker]
407+
image: [api, web, notification-worker, export-worker, automation-worker, fleet-telemetry]
406408
steps:
407409
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
408410

@@ -717,6 +719,9 @@ jobs:
717719
718720
# Automation Worker
719721
docker pull ghcr.io/${{ github.repository }}-automation-worker:${VERSION}
722+
723+
# Fleet Telemetry (event-time preserving)
724+
docker pull ghcr.io/${{ github.repository }}-fleet-telemetry:${VERSION}
720725
\`\`\`
721726
722727
### ⎈ Helm Chart
@@ -772,6 +777,7 @@ jobs:
772777
| Notification Worker | \`ghcr.io/${{ github.repository }}-notification-worker:${VERSION}\` |
773778
| Export Worker | \`ghcr.io/${{ github.repository }}-export-worker:${VERSION}\` |
774779
| Automation Worker | \`ghcr.io/${{ github.repository }}-automation-worker:${VERSION}\` |
780+
| Fleet Telemetry | \`ghcr.io/${{ github.repository }}-fleet-telemetry:${VERSION}\` |
775781
| Helm Chart | \`oci://ghcr.io/${{ github.repository_owner }}/charts/teslasync:${VERSION}\` |
776782
NOTES_EOF
777783

Dockerfile.fleet-telemetry

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# Build Tesla Fleet Telemetry from a pinned upstream revision, applying the
2+
# TeslaSync MQTT event-time envelope patch before compilation.
3+
FROM golang:1.26-bookworm AS build
4+
5+
ARG FLEET_TELEMETRY_REF=8fbaa100bd365936dab6ecbf0e2d7070c4d765cb
6+
ARG FLEET_TELEMETRY_ARCHIVE_SHA256=8a0e2cc31a3c962b4e6296687e75b5fb208fbbe53ccf928af220ec2bf7fdb355
7+
8+
WORKDIR /build
9+
RUN wget -qO fleet-telemetry.tar.gz \
10+
"https://github.com/teslamotors/fleet-telemetry/archive/${FLEET_TELEMETRY_REF}.tar.gz" \
11+
&& echo "${FLEET_TELEMETRY_ARCHIVE_SHA256} fleet-telemetry.tar.gz" | sha256sum -c - \
12+
&& mkdir -p /go/src/fleet-telemetry \
13+
&& tar -xzf fleet-telemetry.tar.gz --strip-components=1 -C /go/src/fleet-telemetry \
14+
&& rm fleet-telemetry.tar.gz
15+
16+
RUN wget -q https://github.com/jedisct1/libsodium/releases/download/1.0.19-RELEASE/libsodium-1.0.19.tar.gz \
17+
&& echo "018d79fe0a045cca07331d37bd0cb57b2e838c51bc48fd837a1472e50068bbea libsodium-1.0.19.tar.gz" | sha256sum -c - \
18+
&& tar -xzf libsodium-1.0.19.tar.gz
19+
WORKDIR /build/libsodium-stable
20+
RUN ./configure --disable-shared --enable-static \
21+
&& make -j"$(nproc)" \
22+
&& make install
23+
24+
WORKDIR /build
25+
RUN wget -q https://github.com/zeromq/libzmq/releases/download/v4.3.4/zeromq-4.3.4.tar.gz \
26+
&& echo "c593001a89f5a85dd2ddf564805deb860e02471171b3f204944857336295c3e5 zeromq-4.3.4.tar.gz" | sha256sum -c - \
27+
&& tar -xzf zeromq-4.3.4.tar.gz
28+
WORKDIR /build/zeromq-4.3.4
29+
RUN ./configure --enable-static --disable-shared --disable-Werror \
30+
&& make -j"$(nproc)" \
31+
&& make install
32+
33+
RUN apt-get update \
34+
&& apt-get install -y --no-install-recommends patch \
35+
&& rm -rf /var/lib/apt/lists/*
36+
37+
WORKDIR /go/src/fleet-telemetry
38+
COPY build/fleet-telemetry/mqtt-event-time.patch /tmp/mqtt-event-time.patch
39+
COPY build/fleet-telemetry/event_time_patch_test.go datastore/mqtt/event_time_patch_test.go
40+
RUN sed -i 's/\r$//' /tmp/mqtt-event-time.patch \
41+
&& patch -p1 < /tmp/mqtt-event-time.patch
42+
43+
ENV CGO_ENABLED=1
44+
ENV CGO_LDFLAGS="-lstdc++"
45+
RUN go test -tags fleettelemetry_patch ./datastore/mqtt -count=1 \
46+
&& make build
47+
48+
FROM gcr.io/distroless/cc-debian12:nonroot
49+
50+
LABEL org.opencontainers.image.source="https://github.com/ev-dev-labs/teslasync" \
51+
org.opencontainers.image.description="Tesla Fleet Telemetry with TeslaSync replay-safe MQTT event timestamps" \
52+
org.opencontainers.image.licenses="Apache-2.0"
53+
54+
COPY --from=build /go/bin/fleet-telemetry /fleet-telemetry
55+
COPY --from=build /go/src/fleet-telemetry/LICENSE /licenses/fleet-telemetry/LICENSE
56+
57+
USER nonroot:nonroot
58+
59+
ENTRYPOINT ["/fleet-telemetry"]
60+
CMD ["-config", "/etc/fleet-telemetry/config.json"]
Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
//go:build fleettelemetry_patch
2+
3+
package mqtt
4+
5+
import (
6+
"encoding/json"
7+
"testing"
8+
"time"
9+
10+
"github.com/teslamotors/fleet-telemetry/protos"
11+
"google.golang.org/protobuf/types/known/timestamppb"
12+
)
13+
14+
func TestTeslaSyncMQTTEventTimeEnvelope(t *testing.T) {
15+
eventTime := time.Date(2026, time.August, 20, 7, 8, 9, 123456789, time.UTC)
16+
payload := &protos.Payload{CreatedAt: timestamppb.New(eventTime)}
17+
18+
body, err := marshalVehicleField(float32(72.5), payload)
19+
if err != nil {
20+
t.Fatalf("marshalVehicleField: %v", err)
21+
}
22+
23+
var envelope struct {
24+
Value float64 `json:"value"`
25+
TS string `json:"ts"`
26+
}
27+
if err := json.Unmarshal(body, &envelope); err != nil {
28+
t.Fatalf("json.Unmarshal: %v", err)
29+
}
30+
if envelope.Value != 72.5 {
31+
t.Fatalf("value = %v, want 72.5", envelope.Value)
32+
}
33+
if envelope.TS != eventTime.Format(time.RFC3339Nano) {
34+
t.Fatalf("ts = %q, want %q", envelope.TS, eventTime.Format(time.RFC3339Nano))
35+
}
36+
37+
body, err = marshalVehicleField(nil, &protos.Payload{})
38+
if err != nil {
39+
t.Fatalf("marshalVehicleField without timestamp: %v", err)
40+
}
41+
if string(body) != `{"value":null}` {
42+
t.Fatalf("body without timestamp = %s, want value-only envelope", body)
43+
}
44+
}
Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
diff --git a/datastore/mqtt/mqtt_payload.go b/datastore/mqtt/mqtt_payload.go
2+
index 4c16039..38ac17e 100644
3+
--- a/datastore/mqtt/mqtt_payload.go
4+
+++ b/datastore/mqtt/mqtt_payload.go
5+
@@ -11,12 +11,35 @@ import (
6+
"google.golang.org/protobuf/reflect/protoreflect"
7+
)
8+
9+
+type vehicleFieldEnvelope struct {
10+
+ Value interface{} `json:"value"`
11+
+ TS string `json:"ts,omitempty"`
12+
+}
13+
+
14+
+// marshalVehicleField preserves Payload.CreatedAt alongside each MQTT value.
15+
+// TeslaSync uses this source timestamp for ordering and keeps broker receipt
16+
+// time separately, so persistent-session replay cannot move old telemetry into
17+
+// the current drive or charging session.
18+
+//
19+
+// TeslaSync modification: the upstream producer publishes only the bare value
20+
+// and therefore discards CreatedAt at this transport boundary.
21+
+func marshalVehicleField(value interface{}, payload *protos.Payload) ([]byte, error) {
22+
+ envelope := vehicleFieldEnvelope{Value: value}
23+
+ if payload != nil {
24+
+ createdAt := payload.GetCreatedAt()
25+
+ if createdAt != nil && createdAt.CheckValid() == nil {
26+
+ envelope.TS = createdAt.AsTime().UTC().Format(time.RFC3339Nano)
27+
+ }
28+
+ }
29+
+ return json.Marshal(envelope)
30+
+}
31+
+
32+
func (p *Producer) processVehicleFields(rec *telemetry.Record, payload *protos.Payload) ([]pahomqtt.Token, error) {
33+
var tokens []pahomqtt.Token
34+
convertedPayload := p.payloadToMap(payload)
35+
for key, value := range convertedPayload {
36+
mqttTopicName := fmt.Sprintf("%s/%s/v/%s", p.config.TopicBase, rec.Vin, key)
37+
- jsonValue, err := json.Marshal(value)
38+
+ jsonValue, err := marshalVehicleField(value, payload)
39+
if err != nil {
40+
return tokens, fmt.Errorf("failed to marshal JSON for MQTT topic %s: %v", mqttTopicName, err)
41+
}
42+
diff --git a/datastore/mqtt/mqtt_test.go b/datastore/mqtt/mqtt_test.go
43+
index a935516..d03f7d8 100644
44+
--- a/datastore/mqtt/mqtt_test.go
45+
+++ b/datastore/mqtt/mqtt_test.go
46+
@@ -219,7 +219,7 @@ var _ = Describe("MQTTProducer", func() {
47+
},
48+
},
49+
},
50+
- CreatedAt: timestamppb.Now(),
51+
+ CreatedAt: timestamppb.New(time.Date(2026, time.August, 20, 7, 8, 9, 123456789, time.UTC)),
52+
}
53+
54+
payloadBytes, err := proto.Marshal(payload)
55+
@@ -248,10 +248,10 @@ var _ = Describe("MQTTProducer", func() {
56+
locationTopic := "test/topic/TEST123/v/Location"
57+
batteryLevelTopic := "test/topic/TEST123/v/BatteryLevel"
58+
59+
- vehicleNameValue := "\"My Tesla\""
60+
- invalidValue := "null"
61+
- locationValue := "{\"latitude\":37.7749,\"longitude\":-122.4194}"
62+
- batterLevelValue := "75.5"
63+
+ vehicleNameValue := `{"value":"My Tesla","ts":"2026-08-20T07:08:09.123456789Z"}`
64+
+ invalidValue := `{"value":null,"ts":"2026-08-20T07:08:09.123456789Z"}`
65+
+ locationValue := `{"value":{"latitude":37.7749,"longitude":-122.4194},"ts":"2026-08-20T07:08:09.123456789Z"}`
66+
+ batterLevelValue := `{"value":75.5,"ts":"2026-08-20T07:08:09.123456789Z"}`
67+
68+
Expect(publishedTopics).To(HaveKey(vehicleNameTopic))
69+
Expect(publishedTopics).To(HaveKey(invalidTopic))

docker-compose.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -649,7 +649,7 @@ services:
649649
# Requires TLS certificates and Tesla Developer account with Fleet Telemetry access.
650650
# See: https://github.com/teslamotors/fleet-telemetry
651651
fleet-telemetry:
652-
image: tesla/fleet-telemetry:latest
652+
image: ghcr.io/ev-dev-labs/teslasync-fleet-telemetry:${TESLASYNC_VERSION:-latest}
653653
container_name: teslasync-fleet-telemetry
654654
restart: unless-stopped
655655
profiles:

docs/guide/fleet-telemetry.md

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ sequenceDiagram
4444
participant DB as TimescaleDB
4545
participant UI as Browser
4646
47-
FT->>MQTT: Publish signal batch
47+
FT->>MQTT: Publish {value, ts} signal envelope
4848
MQTT->>API: Deliver message
4949
API->>API: Decode + normalise to SI
5050
API->>L1: Write-through (FSM, typed rules, sessions)
@@ -61,6 +61,7 @@ sequenceDiagram
6161
| Public HTTPS/WSS endpoint | Vehicles require a publicly trusted TLS certificate |
6262
| Tesla public-key URL | Serve `/.well-known/appspecific/com.tesla.3p.public-key.pem` unauthenticated |
6363
| MQTT broker | Compose and Helm include Mosquitto by default |
64+
| Event-time producer image | Use TeslaSync's Fleet Telemetry image; Tesla's stock MQTT output drops `CreatedAt` |
6465
| API config | Set `FLEET_TELEMETRY_*` envs (see [Configuration](/guide/configuration)) |
6566

6667
## Docker Compose
@@ -73,6 +74,31 @@ docker compose --profile telemetry up -d --build
7374

7475
Configure the public host, TLS certificates, topic base, and Tesla Developer settings in `.env` and `fleet-telemetry/config.json` before enabling it.
7576

77+
TeslaSync ships a pinned Fleet Telemetry build that keeps the upstream
78+
`Payload.CreatedAt` value in every per-field MQTT payload. Do not replace it
79+
with `tesla/fleet-telemetry` unless that upstream image gains the same
80+
`{"value": ..., "ts": ...}` contract; the stock image currently publishes
81+
bare values and queued messages would otherwise be timestamped when replayed.
82+
83+
For the first event-time cutover, deploy the TeslaSync Fleet Telemetry image
84+
**before** the strict API consumer. Do not rely on a simultaneous Helm upgrade:
85+
Kubernetes does not guarantee which Deployment becomes ready first.
86+
87+
1. Update only the Fleet Telemetry Deployment to
88+
`ghcr.io/ev-dev-labs/teslasync-fleet-telemetry:<version>`.
89+
2. Wait for its rollout and inspect a canary MQTT message; its body must contain
90+
both `value` and an RFC3339Nano `ts`.
91+
3. Upgrade the Helm release so the API begins enforcing source timestamps, then
92+
confirm `teslasync_mqtt_telemetry_event_time_total{outcome="source"}`
93+
increases while the `rejected_missing` and `rejected_invalid` outcomes stay
94+
flat.
95+
96+
After the API upgrade, valid signals without `ts` are sent to the MQTT DLQ and
97+
acknowledged rather than written with receipt time. Any bare messages already
98+
queued by the stock producer are intentionally quarantined because MQTT 3.1.1
99+
contains no timestamp from which their original event time can be recovered.
100+
External Fleet Telemetry producers must emit the same envelope contract.
101+
76102
## Kubernetes
77103

78104
Use Helm values for Fleet Telemetry and the ingress/TLS. The web route must allow `/.well-known` without app auth so Tesla can fetch the public key.

0 commit comments

Comments
 (0)