Skip to content

troubleshooting-reference.md

Latest commit

 

History

History
1399 lines (1085 loc) · 46 KB

troubleshooting-reference.md

File metadata and controls

1399 lines (1085 loc) · 46 KB
Raw

Troubleshooting Reference

This is the technical troubleshooting reference for advanced users and support. It preserves the command-level diagnostics, log-event details and failure analysis that are intentionally kept out of the short user guide.

For normal user workflows, see Troubleshooting.

Related references: control-logic.md, control-flow.md, configuration.md, runtime-state.md, safety-model.md, cli.md, influxdb.md.

Docker and container checks

For the recommended Docker Bootstrap setup, run these checks first:

docker compose ps
docker compose logs -f
docker compose exec ems python3 emsctl.py diagnose

Use hardware diagnostics only when you are ready to probe the configured local meter and Zendure devices. It is read-only.

docker compose exec ems python3 emsctl.py diagnose --hardware

Common first fixes:

  • Dashboard not reachable: verify docker compose ps, host port 8080, and container logs.
  • Config still contains placeholders: edit config/config.json, replace example values and YOUR_SN, restart, then run diagnose again.
  • Docker socket permission denied: run Docker with sudo or complete the optional post-install steps in install-docker.md.
  • Grid meter not reachable: check meter IP, meter type, local network routing, and diagnose --hardware.
  • Zendure device not reachable: check device IP, serial number, local network routing, Zendure Local API availability, and diagnose --hardware.
  • Analytics/InfluxDB not reachable: the dashboard still works without InfluxDB; see influxdb.md.
  • Home Assistant not configured: Home Assistant is optional and not required for standalone EMS control.

At least one supported Zendure connection — Local API, Local MQTT, or Zendure cloud MQTT — must be available for EMS control. Do not run Zendure HEMS, Home Assistant automations, MQTT writers, or any other controller in parallel if they write Zendure outputLimit. EMS assumes exclusive write control over outputLimit while active.

The EMS uses structured logs:

event=<name> key=value key=value

Use these logs to validate behavior during dry-run checks and live operation. Change one setting at a time and run a short dry-run or bounded live test after each change.

The template profile is intended for normal standalone live control after real local values are configured and installation limits are reviewed. If required placeholders are still present, EMS forces safe mode: control disabled, dry-run enabled, and hardware writes blocked. Use --dry-run or set system.dry_run=true when you want an explicit no-write validation run.

Docker files owned by root

Symptoms:

  • docker compose exec ems id shows uid=0(root)
  • files in config/ or data/ are owned by 0:0
  • container logs show EMS refuses to start as root.
  • startup reports that /app/data or /app/config is not writable by the non-root runtime user

Stop the container, fix the bind-mounted host directories, and start again with your host UID/GID:

docker compose down
mkdir -p config data
sudo chown -R "$(id -u):$(id -g)" config data
cat > .env <<EOF_ENV
PUID=$(id -u)
PGID=$(id -g)
EOF_ENV
docker compose up -d

Verify PID 1 inside the container:

docker compose exec ems sh -c 'cat /proc/1/status | grep -E "^(Name|Uid|Gid):"'

Verify host ownership:

find config data -maxdepth 1 -type f -printf '%u:%g %p\n'

Generated files should match your non-zero id -u and id -g, not 0:0.

Dashboard not reachable

Check the container and published port:

docker compose ps
docker compose logs -f

The example Compose file publishes 8080:8080. Open http://<host-ip>:8080, or http://127.0.0.1:8080 on the same host.

EMS CLI diagnostics

When you're not sure where to start, diagnostics can give a quick overview.

Docker:

docker compose exec ems python3 emsctl.py diagnose
docker compose exec ems python3 emsctl.py diagnose --deep
docker compose exec ems python3 emsctl.py diagnose --support-bundle

Native Python:

python3 emsctl.py diagnose
python3 emsctl.py diagnose --deep
python3 emsctl.py diagnose --json
python3 emsctl.py diagnose --support-bundle

Normal diagnose is local/read-only. --deep adds local operational checks such as SQLite integrity, table summaries, recent configured log patterns, Docker hints, and dashboard loopback reachability. --hardware is opt-in and performs short-timeout read-only network probes only. --support-bundle writes a redacted ZIP for GitHub/support requests. Exit code 1 means at least one diagnostic error was found; warnings still exit 0.

Full runs from inside Docker:

docker compose exec ems python3 emsctl.py diagnose
docker compose exec ems python3 emsctl.py diagnose --hardware
docker compose exec ems python3 emsctl.py diagnose --control
docker compose exec ems python3 emsctl.py diagnose --control-quality --sample-seconds 60
docker compose exec ems python3 emsctl.py diagnose --deep
docker compose exec ems python3 emsctl.py diagnose --support-bundle

Full emsctl.py reference: cli.md.

Config validation

config.json contains static installation and safety settings.

The configured runtime-state file, data/runtime-state.json in new generated configs, contains temporary runtime/operator values and can override some defaults from config.json after the first start.

Important runtime fields:

{
  "system": {
    "enabled": true,
    "max_total_power": 800,
    "loop_interval": 2,
    "min_output_limit": 35
  },
  "ha": {
    "enabled": false,
    "control_enabled": false
  },
  "devices": {
    "WR1": {
      "enabled": true,
      "max_power": 800,
      "offgrid_socket_mode": "off",
      "pv_priority_factor": 1.0
    }
  }
}

Runtime-editable values are limited to the fields shown above:

  • system enabled, max_total_power, loop_interval, min_output_limit
  • HA runtime enabled and control_enabled
  • winter runtime enabled
  • per-device enabled, max_power, offgrid_socket_mode, and pv_priority_factor

Other safety and tuning values are config-only and require editing config.json plus a restart. Examples: dry_run, allow_hardware_writes, allow_state_reconciliation_writes, deadband, output_control, redistribute_clamped_power, pv_kwp_weighting, pv_charge_balance_enabled, pv_charge_balance_deadband_percent, pv_charge_balance_full_bias_percent, pv_charge_balance_strength, battery_kwh_weighting, soc_reconcile_interval, HA URL/token, and device IP/SN.

Config still contains placeholders

Edit the Docker config:

nano config/config.json
docker compose restart
docker compose exec ems python3 emsctl.py diagnose

Template placeholders force safe mode until replaced. Safe mode disables EMS control, enables dry-run, and blocks hardware writes.

Docker permission denied

If Docker reports permission denied for the daemon socket, run the command with sudo or complete the optional Docker post-install group setup in install-docker.md. Open a new login session after changing group membership.

Reset runtime state

rm data/runtime-state.json

Do this only while the EMS is stopped. On next start, EMS recreates the configured runtime-state file from config.json defaults. Older root-level runtime-state.json files from previous setups are no longer required after switching to data/runtime-state.json and may be removed manually.

Relevant events:

runtime_state_created
runtime_state_loaded
runtime_state_changed
runtime_state_saved
runtime_state_load_error

More detail: runtime-state.md, configuration.md.

Grid meter diagnostics

Check grid_meter.type, the meter endpoint (grid_meter.ip for HTTP meters or grid_meter.mqtt.host/grid_meter.mqtt.topic for Zendure SmartMeter D0 and MQTT), local network routing, and firewall rules. For authenticated MQTT brokers, also verify that the configured username and password match the broker credentials. Then run:

docker compose exec ems python3 emsctl.py diagnose --hardware

If the Shelly is still reachable but repeatedly times out, its HTTP/RPC stack may be slow or temporarily stuck. A Shelly reboot can clear that state. Treat this as a device/network diagnostic step, not as an EMS session-reset workaround.

For MQTT meters, diagnose --hardware checks broker TCP reachability only. Use grid-meter test to wait for a subscribed value and distinguish broker connectivity from a missing or stale topic payload.

Intermittent grid-meter read timeouts (for example a Shelly ReadTimeoutError on /rpc/Shelly.GetStatus) usually point at a slow/unresponsive meter or a flaky network rather than an EMS control problem; power-cycling the meter often clears it. Confirm with a focused read test:

python3 emsctl.py grid-meter test --duration 120 --interval 1

It reports read count, OK/failed counts, and p50/p95/max read latency.

Advanced TCP reachability checks

EMS checks the actual HTTP/API read path. For deeper network-level diagnostics you can optionally use paping-go:

https://github.com/basecubedev/paping-go

Example long-running checks:

# Shelly HTTP/RPC port, run for 6 hours with one probe per second
paping-go -p 80 -d 6h -r 1 192.168.100.93

# EMS dashboard port, run for one day with one probe every 5 seconds
paping-go -p 8080 -d 24h -r 5 <ems-host>

This is optional and not required for normal EMS operation. It is most useful for long-running TCP latency/reachability tests over several hours or days when you need to catch rare timeouts, Wi-Fi drops, router path issues, Shelly HTTP/RPC stalls, inverter reachability problems, or EMS dashboard port instability. Very short smoke tests are fine for a quick check, but the main use case is sustained monitoring over hours or days.

Zendure device diagnostics

Check each device ip and sn in config/config.json, verify the device is on the same reachable network, and run:

docker compose exec ems python3 emsctl.py diagnose --hardware

This runs short-timeout read-only checks for configured Shelly, EcoTracker, and Zendure endpoints. It does not write to devices. Use it when telemetry is missing, a grid meter cannot be parsed, or configured devices appear unreachable.

The output also includes a compact communication-health summary: grid-meter read health and per-device read/write health, each with a status (ok/degraded/failed/unknown), last-success age, consecutive errors, and last read latency. stale value used: yes means a grid-meter read failed and EMS kept the last known value (intended fallback, not a control bug). Read and write health are tracked separately, so a device can read fine while writes fail or are intentionally blocked. These counters are in-memory and reset on restart.

Runtime communication health is kept in memory by the running EMS process. It is intended for live diagnostics, dashboard/API exposure, or future telemetry export. emsctl diagnose --hardware is a fresh read-only probe from a separate CLI process: it checks the currently configured meter and devices at the moment you run it, and does not read historic runtime counters from the running EMS process.

Device is online but does not deliver power

Symptoms:

  • EMS writes a non-zero outputLimit
  • sensor.ems_solarflow_<device>_target is above zero
  • actual output remains zero or much lower
  • battery does not discharge

Important telemetry fields and sensors:

outputHomePower
outputLimit
solarInputPower
packInputPower
outputPackPower
electricLevel
socLimit
packState
acStatus
dcStatus

Home Assistant sensors:

sensor.ems_solarflow_<device>_target
sensor.ems_solarflow_<device>_output
sensor.ems_solarflow_<device>_output_limit
sensor.ems_solarflow_<device>_soc_limit
sensor.ems_solarflow_<device>_pack_state
binary_sensor.<device>_ac_active
binary_sensor.<device>_dc_active
binary_sensor.<device>_available

Common causes:

Cause Check
battery is at or below min_soc device SOC and configured min_soc
device reports no discharge capacity no_discharge_capacity event
telemetry is stale binary_sensor.<device>_available and last_seen_age_s
AC/DC path inactive acStatus, dcStatus
device is runtime-disabled runtime-state.json device enabled=false
target is clamped by max power max_total_power, device max_power

Relevant events:

capability_detection
no_discharge_capacity
night_min_soc_idle_enter
night_min_soc_idle_hold_skip_write
night_min_soc_idle_park_write
min_output_limit_applied

Night/min-SOC idle entry and exit (night_min_soc_idle_enter, night_min_soc_idle_exit) and the parking write (night_min_soc_idle_park_write) are real transitions/writes and stay at info. night_min_soc_idle_hold_skip_write only confirms a device is already parked, so it is a debug trace; set system.log_level=debug to see it.

Related docs: configuration.md, winter-mode.md, safety-model.md.

Output stays at 0 W or device does not wake up

Some installations treat repeated outputLimit=0 like a stop, idle, or sleep state. min_output_limit can keep a small standby/wakeup target while EMS control is enabled.

Check:

{
  "system": {
    "min_output_limit": 35
  }
}

Runtime override:

python3 emsctl.py system min-output-limit 30

Use 0 to disable this behavior.

Relevant events:

min_output_limit_applied
night_min_soc_idle_park_write
night_min_soc_idle_hold_skip_write

Device offline or stale telemetry

Symptoms:

  • device sensors stay visible but do not update
  • writes are skipped for one device
  • target allocation looks lower than expected
  • HA binary_sensor.<device>_available is off

Relevant events:

offline_skip_write
output_control_stale_telemetry
preflight_device_unreachable

The EMS may use cached state for calculation, but it suppresses writes to devices without fresh telemetry.

Check:

  • device IP address
  • local network reachability
  • device Wi-Fi quality
  • telemetry_max_age_seconds
  • HA last_seen_age_s attribute

More detail: configuration.md and home-assistant.md.

Control loop diagnostics

Use control diagnostics when EMS output looks surprising:

Docker:

docker compose exec ems python3 emsctl.py diagnose --control
docker compose exec ems python3 emsctl.py diagnose --control --sample-seconds 30

Native Python:

python3 emsctl.py diagnose --control
python3 emsctl.py diagnose --control --json
python3 emsctl.py diagnose --control --sample-seconds 30

The control report explains the local regulation path: current grid power, filtered grid power, calculated target, final output, deadband state, control enabled/disabled state, dry-run state, device allocation, SOC protection, and likely root causes. Common findings:

  • Control disabled: runtime control is off.
  • Dry run enabled: EMS calculates targets but does not write hardware.
  • Deadband active: filtered meter power is inside the configured threshold.
  • Grid meter signal appears noisy: local samples oscillate or vary strongly.
  • Grid meter values are stale: confirmed meter timestamps stopped updating or repeated read failures were reported. Repeated grid values alone do not mean the meter is stale.
  • Minimum SOC protection active: one or more devices are protected by SOC.

Review Control Snapshot, Decision Explanation, Write Path, and Likely Causes. Runtime-state file age is not used as proof of stale EMS control activity. If no live control timestamp is available, the control staleness check is skipped and shown as informational output.

Control quality

Use control quality diagnostics when the system regulates but the result looks poor over time:

Docker:

docker compose exec ems python3 emsctl.py diagnose --control-quality --sample-seconds 60

Native Python:

python3 emsctl.py diagnose --control-quality --sample-seconds 60
python3 emsctl.py diagnose --quality --json

Interpretation:

  • Export/import quality measures how close the sampled grid power is to the zero-export target. Positive values are import; negative values are export.
  • Export peaks show the strongest negative grid-power samples. Small short peaks can happen; large or long export periods mean EMS is not consistently holding zero export.
  • The regulation quality score is a coarse diagnostic indicator. It penalizes average deviation from zero, export duration, export peak size, and large import peaks, then classifies the result as excellent, good, acceptable, poor, or critical.
  • PV diagnostics can identify missing PV telemetry, likely system/device output limits, or PV that appears unused. It cannot prove a hardware fault on its own.
  • SOC balancing warnings highlight large SOC spread, lower-SOC devices doing too much work, and devices protected by minimum SOC.

Regulation is too slow

Symptoms:

  • house load changes quickly, but EMS target follows too late
  • inverter output lags behind demand
  • target rises or falls only in small steps
  • Home Assistant helper changes take effect late

Check these settings:

{
  "system": {
    "loop_interval": 2,
    "output_control": {
      "filter_enabled": true,
      "ema_alpha": 0.85,
      "ramp_enabled": true,
      "ramp_up_w_per_cycle": 600,
      "ramp_down_w_per_cycle": 700,
      "device_ramp_enabled": true,
      "device_ramp_up_w_per_cycle": 500,
      "device_ramp_down_w_per_cycle": 600,
      "large_import_bypass_w": 600,
      "large_export_bypass_w": 600,
      "bypass_ramp_multiplier": 1.5,
      "telemetry_max_age_seconds": 10,
      "stale_telemetry_ramp_factor": 0.5
    }
  }
}

Tuning hints:

Symptom Setting Direction
Control loop reacts too late loop_interval lower carefully
Filter is too smooth ema_alpha increase
Total target rises too slowly ramp_up_w_per_cycle increase
Total target falls too slowly ramp_down_w_per_cycle increase
Per-device target changes too slowly device_ramp_*_w_per_cycle increase
Stale telemetry slows response telemetry_max_age_seconds / stale_telemetry_ramp_factor check telemetry freshness first

Validate after tuning:

python3 -B ems-solarflow-api-control.py --dry-run --duration 120

Control-chain details: control-logic.md and control-flow.md.

Relevant events (per-cycle control-loop traces; emitted at debug, set system.log_level=debug to see them):

output_control_state
output_control_ramp_limited
output_control_device_ramp_limited
output_control_stale_telemetry
output_control_bypass
output_control_sign_change_fast_response
target_calculation

Regulation oscillates or writes too often

Symptoms:

  • target jumps up and down every cycle
  • many repeated write_output_limit_published events
  • actual output never settles
  • grid import/export alternates quickly

Check these settings:

{
  "system": {
    "deadband": 5,
    "output_control": {
      "load_deadband_w": 5,
      "target_deadband_w": 5,
      "filter_enabled": true,
      "median_window": 3,
      "ema_alpha": 0.85,
      "ramp_enabled": true
    }
  }
}

Tuning hints:

Symptom Setting Direction
too many small target changes target_deadband_w or deadband increase
noisy load input load_deadband_w increase
output follows every spike ema_alpha decrease
target jumps too hard ramp_up_w_per_cycle / ramp_down_w_per_cycle decrease
devices fight each other disable other controllers check Zendure app, HEMS, HA automations

Relevant events (output_control_deadband_hold is a debug trace; the actual write write_output_limit_published stays at info):

output_control_deadband_hold
deadband_skip_write
write_output_limit

Export peaks

Review Export / Import Quality and Regulation Quality. Positive grid power is import. Negative grid power is export. Short small export peaks can be normal; large peaks or long export duration mean EMS is not consistently holding the zero-export target.

Uneven battery usage

Review SOC Balancing. A warning can indicate high SOC spread, a lower-SOC device contributing more than expected, a protected min-SOC device, or runtime max-power limits that prevent balanced distribution.

One device is used too much or too little

Check device metadata:

{
  "name": "WR1",
  "max_power": 800,
  "pv_kwp": 2.0,
  "pv_priority_factor": 1.0,
  "battery_kwh": 1.92,
  "min_soc": 15,
  "max_soc": 100
}

Tuning hints:

Field Effect
max_power hard per-device output limit
pv_kwp PV-size weighting
pv_priority_factor manual PV priority correction
battery_kwh battery weighting
min_soc lower discharge boundary
max_soc upper SOC/headroom boundary

Relevant events:

balance_weight
pv_first_limit
pv_first_limited (DEBUG diagnostic; normal before battery top-up)
pv_first_battery_topup (DEBUG diagnostic; normal allocation detail)
pv_first_battery_topup_unmet (WARNING; unresolved shortfall after top-up)
target_calculation

pv_first_limited means the PV-only allocation pass did not cover the full requested target by itself. This is usually normal in PV-first mode, especially when a device is already charging its battery or derating output. Treat pv_first_battery_topup_unmet as the warning event for an unresolved PV-first shortfall after allowed battery top-up.

Keep pv_priority_factor=1.0 first. Adjust only after confirming realistic pv_kwp, battery_kwh, and SOC limits.

Runtime tuning is available without editing config.json:

python3 emsctl.py device WR1 pv-priority-factor 1.3
python3 emsctl.py device WR2 pv-priority-factor 0.7

This changes PV-first weighting only. It does not create additional PV power and does not override device power limits.

No power changes

Symptoms:

  • target calculation looks correct
  • device output does not change
  • logs show dry-run events only
  • Home Assistant target sensors change, but hardware does not

Check safety flags:

{
  "system": {
    "enabled": true,
    "dry_run": false,
    "simulation_mode": false,
    "allow_hardware_writes": true
  }
}

Also check whether the EMS was started with one of these flags:

--dry-run
--simulate
--replay
--preflight

These modes do not perform normal live output control writes.

Expected dry-run event:

event=dry_run_output_limit

Expected live-write event:

event=write_output_limit_published

For an MQTT device this event means only that the command was dispatched to the transport — never that the device accepted it. Follow the command lifecycle events to see what actually happened:

mqtt_publish_delivered        broker acknowledged the QoS 1 publish (PUBACK)
device_command_acknowledged   correlated device reply (legacy invoke profiles)
telemetry_confirmed           telemetry proved the command effective
confirmation_timed_out        no matching telemetry before the deadline
external_control_suspected    a foreign writer keeps overriding a confirmed target

Other relevant events:

control_disabled_skip_write
device_disabled_skip_write
offline_skip_write
deadband_skip_write
write_output_limit_error
state_reconciliation_skipped

Cloud MQTT writes are published but the inverter never changes

Symptoms:

  • write_output_limit_published appears for the cloud device
  • repeated confirmation_timed_out warnings
  • inverter output never follows the target

Check, in order:

  1. confirmation_timed_out includes broker_delivery. timeout there means the broker never acknowledged the publish — check the cloud credentials: bidirectional cloud MQTT requires the Zendure App / Home Assistant authorization credentials from the device-list login; the public read-only developer account silently drops writes. Broker delivery and device acceptance are independent: broker_delivery=delivered never means the device applied the command, and a late PUBACK still updates its original bounded ledger record after telemetry confirmation and after newer commands. An unresolved MID that crosses a disconnect is quarantined instead of being guessed after reuse, even once its bounded tombstone expires. Confirmation also requires a trustworthy command publish time and a fresh per-property observation time; missing provenance fails closed rather than confirming a cached matching value.
  2. broker_delivery=delivered with no confirmation means the broker accepted the command but the device did not apply it — verify the pinned hardware_profile matches the physical model and the device identifiers (product_key, device_id) are correct.
  3. Obsolete write topic. A profile-backed device that carries a stale mqtt.write_topic cannot misroute control — the canonical iot/<productKey>/<deviceId>/properties/write topic is always used — but a profile_write_topic_obsolete validation warning flags it. Apply the migration/maintenance preview to remove it. (diagnose reports the effective write topic and its source, canonical_profile vs custom_explicit.)
  4. external_control_suspected means another controller is overriding EMS: disable Zendure HEMS, Smart Matching, Zendure schedules and any other system that writes inverter power. Only one controller may run.
  5. Validate the full path with the hardware probe (mqtt-write-latency-probe.md): stop the EMS, run --dry-preview (it shows the canonical topic, whether an obsolete override is ignored, and restore feasibility), then --confirm-writes and check that the setpoint matches the target (movement toward it is not a match) and the required mode properties match. The probe reports broker delivery, setpoint HTTP-match and physical output from the same submission origin, plus the physical delay after setpoint. It polls delivery and HTTP evidence together, refuses to write when any potentially modified initial property is missing, requires an observed away-then-initial HTTP transition before calling restoration verified, has no unsafe atomic-profile partial fallback, and exits non-zero if the initial state cannot be fully restored. A restoration-time HTTP read fault is recorded and cannot suppress the full restore submission; missing verification evidence remains a non-zero result. The probe stops its MQTT runtime in an outer cleanup path even if restoration raises.
  6. Serial-less Cloud device: --api-ip reports "no MQTT control device has a trusted physical serial …" or the write is blocked as unverified binding. A Cloud device without a stored physical serial cannot be auto-selected by the HTTP serial (the Cloud route id and a physical serial are different identity domains). Select it with exact --device-name, --device-id and --broker-ref. The probe then treats the HTTP serial as new, unverified binding evidence and blocks the write until you either pass --confirm-unbound-api-readback (accept it for this run only — never persisted) or bind the physical serial first through Admin discovery. A configured serial that does not match the HTTP readback is a hard identity conflict and always blocks. See mqtt-write-latency-probe.md.

More detail: safety-model.md, configuration.md, runtime-state.md.

Setup or Maintenance shows "Identity conflict" for a rediscovered device

Symptoms:

  • A discovered Cloud MQTT proposal shows a disabled Identity conflict action instead of Add / In config.
  • Preview/apply reports device_identity_conflict.

Cause: the discovered Cloud route is already configured against a different physical serial. The route says "one inverter" while the serials say "two", so Admin refuses to merge or add it as an independent inverter rather than guess. Fresh Setup and Maintenance apply the same rule.

Fix: confirm which physical inverter that Cloud route belongs to. If the existing entry has the wrong serial, correct or remove it, then rediscover. A route-only device gaining its own serial is not a conflict — it is recognized as the same inverter (shown In config), keeps its custom name and dismissal, and is enriched in place. See admin-discovery.md.

A selected Cloud inverter reports "not present in current discovery state"

Symptom: after a rediscovery, a previously selected Cloud MQTT inverter is rejected with zendure_mqtt_proposal_unknown at preview/apply.

Expected behavior: this should not happen for a route-only Cloud selection that is rediscovered on the same scoped route (with or without a new serial). A Cloud proposal's selection id is anchored to its scoped-route token so it stays stable through serial enrichment, and trust resolution additionally remaps a stored selection to the current proposal when a trusted alias token intersects within the same broker scope. If you still see this error, the selection is genuinely stale (a different route, a different broker/account scope, or a tampered id/token) — re-run discovery and select the inverter again.

Cloud MQTT topics look masked in status or support bundles

This is expected. Zendure Cloud account-scoped routes (iot/<product>/<device>/..., including function/invoke and custom suffixes) are masked to iot/…/…/... at every browser and support-export boundary, in structured fields, log/error text and mapping keys alike, so a support bundle never carries account routing secrets. Local MQTT topics are not masked — they contain user-controlled local identifiers, not Cloud secrets, and stay visible as useful diagnostics. A mixed local+Cloud status masks only the Cloud route material, per device. Physical serials and non-secret context are retained; credentials are dropped. Full identifiers are kept internally where required for correct command routing and matching — only external boundaries redact. See admin-discovery.md.

Dashboard values do not add up exactly

Symptoms:

  • home, target, output limit, and actual output differ in the same moment
  • global target and per-device output do not match exactly
  • off-grid socket mode looks like it should affect power totals

home is a calculated runtime/dashboard value. It is not the smoothed control target. The EMS target can be filtered, ramped, clamped, and rate-limited before an outputLimit write is attempted. The actual Zendure output can then lag or remain lower because of device state, available PV/battery power, API timing, or firmware behavior.

Off-grid socket mode is a mode/state value, not power. Do not add it to the home-load, target, output-limit, or actual-output calculation.

More detail: control-logic.md, home-assistant.md, and runtime-state.md.

Unexpected SOC or mode changes

Check whether state reconciliation writes are enabled:

{
  "system": {
    "allow_state_reconciliation_writes": true,
    "soc_reconcile_interval": 10,
    "reconcile_ac_mode_on_start": true,
    "reconcile_smart_mode": true
  }
}

Runtime output writes and persistent state reconciliation writes are separate write paths. Output-limit writes require normal hardware writes to be enabled. State reconciliation writes additionally require allow_state_reconciliation_writes=true.

Relevant events:

dry_run_soc_limits
write_soc_limits
soc_limits_unchanged
dry_run_device_modes
write_device_modes
device_modes_unchanged
dry_run_runtime_device_state_write
write_runtime_device_state

The *_unchanged events (soc_limits_unchanged, device_modes_unchanged, runtime_device_state_unchanged) are healthy idle behavior and are emitted at debug. Actual writes (write_*) and dry-run skips stay visible at info.

Set allow_state_reconciliation_writes=false while validating normal output control only if you deliberately want a conservative troubleshooting variant. The normal template profile keeps it enabled for the full regulation profile after required placeholders are replaced and local limits are reviewed.

Related docs: configuration.md, winter-mode.md, safety-model.md.

Winter mode

Relevant events:

winter_mode_state
winter_ramp
winter_summer_reset
dry_run_winter_ac_charge_limit
write_winter_ac_charge_limit

If no winter event appears, check:

  • winter.enabled
  • runtime winter.enabled
  • current month versus winter.months
  • soc_reconcile_interval
  • current hour versus winter.adjust_hour
  • allow_state_reconciliation_writes

Winter logic runs through SOC reconciliation. It is not a per-cycle output control mechanism.

winter_mode_state is logged at info only when the active state changes or an adjustment is due; otherwise it is a debug trace. Actual winter writes (write_winter_ac_charge_limit, winter_ramp, winter_summer_reset) stay visible at info. Enable system.log_level=debug to see every reconcile.

Home Assistant entities missing

Home Assistant entities are created by REST state writes. They appear after the EMS has published at least once.

After an HA restart, entities can temporarily appear as stale, unavailable, or restored until the EMS publishes fresh states again.

Check:

  • ha.enabled=true in config.json
  • runtime ha.enabled=true
  • valid HA URL and token
  • not running with --no-ha
  • not running simulation or replay
  • Home Assistant is reachable from the EMS host

Relevant events:

ha_publish_no_devices
ha_write_error
runtime_state_ha_write

Home Assistant helpers are ignored

Symptoms:

  • changing HA max power has no effect
  • changing HA enable switch has no effect
  • changing HA loop interval has no effect
  • HA sensors exist, but controls do not change EMS behavior

Check static config and runtime state both have:

{
  "ha": {
    "enabled": true,
    "control_enabled": true
  }
}

Expected helpers:

input_boolean.ems_solarflow_ha_enabled
input_boolean.ems_solarflow_ha_control_enabled
input_boolean.ems_solarflow_enable
input_number.ems_solarflow_max_power
input_number.ems_solarflow_interval
input_number.ems_solarflow_min_output_limit
input_boolean.ems_solarflow_winter_enabled

Per-device helper example:

input_boolean.ems_solarflow_wr1_enabled
input_number.ems_solarflow_wr1_max_power
input_select.ems_solarflow_wr1_offgrid_socket_mode

Relevant events:

runtime_state_ha_sync
runtime_state_ha_read_error
ha_runtime_sync_failed
runtime_state_changed

If HA helper sync fails, EMS continues with the last valid local runtime-state.json values.

HA helper values can update runtime-state.json only when static ha.enabled=true, static ha.control_enabled=true, runtime ha.enabled=true, and runtime ha.control_enabled=true. --no-ha, simulation, and replay disable HA reads and writes for that run.

More detail: home-assistant.md.

Preflight fails

Run:

python3 -B ems-solarflow-api-control.py --preflight --dry-run

Relevant events:

preflight_start
preflight_ha_ok
preflight_shelly_ok
preflight_device_ok
preflight_device_unreachable
preflight_abort
preflight_failed
preflight_ok

Common causes:

Event Meaning
preflight_device_unreachable Zendure device cannot be reached
preflight_abort required preflight input is missing or invalid
preflight_failed at least one required check failed

Backup and restore diagnostics

See backup-restore.md for the full workflow. Common issues:

Backups are stored in data/backups/ by default. Docker users see the same folder on the host via the existing ./data:/app/data mount.

If an update went wrong, inspect the backups under host path data/backups/ before changing files by hand:

docker compose exec ems python3 emsctl.py backup inspect data/backups/<file>.tar.gz.enc
docker compose exec ems python3 emsctl.py backup restore data/backups/<file>.tar.gz.enc --dry-run
docker compose exec ems python3 emsctl.py backup restore data/backups/<file>.tar.gz.enc --on-conflict replace --rollback

For unencrypted backups, use the same commands with data/backups/<file>.tar.gz.

Common findings:

  • "InfluxDB analytics is disabled. Nothing to back up." — backup create --type influxdb only applies when influxdb.enabled is true. Enable bundled analytics first (emsctl.py influx init).
  • "External InfluxDB detected. … not supported for external mode." — Automatic InfluxDB backup/restore covers bundled mode only. Use your external InfluxDB provider's backup tooling. External mode never blocks config or database backups.
  • "no InfluxDB token resolved" — the backup needs a usable token. Set influxdb.token, export the influxdb.token_env variable, or run python3 emsctl.py influx init to generate deploy/docker/influxdb.env.
  • "failed to start bundled InfluxDB via docker compose" — Docker/Compose is required. Check docker compose ps.
  • Encrypted archive won't open — restoring or inspecting a .tar.gz.enc prompts for the password; a wrong password aborts cleanly. The password is never stored and cannot be recovered.
  • After an InfluxDB restore the data is replaced (influx restore --full, bundled mode only — it restores org/buckets/users/tokens and history). Verify with python3 emsctl.py influx status and python3 emsctl.py diagnose --deep. If you created a rollback InfluxDB backup you can restore it the same way.

When migrating or recovering a whole setup, restore in this order so the bundled token and config stay in sync:

  1. Restore the config backup first (ems-config-...).
  2. Verify the bundled InfluxDB secret/config files are present (deploy/docker/influxdb.env, config.json).
  3. Restore the InfluxDB backup (ems-influxdb-...); create a rollback first.
  4. Run python3 emsctl.py influx status and python3 emsctl.py diagnose --deep.

Update and maintenance diagnostics

The Admin Console guided upgrade is a conservative, single-shot workflow: it verifies the target image, optionally backs up, optionally updates config, rewrites the EMS image reference in docker-compose.yml, and force-recreates the ems service. It never removes containers, volumes, or data. For the version detection, build-identity gating, SemVer fallback, release cache and Docker execution details, see admin-discovery.md.

If Upgrade system stops with System Build verification is no longer current (HTTP 409 system_build_verification_stale, or system_build_verification_required when no verification was sent), the target image or build metadata changed after you verified it — most often a mutable tag such as latest re-pushed to a new digest. No preflight, backup, migration, or deployment ran. Select Verify System Build again to re-resolve and re-verify the current pair, then re-plan and retry. This check is deliberate: it guarantees the executed System Build is exactly the one you verified.

For Docker Bootstrap or advanced shell use, the equivalent manual recreate is:

docker compose pull ems
docker compose up -d --force-recreate ems
docker compose exec ems python3 emsctl.py diagnose

Roll back a bad update by restoring a backup; see the backup and restore diagnostics above and backup-restore.md.

Installed release shows as unknown

The Maintenance Overview treats a running EMS container as the active baseline and reads its release from the immutable image identity. If a running container's identity cannot be established (for example a digest-pinned image whose build labels are missing), the overview shows the current release as unknown and adds a short warning instead of borrowing the Compose or last-known-good release. This is intentional: the Compose file and known-good record describe the desired or last-successful state, not the bits that are actually running. They are used only when no EMS container is active (absent, stopped, or Docker unavailable). Recreate the EMS container from the verified System Build to restore a readable release. The custom container name honored here is EMS_CONTAINER_NAME (falling back to the Compose container_name, then the canonical ems-solarflow-api-control).

Guided upgrade digest pull failed

When the exact verified EMS digest is missing locally, guided upgrade pulls ghcr.io/basecubedev/ems-solarflow-api-control@sha256:<digest>. If that pull fails, the typed failure is preserved through the complete upgrade job — the executor step, the job result, the transition record, and the UI all keep the stable error code:

  • image_pull_rate_limited / system_build_registry_rate_limited — a GHCR throttle (see the rate-limit section below);
  • image_pull_network_error — a network problem reaching the registry;
  • image_pull_failed — a generic pull failure (tag/repository/registry);
  • target_digest_mismatch — the pulled content digest did not equal the verified digest (a moved or re-pushed image).

In every case no Compose change is written and the EMS container is not recreated. Any backup or config steps that already ran before the pull are reported honestly in the step list. The verified target stays selected and you can retry. (Untrusted or unknown executor reasons are normalized to ems_upgrade_failed and never copied verbatim into the transition record.)

GitHub Container Registry rate limit reached

When you select Verify System Build (or Update Admin Server) and the System Build images are downloaded, GitHub Container Registry (GHCR) may throttle the request. The Admin Console reports this as a distinct, actionable error:

GitHub Container Registry rate limit reached.

No installation changes were made. Wait before retrying, or authenticate
Docker with a GitHub account to increase the available request quota.

What it means and what to do:

  • Nothing was installed or changed. The build is left unverified, Continue and Update Admin Server stay disabled, and no deployment starts. The full Docker output is in the expandable diagnostics area (credentials are never shown).
  • Just wait and retry. Selecting Verify System Build again after a short wait is the normal fix — anonymous GHCR pulls share a per-IP quota that replenishes over time. Simply browsing builds never consumes GHCR requests, so the wait only applies to the verify/download step.
  • Optional: if you download builds often, authenticating Docker with a GitHub account (docker login ghcr.io) raises the available quota. This is a convenience, not a requirement — normal operation needs no GitHub token.

Support bundle and issue reports

Diagnostics are optional. You can open an issue even if you cannot run them or are not sure which command applies to your setup.

If available, attach a support bundle to help identify issues faster.

Native Python:

python3 emsctl.py diagnose
python3 emsctl.py diagnose --support-bundle

Docker:

docker compose exec ems python3 emsctl.py diagnose
docker compose exec ems python3 emsctl.py diagnose --support-bundle

The support bundle is redacted and helps identify common installation, runtime, hardware, control, and performance problems without exposing tokens, passwords, serial numbers, dashboard auth files, or database contents.

Issue report checklist

Share any of these details that are easy to provide:

EMS version / commit:
Number of devices:
Device model(s):
Firmware version if known:
Home Assistant enabled:
HA control enabled:
dry_run:
allow_hardware_writes:
allow_state_reconciliation_writes:
loop_interval:
deadband:
max_total_power:
min_output_limit:
output_control settings changed from default: yes/no

If available, one complete EMS cycle with these events can help:

startup
runtime_state_loaded
capability_detection
output_control_state
target_calculation
write_output_limit or dry_run_output_limit

Remove secrets before posting logs or config snippets:

Home Assistant token
Zendure serial numbers
local IP addresses if desired

Helpful links:

Optional local validation

If you want to collect more detail, these local checks can help.

Compile:

python3 -m py_compile ems-solarflow-api-control.py emsctl.py

Run self-tests:

python3 -B ems-solarflow-api-control.py --self-test

Run tests from the repository root:

python -m pytest -q

Direct pytest -q is also supported when pytest.ini is present. Both commands require pytest in the active Python environment.

Run simulation:

python3 -B ems-solarflow-api-control.py --simulate --max-cycles 1

Run preflight against live devices without control writes:

python3 -B ems-solarflow-api-control.py --preflight --dry-run

Run one dry-run control cycle:

python3 -B ems-solarflow-api-control.py --dry-run --no-ha --once

Check required events:

python3 scripts/check_log_events.py /tmp/ems-sim.log \
  --require startup \
  --require target_calculation

Safe diagnostic workflow

For deeper local validation, this order is a useful starting point:

python3 -B ems-solarflow-api-control.py --simulate --max-cycles 1
python3 -B ems-solarflow-api-control.py --preflight --dry-run
python3 -B ems-solarflow-api-control.py --dry-run --duration 120
python3 -B ems-solarflow-api-control.py --duration 60

Only run the final live test when these are true:

dry_run=false
simulation_mode=false
allow_hardware_writes=true
runtime system.enabled=true
at least one runtime device enabled=true