assert_no_overlapping_spans in log-analysis/nameres/analyze_nameres_logs.py (added in #107) refuses to load two log exports whose time spans intersect, because CloudWatch Logs Insights exports carry no per-record ID and overlapping windows would silently double-count the same lookups — biasing every statistic in the notebook with no visible symptom.
Rejecting is the right default, but it makes the exports awkward to accumulate: you cannot re-export "the last 10 weeks" on top of an existing "the last 4 weeks" without deleting one, and you cannot combine an autocomplete-only export with a general export covering the same period even though they select disjoint sets of log lines.
What would need designing:
- A record identity. There is no ID field. The candidate key is (
@timestamp, kubernetes.pod_name, the raw log line) — timestamps carry milliseconds and pods are distinguishable, but two genuinely identical lookups from the same pod in the same millisecond would collapse into one. Worth measuring how often that actually happens before assuming it is safe.
- Whether spans are even the right test. The check currently keys on time alone, so it fires on exports that select disjoint log lines (autocomplete-only vs general). A filter-aware comparison would let those coexist.
- Where de-duplication happens — at load, with a reported count of dropped duplicates, so the notebook says plainly how much overlap it absorbed rather than hiding it.
Until then the workaround is to re-export non-overlapping windows, which the error message says.
The current behaviour is covered by tests/log_analysis/test_nameres_log_notebook.py (partial, contained, identical and endpoint-touching overlaps), so any change here has tests to update rather than write from scratch.
assert_no_overlapping_spansinlog-analysis/nameres/analyze_nameres_logs.py(added in #107) refuses to load two log exports whose time spans intersect, because CloudWatch Logs Insights exports carry no per-record ID and overlapping windows would silently double-count the same lookups — biasing every statistic in the notebook with no visible symptom.Rejecting is the right default, but it makes the exports awkward to accumulate: you cannot re-export "the last 10 weeks" on top of an existing "the last 4 weeks" without deleting one, and you cannot combine an autocomplete-only export with a general export covering the same period even though they select disjoint sets of log lines.
What would need designing:
@timestamp,kubernetes.pod_name, the raw log line) — timestamps carry milliseconds and pods are distinguishable, but two genuinely identical lookups from the same pod in the same millisecond would collapse into one. Worth measuring how often that actually happens before assuming it is safe.Until then the workaround is to re-export non-overlapping windows, which the error message says.
The current behaviour is covered by
tests/log_analysis/test_nameres_log_notebook.py(partial, contained, identical and endpoint-touching overlaps), so any change here has tests to update rather than write from scratch.