Skip to content

Commit 2394997

Browse files
erez-c137erez-c137
authored
docs: document the database schema as a public source of truth (#272)
Adds DATABASE.md at the repo root, alongside README/TRANSLATORS/privacy, and links it from the README's transparency bullets. It deliberately does NOT live under Docs/, which is gitignored - a schema reference nobody outside the repo can read would serve nobody. Written against the live schema, not from memory: every CREATE statement is the real DDL, every stat_type and unit was read out of an actual database, and all five example queries were executed before being published. It documents the three-tier rollup and, more usefully, the things that will bite someone querying it directly: - speeds are BYTES per second, not bits - there is no "All interfaces" row, and virtual adapters (WSL, Hyper-V, VPN) are in there alongside the real NIC - there is no min column, so percentiles are impossible beyond the 24h raw tier - which is why the Statistics sheet marks them unavailable rather than fabricating them - avg and max answer different questions, and the long-range graph line is a peak envelope rather than a rate trace - volume must come from avg x bucket_seconds, never sample_count x poll rate, because the poll rate is user-configurable and would retroactively rescale history It also records that bandwidth_history is created, indexed and never written to, so nobody mistakes it for a working lifetime odometer - that is usage_counter. Ends with what is deliberately not stored and why: per-app bytes (Windows cannot attribute them without a driver), LAN-vs-internet (adapter counters carry no addresses), and anything leaving the machine.
1 parent 60fe27d commit 2394997

2 files changed

Lines changed: 287 additions & 0 deletions

File tree

‎DATABASE.md‎

Lines changed: 286 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,286 @@
1+
# The NetSpeedTray database
2+
3+
Your history lives in one SQLite file. It is yours — nothing is uploaded, and this document tells you
4+
exactly what is in it so you can query it yourself.
5+
6+
```
7+
%APPDATA%\NetSpeedTray\speed_history.db
8+
```
9+
10+
Paste that into Explorer's address bar to find it. It opens in any SQLite tool
11+
([DB Browser for SQLite](https://sqlitebrowser.org/) is the usual choice), or from Python with the
12+
standard library and no dependencies.
13+
14+
> **Read it while NetSpeedTray is running, don't write to it.** The database is in WAL mode, so
15+
> reading alongside the running app is safe. Writing to it underneath the app is not.
16+
17+
This file is the source of truth for the schema. If you change the schema, change this file in the
18+
same commit.
19+
20+
---
21+
22+
## The shape of it: three tiers that age
23+
24+
Every second, NetSpeedTray records how many bytes crossed each network adapter. Keeping that forever
25+
would mean ~31 million rows per adapter per year, so older data is rolled up:
26+
27+
| Tier | Resolution | Holds | Then |
28+
|---|---|---|---|
29+
| `speed_history_raw` | per second | the last **24 hours** | averaged into per-minute rows, originals deleted |
30+
| `speed_history_minute` | per minute | the last **30 days** | averaged into per-hour rows, originals deleted |
31+
| `speed_history_hour` | per hour | until your retention setting | deleted |
32+
33+
Retention is **Settings → Advanced → Keep data**, default 365 days. "Keep everything" sets it to
34+
36,500 days, which is a hundred years and means in practice *never delete*.
35+
36+
Hardware statistics use the identical three-tier model in `hardware_stats_raw` / `_minute` / `_hour`.
37+
38+
**The consequence that matters when querying:** a window longer than 24 hours spans more than one
39+
tier, and each tier stores different columns. A query that reads only one table silently misses data.
40+
41+
---
42+
43+
## Tables
44+
45+
### `speed_history_raw` — per-second, last 24h
46+
47+
```sql
48+
CREATE TABLE speed_history_raw (
49+
timestamp INTEGER NOT NULL, -- Unix epoch seconds
50+
interface_name TEXT NOT NULL, -- e.g. 'Wi-Fi 3', 'Ethernet', 'Tailscale'
51+
upload_bytes_sec REAL NOT NULL, -- BYTES per second, not bits
52+
download_bytes_sec REAL NOT NULL,
53+
PRIMARY KEY (timestamp, interface_name)
54+
);
55+
```
56+
57+
### `speed_history_minute` / `speed_history_hour` — the rollups
58+
59+
```sql
60+
CREATE TABLE speed_history_minute ( -- and _hour, identically
61+
timestamp INTEGER NOT NULL, -- start of the bucket
62+
interface_name TEXT NOT NULL,
63+
upload_avg REAL NOT NULL, -- mean bytes/sec across the bucket
64+
download_avg REAL NOT NULL,
65+
upload_max REAL NOT NULL, -- peak bytes/sec seen in the bucket
66+
download_max REAL NOT NULL,
67+
sample_count INTEGER NOT NULL DEFAULT 1, -- how many samples went in
68+
PRIMARY KEY (timestamp, interface_name)
69+
);
70+
```
71+
72+
`sample_count` is how you tell a full bucket from a partial one — 60 samples in a minute bucket means
73+
NetSpeedTray was running the whole minute; 12 means it was not.
74+
75+
### `hardware_stats_raw` / `_minute` / `_hour`
76+
77+
A generic key/value time series. One row per metric per timestamp.
78+
79+
```sql
80+
CREATE TABLE hardware_stats_raw (
81+
timestamp INTEGER NOT NULL,
82+
stat_type TEXT NOT NULL,
83+
value REAL NOT NULL,
84+
PRIMARY KEY (timestamp, stat_type)
85+
);
86+
-- _minute and _hour replace `value` with avg_value, max_value, sample_count
87+
```
88+
89+
`stat_type` values currently recorded, and their units:
90+
91+
| `stat_type` | Unit | Notes |
92+
|---|---|---|
93+
| `cpu`, `gpu`, `ram` | percent | 0–100 |
94+
| `cpu_temp` | °C | needs LibreHardwareMonitor on most systems |
95+
| `cpu_power`, `gpu_power`, `total_power` | watts | |
96+
| `latency_gw` | milliseconds | ping to your router — your LAN |
97+
| `latency_anchor` | milliseconds | ping to a public anchor — true internet latency |
98+
| `latency_gw_timeout` | 0 or 1 | 1 = that probe timed out. **Averaged in the rollups, so in `_minute`/`_hour` this is a packet-loss *rate* between 0 and 1.** |
99+
100+
Because this table is generic, **adding a new metric needs no schema change** — a new `stat_type`
101+
flows through aggregation and retention automatically.
102+
103+
### `usage_counter` — the data-cap odometer
104+
105+
One row, always `id = 1`. Separate from the history tables because a data cap must survive retention
106+
pruning.
107+
108+
```sql
109+
CREATE TABLE usage_counter (
110+
id INTEGER PRIMARY KEY CHECK (id = 1),
111+
cumulative_up REAL NOT NULL DEFAULT 0, -- lifetime bytes, monotonic
112+
cumulative_down REAL NOT NULL DEFAULT 0,
113+
anchor_up REAL NOT NULL DEFAULT 0, -- cumulative at the start of this billing period
114+
anchor_down REAL NOT NULL DEFAULT 0,
115+
period_key TEXT NOT NULL DEFAULT '', -- the billing period this anchor belongs to
116+
updated_ts INTEGER NOT NULL DEFAULT 0
117+
);
118+
```
119+
120+
Usage this period is `cumulative - anchor`. The anchor only ever advances on a genuine **forward**
121+
period rollover, so a clock change or DST shift cannot wipe your running total.
122+
123+
### `metadata` — bookkeeping
124+
125+
`key`/`value` text pairs: `db_version`, `created_at`, `current_retention_days`,
126+
`last_maintenance_at`, `last_vacuum_at`, and the pending-retention keys used to schedule a prune.
127+
128+
### `bandwidth_history` — **not used**
129+
130+
```sql
131+
CREATE TABLE bandwidth_history (
132+
interface_name TEXT PRIMARY KEY,
133+
total_upload_bytes REAL NOT NULL DEFAULT 0,
134+
total_download_bytes REAL NOT NULL DEFAULT 0
135+
);
136+
```
137+
138+
Created and indexed by every install, and **never written to**. Do not build on it without wiring it
139+
up first. It is documented here so nobody mistakes it for a working lifetime odometer — that is
140+
`usage_counter`.
141+
142+
---
143+
144+
## Things that will bite you
145+
146+
**Speeds are bytes per second, not bits.** Multiply by 8 for Mbps: `bytes_sec * 8 / 1e6`.
147+
148+
**There is no "All interfaces" row.** Every row is one real adapter. To aggregate, `SUM` across
149+
interfaces — and note that includes virtual adapters (WSL, Hyper-V, VPN clients like Tailscale). A
150+
typical machine has one adapter carrying ~98% of traffic and several carrying almost none.
151+
152+
**There is no `min` column.** Only `avg` and `max` survive rollup, so percentiles cannot be computed
153+
honestly beyond the 24-hour raw tier. NetSpeedTray refuses to guess them rather than showing a
154+
fabricated p95 — if you see percentiles marked unavailable in the Statistics sheet, this is why.
155+
156+
**`avg` and `max` answer different questions.** The graph plots `max` at minute and hour resolution,
157+
so a long-range line is a *peak envelope*, not a rate trace. Totals use `avg`. The two cannot be
158+
reconciled by eye, and that is expected.
159+
160+
**Compute volume from `avg × bucket_seconds`, not `sample_count × poll_interval`.** The poll rate is
161+
user-configurable; multiplying by it retroactively rescales history if the user ever changed it. Use
162+
60 for minute rows and 3600 for hour rows.
163+
164+
**Timestamps are Unix epoch seconds**, interpreted in local time by the app
165+
(`datetime.fromtimestamp`). They are bucket *start* times in the rollup tables.
166+
167+
---
168+
169+
## Recipes
170+
171+
Read-only, safe to run while the app is open.
172+
173+
**Daily volume per interface, last 30 days**
174+
175+
```sql
176+
SELECT date(timestamp, 'unixepoch', 'localtime') AS day,
177+
interface_name,
178+
ROUND(SUM(download_avg) * 60 / 1e9, 2) AS down_gb,
179+
ROUND(SUM(upload_avg) * 60 / 1e9, 2) AS up_gb
180+
FROM speed_history_minute
181+
GROUP BY day, interface_name
182+
ORDER BY day DESC, down_gb DESC;
183+
```
184+
185+
**Which adapters actually carry traffic**
186+
187+
```sql
188+
SELECT interface_name,
189+
ROUND(SUM(download_avg) * 60 / 1e9, 3) AS down_gb,
190+
ROUND(SUM(upload_avg) * 60 / 1e9, 3) AS up_gb
191+
FROM speed_history_minute
192+
GROUP BY interface_name
193+
ORDER BY down_gb DESC;
194+
```
195+
196+
**Your busiest hour**
197+
198+
```sql
199+
SELECT datetime(timestamp, 'unixepoch', 'localtime') AS hour,
200+
ROUND(MAX(download_max) * 8 / 1e6, 1) AS peak_down_mbps
201+
FROM speed_history_hour
202+
GROUP BY timestamp
203+
ORDER BY peak_down_mbps DESC
204+
LIMIT 10;
205+
```
206+
207+
**Internet latency and packet loss over the last week**
208+
209+
```sql
210+
SELECT datetime(timestamp, 'unixepoch', 'localtime') AS hour,
211+
ROUND(MAX(CASE WHEN stat_type='latency_anchor' THEN avg_value END), 1) AS latency_ms,
212+
ROUND(MAX(CASE WHEN stat_type='latency_gw_timeout' THEN avg_value END) * 100, 1) AS loss_pct
213+
FROM hardware_stats_hour
214+
WHERE timestamp > strftime('%s','now','-7 days')
215+
GROUP BY timestamp
216+
HAVING latency_ms IS NOT NULL
217+
ORDER BY hour;
218+
```
219+
220+
Empty result? The hourly tier only fills once data is older than 30 days, so on a young install
221+
widen the window or query `hardware_stats_minute` instead.
222+
223+
**Percentiles — but only over the last 24 hours**, where per-second data still exists:
224+
225+
```sql
226+
SELECT ROUND(download_bytes_sec * 8 / 1e6, 1) AS mbps
227+
FROM speed_history_raw
228+
WHERE interface_name = 'Wi-Fi 3'
229+
ORDER BY download_bytes_sec
230+
LIMIT 1 OFFSET (SELECT COUNT(*) * 95 / 100 FROM speed_history_raw
231+
WHERE interface_name = 'Wi-Fi 3');
232+
```
233+
234+
Beyond 24 hours this is not possible from the stored columns, by design — see *no `min` column*
235+
above.
236+
237+
---
238+
239+
## Schema versioning
240+
241+
`metadata.db_version` tracks the schema; the current version is **7**. On launch, a database at an
242+
older version is backed up and migrated forward one step at a time (v2→v3→…→v7). Migrations are
243+
additive — no version has ever dropped a column.
244+
245+
| Version | Added |
246+
|---|---|
247+
| 3 | covering indexes, `metadata` table |
248+
| 4 | `sample_count` on the rollup tables |
249+
| 5 | hardware statistics tables |
250+
| 6 | `hardware_stats_hour` |
251+
| 7 | `usage_counter` (the data-cap odometer) |
252+
253+
**If you add a table or column:** bump `_DB_VERSION` in `core/database.py`, add a `_migrate_vN_to_vN+1`
254+
method, and update this file. Old databases must keep working — a migration that assumes a column
255+
exists will break every existing install.
256+
257+
---
258+
259+
## Maintenance
260+
261+
A background pass runs periodically and does three things: roll each tier up into the next, delete
262+
data past your retention setting, and truncate the write-ahead log.
263+
264+
It also runs `VACUUM` to reclaim space, at most once a day and only when there is real slack to
265+
reclaim. Rolling data up deletes rows, and deleted rows leave free pages behind; without a periodic
266+
VACUUM the file keeps its high-water mark forever.
267+
268+
If you ever want to reclaim space by hand — with NetSpeedTray closed:
269+
270+
```sql
271+
VACUUM;
272+
```
273+
274+
---
275+
276+
## What is deliberately *not* stored
277+
278+
NetSpeedTray does not record what it cannot measure honestly:
279+
280+
- **Per-application bytes.** Windows does not attribute network bytes per process without a kernel
281+
driver. The Monitor shows per-app *connections*, which is real, rather than per-app speeds, which
282+
would be a guess.
283+
- **Which traffic was LAN and which was internet.** Adapter byte counters carry no addresses, ports
284+
or routes — the information simply is not there to filter on.
285+
- **Anything that leaves your machine.** No telemetry, no accounts, no uploads. The latency probe
286+
pings your own gateway by default; pinging a public host is opt-in and you name the host.

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -259,6 +259,7 @@ You probably grabbed an unsigned dev build. The official releases ([Setup.exe /
259259
### Security & Privacy
260260
- **Digitally signed** by the [SignPath Foundation](https://signpath.org/) - no SmartScreen warnings
261261
- **100% open source** (GPLv3) - no ads, no tracking, no telemetry · [Privacy Policy](privacy.md)
262+
- **Your history is a plain SQLite file you can query yourself** - schema, units and example queries in [DATABASE.md](DATABASE.md)
262263
- **PII obfuscation** - logs auto-redact paths, IPs (incl. IPv6), MACs, hostnames, and interface GUIDs
263264
- **Latency boundary** - off by default; gateway-only unless you explicitly opt in to a public host you name
264265
- **Verified updates** - the in-app updater authenticates the installer (Windows `WinVerifyTrust` + a publisher pin) before running, with a GitHub fallback

0 commit comments

Comments
 (0)