correctness - #1
Closed
siddharthanagula3 wants to merge 17 commits into
Closed
Conversation
npm ci was failing. The lockfile did not match package.json, so the whole cloud half of CI never got past the install step, which means typecheck, the tests, the portal lint and the portal build have not actually run on a push in a while. Regenerated the lockfile. npm ci now exits clean and the checks behind it run. (cherry picked from commit 133f3aa47bfe5ea8d5178b8c9827267125d21fa1)
Claude Code writes one line to its transcript for every block of a reply: the thinking, the text, each tool call. Every one of those lines repeats the same token count. We were adding them all up, so every number was too big. On a real machine that made the estimated value $40,687 when the true figure is $18,913. Output tokens were 4.4x too high, requests were double, and everything downstream inherited it: the dollar tiles, the per model split, the daily chart, the leaderboard rank, the tiers and the streaks. A reply is now counted once, keyed on its message id. Tool calls still count per line, because each of those lines genuinely carries a different tool call, and dropping them along with the repeated usage would have swapped one wrong number for another. Checked against Claude Code's own totals in stats-cache.json. We were 4.9x above its output figure before this change and within 12% after, which is the normal gap between the two sources and is documented in the README. The stored index is rebuilt once when you upgrade, because everything already in it was counted the old way. (cherry picked from commit 3ea4b83d2028319f59c595cfba36cdead631220b)
Streaks and time windows worked by subtracting 24 hours at a time. Twice a year a day is 23 or 25 hours long, and on those days the arithmetic landed on the wrong date. A streak that crossed a daylight saving change quietly reset, and a seven day window silently gained or lost a day. It now steps through calendar dates instead, so the length of the day does not matter. The test pins a timezone on purpose. Our CI runs in UTC, which has no daylight saving, so the old broken code passes there whether it is fixed or not. A guard that only fires on somebody's laptop is not a guard. (cherry picked from commit 7374cbb876c46a5f16e542e50d8fb8ad762cb4ab)
…enCode The agent loaded every Codex and Copilot session file into memory in full, on every cycle, and it did it twice over because two separate pieces of code each asked whether those tools were installed. On a real machine that is 11 GB across 292 files, one of them 761 MB. Peak memory was 3.27 GB for a program meant to sit quietly in the background. It now reads a line at a time into a reused buffer, remembers files whose size and modified time have not changed, and asks the installed question once. Peak memory is 135 MB and a full pass takes 78 seconds instead of 142. The comments in these files had claimed this behaviour for a while, and now it is true. The cache lives in the process rather than on disk on purpose. The agent's writes are listed in the consent screen and hashed into what you agreed to, and a cache file would widen that promise to buy something a long running loop gets for nothing. Also in here: Codex reported "292 sessions, 189 turns", which cannot be right. Sessions counted every session, turns only counted the newest 60, because the agent sends a shortened list and the board added up what it could see. The real turn count is sent now. It is 996. Codex has been writing a cache_write_input_tokens field that we dropped on the floor, so cache writes were invisible on the Codex board while the Claude board priced them. Now captured. Codex also reports your plan type, whether you have hit a spend limit, and your credit balance, all sitting on disk unread. Now collected. OpenCode has a real reader. It was one of six tools the board claimed to read while having no code to read them with. (cherry picked from commit 15889b5bb9a408eefbb98de6111dff779c724a33)
Codex had no cost anywhere on the board. We ship rates for Anthropic models only, and inventing OpenAI prices would be a guess dressed up as a measurement, which is the one thing this project has always refused to do. So there is now a rate card you fill in yourself at ~/.tokenhud/rates.json, and a Codex value panel that either shows the cost or explains plainly what would give it one. Edit the file and the next reading picks it up. Two rules stop this being a way to cheat. Your rates can only fill gaps. They cannot restate a rate we already ship, so nobody can inflate their Claude spend by editing a file. A figure priced from your own card stays on your own board. Shared and public leaderboards use the rate card that ships with the build, the same one for everybody, because estimated value is a ranked number and ranking strangers on a value any of them can edit would turn the board into a typing contest. Every cost now carries where it came from: our rates, your rates, credits, or not priced at all. "$0.00" used to mean free, no rate available, and included in your subscription, all at once, which is the most misleading thing a usage dashboard can print. (cherry picked from commit 3a98edeef5aee917305d34fee4267d3df8c5e8c7)
…them Some integrations effectively said "here are five steps, go and do them". Almost nobody does, so the tile sits there forever saying one step away. There is now a command that does it: tokenhud-agent enable gemini-cli It reads your existing settings, merges the telemetry block in rather than overwriting the file, takes a backup, shows you the exact diff and asks before writing. It refuses rather than clobbering if the file will not parse, and running it twice does nothing the second time. Aider works the same way. There is also enable --print, which produces the same change as JSON so a coding agent already on the machine can apply it. The paths this writes to are listed on the consent screen in their own section, marked as things that never happen on their own. They are deliberately kept out of the everyday digest, because a command you choose to run is not the same as the reporting loop, and hashing them together would ask everybody to re-consent for a file they may never touch. Also honest now about five tools the board used to describe as installed and readable. There is no code here that reads them, so they say so, and the wording puts it where it belongs: nothing on your side is switched off, we have not written the reader yet. A test now fails if any tool claims to be readable without a module in this crate that reads it. (cherry picked from commit 0960ba9943529c58c87e6951bf30e9d64e65c148)
…two configs The cloud kept exactly one row per machine, overwritten on every heartbeat and thrown away after seven days. There was no history at all. Everything the board showed as "90 days" was riding inside whatever the agent could rebuild from the local disk each time, so if an agent stopped, the history evaporated within a week, and "all time" was not a real number. There is now a small daily rollup per machine, holding only the aggregates that are already allowed to leave, written so that sending the same day twice overwrites instead of adding up. It expires after roughly 400 days rather than never. There is also a public endpoint for the GitHub star count. The page cannot ask GitHub directly because our own content policy blocks it, and asking on every page view would run into GitHub's rate limit anyway. So the server asks once in a while, caches the answer, and keeps serving the last good one if GitHub is down or has cut us off. Zero is a real answer and is treated as one. Two configuration bugs that would have bitten on the next deploy: The content security policy did not allow api.tokenhud.com, which is the whole reason the custom domain setting exists. Turning it on would have broken every call the portal makes. The local development origin was written as port 5173 in two places while the dev server actually runs on 5174, so signing in locally against the cloud was refused before it started. Environment variables the functions need are now checked at startup with a clear message, instead of being cast to strings and failing later as a 500 with no explanation. (cherry picked from commit d7d2d265a0a70248f84c1504b2b4cb0c1a7de20f)
The signed-in app was built twice. Portal.jsx and SelfHost.jsx each contained their own copy of the same shell: the same rails, the same collapse state, the same page switching, differing only in where the data came from. Anything added to one had to be remembered into the other, and four things had already been forgotten. There is one shell now, and both pass it an adapter. That fixes the four forgotten things by construction: Removing a machine from the cloud board did nothing at all. It searched for a field the cloud data has never contained. The upgrade prompt could not appear in the cloud, because the value it depends on was never passed in. One callback was wired to an empty function. The cloud settings page was a heading and a sign out button, while the rail beside it promised connection details, appearance, public links and server information, the documentation described six sections, and the changelog said it was not a stub. The real settings page existed the whole time and was wired up for self-host only. It is shared now. The duplicate sidebar goes with it. There were two rails side by side taking 454 pixels, styled the same, each with its own collapse button, with Machines appearing in both. The outer one is an icon strip now, which is what the README always said it was. The cloud portal also stopped describing signed in users as "Self-hosted", which was a hardcoded string. The opt-in for the public leaderboard was unreachable. It only rendered in a branch that is skipped whenever the board is embedded, and the board is always embedded, so no cloud user could ever join and the public page could never fill up. The server side has been finished this whole time. It lives in settings now. There was no error boundary anywhere in the app. A dashboard that renders whatever an agent sends it had nothing to catch a bad field, so one would blank the page. There is one now. The icon set was trapped inside a file that can no longer render. Moved out. (cherry picked from commit d782ee77759d019bd324024f4e727d31d9c01db3)
The in-account page ranked your own laptops against each other, with medals, a podium, tiers and a rank badge. That is personal analytics wearing a competition's clothes, and it is a different thing from the opt-in board where accounts are ranked against other people. They are separate now. Your own machines are compared without the medals, and the word leaderboard is kept for the public board, which publishes nothing until enough accounts have joined for a ranking to mean anything. Usage can be grouped by app and by model, not just totalled. Tokens are shown as what they are made of rather than one number, because Codex totals are almost entirely cached input while the Claude chart deliberately leaves cache reads out, so the two headline figures were never comparable. Trending has a published formula and a minimum volume, so a tool with almost no usage cannot top the chart by going from two tokens to twenty. There is a methodology section saying what is counted, over what period, how fresh it is, and what it does not prove. Three smaller things. The tier names were written out again by hand in the UI instead of being read from the one place that defines them, which would have drifted the first time anyone changed a band. Three tiles on the live page read a field that is always empty on this path, so they showed zero forever rather than saying they had nothing. And one class name in the markup had no styling behind it at all. (cherry picked from commit 9dbd6e74a71d196366f1577d8f50758b441c6a9c)
The last thing on the page told you to run two scripts that are not in the repository. Anyone who followed it got an error. Fixed to a command that works. The page claimed 9, 26, 4, 7 and 2 supported agents, in five different places. All of them are now derived from the one file that actually knows, so they cannot disagree again. It also described a local-only tool and then had a button that signs you into a hosted service, which was never mentioned. Both halves are on the page now, with what leaves in each case, because the local half is the strongest thing here and the hosted half is opt-in. Presenting only one of them turned a real strength into an apparent contradiction. It said the board is pushed to over SSE. That is true of the local board and not of the hosted one, which polls. The structured data for search engines had been lost in a rewrite. Two files were still sitting in the folder with nothing pointing at them, so search engines saw none of it. It is inline in the page again, and the sitemap lists both pages rather than one. The GitHub star count is on the page, reading it from our own server rather than calling GitHub from the browser, which our content policy blocks and GitHub rate limits by address anyway. Zero is treated as a real answer, and if the count cannot be fetched at all the widget removes itself instead of showing a dash. There is a leaderboard section, which renders an invitation rather than empty boxes when nobody has opted in yet. The dashboard is loaded only when it is needed. Every visitor used to download the whole application, charts, enrollment flow and all, to read a landing page. The main bundle is 228 kB where it was 547 kB. (cherry picked from commit e81115a8322511048f22bd021533dd7d6250a8e8)
…s ring Two colours were used in the stylesheet that were never defined and had no fallback, so those rules did nothing at all and the affected elements quietly inherited whatever was around them. Nothing in the signed-in navigation showed a focus ring, so anyone using a keyboard could not see where they were, while the marketing page did it properly. There is one now, checked for contrast in both themes. The two rails were built to different measurements: different row heights, different collapsed widths, and two text sizes below the smallest one in the type scale, neither of them landing on a whole pixel. The topbar height was written out by hand in four places that all had to agree. These are all named values now. The board had three different left edges depending on which section you were in, one of them a measurement meant for a landing page. One value now. On a narrow screen the inner rail was hidden with nothing in its place, so there was no way to switch machines at all on a phone. Cards had one visual weight, so a page of thirty of them gave the eye nothing to land on. They now have a hierarchy. The chart colours are untouched. They are checked for contrast and colour-blind separation in both themes, and that is worth keeping. (cherry picked from commit 3e58bc5a37f6c6841d9b742c1709ffb2f1514ca0)
Same file in four places so that whichever tool someone uses picks it up. No em dashes, anywhere. Use a comma, a full stop, a colon, brackets or a plain hyphen, and if none of those fit the sentence wants rewriting. With a warning that the long lines in comment headers are box drawing characters rather than dashes, so nobody replaces them and mangles every header in the repository. Nothing hardcoded that can be derived. A literal is a second source of truth and the two drift. The examples are from this codebase rather than invented: the landing page claiming five different agent counts, a rate list doubling as the set of models we are willing to price, a binary size quoted differently in the README and on the website. Two values that must agree belong in one place both read, and if they genuinely cannot share a definition, a test should fail when they disagree. Commit messages in plain language that explain what changed and why. Say the number, name the file, state the effect. And the testing rule this work earned. A test that asserts nothing is worse than no test, because it reports a guarantee that does not exist. Three of those shipped here. A regression test has to fail against the code before the fix, and anything the CI environment would hide has to be pinned in the test itself. The existing rule about not attributing commits to AI tools is carried over unchanged. (cherry picked from commit ae6512b2bfe54d16bdd89492c30dc74f58758fc0)
… em dash Two things at once, because they touch the same files. The documentation had drifted. It described a rate card that covers models it does not, counts of supported tools that no longer matched the catalogue, a settings page called finished that was a stub, a save to PDF feature that cannot be reached, and routes that had changed. The architecture notes still described a layer that was removed. Where the code and the prose disagreed, the code was usually right and the prose is now updated to match it. And every em dash and en dash in the repository is gone, replaced with ordinary punctuation, as the house rules now require. That is most of the size of this change and none of the meaning. Two of them were load bearing in an unexpected way. Replacing a dash at the start of a wrapped comment line turns it into what the Rust linter reads as a bullet point, which fails the build with warnings treated as errors. Those two sentences are rewritten rather than left with a hyphen doing the wrong job. (cherry picked from commit 9652b49de2e3dfedc8893b1c7d07d022b313e7c3)
Nothing behavioural here, just what was needed to keep the build clean. A helper in the Devin reader is now visible to the rest of the crate, because the OpenCode reader added in an earlier commit calls it rather than making a second copy of the same thing. Two uses of map_or in the server rewritten as is_some_and, which the current Rust linter asks for. An unused import removed from a test file.
Several tests change environment variables to point the code at a scratch directory. They run in one process, at the same time, without any lock, so one test could be reading another test's setup. They take a shared lock now, the same one the machine tests already used. Worse, one of them set HOME to a path that does not exist and never put it back, with a comment saying the harness restores it between tests. It does not. Every test that ran afterwards was looking at a machine with no home directory, which is why the OpenCode reader could not find anything in its own fixture. It is restored by hand now.
This test file used the single threaded test runtime while every other test file here uses the multi threaded one. The reads in it block the thread they run on, and on a single threaded runtime that is the only thread there is, so the server it was waiting for never got a chance to reply. The test hung rather than failing, and it took the whole server suite with it. It runs on the multi threaded runtime now, and the sockets have a read timeout so a connection left open cannot stall the run either.
Signed-off-by: Sankara Reddy Telukutla <sankara@reddy.sh>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix