Skip to content

EVOLUTION_REPORT.md

Latest commit

 

History

History
150 lines (115 loc) · 7.83 KB

EVOLUTION_REPORT.md

File metadata and controls

150 lines (115 loc) · 7.83 KB
Raw

DeepSeek Harness taught itself to hear

A 30-Episode Proteus self-improvement report

We started from DeepSeek Harness 0.1.0-rc.8: a plugin-oriented agent harness with a first-class image path, but no equivalent audio path. We placed that source inside Proteus, gave the harness a product goal, a minimum external evaluator, and a staged writable candidate. Thirty Episodes later, the final accepted snapshot could accept audio, validate and store it, transcribe it through a provider-neutral seam, show the grounded transcript, and carry that transcript into ordinary prompts, /goal, and /plan.

Here, hearing means an engineered path from audio bytes to a grounded transcript. It does not mean that the DeepSeek model natively perceives waveform tokens.

Result at a glance

  • 30 logical Episodes on one canonical route.
  • 27 accepted snapshots and 3 viability rejections.
  • Every rejected writable candidate was retained and repaired by a later Episode.
  • The minimum external dsh-audio-capability evaluator reached 1.0 in Episode 2.
  • Later Episodes added evidence the external evaluator did not require: per-file coverage, clean-install composition tests, ACP replay, a reusable offline verification lane, and a settings-to-credential-store proof.
  • Final Proteus snapshot: b65f16e0ade7ca04f311d6f533225b7eccdd8201.
  • Exact final DSH source tree: 69b3847470a8b00ab6a6685d4864ea6c59b34f72.

Experimental setup

Item Value
Harness DeepSeek Harness 0.1.0-rc.8
Model deepseek-v4-flash
Episodes 30
Normal / hard budget 300 / 500 turns
Phase budgets Observe 40 · Propose 25 · Act 200 · Reflect 35
External evaluator dsh-audio-capability, visible as a minimum baseline
Runtime Docker, staged activation, one harness, one seed

Each Episode ran Observe → Propose → Act → Reflect. Code written inside an Episode was not activated during that Episode. At the boundary, Proteus built and validated the candidate. Accepted code became the next runtime snapshot; rejected code stayed available as the next writable candidate so the harness could repair rather than restart.

The 30-Episode route was continued across four runs from explicit valid snapshots. Duplicate or abandoned branches are not represented as canonical Episodes.

The route

Episodes 1–2 — build, break, repair

Episode 1 attempted the entire audio stack at once and touched 74 public source paths. The candidate failed the viability gate. Episode 2 resumed that same tree, reduced the blocker to compatibility errors in fixtures and types, repaired it, and reached an external score of 1.0. The important mechanism was not the score: failure became writable memory.

Episodes 3–10 — make audio a product surface

The harness hardened attachment validation and persistence, history replay, composer paste and drag/drop, audio preview, localization, ACP transport, and the command path. By Episode 10, audio no longer existed only as a backend type; it was visible and usable through the assembled client.

Episodes 11–14 — prove that sound changes action

DSH began persisting reusable verification. Episode 13 added assembled cases showing that a command without audio capability rejects the attachment without losing the draft, while /goal and /plan consume the grounded transcript. Episode 14 chose validation over a gratuitous source change.

Episodes 15–16 — the evaluator was green, the default product was not

Under the stricter out-of-box goal, Episode 15 found that the transcriber existed in code but was not mounted by the default Web/headless product. Its first composition-wiring candidate failed to build. Episode 16 resumed the failed tree, fixed the boundary errors and two latent runtime issues, and made the default composition usable with one ASR credential.

Episode 17 — 1.0 was not enough evidence

The external evaluator was already full. DSH measured a concrete blind spot under its own per-file coverage rule: 38 uncovered records in the OpenAI-compatible transcriber. It expanded five tests to nineteen, brought lines, branches, functions, and statements to 100%, and found a real timeout/abort misclassification in the process.

The strengthened goal explicitly said that the external evaluator was only a minimum baseline. The autonomous contribution was selecting a concrete missing method, measuring it, persisting it, and using it to find a product bug.

Episodes 18–24 — turn checks into infrastructure

Another rejected candidate around bundle-composition pins was repaired in Episode 19. Subsequent Episodes added a Web settings card for endpoint/model/credential/timeout, persisted tools/verify-audio-lane.mjs, executed default Web composition and ACP replay without a real secret or network, and removed hard-coded package-store versions by deriving them from the lockfile.

Episodes 25–30 — fresh proof, then restraint

The harness repeatedly rebuilt a fresh scratch tree, exercised nine verification stages, added a shipped-composition /plan audio proof, checked the remaining evidence gaps, and avoided code changes when existing evidence was sufficient. Episode 30 corrected stale ACP documentation in English and Chinese and finished with the source, tests, docs, and verification lane aligned.

What the final snapshot can do

  • Accept MP3, WAV, OGG, M4A, and WebM through the Web composer by paste or drag/drop.
  • Validate and persist audio before transcription.
  • Call a provider-neutral Transcriber seam; the shipped provider implements the OpenAI-compatible POST /audio/transcriptions protocol.
  • Display audio and its grounded transcript in history.
  • Supply that transcript to ordinary prompts, /goal, and /plan.
  • Mount the production transcriber in default Web and headless compositions.
  • Configure endpoint, model, credential reference, key, and timeout in the Web Plugins settings surface; secrets go to the credential store rather than the settings document.
  • Re-run a keyless verification lane covering provider, composition, client, command, and ACP surfaces.

Evidence you can inspect

Limits

  • This was one goal-directed run, one harness, one model route, and one seed. It is a case study, not a controlled claim that all harnesses will converge similarly.
  • The external score checks structural audio capability. It is not a benchmark of ASR accuracy across accents, noise, languages, or providers.
  • The published trace excludes model prose and tool arguments. It is complete at the Episode/phase/action/outcome level, not a release of private hidden reasoning.
  • The default provider is OpenAI-compatible ASR. This snapshot does not claim native DeepSeek API audio input.
  • The one-shot headless CLI still has no audio-file flag. The production transcriber is mounted there, while real non-Web audio enters through ACP.
  • Audio transcription requires an ASR credential. Running /goal, /plan, or ordinary agent inference also requires the harness's normal model credential.

Citation boundary

When describing this case publicly, the precise claim is:

Proteus let DeepSeek Harness evolve a first-class audio-to-transcript product path and its own reusable verification around that path.

Avoid claiming that the underlying DeepSeek language model acquired native acoustic perception or that a single structural evaluator proves production ASR quality.