Small Agent

A lightweight async log parser. It reads raw log lines, parses them into structured records, buffers them in memory, then ships them asynchronously over gRPC once its internal bus reaches 5 MiB.

The lines it is built for are the ones industrial machines write: a stream of telemetry readings with occasional text mixed in, in files large enough that reading one into memory is not an option.

Time, timestep, iteration, arc status:  21.319 ms, 8.252E-6, 1542600, 1
***** WARNING: Pressure limit @ 0.1 bar has been detected:
Post-processing results...DONE

The shape of it

flowchart LR
    F[("machine.log<br/>any size")]

    subgraph agent ["small-agent — one process, two tasks"]
        direction LR
        subgraph reader ["reader task"]
            direction TB
            L["file::lines<br/><i>one line at a time,<br/>never collected</i>"]
            P["parsing::parse_line"]
            B["Bus<br/><i>accumulate until<br/>5 MiB of payload</i>"]
            L --> P --> B
        end
        Q(["queue<br/>depth 2"])
        S["shipper task<br/><i>one batch in flight</i>"]
        B -- "full batch" --> Q --> S
        Q -. "full queue<br/>stops the reader" .-> B
    end

    C[["collector<br/>gRPC"]]
    S -- "Ship(Batch)" --> C
    C -- "Ack" --> S
    F --> L

Loading

Memory is bounded by the box, not by the file: one bus plus at most two queued batches. When the collector slows down the queue fills, the reader blocks, and the heap stays flat — the dotted arrow is the whole backpressure story.

How one line is classified:

flowchart TB
    IN["a line"] --> BLANK{"blank?"}
    BLANK -- yes --> EMPTY["PEmpty<br/><i>costs nothing</i>"]
    BLANK -- no --> SEV{"subject is one field<br/>and says WARN / ERROR / FATAL?"}
    SEV -- yes --> WARN["PLog(Warn)"]
    SEV -- no --> SHAPE{"equal arity across the colon<br/>and every value a number?"}
    SHAPE -- yes --> EV["PEv(Event)<br/><i>keys read off the line</i>"]
    SHAPE -- no --> INFO["PLog(Info)"]

Loading

Wording is consulted before shape, and that order is the whole design. ERROR: 500 passes the shape test perfectly — one key, one value, 500 is a number — so shape alone would file the alert as a metric named ERROR and nobody would see it again.

Run it

Two terminals: the agent ships, and something has to be listening. Start the collector first — it blocks, which is what a server does.

cargo run --release --bin collector -- --dump

Then, in the second terminal:

cargo run --release -- examples/logs/01-telemetry.log

examples/logs/ holds one small file per behaviour worth seeing: telemetry, severity, plain text, the edge cases, and CRLF. --dump makes the collector print every record rather than a batch tally — useful on those files, and a liability on a real one, where it becomes the bottleneck.

The endpoint defaults to http://127.0.0.1:50051 and can be given as a second argument. cargo build is enough on a clean machine — protoc is vendored, not a prerequisite.

The file and the collector are both checked before a single line is parsed, in that order, so a typo in a path says so instead of reporting that some collector is unreachable. Failures print one line to stderr and exit non-zero:

small-agent: ship: transport error: tcp connect error: Connection refused (os error 111)

Why it is shaped that way

Telemetry lines name their own fields, so nothing needs a catalogue. Keys sit left of the colon, values right of it, in the same order; reading them off the line means one parser covers every format the machine emits, including ones nobody has seen — where a table of per-format patterns needs an entry, and a code change, for each new one.

Two conditions, not one, decide it is telemetry: equal arity across the colon and every value a number. Together they make a test that prose reliably fails — ***** WARNING: Pressure limit @ 0.1 bar splits cleanly on its colon and is still rejected, because Pressure is not a number.

A severity must stand alone to count. The marker is looked for in the subject, and only when that subject is a single field: a telemetry header is several comma-separated fields, a severity is one word on its own. That is what keeps position error, tracking error, count: 0.5, 0.2, 10 a measurement while ERROR: 500 stays an alert, without either being listed anywhere.

The bus owns the task that ships for it, so the only way to learn what the collector accepted is finish, which consumes the bus, sends the tail, and waits. There is no order of operations to remember, and no way to wait on a queue still held open by its own sender.

Every batch carries a run id and a sequence number, so a collector can tell one pass over one file from another, and a gap from a duplicate.

Layers

file	hands out lines, one at a time, never collecting them
parsing	turns a line into a telemetry Event, a text Log, or nothing
server	accumulates records and ships a batch at the threshold
errors	the single channel every unrecoverable failure funnels through

The record shapes are defined once, in proto/client.proto, and generated from there. A hand-written twin of every message plus the conversion between them is a second definition to keep in step, and the first thing to drift.

Measured

x86_64, release build, agent and collector on the same host, synthetic logs of the shape above. parsed is what the agent counted; received is what the collector summed independently.

file	max RSS	wall	parsed	received
51 MB	85.4 MB	0.76 s	758 035	758 035
501 MB	94.7 MB	7.18 s	7 488 598	7 488 598
2001 MB	95.1 MB	28.64 s	29 424 514	29 424 514
401 MB, no newline	8.5 MB	0.08 s	1	1

Forty times the file, and resident memory stays in an 85–95 MB band: the bound is the bus, not the file. Nothing is dropped at any size — the two counts are reconciled, not rounded. Throughput is roughly 74 MB/s, 1.09 M lines/s.

The last row is the one that had to be measured rather than assumed. A file with no newline in it is a single line, and a reader that grows a buffer to the next newline will happily hold the whole file: before lines were capped, that same 400 MB file took 823 MB resident and then died on the transport. It is not a contrived input — a truncated log, a failed rotation, or a binary file handed over by mistake all look exactly like this.

Limits worth knowing

• gRPC's default message ceiling is 4 MiB, below the flush threshold. Left alone, every batch the agent sends is refused on arrival, and no unit test catches it because none crosses a transport. Both ends are told the real number; see MAX_MESSAGE_BYTES, set to twice the threshold.
• A line past 1 MiB keeps its first megabyte and the rest is discarded. This is the cap that makes the memory bound true; see MAX_LINE. One malformed line then costs one malformed record instead of the whole run, and losing the tail of a line is a smaller failure than losing the rest of the file. The truncation is silent — nothing marks the record as clipped.
• The connection is plaintext. There is no TLS: log contents, which routinely carry more than anyone intends, cross the network in the clear. That is fine over loopback or inside a trusted segment and is not fine anywhere else. Adding it means a ClientTlsConfig on the endpoint and a certificate the collector presents — deliberately out of scope here, and named rather than left for someone to discover.
• A batch in flight is not yet durable. If the agent dies between a flush and its acknowledgement, those records are gone. Making them survive means a write-ahead log on disk, which is a different program.
• NaN and inf are refused as metric values and the line goes to the text path instead. They parse as f64 perfectly well, and a single one of them poisons every average computed downstream from that point on.
• Backpressure is evidenced by measurement, not by a unit test. The flat RSS across a forty-fold range of file sizes is what shows a full queue stopping the reader. A test that asserts it would have to assert on timing, and a flaky test that guards a real property is worse than a number that can be re-measured.
• errno: 13 is read as a measurement, not an alert. It carries a number under a name and says none of the three severity words. Adding errno to the list would start exactly the catalogue of magic markers this design avoids.
• A single-field line whose one key contains a severity word — position error: 0.5 — is read as an alert. Between losing a metric and losing an alert, this errs toward keeping the alert.
• The threshold counts record payload, not the framed message, so it means the same thing whatever transport sits underneath.
• A telemetry line with one unparseable value ships as text rather than failing the run. An agent that stops on a line it cannot read is an agent that stops on hour three of a large file.

Tests

cargo test

Seventeen, in three files, each pinning a claim this README makes.

bus_test pushes enough records for three crossings of the threshold and asserts that no batch leaves early, none overshoots by more than a single record, the sequence has no gaps, and every record pushed reaches the collector — plus that a tail below the threshold still ships, which is the case every real file ends on.

file_test holds the memory bound. Capping a line is easy; picking the file back up at the right place afterwards is where it goes wrong, so the test that matters feeds an over-long line followed by an ordinary one and checks the ordinary one still arrives intact.

parsing_test pins the classification rules against the lines that motivated them, including the ones that get it wrong if severity is checked after shape.

License

MIT.