Assemble objects in streaming mode?

[WorldSEnder]

It is sometimes necessary, after some filtering and other transformations, to assemble full json objects for other libraries and "terminate" the json object stream. I'm currently using a simple transformer that uses the Assembler to do this, but I was surprised to not find it in the library itself.

function assembleFullObjects() {
    const asm = new Assembler();
    return new Transform({
        writableObjectMode: true,
        readableObjectMode: true,
        transform(chunk, _, callback) {
            asm.consume(chunk);
            if (asm.done) {
                this.push(asm.current);
            }
            callback();
        }
    });
}

Used as

dataIn
        .pipe(parseJSON.asStream({ jsonStreaming: true }))
        // ... some filtering
        .pipe(assembleFullObjects())
       // ... we now have a stream of objects to pass to third-party code

The other direction of fully assembled objects to stream in Disassembler already works for transforming object streams to streams of json event. Is there any reason that Assembler doesn't have this interface that I'm missing?

[uhop]

Did you try streamValues?

[WorldSEnder]

I see. Due to the key in the emitted objects I was under the impression that this would yield subobjects too when reading docs.

Getting rid of that additional counter and using only the value is simple enough on my side. Thanks for the quick response.

[uhop]

I would appreciate your thoughts on how I can improve the documentation, if it was unclear or hard to follow.

[WorldSEnder]

As a user I start with (bytes) a stream of encoded json that comes in as chunks, or (value) an object to serialize. As I understand, the 'internal' representation is (token) a stream of small tokens representing a traversal over one or more json objects. Mentally then I deal with 3 different stream types and want to transform between them.

(bytes) -> (token)

Parser

These are handled by parser, and quite cleanly. Reading the wiki page though contains a bit of information that I don't really need, mainly it explains a lot of the internal token type, which perhaps deserves its own "internals of tokens" pages, but it's clear in its purpose. The options aren't so clear (for example it's not specified what exactly the parser allows between json objects, except that it most likely allows a record separator control character and new lines. I suppose any whitespace is fine and skipped?). The worked example in "Stream of tokens" should show the json sequence it is based on, and the two different encodings for pack{Keys, Strings, Numbers} should use the same data for both to more easily compare.

It is strange that all values are strings, but null, false and true are represented in their parsed form. You can already tell from the name tag anyway, no?

(Since 1.6.0) If you deal with JSONL, you may want to use jsonl/Parser to improve the performance.

This comment is a bit strange, since the linked page shows a deprecation warning. There is also a duplicate comment earlier that qualifies this advice with "where individual items fit in memory" which sounds like better advice.

Streaming options

I don't understand this section at all. It seems like we get duplicate events, i.e. both the packed and unpacked form of each item, in the stream, even listed as "The default". Why would I want this and is this a compatibility option? It also says earlier that "They have no default values" but the table seems to show a default value of true, aynway.

Verifier

Quite clear in it's description. The backgrounder section tells me no real information as a user. I do have a question on intended use case though: Since I can't stick it in between the source and the parser, should I tee my source into a split stream and run the verifier in parallel? In my mind, the verifier could emit chunks as it reads them in and error out when it sees malformed json and sit between source and parser (or replace it). It could show how an error looks, structurally.

Overall, the docs could show the expected console output whenever a console.log appears in the example. I don't have a sample.json around to test them while I read.

(token) -> (token)

Filters, Pick

I'm a bit biased since the context for this word for me is jq. I think you can pick up a lot more filters from them, as well. The filter option as documented here is inconsistent with the example shown above, the other two are not documented. You might as well just link to the FilterBase docs. Let's see here:

packKeys refers to replace and ignore. You probably mean replacement, not sure about ignore.

Replacement also looks a bit weird, in that it returns "array of semantically valid data items". This should say "tokens" instead of "data items". From a usage perspective, I wonder why I can't return a readable stream here that is piped out as a replacement to the filtered value.

It should document that unpacked keys will get assembled in memory. As I understand it, the main point of unpacked keys is to avoid having the whole key in memory, this should be mentioned.

Filters, Replace + Ignore

As said previously, the replacement option here is unergonomic as I need to understand the internal token format. It should be shown that the filter in the example /\bmeta\b/i filters all keys named "meta" anywhere in the json (incidentally also keys named "-meta" or "-META", which is a bit of a hazard, but workable for a known schema). A lot of the time, I tend to filter on the top-level only, so perhaps this could show an example with /^meta/ or a nested one with /^foo.bar/, too?

Streamers

Now, the part that I initially misunderstood. I thought this was akin to jq'q toEntries from the key, value format 😅 . That would make a stream a (token) -> (token) transformer. Because StreamObject does almost that. But I learned now that it's not this, but actually (token) -> (object).

Streamers, StreamObject

The example here could show more complicated nestings to make it clear what it does and combined with the code in the introduction

{"a": 1, "b": "a", "c": [{"lang": "de"}], "d": {"foo": null}, "e": true}
// ^ in sample.json
const {streamObject} = require('stream-json/streamers/stream-object.js');
const fs = require('fs');

const pipeline = fs.createReadStream('sample.json').pipe(streamObject.withParserAsStream());

pipeline.on('data', data => console.log(data));
// Output:
{key: 'a', value: 1}
{key: 'b', value: 'a'}
{key: 'c', value: [{"lang": "de"}]}
{key: 'd', value: {"foo": null}}
{key: 'e', value: true}

On naming, I find the "streamer" a bit of a misnomer, since it assembles the objects. It seems that most of them combine two operations into one: Restructuring the input token stream (which could stay a token stream) and then assembling the resulting token stream into values. Let me explain a bit more

• StreamObject expects a single top-level json object. It then emits for each (key, value) of that object a { key, value } js object with the fully assembled subobject. Instead, there could be a (token)->(token) transform that expects a stream of json objects, and for each emits an array with objects of the { key, value } form. Notably it doesn't need to reassemble these objects, it only needs to replace a few {start, end}Object tokens with {start, end}Array and emit some extra tokens wrapping each key value pair in subobjects. This is not the full pipeline to understand StreamObject, let's see StreamArray first
• StreamArray is very similar, except it transforms a top-level array and uses indices as keys. The missing piece to recover the current format in a streaming fashion would be a flatten operation, which expects a stream of arrays on the top-level and strips that top-level emitting their entries as is (without keys).
• StreamValues is the most straight-forward of them all, it just wraps the input in an extra { key, value } layer with the index as key. I wonder though, why there is no way to do the (token)->(object) transformation without any extra wrapping such as this extra key?

There is a warning on the streamer doc pages (at least StreamArray which is mentioned first) which I read to conclude that they are actually unsuitable:

Warning: it cannot be used with a stream of objects produced by a JSON Streaming source, or Pick if it picks more than one object.

Well, I need json streaming. So I looked at

(object) -> (token), Disassembler

This one is super clear to me in what it does and what to expect. No notes, and I don't need to understand any internals to use it. Very cool!

Well, I'm looking for the inverse of that, so what's more natural to look for than Assembler?

Assembler

Despite the naming, its purpose is clear from the docs page, but I got a few notes on functionality. There is surely a good reason not to be a writeable stream. As I see it, Assembler.connectTo and the simple onDone callback offer no way to apply backpressure to the stream it is reading from if further processing is done asynchronously and not as part of the onDone callback. Which I suppose is a price for dropping out of the streaming API but it's not a good sink really. The back pressure aware API would probably wait for the consumer to clear the current object when it is done somehow before consuming further from the read stream.

Stringer

Could be more clear if it accepts multiple top-level objects in its input token stream. It's clear and serves the purpose of (token) -> (bytes). What is not clear to me is why it can't automatically figure out the options of the token stream it consumes and how - in a complicated pipeline - I the user am supposed to keep track of the needed use* options to pass. It would also nice to accept an encoding to convert straight to buffer chunks, but that's easy enough to convert, or at least I will model after stringerToFile for piping to a fetch body.

Intro by examples

If I would have seen the output from the different examples, I would have perhaps realised sooner that I was basically asking for the "JSON Streaming and JSONL" example 😰 As I said initially, I was not sure what "each value" was referring to and thought it might mean each recursive subobject (silly in hindsight). Not sure if saying "each top-level value" is better.

I'm already spending a bit too much time fussing over details here, perhaps I was just silly.

A simple action would be to include the expected input and output in each example, since I couldn't be bothered to type out the sample files the wiki refers to data.json and other json data to execute the samples locally :/

[uhop]

I updated README and the wiki to address some of your concerns. I would appreciate if you look them over to see if they work for you better. Additionally I filed some improvement ideas I plan to work on later.

Thank you again for taking time and writing a detailed feedback!

[WorldSEnder]

The notes in Assembler about streamers, the signatures and the caveat about null why values are wrapped in { key, value } objects are all very nice improvements! Thanks for the effort.

[uhop]

3.6.0 (to be released soon): replace's replacement now takes any plain JS value (disassembled for you); null means JSON null.