Processing a json, keeping some attributes as string and others as a stream

[mbenedettini]

I'm trying to parse a json with the following format:

{
  "name": "data #1",
  "description": "description of data #1",
  "files": [
    {
      "filename": "file1",
      "content": "long string base 64 encoded 1"
    },
    {
      "filename": "file2",
      "content": "long string base 64 encoded 2"
    }
  ]
}

where as output I'd like to retain the content as a stream since it can be very long, so the output structure should be something like so that afterwards I can pipe them into an S3 upload (@aws-sdk/lib-storage):

const filesData ={
    "file1": contentStream
}];

Getting the attributes that are not "files" is easy and I omitted that from the following code, where I'm trying to get the filename as a string on one pipeline and the contents as a stream on the other one. The problem is how to connect them both?

I am thinking of doing something with two promises so that one promise is the filename and the other one is the contents stream and once both are fulfilled I would know that one file has been parsed but I'm not sure that's the best way since I will need a pair of promises for each files entry.

Is there an alternative? Thanks in advance!

const processFilesData = (pipeline: Readable) => {
  const filesPipeline = chain([
    pipeline,
    parser(),
    pick({ filter: /^files\.\d+/ }),
  ]);

  const filesData = {};

  const fileNamesPipeline = chain([
    filesPipeline,
    pick({ filter: "filename" }),
    streamValues(),
    ({ key, value }) => {
      // placeholder for content using filename
      filesData[value] = null;
    },
  ]);

  const filesDataPipeline = chain([
    filesPipeline,
    pick({ filter: "content" }),
    // how to identify the file here so that we can assign the stream to it?
    // also, this pipeline could be processed before the filename one
  ]);

  // Once everything is finished, return filesData somehow
};