JSON schema validation errors

[suhrig]

Hi @bentsherman,

Thanks a ton for putting so much effort into this plugin! I am currently searching for a utility to track bioinformatics pipeline results (generation) and this plugin seems super useful to handle all of my Nextflow pipeline outputs, which covers a great deal of my use cases.

While testing the plugin I ran into a few issues regarding the schema validation of BioCompute Objects produced by the plugin:

• When I run the tool bcotool validate on a BCO file from nf-prov, it flags several fields, namely: instance['provenance_domain']['derived_from'], instance['provenance_domain']['obsolete_after'], instance['provenance_domain']['embargo'], instance['provenance_domain']['license']. In these cases, the problem is that they are set to null instead of an empty string. Moreover, the field embargo should be a pair of strings (start and end date).
• While formally valid according to the JSON schema definition (i.e., bcotool validate does not flag it), I think the field execution_domain.script is not formatted the way it is meant to be. According to the spec, this should not be an array of plain strings, but an array of URI objects (see here, which links to there). Currently, the pipeline puts just a string of the main.nf script as the value, e.g., "execution_domain": { "script": [ "file:///tmp/bco_test/crisprseq/main.nf"], ... }. I think it is supposed to be an array of uri objects, i.e., "execution_domain": { "script": [{ "uri": { "uri": "file:///tmp/bco_test/crisprseq/main.nf" } } ], ...}. See supplementary file S1 from a demo publication by the committee of the BCO standard for an example of how this field is meant to be used.

Do you think this could be fixed in the plugin? Happy to contribute a PR if you agree to my suggestions.

Here are the example bco.json file and the output from bcotool validate from my test case (using nf-prov 1.7.0 on the nf-core/crisprseq pipeline).

Lastly, I have a generic question which is not directly related to the plugin, but given your experience in this field, you may have advice: Ultimately, I am not only interested in tracking the provenance/execution of pipelines, but also in rerunning them with the same parameters. Ideally, I am looking for a command that takes a bco.json file and recreates a (nextflow) command with the same parameters as were used for the pipeline run associated with the BCO. The actual command that was used to run the pipeline is not captured in the JSON file. However, the info should be all there in the bco.json file. But I am not aware of a tool to convert it back to a nextflow command. Can you advise of a utility for this purpose or could the nextflow command be recorded automatically by nf-prov in the bco.json file, too (if yes, where)?

Best regards,
Sebastian

[bentsherman]

Hi @suhrig , thanks for the notes. Those validation issues seem simple enough to fix so feel free to submit a PR. If you could include test cases to run in CI that would be ideal

As for your generic question, I'm not sure that you can reconstruct the actual command from the BCO manifest. You might want to ask the BCO maintainers about this more generally, as I recall there was some discussion about supporting re-execution but I think they decided not to prioritize this.

You might have better luck using Nextflow's built-in data lineage. It saves a WorkflowRun lineage record for each run, and it likely has all the necessary information to reconstruct a launch command. You can also get the launch command from nextflow log, for what it's worth