Provenance of data accessed through an API which may require authentication

[thomaszwagerman]

Taking note of guidance on downloadable datasets.

I'm wondering if there are any best practice examples on how to represent data provenance when data is accessed through an API endpoint (which may require authentication through some sort of key like a Personal Access Token).

My use case is that I would like to represent a pipeline which programmatically accesses data from the Copernicus Climate Data Store (for example ERA5 single levels), performs some calculation and creates a new data output. This "new" data output could be ingested into another environmental forecasting model (or Digital Twin system), so an ROCrate is potentially a good way to pass on provenance in such an interaction. Ideally a description of this captures the endpoint used, query used (i.e. geographical area, timespan), license, date accessed, DOI, and the authentication without actually attaching credentials.

I'm new to ROCrate, so my interpretation of how it should work might be wrong, but I'm considering the following approach:

I interpret a script that accesses an API as software used to create files, and describe a script and configuration file (which includes information on the query) as a createAction instruments where the object is a detached Dataset that contains information on the endpoint and result the data downloaded.

        {
            "@id": "Download ERA5 data",
            "@type": "CreateAction",
            "agent": {
                "@id": "https://orcid.org/0000-0000-0000-0000"
            },
            "name": "Downloading required ERA5 data",
            "description": "Script downloading the era5 land sea mask (era5_lsm.nc) and mean sea level pressure data (data/ERA5/monthly/era5_mean_sea_level_pressure_*.nc), referencing ENVS to obtain query parameters.",
            "instrument": [
                {
                    "@id": "src/00_download_era5.sh"
                },
                {
                    "@id": "ENVS"
                }
            ],
            "object": {
                "@id":"https://cds.climate.copernicus.eu/api"
            },
            "result": {
                "@id": "data/"
            }
        }

Where `"data/" is described as downloaded:

        {
            "@id": "data/",
            "@type": "Dataset",
            "description": "Folder containing ERA5 land sea mask (era5_lsm.nc) and ERA5/monthly/era5_mean_sea_level_pressure_monthly_*.nc files.",
            "encodingFormat": "application/netcdf",
            "name": "ERA5 Data",
            "type": "FormalParameter",
            "valueRequired": true
        },

and "https://cds.climate.copernicus.eu/api" is described as a detached dataset (yet to crack this one really, but should contain license etc). After this there are further scripts which process "/data".

I guess my main questions are:

1. Is is good practice to represent API access explicitly like this?
2. If this step is part of a larger computational workflow, does it need to be represented separately in this way?
3. Should guidance on API authentication be linked, or a placeholder file be attached? This would not work "out of the box" without some manual user configuration.

[HarperZ9]

I think keeping the API acquisition as its own CreateAction is useful here, especially because it is the boundary that fixes the query, access time, license context, and exact bytes used downstream.

A shape I would try against the current RO-Crate 1.3 guidance is:

• instrument: the SoftwareSourceCode or workflow step that performs the request, plus a credential-free query/config file
• object: the remote dataset or endpoint and the query/config entity
• result: the local NetCDF file or directory actually produced
• agent: the person or organization that authorized the acquisition
• endTime and actionStatus: the observed completion time and status
• license and DOI: properties of the remote dataset entity, rather than of the action

For authentication, I would not attach a token, a populated credential file, or a URL containing credentials. I would link to the provider's authentication instructions and record only the mechanism or local credential locator, such as "CDS API key supplied from environment variable CDSAPI_KEY." If that needs a machine-readable form, I would make it an explicit profile choice rather than implying it is a core RO-Crate authentication term.

A minimal action could look like this:

{
  "@id": "#download-era5-2026-07-16",
  "@type": "CreateAction",
  "name": "Download the selected ERA5 subset",
  "agent": {"@id": "#authorized-agent"},
  "instrument": [
    {"@id": "src/00_download_era5.sh"},
    {"@id": "config/era5-query.json"}
  ],
  "object": [
    {"@id": "https://cds.climate.copernicus.eu/api"},
    {"@id": "config/era5-query.json"}
  ],
  "result": {"@id": "data/era5-subset.nc"},
  "endTime": "2026-07-16T21:00:00Z",
  "actionStatus": {"@id": "https://schema.org/CompletedActionStatus"}
}

In a larger workflow I would still keep this action separate when reproducibility depends on the remote snapshot. Flattening it into the later calculation step would lose the distinction between "bytes acquired from the service" and "derived output produced from those bytes."

I arrived at this while testing an experimental Gather-to-RO-Crate mapping. Gather records source, ref, method, acquisition time, and a content SHA-256 for authenticated JSON API results, while reading the token from the environment by name and never writing it to the receipt or URL. A conservative mapping is ref to the remote object, fetched_at to endTime or the web entity's sdDatePublished, method to the action/instrument description, and the content digest to the result's integrity metadata.

If this direction matches the community's expectations, I would be happy to prepare a credential-safe minimal crate from a synthetic API request for review. Gather is fair-source rather than OSI open source: https://github.com/HarperZ9/gather