(mirroring original issue because original was transferred accidently)
darothen
Is this a new feature, an improvement, or a change to existing functionality?
New Feature
How would you describe the priority of this feature request
Low (would be nice)
Please provide a clear description of problem you would like to solve.
Hello Earth2Studio team,
I'm reaching out to assess a collaboration to build an E2S integration with Brightband's NNJA-AI dataset as an alternative to the all-BUFR, original NNJA dataset. NNJA-AI builds on NNJA by re-processing all of the BUFR format data into much more easily accessible Parquet files, which are hosted on Google Cloud Storage in a Hive-partitioned hierarchy that makes it easy to subset by sensor/message type and temporal range. Last year, we released v1 of this dataset, which features 11 datasets, fully re-processed through March, 2025. We also append convbufr datasets (adpsfc, adpupa) as soon as they become available, to support collaborators at NOAA.
We released a small Python SDK to make it easy to pull data from the archive, but it's also possible just to query the Parquet files directly. In v1, we published data with a "flattened" schema relative to the original BUFR messages. These are documented at the previous websites linked here.
We're actively preparing v2, which finishes reprocessing the entirety of the remaining BUFR files (including all the PREPBUFR data with full event stacks) and extends the window of available data through the present. Part of this work will also include expanding the near-real-time processing of the data as it is published upstream. Another important note about v2 - we've developed a significantly streamlined struct-of-array-based schema that will make it radically simpler to write SQL or Polars queries to extract data.
Our goal with the NNJA-AI dataset is nothing less than creating the foundational dataset to use for work in ML-based data assimilation, direct observation processing, and similar applications. We'd be thrilled to help build integrations with Earth2Studio to maximize the audience that this dataset reaches - please let us know what the best way to collaborate on this objective might be!
One related note - we've successfully run HealDA directly from both source BUFR data from NNJA (using our own proprietary tools for efficiently reading these data) as well as preview v2 data. I hope that's a strong motivation to leverage our dataset!
(mirroring original issue because original was transferred accidently)
darothen