EM Data

This is a simple project for aggregating different Electron Microscopy files which are hosted over different sources. It uses pooch to download datasets and should be used as a way to host simple example datasets for method validation.

Data is stored in a file "User/em_database" but this can be also set to a custom location.

List of datasets https://cssfrancis.github.io/em_data/datasets.html

Installation

pip install em-database

Usage

Every dataset is a class under em_database.data. Calling download() fetches the file to the data directory, verifies its checksum, and returns a path handle. Files that are already present are not downloaded again.

import em_database.data as data
import hyperspy.api as hs

path = data.LayeredCuNb4DSTEM().download()
s = hs.load(path, lazy=True)

By default the download runs on a background thread so a notebook cell returns immediately. The handle it returns is the file path, so you can hand it straight to a loader as above — it only blocks at the moment the file is actually opened. Call path.done() to check progress without blocking, or path.result() to wait explicitly. Pass download(background=False) to block and get the path as a plain string instead.

Settings

Configuration lives in a live, matplotlib-rcParams-style object, em_database.settings, seeded at import from ~/.em_database/settings.yaml. Change it in memory for an immediate effect, and persist it to remember the choice across sessions:

import em_database

em_database.settings["data_dir"] = "/big/disk/em_data"  # takes effect now
em_database.settings.save()                             # remember it next time

In Jupyter you can also edit them interactively — display(em_database.settings) renders a panel to set (and save) the data directory directly.

The data directory defaults to ~/em_database. Convenience helpers wrap the common case — set_data_dir persists by default:

em_database.get_data_dir()                       # current location
em_database.set_data_dir("/big/disk/em_data")    # set + persist
em_database.set_data_dir("/scratch", persist=False)  # one-off, in-memory only
em_database.reset_data_dir()                     # back to the default, forget the choice

No environment variable is needed. Only values you explicitly set are written to the file, so the default is never frozen in — it keeps being obeyed even if it changes. For a one-off override (e.g. CI) the legacy EM_DATABASE_DATA_DIR is still honored, and EM_DATABASE_CONFIG relocates the settings file.

Shared, system-wide data

Datasets can be installed once for every user. download() and filepath() look in the shared/system locations first, then your own data directory, and only download (into your directory) if the file is nowhere to be found — so a shared copy is reused instead of refetched. Shared locations come from the EM_DATABASE_SHARED_DIR environment variable (an os.pathsep-separated list), a shared_data_dirs list in your settings, or a system config file (/etc/em_database/settings.yaml, or %PROGRAMDATA%\em_database\settings.yaml on Windows).

Adding a dataset

Datasets are described by a YAML file in em_database/datasets/, one entry per file, validated against em_database/datasets/json-schema.json. The class name is generated from the top-level key:

MyDataset:
  description: What the data is, how it was acquired and how it is calibrated.
  source: https://zenodo.org/records/<record>/files
  checksum: md5:<hash>
  file: MyDataset.zspy
  data_size: 1.2 GB
  technique: 4D-STEM
  license: CC-BY-4.0

Open an issue with the new dataset template, or add the YAML file directly.