Cleanup of em_data in prep for more changes - #13
arthurmccray wants to merge 4 commits into
Conversation
… a location field distinguishing shared vs user files
|
Including here the next things to change so we can discuss them (here or on slack) Next PR: the metadata model (breaking)This rewrites every dataset YAML and the public attribute surface.
Non-breaking, same PR: validate the dataset YAML against the shipped JSON The vendor check separates a typo from a new vendor by fuzzy match: After that: search (additive)
Later: config (breaking, at integration)Replace the bespoke |
Correctness fixes, typing, and lint/type-check tooling
Four commits: tooling, a mechanical lint pass, then two rounds of fixes. The linting
is really the main stuff here.
DownloadFutureis renamedDatasetPath, anddownload()now returns one inboth modes, instead of a
strwhen blocking and a path subclass when not.Bug fixes
Fixing DatasetPath pending status: the pending future was stored on the
instance, but pathlib builds a new object for any derived path
(
handle.parent / handle.name,handle.with_suffix(...)). The copy had nofuture, and both
__fspath__anddoneread it withgetattr(..., None),which treats a missing future as finished. A derived path therefore reported
done == Trueduring an active download, and did not wait when passed to areader. Pending downloads now live in a module-level dict keyed by path, so any
path naming that file waits.
Distinguishing shared from user copies:
filepath()searches the shareddirs before the user dir, but
delete()only looks in the user dir. That is thebehaviour we want, but the widget called
delete(), discarded the result andredrew, so on a shared install the button appeared to do nothing. Catalogue
entries now carry
location("user","shared"orNone), and both widgetsshow a different badge and no delete button for a shared copy.
delete()itselfis unchanged.
Fixing subclassing: the generated
__init__usedsuper(self.__class__, self), which resolves against the runtime class and only works whenselfisexactly that class, so subclassing any dataset raised a
TypeError. Classes nowhold their YAML entry as
_specand the base__init__reads it. There is nogenerated
__init__any more.Tooling
[tool.ruff](line length 99,E4, E7, E9, F, I) and[tool.basedpyright](
standard), adevextra, and a pre-commit config.lintjob in CI running both, separate from the test matrix. Both areblocking: the package is at 0 ruff errors and 0 basedpyright errors.
ruff --fix+ruff formatcommit, kept separate from the behaviourchanges. Markdown is excluded so the README is not reformatted.
.pyiended with__all__ = __all__ + [...], which references aname a stub never binds. It is a plain list now.
Breaking changes
Worth a minor version bump.
download()->stror a path subclass, depending onbackgroundDatasetPath(apathlib.Pathsubclass)filepath()->str | NonePath | Noneresult()->strPathDownloadFutureDatasetPathhandle.done()handle.done(property)config.data_dir(),shared_data_dirs(),data_search_dirs()->strPathem_database.get_data_dir()->strPathDownloadableDataset(source, file, ...)DownloadableDataset(**overrides), reading_specOn the two renames:
DownloadFuturedid not implement theFutureinterface(no
cancel(),exception()oradd_done_callback()), and its_futureattribute held an actual
concurrent.futures.Future.doneis a propertybecause as a method
if path.done:is always true, whilepath.done()on aproperty raises straight away.
Settings values are still stored as strings, so existing
~/.em_database/settings.yamlfiles are unaffected.Verification
93 tests pass (7 new), 0 ruff errors, 0 basedpyright errors at
standard.