Skip to content

Show status of file uploads - #113

Merged
jkalayan merged 2 commits into
mainfrom
106-data-submission-to-invenio-times-out-for-large-files-in-production
Nov 6, 2026
Merged

jkalayan merged 2 commits into
mainfrom
106-data-submission-to-invenio-times-out-for-large-files-in-production

Conversation

@jkalayan

@jkalayan jkalayan commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

This pull request introduces a progress reporting system for uploads and extraction workflows in the BioSimDB interface, improving both backend tracking and frontend user experience. The changes span Python backend modules and frontend JavaScript, enabling cancellation, step visualization, and more informative feedback for users.

Changes

Backend progress tracking and reporting:

  • Added a new ProgressReporter class in progress_state.py to manage file-backed, thread-safe progress reporting, allowing both upload and polling threads to share progress via JSON files. Also added a read_progress helper.
  • Updated run_record_upload in client.py to accept a report callback for detailed progress updates at each step (draft creation, file upload, review submission), and integrated calls to this callback throughout the upload process.
  • Extended the multipart and single file upload pipeline (multipart.py) to accept and propagate an on_update progress callback, which is invoked on each progress update, enabling granular progress tracking for both multipart and single PUT uploads.
  • Modified the Progress class in progress.py to accept an on_update callback and call it after each progress increment, allowing external reporting mechanisms to react to progress changes.

Frontend UI improvements for progress and cancellation:

  • Enhanced showLoadingOverlay in core/ui.js to optionally display a stepper panel instead of a spinner, and added a new createStepper function to render and update step-based progress indicators in the UI.
  • Refactored the extraction workflow in extract/extract.js to use the new stepper UI: each file upload and the extraction step are shown as progress steps, with real-time updates and error/cancellation handling. Added support for cancelling extraction, which aborts uploads and clears partial workflow state.

Other improvements:

  • Updated the webform instructions in schema/webform.py to clarify session loss and large file handling for users.

Testing

  • Added tests for progress state reporter.

Impact

  • Progress can now be tracked and reported step-by-step, with real-time updates displayed in the UI during file uploads and metadata extraction.

Checklist

  • Tests added or updated
  • Documentation updated
  • If dependencies in pyproject.toml changed, the lower bounds were reviewed deliberately

Replace spinners with a live step list for extraction and Invenio submission.
The server writes progress to the workflow tmpdir, which the loading page
polls via a new /progress route. The extraction overlay gains a Cancel button.
@jkalayan jkalayan added the enhancement New feature or request label Nov 6, 2026
@jkalayan
jkalayan merged commit 359bf12 into main Nov 6, 2026
12 checks passed
@jkalayan
jkalayan deleted the 106-data-submission-to-invenio-times-out-for-large-files-in-production branch November 6, 2026 20:21
@jkalayan jkalayan changed the title 106 data submission to invenio times out for large files in production Fix times out for large file uploads in production server Nov 6, 2026
@jkalayan jkalayan changed the title Fix times out for large file uploads in production server Show status of file uploads Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant