Draft: UI stability improvements and data catalog updates by theoellingsen · Pull Request #37 · ACCESS-NRI/MED-live-diagnostics

theoellingsen · 2026-08-18T02:23:10Z

Introduces stability improvements to the UI widget and updates the data catalog ingestion. Tagging @headmetal @charles-turner-1 @Whyborn just so you can have a quick look if you get the chance.

Key Changes:

Updated data.py to utilise the use_datastore function from the ACCESS-intake catalog package, removed scheduler set up.

UI Stability improvements:

Replaced pn.bind() with dedicated buttons to trigger dataset plotting.
Pre-loaded the dataset selector UI upon widget initialisation (toggling visibility when required) to resolve reliability issues.
Separated the plotting logic into two distinct functions for reference and user datasets to accommodate their different handling requirements.

Currently working on updating the plots so that some more useful plots can be generated, with more general bug fixes also to come.

…he intake catalogs use_datastore function Making changes to _check_for_new_data and _build_new_catalog to use the intake catalogs use_datastore function, rather than using bespoke functions.

Adds logic to convert AliasedESMCatalog to raw ESMIntake catalog object if passed into the _build_data_object() function. Also allows for the plotting of reference datasets that are single model variants, previously only plotted datasets that had multiple model variants.

Improve reliability of dataset selection appearing when running widget Force widget synchronisation rather than relying on .append() Hopefully will stop issue with dataset selection not appearing as it will synchronise every time. Revert "Force widget synchronisation rather than relying on .append()" This reverts commit 47716bd.

setting figure_exists to true

Splits the plot_dataset function into 2 different functions, for user dataset and reference dataset. Update ref_figure_exists variable in correct spot so that the variable change in for the ref figure, should trigger the update. It doesn't actually fix the problem. Fixed some typos etc., as well.

Change from using pn.bind() function for updating plots, as it was not reliably working and seemed to be a major cause for the instability of the widget. Added plot dataset buttons instead, so the user can select a variable and decide when to replot. This appears to work much more reliably. Also made updates to the user feedback, so that there is more information when data is being loaded, etc.

Remove redundant code left from scheduler, all stuff that was unreachable before anyway

charles-turner-1

Left a couple of minor suggestions and style things.. looking good though!

charles-turner-1 · 2026-08-18T02:53:46Z

+# Testing the _build_new_catalog function with a non-existent directory and a model type of "OM2"
+#_build_new_catalog("blah not a directory", "ESM15")
+
+session = med_diagnostics.session.CreateModelDiagnosticsSession(model_type='CM2', model_path='path/to/your/live/model/data/output', period=20)


I figure we'll need to create some sort of mock for this - again probably one to talk about next week

Co-authored-by: Charles Turner <52199577+charles-turner-1@users.noreply.github.com>

Add new plot options, allows user the choice between plotting a line graph and a heatmap. Adds choices for user to select x axis and y axis (when selecting a heatmap).

Adds button to remove plots, appends new plots rather than overwrites, groups them by whether they are user or reference plots

Add user options to select how data is sliced, finds which dimensions remain after x/y axis are chosen, and prompts the user to choose how they are sliced

Error handling for some user choices that break things and minor bug fixes

…ls etc) Adds important information to plot caption, generated from if the data is sliced, and what model and dataset the plot is created from

…e doing that Making more robust error handling that handles errors caused by invalid entries in the UI.

… reference model UI

…o test UI

… section

…rence datasets Add functionality to plot user data overlayed with any number of reference datasets. Adds UI to add reference model datasets to a dictionary, and plots them.

add animation option to dropdown for model plotting Revert "add animation option to dropdown for model plotting" This reverts commit 39d59c4. Revert "Attempt at adding animated plots" This reverts commit 91456e2. Reapply "Attempt at adding animated plots" This reverts commit 2bc879a. Reapply "add animation option to dropdown for model plotting" This reverts commit b373623. Add animation axes choices for user dataset Fix so that HTML actually loads in widget Trying animation without html with just the hvplot panel object, cleaner if it works Add scrubber to animation Lazy load the animation Add reference animation functionality, and change colour map for animated plots to match xarray heatmap default Update ref UI so that animation options appear when they are supposed to. Initialise fig and fig animated in ref plot data button click function ensure ref plot is actually plotting ref animation rather than user data animation Update to only show the dimensions available to that variable in plot options Update animation plots to use variable as title, and include captions with slice information. Fix animation figure title, make animation bigger?

Adding new option to plot user vs ref data heatmaps on a grid together.

…data and reference data. Works for heat maps and line graphs Add functionality allowing user to plot the difference between their data and reference data. Works for heat maps and line graphs

…erence data

Add toggles for long/ short var names, for all different plot types Add multiplot variable long name toggle Add toggle for reference variable too Add toggle for reference variable too

…h as SeaIce can load. Cleaning stuff up a bit, adding docstrings, and ensuring datasets such as SeaIce can load. Also fixes issue with animation that caused plot to not be created for SeaIce datasets.

charles-turner-1

Just one main comment from me about the length of the ui file - I think we'd probably benefit from simplifying that. I've added a few comments about how we might do that.

With that said, doing so before we have tests on it is a sketchy proposal, so let's not jump the gun there.

charles-turner-1 · 2026-08-26T23:47:39Z

This is definitely getting way too long - but let's not try to refactor/simplify before we've written tests.

charles-turner-1 · 2026-08-27T23:30:34Z

        self.status_textbox = pn.widgets.StaticText(styles={'background': 'lightblue', 'font-size': '18px', 'color': 'black', 'padding': '5px'}, margin=(10, 0, 10, 0))
+        self.warning_textbox = pn.widgets.StaticText(styles={'background': 'darkred', 'font-size': '18px', 'color': 'white', 'padding': '5px'}, margin=(10, 0, 10, 0))
        self.ref_status_textbox = pn.widgets.StaticText(styles={'background': 'lightblue', 'font-size': '18px', 'color': 'black', 'padding': '5px'}, margin=(10, 0, 10, 0))
+        self.ref_warning_textbox = pn.widgets.StaticText(styles={'background': 'darkred', 'font-size': '18px', 'color': 'white', 'padding': '5px'}, margin=(10, 0, 10, 0))


These are duplicating the same styles, so you could consider doing something like

Suggested change

self.status_textbox = pn.widgets.StaticText(styles={'background': 'lightblue', 'font-size': '18px', 'color': 'black', 'padding': '5px'}, margin=(10, 0, 10, 0))

self.warning_textbox = pn.widgets.StaticText(styles={'background': 'darkred', 'font-size': '18px', 'color': 'white', 'padding': '5px'}, margin=(10, 0, 10, 0))

self.ref_status_textbox = pn.widgets.StaticText(styles={'background': 'lightblue', 'font-size': '18px', 'color': 'black', 'padding': '5px'}, margin=(10, 0, 10, 0))

self.ref_warning_textbox = pn.widgets.StaticText(styles={'background': 'darkred', 'font-size': '18px', 'color': 'white', 'padding': '5px'}, margin=(10, 0, 10, 0))

self.STYLES = {

"status": pn.widgets.StaticText(styles={'background': 'lightblue', 'font-size': '18px', 'color': 'black', 'padding': '5px'}, margin=(10, 0, 10, 0)),

"warning": pn.widgets.StaticText(styles={'background': 'darkred', 'font-size': '18px', 'color': 'white', 'padding': '5px'}, margin=(10, 0, 10, 0)),

}

self.status_textbox = self.STYLES.get("status")

self.ref_status_textbox = self.STYLES.get("status")

self.warning_textbox = self.STYLES.get("warning")

self.ref_status_textbox = self.STYLES.get("warning")

charles-turner-1 · 2026-08-27T23:32:53Z

+        self.x_axis_dropdown = pn.widgets.Select()
+        self.y_axis_dropdown = pn.widgets.Select()


Suggested change

self.x_axis_dropdown = pn.widgets.Select()

self.y_axis_dropdown = pn.widgets.Select()

self.x_axis_dropdown, self.y_axis_dropdown = pn.widgets.Select(), pn.widgets.Select()

Only because the x and y axis selects are sort of conceptually the same thing. I probably wouldn't want to merge the plot_type_dropdown here, even though it's also a select - if that makes sense?

charles-turner-1 · 2026-08-27T23:41:42Z

+        self.ref_x_axis_dropdown = pn.widgets.Select()
+        self.ref_y_axis_dropdown = pn.widgets.Select()


Suggested change

self.ref_x_axis_dropdown = pn.widgets.Select()

self.ref_y_axis_dropdown = pn.widgets.Select()

self.x_axis_dropdown, self.y_axis_dropdown = pn.widgets.Select(), pn.widgets.Select()

In fact, it looks like there are probably a heap of these. I might do something like this:

def _setup_plot_selectors(self): self.ref_plot_type_dropdown = pn.widgets.Select() self.x_axis_dropdown, self.y_axis_dropdown = pn.widgets.Select(), pn.widgets.Select() self.animation_axis_dropdown = pn.widgets.Select()

And then call that throughout.

charles-turner-1 · 2026-08-27T23:44:35Z

+    def _update_warning_text(self, text):
+
+            """
+            Update text displayed in status_textbox widget. Private.
+
+            Parameters
+            ----------
+            text : str
+                Text to be displayed in status_textbox
+            """
+
+            # Update status_textbox with text
+            self.warning_textbox.value = str(text)


Suggested change

def _update_warning_text(self, text):

"""

Update text displayed in status_textbox widget. Private.

Parameters

----------

text : str

Text to be displayed in status_textbox

"""

# Update status_textbox with text

self.warning_textbox.value = str(text)

def _update_warning_text(self, text):

"""

Update text displayed in status_textbox widget. Private.

Parameters

----------

text : str

Text to be displayed in status_textbox

"""

self.warning_textbox.value = str(text)

To have these show up as docstrings they need to be right below the definition line. You also don't need the comment below. I think the indentation might be off with these too?

charles-turner-1 · 2026-08-27T23:45:56Z

+        """Toggles the user variable dropdown options between short names and long name."""
+        if self.variable_toggle.value == True:
+            self.long_names = {}
+            for var in list(self.dataset.keys()):


Suggested change

for var in list(self.dataset.keys()):

for var in self.dataset.keys():

No need to coerce to a list here

charles-turner-1 · 2026-08-27T23:46:48Z

+            self.variable_toggle.label = "Display Variable Short Names"
+            self.multiplot_variable_toggle.label = "Display Variable Short Names"
+            self.multiplot_variable_toggle.value = True
+        elif self.variable_toggle.value == False:


Suggested change

elif self.variable_toggle.value == False:

else:

You've done if True above, so this is probably okay to write as an else - unless it can be None too.

charles-turner-1 · 2026-08-27T23:47:26Z

+        if self.ref_variable_toggle.value == True:
+            self.ref_long_names = {}
+            for var in list(self.ref_dataset.keys()):


Suggested change

if self.ref_variable_toggle.value == True:

self.ref_long_names = {}

for var in list(self.ref_dataset.keys()):

if self.ref_variable_toggle.value:

self.ref_long_names = {}

for var in self.ref_dataset.keys():

I won't put any more of this style of suggestion here so I don't swap you, but one to look out for!

charles-turner-1 · 2026-08-27T23:55:36Z

+        self.ref_keys_button.button_type = 'success'

        self.clear_ref_model_data_button.name = 'Clear reference model'
-        self.clear_ref_model_data_button.button_type = 'primary'
+        self.clear_ref_model_data_button.button_type = 'danger'


These could be extracted to shared helpers, eg.

Suggested change

self.ref_keys_button.button_type = 'success'

self.clear_ref_model_data_button.name = 'Clear reference model'

self.clear_ref_model_data_button.button_type = 'primary'

self.clear_ref_model_data_button.button_type = 'danger'

def _set_button_type(button, type):

button.value.button_type = type

...

self._set_button_type(self.ref_keys_button, 'success')

Honest, it doesn't change the number of lines of code much, but it does help with readability & prevent any potential mishaps with not doing the exact same thing.

Make widgets get style from dictionary rather than repeating code Shouldn't change any functionality

Adding testing for ui.py

charles-turner-1 · 2026-09-09T01:49:29Z

@theoellingsen there's a lot to read here so I'm gonna get some in roads from the AI 😅

Copilot

🟡 Changes recommended

Confirmed functional issues in session/UI logic and correctness issues in the new test suite (including duplicate test names and incorrect assertions) should be fixed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR refactors the Panel-based UserInterface to improve widget stability (moving away from pn.bind() to explicit button-driven plotting flows), adds extensive UI test coverage, and updates the data-catalog ingestion path to use access_nri_intake.experiment.use_datastore (while removing the background scheduler from session startup).

Changes:

Refactored UserInterface plotting flows (user, reference, and overlay/multiplot), including slice-selection UI and hvplot-based animations.
Updated live catalog building/checking to rely on use_datastore and expanded supported model builders.
Added a large tests/test_UI.py suite and introduced Pixi workspace/deps metadata.

File summaries

File	Description
`tests/test_UI.py`	Adds comprehensive unit tests for UI flows, plotting validity, slicing UI, and callback wiring.
`src/med_diagnostics/ui.py`	Major UI stability refactor: explicit buttons, prebuilt selection UI, reference + overlay plotting, and animations.
`src/med_diagnostics/session.py`	Removes APScheduler usage and simplifies session lifecycle around a single initial data load.
`src/med_diagnostics/data.py`	Switches catalog building to `use_datastore`, expands supported model types, and adjusts dataset loading behavior.
`src/med_diagnostics/test.py`	Removes placeholder test file.
`pyproject.toml`	Adds Pixi workspace/dependencies/tasks configuration.
`.gitignore`	Ignores Pixi envs, coverage artifacts, and `.vscode/`.
`.gitattributes`	Marks `pixi.lock` as generated/binary-merge and configures linguist settings.

Review details

Files reviewed: 5/9 changed files
Comments generated: 10
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

charles-turner-1 · 2026-09-09T02:00:29Z

Copilots review comments mostly look sensible to me. Have a read through and see if they make sense to you Theo!

Co-authored-by: Charles Turner <52199577+charles-turner-1@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

…eduler, which is now removed. Fixes issue in session._get_data() function as well. Given scheduler was removed, self.model_data was always initialised as an empty list, so the check for new data became redundant.

theoellingsen added 11 commits August 12, 2026 10:56

Making changes to _check_for_new_data and _build_new_catalog to use t…

a2142da

…he intake catalogs use_datastore function Making changes to _check_for_new_data and _build_new_catalog to use the intake catalogs use_datastore function, rather than using bespoke functions.

Allow variable change to update user dataset plot

91ad572

setting figure_exists to true

Update to have both variable selections hopefully working at once

d2974a1

Remove redundant code left from scheduler

ff83340

Remove redundant code left from scheduler, all stuff that was unreachable before anyway

Update .gitignore

a5eb3f4

Update .gitignore

d54471f

Delete settings.json

9eb1f87

charles-turner-1 reviewed Aug 18, 2026

View reviewed changes

theoellingsen and others added 18 commits August 18, 2026 15:45

Update src/med_diagnostics/ui.py

83a3a09

Co-authored-by: Charles Turner <52199577+charles-turner-1@users.noreply.github.com>

Update src/med_diagnostics/test.py

4ab4238

Co-authored-by: Charles Turner <52199577+charles-turner-1@users.noreply.github.com>

Update pyproject.toml

74067bf

Co-authored-by: Charles Turner <52199577+charles-turner-1@users.noreply.github.com>

Update pyproject.toml

d8c4c3e

Co-authored-by: Charles Turner <52199577+charles-turner-1@users.noreply.github.com>

User choices for plots (line or heatmap)

d7495f8

Add new plot options, allows user the choice between plotting a line graph and a heatmap. Adds choices for user to select x axis and y axis (when selecting a heatmap).

Update widget behaviour for appending multiple plots

564a866

Adds button to remove plots, appends new plots rather than overwrites, groups them by whether they are user or reference plots

Add user options to select how data is sliced

460893a

Add user options to select how data is sliced, finds which dimensions remain after x/y axis are chosen, and prompts the user to choose how they are sliced

Error handling and minor bug fixes

005fdb4

Error handling for some user choices that break things and minor bug fixes

Add important information to plot caption (slice details, model detai…

9765bcc

…ls etc) Adds important information to plot caption, generated from if the data is sliced, and what model and dataset the plot is created from

Making more robust error handling, and fixing the bugs I created whil…

763f3f4

…e doing that Making more robust error handling that handles errors caused by invalid entries in the UI.

Fix bug in clearing reference data that caused it to delete the whole…

01a133a

… reference model UI

Update comments

6207680

Added reference model dataset dictionary for adding multiplot. Need t…

8622124

…o test UI

bugs in UI

589eb2d

fixing another bug in building ref model dictionary

984367d

Updates to reference plot widget behaviour to account for new overlay…

0efb637

… section

Change where ref dataset selection UI appears in the widget

2740c96

Add functionality to plot user data overlayed with any number of refe…

609b85e

…rence datasets Add functionality to plot user data overlayed with any number of reference datasets. Adds UI to add reference model datasets to a dictionary, and plots them.

theoellingsen added 7 commits August 27, 2026 09:04

Adding new option to plot user vs ref data heatmaps on a grid together.

0721e09

Adding new option to plot user vs ref data heatmaps on a grid together.

Add functionality allowing user to plot the difference between their …

bdfb6ef

…data and reference data. Works for heat maps and line graphs Add functionality allowing user to plot the difference between their data and reference data. Works for heat maps and line graphs

Add sliders for choosing slice. Also sort dropdowns alphabetically.

108b0b4

adding 'plot both' option to multiplot to plot original data and diff…

3963122

…erence data

Add toggles for long/ short var names, for all different plot types

243c01c

Add toggles for long/ short var names, for all different plot types Add multiplot variable long name toggle Add toggle for reference variable too Add toggle for reference variable too

Cleaning stuff up a bit, adding docstrings, and ensuring datasets suc…

bbb5413

…h as SeaIce can load. Cleaning stuff up a bit, adding docstrings, and ensuring datasets such as SeaIce can load. Also fixes issue with animation that caused plot to not be created for SeaIce datasets.

theoellingsen requested review from Whyborn and charles-turner-1 August 26, 2026 23:11

Fix typos in UI

db5ef7c

charles-turner-1 reviewed Aug 27, 2026

View reviewed changes

Make widgets get style from dictionary rather than repeating code

5bc26ba

Make widgets get style from dictionary rather than repeating code Shouldn't change any functionality

theoellingsen force-pushed the IntakeLoader branch from e4bde03 to 5bc26ba Compare September 3, 2026 02:05

theoellingsen added 2 commits September 3, 2026 13:12

style fix

da696cd

Adding testing ui.py

247d63a

Adding testing for ui.py

theoellingsen force-pushed the IntakeLoader branch from 6da3dc9 to 247d63a Compare September 8, 2026 23:54

theoellingsen requested a review from charles-turner-1 September 8, 2026 23:56

theoellingsen added 2 commits September 9, 2026 11:12

Add tests for all the button click functions in __init__

8da1f44

Increase test coverage slightly

9b54785

charles-turner-1 requested a lite review from Copilot September 9, 2026 01:48

Copilot started reviewing on behalf of charles-turner-1 September 9, 2026 01:49 View session

Copilot AI reviewed Sep 9, 2026

View reviewed changes

theoellingsen and others added 3 commits September 9, 2026 12:21

Apply batched suggestions from code review

952abb7

Co-authored-by: Charles Turner <52199577+charles-turner-1@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Removing unused imports

0604ee5

theoellingsen force-pushed the IntakeLoader branch from f7c2df8 to 2cd8cfc Compare September 10, 2026 22:55

-        self.status_textbox = pn.widgets.StaticText(styles={'background': 'lightblue', 'font-size': '18px', 'color': 'black', 'padding': '5px'}, margin=(10, 0, 10, 0))
-        self.warning_textbox = pn.widgets.StaticText(styles={'background': 'darkred', 'font-size': '18px', 'color': 'white', 'padding': '5px'}, margin=(10, 0, 10, 0))
-        self.ref_status_textbox = pn.widgets.StaticText(styles={'background': 'lightblue', 'font-size': '18px', 'color': 'black', 'padding': '5px'}, margin=(10, 0, 10, 0))
-        self.ref_warning_textbox = pn.widgets.StaticText(styles={'background': 'darkred', 'font-size': '18px', 'color': 'white', 'padding': '5px'}, margin=(10, 0, 10, 0))
+    self.STYLES = {
+        "status": pn.widgets.StaticText(styles={'background': 'lightblue', 'font-size': '18px', 'color': 'black', 'padding': '5px'}, margin=(10, 0, 10, 0)),
+        "warning": pn.widgets.StaticText(styles={'background': 'darkred', 'font-size': '18px', 'color': 'white', 'padding': '5px'}, margin=(10, 0, 10, 0)),
+    }
+    self.status_textbox = self.STYLES.get("status")
+    self.ref_status_textbox = self.STYLES.get("status")
+    self.warning_textbox = self.STYLES.get("warning")
+    self.ref_status_textbox = self.STYLES.get("warning")

		self.x_axis_dropdown = pn.widgets.Select()
		self.y_axis_dropdown = pn.widgets.Select()

	self.x_axis_dropdown = pn.widgets.Select()
	self.y_axis_dropdown = pn.widgets.Select()
	self.x_axis_dropdown, self.y_axis_dropdown = pn.widgets.Select(), pn.widgets.Select()

		self.ref_x_axis_dropdown = pn.widgets.Select()
		self.ref_y_axis_dropdown = pn.widgets.Select()

	for var in list(self.dataset.keys()):
	for var in self.dataset.keys():

Uh oh!

Conversation

theoellingsen commented Aug 18, 2026

Uh oh!

charles-turner-1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

charles-turner-1 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

charles-turner-1 commented Sep 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

charles-turner-1 commented Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants