Merge pull request #139 from BioImageTools/issue-82-aws-profile

Support named AWS profiles for s3:// locations

Author: stefanhahmann

CLAUDE.md

@@ -135,6 +135,23 @@ and half-working link support is worse for users than a clear "not supported". R
 non-`java.net` schemes (e.g. falling back to `LocationService.resolve` when `new URL(p)` throws); links then get `s3:`
 for free, with no change here.
 
+**Named AWS profiles** (issue #82) are how private buckets and non-AWS endpoints are reached: credentials stay in
+`~/.aws/*` (no key fields in any dialog, nothing secret in `PrefService`), and the user only picks a profile *name* in
+`S3Settings` — a dialog of its own, since most users never need it and it means nothing on Fiji-Stable, but stored in
+`OmeZarrOpeningSettings` next to the backend, so each open reads one settings object. The name travels as a
+constructor argument of the backend (`OmeZarrBackend.createBackend( awsProfile )`), so callers only need
+`PyramidBackend`. A setter on `AbstractPyramidBackend` forced callers onto the abstract class, and a setter on the
+interface would have to throw for every backend that cannot read `s3:` (PR #139 review). Both backends build a fresh
+`S3Client` per open, so a changed profile applies to the next open without a restart.
+`AwsProfiles` (in `ome-zarr-imglib2`) parses the two files itself rather than using the SDK's `ProfileFile`: the dialog
+lists the profiles, and it must not link AWS classes on Fiji-Stable. Do not "simplify" it back to the SDK parser; keep
+it agreeing with it instead (the SDK skips profile names outside `[A-Za-z0-9-/.%@_:+]` and merges `region` from both
+files). A chosen profile that has since vanished from the files makes `AbstractPyramidBackend.read` throw
+`AwsProfileNotFoundException`, which `OmeZarr` reports — left to the SDK, it would silently fall back to anonymous
+access on AWS. The remaining SDK wiring (`defaultProfileName` for region and `endpoint_url`, a profile-named credentials
+chain with anonymous fallback) is duplicated in `S3StoreFactory` and `N5PyramidBackend.configureS3`, because no module
+both backends share may depend on the AWS SDK.
+
 An installation without the AWS SDK on the classpath (Fiji-Stable) only notices when the first `s3:` URI is opened —
 see `S3StoreFactory` under `ome-zarr-zarrjava` for the lazy-loading rule and the `S3SupportUnavailableException` it
 throws so the user gets a "get Fiji-Latest" message instead of a linkage stack trace.
@@ -158,10 +175,12 @@ override with their library's display name (`"N5"`, `"zarr-java"`) for user-faci
 implements `read` as a template method – try the multiscales group, fall back to a single array (parent multiscales
 group first, then the array's own `dimension_names`) – and leaves three `protected abstract` hooks for the
 reader-specific steps: `readMultiscale`, `tryReadLevelFromParent` and `tryReadArrayNodeOnly` (the two `try*` hooks
-return `null` for "not applicable, try the next"). `read` and `readSingleArray` are `final`, so the order is fixed for
-every backend. Both `PyramidContents` and the per-level `CachedCellImg`s, transforms, calibration and optional OMERO
-metadata it holds are immutable. Each backend also exposes a static `readPyramid(URI)` convenience entry point for
-outside API users — not named `read`, because Java forbids a static method hiding an inherited instance method.
+return `null` for "not applicable, try the next"; the last one throws `NotAMultiscaleImageException` when there is no
+array at all, so a missing location is never reported as unknown axes). `read` and `readSingleArray` are `final`, so
+the order is fixed for every backend. Both `PyramidContents` and the per-level `CachedCellImg`s, transforms, calibration
+and optional OMERO metadata it holds are immutable. Each backend also exposes a static `readPyramid(URI)` convenience
+entry point for outside API users — not named `read`, because Java forbids a static method hiding an inherited instance
+method.
 
 **A too-old reader library is reported, not thrown at the console.** A Fiji-Stable installation whose N5 stack predates
 this plugin fails inside the backend with a `NoClassDefFoundError` (e.g. on `OmeNgffMetadataParser`), which is useless
@@ -274,8 +293,8 @@ with `Namespace 'ome.zarr' is not allowed`. So going back to Central means switc
 `releaseProfiles`; an org-owned namespace (`io.github.bioimagetools`, or `org.bioimagetools` if the domain is ever
 registered) would be the alternative. Java package names stay `ome.zarr.*` through all of this.
 
-- **`ome-zarr-imglib2`** – package `ome.zarr.imglib2` (+`.metadata`, `.exceptions`); backend-agnostic core. No Fiji or
-  backend dependency.
+- **`ome-zarr-imglib2`** – package `ome.zarr.imglib2` (+`.metadata`, `.exceptions`, `.s3`, `.util`); backend-agnostic
+  core. No Fiji or backend dependency.
 - **`ome-zarr-n5`** – `ome.zarr.n5` (`N5PyramidBackend`, N5-universe, OME-NGFF v0.3–v0.5); depends on imglib2 +
   external N5-universe (codecs `n5-zarr`/`n5-blosc`/zstd arrive transitively via `n5-universe`).
 - **`ome-zarr-zarrjava`** – `ome.zarr.zarrjava` (`ZarrJavaPyramidBackend`, Zarr v2/v3); depends on imglib2 +
@@ -313,9 +332,9 @@ registered) would be the alternative. Java package names stay `ome.zarr.*` throu
   Swing dialogs in `.dialog`, the six built-in `OmeZarrOpener`s in `.open.openers`. The SciJava commands sit in three
   sibling packages under `.plugin.command`, one per menu location: `.fileimport` for the two `File > Import` entries
   plus their shared `FileImportHelper` (package-private, so its tests live there too), `.tools` for the
-  `Plugins > OME-Zarr` entries, and `.settings` for `OpeningBehaviorSettings` and `UserScriptSettings`. Commands are
-  discovered through the `@Plugin` annotation index, not by package, so moving one between these packages does not
-  touch its menu path – but `PrefService` keys off **both** the package and the simple class name
+  `Plugins > OME-Zarr` entries, and `.settings` for `OpeningBehaviorSettings`, `S3Settings` and `UserScriptSettings`.
+  Commands are discovered through the `@Plugin` annotation index, not by package, so moving one between these
+  packages does not touch its menu path – but `PrefService` keys off **both** the package and the simple class name
   (`DefaultPrefService.prefs(Class)` is `userNodeForPackage( c ).node( c.getSimpleName() )`), so *moving or renaming* a
   class used as a preference key (as `ScriptUtils` uses `UserScriptSettings`) resets that stored setting. Renaming
   `ZarrOpeningSettings` to `OmeZarrOpeningSettings` in 0.9 did exactly that, deliberately: opener, preferred width and

README.md

@@ -16,6 +16,7 @@
         - [Open via menu (local folders)](#open-via-menu-local-folders)
         - [Open as `Dataset` (scripting)](#open-as-dataset-scripting)
         - [FIJI links (`fiji://`)](#fiji-links-fiji)
+        - [AWS profiles (private buckets, non-AWS S3 endpoints)](#aws-profiles-private-buckets-non-aws-s3-endpoints)
     - [Opening Behavior Settings](#opening-behavior-settings)
         - [Default opening behavior](#default-opening-behavior)
             - [Selection dialog (Ask me every time)](#selection-dialog-ask-me-every-time)
@@ -57,9 +58,10 @@ Drop a local OME-Zarr folder or an OME-Zarr URI onto Fiji. What happens next is
 * Supports local paths, http(s) URLs, and `s3://` URIs
     * Public (anonymous) S3 buckets work out of the box, e.g.
       `s3://janelia-cosem-datasets/jrc_mus-choroid-plexus-3/jrc_mus-choroid-plexus-3.zarr/recon-1/em/fibsem-uint8`.
-    * Private buckets use your ambient AWS credentials (environment variables, `~/.aws/credentials`,
-      instance profile, etc.); if those are absent, access falls back to anonymous.
-    * The AWS region defaults to `us-east-1`.
+    * Private buckets, S3 storage outside amazonaws.com and multiple accounts: choose a
+      [named AWS profile](#aws-profiles-private-buckets-non-aws-s3-endpoints); region, endpoint and keys then come from it.
+    * Without a chosen profile, your ambient AWS credentials are used (environment variables, `~/.aws/credentials`,
+      instance profile, etc.), falling back to anonymous access; the region is then always `us-east-1`.
 * Three entry points:
     * Paste with `CTRL` / `CMD` / `SHIFT` + `V` (requires FIJI latest)
     * Paste via menu: Plugins -> OME-Zarr -> Paste OME-Zarr URI
@@ -121,6 +123,31 @@ work, check the *Enable web links* setting in `Edit -> Options -> Desktop...`:
 See [doc/fiji-links-demo.html](https://htmlpreview.github.io/?https://raw.githubusercontent.com/BioImageTools/ome-zarr-fiji-java/main/doc/fiji-links-demo.html)
 for a page with clickable examples of each form.
 
+### AWS profiles (private buckets, non-AWS S3 endpoints)
+
+Credentials are expected to live in the standard AWS files. Define a profile like this:
+
+```
+# ~/.aws/config
+[profile my_profile]
+endpoint_url = https://s3.example.org
+region = us-east-1
+
+# ~/.aws/credentials
+[my_profile]
+aws_access_key_id = ...
+aws_secret_access_key = ...
+```
+
+Then choose `my_profile` as *AWS profile for s3:// locations* in `Plugins > OME-Zarr > Settings > S3 Settings`. The
+list shows every profile defined in these two files.
+The next pasted `s3://` URI uses its `endpoint_url`, `region` and keys without needing a restart of Fiji.
+A profile without `region` uses `us-east-1`.
+A profile without keys falls back to anonymous access. 
+*No profile* uses the profile named by the `AWS_PROFILE` environment variable at Fiji start (e.g.
+`AWS_PROFILE=my_profile ./fiji`), else the `default` profile, always with region `us-east-1`. Choose `default` in the
+list to use its own `region`.
+
 ## Opening Behavior Settings
 
 `Plugins -> OME-Zarr -> Settings -> Opening Behavior Settings` sets three things, kept across Fiji sessions: the

doc/DEVELOPERS.md

@@ -39,13 +39,13 @@ graph TD
 
 `ome-zarr-imglib2`, `ome-zarr-n5` and `ome-zarr-zarrjava` are plain Java — **no Fiji, so headless use works**.
 
-| Module              | Packages                                              | Purpose                                                                                                                                                                         | Depends on                                                                           |
-|---------------------|-------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|
-| `ome-zarr-imglib2`  | `ome.zarr.imglib2` `.metadata` `.exceptions`          | Backend-agnostic core: data model, OME-NGFF metadata, Zarr detection. No Fiji, no backend library.                                                                              | imglib2, imglib2-cache, imglib2-realtransform, gson                                  |
-| `ome-zarr-n5`       | `ome.zarr.n5`                                         | N5-universe backed reader, OME-Zarr v0.3 – v0.5.                                                                                                                                | `ome-zarr-imglib2`, the N5 stack                                                     |
-| `ome-zarr-zarrjava` | `ome.zarr.zarrjava`                                   | zarr-java backed reader (the default), Zarr v2/v3 i.e. OME-Zarr v0.4 + v0.5. The only backend that reads `.ozx` archives.                                                       | `ome-zarr-imglib2`, `dev.zarr:zarr-java`                                             |
-| `ome-zarr-fiji`     | `ome.zarr.fiji` `.read` `.open` `.plugins` `.util`    | Opens a pyramid in ImageJ or BigDataViewer, tracks the active window, and defines the `OmeZarrOpener` extension point. No dialogs, no backend dependency — pick one at runtime. | `ome-zarr-imglib2`, bigdataviewer-core, spim_data, imagej-common, ij, scijava-common |
-| `ome-zarr-fiji-ui`  | `ome.zarr.fijiui` `.open` `.plugin` `.dialog` `.util` | The shipped plugin: `IOPlugin`, clipboard, Swing dialogs, SciJava commands, persisted settings, the six built-in openers.                                                       | the four above, scijava-common, script-editor, n5-viewer_fiji                        |
+| Module              | Packages                                                   | Purpose                                                                                                                                                                         | Depends on                                                                           |
+|---------------------|------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|
+| `ome-zarr-imglib2`  | `ome.zarr.imglib2` `.metadata` `.exceptions` `.s3` `.util` | Backend-agnostic core: data model, OME-NGFF metadata, Zarr detection. No Fiji, no backend library.                                                                              | imglib2, imglib2-cache, imglib2-realtransform, gson                                  |
+| `ome-zarr-n5`       | `ome.zarr.n5`                                              | N5-universe backed reader, OME-Zarr v0.3 – v0.5.                                                                                                                                | `ome-zarr-imglib2`, the N5 stack                                                     |
+| `ome-zarr-zarrjava` | `ome.zarr.zarrjava`                                        | zarr-java backed reader (the default), Zarr v2/v3 i.e. OME-Zarr v0.4 + v0.5. The only backend that reads `.ozx` archives.                                                       | `ome-zarr-imglib2`, `dev.zarr:zarr-java`                                             |
+| `ome-zarr-fiji`     | `ome.zarr.fiji` `.read` `.open` `.plugins` `.util`         | Opens a pyramid in ImageJ or BigDataViewer, tracks the active window, and defines the `OmeZarrOpener` extension point. No dialogs, no backend dependency — pick one at runtime. | `ome-zarr-imglib2`, bigdataviewer-core, spim_data, imagej-common, ij, scijava-common |
+| `ome-zarr-fiji-ui`  | `ome.zarr.fijiui` `.open` `.plugin` `.dialog` `.util`      | The shipped plugin: `IOPlugin`, clipboard, Swing dialogs, SciJava commands, persisted settings, the six built-in openers.                                                       | the four above, scijava-common, script-editor, n5-viewer_fiji                        |
 
 Pick the smallest one you need:
 

ome-zarr-fiji-ui/src/main/java/ome/zarr/fijiui/open/OmeZarrOpenActions.java

@@ -189,15 +189,16 @@ public OmeZarrOpenActions( final OmeZarr omeZarr )
 	}
 
 	/**
-	 * Reader for {@code inputUri} using the backend and preferred width from
-	 * {@code settings}, or the defaults when no settings are given.
+	 * Reader for {@code inputUri} using the backend, AWS profile and preferred
+	 * width from {@code settings}, or the defaults when no settings are given.
 	 */
 	private static OmeZarr omeZarrFor( final URI inputUri, final Context context,
 			final OmeZarrOpeningSettings settings, final Consumer< String > errorHandler )
 	{
 		final OmeZarrBackend backend = settings == null ? OmeZarrOpeningSettings.DEFAULT_BACKEND : settings.getBackend();
+		final String awsProfile = settings == null ? null : settings.getAwsProfile();
 		final Integer preferredMaxWidth = settings == null ? null : settings.getPreferredMaxWidth();
-		return new OmeZarr( inputUri, context, backend.createBackend(), preferredMaxWidth, errorHandler );
+		return new OmeZarr( inputUri, context, backend.createBackend( awsProfile ), preferredMaxWidth, errorHandler );
 	}
 
 	private String uriFormattedForN5Dialog( final URI uri )

ome-zarr-fiji-ui/src/main/java/ome/zarr/fijiui/open/PasteToOpenAction.java

@@ -34,7 +34,7 @@
 import org.scijava.Context;
 
 import ome.zarr.fijiui.util.ClipboardUtils;
-import ome.zarr.imglib2.ZarrUtils;
+import ome.zarr.imglib2.util.ZarrUtils;
 
 /**
  * Opening orchestration for the "paste an OME-Zarr location and open it" flow,

ome-zarr-fiji-ui/src/main/java/ome/zarr/fijiui/open/options/OmeZarrBackend.java

@@ -6,13 +6,13 @@
  * %%
  * Redistribution and use in source and binary forms, with or without
  * modification, are permitted provided that the following conditions are met:
- * 
+ *
  * 1. Redistributions of source code must retain the above copyright notice,
  *    this list of conditions and the following disclaimer.
  * 2. Redistributions in binary form must reproduce the above copyright notice,
  *    this list of conditions and the following disclaimer in the documentation
  *    and/or other materials provided with the distribution.
- * 
+ *
  * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
  * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
  * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
@@ -78,20 +78,32 @@ public String getDescription()
 	}
 
 	/**
-	 * Creates a fresh {@link PyramidBackend} for the backend library this constant
-	 * represents.
+	 * Creates a fresh backend for the backend library this constant represents,
+	 * reading {@code s3:} URIs with the AWS SDK default profile.
 	 *
 	 * @return a new backend instance, never {@code null}
 	 */
 	public PyramidBackend createBackend()
+	{
+		return createBackend( null );
+	}
+
+	/**
+	 * Creates a fresh backend for the backend library this constant represents.
+	 *
+	 * @param awsProfile the AWS profile for {@code s3:} URIs, or {@code null} for
+	 *   the AWS SDK default
+	 * @return a new backend instance, never {@code null}
+	 */
+	public PyramidBackend createBackend( final String awsProfile )
 	{
 		switch ( this )
 		{
 		case ZARR_JAVA:
-			return new ZarrJavaPyramidBackend();
+			return new ZarrJavaPyramidBackend( awsProfile );
 		case N5:
 		default:
-			return new N5PyramidBackend();
+			return new N5PyramidBackend( awsProfile );
 		}
 	}
 }