[hailctl] add Spark 4 AWS EMR support by hmkim · Pull Request #15744 · hail-is/hail

hmkim · 2026-09-07T02:14:49Z

Change Description

Adds experimental, community-supported hailctl emr support for running Hail on Amazon EMR on EC2.
This supersedes #15612 by porting the implementation to the Spark 4 baseline and hardening artifact,
network, IAM, storage, and lifecycle behavior using live private-cluster validation.

The supported target is intentionally narrow:

EMR on EC2 only
emr-spark-8.1.0 only (Spark 4.1.1, observed Scala 2.13.18, Java 17)
Hail artifacts built for Spark 4.1.x / Scala 2.13 / Python 3.12
private NAT-backed subnet with pre-created S3 and EMR service endpoints
mandatory idle auto-termination

The command uses an exact artifact manifest containing the Hail git revision, Spark/Scala/Python
versions, S3 wheel and wheelhouse URIs, and SHA256 checksums. Bootstrap installs Python 3.12 and Hail
from the checksum-verified offline wheelhouse without resolving dependencies from PyPI. PySpark is
provided by EMR and excluded from the wheelhouse.

Private-cluster launch validates the release label, subnet routes and DNS, NAT, S3 gateway endpoint,
EMR service endpoint and security group, primary/core security groups, custom service role and EC2
instance profile, and the service role's scoped iam:PassRole access. The request retains the
AmazonEMRServicePolicy_v2 resource tag, exact bootstrap action, S3 logs, and auto-termination policy
after advanced JSON overlays.

AWS paths use Hail's HadoopFS route with HAIL_CLOUD=aws. The final request explicitly routes
s3://, s3n://, and s3a:// through S3A and configures the S3A committer factory, MagicV2,
in-memory commit tracking, and overwrite-and-commit behavior. Advanced overlays cannot remove these
required settings.

Related PR #15692 updates the Google Managed Spark path. Its changed files do not overlap this PR,
and a synthetic merge reports no conflicts.

Validation

Local validation on exact commit 113313f49e48bbd6a11f05c165845d1819241aa3:

103 EMR/config tests
Ruff format/check and Pyright
focused JVM CloudStorageConfigSuite
Spark 4.1.2 / Scala 2.13.18 wheel and JAR build with Java 17 target bytecode
Hail-first and SparkSession-first with local PySpark 4.1.2 and 4.1.1
deterministic 83-wheel offline wheelhouse with PySpark excluded

Live private validation in us-east-1:

exact checksum-verified bootstrap on private-only primary/core instances
runtime Spark 4.1.1-amzn-0, Scala 2.13.18, Java 17.0.20, Python 3.12.13
Hail-first and SparkSession-first S3A Table writes
applicable non-VEP cluster tests (start/stop, tree aggregate, BlockMatrix, SKAT)
published 1000 Genomes tutorial GWAS: minimum p-value 3.576454e-9 at 8:19600329
gnomAD 2.1.1 annotation and Snappy Parquet export
full-autosome 1000G PCA-adjusted regression: 81,625,463 variants × 2,504 samples, all
81,625,463 regression p-values defined

Secondary private validation in ap-northeast-2:

exact final commit artifact and offline bootstrap
all required effective S3A/MagicV2 properties verified
repeated Hail Table and Spark Parquet overwrite succeeded; Parquet _SUCCESS present

All test clusters terminated. Dedicated NAT gateways, VPC endpoints, VPCs, and IAM roles were
removed after validation. Versioned SSE-KMS evidence buckets and their enabled KMS keys were retained.

Security Assessment

This change cannot impact the Hail Batch instance as deployed by Broad Institute in GCP

Port hailctl emr to the Spark 4 Hail baseline and target the emr-spark-8.1.0 release. Add exact artifact manifests, an offline wheelhouse bootstrap, content-addressed scripts, mandatory idle termination, and S3A runtime configuration. Require and preflight a private NAT-backed subnet, S3 and EMR service endpoints, service endpoint security groups, and IAM roles before any cluster launch. Expand unit and JVM coverage for release, artifact, network, and filesystem routing behavior.

Require dedicated private primary, core, and service endpoint security groups and validate that all network resources belong to the selected VPC. Keep IAM preflight compatible with least-privilege callers that can pass roles but cannot read them. Store bootstrap scripts at content-addressed artifact keys, align the CLI and smoke contract with mandatory private-network inputs, and keep NAT-backed bootstrap requirements explicit.

Propagate the AmazonEMRServicePolicy_v2 resource tag through RunJobFlow so the service role can launch tagged instances and volumes in the dedicated private VPC. Cover the required tag in request-builder tests.

Download the Hail wheel using its original tagged filename so pip can validate and install it. Reject malformed wheel URIs before package or network installation work and cover the failure path with a regression test.

ehigham

Thank you for putting this together, it's clear a lot of work has gone into this.

It's going to take me a few passes to get familar with this pr as well as emr, but I think there's some trimming that can be done already.

For a v1, I'm keen to keep this minimal and match what's been done for dataproc wherever possible. If we can do without "advanced features" then all-the-better. We likely won't ever support more than one version of emr at a time, so much of the validation can go I think.

ehigham · 2026-09-10T19:14:12Z


-If you'd like to deploy Hail on AWS, we recommend that you try the `open-source Cloudformation tool <https://github.com/hms-dbmi/hail-on-AWS-spot-instances>`__ maintained by the `Avillach Lab <https://avillach-lab.hms.harvard.edu/>`_ at Harvard Medical School.
+``hailctl emr`` provisions Amazon EMR on EC2 clusters configured for Hail. AWS support is
+experimental and community-supported; it is not part of Hail's required continuous-integration


What's a "continuous-integration matrix"? This feels a little jargon-y to me and can probably be removed.

ehigham · 2026-09-10T19:25:19Z

+Artifact manifest
+~~~~~~~~~~~~~~~~~
+
+Build and upload an exact Hail wheel and an offline dependency wheelhouse. The local manifest passed


I know nothing about aws so you're going to have to help me here.

I want to ensure that our utilities are easy to use and consistent to lower the learning hurdle when switching between clouds; our users are not cloud experts and nor should they be.

In an ideal world, the only difference in the most-basic hailctl command that starts a managed spark service on gcp and aws would be dataproc and emr, ie

$ hailctl dataproc start my-spark-cluster $ hailctl emr start my-spark-cluster

This "wheelhouse" stuff already deviates from that significantly. What is that? Do we really need it? Why can't the wheel be downloaded from a known s3 location?

ehigham · 2026-09-10T19:33:43Z

+        --service-access-security-group sg-EMR_SERVICE_ENDPOINT \
+        --primary-security-group sg-EMR_PRIMARY \
+        --core-security-group sg-EMR_CORE


Are these optional? EMR docs say that you can use the defaults it provides and the cli docs have a --use-default-roles. What's preventing us from using that as a v1?

ehigham · 2026-09-10T20:05:06Z

+    _require_s3_uri(artifact.wheel_uri, 'artifact wheel_uri')
+    _require_s3_uri(artifact.wheelhouse_uri, 'artifact wheelhouse_uri')
+
+    scratch = configuration_of(ConfigVariable.EMR_REMOTE_TMPDIR, s3_scratch, None)


Please move this into start.py and just keep the cli file as typer entry-points.

ehigham · 2026-09-10T20:10:14Z

+    hailctl emr start CLUSTER_NAME \
+        --artifact-manifest ./hail-emr-artifact.json \
+        --s3-scratch s3://my-bucket/hail-tmp/ \
+        --subnet-id subnet-PRIVATE \


Will the default not suffice?

ehigham · 2026-09-10T20:22:48Z

+``s3a://`` paths. The supported EMR Spark 8.1 release therefore does not use EMRFS. Hail routes
+these paths through Spark's Hadoop configuration and sets ``HAIL_CLOUD=aws`` to select its
+HadoopFS-only route. Do not carry EMRFS consistent-view or multipart-cleanup properties into this
+configuration; many have no S3A equivalent. Bucket-specific EMRFS properties are mapped by EMR only
+when the corresponding S3A property is undefined.
+
+``hailctl emr`` explicitly pins all three schemes to ``S3AFileSystem`` and enables the EMR MagicV2
+committer with the S3A committer factory, in-memory commit tracking, and overwrite-and-commit
+behavior. MagicV2 avoids traditional list-and-rename commit operations and writes files to their
+final output location during task commit.
+
+MagicV2 has operational implications. Successful task outputs can remain visible after a failed job,
+so clean the destination before retrying the same output path. A killed JVM can leave incomplete
+multipart uploads; scratch and output buckets should have an S3 lifecycle rule that aborts incomplete
+uploads. MagicV2 also retains a small amount of memory per file until task commit, so workloads that
+write unusually many files per executor may require more container memory or fewer concurrent tasks.
+S3A directory markers end in ``/`` rather than the legacy EMRFS ``_$folder$`` form.


What does all this mean for an end-user? Users should not care that HAIL_CLOUD is set to aws. Other than that, partial uploads is bad - we want to avoid that if at all possible. Hail already checks if a file exists before clobbering it (if so configured), do we need all this?

ehigham · 2026-09-10T20:24:03Z

+Hail requires Python 3.12 or later. EMR on EC2 does not preinstall Python 3.12, so the bootstrap uses
+Amazon Linux 2023 ``dnf`` packages and points ``spark.pyspark.python`` and ``PYSPARK_PYTHON`` at
+``/usr/bin/python3.12``.


do users need to know this?

ehigham · 2026-09-10T20:26:15Z

+Advanced cluster options
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+``--run-job-flow-json`` deep-merges a JSON object into the final ``RunJobFlow`` request. Nested
+objects are merged and lists are replaced. ``InstanceFleets`` replaces default ``InstanceGroups``.
+The final request must retain Spark, the content-addressed Hail bootstrap, the required S3A and
+MagicV2 ``core-site`` properties, an S3 log URI, a supported release, and a valid auto-termination
+policy.
+
+``--off-heap-memory-per-core-mb`` caps Hail's native off-heap allocation per task core. It does not
+reserve YARN container memory or automatically change ``spark.executor.memoryOverhead``.


For v1, no advanced configuration please.
We are not familiar with aws and yet we will need to support this and try not to make breaking changes.

ehigham · 2026-09-10T20:28:55Z

    QUERY_BATCH_WORKER_MEMORY = 'query/batch_worker_memory'
    QUERY_NAME_PREFIX = 'query/name_prefix'
    QUERY_DISABLE_PROGRESS_BAR = 'query/disable_progress_bar'
+    EMR_REGION = 'emr/region'


can this be fetched from aws configure get region rather than cached here?

ehigham · 2026-09-10T20:46:46Z

+@dataclass(frozen=True)
+class EMRReleaseConfig:
+    release_label: str
+    spark_version: str
+    scala_version: str
+    python_executable: str
+    storage_connector: str
+
+
+EMR_RELEASE_CONFIGS = {
+    'emr-spark-8.1.0': EMRReleaseConfig(
+        release_label='emr-spark-8.1.0',
+        spark_version='4.1.1',
+        scala_version='2.13.17',
+        python_executable='/usr/bin/python3.12',
+        storage_connector='s3a',
+    )
+}


I think all this can be removed - we won't support more than one version of emr at a time.

aws-hyunmin added 6 commits September 6, 2026 15:56

fix(emr): Tag resources for V2 policy

bb8f663

Propagate the AmazonEMRServicePolicy_v2 resource tag through RunJobFlow so the service role can launch tagged instances and volumes in the dedicated private VPC. Cover the required tag in request-builder tests.

fix(emr): Preserve artifact wheel filename

64e7c69

Download the Hail wheel using its original tagged filename so pip can validate and install it. Reject malformed wheel URIs before package or network installation work and cover the failure path with a regression test.

fix(emr): Add S3A launcher classpath

76d7663

fix(emr): Configure S3A MagicV2

113313f

hmkim requested a review from a team as a code owner September 7, 2026 02:14

This was referenced Sep 7, 2026

[hailctl] add emr command group for AWS EMR #15612

Closed

feat(emr): Add GRCh38 VEP and LoFTEE profiles hmkim/hail#1

Open

ehigham requested changes Sep 10, 2026

View reviewed changes

Navigation Menu

[hailctl] add Spark 4 AWS EMR support - #15744

[hailctl] add Spark 4 AWS EMR support#15744
hmkim wants to merge 6 commits into
hail-is:mainfrom
hmkim:feature/hailctl-emr-spark4

hmkim commented Sep 7, 2026

Uh oh!

ehigham left a comment

Uh oh!

ehigham Sep 10, 2026

Uh oh!

ehigham Sep 10, 2026

Uh oh!

ehigham Sep 10, 2026

Uh oh!

ehigham Sep 10, 2026

Uh oh!

ehigham Sep 10, 2026

Uh oh!

ehigham Sep 10, 2026

Uh oh!

ehigham Sep 10, 2026

Uh oh!

ehigham Sep 10, 2026

Uh oh!

ehigham Sep 10, 2026

Uh oh!

ehigham Sep 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

hmkim commented Sep 7, 2026

Change Description

Validation

Security Assessment

Uh oh!

ehigham left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants