fix(engine): tighten SNS signature verification by igor-kupczynski · Pull Request #5279 · hatchet-dev/hatchet

igor-kupczynski · 2026-10-22T11:49:50Z

Description

Tightens the checks VerifyPayload runs before trusting an SNS message's signature.

Signed fields other than Message may not contain a newline. The string-to-sign is newline-delimited, so a newline lets a signed string parse as different field values.
SigningCertURL hosts must have a region label (sns.<region>.amazonaws.com[.cn]). A looser pattern matches non-SNS amazonaws.com hosts.
SigningCertURL must be a .pem path with no query. pem.Decode finds a PEM block anywhere in a response body.

Type of change

Bug fix (non-breaking change which fixes an issue)

🤖 AI Disclosure

I acknowledge that an LLM was used in the creation of this Pull Request, in accordance with Hatchet's AI_POLICY.md.
Details: Claude (Opus and Fable) investigated the issue and wrote the fix and test; reviewed by me.

vercel · 2026-10-22T11:49:57Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
hatchet-docs	Ready	Preview	Oct 22, 2026 2:03pm UTC

greptile-apps · 2026-10-22T11:57:40Z

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; no blocking findings remain.

Summary

Tightens SNS signature checks by rejecting newlines outside Message, restricting certificate hosts to regional SNS endpoints, and requiring .pem paths without queries.

SNS messages pass signature checks only with newline-safe fields and regional PEM URLs.

_{Reviews (2) · Last reviewed commit: "test(engine): cover SNS cert URL check" · Reviewed by Greptile}

github-actions · 2026-10-22T12:03:34Z

Runner CPU during the failed `load (optimistic true, race false)` run

Busy is 100 minus idle across all CPUs: near 100 means the runner was saturated. Steal is CPU the host took away from the runner: above a few percent means it was oversubscribed. Either way the latency gate measured the runner, not the scheduler.

# host: Linux 6.14.0-1017-azure x86_64
# cpus: 8
# interval: 2s
# busy% is 100 minus idle; process cpu% is that command's share of all CPUs in the interval
samples: 302
malformed samples: 0
busy%: mean 26.6, max 99.8, samples over 90% busy: 11 (4% of the run)
steal%: mean 6.1, max 33.0, samples over 5% steal: 122 (40% of the run)
iowait%: mean 0.5
load1: max 12.5
top processes by mean share of all CPUs:
  postgres                   8.7%
  hatchet-loadtes            4.2%
  docker-proxy               2.5%
  beam.smp                   1.0%
  compile                    0.3%
  go                         0.2%
  dockerd                    0.1%
  client.test                0.1%

xychart-beta
    title "Runner pressure (302 samples)"
    x-axis "sample time" ["11:51:06", "11:51:21", "11:51:41", "11:51:57", "11:52:15", "11:52:30", "11:52:49", "11:53:05", "11:53:23", "11:53:39", "11:53:58", "11:54:17", "11:54:34", "11:54:52", "11:55:08", "11:55:26", "11:55:42", "11:56:00", "11:56:15", "11:56:33", "11:56:50", "11:57:05", "11:57:22", "11:57:37", "11:57:54", "11:58:10", "11:58:27", "11:58:42", "11:58:59", "11:59:14", "11:59:31", "11:59:48", "12:00:03", "12:00:20", "12:00:35", "12:00:52", "12:01:07", "12:01:24", "12:01:39", "12:01:56"]
    y-axis "percent of all CPUs" 0 --> 100
    line "busy" [8.3, 97.6, 91.4, 69.0, 48.3, 59.8, 49.9, 47.8, 52.0, 57.1, 47.7, 63.6, 56.4, 48.3, 47.4, 55.9, 43.7, 41.3, 35.5, 9.9, 1.6, 1.7, 1.6, 1.6, 1.6, 1.9, 1.7, 1.6, 1.4, 1.7, 1.4, 1.7, 1.4, 1.7, 1.6, 1.4, 1.5, 1.6, 1.6, 2.1]
    line "steal" [0.5, 3.7, 11.5, 14.1, 7.9, 14.5, 14.6, 14.7, 17.0, 21.9, 14.3, 24.4, 17.7, 12.3, 10.8, 19.4, 10.1, 9.2, 2.9, 1.0, 0.1, 0.1, 0.1, 0.0, 0.2, 0.1, 0.2, 0.1, 0.0, 0.0, 0.0, 0.3, 0.2, 0.1, 0.2, 0.1, 0.1, 0.2, 0.1, 0.4]

xychart-beta
    title "Top processes, share of all CPUs"
    x-axis "sample time" ["11:51:06", "11:51:21", "11:51:41", "11:51:57", "11:52:15", "11:52:30", "11:52:49", "11:53:05", "11:53:23", "11:53:39", "11:53:58", "11:54:17", "11:54:34", "11:54:52", "11:55:08", "11:55:26", "11:55:42", "11:56:00", "11:56:15", "11:56:33", "11:56:50", "11:57:05", "11:57:22", "11:57:37", "11:57:54", "11:58:10", "11:58:27", "11:58:42", "11:58:59", "11:59:14", "11:59:31", "11:59:48", "12:00:03", "12:00:20", "12:00:35", "12:00:52", "12:01:07", "12:01:24", "12:01:39", "12:01:56"]
    y-axis "percent of all CPUs" 0 --> 100
    line "postgres" [0.0, 0.0, 0.0, 0.0, 0.1, 17.1, 22.5, 22.7, 27.1, 27.5, 23.4, 33.1, 29.9, 24.7, 24.8, 29.0, 20.7, 18.3, 18.2, 3.9, 0.3, 0.2, 0.3, 0.2, 0.2, 0.2, 0.3, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.1]
    line "hatchet-loadtes" [0.0, 0.0, 0.0, 0.0, 0.5, 7.3, 11.6, 12.0, 12.0, 12.1, 11.5, 14.9, 12.6, 11.7, 11.6, 12.5, 10.4, 10.3, 9.5, 2.2, 0.3, 0.4, 0.4, 0.3, 0.3, 0.4, 0.3, 0.3, 0.3, 0.2, 0.2, 0.3, 0.2, 0.2, 0.3, 0.2, 0.2, 0.2, 0.2, 0.2]
    line "docker-proxy" [0.0, 0.0, 0.0, 0.0, 0.2, 5.2, 7.1, 7.2, 7.3, 8.2, 7.1, 8.8, 7.5, 7.0, 6.7, 7.8, 6.4, 6.4, 5.3, 1.2, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1]
    line "beam.smp" [0.0, 0.0, 0.0, 0.0, 3.3, 4.5, 2.5, 2.3, 2.2, 2.5, 2.3, 3.2, 2.6, 2.4, 2.3, 2.5, 2.1, 2.1, 1.9, 0.5, 0.1, 0.1, 0.1, 0.0, 0.0, 0.1, 0.0, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.0, 0.1, 0.0, 0.0, 0.0, 0.1, 0.0]
    line "compile" [0.0, 7.2, 2.9, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

Process lines, in order: postgres, hatchet-loadtes, docker-proxy, beam.smp, compile

pprof: where the load test process spent its CPU

cpu profiles: 20 of 30s each, merged

top functions by own time, scheduler idle excluded:
Time: 2026-10-22 11:51:49 UTC
Duration: 600.02s, Total samples = 211.80s (35.30%)
Active filters:
   ignore=runtime\.findRunnable
Showing nodes accounting for 88.34s, 41.71% of 211.80s total
Dropped 3056 nodes (cum <= 1.06s)
Showing top 25 nodes out of 460
      flat  flat%   sum%        cum   cum%
    43.50s 20.54% 20.54%     43.50s 20.54%  internal/runtime/syscall/linux.Syscall6
     8.27s  3.90% 24.44%      8.27s  3.90%  runtime.futex
     3.63s  1.71% 26.16%      6.19s  2.92%  runtime.tryDeferToSpanScan
     3.28s  1.55% 27.71%      3.28s  1.55%  runtime.nextFreeFast (inline)
     3.04s  1.44% 29.14%      3.34s  1.58%  runtime.step
     2.39s  1.13% 30.27%      2.39s  1.13%  runtime.memmove
     1.95s  0.92% 31.19%      1.95s  0.92%  runtime.spanClass.sizeclass (inline)
     1.73s  0.82% 32.01%      1.80s  0.85%  runtime.findfunc
     1.62s  0.76% 32.77%      4.40s  2.08%  runtime.selectgo
     1.53s  0.72% 33.49%      5.17s  2.44%  runtime.pcvalue
     1.46s  0.69% 34.18%      1.46s  0.69%  p256MulInternal
     1.43s  0.68% 34.86%      7.57s  3.57%  runtime.newobject
     1.36s  0.64% 35.50%      1.43s  0.68%  internal/runtime/gc/scan.scanSpanPackedAVX512
     1.31s  0.62% 36.12%      3.84s  1.81%  runtime.(*unwinder).resolveInternal
     1.22s  0.58% 36.69%      1.30s  0.61%  runtime.(*mspan).writeHeapBitsSmall
     1.22s  0.58% 37.27%      1.22s  0.58%  runtime.memclrNoHeapPointers
     1.16s  0.55% 37.82%     14.33s  6.77%  runtime.mallocgc
     1.09s  0.51% 38.33%      2.03s  0.96%  encoding/json.checkValid
     1.09s  0.51% 38.85%      4.16s  1.96%  runtime.scanObject
     1.06s   0.5% 39.35%      2.04s  0.96%  runtime.scanObjectSmall
     1.05s   0.5% 39.84%      1.05s   0.5%  internal/runtime/maps.ctrlGroup.matchH2

top hatchet functions by cumulative time (flat, flat%, cum, cum%):
     0.05s 0.024%   51.63s 24.38%  github.com/hatchet-dev/hatchet/internal/msgqueue.(*msgIdBuffer).flush
     0.02s 0.0094%   36.43s 17.20%  github.com/hatchet-dev/hatchet/internal/services/controllers/olap.(*OLAPControllerImpl).handleBufferedMsgs
         0     0%   28.64s 13.52%  github.com/hatchet-dev/hatchet/internal/services/grpc.(*Server).handler.grpcRoutes.unimplemented.func5
         0     0%   28.58s 13.49%  github.com/hatchet-dev/hatchet/internal/services/grpc.(*Server).handler.transportDeadlines.enforce.func4
         0     0%   28.29s 13.36%  github.com/hatchet-dev/hatchet/internal/services/grpc.(*Server).handler.withStreamAbort.func3
         0     0%   28.28s 13.35%  github.com/hatchet-dev/hatchet/internal/services/grpc.(*Server).handler.matchRequestCompression.func2
     0.16s 0.076%   21.88s 10.33%  github.com/hatchet-dev/hatchet/internal/services/controllers/olap.(*OLAPControllerImpl).handleCreateMonitoringEvent
     0.03s 0.014%   18.47s  8.72%  github.com/hatchet-dev/hatchet/internal/services/grpc/middleware.(*handlerInterceptor).WrapUnary.func1
         0     0%   17.92s  8.46%  github.com/hatchet-dev/hatchet/internal/services/grpc/middleware.(*recoveryInterceptor).WrapUnary.func1
     0.01s 0.0047%   15.17s  7.16%  github.com/hatchet-dev/hatchet/internal/services/ingestor/contracts/contractsconnect.NewEventsServiceHandler.func1
     0.01s 0.0047%   14.59s  6.89%  github.com/hatchet-dev/hatchet/internal/services/controllers/task.(*TasksControllerImpl).handleBufferedMsgs
         0     0%   12.95s  6.11%  github.com/hatchet-dev/hatchet/internal/services/dispatcher/contracts/contractsconnect.NewDispatcherHandler.func1
     0.01s 0.0047%   12.89s  6.09%  github.com/hatchet-dev/hatchet/internal/services/ingestor.(*IngestorImpl).Push
     0.02s 0.0094%   12.68s  5.99%  github.com/hatchet-dev/hatchet/internal/services/ingestor.(*IngestorImpl).ingest
         0     0%   12.68s  5.99%  github.com/hatchet-dev/hatchet/internal/services/ingestor.(*IngestorImpl).ingestEventV1
         0     0%   12.67s  5.98%  github.com/hatchet-dev/hatchet/internal/services/ingestor.(*IngestorImpl).IngestEvent
     0.01s 0.0047%   12.10s  5.71%  github.com/hatchet-dev/hatchet/pkg/repository.(*OLAPRepositoryImpl).CreateTaskEvents
     0.05s 0.024%   12.09s  5.71%  github.com/hatchet-dev/hatchet/pkg/repository.(*OLAPRepositoryImpl).writeTaskEventBatch
         0     0%   12.01s  5.67%  github.com/hatchet-dev/hatchet/internal/services/scheduler/v1.(*Scheduler).RunOptimisticSchedulingFromEvents
     0.01s 0.0047%   11.98s  5.66%  github.com/hatchet-dev/hatchet/pkg/scheduling/v1.(*SchedulingPool).RunOptimisticSchedulingFromEvents
     0.07s 0.033%   11.96s  5.65%  github.com/hatchet-dev/hatchet/pkg/scheduling/v1.(*tenantManager).runOptimisticSchedulingFromEvents
     0.08s 0.038%   11.44s  5.40%  github.com/hatchet-dev/hatchet/internal/msgqueue/rabbitmq.(*MessageQueueImpl).subscribe.func1.1
         0     0%    8.82s  4.16%  github.com/hatchet-dev/hatchet/pkg/repository.(*optimisticSchedulingRepositoryImpl).TriggerFromEvents
     0.01s 0.0047%    8.67s  4.09%  github.com/hatchet-dev/hatchet/internal/services/controllers/olap.(*OLAPControllerImpl).writeEngineSpans
     0.03s 0.014%    8.21s  3.88%  github.com/hatchet-dev/hatchet/pkg/repository/sqlchelpers.PrepareTx

goroutines per dump:
  115149: 6
  115219: 16
  115249: 570
  115319: 566
  115349: 587
  115419: 617
  115449: 579
  115519: 574
  115549: 573
  115619: 584
  115649: 502
  115719: 502
  115749: 502
  115819: 502
  115849: 502
  115919: 457
  115949: 457
  120020: 445
  120050: 445
  120120: 445
  120150: 445

pg_stat: where Postgres spent its time

pg_stat snapshots: 19, last one at 12:01:48
total statement exec time: 264.3s over 2694423 calls
database active time: 228.6s, commits: 571285, buffer hits: 19711702, reads: 21337, deadlocks: 0
backends by wait state at the last snapshot:
  Client 17
top statements by total exec time (ms, calls, mean ms, rows):
     21503   25327     0.85        0  -- name: ReleaseRetryQueueItems :batchexec WITH input AS ( SELECT unnest($1::bigint[]) AS task_id, unnest($2::timestamptz[]) AS task_inserte
     17478   46641     0.37    49879  -- name: UpdateTaskStatusesFromMQ :many WITH inputs AS ( SELECT UNNEST($1::UUID[]) AS tenant_id, UNNEST($2::BIGINT[]) AS task_id, UNNEST($3:
     17153   25939     0.66        0  WITH new_slot_rows AS ( SELECT t.id, t.inserted_at, t.retry_count, t.tenant_id, t.workflow_run_id, t.external_id, t.concurrency_parent_strat
     12367   21600     0.57    21600  -- name: CreateTasks :many WITH input AS ( SELECT unnest($1::uuid[]) AS tenant_id, unnest($2::text[]) AS queue, unnest($3::text[]) AS action
     12245   30819     0.40     2400  INSERT INTO outbox.messages (topic, payload) SELECT $1 || nt.tenant_id::text || $2 || nt.strategy_id::text, jsonb_build_object( $3, $4, $5, 
     11701   16394     0.71    19200  -- name: CreateTasksOLAP :exec WITH inputs AS ( SELECT UNNEST($1::UUID[]) AS tenant_id, UNNEST($2::BIGINT[]) AS id, UNNEST($3::TIMESTAMPTZ[]
     10830   46641     0.23   106944  copy "v1_task_events_olap" ( "tenant_id", "task_id", "task_inserted_at", "event_type", "workflow_id", "event_timestamp", "readable_status", 
     10805   43946     0.25    72120  -- name: InsertOtelSpans :exec WITH inputs AS ( SELECT UNNEST($1::UUID[]) AS tenant_id, UNNEST($2::BYTEA[]) AS trace_id, UNNEST($3::BYTEA[])
     10772   25327     0.43     2400  -- name: ReleaseConcurrencySlots :batchexec WITH input AS ( SELECT unnest($1::bigint[]) AS task_id, unnest($2::timestamptz[]) AS task_insert
      8965   25336     0.35     2400  INSERT INTO outbox.messages (topic, payload) SELECT $1 || dr.tenant_id::text || $2 || dr.strategy_id::text, jsonb_build_object( $3, $4, $5, 
      7960   14104     0.56     4800  -- name: UpdateDAGStatusesFromOrchestratorEvents :many WITH inputs AS ( SELECT UNNEST($2::BIGINT[]) AS dag_id, UNNEST($3::TIMESTAMPTZ[]) AS 
      7844   20535     0.38    21728  -- name: FlushAssignedQueueItems :many WITH assign_input AS ( SELECT task_id, task_inserted_at, retry_count, worker_id, step_timeout FROM ( 
      6642   64163     0.10    23820  UPDATE v1_runs_olap r SET readable_status = n.readable_status FROM new_rows n WHERE r.id = n.id AND r.inserted_at = n.inserted_at AND r.kind
      6581   25327     0.26    21728  -- name: ReleaseTasks :batchmany WITH input AS ( SELECT unnest($1::bigint[]) AS task_id, unnest($2::timestamptz[]) AS task_inserted_at, unne
      5806   54076     0.11    76792  -- name: WritePayloads :exec WITH inputs AS ( SELECT DISTINCT UNNEST($1::BIGINT[]) AS id, UNNEST($2::TIMESTAMPTZ[]) AS inserted_at, UNNEST($
      4602   45081     0.10    57592  -- name: PutPayloads :exec WITH inputs AS ( SELECT UNNEST($1::UUID[]) AS external_id, UNNEST($2::TIMESTAMPTZ[]) AS inserted_at, UNNEST($3::J
      3977   16394     0.24    14400  INSERT INTO v1_runs_olap ( tenant_id, id, inserted_at, external_id, readable_status, kind, workflow_id, workflow_version_id, additional_meta
      3975    2400     1.66     2400  -- name: UpdateConcurrencySlotIsFilledBatch :one UPDATE v1_concurrency_slot SET is_filled = $1 WHERE task_id = $2 AND task_inserted_at = $3 
      3741   25315     0.15    31192  -- name: CreateTaskEvents :many -- We get a FOR UPDATE lock on tasks to prevent concurrent writes to the task events -- tables for each task
      3335   16394     0.20     3033  -- name: ReconcileTaskStatusesFromEvents :many WITH inputs AS ( SELECT UNNEST($1::BIGINT[]) AS task_id, UNNEST($2::TIMESTAMPTZ[]) AS task_in
      3273   97004     0.03    28460  INSERT INTO v1_statuses_olap ( external_id, inserted_at, tenant_id, workflow_id, kind, readable_status ) -- DISTINCT ON: nothing constraint-
      3191   39469     0.08    39469  -- name: GetTaskByExternalId :one SELECT t.id, t.inserted_at, t.tenant_id, t.queue, t.action_id, t.step_id, t.step_readable_id, t.workflow_i
      2945   11958     0.25    14400  -- name: BulkCreateEventsOLAP :many WITH to_insert AS ( SELECT UNNEST($1::UUID[]) AS tenant_id, UNNEST($2::UUID[]) AS external_id, UNNEST($3
      2811   14400     0.20    14400  -- name: BulkCreateEvents :many WITH to_insert AS ( SELECT UNNEST($1::UUID[]) AS tenant_id, UNNEST($2::UUID[]) AS external_id, UNNEST($3::TI
      2550   21600     0.12    19200  INSERT INTO v1_queue_item ( tenant_id, queue, task_id, task_inserted_at, external_id, action_id, step_id, workflow_id, workflow_run_id, sche

Raw samples are in the run's load-samples-* artifacts: https://github.com/hatchet-dev/hatchet/actions/runs/39773541583

fix(engine): tighten SNS signature verification

ca94350

github-actions Bot added the engine Related to the core Hatchet engine label Oct 22, 2026

igor-kupczynski marked this pull request as ready for review October 22, 2026 11:50

vercel Bot deployed to Preview October 22, 2026 11:51

View deployment

igor-kupczynski added 2 commits October 22, 2026 13:51

chore(engine): describe SNS host pattern

970668e

chore(engine): explain SNS cert URL check

3abfb8d

vercel Bot deployed to Preview October 22, 2026 11:55

View deployment

greptile-apps Bot reviewed Oct 22, 2026

View reviewed changes

Comment thread internal/integrations/ingestors/sns/sns.go

igor-kupczynski requested a review from juliusgeo October 22, 2026 14:01

test(engine): cover SNS cert URL check

c249432

vercel Bot deployed to Preview October 22, 2026 14:03

View deployment

juliusgeo approved these changes Oct 22, 2026

View reviewed changes

igor-kupczynski merged commit 5964405 into main Oct 22, 2026
56 checks passed

igor-kupczynski deleted the fix/sns-signature-verification branch October 22, 2026 14:16

Navigation Menu

Repository navigation

fix(engine): tighten SNS signature verification - #5279

fix(engine): tighten SNS signature verification#5279
igor-kupczynski merged 4 commits into
mainfrom
fix/sns-signature-verification

igor-kupczynski commented Oct 22, 2026

Uh oh!

vercel Bot commented Oct 22, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Oct 22, 2026 •

edited

Loading

Uh oh!

Uh oh!

github-actions Bot commented Oct 22, 2026

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

igor-kupczynski commented Oct 22, 2026

Description

Type of change

Uh oh!

vercel Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

github-actions Bot commented Oct 22, 2026

Runner CPU during the failed load (optimistic true, race false) run

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vercel Bot commented Oct 22, 2026 •

edited

Loading

greptile-apps Bot commented Oct 22, 2026 •

edited

Loading

Runner CPU during the failed `load (optimistic true, race false)` run