Skip to content

Commit 32c68f1

Browse files
author
Alex Wang
committed
feat(insight): add the remaining Workflow Insight exporters
- DynamoDBExporter: PutItem, pk=executionArn, optional sk=emittedAt - AuroraExporter: RDS Data API upsert, postgresql or mysql dialect - CloudWatchLogsExporter: PutLogEvents into a per-day stream of any log group - OTelExporter: OTLP/HTTP log record, http/json only - FirehoseExporter: PutRecord, one JSON line per record - EventBridgeExporter: PutEvents, DetailType = record status - RedshiftExporter: Redshift Data API MERGE by execution_arn - OpenSearchExporter: Index API PUT, SigV4 or basic auth - SQSExporter: SendMessage, FIFO group and dedup ids - HttpExporter: POST or PUT JSON with a timeout - FileExporter: ndjson append or one json file per execution - OperationsFormat / apply_operations_format shared by the flexible exporters - README: exporter table and one setup block per exporter
1 parent 630b777 commit 32c68f1

29 files changed

Lines changed: 3267 additions & 15 deletions

packages/aws-durable-execution-sdk-python-insight/README.md

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,155 @@ carrying the name-keyed `operationsByName` summary. The `S3Exporter` writes the
4848
lossless per-occurrence `operations` array, one object per execution
4949
(upsert-by-execution-name, so re-emission overwrites rather than appends).
5050

51+
## Exporters
52+
53+
All exporters live in `aws_durable_execution_sdk_python_insight.exporters` and
54+
are re-exported from the package root. Each serializes the record as compact
55+
JSON. Exporters that call AWS accept an injected `client=` for tests and use
56+
the Lambda runtime's boto3 otherwise; none adds a required dependency.
57+
58+
| Exporter | Destination | Upsert | Operations shape | Default size limit |
59+
| --- | --- | --- | --- | --- |
60+
| `LambdaLogExporter` | Function's own log group | No | `operationsByName` | 256 KB |
61+
| `CloudWatchLogsExporter` | Any log group, one stream per day | No | `operationsByName` | 256 KB |
62+
| `S3Exporter` | S3 object per execution | Yes (key) | `operations` | 5 MB |
63+
| `DynamoDBExporter` | DynamoDB item | Configurable | `operationsByName` | 400 KB |
64+
| `AuroraExporter` | Aurora MySQL/PostgreSQL row (Data API) | Yes (upsert) | full record as JSON column | 1 MB |
65+
| `RedshiftExporter` | Redshift row (Data API) | Yes (MERGE) | full record as SUPER column | 1 MB |
66+
| `OpenSearchExporter` | OpenSearch document | Yes (`_id`) | `operations` | 10 MB |
67+
| `FirehoseExporter` | Firehose delivery stream | N/A | `operations_format` | 1 MB |
68+
| `EventBridgeExporter` | EventBridge event | N/A | `operations_format` | 256 KB |
69+
| `SQSExporter` | SQS message | N/A | `operations_format` | 256 KB |
70+
| `OTelExporter` | OTLP/HTTP logs endpoint | N/A | `operations_format` | 1 MB |
71+
| `HttpExporter` | Any HTTP endpoint | N/A | `operations_format` | none |
72+
| `FileExporter` | Directory (EFS, mount, `/tmp`) | Configurable | `operations_format` | none |
73+
74+
`operations_format` is `"array"` (default), `"by-name"`, or `"both"`
75+
(`OperationsFormat`). `max_record_size_bytes` overrides the size limit on every
76+
exporter; `None` disables truncation.
77+
78+
### CloudWatchLogsExporter
79+
80+
Writes one `PutLogEvents` event per record to `log_group_name`, in a stream
81+
named `{log_stream_prefix}{YYYY}/{MM}/{DD}` (default prefix `workflow-insight/`).
82+
IAM: `logs:CreateLogStream`, `logs:PutLogEvents` on the log group.
83+
84+
```python
85+
CloudWatchLogsExporter(log_group_name="/custom/workflow-insight")
86+
```
87+
88+
### DynamoDBExporter
89+
90+
`PutItem` keyed by `partition_key` (default `pk`) = `executionArn`. With the
91+
default `sort_key="sk"` (= `emittedAt`) every export adds an item; pass
92+
`sort_key=None` for a key-only table that upserts. IAM: `dynamodb:PutItem`.
93+
94+
```python
95+
DynamoDBExporter(table_name="workflow-insight")
96+
```
97+
98+
### AuroraExporter
99+
100+
Upserts a row by `execution_arn` through the RDS Data API; `engine` is
101+
`"postgresql"` or `"mysql"` and selects the dialect. Columns: `execution_arn,
102+
execution_name, function_name, status, start_time, end_time, duration_ms,
103+
record_json, emitted_at`. IAM: `rds-data:ExecuteStatement`,
104+
`secretsmanager:GetSecretValue`.
105+
106+
```python
107+
AuroraExporter(
108+
resource_arn="arn:aws:rds:us-east-1:123456789012:cluster:my-cluster",
109+
secret_arn="arn:aws:secretsmanager:us-east-1:123456789012:secret:my-db-creds",
110+
database="insight",
111+
engine="postgresql",
112+
)
113+
```
114+
115+
### RedshiftExporter
116+
117+
`MERGE` by `execution_arn` through the Redshift Data API into
118+
`{schema}.{table}` (default `public.workflow_insight`, same columns as Aurora,
119+
`record_json` as `SUPER`). Provide `workgroup_name` (Serverless) or
120+
`cluster_identifier` (provisioned, with `db_user` or `secret_arn`). IAM:
121+
`redshift-data:ExecuteStatement` plus `redshift-serverless:GetCredentials` or
122+
`secretsmanager:GetSecretValue`.
123+
124+
```python
125+
RedshiftExporter(database="insight", workgroup_name="insight-wg")
126+
```
127+
128+
### OpenSearchExporter
129+
130+
`PUT {endpoint}/{index_name}/_doc/{executionArn}` (default index
131+
`workflow-insight`). `auth="sigv4"` (default) signs with the runtime's
132+
credentials via botocore; `auth="basic"` uses `username`/`password`. IAM:
133+
`es:ESHttpPut` on the domain.
134+
135+
```python
136+
OpenSearchExporter(endpoint="https://my-domain.us-east-1.es.amazonaws.com", region="us-east-1")
137+
```
138+
139+
### FirehoseExporter
140+
141+
`PutRecord` of one JSON line (trailing newline) per record. IAM:
142+
`firehose:PutRecord`.
143+
144+
```python
145+
FirehoseExporter(delivery_stream_name="workflow-insight-stream")
146+
```
147+
148+
### EventBridgeExporter
149+
150+
`PutEvents` with `Source` (default `aws.durable-execution.insight`),
151+
`DetailType` = record status, `Detail` = record. All arguments optional. IAM:
152+
`events:PutEvents`.
153+
154+
```python
155+
EventBridgeExporter(event_bus_name="default")
156+
```
157+
158+
### SQSExporter
159+
160+
`SendMessage` with the record as body and `status`/`functionName` message
161+
attributes. A `.fifo` queue URL enables `MessageGroupId` (default
162+
`executionArn`, or `message_group_id`) and a deduplication id of
163+
`executionArn:emittedAt`. IAM: `sqs:SendMessage`.
164+
165+
```python
166+
SQSExporter(queue_url="https://sqs.us-east-1.amazonaws.com/123456789012/insight")
167+
```
168+
169+
### OTelExporter
170+
171+
POSTs one OTLP `ExportLogsServiceRequest` (`http/json` only) per record to
172+
`endpoint`; identity fields become attributes and the record is the log body.
173+
Pass vendor auth in `headers`. No IAM.
174+
175+
```python
176+
OTelExporter(endpoint="https://otlp.vendor.com/v1/logs", headers={"x-api-key": "..."})
177+
```
178+
179+
### HttpExporter
180+
181+
`POST` (or `method="PUT"`) the record as JSON to `url` with
182+
`Content-Type: application/json` plus `headers`; a non-2xx status raises.
183+
`timeout_ms` defaults to 10000. No IAM.
184+
185+
```python
186+
HttpExporter(url="https://hooks.example.com/insight", headers={"Authorization": "Bearer ..."})
187+
```
188+
189+
### FileExporter
190+
191+
`mode="ndjson"` (default) appends to `{directory}/{YYYY-MM-DD}.ndjson`;
192+
`mode="json"` writes `{directory}/{executionName}.json`, overwriting on update.
193+
For Lambda use an EFS mount (IAM: `elasticfilesystem:ClientMount`,
194+
`elasticfilesystem:ClientWrite`) or `/tmp` for testing.
195+
196+
```python
197+
FileExporter(directory="/mnt/efs/workflow-insight")
198+
```
199+
51200
Emission behavior, record schema (`recordType: WorkflowInsight`,
52201
`schemaVersion: "1.0"`), sampling, content configuration (input/output
53202
omission, `include_errors`, per-operation result opt-in), truncation phases,

packages/aws-durable-execution-sdk-python-insight/src/aws_durable_execution_sdk_python_insight/__init__.py

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,11 +5,29 @@
55

66
from aws_durable_execution_sdk_python_insight.__about__ import __version__
77
from aws_durable_execution_sdk_python_insight.exporters import (
8+
AuroraEngine,
9+
AuroraExporter,
10+
CloudWatchLogsExporter,
11+
DynamoDBExporter,
12+
EventBridgeExporter,
13+
FileExporter,
14+
FileMode,
15+
FirehoseExporter,
16+
HttpExporter,
17+
HttpMethod,
818
LambdaLogExporter,
19+
OpenSearchAuth,
20+
OpenSearchExporter,
21+
OTelExporter,
22+
OTelProtocol,
23+
RedshiftExporter,
924
S3Exporter,
1025
S3Partitioning,
26+
SQSExporter,
1127
)
1228
from aws_durable_execution_sdk_python_insight.operations_index import (
29+
OperationsFormat,
30+
apply_operations_format,
1331
build_operations_by_name,
1432
with_operations_by_name,
1533
)
@@ -31,17 +49,35 @@
3149

3250
__all__ = [
3351
"__version__",
52+
"AuroraEngine",
53+
"AuroraExporter",
54+
"CloudWatchLogsExporter",
3455
"ContentConfig",
3556
"ContentOperations",
57+
"DynamoDBExporter",
3658
"EmitMode",
59+
"EventBridgeExporter",
60+
"FileExporter",
61+
"FileMode",
62+
"FirehoseExporter",
63+
"HttpExporter",
64+
"HttpMethod",
3765
"InsightExporter",
3866
"LambdaLogExporter",
67+
"OTelExporter",
68+
"OTelProtocol",
69+
"OpenSearchAuth",
70+
"OpenSearchExporter",
3971
"OperationDetail",
4072
"OperationOverride",
73+
"OperationsFormat",
74+
"RedshiftExporter",
4175
"S3Exporter",
4276
"S3Partitioning",
77+
"SQSExporter",
4378
"WorkflowInsightConfig",
4479
"WorkflowInsightPlugin",
80+
"apply_operations_format",
4581
"build_operations_by_name",
4682
"truncate_record",
4783
"with_operations_by_name",

packages/aws-durable-execution-sdk-python-insight/src/aws_durable_execution_sdk_python_insight/exporters/__init__.py

Lines changed: 62 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -3,36 +3,84 @@
33
# SPDX-License-Identifier: Apache-2.0
44
"""First-party Workflow Insight exporters.
55
6-
One module per exporter, mirroring the JS package's ``src/exporters/`` layout
7-
(``aws-durable-execution-sdk-js-insight``). Each destination lives in its own
8-
module so the set can grow to the full JS parity surface (S3, CloudWatch Logs,
9-
DynamoDB, Firehose, EventBridge, SQS, OpenSearch, Redshift, Aurora, HTTP, OTel,
10-
file, ...) without any single file accreting every backend's imports and
11-
optional dependencies.
6+
One module per destination, so no single file accretes every backend's imports
7+
and optional dependencies. Concrete exporters are re-exported here so the
8+
public import path is stable:
9+
``from aws_durable_execution_sdk_python_insight.exporters import S3Exporter``.
10+
Shared serialization and transport helpers live in the private ``_common``
11+
module.
1212
13-
Concrete exporters are re-exported here so the public import path is stable:
14-
``from aws_durable_execution_sdk_python_insight.exporters import S3Exporter``
15-
keeps working exactly as before this package was split out of a single module.
16-
Shared serialization helpers live in the private ``_common`` module.
17-
18-
Both shipped exporters serialize the curated record with JS-compatible compact
19-
JSON (no whitespace) so the wire bytes match across SDKs. Records are written
20-
verbatim -- no synthetic emission.
13+
Every exporter serializes the curated record as compact JSON (no whitespace,
14+
non-ASCII preserved). Records are written verbatim -- no synthetic emission.
2115
"""
2216

2317
from __future__ import annotations
2418

19+
from aws_durable_execution_sdk_python_insight.exporters.aurora_exporter import (
20+
AuroraEngine,
21+
AuroraExporter,
22+
)
23+
from aws_durable_execution_sdk_python_insight.exporters.cloudwatch_logs_exporter import (
24+
CloudWatchLogsExporter,
25+
)
26+
from aws_durable_execution_sdk_python_insight.exporters.dynamodb_exporter import (
27+
DynamoDBExporter,
28+
)
29+
from aws_durable_execution_sdk_python_insight.exporters.eventbridge_exporter import (
30+
EventBridgeExporter,
31+
)
32+
from aws_durable_execution_sdk_python_insight.exporters.file_exporter import (
33+
FileExporter,
34+
FileMode,
35+
)
36+
from aws_durable_execution_sdk_python_insight.exporters.firehose_exporter import (
37+
FirehoseExporter,
38+
)
39+
from aws_durable_execution_sdk_python_insight.exporters.http_exporter import (
40+
HttpExporter,
41+
HttpMethod,
42+
)
2543
from aws_durable_execution_sdk_python_insight.exporters.lambda_log_exporter import (
2644
LambdaLogExporter,
2745
)
46+
from aws_durable_execution_sdk_python_insight.exporters.opensearch_exporter import (
47+
OpenSearchAuth,
48+
OpenSearchExporter,
49+
)
50+
from aws_durable_execution_sdk_python_insight.exporters.otel_exporter import (
51+
OTelExporter,
52+
OTelProtocol,
53+
)
54+
from aws_durable_execution_sdk_python_insight.exporters.redshift_exporter import (
55+
RedshiftExporter,
56+
)
2857
from aws_durable_execution_sdk_python_insight.exporters.s3_exporter import (
2958
S3Exporter,
3059
S3Partitioning,
3160
)
61+
from aws_durable_execution_sdk_python_insight.exporters.sqs_exporter import (
62+
SQSExporter,
63+
)
3264

3365

3466
__all__ = [
67+
"AuroraEngine",
68+
"AuroraExporter",
69+
"CloudWatchLogsExporter",
70+
"DynamoDBExporter",
71+
"EventBridgeExporter",
72+
"FileExporter",
73+
"FileMode",
74+
"FirehoseExporter",
75+
"HttpExporter",
76+
"HttpMethod",
3577
"LambdaLogExporter",
78+
"OTelExporter",
79+
"OTelProtocol",
80+
"OpenSearchAuth",
81+
"OpenSearchExporter",
82+
"RedshiftExporter",
3683
"S3Exporter",
3784
"S3Partitioning",
85+
"SQSExporter",
3886
]

packages/aws-durable-execution-sdk-python-insight/src/aws_durable_execution_sdk_python_insight/exporters/_common.py

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,11 +10,17 @@
1010

1111
from __future__ import annotations
1212

13+
import datetime
1314
import json
1415
import re
16+
import urllib.error
17+
import urllib.request
1518
from typing import Any
1619

1720

21+
_SQL_IDENTIFIER = re.compile(r"^[a-zA-Z_][a-zA-Z0-9_]*$")
22+
23+
1824
def compact_dumps(value: Any) -> str:
1925
"""Serialize ``value`` as compact JSON (no whitespace, non-ASCII preserved).
2026
@@ -27,3 +33,56 @@ def compact_dumps(value: Any) -> str:
2733
def sanitize(value: str) -> str:
2834
"""Replace characters unsafe for object keys / file names with ``_``."""
2935
return re.sub(r"[^a-zA-Z0-9._-]", "_", value)
36+
37+
38+
def sql_identifier(name: str) -> str:
39+
"""Return ``name`` if it is a plain SQL identifier, else raise ``ValueError``.
40+
41+
Table and schema names are interpolated into SQL text, so only letters,
42+
digits, and underscores are accepted.
43+
"""
44+
if not _SQL_IDENTIFIER.match(name):
45+
msg = (
46+
f'Invalid SQL identifier: "{name}". '
47+
"Only letters, digits, and underscores are allowed."
48+
)
49+
raise ValueError(msg)
50+
return name
51+
52+
53+
def parse_iso_datetime(value: str) -> datetime.datetime:
54+
"""Parse an ISO-8601 timestamp (``Z`` or offset) into an aware UTC datetime."""
55+
parsed = datetime.datetime.fromisoformat(value.replace("Z", "+00:00"))
56+
if parsed.tzinfo is None:
57+
parsed = parsed.replace(tzinfo=datetime.UTC)
58+
return parsed.astimezone(datetime.UTC)
59+
60+
61+
def http_send(
62+
method: str,
63+
url: str,
64+
headers: dict[str, str],
65+
body: bytes,
66+
timeout: float | None = None,
67+
) -> tuple[int, str, str]:
68+
"""Send one HTTP request and return ``(status, reason, response_text)``.
69+
70+
A non-2xx status is returned, not raised, so callers build their own error
71+
message. Network errors and timeouts propagate.
72+
"""
73+
request = urllib.request.Request(url, data=body, method=method)
74+
for key, value in headers.items():
75+
request.add_header(key, value)
76+
try:
77+
with urllib.request.urlopen(request, timeout=timeout) as response: # noqa: S310
78+
return (
79+
int(response.status),
80+
str(response.reason or ""),
81+
response.read().decode("utf-8", errors="replace"),
82+
)
83+
except urllib.error.HTTPError as exc:
84+
try:
85+
detail = exc.read().decode("utf-8", errors="replace")
86+
except Exception: # noqa: BLE001 - the body is best-effort detail only
87+
detail = ""
88+
return int(exc.code), str(exc.reason or ""), detail

0 commit comments

Comments
 (0)