Skip to content

Commit 3a424e7

Browse files
author
Alex Wang
committed
feat(insight): add the remaining Workflow Insight exporters
- DynamoDBExporter: PutItem, pk=executionArn, optional sk=emittedAt - AuroraExporter: RDS Data API upsert, postgresql or mysql dialect - CloudWatchLogsExporter: PutLogEvents into a per-day stream of any log group - OTelExporter: OTLP/HTTP log record, http/json only - FirehoseExporter: PutRecord, one JSON line per record - EventBridgeExporter: PutEvents, DetailType = record status - RedshiftExporter: Redshift Data API MERGE by execution_arn - OpenSearchExporter: Index API PUT, SigV4 or basic auth - SQSExporter: SendMessage, FIFO group and dedup ids - HttpExporter: POST or PUT JSON with a timeout - FileExporter: ndjson append or one json file per execution - OperationsFormat / apply_operations_format shared by the flexible exporters - README: exporter table and one setup block per exporter
1 parent 8742ad9 commit 3a424e7

32 files changed

Lines changed: 3589 additions & 17 deletions

‎packages/aws-durable-execution-sdk-python-insight/README.md‎

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,160 @@ carrying the name-keyed `operationsByName` summary. The `S3Exporter` writes the
4848
lossless per-occurrence `operations` array, one object per execution
4949
(upsert-by-execution-name, so re-emission overwrites rather than appends).
5050

51+
## Exporters
52+
53+
All exporters live in `aws_durable_execution_sdk_python_insight.exporters` and
54+
are re-exported from the package root. Each serializes the record as compact
55+
JSON. Exporters that call AWS accept an injected `client=` for tests and use
56+
the Lambda runtime's boto3 otherwise; none adds a required dependency.
57+
58+
| Exporter | Destination | Upsert | Operations shape | Default size limit |
59+
| --- | --- | --- | --- | --- |
60+
| `LambdaLogExporter` | Function's own log group | No | `operationsByName` | 256 KB |
61+
| `CloudWatchLogsExporter` | Any log group, one stream per day | No | `operationsByName` | 256 KB |
62+
| `S3Exporter` | S3 object per execution | Yes (key) | `operations` | 5 MB |
63+
| `DynamoDBExporter` | DynamoDB item | Configurable | `operationsByName` | 400 KB |
64+
| `AuroraExporter` | Aurora MySQL/PostgreSQL row (Data API) | Yes (upsert) | full record as JSON column | 1 MB |
65+
| `RedshiftExporter` | Redshift row (Data API) | Yes (MERGE) | full record as SUPER column | 1 MB |
66+
| `OpenSearchExporter` | OpenSearch document | Yes (`_id`) | `operations` | 10 MB |
67+
| `FirehoseExporter` | Firehose delivery stream | N/A | `operations_format` | 1 MB |
68+
| `EventBridgeExporter` | EventBridge event | N/A | `operations_format` | 256 KB |
69+
| `SQSExporter` | SQS message | N/A | `operations_format` | 256 KB |
70+
| `OTelExporter` | OTLP/HTTP logs endpoint | N/A | `operations_format` | 1 MB |
71+
| `HttpExporter` | Any HTTP endpoint | N/A | `operations_format` | none |
72+
| `FileExporter` | Directory (EFS, mount, `/tmp`) | Configurable | `operations_format` | none |
73+
74+
`operations_format` is `"array"` (default), `"by-name"`, or `"both"`
75+
(`OperationsFormat`). `max_record_size_bytes` raises or lowers an exporter's
76+
size limit; omitting it keeps the default. `HttpExporter` and `FileExporter`
77+
have no default and do not truncate unless a limit is set.
78+
79+
### CloudWatchLogsExporter
80+
81+
Writes one `PutLogEvents` event per record to `log_group_name`, in a stream
82+
named `{log_stream_prefix}{YYYY}/{MM}/{DD}` (default prefix `workflow-insight/`).
83+
IAM: `logs:CreateLogStream`, `logs:PutLogEvents` on the log group.
84+
85+
```python
86+
CloudWatchLogsExporter(log_group_name="/custom/workflow-insight")
87+
```
88+
89+
### DynamoDBExporter
90+
91+
`PutItem` keyed by `partition_key` (default `pk`) = `executionArn`. With the
92+
default `sort_key="sk"` (= `emittedAt`) every export adds an item; pass
93+
`sort_key=None` for a key-only table that upserts. IAM: `dynamodb:PutItem`.
94+
95+
```python
96+
DynamoDBExporter(table_name="workflow-insight")
97+
```
98+
99+
### AuroraExporter
100+
101+
Upserts a row by `execution_arn` through the RDS Data API; `engine` is
102+
`"postgresql"` or `"mysql"` and selects the dialect. Columns: `execution_arn,
103+
execution_name, function_name, status, start_time, end_time, duration_ms,
104+
record_json, emitted_at`. IAM: `rds-data:ExecuteStatement`,
105+
`secretsmanager:GetSecretValue`.
106+
107+
```python
108+
AuroraExporter(
109+
resource_arn="arn:aws:rds:us-east-1:123456789012:cluster:my-cluster",
110+
secret_arn="arn:aws:secretsmanager:us-east-1:123456789012:secret:my-db-creds",
111+
database="insight",
112+
engine="postgresql",
113+
)
114+
```
115+
116+
### RedshiftExporter
117+
118+
`MERGE` by `execution_arn` through the Redshift Data API into
119+
`{schema}.{table}` (default `public.workflow_insight`, same columns as Aurora,
120+
`record_json` as `SUPER`). Provide `workgroup_name` (Serverless) or
121+
`cluster_identifier` (provisioned, with `db_user` or `secret_arn`). IAM:
122+
`redshift-data:ExecuteStatement` plus `redshift-serverless:GetCredentials` or
123+
`secretsmanager:GetSecretValue`. The statement is submitted and not awaited, so
124+
statement failures are not reported and `on-change` records may land out of
125+
order; prefer the default `on-complete` emit mode with this exporter.
126+
127+
```python
128+
RedshiftExporter(database="insight", workgroup_name="insight-wg")
129+
```
130+
131+
### OpenSearchExporter
132+
133+
`PUT {endpoint}/{index_name}/_doc/{executionArn}` (default index
134+
`workflow-insight`). `auth="sigv4"` (default) signs with the runtime's
135+
credentials via botocore; `auth="basic"` uses `username`/`password`. IAM:
136+
`es:ESHttpPut` on the domain.
137+
138+
```python
139+
OpenSearchExporter(endpoint="https://my-domain.us-east-1.es.amazonaws.com", region="us-east-1")
140+
```
141+
142+
### FirehoseExporter
143+
144+
`PutRecord` of one JSON line (trailing newline) per record. IAM:
145+
`firehose:PutRecord`.
146+
147+
```python
148+
FirehoseExporter(delivery_stream_name="workflow-insight-stream")
149+
```
150+
151+
### EventBridgeExporter
152+
153+
`PutEvents` with `Source` (default `aws.durable-execution.insight`),
154+
`DetailType` = record status, `Detail` = record. All arguments optional. IAM:
155+
`events:PutEvents`.
156+
157+
```python
158+
EventBridgeExporter(event_bus_name="default")
159+
```
160+
161+
### SQSExporter
162+
163+
`SendMessage` with the record as body and `status`/`functionName` message
164+
attributes. A `.fifo` queue URL enables `MessageGroupId` (default
165+
`executionArn`, or `message_group_id`) and a deduplication id of
166+
`executionArn:emittedAt`. IAM: `sqs:SendMessage`.
167+
168+
```python
169+
SQSExporter(queue_url="https://sqs.us-east-1.amazonaws.com/123456789012/insight")
170+
```
171+
172+
### OTelExporter
173+
174+
POSTs one OTLP `ExportLogsServiceRequest` (`http/json` only) per record to
175+
`endpoint`; identity fields become attributes and the record is the log body.
176+
Pass vendor auth in `headers`. No IAM.
177+
178+
```python
179+
OTelExporter(endpoint="https://otlp.vendor.com/v1/logs", headers={"x-api-key": "..."})
180+
```
181+
182+
### HttpExporter
183+
184+
`POST` (or `method="PUT"`) the record as JSON to `url` with
185+
`Content-Type: application/json` plus `headers`; a non-2xx status raises.
186+
`timeout_ms` defaults to 10000. No IAM.
187+
188+
```python
189+
HttpExporter(url="https://hooks.example.com/insight", headers={"Authorization": "Bearer ..."})
190+
```
191+
192+
### FileExporter
193+
194+
`mode="ndjson"` (default) appends to `{directory}/{YYYY-MM-DD}.ndjson`;
195+
`mode="json"` writes `{directory}/{executionName}.json`, overwriting on update.
196+
For Lambda use an EFS mount (IAM: `elasticfilesystem:ClientMount`,
197+
`elasticfilesystem:ClientWrite`) or `/tmp` for testing. Appends from many
198+
concurrent environments to one NDJSON file on a network file system can
199+
interleave; use `mode="json"` when several environments share a directory.
200+
201+
```python
202+
FileExporter(directory="/mnt/efs/workflow-insight")
203+
```
204+
51205
Emission behavior, record schema (`recordType: WorkflowInsight`,
52206
`schemaVersion: "1.0"`), sampling, content configuration (input/output
53207
omission, `include_errors`, per-operation result opt-in), truncation phases,

‎packages/aws-durable-execution-sdk-python-insight/src/aws_durable_execution_sdk_python_insight/__init__.py‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,11 +5,29 @@
55

66
from aws_durable_execution_sdk_python_insight.__about__ import __version__
77
from aws_durable_execution_sdk_python_insight.exporters import (
8+
AuroraEngine,
9+
AuroraExporter,
10+
CloudWatchLogsExporter,
11+
DynamoDBExporter,
12+
EventBridgeExporter,
13+
FileExporter,
14+
FileMode,
15+
FirehoseExporter,
16+
HttpExporter,
17+
HttpMethod,
818
LambdaLogExporter,
19+
OpenSearchAuth,
20+
OpenSearchExporter,
21+
OTelExporter,
22+
OTelProtocol,
23+
RedshiftExporter,
924
S3Exporter,
1025
S3Partitioning,
26+
SQSExporter,
1127
)
1228
from aws_durable_execution_sdk_python_insight.operations_index import (
29+
OperationsFormat,
30+
apply_operations_format,
1331
build_operations_by_name,
1432
with_operations_by_name,
1533
)
@@ -31,17 +49,35 @@
3149

3250
__all__ = [
3351
"__version__",
52+
"AuroraEngine",
53+
"AuroraExporter",
54+
"CloudWatchLogsExporter",
3455
"ContentConfig",
3556
"ContentOperations",
57+
"DynamoDBExporter",
3658
"EmitMode",
59+
"EventBridgeExporter",
60+
"FileExporter",
61+
"FileMode",
62+
"FirehoseExporter",
63+
"HttpExporter",
64+
"HttpMethod",
3765
"InsightExporter",
3866
"LambdaLogExporter",
67+
"OTelExporter",
68+
"OTelProtocol",
69+
"OpenSearchAuth",
70+
"OpenSearchExporter",
3971
"OperationDetail",
4072
"OperationOverride",
73+
"OperationsFormat",
74+
"RedshiftExporter",
4175
"S3Exporter",
4276
"S3Partitioning",
77+
"SQSExporter",
4378
"WorkflowInsightConfig",
4479
"WorkflowInsightPlugin",
80+
"apply_operations_format",
4581
"build_operations_by_name",
4682
"truncate_record",
4783
"with_operations_by_name",

‎packages/aws-durable-execution-sdk-python-insight/src/aws_durable_execution_sdk_python_insight/exporters/__init__.py‎

Lines changed: 62 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -3,36 +3,84 @@
33
# SPDX-License-Identifier: Apache-2.0
44
"""First-party Workflow Insight exporters.
55
6-
One module per exporter, mirroring the JS package's ``src/exporters/`` layout
7-
(``aws-durable-execution-sdk-js-insight``). Each destination lives in its own
8-
module so the set can grow to the full JS parity surface (S3, CloudWatch Logs,
9-
DynamoDB, Firehose, EventBridge, SQS, OpenSearch, Redshift, Aurora, HTTP, OTel,
10-
file, ...) without any single file accreting every backend's imports and
11-
optional dependencies.
6+
One module per destination, so no single file accretes every backend's imports
7+
and optional dependencies. Concrete exporters are re-exported here so the
8+
public import path is stable:
9+
``from aws_durable_execution_sdk_python_insight.exporters import S3Exporter``.
10+
Shared serialization and transport helpers live in the private ``_common``
11+
module.
1212
13-
Concrete exporters are re-exported here so the public import path is stable:
14-
``from aws_durable_execution_sdk_python_insight.exporters import S3Exporter``
15-
keeps working exactly as before this package was split out of a single module.
16-
Shared serialization helpers live in the private ``_common`` module.
17-
18-
Both shipped exporters serialize the curated record with JS-compatible compact
19-
JSON (no whitespace) so the wire bytes match across SDKs. Records are written
20-
verbatim -- no synthetic emission.
13+
Every exporter serializes the curated record as compact JSON (no whitespace,
14+
non-ASCII preserved). Records are written verbatim -- no synthetic emission.
2115
"""
2216

2317
from __future__ import annotations
2418

19+
from aws_durable_execution_sdk_python_insight.exporters.aurora_exporter import (
20+
AuroraEngine,
21+
AuroraExporter,
22+
)
23+
from aws_durable_execution_sdk_python_insight.exporters.cloudwatch_logs_exporter import (
24+
CloudWatchLogsExporter,
25+
)
26+
from aws_durable_execution_sdk_python_insight.exporters.dynamodb_exporter import (
27+
DynamoDBExporter,
28+
)
29+
from aws_durable_execution_sdk_python_insight.exporters.eventbridge_exporter import (
30+
EventBridgeExporter,
31+
)
32+
from aws_durable_execution_sdk_python_insight.exporters.file_exporter import (
33+
FileExporter,
34+
FileMode,
35+
)
36+
from aws_durable_execution_sdk_python_insight.exporters.firehose_exporter import (
37+
FirehoseExporter,
38+
)
39+
from aws_durable_execution_sdk_python_insight.exporters.http_exporter import (
40+
HttpExporter,
41+
HttpMethod,
42+
)
2543
from aws_durable_execution_sdk_python_insight.exporters.lambda_log_exporter import (
2644
LambdaLogExporter,
2745
)
46+
from aws_durable_execution_sdk_python_insight.exporters.opensearch_exporter import (
47+
OpenSearchAuth,
48+
OpenSearchExporter,
49+
)
50+
from aws_durable_execution_sdk_python_insight.exporters.otel_exporter import (
51+
OTelExporter,
52+
OTelProtocol,
53+
)
54+
from aws_durable_execution_sdk_python_insight.exporters.redshift_exporter import (
55+
RedshiftExporter,
56+
)
2857
from aws_durable_execution_sdk_python_insight.exporters.s3_exporter import (
2958
S3Exporter,
3059
S3Partitioning,
3160
)
61+
from aws_durable_execution_sdk_python_insight.exporters.sqs_exporter import (
62+
SQSExporter,
63+
)
3264

3365

3466
__all__ = [
67+
"AuroraEngine",
68+
"AuroraExporter",
69+
"CloudWatchLogsExporter",
70+
"DynamoDBExporter",
71+
"EventBridgeExporter",
72+
"FileExporter",
73+
"FileMode",
74+
"FirehoseExporter",
75+
"HttpExporter",
76+
"HttpMethod",
3577
"LambdaLogExporter",
78+
"OTelExporter",
79+
"OTelProtocol",
80+
"OpenSearchAuth",
81+
"OpenSearchExporter",
82+
"RedshiftExporter",
3683
"S3Exporter",
3784
"S3Partitioning",
85+
"SQSExporter",
3886
]

0 commit comments

Comments
 (0)