Skip to content

Commit d17ca91

Browse files
authored
Add an Event Hubs fraud detection sample (#104)
* Add an Event Hubs fraud detection sample A payment stream processed end to end on the emulator: producers publish over AMQP, Kafka and HTTPS into one event hub, an Event Hubs-triggered Function App scores each payment and emits alerts to a second hub, Capture archives the stream to Blob Storage as Avro, and a Web App dashboard shows partitions, checkpoint lag, alerts, decoded archives and registered schemas. Key Vault holds the connection strings, Log Analytics and Application Insights collect telemetry. Ships the three deployment paths the repo uses - Azure CLI scripts, Bicep and Terraform - plus validate and end-to-end demo scripts, and registers all three with run-samples.sh. * Make the Event Hubs sample fail when the pipeline does not work run-pipeline.sh described its results instead of asserting them: a dead processor printed 'No alerts yet' and exited 0, so CI could go green with nothing working. Every stage is now an assertion that exits non-zero and prints the Function App's container logs - the payments hub must grow by the events published, alerts must grow past a baseline taken before ingestion, the restarted processor must resume without replaying the backlog, and Capture must produce an archive that decodes. read_capture.py no longer exits 0 when there is nothing to decode, and the two warning-only failures in deploy.sh plus the Bicep validation warning are fatal. The Terraform and Bicep entries had no test command at all, so those shards only provisioned. Both now run scripts/validate.sh, and their watch_folders include the shared scripts directory so a change there re-runs them. That immediately caught Terraform generating different storage and Key Vault names than the other two paths; they are aligned on the same four-character suffix. Adds the Event Hubs packages the sample's host-side scripts need to requirements-runtime.txt - their absence, not the emulator, is what failed the shard - and ignores .deployment-env, which holds live connection strings. * Bump the Event Hubs sample's Terraform to azurerm 4.81.0 Matches the version the other samples were moved to. Verified against the emulator rather than only pinned: init, fmt and validate are clean, deploy.sh applies and converges on a second apply, scripts/validate.sh passes 33 of 33, and destroy completes. No schema or deprecation changes were needed. * Address review feedback on the Event Hubs sample Fixes every comment on PR #104. The dashboard's alerts endpoint could never return: receive_batch() ends only when the client is closed, and the only close was the context-manager exit that came after it. It now receives on a worker thread and closes from the request thread, waiting for every partition to drain before returning. src/producers/requirements.txt was missing azure-storage-blob and fastavro, so the documented quick start could not run its own demo. Cloud-parity corrections: application groups are documented as an emulator-only check (Azure offers them on Premium and Dedicated only), geo-DR now pairs across regions, the Key Vault template asks for soft delete instead of a value Azure refuses, and the schema panel uses the documented versions route rather than a list route that is not in the data plane. The dashboard now holds a namespace-wide Listen-only rule instead of RootManageSharedAccessKey, in all three deployment paths, and every connection string is validated before use. Both IaC paths now write scripts/.deployment-env, so the end-to-end demo runs after a Terraform or Bicep deployment too. Smaller fixes: a wrong recovery hint, a tautological port fallback, TLS verification scoped to the emulator host, the Kafka SASL username, a caveat on the undocumented BrokerProperties header, Bootstrap classes matched to the vendored 5.1.3, and the Terraform Key Vault secret reusing the explicit-endpoint connection string. Also drops a function package that was committed by accident.
1 parent f94022a commit d17ca91

86 files changed

Lines changed: 64921 additions & 3 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -223,10 +223,18 @@ cython_debug/
223223
.idea/
224224
## Terraform
225225
.terraform/
226+
.terraform.lock.hcl
226227
*.tfstate
227228
*.tfstate.backup
228229
tfplan
229230
override.tf
230231
override.tf.json
231232
*_override.tf
232233
*_override.tf.json
234+
235+
## Sample deployment artifacts
236+
# Written by a sample's deploy.sh and read back by its other scripts. It holds live
237+
# connection strings, so it must never be committed.
238+
.deployment-env
239+
# Application packages the deploy scripts build on the fly before publishing them.
240+
*.zip

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ This repository contains comprehensive sample projects demonstrating how to deve
3939
| [Web App with Custom Docker Image](./samples/web-app-custom-image/python/README.md) | Azure Web App running a custom Docker image |
4040
| [ACI and Blob Storage](./samples/aci-blob-storage/python/README.md) | Azure Container Instances with ACR, Key Vault, and Blob Storage |
4141
| [Azure Service Bus with Spring Boot](./samples/servicebus/java/README.md) | Azure Service Bus used by a Spring Boot application |
42+
| [Event Hubs Fraud Detection Pipeline](./samples/eventhubs/python/README.md) | Real-time payment stream processing with Event Hubs (AMQP, Kafka and HTTPS ingestion, Capture, Schema Registry), an Event Hubs-triggered Function App, Key Vault, Storage and a Web App dashboard |
4243

4344
## Sample Structure
4445

‎requirements-runtime.txt‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,5 +6,9 @@ azure-identity
66
azure-storage-blob
77
azure-mgmt-cosmosdb
88
azure-core
9+
azure-eventhub
10+
confluent-kafka
11+
fastavro
12+
requests
913
python-dotenv
1014
localstack

‎run-samples.sh‎

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ PURGE_DOCKER="${PURGE_DOCKER:-0}"
3030
# 1. Define Samples (placed before tool checks so --list works without dependencies)
3131
SAMPLES=(
3232
"samples/servicebus/java|bash scripts/deploy.sh"
33+
"samples/eventhubs/python|bash scripts/deploy.sh|bash scripts/validate.sh && bash scripts/run-pipeline.sh"
3334
"samples/function-app-front-door/python|bash scripts/deploy_all.sh --name-prefix testafd|"
3435
"samples/function-app-managed-identity/python|bash scripts/user-managed-identity.sh|bash scripts/validate.sh && bash scripts/test.sh"
3536
"samples/function-app-service-bus/dotnet|bash scripts/deploy.sh|bash scripts/validate.sh && bash scripts/call-http-trigger.sh"
@@ -46,6 +47,7 @@ SAMPLES=(
4647
# 1a. Define Terraform Samples
4748
TERRAFORM_SAMPLES=(
4849
"samples/servicebus/java/terraform|bash deploy.sh"
50+
"samples/eventhubs/python/terraform|bash deploy.sh|bash ../scripts/validate.sh"
4951
"samples/function-app-managed-identity/python/terraform|bash deploy.sh"
5052
"samples/function-app-service-bus/dotnet/terraform|bash deploy.sh"
5153
"samples/function-app-storage-http/dotnet/terraform|bash deploy.sh"
@@ -60,6 +62,7 @@ TERRAFORM_SAMPLES=(
6062
# 1b. Define Bicep Samples
6163
BICEP_SAMPLES=(
6264
"samples/servicebus/java/bicep|bash deploy.sh"
65+
"samples/eventhubs/python/bicep|bash deploy.sh|bash ../scripts/validate.sh"
6366
#"samples/web-app-sql-database/python/bicep|bash deploy.sh"
6467
"samples/function-app-managed-identity/python/bicep|bash deploy.sh"
6568
"samples/function-app-service-bus/dotnet/bicep|bash deploy.sh"
@@ -84,15 +87,19 @@ if [[ "${1:-}" == "--list" ]]; then
8487
IFS='|' read -r path _ _ <<< "${ALL_SAMPLES[$i]}"
8588

8689
if [[ "$path" == */terraform || "$path" == */bicep ]]; then
87-
watch=("$path" "$(dirname "$path")/src")
90+
# An IaC sample deploys from its own folder, but its test command may run the
91+
# sample's shared scripts, so a change there has to re-run it too. Folders that do
92+
# not exist simply never match a changed file.
93+
watch=("$path" "$(dirname "$path")/src" "$(dirname "$path")/scripts")
8894
name="${path#samples/}"
8995
else
9096
watch=("$path/scripts" "$path/src")
9197
name="${path#samples/}/scripts"
9298
fi
9399

94-
printf ' {"shard":%d,"splits":%d,"name":"%s","watch_folders":["%s","%s"]}' \
95-
$((i+1)) "$TOTAL" "$name" "${watch[0]}" "${watch[1]}"
100+
folders=$(printf '"%s",' "${watch[@]}")
101+
printf ' {"shard":%d,"splits":%d,"name":"%s","watch_folders":[%s]}' \
102+
$((i+1)) "$TOTAL" "$name" "${folders%,}"
96103
(( i < TOTAL-1 )) && echo "," || echo ""
97104
done
98105
echo "]"

‎samples/eventhubs/python/README.md‎

Lines changed: 200 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,200 @@
1+
# Real-time payment fraud detection with Azure Event Hubs
2+
3+
A streaming platform in miniature, running entirely on LocalStack for Azure. Payments
4+
arrive from three different kinds of producer, a serverless processor scores them for
5+
fraud in real time, alerts flow onto a second stream, and every event is archived to a
6+
data lake without a line of code being written for it.
7+
8+
The sample is built around the properties that make Event Hubs different from a queue:
9+
**one log, many protocols, many independent readers, and a replayable history**.
10+
11+
## Architecture
12+
13+
```
14+
Producers (outside Azure) LocalStack for Azure
15+
┌───────────────────────┐ ┌──────────────────────────────────────────────┐
16+
│ POS terminals │ AMQP │ Event Hubs namespace (Standard, Kafka on) │
17+
│ producer_amqp.py ├───────►│ │
18+
│ │ │ payments hub ─ 4 partitions ─── Capture ───┼──► Blob Storage
19+
│ Legacy gateway │ Kafka │ consumer groups: │ Avro archives
20+
│ producer_kafka.py ├───────►│ fraud-detector ──┐ analytics audit │ (cold path)
21+
│ │ │ │ │
22+
│ ATM / IoT devices │ HTTPS │ ▼ │
23+
│ producer_http.py ├───────►│ Function App (Python) │
24+
└───────────────────────┘ │ Event Hubs trigger │
25+
│ fraud rules ──────┐ │
26+
│ ▼ │
27+
│ fraud-alerts hub ─ 2 partitions │
28+
│ ▲ │
29+
│ Key Vault: connection strings │ │
30+
│ Schema Registry: Avro contract│ │
31+
│ App Insights + Log Analytics │ │
32+
│ │ │
33+
│ Web App ── operations dashboard │
34+
└──────────────────────────────────────────────┘
35+
```
36+
37+
**Deployment flow.** The deploy script creates Log Analytics and Application Insights, a
38+
storage account with the Capture container, the Event Hubs namespace with both hubs, three
39+
consumer groups, least-privilege authorization rules and a Schema Registry group; stores
40+
the connection strings in Key Vault; then deploys the Function App and the dashboard Web
41+
App.
42+
43+
**At runtime.** Producers publish payments over AMQP, Kafka and HTTPS to the same hub. The
44+
Function App's Event Hubs trigger reads the `fraud-detector` consumer group in batches,
45+
applies the fraud rules, and writes alerts to the `fraud-alerts` hub through an output
46+
binding. Event Hubs Capture independently archives everything to Blob Storage as Avro. The
47+
dashboard reads the runtime state of both hubs, the checkpoints, the archives and the
48+
schema registry.
49+
50+
## What this demonstrates
51+
52+
| Capability | Where to see it |
53+
|------------|-----------------|
54+
| **Multi-protocol ingestion** - AMQP 1.0, Kafka and HTTPS into one hub | `src/producers/`, step 2 of `run-pipeline.sh` |
55+
| **Kafka compatibility** - an existing librdkafka app with no Azure SDK | `src/producers/producer_kafka.py` |
56+
| **Partition keys and ordering** - all of an account's payments on one partition | `producer_amqp.py`, the velocity rule in `function_app.py` |
57+
| **Consumer groups** - three independent readers of the same stream | `deploy.sh` step 6 |
58+
| **Serverless stream processing** - Event Hubs trigger, batched | `src/functions/function_app.py` |
59+
| **Checkpointing and recovery** - restart resumes where it left off | Step 4 of `run-pipeline.sh` |
60+
| **Event Hubs Capture** - Avro archives to Blob Storage, no consumer code | Step 5 of `run-pipeline.sh`, `scripts/read_capture.py` |
61+
| **Schema Registry** - a versioned Avro contract | `src/producers/schema_register.py` |
62+
| **Least-privilege SAS** - send-only and listen-only rules | `deploy.sh` step 7, `validate.sh` check 5 |
63+
| **Auto-inflate** - throughput units scale with ingress | `deploy.sh` step 4 |
64+
| **Geo-disaster recovery** - namespace pairing behind an alias | `validate.sh` check 11 |
65+
| **Application groups** - client throttling policies | `validate.sh` check 10 |
66+
| **Service integration** - Event Hubs with Storage, Key Vault, Functions, App Service, Monitor | the whole sample |
67+
68+
## Prerequisites
69+
70+
- [LocalStack](https://docs.localstack.cloud/getting-started/installation/) with a valid
71+
`LOCALSTACK_AUTH_TOKEN`
72+
- [Docker](https://docs.docker.com/get-docker/)
73+
- [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) and
74+
[azlocal](https://pypi.org/project/azlocal/) (`pip install azlocal`)
75+
- Python 3.12+ with `src/producers/requirements.txt` installed
76+
- `jq` and `zip`
77+
- [Terraform](https://developer.hashicorp.com/terraform/downloads) or the Bicep CLI, for
78+
the alternative deployments
79+
80+
## Quick start
81+
82+
```bash
83+
# Start the LocalStack Azure emulator
84+
export LOCALSTACK_AUTH_TOKEN=<your_auth_token>
85+
IMAGE_NAME=localstack/localstack-azure localstack start -d
86+
localstack wait -t 120
87+
88+
# Route the Azure CLI to the emulator
89+
azlocal start-interception
90+
91+
cd samples/eventhubs/python
92+
pip install -r src/producers/requirements.txt
93+
94+
bash scripts/deploy.sh # create the pipeline
95+
bash scripts/validate.sh # verify every capability
96+
bash scripts/run-pipeline.sh # run the end-to-end demo
97+
```
98+
99+
Then open the dashboard URL printed at the end of the deployment.
100+
101+
> **Note**
102+
> The first deployment downloads the Azure Functions build image and core tools inside the
103+
> emulator, which can take several minutes. Later runs reuse them.
104+
105+
## Alternative deployments
106+
107+
```bash
108+
bash bicep/deploy.sh # Bicep
109+
bash terraform/deploy.sh # Terraform
110+
```
111+
112+
Both provision the same topology and then publish the same application packages.
113+
114+
## The demo, step by step
115+
116+
`scripts/run-pipeline.sh` walks through the whole story:
117+
118+
1. **Publish the contract.** The Avro payment schema is registered in the Schema Registry
119+
and read back by id, the way a serializer resolves one at runtime.
120+
2. **Ingest over three protocols.** 40 payments over AMQP (keyed by account, plus a burst
121+
that trips the velocity rule), 15 over Kafka, 8 over HTTPS. The partition counts show
122+
they all landed in the same log.
123+
3. **Detect fraud.** The Function App picks up each batch, applies the amount and velocity
124+
rules, and emits alerts to `fraud-alerts`.
125+
4. **Prove checkpointing.** The function app is restarted while new payments arrive. It
126+
resumes from its checkpoint: nothing is lost and nothing is processed twice.
127+
5. **Read the archive.** Once a Capture window flushes, the newest Avro blob is decoded
128+
and its records printed - the cold path, with no consumer code.
129+
6. **Point at the dashboard** for the live view.
130+
131+
Each step asserts its own result rather than reporting it: the script exits non-zero (and
132+
prints the Function App's container logs) if the hub does not grow by the events that were
133+
published, if no alert arrives, if the processor replays instead of resuming, or if Capture
134+
writes nothing decodable. A run that completes is evidence the pipeline works, which is
135+
what makes it usable as a CI test.
136+
137+
## Application
138+
139+
### Producers (`src/producers/`)
140+
141+
| File | Role |
142+
|------|------|
143+
| `common.py` | Shared config, the Avro contract, the payment generator |
144+
| `producer_amqp.py` | POS terminals - official SDK, batches, partition keys |
145+
| `producer_kafka.py` | Legacy gateway - confluent-kafka, bootstrap derived from the connection string |
146+
| `producer_http.py` | ATM/IoT - HTTP runtime API with a hand-built SAS token |
147+
| `schema_register.py` | Registers and resolves the Avro contract |
148+
149+
### Processor (`src/functions/`)
150+
151+
`function_app.py` is an Event Hubs-triggered function with an Event Hubs output binding.
152+
Two rules: an amount threshold, and a per-account velocity window that is only correct
153+
because producers key events by account, which pins each account to one partition.
154+
155+
### Dashboard (`src/dashboard/`)
156+
157+
A Flask app that polls the Event Hubs runtime API, the checkpoint blobs, the alerts hub,
158+
the Capture container and the Schema Registry, and refreshes every five seconds.
159+
160+
| Route | Description |
161+
|-------|-------------|
162+
| `/` | The dashboard |
163+
| `/api/overview` | JSON snapshot of the whole pipeline |
164+
| `/health` | Health check |
165+
166+
## Scripts
167+
168+
| Script | Description |
169+
|--------|-------------|
170+
| `scripts/deploy.sh` | Creates the full topology and deploys both workloads |
171+
| `scripts/validate.sh` | Numbered checks over the control plane, data plane and bonus capabilities |
172+
| `scripts/run-pipeline.sh` | The end-to-end demo |
173+
| `scripts/cleanup.sh` | Deletes the resource group and local artifacts |
174+
| `bicep/deploy.sh`, `terraform/deploy.sh` | The alternative deployment paths |
175+
176+
## Cleanup
177+
178+
```bash
179+
bash scripts/cleanup.sh
180+
```
181+
182+
## Emulator notes
183+
184+
- Kafka is served in plaintext by the emulator; Azure requires `SASL_SSL`. The producer
185+
shows the SASL settings needed against Azure in a comment.
186+
- Data-plane authorization is existence-checked by the emulator, so the send-only and
187+
listen-only rules demonstrate the *deployment* pattern rather than an enforced boundary.
188+
Both rules are used exactly as they would be against Azure.
189+
- The emulator serves the HTTP runtime API on the gateway port; `common.py` derives it
190+
from the connection string so the same code works against Azure unchanged.
191+
192+
## References
193+
194+
- [Azure Event Hubs documentation](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-about)
195+
- [Event Hubs Capture](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-capture-overview)
196+
- [Event Hubs for Apache Kafka](https://learn.microsoft.com/en-us/azure/event-hubs/azure-event-hubs-apache-kafka-overview)
197+
- [Azure Schema Registry](https://learn.microsoft.com/en-us/azure/event-hubs/schema-registry-overview)
198+
- [Azure Functions Event Hubs trigger](https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-event-hubs-trigger)
199+
- [Event Hubs quotas and limits](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-quotas)
200+
- [LocalStack for Azure](https://docs.localstack.cloud/azure/)
Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# Bicep deployment
2+
3+
Provisions the payment fraud detection pipeline with a Bicep template, then publishes the
4+
application code with the Azure CLI.
5+
6+
## Prerequisites
7+
8+
- The LocalStack Azure emulator is running and `azlocal start-interception` has been run.
9+
- The Bicep CLI (`az bicep install`) and `jq` are available.
10+
11+
## Usage
12+
13+
```bash
14+
cd samples/eventhubs/python/bicep
15+
bash deploy.sh
16+
```
17+
18+
`deploy.sh` creates the resource group, validates and deploys `main.bicep`, reads the
19+
outputs, wires the connection strings, zip-deploys the Function App and the Web App, and
20+
verifies the payments hub.
21+
22+
### Why the connection strings are set by the script, not the template
23+
24+
Anything a template emits as an output is retained in the deployment history, so this sample
25+
keeps keys out of the template entirely: `deploy.sh` reads them with
26+
`az ... authorization-rule keys list` once the resources exist and writes them straight into
27+
Key Vault and the app settings. The storage connection string is assembled from the
28+
account's own endpoints, because the Azure Functions host cannot parse the `EndpointSuffix`
29+
form when the suffix carries a port.
30+
31+
## Template layout
32+
33+
| File | Contents |
34+
|------|----------|
35+
| `main.bicep` | Parameters, names, and the module graph |
36+
| `main.bicepparam` | Parameter values for the sample |
37+
| `modules/storage.bicep` | Storage account and the Capture container |
38+
| `modules/event-hubs.bicep` | Namespace, both hubs, Capture, consumer groups, authorization rules, schema group |
39+
| `modules/key-vault.bicep` | Key vault (the secrets themselves are written by `deploy.sh`) |
40+
| `modules/monitoring.bicep` | Log Analytics workspace and Application Insights |
41+
| `modules/workloads.bicep` | App Service plans, the Function App and the Web App |
42+
43+
## Parameters
44+
45+
| Parameter | Default | Notes |
46+
|-----------|---------|-------|
47+
| `prefix` / `suffix` | `local` / `payments` | Compose every resource name |
48+
| `eventHubSkuName` | `Standard` | Capture and Kafka need Standard or higher |
49+
| `partitionCount` | `4` | Partitions are the unit of parallelism and of ordering |
50+
| `retentionTimeInHours` | `24` | Standard tier allows up to 7 days |
51+
| `captureIntervalInSeconds` | `60` | Minimum allowed by Azure |
52+
| `fraudAmountThreshold` | `5000` | A single payment at or above this is flagged |
53+
| `fraudVelocityCount` | `5` | Payments per account per window before flagging |
54+
55+
## Outputs
56+
57+
`main.bicep` returns the resource names and the dashboard URL:
58+
59+
```bash
60+
az deployment group show \
61+
--name eventhubs-fraud-detection \
62+
--resource-group local-eventhubs-rg \
63+
--query properties.outputs
64+
```

0 commit comments

Comments
 (0)