|
| 1 | +# Real-time payment fraud detection with Azure Event Hubs |
| 2 | + |
| 3 | +A streaming platform in miniature, running entirely on LocalStack for Azure. Payments |
| 4 | +arrive from three different kinds of producer, a serverless processor scores them for |
| 5 | +fraud in real time, alerts flow onto a second stream, and every event is archived to a |
| 6 | +data lake without a line of code being written for it. |
| 7 | + |
| 8 | +The sample is built around the properties that make Event Hubs different from a queue: |
| 9 | +**one log, many protocols, many independent readers, and a replayable history**. |
| 10 | + |
| 11 | +## Architecture |
| 12 | + |
| 13 | +``` |
| 14 | + Producers (outside Azure) LocalStack for Azure |
| 15 | + ┌───────────────────────┐ ┌──────────────────────────────────────────────┐ |
| 16 | + │ POS terminals │ AMQP │ Event Hubs namespace (Standard, Kafka on) │ |
| 17 | + │ producer_amqp.py ├───────►│ │ |
| 18 | + │ │ │ payments hub ─ 4 partitions ─── Capture ───┼──► Blob Storage |
| 19 | + │ Legacy gateway │ Kafka │ consumer groups: │ Avro archives |
| 20 | + │ producer_kafka.py ├───────►│ fraud-detector ──┐ analytics audit │ (cold path) |
| 21 | + │ │ │ │ │ |
| 22 | + │ ATM / IoT devices │ HTTPS │ ▼ │ |
| 23 | + │ producer_http.py ├───────►│ Function App (Python) │ |
| 24 | + └───────────────────────┘ │ Event Hubs trigger │ |
| 25 | + │ fraud rules ──────┐ │ |
| 26 | + │ ▼ │ |
| 27 | + │ fraud-alerts hub ─ 2 partitions │ |
| 28 | + │ ▲ │ |
| 29 | + │ Key Vault: connection strings │ │ |
| 30 | + │ Schema Registry: Avro contract│ │ |
| 31 | + │ App Insights + Log Analytics │ │ |
| 32 | + │ │ │ |
| 33 | + │ Web App ── operations dashboard │ |
| 34 | + └──────────────────────────────────────────────┘ |
| 35 | +``` |
| 36 | + |
| 37 | +**Deployment flow.** The deploy script creates Log Analytics and Application Insights, a |
| 38 | +storage account with the Capture container, the Event Hubs namespace with both hubs, three |
| 39 | +consumer groups, least-privilege authorization rules and a Schema Registry group; stores |
| 40 | +the connection strings in Key Vault; then deploys the Function App and the dashboard Web |
| 41 | +App. |
| 42 | + |
| 43 | +**At runtime.** Producers publish payments over AMQP, Kafka and HTTPS to the same hub. The |
| 44 | +Function App's Event Hubs trigger reads the `fraud-detector` consumer group in batches, |
| 45 | +applies the fraud rules, and writes alerts to the `fraud-alerts` hub through an output |
| 46 | +binding. Event Hubs Capture independently archives everything to Blob Storage as Avro. The |
| 47 | +dashboard reads the runtime state of both hubs, the checkpoints, the archives and the |
| 48 | +schema registry. |
| 49 | + |
| 50 | +## What this demonstrates |
| 51 | + |
| 52 | +| Capability | Where to see it | |
| 53 | +|------------|-----------------| |
| 54 | +| **Multi-protocol ingestion** - AMQP 1.0, Kafka and HTTPS into one hub | `src/producers/`, step 2 of `run-pipeline.sh` | |
| 55 | +| **Kafka compatibility** - an existing librdkafka app with no Azure SDK | `src/producers/producer_kafka.py` | |
| 56 | +| **Partition keys and ordering** - all of an account's payments on one partition | `producer_amqp.py`, the velocity rule in `function_app.py` | |
| 57 | +| **Consumer groups** - three independent readers of the same stream | `deploy.sh` step 6 | |
| 58 | +| **Serverless stream processing** - Event Hubs trigger, batched | `src/functions/function_app.py` | |
| 59 | +| **Checkpointing and recovery** - restart resumes where it left off | Step 4 of `run-pipeline.sh` | |
| 60 | +| **Event Hubs Capture** - Avro archives to Blob Storage, no consumer code | Step 5 of `run-pipeline.sh`, `scripts/read_capture.py` | |
| 61 | +| **Schema Registry** - a versioned Avro contract | `src/producers/schema_register.py` | |
| 62 | +| **Least-privilege SAS** - send-only and listen-only rules | `deploy.sh` step 7, `validate.sh` check 5 | |
| 63 | +| **Auto-inflate** - throughput units scale with ingress | `deploy.sh` step 4 | |
| 64 | +| **Geo-disaster recovery** - namespace pairing behind an alias | `validate.sh` check 11 | |
| 65 | +| **Application groups** - client throttling policies | `validate.sh` check 10 | |
| 66 | +| **Service integration** - Event Hubs with Storage, Key Vault, Functions, App Service, Monitor | the whole sample | |
| 67 | + |
| 68 | +## Prerequisites |
| 69 | + |
| 70 | +- [LocalStack](https://docs.localstack.cloud/getting-started/installation/) with a valid |
| 71 | + `LOCALSTACK_AUTH_TOKEN` |
| 72 | +- [Docker](https://docs.docker.com/get-docker/) |
| 73 | +- [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) and |
| 74 | + [azlocal](https://pypi.org/project/azlocal/) (`pip install azlocal`) |
| 75 | +- Python 3.12+ with `src/producers/requirements.txt` installed |
| 76 | +- `jq` and `zip` |
| 77 | +- [Terraform](https://developer.hashicorp.com/terraform/downloads) or the Bicep CLI, for |
| 78 | + the alternative deployments |
| 79 | + |
| 80 | +## Quick start |
| 81 | + |
| 82 | +```bash |
| 83 | +# Start the LocalStack Azure emulator |
| 84 | +export LOCALSTACK_AUTH_TOKEN=<your_auth_token> |
| 85 | +IMAGE_NAME=localstack/localstack-azure localstack start -d |
| 86 | +localstack wait -t 120 |
| 87 | + |
| 88 | +# Route the Azure CLI to the emulator |
| 89 | +azlocal start-interception |
| 90 | + |
| 91 | +cd samples/eventhubs/python |
| 92 | +pip install -r src/producers/requirements.txt |
| 93 | + |
| 94 | +bash scripts/deploy.sh # create the pipeline |
| 95 | +bash scripts/validate.sh # verify every capability |
| 96 | +bash scripts/run-pipeline.sh # run the end-to-end demo |
| 97 | +``` |
| 98 | + |
| 99 | +Then open the dashboard URL printed at the end of the deployment. |
| 100 | + |
| 101 | +> **Note** |
| 102 | +> The first deployment downloads the Azure Functions build image and core tools inside the |
| 103 | +> emulator, which can take several minutes. Later runs reuse them. |
| 104 | +
|
| 105 | +## Alternative deployments |
| 106 | + |
| 107 | +```bash |
| 108 | +bash bicep/deploy.sh # Bicep |
| 109 | +bash terraform/deploy.sh # Terraform |
| 110 | +``` |
| 111 | + |
| 112 | +Both provision the same topology and then publish the same application packages. |
| 113 | + |
| 114 | +## The demo, step by step |
| 115 | + |
| 116 | +`scripts/run-pipeline.sh` walks through the whole story: |
| 117 | + |
| 118 | +1. **Publish the contract.** The Avro payment schema is registered in the Schema Registry |
| 119 | + and read back by id, the way a serializer resolves one at runtime. |
| 120 | +2. **Ingest over three protocols.** 40 payments over AMQP (keyed by account, plus a burst |
| 121 | + that trips the velocity rule), 15 over Kafka, 8 over HTTPS. The partition counts show |
| 122 | + they all landed in the same log. |
| 123 | +3. **Detect fraud.** The Function App picks up each batch, applies the amount and velocity |
| 124 | + rules, and emits alerts to `fraud-alerts`. |
| 125 | +4. **Prove checkpointing.** The function app is restarted while new payments arrive. It |
| 126 | + resumes from its checkpoint: nothing is lost and nothing is processed twice. |
| 127 | +5. **Read the archive.** Once a Capture window flushes, the newest Avro blob is decoded |
| 128 | + and its records printed - the cold path, with no consumer code. |
| 129 | +6. **Point at the dashboard** for the live view. |
| 130 | + |
| 131 | +Each step asserts its own result rather than reporting it: the script exits non-zero (and |
| 132 | +prints the Function App's container logs) if the hub does not grow by the events that were |
| 133 | +published, if no alert arrives, if the processor replays instead of resuming, or if Capture |
| 134 | +writes nothing decodable. A run that completes is evidence the pipeline works, which is |
| 135 | +what makes it usable as a CI test. |
| 136 | + |
| 137 | +## Application |
| 138 | + |
| 139 | +### Producers (`src/producers/`) |
| 140 | + |
| 141 | +| File | Role | |
| 142 | +|------|------| |
| 143 | +| `common.py` | Shared config, the Avro contract, the payment generator | |
| 144 | +| `producer_amqp.py` | POS terminals - official SDK, batches, partition keys | |
| 145 | +| `producer_kafka.py` | Legacy gateway - confluent-kafka, bootstrap derived from the connection string | |
| 146 | +| `producer_http.py` | ATM/IoT - HTTP runtime API with a hand-built SAS token | |
| 147 | +| `schema_register.py` | Registers and resolves the Avro contract | |
| 148 | + |
| 149 | +### Processor (`src/functions/`) |
| 150 | + |
| 151 | +`function_app.py` is an Event Hubs-triggered function with an Event Hubs output binding. |
| 152 | +Two rules: an amount threshold, and a per-account velocity window that is only correct |
| 153 | +because producers key events by account, which pins each account to one partition. |
| 154 | + |
| 155 | +### Dashboard (`src/dashboard/`) |
| 156 | + |
| 157 | +A Flask app that polls the Event Hubs runtime API, the checkpoint blobs, the alerts hub, |
| 158 | +the Capture container and the Schema Registry, and refreshes every five seconds. |
| 159 | + |
| 160 | +| Route | Description | |
| 161 | +|-------|-------------| |
| 162 | +| `/` | The dashboard | |
| 163 | +| `/api/overview` | JSON snapshot of the whole pipeline | |
| 164 | +| `/health` | Health check | |
| 165 | + |
| 166 | +## Scripts |
| 167 | + |
| 168 | +| Script | Description | |
| 169 | +|--------|-------------| |
| 170 | +| `scripts/deploy.sh` | Creates the full topology and deploys both workloads | |
| 171 | +| `scripts/validate.sh` | Numbered checks over the control plane, data plane and bonus capabilities | |
| 172 | +| `scripts/run-pipeline.sh` | The end-to-end demo | |
| 173 | +| `scripts/cleanup.sh` | Deletes the resource group and local artifacts | |
| 174 | +| `bicep/deploy.sh`, `terraform/deploy.sh` | The alternative deployment paths | |
| 175 | + |
| 176 | +## Cleanup |
| 177 | + |
| 178 | +```bash |
| 179 | +bash scripts/cleanup.sh |
| 180 | +``` |
| 181 | + |
| 182 | +## Emulator notes |
| 183 | + |
| 184 | +- Kafka is served in plaintext by the emulator; Azure requires `SASL_SSL`. The producer |
| 185 | + shows the SASL settings needed against Azure in a comment. |
| 186 | +- Data-plane authorization is existence-checked by the emulator, so the send-only and |
| 187 | + listen-only rules demonstrate the *deployment* pattern rather than an enforced boundary. |
| 188 | + Both rules are used exactly as they would be against Azure. |
| 189 | +- The emulator serves the HTTP runtime API on the gateway port; `common.py` derives it |
| 190 | + from the connection string so the same code works against Azure unchanged. |
| 191 | + |
| 192 | +## References |
| 193 | + |
| 194 | +- [Azure Event Hubs documentation](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-about) |
| 195 | +- [Event Hubs Capture](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-capture-overview) |
| 196 | +- [Event Hubs for Apache Kafka](https://learn.microsoft.com/en-us/azure/event-hubs/azure-event-hubs-apache-kafka-overview) |
| 197 | +- [Azure Schema Registry](https://learn.microsoft.com/en-us/azure/event-hubs/schema-registry-overview) |
| 198 | +- [Azure Functions Event Hubs trigger](https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-event-hubs-trigger) |
| 199 | +- [Event Hubs quotas and limits](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-quotas) |
| 200 | +- [LocalStack for Azure](https://docs.localstack.cloud/azure/) |
0 commit comments