A real-time data streaming pipeline built on Microsoft Fabric that simulates live stock price ticks, ingests them through Azure Event Hub and Fabric Eventstream, applies a Bronze → Silver → Gold medallion architecture inside a Fabric Eventhouse (KQL Database), and triggers real-time threshold alerts via Fabric Activator.
This project is part of a broader Microsoft Fabric portfolio with a fintech focus. It specifically targets the real-time/streaming surface of Fabric — Eventstream, Eventhouse/KQL, Reflex/Activator, and streaming-native data quality patterns — as a complement to the batch-oriented medallion architecture built in the COVID-19 Analytics project.
- What is a "tick"?
- Architecture Overview
- Azure Resources
- Security & Secrets Management
- The Producer
- Eventstream Configuration
- Eventhouse: Medallion Architecture
- Data Quality Validation
- Always-Current State: CurrentTickerPriceMV
- Real-Time Alerting: Activator
- Dev / Prod Deployment
- Known Limitations & Design Decisions
- Screenshots
A tick is a single price update event for a stock — one row of data representing "at this moment, here's the price." In real markets, a tick fires every time a trade occurs or the price changes, which can happen many times per second for an active stock.
In this project, a tick is a simulated JSON message generated by a Fabric Notebook and sent to Azure Event Hub:
{
"ticker": "AAPL",
"price": 205.64,
"volume": 2306,
"bid": 205.62,
"ask": 205.66,
"event_time": "2026-07-10T18:05:21.288575+00:00"
}"Tick data" or "tick stream" is the industry term for this kind of granular, event-by-event price feed — as opposed to a daily closing price, which is a summary rather than a tick. The Gold layer of this pipeline (GoldOHLC1Min) takes many individual ticks and aggregates them into 1-minute OHLC (Open/High/Low/Close) bars, which is the standard way tick data is summarized for charting.
Fabric Notebook (producer)
│ simulated ticks (JSON)
▼
Azure Event Hub (stock-ticks)
│
▼
Fabric Eventstream (es_stocktick)
│ field mapping
├──────────────────────────┐
▼ ▼
Eventhouse (eh_stocktick) Activator (activator_stocktick)
│ │ threshold rule
RawTicks (Bronze) └─ Email alert (see limitations)
│ update policy
▼
SilverTicks (Silver)
│ update policy ┌─ Materialized View
├──────────────────────────────┤ CurrentTickerPriceMV
▼ │ (always-current price
GoldOHLC1Min (Gold) │ per ticker)
1-min OHLC bars └─
| Resource | Name | Notes |
|---|---|---|
| Resource Group | rg-fabric-portfolio |
Shared across Fabric portfolio projects |
| Event Hub Namespace | evhns-stocktick-dev |
Standard tier (required for named consumer groups), Canada Central, 1 throughput unit, auto-inflate off |
| Event Hub | stock-ticks |
4 partitions, 1-day retention |
| Consumer Group | eventstream-cg |
Dedicated group for Eventstream, isolated from $Default |
| Shared Access Policy (producer) | producer-send-only |
Send-only scoped policy on the stock-ticks hub — least-privilege, separate from the namespace root key |
| Key Vault | kv-stocktick-dev |
RBAC permission model, Canada Central |
| Key Vault Secret | stocktick-eventhub-send-conn-string |
Stores the producer's Send-only connection string |
Cost note: Azure Event Hub has no pause state — it bills per throughput-unit-hour whenever the namespace exists. At 1 TU on Standard tier this runs roughly $10–12/month if left running continuously, or fractions of a cent per short working session. A budget alert was considered as a safety net rather than deleting/recreating the namespace between sessions, since reconnecting Eventstream each time carries more friction than the marginal cost of leaving it running.
Fabric Eventstream needs a real streaming ingestion endpoint to demonstrate genuine event-driven architecture — Event Hub was chosen as that entry point because it's the standard, natively-supported Azure service for high-throughput event ingestion, and it's what a production fintech tick feed would realistically sit behind (a market data vendor or exchange gateway publishing to a message bus, with downstream consumers subscribing independently). Using Event Hub rather than a static file drop or a direct Eventstream-to-notebook connection was a deliberate choice to model a real streaming source with partitioning, consumer groups, and retention — concepts that don't exist in a batch data source and are core to the "streaming" skill this project set out to demonstrate.
-
Create the namespace (
evhns-stocktick-dev):- Pricing tier: Standard — required because Basic tier does not support named consumer groups (it's limited to
$Default), and a dedicated consumer group was a deliberate isolation choice (see below). - Throughput units: 1, auto-inflate off — sufficient for a low-volume 5-ticker simulated feed; auto-inflate was left off to avoid unexpected cost scaling.
- Region: Canada Central, matching the rest of the portfolio's Azure footprint.
- Pricing tier: Standard — required because Basic tier does not support named consumer groups (it's limited to
-
Create the Event Hub instance (
stock-ticks) inside the namespace:- Partition count: 4 — not one-per-ticker; Event Hub partitioning is about parallelism and ordering guarantees, not per-entity isolation. 4 partitions gives a real partitioning story (partition key = ticker) without overprovisioning for a 5-ticker, low-volume feed.
- Message retention: 1 day — enough for a portfolio project's working sessions; no need for longer retention since data is continuously re-generated by the simulator.
-
Create a dedicated consumer group (
eventstream-cg):- Created separately from
$Defaultso that Eventstream's read position is isolated from any other future consumer of the same hub. This mirrors a real production pattern —$Defaultis what anything unspecified falls back to, so relying on it for a named, intentional consumer (Eventstream) would be sloppy and could cause unexpected offset conflicts if a second consumer were added later.
- Created separately from
-
Create a scoped Shared Access Policy for the producer (
producer-send-only):- Created at the Event Hub level (not namespace level), with Send permission only — no Manage, no Listen.
- This was a deliberate least-privilege decision: the initial setup used the namespace's
RootManageSharedAccessKey(which has full Manage/Send/Listen rights across the entire namespace) to get the pipeline working end-to-end quickly, then a scoped, Send-only policy was created specifically for the notebook producer once the mechanism was proven. The producer only ever needs to send events — it never needs to manage the namespace or read/listen to any hub — so granting it broader rights would violate least-privilege for no functional benefit. - Eventstream's own connection (the consumer side) continues to use a separate, broader-scoped connection since it needs Listen rights — a different credential for a different responsibility, rather than one shared key used everywhere.
Screenshots — Azure Event Hub setup:
The producer notebook never hardcodes the Event Hub connection string. Instead:
- The connection string (scoped to the
producer-send-onlySend-only policy) is stored as a secret in Azure Key Vault (kv-stocktick-dev). - The notebook retrieves it at runtime via:
conn_str = notebookutils.credentials.getSecret(
"https://kv-stocktick-dev.vault.azure.net/",
"stocktick-eventhub-send-conn-string"
)- Key Vault uses the RBAC permission model. The Fabric-portal-signed-in identity (not necessarily the same identity used to manage Azure resources) needs the Key Vault Secrets User role (or Secrets Officer, if also managing secrets) assigned directly — Key Vault does not automatically trust "whoever is logged into Azure," it authorizes by the specific identity making the call, which for
notebookutils.credentials.getSecret()is the Fabric session's own identity.
This means the notebook is safe to commit to Git — no plaintext credentials ever appear in source control.
Screenshots — Key Vault setup:
A Fabric Notebook (nb_stock_tick_producer) simulates price ticks for 5 tickers — AAPL, MSFT, GOOGL, AMZN, NVDA — and sends them to Event Hub using the azure-eventhub Python SDK.
Design note — bounded runs, not 24/7 streaming: Fabric notebook sessions are not built for indefinite long-running processes (session idle timeouts apply). The producer is designed to run in bounded bursts (e.g., a fixed number of minutes or a fixed batch), which is the honest framing for a portfolio project rather than claiming a true always-on producer.
Core tick generation logic:
def next_tick(ticker):
drift = random.uniform(-0.5, 0.5)
prices[ticker] = max(1, round(prices[ticker] + drift, 2))
return {
"ticker": ticker,
"price": prices[ticker],
"volume": random.randint(100, 5000),
"bid": round(prices[ticker] - 0.02, 2),
"ask": round(prices[ticker] + 0.02, 2),
"event_time": datetime.now(timezone.utc).isoformat()
}Package installation: azure-eventhub is installed via the Fabric Environment item (env_stocktick), added through the External repositories → PyPI tab (not the "Custom" tab, which requires uploading .whl files directly — a dead end for a simple named-package install).
Screenshots — Producer notebook:
es_stocktick connects to the stock-ticks Event Hub using the eventstream-cg consumer group and fans out to two parallel destinations from the same mapped stream:
dest_rawticks→ Eventhouse, lands data intoRawTicksdest_activator→ Activator, evaluates the price-threshold rule in real time
Both destinations branch off the Mapper node (not off each other) — Eventhouse's dest_rawticks is a terminal sink once data is written to a table, so a second destination must branch from the still-in-flight mapped stream rather than "after" the first destination.
Field mapping, including ingestion_time, which is deliberately mapped to Eventstream's own EventEnqueuedUtcTime metadata field — this models the distinction between event time (when the producer generated the tick) and processing/ingestion time (when it actually arrived in the pipeline), a standard streaming-systems concept.
Connection setup gotcha: When creating the Event Hub connection in Eventstream, only the key value goes in the Shared Access Key field — pasting the full connection string (including Endpoint=...;SharedAccessKeyName=...) into that field causes a "Duplicate key(s) found in connection string" error, since Eventstream reconstructs the full string internally from the separate Key Name and Key fields.
Screenshots — Eventstream:
Raw landing table. Nothing is filtered here — even malformed or invalid data lands as-is. This is deliberate: Bronze should always reflect exactly what arrived, so there's always a true, unfiltered source of record to audit or debug against.
| Column | Type | Meaning |
|---|---|---|
ticker |
string | Stock ticker symbol |
price |
real | Simulated current price |
volume |
long | Shares "traded" in this tick |
bid |
real | Buyer's offered price |
ask |
real | Seller's asking price |
event_time |
string | When the notebook generated the tick (raw string, as sent) |
ingestion_time |
datetime | When Eventstream landed the row here |
Screenshots — RawTicks:
Cleaned and validated version of Bronze, populated automatically via a KQL update policy — every new row in RawTicks triggers SilverTicksTransform(), which writes its result into SilverTicks. Update policies only apply to ingestion that happens after the policy is attached; they do not retroactively backfill.
Two categories of handling:
- Hard reject (row dropped entirely) — data is unusable:
event_timefails to parse into a valid datetimetickeris null or emptyprice <= 0volume < 0bidoraskis null
- Flag (row kept, labeled) — data is suspicious but not impossible:
event_timemore than 5 seconds in the future →future_timestampbid > priceorprice > ask(crossed spread) →bid_ask_crosstickernot in the known symbol list (AAPL, MSFT, GOOGL, AMZN, NVDA) →unknown_ticker- otherwise →
clean
Duplicate (ticker, event_time) pairs are deduplicated, keeping the row with the latest ingestion_time.
.create-or-alter function SilverTicksTransform() {
RawTicks
| extend parsed_time = todatetime(event_time)
| where isnotnull(parsed_time)
| where isnotempty(ticker)
| where price > 0 and volume >= 0
| where isnotnull(bid) and isnotnull(ask)
| extend flag = case(
parsed_time > now() + 5s, "future_timestamp",
bid > price or price > ask, "bid_ask_cross",
ticker !in ("AAPL", "MSFT", "GOOGL", "AMZN", "NVDA"), "unknown_ticker",
"clean"
)
| summarize arg_max(ingestion_time, *) by ticker, parsed_time
| project ticker, price, volume, bid, ask,
event_time = parsed_time, ingestion_time,
data_quality_flag = flag
}
.alter table SilverTicks policy update
@'[{"IsEnabled": true, "Source": "RawTicks", "Query": "SilverTicksTransform()", "IsTransactional": true, "PropagateIngestionProperties": false}]'The cleaning logic covers every column with independent validation — ticker, price, volume, bid, ask, and event_time each have their own check, rather than relying on comparisons between columns to catch everything indirectly (an earlier version of this function only validated price/volume/timestamp and bid/ask relative to price, leaving ticker and independent bid/ask nulls completely unvalidated — see Data Quality Validation for how this gap was found and fixed).
Screenshots — SilverTicks setup:
Aggregates clean Silver ticks into 1-minute OHLC (Open/High/Low/Close) bars per ticker — the standard structure for charting and analysis. Only rows with data_quality_flag == "clean" are included; flagged rows are excluded from aggregation since questionable data shouldn't feed into summary analytics.
| Column | Meaning |
|---|---|
bucket_start |
Start of the 1-minute window this row summarizes |
open_price |
First tick's price in that minute |
high_price |
Highest price seen in that minute |
low_price |
Lowest price seen in that minute |
close_price |
Last tick's price in that minute |
total_volume |
Sum of volume in that minute |
.create-or-alter function GoldOHLCTransform() {
SilverTicks
| where data_quality_flag == "clean"
| summarize
(open_time, open_price) = arg_min(event_time, price),
high_price = max(price),
low_price = min(price),
(close_time, close_price) = arg_max(event_time, price),
total_volume = sum(volume)
by ticker, bucket_start = bin(event_time, 1m)
| project ticker, bucket_start, open_price, high_price, low_price, close_price, total_volume
}
.alter table GoldOHLC1Min policy update
@'[{"IsEnabled": true, "Source": "SilverTicks", "Query": "GoldOHLCTransform()", "IsTransactional": true, "PropagateIngestionProperties": false}]'KQL syntax trap encountered: arg_min(event_time, price) without explicit tuple-naming binds the alias to the first returned value (the datetime), not the second (the price) — an early version of this function produced open_price/close_price columns that were actually timestamps. The fix is explicit tuple-naming: (open_time, open_price) = arg_min(event_time, price).
Screenshots — GoldOHLC1Min setup:
Rather than assume the cleaning logic worked, it was validated with two rounds of deliberately injected bad data sent through the real pipeline (Notebook → Event Hub → Eventstream → RawTicks → SilverTicks), checking the actual SilverTicks output against expectations each time.
Round 1 — core value validation:
| Test case | Expected | Actual |
|---|---|---|
| Clean tick (AAPL) | clean |
✅ clean |
| Negative price (MSFT) | Hard rejected | ✅ Absent from Silver, present in Raw |
| Negative volume (GOOGL) | Hard rejected | ✅ Absent from Silver, present in Raw |
| Bid higher than price (AMZN) | Flagged, kept | ✅ bid_ask_cross |
| Timestamp 60s in future (NVDA) | Flagged, kept | ✅ future_timestamp |
| Malformed timestamp string (AAPL) | Hard rejected | ✅ Absent from Silver, present in Raw |
Round 2 — after auditing for column coverage gaps (ticker and independent bid/ask nulls had no validation in Round 1):
| Test case | Expected | Actual |
|---|---|---|
| Clean tick (AAPL) | clean |
✅ clean |
| Empty ticker (MSFT) | Hard rejected | ✅ Absent from Silver |
| Null bid (GOOGL) | Hard rejected | ✅ Absent from Silver |
| Null ask (AMZN) | Hard rejected | ✅ Absent from Silver |
| Unknown ticker "TSLA" (NVDA's slot) | Flagged, kept | ✅ unknown_ticker |
In both rounds, RawTicks was confirmed to contain all injected rows (including the rejected ones) — proving Bronze is genuinely unfiltered, and that Silver's filtering happens deliberately at the transform layer, not by accident upstream.
Screenshots — Data quality validation:
A common requirement in streaming systems is answering "what's the latest value right now?" without scanning the full event history. Two approaches were evaluated:
Option considered: Lakehouse Delta table + Change Data Feed + scheduled pipeline.
A CurrentTickerPrice Delta table was created in a Fabric Lakehouse (lh_stocktick) with delta.enableChangeDataFeed = true, and a MERGE INTO upsert was written and tested successfully (pulling the latest clean tick per ticker from SilverTicks via the Kusto Spark connector, then merging into the Delta table). This approach works, but requires an external scheduling mechanism (a Fabric Data Pipeline running the notebook on a timer) to stay current — it's a pull-based pattern, not self-maintaining.
Option chosen: Eventhouse native Materialized View.
.create materialized-view CurrentTickerPriceMV on table SilverTicks
{
SilverTicks
| where data_quality_flag == "clean"
| summarize arg_max(event_time, *) by ticker
}This table automatically stays current as new Silver rows arrive — no external scheduling, no notebook, no pipeline. This was the approach kept in the final architecture, since it achieves the same "always-current state" goal with meaningfully less infrastructure. The Delta/CDF path was fully built and validated but deprioritized as unnecessary complexity once the simpler, Eventhouse-native mechanism proved sufficient — a deliberate trade-off, not an oversight.
Screenshots — CDC evaluation:
An Activator item (activator_stocktick) monitors the same mapped stream in parallel with the Eventhouse destination.
Object: StockTicker, keyed on the ticker field — each of the 5 tickers is tracked as an independent monitored entity.
Rule: es_stocktick-stream alert — fires when price becomes greater than 300 for any tracked ticker.
Validated behavior: a test batch with AAPL at $350 and MSFT at $150 was sent; the rule correctly fired exactly once, for AAPL, and did not fire for MSFT. Confirmed via Activator's History tab, showing 1 activation for the AAPL object ID and 0 for MSFT.
Action / delivery limitation: the configured action is Email, but the Fabric-signed-in identity used in this environment (rutujakadam@rutujaakadam1610outlook.onmicrosoft.com) is an Entra ID guest/Azure-created identity with no licensed Exchange Online mailbox — there is nowhere for the email to be delivered, regardless of correct configuration. This is a genuine tenant-licensing constraint, not a pipeline defect. The rule's condition-evaluation logic was verified directly via Activator's run history instead of via delivered email. In a production deployment, a licensed mailbox (or a Power Automate flow / Teams action tied to a licensed identity) would resolve this.
Screenshots — Activator:
Following the same pattern used across this Fabric portfolio:
ws_stocktick_dev— Git-connected to thedevelopmentbranch, live push/pull sync as work is committedws_stocktick_prod— not Git-connected; updated only via the Fabric Deployment Pipeline (pipeline_stocktick)mainbranch — updated via Pull Request merge fromdevelopment, serving as the reviewed audit trail
Promotion sequence used:
- Commit all Dev workspace changes to
development - Open and merge a Pull Request:
development→main - Promote all 6 items (Notebook, Eventstream, Eventhouse, KQL Database, Activator, Environment) via the Deployment Pipeline's Development → Production stage
Post-deployment verification (not assumed — directly tested): a fresh batch of ticks was sent and traced through the entire Production pipeline independently of Dev:
RawTicksin Prod — 5 rows landed correctlySilverTicksin Prod — all 5 correctly flaggedcleanCurrentTickerPriceMVin Prod — correctly auto-updated- Activator in Prod — correctly fired 3 activations (AMZN, GOOGL, MSFT, all >$300) and correctly ignored AAPL and NVDA (both <$300)
This confirms Production is not just a structural copy of Dev — every layer was functionally re-verified with real data after promotion, including confirming that connections (Event Hub auth, Key Vault access) carried over correctly rather than assuming they would.
Screenshots — Deployment:
-
Eventstream is forward-only. It only processes events from the point it starts actively listening — it does not retroactively backfill from Event Hub's retention window. An early ~10-minute producer test run was lost for this reason, since Eventstream wasn't yet publishing when those events were sent. Not a bug — a characteristic of the service to design around.
-
Update policies are forward-only. The very first test batch sent to
RawTicks, before theSilverTicksupdate policy was attached, never flowed into Silver — a timing artifact of build order, not a defect. -
Producer runs in bounded bursts, not as a true 24/7 process, due to Fabric notebook session lifetime constraints. Documented as an intentional constraint rather than a limitation being hidden.
-
Activator email delivery is untestable in this environment due to the signed-in identity lacking a licensed Exchange mailbox. Rule logic was verified via Activator's run history instead of delivered notifications.
-
CDC was implemented via Eventhouse Materialized View, not Lakehouse Delta CDF, after evaluating both. See Always-Current State for the full rationale.
-
Event Hub has no pause state — cost is proportional to how long the namespace exists, not to actual usage volume. A budget alert is a more practical safeguard than a delete/recreate cycle between sessions, given the reconnection overhead the latter would introduce.