🏗️ Engineering Portfolio | Production Architecture Documentation
This repository documents a production monitoring and observability architecture for a Django/DRF application running on Azure Kubernetes Service (AKS).
The objective is not to share proprietary source code or production credentials. Instead, this documentation preserves the monitoring architecture, implementation approach, operational practices, troubleshooting patterns, engineering decisions, and lessons learned so the same approach can be reused in future projects.
I created this repository for three purposes:
- 📚 Document real production monitoring practices I have implemented.
- 🧠 Build a long-term engineering knowledge base for future projects.
- 🔄 Preserve troubleshooting and observability knowledge so I can reproduce and improve the setup without relearning everything from scratch.
The monitoring approach brings together Sentry, Azure Monitor / Log Analytics, Azure Managed Prometheus, Grafana, Kubernetes health checks, and correlation-based troubleshooting.
I implemented a production monitoring approach covering the complete application lifecycle:
Application
↓
Errors + Logs + Metrics
↓
Monitoring Platforms
↓
Dashboards + Alerts
↓
Incident Investigation
↓
Root Cause
↓
Fix + Verification
↓
Prevention
The monitoring system is designed to answer six simple questions:
| Monitoring Area | Main Question |
|---|---|
| 🐛 Sentry | What broke? |
| 📝 Log Analytics | What happened? |
| 📈 Prometheus | How is the system behaving? |
| 📊 Grafana | What does the system look like now? |
| ❤️ Kubernetes Health | Can the application safely serve traffic? |
| 🚨 Alerts | Do I need to act now? |
A production application cannot depend only on users reporting problems.
Without proper monitoring, issues such as:
- Application exceptions
- Failed background tasks
- High memory usage
- Pod restarts
- Queue backlogs
- 502 / 504 errors
- Slow requests
- External API failures
may only become visible after they affect users.
The goal was therefore to build a monitoring layer that provides:
Visibility
↓
Early Detection
↓
Faster Investigation
↓
Reliable Resolution
↓
Continuous Improvement
The monitoring architecture was designed to:
- 🐛 Detect application errors automatically.
- 📝 Centralize application and infrastructure logs.
- 🔗 Trace requests using correlation IDs.
- 📈 Monitor application and Kubernetes metrics.
- 🔄 Monitor Celery and Redis workloads.
- 🌐 Monitor ingress traffic and HTTP errors.
- 📊 Provide clear operational dashboards.
- 🚨 Detect important problems before they become major incidents.
- ❤️ Verify application health through Kubernetes probes.
- 🧠 Create a repeatable production troubleshooting process.
- 📚 Turn production incidents into reusable engineering knowledge.
- 🐍 Django / Django REST Framework
- 🌿 Celery / Celery Beat
- ❤️ Redis
- ☸️ Azure Kubernetes Service (AKS)
- 🐛 Sentry
- ☁️ Azure Monitor
- 🔎 Azure Log Analytics
- 📈 Azure Managed Prometheus
- 📊 Azure Managed Grafana
- 🌐 NGINX Ingress
- 🔐 Azure Key Vault
- 🗄️ Azure Database
- 📦 Azure Blob Storage
The monitoring architecture observes the application from multiple angles rather than depending on one monitoring tool.
Users
│
▼
NGINX Ingress
│
▼
Django Web Pods
/ \
/ \
▼ ▼
Database Redis / Celery
│
▼
Celery Workers
│
▼
External Services
Application + Kubernetes
│
├──────────► Sentry
│ Errors / Traces / Performance
│
├──────────► Log Analytics
│ Structured Logs / KQL
│
└──────────► Prometheus
Metrics
│
▼
Grafana
│
▼
Alerts
Sentry was used as the application-level visibility layer.
It helps capture problems that ordinary infrastructure monitoring cannot explain clearly.
The monitoring covers areas such as:
- Application exceptions
- Full stack traces
- Request context
- Breadcrumbs
- Slow transactions
- Celery failures
- Performance information
- Release/deployment correlation
A server metric may tell us:
CPU = 90%
Sentry can help answer:
Which request is failing?
Which part of the application failed?
What exception occurred?
What was the user doing?
This makes Sentry the first layer for understanding application-level failures.
Production logs are structured so they can be searched and analyzed centrally.
Example:
{
"timestamp": "2026-08-11T12:00:00Z",
"level": "ERROR",
"message": "Payment verification failed",
"correlation_id": "abc123",
"exception": "PaymentVerificationError"
}Structured logging makes it easier to:
- Search specific requests.
- Filter errors.
- Group similar failures.
- Investigate incidents.
- Connect application logs with infrastructure events.
One of the most useful monitoring patterns is the Correlation ID.
A request can travel through multiple components:
Client
↓
Ingress
↓
Django
↓
Redis
↓
Celery Worker
↓
External API
↓
Webhook
Instead of investigating each component independently, the same correlation ID can be used to follow the complete operation.
X-Correlation-ID
↓
Application Logs
↓
Background Task Logs
↓
External Service Logs
↓
Webhook Logs
This significantly reduces the time required to understand distributed failures.
Container and application logs are centralized in Azure Log Analytics.
AKS Pods
↓
Container Insights
↓
Log Analytics Workspace
↓
KQL
This becomes the main source for answering:
- What happened?
- Which pod handled the request?
- When did it happen?
- Which request caused it?
- What happened immediately before the failure?
The logs can be filtered by:
Correlation ID
Container
Application
Severity
Timestamp
Exception
This provides a central place to investigate production behavior instead of manually checking individual pods.
Prometheus provides the metrics layer of the monitoring architecture.
The important metrics include:
- CPU usage
- Memory usage
- Pod count
- Replica availability
- Container restarts
- Node health
- OOMKilled conditions
- Request rate
- Response latency
- 4xx / 5xx errors
- Application-specific metrics
- Queue depth
- Worker count
- Task duration
- Task failures
- Retry count
- Request rate
- HTTP status codes
- 502 / 504
- Error percentage
The purpose is to understand system behavior over time, not just individual errors.
Grafana provides a visual operational view of the system.
The dashboard is organized around the most important production areas.
Node CPU
Node Memory
Pod Count
Unavailable Pods
Request Rate
Latency
4xx / 5xx
Pod CPU
Pod Memory
Restarts
Queue Depth
Worker Count
Task Failures
Task Duration
Traffic
502 / 504
Error Percentage
The goal is to make important production behavior visible at a glance.
Monitoring becomes useful when important changes can trigger action.
I structured alerts around three major categories.
Memory usage approaching pod limit
↓
Warning
↓
Investigate before OOMKilled
Celery Queue
↓
Queue continuously increasing
↓
Workers cannot keep up
↓
Processing delay
502 / 504 / 5xx increase
↓
Possible upstream/application problem
↓
Investigate before wider impact
Alert thresholds should be based on real production behavior and adjusted over time instead of blindly copying thresholds from another system.
Kubernetes health checks provide another layer of protection.
Startup Probe
↓
Has the application started?
Readiness Probe
↓
Can this pod receive traffic?
Liveness Probe
↓
Should Kubernetes restart this container?
A lightweight health endpoint such as:
/health/
can be used as the basis for these checks.
Monitoring tells engineers something is wrong.
Health probes allow Kubernetes to automatically react to unhealthy workloads.
The overall production monitoring lifecycle is:
Application Request
↓
Application / Kubernetes
↓
Errors + Logs + Metrics
↓
Sentry + Log Analytics + Prometheus
↓
Grafana
↓
Alerts
↓
Incident Investigation
↓
Root Cause
↓
Fix
↓
Verification
↓
Prevention
This creates a complete feedback loop instead of treating monitoring as only a dashboard.
Monitoring is most valuable during production incidents.
When a pod repeatedly crashes:
Pod Starts
↓
Application Fails
↓
Container Restarts
↓
CrashLoopBackOff
Investigation focuses on:
- Application traceback
- Environment configuration
- Secrets
- Database connectivity
- Redis connectivity
- Container command
- Dependency failures
- Health probes
Useful commands:
kubectl logs <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --previous
kubectl describe pod <pod> -n <namespace>When a container exceeds its memory limit:
Memory Usage
↓
Limit Exceeded
↓
Container Killed
↓
Pod Restarts
Investigation should consider:
- Worker concurrency
- Large payloads
- Image processing
- AI/OCR processing
- Memory leaks
- Resource limits
Useful commands:
kubectl top pods -n <namespace>
kubectl describe pod <pod> -n <namespace>Do not immediately increase memory. First understand why memory increased.
When a pod cannot start because its image cannot be pulled:
Deployment
↓
Image
↓
Registry
↓
AKS Pull
Check:
- Repository
- Image tag
- Whether the image exists
- Registry authentication
- AKS pull permissions
- Pod events
The monitoring mindset is:
Observe
↓
Identify failing layer
↓
Verify evidence
↓
Fix only the failing layer
For ingress errors, follow the complete request path:
DNS
↓
Load Balancer
↓
Ingress
↓
Service
↓
Pod
↓
Django
↓
Database / Redis / External API
Check:
- Ingress metrics
- Ingress logs
- Pod availability
- Readiness status
- Application latency
- Database behavior
- Redis behavior
- External dependency behavior
A 502/504 is a symptom at the edge. The actual root cause may be deeper inside the application or infrastructure.
Background workloads require their own monitoring.
Django
↓
Redis Queue
↓
Celery Workers
↓
Background Processing
A useful warning pattern is:
Queue Depth ↑
Worker Capacity ↓
Task Duration ↑
↓
Processing Bottleneck
Investigate:
- Worker health
- Queue depth
- Task duration
- Task failures
- Retry behavior
- Redis
- Database
- External APIs
- Heavy AI/image workloads
Possible improvements:
- Optimize tasks.
- Increase worker capacity.
- Separate heavy workloads.
- Introduce queue-based scaling.
- Improve retry handling.
When a production issue is reported:
1. Get timestamp / correlation ID
↓
2. Check Sentry
↓
3. Search Log Analytics
↓
4. Identify affected pod/component
↓
5. Check Kubernetes events
↓
6. Check Prometheus metrics
↓
7. Check Redis / Celery
↓
8. Check database / external service
↓
9. Identify root cause
↓
10. Fix → Verify → Prevent
Do not randomly change configuration. Follow the request/data path and collect evidence at every layer.
One important production lesson is:
Local → Works
Production → Fails
does not automatically mean the code is wrong.
Compare:
Code Version
Database Data
Environment Variables
Secrets
Redis / Cache
Container Image
Azure Resource
API Deployment
Network
The purpose of monitoring is to identify which layer differs, rather than immediately modifying working application logic.
For Azure Communication Services email:
Application
↓
Azure Communication Services
↓
Email Processing
↓
Delivery Event
When investigating an email issue, collect:
- Recipient
- Timestamp
- Correlation ID
- Event/message ID
- Delivery status
Application success does not automatically prove final email delivery.
The actual delivery event should be verified through Azure monitoring/logs.
AI integrations introduce another important monitoring layer.
Monitor:
Request Count
Latency
404
401 / 403
429
5xx
Token Usage
Deployment
When a request works locally but fails in production, compare:
Endpoint
Deployment Name
API Version
Azure OpenAI Resource
Environment Variables
Authentication
Network
A particularly important distinction is:
Model Name
≠
Azure Deployment Name
Monitoring systems themselves must be secure.
Never expose:
- API keys
- Access tokens
- Database passwords
- Redis credentials
- Key Vault secrets
- Sensitive PII unnecessarily
Use secure secret management:
Application
↓
Managed Identity
↓
Key Vault
↓
Secret
Sentry's PII collection should also be reviewed according to the application's security and compliance requirements.
| Decision | Reason |
|---|---|
| 🐛 Sentry | Application-level errors and performance visibility |
| 📝 Structured JSON logs | Easier centralized searching and investigation |
| 🔗 Correlation IDs | Trace one operation across services |
| ☁️ Log Analytics | Centralized Azure-native log storage |
| 📈 Managed Prometheus | Production metrics without maintaining Prometheus infrastructure |
| 📊 Grafana | Operational dashboards |
| 🚨 Threshold-based alerts | Detect important issues early |
| ❤️ Health probes | Allow Kubernetes to react to unhealthy pods |
| 🔎 Evidence-first troubleshooting | Prevent random production changes |
| 🔐 Key Vault / Managed Identity | Keep monitoring-related secrets secure |
The monitoring architecture provides:
- 👀 Better visibility into production behavior.
- 🐛 Faster identification of application errors.
- 🔎 Centralized log investigation.
- 🔗 End-to-end request tracing.
- 📈 Clear infrastructure and application metrics.
- 📊 Easier operational dashboards.
- 🚨 Earlier detection of production issues.
- 🧯 Faster troubleshooting.
- ❤️ Better Kubernetes workload health.
- 🧠 A repeatable incident investigation process.
- 📚 Reusable knowledge for future projects.
- 👀 Monitoring should be designed together with the application, not added only after incidents occur.
- 🔗 Correlation IDs are extremely valuable in distributed systems.
- 📝 Structured logs make production investigation much easier.
- 📈 Metrics show system behavior that individual logs cannot.
- 📊 Dashboards should focus on actionable information rather than displaying every available metric.
- 🚨 Alerts should identify meaningful problems, not create unnecessary noise.
- 🧯 A production incident should be investigated using evidence from multiple layers.
- 🧠 Local success does not guarantee production success because data, configuration, infrastructure, and dependencies can differ.
- ❤️ Queue-based systems require dedicated monitoring for backlog and worker health.
- 🔐 Monitoring data can contain sensitive information and must be handled securely.
- 📚 Every important incident should become a documented lesson and future improvement.
- Sentry configured
- Django errors captured
- Celery failures captured
- Slow requests visible
- Important business errors monitored
- Structured JSON logging
- Correlation IDs
- Log Analytics configured
- KQL investigation queries available
- CPU
- Memory
- Pod availability
- Restarts
- Queue depth
- Worker health
- Application latency
- Ingress 5xx
- Cluster health
- Application health
- Celery / Redis
- Ingress
- Critical business metrics where required
- Memory pressure
- Queue backlog
- 5xx spike
- Pod availability
- Critical application errors
- Startup probe
- Readiness probe
- Liveness probe
- Incident runbook
- Root-cause documentation
- Verification process
- Prevention action
Every important production issue should eventually become reusable engineering knowledge.
# Incident: <Incident Name>
## Date
<date>
## Impact
<what users/system experienced>
## Symptoms
<what monitoring showed>
## Evidence
<Sentry / Log Analytics / Prometheus / Kubernetes>
## Root Cause
<actual technical cause>
## Resolution
<what was changed>
## Verification
<how the fix was confirmed>
## Prevention
<what will prevent recurrence>
## Lesson Learned
<what should be remembered for future projects>- 📈 Sentry Release Health
- 🔍 Distributed tracing / OpenTelemetry
- 🌐 Synthetic uptime monitoring
- 📊 SLO / SLA dashboards
- 💬 Teams / Slack alert routing
- 🚨 Severity-based alert routing
- ☸️ KEDA queue-based Celery autoscaling
- 🗄️ Database performance monitoring
- ❤️ Redis cache hit/miss monitoring
- 🌐 External API latency and failure monitoring
- 🤖 AI latency, token, and cost monitoring
- 📊 Business KPI monitoring
- 🧠 Automated anomaly detection
- 🔗 Correlation IDs across outbound webhooks
ERROR
↓
Sentry
LOG
↓
Log Analytics
METRIC
↓
Prometheus
DASHBOARD
↓
Grafana
HEALTH
↓
Kubernetes
ALERT
↓
Engineer
INCIDENT
↓
Observe
↓
Correlate
↓
Isolate
↓
Verify
↓
Fix
↓
Prevent
Don't wait for users to report that the system is broken. Build observability that shows what failed, what happened, how the system is behaving, where the problem is, and what should be improved next.
The complete monitoring philosophy is:
OBSERVE
↓
Detect the problem
↓
CORRELATE
↓
Connect the evidence
↓
ISOLATE
↓
Find the failing layer
↓
VERIFY
↓
Confirm the root cause
↓
FIX
↓
PREVENT
↓
Improve the system
This repository is part of my personal Production Engineering Portfolio.
This monitoring architecture is based on real production implementation and operational experience. The focus is deliberately on:
Architecture
↓
Monitoring Strategy
↓
Implementation Approach
↓
Operational Visibility
↓
Troubleshooting
↓
Engineering Decisions
↓
Lessons Learned
↓
Future Improvements
No proprietary application source code, credentials, customer information, or confidential production configuration is included.
The goal is to maintain a long-term engineering reference that helps me understand production systems deeply, troubleshoot incidents systematically, reproduce monitoring patterns in future projects, and continuously improve the reliability of the applications I build.
Sentry tells me what broke, Log Analytics tells me what happened, Prometheus tells me how the system is behaving, Grafana shows me the system, Kubernetes checks its health, and alerts help me act before users are affected.