Why response time alone rarely explains what happened during a performance test.
One of the first graphs shown after almost every performance test is latency:
-
Average response time
-
95th percentile
-
99th percentile.
-
...
These metrics are useful, but they answer only one question:
How fast did the system respond?
Unfortunately, performance investigations usually require answering a different question:
Why did the system behave this way?
Those are fundamentally different problems.
Imagine the following observations:
- Response time increased by 60%.
- CPU utilisation reached 95%.
- Memory usage remained stable.
- Business metrics continued to behave normally.
The response time graph immediately indicates a performance regression.
Infrastructure metrics explain why.
High CPU utilisation strongly suggests that the application has reached the available processing capacity rather than suffering from an application logic issue.
The investigation therefore shifts from application code to infrastructure sizing and scaling.
Another execution produces different observations.
- Response time increased.
- CPU utilisation remained below 40%.
- Memory remained stable.
- PostgreSQL connection count doubled.
- Business metrics remained healthy.
Again, latency identifies the symptom.
This time infrastructure metrics indicate that compute resources are not the limiting factor.
The increasing number of database connections becomes the primary indicator, suggesting database contention or inefficient query execution rather than infrastructure saturation.
Without those additional metrics, the investigation could easily begin in the wrong subsystem.
Now consider a different situation.
- Response times remain stable.
- Error rate remains close to zero.
- Infrastructure metrics remain healthy.
- Agent utilisation drops significantly.
- Waiting conversations continuously increase.
At first glance, the system appears healthy.
Every request succeeds.
Latency remains low.
Infrastructure is under little pressure.
However, business metrics reveal that conversations are no longer progressing through the system.
The application remains responsive, but it no longer performs its primary business function.
This illustrates why business metrics are collected alongside traditional performance metrics.
Performance testing should verify not only that the application remains fast, but also that it continues to behave correctly under load.
Performance improvements deserve the same level of investigation as regressions.
Suppose the next release shows encouraging results.
- Average response time decreased by 20%.
- Error rate remained unchanged.
- Infrastructure metrics remained stable.
At first glance, this appears to be a successful optimisation.
However, another metric tells a different story.
- Response time spread increased dramatically.
Although the average became lower, response times became significantly less consistent.
Some requests completed much faster than before.
Others became considerably slower.
The average hides this behaviour by balancing fast and slow requests into a single number.
From a statistical perspective, the system became less predictable, even though it became faster on average.
This observation introduces an important operational risk.
During periods of moderate load, occasional slow requests may go unnoticed.
During peak traffic, however, those slower executions begin to overlap.
Request queues grow.
Processing latency increases further.
Additional resources become saturated.
Eventually Kubernetes may terminate unhealthy pods.
The replacement pod immediately receives the existing request backlog, preventing it from completing initialization before becoming overloaded again.
Instead of recovering, the service repeatedly enters a crash-restart cycle while the accumulated workload continues to increase.
A dashboard showing only average latency would report an improvement.
The response time spread reveals that the system actually became less stable.
None of the previous examples relied on a single metric.
Every conclusion emerged by correlating multiple independent observations.
Latency identifies that behaviour changed.
Infrastructure metrics explain whether resources became constrained.
Business metrics verify whether the application continues to perform useful work.
Together they provide significantly stronger evidence than any individual graph.
Generating load is relatively straightforward.
Understanding why a system behaves the way it does is considerably more difficult.
Latency remains one of the most valuable performance metrics available to performance engineers.
But it simply isn't sufficient on its own.
The goal of observability is not to collect more dashboards.
The goal is to reduce uncertainty during performance investigations.
Every additional metric should answer a question that latency alone cannot.
When different categories of metrics consistently support the same conclusion, performance analysis becomes faster, more reliable and far easier to trust.
That is the philosophy behind the observability approach used throughout this framework.