Skip to content

Latest commit

 

History

History
167 lines (107 loc) · 5.09 KB

File metadata and controls

167 lines (107 loc) · 5.09 KB

Observability

Why does the framework collect more than just HTTP response times?

What does the framework measure, and why are those measurements useful?

Performance testing often focuses on a single number: response time.

Performance metrics alone are NOT sufficient to evaluate system behaviour.

While latency is certainly important, it represents only one aspect of system behaviour.

Understanding why a system performs the way it does requires observing multiple layers simultaneously.

A successful load test should answer three independent questions:

  • How fast is the system?
  • Is the infrastructure healthy?
  • Is the application still behaving correctly?

This framework collects metrics from each of these areas and correlates them to provide a complete picture of the system under load.


Performance Metrics

Performance metrics describe how efficiently the system processes requests.

These are the metrics most commonly associated with performance testing and are typically the first indicators of performance degradation.

Examples include:

  • HTTP response time
  • throughput
  • request rate
  • error rate
  • WebSocket latency
  • database query duration

These metrics answer questions such as:

  • Did the application become slower?
  • Did the application become unstable?
  • Which operation consumes the most time?
  • Does latency increase with load?
  • Are requests failing?

While essential, performance metrics alone rarely explain why performance changed.


Infrastructure Metrics

Infrastructure metrics describe the health of the environment hosting the application.

Their purpose is to determine whether performance issues originate from resource limitations rather than application logic.

Examples include:

  • CPU utilisation
  • memory utilisation
  • Kubernetes pod resources
  • Kubernetes pod failures
  • database connections
  • database query execution

Infrastructure metrics help answer questions such as:

  • Is the system CPU-bound?
  • Is memory becoming exhausted?
  • Is the database overloaded?
  • Is Kubernetes scaling correctly?
  • Is the application waiting for external resources?

Infrastructure metrics provide valuable context but should never be interpreted independently.


Business Metrics

Business metrics verify that the application continues to function correctly while under load.

Unlike performance metrics, they measure business behaviour rather than technical implementation.

Examples include:

  • currently active Virtual Users
  • Agent utilisation
  • conversation status distribution
  • assigned conversations
  • assignment success rate
  • conversation lifecycle events

Business metrics answer questions such as:

  • Are users actually progressing through the workflow?
  • Are conversations being assigned correctly?
  • Is work distributed evenly between agents?
  • Are messages processed successfully?
  • Does increasing load produce meaningful business activity?

These metrics are particularly useful because a system can remain fast while no longer behaving correctly.

For example:

  • every HTTP request succeeds
  • response times remain low
  • infrastructure remains healthy

yet

  • conversations stop being assigned
  • agents become idle
  • messages are no longer delivered

Without business metrics, such failures can easily remain unnoticed.


Metric Correlation

Individual metrics rarely provide complete answers.

The framework is designed around correlating multiple independent observations.

Rather than analysing a single graph, investigations typically compare several metrics collected during the same time period.

For example:

Observation Possible Interpretation
Response time ↑ + CPU ↑ Infrastructure saturation
Response time ↑ + CPU stable Application logic bottleneck
Response time ↑ + Database connections ↑ Database throttling
Active users ↓ + throughput ↓ Functional regression
WebSocket latency ↑ + HTTP stable = Messaging subsystem degradation
Business activity ↓ + HTTP success rate stable = Functional regression despite healthy APIs
Infrastructure healthy + Business metrics abnormal Application logic failure
Infrastructure healthy + Business metrics healthy + Response time ↑ Performance regression within the application

Correlating independent measurements significantly reduces the number of incorrect conclusions that can be drawn from isolated metrics.


Investigation Examples

Understanding individual metrics is only the first step.

The real value of observability comes from finding real system problems when HTTP metrics aren't enough


Observability Philosophy

This framework treats performance testing as more than measuring latency.

Complex systems deserve a complex approach!

A successful performance test should verify that:

  • the application remains responsive,
  • the infrastructure remains healthy,
  • and the business workflow continues to function correctly.

Only by combining these perspectives can performance investigations produce reliable conclusions.