Skip to content

reporting.md

Latest commit

 

History

History
379 lines (262 loc) · 12.2 KB

reporting.md

File metadata and controls

379 lines (262 loc) · 12.2 KB
Raw

Reporting Service

How are raw performance metrics transformed into structured datasets?

This document explains the responsibilities of the reporting service, the processing flow after a test execution finishes, and how datasets are organized for further analysis.

Introduction

The reporting service is a standalone Node.js application responsible for post-test processing.

Unlike k6, whose responsibility is generating workload and collecting runtime metrics, the reporting service focuses entirely on transforming collected data into persistent datasets.

Rather than generating HTML or PDF reports directly, it produces a structured collection of CSV datasets that can later be consumed by visualization tools, comparison utilities or custom report generators.

Keeping reporting independent from load generation allows both systems to evolve separately.

P.S. I do own a custom HTML report generator but it is out of scope for this project.

Reporting Workflow

Diagram: Reporting Service - Data Flow

graph LR
    subgraph Input["Input"]
        Payload["HTTP Payload<br/>(POST /start-reporting)"]
    end

    subgraph Validation["Validation"]
        Val["validateReportingRequest()"]
        FieldCheck["Check required fields"]
        ResourceCheck["Check resource config<br/>(all-or-nothing)"]
    end

    subgraph Orchestration["Orchestration"]
        Orch["orchestrateReporting()"]
        LoopCat["Loop each category<br/>(Aggregation, WebSocket, etc)"]
        GetPeriods["getPeriodsForCategory()"]
    end

    subgraph InfluxCollection["InfluxDB Collection"]
        InfluxOrch["collectMetricsByPeriod()"]
        LoopGraph["Loop each graph"]
        LoopGroup["Loop each group"]
        IntervalDecide["Determine interval<br/>(group.name includes 'spread'?)"]
        Substitute["substitutePlaceholders()<br/>(dateTimeStart, dateTimeEnd,<br/>interval, CONV_MSG_CYCLES)"]
        Query["runInfluxQuery()"]
        Extract["extractInfluxDBSeries()"]
        Format["seriesToCsvLines()<br/>(include tags if transaction)"]
        WriteCSV["Write CSV file"]
    end

    subgraph ResourceCollection["Resource Collection (Optional)"]
        ResOrch["collectResourcesAndGeneralInfo()"]
        FetchK8s["fetchPods()"]
        ExtractPod["extractPodInfo()"]
        QueryProm["queryPrometheusRange()<br/>(CPU, Memory, Limits)"]
        MergeMetrics["mergeMetricsWithLimits()"]
        WriteResCSV["Write Resources/*.csv"]
        WriteGenInfo["Write GeneralInfo.csv"]
    end

    subgraph Output["Output"]
        Files["CSV Files"]
    end

    Payload -->|1| Val
    Val -->|2| FieldCheck
    FieldCheck -->|3| ResourceCheck
    ResourceCheck -->|Valid| Orch

    Orch -->|4| LoopCat
    LoopCat -->|5| GetPeriods
    GetPeriods -->|6| InfluxOrch

    InfluxOrch -->|7| LoopGraph
    LoopGraph -->|8| LoopGroup
    LoopGroup -->|9| IntervalDecide
    IntervalDecide -->|10| Substitute
    Substitute -->|11| Query
    Query -->|12| Extract
    Extract -->|13| Format
    Format -->|14| WriteCSV

    InfluxOrch -->|15| ResOrch
    ResOrch -->|16| FetchK8s
    FetchK8s -->|17| ExtractPod
    ExtractPod -->|18| QueryProm
    QueryProm -->|19| MergeMetrics
    MergeMetrics -->|20| WriteResCSV
    MergeMetrics -->|21| WriteGenInfo

    WriteCSV -->|22| Files
    WriteResCSV -->|22| Files
    WriteGenInfo -->|22| Files

    style Validation fill:#FFB6C1,stroke:#333
    style Orchestration fill:#87CEEB,stroke:#333
    style InfluxCollection fill:#90EE90,stroke:#333
    style ResourceCollection fill:#FFD700,stroke:#333
    style Output fill:#DDA0DD,stroke:#333
Loading

The reporting process begins during the teardown() phase of the k6 execution.

After all scenarios have finished:

  1. k6 determines the test time boundaries.
  2. A request containing test configuration and timestamps is sent to the reporting service.
  3. The reporting service validates the request.
  4. Metrics are collected from InfluxDB.
  5. Resource information is collected from Kubernetes and Prometheus (optional).
  6. Every collected dataset is formatted as CSV.
  7. CSV files are written to the output directory.

The reporting request is synchronous.

This means the k6 execution does not finish until reporting either succeeds or returns an error.

Time Windows

One of the core responsibilities of the reporting service is dividing a test into meaningful execution periods.

Instead of querying the entire execution as one continuous interval, three independent time windows are created.

  • RampUp — from the beginning of the test until all virtual users have reached their target count.
  • MaxLoad — from the end of the ramp-up until the end of the workload.
  • WholeRun — the complete execution.

Different categories use different reporting strategies.

Aggregation and WebSocket datasets are generated separately for every period.

Metrics and Meta datasets describe the whole execution and therefore are generated only once.

Everything is not set in stone and easily configurable!

Diagram: Period-Based Reporting Strategy

graph TD
    QueryList["Query List<br/>(query_list.json)"]
    
    QueryList -->|Category| Cat1["Aggregation"]
    QueryList -->|Category| Cat2["WebSocket"]
    QueryList -->|Category| Cat3["Metrics"]
    QueryList -->|Category| Cat4["Meta"]

    Cat1 -->|CATEGORY_PERIOD_MAP| Per1["[RampUp, MaxLoad, WholeRun]<br/>(3 directories)"]
    Cat2 -->|CATEGORY_PERIOD_MAP| Per2["[RampUp, MaxLoad, WholeRun]<br/>(3 directories)"]
    Cat3 -->|CATEGORY_PERIOD_MAP| Per3["null<br/>(flat, no periods)"]
    Cat4 -->|CATEGORY_PERIOD_MAP| Per4["null<br/>(flat, no periods)"]

    Per1 --> Window1["RampUp Window:<br/>test_start → test_maxload<br/>(VU ramp-up phase)"]
    Per1 --> Window2["MaxLoad Window:<br/>test_maxload → test_end<br/>(steady-state phase)"]
    Per1 --> Window3["WholeRun Window:<br/>test_start → test_end<br/>(entire test)"]

    Window1 -->|InfluxDB Query| Query1["SELECT ... WHERE<br/>time > RampUp_start AND time < RampUp_end"]
    Window2 -->|InfluxDB Query| Query2["SELECT ... WHERE<br/>time > MaxLoad_start AND time < MaxLoad_end"]
    Window3 -->|InfluxDB Query| Query3["SELECT ... WHERE<br/>time > WholeRun_start AND time < WholeRun_end"]

    Query1 -->|Results| CSV1["Aggregation/RampUp/graph.csv"]
    Query2 -->|Results| CSV2["Aggregation/MaxLoad/graph.csv"]
    Query3 -->|Results| CSV3["Aggregation/WholeRun/graph.csv"]

    Per3 -->|Flat| FlatQ["SELECT ... WHERE<br/>time > test_start AND time < test_end"]
    FlatQ -->|Results| FlatCSV["Metrics/graph.csv<br/>(no period subdirs)"]

    style Cat1 fill:#FF6B6B,stroke:#333,color:#fff
    style Cat2 fill:#4ECDC4,stroke:#333,color:#fff
    style Cat3 fill:#45B7D1,stroke:#333,color:#fff
    style Cat4 fill:#FFA07A,stroke:#333,color:#fff
    style Per1 fill:#FF8C94,stroke:#333,color:#fff
    style Per2 fill:#FF8C94,stroke:#333,color:#fff
    style Per3 fill:#A8D8EA,stroke:#333,color:#fff
    style Per4 fill:#FFB4A2,stroke:#333,color:#fff
Loading

Query Execution

The reporting service is configuration-driven.

It does not contain hardcoded InfluxDB queries.

Instead, every reporting category is described in reporting/query_list.json.

For every configured category the service:

  • determines the reporting periods
  • selects the appropriate aggregation interval
  • substitutes query placeholders
  • executes the InfluxDB query
  • extracts returned series
  • formats the result as CSV

The placeholders replaced during execution include:

  • reporting time window
  • aggregation interval
  • conversation cycle count (for messaging)

This allows the same query template to be reused for different execution periods without duplicating query definitions.

Resource Collection

Infrastructure metrics are collected independently from application metrics.

If Kubernetes and Prometheus are configured, the reporting service:

  • retrieves pod metadata from Kubernetes
  • discovers running pods on namespace
  • queries Prometheus for resource usage
  • combines usage metrics with configured resource limits
  • writes the resulting datasets alongside application metrics

If infrastructure collection fails, reporting continues and application datasets are still generated.

Infrastructure metrics are treated as an optional extension rather than a mandatory dependency

Results Structure

Every test execution produces its own directory identified by the build tag.

results/<BUILD_TAG>/
├── Aggregation/
│   ├── RampUp/
│   │   ├── Aggregation data.csv
│   │   ├── Aggregation by interval data.csv
│   │   ├── Aggregation by transaction.csv
│   │   ├── Aggregation data spread.csv
│   │   └── Aggregation by interval data spread.csv
│   ├── MaxLoad/
│   │   └── (same files as RampUp)
│   └── WholeRun/
│       └── (same files as RampUp)
├── WebSocket/
│   ├── RampUp/
│   │   ├── WebSocket Metrics.csv
│   │   └── WebSocket Counters.csv
│   ├── MaxLoad/
│   │   └── (same files as RampUp)
│   └── WholeRun/
│       └── (same files as RampUp)
├── Metrics/
│   ├── Database requests AND Create Agent.csv
│   └── ... (per logical metric group)
├── Meta/
│   ├── Agent Activity.csv
│   ├── Database Connections.csv
│   └── Conversations by status.csv
├── Resources/
│   ├── myapp-abc123_CPU.csv
│   ├── myapp-abc123_Memory.csv
│   └── ... (per pod, per metric)
└── GeneralInfo.csv

Every directory represents one logical category of collected information.

This organization allows individual datasets to evolve independently while remaining easy to locate during post-test analysis.

Dataset Categories

Aggregation

Provides statistical summaries of collected measurements.

Depending on the dataset, information is grouped:

  • by execution period
  • by transaction
  • by aggregation interval
  • by spread (variability)

These datasets are intended to quickly identify bottlenecks before investigating detailed time-series data.

WebSocket

Contains metrics describing asynchronous communication.

Separate datasets are generated for every execution period, making it possible to compare WebSocket behaviour during ramp-up and steady-state execution.

Metrics

Contains grouped application metrics collected throughout the test.

Unlike Aggregation datasets, these files preserve logical groupings defined in the query configuration and are not divided into execution periods.

Meta

Meta (or Business Metrics) datasets describe the application behaviour under test.

Examples include:

  • agent activity (utilization)
  • database connections
  • conversations by state

These datasets are useful for validating that the application business logic behaved as expected.

Resources

Infrastructure datasets combine Kubernetes metadata with Prometheus measurements.

Each pod produces its own resource files, allowing infrastructure behaviour to be analysed independently of application metrics.

GeneralInfo

Contains metadata describing the execution.

Examples include:

  • test timings
  • workload configuration
  • pod specifications
  • resource limits

This file provides the context required to correctly interpret all generated datasets.

Extending the Reporting Service

The reporting pipeline was designed to be configuration-driven.

Most reporting extensions require changes only to the query configuration.

Typical examples include:

  • adding a new metric category

as simeple as adding 1 line to reporting/query_list.json

  • introducing another graph

as simeple as adding 2 lines - name and InfluxQL query - to reporting/query_list.json

  • extending resource collection

as simple as adding 2 lines — metric name and PromQL query — to getResourceMetricQueries()

  • adding another reporting period

define time window in collectMetricsByPeriod()

wire it up to categories in getPeriodsForCategory()

Because orchestration, querying and CSV generation are separated into independent modules, new datasets can usually be introduced without modifying existing reporting logic.

Moreover, the code is heavily documented.

I can claim that even a person who never coded would understand what is what and how it works!