Report audit instance environment, storage footprint and ingestion health in the usage report (#5965)

* Report audit instance environment data in the usage report

The audit instance serves host and storage facts on a new anonymous GET /api/environment endpoint, gathered from audit-local environment data providers. The primary polls every remote alongside the daily audit verification pass through the new IConfigurationApi.GetRemoteEnvironments, stores the per-instance dictionaries as AuditEnvironmentMetadata in the licensing stores, and the usage report emits them under the Audit.* prefix aggregated across instances: agreement passes through, processor count and memory take the maximum, and disagreement reports Mixed. Remotes that predate the endpoint are omitted rather than guessed.

* Detect whether audit instances share the primary's machine, database or server

Each instance hashes what identifies the machine it runs on and the storage it writes to: the OS machine id on Linux, the machine name elsewhere, NotApplicable in containers, and per engine the RavenDB server URL and database, SQL Server's ServerName and database, or PostgreSQL's cluster system identifier and database, plus the schema. The audit instance serves the hashes on the environment endpoint, the primary compares them with its own during collection, and only the classification reaches the report: Audit.SameMachine as True, False, Mixed, NotApplicable or Unknown, and Audit.DatabaseSharing as SameSchema, SameDatabase, SameServer, SeparateServer, NotApplicable or Unknown, aggregated to the most shared class across instances. A side that cannot be identified reports Unknown rather than guessing from configuration the two sides may spell differently.

* Report storage footprint, edition and service tier for the primary and audit instances

Storage.SizeGB, Storage.MessageCount and Storage.UnresolvedFailedMessages come from catalog statistics scoped to the configured schema on SQL Server and PostgreSQL and from database and collection statistics on RavenDB, never from counting rows. Storage.ServerEdition carries SQL Server's own edition designation and Storage.ServiceObjective the Azure SQL service objective, both NotApplicable elsewhere. The audit instance serves the same keys over the environment endpoint, and the report sums sizes and counts across audit instances with instances sharing a database counted once, keyed by the identity hashes from the sharing detection. A catalog that has no answer, such as a row estimate on a never analysed PostgreSQL table, reports Unknown rather than a guess.

* Collect hourly ingestion rates and busy time for the error and audit sides

Both ingestion pipelines keep running totals in memory: messages stored per completed batch, time spent processing batches, storage write time on the error side, and how far behind the endpoint's own timestamp each message arrived, bucketed at one, ten and sixty minutes. The audit instance serves its counters on the environment endpoint. A new hourly collector in the licensing component polls the primary's counters in process and every audit instance's over the API, folds the deltas into daily records in the licensing store with the audit side summed across instances, and keeps 400 days. A restarted process contributes its whole counters as the delta, a counter set seen for the first time only establishes a baseline, and a collector restart builds on what today's record already holds. The report emits per side the average daily messages, the busiest hour's messages and its busy share, plus storage milliseconds per message for the error side, from the days inside the report window.

Known gap, accepted in the plan: --error-ingestion-only workers keep their own counters and nothing polls them.

* Report health signals for the error and audit sides

Health.Error.FailedImports and Health.Error.RetentionBehindHours come from the primary's persister: failed import rows counted from the store, and how many hours the oldest failed message sits past the retention window on the EF persisters, with RavenDB omitting the retention key because expiry is engine driven. The audit instance serves its failed audit import count on the environment endpoint, summed across instances with shared databases counted once. Restarts are the process start time changes the hourly collector observes between polls, and the lag buckets it accumulates become Health.*.LagOver1MinPercent, LagOver10MinPercent and LagOver60MinPercent over the report window, emitted only when any message carried a usable timestamp.

* Replace hourly ingestion history collector with live counters served on the environment endpoint

Instead of polling ingestion counters hourly, accumulating deltas into daily records in the licensing store, and reading those records back into the report, each instance now computes ingestion and health summaries directly from its running totals at report time.

Both the audit and error sides expose their counters through IEnvironmentDataProvider implementations that call IngestionSummary.Describe, which yields keys like Ingestion.AvgDailyMessages, Ingestion.BusyPercent, Health.UptimeHours, and Health.LagOver*Percent relative to the process uptime. The audit instance serves these on the environment endpoint alongside its existing data; the primary aggregates them under the Audit.* prefix when building the report, summing daily message rates across instances and taking the worst saturation and lag values.

This removes IngestionHistoryCollectorHostedService, IngestionHistory and IngestionDay contracts, IErrorIngestionSnapshotProvider, GetRemoteIngestionCounters from IConfigurationApi, and GetIngestionHistory/SaveIngestionHistory from ILicensingDataStore and both persistence backends.

* Report database hosting, footprint and identity for EF Core audit persisters

SQL Server and PostgreSQL audit persistence now probe how their database is hosted, how much it stores, and what identifies it, the same way the primary instance's EF persisters already did, so the report gets real Storage.* data for audit instances on these backends instead of nothing. SQL Server's edition and service objective are normalised to a small fixed set of categories instead of raw server strings, and both EF persisters resolve the configured schema the same way when none is set.

AuditEnvironmentDataAggregator now passes through only an explicit whitelist of known keys, so a key served by an audit instance of another version can never reach the report unreviewed. ThroughputCollector copies broker metadata into a new dictionary before it reaches the report, so the stored metadata is never mutated. The environment endpoint's machine hash field is renamed from machine_id_hash to machine_name_hash to match what it actually hashes, and RavenDB's storage identity now treats a loopback address as this machine rather than a shared one.

* Fix flaky full-text index size assertion in SQL Server audit footprint test

WaitForFullTextIndex is now exposed as a public method with a default cancellation token so the environment data test can reuse the same wait the search tests rely on, instead of racing the index population. The test now waits for the full-text index to catch up before probing storage footprint, and asserts the footprint is only greater than the tables size rather than an exact sum, since the full-text index size can still drift slightly after the wait. The duplicated inline SQL for computing table and full-text sizes is consolidated into a single SizeGB helper.

* Retry SQL Server storage footprint probe on deadlock or lost scan position

Catalog reads under READ COMMITTED can deadlock with concurrent DDL, so the probe now runs under READ UNCOMMITTED instead. That isolation level can let a scan lose its place when pages move, so both error codes are now retried up to three times with a short backoff before the probe gives up and returns null. Applied to both the primary and audit EF Core SQL Server persisters, and the audit environment data test's inline SQL is updated to match the new isolation level.

* Add tests for mixed and summed database values in throughput report

* Add logging to ConfigurationApi for environment data retrieval errors

---------

Co-authored-by: Warwick Schroeder <warwick.schroeder@particular.net>

Author: johnsimons

src/Particular.LicensingComponent.Contracts/AuditEnvironmentMetadata.cs

@@ -0,0 +1,16 @@
+namespace Particular.LicensingComponent.Contracts;
+
+/// <summary>
+/// The environment data each audit remote served when last polled, one dictionary per remote that
+/// responded. Never saved by versions that predate it, so a null read means unknown and the report
+/// omits the keys.
+/// </summary>
+public record AuditEnvironmentMetadata(List<Dictionary<string, string>> Instances)
+{
+    /// <summary>
+    /// The key under which the primary stores which database an instance's data came from, so that
+    /// instances sharing a database are counted once when sizes and counts are summed. Underscore
+    /// prefixed keys are collection bookkeeping and never reach the report.
+    /// </summary>
+    public const string DatabaseKey = "_DatabaseKey";
+}

src/Particular.LicensingComponent.Persistence.InMemory/InMemoryLicensingDataStore.cs

@@ -14,6 +14,7 @@ public class InMemoryLicensingDataStore : ILicensingDataStore
     readonly Dictionary<EndpointIdentifier, ThroughputData> allThroughput = [];
     BrokerMetadata brokerMetadata = new(null, []);
     AuditServiceMetadata auditServiceMetadata = new([], []);
+    AuditEnvironmentMetadata? auditEnvironmentMetadata;
     List<string> reportMasks = [];
     LicensedEndpointDetails? endpointDetails = null;
     LicensedEndpointDetailsAmendment? endpointDetailsAmendment = null;
@@ -168,6 +169,14 @@ public Task SaveAuditServiceMetadata(AuditServiceMetadata auditServiceMetadata,
         return Task.CompletedTask;
     }
 
+    public Task<AuditEnvironmentMetadata?> GetAuditEnvironmentMetadata(CancellationToken cancellationToken = default) => Task.FromResult(auditEnvironmentMetadata);
+
+    public Task SaveAuditEnvironmentMetadata(AuditEnvironmentMetadata auditEnvironmentMetadata, CancellationToken cancellationToken = default)
+    {
+        this.auditEnvironmentMetadata = auditEnvironmentMetadata;
+        return Task.CompletedTask;
+    }
+
     public Task<List<string>> GetReportMasks(CancellationToken cancellationToken = default) => Task.FromResult(reportMasks);
 
     public Task SaveReportMasks(List<string> reportMasks, CancellationToken cancellationToken = default)

src/Particular.LicensingComponent.Persistence/ILicensingDataStore.cs

@@ -36,6 +36,10 @@ Task RecordEndpointThroughput(string endpointName, ThroughputSource throughputSo
     Task<AuditServiceMetadata> GetAuditServiceMetadata(CancellationToken cancellationToken = default);
 
     Task SaveAuditServiceMetadata(AuditServiceMetadata auditServiceMetadata, CancellationToken cancellationToken = default);
+
+    Task<AuditEnvironmentMetadata?> GetAuditEnvironmentMetadata(CancellationToken cancellationToken = default);
+
+    Task SaveAuditEnvironmentMetadata(AuditEnvironmentMetadata auditEnvironmentMetadata, CancellationToken cancellationToken = default);
     Task<List<string>> GetReportMasks(CancellationToken cancellationToken = default);
     Task SaveReportMasks(List<string> reportMasks, CancellationToken cancellationToken = default);
 

src/Particular.LicensingComponent.UnitTests/AuditQuery_Tests.cs

@@ -228,6 +228,8 @@ public Task<RemoteConfiguration[]> GetRemoteConfigs(CancellationToken cancellati
         }
 
         public Task<RootUrls> GetUrls(string baseUrl, CancellationToken cancellationToken = default) => throw new NotImplementedException();
+
+        public Task<RemoteEnvironment[]> GetRemoteEnvironments(CancellationToken cancellationToken = default) => throw new NotImplementedException();
     }
 
     class ConfigurationApi_ReturningRemotes(params RemoteConfiguration[] remotes) : IConfigurationApi
@@ -237,6 +239,8 @@ class ConfigurationApi_ReturningRemotes(params RemoteConfiguration[] remotes) :
         public Task<RemoteConfiguration[]> GetRemoteConfigs(CancellationToken cancellationToken = default) => Task.FromResult(remotes);
 
         public Task<RootUrls> GetUrls(string baseUrl, CancellationToken cancellationToken = default) => throw new NotImplementedException();
+
+        public Task<RemoteEnvironment[]> GetRemoteEnvironments(CancellationToken cancellationToken = default) => throw new NotImplementedException();
     }
 
     class ConfigurationApi_Configurable : IConfigurationApi
@@ -257,6 +261,8 @@ public Task<RemoteConfiguration[]> GetRemoteConfigs(CancellationToken cancellati
 
         public Task<RootUrls> GetUrls(string baseUrl, CancellationToken cancellationToken = default) => throw new NotImplementedException();
 
+        public Task<RemoteEnvironment[]> GetRemoteEnvironments(CancellationToken cancellationToken = default) => throw new NotImplementedException();
+
         public bool ReturnAuditConfig { get; set; }
         public string RemoteStatus { get; set; }
         public string RemoteVersion { get; set; }

src/Particular.LicensingComponent.UnitTests/AuditThroughputCollectorHostedService_Tests.cs

@@ -272,10 +272,40 @@ public async Task Should_store_how_many_audit_instances_are_configured_and_how_m
         }
     }
 
+    [Test]
+    public async Task Should_store_the_environment_data_of_every_responding_audit_instance()
+    {
+        var auditQuery = new AuditQuery_WithRemotes([new RemoteInstanceInformation { Status = "online" }])
+        {
+            Environments =
+            [
+                new Dictionary<string, string> { ["Storage.Type"] = "RavenDB", ["Host.ProcessorCount"] = "4" },
+                new Dictionary<string, string> { ["Storage.Type"] = "RavenDB", ["Host.ProcessorCount"] = "8" }
+            ]
+        };
+
+        using var auditThroughputCollectorHostedService = new AuditThroughputCollectorHostedService(
+            NullLogger<AuditThroughputCollectorHostedService>.Instance, configuration.ThroughputSettings, DataStore,
+            auditQuery, new FakeTimeProvider())
+        { DelayStart = TimeSpan.Zero };
+
+        await auditThroughputCollectorHostedService.StartAsync(CancellationToken.None);
+        await auditQuery.KnownEndpointsRequested.WaitAsync(TimeSpan.FromSeconds(30));
+        await auditThroughputCollectorHostedService.StopAsync(CancellationToken.None);
+
+        var auditEnvironmentMetadata = await DataStore.GetAuditEnvironmentMetadata();
+
+        Assert.That(auditEnvironmentMetadata, Is.Not.Null);
+        Assert.That(auditEnvironmentMetadata.Instances, Is.EqualTo(auditQuery.Environments));
+    }
+
     class AuditQuery_NoAuditRemotes : IAuditQuery
     {
         public SemanticVersion MinAuditCountsVersion => new(4, 29, 0);
 
+        public Task<List<Dictionary<string, string>>> GetAuditEnvironments(CancellationToken cancellationToken = default) =>
+            Task.FromResult<List<Dictionary<string, string>>>([]);
+
         public Func<RemoteInstanceInformation, bool> ValidRemoteInstances => r => true;
 
         public Task<IEnumerable<AuditCount>> GetAuditCountForEndpoint(string endpointUrlName,
@@ -309,6 +339,9 @@ public AuditQuery_WithOneEndpoint(string endpointName, long throughputCount, Dat
 
         public SemanticVersion MinAuditCountsVersion => new(4, 29, 0);
 
+        public Task<List<Dictionary<string, string>>> GetAuditEnvironments(CancellationToken cancellationToken = default) =>
+            Task.FromResult<List<Dictionary<string, string>>>([]);
+
         public Func<RemoteInstanceInformation, bool> ValidRemoteInstances => r => true;
 
         public Task<IEnumerable<AuditCount>> GetAuditCountForEndpoint(string endpointUrlName,
@@ -341,6 +374,9 @@ class AuditQuery_ThrowingAnExceptionOnKnownEndpointsCall : IAuditQuery
     {
         public SemanticVersion MinAuditCountsVersion => new(4, 29, 0);
 
+        public Task<List<Dictionary<string, string>>> GetAuditEnvironments(CancellationToken cancellationToken = default) =>
+            Task.FromResult<List<Dictionary<string, string>>>([]);
+
         public Func<RemoteInstanceInformation, bool> ValidRemoteInstances => r => true;
 
         public Task<IEnumerable<AuditCount>> GetAuditCountForEndpoint(string endpointUrlName,
@@ -420,6 +456,10 @@ public AuditQuery_WithTwoEndpointsAndSelectiveCounts(
         }
 
         public SemanticVersion MinAuditCountsVersion => new(4, 29, 0);
+
+        public Task<List<Dictionary<string, string>>> GetAuditEnvironments(CancellationToken cancellationToken = default) =>
+            Task.FromResult<List<Dictionary<string, string>>>([]);
+
         public Func<RemoteInstanceInformation, bool> ValidRemoteInstances => _ => true;
 
         public Task<IEnumerable<ServiceControlEndpoint>> GetKnownEndpoints(CancellationToken cancellationToken = default) =>
@@ -456,6 +496,12 @@ class AuditQuery_WithRemotes(List<RemoteInstanceInformation> remotes) : IAuditQu
         public Task KnownEndpointsRequested => knownEndpointsRequested.Task;
 
         public SemanticVersion MinAuditCountsVersion => new(4, 29, 0);
+
+        public List<Dictionary<string, string>> Environments { get; init; } = [];
+
+        public Task<List<Dictionary<string, string>>> GetAuditEnvironments(CancellationToken cancellationToken = default) =>
+            Task.FromResult(Environments);
+
         public Func<RemoteInstanceInformation, bool> ValidRemoteInstances => _ => true;
 
         public Task<List<RemoteInstanceInformation>> GetAuditRemotes(CancellationToken cancellationToken = default) =>

src/Particular.LicensingComponent.UnitTests/Infrastructure/FakeConfigurationApi.cs

@@ -10,6 +10,7 @@ class FakeConfigurationApi : IConfigurationApi
     {
         public Task<object> GetConfig(CancellationToken cancellationToken = default) => throw new NotImplementedException();
         public Task<RemoteConfiguration[]> GetRemoteConfigs(CancellationToken cancellationToken = default) => throw new NotImplementedException();
+        public Task<RemoteEnvironment[]> GetRemoteEnvironments(CancellationToken cancellationToken = default) => throw new NotImplementedException();
         public Task<RootUrls> GetUrls(string baseUrl, CancellationToken cancellationToken = default) => throw new NotImplementedException();
     }
 }

src/Particular.LicensingComponent.UnitTests/MonitoringService_Tests.cs

@@ -252,6 +252,12 @@ public Task<AuditServiceMetadata> GetAuditServiceMetadata(CancellationToken canc
         public Task SaveAuditServiceMetadata(AuditServiceMetadata auditServiceMetadata, CancellationToken cancellationToken = default) =>
             throw new NotSupportedException();
 
+        public Task<AuditEnvironmentMetadata> GetAuditEnvironmentMetadata(CancellationToken cancellationToken = default) =>
+            throw new NotSupportedException();
+
+        public Task SaveAuditEnvironmentMetadata(AuditEnvironmentMetadata auditEnvironmentMetadata, CancellationToken cancellationToken = default) =>
+            throw new NotSupportedException();
+
         public Task<List<string>> GetReportMasks(CancellationToken cancellationToken = default) =>
             throw new NotSupportedException();