feat: system integration rev1 (Vault cert manager + Redis streams + observability)

[miladhzzzz]

Description

This PR represents a major integration milestone (system-integration-rev1) that hardens the core control plane with production-grade secret management, event/metrics streaming, and improved observability across Persys Cloud components.

Key Changes

• Global pkg/certmanager + Vault Manager Service:

  • Automatic certificate issuance, rotation, and management via Vault PKI.
  • Vault Manager gRPC API for credential retrieval and AppRole handling.
  • Removed manual env-based secret injection in favor of dynamic Vault-backed flow with retries and fallbacks.

• Redis Streams Integration:

  • Added dedicated Redis stream for per-workload usage metrics.
  • Enhanced event publishing (node heartbeats, workload lifecycle events) with Node ID identification.
  • Enables downstream services (monitoring, billing, intelligence) to consume real-time data.

• Node & Workload Improvements:

  • Added Node ID to heartbeats and workload ownership tracking.
  • Fixed/added created_at fields in protobuf schemas and models.

• Observability & Debugging:

  • Added pprof endpoints across scheduler, gateway, intelligence, and related services.
  • Updated monitoring configurations and separated Persys monitoring stack.

• Dependencies & Housekeeping:

  • Updated Go modules, submodules, and SDK.
  • Various fixes for config, Docker deployments, and proto fields.

Why

• Establishes secure, automated identity management (mTLS everywhere with rotation).
• Creates a robust event/metrics backbone for the platform.
• Improves debuggability and operational visibility.
• Prepares the system for production workloads and multi-service integrations.

Testing

• Verified Vault credential flow and certificate rotation.
• Tested Redis stream publishing/consumption for usage metrics and events.
• Validated node registration, heartbeat, and workload lifecycle with new fields.
• Local Docker Compose stack passes full integration smoke tests.
• Backward compatibility maintained for existing deployments.

---

Related:
Follows compute-agent per-workload metrics work.
Part of the broader system hardening and production readiness phase for Persys Cloud.

Next steps: Full e2e testing, documentation updates, and potential Vault + Redis HA configuration.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.