🚀 Enterprise TIBCO BWCE Multi-Cluster GitOps Platform with Datadog Observability

An enterprise-grade, multi-cluster continuous integration and progressive delivery platform purpose-built for TIBCO BusinessWorks™ Container Edition (BWCE 2.9.2 / 2.10.0) microservices on Red Hat OpenShift 4.20+. It integrates Jenkins Configuration as Code (JCasC), Dynamic Git Parameter Plugin Dropdowns, Multi-Remote SCM, ArgoCD 3.5 Multi-Cluster GitOps, and Datadog Full-Stack Observability (APM, DogStatsD, CI Visibility, Logs, and Argo Rollouts Metric Analysis).

Important

⚠️ Architectural Blueprint & Sovereign Environment Notice

Non-Production Testing Status: This repository serves as an architectural reference blueprint, educational design pattern, and foundational boilerplate. It has not been tested or executed in a live production environment.
Sovereign & Air-Gapped Boilerplate: This project is engineered as a deterministic, version-controlled Infrastructure-as-Code (IaC) template that can be safely used as a standardized boilerplate in enterprise environments where Agentic AI / LLMs are prohibited due to data sovereignty, compliance regulations, intellectual property protection, or air-gapped security policies.
Enterprise Hardening: Platform engineers must review, adapt, and validate all cluster endpoints, TLS certificates, HashiCorp Vault configurations, Datadog API keys, and RBAC quotas prior to production adoption.
AI Generation Attribution: This architecture, Infrastructure-as-Code implementation, and documentation were generated using Gemini 3.7 Flash with Antigravity.
Origin & Real-World Heritage: This blueprint is based on nubenetes/jenkins-git-parameter and nubenetes/jenkins-git-parameter-global-vars, drawing directly from the author's personal hands-on experience designing and operating enterprise integration platforms with these exact requirements (engineered just before agentic AI became widespread).

Tip

🔗 Companion Repository (Single Source of Truth)

This orchestration platform works in tandem with its centralized configuration repository:
🌐 nubenetes/jenkins-git-parameter-bwce-global-vars
(Houses multi-cluster environment topologies, externalized .substvar BWCE profiles, Helm values, and Vault secret references)

📑 Table of Contents

Executive Summary & Architecture Overview
TIBCO BWCE Cloud-Native Best Practices
Datadog Full-Stack Observability Architecture
Multi-Repository Git Parameter CI/CD Patterns
- Pattern 1: Dual Git Parameter Dropdowns (Multi-Remote SCM)
- Pattern 2: Decoupled CI Build & Multi-Cluster CD Orchestrator
Enterprise Security & Supply Chain Integrity
Repository Structure
Quick Start & Deployment
Decommissioning & Reinstallation
References & Standards

Executive Summary & Architecture Overview

This platform delivers an automated, governed, multi-cluster CI/CD and GitOps delivery pipeline for enterprise TIBCO BWCE workloads across three OpenShift clusters (DEV, STAGING, PROD).

🗺️ Click to expand: End-to-End Multi-Cluster Platform Topology Diagram

flowchart TB
    subgraph DevSystems ["1. Developer & Portals"]
        Dev["Developer / Ops"]
        Backstage["Backstage IDP<br/>Service Catalog"]
        ITSM["Jira Service Mgmt<br/>ServiceNow CMDB"]
    end

    subgraph OCP_DEV ["Cluster 1: OCP DEV"]
        direction TB
        subgraph JenkinsPlatform ["Jenkins Control Plane"]
            Master["Jenkins Master<br/>2.492.2 LTS"]
            Seed["00-Seed-Job<br/>Orchestrator"]
            CIJob["01-CI-Build<br/>BWCE App Git"]
            CDJob["02-CD-Release<br/>Global Vars Git"]
        end

        subgraph Agents ["Agent Pods"]
            BwceAgent["bwce-builder<br/>EAR & Tests"]
            GitOpsAgent["argocd-gitops<br/>Skopeo & Cosign"]
        end

        subgraph OCPDevRegistry ["DEV Registry"]
            DevReg["Internal Registry<br/>dev-bwce"]
        end

        subgraph ArgoCDMaster ["ArgoCD 3.5"]
            ArgoServer["ArgoCD Server<br/>ApplicationSets"]
        end

        subgraph DatadogAgentStack ["Datadog Stack"]
            DDAgent["Datadog Agent<br/>APM:8126 | DSD:8125"]
            DDCluster["Cluster Agent<br/>Metrics Provider"]
        end

        DevApps["BWCE DEV Apps<br/>dev-bwce"]
    end

    subgraph OCP_STG ["Cluster 2: OCP STAGING"]
        StgReg["STG Registry"]
        StgApps["BWCE STAGING Apps<br/>staging-bwce"]
    end

    subgraph OCP_PRD ["Cluster 3: OCP PROD"]
        PrdReg["PROD Registry"]
        PrdApps["BWCE PROD Apps<br/>prod-bwce"]
        RolloutCtrl["Argo Rollouts<br/>Canary Controller"]
    end

    subgraph DatadogCloud ["Datadog Cloud"]
        DDCIVis["CI Visibility<br/>Pipeline Spans"]
        DDAPM["APM Tracing<br/>Live Profiling"]
        DDDash["Dashboards<br/>& Monitors"]
    end

    Dev -->|"Selects Branch"| CIJob
    Dev -->|"Selects Config"| CDJob
    Backstage -->|"REST API"| CDJob
    ITSM -->|"Webhook"| CDJob

    CIJob -->|"Launches"| BwceAgent
    BwceAgent -->|"Pushes Image"| DevReg
    CIJob -->|"Triggers CD"| CDJob

    CDJob -->|"Launches"| GitOpsAgent
    GitOpsAgent -->|"Skopeo Copy"| StgReg
    GitOpsAgent -->|"Skopeo Copy"| PrdReg
    GitOpsAgent -->|"Sync & Health"| ArgoServer

    ArgoServer -->|"GitOps"| DevApps
    ArgoServer -->|"GitOps"| StgApps
    ArgoServer -->|"Sync Waves"| PrdApps

    Master -.->|"CI Spans"| DDAgent
    DevApps -.->|"APM & Metrics"| DDAgent
    StgApps -.->|"APM & Metrics"| DDAgent
    PrdApps -.->|"APM & Metrics"| DDAgent
    DDAgent -->|"HTTPS"| DatadogCloud
    RolloutCtrl -.->|"Query Metrics"| DatadogCloud

TIBCO BWCE Cloud-Native Best Practices

1. 12-Factor Profile Externalization via `.substvar`

In enterprise TIBCO BWCE implementations, building environment-specific EAR files violates 12-Factor principles. Instead:

Build Once: A single immutable .ear artifact (tibco-bwce-order-service_2.1.0.ear) is compiled during CI and packaged into the container image (FROM tibco/bwce:2.9.2).
Externalize Profiles: Configuration tokens (DEV.substvar, STAGING.substvar, PROD.substvar) reside in the Single Source of Truth configuration repository: jenkins-git-parameter-bwce-global-vars.
Runtime Injection: The runtime profile is selected via BW_PROFILE environment variable or ConfigMap volume mount at container startup.

⚙️ Click to expand: TIBCO BWCE Profile Externalization Flow Diagram

flowchart LR
    subgraph BuildTime ["1. CI Packaging"]
        Source[".bwp Sources"] --> Maven["Maven Plugin"]
        Maven --> EAR["Single EAR"]
        EAR --> BaseImage["tibco/bwce:2.9.2"]
        BaseImage --> Image["Container Image<br/>(Immutable)"]
    end

    subgraph GitOpsSSOT ["2. Global Vars SSOT"]
        DevVars["DEV.substvar<br/>(dev.yaml)"]
        StgVars["STAGING.substvar<br/>(staging.yaml)"]
        PrdVars["PROD.substvar<br/>(prod.yaml)"]
    end

    subgraph RuntimeDeploy ["3. OpenShift Runtime"]
        Image --> OCPDev["DEV Pod<br/>Profile: DEV"]
        Image --> OCPStg["STG Pod<br/>Profile: STG"]
        Image --> OCPPrd["PROD Pod<br/>Profile: PROD"]

        DevVars -.->|"ConfigMap"| OCPDev
        StgVars -.->|"ConfigMap"| OCPStg
        PrdVars -.->|"ConfigMap"| OCPPrd
    end

2. Engine Sizing & Performance Tuning

The BusinessWorks Container Edition runtime engine requires deliberate sizing parameters adapted to container cgroups:

Environment	Replicas	CPU Request (Burstable)	Memory Req / Limit	`BW_ENGINE_THREADCOUNT`	`BW_STEP_FLOWLIMIT`	`BW_LOGLEVEL`
DEV	2	`250m` (No CPU Limit)	512Mi / 1024Mi	`16`	`50`	`DEBUG`
STAGING	3	`500m` (No CPU Limit)	768Mi / 1536Mi	`32`	`100`	`INFO`
PROD	6	`1000m` (No CPU Limit)	1024Mi / 2048Mi	`64`	`250`	`WARN`

Tip

💡 Why CPU Limits are Omitted at the Pod Level (Preventing Linux CFS Bandwidth Throttling)

The CFS Quota Problem in Multi-Threaded Runtimes: Setting resources.limits.cpu instructs the Linux kernel Completely Fair Scheduler (CFS) to enforce hard bandwidth quotas (cfs_quota_us over 100ms periods). Because TIBCO BWCE and JVM garbage collection are highly concurrent (e.g. 16–64 engine threads), parallel thread execution can consume the quota in milliseconds, causing the container to be hard-throttled for the remainder of the period. This induces artificial p99 latency spikes and timeouts even when the OpenShift worker node has surplus CPU capacity.
Namespace-Level ResourceQuota Governance: Instead of per-pod CPU limits, capacity governance and noisy-neighbor protection are enforced at the OpenShift Project/Namespace boundary using security/openshift-namespace-resource-quota.yaml (ResourceQuota for aggregate requests.cpu, requests.memory, and limits.memory).
Deterministic Memory Limits: Unlike CPU (which is compressible), Memory is strictly capped with limits.memory to prevent out-of-memory cascading to neighbor pods.

BW_ENGINE_THREADCOUNT: Specifies the number of engine worker threads executing process instances concurrently.
BW_STEP_FLOWLIMIT: Prevents unbounded memory growth during traffic bursts by capping active in-memory process transitions.
BW_CONTAINER_SHUTDOWN_TIMEOUT_SECONDS: Set to 30s (DEV/STAGING) and 45s (PROD) for clean process draining upon SIGTERM.
JVM Options: -XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0 -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent ensuring GC predictability inside Linux container cgroups.

3. OpenShift Hardening & Health Probing

Security Context Constraints: Complies with OpenShift restricted-v2 SCC (non-root uid: 1001, dropped capabilities ALL, read-only root filesystems).
Probes:
- Liveness Probe: HTTP GET http://:8090/health (Initial delay: 45s, Period: 15s, Timeout: 5s).
- Readiness Probe: HTTP GET http://:8090/health (Initial delay: 20s, Period: 10s, Timeout: 3s).

Datadog Full-Stack Observability Architecture

Replaces OpenTelemetry and Grafana with an enterprise Datadog observability stack.

📊 Click to expand: Datadog Full-Stack Observability Architecture Diagram

flowchart TB
    subgraph Sources ["1. OpenShift Workloads"]
        direction TB
        Jenkins["Jenkins Master<br/>(Datadog Plugin)"]
        BWCEApp["TIBCO BWCE App<br/>(dd-java-agent)"]
        RolloutCtrl["Argo Rollouts<br/>(Canary Engine)"]
    end

    subgraph DaemonSetLayer ["2. Node Agent Layer"]
        direction TB
        DDAgent["Datadog Agent<br/>(DaemonSet)<br/>──────────────<br/>APM Port: 8126<br/>StatsD Port: 8125<br/>Metrics: 8090<br/>Logs: JSON"]
    end

    subgraph CloudPlatform ["3. Datadog Cloud Platform"]
        direction TB
        CIVisibility["CI Visibility<br/>Pipeline Spans<br/>Queue Metrics"]
        APMTrace["APM Tracing<br/>HTTP & JMS<br/>Distributed Spans"]
        OpenMetrics["Engine Metrics<br/>Thread Pools<br/>JVM GC Pauses"]
        Dashboards["Dashboards<br/>• Jenkins CI<br/>• BWCE Engine<br/>• ArgoCD Sync"]
        Monitors["Alert Monitors<br/>• 5xx > 1.0%<br/>• p99 > 500ms<br/>• Queue Spikes"]
    end

    Jenkins -->|"CI Spans"| DDAgent
    BWCEApp -->|"APM: 8126"| DDAgent
    BWCEApp -->|"Metrics: 8090"| DDAgent
    BWCEApp -->|"Logs"| DDAgent

    DDAgent -->|"HTTPS"| CIVisibility
    DDAgent -->|"HTTPS"| APMTrace
    DDAgent -->|"HTTPS"| OpenMetrics

    CIVisibility --> Dashboards
    APMTrace --> Dashboards
    OpenMetrics --> Dashboards

    CIVisibility --> Monitors
    APMTrace --> Monitors
    OpenMetrics --> Monitors

    RolloutCtrl -.->|"Query SLA"| APMTrace

1. Jenkins CI/CD Visibility Plugin

The official Datadog Jenkins Plugin (datadog:5.9.0) is configured via JCasC (jcasc/jenkins-jcasc.yaml):

Automatically correlates pipeline builds, stage execution times, agent queue wait times, and test results.
Injects Datadog Trace IDs (x-datadog-trace-id) across pipeline steps.
Global tags: env:dev, team:integration-platform, tech:tibco-bwce, cluster:ocp-dev.

2. TIBCO BWCE Java APM Tracer & DogStatsD

Java Tracer: The Datadog Java APM agent (dd-java-agent.jar) is injected into the container image and attached to the BWCE engine via:

BW_JAVA_OPTS="-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0 -XX:+UseG1GC -javaagent:/opt/datadog/dd-java-agent.jar -Ddd.service=tibco-bwce-order-service -Ddd.logs.injection=true -Ddd.profiling.enabled=true"

OpenMetrics Autodiscovery: Pod annotations configure the Datadog Agent to scrape BWCE management metrics from port 8090:

annotations:
  ad.datadoghq.com/tibco-bwce-order-service.logs: '[{"source": "tibco-bwce", "service": "tibco-bwce-order-service"}]'
  ad.datadoghq.com/tibco-bwce-order-service.check_names: '["openmetrics"]'
  ad.datadoghq.com/tibco-bwce-order-service.init_configs: '[{}]'
  ad.datadoghq.com/tibco-bwce-order-service.instances: |
    [{"openmetrics_endpoint": "http://%%host%%:8090/metrics", "namespace": "tibco_bwce", "metrics": [".*"]}]

3. Datadog Live Dashboards & Monitors

Preconfigured Datadog JSON dashboards and alert monitors located in observability/:

Jenkins CI Visibility: observability/dashboards/datadog-jenkins-ci-visibility.json
BWCE Engine Performance: observability/dashboards/datadog-bwce-runtime-performance.json
ArgoCD GitOps Sync: observability/dashboards/datadog-argocd-gitops-sync.json
Automated Monitors: observability/monitors/datadog-monitors-bwce.yaml (Monitors HTTP 5xx error rate > 1%, p99 latency > 500ms, and build agent queue bottlenecks).

4. Progressive Delivery with Argo Rollouts & Datadog Metrics SLA

During canary deployments, Argo Rollouts queries Datadog metrics directly using sample-apps/tibco-bwce-order-service/rollout/analysis-template-datadog.yaml:

Error Rate SLA: Evaluates that 5xx errors $\le 0.1%$ via sum:trace.tibco_bwce_order_service.request.errors{env:prod}.as_count() / sum:trace.tibco_bwce_order_service.request.hits{env:prod}.as_count().
Latency SLA: Evaluates that p99 latency $\le 250 ext{ms}$ via p99:trace.tibco_bwce_order_service.request.duration{env:prod} before traffic step promotion.

🐣 Click to expand: Argo Rollouts Canary Traffic Splitting & Datadog Metric Analysis Diagram

flowchart LR
    Ingress["OpenShift Route<br/>Ingress Traffic"]
    Canary["Canary Pods<br/>10% -> 25% -> 50%<br/>v2.1.0 (New EAR)"]
    Stable["Stable Pods<br/>90% -> 75% -> 50%<br/>v2.0.8 (Stable EAR)"]
    DatadogMetrics["Datadog APM API<br/>5xx <= 0.1%<br/>p99 <= 250ms"]
    RolloutCtrl["Argo Rollouts<br/>Controller"]

    Ingress -->|"Traffic Split"| Canary
    Ingress -->|"Traffic Split"| Stable
    Canary -.->|"APM Metrics"| DatadogMetrics
    DatadogMetrics -.->|"SLA Evaluation"| RolloutCtrl
    RolloutCtrl -->|"Pass: Promote<br/>Fail: Rollback"| Ingress

Multi-Repository Git Parameter CI/CD Patterns

The Core Problem: Why Standard Pipelines Fail with Multiple Git Repositories

In cloud-native architectures, applications separate source code (tibco-bwce-order-service) from centralized configuration and Helm values (jenkins-git-parameter-bwce-global-vars).

When engineering teams attempt to introduce multiple gitParameter dropdowns into a single Jenkins Pipeline, they encounter a hard blocker where GitSCM appears to only support one repository, or secondary dropdowns fail.

⚠️

Click to expand: Jenkins SCM Lifecycle & Pre-Execution Blindspot Diagram

flowchart TB
    subgraph UI_Phase ["1. Pre-Execution (Master)"]
        direction TB
        User["User opens<br/>Build UI Form"]
        Master["Master reads<br/>Job XML SCM"]
        GitParam["git-parameter<br/>queries remote refs"]
        Dropdown["Renders Dropdown<br/>for Primary Repo"]

        User --> Master --> GitParam --> Dropdown
    end

    subgraph Runtime_Phase ["2. Runtime Phase (Agent)"]
        direction TB
        AllocAgent["Ephemeral Agent<br/>Pod Allocated"]
        RunStage["Pipeline Stage:<br/>checkout(repo-2)"]

        AllocAgent --> RunStage
    end

    Gap["SCM Blindspot:<br/>Dynamic checkouts<br/>run during build and are<br/>invisible at UI render"]

    Dropdown -.-> Gap
    Gap -.-> RunStage

Master Seed Job & JCasC Automation Architecture

The entire CI/CD platform is self-bootstrapping and self-healing using Jenkins Configuration as Code (JCasC) combined with the Job DSL Plugin.

⚙️ Click to expand: Seed Job & Job DSL Pipeline Provisioning Flow Diagram

flowchart TB
    subgraph JCasC_Phase ["1. Controller Bootstrap (JCasC)"]
        direction TB
        JCasC["jenkins-jcasc.yaml<br/>(Master Config)"] -->|"Registers"| SeedJob["00-Seed-Job<br/>(Platform Seed)"]
    end

    subgraph SCM_Phase ["2. SCM Synchronization"]
        direction TB
        GitRepo["Git Repository<br/>(Platform Code)"]
        SeedJob -->|"Polls H/15 * * * *"| GitRepo
        SeedJob -->|"Evaluates"| JobDSLFiles["Target Scripts<br/>(jobdsl/*.groovy)"]
    end

    subgraph Materialization ["3. Pipeline Materialization"]
        direction TB
        JobDSLFiles -->|"seed-job.groovy"| Folders["Creates Folders:<br/>• CI Pipelines<br/>• CD Orchestrators"]
        JobDSLFiles -->|"pipelines-ci.groovy"| CIPipelines["Generates CI:<br/>• order-service-ci<br/>• customer-api-ci"]
        JobDSLFiles -->|"pipelines-cd.groovy"| CDPipelines["Generates CD:<br/>• release-orchestrator<br/>• hotfix-deploy"]
    end

    Folders -.->|"Parent"| CIPipelines
    Folders -.->|"Parent"| CDPipelines

1. Why `jobdsl/seed-job.groovy` Only Declares Folders (Separation of Concerns)

In enterprise Jenkins-as-Code implementations, we deliberately apply the Separation of Concerns (SoC) principle:

Taxonomy & Hierarchy Initialization: jobdsl/seed-job.groovy acts as the root taxonomy initializer. It creates and configures the top-level folder containers (01-CI-Build-Pipelines and 02-CD-Release-Orchestrators) along with their descriptions and metadata.
Decoupled Pipeline Generation Logic: The application CI pipelines (jobdsl/pipelines-ci.groovy) and CD release orchestrators (jobdsl/pipelines-cd.groovy) contain iterative business logic (looping over the apps inventory, configuring multi-remote refspecs, Git Parameter dropdowns, log rotation, and Datadog CI visibility). Keeping folder declarations separate prevents monolithic scripts and enables modular additions of new business domains.

2. Managing Folders in Job DSL vs. Static JCasC

While folders can technically be defined directly in jenkins-jcasc.yaml, managing them through Job DSL is the recommended SRE practice for multi-tenant platforms:

Architectural Metric	Static JCasC Management (`jenkins-jcasc.yaml`)	Dynamic Job DSL Management (`jobdsl/seed-job.groovy`)
Creation Lifecycle	Executed strictly upon Jenkins Controller startup or manual JCasC reload.	Evaluated continuously on every Seed Job execution (automated SCM polling or manual trigger).
Orphaned Item Pruning	Does not automatically purge folders or child jobs if deleted from configuration.	Automated Pruning: `removedJobAction('DELETE')` and `removedViewAction('DELETE')` cleanly destroy removed jobs/folders.
Configuration Ownership	Belongs to Controller System Configuration (Platform Admin scope).	Belongs to Version-Controlled GitOps Code (Developer / Application Domain scope).
Zero-Downtime Evolution	Renaming or restructuring folders requires reloading master JCasC.	Restructuring folders is applied dynamically via Git commit without touching master configuration.

3. Step-by-Step Technical Chain & Execution Flow

JCasC Master Bootstrap:
When the Jenkins container starts, jcasc/jenkins-jcasc.yaml configures the master Seed Job:

jobs:
  - script: |
      job('00-Seed-Job-BWCE-Platform-Orchestrator') {
        description('Master Seed Job provisioning TIBCO BWCE Pipelines as Code.')
        scm {
          git {
            remote { url('https://github.com/nubenetes/jenkins-git-parameter-bwce.git') }
            branch('*/main')
          }
        }
        steps {
          jobDsl {
            targets('jobdsl/seed-job.groovy\njobdsl/pipelines-ci.groovy\njobdsl/pipelines-cd.groovy')
            removedJobAction('DELETE')
            removedViewAction('DELETE')
            lookupStrategy('JENKINS_ROOT')
          }
        }
        triggers { scm('H/15 * * * *') }
      }

lookupStrategy('JENKINS_ROOT') Context Resolution:
By setting lookupStrategy('JENKINS_ROOT'), Job DSL resolves all relative job paths from the root of Jenkins. This allows pipelines-ci.groovy to target "01-CI-Build-Pipelines/${app.name}-ci-build" and nest jobs into the folders previously created by seed-job.groovy.
Automated Drift Correction via SCM Polling:
The trigger scm('H/15 * * * *') polls this repository every 15 minutes. When an engineer adds a new microservice to the def apps = [...] inventory in pipelines-ci.groovy and pushes to main, the Seed Job automatically generates the new CI/CD pipelines in Jenkins with zero manual UI configuration.

Pattern 1: Dual Git Parameter Dropdowns (Multi-Remote SCM)

Target: Developer Sandboxes & Feature Preview Environments

Dropdown 1: Queries application repository (APP_GIT_REVISION).
Dropdown 2: Queries global configuration repository (GLOBAL_VARS_REVISION).
Multi-remote refspecs configured via Job DSL in jobdsl/pipelines-ci.groovy.

Pattern 2: Decoupled CI Build & Multi-Cluster CD Orchestrator

Target: Enterprise Production Release Pipeline (Recommended)

🧪 Click to expand: Inner-Loop (Pattern 1) vs Outer-Loop (Pattern 2) Coexistence Diagram

flowchart TD
    subgraph InnerLoop ["Pattern 1: Dual-Dropdown"]
        Dev1["Feature Dev"]
        DualJob["Dual-Dropdown Job<br/>App & Config Refs"]
        DevSandbox["Dev Sandbox<br/>dev-bwce"]

        Dev1 -->|"Trigger"| DualJob
        DualJob -->|"Deploy"| DevSandbox
    end

    subgraph OuterLoop ["Pattern 2: Decoupled CI/CD"]
        Dev2["Release Lead"]
        CIJob["01-CI-Build<br/>App Repo"]
        Registry["DEV Registry"]
        CDOrchestrator["02-CD-Orchestrator<br/>Global Vars SSOT"]
        ArgoEngine["ArgoCD Engine"]
        Clusters["Promotion Chain<br/>DEV->STG->PROD"]

        Dev2 -->|"Trigger CI"| CIJob
        CIJob -->|"Push Image"| Registry
        CIJob -->|"Dispatch"| CDOrchestrator
        CDOrchestrator -->|"Sync"| ArgoEngine
        ArgoEngine -->|"Promote"| Clusters
    end

🔄 Click to expand: End-to-End Hand-off Sequence Diagram

sequenceDiagram
    autonumber
    actor Dev as Developer / Operator
    participant CI as 01-CI-Build-Pipeline
    participant DevReg as OCP DEV Registry
    participant CD as 02-CD-Release-Orchestrator
    participant StgReg as OCP Staging Registry
    participant PrdReg as OCP Prod Registry
    participant Argo as ArgoCD 3.5 Control Plane
    participant DDog as Datadog Cloud APM

    Dev->>CI: Trigger Build (Select App Branch)
    activate CI
    CI->>CI: Execute BWUnit Mock Tests
    CI->>CI: Package EAR and Build Container
    CI->>CI: Scan with Trivy and Sign with Cosign
    CI->>DevReg: Push image (e.g. 2.1.0-42)
    CI->>DDog: Emit CI Span (ci.bwce.pipeline_completed)
    CI->>CD: Dispatch downstream CD Orchestrator
    deactivate CI

    activate CD
    CD->>Argo: Sync DEV Application (Wave 0 -> Wave 1)
    Argo-->>CD: DEV Health Check Passed (HTTP 200)
    CD->>StgReg: Skopeo copy (DEV -> STAGING)
    CD->>Argo: Sync STAGING Application
    Argo-->>CD: STAGING Integration Suite Passed (100%)
    
    CD->>Dev: Production Approval Gate (Interactive Input)
    Dev-->>CD: Approved by Release Engineer

    CD->>PrdReg: Skopeo copy (STAGING -> PROD)
    CD->>Argo: Sync PROD Application (Argo Rollouts Canary)
    Argo->>DDog: Query Datadog Metric Analysis (p99 < 250ms, 5xx < 0.1%)
    DDog-->>Argo: Metrics Passed (100% Promoted)
    CD->>DDog: Emit Release Metric (cd.bwce.release_completed)
    deactivate CD

Enterprise Security & Supply Chain Integrity

1. Cosign Image Signing & SLSA Level 3 Attestation

All container images layered on the BWCE base runtime are cryptographically signed using Sigstore Cosign during CI execution, guaranteeing supply chain integrity before promotion into production.

🔏 Click to expand: Cosign Image Signing & SLSA Level 3 Attestation Flow Diagram

flowchart LR
    CI["CI Build<br/>(Maven BWCE)"]
    Trivy["Trivy Scan<br/>& Syft SBOM"]
    Cosign["Cosign Sign<br/>& Attestation"]
    Registry["OCP DEV<br/>Registry"]
    Policy["OpenShift Policy<br/>Block Unsigned"]

    CI --> Trivy
    Trivy --> Cosign
    Cosign --> Registry
    Registry -.->|"Enforced"| Policy

2. Zero-Trust Secrets with External Secrets Operator & Vault

Plaintext credentials (database passwords, JMS connection tokens, Datadog API keys) are never stored in Git. Instead, the External Secrets Operator (ESO) synchronizes encrypted secrets from HashiCorp Vault directly into OpenShift namespaces.

🔐 Click to expand: External Secrets Operator & Vault Synchronization Flow Diagram

flowchart LR
    Vault["HashiCorp Vault<br/>Secrets & Keys"]
    GlobalVars["Global Vars Repo<br/>ExternalSecret CRD"]
    ESO["External Secrets<br/>Operator (ESO)"]
    K8sSecret["Kubernetes Secret<br/>(In-Memory)"]
    BWCEPod["TIBCO BWCE Pod<br/>(Workload)"]

    Vault -->|"Token Fetch"| ESO
    GlobalVars -->|"Ref Spec"| ESO
    ESO -->|"Sync/Rotate"| K8sSecret
    K8sSecret -->|"Env Mount"| BWCEPod

3. Ephemeral PR Preview Environments via ArgoCD ApplicationSets

Pull Requests targeting application repositories automatically trigger the ArgoCD PR Generator ApplicationSet, creating ephemeral preview namespaces for end-to-end integration testing before merging.

⚡ Click to expand: Ephemeral PR Preview Provisioning Flow Diagram

flowchart LR
    Dev["Developer"]
    PR["GitHub PR #42"]
    AppSet["ArgoCD AppSet<br/>(PR Generator)"]
    PreviewNS["Preview Env<br/>pr-42-preview"]
    Datadog["Datadog<br/>PR Dashboard"]

    Dev -->|"Opens"| PR
    PR -->|"Discovers"| AppSet
    AppSet -->|"Deploys"| PreviewNS
    PreviewNS -->|"DD_ENV"| Datadog

📂 Repository Structure

.
├── config/
│   ├── environments.env              # Pinned versions, domain names, Datadog credentials
│   └── clusters.yaml                 # ConfigMap defining OCP DEV, STAGING, PROD topologies
├── helm/
│   ├── jenkins/                      # Official Jenkins chart values with Datadog plugin
│   ├── observability/                # Datadog Agent Helm values (OpenShift hardened)
│   └── argocd/                       # ArgoCD 3.5 Helm values
├── jcasc/
│   ├── jenkins-jcasc.yaml            # JCasC security matrix, Datadog CI visibility, seed job
│   ├── pod-templates.yaml            # Ephemeral Kubernetes agent pod templates (bwce-builder)
│   └── github-app-credentials.yaml   # GitHub App credential rotation
├── jobdsl/
│   ├── seed-job.groovy               # Master Seed Job definition
│   ├── pipelines-ci.groovy           # Pattern 1 and Pattern 2 CI pipelines
│   └── pipelines-cd.groovy           # CD Release Orchestrator & Hotfix pipelines
├── jenkinsfiles/
│   ├── ci/
│   │   ├── Jenkinsfile.app-bwce
│   │   └── Jenkinsfile.app-bwce-dual-dropdown
│   └── cd/
│       ├── Jenkinsfile.release-orchestrator
│       └── Jenkinsfile.hotfix-deploy
├── shared-library/
│   ├── vars/
│   │   ├── datadogLogEvent.groovy    # Emits Datadog CI spans & DogStatsD metrics
│   │   ├── bwceEarBuild.groovy       # Compiles and packages BWCE EAR
│   │   ├── bwceProfileOverride.groovy# Injects .substvar profile tokens
│   │   ├── argoAppSync.groovy        # ArgoCD 3.5 synchronization
│   │   ├── skopeoPromote.groovy      # Cross-cluster image promotion
│   │   ├── cosignSign.groovy         # Cryptographic image signing (SLSA Level 3)
│   │   ├── sbomGenerate.groovy       # CycloneDX SBOM generation
│   │   └── gitopsCommit.groovy       # Updates Global Vars SSOT repository
│   └── src/com/nubenetes/gitops/
├── sample-apps/
│   ├── tibco-bwce-order-service/     # Complete TIBCO BWCE project + Dockerfile + K8s + Rollout
│   └── tibco-bwce-customer-api/
├── observability/
│   ├── dashboards/                   # Datadog dashboards (Jenkins, BWCE Engine, ArgoCD)
│   └── monitors/                     # Datadog automated alert monitors
├── argocd-apps/                      # ArgoCD Root App-of-Apps & ApplicationSets
├── security/                         # Image signature policy & External Secrets Operator
├── scripts/                          # Automation & setup scripts
├── deploy.sh                         # 1-Click Platform Deployment Script
├── destroy.sh                        # Clean Decommission Script
├── reinstall.sh                      # Full Wipe & Reinstall Script
└── Makefile

🚀 Quick Start & Deployment

1. Prerequisites

Red Hat OpenShift 4.20+ cluster (or Kubernetes 1.31+)
kubectl, oc, and helm v3 CLI tools installed
Datadog Account & API/App Keys

2. Configure Credentials

Set your Datadog credentials in config/environments.env or export environment variables:

export DATADOG_API_KEY="your-datadog-api-key"
export DATADOG_APP_KEY="your-datadog-app-key"

3. Deploy Platform

./deploy.sh
# or
make deploy

4. Access Platform Services

Jenkins Controller: https://jenkins-jenkins.apps.ocp-dev.nubenetes.internal (Default login: admin / admin123!)
ArgoCD UI: https://argocd-server.apps.ocp-dev.nubenetes.internal
Datadog Dashboard: https://app.datadoghq.eu

Decommissioning & Reinstallation

Clean Decommission

./destroy.sh
# or
make destroy

Full Reinstallation

./reinstall.sh
# or
make reinstall

References & Standards

TIBCO BWCE Documentation: TIBCO BusinessWorks™ Container Edition Official Docs
Datadog Jenkins Integration: Datadog CI Visibility & Jenkins Plugin
Datadog Java Tracer APM: Datadog Java Agent Documentation
ArgoCD 3.5 & ApplicationSets: ArgoCD Documentation
Sigstore Cosign: Cosign Container Signing Documentation
External Secrets Operator: External Secrets Operator Docs
Jenkins Job DSL & Git Parameter: Jenkins Job DSL API

🔗 Related Repositories

Global Variables SSOT: jenkins-git-parameter-bwce-global-vars
Base Generic CI/CD Blueprint: nubenetes/jenkins-git-parameter
Base Generic Global Variables SSOT: nubenetes/jenkins-git-parameter-global-vars

Name		Name	Last commit message	Last commit date
argocd-apps		argocd-apps
config		config
helm		helm
jcasc		jcasc
jenkinsfiles		jenkinsfiles
jobdsl		jobdsl
observability		observability
sample-apps		sample-apps
scripts		scripts
security		security
shared-library		shared-library
Makefile		Makefile
README.md		README.md
deploy.sh		deploy.sh
destroy.sh		destroy.sh
reinstall.sh		reinstall.sh

Uh oh!

Latest commit

History

Folders and files

Repository files navigation