Skip to content

production-engineering-scorecard.md

Latest commit

 

History

History
139 lines (101 loc) · 8.06 KB

production-engineering-scorecard.md

File metadata and controls

139 lines (101 loc) · 8.06 KB
Raw

Production Engineering Scorecard

This scorecard gives reviewers a fast, evidence-backed view of DevHire Cloud as a production engineering portfolio. Scores are intentionally conservative: runtime and owner-permission gaps stay visible instead of being hidden behind aspirational language.

Summary

Category Score Evidence
Architecture and service boundaries 9/10 Multi-module Spring Boot services, service-owned databases, Flyway migrations, gateway routing, no shared JPA entities, architecture tests
Security and identity 8/10 JWT access tokens, refresh rotation, logout blacklist, BCrypt, role checks, security headers, Gitleaks, blocking Trivy image scans with throttled/cache-backed image builds, CodeQL
Reliability and event delivery 8/10 Kafka, transactional outbox, retry/dead-letter states, idempotent notification/audit consumers, chaos smoke scripts
Observability and SLOs 9/10 Actuator, Prometheus rules, Grafana SLO/domain dashboards, Loki, Tempo, OpenTelemetry, runtime metrics smoke, seeded funnel metrics
CI/CD and release governance 8/10 Maven verify, frontend build, ratcheted coverage gate, Docker matrix, docs/security/terraform workflows, release notes, release evidence, v0.6.0 public release
Cloud readiness 9/10 Docker Compose, Kubernetes manifests without latest, ai-service raw K8s coverage, Helm chart with immutable defaults, Argo CD samples, AWS Terraform blueprint, External Secrets wiring, race-safe Terraform validation, cloud policy audit
Runtime reviewer proof 9/10 Self-starting frontend E2E smoke, portfolio verification scripts, curated demo evidence pack, API smoke, AI eval, Mailpit smoke, OpenAPI verify, performance and chaos smoke wrappers
AI portfolio layer 8/10 Claude Haiku assistant, RAG citations, fallback mode, tool traces, AI safety docs, eval dataset
Public GitHub facade 9.5/10 About/Homepage/Topics and master branch protection are applied; facade assertion handles public-limited protection details correctly, and settings-as-code disables admin bypass

Overall portfolio posture: 9.2/10 production engineering evidence, with the main remaining gap being future real-cloud deployment evidence in an AWS account.

Current public release: v0.6.0. Current development cycle: 0.6.0-SNAPSHOT release cut. The v1.0.0 files are roadmap and acceptance evidence only.

Architecture

DevHire Cloud demonstrates a real microservice decomposition instead of a folder-only split. The project has independent modules for gateway, auth, users, companies, jobs, applications, notifications, audit, AI, and common contracts. Each business service owns its database and Flyway migrations. The common library contains DTOs, event contracts, errors, and security helpers, but not business entities.

Evidence:

Security

Security is present at the API edge, service layer, CI pipeline, and documentation layer. Secrets are environment driven and ignored by repository hygiene checks. The project uses BCrypt, JWT validation, refresh token rotation, role-based authorization, gateway security headers, dependency review, secret scanning, CodeQL, SBOM generation, and Trivy scanning.

Evidence:

Reliability

The event path uses Kafka and transactional outbox publishing so important business events are stored before they are published. Consumers track event ids for idempotency. Runtime scripts cover degraded OpenSearch, Kafka, AI provider, and SMTP scenarios. Backup and disaster recovery runbooks are present for PostgreSQL-owned databases.

Evidence:

Observability

Services expose actuator health and metrics. The Docker stack includes Prometheus, Grafana, Loki, Tempo, and OpenTelemetry Collector. Alerts and dashboards focus on availability, latency, error rate, JVM pressure, DB pool pressure, outbox health, recruitment funnel state, notification delivery, audit ingestion, search fallback, and AI assistant provider behavior.

Evidence:

  • SLO documentation
  • infra/prometheus/rules/devhire-slo.yml
  • infra/grafana/dashboards/devhire-slo-overview.json
  • infra/grafana/dashboards/devhire-recruitment-funnel.json
  • infra/grafana/dashboards/devhire-event-reliability.json
  • infra/grafana/dashboards/devhire-search-and-ai.json
  • Runtime acceptance matrix
  • scripts/runtime-observability-smoke.ps1

Delivery

The release path includes Maven verification, frontend typecheck/build, Docker image matrix builds, security workflows, docs quality gates, Terraform validation, release notes, and evidence files. Dockerfiles now include OCI labels for source, revision, version, created timestamp, title, description, and license.

Evidence:

Cloud Readiness

Cloud deployment is intentionally blueprint-safe. The repo contains Docker Compose for local runtime, Kubernetes raw manifests, Helm values for local/staging/prod/AWS, Argo CD samples, External Secrets wiring, and an AWS Terraform blueprint for EKS, RDS, Redis, MSK, OpenSearch, ECR, and Secrets Manager. The current master branch includes race-safe Terraform validation, stricter cloud policy auditing, explicit kubeconform CRD skip handling, AWS account bootstrap docs, and an apply runbook. No cloud apply or secret commit is required.

Evidence:

Runtime Proof

Runtime proof is script-first so reviewers can run evidence without reading every service. Pull requests get a lightweight frontend preview smoke that does not need Docker, while full runtime proof depends on Docker being available locally; when Docker is unavailable, the repo records the blocker honestly.

Evidence:

  • scripts/portfolio-verify.ps1
  • scripts/portfolio-demo-evidence.ps1
  • scripts/docs-parity.ps1
  • cd frontend && npm run e2e:all
  • scripts/public-portfolio-audit.ps1
  • scripts/api-smoke.ps1
  • scripts/ai-eval.ps1
  • scripts/email-smoke.ps1
  • scripts/openapi-verify.ps1
  • scripts/perf-suite.ps1
  • scripts/chaos-smoke.ps1

Remaining Owner Actions

These are not code gaps. They are account or release-operation follow-ups:

  • confirm GHCR package visibility if public package browsing is part of the demo.
  • deploy the AWS blueprint to a real staging account when cloud credentials and budget are available.
  • review the explicit remaining gaps and roadmap before presenting the project as a live production SaaS.

Evidence: