Thank you for your interest in contributing to qubinode-pipelines! This repository serves as the middleware layer in the three-tier qubinode architecture, hosting deployment DAGs and scripts that integrate with qubinode_navigator.
┌─────────────────────────────────────────────────────────────────────────┐
│ TIER 1: DOMAIN PROJECTS │
│ (ocp4-disconnected-helper, freeipa-workshop-deployer) │
│ │
│ Own: Domain-specific playbooks, automation logic │
│ Contribute: DAGs and scripts to qubinode-pipelines via PR │
└─────────────────────────────────────────────────────────────────────────┘
│
│ PR-based contribution
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ TIER 2: QUBINODE-PIPELINES │
│ (this repo - middleware layer) │
│ │
│ Own: │
│ - Deployment scripts (scripts/*/deploy.sh) │
│ - Deployment DAGs (dags/ocp/*.py, dags/infrastructure/*.py) │
│ - DAG registry (dags/registry.yaml) │
│ │
│ Mounted at: /opt/qubinode-pipelines │
└─────────────────────────────────────────────────────────────────────────┘
│
│ Volume mount
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ TIER 3: QUBINODE_NAVIGATOR │
│ (platform / runtime) │
│ │
│ Own: │
│ - Airflow infrastructure (docker-compose, containers) │
│ - Platform DAGs (rag_*.py, dag_factory.py, dag_loader.py) │
│ - ADRs, standards, validation tools │
│ - AI Assistant, MCP server │
└─────────────────────────────────────────────────────────────────────────┘
Deployment DAGs automate infrastructure and application deployments. They should:
- Call deployment scripts in
scripts/directory via SSH to host - Follow ADR-0045 standards (snake_case, SSH pattern, ASCII output)
- Be categorized into one of:
ocp/- OpenShift deployment and managementinfrastructure/- Core services (DNS, VMs, certificates, registries)networking/- Network configuration and managementstorage/- Storage clusters (Ceph, NFS, etc.)security/- Security scanning, compliance, hardening
Deployment scripts (scripts/*/deploy.sh) should:
- Support ACTION variable:
create,delete,status - Use standard exit codes: 0 for success, non-zero for failure
- Output ASCII markers:
[OK],[ERROR],[WARN],[INFO] - Be idempotent: Safe to run multiple times
- Source environment: Load
scripts/helper_scripts/default.env
Develop and test your DAG in your project repository first:
# In your project (e.g., ocp4-disconnected-helper)
cd my-project
mkdir -p dags
vim dags/my_new_deployment.pyUse qubinode_navigator validation tools:
# Clone qubinode_navigator if not already available
git clone https://github.com/Qubinode/qubinode_navigator.git
# Validate DAG syntax and standards
./qubinode_navigator/airflow/scripts/validate-dag.sh dags/my_new_deployment.py
# Lint the DAG
./qubinode_navigator/airflow/scripts/lint-dags.sh dags/my_new_deployment.pyBoth scripts must pass before submitting a PR.
- Fork this repository
- Create a branch:
git checkout -b feature/my-new-dag - Add your DAG to the appropriate category:
cp dags/my_new_deployment.py qubinode-pipelines/dags/ocp/
- Update registry.yaml:
ocp: description: "OpenShift deployment and management DAGs" dags: - name: my_new_deployment file: ocp/my_new_deployment.py description: "Description of what this DAG does" contributed_by: my-project-name status: tested prerequisites: - List any prerequisites
- Commit and push:
git add dags/ocp/my_new_deployment.py dags/registry.yaml git commit -m "feat: add my_new_deployment DAG" git push origin feature/my-new-dag - Open PR with:
- Clear description of what the DAG does
- Evidence of validation (paste output from validate-dag.sh and lint-dags.sh)
- Links to related issues or projects
- Testing instructions
All DAGs must comply with these standards:
- Filename: Snake case matching DAG ID (e.g.,
ocp_agent_deployment.py) - DAG ID: Snake case (e.g.,
ocp_agent_deployment)
# ✅ CORRECT: Use triple double quotes for bash_command
bash_command="""
echo "Hello World"
"""
# ❌ WRONG: Never use triple single quotes
bash_command='''
echo "Hello World"
'''Execute commands on host via SSH to avoid container limitations:
bash_command="""
ssh -o StrictHostKeyChecking=no -o LogLevel=ERROR root@localhost \
"cd /opt/qubinode-pipelines/scripts/my-script && \
export ACTION=create && \
./deploy.sh"
"""Use ASCII-only output markers:
[OK]- Success message[ERROR]- Error message[WARN]- Warning message[INFO]- Informational message
echo "[OK] Deployment completed successfully"
echo "[ERROR] Failed to connect to server"
echo "[WARN] FreeIPA not found - DNS registration skipped"
echo "[INFO] Using default configuration"Include a docstring at the top of your DAG:
"""
Airflow DAG: My Deployment
Category: ocp
This DAG automates the deployment of...
Features:
- Feature 1
- Feature 2
Prerequisites:
- Prerequisite 1
- Prerequisite 2
"""Use this template for new DAGs:
"""
Airflow DAG: [DAG Name]
Category: [ocp|infrastructure|networking|storage|security]
[Brief description of what this DAG does]
Prerequisites:
- List prerequisites here
Related ADRs:
- ADR-XXXX: [ADR Title]
"""
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.operators.python import BranchPythonOperator
default_args = {
'owner': 'qubinode',
'depends_on_past': False,
'start_date': datetime(2025, 1, 1),
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=3),
}
dag = DAG(
'my_deployment',
default_args=default_args,
description='Short description',
schedule=None,
catchup=False,
tags=['qubinode', 'category', 'component'],
params={
'action': 'create', # create or destroy
},
)
def decide_action(**context):
"""Branch based on action parameter"""
action = context['params'].get('action', 'create')
if action == 'destroy':
return 'destroy_task'
return 'validate_environment'
decide_action_task = BranchPythonOperator(
task_id='decide_action',
python_callable=decide_action,
dag=dag,
)
validate_environment = BashOperator(
task_id='validate_environment',
bash_command="""
echo "========================================"
echo "Validating Environment"
echo "========================================"
# Add validation logic here
echo "[OK] Environment validation complete"
""",
dag=dag,
)
# Add more tasks here...
# Define workflow
decide_action_task >> validate_environmentUse this template for deployment scripts:
#!/bin/bash
# Component Deployment Script
# Usage: ACTION=create ./deploy.sh
#
# Environment Variables:
# ACTION - create, delete, or status (required)
# VM_NAME - Custom VM name (optional)
# Other component-specific variables
set -euo pipefail
# Default configuration
ACTION="${ACTION:-create}"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# Source environment
if [ -f /opt/qubinode-pipelines/scripts/helper_scripts/default.env ]; then
source /opt/qubinode-pipelines/scripts/helper_scripts/default.env
fi
echo "========================================"
echo "Component Deployment"
echo "========================================"
echo "Action: ${ACTION}"
echo "========================================"
function create() {
echo "[INFO] Creating component..."
# Add creation logic here
echo "[OK] Component created successfully"
}
function delete() {
echo "[INFO] Deleting component..."
# Add deletion logic here
echo "[OK] Component deleted successfully"
}
function status() {
echo "[INFO] Checking component status..."
# Add status check logic here
echo "[OK] Status check complete"
}
# Main execution
case "${ACTION}" in
create)
create
;;
delete)
delete
;;
status)
status
;;
*)
echo "[ERROR] Invalid action: ${ACTION}"
echo "Usage: ACTION=[create|delete|status] ./deploy.sh"
exit 1
;;
esac-
Mount this repo in qubinode_navigator:
# In qubinode_navigator directory vim docker-compose.yml # Add volume mount: # - /path/to/qubinode-pipelines:/opt/qubinode-pipelines
-
Restart Airflow:
cd qubinode_navigator docker compose down docker compose up -d -
Test your DAG:
- Open Airflow UI: http://localhost:8080
- Find your DAG in the list
- Trigger it with test parameters
- Verify it completes successfully
Once you submit a PR, GitHub Actions will automatically:
- Validate DAG syntax
- Check code style (linting)
- Verify registry.yaml is updated
- Check for proper documentation
- Documentation: See README.md for architecture overview
- ADRs: Read qubinode_navigator ADRs
- Issues: Open an issue in this repository
- Discussions: Use GitHub Discussions for questions
Please be respectful and professional in all interactions. We welcome contributions from everyone.
By contributing, you agree that your contributions will be licensed under the same license as this project (Apache 2.0).