ClickHouse on Amazon EKS

A small, anonymized reference for running ClickHouse on Amazon EKS with the Altinity ClickHouse Operator.

This repository is based on lessons from a real EKS deployment, but it is intentionally generic. Organization names, AWS accounts, regions, domains, namespaces, image registries, database names, secret paths, resource sizes, and workload-specific SQL have been removed or replaced.

Why this repository exists

ClickHouse is straightforward to start but requires deliberate decisions around storage, replication, credentials, backups, monitoring, and upgrades before it becomes a reliable Kubernetes service. This repository documents a practical starting point without exposing a private infrastructure repository.

It includes a small single-node demo for learning and a production-oriented architecture guide. The demo is not a production template.

What is ClickHouse?

ClickHouse is an open-source, column-oriented SQL database designed for fast analytical queries over large datasets. Unlike a transactional database used for individual application records, ClickHouse is optimized for reading and aggregating many rows at once.

Typical uses include:

application, access, and audit-log analysis;
product and visitor analytics;
time-series events and operational metrics;
dashboards and near-real-time reporting;
high-volume event ingestion;
long-running aggregations that would be expensive in an OLTP database.

ClickHouse is usually a poor primary database for workflows dominated by frequent single-row updates, strict multi-row transactions, or relational application state. It often complements PostgreSQL or MySQL rather than replacing them.

Architecture

The included demo deploys one ClickHouse pod and one EBS-backed persistent volume behind an internal Kubernetes service.

flowchart LR
    C[Applications and data collectors] -->|HTTP 8123 or native 9000| S[ClusterIP Service]
    S --> CH[ClickHouse pod]
    CH --> EBS[(Amazon EBS PVC)]
    CH -->|Metrics| P[Prometheus]
    SM[AWS Secrets Manager] -. optional External Secrets .-> K8S[Kubernetes Secret]
    K8S --> CH
    CH -. native BACKUP / RESTORE .-> S3[(Amazon S3)]

Recommended production shape

A common highly available starting point is:

one shard with two ClickHouse replicas;
three ClickHouse Keeper replicas for coordination and quorum;
ClickHouse replicas spread across Availability Zones;
one EBS volume per ClickHouse and Keeper pod;
internal-only services;
credentials synchronized from AWS Secrets Manager;
Prometheus scraping ClickHouse and Keeper metrics;
scheduled native ClickHouse backups to S3;
restore tests performed regularly.

flowchart TB
    APP[Writers and readers] --> LB[Internal Kubernetes Service]
    LB --> CH1[ClickHouse replica 1]
    LB --> CH2[ClickHouse replica 2]
    CH1 --> V1[(EBS volume A)]
    CH2 --> V2[(EBS volume B)]
    CH1 <--> K[3-node ClickHouse Keeper quorum]
    CH2 <--> K
    CH1 -. backup .-> S3[(S3 backup bucket)]

Replicas improve availability and hold additional copies of data. Shards divide data and increase capacity. Do not add shards merely to obtain high availability.

Repository contents

.
├── manifests/
│   └── demo/
│       ├── clickhouse-installation.yaml
│       ├── kustomization.yaml
│       └── namespace.yaml
└── scripts/
    ├── deploy-demo.sh
    ├── install-operator.sh
    ├── uninstall-demo.sh
    └── verify-demo.sh

The demo uses:

a ClickHouseInstallation custom resource;
one shard and one replica;
an internal ClusterIP service;
an EBS-backed ReadWriteOnce volume;
a non-default administrator whose password comes from a Kubernetes Secret;
the default ClickHouse user disabled;
a Prometheus metrics endpoint;
a retained volume policy to reduce accidental data loss.

Prerequisites

You need:

an existing EKS cluster with EC2 worker nodes;
access to the cluster through kubectl;
Helm 3;
a working EBS CSI driver;
a suitable EBS StorageClass, commonly a gp3 class;
openssl for local password generation;
permissions to install cluster-wide CRDs and operator RBAC.

The EBS CSI driver requires AWS IAM permissions. AWS recommends installing it as an EKS add-on. EKS Auto Mode uses a different EBS provisioner, and EBS volumes cannot be attached to Fargate pods.

Verify the basics:

kubectl cluster-info
kubectl get nodes
kubectl get storageclass
helm version

Make sure kubectl points to the intended cluster before running any script:

aws eks update-kubeconfig \
  --name <cluster-name> \
  --region <aws-region>

kubectl config current-context

Quick start

1. Install the ClickHouse Operator

./scripts/install-operator.sh

The script installs the official Altinity Helm chart in a dedicated namespace and waits for the ClickHouse CRD. For repeatable production builds, set and test a specific chart version:

OPERATOR_CHART_VERSION=<tested-version> \
  ./scripts/install-operator.sh

2. Deploy the demo

The demo script creates a namespace-local Kubernetes Secret and applies the Kustomize manifests.

CLICKHOUSE_ADMIN_PASSWORD="$(openssl rand -base64 32)" \
  ./scripts/deploy-demo.sh

The password is not written to this repository. Store it in a password manager for the duration of the demo.

Optional settings:

CLICKHOUSE_ADMIN_PASSWORD="<strong-password>" \
  ./scripts/deploy-demo.sh

If your cluster uses another namespace or StorageClass, update the demo manifests before deployment. Keep the namespace value consistent across namespace.yaml, kustomization.yaml, and the helper scripts.

3. Verify the deployment

./scripts/verify-demo.sh

The script waits for the pod, displays the generated services and PVCs, and executes a basic query from inside the ClickHouse pod.

4. Connect locally

Find the pod and open a temporary local tunnel:

POD="$(kubectl get pod \
  --namespace clickhouse-demo \
  --selector clickhouse.altinity.com/chi=clickhouse-demo \
  --output jsonpath='{.items[0].metadata.name}')"

kubectl port-forward \
  --namespace clickhouse-demo \
  "pod/${POD}" \
  8123:8123 9000:9000

In another terminal, obtain the demo password and query the HTTP interface:

CLICKHOUSE_PASSWORD="$(kubectl get secret clickhouse-credentials \
  --namespace clickhouse-demo \
  --output jsonpath='{.data.admin-password}' | base64 --decode)"

curl --fail --silent --show-error \
  --user "admin:${CLICKHOUSE_PASSWORD}" \
  'http://127.0.0.1:8123/?query=SELECT%20version()'

Do not expose ports 8123 or 9000 directly to the public internet.

Try a small dataset

Open a ClickHouse client in the pod:

kubectl exec --namespace clickhouse-demo --stdin --tty "${POD}" -- \
  clickhouse-client --user admin --password "${CLICKHOUSE_PASSWORD}"

Then create and query a simple events table:

CREATE DATABASE IF NOT EXISTS analytics;

CREATE TABLE IF NOT EXISTS analytics.events
(
    occurred_at DateTime,
    event_name LowCardinality(String),
    account_id UInt64,
    properties String
)
ENGINE = MergeTree
PARTITION BY toYYYYMM(occurred_at)
ORDER BY (event_name, occurred_at)
TTL occurred_at + INTERVAL 90 DAY;

INSERT INTO analytics.events VALUES
    (now(), 'page_view', 1001, '{"path":"/docs"}'),
    (now(), 'sign_in', 1001, '{}'),
    (now(), 'page_view', 1002, '{"path":"/pricing"}');

SELECT
    event_name,
    count() AS events
FROM analytics.events
GROUP BY event_name
ORDER BY events DESC;

The partition, ordering key, and TTL are part of the data model. Choose them from real query and retention requirements, not by copying this example.

Installation options

Helm installation used by this repository

The helper script wraps the operator's official Helm chart:

helm repo add clickhouse-operator https://helm.altinity.com
helm repo update clickhouse-operator
helm upgrade --install clickhouse-operator \
  clickhouse-operator/altinity-clickhouse-operator \
  --namespace clickhouse-operator \
  --create-namespace

Review chart values and RBAC before using this in a controlled environment. Pin the chart version in GitOps or CI instead of silently following the latest release.

Direct manifest installation

The operator also publishes an installation bundle. Pin a release tag rather than referencing the moving master branch:

kubectl apply --filename \
  "https://raw.githubusercontent.com/Altinity/clickhouse-operator/<operator-version>/deploy/operator/clickhouse-operator-install-bundle.yaml"

Use either Helm or the bundle, not both.

AWS considerations

EBS storage

Use the EBS CSI driver and an encrypted StorageClass.
Use volumeBindingMode: WaitForFirstConsumer so scheduling can select an appropriate Availability Zone.
EBS volumes are Availability Zone scoped. A pod cannot attach the same volume in another zone without recovery or migration.
Spread replicas across zones; do not expect one EBS volume to provide multi-zone availability.
Use a reclaim policy appropriate for the data. Production database volumes should normally be retained when a claim or workload is removed.
Monitor free disk space, part counts, merge pressure, and volume limits.

Node scheduling

For a busy production cluster, use a dedicated EKS node group with labels and taints. Configure node selectors, tolerations, and topology spread constraints in the ClickHouse pod template.

Do not add dedicated-node constraints to the demo until matching nodes exist, or the pods will remain Pending.

Secrets

The demo creates a normal Kubernetes Secret for simplicity. For production, use AWS Secrets Manager with a controller such as External Secrets Operator and EKS Pod Identity or IRSA.

Use separate identities for:

administration;
application ingestion;
read-only dashboards;
metrics exporters;
backups.

Grant each identity access only to the required databases and operations. Disable or secure the default user and limit allowed networks.

Never commit passwords, AWS keys, S3 credentials, internal hostnames, or rendered Secrets.

Networking

Prefer ClusterIP for clients running in the same cluster.
If clients require VPC access, use an internal load balancer and restrictive security groups.
Apply Kubernetes NetworkPolicies where the cluster networking plugin supports them.
Use TLS for traffic that crosses trust boundaries.
Limit access to HTTP 8123, native TCP 9000, Keeper, and metrics ports separately.

Production hardening

Before adapting this example for production:

Replication and Keeper

ClickHouse Keeper coordinates replication and distributed DDL. For a production replicated deployment, use an odd number of Keeper nodes so the cluster can maintain quorum; three is a common starting point.

The production pattern that informed this repository uses one shard with two replicas and a separate three-member Keeper quorum. That topology protects against a single ClickHouse replica failure without introducing sharding complexity.

Use replicated engines for replicated data, for example:

CREATE TABLE analytics.events_local
ON CLUSTER analytics_cluster
(
    occurred_at DateTime,
    event_name LowCardinality(String),
    account_id UInt64
)
ENGINE = ReplicatedMergeTree
PARTITION BY toYYYYMM(occurred_at)
ORDER BY (event_name, occurred_at);

Do not change replica counts or Keeper topology casually on a stateful production cluster. Test operator behavior, persistent-volume retention, schema engines, and recovery procedures first.

Monitoring

The demo enables ClickHouse's Prometheus endpoint. If Prometheus Operator is installed, create a ServiceMonitor matching the generated ClickHouse service labels and metrics port.

At minimum, alert on:

pod restarts and unavailable replicas;
Keeper quorum or session failures;
free disk space and inode exhaustion;
memory pressure and killed queries;
delayed inserts and rejected connections;
replication queue growth and replica delay;
excessive active parts and failed merges;
backup failures and missing restore tests;
query latency and error-rate changes.

Application-specific SQL metrics can be exported separately, but protect exporter credentials with a read-only profile and short query timeouts.

Backups and recovery

Replication is not a backup. It can reproduce accidental deletions and bad writes across replicas.

ClickHouse supports native BACKUP and RESTORE to S3. A practical schedule may combine periodic full backups with more frequent incremental backups. Give the backup pod or ClickHouse workload only the S3 permissions it requires, preferably through EKS Pod Identity or IRSA rather than static AWS keys.

Example syntax:

BACKUP DATABASE analytics
TO S3('https://<bucket>.s3.<aws-region>.amazonaws.com/clickhouse/full-<timestamp>');

RESTORE DATABASE analytics AS analytics_restored
FROM S3('https://<bucket>.s3.<aws-region>.amazonaws.com/clickhouse/full-<timestamp>');

Exact authentication and endpoint syntax depends on the AWS identity model and ClickHouse version. Review the official backup documentation before implementation.

Backups are complete only when you can restore them. Regularly restore into an isolated namespace and verify schemas, row counts, representative queries, permissions, and recovery time.

Upgrades

Read the ClickHouse and operator release notes.
Back up the databases and verify recent restore evidence.
Test the exact version transition on non-production data.
Render and review the manifests before applying them.
Upgrade one controlled step at a time.
Verify Keeper, replication queues, merges, queries, and ingestion before continuing.
Keep a documented rollback or recovery plan.

Avoid unpinned latest image tags.

Troubleshooting

Pod remains Pending

kubectl describe pod --namespace clickhouse-demo <pod-name>
kubectl get pvc --namespace clickhouse-demo
kubectl get storageclass

Common causes are a missing StorageClass, EBS CSI permissions, incompatible node selectors or taints, insufficient capacity, and Availability Zone constraints.

Secret is missing

kubectl get secret clickhouse-credentials --namespace clickhouse-demo

Run deploy-demo.sh with CLICKHOUSE_ADMIN_PASSWORD set. For External Secrets, inspect the ExternalSecret, SecretStore, controller logs, and AWS IAM permissions.

Operator does not reconcile

kubectl get crd clickhouseinstallations.clickhouse.altinity.com
kubectl get pods --namespace clickhouse-operator
kubectl get clickhouseinstallation --all-namespaces

Then inspect the ClickHouseInstallation status and operator logs.

ClickHouse is running but unreachable

kubectl get service --namespace clickhouse-demo
kubectl get endpoints --namespace clickhouse-demo
kubectl get networkpolicy --namespace clickhouse-demo

Verify the selected port, service type, security groups, NetworkPolicies, user network rules, and TLS settings.

Uninstall the demo

./scripts/uninstall-demo.sh

The script removes the demo custom resource, Secret, and namespace only after confirmation. Persistent volumes may remain because the example favors retention. Inspect and delete retained EBS volumes manually only after confirming the data is disposable.

The operator is shared infrastructure and is not removed by the demo uninstall script.

Security and anonymization

This repository contains no credentials and is not intended to identify or reproduce a specific private environment. Before publishing a fork, scan the entire Git history as well as the current files for:

AWS account IDs and ARNs;
private registry URLs;
internal DNS names and domains;
cluster, namespace, database, and customer names;
Secrets Manager paths and Kubernetes Secret values;
source comments that describe private systems;
dashboards, queries, or schemas containing business identifiers.

References

Disclaimer

This is an educational reference, not a drop-in production platform. Capacity, topology, durability, security, and recovery requirements vary by workload. Validate every change in a non-production EKS cluster and obtain an experienced review before storing important data.

Name		Name	Last commit message	Last commit date
manifests/demo		manifests/demo
scripts		scripts
README.md		README.md

Latest commit

History

Folders and files

Repository files navigation