Skip to content

UPGRADING.md

Latest commit

 

History

History
1444 lines (1125 loc) · 59.8 KB

UPGRADING.md

File metadata and controls

1444 lines (1125 loc) · 59.8 KB
Raw

Upgrade guidelines

0.157.0 to 0.158.0

Deployment environment resource attribute renamed

When the optional environment value is set, the chart now emits the stable deployment.environment.name resource attribute instead of the deprecated deployment.environment attribute. The value is also added to the agent, gateway, and cluster receiver service.telemetry.resource.attributes, so Collector's internal telemetry now includes the deployment environment.

If deployment.environment is still required, use Collector configuration overrides to restore the deprecated name. Keep environment set so the chart continues to configure the attribute and applicable pipeline references, then override resource/add_environment and service.telemetry.resource.attributes for each enabled component. For example:

environment: production

agent:
  config:
    processors:
      resource/add_environment:
        attributes:
          - action: insert
            key: deployment.environment
            value: production
    service:
      telemetry:
        resource:
          attributes:
            - name: deployment.environment
              value: production
            # Copy rest of internal telemetry resource attributes here

Because overriding service.telemetry.resource.attributes replaces the complete list, preserve all existing attributes and change only deployment.environment.name to deployment.environment. Use helm template to render the chart using the same value arguments as your install command, then extract the resource section for each enabled component. For example, for the agent component:

helm template my-splunk-otel-collector --values my_values.yaml splunk-otel-collector-chart/splunk-otel-collector --show-only templates/configmap-agent.yaml | yq '.data.relay | from_yaml | .service.telemetry.resource'

Use templates/configmap-gateway.yaml for the gateway and templates/configmap-cluster-receiver.yaml for the cluster receiver. Copy the complete output under the corresponding <component>.config.service.telemetry.resource section and change only the deployment.environment.name entry to deployment.environment.

cert-manager subchart is removed

The deprecated cert-manager subchart and the top-level certmanager values have been removed. The operator still supports certificates from a separately managed cert-manager installation through operator.admissionWebhooks.certManager. Deployments using the default Helm-generated operator webhook certificate do not need to install cert-manager.

If certmanager.enabled=true in your current release, migrate to a standalone cert-manager installation before upgrading. Follow the standalone cert-manager migration steps while the Splunk OpenTelemetry Collector chart is still at version 0.157.0. Transfer any required settings under certmanager.* to the standalone jetstack/cert-manager release.

The removed certmanager key is no longer accepted by the chart's values schema. Do not use --reuse-values for this upgrade if the release's saved values contain that key. Export the current user-supplied values and remove the entire certmanager block:

helm get values <release> \
  --namespace <namespace> \
  --output yaml > values-0.158.0.yaml

Then upgrade with the cleaned values file:

helm upgrade <release> splunk-otel-collector-chart/splunk-otel-collector \
  --namespace <namespace> \
  --reset-values \
  --values values-0.158.0.yaml

If the release values are managed in source control, remove the certmanager block there and use that values file instead. Keep operator.admissionWebhooks.certManager.enabled=true when the operator should continue using the standalone cert-manager installation.

0.154.0 to 0.155.0

cert-manager subchart is deprecated

Installing cert-manager through the Splunk OpenTelemetry Collector chart by setting certmanager.enabled=true is deprecated and will be removed in a future release. Use a separately managed cert-manager installation instead.

This affects deployments that use all of these settings:

  • operator.enabled=true
  • operator.admissionWebhooks.certManager.enabled=true
  • certmanager.enabled=true

For new installs, follow the operator webhook certificate guidance in the auto-instrumentation install guide. Do not enable the deprecated cert-manager subchart.

The examples in this section show only the values related to cert-manager and the operator webhook certificate. Keep your existing required chart values, such as clusterName and your Splunk destination configuration, when running these commands.

Before you migrate

The operator webhook keeps serving with its current certificate throughout the migration. cert-manager cannot issue or renew certificates in the short gap between disabling the subchart and installing standalone cert-manager, so only proceed if the webhook certificate is not close to expiry:

kubectl get certificate -n <namespace> \
  -l "app.kubernetes.io/instance=<release>,app.kubernetes.io/component=webhook" \
  -o jsonpath='{.items[0].status.notAfter}'

An in-place handoff of the cert-manager Deployments is not supported with normal Helm commands. --take-ownership is still useful later in the migration to let the standalone cert-manager release adopt retained CRDs.

Migration steps

During this migration, disabling the subchart removes the old cert-manager controller, cainjector, and webhook Deployments before the standalone cert-manager release is installed.

The examples below use cert-manager as the standalone cert-manager release name and namespace, which is the common namespace for a cluster-wide cert-manager installation. Replace <release> and <namespace> with the Splunk OpenTelemetry Collector Helm release name and namespace. If cert-manager is already managed separately in your cluster, use that installation instead of installing a second standalone cert-manager release. Adjust or skip the cert-manager Helm commands below if your cert-manager installation is not managed by Helm.

Upgrade the Splunk chart with the cert-manager subchart disabled, while keeping the operator configured to use cert-manager:

helm upgrade <release> splunk-otel-collector-chart/splunk-otel-collector \
  --namespace <namespace> \
  --reuse-values \
  --set certmanager.enabled=false \
  --set operator.admissionWebhooks.autoGenerateCert.enabled=false \
  --set operator.admissionWebhooks.certManager.enabled=true

Confirm that the old subchart cert-manager Deployments were removed:

kubectl get deployment -n <namespace> \
  -l "app.kubernetes.io/instance=<release>,app.kubernetes.io/component in (controller,cainjector,webhook)"

The command should return no resources.

Install cert-manager as a standalone release. Choose the cert-manager version explicitly. To minimize version changes during the migration, use the version from this chart's cert-manager dependency, or use the version approved for your cluster if cert-manager is managed by your platform team.

If you customized the cert-manager subchart with values under certmanager.*, apply equivalent values to the standalone jetstack/cert-manager release. The migration adopts retained CRDs, but it does not copy cert-manager Deployment, Service, scheduling, image, monitoring, or security settings from the old subchart release:

helm repo add jetstack https://charts.jetstack.io
helm repo update jetstack

helm upgrade --install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version <cert-manager-version> \
  --take-ownership \
  --set crds.enabled=true \
  --set crds.keep=true \
  --wait

--take-ownership lets the standalone release adopt cert-manager CRDs that were kept from the old subchart release. It is only for retained resources such as CRDs. Use a Helm version that supports --take-ownership. If cert-manager CRDs are managed separately in your cluster, do not adopt them into this standalone release. Install cert-manager with crds.enabled=false and follow your existing CRD management process instead.

This migration relies on the cert-manager CRDs being retained. Do not change crds.keep=true to crds.keep=false when installing the standalone cert-manager release. Deleting cert-manager CRDs also deletes cert-manager custom resources, including the operator webhook Issuer and Certificate.

If your existing values use the deprecated hook workaround with certificateAnnotations or issuerAnnotations, the --reuse-values upgrade keeps those annotations during the migration. Helm may delete and recreate the operator webhook Certificate and Issuer during that upgrade. This is expected. The webhook serving Secret normally remains in place, and the delete does not normally wait for the cert-manager controller. The webhook keeps serving with the existing Secret and injected CA bundle until the recreated resources are reconciled by the standalone cert-manager release.

If cert-manager was installed with enableCertificateOwnerRef=true, deleting the Certificate may also delete the serving Secret. The operator should continue serving the certificate it already loaded, but do not restart the operator pod until standalone cert-manager is running and has reissued the Secret.

Validation

After the migration:

helm status <release> -n <namespace>
helm status cert-manager -n cert-manager

kubectl wait --for=condition=Available deployment \
  -n cert-manager \
  -l "app.kubernetes.io/instance=cert-manager,app.kubernetes.io/component in (controller,cainjector,webhook)" \
  --timeout=5m

kubectl wait --for=condition=Available deployment \
  -n <namespace> \
  -l "app.kubernetes.io/instance=<release>,app.kubernetes.io/name=operator" \
  --timeout=5m

kubectl wait --for=condition=Ready certificate \
  -n <namespace> \
  -l "app.kubernetes.io/instance=<release>,app.kubernetes.io/component=webhook" \
  --timeout=5m

Also verify that the old subchart cert-manager Deployments are still absent from the Splunk chart namespace:

kubectl get deployment -n <namespace> \
  -l "app.kubernetes.io/instance=<release>,app.kubernetes.io/component in (controller,cainjector,webhook)"

The command should return no resources. The Splunk chart can still include cert-manager API resources such as Certificate and Issuer for the operator webhook certificate; the controller, cainjector, and webhook Deployments should come from the standalone cert-manager release.

Optional: remove the hook workaround

After cert-manager is healthy, you can remove the deprecated Helm hook annotations from the operator webhook Certificate and Issuer. Hook-created resources do not have Helm's normal ownership annotations, so add those annotations before upgrading without the hook values:

kubectl annotate certificate,issuer -n <namespace> \
  -l "app.kubernetes.io/instance=<release>,app.kubernetes.io/component=webhook" \
  meta.helm.sh/release-name=<release> \
  meta.helm.sh/release-namespace=<namespace> \
  --overwrite

kubectl label certificate,issuer -n <namespace> \
  -l "app.kubernetes.io/instance=<release>,app.kubernetes.io/component=webhook" \
  app.kubernetes.io/managed-by=Helm \
  --overwrite

Then remove certificateAnnotations and issuerAnnotations from your values file, or override them with empty maps:

helm upgrade <release> splunk-otel-collector-chart/splunk-otel-collector \
  --namespace <namespace> \
  --reuse-values \
  --set-json 'operator.admissionWebhooks.certManager.certificateAnnotations={}' \
  --set-json 'operator.admissionWebhooks.certManager.issuerAnnotations={}'

Rollback

If the standalone cert-manager install fails after the Splunk chart upgrade, restore cert-manager first. The operator may continue serving webhooks while the existing TLS Secret is valid, but certificate renewal will not work until cert-manager is healthy again.

0.153.1 to 0.154.0

Target allocator functionality now uses upstream chart as subchart

Instead of relying on local versioning and implementation of target allocator functionality, the chart has moved to using the upstream Target Allocator helm chart directly as a subchart. This will help keep up to date with upstream features and functionality.

This resulted in breaking changes to the helm chart's configuration for the target allocator, as outlined below.

Old option New option
targetAllocator targetallocator
image.imagePullSecrets targetallocator.targetAllocator.imagePullSecrets
targetAllocator.image targetallocator.targetAllocator.image.repository + targetallocator.targetAllocator.image.tag
targetAllocator.resources targetallocator.targetAllocator.resources
targetAllocator.serviceAccount targetallocator.targetAllocator.serviceAccount
targetAllocator.config targetallocator.targetAllocator.config

Note

The image.imagePullSecrets option is still valid for non-target allocator service accounts, but the new option must ALSO be configured to attach secrets to the target allocator's service account.

Example old config:

image:
  imagePullSecrets:
    - my-registry-secret

targetAllocator:
  enabled: true
  image: ghcr.io/open-telemetry/opentelemetry-operator/target-allocator:v0.132.0
  config:
    allocation_strategy: per-node
    prometheus_cr:
      enabled: true
    filter_strategy: relabel-config

New config equivalent:

image:
  imagePullSecrets:
    - my-registry-secret

targetallocator:
  enabled: true
  targetAllocator:
    imagePullSecrets:
      - my-registry-secret
    image:
      repository: ghcr.io/open-telemetry/opentelemetry-operator/target-allocator
      tag: v0.132.0
    config:
      allocation_strategy: per-node
      prometheus_cr:
        enabled: true
      filter_strategy: relabel-config

Changed target allocator defaults (when enabled):

Old option default New option default Notes
N/A targetallocator.targetAllocator.livenessProbe Reference
N/A targetallocator.targetAllocator.readinessProbe Reference

Refer to the upstream target allocator helm chart's values.yaml for the full list of valid configuration options.

0.151.0 to 0.152.0

Container log parsing now uses the container operator

The filelog operator chain for container log parsing now uses the container operator in place of the runtime-specific parser chain, with a small docker recombine step and optional multiline handling still present where applicable. This affects users who use logsCollection.containers.extraOperators or logsCollection.containers.multilineConfigs.

containerRuntime: cri-o must be changed to crio

The accepted value for CRI-O has changed:

# Before
logsCollection:
  containers:
    containerRuntime: cri-o

# After
logsCollection:
  containers:
    containerRuntime: crio

extraOperators referencing attributes.log must use body

Previously the parsed log line was available as attributes.log. It is now set directly as body:

# Before
logsCollection:
  containers:
    extraOperators:
      - type: filter
        expr: 'attributes.log matches "health_check"'

# After
logsCollection:
  containers:
    extraOperators:
      - type: filter
        expr: 'body matches "health_check"'

If attributes.log is still referenced, no error is raised — the value silently evaluates to null.

attributes.time is no longer available in extraOperators, use field timestamp instead

Previously attributes.time contained the raw timestamp string and was available to extraOperators. The container operator sets the timestamp directly on the log record's Timestamp field and does not set attributes.time. Replace any references to attributes.time with timestamp, which accesses the log record's Timestamp field directly in stanza expressions. For example:

# Before
- type: filter
  expr: 'attributes.time < "2026-01-01"'

# After
- type: filter
  expr: 'timestamp.Before(date("2026-01-01T00:00:00Z"))'

If attributes.time is still referenced, no error is raised — the value silently evaluates to null.

extraOperators using old operator IDs as output targets

The following operator IDs no longer exist: get-format, parser-docker, parser-crio, parser-containerd, crio-recombine, containerd-recombine, clean-up-log-record. Remove any output: references to these IDs from your extraOperators.

New feature gate: component label in agent DaemonSet selector

A new feature gate featureGates.daemonSetComponentSelector (disabled by default) adds component: otel-collector-agent to the agent DaemonSet's spec.selector.matchLabels. This prevents workload management tools from incorrectly grouping gateway and cluster receiver pods under the agent daemonset.

The feature gate is disabled by default so that helm upgrade works without any manual steps. To opt in, set:

featureGates:
  daemonSetComponentSelector: true

Because selector.matchLabels is immutable on DaemonSets, enabling this on an existing installation requires deleting the DaemonSet first. Choose one of the options below.

Option A: Zero-downtime upgrade (recommended)

Delete the existing agent DaemonSet with --cascade=orphan before upgrading. This removes the DaemonSet object but keeps all agent pods running, so there is no gap in data collection:

kubectl delete daemonset \
  -l app=splunk-otel-collector,component=otel-collector-agent,release=<release> \
  --cascade=orphan \
  -n <namespace>
helm upgrade <release> splunk-otel-collector-chart/splunk-otel-collector \
  --set featureGates.daemonSetComponentSelector=true ... -n <namespace>

Helm creates the new DaemonSet with the three-label selector and adopts the orphaned pods (they already carry component: otel-collector-agent in their labels). The DaemonSet controller reconciles: pods on nodes that already have a matching pod are left alone; any missing nodes get new pods scheduled.

Option B: Delete the DaemonSet and upgrade (brief data-collection gap)

Delete the agent DaemonSet (without --cascade=orphan) before upgrading. This is simpler than Option A but terminates existing agent pods — there will be a brief data-collection gap on each node while new pods start:

kubectl delete daemonset \
  -l app=splunk-otel-collector,component=otel-collector-agent,release=<release> \
  -n <namespace>
helm upgrade <release> splunk-otel-collector-chart/splunk-otel-collector \
  --set featureGates.daemonSetComponentSelector=true ... -n <namespace>

Option C: Uninstall and reinstall (simple environments)

helm uninstall <release> -n <namespace>
helm install <release> splunk-otel-collector-chart/splunk-otel-collector \
  --set featureGates.daemonSetComponentSelector=true ... -n <namespace>

This removes all chart-managed resources (agent, gateway, cluster receiver, RBAC, etc.) and recreates them from scratch.

0.137.0 to 0.138.0

Fluentd sidecar has been removed

The legacy fluentd sidecar container and related configuration options have been removed from the chart. All users should now use the native OpenTelemetry logs collection, which is the default and provides better performance and resource efficiency.

If you still need to use fluentd for log collection, you can run it as a separate deployment using the official fluentd helm chart and forward logs to the Splunk OpenTelemetry Collector using the fluentforward receiver.

Running fluentd separately with fluentforward

  1. Enable the fluentforward receiver and port in the collector:
agent:
  ports:
    fluentforward:
      containerPort: 8006
      hostPort: 8006
      protocol: TCP
      enabled_for: [logs]
  config:
    receivers:
      fluentforward:
        endpoint: 0.0.0.0:8006
    service:
      pipelines:
        logs:
          receivers: [fluentforward, otlp]
  1. Deploy fluentd separately using the fluentd helm chart and configure it to forward logs to the node IP on port 8006 (the collector agent listens on hostPort).

0.134.0 to 0.135.0

This release includes the breaking change of removing the splunkObservability.logsEnabled option.

0.128.0 to 0.129.0

This release includes a breaking change related to the operator instrumentation configuration.

  • Starting with version 0.129.0, the operator.instrumentation.* configuration has been moved to operator.instrumentation.spec.*.
  • User can now define any spec setting supported by the OpenTelemetry Operator
  • The helm chart defines the minimal default spec settings for the operator instrumentation, which can be overridden by the user.
  • If you used custom values under instrumentation.spec.* in your values.yaml, you will need to migrate them to the new path:
    • instrumentation.endpoint -> instrumentation.spec.exporter.endpoint
    • instrumentation.propagators.* -> instrumentation.spec.propagators.*
    • instrumentation.sampler.* -> instrumentation.spec.sampler.*
    • instrumentation.env.* -> instrumentation.spec.env.* <libname> : name of the library you are instrumenting (java, python, nodejs, dotnet, go, apache-httpd and nginx).
    • instrumentation.<libname>.* -> instrumentation.spec.<libname>.*
      • instrumentation.<libname>.repository and -> instrumentation.<libname>.tag -> instrumentation.spec.java.image

Example 1: Migrating custom values

v0.128.0 and earlier

instrumentation:
  propagators:
    - "b3multi"
    - "baggage"
  env:
    - name: "SPLUNK_ACCESS"
      value: "my-access-token"
  java:
    repository: ghcr.io/personal/splunk-otel-java
    tag: v1.27.0
    env:
      - name: "JAVA_TOOL_OPTIONS"
        value: "-javaagent:/opt/splunk/splunk-otel-javaagent.jar"

v0.129.0 and later

instrumentation:
  spec:
    propagators:
      - "b3multi"
      - "baggage"
    env:
      - name: "SPLUNK_ACCESS"
        value: "my-access-token"
    java:
      image: ghcr.io/personal/splunk-otel-java:v1.27.0
      env:
        - name: "JAVA_TOOL_OPTIONS"
          value: "-javaagent:/opt/splunk/splunk-otel-javaagent.jar"

Example 2: Adding resources limits to the java instrumentation and removing python instrumentation:

instrumentation:
  spec:
    python:
    java:
      resources:
        limits:
          cpu: 500m
          memory: 500Mi
        requests:
          cpu: 100m
          memory: 200Mi

0.125.0 to 0.126.0

This release includes a breaking change related to Prometheus receivers utilized by the agent and clusterReceiver.

Restoring Previous Behavior

You can disable the feature gate to restore previous behavior, for example via Helm CLI:

For the agent: --set agent.featureGates=-receiver.prometheusreceiver.RemoveLegacyResourceAttributes

For the clusterReceiver: --set clusterReceiver.featureGates=-receiver.prometheusreceiver.RemoveLegacyResourceAttributes

Note: This feature gate will be removed in a future release. It’s recommended to migrate to the new attributes (server.address, server.port, url.scheme).

0.119.0 to 0.120.0

This guide provides steps for new users, transitioning users, and those maintaining previously deployed Operator-related TLS certificates and configurations.

  • New users: No migration is required for Operator TLS certificates.
  • Previous users: Migration may be needed if using operator.enabled=true or certmanager.enabled=true.

To maintain previous functionality and avoid breaking changes, review the following sections.

Maintaining Previous Functionality via Helm Values Update

Scenario 1: Operator and cert-manager Deployed via This Helm Chart

If you previously deployed both the Operator and cert-manager via this Helm chart (operator.enabled=true and certmanager.enabled=true), you can preserve functionality by adding the following values:

operator:
  enabled: true
  admissionWebhooks:
    certManager:
      enabled: true
      certificateAnnotations:
        "helm.sh/hook": post-install,post-upgrade
        "helm.sh/hook-weight": "1"
      issuerAnnotations:
        "helm.sh/hook": post-install,post-upgrade
        "helm.sh/hook-weight": "1"
certmanager:
  enabled: true
  installCRDs: true

Scenario 2: Operator Deployed with External cert-manager (Not Managed by This Helm Chart)

If you previously deployed the Operator and used an externally managed cert-manager (operator.enabled=true and certmanager.enabled=false), you can preserve functionality by adding the following values:

operator:
  enabled: true
  admissionWebhooks:
    certManager:
      enabled: true

Adopting New Functionality (Requires Migration Steps)

If you want to migrate from cert-manager managed certificates to the now default Helm-generated certificates, additional steps may be required to avoid conflicts.

Potential Upgrade Issue: Existing Secret Conflict

If you see an error message like the following during a Helm install or upgrade:

warning: Upgrade "{helm_release_name}" failed: pre-upgrade hooks failed: warning: Hook pre-upgrade splunk-otel-collector/charts/operator/templates/admission-webhooks/operator-webhook.yaml failed: 1 error occurred:* secrets "splunk-otel-collector-operator-controller-manager-service-cert" already exists

This typically occurs because:

  • cert-manager deletes its Certificate resources immediately.
  • However, cert-manager does not delete the associated secrets instantly. It waits for its garbage collector process to remove them.

You will first have to delete this chart, wait for cert-manager to do garbage collection, and then install the latest version of this chart. With the assumption your Helm release is named "splunk-otel-collector", we show the commands to run below.

  • Be aware these steps likely include the operator being unavailable and having down time for this service in your environment.

Step 1: Delete this Helm Chart

Use a command like this to delete the chart in your namespace:

helm uninstall splunk-otel-collector --namespace <your_namespace>

Step 2: Verify If the Old Cert Manager Secret Does Not Exists Anymore

Use the following command to check if the certificate secret remains in your namespace:

kubectl get secret splunk-otel-collector-operator-controller-manager-service-cert --namespace <your_namespace>

Step 3: Wait for Secret Removal or Manually Delete It

If the secret still exists, you must wait for cert-manager to remove it or delete it manually:

kubectl delete secret splunk-otel-collector-operator-controller-manager-service-cert --namespace <your_namespace>

Step 4: Proceed with Helm Install

Once the secret is no longer present, you can install the chart with the latest version (0.120.0) successfully:

helm install splunk-otel-collector splunk-otel-collector-chart/splunk-otel-collector --values ~/values.yaml --namespace <your_namespace>

Step 5 (Optional): Delete cert-manager CRDs

Helm delete will not remove CRDs objects created as part of the cert-manager installation. You can find the command to delete cert-manager CRDs in their official documentation here.

0.113.0 to 0.116.0

This guide provides steps for new users, transitioning users, and those maintaining previous operator CRD configurations:

  • New users: No migration for CRDs is required.
  • Previous users: Migration may be needed if using operator.enabled=true.

CRD deployment has evolved over chart versions:

  • Before 0.110.0: CRDs were deployed via a crds/ directory (upstream default).
  • 0.110.0 to 1.113.0: CRDs were deployed using Helm templates (upstream default), which had reported issues.
  • 0.116.0 and later: Users must now explicitly configure their preferred CRD deployment method or deploy the CRDs manually to avoid potential issues. Users can deploy CRDs via a crds/ directory again by enabling a newly added value.

New Users

New users are advised to deploy CRDs via the crds/ directory. For a fresh installation, use the following Helm values:

operatorcrds:
  install: true
operator:
  enabled: true

To install the chart:

helm install <release-name> splunk-otel-collector-chart/splunk-otel-collector --set operatorcrds.install=true,operator.enabled=true <extra_args>

Current Users (Recommended Migration to crds/ Directory)

If you're using chart versions 0.110.0 to 1.113.0, CRDs are likely deployed via Helm templates. To migrate to the recommended crds/ directory deployment:

Step 1: Delete the Existing Chart

Remove the chart to prepare for a fresh installation:

helm uninstall <release-name>

Step 2: Verify or Remove Existing CRDs

Check if the following CRDs are present and delete them if necessary:

kubectl get crds | grep opentelemetry
kubectl delete crd opentelemetrycollectors.opentelemetry.io
kubectl delete crd opampbridges.opentelemetry.io
kubectl delete crd instrumentations.opentelemetry.io

Step 3: Reinstall with Recommended Values

Reinstall the chart with the updated configuration:

helm install <release-name> splunk-otel-collector --set operatorcrds.install=true,operator.enabled=true <extra_args>

Previous Users (Maintaining Legacy Helm Templates)

If you're using chart versions 0.110.0 to 1.113.0 and prefer to continue deploying CRDs via Helm templates (not recommended), you can do so with the following values:

operator:
  enabled: true
operator:
  crds:
    create: true

Warning: This method may cause race conditions during installation or upgrades, leading to errors like:

ERROR: INSTALLATION FAILED: failed post-install: warning: Hook post-install splunk-otel-collector/templates/operator/instrumentation.yaml failed: 1 error occurred:
* Internal error occurred: failed calling webhook "minstrumentation.kb.io": failed to call webhook: Post "https://splunk-otel-collector-operator-webhook.default.svc:443/mutate-opentelemetry-io-v1alpha1-instrumentation?timeout=10s": dial tcp X.X.X.X:443: connect: connection refused

0.105.5 to 0.108.0

We've simplified the Helm chart configuration for operator auto-instrumentation. The values previously under .Values.operator.instrumentation.spec.* have been moved to .Values.instrumentation.*.

  • No Action Needed: If you have no customizations under .Values.operator.instrumentation.spec.*, no migration is required.
  • Action Required: Continuing to use the old values path will result in a Helm install or upgrade error, blocking the process.

Migration Steps:

  1. Find any references to .Values.operator.instrumentation.spec.* in your Helm values with custom values.
  2. Migrate them from .Values.operator.instrumentation.spec.* to .Values.instrumentation.*.

Example Migration:

Before (Deprecated Path):

operator:
  instrumentation:
    spec:
      endpoint: XXX
      ...

After (Updated Path):

instrumentation:
  endpoint: XXX
  ...

0.105.3 to 0.105.4

The Java instrumentation for Operator auto-instrumentation has been upgraded from v1.32.2 to v2.7.0. This major update introduces several breaking changes. Below we have supplied a customer migration guide and outlined the key changes to highlight the impact.

Please refer to the Migration guide for OpenTelemetry Java 2.x to update your custom dashboards, detectors, or alerts using Java application telemetry data.

Breaking Changes Overview

  • Runtime metrics will now be enabled by default, this can increase the number of metrics collected.
  • The default protocol changed from gRPC to http/protobuf. For custom Java exporter endpoint configurations, verify that you’re sending data to http/protobuf endpoints like this example.
  • Span Attribute Name Changes:
Old Attribute (1.x) New Attribute (2.x)
http.method http.request.method
http.status_code http.response.status_code
http.request_content_length http.request.body.size
http.response_content_length http.response.body.size
http.target url.path and url.query
http.scheme url.scheme
http.client_ip client.address
  • Metric Name Changes:
Old Metric (1.x) New Metric (2.x)
db.pool.connections.create_time db.client.connections.create_time (Histogram, ms)
db.pool.connections.idle.max db.client.connections.idle.max
db.pool.connections.idle.min db.client.connections.idle.min
db.pool.connections.max db.client.connections.max
db.pool.connections.pending_threads db.client.connections.pending_requests
db.pool.connections.timeouts db.client.connections.timeouts
db.pool.connections.idle db.client.connections.usage[state=idle]
db.pool.connections.active db.client.connections.usage[state=used]
db.pool.connections.use_time db.client.connections.use_time (Histogram, ms)
db.pool.connections.wait_time db.client.connections.wait_time (Histogram, ms)
runtime.jvm.buffer.count jvm.buffer.count
runtime.jvm.buffer.total.capacity jvm.buffer.memory.limit
runtime.jvm.buffer.memory.used jvm.buffer.memory.usage
runtime.jvm.classes.loaded jvm.class.count
runtime.jvm.classes.unloaded jvm.class.unloaded
runtime.jvm.gc.concurrent.phase.time jvm.gc.duration (Histogram, )
runtime.jvm.gc.pause jvm.gc.duration ()
runtime.jvm.gc.memory.allocated | process.runtime.jvm.memory.allocated jvm.memory.allocated*
runtime.jvm.memory.committed jvm.memory.committed
runtime.jvm.memory.max jvm.memory.limit
runtime.jvm.gc.max.data.size jvm.memory.limit{jvm.memory.pool.name=}
runtime.jvm.memory.used jvm.memory.used
runtime.jvm.gc.live.data.size jvm.memory.used_after_last_gc{jvm.memory.pool.name=}
runtime.jvm.threads.daemon | runtime.jvm.threads.live jvm.thread.count
  • Dropped Metrics:
    • executor.tasks.completed
    • executor.tasks.submitted
    • executor.threads
    • executor.threads.active
    • executor.threads.core
    • executor.threads.idle
    • executor.threads.max
    • runtime.jvm.memory.usage.after.gc
    • runtime.jvm.gc.memory.promoted
    • runtime.jvm.gc.overhead
    • runtime.jvm.threads.peak
    • runtime.jvm.threads.states

0.93.0 to 0.94.0

The networkExplorer option is removed.

0.87.0 to 0.88.0

The networkExplorer option is deprecated now. Please use the upstream OpenTelemetry eBPF Helm chart to collect the network metrics by following the next steps:

  1. Make sure the Splunk OpenTelemetry Collector helm chart is installed with the gateway enabled:
gateway:
  enabled: true
  1. Disable the network explorer:
networkExplorer:
  enabled: false
  1. Grab name of the Splunk OpenTelemetry Collector gateway service:
kubectl get svc | grep splunk-otel-collector-gateway
  1. Install the upstream OpenTelemetry eBPF helm chart pointing to the Splunk OpenTelemetry Collector gateway service:
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update open-telemetry
helm install my-opentelemetry-ebpf -f ./otel-ebpf-values.yaml open-telemetry/opentelemetry-ebpf

otel-ebpf-values.yaml must at least have endpoint.address option set to the Splunk OpenTelemetry Collector gateway service name captured in the step 2. Additionally, if you had any custom configurations in the networkExplorer section, you need to move them to the otel-ebpf-values.yaml file.

endpoint:
  address: <my-splunk-otel-collector-gateway>

# additional custom configuration moved from the networkExplorer section in Splunk OpenTelemetry Collector helm chart.

0.85.0 to 0.86.0

The default logs collection engine (logsEngine) changed from fluentd to the native OpenTelemetry logs collection (otel). If you want to keep using Fluentd sidecar for the logs collection, set logsEngine: fluentd in your values.yaml.

0.84.0 to 0.85.0

The format for defining auto-instrumentation images has been refactored. Previously, the image was defined using the operator.instrumentation.spec.{library}.image format. This has been changed to separate the repository and tag into two distinct fields: operator.instrumentation.spec.{library}.repository and operator.instrumentation.spec.{library}.tag.

If you were defining a custom image under operator.instrumentation.spec.{library}.image, update your values.yaml to accommodate this change.

  • Before:
operator:
  instrumentation:
    spec:
      java:
        image: ghcr.io/custom-owner/splunk-otel-java/custom-splunk-otel-java:v1.27.0
  • After:
operator:
  instrumentation:
    spec:
      java:
        repository: ghcr.io/custom-owner/splunk-otel-java/custom-splunk-otel-java
        tag: v1.27.0

0.67.0 to 0.68.0

There is a new receiver: Kubernetes Objects Receiver that can pull or watch any object from Kubernetes API server.

To use the Kubernetes Object Receiver instead of the Kubernetes Events Receiver to collect k8s events, configure clusterReceiver values.yaml section with:

k8sObjects:
  - mode: watch
    name: events

There are differences in the log record formatting between the previous k8s_events receiver and the now adopted k8sobjects receiver results. The k8s_events receiver stores event messages their log body, with the following fields added as attributes:

  • k8s.object.kind
  • k8s.object.name
  • k8s.object.uid
  • k8s.object.fieldpath
  • k8s.object.api_version
  • k8s.object.resource_version
  • k8s.event.reason
  • k8s.event.action
  • k8s.event.start_time
  • k8s.event.name
  • k8s.event.uid
  • k8s.namespace.name

Now with the k8sobjects receiver, the whole payload is stored in the log body and object.message refers to the event message.

You can monitor more Kubernetes objects configuring by clusterReceiver.k8sObjects according to the instructions from the Kubernetes Objects Receiver documentation.

Remember to define rbac.customRules when needed. For example, when configuring:

objectsEnabled: true
k8sObjects:
  - name: events
    mode: watch
    group: events.k8s.io
    namespaces: [default]

You should add events.k8s.io API group to the rbac.customRules:

rbac:
  customRules:
    - apiGroups:
      - "events.k8s.io"
      resources:
      - events
      verbs:
      - get
      - list
      - watch

0.58.0 to 0.59.0

[receiver/filelogreceiver] Datatype for force_flush_period and poll_interval were changed from map to string.

If you are using custom filelog receiver plugin, you need to change the config from:

filelog:
  poll_interval:
    duration: 200ms
  force_flush_period:
    duration: "0"

to:

filelog:
  poll_interval: 200ms
  force_flush_period: "0"

0.57.1 to 0.58.0

[receiver/filelogreceiver] Datatype for force_flush_period and poll_interval were changed from string to map. Because of that, the default values in Helm Chart were causing problems #519

If you are using custom filelog receiver plugin, you need to change the config from:

filelog:
  poll_interval: 200ms
  force_flush_period: "0"

to:

filelog:
  poll_interval:
    duration: 200ms
  force_flush_period:
    duration: "0"

0.54.0 to 0.55.0

[receiver/k8sclusterreceiver] The receiver.k8sclusterreceiver.reportCpuMetricsAsDouble feature gate has been removed

If you are disabling this feature gate to keep previous functionality, you will have to complete the steps in upgrade guidelines 0.47.0 to 0.47.1 to upgrade since the feature gate no longer exists.

0.53.2 to 0.54.0

OTel Kubernetes receiver is now used for events collection instead of Signalfx events receiver

Before this change, if clusterReceiver.k8sEventsEnabled=true, Kubernetes events used to be collected by a Signalfx receiver and sent both to Splunk Observability Infrastructure Monitoring and Splunk Observability Log Observer. Now we utilize a native OpenTelemetry receiver for collecting Kubernetes events. Therefore clusterReceiver.k8sEventsEnabled option is now deprecated and replaced by the following two options:

  • clusterReceiver.eventsEnabled: to send Kubernetes events in the new OTel format to Splunk Observability Log Observer (if splunkObservability.logsEnabled=true) or to Splunk Platform (if splunkPlatform.logsEnabled=true).
  • splunkObservability.infrastructureMonitoringEventsEnabled: to collect Kubernetes events using the Signalfx Kubernetes events receiver and send them to Splunk Observability Infrastructure Monitoring.

If you have clusterReceiver.k8sEventsEnabled set to true to send Kubernetes events to both Splunk Observability Infrastructure Monitoring and Splunk Observability Log Observer, remove clusterReceiver.k8sEventsEnabled from your custom values.yaml enable both clusterReceiver.eventsEnabled and splunkObservability.infrastructureMonitoringEventsEnabled options. This will send the Kubernetes events to Splunk Observability Log Observer in the new OpenTelemetry format.

If you want to keep sending Kubernetes events to Splunk Observability Log Observer in the old Signalfx format to keep exactly the same behavior as before, remove clusterReceiver.k8sEventsEnabled from your custom values.yaml and add the following configuration:

splunkObservability:
  logsEnabled: true
  infrastructureMonitoringEventsEnabled: true
clusterReceiver:
  config:
    exporters:
      splunk_hec/events:
        endpoint: https://ingest.<SPLUNK_OBSERVABILITY_REALM>.observability.splunkcloud.com/v1/log
        log_data_enabled: true
        profiling_data_enabled: false
        source: kubelet
        sourcetype: kube:events
        token: ${SPLUNK_OBSERVABILITY_ACCESS_TOKEN}
    service:
      pipelines:
        logs/events:
          exporters:
            - signalfx
            - splunk_hec/events

where SPLUNK_OBSERVABILITY_REALM must be replaced by splunkObservability.realm value.

0.48.0 to 0.49.0

New releases of opentelemetry-log-collection ( v0.29.0, v0.28.0 ) have breaking changes

Several of the logging receivers supported by the Splunk Otel Collector Chart were updated to use v0.29.0 instead v0.27.2 of opentelemetry-log-collection.

  1. Check to see if you have any custom log monitoring setup with the extraFileLogs config, the logsCollection.containers.extraOperators config, or any of the affected receivers. If you don't have any custom log monitoring setup, you can stop here.
  2. Read the documentation for upgrading to opentelemetry-log-collection v0.29.0.
  3. If opentelemetry-log-collection v0.29.0 or v0.28.0 will break any of your custom log monitoring, update your log monitoring to accommodate the breaking changes.

[receiver/k8sclusterreceiver] The receiver.k8sclusterreceiver.reportCpuMetricsAsDouble feature gate is now enabled by default

If you haven't already completed the steps in upgrade guidelines 0.47.0 to 0.47.1 , then complete them.

0.47.0 to 0.47.1

[receiver/k8sclusterreceiver] Fix k8s node and container cpu metrics not being reported properly

The Splunk Otel Collector added a feature gate to enable a bug fix for three metrics. These metrics have a current and a legacy name, we list both as pairs (current, legacy) below.

  • Affected Metrics
    • k8s.container.cpu_request, kubernetes.container_cpu_request
    • k8s.container.cpu_limit, kubernetes.container_cpu_limit
    • k8s.node.allocatable_cpu, kubernetes.node_allocatable_cpu
  • Upgrade Steps
    1. Check to see if any of your custom monitoring uses the affected metrics. Check for the current and legacy names of the affected metrics. If you don't use the affected metrics in your custom monitoring, you can stop here.
    2. Read the documentation for the receiver.k8sclusterreceiver.reportCpuMetricsAsDouble feature gate and the bug fix it applies.
    3. If the bug fix will break any of your custom monitoring for the affected metrics, update your monitoring to accommodate the bug fix.
  • Feature Gate Stages and Versions
    • Alpha (versions 0.47.1-0.48.0):
      • The feature gate is disabled by default. Use the --set clusterReceiver.featureGates=receiver.k8sclusterreceiver.reportCpuMetricsAsDouble argument with the helm install/upgrade command, or add the following line to your custom values.yaml to enable the feature gate:
      clusterReceiver:
  featureGates: receiver.k8sclusterreceiver.reportCpuMetricsAsDouble
    • Beta (versions 0.49.0-0.54.0):
      • The feature gate is enabled by default. Use the --set clusterReceiver.featureGates=-receiver.k8sclusterreceiver.reportCpuMetricsAsDouble argument with the helm install/upgrade command, or add the following line to your custom values.yaml to disable the feature gate:
      clusterReceiver:
  featureGates: -receiver.k8sclusterreceiver.reportCpuMetricsAsDouble
    • Generally Available (versions +0.55.0):
      • The receiver.k8sclusterreceiver.reportCpuMetricsAsDouble feature gate functionality is permanently enabled and the feature gate is no longer available for anyone.

0.44.1 to 0.45.0

[receiver/k8sclusterreceiver] Use newer batch and autoscaling APIs

Kubernetes clusters with version 1.20 stopped having active support on 2021-12-28 and had an end of life date on 2022-02-28. The k8s_cluster receiver was refactored to use newer Kubernetes APIs that are available starting in Kubernetes version 1.21. The latest version of the k8s_cluster receiver will no longer be able to collect all the previously available metrics with Kubernetes clusters that have versions below 1.21.

If version 0.45.0 of the chart cannot collect metrics from your Kubernetes cluster that is a version below 1.21, you will see error messages in your cluster receiver logs that look like this.

Failed to watch *v1.CronJob: failed to list *v1.CronJob: the server could not find the requested resource

To better support users, in a future release we are adding a feature that will allow users to use the last version of the k8s_cluster receiver that supported Kubernetes clusters below version 1.21.

If you still want to keep the previous behavior of the k8s_cluster receiver and upgrade to v0.45.0 of the chart, make sure your Kubernetes cluster uses one of the following versions.

  • kubernetes, aks, eks, eks/fargate, gke, gke/autopilot
    • Use version 1.21 or above
  • openshift
    • Use version 4.8 or above

0.43.1 to 0.43.2

#375 Resource detection processor is configured to override all host and cloud attributes

If you still want to keep the previous behavior, use the following custom values.yaml configuration:

agent:
  config:
    processors:
      resourcedetection:
        override: false

0.41.0 to 0.42.0

#357 Double expansion issue in splunk-otel-collector is fixed

If you use OTel native logs collection with any custom log processing operators in filelog receiver, please replace any occurrences of $$$$ with $$.

0.38.0 to 0.39.0

#325 Logs collection is now disabled by default for Splunk Observability destination

If you send logs to Splunk Observability destination, make sure to enable logs. Use --set="splunkObservability.logsEnabled=true" argument with helm install/upgrade command, or add the following line to your custom values.yaml:

splunkObservability:
  logsEnabled: true

0.37.1 to 0.38.0

#297, #301 Several parameters in values.yaml configuration were renamed according to Splunk GDI Specification

If you use the following parameters in your custom values.yaml, please rename them accordingly:

  • provider -> cloudProvider
  • distro -> distribution
  • otelAgent -> agent
  • otelCollector -> gateway
  • otelK8sClusterReceiver -> clusterReceiver

#306 Some parameters under splunkPlatform group were renamed

If you use the following parameters under splunkPlatform group, please make sure they are updated:

  • metrics_index -> metricsIndex
  • max_connections -> maxConnections
  • disable_compression -> disableCompression
  • insecure_skip_verify -> insecureSkipVerify

#295 Secret names are changed according to the GDI specification

If you provide access token for Splunk Observability using a custom Kubernetes secret (secter.create=false), please update the secret key from splunk_o11y_access_token to splunk_observability_access_token

#273 Changed configuration to fetch attributes from labels and annotations of pods and namespaces

podLabels parameter under the extraAttributes group is now deprecated. in favor of fromLabels. Please update your custom values.yaml accordingly.

For example, the following config:

extraAttributes:
  podLabels:
    - app
    - git_sha

Should be changed to:

extraAttributes:
  fromLabels:
    - key: app
    - key: git_sha

#316 Busybox dependency is removed, splunk/fluentd-hec image is used in init container instead

image.fluentd.initContainer is not being used anymore. Please remove it from your custom values.yaml.

0.36.2 to 0.37.0

#232 Access to underlying node's filesystem was reduced to the minimum scope required for default functionality: host metrics and logs collection

If you have any extra receivers that require access to node's files or directories that are not mounted by default, you need to setup additional volume mounts.

For example, if you have the following smartagent/docker-container-stats receiver added to your configuration:

agent:
  config:
    receivers:
      smartagent/docker-container-stats:
        type: docker-container-stats
        dockerURL: unix:///hostfs/var/run/docker.sock

You need to mount the docker socket to your container as follows:

  extraVolumeMounts:
    - mountPath: /hostfs/var/run/docker.sock
      name: host-var-run-docker
      readOnly: true
  extraVolumes:
    - name: host-var-run-docker
      hostPath:
        path: /var/run/docker.sock

#246 Simplify configuration for switching to native OTel logs collection

The config to enable native OTel logs collection was changed from

fluentd:
  enabled: false
logsCollection:
  enabled: true

to

logsEngine: otel

Enabling both engines is not supported anymore. If you need that, you can install fluentd separately.

0.35.3 to 0.36.0

#209 Configuration interface changed to support both Splunk Enterprise/Cloud and Splunk Observability destinations

The following parameters are now deprecated and moved under splunkObservability group. They need to be updated in your custom values.yaml files before backward compatibility is discontinued.

Required parameters:

  • splunkRealm changed to splunkObservability.realm
  • splunkAccessToken changed to splunkObservability.accessToken

Optional parameters:

  • ingestUrl changed to splunkObservability.ingestUrl
  • apiUrl changed to splunkObservability.apiUrl
  • metricsEnabled changed to splunkObservability.metricsEnabled
  • tracesEnabled changed to splunkObservability.tracesEnabled
  • logsEnabled changed to splunkObservability.logsEnabled

0.26.4 to 0.27.0

#163 Auto-detection of prometheus metrics is disabled by default: If you rely on automatic prometheus endpoints detection to scrape prometheus metrics from pods in your k8s cluster, make sure to add this configuration to your values.yaml:

autodetect:
  prometheus: true