Improve InvalidSequenceTokenException Being Logged Frequently

[ingshtrom]

Problem

I know there are a lot of other tickets talking about this, but they either accept the log messages/metrics, or are very old.

We are getting a lot of InvalidSequenceTokenException errors in our logs and then our Prometheus metrics are also filled with these errors. I believe they are recovering fine with the retries that have been put in place, but I'm curious if there is a better way to work around this?

We use flush_threads: 4 in our buffer configuration and then run two replicas of this FluentD configuration. It seems that the only way to fix this is to run a single replica with a single thread, but then that wouldn't be redundant 😞

Lastly, I am curious what the difference between the concurrency parameter and the Buffer's parameter flush_threads is. They sound like they are doing the same thing, but maybe I should be using one over the other?

Steps to replicate

# that is a Kubernetes Pod's logs
# These are aggregated by a fluent-bit running on each
# Kubernetes node, and then forwarded to central processing, 
# which includes this configuration snippet

# NOTE: I have excluded prometheus and other non-essential pieces of the config

<source>
  @type forward
  port 24284
  bind 0.0.0.0
  tag pod.source
  @label @POD_SOURCE
</source>

<label @POD_SOURCE>
  <filter **>
    @type record_transformer
    enable_ruby true
    <record>
      namespace ${record["kubernetes"]["namespace_name"]}
      pod ${record["kubernetes"]["pod_name"]}
    </record>
  </filter>
  <match **>
    @type rewrite_tag_filter
    <rule>
      key     namespace
      pattern /(.+)/
      tag     $1
    </rule>
    @label @POD_STEP2
  </match>
</label>

<label @POD_STEP2>
  <match **>
    @type rewrite_tag_filter
    <rule>
      key     pod
      pattern /(.+)/
      tag     ${tag}_$1
    </rule>
    @label @POD_OUTPUT
  </match>
</label>

<label @POD_OUTPUT>
  <match **>
    @type copy
    <store>
      @type s3
      s3_bucket foobar
      s3_region us-east-1
      s3_object_key_format "#{ENV['ENVIRONMENT']}/eks-pod-logs/%Y-%m-%d/${tag}/%H_%{index}_%{uuid_flush}.%{file_extension}"
      <format>
        @type json
      </format>
      <buffer tag,time>
        timekey 1h
        @type memory
        flush_mode interval
        retry_type exponential_backoff
        flush_thread_count 4
        flush_interval 5s
        retry_forever false
        retry_max_interval 30
        chunk_limit_size 8MB
        chunk_full_threshold 0.90
        overflow_action throw_exception
        compress gzip
      </buffer>
    </store>
    <store>
      @type cloudwatch_logs
      log_group_name /infra/logs/eks/pods/stage
      log_stream_name %Y-%m-%d-%H-${tag}
      auto_create_stream true
      region us-east-1
      <buffer tag, time>
        timekey 1m
        @type memory
        flush_mode interval
        retry_type exponential_backoff
        flush_thread_count 4
        flush_interval 5s
        retry_forever false
        retry_max_interval 30
        chunk_limit_size 8MB
        chunk_full_threshold 0.90
        overflow_action throw_exception
        compress gzip
      </buffer>
    </store>
  </match>
</label>

Expected Behavior or What you need to ask

The s3 logging works just fine, but cloudwatch has a lot of errors about the token being out of sequence. This might come down to how AWS implements their services, but I feel like there must be a better way than retrying a bunch and having logs and metrics be filled up with errors, but maybe that would require a lot more work than it is worth? 🤷 Any suggestions are welcome, too.

Using Fluentd and CloudWatchLogs plugin versions

• OS version: Docker image fluentd:v1.9.1-1.0
• Bare Metal or within Docker or Kubernetes or others? within Kubernetes in AWS EKS
• Fluentd v0.12 or v0.14/v1.0
  • paste result of fluentd --version or td-agent --version => fluentd 1.9.1
• Dependent gem versions
  • paste boot log of fluentd or td-agent
  • paste result of fluent-gem list, td-agent-gem list or your Gemfile.lock

/ $ fluent-gem list

*** LOCAL GEMS ***

async (1.25.0)
async-http (0.50.0)
async-io (1.27.7)
async-pool (0.3.0)
aws-eventstream (1.1.0)
aws-partitions (1.366.0)
aws-sdk-cloudwatchlogs (1.36.0)
aws-sdk-core (3.105.0)
aws-sdk-kms (1.37.0)
aws-sdk-s3 (1.79.1)
aws-sdk-sqs (1.32.0)
aws-sigv4 (1.2.2)
bigdecimal (1.4.4)
cmath (default: 1.0.0)
concurrent-ruby (1.1.6)
console (1.8.2)
cool.io (1.6.0)
csv (default: 1.0.0)
date (default: 1.0.0)
etc (default: 1.0.0)
ext_monitor (0.1.2)
fcntl (default: 1.0.0)
fileutils (default: 1.0.2)
fluent-config-regexp-type (1.0.0)
fluent-plugin-cloudwatch-logs (0.10.2)
fluent-plugin-prometheus (1.8.3)
fluent-plugin-rewrite-tag-filter (2.3.0)
fluent-plugin-s3 (1.4.0)
fluentd (1.9.1)
http_parser.rb (0.6.0)
ipaddr (default: 1.2.0)
jmespath (1.4.0)
json (2.3.0)
msgpack (1.3.3)
nio4r (2.5.2)
oj (3.8.1)
openssl (default: 2.1.2)
prometheus-client (0.9.0)
protocol-hpack (1.4.2)
protocol-http (0.13.1)
protocol-http1 (0.10.2)
protocol-http2 (0.10.4)
psych (default: 3.0.2)
quantile (0.2.1)
scanf (default: 1.0.0)
serverengine (2.2.1)
sigdump (0.2.4)
stringio (default: 0.0.1)
strptime (0.2.3)
strscan (default: 1.0.0)
timers (4.3.0)
tzinfo (2.0.2)
tzinfo-data (1.2019.3)
webrick (default: 1.4.2)
yajl-ruby (1.4.1)
zlib (default: 1.0.0)

[Watson1978]

Thanks for the detailed report — here's a summary of what's going on and how I'd treat this.

What causes it

InvalidSequenceTokenException is raised when more than one writer sends PutLogEvents to the same log stream concurrently. Under the old CloudWatch Logs API every call had to carry the current sequence_token, and only one in-flight request could hold the valid token at a time. With 2 Fluentd replicas × flush_thread_count: 4, several threads race for the same stream's token, so some requests come back with this error. The plugin rescues it, picks up the correct token from err.expected_sequence_token, and retries — which is why your data still arrives and you see retry succeeded. So it is noisy, but it is not data loss.

concurrency vs buffer flush_thread_count

• flush_thread_count (Fluentd buffer): how many flush threads call the output's write in parallel, one chunk each.
• concurrency (this plugin, default 1): within a single write, events are grouped by log group/stream and up to concurrency worker threads call PutLogEvents in parallel.

Both increase the parallelism against the same stream, so raising either tends to increase token contention rather than reduce it.

The important update

In January 2023 AWS removed the sequence-token requirement, and CloudWatch Logs no longer returns InvalidSequenceTokenException regardless of the token supplied (tokens are still accepted for backward compatibility). See: https://aws.amazon.com/about-aws/whats-new/2023/01/amazon-cloudwatch-logs-log-stream-transaction-quota-sequencetoken-requirement/

On any reasonably current aws-sdk-cloudwatchlogs these warnings should stop on their own. Your report is on a pre-2023 stack (fluent-plugin-cloudwatch-logs 0.10.2, aws-sdk-cloudwatchlogs 1.36.0), which is why you were seeing them. The recommended fix is simply to upgrade aws-sdk-cloudwatchlogs and the plugin.

If you must stay on the old stack and want to quiet the warnings

Avoid multiple concurrent writers per stream — either lower the parallelism, or give each source its own stream so every stream has a single writer, for example:

<match your.tag>
  @type cloudwatch_logs
  log_group_name my-group
  use_tag_as_stream true   # one stream per tag => single writer per stream
  auto_create_stream true
  concurrency 1
  <buffer>
    flush_thread_count 1
  </buffer>
</match>

Given the upstream AWS change, I'll close this as resolved; please reopen if you still reproduce it on a current SDK/endpoint.