Warnings "stream_name is nil" generated when using the log_stream_name_key option

[anosek-verimatrix]

This is my configuration:

<match *.**>
  @type copy
  <store>
    @type cloudwatch_logs
    log_group_name mygroup
    log_stream_name_key mykey
    auto_create_stream true
    localtime false
    region us-east-2
  </store>
</match>

The logs are sent to CloudWatch and both log group and log stream are created in CloudWatch as expected. The only problem is periodic warnings in the td-agent.log:

2018-02-23 17:01:01 +0000 [info]: #0 flushing all buffer forcedly
2018-02-23 17:01:01 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:02:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:03:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:04:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:05:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:06:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:07:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:08:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:09:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:10:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:11:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:12:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:13:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:14:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:15:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:16:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:17:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:18:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:19:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:20:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:21:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:22:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:23:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:24:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:25:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:26:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:27:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)
2018-02-23 17:28:04 +0000 [warn]: #0 stream_name is nil (group_name=mygroup)

It would be great if the warnings could be fixed. Thanks.

[cosmo0920]

I couldn't reproduce this issue with fluent-plugin-cloudwatch-logs v0.10.0:

<source>
  @type forward
</source>
<filter **>
 @type stdout
</filter>
<match cloudwatch.**>
  @type cloudwatch_logs
  region ap-northeast-1
  aws_key_id <obfuscated>
  aws_sec_key <obfuscated>
  auto_create_stream true
  log_group_name fluentd
  log_stream_name_key mykey
  auto_create_stream true
  message_keys message
  <buffer tag>
    @type memory
  </buffer>
</match>

echo '{"elmo":"3years old", "abby":"a fairy","message":"ｲｪ━━━━━ヽ( ﾟДﾟ)人(ﾟДﾟ )ﾉ━━━━━━ｲ!!🤝 from fluentd!!!!","mykey":"cloudwatchlogs_test_restrict"}' | bundle exec fluent-cat cloudwatch.test    

AWS CloudWatchLogs console:

  | 2020-07-10T15:33:40.302+09:00 | ｲｪ━━━━━ヽ( ﾟДﾟ)人(ﾟДﾟ )ﾉ━━━━━━ｲ!!🤝 from fluentd!!!!
-- | -- | --

[aleskiontherun]

I'm having the same issue using official fluent/fluentd:v1.11-1 docker image. In my case the logs are generated by docker containers and log_stream_name_key is set to a container label. My config is:

<source>
  @type forward
  port 24224
</source>

<filter **>
  @type parser
  format json
  key_name log
  reserve_data true
  hash_value_field data
  emit_invalid_record_to_error false
</filter>

<match docker.**>
  @type cloudwatch_logs
  log_group_name "#{ENV['CLOUDWATCH_LOG_GROUP_NAME']}"
  log_stream_name_key "#{ENV['CLOUDWATCH_LOG_STREAM_KEY']}"
  auto_create_stream true
  <buffer>
    flush_interval "#{ENV['CLOUDWATCH_LOG_FLUSH_INTERVAL']}"
  </buffer>
</match>

<match **>
  @type stdout
</match>

It works fine, the log group and stream are created and logs are getting there, but the fluentd container output is full of [warn]: #0 stream_name is nil (group_name=/my/group)

Example of a docker-compose.yml file with fluentd as a logging driver:

version: '2.3'
services:
  redis:
    hostname: redis.development
    image: redis:alpine
    labels:
      host_name: vagrant
      instance_name: development
      service_name: redis
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        fluentd-async-connect: 'true'
        fluentd-retry-wait: 5s
        labels: instance_name,service_name,host_name
        tag: docker.container-log

[szymonrychu]

For anyone who will stumble on this issue:

1. the software -issue wise- seems like an abandonware.. this issue is from 2018 and nobody even dared to answer with a solution
2. the solution is as follows: it seems, that developers abandonded the log_stream_name_key functionality- instead you should use log_group_name prefix-${tag}- it will get you what you want. How I've found it out? there is similar plugin for fluent-bit and it configuration notes pretty much what I've did: log_stream_prefix: (deprecated) Prefix for the Log Stream name. Setting this to prefix- is the same as setting log_stream_name = prefix-$(tag)

cheers!