[BUG] Log Links should work with CloudWatch FluentD Out Of the Box

[EngHabu]

Describe the bug

As it stands, users wishing to enable CloudWatch logs will need to go through the AWS Process to do so here and then manually modify the fluent-bit-config configmap to alter the generated log stream to match what Flyte expects.

kubectl edit cm -n amazon-cloudwatch fluent-bit-config 

[OUTPUT]
    Name                cloudwatch_logs
    Match               application.*
    region              ${AWS_REGION}
    log_group_name      /aws/containerinsights/${CLUSTER_NAME}/application
    log_stream_prefix   var.
    auto_create_group   true
    extra_user_agent    container-insights

And then use this configmap for flyte:

  task_logs.yaml: |
    plugins:
      logs:
        cloudwatch-template-uri: 'https://{vars.region}.console.aws.amazon.com/cloudwatch/home?region={vars.region}#logsV2:log-groups/log-group/$252Faws$252Fcontainerinsights$252F<log group name>$252Fapplication$3FlogStreamNameFilter$3Dvar.application.var.log.containers.{{ .podName }}_{{ .namespace }}_{{ .containerName }}'

We should change the default template defined here to be: https://console.aws.amazon.com/cloudwatch/home?region=%s#logsV2:log-groups/log-group/%s$3FlogStreamNameFilter=var.log.containers.{{ .podName }}_{{ .namespace }}_{{ .containerName }}

Expected behavior

1. Follow AWS Guide to deploy FluentD: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html
2. Enable CloudWatch logs in Flyte:

     task_logs.yaml: |
       plugins:
         logs:
           cloudwatch-enabled: true
           cloudwatch-log-group: 'bv-ml-pipelines'
           cloudwatch-region: 'us-east-1'
           kubernetes-enabled: false
   

3. SUCCESS ✔️

Additional context to reproduce

No response

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[Azanul]

I'd like to work on this.

[Azanul]

#take

[Azanul]

#self-assign

[Azanul]

@samhita-alla here