Skip to content

Add structured logs and Prometheus metrics for blobstore and packer operations - #5575

Open
WeiQuan0605 wants to merge 6 commits into
mainfrom
observability-blobstore-packing-logs
Open

WeiQuan0605 wants to merge 6 commits into
mainfrom
observability-blobstore-packing-logs

Conversation

@WeiQuan0605

Copy link
Copy Markdown
Contributor
  • A short explanation of the proposed change:
    Add structured log entries and Prometheus metrics to blobstore and packer operations. cp_to_blobstore now emits cp-start, cp-finish, cp-skip, and cp-error log events with destination key, duration, and blob size. cp_r_to_blobstore emits per-phase timing logs (blobstore.cp_r.digest, .exists, .upload). Three new Prometheus metrics track storage-cli subprocess invocations: cc_storage_cli_operations_total, cc_storage_cli_duration_seconds, and cc_storage_cli_blob_size_bytes.

  • An explanation of the use cases your change solves
    Blobstore upload failures are currently silent: operators cannot tell from logs whether a droplet/package upload was attempted, skipped, or how long it took. This change makes every upload observable: structured logs allow filtering by destination key or bucket in Kibana, and Prometheus metrics enable alerting on non-zero exit codes and latency spikes across operations (put, get, exists, delete, etc.).

  • I have reviewed the contributing guide

  • I have viewed, signed, and submitted the Contributor License Agreement

  • I have made this pull request to the main branch

  • I have run all the unit tests using bundle exec rake

  • I have run CF Acceptance Tests

@stephanme stephanme left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are number of unrelated files touched in this PR.

…invocation

Log one structured entry per storage-cli subprocess call on
cc.blobstore.storage_cli_client with operation, bucket, resource_type,
operation-specific fields (key/source_path/destination_key etc.),
duration_seconds, exit_code, and success. Linked to vcap-request-id
so all storage-cli calls for a CF API request are traceable in Kibana,
comparable to CCDB statement logs.

Remove unrelated delayed_jobs_log_redaction changes

These changes belong to PR #5513 (query log redaction), not this observability PR.
@WeiQuan0605
WeiQuan0605 force-pushed the observability-blobstore-packing-logs branch from a2c5ff2 to 7eb72ea Compare November 6, 2026 09:33
- Add operation and key args to storage_cli.call.finish message for quick scanning
- Use labeled args (src=, dest=, key=, verb=, expires_in=) for clarity
- Use File.basename to keep message concise without losing context
- Log spawn errors with storage_cli.call.error including duration and error detail
- Fix symbol key serialization bug in operation_log_fields (was logging :operation instead of operation)

Revert user_audit_info changes unrelated to this PR
@WeiQuan0605
WeiQuan0605 force-pushed the observability-blobstore-packing-logs branch from 7eb72ea to b8427c3 Compare November 6, 2026 09:39

@stephanme stephanme left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had some conversation with Claude how to centralise the metrics in run_cli.

Make run_cli the single component that guarantees "exactly one metric per invocation, on every exit path." Wrap the subprocess call in one begin/rescue with a local emitted guard. Then cp_to_blobstore keeps only the one thing that is genuinely not a CLI call: the skip.

It would make obsolete: new_cp_state, mark_run_cli_called!, mark_run_cli_succeeded!, mark_skip_metric_emitted!, the three predicate readers, perform_put_or_skip, run_put_with_fatal_rescue, handle_fatal_put_exception, emit_signal_metric_if_needed, signal_exception?, and capture_cli — roughly 70 lines of coordination logic for one net metric (the skip).

Let's discuss this offline.


def capture_cli(command, args, start_time)
Open3.capture3(@cli_path, '-s', @storage_type, '-c', @config_file, *additional_flags, command, *args)
rescue StandardError => e

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Metrics and logs are not written on Signals only on StandardError.

fields
end

def capture_cli(command, args, start_time)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Naming: maybe better run_and_capture_cli
Might be my problem as I'm not too deep into the ruby open3 lib. But the program flow was more difficult to understand (where is storage_cli called in run_cli()?)

return 'cp-skip'
end

mark_run_cli_called!(state)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mark_run_cli_called! is set before the CLI is actually invoked, so a pre-CLI StandardError suppresses the put failure/-3 metric.

Failure scenario: perform_put_or_skip calls mark_run_cli_called!(state) before run_put_with_fatal_rescue, i.e. before partitioned_key(destination_key) and before run_cli/Open3 ever run. If partitioned_key (or any code before Open3) raises a StandardError, run_put_with_fatal_rescue's rescue StandardError re-raises on the assumption 'run_cli already emitted a metric' (it did not), and cp_to_blobstore's rescue StandardError skips the -3 emit because run_cli_called?(state) is true. The put failure is logged as cp-error but no storage_cli metric is emitted at all — 'run_cli_called' actually means 'passed the size check', not 'CLI invoked'.

@WeiQuan0605
WeiQuan0605 force-pushed the observability-blobstore-packing-logs branch from b3fd872 to f479098 Compare November 9, 2026 13:56
success = status.success? || (allow_exit_code_three && status.exitstatus == 3)
metric_exit_code = allow_exit_code_three && exit_code == 3 ? 0 : exit_code
safe_emit_metrics(command, metric_exit_code, duration, 'run_cli', blob_size: blob_size)
logger.info("storage_cli.call.finish #{command}#{log_message_arg_suffix(command, args)}",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really want to log every successful runs, wouldn't it be too noisy. Since we already have cf-finish, cp-finish, cf-skip and cf-error in cp_to_blobstore method

@WeiQuan0605 WeiQuan0605 Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

storage_cli.call.finish serves a different purpose than cp-finish:

  • cp-finish is one entry per upload decision, it records what cp_to_blobstore did (skipped or uploaded), with file size and total duration.
  • storage_cli.call.finish is one entry per CLI subprocess call, it records the actual exit code, subprocess duration, bucket, and resource type. This is the log that lets you correlate a Prometheus metric spike to a specific call.

For operations like exists, delete, sign, copy, there is no higher-level log at all. storage_cli.call.finish is the only trace.

@stephanme also explicitly requested this log in an earlier comment.

If volume is a concern, we can move success logs to debug level, open to discuss.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea was to have storage-cli logs similar to CCDB stmt logs. Can become verbose but we see if there are too many calls per CF API request so that we can optimise e.g. by adding bulk endpoints.

We should be able to configure the log level per logger. IIRC, this is possible for several logs (like DB stmts).

VCAP::CloudController::ExecutionContext::API_PUMA_WORKER
].freeze

STORAGE_CLI_OPERATIONS_TOTAL_DOCSTRING =

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so i see that there are a lot of magic numbers are used in the code (e.g. inside def mid_execution_exit_code(capture3_called)), so would not be better to create a constant module and using constants in the code and would be much more descriptive, here is generated example

module CloudController
  module Blobstore
    # Named exit codes reported as the `exit_code` metric label for storage-cli
    # invocations. Non-negative values are real process exit statuses; negative
    # values are sentinels for failures where no genuine process exit status exists.
    module StorageCliExitCode
      SUCCESS          =  0
      NOT_FOUND        =  3
      SPAWN_FAILURE    = -1
      PLATFORM_ANOMALY = -2
      PRE_SPAWN_ERROR  = -3
      SIZE_SKIP        = -4
      SIGNAL_PRE_SPAWN = -5
      SIGNAL_MID_EXEC  = -6
      FATAL_NO_STATUS  = -7

      DESCRIPTIONS = {
        SUCCESS => 'success',
        NOT_FOUND => 'not-found (expected for exists)',
        SPAWN_FAILURE => 'spawn failure (binary missing or permission denied) after setup succeeds',
        PLATFORM_ANOMALY => 'process exited with neither exitstatus nor termsig (platform anomaly)',
        PRE_SPAWN_ERROR => 'pre-spawn StandardError (config load, flags, or source file read failure)',
        SIZE_SKIP => 'upload skipped locally because blob size is outside configured limits',
        SIGNAL_PRE_SPAWN => 'Interrupt or SignalException before Open3.capture3 was called',
        SIGNAL_MID_EXEC => 'signal received after CLI invocation began (Interrupt or SignalException mid-execution)',
        FATAL_NO_STATUS => 'fatal non-signal exception escaping run_cli (e.g. NoMemoryError)'
      }.freeze

      def self.docstring(prefix)
        "#{prefix} " + DESCRIPTIONS.map { |code, desc| "exit_code=#{code}: #{desc}" }.join('; ')
      end
    end
  end
end

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sentinel codes are documented in the STORAGE_CLI_OPERATIONS_TOTAL_DOCSTRING in prometheus_updater.rb, each value has a named description there. Since each code appears only once in the implementation, inline constants would add lines without improving readability much.

counters[:upload_duration] += Time.now.utc - t
end

def log_cp_r_phases(ctrs, completed)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider refactoring this function, the math calculations are mixed into the logging, which makes it a little hard to follow. Here is example refactored code might look like:

def log_cp_r_phases(ctrs, completed)
        logger.info('blobstore.cp_r.digest',
                    files_scanned: ctrs[:digest_count],
                    duration_seconds: ctrs[:digest_duration],
                    completed: completed)

        log_exists_phase(ctrs, completed)

        ua = ctrs[:upload_attempts]
        logger.info('blobstore.cp_r.upload',
                    files_to_upload: ua,
                    files_uploaded: ctrs[:upload_count],
                    files_failed: ua - ctrs[:upload_count],
                    duration_seconds: ctrs[:upload_duration],
                    completed: completed)
      end

def log_exists_phase(ctrs, completed)
        checks   = ctrs[:exists_count]
        attempts = ctrs[:upload_attempts]
        errors   = ctrs[:exists_errors]
        hits     = checks - attempts - errors
        resolved = checks - errors # checks that did not error

        logger.info('blobstore.cp_r.exists',
                    files_checked: checks,
                    duration_seconds: ctrs[:exists_duration],
                    avg_duration_seconds: safe_div(ctrs[:exists_duration], checks),
                    cache_hits: hits,
                    cache_misses: attempts,
                    cache_hit_rate: safe_div(hits, resolved),
                    process_spawns: checks,
                    error_count: errors,
                    completed: completed)
      end

      def safe_div(num, denom) = denom.positive? ? num.to_f / denom : 0

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth doing. I will do the refactor.

Comment thread lib/tasks/jobs.rake
require 'prometheus/client/data_stores/direct_file_store'
prometheus_dir = File.join(RakeConfig.config.get(:directories, :tmpdir), 'prometheus')
FileUtils.mkdir_p(prometheus_dir)
Prometheus::Client.config.data_store = Prometheus::Client::DataStores::DirectFileStore.new(dir: prometheus_dir)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shared metrics files can be overwritten.

Local BPM workers share one Prometheus directory while having identical container PIDs. Unlike generic workers, this setup provides no unique process identity.

sha256: Digester.new(algorithm: OpenSSL::Digest::SHA256).digest_path(complete_package_path)
}
ensure
logger.info('packer.pack', duration_seconds: Time.now.utc - pack_start, blobstore_key: blobstore_key)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If logging raises (steno doesn't catch exceptions), the directory is not removed.

logger.error("storage_cli.call.error #{command}#{log_message_arg_suffix(command, args)}",
log_fields(command, args, blob_size: blob_size).merge('duration_seconds' => duration, 'error' => e.inspect))
raise BlobstoreError.new(e.inspect)
ensure

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cancellation can become a retry

If logging in this ensure block raises an IOError while this ensure block runs for a propagating Interrupt, the new IOError will replace the original Interrupt. The caller (RetryableClient) will retry the operation on an IOError instead of stopping.

Logging and telemetry need to become "non-throwing".

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants