Repository navigation
Add structured logs and Prometheus metrics for blobstore and packer operations - #5575
WeiQuan0605 wants to merge 6 commits into
Conversation
…lobstore and packer operations
stephanme
left a comment
There was a problem hiding this comment.
There are number of unrelated files touched in this PR.
…invocation Log one structured entry per storage-cli subprocess call on cc.blobstore.storage_cli_client with operation, bucket, resource_type, operation-specific fields (key/source_path/destination_key etc.), duration_seconds, exit_code, and success. Linked to vcap-request-id so all storage-cli calls for a CF API request are traceable in Kibana, comparable to CCDB statement logs. Remove unrelated delayed_jobs_log_redaction changes These changes belong to PR #5513 (query log redaction), not this observability PR.
a2c5ff2 to
7eb72ea
Compare
- Add operation and key args to storage_cli.call.finish message for quick scanning - Use labeled args (src=, dest=, key=, verb=, expires_in=) for clarity - Use File.basename to keep message concise without losing context - Log spawn errors with storage_cli.call.error including duration and error detail - Fix symbol key serialization bug in operation_log_fields (was logging :operation instead of operation) Revert user_audit_info changes unrelated to this PR
7eb72ea to
b8427c3
Compare
stephanme
left a comment
There was a problem hiding this comment.
I had some conversation with Claude how to centralise the metrics in run_cli.
Make run_cli the single component that guarantees "exactly one metric per invocation, on every exit path." Wrap the subprocess call in one begin/rescue with a local emitted guard. Then cp_to_blobstore keeps only the one thing that is genuinely not a CLI call: the skip.
It would make obsolete: new_cp_state, mark_run_cli_called!, mark_run_cli_succeeded!, mark_skip_metric_emitted!, the three predicate readers, perform_put_or_skip, run_put_with_fatal_rescue, handle_fatal_put_exception, emit_signal_metric_if_needed, signal_exception?, and capture_cli — roughly 70 lines of coordination logic for one net metric (the skip).
Let's discuss this offline.
|
|
||
| def capture_cli(command, args, start_time) | ||
| Open3.capture3(@cli_path, '-s', @storage_type, '-c', @config_file, *additional_flags, command, *args) | ||
| rescue StandardError => e |
There was a problem hiding this comment.
Metrics and logs are not written on Signals only on StandardError.
| fields | ||
| end | ||
|
|
||
| def capture_cli(command, args, start_time) |
There was a problem hiding this comment.
Naming: maybe better run_and_capture_cli
Might be my problem as I'm not too deep into the ruby open3 lib. But the program flow was more difficult to understand (where is storage_cli called in run_cli()?)
| return 'cp-skip' | ||
| end | ||
|
|
||
| mark_run_cli_called!(state) |
There was a problem hiding this comment.
mark_run_cli_called! is set before the CLI is actually invoked, so a pre-CLI StandardError suppresses the put failure/-3 metric.
Failure scenario: perform_put_or_skip calls mark_run_cli_called!(state) before run_put_with_fatal_rescue, i.e. before partitioned_key(destination_key) and before run_cli/Open3 ever run. If partitioned_key (or any code before Open3) raises a StandardError, run_put_with_fatal_rescue's rescue StandardError re-raises on the assumption 'run_cli already emitted a metric' (it did not), and cp_to_blobstore's rescue StandardError skips the -3 emit because run_cli_called?(state) is true. The put failure is logged as cp-error but no storage_cli metric is emitted at all — 'run_cli_called' actually means 'passed the size check', not 'CLI invoked'.
b3fd872 to
f479098
Compare
| success = status.success? || (allow_exit_code_three && status.exitstatus == 3) | ||
| metric_exit_code = allow_exit_code_three && exit_code == 3 ? 0 : exit_code | ||
| safe_emit_metrics(command, metric_exit_code, duration, 'run_cli', blob_size: blob_size) | ||
| logger.info("storage_cli.call.finish #{command}#{log_message_arg_suffix(command, args)}", |
There was a problem hiding this comment.
Do we really want to log every successful runs, wouldn't it be too noisy. Since we already have cf-finish, cp-finish, cf-skip and cf-error in cp_to_blobstore method
There was a problem hiding this comment.
storage_cli.call.finish serves a different purpose than cp-finish:
cp-finishis one entry per upload decision, it records what cp_to_blobstore did (skipped or uploaded), with file size and total duration.storage_cli.call.finishis one entry per CLI subprocess call, it records the actual exit code, subprocess duration, bucket, and resource type. This is the log that lets you correlate a Prometheus metric spike to a specific call.
For operations like exists, delete, sign, copy, there is no higher-level log at all. storage_cli.call.finish is the only trace.
@stephanme also explicitly requested this log in an earlier comment.
If volume is a concern, we can move success logs to debug level, open to discuss.
There was a problem hiding this comment.
The idea was to have storage-cli logs similar to CCDB stmt logs. Can become verbose but we see if there are too many calls per CF API request so that we can optimise e.g. by adding bulk endpoints.
We should be able to configure the log level per logger. IIRC, this is possible for several logs (like DB stmts).
| VCAP::CloudController::ExecutionContext::API_PUMA_WORKER | ||
| ].freeze | ||
|
|
||
| STORAGE_CLI_OPERATIONS_TOTAL_DOCSTRING = |
There was a problem hiding this comment.
so i see that there are a lot of magic numbers are used in the code (e.g. inside def mid_execution_exit_code(capture3_called)), so would not be better to create a constant module and using constants in the code and would be much more descriptive, here is generated example
module CloudController
module Blobstore
# Named exit codes reported as the `exit_code` metric label for storage-cli
# invocations. Non-negative values are real process exit statuses; negative
# values are sentinels for failures where no genuine process exit status exists.
module StorageCliExitCode
SUCCESS = 0
NOT_FOUND = 3
SPAWN_FAILURE = -1
PLATFORM_ANOMALY = -2
PRE_SPAWN_ERROR = -3
SIZE_SKIP = -4
SIGNAL_PRE_SPAWN = -5
SIGNAL_MID_EXEC = -6
FATAL_NO_STATUS = -7
DESCRIPTIONS = {
SUCCESS => 'success',
NOT_FOUND => 'not-found (expected for exists)',
SPAWN_FAILURE => 'spawn failure (binary missing or permission denied) after setup succeeds',
PLATFORM_ANOMALY => 'process exited with neither exitstatus nor termsig (platform anomaly)',
PRE_SPAWN_ERROR => 'pre-spawn StandardError (config load, flags, or source file read failure)',
SIZE_SKIP => 'upload skipped locally because blob size is outside configured limits',
SIGNAL_PRE_SPAWN => 'Interrupt or SignalException before Open3.capture3 was called',
SIGNAL_MID_EXEC => 'signal received after CLI invocation began (Interrupt or SignalException mid-execution)',
FATAL_NO_STATUS => 'fatal non-signal exception escaping run_cli (e.g. NoMemoryError)'
}.freeze
def self.docstring(prefix)
"#{prefix} " + DESCRIPTIONS.map { |code, desc| "exit_code=#{code}: #{desc}" }.join('; ')
end
end
end
end
There was a problem hiding this comment.
The sentinel codes are documented in the STORAGE_CLI_OPERATIONS_TOTAL_DOCSTRING in prometheus_updater.rb, each value has a named description there. Since each code appears only once in the implementation, inline constants would add lines without improving readability much.
| counters[:upload_duration] += Time.now.utc - t | ||
| end | ||
|
|
||
| def log_cp_r_phases(ctrs, completed) |
There was a problem hiding this comment.
Consider refactoring this function, the math calculations are mixed into the logging, which makes it a little hard to follow. Here is example refactored code might look like:
def log_cp_r_phases(ctrs, completed)
logger.info('blobstore.cp_r.digest',
files_scanned: ctrs[:digest_count],
duration_seconds: ctrs[:digest_duration],
completed: completed)
log_exists_phase(ctrs, completed)
ua = ctrs[:upload_attempts]
logger.info('blobstore.cp_r.upload',
files_to_upload: ua,
files_uploaded: ctrs[:upload_count],
files_failed: ua - ctrs[:upload_count],
duration_seconds: ctrs[:upload_duration],
completed: completed)
end
def log_exists_phase(ctrs, completed)
checks = ctrs[:exists_count]
attempts = ctrs[:upload_attempts]
errors = ctrs[:exists_errors]
hits = checks - attempts - errors
resolved = checks - errors # checks that did not error
logger.info('blobstore.cp_r.exists',
files_checked: checks,
duration_seconds: ctrs[:exists_duration],
avg_duration_seconds: safe_div(ctrs[:exists_duration], checks),
cache_hits: hits,
cache_misses: attempts,
cache_hit_rate: safe_div(hits, resolved),
process_spawns: checks,
error_count: errors,
completed: completed)
end
def safe_div(num, denom) = denom.positive? ? num.to_f / denom : 0
There was a problem hiding this comment.
Worth doing. I will do the refactor.
| require 'prometheus/client/data_stores/direct_file_store' | ||
| prometheus_dir = File.join(RakeConfig.config.get(:directories, :tmpdir), 'prometheus') | ||
| FileUtils.mkdir_p(prometheus_dir) | ||
| Prometheus::Client.config.data_store = Prometheus::Client::DataStores::DirectFileStore.new(dir: prometheus_dir) |
There was a problem hiding this comment.
Shared metrics files can be overwritten.
Local BPM workers share one Prometheus directory while having identical container PIDs. Unlike generic workers, this setup provides no unique process identity.
| sha256: Digester.new(algorithm: OpenSSL::Digest::SHA256).digest_path(complete_package_path) | ||
| } | ||
| ensure | ||
| logger.info('packer.pack', duration_seconds: Time.now.utc - pack_start, blobstore_key: blobstore_key) |
There was a problem hiding this comment.
If logging raises (steno doesn't catch exceptions), the directory is not removed.
| logger.error("storage_cli.call.error #{command}#{log_message_arg_suffix(command, args)}", | ||
| log_fields(command, args, blob_size: blob_size).merge('duration_seconds' => duration, 'error' => e.inspect)) | ||
| raise BlobstoreError.new(e.inspect) | ||
| ensure |
There was a problem hiding this comment.
Cancellation can become a retry
If logging in this ensure block raises an IOError while this ensure block runs for a propagating Interrupt, the new IOError will replace the original Interrupt. The caller (RetryableClient) will retry the operation on an IOError instead of stopping.
Logging and telemetry need to become "non-throwing".
A short explanation of the proposed change:
Add structured log entries and Prometheus metrics to blobstore and packer operations.
cp_to_blobstorenow emits cp-start, cp-finish, cp-skip, and cp-error log events with destination key, duration, and blob size.cp_r_to_blobstoreemits per-phase timing logs (blobstore.cp_r.digest, .exists, .upload). Three new Prometheus metrics track storage-cli subprocess invocations:cc_storage_cli_operations_total,cc_storage_cli_duration_seconds, andcc_storage_cli_blob_size_bytes.An explanation of the use cases your change solves
Blobstore upload failures are currently silent: operators cannot tell from logs whether a droplet/package upload was attempted, skipped, or how long it took. This change makes every upload observable: structured logs allow filtering by destination key or bucket in Kibana, and Prometheus metrics enable alerting on non-zero exit codes and latency spikes across operations (put, get, exists, delete, etc.).
I have reviewed the contributing guide
I have viewed, signed, and submitted the Contributor License Agreement
I have made this pull request to the
mainbranchI have run all the unit tests using
bundle exec rakeI have run CF Acceptance Tests