spark-BlockManager-ShuffleMetricsSource.adoc

ShuffleMetricsSource — Metrics Source of BlockManager for Shuffle-Related Metrics

ShuffleMetricsSource is the metrics source of a BlockManager for shuffle-related metrics.

ShuffleMetricsSource lives on a Spark executor and is registered only when a Spark application runs in a non-local / cluster mode.

Figure 1. Registering ShuffleMetricsSource with "executor" MetricsSystem

ShuffleMetricsSource is created exclusively when BlockManager is requested for the shuffleMetricsSource.

When created, ShuffleMetricsSource gets a MetricSet that BlockManager requests from the ShuffleClient (only when in a non-local / cluster mode).

ShuffleMetricsSource is registered under the following source name per the type of a BlockManager:

• NettyBlockTransfer when spark.shuffle.service.enabled configuration property is off (false)

• ExternalShuffle when spark.shuffle.service.enabled configuration property is on (true)

Note	spark.shuffle.service.enabled configuration property is off (false) by default.

Table 1. ShuffleMetricsSource’s Gauge Metrics (in alphabetical order)

Name	Type	Description

Note	Since Executor does not have a web UI attached you cannot access the metrics using the HTTP protocol (through MetricsServlet JSON metrics sink).

Creating ShuffleMetricsSource Instance

ShuffleMetricsSource takes the following when created:

• Source name

• Dropwizard Metrics' MetricSet