Home

Welcome to the Data Skew Intelligence Engine Wiki

This wiki contains the detailed technical documentation for the Data Skew Intelligence Engine, an intelligent, automated framework for detecting and mitigating partition skew in Apache Spark using Machine Learning.

Quick Links to Core Sections

• Architecture and Core Engine
• ML Severity Classification
• Mitigation Algorithms (Salting and Repartitioning)

High-Level System Overview

The engine acts as a middleware pipeline that monitors Spark stage metrics, detects unbalanced partition partitions, and automatically decides the optimal mitigation strategy (Salting vs Repartitioning) using an ML classifier.

Key Mathematical Formulas

We calculate partition health using a multi-factor SkewScore:

$$\text{SkewScore} = w_1 \cdot \text{norm}(\text{skew_ratio}) + w_2 \cdot \text{norm}(\text{CV}) + w_3 \cdot \text{top1_concentration}$$

Where:

• $\text{skew_ratio} = \frac{\text{max_partition_size}}{\text{median_partition_size}}$
• $\text{CV} = \frac{\sigma}{\mu}$ (Coefficient of Variation of partition sizes)
• $\text{top1_concentration} = \frac{\text{max_partition_size}}{\text{total_data_size}}$

Mitigation Strategy Decision Table

Skew Severity	Score Range	Action Required	Mitigation Algorithm
NORMAL	$[0.0, 0.3)$	None	None (Keep default partitioning)
LOW_SKEW	$[0.3, 0.6)$	Moderate	Repartitioning (Spark Shuffle Repartition)
HIGH_SKEW	$[0.6, 1.0]$	Severe	Salting (Salt Keys + Two-Phase Aggregation)