Skip to content

Commit b238d72

Browse files
atxtechbroatxtechbro
authored
feat: implement MLflow observability for LLM-driven CLI workflows with interactive session tracking (#1321)
* feat: add MLflow tracking for automation procedures - Implement MLflow tracking wrappers for close-issue and extract-best-frame - Track parameters, metrics, and artifacts for all procedure runs - Add comprehensive documentation and test script - Provide observability and debugging capabilities Closes #1317 * cleanup: remove deprecated MCP dashboard and apply DRY principle - Remove MCP dashboard (deprecated in experiment #1213) - Delete knowledge/tools/mcp-dashboard.md - Clean up setup.sh commented code - Remove mcp-dashboard-go directory - Apply DRY principle to documentation - Remove duplicate implementation details from README - Point to source files instead of duplicating - Reference knowledge/procedures/ for procedure definitions - Keep implementation details only in code * cleanup: remove knowledge/tools directory entirely - Remove knowledge/tools/mlflow-tracking.md - No files should be in knowledge/tools/ - MLflow documentation lives in tracking/README.md instead * feat: add self-healing MLflow auto-start to setup.sh - Create bin/start-mlflow with spilled coffee principle - Auto-installs MLflow via uv if not present - Checks if already running (idempotent) - Starts in background if needed - Silent when already running - Integrate into setup.sh workflow - Replaces deprecated MCP dashboard location - Automatic startup on source setup.sh - No manual intervention required Implements self-healing infrastructure that ensures MLflow is always available for tracking automation procedures. * refactor: apply DRY principle to MLflow trackers - Remove all business logic from mlflow_tracker.py - Convert to thin wrappers that only track metrics - Remove duplicated implementations (gh commands, worktree logic) - Add clear documentation about wrapper-only nature - Point to knowledge/procedures/ for actual implementations MLflow trackers now follow single source of truth principle: - Procedures define HOW to do things - Trackers only OBSERVE what happened - No duplicated logic between tracking and implementation * feat: add interactive Claude session tracking with MLflow - Create claude-with-tracking wrapper for interactive sessions - Preserves full Claude CLI interactivity (plan mode, permissions) - Captures complete session transcript - Runs Claude normally while logging in background - Add session parser to extract metrics - Commands executed, files modified, git operations - Errors encountered, user interactions - Plan mode activations, tool uses - Full transcript and summary saved to MLflow - Solves the real tracking problem - No more mock tracking with fake success metrics - Actual session data from real Claude executions - Queryable history of all Claude sessions - Works WITH existing workflow, not against it This enables true observability while maintaining the interactive Claude CLI experience we rely on. * cleanup: remove mock tracking code, keep real session tracking - Remove tracking/mlflow_tracker.py (mock functions) - Remove tracking/test_mlflow.py (tests for mocks) - Remove tracking/__init__.py (imported mocks) - Update README to focus on real session tracking These mock functions didn't actually track anything real, just returned fake success metrics. Our real value is the interactive Claude session tracking that preserves workflow while adding observability.
1 parent 2027306 commit b238d72

6 files changed

Lines changed: 568 additions & 37 deletions

File tree

‎bin/claude-with-tracking‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
#!/bin/bash
2+
# claude-with-tracking - Run Claude CLI with MLflow session tracking
3+
#
4+
# This wrapper preserves Claude's full interactive experience while
5+
# capturing the session for MLflow tracking and analysis.
6+
#
7+
# Usage: claude-with-tracking "your claude command here"
8+
# Example: claude-with-tracking "close-issue 583"
9+
10+
set -euo pipefail
11+
12+
# Colors for output
13+
GREEN='\033[0;32m'
14+
YELLOW='\033[0;33m'
15+
BLUE='\033[0;34m'
16+
NC='\033[0m' # No Color
17+
18+
# Determine dotfiles root
19+
DOT_DEN="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
20+
21+
# Ensure MLflow is running
22+
if [[ -x "$DOT_DEN/bin/start-mlflow" ]]; then
23+
"$DOT_DEN/bin/start-mlflow" start >/dev/null 2>&1
24+
fi
25+
26+
# Generate session ID and paths
27+
SESSION_ID="claude_$(date +%Y%m%d_%H%M%S)_$$"
28+
SESSION_LOG="/tmp/${SESSION_ID}.log"
29+
SESSION_METADATA="/tmp/${SESSION_ID}.meta"
30+
31+
# Function to cleanup on exit
32+
cleanup() {
33+
local exit_code=$?
34+
35+
# Parse and send session to MLflow (in background to not block)
36+
if [[ -f "$SESSION_LOG" ]]; then
37+
echo -e "\n${BLUE}📊 Sending session to MLflow...${NC}"
38+
39+
# Run parser with uv to ensure mlflow is available
40+
uv run --with mlflow python "$DOT_DEN/tracking/parse_claude_session.py" \
41+
"$SESSION_LOG" \
42+
"$SESSION_METADATA" \
43+
"$exit_code" 2>/dev/null &
44+
45+
# Brief pause to let it start
46+
sleep 1
47+
echo -e "${GREEN}✓ Session logged to MLflow${NC}"
48+
echo -e "${BLUE}View at: http://localhost:5000${NC}"
49+
fi
50+
51+
# Clean up temporary files after a delay (let MLflow finish first)
52+
(sleep 10 && rm -f "$SESSION_LOG" "$SESSION_METADATA" 2>/dev/null) &
53+
}
54+
55+
# Set up exit trap
56+
trap cleanup EXIT
57+
58+
# Save metadata about the session
59+
cat > "$SESSION_METADATA" <<EOF
60+
{
61+
"session_id": "$SESSION_ID",
62+
"command": "$*",
63+
"start_time": "$(date -Iseconds)",
64+
"user": "$USER",
65+
"pwd": "$PWD",
66+
"dotfiles_root": "$DOT_DEN"
67+
}
68+
EOF
69+
70+
# Inform user
71+
echo -e "${BLUE}╭────────────────────────────────────────╮${NC}"
72+
echo -e "${BLUE}│ 🚀 Claude with MLflow Tracking Active │${NC}"
73+
echo -e "${BLUE}├────────────────────────────────────────┤${NC}"
74+
echo -e "${BLUE}│ Session: ${SESSION_ID:0:20}... │${NC}"
75+
echo -e "${BLUE}│ Tracking to: http://localhost:5000 │${NC}"
76+
echo -e "${BLUE}╰────────────────────────────────────────╯${NC}"
77+
echo ""
78+
79+
# Run Claude with full interactivity, capturing output
80+
# Using 'script' to capture terminal session including colors and control chars
81+
script -q -c "claude $*" "$SESSION_LOG"
82+
83+
# Exit code is preserved by trap

‎bin/start-mlflow‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
#!/bin/bash
2+
# Self-healing MLflow starter - installs if missing, starts if not running
3+
# Follows spilled coffee principle: system should repair itself
4+
5+
set -euo pipefail
6+
7+
# Colors for output
8+
GREEN='\033[0;32m'
9+
YELLOW='\033[0;33m'
10+
NC='\033[0m' # No Color
11+
12+
# Determine dotfiles root
13+
DOT_DEN="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
14+
15+
# Function to check if MLflow is installed via uv
16+
check_mlflow_installed() {
17+
uv tool list 2>/dev/null | grep -q "mlflow" || return 1
18+
}
19+
20+
# Function to check if MLflow is running
21+
check_mlflow_running() {
22+
# Check if port 5000 is in use by MLflow
23+
lsof -i :5000 -P -n 2>/dev/null | grep -q LISTEN || return 1
24+
}
25+
26+
# Function to start MLflow
27+
start_mlflow_server() {
28+
local mlruns_dir="$DOT_DEN/mlruns"
29+
30+
# Ensure mlruns directory exists
31+
mkdir -p "$mlruns_dir"
32+
33+
# Start MLflow UI in background using uv run
34+
nohup uv run --with mlflow mlflow ui \
35+
--backend-store-uri "file://$mlruns_dir" \
36+
--host 0.0.0.0 \
37+
--port 5000 \
38+
> /dev/null 2>&1 &
39+
40+
# Give it a moment to start
41+
sleep 2
42+
43+
# Verify it started successfully
44+
if curl -s http://localhost:5000 > /dev/null 2>&1; then
45+
echo -e "${GREEN}✓ MLflow UI started at http://localhost:5000${NC}"
46+
return 0
47+
else
48+
# Silent failure - don't break setup flow
49+
return 0
50+
fi
51+
}
52+
53+
# Main execution
54+
main() {
55+
local action="${1:-start}"
56+
57+
case "$action" in
58+
start)
59+
# Check if MLflow is installed, install if not (spilled coffee principle)
60+
if ! check_mlflow_installed; then
61+
echo "MLflow not found. Installing via uv..."
62+
if uv tool install mlflow > /dev/null 2>&1; then
63+
echo -e "${GREEN}✓ MLflow installed successfully${NC}"
64+
else
65+
echo -e "${YELLOW}Could not install MLflow. Skipping...${NC}"
66+
exit 0
67+
fi
68+
fi
69+
70+
# Check if already running
71+
if check_mlflow_running; then
72+
# Already running - silent success (no output to avoid noise)
73+
exit 0
74+
fi
75+
76+
# Start MLflow
77+
echo "Starting MLflow UI..."
78+
start_mlflow_server
79+
;;
80+
81+
stop)
82+
# Stop MLflow if running
83+
if pgrep -f "mlflow ui" > /dev/null 2>&1; then
84+
pkill -f "mlflow ui"
85+
echo -e "${GREEN}✓ MLflow stopped${NC}"
86+
else
87+
echo "MLflow is not running"
88+
fi
89+
;;
90+
91+
status)
92+
if check_mlflow_running; then
93+
echo -e "${GREEN}MLflow is running at http://localhost:5000${NC}"
94+
else
95+
echo "MLflow is not running"
96+
fi
97+
;;
98+
99+
*)
100+
echo "Usage: $0 {start|stop|status}"
101+
exit 1
102+
;;
103+
esac
104+
}
105+
106+
# Run main function with all arguments
107+
main "$@"

‎knowledge/tools/mcp-dashboard.md‎

Lines changed: 0 additions & 20 deletions
This file was deleted.

‎setup.sh‎

Lines changed: 16 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -596,23 +596,22 @@ if [[ -f ~/.bash_aliases ]]; then
596596
echo -e "${GREEN}✓ Bash aliases loaded successfully${NC}"
597597
fi
598598

599-
# MCP Dashboard setup
600-
# EXPERIMENT #1213: Temporarily disabled - testing without MCP servers/dashboard
601-
# echo -e "${DIVIDER}"
602-
# echo "Setting up MCP Dashboard..."
603-
#
604-
# # Check if start-mcp-dashboard script exists
605-
# if [[ -x "$DOT_DEN/bin/start-mcp-dashboard" ]]; then
606-
# # Use the start-mcp-dashboard script which handles all checks
607-
# "$DOT_DEN/bin/start-mcp-dashboard" start
608-
# # The script handles:
609-
# # - Checking if dashboard is already running
610-
# # - Verifying the binary exists
611-
# # - Starting with proper health checks
612-
# # - Displaying clear status messages
613-
# else
614-
# echo -e "${YELLOW}start-mcp-dashboard script not found. Skipping dashboard setup.${NC}"
615-
# fi
599+
# MLflow tracking setup (self-healing with spilled coffee principle)
600+
echo -e "${DIVIDER}"
601+
echo "Setting up MLflow tracking..."
602+
603+
# Check if start-mlflow script exists
604+
if [[ -x "$DOT_DEN/bin/start-mlflow" ]]; then
605+
# Use the start-mlflow script which handles all checks
606+
"$DOT_DEN/bin/start-mlflow" start
607+
# The script handles:
608+
# - Installing MLflow via uv if not present (spilled coffee principle)
609+
# - Checking if already running (idempotent)
610+
# - Starting in background if needed
611+
# - Silent operation to avoid noise
612+
else
613+
echo -e "${YELLOW}start-mlflow script not found. Skipping MLflow setup.${NC}"
614+
fi
616615

617616
# Configure git hooks
618617
if [[ -d "$DOT_DEN/.githooks" ]]; then

0 commit comments

Comments
 (0)