TASKS.MD

Coldplay Spotify Analytics Project Tasks

Project Overview

Building a DLT + Visivo project to analyze Coldplay's music data from Spotify, store it in DuckDB, and create interactive dashboards.

Phase 1: Project Setup ✅

• Create TASKS.MD file with detailed project tasks
• Initialize UV project with pyproject.toml
• Add project dependencies using UV (dlt[duckdb], visivo, requests, python-dotenv)
• Create .env file structure for Spotify credentials
• Set up UV virtual environment and install dependencies

Phase 2: Data Extraction with DLT (Using REST API Source) ✅

• Create DLT pipeline using REST API source
  • Create extract_spotify_data.py file
  • Set up REST API source configuration for Spotify
  • Test: Verify basic pipeline structure works
• Configure and customize generated pipeline
  • Set up Client Credentials authentication (add to .env)
  • Add Coldplay's artist ID (4gzpq5DPGxSnKTe4SA8HAU) to configuration
  • Filter to only available endpoints (Note: audio-features and related-artists restricted as of Nov 2024):
    • /artists/{id} - Artist info ✅
    • /artists/{id}/albums - Artist albums ✅
    • /artists/{id}/top-tracks - Top tracks ✅
    • /artists/{id}/related-artists - Related artists (RESTRICTED)
    • /audio-features - Audio features for tracks (RESTRICTED)
    • /tracks - Track details ✅
  • Test: Verify authentication works with simple API call
• Extract and store all data:
  • Run pipeline to extract Coldplay data
  • Verify data includes:
    • Artist profile (followers, popularity, genres, images) ✅
    • All albums with metadata (110 albums extracted) ✅
    • All tracks with popularity and duration (405 tracks) ✅
    • Audio features for all tracks (API restricted)
    • Top 10 tracks ✅
    • Related artists (API restricted)
  • Test: Query DuckDB tables to verify data integrity
  • Test: Verify track count matches album track totals
  • Note: Audio features not available due to API restrictions

Phase 3: Visivo Dashboard Creation

• Create project.visivo.yml with DuckDB connection configuration
  • Test: Run visivo serve to verify connection ✅ (Running at http://localhost:8000)
• Create artist_overview.visivo.yml dashboard:
  • Follower count display
  • Popularity metrics
  • Genre tags visualization
  • Artist image display (images in separate table, not implemented)
  • Test: Verify dashboard renders with live data
• Create album_analysis.visivo.yml dashboard:
  • Album timeline visualization
  • Album popularity comparison (via timeline)
  • Album type distribution (album vs single vs compilation)
  • Track count per album chart
  • Test: Verify all albums appear correctly
• Create musical_evolution.visivo.yml dashboard:
  • Audio features over time (API restricted - replaced with duration/popularity trends)
  • Energy vs Valence scatter plot (API restricted)
  • Danceability trends (API restricted)
  • Acousticness evolution (API restricted)
  • Tempo distribution histogram (API restricted)
  • Track duration evolution over time
  • Album structure changes (tracks per album)
  • Release strategy evolution (singles vs albums)
  • Test: Verify time-based visualizations work
• Create top_tracks.visivo.yml dashboard:
  • Top 10 tracks ranking
  • Audio features radar chart (API restricted)
  • Popularity vs duration analysis (instead of audio features)
  • Track duration analysis
  • Test: Verify top tracks match Spotify's top tracks

Phase 4: Testing and Documentation ✅

• Test DLT pipeline execution ✅
• Verify data quality in DuckDB ✅
• Test all Visivo dashboards ✅ (Server running at http://localhost:8000)
• Create README.md with:
  • Project description
  • Setup instructions
  • How to run the pipeline
  • How to view dashboards
  • Data schema documentation

Phase 5: Enhancements (Optional)

• Add incremental loading to update data periodically
• Create automated refresh schedule
• Add more artists for comparison
• Create playlist analysis features
• Add concert/tour data if available

Phase 6: CI/CD Deployment with Netlify ✅

• Create netlify.toml configuration file
  • Set build command to use build.sh
  • Configure Python and Node versions
  • Set publish directory to dist/
  • Add security headers
  • Configure SPA routing
• Create build.sh script
  • Install UV if not present
  • Set up Python environment
  • Install dependencies
  • Handle optional Spotify credentials
  • Run visivo run and visivo dist
  • Verify dist directory creation
• Create .env.example file
  • Document required environment variables
  • Add instructions for Netlify configuration
• Verify .gitignore includes dist/
• Create GitHub Actions workflow (.github/workflows/deploy.yml)
  • Trigger on push to main branch
  • Allow manual triggers
  • Set up Python and UV
  • Handle secrets for Spotify credentials
  • Build and deploy to Netlify
• Update README.md with deployment instructions
  • Add Netlify deployment section
  • Document both deployment options
  • Include local build instructions
  • Note about pre-populated database

Notes

• Using Spotify Client Credentials flow (no user-specific data)
• DuckDB database will be committed to repo for easy sharing
• All Visivo files use .visivo.yml extension
• Focus on Coldplay's musical evolution and characteristics
• Netlify deployment works with or without Spotify credentials
• Static site can be hosted anywhere that supports static files