Coldplay Spotify Analytics

A data analytics project that extracts Coldplay's music data from Spotify, stores it in DuckDB, and creates interactive dashboards using Visivo.

Overview

This project uses:

• DLT (Data Load Tool) to extract data from Spotify's Web API
• DuckDB as the data warehouse
• Visivo for creating interactive data visualizations
• UV for Python dependency management

Data Collected

The pipeline extracts:

• Artist information (followers, popularity, genres)
• All albums (110+ albums including singles and compilations)
• All tracks (405+ tracks with metadata)
• Top 10 most popular tracks

Note: Due to Spotify API restrictions as of November 2024, audio features and related artists data are not available for new applications.

Project Structure

coldplay-spotify/
├── .env                        # Spotify API credentials
├── pyproject.toml             # UV project configuration
├── extract_spotify_data.py    # DLT pipeline script
├── spotify_coldplay.duckdb    # DuckDB database (generated)
├── project.visivo.yml         # Main Visivo configuration
├── artist_overview.visivo.yml # Artist metrics dashboard
├── album_analysis.visivo.yml  # Album analysis dashboard
├── musical_evolution.visivo.yml # Musical evolution over time
├── top_tracks.visivo.yml      # Top tracks analysis
├── TASKS.MD                   # Project task tracking
└── README.md                  # This file

Setup Instructions

Prerequisites

• Python 3.10 or higher
• UV package manager
• Spotify Developer Account

1. Clone the Repository

git clone <repository-url>
cd coldplay-spotify

2. Set Up Spotify API Credentials

1. Go to Spotify Developer Dashboard
2. Create a new app
3. Copy your Client ID and Client Secret
4. Update the .env file with your credentials:

SPOTIFY_CLIENT_ID=your_client_id_here
SPOTIFY_CLIENT_SECRET=your_client_secret_here

3. Install Dependencies

uv sync

This will create a virtual environment and install all required packages.

Running the Pipeline

Extract Data from Spotify

Run the DLT pipeline to extract Coldplay's data:

uv run python extract_spotify_data.py

This will:

• Authenticate with Spotify using Client Credentials flow
• Extract artist, album, track, and top tracks data
• Store everything in spotify_coldplay.duckdb

Expected output:

Pipeline completed successfully!
Data loaded to: spotify_coldplay.duckdb

Data Statistics:
  artists: 1 records
  albums: 110 records
  tracks: 405 records
  top_tracks: 10 records

View Dashboards

Start the Visivo server:

uv run visivo serve

Then open your browser to: http://localhost:8000

Available Dashboards

1. Artist Overview (artist_overview.visivo.yml)

• Total follower count
• Spotify popularity score gauge
• Genre distribution
• Artist name display

2. Album Analysis (album_analysis.visivo.yml)

• Album release timeline scatter plot
• Album type distribution (albums vs singles vs compilations)
• Releases by year and type
• Recent studio albums overview

3. Musical Evolution (musical_evolution.visivo.yml)

• Average track duration over time
• Album structure changes (tracks per album)
• Release strategy evolution
• All tracks scatter plot (duration vs year, colored by popularity)

4. Top Tracks (top_tracks.visivo.yml)

• Top 10 tracks ranking by popularity
• Track popularity vs duration analysis
• Duration distribution histogram for all tracks
• Interactive bubble chart of top tracks

Data Schema

DuckDB Tables

The database contains the following tables in the coldplay_data_20250718022923 schema:

artists

• id: Spotify artist ID
• name: Artist name
• popularity: Popularity score (0-100)
• followers__total: Total follower count
• genres: Array of genre tags

albums

• id: Spotify album ID
• name: Album name
• release_date: Release date
• total_tracks: Number of tracks
• album_type: Type (album/single/compilation)

tracks

• id: Spotify track ID
• name: Track name
• album_id: Associated album ID
• album_name: Album name
• duration_ms: Track duration in milliseconds
• popularity: Popularity score (0-100)
• track_number: Position on album

top_tracks

• rank: Ranking position (1-10)
• name: Track name
• popularity: Popularity score
• album__name: Album name

Troubleshooting

Visivo Server Issues

If the Visivo server fails to start:

1. Make sure you're in the project directory
2. Check that the virtual environment is activated
3. Verify that spotify_coldplay.duckdb exists

API Authentication Errors

If you get 401/403 errors:

1. Verify your Spotify credentials in .env
2. Make sure your app is active in Spotify Developer Dashboard
3. Note that some endpoints (audio-features, related-artists) are restricted for new apps

Data Issues

To verify data integrity:

echo "SELECT COUNT(*) FROM coldplay_data_20250718022923.tracks;" | duckdb spotify_coldplay.duckdb

Future Enhancements

• Add incremental data loading
• Implement automated refresh schedule
• Add more artists for comparison
• Create playlist analysis (if API access available)
• Add concert/tour data integration

Deployment

Netlify Deployment

This project is configured for easy deployment to Netlify. The repository includes:

• netlify.toml - Netlify configuration
• build.sh - Build script that handles dependencies and site generation
• .github/workflows/deploy.yml - GitHub Actions workflow for automated deployments

Deploy to Netlify (Option 1: Direct from GitHub)

1. Connect to Netlify:

   • Log in to Netlify
   • Click "Add new site" → "Import an existing project"
   • Connect your GitHub account and select this repository

2. Configure Environment Variables:

   • Go to Site settings → Environment variables
   • Add the following (optional - site will work without them):
     • SPOTIFY_CLIENT_ID: Your Spotify app client ID
     • SPOTIFY_CLIENT_SECRET: Your Spotify app client secret

3. Deploy:

   • Netlify will automatically build and deploy your site
   • Future pushes to main branch will trigger automatic deployments

Deploy to Netlify (Option 2: GitHub Actions)

1. Get Netlify Credentials:

   • In Netlify, go to User settings → Applications → Create new token
   • Copy the token for NETLIFY_AUTH_TOKEN
   • In your Netlify site, go to Site settings → General → Copy the Site ID

2. Configure GitHub Secrets:

   • In your GitHub repository, go to Settings → Secrets and variables → Actions
   • Add the following secrets:
     • NETLIFY_AUTH_TOKEN: Your Netlify personal access token
     • NETLIFY_SITE_ID: Your Netlify site ID
     • SPOTIFY_CLIENT_ID: (Optional) Your Spotify client ID
     • SPOTIFY_CLIENT_SECRET: (Optional) Your Spotify client secret

3. Deploy:

   • Push to the main branch or manually trigger the workflow
   • Check the Actions tab to monitor deployment progress

Local Build

To build the static site locally:

# Run the data pipeline and generate trace data
uv run visivo run

# Generate the static site
uv run visivo dist

# The static site will be in the dist/ directory
# You can serve it locally with:
python -m http.server -d dist 8080

Build Without Spotify Credentials

The project includes a pre-populated DuckDB database (spotify_coldplay.duckdb), so the site can be built and deployed without Spotify API credentials. The build process will:

1. Use the existing database if no credentials are provided
2. Optionally refresh data if credentials are available

License

This project is for educational and personal use only. Spotify data is subject to Spotify's Terms of Service.