🎵 MMMD — Multimodal Music Intelligence System

An AI-powered multimodal music intelligence platform that identifies, analyzes, and enriches music information from audio, text, and image inputs. The system uses a custom MCP (Model Context Protocol) server to orchestrate multiple tools and external APIs, enabling intelligent music discovery, metadata enrichment, Indian classical music analysis, and contextual insights.

---

✨ Features

🎵 Audio-Based Music Recognition

• Song identification using ACRCloud audio fingerprinting
• Metadata extraction and enrichment
• Recognition confidence scoring

🔍 Text-Based Music Intelligence

• Natural language music queries
• Artist-based search with curated song recommendations
• Song metadata retrieval and enrichment

🖼️ Image-Based Music Recognition

• Claude Vision integration for music-related image understanding
• Multimodal analysis pipeline

🎼 Indian Classical Music Analysis

• Raaga detection and enrichment
• Taala and rasa analysis
• Composer and music system identification
• Carnatic music knowledge retrieval

🤖 Music Intelligence

• AI-generated song summaries
• Similar song recommendations
• Contextual song insights
• Metadata aggregation from multiple sources

---

🏗️ Architecture

        Input Layer
   [Audio] [Text] [Image]
              │
              ▼
     ┌─────────────────┐
     │  Custom MCP     │
     │  Server         │
     │  (7 AI Tools)   │
     └────────┬────────┘
              │
              ▼
         Tool Router
    ┌─────────┼─────────┐
    │         │         │
    ▼         ▼         ▼
ACRCloud   Spotify  MusicBrainz
(Audio ID) (Meta)  (Credits)
    │         │         │
    └─────────┼─────────┘
              ▼
   Metadata Aggregation
              │
              ▼
  ┌───────────────────────┐
  │  Music Intelligence   │
  │  Layer (Claude API +  │
  │  LLM Reasoning)       │
  └───────────┬───────────┘
              │
              ▼
  Indian Classical Analysis
  (Raaga Detection)
              │
              ▼
  Summary & Recommendations
              │
              ▼
      Unified Response

---

🛠️ Tech Stack

Frontend

• React
• TypeScript
• Vite
• Tailwind CSS

Backend

• Node.js
• Express.js
• TypeScript

AI & Orchestration

• Claude API
• MCP (Model Context Protocol)
• ACRCloud

External APIs

• Spotify API
• MusicBrainz API

---

🚀 Key Capabilities

• Multimodal music understanding through audio, text, and image inputs
• Custom MCP orchestration layer coordinating multiple intelligence tools
• Indian classical music enrichment beyond traditional music metadata systems
• Unified analysis pipeline combining external APIs with domain-specific knowledge bases
• Full-stack TypeScript architecture with modular service design

---

📸 Screenshots

Home Page

Audio Recognition

Text Search

Indian Music Analysis

Similar Song Recommendations

Artist Lookup

AI Summary

---

⚙️ Setup

Clone Repository

git clone https://github.com/SriChandhana/multimodal-music-intelligence-system.git

Backend

cd backend
npm install
npm run dev

Frontend

cd frontend
npm install
npm run dev

Environment Variables

Create:

PORT=5000

ACR_HOST=
ACR_ACCESS_KEY=
ACR_ACCESS_SECRET=

SPOTIFY_CLIENT_ID=
SPOTIFY_CLIENT_SECRET=

---

🔮 Future Enhancements

• Enhanced recommendation engine
• Expanded Indian classical music knowledge base
• Artist profile enrichment
• Advanced semantic music search
• Production deployment

👩‍💻 Author

Kandula Sri Chandhana

B.Tech CSE (AI & ML), VNR VJIET