Skip to content

Readme.md

Latest commit

 

History

History
91 lines (61 loc) · 3.63 KB

Readme.md

File metadata and controls

91 lines (61 loc) · 3.63 KB
Raw

Chat With Video

This repository contains a Streamlit application that allows users to interact with video content through a chat interface. The application uses OpenAI's Whisper model for transcribing audio from videos and LangChain for creating a conversational AI. Users can upload videos, which are then processed and transcribed. Once transcribed, users can select these videos and ask questions about their content. The key features of the application include:

  • Chat With Video: Users can select transcribed videos and ask questions to get insights based on the video content.
  • Upload Video: Users can upload new videos for processing and transcription, making them available for the chat functionality.

Installation

  1. Clone the repository
git clone https://github.com/aimaster-dev/langchain-agent.git
cd langchain-apps/chat-with-video
  1. Create a virtual environment and activate it
python -m venv env
source env/bin/activate
  1. Install required dependencies
pip install -r requirements.txt
  1. Setup environment variables for OpenAI API key, LangSmith key and Pinecone key
cp .env.example .env
# Edit the .env file to include your OpenAI API key and LangSmith key

Usage

Run the streamlit app

streamlit run Home.py

Demo

Langsmith Trace

Process Overview

Video Processing Overview Video Processing Overview

QA Process Overview QA Process Overview

Here’s an overview of the process:

  1. Video Upload and Audio Extraction:
  • Users upload a video file.
  • The application uses the ffmpeg-python library to extract audio from the uploaded video.
  1. Audio Transcription:
  • The extracted audio is transcribed using the OpenAI Whisper API.
  • Since Whisper has a 25 MB limit for audio files, larger audio files are split into smaller chunks using ffmpeg-python.
  • Each chunk is transcribed individually, and the transcriptions are merged to form the complete transcription for the video.
  1. Creating Embeddings:
  • The transcriptions are split into smaller chunks using LangChain’s RecursiveCharacterTextSplitter with a chunk size of 1000 and a chunk overlap of 100. Splitting the transcription into chunks improves the accuracy and efficiency of information retrieval.
  • Each chunk is converted into embeddings using OpenAI's text-embedding-ada-002 model.
  • The embeddings are stored in a Pinecone vector database along with the video name as metadata.
  1. Chatting with the Video:
  • Users can select one or more videos for querying.
  • The application uses the Pinecone vector store as a retriever, filtering documents based on the selected videos.
  • A custom ChatPromptTemplate and a QAChain are created, which take the context and user query, and pass them to the prompt.
  • The prompt uses the OpenAI GPT-3.5 model to generate a response in JSON format: {answer: "", references: ["", "", ...]}.
  • The response is parsed using a JSON output parser.

References