# Paper Presenter – End-to-End Generative AI System

## Stage 1 - Modular Video-Ready Pipeline

### Output Structure (for TTS + Video)

Each section generates:
- **Separate text files** (oral scripts for TTS)
- **Separate image files** (for video frames)
- **Backbone structure** (sequence for video assembly)

```
outputs/stage1_v2/
├── presenter.png                    # Presenter image
├── background_text.txt              # Oral script
├── background_images/
│   ├── background_1.png
│   ├── background_2.png
│   └── background_3.png
├── methodology_text.txt             # Oral script
├── experiments_text.txt             # Oral script
├── conclusion_text.txt              # Oral script
├── summary_text.txt                 # Oral script
├── images/                          # Extracted figures
│   ├── fig01-*.png
│   ├── fig02-*.png
│   └── fig03-*.png
├── tables/                          # Extracted tables (markdown)
│   ├── table01.md
│   ├── table02.md
│   ├── table03.md
│   └── table04.md
└── backbone.json                    # Video sequence structure
```

### Backbone Structure

```json
{
  "sequence": [
    {"type": "image", "file": "presenter.png", "duration": 3},
    {"type": "audio+image", "text": "background_text.txt", "image": "background_images/background_1.png"},
    {"type": "audio+image", "text": "background_text.txt", "image": "background_images/background_2.png"},
    {"type": "audio+image", "text": "methodology_text.txt", "image": "images/fig01-*.png"},
    {"type": "audio+image", "text": "methodology_text.txt", "image": "images/fig02-*.png"},
    {"type": "audio+table", "text": "experiments_text.txt", "image": "tables/table01.png"},
    ...
  ]
}
```

## Quick Start

### 1. Extract Images and Tables (One-time)

```bash
conda activate nova

# Extract images from PDF
python -c "
from pathlib import Path
from src.utils.image_utils import extract_images_from_pdf
extract_images_from_pdf(Path('paper.pdf'), Path('outputs/stage1_v2/images'))
"

# Extract tables with GPT-5 (concurrent, uses existing PNG images)
python extract_tables.py
```

### 2. Generate Background (Oral + Images)

```bash
python test_background.py
```

**Output:**
- `outputs/stage1_v2/background_text.txt` - Oral script (TTS-ready)
- `outputs/stage1_v2/background_images/*.png` - Generated images

### 3. Generate Other Sections

```bash
# TODO: Methodology
python test_methodology.py

# TODO: Experiments
python test_experiments.py

# TODO: Conclusion
python test_conclusion.py

# TODO: Summary
python test_summary.py
```

## Configuration

- **API Key:** `src/config/settings.py`
- **Models:** GPT-5 for text, DALL-E 3 for images
- **Environment:** `nova` conda environment

## Current Status

✅ **Completed:**
- Image extraction (3 figures, 4 tables)
- Table extraction (GPT-5, markdown format)
- Background generator (oral script, image generation)
- Modular structure for video pipeline

⏳ **TODO:**
- Methodology generator (oral + figure sequence)
- Experiments generator (oral + table/figure sequence)
- Conclusion generator (oral + conclusion image)
- Summary generator (oral script)
- Backbone JSON generator (video sequence)

## Configuration 配置

### API Keys and Settings

OpenRouter credentials are stored in `src/config/settings.py`. For this hackathon prototype:

- **OpenRouter API Key**: `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`
- **Default Model**: `openai/gpt-5` (for text generation and structured outputs)
- **Image Generation**: Uses Unsplash search API (free, no key required) with fallback to DALL·E 3

For production scenarios, move secrets into environment variables or a secure secrets manager.

## Usage 使用方法

### Quick Start (Recommended)

Use the modular runner script for easy component execution:

```bash
conda activate nova

# Run the complete pipeline
python run_stage1.py --full

# Or run individual components (each is re-runnable)
python run_stage1.py --component extract_images
python run_stage1.py --component presenter
python run_stage1.py --component background
python run_stage1.py --component tables
```

### Full Stage 1 Pipeline

Run the complete Stage 1 pipeline to generate a presentation from a PDF:

```bash
conda activate nova
python -m src.main \
  --pdf /Users/xiangzheng/Desktop/nova/paper_presenter/paper.pdf \
  --presenter /Users/xiangzheng/Desktop/nova/paper_presenter/presenter.webp
```

**Or use the modular runner:**

```bash
python run_stage1.py --full
```

**Optional Arguments:**
- `--output`: Override the artifact output directory (default: `outputs/stage1/`)
- `--video`: Path to accompanying video clip (for future stages)

### Individual Components (Re-runnable)

Each component can be run independently for iterative development:

#### 1. Extract Images and Tables Only

```bash
python run_stage1.py --component extract_images
```

Images extracted to: `outputs/stage1/images/pdf/`

#### 2. Convert Presenter Image

```bash
python run_stage1.py --component presenter
```

Converts WEBP/JPG/PNG to PNG format in: `outputs/stage1/presenter.png`

#### 3. Generate 5W2H Background with Illustrations

```bash
python run_stage1.py --component background
```

**With automatic injection into presentation:**

```bash
python run_stage1.py --component background --apply
```

Or use the dedicated script:

```bash
python -m src.background_generator \
  --pdf paper.pdf \
  --apply \
  --target outputs/stage1/stage1_presentation.md
```

#### 4. Convert Tables to Markdown

```bash
python run_stage1.py --component tables
```

Processes all extracted table images and generates markdown files.

#### 5. Update Background Section in Existing Presentation

```bash
python scripts/update_background.py
```

This replaces content between `<!-- BACKGROUND_START -->` and `<!-- BACKGROUND_END -->` markers.

### Figure/Table Extraction 命名规范

- 所有从 PDF 提取的图像与表格按顺序命名为 `figXX-<slug>.png`、`tableXX-<slug>.png`。
- `XX` 为两位序号（01, 02, ...），`<slug>` 取自原 caption，经小写化和字符清洗后的摘要。
- 若无法获取 caption，则使用默认占位文本生成 slug，仍保持 `figXX` / `tableXX` 前缀以保证唯一性。

## Technical Architecture 技术架构

### Core Components

1. **PDF Processing** (`src/utils/pdf_utils.py`)
   - Text extraction using `pypdf`
   - Intelligent chunking for LLM context
   - Section identification

2. **Image Extraction** (`src/utils/image_utils.py`)
   - Figure extraction using PyMuPDF (`fitz`)
   - Table region detection with caption matching
   - Automatic image quality filtering
   - Presenter image conversion (WEBP → PNG)

3. **LLM Integration** (`src/utils/llm.py`)
   - OpenRouter API client
   - Structured JSON completions
   - Image generation support
   - Streaming response handling

4. **Table Processing** (`src/utils/table_ai.py`)
   - Multimodal AI for table-to-markdown conversion
   - Vision model analysis of table images
   - Structured data extraction

5. **Background Generation** (`src/background_generator.py`)
   - 5W2H framework (Who, What, When, Where, Why, How, How much)
   - Illustration generation via Unsplash/DALL·E
   - Markdown formatting with embedded images

6. **Stage 1 Pipeline** (`src/pipeline/stage1.py`)
   - End-to-end orchestration
   - Asset categorization and mapping
   - Markdown deck assembly

### Backend API Integration (For Future Development)

The system is designed to integrate with a backend API for frontend-driven workflows:

**Endpoint:** `POST /chat_streaming`

**Request Schema:**
```python
{
  "model_id": str,              # OpenRouter model ID (e.g., "openai/gpt-5")
  "chat_history": List[Message], # Conversation context
  "use_mcp": bool,              # Enable tool calls
  "approved_tool_calls": List[dict],  # Pre-approved tools
  "mcp_auto_approve": bool      # Auto-approve all tools
}
```

**Response:** Server-Sent Events (SSE) stream with JSON chunks

**Message Format:**
```python
{
  "role": "user" | "assistant" | "system",
  "content": str,
  "image": Optional[{"format": str, "data": base64_str}],
  "pdf": Optional[{"name": str, "data": base64_str}]
}
```

### OpenRouter Model Support

Supported output modalities from OpenRouter models:
- `text`: Standard text generation
- `image`: Image generation (DALL·E, Stable Diffusion, etc.)
- `audio`: Audio generation (for Stage 2)

The system automatically selects appropriate modalities based on model capabilities.

## Project Structure 项目结构

```
paper_presenter/
├── data/                          # Input data directory
│   └── inputs/                    # User-provided inputs
├── outputs/                       # Generated artifacts
│   ├── stage1/                    # Stage 1 outputs
│   │   ├── images/                # All image assets
│   │   │   └── pdf/              # Extracted PDF figures/tables
│   │   ├── tables/                # Table markdown files
│   │   ├── background_images/     # Generated background illustrations
│   │   ├── background_5w2h.md     # 5W2H background document
│   │   ├── stage1_presentation.md # Final presentation markdown
│   │   └── presenter.png          # Converted presenter image
│   └── tmp/                       # Temporary working directory
├── src/                           # Source code
│   ├── config/                    # Configuration and settings
│   │   └── settings.py           # API keys, paths, model config
│   ├── pipeline/                  # Pipeline implementations
│   │   └── stage1.py             # Stage 1 orchestration
│   ├── utils/                     # Utility modules
│   │   ├── pdf_utils.py          # PDF text extraction
│   │   ├── image_utils.py        # Image processing
│   │   ├── llm.py                # OpenRouter client
│   │   └── table_ai.py           # Table-to-markdown conversion
│   ├── background_generator.py    # 5W2H background generator
│   └── main.py                    # CLI entry point
├── scripts/                       # Helper scripts
│   └── update_background.py       # Background injection script
├── paper.pdf                      # Input: Research paper
├── presenter.webp                 # Input: Presenter image
├── requirements.txt               # Python dependencies
└── README.me                      # This file
```

## Example Output 示例输出

After running the Stage 1 pipeline, you'll get:

1. **Main Presentation** (`stage1_presentation.md`):
   - Title slide with presenter image
   - Background section (5W2H format with illustrations)
   - Methodology section with sub-steps and figure references
   - Experiments section with results and table references
   - Conclusion and summary

2. **Extracted Assets**:
   - `fig01-*.png`, `fig02-*.png`, ... (paper figures)
   - `table01-*.png`, `table02-*.png`, ... (paper tables as images)
   - `tables/table1.md`, `table2.md`, ... (tables as markdown)
   - `background_images/*.png` (generated illustrations)

3. **Structured Data**:
   - All figures and tables properly labeled and captioned
   - Automatic reference linking in the markdown
   - Relative paths for portability

## Key Features 核心特性

✅ **Fully Automated**: One command generates complete presentation  
✅ **Re-runnable**: Each component can be run independently  
✅ **Smart Extraction**: Automatic caption detection for figures/tables  
✅ **AI-Powered**: Uses GPT-5 for intelligent content structuring  
✅ **Multimodal**: Handles text, images, and tables seamlessly  
✅ **5W2H Framework**: Structured background generation for clarity  
✅ **Quality Filtering**: Removes low-variance/blank images automatically  
✅ **Markdown Output**: Easy to edit, version control, and convert

## Next Steps 后续计划

### Stage 2: TTS + Video Generation
- Integrate text-to-speech (TTS) API
- Align audio narration with slide timing
- Generate MP4 video with static images and voiceover
- Add slide transitions and pacing control

### Stage 3: Avatar Animation
- Generate presenter avatar from portrait
- Implement lip-sync animation
- Add gesture recognition and emphasis highlighting
- Real-time keyword highlighting on slides

欢迎继续扩展本项目，逐步实现完整的多模态演示生成管线。

## Troubleshooting 故障排除

### Common Issues

1. **Missing Dependencies**: Ensure all packages are installed in the `nova` environment
   ```bash
   conda activate nova
   pip install -r requirements.txt
   ```

2. **API Rate Limits**: OpenRouter has rate limits; wait and retry if you hit them

3. **Image Extraction Fails**: Ensure the PDF contains actual images (not just text)

4. **Table Conversion Errors**: Complex tables may require manual review/editing

### Development Notes

- The system uses fallback mechanisms for robustness
- All file operations use absolute paths for reliability
- Each component logs its progress to stdout
- Re-running the pipeline will clean and regenerate all outputs

## License & Credits

This is a hackathon prototype for educational and demonstration purposes.

**Technologies Used:**
- OpenRouter API (GPT-5, DALL·E 3)
- PyMuPDF for PDF processing
- Pillow for image manipulation
- Unsplash for free stock imagery

---

**Built for Nova Hackathon 2025** 🚀