Skip to content

Commit ac35e20

Browse files
committed
Added VIBE blog post
1 parent 4a42a45 commit ac35e20

5 files changed

Lines changed: 81 additions & 0 deletions

File tree

‎_posts/2025-09-21-VIBE.md‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
---
2+
title: 'VIBE: Annotation-Free Video-to-Text Information Bottleneck Evaluation for TL;DR'
3+
description: NeurIPS 2025 paper
4+
categories: blog
5+
---
6+
7+
*By Po-han*
8+
9+
# VIBE: Annotation-Free Video-to-Text Information Bottleneck Evaluation for TL;DR
10+
11+
Proceedings of the 39th Conference on Neural Information Processing Systems (NeurIPS 2025)
12+
13+
**TLDR:** Annotation-free video summarization evaluation that boosts human decision-making accuracy by up to 61% and cuts response time by 75%!
14+
15+
[Project Website](https://vivianchen98.github.io/VIBE_website/) |
16+
[arXiv](https://arxiv.org/abs/2505.17423) |
17+
[Code](https://github.com/UTAustin-SwarmLab/Task-aware-TLDR-Public) |
18+
[Dataset](https://huggingface.co/datasets/vivianchen98/LearningPaper24)
19+
20+
## Motivation
21+
Many real-world tasks still require human oversight: a traffic officer sifting through dashcam footage, or a researcher screening long conference videos. Watching raw video is slow, and existing vision-language models (VLMs) often produce verbose, redundant captions that hinder efficiency.
22+
Current video-to-text evaluation methods depend on costly human annotations and ignore whether summaries actually help humans make decisions. We ask:
23+
- Q1 (Annotation-Free): Can we evaluate video summaries without relying on gold-standard human captions?
24+
- Q2 (Task-Aware): Can we measure how much a summary improves human performance on downstream tasks?
25+
26+
### System Plot:
27+
<figure style="text-align: center;">
28+
<img src="{{site.baseurl}}/images/post/TLDR_system.png" alt="TLDR System Plot" height="auto" style="margin: auto; display: block;">
29+
<figcaption>VIBE Framework Overview</figcaption>
30+
<p></p>
31+
</figure>
32+
33+
## Contributions
34+
We introduce VIBE (Video-to-text Information Bottleneck Evaluation), a novel framework that evaluates and selects VLM summaries without annotations or retraining.
35+
- Grounding Score: Measures how faithfully a summary reflects the video using pointwise mutual information between video and text.
36+
- Utility Score: Measures how informative the summary is for a downstream task.
37+
38+
- Annotation-Free Rejection Sampling: VIBE ranks multiple VLM-generated summaries and selects the one that maximizes grounding and/or utility, supporting efficient human decision-making.
39+
40+
## VIBE Framework
41+
<figure style="text-align: center;">
42+
<img src="{{site.baseurl}}/images/post/VIBE_figure.png" alt="VIBE Mechanism Overview" height="auto" style="margin: auto; display: block;">
43+
<figcaption>VIBE: Mechanism Overview</figcaption>
44+
<p></p>
45+
</figure>
46+
47+
VIBE adapts the information bottleneck principle to video summarization:
48+
49+
- Grounding approximates I(V;T), testing how well the video supports reconstruction of a masked summary.
50+
- Utility approximates I(T;Y), testing how well the summary compensates for missing video information to predict a task label.
51+
52+
By maximizing both, VIBE selects concise, task-relevant summaries—without gold labels or retraining.
53+
54+
## Results
55+
<figure style="text-align: center;">
56+
<img src="{{site.baseurl}}/images/post/VIBE_table.png" alt="VIBE Results Table" height="auto" style="margin: auto; display: block;">
57+
<figcaption>VIBE: Results Overview</figcaption>
58+
<p></p>
59+
</figure>
60+
61+
We validate VIBE through user studies with 243 participants across three datasets:
62+
- LearningPaper24 (research talk videos)
63+
- LongVideoBench (long instructional clips)
64+
- SUTD-TrafficQA (short traffic clips)
65+
66+
<figure style="text-align: center;">
67+
<img src="{{site.baseurl}}/images/post/longvideobench_main.png" alt="LongVideoBench Results" height="auto" style="margin: auto; display: block;">
68+
<figcaption>Correlation between Accuracy, Grounding, and Utility Scores</figcaption>
69+
<p></p>
70+
</figure>
71+
72+
Key findings:
73+
- Accuracy Gains: VIBE-selected summaries boost task accuracy by up to 61.23%.
74+
- Speed Improvements: Response time drops by up to 75.77% compared to raw video.
75+
- Efficiency: VIBE achieves lower inverse efficiency scores (time/accuracy), showing superior speed–accuracy trade-offs.
76+
- Scalability: Grounding works in a fully self-supervised manner, while utility correlates strongly with higher task accuracy.
77+
78+
## Impact
79+
VIBE reframes video caption evaluation from a human decision support perspective. Unlike reference-based metrics, it scales to unseen data, works with black-box VLMs, and requires no human annotations.
80+
This makes VIBE a practical plug-in for improving video summarization in real-world settings: from scientific video search to public safety monitoring.
81+

‎images/post/TLDR_system.png‎

1.35 MB
Loading

‎images/post/VIBE_figure.png‎

669 KB
Loading

‎images/post/VIBE_table.png‎

214 KB
Loading
359 KB
Loading

0 commit comments

Comments
 (0)