|
| 1 | +--- |
| 2 | +title: 'VIBE: Annotation-Free Video-to-Text Information Bottleneck Evaluation for TL;DR' |
| 3 | +description: NeurIPS 2025 paper |
| 4 | +categories: blog |
| 5 | +--- |
| 6 | + |
| 7 | +*By Po-han* |
| 8 | + |
| 9 | +# VIBE: Annotation-Free Video-to-Text Information Bottleneck Evaluation for TL;DR |
| 10 | + |
| 11 | +Proceedings of the 39th Conference on Neural Information Processing Systems (NeurIPS 2025) |
| 12 | + |
| 13 | +**TLDR:** Annotation-free video summarization evaluation that boosts human decision-making accuracy by up to 61% and cuts response time by 75%! |
| 14 | + |
| 15 | +[Project Website](https://vivianchen98.github.io/VIBE_website/) | |
| 16 | +[arXiv](https://arxiv.org/abs/2505.17423) | |
| 17 | +[Code](https://github.com/UTAustin-SwarmLab/Task-aware-TLDR-Public) | |
| 18 | +[Dataset](https://huggingface.co/datasets/vivianchen98/LearningPaper24) |
| 19 | + |
| 20 | +## Motivation |
| 21 | +Many real-world tasks still require human oversight: a traffic officer sifting through dashcam footage, or a researcher screening long conference videos. Watching raw video is slow, and existing vision-language models (VLMs) often produce verbose, redundant captions that hinder efficiency. |
| 22 | +Current video-to-text evaluation methods depend on costly human annotations and ignore whether summaries actually help humans make decisions. We ask: |
| 23 | +- Q1 (Annotation-Free): Can we evaluate video summaries without relying on gold-standard human captions? |
| 24 | +- Q2 (Task-Aware): Can we measure how much a summary improves human performance on downstream tasks? |
| 25 | + |
| 26 | +### System Plot: |
| 27 | +<figure style="text-align: center;"> |
| 28 | + <img src="{{site.baseurl}}/images/post/TLDR_system.png" alt="TLDR System Plot" height="auto" style="margin: auto; display: block;"> |
| 29 | + <figcaption>VIBE Framework Overview</figcaption> |
| 30 | + <p></p> |
| 31 | +</figure> |
| 32 | + |
| 33 | +## Contributions |
| 34 | +We introduce VIBE (Video-to-text Information Bottleneck Evaluation), a novel framework that evaluates and selects VLM summaries without annotations or retraining. |
| 35 | +- Grounding Score: Measures how faithfully a summary reflects the video using pointwise mutual information between video and text. |
| 36 | +- Utility Score: Measures how informative the summary is for a downstream task. |
| 37 | + |
| 38 | +- Annotation-Free Rejection Sampling: VIBE ranks multiple VLM-generated summaries and selects the one that maximizes grounding and/or utility, supporting efficient human decision-making. |
| 39 | + |
| 40 | +## VIBE Framework |
| 41 | +<figure style="text-align: center;"> |
| 42 | + <img src="{{site.baseurl}}/images/post/VIBE_figure.png" alt="VIBE Mechanism Overview" height="auto" style="margin: auto; display: block;"> |
| 43 | + <figcaption>VIBE: Mechanism Overview</figcaption> |
| 44 | + <p></p> |
| 45 | +</figure> |
| 46 | + |
| 47 | +VIBE adapts the information bottleneck principle to video summarization: |
| 48 | + |
| 49 | +- Grounding approximates I(V;T), testing how well the video supports reconstruction of a masked summary. |
| 50 | +- Utility approximates I(T;Y), testing how well the summary compensates for missing video information to predict a task label. |
| 51 | + |
| 52 | +By maximizing both, VIBE selects concise, task-relevant summaries—without gold labels or retraining. |
| 53 | + |
| 54 | +## Results |
| 55 | +<figure style="text-align: center;"> |
| 56 | + <img src="{{site.baseurl}}/images/post/VIBE_table.png" alt="VIBE Results Table" height="auto" style="margin: auto; display: block;"> |
| 57 | + <figcaption>VIBE: Results Overview</figcaption> |
| 58 | + <p></p> |
| 59 | +</figure> |
| 60 | + |
| 61 | +We validate VIBE through user studies with 243 participants across three datasets: |
| 62 | +- LearningPaper24 (research talk videos) |
| 63 | +- LongVideoBench (long instructional clips) |
| 64 | +- SUTD-TrafficQA (short traffic clips) |
| 65 | + |
| 66 | +<figure style="text-align: center;"> |
| 67 | + <img src="{{site.baseurl}}/images/post/longvideobench_main.png" alt="LongVideoBench Results" height="auto" style="margin: auto; display: block;"> |
| 68 | + <figcaption>Correlation between Accuracy, Grounding, and Utility Scores</figcaption> |
| 69 | + <p></p> |
| 70 | +</figure> |
| 71 | + |
| 72 | +Key findings: |
| 73 | +- Accuracy Gains: VIBE-selected summaries boost task accuracy by up to 61.23%. |
| 74 | +- Speed Improvements: Response time drops by up to 75.77% compared to raw video. |
| 75 | +- Efficiency: VIBE achieves lower inverse efficiency scores (time/accuracy), showing superior speed–accuracy trade-offs. |
| 76 | +- Scalability: Grounding works in a fully self-supervised manner, while utility correlates strongly with higher task accuracy. |
| 77 | + |
| 78 | +## Impact |
| 79 | +VIBE reframes video caption evaluation from a human decision support perspective. Unlike reference-based metrics, it scales to unseen data, works with black-box VLMs, and requires no human annotations. |
| 80 | +This makes VIBE a practical plug-in for improving video summarization in real-world settings: from scientific video search to public safety monitoring. |
| 81 | + |
0 commit comments