You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 9b642ab
Browse filesBrowse the repository at this point in the historyBrowse files
Browse by **who receives assistance and what the assistant helps with**. A personal assistant may run on a phone, watch, glasses, desktop, or connected service. Streaming describes an interaction capability; it can support any of these applications. These routes overlap and do not assign each work to a single exclusive category.
6
+
7
+
## Personal Assistance
8
+
9
+
Persistent goals, routines, preferences, and everyday context make assistance useful to a particular person. Start with the [Personal Agents guide](PERSONAL_AGENTS.md), which connects research methods, implemented assistants, streaming models, and evaluation across devices.
| Everyday digital workflows |[Proactive Agent](papers/conference/ICLR2025/proactive-agent-shifting-llm.md); [dot, GAIA, MineContext](PROJECTS.md)| Which unstated task, contextual suggestion, or background update is worth surfacing? |
14
+
| Mobile preferences and routines |[PersonalAlign](papers/conference/ACL2026/personalalign-long-term-gui-intent.md); [FingerTip 20K](papers/conference/ICLR2026/fingertip-20k.md); [KnowU-Bench](papers/arxiv/2026-04/knowu-bench.md)| When does history justify a suggestion, and when should the assistant ask or respect rejection? |
15
+
| Wearable and physical-world assistance |[Satori](papers/conference/CHI2025/satori-proactive-ar-bdi.md); [ProMemAssist](papers/conference/UIST2025/promemassist-working-memory-wearable.md); [WatchGuardian](papers/arxiv/2025-02/watchguardian-personalized-jiti.md)| Does the user's inferred task, cognitive state, or user-defined behavior warrant an intervention? |
|[Need Help?](papers/conference/CHI2025/need-help-proactive-programming.md), [CodingGenie](papers/arxiv/2025-03/codinggenie-proactive-programming-assistant.md)| Offer programming help at useful moments while preserving user control. | Human studies, interruption cost, acceptance, and task outcomes. |
22
+
|[Ask or Assume?](papers/conference/EMNLP2026/ask-or-assume-coding-agents.md), [HiSentinel](papers/arxiv/2026-09/hisentinel-coding-agent-intervention.md)| Ask about missing requirements, redirect a risky action, or continue execution. | Clarification burden and downstream success; distinguish intervention on another agent from assistance initiated for a human. |
23
+
|[ProAgentBench](papers/arxiv/2026-02/proagentbench.md), [ProCodeBench](papers/arxiv/2026-05/procodebench-proactive-coding-assistants.md)| Assess assistance opportunities in desktop or coding workflows. | See protocol and data columns in the [Benchmark Matrix](BENCHMARKS.md). |
|[InsightToast](papers/arxiv/2026-08/insighttoast-meeting-side-channel.md)| Retrieve and surface missing context in a peripheral meeting channel. | Information utility and disruption during collaboration. |
30
+
|[Speak for Me](papers/arxiv/2026-09/speak-for-me-meeting-delegation.md)| Decide whether, when, and what to contribute on a participant's behalf. | Situational awareness, floor control, and delegation boundaries. |
31
+
|[DocuTeam](papers/arxiv/2026-09/docuteam-evolving-documents.md)| Use document changes to initiate or redirect collaboration. | Relevance of contributions to the evolving artifact. |
|[PACE](papers/conference/ICRA2025/pace-action-completion-assistance.md)| Use estimated human action completion to time physical assistance. | Idle time and fluency in structured collaboration. |
38
+
|[PACT](papers/arxiv/2026-05/pact-continual-robot-assistance.md)| Ask for missing context or act using cross-day interaction history. | Assistance accuracy versus clarification cost. |
39
+
|[ProRobo](papers/arxiv/2026-09/prorobo-cognitive-action-reasoning.md)| Anticipate needs in embodied interaction. | Consult the evidence card for task and deployment limits. |
40
+
41
+
Personalization can overlap with this route, but a collaborative robot is not automatically a personal assistant. Household assistance is an intended application to track; controlled assembly or simulated multi-day routines do not establish an open-ended home deployment.
42
+
43
+
## Domain-Specific Assistance
44
+
45
+
| Starting resources | Why it belongs here | Evidence boundary |
46
+
|---|---|---|
47
+
|[See, Infer, Intervene](papers/arxiv/2026-06/see-infer-intervene-retail.md)| Retail assistance explicitly selects engagement actions or `HOLD`. | Scripted and small-scale settings; inspect the perception-to-policy gap. |
48
+
|[AI Assistants Overassist](papers/arxiv/2026-07/ai-assistants-overassist.md)| Tutoring tests whether intervention arrives too early or supplies too much. | Learning outcomes and user independence matter alongside immediate correctness. |
49
+
|[Oops, Not Now / PEARL](papers/arxiv/2026-09/pearl-proactive-gameplay-support.md); [Proact-VL](papers/arxiv/2026-03/proact-vl-realtime-companions.md)| Gameplay assistance and commentary expose timing and unwanted-interruption trade-offs. | Commentary, requested guidance, and spontaneous assistance have different participation expectations. |
50
+
51
+
## How These Routes Connect
52
+
53
+
Use [Projects & Products](PROJECTS.md) for implementation/access evidence, [Streaming Proactivity](STREAMING.md) for model mechanisms, [Supporting Infrastructure](INFRASTRUCTURE.md) for sensing/memory/runtime roles, and the [Research Map](RESEARCH_MAP.md) for scientific questions. The [full bibliography](README.md#papers) remains the source of publication metadata; these routes select representative entries rather than duplicate that catalog.
-**Streaming multimodal benchmarks by task** (see [evaluation routes](STREAMING.md#evaluation-routes)): ESTP-Bench, StreamGaze, SOVBench, Live Gaming Benchmark, APM-Bench, DuplexAct-Bench, Full-Duplex Floor Selection, Live Assistant, TRACE, TIMELI, ProReady-QA, OmniAssistBench, OmniPro, EgoPro-Bench, OmniMMI, StreamArena, IPIBench, EgoServe, StreamSoccer.
13
15
14
16
## Benchmark Matrix
15
17
@@ -22,6 +24,10 @@ This page compares benchmarks by what they actually test. The goal is to make be
22
24
|**Ambig-SWE**| Ambig-SWE | Software engineering | Repository, underspecified issue, and interactive user responses | Detect underspecificity, ask targeted questions, and use the answer | Simulator backed by withheld issue information | SWE-bench Verified-derived interactive benchmark | Detection, clarification quality, post-interaction task resolution |[OpenReview](https://openreview.net/forum?id=X2yzXtH4wp) · [GitHub](https://github.com/sani903/InteractiveSWEAgents)|
23
25
|**ProactiveBench**| Proactive Agent | Desktop activity | Event streams and desktop context | Predict useful next tasks | Simulated user feedback / reward model | Synthetic + event-derived | Task prediction usefulness, acceptance proxy |[GitHub](https://github.com/thunlp/ProactiveAgent)|
24
26
|**OmniMMI**| OmniMMI | Streaming multimodal interaction | Continuous video, audio, speech, and dialogue history | Trigger proactive alerts and turn-taking while supporting reactive QA | Standing user queries | Human-annotated benchmark | Alert timing, turn-taking, streaming QA, multi-turn interaction |[CVPR 2025](https://openaccess.thecvf.com/content/CVPR2025/html/Wang_OmniMMI_A_Comprehensive_Multi-modal_Interaction_Benchmark_in_Streaming_Video_Contexts_CVPR_2025_paper.html) · [Website](https://omnimmi.github.io/)|
27
+
|**ESTP-Bench**| Eyes Wide Open | Egocentric streaming video | Past/current first-person frames and evolving questions | Coherent just-in-time responses during perception | Evolving question context | Benchmark with instruction-tuning resources | ESTP-F1; coherence, response timing, efficiency |[Paper](https://arxiv.org/abs/2510.14560) · [Code](https://github.com/SooLab/EyeWO)|
28
+
|**StreamGaze**| StreamGaze | Gaze-conditioned egocentric video | Causal video plus gaze trajectories | Anticipation and gaze-triggered alerts, alongside past/present reasoning | Measured gaze as an attention cue | 285 videos; 8,521 QA pairs across 10 tasks | Task-specific reasoning/alert evaluation; proactive tasks reported separately |[Paper](https://arxiv.org/abs/2512.01707) · [Code](https://github.com/daeunni/StreamGaze)|
29
+
|**SOVBench-O / SOVBench-T**| StreamOV | Streaming omni-video interaction | Continuous audio/video and multi-turn queries | Contextual inference; trigger on present evidence, stay silent for absent evidence | Standing queries and dialogue history | O: 172 sessions / 1,739 QA turns; T: 226 positive/negative samples | QA accuracy; trigger precision, recall, F1 |[Paper / protocol](https://arxiv.org/html/2605.25621v1)|
30
+
|**Live Gaming Benchmark**| Proact-VL | Live companion commentary/guidance | Game video and scenario-conditioned interaction | Choose timely commentary, co-commentary, or guidance | Scenario roles / user guidance request | Game and guidance evaluation data | LLM Score, CC, F1, Time-Diff, PAUC |[Code / protocol](https://github.com/microsoft/AnthropomorphicIntelligence/tree/main/Proact-VL) · [Dataset](https://huggingface.co/datasets/oaaoaa/LiveGamingBenchmark)|
25
31
|**Full-Duplex Floor Selection**| Full-Duplex Speech Models Take the Floor When Asked, Not When Needed | Always-on speech interaction | Context-matched monologues with controlled false-fact, hazard, missing-word, address, and silence triggers | Self-select to speak because intervention is needed, not merely because the floor is open | Speaker monologues rather than personalized users | Ten-condition controlled diagnostic | Speech-initiation probability, non-empty response rate, false-fact challenge, hazard warning |[arXiv](https://arxiv.org/abs/2609.19596)|
26
32
|**Live Assistant**| Live Assistant | Livestream social assistance | Native audio-video, comments, gifts, viewer dynamics, and room metadata | Choose silence, private memory, or a grounded message to the right recipient | Hosts and viewers reconstructed from real livestreams | 275 clips and 13,812 human-reviewed decision intervals | State, recipient, and task accuracy |[arXiv](https://arxiv.org/abs/2609.27303)|
27
33
|**ProAction**| Cognitive Action Reasoning for Proactive Robots | Human-centered embodied assistance | Visual observations, audio, and text without an action instruction | Infer a cognitively grounded high-level robot action from human and environmental cues | Human-refined action judgments informed by affective theory of mind | 10,000 samples across 12 scenarios and five scenes | Action reasoning, modality contribution, subject-disjoint transfer, human evaluation |[arXiv](https://arxiv.org/abs/2609.23486)|
@@ -94,7 +100,7 @@ This page compares benchmarks by what they actually test. The goal is to make be
94
100
| How do we evaluate personalization? | RobotEQ 3.0, KnowU-Bench, FingerTip 20K, VitaBench 2.0, UserVille, Ψ-Bench | They include profiles, preferences, or user-specific trajectories. |
95
101
| How do we evaluate memory as a proactive substrate? | APM-Bench, CogEval-Bench, MemEye, TWIST, VitaBench 2.0, ProActEval | They test memory formation, retrieval, evidence preparation, intent emergence, missing-evidence awareness, or whether memory should trigger an intervention. |
96
102
| How do we evaluate computer-use agents? | Act2Intention Bench, CONFLICTGUI, SWE-Intervene, OverAct, ProAgentBench, GUIDE, PIRA-Bench, KnowU-Bench, Pare-Bench, ProCodeBench | They connect proactive behavior to GUI, mobile, IDE, tool calls, or workflow contexts; CONFLICTGUI and OverAct test restraint, while SWE-Intervene tests pre-execution control. |
97
-
| How do we evaluate proactive streaming video or speech? | APM-Bench, DuplexAct-Bench, Live Assistant, TRACE, TIMELI, ProReady-QA, OmniAssistBench, OmniPro, EgoPro-Bench, OmniMMI, StreamArena, IPIBench, EgoServe, StreamSoccer | They require causal processing of an ongoing stream and timely outputs rather than offline clip answering; TRACE audits execution conditions and DuplexAct-Bench adds proactive initiation and active silence. |
103
+
| How do we evaluate proactive streaming video or speech? |ESTP-Bench, StreamGaze, SOVBench, Live Gaming Benchmark, APM-Bench, DuplexAct-Bench, Live Assistant, TRACE, TIMELI, ProReady-QA, OmniAssistBench, OmniPro, EgoPro-Bench, OmniMMI, StreamArena, IPIBench, EgoServe, StreamSoccer | They require causal processing of an ongoing stream and timely outputs rather than offline clip answering; TRACE audits execution conditions and DuplexAct-Bench adds proactive initiation and active silence. |
98
104
| How do we evaluate proactive critique? | RPCBench, MMPCBench, AskBench | They require the assistant to challenge or repair a faulty premise instead of complying silently; RPCBench additionally tests localization, handling strategy, and evidence faithfulness in recommendation. |
99
105
| How do we evaluate ask-versus-stop clarification? | FinInteract, IdeaAMBIG, OR-Clarify, ClarifyBench, AskBench | They evaluate whether missing information justifies another question and whether the agent can stop without silently assuming critical information; FinInteract additionally tests whether the answer is integrated under competing valid interpretations. |
100
106
| How do we evaluate social intervention outcomes? | ProMediConv, CC-Mediation, ProMediate, ProACT Collaboration Bench | They evaluate intervention strategies or points and downstream collaborative or interpersonal effects rather than response quality alone. |
Help keep the collection useful, traceable, and easy to maintain. Bibliographic content and the LowEntropyAI website share the same Markdown sources.
4
+
5
+
## Add or correct a paper
6
+
7
+
1. Check `README.md` for the title and canonical source URL to avoid duplicates.
8
+
2. Confirm a user-facing proactive decision: infer an unstated need, choose an intervention moment, ask, suggest, remind, act, or deliberately stay silent. Describe the actual decision and its evaluation.
9
+
3. Add a row under the best-fitting `## Papers` section using the existing five-column schema: date, title, venue/source, tags, resources. Use existing tags from the vocabulary.
10
+
4. Preserve publication uncertainty: a submission, preprint, repository claim, and verified conference acceptance are distinct statuses.
11
+
5. Add a note under `papers/arxiv/YYYY-MM/` or `papers/conference/VENUE/`. Use the exact headings `Why It Matters`, `Proactivity Signal`, `Evaluation Setup`, `Key Limitations`, and `Use For`. Ground results in primary sources and state evaluation limits.
12
+
6. Link the note from the paper's README row using `[Notes](papers/.../note.md)` or the existing Notes badge format.
13
+
7. Update `RESEARCH_MAP.md`, `BENCHMARKS.md`, or `STREAMING.md` when the work adds a useful research route or evaluation target.
14
+
15
+
For a benchmark, keep all nine columns in `BENCHMARKS.md`. Do not compare scores across incompatible datasets, hardware, protocols, or timing boundaries.
16
+
17
+
## Add a project
18
+
19
+
Use `PROJECTS.md`. Record what triggers the assistant, what context it sees, what user-facing behavior it chooses, release/access status, evidence boundaries, canonical links, and checked date. Distinguish a memory/capture/serving component from a complete proactive assistant.
20
+
21
+
## Place a resource in the collection
22
+
23
+
Use [Applications](APPLICATIONS.md) and [Personal Agents](PERSONAL_AGENTS.md) for selective reading routes. A paper keeps its canonical metadata in the README; projects keep implementation/access facts in PROJECTS; streaming mechanisms stay in STREAMING; benchmark protocols stay in BENCHMARKS. Link between these views instead of copying full records.
24
+
25
+
Record application, device/modality, decision mechanism, and resource type separately. Do not infer personalization from a camera interface or learned proactivity from a scheduler. Include supporting components through [Infrastructure](INFRASTRUCTURE.md) with their specific role. A model may support personal interaction without being a complete personal assistant. The [restructuring design](docs/RESTRUCTURE_PLAN.md) defines the scope and staged data plan.
26
+
27
+
## Verify
28
+
29
+
```sh
30
+
npm ci
31
+
npm run check
32
+
git diff --check
33
+
```
34
+
35
+
Preview website changes with `npm run dev`. Verify desktop and mobile reading layouts and relevant interactions. See [WEBSITE.md](WEBSITE.md) for architecture and deployment.
0 commit comments