Skip to content

feat(message-toolbar): add assistant reply actions and read aloud - #518

Open
AtomsH4 wants to merge 29 commits into
CherryHQ:v0.2from
AtomsH4:codex/message-action-toolbar
Open

feat(message-toolbar): add assistant reply actions and read aloud#518
AtomsH4 wants to merge 29 commits into
CherryHQ:v0.2from
AtomsH4:codex/message-action-toolbar

Conversation

@AtomsH4

@AtomsH4 AtomsH4 commented Aug 13, 2026

Copy link
Copy Markdown

Summary

  • add controlled copy, read-aloud, stop, and regenerate actions to the assistant message toolbar
  • project speakable reply text from structured message parts while excluding non-spoken Markdown and citation content
  • select matching system TTS voices using explicit translation language first, then lightweight Chinese, English, Japanese, and Korean character signals
  • manage chunked speech sessions, lifecycle cancellation, stale callbacks, and actionable missing-voice feedback without changing message persistence
  • document message-presentation and chat-workspace ownership boundaries
image

Why

Assistant replies lacked mobile toolbar parity with the desktop implementation. Using the system default TTS language also caused Chinese and other CJK replies to be spoken with an incorrect voice or cadence.

User impact

Users can copy, regenerate, start, and stop reading assistant replies from the message toolbar. Read-aloud now prefers a compatible installed system voice and explains how to install a missing voice when the device cannot provide one.

Validation

  • 8 focused Jest suites: 123 tests passed
  • pnpm typecheck:app
  • targeted oxfmt and oxlint
  • pnpm packages:build
  • Android ARM64 release: :app:assembleRelease
  • APK signature, architecture, embedded bundle, speech module, and icon font verified
  • release APK installed and cold-started successfully on an ARM64 emulator

AtomsH4 added 26 commits August 13, 2026 17:28
@AtomsH4
AtomsH4 marked this pull request as ready for review August 13, 2026 10:51
Copilot AI lite review requested due to automatic review settings August 13, 2026 10:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds mobile-parity assistant reply actions (copy / regenerate / read aloud / stop) to the chat message toolbar, including speakable-text projection, language hinting, and voice selection for Expo TTS, while keeping playback state ephemeral to the chat workspace.

Changes:

  • Add assistant message toolbar actions (copy, regenerate, read aloud/stop) and wire them from ChatWorkspace into MessageList / AssistantMessageRow.
  • Implement read-aloud text projection + Markdown/citation stripping, language hinting, voice resolution, and chunked speech session management with lifecycle cleanup.
  • Add i18n strings and Jest coverage for the new TTS utilities and hook (with limited coverage for the UI toolbar wiring).

Reviewed changes

Copilot reviewed 24 out of 25 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/frontend/i18n/locales/zh-cn.json Adds localized strings for new assistant message actions and read-aloud errors.
src/frontend/i18n/locales/en-us.json Adds English strings for new assistant message actions and read-aloud errors.
src/frontend/features/chat/workspace/utils/splitReplyReadAloudText.ts Splits spoken text into safe chunks with preferred boundaries and surrogate-pair safety.
src/frontend/features/chat/workspace/utils/resolveReplyReadAloudVoice.ts Selects the best matching system voice by base language, exact locale, and quality.
src/frontend/features/chat/workspace/utils/tests/splitReplyReadAloudText.test.ts Tests chunk sizing and boundary selection, including emoji/surrogate handling.
src/frontend/features/chat/workspace/utils/tests/resolveReplyReadAloudVoice.test.ts Tests locale normalization, exact-match preference, quality ordering, and non-mutation.
src/frontend/features/chat/workspace/hooks/useReplyReadAloud.ts Introduces the read-aloud session hook with chunk playback, cancellation, and lifecycle cleanup.
src/frontend/features/chat/workspace/hooks/tests/useReplyReadAloud.test.tsx Adds extensive tests for session races, cleanup behavior, chunk sequencing, and errors.
src/frontend/features/chat/workspace/ChatWorkspace.tsx Wires toolbar actions (copy/regenerate/read-aloud) into MessageList, with user feedback alerts.
src/frontend/features/chat/workspace/tests/ChatWorkspace.test.tsx Updates workspace test scaffolding/mocks for regenerate and read-aloud hook presence.
src/frontend/features/chat/README.md Documents workspace ownership boundaries and read-aloud session behavior.
src/frontend/components/messagePresentation/types.ts Adds AssistantMessageActions + AssistantReadAloudInput and threads actions through list props.
src/frontend/components/messagePresentation/README.md Documents read-aloud projection boundaries and consumer-owned playback state.
src/frontend/components/messagePresentation/messageRow/utils/resolveAssistantReadAloudLanguage.ts Adds script-signal language hinting (explicit targetLanguage first, then CJK/Latin heuristics).
src/frontend/components/messagePresentation/messageRow/utils/projectAssistantMessageReadAloud.ts Projects speakable assistant reply text while excluding code, citations, URLs, etc.
src/frontend/components/messagePresentation/messageRow/utils/copyAssistantMessageText.ts Produces a “controlled copy” payload from visible assistant message parts.
src/frontend/components/messagePresentation/messageRow/utils/tests/resolveAssistantReadAloudLanguage.test.ts Tests explicit language, script detection, thresholds, and documented limitations.
src/frontend/components/messagePresentation/messageRow/utils/tests/projectAssistantMessageReadAloud.test.ts Tests translation preference, Markdown cleanup, citation stripping, and reject cases.
src/frontend/components/messagePresentation/messageRow/utils/tests/copyAssistantMessageText.test.ts Tests copy projection includes only intended visible content.
src/frontend/components/messagePresentation/messageRow/components/AssistantMessageRow.tsx Adds the assistant message toolbar UI and connects to AssistantMessageActions.
src/frontend/components/messagePresentation/messageRow/components/tests/AssistantMessageRow.test.tsx Adds i18n mocking but does not yet cover new toolbar interactions.
src/frontend/components/messagePresentation/index.ts Re-exports new action/input types from the module entrypoint.
src/frontend/components/messagePresentation/components/MessageList.tsx Accepts assistantActions, passes to assistant rows, and updates list extraData.
pnpm-lock.yaml Locks the new expo-speech dependency.
package.json Adds expo-speech dependency for read-aloud support.
Files not reviewed (1)
  • pnpm-lock.yaml: Generated file

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +67 to +74
{actions && message.status !== 'pending' ? (
<View className="flex-row items-center" testID="assistant-message-toolbar">
{copyText ? (
<Button
accessibilityLabel={t(isCopied ? 'chat.messageActions.copied' : 'common.copy')}
hitSlop={6}
icon={isCopied ? <CheckIcon /> : <CopyIcon />}
onPress={() => actions.onCopy({ messageId: message.id, text: copyText })}
Comment on lines 208 to 210
<MessageList
assistantActions={assistantActions}
key={listRenderKey}
@eeee0717

Copy link
Copy Markdown
Collaborator

补充从项目规范、React composition 和 React Native 实践角度的 review。整体方向合理,但建议合并前处理下面这些问题:

  1. Android 冷初始化的取消竞态useReplyReadAloud.ts 在没有语言提示时会直接调用 Speech.speak()expo-speech@57.0.1 的 Android TTS 尚未 ready 时会保留待播放内容;此时 stop、退后台或失焦虽然能让 JS session 失效,却未必能撤销 native 初始化完成后的播放。建议统一等待 TTS 初始化,等待后再次校验 session token,再调用 speak,并补一个冷启动后立即停止/退后台的设备回归。

  2. 相邻按钮的命中区域重叠AssistantMessageRow.tsx 中三个 sm 纯图标按钮零间距排列。该尺寸实际是 32x32,hitSlop={6} 会让横向命中区重叠 12pt,纵向扩展则会被同高父容器裁掉;React Native 在重叠区域按 sibling 层级选择目标,按钮边缘可能触发相邻的朗读或重新生成。建议在布局中给每个 action 分配互不重叠的 44x44 命中空间。

  3. 工具栏状态使整张虚拟列表失效ChatWorkspace.tsx 把易变 ID 和 handlers 打成一个 assistantActions 对象,MessageList.tsx 又把它放进 renderItem 依赖和 extraData。复制反馈、朗读切换和 busy 状态都会使所有可见消息行失效,每个助手行还会重新执行完整的 copy/read-aloud projection。建议抽出 memoized toolbar leaf,传入每行 primitive 状态和稳定 callbacks,并按稳定的 message parts 缓存 projection。

  4. 文本投影存在错误结果resolveAssistantReadAloudLanguage.ts 会把所有主要使用 Latin 字符的语言判成 en-US,纯汉字日语会判成 zh-CN;没有可靠元数据时返回 undefined 更稳妥。projectAssistantMessageReadAloud.ts 还会无条件删除全部 *_,例如 snake_case 变成 snakecase2 * 3 变成 2 3。这里更适合基于 Markdown 解析结果做 projection,而不是继续扩展正则清理器。

  5. Composition API 有冲突状态MessageListProps 同时允许 assistantActionsrenderAssistantMessage,但运行时后者会静默覆盖前者。动态列表使用 render prop 本身没有问题,也不需要为了模式而引入 compound components;建议保留单一显式组合入口,或用 discriminated union 表达 default/custom assistant row 两种互斥变体。

  6. 测试不符合仓库规则:新增的 642 行 hook suite 中存在大量 toHaveBeenCalled* 断言,而仓库 AGENTS.md 明确要求不要测试 mock 是否被调用。建议抽出纯 session/reducer 逻辑测试状态转换,native 生命周期与工具栏交互由 agent-device 覆盖。

Cherry UI 使用是符合规范的:这里正确使用了 @cherrystudio/ui/componentsButton/PrismSweep@cherrystudio/app-icons 和无障碍标签;直接用 View 做布局也合理。上述触摸问题应继续基于 Cherry UI Button 修复,不建议换成裸 Pressable。如果视觉上允许,目标分支已有语义合适的 SpeechIcon,复用它也可以避免当前唯一的 MaterialSymbols.ttf 二进制冲突。

这个 PR 涉及 30 个文件、约 2.1k 行新增,把工具栏、文本 projection、语言/voice 策略和 native TTS 生命周期放在了一起,review 和回归范围都偏大。建议作者用 gh-stack 拆为同一 topic 前缀下的三层 PR(bottom -> top):

v0.2
<- message-actions/toolbar
<- message-actions/read-aloud-projection
<- message-actions/read-aloud-runtime

拆好分支后可以用 gh stack init --base v0.2 <bottom> <middle> <top> 建立 stack,再用 gh stack submit --auto --open 发布。CI 目前是绿的,但上述 device race、触摸几何和列表失效范围并未被现有检查覆盖。

@eeee0717

Copy link
Copy Markdown
Collaborator

修改后需要按照最新的AGENTS.md对代码进行调整,添加了一些规范

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants