This study proposes "Speak2UI", a novel voice-based mobile UI control system that leverages pre-trained large language models to minimize data dependency and enable direct voice control of low-level UI elements. The system comprises Voice Activity Detection-based speech recognition, GPT-4o-mini command parsing supporting ten action types with context-aware filtering, and UI control modules. Performance evaluation compared command recognition accuracy with the existing system.
speak2ui_demo.mp4
Test Environment: Samsung S8+ (Android API level 34)
This section outlines the environment settings required to build the project successfully.
- Compile Sdk Version: 36
- Min Sdk Version: 30
- Java Version: 21
- Kotlin Compiler Extension Version: 1.5.0
-
Clone the project
git clone https://github.com/Jin-A-Park/speak2ui.git
-
Set up the API Key This project requires an OpenAI API key to function.
- Create a file named
local.propertiesin the root directory of the project. - Add your OpenAI API key to the file as shown below:
OPENAI_API_KEY=YOUR_OPENAI_API_KEY_HERE - Create a file named
-
Open the project in Android Studio
- Launch Android Studio and select
Opento open the cloned project folder. - Wait for the Gradle sync to complete.
- Launch Android Studio and select
-
Run the app
- Select your desired emulator or physical device.
- Click the 'Run' button to build and run the application.
- Enable
Speak2UIin Settings > Accessibility. - Turn on
Display over other appsin Settings > App info > Speak2UI. - Tap the MIC button in the top-left corner.
- Tap
On, and you’re all set!
@article{ART003300241,
author = {Sung-Jin Park and Jin-A Park and Ha-Young Oh},
title = {Speak2UI: Voice Assistance Technology for Mobile Accessibility},
journal = {Journal of the Korea Institute of Information and Communication Engineering},
year = {2026},
volume = {30},
number = {1},
pages = {46--53},
issn = {2234-4772}
}