Bert-VITS2-MNN

✨ Android port of Bert-VITS2, powered by alibaba-MNN for on-device inference.

🌐 Language: 中文 · English

🧠 Overview

This project provides a sample implementation of Bert-VITS2 (v2.3) running fully offline on Android. It now supports TTS in Chinese / Japanese / English, and some Chinese speakers also accept mixed Chinese + English input:

🧠 Distilled multilingual BERT — The Chinese BERT is a self-distilled variant trained on Wikipedia-zh and SkyPile (~10M sentences), shrunk down to ~30MB. The Japanese / English BERTs are compressed in the same fashion so they all fit on-device.
🏗 MNN — The whole BV2 pipeline is rebuilt on top of MNN, referencing the official ONNX inference code. (Direct .pth → MNN doesn't work, so ONNX → MNN it is.)
🧹 cppjieba / cpptokenizer / openjtalk — Replace the Python-side jieba, HuggingFace tokenizers and open-jtalk respectively. BV2-specific text preprocessing is reimplemented in Kotlin.
📦 AAR distribution — All inference code has been extracted into bertvits2-infer-wrapper, which can be published as an AAR (with its native dependencies) so other apps can integrate with one coordinate. See PUBLISHING.md.

The entire pipeline runs fully offline on the device — no network required.

🔬 Pipeline

Input Text (ZH / JP / EN / ZH+EN Mix)
   ↓
Tokenization + G2P (cppjieba + cpptokenizer + openjtalk + kotlin code)
   ↓
BERT embedding (distilled ZH / JP / EN model)
   ↓
Encoder + Emb + DP/SDP + Flow + Decoder (BV2 infer by MNN)
   ↓
Waveform output (.wav)

🎵 Sample Audio

The speakers bundled in the demo app are listed below. The full list lives in VoiceViewModel.kt. Chinese speakers are trained on public voice sets from Arknights, Wuthering Waves, Bilibili New Year Gala videos and Genshin Impact; Japanese / English speakers come from the public voice sets of their respective games. Some English samples have minor distortion, and the Japanese samples sound a bit flat — most likely because the fine-tuning data mixed in some messy stuff and the model learned the wrong things (fog). Feel free to train your own characters and you should get noticeably better results~

Character	Language	Sample Rate	Text	Audio
Ch'en	Chinese	44100 Hz	博士，当初在龙门，我不该放你走的。	🔊 Play
Faruzan	Chinese	44100 Hz	旅行者，好久不见。	🔊 Play
Ganyu	Chinese	44100 Hz	工作还没有做完，又要开始搬砖了。	🔊 Play
22	ZH + EN mix	22050 Hz	RTX 5090 将于明年发布，敬请期待！	🔊 Play
APPLe	English	44100 Hz	Greetings, madam. I am here. Clouds help predict the weather.	🔊 Play
Sonetto	English	44100 Hz	Timekeeper, at your service. The stars shine bright tonight.	🔊 Play
Vertin	English	44100 Hz	The storm is coming. We must prepare ourselves.	🔊 Play
Yae Miko	Japanese	44100 Hz	たびびと、きょうはどんなおもしろいほんをもってきてくれたの？もしないようがつまらなかったら、わたし、へんしゅうぶに『しげき』にいこうかな～？	🔊 Play
Yoimiya	Japanese	44100 Hz	こんにちは、皆さん。今日は素晴らしい一日ですね。	🔊 Play
Tsubaki	Japanese	44100 Hz	あなたといると、なぜか落ち着くの。	🔊 Play
Yajuu Senpai	Japanese	44100 Hz	にじゅうよんさいはがくせいです	🔊 Play

🎯 22 kHz Base Model

A new 22 kHz base model lives under base_model_22k/.

After plenty of testing and trade-offs, 22 kHz turned out to strike the best balance between performance and quality. The base model is trained on character voices from four games — Genshin Impact, Wuthering Waves, Zenless Zone Zero and Honkai: Star Rail — and is provided for study / non-commercial use only. The decoder hyper-parameters have been adjusted accordingly; the upstream training code contains some hard-coded values that may need small tweaks, which are not covered here.

If you want to fine-tune from this base model, please pair it with BertVITS2 v2.3.

📊 On-device Inference Performance

A reference measurement on a mid-flagship SoC, intended as a rough order-of-magnitude indicator:

Item	Value
Device	Qualcomm Snapdragon 888
Model sample rate	22050 Hz
Model size	≈ 29.7 MB (full 22050 Hz BV2 modules in MNN, int8 weight-quant)
Test text	"RTX 5090 将于明年发布，敬请期待" (ZH + EN mixed, ~10 CJK chars + 1 English token)
End-to-end latency	≈ 1856 ms (text preprocessing + Encoder + Flow + Decoder)
Synthesized audio	≈ 5.20 s (22050 Hz × 114688 frames)
RTF	≈ 0.357 (< 1 means inference is faster than real-time playback, suitable for streaming use)
Estimated throughput	≈ 2.80 s of audio synthesized per wall-clock second

Real numbers fluctuate with SoC tier, thermal state, scheduler behavior, text length, and backendConfig difference — the figures above are order-of-magnitude only. Newer platforms such as Snapdragon 8 Gen2 / 8 Gen3 are typically another 30%–50% faster.

📦 Integrating as AAR

The bertvits2-infer-wrapper module (along with all its native dependencies) is maven-publish ready. Producing AARs for downstream apps is a single command:

./gradlew publishAars

See PUBLISHING.md for the full guide — custom coordinates, local repository layout, downstream Gradle wiring, etc.

⚡ Build Guide

Clone with submodules

GIT_LFS_SKIP_SMUDGE=1 git clone --recurse-submodules git@github.com:Voine/Bert-VITS2-MNN.git

# for windows powershell
$env:GIT_LFS_SKIP_SMUDGE=1; git clone --recurse-submodules git@github.com:Voine/Bert-VITS2-MNN.git

cd Bert-VITS2-MNN

If already cloned:

git submodule update --init --recursive

Build for Android

📦 Android Studio is recommended — just open the root directory.

# From project root
./gradlew assembleRelease

🛁 Git LFS

Some assets (e.g. .mnn) are stored via Git LFS:

git lfs install
git lfs pull

To track files (if contributing):

git lfs track "*.mnn"

🛠️ Submodules

Library	Path
MNN	`third_party/MNN`
cppjieba	`third_party/cppjieba`
tokenizer-cpp	`third_party/tokenizers-cpp`

💡 Notes — BERT Distillation

The Chinese BERT is distilled from chinese-roberta-wwm-ext-large and aggressively compressed for mobile (the naïve port is ~1.2 GB...). The JP / EN BERTs go through the same distill + quantize flow.

See distill/README.md for the distillation scripts.

💡 Notes — Replacing With Your Own Model

Train a model following the BertVITS2 instructions. Only v2.3 is supported; this project is built against commit 13424595, so please stick to that version. If you target the 22 kHz sample rate, use the config.json / G_0.pth shipped in base_model_22k/.
Convert your .pth to ONNX with the official export_onnx.py.
Convert each ONNX module to MNN via MNN Convert:

./MNNConvert --modelFile your_path_to_onnx.onnx --MNNModel your_path_to_mnn.mnn --framework ONNX --bizCode MNN --weightQuantBits 8 --weightQuantAsymmetric

Drop the .mnn files under bertvits2-jni/src/main/assets/bv2_model, then update the loading code in BertVITS2SimpleInferImpl.kt accordingly.

💡 Notes — third_party

third_party/{cppjieba, tokenizers-cpp, MNN} is only there to provide headers. If you want to rebuild tokenizer-cpp and replace the prebuilt libtokenizers_c.a / libtokenizers_cpp.a, remember to flip add_special_tokens to true in huggingface_tokenizer.cc.

💡 Other Notes

The BERT model mainly serves as an auxiliary input to the whole system — in many cases dropping it doesn't break inference, the voice just ends up a bit duller / flatter / slightly off. Even after distillation the BERT model can still be sizeable, so when integrating into a real product, feel free to trade it off based on your size budget.
Since the demo app bundles inference for multiple languages, a lot of components are wired up as lazy-init. As a result the first inference is noticeably slower. In a real product you can either hide the lazy-init work behind some other flow (e.g. splash / preload), or optimize the lazy-init path itself.

📋 Project Layout

├── app/                              # Demo app
│   └── src/main/
│       ├── assets                    # mnn bert model, cppjieba dict, mnn bv2 model
│       └── java/                     # UI / ViewModel
├── bertvits2-infer-wrapper/          # Public inference entry (publishable as AAR)
├── bertvits2-jni/                    # Bert-VITS2 inference JNI
├── text-preprocess/                  # ZH / JP / EN / mixed text preprocessing
├── cppjieba/                         # cppjieba interface
├── cpptokenizer/                     # cpptokenizer interface
├── openjtalk/                        # open-jtalk interface (Japanese G2P)
├── base_model_22k/                   # 22 kHz base model
├── distill/                          # BERT distillation scripts
└── third_party/                      # Vendored headers

🙌 Credits

Standing on the shoulders of these giants:

🛠️ Roadmap

Integrate into ChatWaifu_Mobile

✨ Intro Video

Video

Navigation Menu

Bert-VITS2-MNN

🧠 Overview

🔬 Pipeline

🎵 Sample Audio

🎯 22 kHz Base Model

📊 On-device Inference Performance

📦 Integrating as AAR

⚡ Build Guide

Clone with submodules

Build for Android

🛁 Git LFS

🛠️ Submodules

💡 Notes — BERT Distillation

💡 Notes — Replacing With Your Own Model

💡 Notes — third_party

💡 Other Notes

📋 Project Layout

🙌 Credits

🛠️ Roadmap

✨ Intro Video

Disclaimer

This project is for study and personal experimentation only. Commercial use is not permitted.

It must not be used for any purpose that violates the laws or regulations of your jurisdiction, nor for any politically sensitive purpose.

FilesExpand file tree

README_EN.md

Latest commit

History

README_EN.md

File metadata and controls

Bert-VITS2-MNN

🧠 Overview

🔬 Pipeline

🎵 Sample Audio

🎯 22 kHz Base Model

📊 On-device Inference Performance

📦 Integrating as AAR

⚡ Build Guide

Clone with submodules

Build for Android

🛁 Git LFS

🛠️ Submodules

💡 Notes — BERT Distillation

💡 Notes — Replacing With Your Own Model

💡 Notes — third_party

💡 Other Notes

📋 Project Layout

🙌 Credits

🛠️ Roadmap

✨ Intro Video

Disclaimer

This project is for study and personal experimentation only. Commercial use is not permitted.

It must not be used for any purpose that violates the laws or regulations of your jurisdiction, nor for any politically sensitive purpose.