Laya on Rockchip: FP16 RKNN adapter and full-API benchmarks

[Mirrorium227]

Hi everyone, I'd like to share an RKNN adapter for running Laya multilingual on Rockchip RK3576 boards, with Chinese intent classification and routing as the main use case.

Repository: Mirrorium227/Laya-rknn

Full-API performance

The adapter provides 96-, 256-, and 1024-token buckets, with three FP16 RKNN graphs per bucket. Here are the measured latencies for warmed-up Agent.predict calls on RK3576:

Token bucket	Original CPU P50 / P95 (ms)	RKNN adapter P50 / P95 (ms)	P50 speedup
96	844.39 / 950.37	102.09 / 103.87	8.27×
256	1994.87 / 2452.93	290.38 / 363.26	6.87×
1024	8640.98 / 11542.18	2809.23 / 3094.45	3.08×

Timing includes tokenization, CPU preprocessing and postprocessing, RKNN execution, and transfers between graphs. The CPU baseline uses eight threads. Each NPU call processes one row. The 96-token bucket uses 30 inputs with three runs each; the other buckets use 12 inputs with three runs each. Operator profiling runs separately.

How the adapter works

It keeps Agent.predict(state, questions) and the complete output, including answers, probabilities, action, and usage. The original multilingual checkpoint supplies the weights.

The CPU handles vocabulary lookup, two floating-point NaN guards, attention head key bias, the original action head, and output decoding. RKNN handles embedding normalization, the 22-layer mmBERT encoder, type embeddings, two decision Transformer layers, and the scorer. Device profiling records 531 NPU nodes and 26 CPU nodes per bucket; the NPU dtype breakdown is 529 F16, one FLOAT, and one INT8 mask Where.

The question, candidates, state, and special tokens determine the total encoded length. The adapter selects the smallest fitting bucket. Each question supports up to six candidates; inputs above 1024 tokens or six candidates raise ValueError.

Output checks

Evaluation covers 54 primary inputs and 144 additional MASSIVE inputs. FP16 changes one class relative to the original CPU model across these 198 inputs, a 0.51% flip rate. CPU and FP16 both score 13/18 on the existing manually labeled fixtures. The MASSIVE subset maps 17 intents to six functional domains; those mappings and the new long-input labels are pending human review. Calibration uses a separate set of 36 inputs.

Getting started

The tested environment uses Linux ARM64, Python 3.11.2, Laya 0.3.22, RKNN Toolkit/Runtime 2.3.2, and driver 0.9.8. Runtime is loaded from the project's own directory.

• English setup and usage guide
• 中文安装与使用说明
• FP16 prediction example
• Model and Runtime checksum manifest

The guides include model bundle downloads, preparation of the pinned Laya checkpoint, uv-based dependency setup, and command-line examples.

Thanks to the Laya authors for making the model available. I'd welcome feedback from other Rockchip users, especially on the CPU/NPU graph boundaries and Chinese intent-routing use cases. Results and reproducible examples can also be shared through the project's Issues.