Skip to content

Commit 9510334

Browse files
authored
Timestamps in parakeet_runner (pytorch#16545)
### Summary Enable computation of timestamps within `parakeet_runner` through a `--timestamps` flag (`none|token|word|segment|all`). Followed reference implementation from [NVIDIA-NeMo/NeMo](https://github.com/NVIDIA-NeMo/NeMo/tree/bf583c980b70cecc184fa8a083a9c3ddb87f905e), which is cited as the way to run parakeet and compute timestamps for [nvidia/parakeet-tdt-0.6b-v3 on HF](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3#transcribing-with-timestamps). Requires meta-pytorch/tokenizers#163 for the `id_to_piece` method on `Tokenizers`. ### Test plan Outputs the exact same transcription/timestamps as [NVIDIA-NeMo/NeMo](https://github.com/NVIDIA-NeMo/NeMo/tree/bf583c980b70cecc184fa8a083a9c3ddb87f905e) for audio files [basketball.wav](https://github.com/user-attachments/files/24569675/basketball.wav) and [audio.wav](https://github.com/user-attachments/files/24569979/audio.wav) #### NeMo (basketball.wav) ``` import nemo.collections.asr as nemo_asr asr_model = nemo_asr.models.ASRModel.from_pretrained(model_name="nvidia/parakeet-tdt-0.6b-v3") output = asr_model.transcribe(['basketball.wav'], timestamps=True) # by default, timestamps are enabled for char, word and segment level word_timestamps = output[0].timestamp['word'] # word level timestamps for first sample segment_timestamps = output[0].timestamp['segment'] # segment level timestamps char_timestamps = output[0].timestamp['char'] # char level timestamps for stamp in segment_timestamps: print(f"{stamp['start']}s - {stamp['end']}s : {stamp['segment']}") for stamp in word_timestamps: print(f"{stamp['start']}s - {stamp['end']}s : {stamp['word']}") for stamp in char_timestamps: print(f"{stamp['start']}s - {stamp['end']}s : {stamp['char']}") ``` <details> <summary>Click to see output</summary> ``` Transcribing: 1it [00:01, 1.82s/it] 0.56s - 2.88s : All right, ballerina for the Nuggets and Mavs tonight. 2.96s - 3.6s : Late fourth quarter. 3.68s - 4.88s : Joker to MPJ. 4.96s - 5.68s : Back to you. 5.92s - 7.92s : Joker spins, fakes, scores. 8.08s - 10.24s : Was tied at 118 with two minutes to go. 10.32s - 11.52s : Then the Mavs by two. 11.6s - 12.48s : Joker drives. 12.56s - 14.64s : He'll miss but clean up his own mess. 14.8s - 16.080000000000002s : Tied at 120. 16.240000000000002s - 17.44s : Then time running out. 17.68s - 19.44s : Jamal to MPJ. 20.0s - 22.080000000000002s : He drives, stops, and pops. 22.16s - 23.04s : He had 17. 23.12s - 25.44s : The Nuggets had the lead with 6.5 to go. 25.6s - 26.8s : Mavs had a last chance. 26.88s - 31.28s : Kyrie Irving scored 43 points, but didn't hit the game winner. 31.44s - 32.24s : Joker got it. 32.32s - 34.56s : The Nuggets win at 122-120. 34.800000000000004s - 36.480000000000004s : Incredible stat line for Jokic. 36.72s - 37.36s : One of a kind. 37.6s - 40.88s : 37 points, 18 rebounds, 15 assists. 41.04s - 42.24s : He is simply the best. 42.4s - 45.92s : And the Nuggets have won four in a row and six of their last seven. 0.56s - 0.72s : All 0.72s - 0.8s : right, 0.96s - 1.76s : ballerina 1.76s - 1.84s : for 1.84s - 1.92s : the 1.92s - 2.24s : Nuggets 2.24s - 2.32s : and 2.32s - 2.64s : Mavs 2.64s - 2.88s : tonight. 2.96s - 3.2s : Late 3.2s - 3.44s : fourth 3.44s - 3.6s : quarter. 3.68s - 4.16s : Joker 4.16s - 4.32s : to 4.32s - 4.88s : MPJ. 4.96s - 5.2s : Back 5.2s - 5.44s : to 5.44s - 5.68s : you. 5.92s - 6.48s : Joker 6.48s - 6.88s : spins, 6.96s - 7.36s : fakes, 7.5200000000000005s - 7.92s : scores. 8.08s - 8.32s : Was 8.32s - 8.64s : tied 8.64s - 8.72s : at 8.72s - 9.36s : 118 9.36s - 9.52s : with 9.52s - 9.68s : two 9.68s - 9.92s : minutes 9.92s - 10.0s : to 10.0s - 10.24s : go. 10.32s - 10.56s : Then 10.56s - 10.64s : the 10.64s - 11.120000000000001s : Mavs 11.120000000000001s - 11.200000000000001s : by 11.28s - 11.52s : two. 11.6s - 12.16s : Joker 12.16s - 12.48s : drives. 12.56s - 12.96s : He'll 12.96s - 13.200000000000001s : miss 13.280000000000001s - 13.44s : but 13.44s - 13.68s : clean 13.84s - 13.92s : up 13.92s - 14.08s : his 14.08s - 14.32s : own 14.32s - 14.64s : mess. 14.8s - 15.200000000000001s : Tied 15.200000000000001s - 15.36s : at 15.36s - 16.080000000000002s : 120. 16.240000000000002s - 16.48s : Then 16.48s - 16.72s : time 16.88s - 17.28s : running 17.28s - 17.44s : out. 17.68s - 18.400000000000002s : Jamal 18.400000000000002s - 18.72s : to 18.72s - 19.44s : MPJ. 20.0s - 20.240000000000002s : He 20.240000000000002s - 20.72s : drives, 20.96s - 21.28s : stops, 21.36s - 21.6s : and 21.6s - 22.080000000000002s : pops. 22.16s - 22.32s : He 22.400000000000002s - 22.64s : had 22.64s - 23.04s : 17. 23.12s - 23.28s : The 23.28s - 23.68s : Nuggets 23.68s - 23.84s : had 23.84s - 23.92s : the 23.92s - 24.240000000000002s : lead 24.240000000000002s - 24.400000000000002s : with 24.400000000000002s - 25.12s : 6.5 25.12s - 25.28s : to 25.28s - 25.44s : go. 25.6s - 26.16s : Mavs 26.16s - 26.32s : had 26.32s - 26.400000000000002s : a 26.400000000000002s - 26.560000000000002s : last 26.560000000000002s - 26.8s : chance. 26.88s - 27.36s : Kyrie 27.52s - 28.0s : Irving 28.16s - 28.560000000000002s : scored 28.560000000000002s - 29.2s : 43 29.2s - 29.6s : points, 29.76s - 30.080000000000002s : but 30.080000000000002s - 30.48s : didn't 30.48s - 30.64s : hit 30.64s - 30.72s : the 30.72s - 30.96s : game 30.96s - 31.28s : winner. 31.44s - 32.0s : Joker 32.0s - 32.160000000000004s : got 32.160000000000004s - 32.24s : it. 32.32s - 32.480000000000004s : The 32.480000000000004s - 32.88s : Nuggets 32.88s - 33.12s : win 33.12s - 33.28s : at 33.28s - 34.56s : 122-120. 34.800000000000004s - 35.36s : Incredible 35.36s - 35.6s : stat 35.6s - 35.84s : line 35.84s - 36.0s : for 36.0s - 36.480000000000004s : Jokic. 36.72s - 36.88s : One 36.88s - 37.04s : of 37.04s - 37.12s : a 37.12s - 37.36s : kind. 37.6s - 38.160000000000004s : 37 38.160000000000004s - 38.56s : points, 38.72s - 39.2s : 18 39.2s - 39.68s : rebounds, 39.84s - 40.32s : 15 40.32s - 40.88s : assists. 41.04s - 41.2s : He 41.2s - 41.44s : is 41.44s - 41.76s : simply 41.76s - 42.0s : the 42.0s - 42.24s : best. 42.4s - 42.56s : And 42.56s - 42.64s : the 42.64s - 43.12s : Nuggets 43.12s - 43.28s : have 43.28s - 43.52s : won 43.52s - 43.76s : four 43.76s - 43.92s : in 43.92s - 44.0s : a 44.0s - 44.24s : row 44.24s - 44.56s : and 44.56s - 44.96s : six 44.96s - 45.2s : of 45.2s - 45.36s : their 45.36s - 45.6s : last 45.6s - 45.92s : seven. 0.56s - 0.72s : ['All'] 0.72s - 0.8s : ['right'] 0.8s - 0.8s : [','] 0.96s - 1.12s : ['b'] 1.12s - 1.28s : ['all'] 1.28s - 1.36s : ['er'] 1.6s - 1.76s : ['ina'] 1.76s - 1.84s : ['for'] 1.84s - 1.92s : ['the'] 1.92s - 2.0s : ['N'] 2.0s - 2.08s : ['ug'] 2.08s - 2.16s : ['g'] 2.16s - 2.24s : ['ets'] 2.24s - 2.32s : ['and'] 2.32s - 2.4s : ['M'] 2.4s - 2.48s : ['av'] 2.48s - 2.64s : ['s'] 2.64s - 2.72s : ['ton'] 2.72s - 2.88s : ['ight'] 2.88s - 2.88s : ['.'] 2.96s - 3.04s : ['L'] 3.04s - 3.2s : ['ate'] 3.2s - 3.2800000000000002s : ['fo'] 3.2800000000000002s - 3.36s : ['urt'] 3.36s - 3.44s : ['h'] 3.44s - 3.52s : ['quar'] 3.52s - 3.6s : ['ter'] 3.6s - 3.6s : ['.'] 3.68s - 3.84s : ['J'] 3.84s - 4.0s : ['ok'] 4.0s - 4.16s : ['er'] 4.16s - 4.32s : ['to'] 4.32s - 4.48s : ['M'] 4.48s - 4.64s : ['P'] 4.64s - 4.88s : ['J'] 4.88s - 4.88s : ['.'] 4.96s - 5.04s : ['B'] 5.04s - 5.2s : ['ack'] 5.2s - 5.44s : ['to'] 5.44s - 5.68s : ['you'] 5.68s - 5.68s : ['.'] 5.92s - 6.16s : ['J'] 6.16s - 6.32s : ['ok'] 6.32s - 6.48s : ['er'] 6.48s - 6.72s : ['sp'] 6.72s - 6.88s : ['ins'] 6.88s - 6.88s : [','] 6.96s - 7.12s : ['fak'] 7.28s - 7.36s : ['es'] 7.36s - 7.36s : [','] 7.5200000000000005s - 7.76s : ['sc'] 7.76s - 7.92s : ['ores'] 7.92s - 7.92s : ['.'] 8.08s - 8.32s : ['Was'] 8.32s - 8.64s : ['tied'] 8.64s - 8.72s : ['at'] 8.72s - 8.8s : [''] 8.8s - 8.88s : ['1'] 8.96s - 9.040000000000001s : ['1'] 9.120000000000001s - 9.36s : ['8'] 9.36s - 9.52s : ['with'] 9.52s - 9.68s : ['two'] 9.68s - 9.76s : ['minut'] 9.76s - 9.92s : ['es'] 9.92s - 10.0s : ['to'] 10.0s - 10.24s : ['go'] 10.24s - 10.24s : ['.'] 10.32s - 10.4s : ['T'] 10.4s - 10.56s : ['hen'] 10.56s - 10.64s : ['the'] 10.64s - 10.8s : ['M'] 10.8s - 10.96s : ['av'] 10.96s - 11.120000000000001s : ['s'] 11.120000000000001s - 11.200000000000001s : ['by'] 11.28s - 11.52s : ['two'] 11.52s - 11.52s : ['.'] 11.6s - 11.76s : ['J'] 11.76s - 12.0s : ['ok'] 12.0s - 12.16s : ['er'] 12.16s - 12.24s : ['d'] 12.24s - 12.32s : ['ri'] 12.32s - 12.48s : ['ves'] 12.48s - 12.48s : ['.'] 12.56s - 12.72s : ['He'] 12.72s - 12.72s : ["'"] 12.8s - 12.96s : ['ll'] 12.96s - 13.200000000000001s : ['miss'] 13.280000000000001s - 13.44s : ['but'] 13.44s - 13.52s : ['c'] 13.52s - 13.6s : ['le'] 13.6s - 13.68s : ['an'] 13.84s - 13.92s : ['up'] 13.92s - 14.08s : ['his'] 14.08s - 14.16s : ['o'] 14.16s - 14.32s : ['wn'] 14.32s - 14.48s : ['m'] 14.48s - 14.64s : ['ess'] 14.64s - 14.64s : ['.'] 14.8s - 14.96s : ['T'] 14.96s - 15.200000000000001s : ['ied'] 15.200000000000001s - 15.36s : ['at'] 15.36s - 15.44s : [''] 15.44s - 15.6s : ['1'] 15.68s - 15.92s : ['2'] 15.92s - 16.080000000000002s : ['0'] 16.080000000000002s - 16.080000000000002s : ['.'] 16.240000000000002s - 16.32s : ['T'] 16.32s - 16.48s : ['hen'] 16.48s - 16.72s : ['time'] 16.88s - 17.04s : ['run'] 17.04s - 17.28s : ['ning'] 17.28s - 17.44s : ['out'] 17.44s - 17.44s : ['.'] 17.68s - 17.84s : ['J'] 17.84s - 18.080000000000002s : ['am'] 18.080000000000002s - 18.400000000000002s : ['al'] 18.400000000000002s - 18.72s : ['to'] 18.72s - 18.88s : ['M'] 18.88s - 19.04s : ['P'] 19.2s - 19.44s : ['J'] 19.44s - 19.44s : ['.'] 20.0s - 20.240000000000002s : ['He'] 20.240000000000002s - 20.400000000000002s : ['d'] 20.400000000000002s - 20.64s : ['ri'] 20.64s - 20.72s : ['ves'] 20.72s - 20.72s : [','] 20.96s - 21.12s : ['stop'] 21.12s - 21.28s : ['s'] 21.28s - 21.28s : [','] 21.36s - 21.6s : ['and'] 21.6s - 21.84s : ['po'] 21.84s - 22.080000000000002s : ['ps'] 22.080000000000002s - 22.080000000000002s : ['.'] 22.16s - 22.32s : ['He'] 22.400000000000002s - 22.64s : ['had'] 22.64s - 22.72s : [''] 22.72s - 22.88s : ['1'] 22.88s - 23.04s : ['7'] 23.04s - 23.04s : ['.'] 23.12s - 23.28s : ['The'] 23.28s - 23.36s : ['N'] 23.36s - 23.44s : ['ug'] 23.44s - 23.6s : ['g'] 23.6s - 23.68s : ['ets'] 23.68s - 23.84s : ['had'] 23.84s - 23.92s : ['the'] 23.92s - 24.080000000000002s : ['le'] 24.080000000000002s - 24.240000000000002s : ['ad'] 24.240000000000002s - 24.400000000000002s : ['with'] 24.400000000000002s - 24.48s : [''] 24.48s - 24.72s : ['6'] 24.72s - 24.72s : ['.'] 24.88s - 25.12s : ['5'] 25.12s - 25.28s : ['to'] 25.28s - 25.44s : ['go'] 25.44s - 25.44s : ['.'] 25.6s - 25.76s : ['M'] 25.76s - 26.0s : ['av'] 26.0s - 26.16s : ['s'] 26.16s - 26.32s : ['had'] 26.32s - 26.400000000000002s : ['a'] 26.400000000000002s - 26.560000000000002s : ['last'] 26.560000000000002s - 26.64s : ['ch'] 26.64s - 26.8s : ['ance'] 26.8s - 26.8s : ['.'] 26.88s - 27.04s : ['K'] 27.04s - 27.28s : ['y'] 27.28s - 27.36s : ['rie'] 27.52s - 27.76s : ['Ir'] 27.76s - 27.92s : ['v'] 27.92s - 28.0s : ['ing'] 28.16s - 28.32s : ['sc'] 28.32s - 28.400000000000002s : ['or'] 28.400000000000002s - 28.560000000000002s : ['ed'] 28.560000000000002s - 28.72s : [''] 28.72s - 28.88s : ['4'] 28.88s - 29.2s : ['3'] 29.2s - 29.28s : ['po'] 29.28s - 29.44s : ['in'] 29.44s - 29.6s : ['ts'] 29.6s - 29.6s : [','] 29.76s - 30.080000000000002s : ['but'] 30.080000000000002s - 30.240000000000002s : ['did'] 30.240000000000002s - 30.32s : ['n'] 30.32s - 30.32s : ["'"] 30.48s - 30.48s : ['t'] 30.48s - 30.64s : ['hit'] 30.64s - 30.72s : ['the'] 30.72s - 30.8s : ['g'] 30.8s - 30.96s : ['ame'] 30.96s - 31.04s : ['w'] 31.04s - 31.2s : ['inn'] 31.2s - 31.28s : ['er'] 31.28s - 31.28s : ['.'] 31.44s - 31.68s : ['J'] 31.68s - 31.84s : ['ok'] 31.84s - 32.0s : ['er'] 32.0s - 32.160000000000004s : ['got'] 32.160000000000004s - 32.24s : ['it'] 32.24s - 32.24s : ['.'] 32.32s - 32.480000000000004s : ['The'] 32.480000000000004s - 32.56s : ['N'] 32.56s - 32.64s : ['ug'] 32.64s - 32.8s : ['g'] 32.8s - 32.88s : ['ets'] 32.88s - 32.96s : ['w'] 32.96s - 33.12s : ['in'] 33.12s - 33.28s : ['at'] 33.28s - 33.36s : [''] 33.36s - 33.52s : ['1'] 33.52s - 33.76s : ['2'] 33.76s - 33.92s : ['2'] 33.92s - 33.92s : ['-'] 34.0s - 34.08s : ['1'] 34.24s - 34.4s : ['2'] 34.4s - 34.56s : ['0'] 34.56s - 34.56s : ['.'] 34.800000000000004s - 34.88s : ['In'] 34.88s - 34.96s : ['c'] 34.96s - 35.04s : ['re'] 35.04s - 35.12s : ['di'] 35.12s - 35.36s : ['ble'] 35.36s - 35.6s : ['stat'] 35.6s - 35.68s : ['l'] 35.68s - 35.84s : ['ine'] 35.84s - 36.0s : ['for'] 36.0s - 36.160000000000004s : ['J'] 36.160000000000004s - 36.32s : ['ok'] 36.32s - 36.480000000000004s : ['ic'] 36.480000000000004s - 36.480000000000004s : ['.'] 36.72s - 36.800000000000004s : ['O'] 36.800000000000004s - 36.88s : ['ne'] 36.88s - 37.04s : ['of'] 37.04s - 37.12s : ['a'] 37.12s - 37.36s : ['kind'] 37.36s - 37.36s : ['.'] 37.6s - 37.68s : [''] 37.68s - 37.84s : ['3'] 37.92s - 38.160000000000004s : ['7'] 38.160000000000004s - 38.32s : ['po'] 38.32s - 38.4s : ['in'] 38.4s - 38.56s : ['ts'] 38.56s - 38.56s : [','] 38.72s - 38.800000000000004s : [''] 38.800000000000004s - 38.96s : ['1'] 38.96s - 39.2s : ['8'] 39.2s - 39.36s : ['re'] 39.36s - 39.44s : ['bo'] 39.44s - 39.6s : ['und'] 39.6s - 39.68s : ['s'] 39.68s - 39.68s : [','] 39.84s - 39.92s : [''] 39.92s - 40.08s : ['1'] 40.08s - 40.32s : ['5'] 40.32s - 40.480000000000004s : ['ass'] 40.480000000000004s - 40.64s : ['ist'] 40.64s - 40.88s : ['s'] 40.88s - 40.88s : ['.'] 41.04s - 41.2s : ['He'] 41.2s - 41.44s : ['is'] 41.44s - 41.6s : ['simpl'] 41.6s - 41.76s : ['y'] 41.76s - 42.0s : ['the'] 42.0s - 42.24s : ['best'] 42.24s - 42.24s : ['.'] 42.4s - 42.56s : ['And'] 42.56s - 42.64s : ['the'] 42.64s - 42.72s : ['N'] 42.72s - 42.88s : ['ug'] 42.88s - 43.04s : ['g'] 43.04s - 43.12s : ['ets'] 43.12s - 43.28s : ['have'] 43.28s - 43.36s : ['w'] 43.36s - 43.52s : ['on'] 43.52s - 43.6s : ['fo'] 43.6s - 43.76s : ['ur'] 43.76s - 43.92s : ['in'] 43.92s - 44.0s : ['a'] 44.0s - 44.08s : ['ro'] 44.08s - 44.24s : ['w'] 44.24s - 44.56s : ['and'] 44.56s - 44.72s : ['si'] 44.72s - 44.96s : ['x'] 44.96s - 45.2s : ['of'] 45.2s - 45.36s : ['their'] 45.36s - 45.6s : ['last'] 45.6s - 45.76s : ['se'] 45.76s - 45.92s : ['ven'] 45.92s - 45.92s : ['.'] ``` </details> #### `parakeet_runner` (basketball.wav) ``` ./cmake-out/examples/models/parakeet/parakeet_runner --model_path examples/models/parakeet/parakeet_tdt_exports/parakeet_tdt.pte --tokenizer_path /Users/matt/Workspace/executorch/examples/models/parakeet/parakeet_tdt_exports/tokenizer.model --audio_path basketball.wav --timestamps all ``` <details> <summary>Click to see output</summary> ``` -> % ./cmake-out/examples/models/parakeet/parakeet_runner --model_path examples/models/parakeet/parakeet_tdt_exports/parakeet_tdt.pte --tokenizer_path /Users/matt/Workspace/executorch/examples/models/parakeet/parakeet_tdt_exports/tokenizer.model --audio_path /Users/matt/Documents/parakeet_test_audio/basketball.wav --timestamps all I tokenizers:regex.cpp:27] Registering override fallback regex I 00:00:00.005332 executorch:main.cpp:717] Loading model from: examples/models/parakeet/parakeet_tdt_exports/parakeet_tdt.pte I 00:00:00.005729 executorch:main.cpp:733] Loading audio from: /Users/matt/Documents/parakeet_test_audio/basketball.wav I 00:00:00.008822 executorch:wav_loader.h:98] WAV header detected, getting raw audio data. I 00:00:00.008832 executorch:wav_loader.h:105] RIFF Header: RIFF I 00:00:00.008835 executorch:wav_loader.h:106] Chunk Size: 1476488 I 00:00:00.008836 executorch:wav_loader.h:113] WAVE Header: WAVE I 00:00:00.008840 executorch:wav_loader.h:120] Format Header: fmt I 00:00:00.008841 executorch:wav_loader.h:121] Format Chunk Size: 16 I 00:00:00.008843 executorch:wav_loader.h:122] Audio Format: 1 I 00:00:00.008845 executorch:wav_loader.h:123] Number of Channels: 1 I 00:00:00.008846 executorch:wav_loader.h:124] Sample Rate: 16000 I 00:00:00.008848 executorch:wav_loader.h:125] Byte Rate: 32000 I 00:00:00.008849 executorch:wav_loader.h:126] Block Align: 2 I 00:00:00.008851 executorch:wav_loader.h:127] Bits per Sample: 16 I 00:00:00.008852 executorch:wav_loader.h:132] Subchunk2Size: 1476418 I 00:00:00.009391 executorch:wav_loader.h:226] Loaded 738209 audio samples from WAV file: /Users/matt/Documents/parakeet_test_audio/basketball.wav I 00:00:00.009403 executorch:main.cpp:736] Loaded 738209 audio samples I 00:00:00.009432 executorch:main.cpp:747] Running preprocessor... I 00:00:00.033216 executorch:cpuinfo_utils.cpp:71] Reading file /sys/devices/soc0/image_version I 00:00:00.033243 executorch:cpuinfo_utils.cpp:87] Failed to open midr file /sys/devices/soc0/image_version I 00:00:00.065268 executorch:main.cpp:772] Mel spectrogram shape: [1, 128, 4614], mel_len: 4613 I 00:00:00.065280 executorch:main.cpp:775] Running encoder... I 00:01:19.425587 executorch:main.cpp:792] Encoder output shape: [1, 1024, 577], len=577 I 00:01:19.425620 executorch:main.cpp:833] Model metadata: vocab_size=8192, blank_id=8192, num_rnn_layers=2, pred_hidden=640, sample_rate=16000, window_stride=0.010000, encoder_subsampling_factor=8 I 00:01:19.425627 executorch:main.cpp:835] Running TDT greedy decode... I 00:01:32.149805 executorch:main.cpp:845] Decoded 289 tokens I 00:01:32.149823 executorch:main.cpp:848] Loading tokenizer from: /Users/matt/Workspace/executorch/examples/models/parakeet/parakeet_tdt_exports/tokenizer.model E tokenizers:hf_tokenizer.cpp:82] Error parsing json file: [json.exception.parse_error.101] parse error at line 2, column 1: syntax error while parsing value - invalid literal; last read: '<U+000A><U+000E>' E tokenizers:tiktoken.cpp:59] invalid tiktoken line: I 00:01:32.154983 executorch:llm_runner_helper.cpp:77] Loaded Sentencepiece tokenizer Transcribed text: All right, ballerina for the Nuggets and Mavs tonight. Late fourth quarter. Joker to MPJ. Back to you. Joker spins, fakes, scores. Was tied at 118 with two minutes to go. Then the Mavs by two. Joker drives. He'll miss but clean up his own mess. Tied at 120. Then time running out. Jamal to MPJ. He drives, stops, and pops. He had 17. The Nuggets had the lead with 6.5 to go. Mavs had a last chance. Kyrie Irving scored 43 points, but didn't hit the game winner. Joker got it. The Nuggets win at 122-120. Incredible stat line for Jokic. One of a kind. 37 points, 18 rebounds, 15 assists. He is simply the best. And the Nuggets have won four in a row and six of their last seven. I 00:01:32.155043 executorch:main.cpp:867] Computing timestamps... I 00:01:32.155828 executorch:main.cpp:873] Derived supported_punctuation size=11 Segment timestamps: 0.56s - 2.88s : All right, ballerina for the Nuggets and Mavs tonight. 2.96s - 3.6s : Late fourth quarter. 3.68s - 4.88s : Joker to MPJ. 4.96s - 5.68s : Back to you. 5.92s - 7.92s : Joker spins, fakes, scores. 8.08s - 10.24s : Was tied at 118 with two minutes to go. 10.32s - 11.52s : Then the Mavs by two. 11.6s - 12.48s : Joker drives. 12.56s - 14.64s : He'll miss but clean up his own mess. 14.8s - 16.08s : Tied at 120. 16.24s - 17.44s : Then time running out. 17.68s - 19.44s : Jamal to MPJ. 20s - 22.08s : He drives, stops, and pops. 22.16s - 23.04s : He had 17. 23.12s - 25.44s : The Nuggets had the lead with 6.5 to go. 25.6s - 26.8s : Mavs had a last chance. 26.88s - 31.28s : Kyrie Irving scored 43 points, but didn't hit the game winner. 31.44s - 32.24s : Joker got it. 32.32s - 34.56s : The Nuggets win at 122-120. 34.8s - 36.48s : Incredible stat line for Jokic. 36.72s - 37.36s : One of a kind. 37.6s - 40.88s : 37 points, 18 rebounds, 15 assists. 41.04s - 42.24s : He is simply the best. 42.4s - 45.92s : And the Nuggets have won four in a row and six of their last seven. Word timestamps: 0.56s - 0.72s : All 0.72s - 0.8s : right, 0.96s - 1.76s : ballerina 1.76s - 1.84s : for 1.84s - 1.92s : the 1.92s - 2.24s : Nuggets 2.24s - 2.32s : and 2.32s - 2.64s : Mavs 2.64s - 2.88s : tonight. 2.96s - 3.2s : Late 3.2s - 3.44s : fourth 3.44s - 3.6s : quarter. 3.68s - 4.16s : Joker 4.16s - 4.32s : to 4.32s - 4.88s : MPJ. 4.96s - 5.2s : Back 5.2s - 5.44s : to 5.44s - 5.68s : you. 5.92s - 6.48s : Joker 6.48s - 6.88s : spins, 6.96s - 7.36s : fakes, 7.52s - 7.92s : scores. 8.08s - 8.32s : Was 8.32s - 8.64s : tied 8.64s - 8.72s : at 8.72s - 9.36s : 118 9.36s - 9.52s : with 9.52s - 9.68s : two 9.68s - 9.92s : minutes 9.92s - 10s : to 10s - 10.24s : go. 10.32s - 10.56s : Then 10.56s - 10.64s : the 10.64s - 11.12s : Mavs 11.12s - 11.2s : by 11.28s - 11.52s : two. 11.6s - 12.16s : Joker 12.16s - 12.48s : drives. 12.56s - 12.96s : He'll 12.96s - 13.2s : miss 13.28s - 13.44s : but 13.44s - 13.68s : clean 13.84s - 13.92s : up 13.92s - 14.08s : his 14.08s - 14.32s : own 14.32s - 14.64s : mess. 14.8s - 15.2s : Tied 15.2s - 15.36s : at 15.36s - 16.08s : 120. 16.24s - 16.48s : Then 16.48s - 16.72s : time 16.88s - 17.28s : running 17.28s - 17.44s : out. 17.68s - 18.4s : Jamal 18.4s - 18.72s : to 18.72s - 19.44s : MPJ. 20s - 20.24s : He 20.24s - 20.72s : drives, 20.96s - 21.28s : stops, 21.36s - 21.6s : and 21.6s - 22.08s : pops. 22.16s - 22.32s : He 22.4s - 22.64s : had 22.64s - 23.04s : 17. 23.12s - 23.28s : The 23.28s - 23.68s : Nuggets 23.68s - 23.84s : had 23.84s - 23.92s : the 23.92s - 24.24s : lead 24.24s - 24.4s : with 24.4s - 25.12s : 6.5 25.12s - 25.28s : to 25.28s - 25.44s : go. 25.6s - 26.16s : Mavs 26.16s - 26.32s : had 26.32s - 26.4s : a 26.4s - 26.56s : last 26.56s - 26.8s : chance. 26.88s - 27.36s : Kyrie 27.52s - 28s : Irving 28.16s - 28.56s : scored 28.56s - 29.2s : 43 29.2s - 29.6s : points, 29.76s - 30.08s : but 30.08s - 30.48s : didn't 30.48s - 30.64s : hit 30.64s - 30.72s : the 30.72s - 30.96s : game 30.96s - 31.28s : winner. 31.44s - 32s : Joker 32s - 32.16s : got 32.16s - 32.24s : it. 32.32s - 32.48s : The 32.48s - 32.88s : Nuggets 32.88s - 33.12s : win 33.12s - 33.28s : at 33.28s - 34.56s : 122-120. 34.8s - 35.36s : Incredible 35.36s - 35.6s : stat 35.6s - 35.84s : line 35.84s - 36s : for 36s - 36.48s : Jokic. 36.72s - 36.88s : One 36.88s - 37.04s : of 37.04s - 37.12s : a 37.12s - 37.36s : kind. 37.6s - 38.16s : 37 38.16s - 38.56s : points, 38.72s - 39.2s : 18 39.2s - 39.68s : rebounds, 39.84s - 40.32s : 15 40.32s - 40.88s : assists. 41.04s - 41.2s : He 41.2s - 41.44s : is 41.44s - 41.76s : simply 41.76s - 42s : the 42s - 42.24s : best. 42.4s - 42.56s : And 42.56s - 42.64s : the 42.64s - 43.12s : Nuggets 43.12s - 43.28s : have 43.28s - 43.52s : won 43.52s - 43.76s : four 43.76s - 43.92s : in 43.92s - 44s : a 44s - 44.24s : row 44.24s - 44.56s : and 44.56s - 44.96s : six 44.96s - 45.2s : of 45.2s - 45.36s : their 45.36s - 45.6s : last 45.6s - 45.92s : seven. Token timestamps: 0.56s - 0.72s : All 0.72s - 0.8s : right 0.8s - 0.8s : , 0.96s - 1.12s : b 1.12s - 1.28s : all 1.28s - 1.36s : er 1.6s - 1.76s : ina 1.76s - 1.84s : for 1.84s - 1.92s : the 1.92s - 2s : N 2s - 2.08s : ug 2.08s - 2.16s : g 2.16s - 2.24s : ets 2.24s - 2.32s : and 2.32s - 2.4s : M 2.4s - 2.48s : av 2.48s - 2.64s : s 2.64s - 2.72s : ton 2.72s - 2.88s : ight 2.88s - 2.88s : . 2.96s - 3.04s : L 3.04s - 3.2s : ate 3.2s - 3.28s : fo 3.28s - 3.36s : urt 3.36s - 3.44s : h 3.44s - 3.52s : quar 3.52s - 3.6s : ter 3.6s - 3.6s : . 3.68s - 3.84s : J 3.84s - 4s : ok 4s - 4.16s : er 4.16s - 4.32s : to 4.32s - 4.48s : M 4.48s - 4.64s : P 4.64s - 4.88s : J 4.88s - 4.88s : . 4.96s - 5.04s : B 5.04s - 5.2s : ack 5.2s - 5.44s : to 5.44s - 5.68s : you 5.68s - 5.68s : . 5.92s - 6.16s : J 6.16s - 6.32s : ok 6.32s - 6.48s : er 6.48s - 6.72s : sp 6.72s - 6.88s : ins 6.88s - 6.88s : , 6.96s - 7.12s : fak 7.28s - 7.36s : es 7.36s - 7.36s : , 7.52s - 7.76s : sc 7.76s - 7.92s : ores 7.92s - 7.92s : . 8.08s - 8.32s : Was 8.32s - 8.64s : tied 8.64s - 8.72s : at 8.72s - 8.8s : 8.8s - 8.88s : 1 8.96s - 9.04s : 1 9.12s - 9.36s : 8 9.36s - 9.52s : with 9.52s - 9.68s : two 9.68s - 9.76s : minut 9.76s - 9.92s : es 9.92s - 10s : to 10s - 10.24s : go 10.24s - 10.24s : . 10.32s - 10.4s : T 10.4s - 10.56s : hen 10.56s - 10.64s : the 10.64s - 10.8s : M 10.8s - 10.96s : av 10.96s - 11.12s : s 11.12s - 11.2s : by 11.28s - 11.52s : two 11.52s - 11.52s : . 11.6s - 11.76s : J 11.76s - 12s : ok 12s - 12.16s : er 12.16s - 12.24s : d 12.24s - 12.32s : ri 12.32s - 12.48s : ves 12.48s - 12.48s : . 12.56s - 12.72s : He 12.72s - 12.72s : ' 12.8s - 12.96s : ll 12.96s - 13.2s : miss 13.28s - 13.44s : but 13.44s - 13.52s : c 13.52s - 13.6s : le 13.6s - 13.68s : an 13.84s - 13.92s : up 13.92s - 14.08s : his 14.08s - 14.16s : o 14.16s - 14.32s : wn 14.32s - 14.48s : m 14.48s - 14.64s : ess 14.64s - 14.64s : . 14.8s - 14.96s : T 14.96s - 15.2s : ied 15.2s - 15.36s : at 15.36s - 15.44s : 15.44s - 15.6s : 1 15.68s - 15.92s : 2 15.92s - 16.08s : 0 16.08s - 16.08s : . 16.24s - 16.32s : T 16.32s - 16.48s : hen 16.48s - 16.72s : time 16.88s - 17.04s : run 17.04s - 17.28s : ning 17.28s - 17.44s : out 17.44s - 17.44s : . 17.68s - 17.84s : J 17.84s - 18.08s : am 18.08s - 18.4s : al 18.4s - 18.72s : to 18.72s - 18.88s : M 18.88s - 19.04s : P 19.2s - 19.44s : J 19.44s - 19.44s : . 20s - 20.24s : He 20.24s - 20.4s : d 20.4s - 20.64s : ri 20.64s - 20.72s : ves 20.72s - 20.72s : , 20.96s - 21.12s : stop 21.12s - 21.28s : s 21.28s - 21.28s : , 21.36s - 21.6s : and 21.6s - 21.84s : po 21.84s - 22.08s : ps 22.08s - 22.08s : . 22.16s - 22.32s : He 22.4s - 22.64s : had 22.64s - 22.72s : 22.72s - 22.88s : 1 22.88s - 23.04s : 7 23.04s - 23.04s : . 23.12s - 23.28s : The 23.28s - 23.36s : N 23.36s - 23.44s : ug 23.44s - 23.6s : g 23.6s - 23.68s : ets 23.68s - 23.84s : had 23.84s - 23.92s : the 23.92s - 24.08s : le 24.08s - 24.24s : ad 24.24s - 24.4s : with 24.4s - 24.48s : 24.48s - 24.72s : 6 24.72s - 24.72s : . 24.88s - 25.12s : 5 25.12s - 25.28s : to 25.28s - 25.44s : go 25.44s - 25.44s : . 25.6s - 25.76s : M 25.76s - 26s : av 26s - 26.16s : s 26.16s - 26.32s : had 26.32s - 26.4s : a 26.4s - 26.56s : last 26.56s - 26.64s : ch 26.64s - 26.8s : ance 26.8s - 26.8s : . 26.88s - 27.04s : K 27.04s - 27.28s : y 27.28s - 27.36s : rie 27.52s - 27.76s : Ir 27.76s - 27.92s : v 27.92s - 28s : ing 28.16s - 28.32s : sc 28.32s - 28.4s : or 28.4s - 28.56s : ed 28.56s - 28.72s : 28.72s - 28.88s : 4 28.88s - 29.2s : 3 29.2s - 29.28s : po 29.28s - 29.44s : in 29.44s - 29.6s : ts 29.6s - 29.6s : , 29.76s - 30.08s : but 30.08s - 30.24s : did 30.24s - 30.32s : n 30.32s - 30.32s : ' 30.48s - 30.48s : t 30.48s - 30.64s : hit 30.64s - 30.72s : the 30.72s - 30.8s : g 30.8s - 30.96s : ame 30.96s - 31.04s : w 31.04s - 31.2s : inn 31.2s - 31.28s : er 31.28s - 31.28s : . 31.44s - 31.68s : J 31.68s - 31.84s : ok 31.84s - 32s : er 32s - 32.16s : got 32.16s - 32.24s : it 32.24s - 32.24s : . 32.32s - 32.48s : The 32.48s - 32.56s : N 32.56s - 32.64s : ug 32.64s - 32.8s : g 32.8s - 32.88s : ets 32.88s - 32.96s : w 32.96s - 33.12s : in 33.12s - 33.28s : at 33.28s - 33.36s : 33.36s - 33.52s : 1 33.52s - 33.76s : 2 33.76s - 33.92s : 2 33.92s - 33.92s : - 34s - 34.08s : 1 34.24s - 34.4s : 2 34.4s - 34.56s : 0 34.56s - 34.56s : . 34.8s - 34.88s : In 34.88s - 34.96s : c 34.96s - 35.04s : re 35.04s - 35.12s : di 35.12s - 35.36s : ble 35.36s - 35.6s : stat 35.6s - 35.68s : l 35.68s - 35.84s : ine 35.84s - 36s : for 36s - 36.16s : J 36.16s - 36.32s : ok 36.32s - 36.48s : ic 36.48s - 36.48s : . 36.72s - 36.8s : O 36.8s - 36.88s : ne 36.88s - 37.04s : of 37.04s - 37.12s : a 37.12s - 37.36s : kind 37.36s - 37.36s : . 37.6s - 37.68s : 37.68s - 37.84s : 3 37.92s - 38.16s : 7 38.16s - 38.32s : po 38.32s - 38.4s : in 38.4s - 38.56s : ts 38.56s - 38.56s : , 38.72s - 38.8s : 38.8s - 38.96s : 1 38.96s - 39.2s : 8 39.2s - 39.36s : re 39.36s - 39.44s : bo 39.44s - 39.6s : und 39.6s - 39.68s : s 39.68s - 39.68s : , 39.84s - 39.92s : 39.92s - 40.08s : 1 40.08s - 40.32s : 5 40.32s - 40.48s : ass 40.48s - 40.64s : ist 40.64s - 40.88s : s 40.88s - 40.88s : . 41.04s - 41.2s : He 41.2s - 41.44s : is 41.44s - 41.6s : simpl 41.6s - 41.76s : y 41.76s - 42s : the 42s - 42.24s : best 42.24s - 42.24s : . 42.4s - 42.56s : And 42.56s - 42.64s : the 42.64s - 42.72s : N 42.72s - 42.88s : ug 42.88s - 43.04s : g 43.04s - 43.12s : ets 43.12s - 43.28s : have 43.28s - 43.36s : w 43.36s - 43.52s : on 43.52s - 43.6s : fo 43.6s - 43.76s : ur 43.76s - 43.92s : in 43.92s - 44s : a 44s - 44.08s : ro 44.08s - 44.24s : w 44.24s - 44.56s : and 44.56s - 44.72s : si 44.72s - 44.96s : x 44.96s - 45.2s : of 45.2s - 45.36s : their 45.36s - 45.6s : last 45.6s - 45.76s : se 45.76s - 45.92s : ven 45.92s - 45.92s : . ``` </details>
1 parent 7ed453f commit 9510334

9 files changed

Lines changed: 620 additions & 33 deletions

File tree

‎examples/models/parakeet/CMakeLists.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ if(EXECUTORCH_BUILD_METAL)
8080
executorch_target_link_options_shared_lib(metal_backend)
8181
endif()
8282

83-
add_executable(parakeet_runner main.cpp)
83+
add_executable(parakeet_runner main.cpp timestamp_utils.cpp tokenizer_utils.cpp)
8484
if(NOT CMAKE_BUILD_TYPE STREQUAL "Debug")
8585
target_link_options_gc_sections(parakeet_runner)
8686
if(NOT APPLE AND NOT MSVC)

‎examples/models/parakeet/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,3 +71,4 @@ From the executorch root directory:
7171
| `--audio_path` | Path to input audio file (.wav) |
7272
| `--tokenizer_path` | Path to tokenizer file (default: `tokenizer.json`) |
7373
| `--data_path` | Path to data file (.ptd) for delegate data (optional, required for CUDA) |
74+
| `--timestamps` | Timestamp output mode: `none\|token\|word\|segment\|all` (default: `segment`) |

‎examples/models/parakeet/export_parakeet_tdt.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -351,13 +351,18 @@ def export_all(model):
351351
)
352352

353353
sample_rate = model.preprocessor._cfg.sample_rate
354+
window_stride = float(model.preprocessor._cfg.window_stride)
355+
encoder_subsampling_factor = int(getattr(model.encoder, "subsampling_factor", 8))
356+
354357
metadata = {
355358
"num_rnn_layers": num_layers,
356359
"pred_hidden": pred_hidden,
357360
"joint_hidden": joint_hidden,
358361
"vocab_size": model.tokenizer.vocab_size,
359362
"blank_id": model.tokenizer.vocab_size,
360363
"sample_rate": sample_rate,
364+
"window_stride": window_stride,
365+
"encoder_subsampling_factor": encoder_subsampling_factor,
361366
}
362367

363368
return programs, metadata

‎examples/models/parakeet/main.cpp‎

Lines changed: 151 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -6,18 +6,28 @@
66
* LICENSE file in the root directory of this source tree.
77
*/
88

9+
#include <algorithm>
910
#include <cmath>
11+
#include <cstdint>
1012
#include <cstring>
13+
#include <exception>
1114
#include <fstream>
1215
#include <iostream>
1316
#include <memory>
17+
#include <optional>
1418
#include <string>
19+
#include <unordered_set>
1520
#include <vector>
1621

1722
#include <gflags/gflags.h>
1823

24+
#include "timestamp_utils.h"
25+
#include "tokenizer_utils.h"
26+
#include "types.h"
27+
1928
#include <executorch/extension/llm/runner/llm_runner_helper.h>
2029
#include <executorch/extension/llm/runner/wav_loader.h>
30+
#include <executorch/extension/llm/tokenizers/third-party/llama.cpp-unicode/include/unicode.h>
2131
#include <executorch/extension/module/module.h>
2232
#include <executorch/extension/tensor/tensor_ptr_maker.h>
2333
#include <executorch/runtime/core/evalue.h>
@@ -27,24 +37,75 @@ DEFINE_string(model_path, "parakeet.pte", "Path to Parakeet model (.pte).");
2737
DEFINE_string(audio_path, "", "Path to input audio file (.wav).");
2838
DEFINE_string(
2939
tokenizer_path,
30-
"tokenizer.json",
40+
"tokenizer.model",
3141
"Path to SentencePiece tokenizer model file.");
3242
DEFINE_string(
3343
data_path,
3444
"",
3545
"Path to data file (.ptd) for delegate data (optional, required for CUDA).");
46+
DEFINE_string(
47+
timestamps,
48+
"segment",
49+
"Timestamp output mode: none|token|word|segment|all");
3650

3751
using ::executorch::extension::from_blob;
3852
using ::executorch::extension::Module;
3953
using ::executorch::runtime::Error;
4054
using ::executorch::runtime::EValue;
4155

42-
namespace {
56+
using ::parakeet::TextWithOffsets;
57+
using ::parakeet::Token;
58+
using ::parakeet::TokenId;
59+
using ::parakeet::TokenWithTextInfo;
4360

61+
namespace {
4462
// TDT duration values
4563
const std::vector<int> DURATIONS = {0, 1, 2, 3, 4};
4664

47-
std::vector<int64_t> greedy_decode_executorch(
65+
struct TimestampOutputMode {
66+
bool token = false;
67+
bool word = false;
68+
bool segment = false;
69+
70+
bool enabled() const {
71+
return token || word || segment;
72+
}
73+
};
74+
75+
std::string to_lower_ascii(std::string s) {
76+
for (char& ch : s) {
77+
ch = static_cast<char>(std::tolower(static_cast<unsigned char>(ch)));
78+
}
79+
return s;
80+
}
81+
82+
TimestampOutputMode parse_timestamp_output_mode(const std::string& raw_arg) {
83+
if (raw_arg.empty()) {
84+
throw std::invalid_argument(
85+
"Invalid --timestamps value (empty). Expected: token, word, segment, all.");
86+
}
87+
const std::string mode = to_lower_ascii(raw_arg);
88+
if (mode == "none") {
89+
return {false, false, false};
90+
}
91+
if (mode == "token") {
92+
return {true, false, false};
93+
}
94+
if (mode == "word") {
95+
return {false, true, false};
96+
}
97+
if (mode == "segment") {
98+
return {false, false, true};
99+
}
100+
if (mode == "all") {
101+
return {true, true, true};
102+
}
103+
throw std::invalid_argument(
104+
"Invalid --timestamps value '" + raw_arg +
105+
"'. Expected: token, word, segment, all.");
106+
}
107+
108+
std::vector<Token> greedy_decode_executorch(
48109
Module& model,
49110
const ::executorch::aten::Tensor& encoder_output,
50111
int64_t encoder_len,
@@ -53,7 +114,7 @@ std::vector<int64_t> greedy_decode_executorch(
53114
int64_t num_rnn_layers = 2,
54115
int64_t pred_hidden = 640,
55116
int64_t max_symbols_per_step = 10) {
56-
std::vector<int64_t> hypothesis;
117+
std::vector<Token> hypothesis;
57118
int64_t num_token_classes = vocab_size + 1;
58119

59120
// Transpose encoder output from [1, enc_dim, time] to [1, time, enc_dim]
@@ -205,10 +266,10 @@ std::vector<int64_t> greedy_decode_executorch(
205266
int64_t dur = DURATIONS[dur_idx];
206267

207268
if (k == blank_id) {
208-
t += std::max(dur, (int64_t)1);
269+
t += std::max(dur, static_cast<int64_t>(1));
209270
symbols_on_frame = 0;
210271
} else {
211-
hypothesis.push_back(k);
272+
hypothesis.push_back({static_cast<TokenId>(k), t, dur});
212273

213274
// Update decoder state
214275
std::vector<int64_t> token_data = {k};
@@ -268,29 +329,19 @@ std::vector<int64_t> greedy_decode_executorch(
268329
return hypothesis;
269330
}
270331

271-
std::string tokens_to_text(
272-
const std::vector<int64_t>& tokens,
273-
tokenizers::Tokenizer* tokenizer) {
274-
// Decode tokens to text one by one
275-
std::string result;
276-
uint64_t prev_token = 0;
277-
for (size_t i = 0; i < tokens.size(); i++) {
278-
uint64_t token = static_cast<uint64_t>(tokens[i]);
279-
auto decode_result = tokenizer->decode(prev_token, token);
280-
if (decode_result.ok()) {
281-
result += decode_result.get();
282-
}
283-
prev_token = token;
284-
}
285-
286-
return result;
287-
}
288-
289332
} // namespace
290333

291334
int main(int argc, char** argv) {
292335
gflags::ParseCommandLineFlags(&argc, &argv, true);
293336

337+
TimestampOutputMode timestamp_mode;
338+
try {
339+
timestamp_mode = parse_timestamp_output_mode(FLAGS_timestamps);
340+
} catch (const std::invalid_argument& e) {
341+
ET_LOG(Error, "%s", e.what());
342+
return 1;
343+
}
344+
294345
if (FLAGS_audio_path.empty()) {
295346
ET_LOG(Error, "audio_path flag must be provided.");
296347
return 1;
@@ -381,10 +432,14 @@ int main(int argc, char** argv) {
381432
auto vocab_size_result = model->execute("vocab_size", empty_inputs);
382433
auto blank_id_result = model->execute("blank_id", empty_inputs);
383434
auto sample_rate_result = model->execute("sample_rate", empty_inputs);
435+
auto window_stride_result = model->execute("window_stride", empty_inputs);
436+
auto encoder_subsampling_factor_result =
437+
model->execute("encoder_subsampling_factor", empty_inputs);
384438

385439
if (!num_rnn_layers_result.ok() || !pred_hidden_result.ok() ||
386440
!vocab_size_result.ok() || !blank_id_result.ok() ||
387-
!sample_rate_result.ok()) {
441+
!sample_rate_result.ok() || !window_stride_result.ok() ||
442+
!encoder_subsampling_factor_result.ok()) {
388443
ET_LOG(
389444
Error,
390445
"Failed to query model metadata. Make sure the model was exported with constant_methods.");
@@ -396,18 +451,23 @@ int main(int argc, char** argv) {
396451
int64_t num_rnn_layers = num_rnn_layers_result.get()[0].toInt();
397452
int64_t pred_hidden = pred_hidden_result.get()[0].toInt();
398453
int64_t sample_rate = sample_rate_result.get()[0].toInt();
454+
double window_stride = window_stride_result.get()[0].toDouble();
455+
int64_t encoder_subsampling_factor =
456+
encoder_subsampling_factor_result.get()[0].toInt();
399457

400458
ET_LOG(
401459
Info,
402-
"Model metadata: vocab_size=%lld, blank_id=%lld, num_rnn_layers=%lld, pred_hidden=%lld, sample_rate=%lld",
460+
"Model metadata: vocab_size=%lld, blank_id=%lld, num_rnn_layers=%lld, pred_hidden=%lld, sample_rate=%lld, window_stride=%.6f, encoder_subsampling_factor=%lld",
403461
static_cast<long long>(vocab_size),
404462
static_cast<long long>(blank_id),
405463
static_cast<long long>(num_rnn_layers),
406464
static_cast<long long>(pred_hidden),
407-
static_cast<long long>(sample_rate));
465+
static_cast<long long>(sample_rate),
466+
window_stride,
467+
encoder_subsampling_factor);
408468

409469
ET_LOG(Info, "Running TDT greedy decode...");
410-
auto tokens = greedy_decode_executorch(
470+
auto decoded_tokens = greedy_decode_executorch(
411471
*model,
412472
encoded,
413473
encoded_len,
@@ -416,7 +476,7 @@ int main(int argc, char** argv) {
416476
num_rnn_layers,
417477
pred_hidden);
418478

419-
ET_LOG(Info, "Decoded %zu tokens", tokens.size());
479+
ET_LOG(Info, "Decoded %zu tokens", decoded_tokens.size());
420480

421481
// Load tokenizer
422482
ET_LOG(Info, "Loading tokenizer from: %s", FLAGS_tokenizer_path.c_str());
@@ -431,9 +491,68 @@ int main(int argc, char** argv) {
431491
}
432492

433493
// Convert tokens to text
434-
std::string text = tokens_to_text(tokens, tokenizer.get());
435-
std::cout << "Transcription tokens: " << text << std::endl;
494+
std::string text = parakeet::tokenizer_utils::decode_token_sequence(
495+
decoded_tokens, *tokenizer);
496+
std::cout << "Transcribed text: " << text << std::endl;
497+
498+
if (!timestamp_mode.enabled()) {
499+
return 0;
500+
}
501+
502+
ET_LOG(Info, "Computing timestamps...");
503+
std::unordered_set<std::string> supported_punctuation =
504+
parakeet::tokenizer_utils::derive_supported_punctuation(*tokenizer);
505+
ET_LOG(
506+
Info,
507+
"Derived supported_punctuation size=%zu",
508+
supported_punctuation.size());
509+
510+
// for simplicity, compute all levels of timestamps regardless of mode
511+
std::vector<TokenWithTextInfo> tokens_with_text_info;
512+
try {
513+
tokens_with_text_info =
514+
parakeet::timestamp_utils::get_tokens_with_text_info(
515+
decoded_tokens, *tokenizer, supported_punctuation);
516+
} catch (const std::exception& e) {
517+
ET_LOG(Error, "Failed to get tokens with text info: %s", e.what());
518+
return 1;
519+
}
520+
const auto word_offsets = parakeet::timestamp_utils::get_words_offsets(
521+
tokens_with_text_info, *tokenizer, supported_punctuation);
522+
const auto segment_offsets =
523+
parakeet::timestamp_utils::get_segment_offsets(word_offsets);
524+
525+
const double frame_to_seconds =
526+
window_stride * static_cast<double>(encoder_subsampling_factor);
527+
528+
if (timestamp_mode.segment) {
529+
std::cout << "\nSegment timestamps:" << std::endl;
530+
for (const auto& segment : segment_offsets) {
531+
const double start = segment.start_offset * frame_to_seconds;
532+
const double end = segment.end_offset * frame_to_seconds;
533+
std::cout << start << "s - " << end << "s : " << segment.text
534+
<< std::endl;
535+
}
536+
}
537+
538+
if (timestamp_mode.word) {
539+
std::cout << "\nWord timestamps:" << std::endl;
540+
for (const auto& word : word_offsets) {
541+
const double start = word.start_offset * frame_to_seconds;
542+
const double end = word.end_offset * frame_to_seconds;
543+
std::cout << start << "s - " << end << "s : " << word.text << std::endl;
544+
}
545+
}
546+
547+
if (timestamp_mode.token) {
548+
std::cout << "\nToken timestamps:" << std::endl;
549+
for (const auto& token : tokens_with_text_info) {
550+
const double start = token.start_offset * frame_to_seconds;
551+
const double end = token.end_offset * frame_to_seconds;
552+
std::cout << start << "s - " << end << "s : " << token.decoded_text
553+
<< std::endl;
554+
}
555+
}
436556

437-
ET_LOG(Info, "Done!");
438557
return 0;
439558
}

0 commit comments

Comments
 (0)