Integrating models with Hugging Face (#957)

* Add dependencies for huggingface_hub

* Update alignment analysis calculations

* Add command for migrating models to hf format

* Bump kalpy versions

Author: mmcauliffe

README.md

@@ -12,6 +12,45 @@ Please see the documentation http://montreal-forced-aligner.readthedocs.io for i
 
 If you run into any issues, please check the [mailing list](https://groups.google.com/forum/#!forum/mfa-users) for fixes/workarounds or to post a [new issue](https://github.com/MontrealCorpusTools/Montreal-Forced-Aligner/issues).
 
+## Citation
+
+### MFA 3.0
+
+For citing the latest version of MFA and its pretrained models, please use:
+
+McAuliffe, Michael, Kaylynn Gunter, Michael Wagner, and Morgan Sonderegger (2026).
+Montreal Forced Aligner and the state of speech-to-text alignment in 2026. In
+*Proceedings of Interspeech 2026*.
+
+
+```
+@inproceedings{mcauliffe26_interspeech,
+ author={McAuliffe, Michael and Gunter, Kaylynn and Wagner, Michael and Sonderegger, Morgan},
+ title={{Montreal Forced Aligner and the state of speech-to-text alignment in 2026}},
+ year=2026,
+ booktitle={Proc. Interspeech 2026}
+}
+```
+
+### MFA 1.0
+
+For citing the original software and models (upgrading to the MFA 3.X is recommended!), please use:
+
+McAuliffe, Michael, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger (2017).
+Montreal Forced Aligner: trainable text-speech alignment using Kaldi. In
+*Proceedings of Interspeech 2017*.
+
+```
+@inproceedings{mcauliffe17_interspeech,
+ author={McAuliffe, Michael and Socolof, Michaela and Mihuc, Sarah and Wagner, Michael and Sonderegger, Morgan},
+ title={{Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi}},
+ year=2017,
+ booktitle={Proc. Interspeech 2017},
+ pages={498--502},
+ doi={10.21437/Interspeech.2017-1386}
+}
+```
+
 ## Installation
 
 You can install MFA either entirely through [conda](https://docs.conda.io/en/latest/) or a mix of conda for Kaldi and Pynini dependencies and Python packaging for MFA itself

ci/docker_environment.yaml

@@ -38,12 +38,14 @@ dependencies:
   - sqlite
   - rich
   - rich-click
-  - kalpy>=0.8
+  - kalpy>=0.10
   - spacy
   - sudachipy
   - sudachidict-core
   - spacy-pkuseg
   - sentencepiece
+  - huggingface_hub
+  - jinja2
   - pip
   - pip:
       - speechbrain==1.0.3

docs/source/changelog/changelog_3.4.rst

@@ -0,0 +1,18 @@
+
+.. _changelog_3.4:
+
+*************
+3.4 Changelog
+*************
+
+3.4.0
+-----
+
+- Compatibility with Kalpy 0.9.0
+- Introduced new command for :ref:`compare_alignments`
+- Introduced new MFA model format for distribution on HuggingFace
+- Introduced new commands that use the new MFA model format (i.e. :ref:`pretrained_alignment_hf`) which will replace the default commands (i.e. :ref:`pretrained_alignment`) in MFA 4.0.  The default commands will still be available using a legacy command and will be removed in MFA 5.0.
+- Deprecated :ref:`train_tokenizer_cli` and :ref:`tokenize_cli` for removal in MFA 4.0
+- Added calculation for :ref:`alignment_analysis_intensity_deviation` and :ref:`alignment_analysis_snr` to be included as diagnostic metrics for alignment
+- Updated calculation for phone duration deviation to use log-duration to better model the duration distribution
+- Added column for :ref:`alignment_max_running_short_intervals` as a measure of alignment quality

docs/source/changelog/index.md

@@ -7,52 +7,32 @@
    Please bear in mind that all plans below are tentative and subject to change.
 ```
 
-### Version 3.1
+### Version 5.0 (Planned for 2027)
 
-* Persistent server for sending audio/text files to
-  * May not be necessary/prioritized with {ref}`align_one` command
-* Update tokenization to use spacy tokenizers instead of custom specification
-  * Should be more robust than MFA's custom rules
-  * Some languages are more finely tokenized than others (i.e., Japanese and Korean tokens are largely morphemes, while the English one doesn't do morpheme analysis), but the ideal would be some morphologically-aware G2P of phonological words
-* Add option for training SpeechBrain ASR model on phone strings of MFA models
-  * Should allow for better single-pass alignment and faster with GPUs
-* Release Anchor compatible with the latest versions of MFA
+* Removal of legacy alignment commands
+  * Legacy models can still be used with previous versions of MFA
+* Support for neural network acoustic models
 
-### Future
+### Version 4.0 (Planned for end of 2026)
+
+* Deprecation of legacy model usage
+  * Commands for {ref}`align_hf`, {ref}`adapt_hf`, and {ref}`align_one_hf` will be renamed to {ref}`align`, {ref}`adapt`, and {ref}`align_one` and 3.4 {ref}`align`, {ref}`adapt`, and {ref}`align_one` will be renamed {ref}`align_legacy`, {ref}`adapt_legacy`, and {ref}`align_one_legacy`
+* Refactor and code clean up
+* Broad phone sets with modular implementation of phonological rules
+* Richer lexicon support
+
+### Version 3.4 (Released)
+
+* {ref}`changelog_3.4`
 
-* Retrain existing acoustic models with new phone groups and rules features
-* Begin work on expanding to new languages
-    * Japanese (in progress)
-    * Arabic
-    * Tamil
-* Localize documentation
-    * I'll initially do a pass at localizing the documentation to Japanese and see if I can crowd source other languages (and fixing my initial Japanese pass)
-* Update pitch feature calculation to use speaker-adjusted min and max f0 ranges
 
-* Moving away from Kaldi-based dependencies
-  * Kaldi is not being actively developed and I don't have much of a desire to depend on it long term
-  * Most actively developed ASR toolkits and libraries are based around neural networks
-    * I'm not the biggest fan of using these for alignment, as most of the research is geared towards improving end-to-end signal to orthographic text models that don't have intermediate representations of phones
-    * That said, if alignment were the task that was being optimized for rather than some "word error rate" style metric, then alignment performance could improve significantly
-      * One particular direction would be towards sample-based or waveform-based alignment rather than frame-based
-        * Frame-based methods are time-smeared, so providing an exact time for voicing onset or stop closure is murky
-        * Phoneticians use spectrograms for gross boundaries, but more accurate manual alignments are determined based on the waveform
-      * Perhaps combining a model that performs language-independent boundary insertion combined with per-language models to combine resulting segments might perform better ({ipa_inline}`[a]` + {ipa_inline}`[j]` becomes {ipa_inline}`[aj]` in English, but not in other languages like Japanese, Spanish, or Portuguese, etc)
-    * Additionally, neural networks might allow for better modeling of phone symbols, so embedding {ipa_inline}`[pʲ]` could result in a more compositional "voiceless bilabial stop plus palatalization"
-  * Other options for toolkits to support MFA are
-    * [SpeechBrain](https://speechbrain.github.io/)
-    * Custom PyTorch code
-    * Custom tensorflow code
-* Update dictionary model format to move away from the current plain-text lexicons to a more robust compressed format
-  * With extra meta data and capabilities in the form of phonological rules and phone groupings, it makes more sense to package those with the lexicon rather than the acoustic model
-  * Another option would be to package up the lexicon (and maybe G2P models) with the acoustic model into a complete MFA model
-  * As part of any update, I would expand the {ref}`MFA model CLI <pretrained_models>` with functionality for adding new pronunciations to internal lexicons
-    * Something like {code}`mfa model update /path/to/g2pped_file.txt`
 
 ```{toctree}
 :hidden:
 :maxdepth: 1
 
+news_3.4.rst
+changelog_3.4.rst
 changelog_3.3.rst
 changelog_3.2.rst
 changelog_3.1.rst

docs/source/changelog/news_3.4.rst

@@ -0,0 +1,11 @@
+
+.. _whats_new_3_4:
+
+What's new in 3.4
+=================
+
+Version 3.4 of the Montreal Forced Aligner introduces a new model format to allow for distribution via Hugging Face.  See :ref:`mfa_model_versions` for more details on the changes and how to use and distribute these models.
+
+Additionally, MFA-trained tokenizers are being deprecated and will be removed in MFA 4.0, as the language-specific tokenization in training has yielded much better and consistent results (see :ref:`language_tokenization` for more details).
+
+Finally, a new MFA paper reflecting the updates over the past 10 years along with benchmarks with recent and other widely-used aligners across three languages will be presented at Interspeech 2026!  See :ref:`citing_mfa` for more details on the updated citation.

docs/source/conf.py

@@ -22,8 +22,11 @@
 from datetime import date
 
 sys.path.insert(0, os.path.abspath("../../"))
-import montreal_forced_aligner  # noqa
-from montreal_forced_aligner.utils import get_mfa_version  # noqa
+try:
+    from montreal_forced_aligner.utils import get_mfa_version  # noqa
+    release = get_mfa_version()
+except ImportError:
+    release = "3.4.0"
 
 # -- General configuration ------------------------------------------------
 
@@ -76,6 +79,7 @@
 }
 
 xref_links = {
+    "hf": ("Hugging Face", "https://huggingface.co/"),
     "mfa_models": ("MFA Models", "https://mfa-models.readthedocs.io/"),
     "mfa_models_repo": (
         "MFA Models repository",
@@ -229,6 +233,7 @@
     "mp.Process": "multiprocessing.Process",
     "MetaDict": "dict[str, Any]",
 }
+autodoc_mock_imports = ['kalpy', 'kalpy.evaluation.naive_boundary_f1']
 
 napoleon_preprocess_types = False
 napoleon_attr_annotations = False
@@ -274,9 +279,8 @@
 # built documents.
 #
 # The short X.Y version.
-version = ".".join(get_mfa_version().split(".", maxsplit=2)[:2])
+version = ".".join(release.split(".", maxsplit=2)[:2])
 # The full version, including alpha/beta/rc tags.
-release = get_mfa_version()
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
@@ -397,6 +401,8 @@
     # "navigation_depth": 4,
     # "show_toc_level": 2,
     # "collapse_navigation": True,
+   "announcement": "Starting with Version 3.4, we are rolling out new model formats that integrate with Hugging Face! "
+                   "See <a href='changelog/news_3.4.html'>here</a> for the latest updates in MFA 3.4.",
 }
 html_context = {
     "github_user": "MontrealCorpusTools",

docs/source/first_steps/index.rst

@@ -82,10 +82,6 @@ There are several broad use cases that you might want to use MFA for.  Take a lo
    #. Use the trained G2P model in :ref:`first_steps_g2p_pretrained` to generate a pronunciation dictionary
    #. Use the generated pronunciation dictionary in :ref:`first_steps_align_train_acoustic_model` to generate aligned TextGrids
 
-#. **Use case 6:** You have a :ref:`speech corpus <corpus_structure>` and the language has a :xref:`pretrained acoustic model <pretrained_acoustic_models>`, but the language does not mark word boundaries in its orthography (and the language has a :xref:`pretrained tokenizer model <pretrained_tokenizer_models>`).
-
-   #. Follow :ref:`first_steps_tokenize` to tokenize the corpus
-   #. Use the tokenized transcripts and follow :ref:`first_steps_align_pretrained`
 
 .. _first_steps_align_pretrained:
 
@@ -337,36 +333,4 @@ Once the G2P model is trained, you should see the exported archive in the folder
 
    mfa g2p ~/mfa_data/my_new_word_list.txt ~/mfa_data/my_g2p_model.zip ~/mfa_data/my_new_dictionary.txt
 
-Take a look at :ref:`first_steps_g2p_pretrained` with this new model for a more detailed walk-through of generating a dictionary.
-.. _first_steps_tokenize:
-
-Tokenize a corpus to add word boundaries
-----------------------------------------
-
-For the purposes of this example, we'll also assume that you have done nothing else with MFA other than follow the :ref:`installation` instructions and you have the :code:`mfa` command working.  Finally, we'll assume that your corpus is in Japanese and is stored in the folder :code:`~/mfa_data/my_corpus`, so when working with your data, this will be the main thing to update.
-
-To tokenize the Japanese text to add spaces, first download the Japanese tokenizer model via:
-
-
-.. code-block::
-
-   mfa model download tokenizer japanese_mfa
-
-Once you have the model downloaded, you can tokenize your corpus via:
-
-.. code-block::
-
-   mfa tokenize ~/mfa_data/my_corpus japanese_mfa ~/mfa_data/tokenized_version
-
-You can check the tokenized text in :code:`~/mfa_data/tokenized_version`, verify that it looks good, and copy the files to replace the untokenized files in :code:`~/mfa_data/my_corpus` for use in alignment.
-
-.. warning::
-
-   MFA's tokenizer models are nowhere near state of the art, and I recommend using other tokenizers as they make sense:
-
-   * Japanese: `nagisa <https://nagisa.readthedocs.io/en/latest/>`_
-   * Chinese: `spacy-pkuseg <https://github.com/explosion/spacy-pkuseg/blob/master/readme/readme_english.md>`_
-   * Thai: `sertiscorp/thai-word-segmentation <https://github.com/sertiscorp/thai-word-segmentation>`_
-
-   The above were used in the initial construction of the training corpora for MFA, though the training segmentations for Japanese have begun to diverge from :code:`nagisa`, as they break up phonological words into morphological parses where for the purposes of acoustic model training and alignment it makes more sense to not split (nagisa: :ipa_inline:`使っ て [ts ɨ k a Q t e]` vs mfa: :ipa_inline:`使って [ts ɨ k a tː e]`). The MFA tokenizer models are provided as an easy start up path as the ones listed above may have extra dependencies and platform restrictions.
-
+Take a look at :ref:`first_steps_g2p_oovs` with this new model for a more detailed walk-through of generating a dictionary.

docs/source/user_guide/corpus_creation/tokenize.rst

@@ -4,6 +4,10 @@
 Tokenize utterances ``(mfa tokenize)``
 =========================================
 
+.. deprecated:: 3.4
+
+   The functionality for training tokenizers in MFA is deprecated and slated to be removed in MFA 4.0. For better solutions for tokenizing a given language, see :ref:`language_tokenization` for how to use dedicated packages and models for various languages.
+
 Use a model trained from :ref:`train_tokenizer_cli` to tokenize a corpus (i.e. insert spaces as word boundaries for orthographic systems that do not require them).
 
 Command reference

docs/source/user_guide/corpus_creation/train_tokenizer.rst

@@ -4,6 +4,10 @@
 Train a word tokenizer ``(mfa train_tokenizer)``
 ================================================
 
+.. deprecated:: 3.4
+
+   The functionality for training tokenizers in MFA is deprecated and slated to be removed in MFA 4.0. For better solutions for tokenizing a given language, see :ref:`language_tokenization` for how to use dedicated packages and models for various languages.
+
 Training a tokenizer uses a simplified sequence-to-sequence model like G2P, but with the following differences:
 
 * Both the input and output symbols are graphemes

docs/source/user_guide/implementations/alignment_analysis.md

@@ -4,6 +4,7 @@
 
 When exporting textgrids following alignment, an additional file named `alignment_analysis.csv` will be exported.  I am still currently working to refine what are the best measures for analyzing alignments as it's not entirely as straightforward as taking the overall alignment log-likelihood.
 
+(alignment_analysis_log_likelihood)=
 ## Alignment log-likelihood
 
 The first measure provided for each utterance is the alignment log-likelihood.  This represents overall the objective measure that was optimized for alignment. However, it is extremely important to note that this log-likelihood is a relative measure for the best path of alignment for this particular utterance compared to other possible alignments.
@@ -16,14 +17,35 @@ Additionally, log-likelihood reflects differences in the training data versus al
 
 The overall alignment log-likelihood represents the best path including all sections of silence.  In general when we're thinking about how good an alignment is, we don't necessarily care how good of a match the silence intervals in a given utterance are to the trained silence model.  So the speech log-likelihood measure takes out all log-likelihoods from silence intervals and is the average of per-phone log-likelihoods in the utterance.
 
+(alignment_analysis_duration_deviation)=
 ## Phone duration deviation
 
-Stepping back from log-likelihoods generated by the model, we can take a look at statistics of the duration of phones in the aligned corpus.  By calculating the mean and standard deviation of durations per phone, we can z-score the individual phone's duration to see how unexpected it is relative to the corpus overall.  The phone duration deviation measure is an average of the absolute z-score of each phones duration.
+Stepping back from log-likelihoods generated by the model, we can take a look at statistics of the duration of phones in the aligned corpus.  By calculating the mean and standard deviation of durations per phone, we can z-score the individual phone's duration to see how unexpected it is relative to the corpus overall.  The phone duration deviation measure is the maximum of the absolute z-score of each phone's log-duration.
 
-We use the absolute value of the z-score because often excessive durations due to misalignment will also result in excessively small durations on other phones. The average of raw z-scores in these cases will trend towards zero, when really we want these deviations to aggregate to utterances that clearly had something go wrong.
+We use the absolute value of the z-score because often excessive durations due to misalignment will also result in excessively small durations on other phones. The average of raw z-scores in these cases will trend towards zero, when really we want these deviations to aggregate to utterances that clearly had something go wrong.  The log-duration is used to better model the distribution of durations, as they are bounded by 0 (i.e., always positive), and skewed heavily towards lower durations with a long tail of longer durations, so a log-normal distribution provides a distribution that matches the assumptions of z-scores.
 
 It is important to note that there stylistic and speaker influences on duration, and statistics are gathered for the whole corpus, not normalized per speaker, so false positives are likely to pop up when sorting by this metric.  Normalizing per-speaker, however, might minimize the magnitude of duration deviation if a given speaker's utterances are all poorly aligned.  This would increase the likelihood of false negatives, and false positives are more acceptable than false negatives.
 
+(alignment_max_running_short_intervals)=
+## Max running short intervals
+
+This measure provides a sense of how many short intervals (10 ms or less) in a row are in the file. For long-running stretches of 10ms intervals in a row, this indicates that significant parts of the transcript were not correctly aligned or shoved together at the beginning/end of the file.
+
+(alignment_analysis_intensity_deviation)=
+## Intensity deviation
+
+Similar to {ref}`alignment_analysis_duration_deviation`, intensity deviation first calculates the mean and standard deviation of intensity over phones.  The mean and standard deviation are then used to create z-scores of individual interval intensities to allow for querying and search of phones/utterances that unexpectedly high or low intensities for a given interval, which could indicate an issue in alignment, such as if all the speech intervals are aligned to stretches of silence.
+
+However, there can be valid reasons for intensity to deviate related to speech style in particular, either if the speaker is excited or is whispering or is otherwise adopting an affect outside the rest of the speech.
+
+(alignment_analysis_snr)=
+## Signal-to-Noise Ratio (SNR)
+
+Signal-to-Noise Ratio (SNR) is calculated as the ratio of the mean intensity of speech phones to the mean intensity of the silence in an utterance.  If there is no silence aligned, then the calculation will use the minimum intensity in the utterance, so it may not be entirely accurate.
+
+SNR can be used in certain circumstances to help find alignment or transcript errors, such as when a word is missing from a transcript, and that section of speech is aligned as silence, resulting in a lower than expected SNR.
+
+It is important to note that SNR can vary due to recording conditions and may not be indicative of alignment errors alone.  Knowing the source data is particularly important for utilizing this metric to find errors, such as knowing that the data should be generally clean speech in quiet recording conditions vs uncontrolled or variable recording conditions.
 
 ## Ideas for the future that need a lot more thinking before I implement them