AI Translation for German

A curated list of neural translation tools and multilingual corpus alignment techniques for German language processing.

Check out the German-English Dictionary App for Scientifically-Efficient Modern Learning 💡

Read this in Deutsch | 日本語

Contents

• Neural Translation Tools
  • Machine Translation Systems
  • Translation APIs
  • Open Source Translation Models
  • Translation Quality Evaluation
• Multilingual Corpus Alignment
  • Sentence Alignment Tools
  • Word Alignment Tools
  • Document Alignment
• Parallel Corpora
  • German-English Corpora
  • Multilingual Corpora
• Preprocessing Tools
  • Tokenization
  • Text Normalization
• Language Models for German
  • Transformer Models
  • Pre-trained Models
• Datasets
  • Translation Datasets
  • Evaluation Benchmarks
• Research Papers
• Learning Resources
  • Tutorials
  • Courses
  • Books
• Community

Neural Translation Tools

Machine Translation Systems

• DeepL - High-quality neural machine translation service with excellent German support.
• Google Translate - Widely-used translation service supporting German and 100+ languages.
• Microsoft Translator - Cloud-based machine translation service with German language support.
• SYSTRAN - Professional translation software with neural MT capabilities.
• ModernMT - Open-source neural machine translation platform with domain adaptation.
• Apertium - Free/open-source rule-based machine translation platform supporting German.

Translation APIs

• DeepL API - REST API for DeepL's neural translation engine.
• Google Cloud Translation API - Scalable translation API with AutoML capabilities.
• Amazon Translate - Neural machine translation service by AWS.
• Microsoft Translator Text API - Cloud translation API with customization options.
• LibreTranslate - Free and open-source machine translation API.

Open Source Translation Models

• MarianMT - Neural machine translation models from the University of Helsinki, includes German pairs.
• OPUS-MT-train - Training pipeline for Marian NMT models.
• fairseq - Facebook's sequence modeling toolkit with translation models.
• OpenNMT - Open-source neural machine translation framework.
• Sockeye - Sequence-to-sequence framework built on Apache MXNet.
• JoeyNMT - Minimalist neural machine translation toolkit.

Translation Quality Evaluation

• SacreBLEU - Standard BLEU implementation for comparing MT systems.
• COMET - Neural framework for training multilingual machine translation evaluation models.
• BERTScore - Automatic evaluation metric using BERT embeddings.
• BLEURT - Learned evaluation metric for natural language generation.
• Prism - Automatic MT metric based on sequence-to-sequence paraphrasing.

Multilingual Corpus Alignment

Sentence Alignment Tools

• Bleualign - Sentence alignment using MT and BLEU scores.
• Hunalign - Sentence-level alignment based on automatic dictionary creation.
• vecalign - Accurate sentence alignment using multilingual embeddings.
• Gargantua - Tool for aligning parallel texts.
• LF Aligner - Document and sentence alignment tool.

Word Alignment Tools

• fast_align - Fast, unsupervised word alignment implementation.
• eflomal - Efficient low-memory aligner for word alignment.
• awesome-align - Neural word alignment based on multilingual BERT.
• SimAlign - Word alignment based on static and contextualized embeddings.

Document Alignment

• bitextor - Tool for harvesting parallel documents from the web.
• WMT Document Alignment - Tools for document-level alignment in web crawling.

Parallel Corpora

German-English Corpora

• OPUS - Large collection of parallel corpora including German-English pairs.
• Europarl - Parallel corpus extracted from European Parliament proceedings.
• News Commentary - Political and economic commentary corpus.
• Tatoeba - Collection of sentences and translations in many languages.
• OpenSubtitles - Large parallel corpus based on movie subtitles.
• WMT Datasets - Annual shared task datasets for machine translation.

Multilingual Corpora

• OPUS-100 - English-centric multilingual corpus covering 100 languages.
• CCAligned - Multilingual parallel corpus mined from Common Crawl.
• WikiMatrix - Parallel sentences mined from Wikipedia.
• JW300 - Parallel corpus from jw.org in 300+ languages.
• TED Talks - Parallel corpus of TED talk transcripts.

Preprocessing Tools

Tokenization

• SentencePiece - Unsupervised text tokenizer and detokenizer.
• subword-nmt - Byte Pair Encoding for neural machine translation.
• SoMaJo - Tokenizer and sentence splitter for German.
• spaCy - Industrial-strength NLP with German language support.
• NLTK - Natural language toolkit with German tokenization.

Text Normalization

• Moses - Statistical machine translation toolkit with normalization scripts.
• clean-text - Python library for text cleaning and normalization.
• ftfy - Fixes text for you - handles encoding issues and mojibake.

Language Models for German

Transformer Models

• BERT German - German BERT models by deepset.
• German GPT-2 - German version of GPT-2 by dbmdz.
• GermanBERT - BERT model trained on German Wikipedia.
• GBERT - German BERT variants for downstream tasks.
• mBERT - Multilingual BERT supporting German.
• XLM-RoBERTa - Multilingual RoBERTa model.

Pre-trained Models

• German ELMo - German language ELMo embeddings.
• fastText German - Pre-trained word vectors for German.
• GloVe German - German GloVe word embeddings.

Datasets

Translation Datasets

• WMT Shared Tasks - Annual machine translation competition datasets.
• IWSLT - International Conference on Spoken Language Translation datasets.
• Multi30k - Multilingual image description dataset (English-German-French).
• Medical Translation - Medical domain parallel corpus.

Evaluation Benchmarks

• WMT Test Sets - Standard test sets for translation evaluation.
• FLORES-101 - Many-to-many multilingual translation benchmark.
• TICO-19 - Translation initiative for COVID-19 dataset.

Research Papers

• Attention Is All You Need - Foundational transformer architecture paper.
• Neural Machine Translation by Jointly Learning to Align and Translate - Attention mechanism for NMT.
• Google's Neural Machine Translation System - GNMT architecture.
• The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation - Hybrid NMT approaches.
• Unsupervised Cross-lingual Representation Learning at Scale - XLM-R paper.
• Massively Multilingual Neural Machine Translation in the Wild - Multilingual NMT findings.

Learning Resources

Tutorials

• Neural Machine Translation Tutorial - PyTorch NMT tutorial.
• OpenNMT Tutorial - Getting started with OpenNMT.
• Hugging Face Translation - Translation with transformers library.
• fairseq Tutorial - Translation examples with fairseq.

Courses

• Neural Machine Translation - Stanford CS224N - Stanford NLP course including NMT.
• Deep Learning for NLP - Carnegie Mellon course on neural NLP.
• Machine Translation - Edinburgh - University of Edinburgh MT course.

Books

• Neural Machine Translation - Comprehensive textbook by Philipp Koehn.
• Statistical Machine Translation - Classical MT textbook by Philipp Koehn.
• Foundations of Statistical Natural Language Processing - Essential NLP reference.

Community

• r/LanguageTechnology - Reddit community for NLP and MT.
• ACL Special Interest Group on Machine Translation - Professional organization for MT research.
• WMT Mailing List - Discussion group for MT shared tasks.
• German NLP Community - Resources for German language processing.

Contributing

Contributions are welcome! Please read the contribution guidelines first.

---

Maintained by the awesome-german community