Modernization: filelock + atomic writes, Python 3.10+/uv tooling, hardened C++ core, native token/terminal matching (2.2x faster parsing)

.gitattributes

@@ -1,6 +1,3 @@
-src/reynir/resources/ordalisti-*.bin filter=lfs diff=lfs merge=lfs -text
-src/reynir/resources/ord.compressed filter=lfs diff=lfs merge=lfs -text
-
 # Set the default line ending behavior to auto
 * text=auto
 

.github/workflows/python-package.yml

@@ -2,9 +2,10 @@ name: tests
 
 on:
   push:
-    branches: [ "*" ]
+    # Note: "**" is required to match branch names containing '/'
+    branches: [ "**" ]
   pull_request:
-    branches: [ "*" ]
+    branches: [ "**" ]
 
 jobs:
   build:
@@ -13,25 +14,32 @@ jobs:
     strategy:
       matrix:
         os: [ubuntu-latest]
-        python-version: ["3.9", "3.10", "3.11", "3.12", "3.13", "3.14", "pypy-3.11"]
+        python-version: ["3.10", "3.11", "3.12", "3.13", "3.14", "pypy3.11"]
+        # The Windows and macOS wheels are built from the same sources,
+        # so test at least one Python version on each of those platforms
+        include:
+          - os: windows-latest
+            python-version: "3.13"
+          - os: macos-latest
+            python-version: "3.13"
 
     steps:
     - uses: actions/checkout@v4
-    - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v5
+    - name: Install uv
+      # Exact tag: setup-uv has no floating 'v8' major tag (unlike v6/v7)
+      uses: astral-sh/setup-uv@v8.3.2
       with:
         python-version: ${{ matrix.python-version }}
     - name: Install GreynirEngine
-      run: |
-        python -m pip install uv
-        uv pip install --system wheel setuptools pytest ruff
-        uv pip install --system -e .
+      run: uv sync --locked
     - name: Lint with ruff
-      run: |
-        ruff check src/reynir
+      run: uv run ruff check src/reynir
+    - name: Type check with mypy
+      # mypy is not installed on PyPy (see [dependency-groups] in pyproject.toml)
+      if: ${{ !startsWith(matrix.python-version, 'pypy') }}
+      run: uv run mypy src/reynir
     - name: Test with pytest
-      run: |
-        python -m pytest
+      run: uv run pytest
     - name: Slack notification
       uses: 8398a7/action-slack@v3
       with:

.github/workflows/wheels.yml

@@ -16,26 +16,25 @@ jobs:
 
     steps:
       - uses: actions/checkout@v4
-        with:
-          lfs: true
 
       - uses: actions/setup-python@v5
         with:
           python-version: '3.11'
 
       - name: Install cibuildwheel
-        run: python -m pip install --upgrade pip wheel setuptools cibuildwheel
+        # Pinned so that releases only change toolchain when we choose to
+        run: python -m pip install --upgrade pip wheel setuptools cibuildwheel==4.1.0
 
       - name: Build wheels
         run: python -m cibuildwheel --output-dir wheelhouse
         # Options (https://cibuildwheel.readthedocs.io/en/stable/options/)
         env:
-          # Build abi3 wheel for CPython 3.9+ (one wheel for all versions)
+          # Build abi3 wheel for CPython 3.10+ (one wheel for all versions)
           # Build version-specific wheel for PyPy (which doesn't support abi3)
-          CIBW_BUILD: cp39-* pp311-*
+          CIBW_BUILD: cp310-* pp311-*
           CIBW_SKIP: "*musllinux*"
           CIBW_ENABLE: pypy
-          CIBW_PROJECT_REQUIRES_PYTHON: ">=3.9"
+          CIBW_PROJECT_REQUIRES_PYTHON: ">=3.10"
           CIBW_BEFORE_BUILD_MACOS: python3 -m pip install --upgrade setuptools wheel cffi
           CIBW_ARCHS_MACOS: "x86_64 arm64"
           CIBW_ARCHS_WINDOWS: "AMD64"
@@ -54,8 +53,6 @@ jobs:
 
     steps:
       - uses: actions/checkout@v4
-        with:
-          lfs: true
 
       - uses: actions/setup-python@v5
         with:

.gitignore

@@ -52,6 +52,8 @@ test/test_corpus/handpsd/annotaldLog.txt
 src/reynir/_eparser.cpp
 *.bin
 !ordalisti-*.dawg.bin
+*.bin.lock
+*.bin.tmp.*
 
 *.sublime-project
 *.sublime-workspace
@@ -93,7 +95,6 @@ p37/
 pypy*
 
 # uv stuff
-uv.lock
 .python-version
 
 # Installer logs

CLAUDE.md

@@ -0,0 +1,106 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Overview
+
+GreynirEngine (PyPI package name: `reynir`, imported as `reynir`) is an NLP engine
+for parsing Icelandic text into sentence trees. It combines a hand-written
+context-free grammar for Icelandic with a C++ Earley-Scott parser core, wrapped
+via CFFI. Source lives in `src/reynir/`. Supports CPython 3.10+ and PyPy 3.11+.
+
+## Commands
+
+The project uses [uv](https://docs.astral.sh/uv/) with a committed `uv.lock`;
+dev dependencies live in `[dependency-groups]` in `pyproject.toml`.
+
+```sh
+# Set up the dev environment (installs dev deps from uv.lock and
+# compiles the C++ parser extension via CFFI)
+uv sync
+
+# Run all tests
+uv run pytest
+
+# Run a single test file / single test
+uv run pytest test/test_parse.py
+uv run pytest test/test_parse.py::test_long_parse
+
+# Lint (CI runs this)
+uv run ruff check src/reynir
+
+# Type check (config in pyproject.toml [tool.mypy]; carries a handful of
+# known pre-existing errors, so it is not a CI gate)
+uv run mypy src/reynir
+```
+
+Note: after changing the C++ sources (`eparser.cpp`, `eparser.h`) or
+`eparser_build.py`, force a rebuild of the `_eparser` CFFI extension with
+`uv sync --reinstall-package reynir` (the extension is compiled at install
+time; see `setup.py`, which exists only for the `cffi_modules` hook — all
+other metadata is in `pyproject.toml`).
+
+## Architecture
+
+The parsing pipeline, orchestrated by the `Greynir` class in `reynir.py`
+(`parse()`, `parse_single()`, `submit()`):
+
+1. **Tokenization** — the external `tokenizer` package splits text into tokens;
+   `bintokenizer.py` is a dictionary-aware layer that annotates each token with
+   its possible meanings from BÍN (the Database of Icelandic Morphology), via
+   `bindb.py`, which wraps the external `islenska`/BinPackage vocabulary database.
+
+2. **Grammar** — `Greynir.grammar` is a ~7,500-line CFG for Icelandic in extended
+   BNF, read by `grammar.py`. It is compiled to a binary form
+   (`Greynir.grammar.bin`, gitignored) automatically at runtime whenever the
+   source grammar file is newer. The binary is written atomically (temp file +
+   `os.replace`), and regeneration is serialized across processes with a
+   `filelock` lock file next to the binary (`Greynir.grammar.bin.lock`);
+   acquisition times out with a `GrammarError` instead of hanging. The lock
+   is only taken when the binary is missing or stale — the warm path is
+   lock-free, guarded only by an in-process `threading.Lock` protecting the
+   class-level grammar caches. `glock.py` is a deprecated compatibility shim
+   over `filelock`, no longer used internally.
+
+3. **Parsing** — `fastparser.py` (`Fast_Parser`) wraps the C++ Earley-Scott
+   parser (`eparser.cpp`) through the `_eparser` CFFI extension, producing a
+   parse forest (SPPF) of all possible parses. `binparser.py` (`BIN_Parser`)
+   maps BÍN token meanings to grammar terminals (e.g. `no_et_nf_kvk` = noun,
+   singular, nominative, feminine); terminal matching logic lives here.
+   `baseparser.py` holds a pure-Python reference parser. `incparser.py`
+   handles incremental parsing of token streams by paragraph/sentence.
+
+4. **Reduction** — `reducer.py` scores the parse forest and reduces it to the
+   single most likely tree, using preferences from `config/Prefs.conf`,
+   production priorities and `$score()` pragmas in the grammar, and
+   verb-preposition matching driven by `config/Verbs.conf`.
+
+5. **Output/API** — `simpletree.py` provides `SimpleTree`, the simplified tree
+   API users interact with (`.tree.S.IP.NP_SUBJ`, `.lemmas`, `.nouns`, `.flat`,
+   etc.). `matcher.py` implements pattern matching over these trees.
+   `nounphrase.py` provides the `NounPhrase` class with case inflection through
+   `__format__` (e.g. `f"{np:þgf}"`). `lemmatize.py`, `verbframe.py` and
+   `ifdtagger.py` provide lemmatization, verb frames and IFD-style POS tagging.
+
+Supporting data:
+
+- `src/reynir/config/*.conf` — linguistic configuration (verbs with their
+  argument cases and prepositions, phrase preferences, name preferences,
+  adjective/noun predicates, etc.), loaded by `settings.py`.
+- `src/reynir/resources/ord*.csv` — vocabulary additions layered on top of BÍN.
+
+The public API is defined by the exports in `src/reynir/__init__.py`.
+`Reynir` is retained as a compatibility alias for `Greynir`.
+
+## Conventions
+
+- Grammar terminal and category names are Icelandic abbreviations (`no`=noun,
+  `so`=verb, `nf`/`þf`/`þgf`/`ef`=cases, `et`/`ft`=number, `kk`/`kvk`/`hk`=gender);
+  these appear throughout the code, tests and grammar files.
+- Ruff line length is 88; `E731` (lambda assignment) is ignored.
+- CI (`.github/workflows/python-package.yml`) runs ruff + pytest via
+  `uv sync --locked` on Python 3.10–3.14 and PyPy 3.11 on Linux, plus one
+  job each on Windows and macOS. Wheels are built on tag push via a pinned
+  cibuildwheel (`cp310` abi3 wheel for CPython, version-specific for PyPy).
+- The `old/` and `build/` directories contain legacy/build artifacts — do not
+  edit code there.

MANIFEST.in

@@ -4,6 +4,8 @@ prune src/reynir/.mypy_cache
 prune test/test_corpus
 include src/reynir/Greynir.grammar
 exclude src/reynir/Greynir.*.bin
+exclude src/reynir/*.bin.lock
+exclude src/reynir/*.bin.tmp.*
 include src/reynir/eparser.h
 exclude src/reynir/_eparser.cpp
 include src/reynir/config/*.conf

README.md

@@ -1,5 +1,5 @@
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![Python 3.9](https://img.shields.io/badge/python-3.9-blue.svg)](https://www.python.org/downloads/release/python-3817/)
+[![Python 3.10](https://img.shields.io/badge/python-3.10-blue.svg)](https://www.python.org/downloads/)
 ![Release](https://shields.io/github/v/release/mideind/GreynirEngine?display_name=tag)
 ![PyPI](https://img.shields.io/pypi/v/reynir)
 [![Build](https://github.com/mideind/GreynirEngine/actions/workflows/python-package.yml/badge.svg)]()
@@ -12,7 +12,7 @@
 
 ## Overview
 
-Greynir is a Python 3 (>=3.9) package,
+Greynir is a Python 3 (>=3.10) package,
 published by [Miðeind ehf.](https://mideind.is), for
 **working with Icelandic natural language text**.
 Greynir can parse text into **sentence trees**, find **lemmas**,
@@ -147,7 +147,7 @@ and each token annotated with its lemma and POS tag (`no`=noun, `so`=verb):
 
 ## Prerequisites
 
-This package runs on CPython 3.9 or newer, and on PyPy 3.11 or newer.
+This package runs on CPython 3.10 or newer, and on PyPy 3.11 or newer.
 
 To find out which version of Python you have, enter:
 
@@ -195,14 +195,17 @@ The package source code is in `GreynirEngine/src/reynir`.
 
 ## Tests
 
-To run the built-in tests, install [pytest](https://docs.pytest.org/en/latest),
-`cd` to your `GreynirEngine` subdirectory (and optionally activate your
-virtualenv), then run:
+To run the built-in tests, `cd` to your `GreynirEngine` subdirectory and,
+using [uv](https://docs.astral.sh/uv/), run:
 
 ````sh
-python -m pytest
+uv sync
+uv run pytest
 ````
 
+Alternatively, install [pytest](https://docs.pytest.org/en/latest) into
+your virtualenv and run `python -m pytest`.
+
 ## Evaluation
 
 A parsing test pipeline for different parsing schemas, including the Greynir schema,
@@ -219,22 +222,13 @@ as well as important information about
 
 ## Troubleshooting
 
-If parsing seems to hang, it is possible that a lock file that GreynirEngine
-uses has been left locked. This can happen if a Python process that uses
-GreynirEngine is killed abruptly. The solution is to delete the lock file
-and try again:
-
-On Linux and macOS:
-
-````sh
-rm /tmp/greynir-grammar  # May require sudo privileges
-````
-
-On Windows:
-
-````cmd
-del %TEMP%\greynir-grammar
-````
+GreynirEngine uses a lock file, located alongside the binary grammar file
+within the package directory (`reynir/Greynir.grammar.bin.lock`), to
+serialize the (re)generation of the binary grammar between processes.
+If a process gets stuck holding the lock, parser initialization fails
+after a timeout with an error message identifying the lock file.
+Terminate the process holding the lock, or - if no such process exists -
+delete the lock file named in the error message, and try again.
 
 ## Copyright and licensing