Skip to content

Commit 0ff8d45

Browse files
committed
feat: implement POS tag sync with C++ upstream (closes #40)
- Add tree-sitter based script to extract POS tags from C++ source - Add Makefile targets: sync-postypes, check-postypes - Update postype.go with tags from Kiwi v0.23.2 - Add new tags: POS_P, POS_PV, POS_PA, POS_PV_I, POS_PA_I - Remove non-existent tags: POS_V, POS_VV_R, POS_VA_R, POS_VX_R, POS_XSA_R - Fix naming: POS_USER_0 -> POS_USER0 (match C++ enum)
1 parent b869bf7 commit 0ff8d45

3 files changed

Lines changed: 361 additions & 117 deletions

File tree

Makefile

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,3 +25,23 @@ format:
2525
# go install mvdan.cc/gofumpt@latest
2626
gofumpt -l -w .
2727

28+
.PHONY: sync-postypes
29+
sync-postypes:
30+
@echo "Extracting POS tags from Kiwi $(KIWI_VERSION)..."
31+
source .venv/bin/activate && python scripts/extract_postags.py $(KIWI_VERSION)
32+
@echo "Generated postype_generated.go"
33+
@echo "Comparing with current postype.go..."
34+
@diff -u postype.go postype_generated.go || true
35+
@echo "To apply changes, run: mv postype_generated.go postype.go"
36+
37+
.PHONY: check-postypes
38+
check-postypes:
39+
@source .venv/bin/activate && python scripts/extract_postags.py $(KIWI_VERSION)
40+
@if diff -q postype.go postype_generated.go > /dev/null 2>&1; then \
41+
echo "POS types are in sync with Kiwi $(KIWI_VERSION)"; \
42+
else \
43+
echo "POS types are OUT OF SYNC with Kiwi $(KIWI_VERSION)"; \
44+
echo "Run 'make sync-postypes' to update"; \
45+
exit 1; \
46+
fi
47+

postype.go

Lines changed: 99 additions & 117 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)