Initial commit

Author: nshkrdotcom

.formatter.exs

@@ -0,0 +1,4 @@
+# Used by "mix format"
+[
+  inputs: ["{mix,.formatter}.exs", "{config,lib,test}/**/*.{ex,exs}"]
+]

.github/workflows/ci.yml

@@ -0,0 +1,35 @@
+name: CI
+
+on:
+  push:
+    branches:
+      - main
+      - master
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up Elixir
+        uses: erlef/setup-beam@v1
+        with:
+          elixir-version: 1.18.4
+          otp-version: 27.0
+
+      - name: Install dependencies
+        run: |
+          mix deps.get
+          mix deps.compile
+
+      - name: Check formatting
+        run: mix format --check-formatted
+
+      - name: Run tests
+        run: mix test --exclude oracle
+
+      - name: Run dialyzer
+        run: mix dialyzer

.gitignore

@@ -0,0 +1,27 @@
+# The directory Mix will write compiled artifacts to.
+/_build/
+
+# If you run "mix test --cover", coverage assets end up here.
+/cover/
+
+# The directory Mix downloads your dependencies sources to.
+/deps/
+
+# Where third-party dependencies like ExDoc output generated docs.
+/doc/
+
+# If the VM crashes, it generates a dump, let's ignore it too.
+erl_crash.dump
+
+# Also ignore archive artifacts (built via "mix archive.build").
+*.ez
+
+# Ignore package tarball (built via "mix hex.build").
+tiktoken_ex-*.tar
+
+# Temporary files, for example, from tests.
+/tmp/
+
+# Downloaded Kimi tokenizer oracle files (HF artifacts, venv, etc.)
+/oracle/kimi/
+/oracle/.venv/

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 North-Shore-AI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,101 @@
+<div align="center">
+  <img src="assets/tiktoken_ex.svg" width="400" alt="TiktokenEx Logo" />
+</div>
+
+# TiktokenEx
+
+**Pure Elixir TikToken-style byte-level BPE tokenizer (Kimi K2 compatible).**
+
+[![CI](https://github.com/North-Shore-AI/tiktoken_ex/actions/workflows/ci.yml/badge.svg?branch=master)](https://github.com/North-Shore-AI/tiktoken_ex/actions/workflows/ci.yml)
+[![Hex.pm](https://img.shields.io/hexpm/v/tiktoken_ex.svg)](https://hex.pm/packages/tiktoken_ex)
+[![Docs](https://img.shields.io/badge/docs-hexdocs.pm-blue.svg)](https://hexdocs.pm/tiktoken_ex)
+[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+
+TiktokenEx is a small, dependency-light implementation of the core TikToken
+idea:
+
+- Split text with a Unicode-aware regex (`pat_str`)
+- Encode pieces with byte-pair encoding (BPE) using `mergeable_ranks`
+- Optionally recognize special tokens (e.g. `<|im_end|>`)
+
+It’s focused on matching the behavior of MoonshotAI’s **Kimi K2** tokenizers
+that ship a `tiktoken.model` file and a TikToken-compatible `pat_str`.
+
+## Installation
+
+Add `tiktoken_ex` to your dependencies:
+
+```elixir
+def deps do
+  [
+    {:tiktoken_ex, "~> 0.1.0"}
+  ]
+end
+```
+
+## Usage
+
+### Build an encoding directly
+
+```elixir
+alias TiktokenEx.Encoding
+
+mergeable_ranks = %{
+  "He" => 0,
+  "ll" => 1,
+  "llo" => 2,
+  "H" => 10,
+  "e" => 11,
+  "l" => 12,
+  "o" => 13
+}
+
+{:ok, enc} = Encoding.new(pat_str: ".+", mergeable_ranks: mergeable_ranks)
+
+{:ok, ids} = Encoding.encode(enc, "Hello")
+{:ok, text} = Encoding.decode(enc, ids)
+```
+
+### Load a Kimi K2 encoding from local HuggingFace artifacts
+
+Kimi provides:
+
+- `tiktoken.model` (mergeable ranks)
+- `tokenizer_config.json` (special tokens, etc)
+
+```elixir
+alias TiktokenEx.{Encoding, Kimi}
+
+{:ok, enc} =
+  Kimi.from_hf_files(
+    tiktoken_model_path: "/path/to/tiktoken.model",
+    tokenizer_config_path: "/path/to/tokenizer_config.json"
+  )
+
+{:ok, ids} = Encoding.encode(enc, "Say hi")
+{:ok, decoded} = Encoding.decode(enc, ids)
+```
+
+### Special tokens
+
+Special tokens are recognized by default. To treat them as plain text:
+
+```elixir
+{:ok, ids} = TiktokenEx.Encoding.encode(enc, "<|im_end|>", allow_special_tokens: false)
+```
+
+### Regex compatibility note
+
+Kimi’s upstream `pat_str` uses character-class intersections (`&&`), which are
+not supported by Erlang’s PCRE engine. `TiktokenEx.Kimi.pat_str/0` provides a
+PCRE-compatible translation.
+
+## Development
+
+- Run tests: `mix test`
+- Run oracle parity tests (downloads HF artifacts): `mix test --include oracle`
+- Run dialyzer: `mix dialyzer`
+
+## License
+
+MIT © 2025 North-Shore-AI

assets/tiktoken_ex.svg

@@ -0,0 +1,8 @@
+defmodule TiktokenEx do
+  @moduledoc """
+  Pure-Elixir implementation of TikToken-style byte-level BPE encoding.
+
+  This project targets the subset needed to match MoonshotAI Kimi tokenizers
+  that ship a `tiktoken.model` file and a TikToken-compatible `pat_str`.
+  """
+end