safetensors.mojo

safetensors.mojo is an early, pure-Mojo implementation of the Safetensors file format. It provides a strictly validated runtime-independent format core, local single-file and sharded random-access readers, and POSIX memory-mapped zero-copy views for raw tensor bytes and selected exact native scalar encodings. It also provides a deterministic one-shot writer with atomic local-file replacement. It does not load values into a tensor runtime.

Disclaimer: safetensors.mojo is an independent implementation of the Safetensors file format for Mojo and is not affiliated with or endorsed by Hugging Face.

Naming

The project uses distinct names at each packaging layer:

• Repository and project brand: safetensors.mojo
• Pixi/Conda distribution: safetensors-mojo
• Importable Mojo package: safetensors

The project does not create or use a mojo.safetensors namespace.

Current scope

The current API provides:

• the independent SafeDType wire-format model;
• raw and validated metadata structures;
• decoding of the 8-byte little-endian header length;
• schema-directed UTF-8 and Safetensors JSON header parsing;
• reference-compatible leading and trailing JSON whitespace handling;
• bounded skipping of unknown tensor descriptor fields, with an opt-in canonical-schema strict mode;
• decoded-key duplicate detection;
• exact unsigned integer parsing without a floating-point intermediate;
• checked shape, bit-length, byte-length, offset, and full-coverage validation;
• valid and malformed compatibility fixtures;
• a byte-exact differential matrix for the 20 dtypes emitted by the pinned reference serializer, plus reference-deserializer coverage for both F6 dtypes, using valid scalar, multidimensional, and zero-element shapes;
• metadata-only opening through an owned read-only file handle;
• exact named-tensor reads into caller-owned byte buffers;
• explicit owned loads of one tensor payload;
• whole-file read-only private mappings on every supported platform;
• immutable named-tensor byte spans whose Mojo origins are tied to the mapping owner;
• exact immutable native scalar spans with checked dtype, size, endianness, and actual-address alignment;
• an owned raw tensor input model covering every recognized Safetensors dtype;
• deterministic checked serialization through atomic local-file replacement;
• standard Safetensors shard-index parsing with exact global routing and size validation;
• buffered sharded reads with at most one active shard reader; and
• eager zero-copy sharded mappings whose views remain tied to the aggregate owner.

Reader compatibility does not relax dtype, shape, size, offset, or complete coverage validation. The architecture guide describes how the implementation works today. The numbered decision history preserves why important choices were made and how they evolved. Curated change history is published in GitHub Releases, while planned work belongs in GitHub Issues.

Usage

The 0.7.0 Conda package targets linux-64, linux-aarch64, and osx-arm64 in the shared modular-community Prefix.dev channel. Add that channel before the Modular and conda-forge channels, then install the distribution with the supported compiler:

[workspace]
channels = [
  "https://prefix.dev/modular-community",
  "https://conda.modular.com/max",
  "conda-forge",
]
platforms = ["linux-64", "linux-aarch64", "osx-arm64"]

[dependencies]
safetensors-mojo = "==0.7.0"
mojo-compiler = "==1.0.0"

The installed Mojo package is imported as safetensors. These are native artifacts: select the platform matching the host rather than cross-building one target on another.

Create a file from raw tensor wire bytes with the one-shot writer:

from safetensors import SafeDType, SafeTensorData, save_safetensors


def main() raises:
    var tensors: List[SafeTensorData] = [
        SafeTensorData(
            "values",
            SafeDType.F32,
            [UInt64(2)],
            [UInt8(0), 0, 0x80, 0x3F, 0, 0, 0, 0xC0],
        )
    ]
    var metadata = Dict[String, String]()
    metadata["producer"] = "example"
    save_safetensors("model.safetensors", tensors, metadata)

SafeTensorData.data must already contain packed C-order little-endian bytes for the declared dtype and shape. The writer validates every input and computes the complete layout before touching the filesystem. It emits canonical compact JSON, deterministic tensor and metadata ordering, and 8-byte header padding.

The destination's parent directory must already exist. A successful write creates an exclusive sibling temporary file requesting mode 0600, closes it, and atomically replaces the destination entry with rename(2). Existing readers and mappings remain attached to the previous inode. This is an atomic visibility guarantee, not a crash-durability guarantee: the writer does not call fsync.

Open a local file without loading its tensor data, inspect the validated metadata, and explicitly load one tensor as owned raw wire bytes:

from safetensors import open_safetensors


def main() raises:
    var reader = open_safetensors("model.safetensors")
    var metadata = reader.metadata()
    var info = metadata.info("weights")
    var bytes = reader.load_tensor("weights")

    print(info.dtype, info.shape, info.byte_length)
    print("loaded bytes:", len(bytes))

To reuse caller-owned storage, pass a mutable byte buffer whose length exactly matches the validated tensor byte length:

var destination = List[UInt8](length=16, fill=0)
reader.read_tensor_into("weights", destination)

SafeTensorReader owns one file handle and is movable but not copyable. Calls on one reader share its seek cursor and must not execute concurrently. Opening reads only the 8-byte prefix and declared JSON header; tensor data is read only by read_tensor_into() or load_tensor().

Readers accept the same leading and trailing JSON whitespace as the reference implementation and ignore syntactically valid unknown tensor descriptor fields by default. Pass strict=True to a single-file or sharded parser or reader to require byte zero of each Safetensors header to be {, allow only ASCII-space padding after its root object, and reject unknown descriptor fields. Both modes apply the same exact-integer and complete metadata validation. Shard index JSON itself always uses its fixed compatibility and security policy.

Memory-mapped views

from safetensors import map_safetensors


def main() raises:
    var mapped = map_safetensors("model.safetensors")
    var info = mapped.metadata().info("weights")
    var bytes = mapped.tensor_bytes("weights")
    var values = mapped.tensor_view[DType.float32]("weights")

    print(info.dtype, info.shape, info.byte_length)
    print("mapped bytes:", len(bytes), bytes[0])
    print("native values:", len(values), values[0])

MappedSafeTensorFile owns one POSIX PROT_READ | MAP_PRIVATE whole-file mapping and is movable but not copyable. tensor_bytes() returns an immutable Span[UInt8] without copying the payload. Its Mojo origin prevents subsequent use after the mapping owner is consumed.

tensor_view[DType.float32]() returns a flat immutable native scalar span with the same origin and no payload copy. The requested compile-time DType must exactly match the file metadata. Supported mappings are signed and unsigned 8-, 16-, 32-, and 64-bit integers; F16, BF16, F32, and F64; and Mojo 1.0's five matching float8 encodings. BOOL, F4, both F6 encodings, and C64 remain raw-byte-only.

Non-empty multi-byte views require a little-endian host, and every non-empty view requires an actually aligned mapped address. An otherwise valid unaligned tensor still has raw byte access. Empty exact-dtype views skip endianness and address checks because they contain no dereferenceable element. Unsupported dtypes, mismatches, incompatible byte order, and misalignment produce typed SafeTensorError values. The tensor's logical shape remains in its validated metadata; the returned span is deliberately one-dimensional.

The backing inode must remain unchanged from before map_safetensors() starts until the mapping owner and every borrowed span are dead. A length check before returning a span catches an already-observed growth or truncation, but cannot eliminate the later race: dereferencing pages after external truncation can terminate the process with SIGBUS. Renaming, unlinking, or replacing the path does not redirect an existing mapping.

Sharded archives

Open a standard local *.safetensors.index.json archive through one global tensor namespace:

from safetensors import open_safetensors_index


def main() raises:
    var reader = open_safetensors_index("model.safetensors.index.json")
    var info = reader.metadata().info("decoder.weight")
    var bytes = reader.load_tensor("decoder.weight")
    print(info.shard, info.dtype, len(bytes))

The index reader treats decoded weight_map filenames as untrusted. Each must be one .safetensors basename and is opened beside the lexical index path without following a shard symlink. Every referenced file is validated, every physical tensor must have exactly the declared route, duplicate names are rejected, and metadata.total_size is checked exactly when present. The default index-size, weight_map entry-count, and unique-shard limits are available as DEFAULT_MAX_INDEX_BYTES, DEFAULT_MAX_INDEX_ENTRIES, and DEFAULT_MAX_SHARDS. This resolution policy is not a sandbox for an attacker-writable archive directory: hard links to regular files cannot be distinguished from entries created directly in that directory.

Hugging Face cache snapshots commonly expose shard files as symlinks into blob storage. For that trusted application layout, resolve or enumerate the paths in the application and use the explicit-list API:

from safetensors import open_sharded_safetensors


def main() raises:
    var reader = open_sharded_safetensors([
        "snapshot/model-00001-of-00002.safetensors",
        "snapshot/model-00002-of-00002.safetensors",
    ])
    print(reader.metadata().len())

Explicit path arguments are trusted and may follow symlinks; never copy untrusted weight_map strings into that API. map_safetensors_index() and map_sharded_safetensors() provide the corresponding zero-copy interface. Mapped sharded archives eagerly retain one descriptor and one whole-file mapping per unique shard so immutable views from different shards can coexist. Buffered sharded readers instead keep at most one active shard descriptor and revalidate a shard against its opening identity, length, and metadata whenever they switch files. names() remains globally lexicographic; for batch buffered reads, iterate reader.metadata().shard_grouped_names() to receive a deterministic order grouped by shard and avoid reopening the same shard for each interleaved tensor name. The complete contract is documented in the sharded-reader architecture.

The format core remains available for caller-owned buffers containing a complete .safetensors file:

from std.pathlib import Path

from safetensors import parse_metadata


def main() raises:
    var bytes = Path("model.safetensors").read_bytes()
    var metadata = parse_metadata(bytes)

    for name in metadata.names():
        var info = metadata.info(name)
        print(name, info.dtype, info.shape, info.begin, info.end)

The parser validates ranges against the remaining data-buffer length but neither copies nor interprets tensor data. File-reader results are raw wire bytes. Wire data is defined as packed C-order and little-endian.

Validated metadata accessors return copies. Mojo 1.0 does not enforce field visibility, so underscore-prefixed fields and direct SafeTensorMetadata or SafeTensorReader or MappedSafeTensorFile construction are implementation details. Mutating or constructing this state outside the public parsing and opening functions is unsupported and can invalidate the validated-state contract. The supported API is exported from the root safetensors package; nested module paths are internal and may change between releases.

Performance

pixi run -e benchmark benchmark generates a sparse archive with 193 F32 tensors and a 967 MiB logical payload, builds native Mojo and Rust workers, and compares the same operation through safetensors.mojo, the official Rust safetensors crate, and the Python package: open or map the file, validate the complete header, obtain the one-element first tensor, touch its value, and close. Both reference implementations are pinned to Safetensors 0.8.0. The first tensor intentionally contains one F32 value so the measurement isolates opening and header validation instead of a framework-specific payload copy. The remaining sparse payload is not scanned.

The three-way reports were collected on 2026-09-06 using safetensors.mojo 0.7.0, Rust 1.98.0, Mojo 1.0.0, Python 3.12.14, Safetensors 0.8.0, and NumPy 2.5.2 on Linux 7.2.2-1-cachyos. Each contains 500 warm samples per implementation across six batches with 50 warmups each, and 30 fresh-process samples per implementation after three warmup rounds. The execution order rotates among Mojo, Rust, and Python to reduce ordering bias. Each cell is median / p95.

Intel Core i7-1255U (JSON report, clean commit 6088655):

Operation	Mojo	Rust	Python
Warm process: open/map, validate, first-value touch	0.696 / 0.746 ms	0.229 / 0.252 ms	0.274 / 0.298 ms
Fresh process plus the same operation	16.287 / 17.934 ms	2.638 / 2.917 ms	215.401 / 243.953 ms

Intel Core i5-12400F (JSON report, clean commit 98e2e2b):

Operation	Mojo	Rust	Python
Warm process: open/map, validate, first-value touch	0.290 / 0.307 ms	0.096 / 0.102 ms	0.116 / 0.138 ms
Fresh process plus the same operation	7.336 / 7.679 ms	0.800 / 1.168 ms	90.728 / 109.504 ms

The Rust reference is the fastest implementation on both machines in this workload. Python is close to Rust once its runtime and imports are warm. Mojo does not beat the native reference parser, but its fresh process avoids the CPython startup and imports measured by the Python worker. This is an end-to-end public API comparison, not a parser-only benchmark: the Mojo mapping path reads and validates the header before mapping, checks that the file length remains stable around mapping, and checks it again during typed access. The Rust worker maps first and deserializes directly from that mapping. These additional integrity checks account for part of Mojo's measured cost, but this benchmark does not isolate their individual contribution.

Earlier two-way reports from clean commit 560a286 are retained for cross-machine context. They used four alternating batches and did not include the Rust worker. Each value is median / p95:

CPU	Mojo warm	Python warm	Mojo fresh	Python fresh
Intel Core i7-1255U	0.688 / 0.835 ms	0.273 / 0.293 ms	15.622 / 17.567 ms	175.383 / 205.338 ms
Intel Core i5-12400F	0.291 / 0.437 ms	0.113 / 0.118 ms	7.280 / 7.639 ms	90.499 / 106.481 ms

Fresh-process timings include process startup and, for Python, Safetensors and NumPy imports. Every operation uses a warm page cache for the header and first payload page. These measurements cover opening, validation, and a first-value touch, not a complete payload load. Results depend on the machine, environment, and workload.

The harness stores its configuration, environment, summaries, and raw samples in the ignored .pixi/benchmarks/latest.json report. Selected measurements are published under benchmarks/results/ with dated filenames; checkout-specific paths are made relative to the repository, while timings and all other fields are preserved. Rust and Cargo are confined to the separate Pixi benchmark environment and are not package or default development dependencies.

Deliberate limitations

Mapped access exposes borrowed raw byte spans and an exact whitelist of native scalar spans on supported platforms. It does not provide a decoded or byte-swapped fallback for other encodings or layouts. The writer does not provide serialization to a complete in-memory archive, typed-value encoding, byte swapping, incremental or stateful writes, append/update-in-place behavior, or mmap writes. Remote Hub downloads, index writing, automatic shard planning, slicing, MAX adapters, and other tensor-runtime adapters remain outside the current scope. Parser, reader, and writer APIs do not interpret tensor values.

Safetensors prevents arbitrary code execution through its data format, but it does not provide authenticity, integrity, signatures, encryption, or protection against in-place mutation. A retained reader handle prevents path replacement from redirecting later reads and detects ordinary file-length changes around a read, but it cannot detect same-length changes to already opened file contents. A read-only private mapping is likewise not an immutable snapshot and requires a stable backing file for its entire lifetime.

Development

The supported toolchain is Mojo 1.0.0 on these native hosts:

Pixi platform	Mojo 1.0 host requirements
linux-64	Linux with glibc 2.34 or later, an x86-64-v3 (Haswell-class or newer) CPU, and a C compiler available as the linker
linux-aarch64	Linux with glibc 2.34 or later, an ARM64 Neoverse N1-class or newer CPU, and a C compiler available as the linker
osx-arm64	Apple silicon running macOS 15 or later, with Xcode or Xcode Command Line Tools 16 or later

The project does not support osx-64; Mojo 1.0 supports macOS on Apple silicon only. See the upstream Mojo system requirements for the complete host requirements. Pixi installs the exact compiler version and the Python-only development dependencies used to generate reference fixtures. The generated .mojoc package is compiler-version-specific and must be consumed with Mojo 1.0.0. Release builds and tests must run natively for each package platform; cross-building is unsupported.

The repository is organized by responsibility:

src/safetensors/
  format/       # Runtime-independent parsing, validation, and write planning
  io/           # Buffered, mapped, and atomic local-file access
  sharding/     # Index validation and aggregate buffered/mapped readers
benchmarks/      # Reproducible Mojo, Rust, and Python performance comparisons
tests/
  unit/         # Focused format-core behavior
  integration/  # Fixture and local-I/O behavior
  contracts/    # Compile-time public and ownership contracts
  tooling/      # Python tooling tests
  fuzz/         # Standalone hostile-corpus survival harness
tools/
  checks/       # Formatting and test orchestration
  fixtures/     # Reproducible fixture generation
  fuzz/         # Deterministic hostile-corpus generation
  packaging/    # Clean-install package smoke tests
  release/      # Release-policy and channel preflight checks

Mojo production modules use absolute safetensors.* imports. The format subpackage depends only on shared errors, io depends on the format core, and sharding composes both layers. The root package re-exports the supported API from all three subpackages.

pixi install
pixi run check
pixi run fuzz
pixi run fuzz-index
pixi run -e benchmark benchmark
pixi run all

pixi run check verifies formatting, runs the Python tooling and compile-time API contract tests, compiles the importable package, runs the Mojo tests, and checks that fixtures are reproducible. pixi run all additionally builds the safetensors-mojo Conda package and verifies it in a clean Pixi workspace. Individual tasks include compile, test, format-check, fixtures-check, fuzz, fuzz-index, and package-build; benchmark belongs to the separate benchmark environment. Fuzzing and benchmarking stay outside check and all because each generates its own ignored data. The deterministic fuzz tasks have a separate CI job; the machine-sensitive benchmark remains manual and outside CI. The fuzz failure model and randomized triage commands are documented in docs/fuzzing.md.

Release artifacts are published by the tag workflow after a clean package installation test. Maintainer setup and the release checklist are documented in docs/releasing.md.

To rewrite Mojo sources with the canonical formatter, run pixi run format. To regenerate the committed fixture corpus, run pixi run fixtures.

License

Licensed under the Apache License, Version 2.0. See LICENSE.