Multikernel inference: RTX 5070 Ti + Tesla P40

An engineering experiment running one language model across two Linux kernels inside a KVM virtual machine. Each kernel owns a different passed-through GPU and uses the NVIDIA driver appropriate for that GPU. Patched llama.cpp connects the two through native Multikernel AF_VSOCK/IPI RPC.

This repository preserves the tested source changes, configuration, probes and engineering record. It is not a turnkey installer or a bootable VM backup.

Architecture

KVM VM — Shiba
│
├── Multikernel primary: 7.0.0-mk2-shiba-mk1
│   ├── RTX 5070 Ti → nvidia-open 595.84 → CUDA
│   └── llama.cpp coordinator / local CUDA backend
│                  │
│                  │ native AF_VSOCK / shared-memory IPI
│                  │ mkvsock:1:5002
│                  ▼
└── Multikernel secondary
    ├── Tesla P40 → proprietary NVIDIA 580.173.02 → CUDA
    └── llama.cpp RPC worker

The two drivers run in separate kernels. The experiment does not require changes to the physical Proxmox host. NVIDIA versions above describe the tested pre-upgrade configuration, not a compatibility claim for arbitrary newer packages.

Current status

The VM has upgraded to Ubuntu 26.04.1 LTS and successfully rebooted into 7.0.0-mk2-shiba-mk1. Primary NVIDIA-open modules now match userspace 595.91.07; the P40 secondary remains on 580.173.02. Kerf was repaired for Python3.14 (330 tests passed). Both RPC listeners passed exact CUDA tensor tests through16MiB.

A post-upgrade50/50 Nemotron CLI test reached 70.3 prompt /76.0 generation tokens/s, with exact visible-output agreement against the pre-upgrade reference. This is one16-token short-context run, not a replacement for the repeated72.0tokens/s reference or evidence of long-context/API throughput. The model-switching API is enabled and healthy with15 entries and no resident model.

Recovery is functional but not warning-free: the primary emitted one softirq WARN in the Multikernel VSOCK hard-IRQ receive/socket-lookup path before benchmarking. No kernel changes were made; the call stack is documented in the audit for separate investigation.

Measured results

Configuration	Prompt tokens/s	Generation tokens/s
RTX CUDA + host RAM baseline	55.6	15.5
P40 Nouveau/NVK + host RAM baseline	12.1	8.0
RTX CUDA + P40 NVK split baseline	6.0	8.3
Native Multikernel RPC, fast transport, 48/52 split	69.2	66.0
Native Multikernel RPC + backend sampler optimization, 50/50 split	—	72.0

The final 50/50 CLI reference had generation runs of 71.9 and 72.1 tokens/s. These are Nemotron Q8 measurements under the conditions recorded in the engineering audit. API serving, allocated context and split settings affect results: the separately tested 160K-context API profile reached approximately 58 tokens/s and should not be presented as the 72 tokens/s CLI result.

Native transport initially measured a 52.920 µs ping/pong RTT, 135.47 MiB/s for 1 MiB and 129.02 MiB/s for 16 MiB. Removing unnecessary read and secondary-send progress sleeps brought a verified 512 KiB return to 2.452 ms. These measurements come from different isolated tests, not one simultaneous benchmark.

Other split-model checks after the upgrade

Isolated4K-context tests, one warmup and two measured128token greedy completions per placement, with exact local/split output agreement on the tested prompt:

Model	RTX/P40 split	Local RTX+CPU decode	Split decode	Split load
GLM4.7Flash Q6_XL	45/55	6.80tok/s	48.06tok/s	96.03s
Ling3.0Flash IQ2_S	40/60	4.87tok/s	30.22tok/s	154.09s

These are short-context mean rates, not validated large-context API profiles. The live catalog is unchanged. Mistral remainsRTX-only; distributed Qwen27B output correctness, Muse secondaryRAM capacity, and K2 native transport remain unresolved. Reports and the isolated harness are in model-api/.

Transport and safeguards

The socket explicitly selects Multikernel transport:

int transport = 1;
setsockopt(fd, AF_VSOCK, 9, &transport, sizeof(transport));

Without this selection, generic AF_VSOCK can choose the VM transport and return ENODEV before entering Multikernel. The llama.cpp implementation is isolated in the RPC transport layer and handles partial reads/writes. It uses native sockets; socat is not the inference transport.

• Port 5000 remains the conservative RPC fallback.
• Port 5002 is the tested fast configuration.
• Primary upload/write pacing remains enabled, using approximately 32 KiB chunks. Removing it caused sustained IPI-ring ENOSPC during model loading and was rejected.
• Primary read sleeps and secondary read/send sleeps are disabled in the fast configuration.
• The stable backend sampler optimization is included in llama.cpp commit 3e88755aa.

The experiment runs with virtualized PCI passthrough. Reset support, interrupts, DMA and guest IOMMU facilities need separate inspection when reproducing it. A failed secondary shutdown can require a full VM reboot to recover the P40. Guest transport success is not evidence of DMA isolation suitable for production.

Repository map

Path	Contents
manifest.json	Pinned upstream bases, tested commits, driver versions and local binary hashes
patches/linux	Ten committed custom-kernel patches
patches/llama.cpp	Fifteen committed transport, profiling and sampler patches
configs	Pre-upgrade kernel/GRUB settings, package inventory and API service
scripts	Build, lifecycle, profiling and upgrade-preparation helpers
tests	CUDA, RPC, transport and P40 correctness/performance probes
lazy_cma	Experimental resource-allocation helper sources
model-api	Authenticated model-switching API code/configuration, without credentials
multikernel-engineering-audit.md	Detailed experiments, failures, correctness checks and benchmark evidence

Configuration and package inventory files describe the snapshot date; they are not automatically synchronized with the upgraded VM. Some research scripts refer to experimental branches or models outside the stable patch series.

Reconstruct the tested source

Use the exact bases in manifest.json. For example, from this repository's root:

snapshot_dir="$PWD"
git clone https://github.com/multikernel/linux.git ../multikernel-linux
cd ../multikernel-linux
git switch -c experiment/recovery 3bdd35b64413da0b4e089ce931bfc2e8b031cbf7
git am "$snapshot_dir"/patches/linux/*.patch

cd "$snapshot_dir"
git clone https://github.com/ggml-org/llama.cpp.git ../llama-multikernel
cd ../llama-multikernel
git switch -c experiment/recovery c841aeeb8bb2fe417038dadfa9b007cf1a9ef950
git am "$snapshot_dir"/patches/llama.cpp/*.patch

Kerf and daxfs are pinned directly to their recorded upstream commits. Apply the saved kernel configuration and consult the audit/build scripts for the actual build sequence. Inspect absolute paths and local assumptions before running scripts on another machine. Applying patches reconstructs source; it does not install or boot the environment. Replayed commit IDs may differ from the original IDs.

For the custom primary NVIDIA build, the tested source/output arguments were:

SYSSRC=/home/shiba/multikernel-experiment/source
SYSOUT=/home/shiba/multikernel-experiment/build

The pre-upgrade custom-kernel modules were manually installed and were not registered in DKMS for that kernel. Updating NVIDIA userspace independently can break the module/userspace match. The audit records the build details and the previously failed DKMS configuration.

Upgrade and recovery

Before the release upgrade, stock 6.8.0-142-generic was selected as the default with a 15-second GRUB menu. The custom 7.0.0-mk2-shiba-mk1 entry was retained. Stopping the secondary alone would have left the custom primary active, so it would not have provided a normal Ubuntu kernel/driver upgrade path.

A checksum-verified local recovery archive was saved under:

/home/shiba/multikernel-upgrade-recovery-20261006/

It contains boot files, custom-kernel modules and selected GRUB/apt configuration. It is intentionally absent from GitHub. Retain that archive and the VM backup: source patches and binary hashes cannot replace compiled boot artifacts.

After the upgrade, verify the installed kernel, NVIDIA package/module versions, GPU access and secondary lifecycle before restoring the model API. Restore the custom GRUB default only after its driver compatibility is established. The saved original GRUB fragment and the preparation script document the reversible change; do not run that machine-specific script blindly on another installation.

Once the desired runtime has been verified, the previously enabled API can be restored with:

systemctl --user enable --now shiba-model-api.service

Scope and licensing

Model weights, NVIDIA installers, compiled kernels/modules, secondary root filesystem, API keys and other credentials are excluded. This snapshot preserves the stable Nemotron/Multikernel source branch; it does not claim to include or validate every later Qwen/MTP research branch.

The repository-level license does not relicense third-party patches or copied sources. Their upstream notices and licenses continue to apply.