Skip to content

architecture.md

Latest commit

 

History

History
67 lines (48 loc) · 1.69 KB

architecture.md

File metadata and controls

67 lines (48 loc) · 1.69 KB
Raw

Architecture

Role In The Stack

llama_cpp_sdk is the provider-specific runtime kit for llama.cpp inside the self-hosted inference stack.

execution_plane
  -> self_hosted_inference_core
  -> llama_cpp_sdk
  -> req_llm consumers via published EndpointDescriptor values

The package owns the backend-specific service semantics that the shared kernel must not guess:

  • canonical boot-spec normalization
  • llama-server flag rendering
  • readiness probes
  • health probes
  • stop strategy for spawned services
  • backend manifest details

Boundary Rules

execution_plane

Transport remains below this package:

  • subprocess start and shutdown mechanics
  • stdout and stderr delivery
  • interrupt and force-close behavior
  • execution-surface routing

self_hosted_inference_core

The shared kernel remains above this package:

  • backend registry
  • runtime instance reuse
  • attachability and lease reuse semantics
  • lease semantics
  • compatibility calculation
  • endpoint publication contracts

llama_cpp_sdk

This package sits between the two:

  • it decides how llama-server should be launched
  • it decides when the service is truly ready
  • it decides how health should be interpreted
  • it exposes honest backend capabilities northbound

Current Topology

The first release supports one truthful runtime posture:

  • startup kind: :spawned
  • management mode: :jido_managed
  • execution surface: :local_subprocess

llama-server can be launched remotely later, but :ssh_exec is not claimed yet because remote path semantics, reachability, and shutdown guarantees still need backend-specific verification. The boot spec rejects those non-local surfaces instead of silently accepting them.