dynamic-batching

Deep Learning Deployment Framework: Supports tf/torch/trt/trtllm/vllm and other NN frameworks. Support dynamic batching, and streaming modes. It is dual-language compatible with Python and C++, offering scalability, extensibility, and high performance. It helps users quickly deploy models and provide services through HTTP/RPC interfaces.

Dynamic batching library for Deep Learning inference. Tutorials for LLM, GPT scenarios.

InsNet Runs Instance-dependent Neural Networks with Padding-free Dynamic Batching.

RWKV7 推理服务 | 高并发批量推理，动态扩缩容，兼容OpenAI API，CUDA优化，低显存部署大模型推理 RWKV linear attention batch inference server — auto-queue, dynamic capacity, OpenAI compatible.

KVTide is a Kubernetes-native LLM serving system exploring cache-aware scheduling and proactive peer-to-peer KV mobility.

A high-performance deep learning model inference server based on TensorRT, supporting fast inference for Embedding, Reranker, and NLI models.

GPU-aware LLM serving platform with multi-worker routing, dynamic batching, model placement, streaming, admission control, and reproducible performance benchmarks.

"A practical guide to NLP model optimization and serving with NVIDIA Triton." / "실습으로 익히는 고성능 NLP 모델 최적화 및 NVIDIA Triton 서버 배포."

Online dynamic batching for faster LLM/VLM fine-tuning

Distributed ML training and serving platform

PyTorch/Hugging Face batching utility that sorts variable-length text by difficulty, then dynamically increases batch size on easier samples using a pre-trained VRAM predictor to improve GPU utilization and throughput while reducing OOM risk with fallback handling.

Production-grade LLM inference API built from scratch. NestJS gateway + Python GPU workers. Scheduling, batching, KV cache, tensor parallelism, multi-modal — all against real GPUs.

LM Multi-Bin Dynamic Scheduler Simulator - Implementation combining Multi-Bin batching with SLA-constrained dynamic batching

Production-style model inference platform — versioned registry, dynamic micro-batching (measured 8x throughput), canary + shadow deployments with instant rollback, load shedding (429+Retry-After), Prometheus metrics, async load-test harness

Vram budgeting and tail-latency simulation for three models sharing one triton gpu

A model serving system built around the dynamic batcher: priorities, load shedding, and the measured saturation cliff (p50 3ms to 757ms) that batching prevents

A reference implementation of the SONIC architectural pattern for decoupled GPU inference in HEP trigger systems.

Model serving platform with FastAPI, Docker, Prometheus, Grafana, canary releases, A/B testing, drift checks and dynamic batching.

Dynamic batching inference server with FastAPI, dynamic padding, and benchmark visualizations

cost-aware-inference-cluster: FastAPI inference-serving prototype with Redis queues, worker heartbeats, dynamic batching, autoscaling simulation, and local Docker Compose benchmark evidence.