Enhancement: Dynamic MIG

[FouoF]

What would you like to be added:

HAMi already supports on-demand MIG instance creation through Dynamic MIG. However, there is still room to improve NVML usage, MIG template selection, device injection, and instance lifecycle management.

This issue proposes the following improvements.

1. Optimize NVML initialization and shutdown

Centralize the NVML lifecycle for Dynamic MIG:

• Initialize NVML once during component or MIG manager startup.
• Shut down NVML once when the component or manager exits.
• Avoid repeated nvml.Init() and nvml.Shutdown() calls during individual query, allocation, creation, or release operations.
• Define clear ownership of the NVML session so that different modules do not initialize or close it independently.
• Ensure NVML resources are handled correctly during startup failures, unexpected errors, and graceful shutdown.

This should reduce unnecessary driver calls, runtime overhead, and potential concurrency issues.

2. Optimize the MIG template allocation algorithm

Improve MIG profile placement and template selection so that each allocation preserves the largest possible contiguous free region and reduces fragmentation.

For common seven-slice GPUs, such as an A100 configured as 7 × 1g, a greedy strategy can be used:

1. Process profiles that require more slices first.
2. Evaluate the remaining contiguous space after each candidate placement.
3. Prefer placements that do not split a large free region into smaller fragments.
4. Use a stable and deterministic placement order when multiple candidates have the same score.
5. Fall back to backtracking or exhaustive search when the greedy strategy cannot find a valid solution.

For example, a 1g instance should preferably be allocated from the edge of a free region so that the remaining contiguous slices can still accommodate 2g, 3g, or larger profiles.

The algorithm should cover:

• Multiple allocations of the same profile.
• Mixed small and large profiles.
• GPUs with existing active MIG instances.
• Cases where greedy selection fails but backtracking can still find a valid placement.
• Different GPU models, slice counts, and placement constraints reported by NVML.

3. Add CDI support

Add Container Device Interface support for dynamically created MIG instances.

Expected behavior:

• Generate or update a CDI device spec after creating a MIG GI/CI pair.
• Maintain a stable mapping between the CDI device name and the MIG UUID.
• Return a CDI-qualified device name during allocation so that CDI-enabled container runtimes can inject the device.
• Reuse an existing valid CDI entry when reusing an idle MIG instance.
• Remove the corresponding CDI device entry when a MIG instance is permanently destroyed.
• Update CDI files atomically to prevent kubelet or the container runtime from reading incomplete specifications.
• Preserve compatibility with the existing non-CDI injection path and provide explicit configuration or capability detection.

The implementation can refer to the Dynamic MIG implementation in the NVIDIA DRA Driver:

• https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu
• https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu/releases

In particular, we should review how it creates, caches, refreshes, and removes CDI specifications for dynamic MIG devices.

4. Optimize the MIG instance lifecycle with lazy reclamation

Currently, destroying a MIG GI/CI pair immediately after a Pod releases it may cause unnecessary overhead when a subsequent workload requests the same profile.

Introduce a lazy reclamation mechanism:

• Mark a released MIG instance as Idle instead of destroying it immediately.
• Reuse an idle instance when a new request matches its profile and placement.
• Destroy or reconfigure idle instances only when they prevent a new profile from being allocated.
• Optionally support an idle TTL, after which an unused instance is reclaimed.
• Optionally enforce per-GPU cache limits to avoid retaining too many idle instances.
• Restore the actual MIG state from NVML after a component restart and reconcile it with active Pod allocations.
• Bind the CDI specification lifecycle to the MIG instance lifecycle rather than to an individual Pod lifecycle.

Suggested instance states:

• Creating
• Active
• Idle
• Reclaiming
• Deleting
• Error

Why is this needed:

The current implementation may have the following limitations:

• Frequent NVML initialization and shutdown increases driver interaction overhead.
• Suboptimal placement selection can cause fragmentation, preventing larger profiles from being allocated even when sufficient total capacity remains.
• Dynamically created MIG instances do not yet have complete CDI integration.
• Binding the MIG instance lifecycle directly to the Pod lifecycle causes unnecessary GI/CI creation and destruction.
• The lack of instance reuse increases allocation latency for short-lived workloads, autoscaling scenarios, and workloads that repeatedly request the same profiles.

These improvements should reduce allocation latency and driver calls, improve slice utilization, and provide a more standardized device injection mechanism.

Acceptance Criteria:

• NVML is initialized and shut down only once during the Dynamic MIG manager lifecycle.
• Common seven-slice scenarios use a greedy strategy that preserves large contiguous free regions.
• The allocator falls back to a complete search when greedy placement cannot find a valid solution.
• Dynamically created MIG instances produce valid CDI device specifications.
• MIG devices can be injected using CDI-qualified device names.
• CDI creation, update, and deletion follow the MIG instance lifecycle.
• Released MIG instances can enter the Idle state and be reused by compatible requests.
• Idle instances can be reclaimed safely under placement pressure, cache limits, or TTL expiration.
• Existing MIG state can be restored and reconciled after a component restart.

Anything else we need to know?:

The implementation can be split into several independent pull requests:

1. Centralize the NVML lifecycle.
2. Optimize MIG placement and template selection.
3. Add MIG instance state management and lazy reclamation.
4. Add CDI specification generation and container injection.
5. Add metrics, logging, unit tests, and end-to-end tests.

Lazy reclamation and CDI support should share a single authoritative MIG instance state to prevent inconsistencies between scheduler reservations, the actual NVML state, and CDI specifications.

[maishivamhoo123]

@FouoF these are Great Enhancement . i love to take the second task (Optimize MIG placement and template selection).
can i take this?

[maishivamhoo123]

Tasks :-

• Centralize the NVML lifecycle.
• Optimize MIG placement and template selection.(@maishivamhoo123 ).
• Add MIG instance state management and lazy reclamation.
• Add CDI specification generation and container injection.
• Add metrics, logging, unit tests, and end-to-end tests.

[FouoF]

@maishivamhoo123 Of course you can take it. I create a sub issue #2831 we can discuss more details there.

[asadjan4611]

@FouoF can you please assign this issue #2833 to me,after the deep search on it
I would approach this issue as follows:

• Centralize NVML initialization and shutdown within a single Dynamic MIG manager.
• Implement fragmentation-aware greedy placement with backtracking as a fallback.
• Reuse idle MIG instances and reclaim them only under TTL, cache, or placement pressure.
• Atomically synchronize CDI specifications with every MIG instance lifecycle change.

i Would love to work on it