Repository navigation
feat: keep the credit hold alive while inference runs - #10
Merged
Merged
Conversation
Metering expires a credit reservation after its TTL even while the request is still running. Gateway now renews the hold for the life of the request when Metering reports a lease (lease_ms): - renewals at a third of the remaining lease, retried on transient errors no more often than once per second or a tenth of the lease; - the lease is measured on the Gateway's own clock from before the reserve request, so clock skew (an enclave) cannot shorten or extend it; - if Metering refuses a renewal (the hold is gone) or the lease runs out, inference stops with an internal error rather than a user cancellation. Metering versions without lease_ms are unaffected: nothing is renewed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Metering expires a credit reservation after its TTL even while the request is still running. When Metering reports a lease (
lease_ms, from dappnode/dappnode-nexus#143), the Gateway now renews the hold for as long as the request runs.renew, without reservation IDs.Compatibility
With a Metering that doesn't send
lease_ms, nothing is renewed and behaviour is unchanged. Long requests need dappnode/dappnode-nexus#143 deployed first.Tests
🤖 Generated with Claude Code