Skip to content

Use partial, sparse clone in the git resolver - #10665

Open
sukkola wants to merge 1 commit into
tektoncd:mainfrom
sukkola:sparse-resolver-checkout
Open

Use partial, sparse clone in the git resolver#10665
sukkola wants to merge 1 commit into
tektoncd:mainfrom
sukkola:sparse-resolver-checkout

Conversation

@sukkola

@sukkola sukkola commented Aug 27, 2026

Copy link
Copy Markdown

Changes

The git resolver clones the whole repository to read one file. For a 300MB
repo this costs 601MB of resolver disk and 46s per resolution, and the
resolver's /tmp emptyDir is capped at 4Gi, so a handful of concurrent
large-repo resolutions risk evicting the resolver pod and stalling every
build in the cluster.

Since #8677 replaced go-git with the native git binary, the resolver can use
git's own partial + sparse clone support directly (--filter=blob:none --sparse), deriving the sparse cone from the existing pathInRepo param
rather than adding a new resolver parameter. Sparse checkout alone is not
enough — the transfer, not the checkout, is the cost (measured 301MB/44s
without --filter, against 1MB/<1s with it, same commit, same repo).

Resolves #6176.

Fallback and reliability

Falls back to today's exact unfiltered clone sequence whenever the filtered
clone or the subsequent filtered fetch fails, without pattern-matching git's
error text. Verified this fallback engages for real, not only in a mock:
live against Forgejo with partial clone explicitly disabled
(DISABLE_PARTIAL_CLONE), against a raw git daemon with direct control
over uploadpack.allowFilter/allowAnySHA1InWant, and against a genuine
auth/network failure through the patched resolver in a running cluster.

A single transient failure of the filtered clone or fetch is retried once
before escalating to the more expensive unfiltered fallback, so a flaky
connection does not by itself force a full clone of what may be a very
large repository.

Open question for maintainers: the retry count (currently a fixed 1
retry) and whether it should use the resolver's existing backoff-* config
instead of an immediate retry are not resolved here — raising this
explicitly rather than assuming an answer. If configurability is wanted,
resolvers-feature-flags/git-resolver-config (used already for
enable-git-resolver, backoff-*, etc.) looks like the natural place, as
an operator-facing setting rather than a resolver param.

Scope

Applies to both anonymous and gitToken-authenticated clones — the cost is
identical and it's the same code path. Issue #6176 titles this "for
anonymous git resolutions"; restricting the fix to anonymous clones would
leave private repositories, arguably the case that motivates it most in
practice, paying full price.

Out of scope: the resolver's authenticated API mode (org/repo +
token), which already fetches only the file; the resolver cache contract
(cache keys are untouched — this changes only how a file is fetched, never
what's returned); and the stale docs line claiming clone mode "clones the
entire repository in memory" (true of the old go-git implementation, false
since #8677) — worth a separate one-line docs PR.

Submitter Checklist

  • Has Docs if any changes are user facing, including updates to minimum requirements e.g. Kubernetes version bumps
  • Has Tests included if any functionality added or changed
  • pre-commit Passed
  • Follows the commit message standard
  • Meets the Tekton contributor standards (including functionality, content, code)
  • Has a kind label. You can add one by adding a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
  • Release notes block below has been updated with any user facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings). See some examples of good release notes.
  • Release notes contains the string "action required" if the change requires additional action from users switching to the new release

Release Notes

The git resolver now uses a partial, sparse clone (`--filter=blob:none --sparse`) instead of a full clone, fetching only the directory containing the requested file. This significantly reduces resolver disk usage and resolution time for large repositories. Falls back automatically to the previous full-clone behavior if the server or a transient failure prevents the filtered clone/fetch from succeeding. No user action is required.

This contribution was substantially AI-assisted (Claude Sonnet 5), disclosed per Tekton's AI contribution policy and via the Assisted-by: commit trailer. I've read and understand the change; happy to answer questions about any part of it.

The git resolver clones the whole repository to read one file. For a
300MB repo this costs 601MB of resolver disk and 46s per resolution,
and the resolver's /tmp emptyDir is capped at 4Gi, so a handful of
concurrent large-repo resolutions risk evicting the resolver pod and
stalling every build in the cluster.

Since tektoncd#8677 replaced go-git with the native git binary, the resolver
can use git's own partial + sparse clone support directly: derive the
sparse cone from the existing pathInRepo param rather than adding a
new resolver parameter. Sparse checkout alone is not enough - the
transfer, not the checkout, is the cost (measured 301MB/44s without
--filter, against 1MB/<1s with it, same commit).

Falls back to today's exact unfiltered clone sequence whenever the
filtered clone or the subsequent filtered fetch fails, without
pattern-matching git's error text. Verified this fallback engages for
real (not only in a mock): live against Forgejo with partial clone
disabled, against a raw git-daemon with direct control over
uploadpack.allowFilter/allowAnySHA1InWant, and against a genuine
network/auth failure through the patched resolver in a running
cluster. A single transient failure retries the same filtered
operation once before escalating to the more expensive fallback, so a
flaky connection does not by itself force a full clone of what may be
a very large repository.

Resolves tektoncd#6176.

Assisted-by: Claude Sonnet 5
@tekton-robot tekton-robot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Aug 27, 2026
@linux-foundation-easycla

linux-foundation-easycla Bot commented Aug 27, 2026

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

  • ✅ login: sukkola / name: Sampo Ukkola (d41f592)

@tekton-robot

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign wlynch after the PR has been reviewed.
You can assign the PR to them by writing /assign @wlynch in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tekton-robot tekton-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Consider using sparse checkout for anonymous git resolutions

2 participants