Skip to content

Avoid resolver stalls in the git cache by pinning its identity - #734

Open
felipecrs wants to merge 1 commit into
git-pkgs:mainfrom
felipecrs:fix/reftable-ident-stall
Open

felipecrs wants to merge 1 commit into
git-pkgs:mainfrom
felipecrs:fix/reftable-ident-stall

Conversation

@felipecrs

@felipecrs felipecrs commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Configures a static Git user.name and user.email for all cache Git processes.

With reftable enabled in Git 2.56+ (#723), Git records a writer identity on every ref write and falls back to a hostname DNS lookup when none is set—stalling clones, fetches, and startup migrations on slow resolvers. Pinning a static identity eliminates this lookup safely, as the cache is a mirror and the identity never leaves the internal reftables.

Benchmarks

A fabricated repository with git 2.56.1 and a slow DNS resolver:

before after
cold clone 9.2 s 0.1 s
warm clone, nothing changed 0 s 0 s
refresh, 200 new commits 3.1 s 0.1 s
fetch of a commit by ID 3.1 s 0 s
Reproduce it

Needs Linux, for the LD_PRELOAD shim. Save dnstall.c and bench-ident.go, below, into an empty directory, and start the container there:

docker run --rm -it -v "$PWD:/kit:ro" alpine:edge sh

Then, inside it:

apk add git git-daemon build-base go
mkdir /x
git clone https://github.com/git-pkgs/proxy.git src
go -C src build -o /x/proxy-before ./cmd/proxy
git -C src fetch https://github.com/felipecrs/proxy.git fix/reftable-ident-stall
git -C src checkout FETCH_HEAD
go -C src build -o /x/proxy-after ./cmd/proxy
gcc -shared -fPIC -O2 -o /x/dnstall.so /kit/dnstall.c -ldl
go run /kit/bench-ident.go -before /x/proxy-before -after /x/proxy-after -shim /x/dnstall.so

dnstall.c:

// Adds DNSTALL_DELAY_US of latency to every getaddrinfo call whose host is
// not a numeric address: the wait of a slow resolver per name lookup.
// Build: gcc -shared -fPIC -O2 -o dnstall.so dnstall.c -ldl
#define _GNU_SOURCE
#include <arpa/inet.h>
#include <dlfcn.h>
#include <stdlib.h>
#include <netdb.h>
#include <unistd.h>

static void delay(void) {
	static long us = -1;
	if (us < 0) {
		const char *s = getenv("DNSTALL_DELAY_US");
		us = s ? atol(s) : 3000000;
	}
	usleep(us);
}

int getaddrinfo(const char *node, const char *service,
		const struct addrinfo *hints, struct addrinfo **res) {
	static int (*real)(const char *, const char *,
		const struct addrinfo *, struct addrinfo **);
	if (!real)
		real = dlsym(RTLD_NEXT, "getaddrinfo");
	struct in_addr v4;
	struct in6_addr v6;
	if (!node || !*node ||
	    inet_pton(AF_INET, node, &v4) > 0 ||
	    inet_pton(AF_INET6, node, &v6) > 0)
		return real(node, service, hints, res);
	delay();
	return real(node, service, hints, res);
}

bench-ident.go:

// Proves the reftable identity stall of the git cache in a slow-resolver
// environment, comparing proxy builds with and without the fix that pins
// the committer identity: with Git 2.56 the cache keeps its repositories
// in the reftable format, and every Git process that writes refs, so
// every fetch, resolves the committer email from the machine hostname,
// which a slow resolver turns into a stall on top of every wait a client
// would notice. The bench's own Git, the client's and the upstream
// server's stay outside the shim, and their loopback traffic is numeric
// and unaffected.
//
// The stall is simulated by an LD_PRELOAD shim delaying getaddrinfo for
// non-numeric hosts (dnstall.c), delivered only to the cache's Git
// through a wrapper first in the proxy's PATH, the one environment
// variable the cache passes on to its Git.
//
//	Run it from a checkout of the proxy:
//
//	go run /path/bench-ident.go -before /path/proxy-before -after /path/proxy-after -shim /path/dnstall.so
package main

import (
	"bufio"
	"bytes"
	"flag"
	"fmt"
	"io"
	"math/rand"
	"net"
	"net/http"
	"net/http/cgi"
	"os"
	"os/exec"
	"path/filepath"
	"strings"
	"time"
)

func main() {
	commits := flag.Int("commits", 200, "commits in each repository")
	kib := flag.Int("kib", 8, "KiB of incompressible content per commit")
	before := flag.String("before", "proxy-before", "proxy binary without the fix")
	after := flag.String("after", "proxy-after", "proxy binary with the fix")
	shim := flag.String("shim", "dnstall.so", "the compiled dnstall.c, an absolute path; loads into the cache's Git")
	keep := flag.Bool("keep", false, "print and keep the run directory")
	flag.Parse()
	shimPath, err := filepath.Abs(*shim)
	check(err)
	dir, err := os.MkdirTemp("", "bench-ident-")
	check(err)
	if !*keep {
		defer os.RemoveAll(dir)
	}
	fmt.Printf("dir: %s\n", dir)

	// The wrapper first in the proxy's PATH, and the shim it loads: the
	// cache runs its Git with a fresh environment passing only PATH, so
	// this is the shim's way in.
	wrapDir := filepath.Join(dir, "bin")
	check(os.MkdirAll(wrapDir, 0o755))
	wrapper := fmt.Sprintf("#!/bin/sh\nLD_PRELOAD=%s\nexport LD_PRELOAD\nexec /usr/bin/git \"$@\"\n", shimPath)
	check(os.WriteFile(filepath.Join(wrapDir, "git"), []byte(wrapper), 0o755))

	// The two upstreams, one per leg, so both legs see the same history.
	tips := map[string]string{}
	for _, name := range []string{"before", "after"} {
		repo := filepath.Join(dir, "up", "repo-"+name+".git")
		git("", "init", "-q", "--bare", "-b", "main", repo)
		// The leg's stray-commit fetch asks this upstream for a commit
		// no ref points to, as a pull request head is.
		git("", "-C", repo, "config", "uploadpack.allowAnySHA1InWant", "true")
		tips[name] = fastImport(repo, "", 0, *commits, *kib)
	}

	// The upstreams' http backend, as the cache's own would be.
	ln, err := net.Listen("tcp", "127.0.0.1:0")
	check(err)
	backend := &cgi.Handler{
		Path: filepath.Join(git("", "--exec-path"), "git-http-backend"),
		Env: []string{"GIT_PROJECT_ROOT=" + filepath.Join(dir, "up"), "GIT_HTTP_EXPORT_ALL=1",
			"GIT_HTTP_MAX_REQUEST_BUFFER=1073741824"},
	}
	go http.Serve(ln, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		// net/http/cgi refuses the chunked bodies git sends for large requests.
		body, _ := io.ReadAll(r.Body)
		r.Body, r.ContentLength, r.TransferEncoding = io.NopCloser(bytes.NewReader(body)), int64(len(body)), nil
		backend.ServeHTTP(w, r)
	}))

	leg := func(name, bin string) []result {
		config := filepath.Join(dir, "config-"+name+".yaml")
		check(os.WriteFile(config, []byte(fmt.Sprintf(`listen: 127.0.0.1:19080
upstream:
  allow_private_hosts: [127.0.0.1]
  allow_loopback: true
  git:
    up: http://%s
git:
  cache_dir: %s
`, ln.Addr(), filepath.Join(dir, "git-"+name))), 0o600))
		cmd := exec.Command(bin, "serve", "-config", config, "-log-level", "warn",
			"-storage-url", "file://"+filepath.Join(dir, "artifacts-"+name),
			"-database-path", filepath.Join(dir, "proxy-"+name+".db"))
		cmd.Env = proxyEnv(wrapDir)
		cmd.Stderr = os.Stderr
		check(cmd.Start())
		defer func() { _ = cmd.Process.Kill(); _ = cmd.Wait() }()
		for i := 0; ; i++ {
			if resp, err := http.Get("http://127.0.0.1:19080/health"); err == nil && resp.StatusCode == http.StatusOK {
				break
			}
			if i >= 300 {
				panic("the proxy did not start")
			}
			time.Sleep(100 * time.Millisecond)
		}

		url := "http://127.0.0.1:19080/git/up/repo-" + name + ".git"
		var results []result
		timed := func(op string, f func()) {
			start := time.Now()
			f()
			results = append(results, result{op, time.Since(start)})
		}
		clone := func() {
			dest, err := os.MkdirTemp(dir, "clone-")
			check(err)
			git("", "clone", "-q", "--single-branch", url, dest)
		}
		timed("cold clone", clone)
		// A request that writes no refs: the ref check goes over the proxy's
		// own HTTP, so both legs stay flat, and the row shows only the ref
		// writers stall.
		timed("warm clone, nothing changed", clone)

		// A client that stays cloned, to fetch from afterwards.
		client := filepath.Join(dir, "client-"+name)
		git("", "clone", "-q", "--single-branch", url, client)

		// The ls-remote of a changed upstream waits for the fetch: the
		// refresh a clone right after a push gets. Its answer is checked
		// against the new tip, so the row proves the wait was the real one.
		next := fastImport(filepath.Join(dir, "up", "repo-"+name+".git"), tips[name], *commits, *commits, *kib)
		timed(fmt.Sprintf("refresh, %d new commits", *commits), func() {
			if sha, _, _ := strings.Cut(git("", "ls-remote", url, "refs/heads/main"), "\t"); sha != next {
				panic("the refresh did not bring the new tip")
			}
		})

		// A commit on no branch, as a pull request head is, fetched by ID:
		// the actions/checkout path. The cache fetches it under a wanted
		// ref, a ref write in itself.
		stray := git("", "-C", filepath.Join(dir, "up", "repo-"+name+".git"),
			"commit-tree", "-p", "refs/heads/main", "-m", "stray", "refs/heads/main^{tree}")
		timed("fetch of a commit by ID", func() {
			git("", "-C", client, "fetch", "-q", "origin", stray)
		})
		return results
	}

	legs := [2][]result{leg("before", *before), leg("after", *after)}
	fmt.Printf("%-32s %10s %10s\n", "", "before", "the fix")
	for i, r := range legs[0] {
		fmt.Printf("%-32s %10s %10s\n", r.op, r.d.Round(100*time.Millisecond), legs[1][i].d.Round(100*time.Millisecond))
	}
}

// result is the time one measured operation took in one leg.
type result struct {
	op string
	d  time.Duration
}

// proxyEnv is the environment of the proxy: the same as the bench's own,
// except the wrapper directory comes first in PATH, so the cache's Git,
// and only it, loads the shim.
func proxyEnv(wrapDir string) []string {
	env := make([]string, 0, len(os.Environ())+1)
	for _, v := range os.Environ() {
		if !strings.HasPrefix(v, "PATH=") {
			env = append(env, v)
		}
	}
	return append(env, "PATH="+wrapDir+":"+os.Getenv("PATH"))
}

// fastImport writes count commits to refs/heads/main in repo, each replacing
// the file f with fresh incompressible content, and returns the new tip.
// from is the commit the first one builds on, "" if the branch is empty.
func fastImport(repo, from string, first, count, kib int) string {
	cmd := exec.Command("git", "-C", repo, "fast-import", "--quiet")
	cmd.Env = gitEnv()
	cmd.Stdin = importStream(from, first, count, kib)
	cmd.Stderr = os.Stderr
	check(cmd.Run())
	return git("", "-C", repo, "rev-parse", "refs/heads/main")
}

// importStream returns the fast-import stream of count commits, each
// replacing f with kib KiB of pseudorandom, incompressible content.
func importStream(from string, first, count, kib int) io.Reader {
	pr, pw := io.Pipe()
	go func() {
		defer pw.Close()
		w := bufio.NewWriterSize(pw, 1<<20)
		blob := make([]byte, kib<<10)
		for i := first; i < first+count; i++ {
			fmt.Fprintf(w, "commit refs/heads/main\n")
			fmt.Fprintf(w, "author bench <bench@example.com> %d +0000\n", 1700000000+i)
			fmt.Fprintf(w, "committer bench <bench@example.com> %d +0000\n", 1700000000+i)
			message := fmt.Sprintf("position %d\n", i)
			fmt.Fprintf(w, "data %d\n%s", len(message), message)
			if i == first && from != "" {
				fmt.Fprintf(w, "from %s\n", from)
			}
			fmt.Fprintf(w, "M 100644 inline f\ndata %d\n", len(blob))
			rng := rand.New(rand.NewSource(int64(i)))
			rng.Read(blob)
			w.Write(blob)
		}
		w.Flush()
	}()
	return pr
}

// gitEnv is the environment for the bench's own git: the machine's configs
// stay out, so no setting outside this script changes what is measured.
func gitEnv() []string {
	env := make([]string, 0, len(os.Environ()))
	for _, v := range os.Environ() {
		if !strings.HasPrefix(v, "GIT_") {
			env = append(env, v)
		}
	}
	return append(env, "GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL="+os.DevNull,
		"GIT_AUTHOR_NAME=bench", "GIT_AUTHOR_EMAIL=bench@example.com",
		"GIT_COMMITTER_NAME=bench", "GIT_COMMITTER_EMAIL=bench@example.com")
}

func git(stdin string, args ...string) string {
	cmd := exec.Command("git", args...)
	cmd.Stdin = strings.NewReader(stdin)
	cmd.Env = gitEnv()
	return run(cmd)
}

func run(cmd *exec.Cmd) string {
	cmd.Stderr = os.Stderr
	out, err := cmd.Output()
	check(err)
	return strings.TrimSpace(string(out))
}

func check(err error) {
	if err != nil {
		panic(err) // unlike log.Fatal, runs the deferred cleanup
	}
}

Summary by CodeRabbit

  • Updates
    • Git operations performed by the cache now use a consistent proxy identity. This may affect the identity recorded for Git actions initiated by the cache; it does not change the identity configured for other Git operations.

@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The Git cache configuration now sets a fixed Git user name and email for Git processes run by the cache.

Changes: Git cache identity

Layer / File(s) Summary
Set Git identity
internal/gitcache/git.go
gitConfig adds fixed values for Git’s user.name and user.email.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Change: Bug fix

Suggested reviewers: montehurd

Merge Risk: Minimal · up to f2b49

The change pins a fixed Git identity for cache Git processes to avoid slow hostname lookups. It is small and low risk, and no merge-blocking issue was found.

Architecture risk: Low · up to f2b49

The change affects 1 system.

Changed systems: internal

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

  • observed — internal (service) was modified; 1 changed file maps to changed impact.

Before / after behavior

  • observed — Modified behavior in internal/gitcache/git.go: gitConfig adds fixed Git author name and email settings for cache processes.
Pre-merge checks: Passed 4

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title names the main change: the git cache now pins a static Git identity (user.name and user.email) for its processes. The stated goal is to avoid resolver stalls, which matches the PR objectives…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main change: the git cache now pins a static Git identity (user.name and user.email) for its processes. The stated goal is to avoid resolver stalls, which matches the PR objectives. The changeset adds the identity settings in gitConfig, so the title is related to the change and specific enough for a reader scanning history.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 65e0e806-826e-46f5-83ba-03396f193aa2
Commits

Reviewing files that changed from the base of the PR and between 75b2e0c and f2b4946.

Files selected for processing: 1

  • internal/gitcache/git.go

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.

@felipecrs
felipecrs force-pushed the fix/reftable-ident-stall branch from 75b2e0c to f2b4946 Compare November 16, 2026 02:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant