Git-Sync Worker Saturation and Ref Sync

I found why each of these syncs costs the same whether 1 ref changed or 50k did. I couldn't confirm what started the 2–4× increase at around 17:00Z.

Every sync re-checks the whole repo, by design. A sync job (pkg/gitjobs/jobs.go:105) names only the source and target, not the ref that changed. So each webhook runs a full comparison, and in git-sync that costs time proportional to the total number of refs:

1. Listing the GitHub refs. The worker asks for all refs (AllRefs: true, mirror-pipeline/cmd/worker/syncer.go:832), which makes git-sync request everything under refs/ (internal/planner/types.go:248). That includes refs/pull/*, which the worker only drops after the list arrives. On busy repos the PR refs can outnumber the branches, so GitHub may be sending well over the 20–65k refs that get compared.
2. Listing the mirror's refs. git-sync fetches the target's complete ref list (internal/syncer/syncer.go:918). The protocol version it uses for this has no way to ask for less.
3. Fetching from GitHub. The fetch sends one line for the changed ref plus a "have" line for every ref tip already on the mirror (internal/gitproto/fetch.go:251 and :272). That's tens of thousands of lines, about 3 MB of request body for 65k refs, and GitHub has to check every one before it sends back a tiny pack. When one ref is being updated, its old value on the mirror is enough.

So more of these syncs means p95 goes up, even though the total job count barely moved.

Repeat jobs. I couldn't find the "Target already being synced, deferred" log line in my local mirror-pipeline checkout or in entiredb, so the deployed code may differ. Since every sync reconciles the whole repo, a job that arrives during a running sync of the same target needs no work beyond one follow-up sync. If deferral puts each job back in the queue instead of merging them, 10 webhooks during one slow sync become 10 more full syncs later. A rise from 181 to 1,351 deferrals an hour fits that. A busy repo can then stay at full-repo cost almost continuously.

What I'd do, cheapest first:

• Now, in the worker: keep at most one pending sync per target and drop the rest. Nothing is lost, because the next sync picks up every change. This is the most likely way to bring the queue back down.
• git-sync: when only a few refs are being created or updated, send far fewer "haves": the old value of each updated ref plus the default branch tip. This cuts the 3 MB request and the matching work on GitHub's side. It's a small change in fetchPackV2 and its callers.
• git-sync and worker: avoid listing refs/pull/* from GitHub. The listing can't exclude a namespace, only include prefixes. So this means listing named namespaces instead of refs/, which conflicts with the empty-source safety check that requires the full listing. That needs a design decision.
• Later: put the changed refs in the job and let git-sync list only those (narrowing the GitHub side by exact ref name). The mirror's full ref list would still be fetched each time, but that request stays inside our own servers.

Still open: why these repos jumped 2–4× at 17:00Z. Possibilities include a merge-queue or CI bot pushing more often, the re-kicker, or retries after "raced" outcomes. Grouping the incoming webhooks for the top repos by event type and sender would show which. I can query Datadog for that, or write the fetchPackV2 change. Which would help most?

Yes, the update changes part of what I told you. The part about the repeat jobs was wrong. The part about git-sync still holds, and it now applies directly to what the worker runs.

The "deferred" jobs already collapse, so they aren't extra work. The line comes from the new per-target lease (cmd/worker/lease.go:420, design in docs/design/mirror-sync-target-lease.md). Each target has a lease, and only the worker holding it can sync that target. A job that arrives while the lease is held writes a pending note and acks. There is one note slot per target per source kind, so a newer note overwrites the older one. When the sync finishes, the worker publishes each remaining note as one new job (handOffNotes, line 633). So 10 webhooks during one sync become one follow-up sync, not 10. I'd guessed the opposite earlier, and that was wrong.

That changes how to read your numbers:

• Each repo runs at most one sync at a time. A hot repo gets one sync running and at most one waiting (per kind).
• The deferral count shows how often pushes overlap a running sync. It rose because these repos get pushed more often and each sync takes longer. It isn't creating load.
• The 2–4× is driven by push volume. Once a repo's pushes arrive faster than one sync finishes, it is synced back-to-back. Slower syncs would mean fewer syncs per hour, not more, so the extra syncs most likely come from more pushes on those repos since about 17:00Z. That's an upstream cause (a bot, a merge queue, CI), and the webhook breakdown by sender and event type is still the query that would find it.

The git-sync analysis now matches what's deployed. The worker now uses git-sync at d958639b, the current HEAD of this repo, assuming that build is what's running in us-east-2. The three full-repo costs per sync are the same as before:

• listing everything under refs/ on GitHub, including refs/pull/*;
• fetching the mirror's complete ref list;
• a fetch that sends one "have" line for every ref tip already on the mirror.

Nothing merged between the old version and the new one changes that path for a one-ref update.

What changes in the recommendations:

• Drop: "keep at most one pending sync per target." The worker already does this.
• New option, worker side: wait before publishing the follow-up. Right now a hot repo's next sync is queued the moment the previous one ends, so it is effectively always syncing. A minimum gap per target of, say, 30–60s would fold more pushes into each follow-up and cap how often a single repo can be synced. The cost is up to that much extra mirror lag on busy repos.
• Unchanged, git-sync side: cut the per-sync cost by sending only the "haves" that matter (the old value of each updated ref plus the default branch tip), and stop listing refs/pull/* where the safety rules allow. These matter more now, because back-to-back syncs mean a hot repo pays this cost continuously.

I can draft the "haves" change in git-sync, or the delay on follow-up jobs in mirror-pipeline. The delay is probably the faster relief for the queue.

No, PR refs aren't written to Entire. They're downloaded on every sync and then thrown away before anything is planned or pushed.

• The worker excludes them. refs/pull/ is the first entry in excludeRefPrefixes (mirror-pipeline/cmd/worker/syncer.go:88). The ref scope sent to git-sync uses that list for GitHub syncs (refScopeFor, syncer.go:832).
• git-sync drops them before planning. Every listed source ref goes through planner.InScope (internal/planner/planner.go:91 and :123). A ref under an excluded prefix never becomes something to push.
• Pruning ignores them too. Mirror-side refs go through InTargetScope (planner.go:327 and :369). So git-sync never treats PR refs as missing and never tries to delete them on the mirror.

They do still cost something, only on the GitHub side. Because the scope is AllRefs, git-sync asks GitHub for everything under refs/, so the GitHub ref list on every sync includes all refs/pull/*/head and refs/pull/*/merge refs. They're filtered out only after the whole list has arrived and been parsed. They don't affect the fetch's "have" lines, which come only from refs already on the mirror, and there are no PR refs there.

So on big repos with many PRs, the GitHub ref list may be much larger than the 20–65k refs that actually get compared. Comparing ref counts with the worker's "applied/skipped" numbers for the top repos would show how much larger. Listing only specific namespaces instead of refs/ would avoid downloading them. That needs a decision, because the empty-source safety check requires AllRefs.

Then the source list is about as big as the refs you actually mirror, and narrowing the GitHub listing gains almost nothing. Scrap that idea. For a 50k-ref repo where one ref changed, each sync still pays three costs that grow with the ref count:

1. Listing GitHub's refs (about 50k). This can't be avoided while the job doesn't say which ref changed.
2. Listing the mirror's refs (about 50k). The same applies. This one stays inside our own network, so it's probably the cheapest of the three.
3. The fetch's "have" lines (about 50k, roughly 3 MB). git-sync sends one for every distinct tip on the mirror, and GitHub has to check each one before it sends back a pack containing one commit. Of the three, only this one can be cut without changing the job format.

So I'd go with two changes:

• git-sync: when the plan is only a few creates or updates, send as "haves" the mirror's current value of each updated ref plus the default branch tip, not every tip. The pack stays the same size or close to it, because an updated ref's old value is the best base there is. A brand-new branch off the trunk is covered by the trunk tip. This mostly changes how fetchPackV2 gets its haves from the replicate path (internal/strategy/replicate/replicate.go:64 passes all of p.TargetRefs). One risk: a create whose history shares nothing with the trunk could get a bigger pack. Capping the reduced list and falling back to the full list above some plan size covers that.
• mirror-pipeline: add a short delay before the follow-up sync is published, so hot repos aren't synced back-to-back.

Before writing the git-sync change, it's worth confirming the fetch is where the time goes. git-sync records request and response bytes per step, and the stats: lines show them per sync when stats are on. If the worker exposes those, or has spans per step, a few slow syncs of this repo would split the time between listing and fetch. If the two listings dominate, cutting haves won't help much, and the real fix is putting the changed refs in the job.

Should I write the git-sync change now, or check where the time goes first?

It doesn't need to be told. It works out which refs changed by comparing the two lists, and that comparison is the reason each sync lists all 50k refs on both sides.

What one sync does now:

1. List GitHub's refs. This gives every ref name and its commit (newSession, internal/syncer/syncer.go:897).

2. List the mirror's refs, the same way (syncer.go:918).

3. Compare the two lists. BuildReplicationPlans goes through them ref by ref (internal/planner/planner.go:221):

   • same commit on both sides: skip;
   • on GitHub only: create;
   • on both, different commit: update;
   • on the mirror only: delete (pruning).

   With 50k refs and one push, you get 49,999 skips and one update. Only the update goes on to the fetch (classifyPlans).

4. Fetch from GitHub. git-sync asks for the commits it's missing ("want") and lists what the mirror already has ("have"), so GitHub sends only the difference. This is the wasteful step: the "have" list is every ref tip on the mirror (replicate.go:64 passes p.TargetRefs, and fetch.go:251 sends all of them).

5. Push the pack and the ref update to the mirror.

The change is only in step 4. By then the plan already says which refs are being created or updated, and the "have" list can be built from that:

• Updated ref: send the mirror's current commit for that ref (plan.TargetHash). If main moved from A to B, saying "I have A" is enough for GitHub to send A..B. The other 49,999 tips add nothing, because GitHub only needs a commit on the path back from B to stop at.
• New branch: there's no old value, so send the mirror's default-branch tip (and maybe a few other likely bases). Most new branches start from the trunk.
• Fallback: for a large plan, such as many creates or a first sync, keep sending all tips. It's the safe choice, and the request size matters less there.

Roughly, for an update of main on a 50k-ref repo:

The trade-off is that GitHub can only leave out what we tell it the mirror has. If a new branch shares history with some other branch but not the trunk, the smaller list misses that, and GitHub sends a larger pack than needed. The result is still correct, because the pack is complete. That's why the fallback threshold and a few extra bases matter, and why updates, which have an exact old value, are the safe case.

What it doesn't fix: steps 1–3 still list and compare 50k refs on both sides. To skip those, the job itself would have to carry the changed refs from the webhook, so git-sync could ask GitHub for just those names. That's a bigger change across fanout, the job format and git-sync, and the mirror side would still list everything.

Yes. The "have" list is deduplicated by commit hash. refValues collects the mirror's ref values and skips zero hashes (internal/gitproto/fetch.go:620). SortedUniqueHashes then removes duplicates and sorts them (fetch.go:631), and only that list is sent (fetch.go:251).

So the request holds one line per distinct commit or tag object on the mirror, not one per ref. How much that saves depends on the repo:

• Many branches at the same commit (stale branches never moved off the trunk, release branches cut and left alone) collapse to one line each. Deduplication helps a lot here.
• Tags are almost always distinct, and annotated tags are sent as the tag object's own hash. A repo whose 50k refs are mostly tags still sends close to 50k lines.
• Active feature branches each have their own tip, so there's nothing to collapse.

All of the lines go in one request ending with "done", so there's no back-and-forth negotiation. The cost is one large request body, plus GitHub looking up each hash before it builds the pack.

For the 50k-ref repo you checked, the number of distinct hashes would tell you how much the "haves" change can save. git ls-remote <url> | cut -f1 | sort -u | wc -l against the mirror gives it (run it against GitHub, minus PR refs, if that's easier). If it's close to 50k, the change saves the full 3 MB per sync. If it's a few thousand, deduplication already takes care of most of it, and the time is going into the two ref listings.

Yes. Every line of both ref lists is a commit hash plus a ref name, so the tips come back with the lists:

• From GitHub: git-sync's ref-listing request (protocol v2 ls-refs) asks for peel and symrefs (internal/gitproto/refs.go:238). Each ref arrives with its hash. Annotated tags also carry the commit they point to, and HEAD comes with the branch it points at.
• From the mirror: the older-protocol advertisement lists one <hash> <refname> line per ref, which git-sync reads in AdvRefsToSlice (refs.go:181). For an annotated tag, the hash is the tag object, not the commit.

Those hashes are everything git-sync uses:

• Comparing: the planner matches GitHub's hash and the mirror's hash for each ref name to decide skip, create, update or delete.
• "Haves": the fetch sends the mirror's hashes, deduplicated, back to GitHub.

So nothing extra needs to be fetched for the smaller "have" list. When main goes from A to B, both A (from the mirror list) and B (from the GitHub list) are already known before the fetch. The change only decides which of the hashes git-sync already holds go into the request: A and the trunk tip, rather than all ~50k.

What the lists don't contain is the commits between the tips, meaning the history. git-sync never stores that for a replicate sync: it streams the pack from GitHub straight to the mirror. That's why it can't work out the smallest set of "haves" on its own and has to choose by rule (old value of each updated ref, plus the trunk tip for new branches).

Mostly yes, and that's close to what I'm proposing. There's one exception that matters a lot.

If you drop every ref whose hash is the same on both sides (the skips), what remains on the mirror side is:

• the old value of each updated ref. That's the useful part: it's the commit the new tip builds on.
• the value of each ref about to be deleted. It's harmless, and could be dropped too.

For the common case, a push that moves main from A to B, that gives want B, have A, which is all GitHub needs.

The catch is that the unchanged refs are exactly the shared history. A "have" tells GitHub where it can stop walking back. If all of them are gone, GitHub has nowhere to stop, and in these cases it would send the whole history:

• A new branch. It has no old value, so after dropping the unchanged refs, the "have" list for it is empty. The branch was almost certainly cut from main, which didn't change, so main's tip was dropped too. GitHub would send every commit reachable from the new branch: in effect a full clone, possibly gigabytes, close to the 10 GiB push ceiling.
• A force push or rebase. feature is rebased from A onto the current main and becomes B. A is no longer in B's history, so have A doesn't help. The real base, main's tip, was dropped as unchanged, and the result is the same as above.
• A merge. main merges release. The old value of main covers one side. The release side is cut short only if release's tip is in the list, and it was dropped as unchanged.

So the rule I'd use is "drop the equal lines, but keep a few of them":

• the mirror's old value of each updated ref;
• the default branch's tip, even when it didn't change, as the base for new branches and rebases;
• optionally, a small number of other tips (for example the most recently changed branches, or release branches) to cover merges;
• all tips, as today, when the plan is large.

The bad case can't break anything, only make one sync expensive. A "have" list that's too short makes the pack larger but never incorrect. It's still the reason the default-branch tip has to stay in the list even though its line is "equal".

Yes, with one refinement: it isn't all-or-nothing. Leaving out a base costs only the commits between the new tip and the nearest base still in the list. So:

• A new branch cut from another feature branch is fine. That branch's history is mostly on main, so the cost is the few extra commits of the parent branch, usually small.
• A long-lived branch that split from main long ago is the expensive case. Think of release/1.x maintained for years, or a fork-style branch. Pushing a new tag or branch from it would send everything since the split, which can be large.
• A branch with no shared history at all, like gh-pages or other orphan branches, means the full history of that branch.

There is a way to cover those cases without guessing. The protocol has negotiation rounds, and git-sync skips them today by always sending "done". Without "done":

1. Round 1: send the small list (old values of updated refs plus the default-branch tip) with no "done". GitHub replies with which of those commits it shares, and says "ready" once every wanted commit reaches one of them.
2. If "ready": GitHub sends the pack. This is the common case, and it's one request, as today, with a tiny body.
3. If not: send the remaining tips in the next round, or all of them as today, then "done". That costs one extra round trip, only in the diverged-branch case.

So GitHub's own reply decides whether the small list was enough, and the bad case falls back to today's behaviour instead of producing a huge pack. The extra work in git-sync is handling multiple rounds. Over HTTP each round is a new request that has to repeat the shared commits found so far, so it's a bit more code than just shortening the list.

I haven't checked one detail: exactly when GitHub declares "ready". Stock git checks that each wanted commit can reach a commit we listed. I'm not certain whether GitHub's server does the same, so it's worth testing against GitHub with a diverged branch before relying on it.

I've run the first test, sending raw fetch requests to GitHub against entireio/cli, and the results change the design. The script is at /private/tmp/claude-501/-Users-soph--herdr-worktrees-git-sync-worktree-brave-forest-43d8/7915da1e-91f7-48c7-86de-a78b6561fbd0/scratchpad/negotiate.py.

Pack sizes, short "have" list versus every mirror tip:

(The plain-update numbers came out unfairly: other refs already contained main's new tip, so the full list produced an empty pack. Those numbers aren't comparable.)

Time spent on the "haves":

The 50k list was padded with random hashes that GitHub won't find, so real tips (found and walked) may cost more. These were timed from my machine, so the upload share will be smaller from us-east-2. That's roughly 1.5 s or more per sync on a 50k-ref repo, a real part of the 16 s p95 but not all of it. The two ref listings are the rest.

What this changes:

1. The "ready" fallback doesn't work. GitHub says "ready" only when the "have" is in the new tip's history. It didn't say it for the new branch, even though the short list gave the same pack as the full one. It did say it in the merge case, where the short list was 12× worse. So it rejects good cases and accepts bad ones. I'd drop negotiation rounds and keep the current single request with "done".
2. The real risk is commits already on the mirror under another ref, not orphan branches. Orphans are expensive either way. The common real form is merging a branch the mirror already has: GitHub resends that branch's commits. That's correct, just bigger, and capped at the size of the merged branch.
3. Cheap improvements: also send the tips of refs this sync deletes, since a merged branch is often deleted in the same push window. Keep the full list for large plans.

So the trade is about 1.5–2 s off every sync on 50k-ref repos, in exchange for occasionally larger packs when a merged branch was already mirrored. On the merge-heavy repos driving the queue, those packs could add up.

For building it, I'd turn these cases into a git-sync integration test using the git-http-backend fixtures the repo already has (internal/syncer/git_http_backend_test.go). The test would assert how many haves are sent and that the mirror ends up with exactly the same refs, for update, new branch, merge of a mirrored branch, and orphan.

Do you want the shorter list with that trade-off? The alternative is to first measure where the time goes in a real slow sync, in case the ref listings dominate.

The slowdown isn't caused by ref counts. GitHub's git endpoints got slow for every repo, but only for requests from us-east-2, and they're still slow now (median about 5.3 s at 19:40).

GitHub requests, last hour:

Timeline, us-east-2 GET info/refs median:

The upload-pack POSTs follow the same curve, going from a 0.25 s median to 5.4 s.

Why it isn't ref size: in protocol v2, GET info/refs returns only GitHub's capability list, the same few hundred bytes for any repo, and it went from 0.1 s to 5 s. Small repos are as slow as big ones. The slowness is the same on all 25 pods, so it isn't one bad node. The mirror-side requests stayed fast (p95 0.08 s on the entry host).

How it explains the incident: each sync makes about 2.8 GitHub requests (one GET, then ref listing and usually a fetch as POSTs). At about 5 s extra each, that adds roughly 14 s per sync, which matches the p95 going from 5 s to 16 s. Longer syncs on hot repos overlap more pushes, so deferrals went up. Throughput per pod dropped and the queue grew. Big repos show up as "most total sync work" because they get the most pushes, not because their ref counts changed.

What's causing it is still open. Only github.com, only from us-east-2, and api.github.com from the same region is fine. That points to either GitHub throttling our egress IPs (more likely if us-east-2 traffic from a few NAT IPs is heavy, and the 2–4× sync rate may have triggered it), or something on our egress path. The durations spread from 1 to 10 s rather than bunching at 5 s, so a DNS timeout is unlikely. Things worth checking:

• NAT gateway metrics for us-east-2 (ErrorPortAllocation, active connections, how many elastic IPs).
• Whether the time goes to connecting or waiting for GitHub's reply. Adding otelhttptrace to the worker's GitHub transport would split each span into DNS, connect, TLS and time to first byte, and would settle this next time.

On git-sync: before 17:00, a GitHub POST was 0.25 s at p50 and 3–6 s at p95. The "haves" change only affects part of that p95 on large repos, so it's a modest improvement and wouldn't have prevented this incident. I'd hold it until the GitHub latency is explained. One related change would help in exactly this situation: fewer GitHub round trips per sync. Each request now costs about 5 s, so going from 3 requests to 2 saves more than a smaller "have" list would.

These numbers come from sampled traces (about 1 in 3 syncs), so they're good for latencies and proportions, not totals.

Yes, I'd leave the "haves" change out. When GitHub is healthy, a GitHub POST takes 0.25 s at p50 and 3–6 s at p95. The big "have" list only costs on the few repos with tens of thousands of distinct tips: about 1.5–2 s in my test against GitHub, and that test used random hashes, not real tips. The downside isn't limited to orphan branches either. Any merge of a branch the mirror already has would make GitHub resend that branch's commits, so the more common case would get worse in exchange for a small gain on a few repos.

If we revisit it, two things would be needed first:

• Measurements on the big repos: traces split into DNS, connect, TLS and time to first byte, to show how much of their POST time is really the "have" list.
• A narrower rule: keep the full list and only drop tips when a repo has more than, say, 10k distinct tips. That limits the risk to the repos where the saving is real.

The useful follow-up from today is on the worker side, not in git-sync:

1. Find out why github.com is slow only from us-east-2: NAT gateway metrics, egress IPs, and whether GitHub is throttling those IPs.
2. Add otelhttptrace to the worker's GitHub transport, so the next occurrence shows where the time goes.

The test script is at /private/tmp/claude-501/-Users-soph--herdr-worktrees-git-sync-worktree-brave-forest-43d8/7915da1e-91f7-48c7-86de-a78b6561fbd0/scratchpad/negotiate.py, in case you want it later. Do you want me to shelve the "haves" idea and the egress investigation as priorities?

No. ls-refs has no option like that. The protocol v2 spec defines four arguments:

• symrefs: include where symbolic refs like HEAD point.
• peel: include the commit an annotated tag points to.
• ref-prefix <prefix>: only return refs under this name prefix (can be repeated).
• unborn: report HEAD even if it points at a branch with no commits yet.

All of them filter by name or add detail to a ref. None looks at history. GitHub's capability list for the cli repo confirms it supports nothing beyond the standard set:

ls-refs=unborn means only the optional unborn argument is offered. git-sync already sends it (internal/gitproto/refs.go:245) along with peel, symrefs and ref-prefix.

Returning only the tips that aren't in another ref's history would mean walking commit history on every listing, which is why ref listing never does it. Even if GitHub could, it would describe GitHub's history, and the "have" list has to describe the mirror's.

The mirror is the one place this could work, because entiredb is our server. A small endpoint (or a receive-pack extension) could return the mirror's tips minus any tip already contained in another tip's history. That's exactly the smallest correct "have" list, with no guessing and no merge-case penalty. entiredb would have to walk history to produce it, though, and the gain would be the same 1–2 s on a few repos. I'd only consider it if the trace measurements show the "have" list actually matters for the big repos.

For cli the result is what you predicted for tags, and less so for branches:

Tags collapse to one. Branches barely shrink: most are unmerged feature branches, so each is its own tip. So how much this helps depends on the repo. A repo whose 50k refs are mostly tags would drop to a few hundred "haves". A repo with 50k live branches wouldn't shrink much.

Can we find the maximal tips by negotiating with the mirror? Not cheaply:

• Acknowledgments don't help. When the mirror acknowledges a "have", it only means "I have this object". It says nothing about whether one tip contains another, so one round can't tell us which tips are maximal.
• Probing one tip at a time works: "want T, have all the others", and an empty pack means T is contained. But that's one request per tip, tens of thousands for a big repo.
• Fetching only commits from the mirror (filter tree:0, which git-sync already uses in FetchCommitParents) and computing locally gives the full answer. It downloads the whole commit history on every sync, though, which costs more than the "have" list it saves.

What would work is computing it inside entiredb. The mirror already has the history locally. With commit-graph generation numbers, the equivalent of merge-base --independent is fast, and it updates cheaply on each push. When a ref moves to X, drop the maximal tips that X contains, and add X unless an existing tip contains it. entiredb would keep that set per repo and expose it, for example through a custom capability on the ref advertisement or a small endpoint. git-sync would then send that set as its "haves". The pack would be identical to today's, with no guessing and no merge-case penalty.

Whether it's worth building depends on what the big repos are made of. For the 50k-ref repo you looked at, this would give the split:

Mostly refs/tags would make it a big win, since those collapse to a handful. Mostly refs/heads would make it a small one, and the 1–2 s from yesterday's estimate wouldn't move much.

Confirmed in cli: the checkpoint refs share no history with each other. For example:

• v2/full/0000000000032: 6 commits, 1 root commit.
• v2/full/0000000000033: 5 commits, 1 root commit.
• v2/full/current: 198 commits, 1 root commit.

git merge-base finds no common commit between any pair I tried. There are also 37 more refs/entire/checkpoints/<XX> refs besides the 21 under v2/full. Two things follow:

1. They can't be shrunk away. None of them contains another, so every one is a maximal tip. A maximal-tips list keeps all of them, which is why all 59 survived in cli. What can't be removed scales with the number of checkpoint refs. Each archived generation adds one, so that number keeps growing for repos that use Entire heavily.

2. They still matter as "haves", even with no shared history. GitHub leaves out every object reachable from a "have", including file contents and directory listings, not only commits. Generations 0032 and 0033 have no common commit but share 128 objects. That's why, in yesterday's test, the full list saved 4 MB (56 MB versus 60 MB) on a new checkpoint ref compared with sending only main's tip. So sending only tips from the same history would make every new checkpoint generation more expensive.

For a "have" list this splits into two parts:

• Code history (branches and tags): maximal tips, computed in entiredb, would shrink it a lot on tag-heavy repos.
• Checkpoint refs: every one stays, unless we rely on our own layout. The format is ours, so a namespace rule is defensible. For a new or updated ref under refs/entire/checkpoints/, send v2/full/current and the most recent archived generation or two as "haves", not all of them. That's probably where nearly all the shared content is. It would need measuring, by comparing pack sizes for a new generation using all checkpoint tips versus only current plus the previous generation.

Both parts make the change bigger, and the gain on a healthy day is still small. I'd note this in the priority entry and keep the decision where we left it: measure on the big repos first. If those repos turn out to carry thousands of checkpoint refs, the namespace rule is the cheap first step, because it needs nothing from entiredb.

Yes, that one is safe, with one exception. Here's why, case by case:

• A code branch or tag changes. Its history shares nothing with the checkpoint histories: different commits, and in practice different files. Unchanged checkpoint tips can't make the pack smaller, so dropping them costs nothing.
• An existing checkpoint ref is updated (for example v2/full/current moves). The "have" that matters is that ref's own old value, which stays because it's a changed ref, not an equal one. New checkpoint commits add new session data that older generations won't contain, so the other checkpoint tips add next to nothing.
• A new checkpoint ref is created (a new archived generation). This is the exception. It has no old value, and its content overlaps with current and earlier generations. That overlap was the 4 MB in yesterday's test. For this case, keep the checkpoint tips, or at least current and the latest generations.

So the rule: unchanged checkpoint tips are left out of the "have" list unless the sync creates a checkpoint ref. Everything else stays as it is today, including every branch and tag tip. That keeps the full list exactly where it matters (branches and tags, which share history) and drops it only where there's nothing to share.

How I'd build it:

• Generic in git-sync. Neither git-sync nor the worker knows about refs/entire/ today, and git-sync is a general library. I'd add a request option, something like IndependentRefPrefixes: "refs under these prefixes have histories independent of everything else". The "have" filter lives where the list is built (fetchPackV2 and its callers in the replicate path). The worker passes refs/entire/checkpoints/.
• Tests, on the git-http-backend fixtures: a branch update with many unchanged checkpoint refs should send no checkpoint "haves" and give the same pack size as today; a new checkpoint generation should still send them; and the mirror's refs must match exactly in every case.

How much it saves is the number of checkpoint refs per repo: 59 of 801 tips in cli, so not much there. It only pays off if the big mirrors carry hundreds or thousands of checkpoint refs. A count from one of them would settle it:

The risk is low and the change is small, so I'm happy to build it either way. Should I go ahead, or do you want to check that count first?

Without v2/, the checkpoint refs are fully independent, so the rule gets simpler:

• Each ref is one checkpoint: 37 refs, 1–4 commits each, each with its own root commit.
• None contains another: all 37 are maximal tips.
• Almost nothing is shared: of 411 objects across all of them, 17 appear under more than one ref (probably small identical files).

So the exception I proposed for new checkpoint refs isn't needed. A new checkpoint shares at most a handful of tiny objects with the existing ones, so leaving their tips out of the "have" list can't make its pack noticeably bigger.

The rule becomes: for refs under the checkpoint prefix, send only the old value of a checkpoint ref being updated, and drop every unchanged one. Branch and tag tips stay exactly as today. In the cases that matter:

How much it saves is the number of checkpoint refs on a repo. That's only 37 on cli, but it grows by one per checkpoint, so for active Entire users it's the namespace that grows the "have" list fastest. Implementation as before: a generic option in git-sync (named for refs with independent histories, not for refs/entire/), which the worker sets to refs/entire/checkpoints/, with tests on the git-http-backend fixtures.

On deleting v2/: there are 22 refs under refs/entire/checkpoints/v2/ in entireio/cli on GitHub. Deleting them is a push to a shared repo and can't be undone, so I won't do it without your go-ahead. If they're deleted on GitHub, the mirror drops them on its next sync, since pruning is on. Should I delete them, or will you or the cli owners handle it?

Should I go ahead with the git-sync change?

entireio/cli-checkpoints is where they live, and it's the strongest case for the change.

On GitHub:

• 4,048 refs under refs/entire/checkpoints/: 4,026 checkpoints plus 22 under v2/. Apart from those there's only refs/heads/entire and HEAD.
• 4,049 distinct tips, so each sync of this repo sends about 4,050 "haves".
• The ref list alone is about 384 KB.

Sample of 21 checkpoint refs, every 200th:

• 1–3 commits each, each with its own root commit.
• All 21 are maximal tips.
• 0 shared objects across 352 objects (about 27 MB in total).

So for this repo, the checkpoint rule cuts the "have" list from about 4,050 to the old value of whatever ref changed, usually none, because a new checkpoint is a new ref. Leaving those tips out can't make the pack bigger, since nothing is shared. Today's cost is moderate, about 200 KB of request body and probably a few hundred milliseconds at GitHub, going by yesterday's 50k-hash test. But it grows by one "have" per checkpoint, forever, and every customer repo that stores checkpoints the same way grows with it.

Two limits on what it fixes:

• The ref listings still grow. git-sync still lists all 4,000+ refs on both sides every sync, and that also grows with each checkpoint. Fixing that would need the changed refs in the job, or a different ref layout for checkpoints.
• Pruning stays as it is. The full ref lists are still compared, so deleted checkpoints are still pruned from the mirror. Only the fetch's "have" list changes.

The change itself is what I described before:

• git-sync: a generic request option for ref prefixes whose histories are independent. Under those prefixes, unchanged tips are left out of the "have" list and an updated ref keeps its own old value.
• Worker: sets the option to refs/entire/checkpoints/.
• Tests, on the git-http-backend fixtures, covering:
  • a branch update with many unchanged checkpoint refs (no checkpoint "haves", same pack);
  • a new checkpoint ref (no checkpoint "haves");
  • an update to an existing checkpoint ref (only its old value);
  • the mirror's refs matching exactly in every case.

Should I build it? The git-sync part goes first, and then the worker change in mirror-pipeline once a git-sync version with it is available.

I've built the change in git-sync. It's not committed yet. The race-detector suite, the git-http-backend tests and the linter all pass.

What it does: there's a new request option, RefScope.IndependentHistoryRefPrefixes. Under those prefixes, a fetch from GitHub no longer offers the mirror's unchanged tips as "haves". A ref the run updates is still offered, as its own old value. Which refs get synced or pruned doesn't change, and with the option empty, behaviour is exactly as before.

Where it applies: the replicate path, which is what the worker uses, and sync mode's incremental relay. Two other fetches still send every tip:

• Sync mode's earlier fetch: before planning, sync mode fetches history to check fast-forwards. No plan exists yet at that point, and that check is the wrong place to experiment.
• Bootstrap: first syncs and batched retries.

Code changes:

• internal/planner/haves.go: FetchHaves decides which mirror tips are offered.
• internal/syncer/syncer.go: executeReplicate and executeIncremental use it, and the session validates the prefixes.
• types.go, client.go, unstable/client.go: the new field is documented, validated (it must start with refs/) and passed through every config builder, as the existing guard test requires.
• CHANGELOG.md: an entry under Unreleased.

Tests:

• Unit tests for the "have" rule and for validation.
• Integration tests (internal/syncer/independent_history_test.go), each on a source with a branch and three checkpoint refs. Every test records the exact "have" lines each fetch sent and checks that the mirror's refs match the source exactly:

Still to do:

1. Commit this and open a git-sync PR. Say if you want me to.
2. Once a git-sync version with this is available, the worker change in mirror-pipeline: bump the pin and set IndependentHistoryRefPrefixes: []string{"refs/entire/checkpoints/"} in refScopeFor.
3. Deleting the 22 v2/ refs in entireio/cli and entireio/cli-checkpoints, if you want that done. I haven't touched them.

SSH push was refused here, the same publickey error as earlier with cli-checkpoints. I'll push over HTTPS using the gh login for this one command, without changing your git config:

SSH still fails from my shell (Permission denied (publickey)). It probably can't reach your SSH agent. Could you run the push yourself in this session?

Once the branch is on GitHub, I'll open the PR with gh. This is the description I'll use:

Title: relay: leave unchanged independent-history tips out of the haves

A relay fetch offers the source every target tip as a have. For refs that each carry their own history, such as one orphan history per session checkpoint, those tips cannot shorten the pack the run needs. A repository with about 4,000 checkpoint refs sends about 4,000 have lines on a run that moved one branch, and the count grows by one per checkpoint.

This adds RefScope.IndependentHistoryRefPrefixes. Under those prefixes, a target ref is offered as a have only when the run updates it, as its own old value. Every other target tip is offered as before. Which refs are synced or pruned does not change, and with the option empty the behaviour is identical to today.

Where it applies:

• Replicate relays and sync's incremental relay.
• Not sync's closure fetch for fast-forward checks, which runs before any plan exists, and not bootstrap. Both still offer every tip.

Why it's safe: in a sample of 21 checkpoint refs from a real checkpoint repository, each had 1–3 commits and its own root, and none shared an object with another. So leaving their unchanged tips out cannot make a pack larger by more than a few small identical objects.

Tests:

• Unit tests for planner.FetchHaves and for prefix validation.
• In-process integration tests that record the have lines of every source fetch and check that the target's refs match the source exactly. They cover a branch update (with the option and as a baseline without it), a new checkpoint ref, an update to an existing checkpoint ref, sync mode's two fetches, and a prune-only run (no fetch).
• go test -race ./..., the git-http-backend suite and mise run lint pass.

🤖 Generated with Claude Code

I opened the trail: https://entire.com/gh/entireio/-/trails/2748

• Branch: soph/independent-history-haves is linked as change git-sync/8 (https://entire.com/gh/entireio/-/trails/2748/changes/git-sync/8), at commit 2aa64842.
• Description: the PR text from before, plus a line noting the mirror-pipeline follow-up.
• Merge gates: it isn't mergeable yet. It needs a reviewer's approval, and the checks and findings gates haven't run yet. It's up to date with main.

I didn't open a separate GitHub PR. Say if you want one as well.

Once this is merged and released, the worker change is next: bump the git-sync pin in mirror-pipeline and set IndependentHistoryRefPrefixes: []string{"refs/entire/checkpoints/"} in refScopeFor.

I addressed all four review points, pushed them as commit 601350fa, and updated the trail description to match: https://entire.com/gh/entireio/-/trails/2748

1. Tests where the named refs share history. Two new integration tests configure the prefix over refs that are not independent:

   • a new ref under the prefix built on an existing one;
   • a branch moved onto a commit the mirror holds only under the prefix.

   Both leave a complete copy; the source just resends some objects. The shared check (assertSameRefs) now reads every commit, tree and blob reachable from each mirror ref, not only the tips, so the six earlier tests check the full history too. I didn't confirm that the deeper check actually fails on a broken copy, by removing an object on purpose.

2. How prefixes match (types.go): the doc now says each entry is a literal string prefix of the mirror's ref names (after Mappings), with no globs. It also says to end a namespace with a slash, for example refs/entire/checkpoints/, because otherwise it also matches refs/entire/checkpoints-old.

3. "Never wrong, only costly" is qualified. Naming refs that share history keeps the copy correct, but the larger pack can exceed a pack size limit (unstable's MaxPackBytes) or a server's request limit. So a transfer that worked without the option can fail with it. The CHANGELOG entry says the same.

4. Validator doc (validation.go): it now says what the check enforces (each entry, once trimmed, starts with refs/) and what it doesn't. Something like refs/foo* passes and only matches names that literally contain *.

Pushing over SSH worked from my shell this time.

The trail still needs a reviewer's approval. Its findings gate shows an error because reviews don't start automatically on this repository, so someone has to start one by hand.

I updated PR #138 (https://github.com/entireio/git-sync/pull/138):

• Title: it was the branch name, so I changed it to "relay: leave unchanged independent-history tips out of the haves".
• Description: I replaced Entire's placeholder text with the trail description. That covers what the option does, where it applies, the checkpoint sample behind it, the tests, and a link back to the trail. Because this repo is public, I left out the line about the private mirror-pipeline follow-up.
• Kept as is: Entire's hidden marker, which it uses to close the PR when the trail closes, and Cursor Bugbot's summary block. Bugbot's summary still describes the first commit and should refresh on its own.

I re-ran the git-http-backend tests on the current head first, and they pass, as the description says.

The PR is still a draft. It's ready for review whenever you want to mark it.

I fixed the test weakness and pushed it as commit e7f85951 to PR #138 (https://github.com/entireio/git-sync/pull/138). The reviewer's diagnosis was right: in the pinned go-git (v6.0.0-alpha.5), TreeWalker.Next turns a failed subtree lookup into io.EOF (plumbing/object/tree.go:704-706). So the old check stopped early on a missing subdirectory and reported success.

• Explicit walk: assertClosureReadable now calls a new readClosure. It follows parent commits and tree entries itself and reads every commit, tree and blob, failing as soon as one is missing. An unexpected file mode is reported as an error instead of being skipped.
• Nested fixture: checkpoint commits now have a 0/ directory next to metadata.json, like real checkpoints. Every integration test therefore checks a subdirectory.
• Missing objects are caught: the new TestReadClosureDetectsMissingObjects removes a subtree, a blob inside the subdirectory, and a parent commit, one at a time. Each time, readClosure has to name the missing object, and it does. In the subtree case, go-git's own iterator still reported success when the subtree was gone.

The race-detector run of the syncer tests, the git-http-backend tests and the linter all pass. The full race-detector suite last ran on the commit before this one; the only file changed since is this test file. The PR description still holds, so I didn't change it.

The merge is clean, and everything passes on the merged branch: the race-detector suite, the git-http-backend tests and the linter (0 issues).

Merged to main in #139, as you asked me to check:

• Ref names starting with a dash are now mirrored, and the mirror accepts them. entiredb validates pushed names with go-git's Validate and already pins a go-git from after beta.1. I tested refs/heads/-foo: alpha.5 rejects it, while beta.1 and entiredb's version both accept it. That matters because entiredb refuses a whole push with HTTP 400 if any name is invalid. This assumes production runs the go-git in my local entiredb checkout, which I haven't verified.
• No subprocess can mistake one for an option. The PR's claim holds: git-sync's subprocesses get URLs, or names starting with refs/ or converted/.
• URLs: HTTPS and ssh:// parse the same as before. Two scp-style shapes now parse as git does. git@host:22:path used to read 22 as a port; now 22:path is the path. Bracketed IPv6 hosts used to parse as garbage and now parse correctly.
• Packs: applying a delta now checks it first, so a malformed pack fails cleanly. Each delta is read fully into memory, which matters little for git-sync.
• Unchanged: go-git's tree walker still turns a missing subtree into a normal "done", so #138's explicit closure walk is still needed.

On #138's branch, I merged origin/main (commit d51b8803) rather than rebasing, so it can be pushed without a force-push. The only conflict was CHANGELOG.md: both PRs added an entry under Unreleased, and I kept both. Main also brought #137 (settle a fast-forward against the source when the local planning store can't). It touches the syncer next to #138's call sites but doesn't change any "have" behaviour #138 tests.

The merge commit is only local so far. Should I push it to soph/independent-history-haves, which updates PR #138 and the trail?

There's no overlap between #138 and #123 beyond CHANGELOG.md, and that conflict isn't caused by #138.

Textually:

• internal/syncer/syncer.go merges cleanly. #123 reorders newSession to list the target before the source. #138's prefix validation sits above both listings and stays there (syncer.go:887, before the target listing at :966 and the source listing at :1012).
• The only conflict is in CHANGELOG.md. #123 hits the same conflict merged onto plain main, because it's 55 commits behind and every recent PR has added an Unreleased entry. Whoever lands second just keeps both entries.

Behaviour: the two touch different stages of a sync.

• #138 changes only which mirror tips are sent as "haves" on the relay fetch. That set is computed from the mirror's refs and the plan before anything is pushed (FetchHaves at syncer.go:1671 and :1711 after the merge).
• #123 changes the order of the two ref listings, and what happens after a push is rejected (it re-reads the mirror, and if the ref already holds the wanted commit, counts it as skipped).
• Neither depends on the other's result. Listing the mirror first doesn't change the snapshot the "haves" are built from. #123's re-read happens after the fetch and push, and doesn't use the "have" list.

Tested: on #138 combined with #123, the full race-detector suite passes, as do all 12 of #138's test results (counting subtests). I've removed the scratch checkout.

The merge of main into #138 (d51b8803) is still only local. Should I push it to update PR #138?

I rebased the branch onto the latest main (a870385c, which also includes #131 and the change removing file-based runners). The three commits now sit on top of main without the merge commit.

The only conflict was in CHANGELOG.md, in the Unreleased section. I kept both sides: our ### Added entry comes first, then the ### Changed entries from #139, then the existing ### Fixed entries. Nothing else conflicted.

After the rebase, the full test suite passes with the race detector on, and mise run lint reports 0 issues.

The rebase rewrote the commits, so updating PR #138 needs a force push. Should I run git push --force-with-lease to soph/independent-history-haves?

I haven't filed anything yet. Here's the draft. The body is in /private/tmp/claude-501/-Users-soph--herdr-worktrees-git-sync-worktree-brave-forest-43d8/7915da1e-91f7-48c7-86de-a78b6561fbd0/scratchpad/gogit-issue.md.

I ran a small repro against v6.0.0-beta.1 before writing it, and the bug is worse than the PR commit says. A tree had a.txt, then dir/ (whose tree object I deleted), then z.txt. Files().ForEach returned [a.txt] with no error, so z.txt was lost too, not just the contents of dir/. The same code is on go-git main, and searching go-git's issues for "TreeWalker io.EOF" and "missing subtree" turned up nothing.

Repo: go-git/go-git · Assignee: Soph (go-git accepts you as an assignee)

Title: TreeWalker.Next ends the walk with io.EOF when a subtree is missing

TreeWalker.Next turns a failed subtree lookup into io.EOF, so a recursive walk over a tree with a missing subtree ends early and reports success. Tree.Files().ForEach and anything else built on FileIter returns nil for an incomplete tree, and every entry after the missing subtree is silently dropped, not only the subtree's own contents.

The code is in plumbing/object/tree.go (lines 704-706 in v6.0.0-beta.1, 726-728 on main at ae4b49cdd0; the same in v6.0.0-alpha.5):

The doc comment on Next says objects that cannot be found "will be skipped automatically", but the walk does not skip the entry, it stops. Any GetTree error is mapped to io.EOF, not only plumbing.ErrObjectNotFound, so a decode or storage error also reads as a clean end of iteration.

Reproduction

A root tree with a.txt, dir/inner.txt and z.txt, with the dir tree object removed from a memory.Storage:

Output on v6.0.0-beta.1:

Expected: an error that wraps plumbing.ErrObjectNotFound and names the missing tree. At the least, z.txt should still be visited if skipping stays the documented behaviour.

Impact

Callers that use Files() or TreeWalker to check that a tree is complete (after a fetch, a partial clone, or a pack import) get a false pass. We hit this in a test helper meant to prove every object reachable from a ref was readable. It kept passing with a subtree deleted from storage, so we replaced it with an explicit walk.

Suggested fix

Return the GetTree error from Next instead of io.EOF, and update the doc comment. Callers that relied on the skip behaviour would see an error where they saw a short walk, so this may want a mention in the v6 release notes. If the skip is meant to stay, Next should skip that one entry and continue with its siblings, and only plumbing.ErrObjectNotFound should be skipped.

store and blob in the snippet are small helpers that encode an object into storage; the full runnable program is in the scratchpad if you want it attached.

Should I file it? Separately, the force push of the rebased #138 branch is still waiting on your go-ahead.

I filed the go-git issue and assigned it to you: https://github.com/go-git/go-git/issues/2555

The rebased #138 branch still hasn't been pushed. Should I run git push --force-with-lease to update the PR?

You're right, and nothing more is needed. The PR was merged from 8d4f734b, the head of the rebased branch, so the rebased commits are what landed. origin/main (23568c7d) has the same content as the local branch.

Still open:

• mirror-pipeline: once a git-sync release includes #138, the worker can bump git-sync and set IndependentHistoryRefPrefixes: []string{"refs/entire/checkpoints/"} in refScopeFor.
• Old checkpoint refs: you said earlier the 22 refs/entire/checkpoints/v2/* refs in entireio/cli and entireio/cli-checkpoints should be deleted. I haven't touched them, and I'll wait for your go-ahead before deleting anything.

I checked how git-sync uses go-git's tree walker. My recommendation is not to move to go-git main for this fix: git-sync doesn't need it, and pinning an untagged commit has a cost.

• No git-sync code hits the bug. Only one non-test place reads trees, the object collector at internal/planner/planner.go:814. It reads tree.Entries itself and returns an error when a lookup fails. Nothing outside tests calls TreeWalker, Files(), DiffTree or merkletrie, which is the code the fix (go-git #2556) changes.
• The tests already cover it. The only Files().ForEach call is in the regression test at internal/syncer/independent_history_test.go:498. The real check there is readClosure, which works on either version. That line just logs a note once go-git reports the missing subtree.
• The cost: go-git main is 89 commits ahead of v6.0.0-beta.1, and nothing newer has been tagged. A public library pinned to an unreleased commit takes on all 89 untested changes for its consumers, including the mirror-pipeline bump that's coming.

I'd stay on beta.1 and move when go-git tags the next release (beta.2 or whatever it is called). At that point the test's log note should start appearing, and the explicit walk can stay as a stricter check or be simplified.

If you want to try main early anyway, I can run go get github.com/go-git/go-git/v6@main on a branch and run the race test suite. It won't change behaviour for the reasons above.

Tried it: moving to go-git main works and changes nothing you'd notice. I'd still wait for the next go-git tag before shipping it.

What I did: made a local branch soph/go-git-main off main with one commit that moves go-git and go-billy to their latest main commits. Nothing else in go.mod changed, and nothing is pushed.

Results:

• Build, go vet, the full race test suite, mise run lint and the git-http-backend suite all pass.
• The go-git fix is confirmed. The Files().ForEach call in the regression test now returns object not found for the deleted subtree, and the test logs that the workaround is no longer needed.

What else comes with it: go-git main is 89 commits past beta.1. From the commit titles, the ones in code git-sync uses:

• Stricter pack parsing. go-git now rejects truncated packs and objects shorter than their declared size. That turns bad input into clear errors, and our tests don't hit it.
• HTTP push framing. Large pushes now go out without a Content-Length, as git does. This doesn't reach git-sync: it builds its own HTTP requests in internal/gitproto/smarthttp.go.
• Tree entry modes, in convert-sha256. This is the only real behaviour change. go-git now reads tree entry modes the way git does, and convert-sha256 decodes and re-encodes every tree.
  • A file with an execute bit set but not the owner's, such as 100011, used to be written as 100755 and is now written as 100644.
  • Trees with zero-padded modes, which used to fail to decode, now convert.
  • So for those rare trees, the SHA-256 hash changes between git-sync versions. The tests have no such trees.
  • The conversion already didn't copy modes byte for byte, so this widens an existing difference rather than adding a new one.

Recommendation: keep the branch as a preview, and bump when go-git tags beta.2 or whatever comes next. At that point we can drop the Files().ForEach note from the test and mention the convert-sha256 mode change in the CHANGELOG. Shipping now would pin a public library to an untagged commit and gain nothing, since no git-sync code uses the fixed walker.

Should I keep the branch as is, open a draft PR from it, or delete it?

I bumped go-git and go-billy to their latest main and opened the trail: https://entire.com/gh/entireio/-/trails/2763/changes/git-sync/10

The branch soph/go-git-main is pushed as one commit, 3e978daa. It contains:

• The bump. go.mod and go.sum now point at go-git bbee1d214130 and go-billy 3dcdd520bf27; no other module moved.
• The test. The missing-subtree regression test now fails unless go-git's Files().ForEach reports the missing subtree as an error. Before, it only logged a note. The comments no longer describe the old go-git behaviour.
• The CHANGELOG. A ### Changed entry links go-git/go-git#2555 and covers the stricter pack checks. It also covers the convert-sha256 tree-mode change: a file mode like 100011 now converts as 100644 instead of 100755, so such a tree gets a different SHA-256 hash.

The full race test suite, mise run lint and the git-http-backend suite all pass. The trail description says git-sync now pins untagged go-git commits until go-git tags its next release.

I removed both comments, along with the Files().ForEach check that existed only to show the old go-git behaviour. Compared with main, the test file now only loses lines:

• The Files().ForEach probe and its comment are gone from the missing-subtree test. The test still requires readClosure to name the missing subtree.
• The readClosure doc comment now just says what the function does: it walks parents and tree entries, reports a missing object as an error, and reads every blob.

The syncer tests (with the race detector) and lint pass. The change isn't committed yet.

Two other places still say the test "now requires that error", and that's no longer true: the pushed commit message, and the sentence in the trail description. I'd amend the commit, update that sentence, and git push --force-with-lease. The branch has one commit and was only just pushed. Should I go ahead?