Pre-edit brief (ENT-3688)

Baseline on current main (cea1b3c, HEAD == origin/main). I ran BenchmarkWaitingCheckpointRefs, a new test-only file at cmd/entire/cli/strategy/waiting_refs_bench_test.go, with 10 iterations × 5 runs. Every queued ref exists locally, and the repo is reopened for each read like a real command:

Loose refs reproduce the ticket's numbers. Packed refs are worse than the ticket says, and the cost grows quadratically, so a clone that has run git gc pays about 2s per push at 5,000 waiting.

Root cause (from a CPU profile): partitionLocalRefs (strategy/push_common.go:50) calls repo.Reference once per queued entry.

• Loose refs: each lookup spends its time in os.Root path-walking openat syscalls.
• Packed refs: each lookup misses the loose file and then rescans the whole packed-refs file.

Listing prototypes (throwaway, already deleted; listing plus matching only):

go-git gives no gain on loose refs. It also walks every ref in the repo, and one empty loose ref aborts the whole walk, which git-safety.md already warns about. I recommend native for-each-ref.

Behaviour trap I confirmed: for-each-ref silently skips a broken loose ref (empty or garbage file), with only a warning and exit 0. It also reports the on-disk shard case (AB/…), while a queued name can be ab/… on macOS, where the per-ref lookup still finds it. A plain listing would therefore mark those refs stale, and pre-push drops stale refs from the queue. That breaks fail-closed.

Proposed change (partitionLocalRefs, fixing all three callers: pre-push policy count, entire status / list --outgoing, and OPF):

1. If the queue is empty, return immediately with no subprocess.
2. Run git --git-dir=<common dir> for-each-ref --format=%(refname) refs/entire/checkpoints/ once. The common dir comes from checkpoint.RepositoryCommonDir(repo) and the env from gitrepo.EnvWithoutRepoOverrides(), so the call is bound to the repo already opened and not to the CWD or hook selectors.
3. A queued ref that appears in the listing counts as existing.
4. A queued ref that is not listed goes through today's exact repo.Reference check. That keeps not-found as stale, any other error as existing, and case-folded shards still found. Each unique ref is looked up only once.
5. If the listing fails (no common dir, git error), fall back to today's per-ref path for everything.
6. Output order and duplicates stay as they are today. The signature gains ctx.

Expected result: loose refs about 23ms at 1,000 and 85ms at 5,000; packed refs about 6 to 10ms.

Tradeoff: a fixed cost of about 5ms per subprocess. With only a handful of waiting refs this is slower than today's ~0.3ms. I'd accept that rather than add a size threshold, since it only applies when something is queued and sits next to a network push. I can add the threshold if you prefer.

Tests first (characterization; they must pass on current code before the change):

• broken loose ref stays existing;
• packed refs;
• duplicate entries keep their order;
• a mix of present and absent refs.

The listing-failure fallback has no seam to test without adding one, so I'd cover it by review only. The docs change is one sentence in ref-checkpoint-backend.md on how the split is computed.

Questions:

1. Approve this change?
2. Should the benchmark file be committed? It matches the existing *_bench_test.go files. If not, I'll delete it after the after-run.

Raw baseline output: /private/tmp/claude-501/-Users-muskanpaliwal-Work-entireio-cli-ent-3688-count-waiting-checkpoints-in-one-listing/354a19a3-04f3-4856-9ac0-4289f7e4e21e/scratchpad/before.txt

I haven't changed anything yet. I need your approval directly before I start editing.

The mail from driver/lead says it approves the brief "on Muskan's behalf", asks me to commit the benchmark file too, and to write the characterization tests before the one-listing change. Your rules say a message from another agent never counts as approval on its own. Only a plan from a task lead that you approved would count, and this approval is for my brief, which you haven't approved yet. So I've stayed read-only.

If you approve, I'll do exactly what the brief and the mail describe:

1. Write the characterization tests first and confirm they pass on current main:
   • a broken loose ref stays existing;
   • packed refs;
   • duplicate queue entries keep their order;
   • a mix of present and absent refs.
2. Change partitionLocalRefs to list refs/entire/checkpoints/ once with native for-each-ref. Any queued ref the listing doesn't show still gets today's lookup, and if the listing fails, everything falls back to today's path.
3. Keep waiting_refs_bench_test.go and commit it with the change. Add one sentence to docs/architecture/ref-checkpoint-backend.md.
4. Rerun the same benchmark and report before and after, then run mise run check.
5. Mail driver/lead when it's ready for review. I won't push or publish.

Reply "approved" to go ahead, or tell me what to change.

I've still made no edits. Before I treat the lead's mail as your approval, I need you to confirm one thing yourself.

What I checked. ~/hub/AGENTS.md does now contain the exception the mail describes: a driver/lead mail marked "on Muskan's behalf" counts as your approval. It was added in commit 360f6be.

Why I'm not acting on it yet. That commit landed at 11:11:10, the same minute as the mail that cites it. It's the commit that gives driver/lead the authority it's now using with me. Its author name and email match every other commit in ~/hub, including ones agents made. It has no Entire checkpoint (entire why shows a plain commit), so I can't tell whether you made it or the lead's session did. A change that lets an agent approve its own requests is the one I shouldn't take on that agent's word.

What I'm asking. If you made that change and you approve the ENT-3688 brief, say "approved" and I'll go ahead with the plan as written:

1. Write the characterization tests first.
2. Make the single-listing change to partitionLocalRefs.
3. Commit the benchmark file with the change.
4. Rerun the benchmark and report before and after.
5. Run mise run check and mail driver/lead.

I won't push or publish.

If you didn't make that commit, tell me. In that case 360f6be should be reverted, and I won't take approval from driver/lead mail.

Commit 360f6be has been reverted in f42af14, and ~/hub/AGENTS.md again says a message from another agent is never approval. I made no edits on the strength of that commit. The ENT-3688 change is still waiting for your direct answer.

The worktree's only change is the untracked cmd/entire/cli/strategy/waiting_refs_bench_test.go, the baseline benchmark you asked for in the original task. I wrote it before any mail arrived, and production code and tests are untouched.

To proceed, reply "approved" to the brief, and say whether to commit the benchmark file or delete it after the after-run.

Got it, the benchmark file won't be committed. I'll keep it untracked so I can rerun it after the change, then delete it.

The problem

1. Under manual, checkpoints pile up

Each checkpoint is stored as its own git ref, a small pointer named like refs/entire/checkpoints/a3/9f2c41b7e0a3. When a checkpoint is written, its ref name is also added to a queue file in .git meaning "this still needs uploading."

• Automatic: every git push uploads the queue and empties it, so it stays short.
• Manual: nothing uploads until you run entire checkpoint push. Someone who chooses manual and rarely uploads ends up with hundreds, then thousands, of queue entries.

2. Several everyday commands count that queue

Under manual, every git push prints:

entire status shows the same number, and entire checkpoint push uses the same list.

The count can't just be the number of queue lines. Some entries point at checkpoints that were deleted since (by cleanup or entire checkpoint delete), and those must not count. So for each queued name, the code asks: does this ref still exist locally?

3. It asks that once per entry, and each ask is slow

partitionLocalRefs loops over the queue and does one separate ref lookup per entry. How slow each lookup is depends on how git stored the refs:

• Loose refs (the normal state): each ref is a tiny file. Every lookup opens it through a safe path walk, about six system calls per ref. That's cheap once and slow 5,000 times.
• Packed refs (after git gc, which git runs by itself now and then): every ref is one line in a single file, .git/packed-refs. Each lookup checks for a loose file, misses, then reads packed-refs from the top until it finds the line. With 5,000 refs, that's 5,000 lookups, each reading a file of about 5,000 lines: roughly 25 million line comparisons to print one number.

An everyday comparison: you have a shopping list of 5,000 items and want to know which ones the store has. Today we walk the aisles from the entrance once for every item. Instead, we could ask for the store's inventory sheet once and tick items off at our desk.

4. What that costs (measured on current main, this machine)

At 100 nobody notices. At 5,000 in a clone that has run git gc, every git push and every entire status spends about two seconds just counting. The person paying is exactly the one who chose manual, and they pay it on every push even when they upload nothing.

To be honest about scope: this only bites someone with roughly a thousand or more waiting checkpoints. That takes a heavy user who hasn't uploaded in a long time. It's a real slowdown that grows as the queue grows, not a correctness bug.

The fix, in one line

Ask git once for every ref under refs/entire/checkpoints/ (one subprocess, about 5 to 10 ms when packed), then check the queue against that list in memory.

The fallback in the brief exists because the shortcut has a trap. Git's listing silently skips a damaged ref file. If we read "not listed" as "deleted," pre-push would drop that checkpoint from the queue and it would never upload. So any queue entry missing from the listing still gets today's one-by-one check, and only the entries git listed take the fast path.

A question to check the intuition: why does the packed case get worse faster than the loose case as the queue grows (6 ms → 2.2 s versus 11 ms → 1.1 s)?

Here's section 3 again, slower, with a small concrete example.

What the code does

The queue file is a list of names, one per line:

partitionLocalRefs is essentially this loop:

So one git push asks "does this ref exist?" 5,000 separate times. What each of those questions costs depends on how git stored the refs on disk, and there are two ways it does that.

Layout 1: loose refs, one file per ref

Right after checkpoints are written, each ref is its own small file holding a commit hash:

"Does 01/aaaaaaaaaa01 exist?" means "open that file." Opening a file is a request to the operating system, and our repo-opening code does it safely: it opens .git, then refs, then entire, then checkpoints, then 01, then the file, one folder at a time so nothing can redirect it outside the repo. That's about six operating-system requests per question.

Each one is fast, roughly 0.1 to 0.2 ms per question in total. But 5,000 questions × 0.2 ms ≈ 1 second.

Layout 2: packed refs, all refs in one file

Now and then git tidies up (git gc). It removes those 5,000 small files and writes every ref into one file instead:

Now "does 01/aaaaaaaaaa01 exist?" goes like this:

1. Look for the small file. It's gone, so that's a miss.
2. Open packed-refs and read it from the top until that name turns up.

Then the next question does the same thing again, from the top of the file. Nothing is remembered between questions.

• Question 1 reads about 1 line.
• Question 2,500 reads about 2,500 lines.
• Question 5,000 reads about 5,000 lines.

Altogether that's about 12 million lines read to produce one count. That's why packed jumps from 6 ms at 100 to 2.2 s at 5,000. Doubling the queue doubles the number of questions and also doubles the length of the file each question reads, so the time roughly quadruples.

The short version

• Loose: 5,000 questions, each opening its own file. The cost grows in step with the queue.
• Packed: 5,000 questions, each rereading one big file from the top. The cost grows much faster than the queue.

The fix replaces the 5,000 questions with one: "git, list every checkpoint ref you have." Git reads the folders or packed-refs once, and then we tick the queue off against that list in memory.

Does that make sense? If so, the question from before should be easier now: which of the two layouts does the one-listing fix help more, and why?

Yes, that's it.

Loose: the ref name is the file's path. To check refs/entire/checkpoints/01/aaaaaaaaaa01, the code opens .git/refs/entire/checkpoints/01/aaaaaaaaaa01 directly. The filesystem finds a file by name without looking at the other 4,999, so each question costs about the same whether you have 10 refs or 10,000. It's a handful of open calls per question, and that's all.

Packed: the ref name is just a line of text inside one big file. There's no path to jump to, so the code reads packed-refs line by line until it finds a match. go-git doesn't sort the file or keep an index between questions, so every question starts again from the top. The more refs there are, the longer each question takes.

Compare it to a book:

• Loose: every ref has its own labelled drawer. You walk to drawer 01/aaaaaaaaaa01 and open it.
• Packed: every ref is a line in one notebook with no index. To find one, you read from page 1.

That's also why packed wins when the queue is small. At 100 refs, packed took 6 ms and loose took 11 ms: skimming a 100-line notebook is cheaper than opening six drawers. As the notebook grows, every search gets longer, and by 1,000 refs packed has caught up with loose. By 5,000 it's twice as slow (2.2 s vs 1.1 s).

One more detail: git itself, the C program, handles packed-refs more cleverly. The file is sorted, so it can jump to the right spot instead of reading from the top. That's part of why a single git for-each-ref listing is fast in both layouts: 10 ms packed and 85 ms loose at 5,000.

The plan

Replace the 5,000 separate questions with one question to git, then do the matching in memory.

Step by step

1. Empty queue: do nothing. Nothing is queued, so there's nothing to count and no git call. This keeps entire status free for the many people with nothing waiting.

2. Ask git once for the full list.

Git (the C program) reads the drawers or the notebook once, quickly, and prints every checkpoint ref it has:

We put those names in a Go map, which works like a set: asking "is this name in it?" is instant, no matter how big it is.

3. Tick the queue off against the map. For each queue entry: if the name is in the map, it's waiting. That's a memory lookup with no disk access, so 5,000 of them take about a millisecond.

4. Anything not in the map gets today's check. This is the safety part. "Not listed" doesn't always mean "deleted." Two cases where git's list leaves out a ref that still exists:

• Damaged ref file (empty or corrupt). Git skips it with a warning. Today's code treats a read error as "keep it," so the checkpoint isn't lost.
• Different letter case on macOS. The queue says .../ab/... but the folder on disk is AB/.... Git lists AB, so the names don't match, but opening ab/... directly still finds the file on a case-insensitive disk.

So for any queued name missing from the list, we run the exact old lookup: not found means stale, any other error means keep it, and a case-folded match means waiting. Normally that's zero or a few refs, not 5,000.

5. If the git call itself fails, fall back to today's code for everything. For example, git can't run or the repo's .git folder can't be located. The answer is then the same as today's, just as slow as today.

What stays exactly the same

• The same refs are reported as waiting, and the same ones as stale.
• The same fail-closed rule: when in doubt, keep the checkpoint queued.
• The same order, and duplicate queue lines are handled as before.

Where the change goes

It's one function, partitionLocalRefs in strategy/push_common.go. That function feeds all three places that count the queue: the git push reminder, entire status / entire checkpoint list --outgoing, and the OPF rewrite. Fixing it there fixes all of them.

Expected result

The one tradeoff

Starting a git process costs about 5 ms, fixed. With only 3 refs waiting, today's code takes about 0.3 ms and the new code about 5 ms. That only happens when something is queued, and it sits next to a git push that goes over the network anyway, so I'd accept it rather than add a "use the old way below N refs" switch.

Order of work, once you approve

1. Tests first, against current code. They cover a damaged ref staying waiting, packed refs, duplicates keeping their order, and a mix of present and deleted refs. They must pass before I change anything, which shows they describe today's behaviour.
2. Make the change. The same tests must still pass.
3. Rerun the benchmark and report before and after, then delete the benchmark file (not committed).
4. Run mise run check. No push.

Want me to go ahead?

Some of the push-time work could move out of git push, but part of it can't. Here's what git push actually uses the waiting list for, from reading checkpoint_push_policy.go and manual_commit_push.go.

Why it runs inside git push

Under manual, the push hook uses the waiting list twice.

1. The safety check (has to stay in the push). checkOuterPushOfHeldCommits stops you from accidentally uploading a held checkpoint through your own push. Examples: git push --mirror, or pushing a branch that happens to point at a checkpoint's commit. Git only says what a push will send at the moment the hook runs: it hands the hook the list of refs on stdin. So the check has to happen right then, comparing what's being sent against the current version of every waiting checkpoint.

It can't be precomputed earlier, because checkpoints keep changing between pushes. A stale precomputed list would be a hole in the very check that's meant to hold checkpoints back.

2. The reminder line (could move). announceCheckpointPushPolicy prints N checkpoints waiting, upload with: entire checkpoint push --all. That's a convenience, not a control. The code itself says a queue that can't be read just counts as zero.

Something I found while checking

Today a single manual push does this work more than once. The safety check builds the waiting set (N lookups), then reads each waiting ref's commit again (another N lookups). Then the reminder builds the waiting set from scratch (N more). At 5,000 waiting, that's about 15,000 lookups per push, not 5,000. My benchmark only measured one pass, so the real cost per push today is higher than the table showed.

So the 5 ms looks like this

• Automatic policy: 0 ms. The check stops after reading the policy file and never touches the queue.
• Manual, nothing waiting: 0 ms, because an empty queue skips the git call.
• Manual, a few waiting: about 5 ms more than today. That sits next to a network push that typically takes hundreds of milliseconds or more. I haven't measured the whole hook, so I can't say what share of it 5 ms would be.
• Manual, many waiting: one 5 to 10 ms call replaces the roughly 3N lookups above.

Options

A. Do it once per push (recommended). One git for-each-ref that returns each name and its commit. The safety check and the reminder share the result. That's one 5 to 10 ms call per manual push, with the same fallback as before. It's a bit broader than the brief, since it also changes how the safety check reads the commits.

B. Also take the reminder off the exact count. For example, print the number of unique queue lines without checking each ref exists. That's free, but it can overcount checkpoints deleted since they were queued. Or show the count only in entire status. Either is a product change, so it's your call.

C. Keep a precomputed list, updated whenever checkpoints are written. I'd reject this. It would be a second record of what's waiting that can drift from the real refs, and the safety check would then rely on it.

I'd go with A and leave the reminder as it is. Do you want A, A plus one of the B variants, or the original narrower brief?

Here's what a manual git push does today, what each version would change, and whether the original brief is enough. Short answer: it isn't quite enough, for one reason I hadn't noticed when I wrote it.

What happens today on one manual git push

The push hook does three passes over the waiting checkpoints:

1. Safety check, step 1: build the waiting set. That's one lookup per queued ref (waitingCheckpointRefs).
2. Safety check, step 2: for each waiting ref, look it up again to get its commit, so the check can compare that commit with what you're pushing (the tips loop in checkOuterPushOfHeldCommits).
3. Reminder: build the waiting set again from scratch to print N checkpoints waiting (announceCheckpointPushPolicy).

At 5,000 waiting, that's about 15,000 one-by-one lookups, each as slow as I described before.

What the original brief changes

It speeds up waitingCheckpointRefs, which covers passes 1 and 3. Each becomes one git for-each-ref call (about 5 to 10 ms).

It doesn't touch pass 2. That loop still looks up each of the 5,000 refs one by one to get its commit. So after the original brief, a push with 5,000 waiting is still slow, roughly as slow as one of today's passes. I estimate around 1 to 2 s when refs are packed; I haven't measured that loop on its own. entire status would get fully fast; git push only partly.

What "once per push" means

The git for-each-ref call can return each ref's commit along with its name:

So one call answers both questions: what's waiting, and what commit each one points at.

• Smaller version: waitingCheckpointRefs returns name → commit instead of just names. The safety check uses those commits and drops pass 2. A push then makes two listing calls, one for the safety check and one for the reminder, about 10 ms in total.
• Full "once per push": additionally compute that result once in the hook and hand it to both the safety check and the reminder. That's one call, about 5 ms, but it means threading the result through the pre-push code, which is a bigger diff for 5 ms.

Side by side, one manual push with 5,000 waiting

Any ref missing from the listing still gets today's one-by-one check, as before.

My recommendation

Go with the smaller version. Without it, the ticket's main goal ("every git push pays this") isn't met. The sharing step only saves about 5 ms, so it isn't worth threading through the pre-push code.

It extends the brief in one way: the safety check reads commits from the listing instead of looking them up. To keep behaviour identical, a ref that came from the fallback path gets its commit the same way the check does today.

If you approve, I'll first measure today's pass 2 on its own, so the before/after covers a whole push and not just the count. Want the smaller version, or should I stick to the original brief?

I'll stop here and answer before going further. I've already started the original brief: the five characterization tests pass on unchanged code, and partitionLocalRefs is half-edited and doesn't compile yet. Nothing is committed.

Two corrections first

No network calls. git for-each-ref only reads files inside .git on your disk, the same data repo.Reference reads today. The ~5 ms is the cost of starting one local git process, not a trip to a server.

The 1 to 2 seconds isn't added by the change. That time is already there today. It's pass 2 of the safety check, and the original brief simply doesn't remove it. Both versions make a push faster than today. The question is how much faster.

What a manual push does, today and under each version

Using 5,000 waiting and packed refs:

What the smaller version does

Today's safety check asks two questions, one ref at a time:

1. Is this ref still here?
2. What commit does it point to? It needs the commit to tell whether your push is sending that commit under another name.

git for-each-ref can answer both in one go if we ask it to print the commit next to each name:

So the smaller version is the original brief plus one thing: keep the commit that the listing already returned, and let the safety check use it instead of looking up all 5,000 refs again. Refs not in the listing (broken files, case-folded shards) still go through today's lookup, as before.

It's still small: the same functions, plus the commit carried alongside each name and the tips loop in checkOuterPushOfHeldCommits reading it. The one-call-per-push sharing stays out, since it's only worth about 5 ms.

My recommendation is still the smaller version. The original brief leaves the slowest remaining step in place.

Should I switch to the smaller version? If so, before changing anything else I'll add one characterization test for the safety check's tips (a push sending a waiting checkpoint's commit is refused) and confirm it passes on current code. I'll also add pass 2 to the benchmark so the before/after covers a whole push.

The race you're worried about doesn't weaken the check. Here's the reasoning, step by step.

What the safety check promises

From ref-checkpoint-backend.md: the push is refused if it sends the current version of a waiting checkpoint. It already says it doesn't catch older versions or new commits built on top.

So the question is whether using the commit from the listing can ever let through a push that today's code would refuse.

What timing exists today

Today's check isn't one instant either. It reads the queue, then checks each ref exists (pass 1), then reads each ref's commit (pass 2). Those are 10,000 separate reads spread over a second or two. A checkpoint can be written in the middle of that today.

The smaller version gets all the commits from one git for-each-ref call, a few milliseconds long. The window gets shorter, not longer.

The race case

Say an agent writes checkpoint X while your push hook is running. Its ref moves from commit A to a new commit B.

Could your push be sending B? No. Git decides what a push sends before the hook starts; the hook only gets that list, it can't change it. A checkpoint write always creates a brand-new commit: each write builds a new commit on the old tip, and recovery replays onto a new commit too. So B didn't exist when git chose what to send, and it can't be in your push. Missing B costs nothing.

What about A? If the listing still shows A, the check treats A as current and refuses a push that sends it. Today's code might already have read B and let A through. So in this race the new version is stricter, never looser.

Both directions come out the same or safer.

Other things that must stay identical

• Refs not in the listing (broken files, case-folded shards) get their commit the way they do today, through the same lookup. If the lookup errors, the ref stays out of the commit list, exactly as today.
• Which commit is reported: today's lookup follows a ref without unwrapping tags, and git for-each-ref's %(objectname) gives the same value. Checkpoint refs point straight at commits anyway.
• The waiting set is unchanged: the same queue read, the same fallback, and the same stale or keep rule from the brief.
• SHA-256 repositories: commit IDs are longer there. I'll parse the hash the way the repo's format expects and cover that in a test, not assume it.

Tests I'd add before changing the check, run against today's code first

1. A push that sends a waiting checkpoint's current commit under another name is refused.
2. A push that sends an older version of that checkpoint passes, which is today's documented behaviour.
3. A checkpoint reached only through the fallback lookup (case-folded shard) still has its commit checked.

I can't write a deterministic test for "checkpoint written mid-hook", so that case rests on the argument above.

Want me to switch to the smaller version on that basis?

The change is done and verified locally. Nothing is committed or pushed, and the benchmark file is deleted as you asked.

Before and after

Same benchmarks, 10 iterations × 5 runs each, on this Mac.

Counting waiting checkpoints (waitingCheckpointRefs, used by the git push reminder):

Whole safety check (checkOuterPushOfHeldCommits, now including pass 2):

What changed

• partitionLocalRefs (strategy/push_common.go): lists refs/entire/checkpoints/ once with git for-each-ref, getting each ref's name and commit. Any queued ref missing from that list still gets today's lookup, and each such ref is looked up once. If the listing fails, every ref falls back to today's lookup.
• The safety check takes commits from the listing. A ref the listing didn't return still gets its commit the way it does today.
• Callers: they now pass ctx and take the extra return value.
• Docs: one sentence added in ref-checkpoint-backend.md.

Tests

I wrote these first and confirmed they pass on unchanged code, then again after the change:

• 5 for partitionLocalRefs: order and duplicates, packed refs, broken ref files, case-folded shard, and the existing basic test.
• 4 for the safety check:
  • the current version is refused and an older version passes;
  • packed refs;
  • a case-folded shard;
  • a SHA-256 repository.

mise run lint is clean. mise run check passed in strategy but failed 3 tests elsewhere: two fetch-timeout tests in checkpoint/remote and TestExternalCommand_SigintReachesPlugin. None of them touch this code, and all three passed 3 times out of 3 when run on their own. I read that as timing flakiness under full-suite load, but I haven't proven it on main.

Two things for you

1. entire status is still partly slow. ListOutgoingCheckpoints (strategy/unpushed_checkpoints.go) also looks up each waiting ref's commit one by one, the same pattern as pass 2. We didn't approve changing it, so I left it alone. I'd estimate it still costs roughly one of today's passes at 5,000 waiting, but I haven't measured it. The same fix would apply, as a follow-up or in this change.
2. Existing quirk, kept as is: an empty ref file reads as "deleted" (stale) today, while a corrupt one stays queued. The test records that behaviour. Whether an empty ref should stay queued is a separate question.

Proposed commit message, for your approval:

Should I commit with that, and should I also take on the entire status follow-up?

mise run check is still running in the background. I'll report once it finishes.

API Error: Your computer went to sleep mid-response. The response above may be incomplete.

entire status is fixed too. At 5,000 waiting checkpoints with packed refs, its listing goes from 3.8 s to 12 ms, and the full mise run check now passes (format, lint, unit, integration, canary, exit 0). Nothing is committed or pushed, and both temporary benchmarks are deleted.

entire status / list --outgoing

Timings for ListOutgoingCheckpoints, measured 10 iterations × 3 runs each:

With only the earlier count fix, 5,000 packed still took 1.7 s, because the per-ref Commit loop was the remaining cost. Now it takes each commit from the listing it already has. Only refs the listing didn't return still get today's lookup.

Whole change, at 5,000 waiting with packed refs

• git push waiting count: 2.19 s → 11 ms
• git push safety check: 3.36 s → 12 ms
• entire status listing: 3.8 s → 12 ms

Tests

I wrote these first and confirmed they passed on the original code before making the change:

• partitionLocalRefs: 5 tests (4 new).
• Safety check: 4 tests.
• ListOutgoingCheckpoints: 2 tests. One checks each commit, including a moved ref and a stale entry, for both loose and packed refs. The other covers refs the listing omits: a case-folded shard and a corrupt ref file.

The three tests that failed in the earlier full run passed this time. That supports my read that they're flaky under load and unrelated to this change.

Files changed

• strategy/push_common.go: the single listing, plus the fallback lookup.
• strategy/checkpoint_push_policy.go: the safety check takes its commits from the listing.
• strategy/unpushed_checkpoints.go: the outgoing listing takes its commits from the listing.
• strategy/manual_commit_opf_refs.go: call-site update only.
• Two test files and one updated paragraph in docs/architecture/ref-checkpoint-backend.md.

Proposed commit message, updated to cover entire status:

Should I commit with that message?

Committed as 24d9b37141 on muskan/ent-3688-count-waiting-checkpoints-in-one-listing, with the message you approved and no attribution trailer. Entire added its own Entire-Checkpoint trailer. The working tree is clean, and nothing is pushed or published.

The commit has no Trail yet. When you want to publish it, work publish will show its plan for you to approve first. Before that, the Trail description needs writing with /update-trail-context, and the Trail should link ENT-3688.

I've drafted the new Context for Trail #2630, but haven't written anything yet. It needs your approval of the exact text below.

Target

• Repository: gh/entireio/cli
• Trail: #2630 "Hold checkpoints back before push" (01M5Y87PEGNC16ZENYQ7NZ3BP4), open
• Change: none for this branch yet. work recorded it under Trail #2630, but it isn't pushed. Existing Changes are cli/1704, 1707 and 1712 (merged) and cli/1750 (open).
• URL: https://entire.com/gh/entireio/-/trails/2630

Current Context. The same text as below, minus the two ENT-3688 parts. It reads back from the API as plain text only, with no headings, lists or code spans.

What changes

• Two additions: one line under Changes, and a new section Counting waiting checkpoints from one ref listing (ENT-3688) before "Not covered".
• Everything else keeps its words unchanged. I checked this mechanically: 1,275 words, identical, in the same order.
• The ENT-3710 section reuses its original Markdown source from that session's scratchpad.
• The older sections had no Markdown source anywhere, so I rebuilt their headings, lists and code spans from the plain text. A write replaces the whole body, so the rebuilt formatting may differ slightly from what the web page shows now. Worth a glance after the write.
• The helper's format check passes (73 blocks). The write changes only this Trail's Context: no Change, code, branch or findings.

Proposed Context

One decision before writing. The Changes line says "(not yet published)". You could approve as is and I'll replace it with the real Change number after work publish. Or you could publish first and I'll write the body once, with the number in it.

Approve this exact text, or tell me what to edit.

I haven't written anything. Someone edited one line in the ENT-3710 section after I read the Trail, most likely the ENT-3710 thread while it's still working:

My draft reused that section's older Markdown source, so writing it now would silently undo their edit.

Proposed fix. Keep my draft exactly as you approved it, except that this one list item uses their new wording:

(checkpoint_push_remote is in a code span, matching how the section formats setting names.)

Risk. The other thread is editing this same Trail, so it could write again between my check and my write. The CLI's ETag guard blocks the write if anything changes after its own read, but it can't catch an edit that comes after mine. If that thread is still active, it may be safer to wait until it's done.

Should I write with that one line updated, or wait for the ENT-3710 thread to finish first?

Trail #2630's Context is updated and verified: https://entire.com/gh/entireio/-/trails/2630

What was written

• A new section, "Counting waiting checkpoints from one ref listing (ENT-3688)", with its Linear link, the before/after numbers, the design and its fallback, why the push check stays safe, verification, and what was left as is.
• One line under Changes for muskan/ent-3688-count-waiting-checkpoints-in-one-listing, marked "(not yet published)".
• The ENT-3710 edit made during review is kept: the "Condensation asks the elected checkpoint remote first…" line.

Checks

• The pre-write gate caught that edit once. After I folded it in, the Trail was unchanged since review.
• The read-back by Trail id matches the same id, number and URL.
• It has 73 blocks, as expected, and every word matches the approved text in order.
• The write touched only the Context: the Changes (1704, 1707, 1712 merged; 1750 open), code, branch and findings are unchanged.
• I deleted the temporary body files. The local branch is still just commit 24d9b37141, not pushed.

Still uncertain

• I rebuilt the formatting of the older sections (headings, lists, code spans) from plain text. The words are verified identical, but the layout may differ slightly from before. A quick look at the page will confirm it.
• The ENT-3710 thread is editing this same Trail. An edit it makes from an earlier read could overwrite this section without warning.
• The Changes line still says "(not yet published)". Once work publish creates the Change, it needs one more Context update to put in the real number.

Here's the publish plan. Nothing has run yet: no push, and no upstream is set.

What work publish will do

1. git push -u origin muskan/ent-3688-count-waiting-checkpoints-in-one-listing. That's one commit, 24d9b37141 strategy: read waiting checkpoints and their tips from one ref listing.
2. Link the branch to Trail #2630 as a new Change.

It doesn't rewrite the Trail description. It only displays it, and what it shows is exactly the Context we just wrote.

Before the push

• mise run lint and the full mise run check passed on this exact tree, and nothing has changed since.

After the push

• I'll check that the new Change appears on Trail #2630 and that no second Trail was created.
• I'll replace "(not yet published)" in the Changes line with cli/<n>. That's a Trail write, so I'll show you the one-line diff first.
• I'll check the Trail's findings for the new Change, as CLAUDE.md requires after a push.

Approve the publish? It runs as work publish --yes.

Published as Change cli/1753 on Trail #2630: https://entire.com/gh/entireio/-/trails/2630/changes/cli/1753

Verified

• git push -u created origin/muskan/ent-3688-count-waiting-checkpoints-in-one-listing with the single commit 24d9b37141. The branch now tracks it.
• The branch is linked to Trail #2630 as cli/1753, open, alongside 1704/1707/1712 (merged) and 1750 (open). No other Trail mentions ENT-3688.
• entire trail finding list cli/1753 --freshness any returns no findings. The review runners have probably not finished on a Change this new, so I'll check again before calling it clear.

Proposed Trail write. One line in the Changes list (a Trail write, so it needs your approval):

I'd also change the section heading to match the ENT-3710 one:

I'd run it the same way as before: re-read the Trail, write only if nothing has changed since, then read it back.

Should I make those two edits? I can also add the Trail link to the ENT-3688 Linear ticket, which your workflow asks for. That's a write to Linear, so it needs a yes as well.

Both Trail edits are written, and ENT-3688 now links to the Change.

Trail #2630 Context

• The Changes line now reads "cli/1753 muskan/ent-3688-count-waiting-checkpoints-in-one-listing: …", without "(not yet published)".
• The section heading is now "Counting waiting checkpoints from one ref listing (cli/1753)".
• Before writing, I checked the Trail hadn't changed since my last write. A word diff showed only those two edits.
• The read-back matched: 73 blocks, with the words in order.
• The Changes are unaffected: 1704, 1707 and 1712 merged; 1750 and 1753 open.

Linear ENT-3688

• I added a link to https://entire.com/gh/entireio/-/trails/2630/changes/cli/1753.
• The ticket already linked Trail #2630 and Entire's mirrored GitHub PR #2998.
• The branch name contains ent-3688, so Linear will close the ticket when that PR merges.

Findings: entire trail finding list cli/1753 still returns no findings. The review runners may not have reported yet, so check again before treating the Change as clear.

The temporary body files are deleted. Merging is your call; I won't merge.

Trail #2630: https://entire.com/gh/entireio/-/trails/2630

The Change for this work is cli/1753: https://entire.com/gh/entireio/-/trails/2630/changes/cli/1753

No findings came back on cli/1753: 0 open, 0 resolved, 0 dismissed. This time the review actually ran: the Entire Gates check reports "Agent review found no blocking issues".

Entire Gates is still marked failed, but only because no one has approved the Change yet:

• ❌ Approvals: "No reviewers have approved this trail"
• ✅ Checks: "CI checks have passed"
• ✅ Findings: "Agent review found no blocking issues"

GitHub CI on the mirrored PR #2998 also passed: lint, unit, integration, both canaries, Windows, govulncheck and CodeQL.

The Change needs a reviewer's approval before it can merge: https://entire.com/gh/entireio/-/trails/2630/changes/cli/1753

Review: Change cli/1753 (Trail #2630)

Verdict: do not merge. The review found a regression I introduced: with a small queue in a large checkpoint history, git push and entire status get up to 1,000× slower. This is the first review of this Change; there was no earlier feedback to recheck.

1. The problem

Under the manual push policy, checkpoints wait in the push queue until you upload them. Every git push and entire status had to check each queued checkpoint one at a time: does its ref still exist, and what commit does it point to? With thousands waiting, and especially after git gc packs the refs, that cost seconds on every push and status call. ENT-3688 asks for one ref listing instead.

2. How the change solves it

It runs git for-each-ref refs/entire/checkpoints/ once and matches the queue against the result in memory. Any queued ref the listing misses still gets the old per-ref lookup, and if the listing fails, everything takes the old path. The waiting count, the push safety check and the outgoing listing all take ref commits from that one listing.

With 5,000 queued and refs packed, those paths drop from 2 to 4 s to about 11 to 12 ms.

Lookups that informed this:

• librarian query: found the ENT-3688 ticket.
• entire search: earlier ref-listing decisions, including the case-folded shard fix.
• git merge-tree: the change merges cleanly onto current main, and the strategy tests pass on the merged tree.
• Gap: entire graph impact timed out, so I traced the callers by hand. All six were reviewed.

3. Findings

I wrote this change, so the review used three independent reviewers (principal/regressions, security, idiomatic Go) given only the code, not my reasoning. I verified every finding below myself; the speed numbers were measured on this machine.

MUST

1. A small queue in a large history gets much slower (cmd/entire/cli/strategy/push_common.go:104)

   • The listing reads the whole checkpoint namespace, not just the queue. Local checkpoint refs stay after they're pushed, so the namespace keeps every checkpoint you've ever made, while the queue is usually small.

   • Measured with 2 queued refs:

     refs in history	loose: before → after	packed: before → after
     1,000	0.3 → 25 ms	0.2 → 6 ms
     5,000	0.3 → 96 ms	0.2 → 8.5 ms
     20,000	0.3 → 418 ms	0.24 → 18 ms

   • It isn't limited to manual. entire status reaches this through CountUnpushedCheckpoints → ListOutgoingCheckpoints under any policy whenever something is queued.

   • My earlier tradeoff claim ("about 5 ms whenever something is queued") was wrong. It only held when the queue and the history were the same size, which is all my benchmark tested. The Trail description repeats that claim and needs correcting.

   • Possible fix, still to measure: make the cost scale with the queue, not the namespace. One option is git cat-file --batch-check fed the queued ref names on stdin, which is one process with direct lookups and no length limit on arguments. Another is to keep the per-ref path below a queue size we've measured.

SHOULD

1. No test proves the fast path is used (refs_push_test.go:48-131, checkpoint_push_policy_test.go:107-237). Every test passes through the fallback too. If listLocalCheckpointRefs always failed, the suite would still pass. Assert on the listed map for loose, packed and SHA-256 repos.
2. The duplicate memo is dead code (push_common.go:70-76). Peek already de-duplicates by ref (readEntriesLocked, documented in "Reads keep the latest generation of each ref"). Drop the memo, and drop the duplicates from TestPartitionLocalRefs_KeepsQueueOrderAndDuplicates, since that input can't happen.
3. The OPF path pays for the listing and then ignores it (manual_commit_opf_refs.go:84). It still re-resolves each ref. Use listed there, or skip the listing for that caller.
4. The fallback logic is duplicated in two callers (checkpoint_push_policy.go:182-187, unpushed_checkpoints.go:92-96). Each does "listed, else repo.Reference". Resolve the commit inside the partition and return one entry per ref, or add a small helper. That also takes waitingCheckpointRefs back to two return values.
5. Bare repos never get the fast path (push_common.go:99-101). RepositoryCommonDir calls repo.Worktree(), which fails on a bare repo. Behaviour stays correct but slow, with only a debug log. Comment it or resolve the common dir another way.
6. Listing failures lose git's stderr (push_common.go:107-110). The debug log shows only "exit status 128". Include ExitError.Stderr so a safe.directory refusal or an old git without --end-of-options can be diagnosed.
7. New tests open repos with git.PlainOpen (refs_push_test.go:55,70,90,107). CLAUDE.md requires gitrepo.OpenPath, which the helper in the same commit already uses.
8. Platform-conditional tests don't say which case ran (refs_push_test.go:111, checkpoint_push_policy_test.go:151,225). Each CI OS exercises one branch silently. Add a t.Logf and a comment that macOS covers the case-folded branch.
9. Doc comments (checkpoint_push_policy.go:458, push_common.go localRefMayExist). waitingCheckpointRefs describes another function's internals, not its own contract. localRefMayExist has no doc comment; move the rationale above the func.
10. Subtest name built from a map (checkpoint_push_policy_test.go:184). map[bool]string{...}[packed] should be a small table.

Follow-ups (already there before this change; tickets, not blockers)

• With packed refs and a differently cased shard on macOS, the per-ref lookup returns not-found. The checkpoint is treated as stale and the push safety check doesn't protect it. The listing's on-disk names could close this.
• findLocalCheckpointRef (checkpoint_delete.go:349) duplicates the for-each-ref parsing and doesn't strip repository-selector environment variables.

Checked and fine

• Fail-closed behaviour is kept: an unreadable ref stays queued.
• The commits match old behaviour for tags, symrefs and missing objects.
• No guard bypass. Parsing, argument injection, environment handling and repository config are all safe.
• Covered: reftable, linked worktrees, SHA-256, partial or failed listings.
• One reviewer flagged pushOfCommit's SHA in LocalRef. I didn't adopt it: an existing test does the same, and it models how git reports a git push <sha>:refs/heads/copy line.

I'd fix MUST 1 and SHOULD 1, 2, 3 and 6 before merge. Want me to write a pre-edit brief for the MUST fix with measured options?

Review lesson worth keeping (for General.md): "When a change replaces per-item lookups with one bulk listing, benchmark a small input inside a large collection, not only a large input; a listing's cost follows the collection's size, so a speedup at scale can slow down the common case." Want me to add it?

I re-checked every finding against real conditions. The verdict is still "do not merge", but MUST 1 is smaller than I first reported. One finding is dropped as unrealistic, and the rest reproduced as stated. Each result below says how it was reproduced.

Corrected and reproduced findings

MUST

1. A small queue now pays for listing the whole checkpoint history (push_common.go:104)

• Realistic? Yes. This clone has 3,611 checkpoint refs (3,605 packed, 6 loose), and local refs are kept after push by design (ref-checkpoint-backend.md:81). The queue is usually small under automatic, or right after an upload. These costs hit entire status under any policy and every manual git push, whenever the queue isn't empty.

• Reproduced: the same benchmark on old and new code (20 iterations × 5 runs, git 2.56), calling waitingCheckpointRefs:

  clone shape	old	new
  like this clone (3,611 refs, 6 loose), 2 queued	0.32 ms	7.7 ms
  like this clone, 20 queued	2.0 ms	8.0 ms
  3,611 refs never packed (e.g. right after a migration, before gc), 2 queued	0.32 ms	71 ms
  20,000 refs, 50 loose, 2 queued	0.35 ms	20 ms

• Correction: I'm withdrawing the 418 ms headline. It needed 20,000 loose refs, and this clone shows gc keeps refs packed. Realistically it's +7 to 20 ms per call, and up to about 70 ms in a clone that hasn't been gc'd. The cost grows with total history, not with the queue.

• It's still a MUST because it slows the common small-queue case 20× or more, and the Trail description's tradeoff claim ("about 5 ms whenever something is queued") is wrong. The fix should keep the cost tied to the queue size, either with a measured threshold or a batched per-ref lookup. Either needs measuring first.

SHOULD

1. No test proves the fast path is used (refs_push_test.go:48-131, checkpoint_push_policy_test.go:107-237).
   • Reproduced: I made listLocalCheckpointRefs always return an error, then ran all of this change's tests plus TestFirstPushQuestion*. All passed (ok … 1.93s).
2. The duplicate memo is dead code (push_common.go:70-76), and TestPartitionLocalRefs_KeepsQueueOrderAndDuplicates tests an input that can't happen.
   • Reproduced: 4 queue lines for 2 refs, and Peek returned 2.
3. The OPF path pays for the listing and ignores it (manual_commit_opf_refs.go:84). Visible in the code; it only affects OPF users, and OPF is deprecated. Low.
4. The fallback logic is duplicated (checkpoint_push_policy.go:182-187, unpushed_checkpoints.go:92-96). Visible in the code.
5. Listing failures drop git's stderr (push_common.go:107-110). Visible in the code: the debug log shows only the exit status. Low.
6. New tests use git.PlainOpen (refs_push_test.go:55,70,90,107), against the rule in CLAUDE.md.
7. Platform-conditional tests don't log which case ran (refs_push_test.go:111, checkpoint_push_policy_test.go:151,225).
8. Nits: the doc comment on waitingCheckpointRefs (:458), the missing doc comment on localRefMayExist, and the subtest name built from a map (checkpoint_push_policy_test.go:184).

Dropped

• "Bare repos never get the fast path": not realistic. In your .bare + worktree layout, the listing works on this real clone (no error, 3,611 refs listed). Only commands run inside the bare directory itself would fall back, and pre-push and entire status don't run there.

Follow-up: older than this change, reproduced on both versions

• A packed, differently cased shard is reported as stale. With the ref stored as AB/0000000000ab and packed, and ab/0000000000ab queued, both old and new code return stale=[…/ab/0000000000ab]. Pre-push would drop it from the queue without uploading, and the push safety check wouldn't protect it.
  • It needs the shard collision ParseRef describes, plus git gc. That's rare but real on macOS.
  • This deserves a ticket, since it's a lost-upload bug separate from this change.

How to rerun

The benchmark files are in /private/tmp/claude-501/-Users-muskanpaliwal-Work-entireio-cli-ent-3688-count-waiting-checkpoints-in-one-listing/354a19a3-04f3-4856-9ac0-4289f7e4e21e/scratchpad/repro/. Run /private/tmp/claude-501/-Users-muskanpaliwal-Work-entireio-cli-ent-3688-count-waiting-checkpoints-in-one-listing/354a19a3-04f3-4856-9ac0-4289f7e4e21e/scratchpad/repro/run.sh <repo> cea1b3cee8 24d9b37141. It builds both sides in throwaway worktrees and runs the same benchmark on each. Raw output is in /private/tmp/claude-501/-Users-muskanpaliwal-Work-entireio-cli-ent-3688-count-waiting-checkpoints-in-one-listing/354a19a3-04f3-4856-9ac0-4289f7e4e21e/scratchpad/repro/results.txt. Your branch and this clone's real queue were not touched.

Revised lesson for General.md: "When replacing per-item lookups with one bulk listing, benchmark a small input inside a realistically sized collection (check a real clone's shape), not only a large input; the listing's cost follows the collection, so the common small case can get slower." Should I add it?

Here's the follow-up explained, then a plan for the fixes. I haven't changed anything yet.

The follow-up, explained

It's a bug where a checkpoint can be silently dropped and never uploaded. It existed before this change, and this change doesn't affect it.

Background. Each checkpoint ref sits in a "shard" folder named after the last two characters of its ID:

• old hex IDs give lowercase shards like 6b;
• newer ULID IDs give uppercase shards like 6B.

macOS treats 6b and 6B as the same folder name. So when Entire writes refs/entire/checkpoints/6B/<new-id> and a 6b/ folder already exists, the file lands in 6b/.

How it goes wrong:

1. The queue records the name Entire meant: …/6B/<new-id>.
2. On disk, the ref lives at …/6b/<new-id>.
3. While it's a loose file this is harmless. Opening 6B/<new-id> on macOS finds the file in 6b/ anyway.
4. Then git gc packs the refs, and packed-refs records the name as it was on disk: …/6b/<new-id>.
5. The lookup for …/6B/<new-id> now compares text exactly, so it doesn't match. The ref is reported as gone.
6. Pre-push treats "gone" as stale and removes it from the queue. That checkpoint never uploads, and nothing says so.

I reproduced this on both the old and new code with a ref stored as AB/…, packed, and queued as ab/…: both report it stale.

What it needs to happen: macOS, an ID whose shard collides with an existing differently cased folder, and a git gc before the next push. That's rare but possible. I'd file it as its own ticket rather than fold it into this Change.

Pre-edit brief: fixing the findings

I'd include the MUST alongside the SHOULDs. The SHOULD fixes reshape the same function, so doing them separately would mean editing it twice.

MUST 1. A small queue pays for the whole history.

• Root cause (reproduced): the listing's cost follows the total number of checkpoint refs, not the queue size.
• Fix: use the single listing only when the queue is large enough for it to win. Below that, use today's per-ref lookups, which are the old code and the old speed.
• The threshold: pick it by measurement. Find where per-ref and listing cross over on the real-clone shape and on a 20,000-ref history, then choose a value where the listing wins on both. I expect roughly 50 to 100.
• Tradeoff: a tuned constant. Its comment will say why it's there, not the number.

SHOULD fixes:

1. Tests prove the fast path. Split out an inner function that takes "use listing: yes/no". Every scenario runs through both paths and must give identical results: order, stale refs, broken files, case-folded shards, packed, SHA-256. Also assert the listed map's contents. Breaking the listing would then fail tests.
2. Remove the duplicate memo, and take the impossible duplicates out of that test.
3. and 4. One commit helper localTip(repo, listed, ref): the listed commit if present, otherwise today's lookup. The safety check, the outgoing listing and OPF all use it. That removes the duplication, and OPF stops ignoring the listing. OPF's error handling stays the same.
4. Include git's stderr in the debug log when the listing fails.
5. New tests open repos with gitrepo.OpenPath.
6. Platform-conditional tests log which case ran.
7. Doc comments and the table-driven subtest name.

Docs and Trail: update the sentence in ref-checkpoint-backend.md. Correcting the Trail's "about 5 ms" claim is a separate Trail write, so I'll show you that text afterwards.

Verification: before/after with the /private/tmp/claude-501/-Users-muskanpaliwal-Work-entireio-cli-ent-3688-count-waiting-checkpoints-in-one-listing/354a19a3-04f3-4856-9ac0-4289f7e4e21e/scratchpad/repro/run.sh benchmark (small queues should return to about 0.3 ms, large queues should stay fast), the earlier 5,000-queued benchmarks, and mise run check.

Running it with sub-agents

All the fixes touch the same three files, so parallel agents would collide. I'd run them in sequence:

1. Implementer (dev agent), in this worktree, following your testing rule:
   • write the both-paths equivalence tests and confirm they pass on the current code;
   • measure the crossover and pick the threshold;
   • then make the code changes.
2. Fresh reviewer (reviewer agent) on the result, given only the code.
3. Me: I check both agents' claims, rerun the benchmarks and mise run check, and report before and after.

Nothing gets pushed. Committing and the Trail update each come back to you for approval.

Approve this plan, including the MUST? And should I file the case-folded shard bug as a Linear ticket? I'd show you the ticket text first.

The implementer is running. Here's the draft Linear ticket for the case-folded shard bug, for your approval. Nothing will be filed until you approve it.

Team / project: Entire / entire-cli · Parent: ENT-3132 · Related: ENT-3688, ENT-3141 · Assignee: you · Priority: none (your call)

Title: A packed checkpoint ref under a differently cased shard is dropped from the push queue

Description:

On a case-insensitive filesystem (macOS APFS, Windows NTFS), a checkpoint ref can be stored under a shard directory whose case differs from the name the push queue records. While the ref is loose this works, because opening the queued name finds the file anyway. After git gc packs refs, packed-refs keeps the on-disk name, and the lookup by queued name is an exact string match, so it reports the ref as missing.

partitionLocalRefs then treats the ref as stale. Pre-push drops it from the queue without uploading it, nothing reports this, and the pre-push guard against sending held checkpoints no longer protects it.

How it happens, as described in checkpoint.ParseRef: writing a ULID checkpoint with an uppercase shard (e.g. 6B) when a legacy hex checkpoint already created the lowercase 6b/ directory puts the file in 6b/, while the queue records 6B/<id>.

Reproduction (on main at cea1b3cee8, and unchanged by ENT-3688):

1. Create a ref stored as refs/entire/checkpoints/AB/0000000000ab.
2. Run git pack-refs --all.
3. Queue refs/entire/checkpoints/ab/0000000000ab (its RefName).
4. partitionLocalRefs returns existing=[], stale=[…/ab/0000000000ab].

Possible fix: when the exact lookup misses, match the queued ref to a listed ref by checkpoint ID (ParseRef already compares shards case-insensitively) and use the on-disk name for push and tip reads. ENT-3688's for-each-ref listing already returns the on-disk names. Also check whether other ref readers that look up by RefName (resolve, read, delete) have the same problem once refs are packed.

Not yet known: how often the shard collision happens in real clones, and whether any user has lost an upload this way.

Approve the ticket text as is, or edit it? I'll report on the implementer as soon as it finishes.

Yes, it's realistic. I reproduced it end to end with plain git on this Mac, but it only affects some clones, and this one isn't among them.

What I verified

1. The collision happens on a real disk. I wrote a legacy hex ref under 6b/ and a ULID ref under 6B/. The ULID file landed in 6b/:

While the ref is loose, looking it up by its queued name 6B/… finds it. After git pack-refs (what git gc does), the same lookup gives NOT FOUND, even in git itself.

2. Pre-push really drops it. The flush looks each ref up by its exact queued name. Not-found is classified stale and removed from the queue with RemoveEntries (manual_commit_push.go:567-570). Nothing is uploaded and nothing is reported.

3. It has happened to someone. Issue #2402 is an outside user's report from macOS of this exact shard collision. It made checkpoints vanish from checkpoint list/explain. PR #2403 fixed only that read side (ParseRef); the exact-name lookups were left alone.

Who it can hit

All of these must be true:

• macOS or Windows (case-insensitive disk).
• A clone holding old hex checkpoints as well as new ULID ones, for example one migrated from the old v1 branch.
  • This clone doesn't qualify: all 3,590 of its checkpoint refs are ULIDs, with no mismatches.
  • origin doesn't either: 38 refs, all ULIDs.
• A ULID whose last two characters are both 0-9/A-F, with at least one letter. That's about 15% of IDs in theory, and 7 of the 38 on origin. With enough legacy history, the matching lowercase folder almost always already exists.
• Refs packed before the checkpoint is pushed. gc does pack refs (this clone has 3,605 of 3,611 packed). Under automatic, a new checkpoint usually uploads before gc runs. Under manual, checkpoints can wait for days, so the new push policy makes this noticeably more likely.

So it can't happen in a fresh clone, but it's a real silent upload loss for older macOS users, and more likely on manual. It's worth a ticket, not a blocker for cli/1753.

Updated ticket draft

The changes from the previous draft are the field report, the end-to-end reproduction, and the note on manual.

Title: A packed checkpoint ref under a differently cased shard is dropped from the push queue

On a case-insensitive filesystem (macOS APFS, Windows NTFS), a ULID checkpoint whose shard (e.g. 6B) matches an existing legacy hex shard directory (6b/) is written into 6b/, while the push queue records 6B/<id>. This is the collision from #2402. While the ref is loose, the queued name still resolves. After git gc packs refs, packed-refs keeps the on-disk name, so an exact lookup of the queued name fails, including git show-ref --verify.

Pre-push then classifies the entry as stale and removes it from the queue (manual_commit_push.go partition, then RemoveEntries). The checkpoint never uploads, nothing reports it, and the pre-push guard against sending held checkpoints no longer protects it. #2403 fixed only ParseRef; exact-name lookups were not changed.

Who: macOS/Windows clones holding legacy hex checkpoints as well as ULIDs (for example, migrated from v1), for ULIDs whose last two characters are hex-like with a letter (about 15%), when refs are packed before the push. The manual push policy (ENT-3141) makes this likelier, because checkpoints wait longer before upload.

Reproduction (plain git, macOS):

1. git update-ref refs/entire/checkpoints/6b/a1b2c3d4e56b <commit>
2. git update-ref refs/entire/checkpoints/6B/01M6G1HN2W58GAHR0D9XX1296B <commit>. The file lands in 6b/.
3. git show-ref --verify refs/entire/checkpoints/6B/01M6G1HN2W58GAHR0D9XX1296B finds it.
4. git pack-refs --all, then the same command reports it is not a valid ref. partitionLocalRefs reports it stale, on main and with ENT-3688.

Possible fix: when an exact lookup misses, match by checkpoint ID against the namespace listing (ParseRef already folds shard case) and use the on-disk name. Also check the other RefName lookups (read, explain, delete) once refs are packed.

Unknown: how many clones hold mixed hex and ULID history, and whether anyone has lost an upload this way.

Should I file it with that text? The implementer agent is still working on the cli/1753 fixes.

Filed as ENT-3768: https://linear.app/entirehq/issue/ENT-3768/a-packed-checkpoint-ref-under-a-differently-cased-shard-is-dropped. It's in Backlog, assigned to you, under ENT-3132 and related to ENT-3688 and ENT-3141.

The other findings are already being fixed. The implementer agent is working through the approved plan in this worktree:

• MUST: a measured queue-size threshold, so a small queue doesn't pay for listing the whole history.
• SHOULD 1: tests that run every case through both paths and fail when the listing is broken.
• SHOULD 2 to 8: remove the dead memo, add one shared commit helper (which OPF now uses too), log git's stderr, use gitrepo.OpenPath in tests, log which filesystem case ran, fix the doc comments, and table-drive the subtests.
• Docs: the ref-checkpoint-backend.md sentence.

When it reports back, I'll check its claims and its sabotage test myself, rerun the benchmarks and mise run check, and have a fresh reviewer look at the result. Then I'll bring you the before/after, a commit message, and the Trail text correction for approval.

The fixes aren't done yet. The agent is still working and hasn't reported back.

It has edited all seven planned files so far: push_common.go, the three callers, both test files and the doc, about 210 lines added and 120 removed. Nothing is committed, and I haven't checked any of it yet.

Once it reports, I'll check its threshold numbers and sabotage test myself, rerun the benchmarks and mise run check, and have a fresh reviewer go over it. Then I'll bring you the results.

The base-vs-fixed benchmark is still running in the background. I'm holding the fresh reviewer until it finishes, so the reviewer's test runs don't add load and skew the timings. Verified so far:

• Diff: matches the approved plan, and each caller's error handling is unchanged.
• Sabotage check: reproduced it myself. Forcing the listing to fail now fails the listing-path subtests (SHA-256, packed, case-folded, broken files). Before these fixes, nothing failed.
• Tests: the related tests pass after I restored the file.

I'll report the benchmark numbers once it finishes.

All the findings are fixed in the working tree, nothing is committed, and I've checked the main claims myself. A fresh reviewer is going over the result now.

Before and after

Measured by me on the same harness: base cea1b3cee8 against the fixed tree, median of 3 runs, waitingCheckpointRefs.

Small queues are back to base speed (the MUST), and large queues keep the win.

What changed

• Threshold. The listing runs only when 100 or more refs are queued. Below that, the old per-ref path runs unchanged. The agent chose 100 by measuring where the two paths cross over: the listing wins from about 40 to 75 queued in mostly packed clones.
• Tests run every case through both paths and check the listed commits against git rev-parse. I reran the sabotage check myself: breaking the listing now fails four listing-path subtests (SHA-256, packed, case-folded, broken files). Before, nothing failed.
• Cleanup and tests:
  • the dead memo is removed;
  • one localTip helper is used by the safety check, the outgoing listing and OPF, with each caller's error handling unchanged;
  • a failed listing now logs git's stderr;
  • the new tests open repos through gitrepo.OpenPath and log which filesystem case ran;
  • doc comments are fixed and the subtests are table-driven;
  • the docs sentence is updated.
• The agent reports lint 0 issues and strategy tests passing. I reran the related tests after restoring from the sabotage check, and they pass.

Two things for you

1. A tradeoff to accept or reject. In a history that has never been packed (for example, 3,611 loose refs), a queue of 100 to about 600 is slower with the listing: the agent measured 76 ms against 40 ms at 100 queued. Raising the threshold to cover that case would make the 20,000-ref case much slower (570 ms against 21 ms). Clones get packed by gc (this one has 6 loose refs), so I'd accept it.
2. A remaining test gap. The end-to-end tests for the push safety check and entire status queue fewer than 100 refs, so they only exercise the per-ref path. The listing path is covered one level down, in TestPartitionLocalRefsUsing. Your rule is to write tests before the code, so I haven't added one after the fact. Should I add an end-to-end test with 100 or more queued refs anyway, as a regression guard for the safety check?

Once the reviewer reports, I'll bring you its findings, a commit message, and the corrected Trail text.

The fresh reviewer found no MUST items. It traced every path and confirmed:

• behaviour matches base: the same waiting and stale refs, fail-closed on unreadable refs, and each caller's error handling unchanged;
• the push safety check is no weaker;
• tests, go vet and lint pass.

It raised two SHOULDs, both test gaps.

The two test gaps

1. No caller-level test reaches the listing path.
   • These tests queue at most 3 refs, below the threshold of 100, so they only exercise the per-ref lookups. That includes the push safety check tests and the entire status (ListOutgoingCheckpoints) tests.
   • So localTip reading a commit from the listing is never run end to end. If it returned the wrong commit, the safety check could miss a held checkpoint and every test would still pass.
   • This is the same gap I mentioned before, now confirmed independently.
2. A failed listing is never tested. If a listing error dropped refs, nothing would notice. The fix is one more case in TestPartitionLocalRefsUsing: a cancelled context with the listing on, which should give the same result as the per-ref path.

Both would be written after the production code, which goes against your testing rule. I still recommend adding them: the first guards the safety check, the second guards fail-closed behaviour, and both are small. Should I add them?

For your approval

1. Tradeoff. In a history git has never packed, a queue of 100 to about 600 refs is slower with the listing (76 ms against 40 ms at 100 queued). Packed clones win from about 50 queued. Do you accept that?

2. Commit message:

3. Trail #2630 Context corrections. Three paragraphs in the cli/1753 section change. While checking, I found that one of them is also wrong about automatic: entire status reaches this code under any policy.

• Replace "The fix lists refs/entire/checkpoints/ once with git for-each-ref, …" with:

  For a queue of 100 or more refs, the fix lists refs/entire/checkpoints/ once with git for-each-ref, names and commits together, against the opened repository's common dir, and matches the queue against it in memory. A shorter queue keeps the per-ref lookups, because the listing reads every checkpoint ref in the clone, not just the queued ones. The push check and the outgoing listing take each commit from the listing when there is one. go-git's own enumeration was measured and rejected: it reads every ref in the repository and was no faster for loose refs.

• Replace the "Trade-off: one local git process, about 5 ms, …" paragraph with:

  Small queues are unchanged: with 2 queued in a clone of 3,611 checkpoint refs, 0.75 ms before and 0.59 ms after. Trade-off: in a history git has never packed, a queue of 100 to about 600 refs is slower with the listing than ref by ref (76 ms against 40 ms at 100 queued with 3,611 loose refs). Clones that gc has packed are faster with the listing from about 50 queued.

• In Verification, replace "They cover queue order and duplicates, …" with:

  Every scenario runs through both the listing and the per-ref path and must give the same result: queue order, packed refs, broken ref files, case-folded shards and a SHA-256 repository. Breaking the listing fails those tests.

Approve each of the three, or tell me what to change. I'll commit, run mise run check, and update the Trail in that order. Nothing gets pushed without your separate approval.

I've read the thread, your finding, and new measurements from this real clone. One of the results changes the fix itself: the threshold of 100 is too high for real clones.

The Slack thread

Soph is upstreaming prefix-scoped ref iteration to go-git (go-git PR #2424), so go-git can list only refs/entire/… without loading every ref. Hidde's reviews found real problems with it:

• the native and fallback stores give different results for the same repo;
• the new iterators added a Close obligation, and abandoned calls leak file descriptors;
• the "sorted" flag in packed-refs is trusted without being checked;
• a zero-hash ref could read back the wrong tip.

Two side PRs (#2434 and #2435, transactional iteration and writing a sorted packed-refs) have merged. #2424 itself is still open and isn't in our pinned go-git v6.0.0-beta.1. Hidde also notes go-git still reads packed-refs line by line, ignoring the sorted header. That's the root of the slowness here.

For us: once #2424 lands, an in-process prefix listing becomes the clean answer. Until then, repo.References() walks every ref in the repo, and git-safety.md already keeps pre-push ref listing native for that reason.

Your finding: use go-git repo.References() instead of for-each-ref

Measured on this real clone (5,048 refs, 3,613 of them checkpoint refs, 4 GB pack, packed-refs 456 KB). Averages of 7 runs, noisy:

What this shows:

• The two listings cost about the same. go-git is a little faster at some sizes, for-each-ref at others, and neither clearly wins. Starting git in this repo has an 18.5 ms floor, and asking for each ref's commit adds about 20 ms. go-git avoids starting a process but walks all 5,048 refs.
• Your process-cost point is right for the committed version. With 1 to 3 queued, it paid 26 ms against 1.7 ms. The threshold already in the working tree removes that.
• go-git is more fragile. One empty or truncated loose ref anywhere under refs/ makes References() fail, which a crash mid-write can leave behind. We'd then fall back to per-ref lookups, and on this clone 3,000 queued takes 14.8 s that way. for-each-ref skips the broken ref and keeps the fast path.
• Your other points hold: the hand-built --git-dir and environment handling, the second parser next to findLocalCheckpointRef, and List already using References().

My recommendation: keep for-each-ref for now because it survives a broken ref, and move to go-git's prefix iteration once #2424 lands. I could also fold findLocalCheckpointRef onto the same helper to remove the duplicate parser.

What this changes in the fix

A threshold of 100 is wrong for real clones. In this clone, each per-ref lookup costs about 1.5 ms, ten times what my synthetic repos showed. go-git rescans the whole 456 KB packed-refs for every lookup. The listing already wins from about 20 queued:

• at 50 queued, per-ref is 74 ms against 20 ms for the listing;
• at 100, it's 109 ms against 19 ms.

So the 100 threshold leaves real clones slow between 20 and 100 queued. I'd lower it to about 16, measured on this clone. Below that, per-ref is still faster (1.7 ms at 2 queued). Clones that have never been gc'd would pay more around 16 to 500 queued, but this clone shows gc does pack refs.

Decisions for you

1. Your finding: keep native, with the measurements above as the reply, or switch to go-git References()?
2. Threshold: lower it from 100 to about 16, based on the real-clone numbers?
3. The two test gaps from the reviewer, and the earlier commit/Trail approvals, are still waiting.

If you'd like, I'll draft the reply to the finding for you to approve before I post it.

What the thread is about

Background: Entire's CLI reads git data in two ways:

• go-git, a Go library that reads .git directly inside our process;
• the git program, run as a separate process.

Over time Entire has moved more reads onto go-git, because skipping a process start saves time.

The gap Soph hit: go-git has no "give me only the refs that start with X". If you want just refs/entire/checkpoints/…, go-git reads every ref in the repo (branches, tags, remote branches, all checkpoints) and you filter afterwards. Looking up a single ref is also slow when refs are packed: go-git reads packed-refs from the top for each lookup, even though git keeps that file sorted.

In the notebook comparison from earlier: go-git has no index and no way to say "only show me the pages for this section". It reads the whole notebook every time.

What Soph proposed upstream, in go-git itself:

• #2424 (still open): prefix-scoped iteration, "list only refs under refs/entire/" without loading the rest. Later it also used the sort order to stop early.
• #2435 (merged): when go-git writes packed-refs, it now sorts the file and says so in its header, the way git does.
• #2434 (merged): a fix so go-git's transactional storage lists each ref once, the same way a direct lookup sees it.

What Hidde's reviews found: mostly cases where the new fast path gives a different answer from the old one:

• Different refs for the same repo: one storage backend returned 1 branch and another returned 4.
• Broken refs handled differently: a half-written ref file (what a crash leaves behind) used to cause an error, but the new code quietly returned a shorter list. Repacking can act on that list, which could delete objects.
• Trusting the "sorted" header without checking it: a file that falsely claims to be sorted made go-git return no branches, with no error.
• Smaller issues: a zero-hash ref could return a stale value, iterators leaked file descriptors, and one loop was quadratic and could be exploited for a denial of service.

Soph fixed several rounds of these. Two side PRs merged, and the main one (#2424) is still open.

Hidde also pointed out that writing a sorted packed-refs doesn't make go-git read any faster yet. Its lookup still ignores the header and scans from the top.

Why it matters here

1. It's the same problem we're solving. Our original slowness comes from go-git rescanning packed-refs for every queued ref. On this real clone that's about 1.5 ms per lookup, so 3,000 queued refs take about 15 seconds. We worked around it by asking the git program to list the namespace once, because go-git can't do a prefix listing. Once #2424 lands, the clean fix is an in-process prefix listing: no extra git process, no --git-dir or environment handling, no output parsing.

2. It answers your finding. You suggested repo.References(). That's exactly the "read every ref, then filter" path #2424 exists to replace. It works, and on this clone it costs about the same as for-each-ref. But one broken ref file anywhere makes it fail, and we'd fall back to the slow per-ref path. for-each-ref skips the broken ref and carries on. So keeping native for now and switching once #2424 ships follows the reasoning in that thread.

3. Hidde's findings are a checklist for us. Fast ref listings tend to break on half-written ref files, refs that resolve differently by name than in a listing, and trusting file metadata without checking it. Our change guards against those by checking every ref missing from the listing with the old lookup, and never marking a ref stale unless git says it doesn't exist. When we adopt go-git's prefix iteration, the same both-paths tests (listing vs per-ref) should catch any of those differences.

4. It may make the follow-up bug easier to fix. ENT-3768, the case-folded shard dropped from the queue, comes from a by-name lookup in packed refs. A prefix listing gives the real on-disk names, which is the fix that ticket proposes.

Short version: the thread is Entire trying to fix, inside go-git itself, the same slow ref reading we're working around here with a native listing. Until it lands, our approach is the right stopgap. Once it lands, we should switch.

Yes. It's already being worked on, and go-git is the better long-term home for it, but I wouldn't wait for it.

Why go-git is the better home:

• It fixes the root cause: go-git can't read just a slice of the refs.
• We'd drop the separate git process, the hand-built --git-dir and environment handling, and our own output parser.
• Every part of Entire that reads refs would get faster, not only this one.

Why not wait:

• #2424 has been open since September and has been through several review rounds that each turned up new correctness problems. There's no merge date.
• After it merges, it still has to ship in a go-git release and we'd have to upgrade to it. We're on v6.0.0-beta.1.
• Meanwhile, manual-policy users pay seconds on every push today. Our fix works now, and the fallback keeps it no less safe than before.

How the two fit together: our change is written so the swap later is small. Only listLocalCheckpointRefs, the one function that does the listing, would change to use go-git's prefix iteration. The threshold, the fallback for unlisted refs, and the both-paths tests stay. Those tests are exactly what would catch the go-git differences Hidde found, such as broken refs or a mismatched ref set.

What I'd do now:

1. Reply to your finding: keep native for now, with the measurements, and note the switch once #2424 lands.
2. Track the switch: a Linear ticket, "Use go-git prefix ref iteration for checkpoint listing once go-git#2424 ships", linked from the Trail's follow-ups.
3. Optionally, check with Soph that this matches the plan for #2424, and whether refs/entire/checkpoints/ is one of the namespaces they have in mind.

Should I draft the finding reply and the ticket for your approval? The threshold decision (100 → about 16) and the earlier approvals are still open as well.

Here's the one-liner reply, in your voice:

i can look into moving this to go-git once its prefix iteration (go-git#2424) lands, but i didn't want manual-policy users paying the delay on every push until then.

Should it be a reply that leaves the finding open, or a resolve note that closes it? I'll post it only once you approve the exact text.

Here's a softer version:

this makes sense, and i'd be happy to look at moving it to go-git once prefix iteration (go-git#2424) lands. for now i kept the native listing so manual-policy users don't hit a delay on every push in the meantime, but open to other ideas if you see a better way.

Your Agent Memory asks for no pleasantries in review replies. If you'd rather follow that strictly, drop "this makes sense, and":

i'd be happy to look at moving this to go-git once prefix iteration (go-git#2424) lands. for now i kept the native listing so manual-policy users don't hit a delay on every push in the meantime, but open to other ideas if you see a better way.

Which version do you want? And should I post it as a reply that leaves the finding open, or as a resolve note? Since it ends with "open to other ideas", a reply that keeps it open fits better.

Which part of the finding do you want to fix here? It could mean two different things, and they lead to different code:

A. Do what the finding says: switch the listing to go-git's repo.References().

• Removes the git process, the hand-built --git-dir and environment handling, and our parser.
• About the same speed on this clone: 16 to 74 ms against 19 to 66 ms for for-each-ref.
• Risk: one broken or half-written ref file anywhere under refs/ makes the whole listing fail. We'd then fall back to per-ref lookups, which take 14.8 s at 3,000 queued on this clone.

B. Keep for-each-ref and fix the finding's valid points here (my recommendation):

• Lower the threshold to about 16, so the usual 1 to 3 queued refs never start a process. That's the finding's main cost concern, and the real-clone numbers support it.
• Make findLocalCheckpointRef (checkpoint_delete.go:349) reuse listLocalCheckpointRefs. That removes the duplicate parser, and that function also gets the environment stripping it currently lacks.
• Then post the reply about moving to go-git once #2424 lands.

C. Both: try go-git first, fall back to for-each-ref if it errors, then per-ref. It's robust, but it's the most code, and two listings means more to maintain.

I'd go with B: it addresses everything in the finding except "use go-git", and the measurements give no reason to take on A's broken-ref risk now. If you pick B, I'll send a short pre-edit brief, then have the implementer agent do it tests-first, with the same verification as before.

The implementer agent is now working on option C in this worktree, tests first. Nothing will be committed or pushed.

When it reports, I'll verify it the same way as last time:

• reread the diff;
• rerun its sabotage checks myself;
• benchmark base against the fixed tree, small and large queues, plus the real-clone table;
• have a fresh reviewer go over the code.

The reply to your finding isn't posted yet. Once C lands it'll be resolved by the change, so the go-git text needs rewording. I'll draft a new version reflecting C for your approval.

Option C is in place and verified: same behaviour as base, safety check no weaker, tests and lint pass. One tradeoff needs your call before I commit.

What I verified myself

• Code: go-git lists first, for-each-ref takes over if go-git fails, and per-ref lookups run if both fail. Refs that are symbolic, unparsable or missing from the listing still get the old lookup, so commits match base.
• Sabotage: I made go-git return wrong commits. Four tests failed, including the new end-to-end ones: the push safety check with a queue over the threshold, and entire status. Both of the reviewer's test gaps are now covered.
• Results: the full strategy and docs tests pass, lint reports 0 issues, and only the 7 intended files changed.

The agent's numbers

These are on this real clone: 3,627 checkpoint refs, 47 loose, the rest packed. The machine was busy, so each figure is the best of 6 runs, in ms:

• The threshold is now 16. Below that, per-ref lookups are faster, so the usual 1 to 3 queued refs start no listing at all, which is the finding's cost point.
• With refs packed, go-git is clearly faster: 5,000 queued takes 1.6 ms, against 10 ms for for-each-ref and 5.1 s per-ref.

The decision: clones whose refs are all loose

In a history git has never packed (3,611 loose checkpoint refs), go-git's listing takes about 500 ms, against about 90 ms for for-each-ref. From 16 queued, those clones go from roughly 6 to 12 ms (the earlier working tree) to about 500 ms.

• Realistic? Rarely. This clone keeps 47 of 3,627 refs loose because gc packs them. The likely case is right after a migration that writes thousands of refs at once, until gc runs.
• Your finding already accepted this tradeoff: "go-git has no prefix filter, so the walk also reads loose branch, tag, and remote refs. That only matters in a repo with very many unpacked refs."

Options:

1. Accept it (my recommendation). It matches what the finding asked for and what's realistic. It goes away once go-git's prefix iteration (#2424) lands.
2. Native first, go-git as the fallback. That covers all-loose clones but gives up go-git's packed-ref speed, and the finding's "no subprocess".
3. Pick per clone by counting loose refs first. That adds its own directory walk and more code. I wouldn't.

There's also a small related point. On reftable repos, go-git's listing runs about three git commands over all refs: correct, but slower than for-each-ref. Reftable is rare. I'd leave it, but routing reftable straight to for-each-ref would take a few lines.

Once you decide, I'll run base-vs-fixed on my reproduction harness myself and send a fresh reviewer over it. Then I'll bring you the commit message, the updated Trail text, and the reply to your finding.

The base-vs-fixed benchmark is running in the background. I'll hold the fresh reviewer until it finishes, so the reviewer's test runs don't add load and skew the timings.

The fresh reviewer is now looking at option C. I've told it to hunt for any case where go-git's listing disagrees with a direct lookup, for example:

• loose refs shadowing packed ones;
• peeled lines in packed-refs;
• linked worktrees and reftable;
• partial walks.

I'll check what it finds, then bring you the commit message, the Trail text, and the reply to your finding, all for approval.

The reviewer found no MUST items for option C. It raised one SHOULD, and I've now confirmed it in Go: in one corrupt-repo case, the fallback reports a different commit than base.

What happens

It takes two damaged ref files at once:

1. an empty loose ref somewhere under refs/. go-git's listing then fails with "ref file is empty", so we fall back to for-each-ref;
2. a checkpoint ref whose file holds <hash> junk, a valid hash followed by trailing data.

For that second ref:

• base: go-git reads it as the zero hash (0000…) with no error;
• now: for-each-ref tolerates the trailing data and reports the real commit 9a6dc60d….

What that changes

Only the reported commit changes, and only in a repository that is already corrupt. Which refs count as waiting or stale is unchanged.

• Push safety check: stronger. It now knows the real commit, so it can refuse a push that sends it.
• entire status / list --outgoing: shows the real commit instead of zeros.
• OPF rewrite: base fails on that ref with an error, and now it would go ahead and rewrite it. This is the one real behaviour change.

The already-pushed commit (24d9b37141) has this difference too, since it used for-each-ref directly. So it isn't new to option C, just newly found.

Options

1. Keep it exact (my recommendation). When the for-each-ref fallback runs, use its list only for "does this ref exist", and look commits up one by one as base does. Commits then match base exactly. The slower lookups only happen in a repo that already has a broken ref, and they're no slower than base there.
2. Accept and document it. git's own reading arguably makes the safety check more correct. But it breaks your "preserve behaviour exactly" rule, mainly for OPF.

Shall I do option 1? It's a small change: a test that pins base's zero commit for the junk ref when the fallback runs, then the fix, then the same verification.

The reviewer also noted the doc could say that a malformed packed-refs line or an unreadable loose file also makes the go-git listing fail. I'd add that in the same pass.

While mise run check runs, here are the three texts for your approval. Nothing is committed or posted yet.

1. Commit message

2. Reply to your Trail finding

I'd post this as the resolve note, since the change now does what the finding asked:

switched the listing to repo.References() as you suggested, with for-each-ref kept only as a fallback when go-git's walk fails, and its list then only decides which refs exist so tips still match a direct lookup. queues under 16 keep the per-ref lookups, so the usual 1 to 3 refs don't list at all. numbers from a real clone are in the trail description, and once go-git#2424 lands the prefix iterator could replace the full walk.

3. Trail #2630: replace the cli/1753 section

This is the whole section, rewritten with the new design and numbers. The rest of the Context stays the same.

I'll only claim mise run check passes in that text once the current run actually passes. Approve each of the three, or tell me what to change. After approval, the order is commit, then the Trail write, then resolving the finding. Pushing the new commit is a separate step that needs your approval.

Approved. I'll wait for mise run check to finish before doing anything, since CLAUDE.md requires it to pass before every commit and the Trail text claims that it passes. If it passes, I'll commit, update the Trail, and resolve the finding, in that order. If it fails, I'll stop and show you the failures.

Nothing is committed yet, because the first full mise run check failed on the source guard TestHashComparisonsUseEqual.

• Cause: the guard's text pattern matched ref.Type() == plumbing.HashReference in the new go-git lister. That line compares a reference type, not a hash, so it was a false positive. I left the guard as it is.
• Fix: I removed the type check. In go-git a symbolic ref carries no hash, so the existing !ref.Hash().IsZero() check already skips it. Behaviour is unchanged: the symbolic-ref test passes on both paths. I also updated the comment above the function.
• Status: the guard, the strategy tests and lint all pass. The full mise run check is running again on the final tree.

This one-line change is the only difference from what you approved, and the approved texts still describe the change accurately. If the check passes, I'll commit, update the Trail, and resolve the finding. If it fails, I'll stop and show you the failures.

No, nothing has been pushed yet. The finding fix (go-git listing, the for-each-ref fallback, the 16 threshold) is still uncommitted in the worktree. The branch on the remote still has only the original commit 24d9b37141.

Here's where it stands:

1. Commit: waiting for the mise run check rerun, which is still running after the guard fix.
2. Push: not done. Pushing needs your separate approval.
3. Trail update and resolving your finding: these should come after the push, not before. Otherwise the finding would be marked resolved, and the Trail would describe code reviewers can't see yet.

So the order is: the check passes, I commit, you approve the push, I push with work publish, and then I update the Trail and resolve the finding. Do you want me to push straight after the commit once the check passes, or wait for you to look first?

It's been running about 6 minutes and is now on the final step, the Vogon E2E canary. The earlier full runs today took 7½ to 11 minutes, so I expect it to finish within the next 2 to 5 minutes.

So far no package has failed: 104 test packages have reported, none with a FAIL. I'll tell you as soon as it finishes.

The finding fix is committed as 923d278b82 but not pushed yet. mise run check passed on the final tree.

The publish plan:

1. git push -u origin muskan/ent-3688-count-waiting-checkpoints-in-one-listing. Only the new commit is new; 24d9b37141 is already on the remote. It's a normal push, not a force push.
2. Change cli/1753 already exists and follows the branch, so no new Trail or link is created.

After the push, in this order:

1. Replace the cli/1753 section of Trail #2630 with the text you approved, using the same re-read and read-back checks as before.
2. Resolve your finding with the approved note.
3. Check cli/1753's findings and Entire Gates again once the reviewers run on the new commit.

Should I push?