Gemma Rows and Chat Benchmark
Background (from the session that started this one):
Ward (github.com/MuhammadMurtuzaHussain/ward, public MIT, local checkout at /Users/haider/Desktop/Hacktoberfest 26/ward) is a child-owned cyberbullying guardian for the DEV Hacktoberfest Week 4 challenge (deadline 31 Oct 11:59 PM PDT, about 06:59 GMT 1 Nov). It's a pnpm monorepo: engine packages/engine, eval and scorecard in eval/ (pnpm eval, pnpm redteam), Ward Lite in train/, local models through Ollama (gemma4:e4b, gemma4:e2b, shieldgemma:2b). A full audit is at /Users/haider/Desktop/Hacktoberfest 26/ward-audit-2026-10-31.md. Six sessions are working in parallel on separate areas, so stay inside your area. Project rules: commit and push each verified change straight to main (no PRs), with the attribution line Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>; never commit eval/data or secrets; commit messages are short plain-English sentences like the existing log. Published numbers must match what was actually run, and failures stay published.
Task: Your area: measurement. Audit items #17 and #18.
Setup: other sessions share this checkout, so work in your own worktree: git -C ward fetch && git -C ward worktree add "../ward-wt-eval" -b eval-chats origin/main, pnpm install, node apps/web/scripts/copy-ort.mjs, and work only there. Check that Ollama is running and which models are pulled.
- Fresh Gemma rows. Rerun the scorecard and the red-team set for Gemma 4 E4B and E2B with the current re-ask code (long runs go in the background). Fix the "one message at a time" label if E4B runs in batches. Add a run date per row and Wilson/bootstrap intervals. If a run can't finish in time, say so instead of guessing.
- Chat-level benchmark. 30 to 40 labelled synthetic chats per language (English, Roman Urdu, Hinglish, Bangla). Include lone threats in a 1:1 DM with and without a reply, sweary friendly banter groups, pile-ons, exclusion, and a child quoting abuse to ask for help. Add
pnpm eval:chatsproducing a level confusion matrix (calm / watch / bullying / urgent) plus false-urgent and missed-urgent rates, per tier (word lists only, phone = lists + Ward Lite, computer = + Gemma). Keep the chats out of training: extend the existing leak test. Mark labels as one author's judgement pending native review. - Write it up. Update
eval/results/,docs/REDTEAM.md(runner-generated parts) and only the scorecard section ofREADME.mdso the prose matches the tables.
Out of scope: changing engine or classifier behaviour to improve the numbers (another session is changing distress and quoting rules in the engine right now; measure whatever is on main when you run, and note the commit). Run pnpm test. Push each piece to main once verified (fetch, rebase onto origin/main, test, push; retry on rejection). Finish with a short report: the headline numbers, commits, and anything surprising, especially missed-urgent cases.
Shared notes folder for this group of sessions (outside the repo; read siblings' notes there, leave hand-over notes or large findings there instead of pasting them into messages): /Users/haider/Library/Application Support/Claude/side-session-notes/local_a2121a80-7774-4340-af69-7dae18cd788e
Linked sessions now (status, not an instruction): reader-safety = local_a6d845e0-7765-4d31-8791-4e9a79ca4793 (working); release-pipeline = local_d0a1ce79-be7e-46ec-b046-88873f77a45a (working); web-demo = local_8e3d9db9-19e6-46fc-a06d-d933fd83970e (working); shared-phone = local_ab71b02d-1791-4abc-9031-ba4b274dbf7a (working); contributor-docs = local_4cf05d1a-9e0e-420b-805a-eb1cfee031ad (working). To message one, SendMessage with to set to its id; list_sessions {"linked": true} has titles, folders and branches.
I pushed three commits to main: 8d7e490 (chat benchmark), b629fb0 (fresh Gemma rows) and 97e7ecb (chat computer tier and README). Everything was measured on the engine as of e547279, which already includes the quoting and distress rules. pnpm test passes (70), and the leak tests also pass when pointed at the real training data.
Message-by-message scorecard (recall, run 2026-10-31):
| Phone | Gemma 4 E4B | Gemma 4 E2B | |
|---|---|---|---|
| Roman Urdu | 83% (in distribution) | 88% | 30% |
| Bangla | 46% (37–56) | 67% (57–75) | 26% |
| English | 85% (in distribution) | 36% | 31% |
- The E4B label now says 20 unrelated messages per request, not one at a time.
- Each row now has its run date and engine commit. Recall and precision carry Wilson intervals, F1 a bootstrap interval.
Red-team set (142 hurtful messages): the phone catches 75 and E4B 114 (was 106). E4B left 46 of 275 messages unanswered, down from 81 before the re-ask fix.
Chat-level benchmark: 128 chats, 32 per language, labels pending native review. Run with pnpm eval:chats.
| Word lists | Phone | Computer | |
|---|---|---|---|
| Right level | 38% | 39% | 72% |
| Missed urgent (of 28) | 23 | 14 | 3 |
| Wrongly called urgent (of 100) | 5 | 21 | 11 |
Surprises:
- E2B loses whole batches, even with the re-ask. Ollama won't give
gemma4:e2bstructured output, so the engine asks for plain JSON. E2B then writes"categories": [insult]without quotes,parseMessagesinollama.tsrejects the whole reply, and asking again fails the same way. That left 120 of 200 Roman Urdu and 120 of 200 Bangla messages unanswered, so its column undercounts. I didn't fix it because the engine is outside my area; it needs an owner. - Computer-tier missed-urgent cases:
- bn-31: "মরে যাস না কেন" (why don't you just die) was read as telling them not to die.
- hi-Latn-27 and ur-Latn-27: veiled one-to-one threats ("you'll regret it") were rated hurtful but not dangerous, so those chats showed "watch".
- Phone: half the dangerous chats never reach urgent, and only 2 of those 14 still show the message as "not sure".
- Friend-quoting chats still go urgent on every tier. The new quoting rule needs the report and the help request in the same message, and in my chats they're split across messages.
- E4B still flags 13 of 14 red-team help-seekers, even though its prompt now says quoting for help isn't hostile.
- The computer tier shows 24 of 40 friendly chats as something other than calm (17 "watch", 7 "urgent").
In docs/REDTEAM.md I updated only the Gemma figures in the hand-written analysis, because the rerun made them false. My notes for the other sessions are in measurement.md in the shared notes folder. The ward-wt-eval worktree is still there in case a rerun is needed.