logit_staircase.py - Manual

Module: logit_staircase.py @version 1.3.0 @date 2026-11-01 Another collaboration between Andrew.human and Claude.ai

Version history

Version	Date	Change
1.0.0	2026-10-31	First release. Ran on the Clone (staircase_20261031-154509).
1.1.0	2026-11-01	Step-by-step mode: --gate N, --follow LEVEL, --stepwise. Default behaviour unchanged. Its 24-step self-test failed on the Clone at step 12 (a near-tie), so it never ran.
1.1.1	2026-11-01	Step-by-step mode now makes its own no-push baseline (baseline_stepwise.json) and compares against that. A difference from the original baseline is accepted only at a near-tie (--tie-gap, default 0.3), otherwise it stops. Mode printed on the console. --selftest-steps removed. Tested against a stand-in server only.
1.2.0	2026-11-01	New "repeated 4-word runs" measure (catches sentence loops that "longest repeat" misses; baseline reference is in the table). A biased first piece inside an already completed phrase is no longer a stray (new count "inside"). Summary table gains one column. Checked offline against the real v1.1.1 logs and end to end on the stand-in server.
1.3.0	2026-11-01	New --mid LO-HI: at every step, whatever the model ranks LO to HI gets the level bonus (candidates or not). One group, mid. Stand-in server test only.
1.3.0	2026-11-01	New --mid LO-HI: at every step, whatever the model ranks LO to HI (not mem-toks) gets the level bonus. Special tokens (`<	...	>`, e.g. end of text) are never pushed and end the reply. Tested on the stand-in server only.

What it does (plain words)

It asks LLM2 (Qwen3-4B on port 8082) the standard prompt again and again, each time pushing up the score of the first token of every mem-tok candidate by a fixed amount (+1, +2 ... +6). It also does the same with a control list of unrelated words. For each push it compares the reply with the no-bias baseline and reports where the reply first changed and how much damage was done. The question it answers: how big a push does the model tolerate before the reply breaks, and is the damage specific to mem-toks or happens to any word?

Everything runs at temperature 0, so a given setting always gives the same reply.

Requirements

• Python 3.6 or newer, standard library only (checked with vermin: minimum 3.2 syntax; actually tested on 3.11-3.13).
• llama-server reachable (default http://localhost:8082) with /health, /tokenize and /completion.
• Files in the working folder (or given by option): standard_prompt.txt, memtok_candidates.txt, qwen_llm2_std_120a.json (the verified no-bias baseline, temperature 0, 50 top probabilities, at least 120 steps). control_words.txt is found in the current folder or next to the script.

Usage

First look at the plan, generating nothing:

python3 logit_staircase.py --dry-run

Then the full run (6 levels x 2 groups = 12 runs, about 13 minutes if the prompt is cached; the first run may take about 150 s if not):

python3 logit_staircase.py

Options: --stepwise, --gate, --follow, --tie-gap (see Step-by-step mode below), --server, --prompt, --candidates, --control, --baseline, --levels 1,2,3,4,5,6, --groups mem,control (also mem_in = candidates already in the prompt, mem_new = candidates not in the prompt), --n 120, --nprobs 50, --flag-steps 20, --include-flagged, --dry-run, --outdir, --wait 120, --timeout 900.

The output folder name is new every run (default staircase_YYYYMMDD-HHMMSS) and an existing folder is refused. Ctrl-C stops cleanly and writes a partial summary marked PARTIAL (exit 130).

Output files

• plan.md - candidates, first-token ids, groups, flagged tokens.
• run_<group>_L<level>.json and .txt - compact log of each run (the .txt starts with "temperature 0").
• summary.md and summary.json - the tables to read first.

Checks before running

1. The prompt echoed in the baseline must equal the standard prompt, temperature 0, no bias, enough steps; otherwise it stops.
2. Health wait; unreachable server stops with a message.
3. Self-check of score reporting: some builds report scores before the bias, others after. The harness detects which and says so in the summary ("server reports scores before the bias: yes/no"); where scores are pre-bias it adds the bias back to give effective scores.

Groups

• mem: first token of each of the 29 candidates (with leading space).
• control: the first usable control words whose first token is not a candidate first token, not in the prompt and not flagged. Same count of distinct tokens as mem.
• Flagged: a first token that sits in the baseline top-50 at 20 or more of 120 steps (a common word piece). Excluded unless --include-flagged.
• Candidates already in the prompt by the phrase-level rule are 5: base rate neglect, survivorship bias, false positives, percentage of what, relative reduction.

Metrics per run

• First differing step vs baseline (cascade: after that, text differs, so compare log-probs, not text).
• Forced picks: steps where the chosen token is a biased token. Completed = the whole candidate phrase followed; stray = a first token alone.
• Hits in prompt / new: biased phrases that appear in the reply, split by already-in-prompt or not.
• Distinct-word ratio and longest repeat run (signs of breakdown).
• Inside: a biased first piece that is part of a phrase already completed (not a leak).
• Repeated 4-word runs: share of 4-word sequences already seen earlier in the reply. Compare with the baseline row.
• Mean and max log-prob shift vs baseline.
• Overlap with baseline text.
• Top-1 not chosen count.
• Candidate track: per-candidate effective score.

Step-by-step mode (v1.1.0)

Why: a fixed push lands on the same word piece at every step, and it only ever touches the first piece of a phrase (v1.0.0 result: first words leaked, no phrase completed). The server cannot make a push depend on what was just chosen, so this mode asks for ONE token at a time, sending the prompt as token ids plus everything chosen so far, and sets the push fresh at every step.

• --follow LEVEL - once the first k pieces of a candidate have been chosen, push piece k+1 by LEVEL for the next step. This is what can pull a whole phrase through. Example: --levels 3 --follow 8.
• --gate N - push a first piece (by the run's level) only on steps where it is already ranked N or better in an unpushed probe of that step. Costs one extra request on steps where something qualifies. Example: --levels 3,4,5 --gate 10.
• --stepwise - step-by-step with neither of the above (a check that it matches the normal mode).
• Either option switches the mode on. Run files are named run_<group>_g<gate>_f<follow>_L<level>.
• Before the runs it generates the full reply one token at a time with NO push and saves it as baseline_stepwise.json. Every step-by-step run is compared with that file, not with the original baseline, because generating one token per request rounds slightly differently from one long request and can flip a near-tie (seen on the Clone: step 12, gap 0.12). The new baseline is checked against the original: identical is fine; a difference where the original's two best scores were closer than --tie-gap (0.3) is reported as rounding; a difference anywhere else stops the run (token-id prompt does not reproduce the text prompt). A probe request then finds whether scores are reported before the push.
• Time: about 1.2x (follow only) to 1.5x (gate) the normal run, plus about 2 minutes for the no-push baseline.
• First real result (v1.1.1, --follow 8, no gate, mem only): +2 reads naturally and completes "relative and absolute risk"; +3 and +4 hijack at step 4 ("false positives") and loop (repeated 4-word runs 55% / 25% against 23% for the baseline). Next test: add --gate 10.
• Differences from the normal mode: "forced" and "completed" are counted the same way; the candidate-track "effective score" equals the reported score (the push varies by step, so it is not added back); mean shift is not meaningful.

Examples:

python3 logit_staircase.py --dry-run --gate 10 --follow 8
python3 logit_staircase.py --groups mem --levels 2,3,4 --follow 8
python3 logit_staircase.py --groups mem,control --levels 2,3,4 --gate 10 --follow 8

Middling-interest push (v1.3.0)

python3 logit_staircase.py --mid 3-15 --levels 1,2,3

At every step the harness asks the server for the scores with no push, takes the tokens ranked LO to HI (3rd to 15th here; the model's favourites 1-2 and the long tail are left alone; special tokens such as end-of-reply are skipped), and pushes all of them by the level. Then it takes the step. Reads: how the reply drifts when tokens the model finds "somewhat interesting" are promoted. Each run is one reply per level; read the .txt files.

Mid-range push (v1.3.0)

The blunt version of the experiment: forget the mem-tok list and, at every step, push the tokens the model ranks LO to HI (the "fair to middling" ones, plausible but not leading) by the level. Shows how far and how fast replies bend when plausible-but-not-leading words are favoured, whatever they are.

python3 logit_staircase.py --mid 4-30 --levels 1,2,3
python3 logit_staircase.py --mid 3-15 --levels 1,2,3,4

• Step-by-step: each step makes an unpushed probe, then a second request with the band pushed. About 2 minutes per level plus 1-2 minutes for the no-push baseline.
• Run files: run_mid_mid<LO>-<HI>_L<level>. Compare replies against baseline_stepwise.json. Forced/completed counts are zero by design (no candidates); read the reply text, "first differing step", "distinct-word ratio" and "repeated 4-word runs".
• --follow does nothing here (no candidates).

Limits (read before trusting numbers)

• The push is static: it applies to the same token at every step, so common first tokens are pushed everywhere (hence the flag).
• Only the first token of a phrase is pushed.
• Capitalised and no-space variants of a token are not pushed (risk 54).
• The control is not matched on word shape (risk 53 tuning on a tiny sample also applies to floor/weight in hotwords).
• The control list is fixed (40 words).
• Temperature is always 0.
• Step-by-step mode: a follow push applies after any matching first pieces, even if the word came up by chance; the gate probe ranks by the server's top-50 list only.
• v1.1.0 is mock-tested only; v1.0.0 normal mode has run on the real server.
• (v1.0.0 note) Mock-tested only. The real server may differ (score reporting mode, tokenisation), so run --dry-run first and check the plan.