I started this because I could not tell whether I should be fine-tuning at all. Everything written about training a model assumes you have already decided to, and the deciding is the part that costs a night and a GPU bill when you get it wrong. Most people who think they need a fine-tune need a better prompt, a retrieval step, or an evaluation loop run a hundred times. Nobody was going to tell me which, so I built the thing that would.
What it does is walkable and unglamorous. You attach a model — a local one, or any OpenAI-compatible endpoint — and describe what you want. It proposes an approach and names which of the three things it is, because pretraining, fine-tuning and a low-rank adapter are not interchangeable and the difference is most of the cost. It finds or generates data. It reads your hardware and prices the run against it. Then it either trains, handing execution to a pinned backend rather than reimplementing one, or it tells you not to.
The last clause is the product. Everything else here is available elsewhere and better funded.
Every refusal has cost me something I wanted
The one that stung is the flagship dataset. Sixty-nine rows, and the tool that carves a held-out set refused to touch it: an evaluation needs about thirty rows before it can resolve anything, which would leave thirty-nine to train on, and below roughly a hundred rows a fine-tune learns your examples rather than your task. The threshold is a constant in the code, not a judgement call made at the door. So the answer is that this dataset cannot be both trained on and honestly measured, and the fix is more data rather than a smaller split. I wanted the run. I did not get the run.
The publishing step refuses too, and the refusal is the half that has a test. Without a credential it exits with its own code and prints where to get one, rather than crashing or — worse — quietly writing nothing and reporting success. The test that covers it also checks that the token reader returns the name of an environment variable and never its value, which is the kind of thing you only write after imagining the version that logs a secret into a run record.
The refusal that withdrew my best result
Then the door that refuses leaked data, which cost me the only statistically resolved finding this project had produced.
A judged comparison had come back at nine discordant rows to two, a p-value of 0.0391, and the word resolved beside it. Running a leak check across the split afterwards found three of the twenty-four evaluation rows also present in the training file, at up to 0.944 similarity by exact Jaccard — and not one of them an exact string match, which is precisely why reading the files by eye had not caught it. Remove those three rows and the same comparison reads eight to two, p = 0.0703, and no evidence.
Nothing about the model changed. The only thing that changed was which rows were allowed to count. A result that survives only because three rows were on both sides of the wall was never a result, and I would have gone on quoting it, because it had a number and the number was under 0.05.
The check now runs inside the carving step, on its own output, and refuses a split it would have produced. That is the shape I want everywhere: a check that runs where the mistake is made rather than in a report written afterwards, because a report is read by someone who already believes the number.
What I actually think this is for
The honest summary is that this project has trained twenty-one adapters and has no verified win on held-out data. Both halves of that sentence matter. The machinery runs — real adapters, two model sizes, a full fine-tune-then-align pipeline, a preflight that declines a CPU-only install rather than running twenty times slower in silence — and the thing it is supposed to demonstrate has not happened yet.
There is a second result I am deliberately not putting a number on. Four-bit training landed recently and made a configuration fit that had previously exhausted an eight-gigabyte card. That is worth knowing and the figure is not on this page, because no run record on this disk carries the flag that would have produced it and the newest adapter predates the change. The shape is real; the number does not yet have an artefact, and a number without one is how the leaked-row result happened.
A harness that only helps you train will help you train the wrong thing extremely well. The one worth having reads your own numbers back and tells you the run you are about to start cannot answer the question you are asking. Mine has now done that to me three times, on days I would much rather it had agreed, and the third time it took a published result with it.
The repository is private, so this is a page you can read rather than a thing you can run — which means everything above is testimony, and the refusals are the part I would ask you to weigh it on.