The 125M model's scaled-up, more honest successor.
Trained from a random initialization on 14.7 billion tokens across five sources of US case law, SEC filings, and educational web text — with a staged data schedule, not a flat blend.
Built end-to-end by Deependra Verma — Generative AI Researcher, AI Engineer, and AI Team Lead
Ask it a question
Fine-tuned on the exact same 21,186-pair SFT datasetas the 125M model — CUAD contract clause extraction, LEDGAR clause classification, and distilled legal/financial Q&A — so any difference in behavior below comes from the bigger, better-pretrained base, not different fine-tuning data. Runs entirely in your browser via WebAssembly; expect a larger download and slower generation than the 125M model.
runs entirely in your browser
Ask a legal or financial question. Pick a starter below or type your own.
A 528.5M fine-tuned model. Meaningfully better than the 125M model, but still not reliable closed-book — see the comparison below. Not legal or financial advice.
Bigger, and honestly measured
Both models answering the identical 1,059 held-out questions, scored the same way — extraction/classification against ground truth, general Q&A judged by an independent local Meta-Llama-3.1-70B-Instruct.
| Task | 125M | 500M |
|---|---|---|
| CUAD clause extraction (F1) | 0.711 | 0.761 |
| CUAD refusal accuracy | 90.1% | 87.7% |
| LEDGAR classification (exact-match) | 74.1% | 77.4% |
| Case-law Q&A (closed-book) | 35.6% | 42.2% |
| SEC filings Q&A (closed-book) | 43.1% | 46.9% |
| General web Q&A (closed-book) | 18.2% | 30.3% |
The 500M model wins net in every category, but not as a clean sweep — e.g. on CUAD, 46 questions flipped from wrong to right, but 25 flipped the other way. One concrete regression: asked what changed conditions a defendant cited, the 125M model correctly listed them from the source text; the 500M model gave a vaguer, legal-sounding but non-responsive answer. Bigger and better on net does not mean strictly better on every question.
Held-out perplexity, per source
The pretraining base model, not the fine-tuned one — measured on shared domains from the 125M model's own validation set, so the improvement isn't a val-set-composition artifact.
A bigger model, honest accounting
Honesty note on the aggregate perplexity: it looks like a big win partly because one bulk source (SEC-EDGAR) is ~61% of the val set by window count and is inherently easier to predict (repetitive, formulaic filing text) than case law or general web text. The per-source numbers above — where this model beats the 125M model on all three shared domains — are the trustworthy signal, not the aggregate alone.
Same recipe, scaled up
- Architecture
- Llama-style decoder
- Layers · dim · heads
- 24 · 1152 · 18
- Head dimension
- 64 (multi-head)
- Context length
- 1,024 tokens
- Positional
- RoPE (θ = 10,000)
- Normalization
- RMSNorm (1e-5)
- Activation
- SwiGLU (silu), intermediate 4,608
- Vocabulary
- 16,384 byte-level BPE (shared w/ 125M)
- Embeddings
- tied input / output
- Precision
- bfloat16
Pretrained on 8 × B200 GPUs (on-prem) · bfloat16 compute · shares the 125M model's tokenizer
Five sources, staged — not flat
A flat/proportional mix of these five sources was tried first and measurably diluted the original case-law/SEC/fineweb-edu identity down to ~13% of training data — fineweb-edu perplexity got worse (22.82 vs 20.73) as a direct result. The fix, adopted from MiniCPM's and SmolLM2's published recipes: a staged (Warmup-Stable-Decay) schedule — broad, balanced sampling for 85% of training, then a decay phase for the last 15% that upweights the curated core specifically. The donut below shows the time-averaged sampling weight across both phases.
- 27%SEC-EDGAR filings (bulk)TeraflopAI/SEC-EDGAR
- 24%US case law (bulk)common-pile/caselaw_access_project
- 17%Educational webfineweb-edu
- 17%SEC 10-K filingsPleIAs/SEC
- 15%US case lawHFforLegal/case-law
What this is, and is not
Open-book contract tasks are the trustworthy mode — give it an excerpt, and extraction/classification is measurably better than the 125M model across the board.
Closed-book general Q&A improved but is still fundamentally limited.A 528.5M-parameter model can store at most ~2 bits of knowledge per parameter (Allen-Zhu & Li, "Physics of Language Models: Knowledge Capacity Scaling Laws") — a hard capacity ceiling, not a training defect.
Confident fabrication is real and was directly observed, not hypothetical — invented specifics contradicting the real source, and the vague non-answer regression cited above. Never rely on its output as legal, financial, or factual advice.
Who built this, and why
Deependra Verma — a Generative AI Researcher, AI Engineer, and AI Team Lead — built this as the scaled-up successor to legal-slm-125M ↗, to test whether more parameters and more (better-balanced) legal/financial data would produce a genuinely stronger model, not just a bigger one. Every number on this page — including the ones that don't flatter the 500M model — comes from a real training or evaluation run.
Questions worth answering honestly
What's actually different from the 125M model?
More parameters (528.5M vs 125.8M) and more, better-balanced training data — 14.7B unique tokens across 5 sources vs 2.04B across 3, mixed with a staged (Warmup-Stable-Decay) schedule instead of a flat blend. Both share the exact same tokenizer and the exact same SFT dataset, so the comparison below is apples-to-apples.
Is it actually better, or just bigger?
Better on every shared metric measured — but not a clean sweep on every individual question. A direct, paired comparison on the identical 1,059 held-out questions found real regressions alongside the improvements (see the comparison table below). Bigger and better on net does not mean strictly better on every question — that nuance is documented, not glossed over.
Should I trust its answers?
For open-book tasks — give it a contract excerpt and ask about it — yes, that's the reliable mode (CUAD F1 0.761, LEDGAR 77.4%). For closed-book questions with no source text, no — it still confidently fabricates specifics on unfamiliar prompts, a hard capacity limitation at this parameter count, not a bug. Never use its output as legal, financial, or factual advice.