Legal & financial language, learned from nothing.
Trained from a random initialization on 2.04 billion tokens of US case law, SEC filings and educational web text, then asked to keep writing.
Built end-to-end by Deependra Verma — Generative AI Researcher, AI Engineer, and AI Team Lead
Complete the passage
Give it the opening of a brief, a filing, or an opinion, then watch the model continue it, one token at a time. These are the real 125M model weights, generating live.
runs entirely in your browser
This is a base model: it continues text, it does not answer questions. It will confidently invent case names, citations and figures. Never rely on its output as legal, financial or factual advice.
Now ask it a question
Fine-tuned on 5,846 grounded legal & financial Q&A pairs, the same 125M model stops rambling and starts answering. It is a separate model, legal-slm-125m-sft ↗, that streams its reply as you watch.
runs entirely in your browser
Ask a legal or financial question. Pick a starter below or type your own.
A 125M fine-tuned model. It answers one question at a time and will confidently invent case names and figures. Not legal or financial advice.
Small model, honest accounting
Perplexity, falling
Held-out perplexity measured on a 20.6-million-token validation set the model never trained on. Two epochs, 38,890 steps, from a random start to 7.76.
A Llama, in miniature
- Architecture
- Llama-style decoder
- Layers · dim · heads
- 12 · 768 · 12
- Head dimension
- 64 (multi-head)
- Context length
- 1,024 tokens
- Positional
- RoPE (θ = 10,000)
- Normalization
- RMSNorm (1e-5)
- Activation
- SwiGLU (silu)
- Vocabulary
- 16,384 byte-level BPE
- Embeddings
- tied input / output
- Precision
- bfloat16
Pretrained on 8 × B200 GPUs (on-prem) · bfloat16 compute
Two billion tokens, hand-cleaned
Streamed from public datasets, then run through a deterministic pipeline: rule-based cleaning, an OCR-garble gate, MinHash-LSH near-duplicate removal, and 13-gram decontamination against the CaseHOLD and LexGLUE benchmarks.
- 35%US case lawHFforLegal/case-law
- 42%SEC filingsPleIAs/SEC
- 23%Educational webfineweb-edu
What this is, and is not
It is a base (pretrained) model, a next-token predictor. It has never been instruction-tuned, aligned, or shown a single question-answer pair.
It will fabricate case names, docket numbers, statutes and financial figures with total confidence. Everything it writes is fiction shaped like law.
English only, 1,024-token context, 125M parameters. It is a study in doing a lot with very little. It is not a product, and never legal or financial advice.
Who built this, and why
Deependra Verma — a Generative AI Researcher, AI Engineer, and AI Team Lead— designed and built legal-slm-125M end to end: the data pipeline, the byte-level tokenizer, pretraining from a random initialization, honest held-out evaluation, this live demo, and the supervised fine-tuning that turned it into a Q&A assistant.
It exists as a hands-on study in language-model engineering — proof of being able to take a model from nothing to a working, evaluated, deployed system, with every number on this page coming from a real training or evaluation run, not a projection.
Questions worth answering honestly
Who built legal-slm-125M?
It was designed and built end-to-end by Deependra Verma — a Generative AI Researcher, AI Engineer, and AI Team Lead — covering the full pipeline from data cleaning and tokenizer training through pretraining, evaluation, and fine-tuning.
What is legal-slm-125M?
A 125.8-million-parameter Llama-style decoder language model trained from a random initialization on 2.04 billion unique tokens of US case law, SEC filings, and educational web text, reaching a held-out perplexity of 7.76 after two epochs of pretraining (38,890 steps).
What's the difference between the base model and the fine-tuned (chat) model?
The base model (slm-125m-base) only continues text — it was never shown a question-answer pair and cannot answer questions. The fine-tuned model (legal-slm-125m-sft) was further trained via supervised fine-tuning on a teacher-LLM-distilled, judge-filtered legal/financial Q&A dataset, so it can actually answer questions, though both models still fabricate specifics on unfamiliar prompts.
Is the fine-tuned model production-ready or safe to rely on for legal or financial advice?
No. It answers reasonably on prompts similar to its training distribution, but like any 125-million-parameter model fine-tuned on a few thousand examples, it confidently invents case names, citations, and figures on out-of-distribution or adversarial questions. This is a research and portfolio project, not a source of legal, financial, or factual advice.
How was the model trained, and what did it cost?
Pretraining ran for two epochs on a from-scratch cleaned and deduplicated corpus, and the fine-tuning stage distilled a Q&A dataset from a teacher LLM, filtered by an LLM-as-judge. The compute-heavy phases (pretraining, evaluation, fine-tuning) ran on the author's own on-prem GPUs rather than rented cloud instances, and the fine-tuning dataset was built entirely on free-tier APIs — real out-of-pocket cost for this build was $0.