It doesn’t guess one word at a time. It fixes the whole sentence.
Hindi typed in English letters → Devanagari, decided sentence by sentence. The word model proposes spellings; a 2.77M-parameter neural Hindi language model chooses the best one for every word at once. 10.85 MB in total, and it runs inside this tab — nothing you type ever leaves your device.
Dakshina Hindi test, 4,976 sentences
95% CI 0.40–1.65 · significant
no server and no GPU
One-shot evaluation, every number in this page measured by us; v2 and v3 scored with the same harness on the same sentences (paired bootstrap, 2,000 resamples, seed 0).
Same sentence, two decoders
One click runs the v2 word decoder and the v3 sentence decoder on the same input, in this tab.
Sentences, measured
Dakshina Hindi sentence test (4,976 sentences) for word accuracy and whole-sentence exact match; internal casual-chat and context sets for tokens and sentences. Brackets are 95% paired-bootstrap intervals.
| Set | Sentences | v2 word % | v3 word % | v2 sentence % | v3 sentence % | Δ sentence |
|---|
Sentence exact match = every word of the sentence matches the reference (Unicode-normalised). Settings and weights were chosen on validation and internal dev data only; the test column is the single run after that.
What the sentence LM changes
Casual-chat dev set B, 480 held-out sentences, word accuracy per spelling pattern. Gains and regressions both shown.
| Pattern | v2 % | v3 % | Δ |
|---|
Small per-pattern counts (25–26 sentences each), so single rows are indicative; the overall devB gain (+0.68 tokens, CI −0.20 to 1.64) is not significant. The sentence test is, and so is the context set on words.
It reorders candidates. It never invents words.
Hard guardrail — checked on 132,299 output words
Every word in the sentence is a candidate the word model already produced for that typed word, or a token passed through untouched (punctuation, digits, kept-as-is Latin). The sentence LM can only choose between candidates; it cannot write a spelling the word model never proposed. Across every evaluation set — 132,299 output words — 0 fell outside their slot’s candidate list. The badge in the playground re-checks this on every keystroke.
10.85 MB, all in.
The word-only package is 7.92 MB. Sentence mode adds the neural LM and two small modules: +2.93 MB gzipped — 10.85 MB in total.
| Part | On disk | gzip -9 |
|---|---|---|
| Word model package (v2: core + 3M-word dictionary pack) | 16.26 MB | 7.92 MB |
nlm.q8 — sentence LM, 2,770,752 parameters, tokenizer embedded | 3.24 MB | 2.92 MB |
nlm.mjs — LM loader + inference | 8.4 KB | 3.1 KB |
v3.mjs — lattice search (beam) | 4.9 KB | 2.1 KB |
| Add-on total (v3) | 3.25 MB | 2.93 MB |
| v2 word package | 16.26 MB | 7.92 MB |
| Complete sentence package (v3) | 19.51 MB | 10.85 MB |
Smaller LM instead: nlm.q4 (4-bit) | 1.86 MB | 1.54 MB |
| Same system with the q4 LM | 18.14 MB | 9.46 MB |
gzip -9 sizes, measured (plus sent.mjs, 3.7 KB / 1.8 KB gz, the small glue module this demo uses to switch decoders and audit the guardrail). The 4-bit LM scores within tuning noise of the 8-bit one (val word accuracy 89.45 vs 89.35 at margin 4) and saves 1.39 MB (full package 9.46 MB); the 8-bit LM ships because it is the one the final evaluation was run with. This Space loads the 8-bit file.
Try one
Every preview is computed live by both decoders in your browser.
What sentence mode brings on-device
No server in the loop. Every example below is computed right now, inside this page.
Four lines of JavaScript
Load the word package, then attach the sentence LM. Both are static files; run them in a Web Worker as this page does.
import { AksharaLP } from "https://huggingface.co/sourojitdotcom/AksharaLP-Sentence-2.8M/resolve/main/aksharalp.mjs";
import { attachSentence } from "https://huggingface.co/sourojitdotcom/AksharaLP-Sentence-2.8M/resolve/main/sent.mjs";
const base = "https://huggingface.co/sourojitdotcom/AksharaLP-Sentence-2.8M/resolve/main/";
const aklp = await AksharaLP.load(base); // word package (7.92 MB gz)
attachSentence(aklp, await (await fetch(base + "nlm.q8")).arrayBuffer()); // + sentence LM (2.93 MB gz)
const r = aklp.transliterate("mai ghr pe hu"); // → मैं घर पे हूं
r.guard; // {ok: true, words: 4, outside: 0}
aklp.sentenceMode = false; // back to word mode: मैं घर पे हुआ
How it works
- 1Propose
The v2 word model (a 4-bit character Transformer, 989K parameters) proposes a short ranked list of Devanagari spellings for every typed word — unchanged from the word-only release.
- 2Re-score whole sentences
A 2-layer LSTMP language model with a tied embedding (2.77M parameters, 8,000 pieces) scores every candidate in the sentence it would form, and a beam search picks one candidate per word jointly.
- 3Stay inside the lattice
Only the word model’s own candidates for that word are ever considered, plus the v2 bigram LM and agreement rules. Punctuation, numbers and kept-as-is tokens pass straight through.
License & attribution
Free for commercial use. Credit AksharaLP and Sourojit Dhua when you ship it.
Commercial use allowed
Code, model weights and the sentence LM are released under the Apache License 2.0. The sentence LM was trained only on IndicCorp v2 Hindi (CC0) with our own code and tokenizer: it is commercially clean.
Attribution required
Apache-2.0 §4(d): every redistribution or derivative (apps, web pages, extensions, SDKs, fine-tuned or converted models) must include the NOTICE file. The runtime keeps its /*! … */ header, including when minified.
Transliteration by AksharaLP (Sourojit Dhua) — https://huggingface.co/sourojitdotcom/AksharaLP-Sentence-2.8M
Hindi typing powered by <a href="https://huggingface.co/sourojitdotcom/AksharaLP-Sentence-2.8M">AksharaLP by Sourojit Dhua</a>