Case study · Open-source AI
Open Vernacular AI Kit
A production-first preprocessing layer that makes AI systems more reliable for people who write in more than one language at once.
- Type
- Open-source toolkit
- Problem
- Code-mixed, multi-script user text
- Languages
- Gujarati reference, Hindi beta
- Ships as
- API, Docker service and CLI
The challenge
Real user text is messy.
People don't write in one language, one script or one style. A single message can mix Gujarati and English, romanized words and native script, shorthand spellings and dialect. AI pipelines that look great in demos break on it in production.
That mess directly hurts:
- Retrieval quality
- Intent routing
- LLM response reliability
- Analytics consistency
The approach
Normalize once, improve everything downstream.
Instead of making every component cope with linguistic chaos, a dedicated layer cleans the text before retrieval, routing and prompting. It doesn't replace your model: it helps the model stack you already have perform better on real vernacular input.
Input "che gujarat kayu nu paatnagar"
Step POST /normalize { lang: "gu" }
Output canonical native script, plus code-mix signals
What we built
Engineered like production infrastructure.
Three API endpoints
/normalize, /codemix and /analyze, for canonical text and the language signals behind it.
Service mode
A Dockerized service that drops in ahead of any retrieval, routing or LLM call.
CLI and recipes
Copy-paste integration examples, RAG preprocessing recipes and batch processing for CSV and JSONL.
Safe evolution
Schema versioning and backward compatibility tests, so upgrades don't break the systems that depend on it.
Language packs
A language-pack interface with fail-safe fallbacks, so new languages are added without bloat.
Benchmarks
Benchmark snapshots and evaluation slices, built in public with transparent benchmarks.
Inside the toolkit
See every decision it makes.
How it's measured
Three numbers it's held to.
Transliteration success
How often romanized words become the right native-script form.
Dialect accuracy
How reliably dialect-heavy input is recognized and handled.
p95 latency
Kept low, so it can sit in front of every request in production.
What it means for your project
The same discipline, on your AI.
Clear metrics, evaluations you can read, versioned interfaces and safe fallbacks are how we build every agent and assistant. And if your customers write in more than one language, we already know where AI pipelines break, and how to fix it.
Building AI for multilingual customers?
Tell us about your users and your stack. We'll show you how to make your assistants, search and routing work with the way people really write.