Skip to content

Language Detection (LID)

Five language-identification models, from a 1.1 MB pure-Python classifier to 25-language neural models โ€” pick by coverage, latency, and accuracy.

from olaverse import detect_language

detect_language("Bawo ni, se daadaa ni?")   # โ†’ 'yor'

Who is this for?

  • Routing user messages to the right model, translator, or support queue
  • Filtering or labelling multilingual corpora at scale
  • The first stage of pipelines (e.g. Diacritizer(model="auto") uses LIDLite5 internally)

Which model should I use?

Need Model Install
Nigerian languages + English, zero GPU, instant LIDLite5 olaverse
Nigerian languages + English, best accuracy LIDNeural5 olaverse[deeplearning]
25 languages, CPU-only, sub-millisecond LIDLite25 olaverse[lid]
25 languages, best short-text accuracy LIDNeural25 olaverse[deeplearning]
Only the 4 Nigerian languages โ€” input never contains English LIDNeural5_1 olaverse[deeplearning]

LIDLite5 vs LIDNeural5

Both cover Yoruba, Igbo, Hausa, Nigerian Pidgin, and English.

LIDLite5 LIDNeural5
Size 1.1 MB JSON 484 MB (XLM-RoBERTa 125M)
Speed 0.014 ms/sentence 13.3 ms/sentence
Macro accuracy 98.12% 98.96%
Architecture TF-IDF + Logistic Regression Fine-tuned afriberta_large
Dependencies Pure Python transformers + torch

Model Cards: olaverse/lid-lite-5 ยท olaverse/lid-neural-5

from olaverse import LIDLite5, LIDNeural5

lite = LIDLite5()
lite.predict("Sannu, yaya kake?")            # โ†’ 'hau'
lite.predict_proba("Kedu, แป dแป‹ mma?")
# โ†’ {'ibo': 0.993, 'pcm': 0.003, 'eng': 0.002, ...}

neural = LIDNeural5()
neural.load()   # one-time download, cached
neural.predict("Kedu ka แป‹ mere today?")      # โ†’ 'ibo'

# Batched โ€” single forward pass, much faster on datasets
neural.predict_batch(["Bawo ni?", "Kedu แป dแป‹?", "How far?", "Sannu dai."])
# โ†’ ['yor', 'ibo', 'pcm', 'hau']

LIDLite25 / LIDNeural25 โ€” 25 languages

Coverage across Africa, Europe, and Asia. Both come in two checkpoints tuned for input length โ€” pick variant= to match your traffic:

variant= Use for
"passages" Documents, articles, paragraph-length text
"questions" (default) Search queries, chat messages, short user input

Model Cards: olaverse/lid-lite-25 ยท olaverse/lid-neural-25.1 ยท olaverse/lid-neural-25.2

from olaverse import LIDLite25, LIDNeural25

lite = LIDLite25(variant="questions")    # fastText, CPU, ~5-10 MB
lite.predict("What causes ocean tides?")   # โ†’ 'eng'

neural = LIDNeural25(variant="questions")  # XLM-RoBERTa-base
neural.load()
neural.predict_proba("What causes ocean tides?")
# โ†’ {'eng': 0.999, 'fra': 0.0003, ...}

LIDNeural25 is more accurate on short text (98.2% vs 97.3%) at the cost of needing transformers/torch.

Zulu/Xhosa confusion on short text

Both models score noticeably lower on Zulu/Xhosa short-text classification (F1 ~0.77-0.79) than every other language (โ‰ฅ0.98) โ€” the two languages are closely related with substantial shared vocabulary. Treat predictions between these two with reduced confidence on short input.

LIDNeural5_1 โ€” Nigerian-only

A compact (~31M parameter) classifier built on mist-encoder-base-ng, covering only Yoruba, Igbo, Hausa, and Nigerian Pidgin.

Model Card: olaverse/lid-neural-5.1

from olaverse import LIDNeural5_1

detector = LIDNeural5_1()
detector.predict("Ina kwana?")   # โ†’ 'Hausa'

No English class

Out-of-set input (English or any other language) will be confidently mislabelled, most often as Nigerian Pidgin. If your input may include English, use LIDNeural5 or the 25-language models instead.

Benchmarks

Model Size Speed Macro F1
LIDLite5 1.1 MB 0.014 ms 98.12%
LIDNeural5 484 MB 13.3 ms 98.96%

Per-language precision/recall and the 25-language numbers: Benchmarks โ†’

Applications

  • โœ… Chat & support routing โ€” send Hausa messages to Hausa-speaking agents
  • โœ… Corpus building โ€” label scraped text by language before training
  • โœ… Pipeline routing โ€” Diacritizer(model="auto") and other language-conditional steps
  • โœ… Content moderation โ€” apply the right language-specific rules
  • โœ… Search โ€” pick per-language analyzers and tokenizers at query time

API Reference

Full class reference: NLP & Tokenization โ†’ Language Detection