Bhasha v2 Enhances Tamil Identification in Mixed Text

The open-source model Bhasha identifies the language of each word in mixed Indian texts but initially confused Tamil and Telugu. The addition of annotated YouTube comments and a language pair index in v2 improves accuracy. The system also provides message summaries and usage tips, along with publication and access links.

Tamil–Telugu Confusions, Then Improvement in v2

Bhasha's evaluations show good overall performance but reveal that many Tamil words are misidentified as Telugu in human-annotated data. This error is linked to the predominance of written press-style Tamil in the training data, while spoken forms are common in comments. The language pair index (langs) option helps improve identification. With the addition of real annotated YouTube comment data at the word level, version v2 significantly enhances Tamil recognition.

Word and Message Outputs for Precise Uses

Language identification at the message level is not sufficient for certain tasks that require word-by-word labels. Bhasha assigns a language label to each word and generates, for each message, summaries that include the dominant language, the type of script used, and the distribution of languages. These results are suitable for applications such as transliteration, tokenization, translation modeling, code-mixing evaluation, and corpus filtering.

Training Set, Language Scope, and Access

Bhasha was trained on 1.65 million mixed sentences programmatically generated from public texts. It targets mixed Indian text, including Hinglish, Tanglish, Kanglish, and other combinations of Romanized and native scripts. Guidance on optimal usage contexts, lessons learned from error analysis in Tamil, as well as details on availability and links to try Bhasha and related models are provided.