Skip to content
/ inltk Public

Natural Language Toolkit for Indic Languages aims to provide out of the box support for various NLP tasks that an application developer might need

License

Notifications You must be signed in to change notification settings

goru001/inltk

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

97 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Natural Language Toolkit for Indic Languages (iNLTK)

Gitter Downloads

iNLTK aims to provide out of the box support for various NLP tasks that an application developer might need for Indic languages. Paper for iNLTK library has been accepted at EMNLP-2020's NLP-OSS workshop. Here's the link to the paper

Documentation

Checkout detailed docs along with Installation instructions at https://inltk.readthedocs.io

Supported languages

Native languages

Language Code
Hindi hi
Punjabi pa
Gujarati gu
Kannada kn
Malayalam ml
Oriya or
Marathi mr
Bengali bn
Tamil ta
Urdu ur
Nepali ne
Sanskrit sa
English en
Telugu te

Code Mixed languages

Language Script Code
Hinglish (Hindi+English) Latin hi-en
Tanglish (Tamil+English) Latin ta-en
Manglish (Malayalam+English) Latin ml-en

Repositories containing models used in iNLTK

Language Repository Dataset used for Language modeling Perplexity of ULMFiT LM
(on validation set)
Perplexity of TransformerXL LM
(on validation set)
Dataset used for Classification Classification:
Test set Accuracy
Classification:
Test set MCC
Classification: Notebook
for Reproducibility
ULMFiT Embeddings visualization TransformerXL Embeddings visualization
Hindi NLP for Hindi Hindi Wikipedia Articles - 172k


Hindi Wikipedia Articles - 55k
34.06


35.87
26.09


34.78
BBC News Articles


IIT Patna Movie Reviews


IIT Patna Product Reviews
78.75


57.74


75.71
0.71


0.37


0.59
Notebook


Notebook


Notebook
Hindi Embeddings projection Hindi Embeddings projection
Bengali NLP for Bengali Bengali Wikipedia Articles 41.2 39.3 Bengali News Articles (Soham Articles) 90.71 0.87 Notebook Bengali Embeddings projection Bengali Embeddings projection
Gujarati NLP for Gujarati Gujarati Wikipedia Articles 34.12 28.12 iNLTK Headlines Corpus - Gujarati 91.05 0.86 Notebook Gujarati Embeddings projection Gujarati Embeddings projection
Malayalam NLP for Malayalam Malayalam Wikipedia Articles 26.39 25.79 iNLTK Headlines Corpus - Malayalam 95.56 0.93 Notebook Malayalam Embeddings projection Malayalam Embeddings projection
Marathi NLP for Marathi Marathi Wikipedia Articles 18 17.42 iNLTK Headlines Corpus - Marathi 92.40 0.85 Notebook Marathi Embeddings projection Marathi Embeddings projection
Tamil NLP for Tamil Tamil Wikipedia Articles 19.80 17.22 iNLTK Headlines Corpus - Tamil 95.22 0.92 Notebook Tamil Embeddings projection Tamil Embeddings projection
Punjabi NLP for Punjabi Punjabi Wikipedia Articles 24.40 14.03 IndicNLP News Article Classification Dataset - Punjabi 97.12 0.96 Notebook Punjabi Embeddings projection Punjabi Embeddings projection
Kannada NLP for Kannada Kannada Wikipedia Articles 70.10 61.97 IndicNLP News Article Classification Dataset - Kannada 98.87 0.98 Notebook Kannada Embeddings projection Kannada Embeddings projection
Oriya NLP for Oriya Oriya Wikipedia Articles 26.57 26.81 IndicNLP News Article Classification Dataset - Oriya 98.83 0.98 Notebook Oriya Embeddings Projection Oriya Embeddings Projection
Sanskrit NLP for Sanskrit Sanskrit Wikipedia Articles ~6 ~3 Sanskrit Shlokas Dataset 84.3 (valid set) Sanskrit Embeddings projection Sanskrit Embeddings projection
Nepali NLP for Nepali Nepali Wikipedia Articles 31.5 29.3 Nepali News Dataset 98.5 (valid set) Nepali Embeddings projection Nepali Embeddings projection
Urdu NLP for Urdu Urdu Wikipedia Articles 13.19 12.55 Urdu News Dataset 95.28 (valid set) Urdu Embeddings projection Urdu Embeddings projection
Telugu NLP for Telugu Telugu Wikipedia Articles 27.47 29.44 Telugu News Dataset


Telugu News Andhra Jyoti
95.4


92.09
Notebook


Notebook
Telugu Embeddings projection Telugu Embeddings projection
Tanglish NLP for Tanglish Synthetic Tanglish Dataset 37.50 - Dravidian Codemix HASOC @ FIRE 2020

Dravidian Codemix Sentiment Analysis @ FIRE 2020
F1 Score: 0.88

F1 Score: 0.62
- Notebook

Notebook
Tanglish Embeddings Projection -
Manglish NLP for Manglish Synthetic Manglish Dataset 45.84 - Dravidian Codemix HASOC @ FIRE 2020

Dravidian Codemix Sentiment Analysis @ FIRE 2020
F1 Score: 0.74

F1 Score: 0.69
- Notebook

Notebook
Manglish Embeddings Projection -
Hinglish NLP for Hinglish Synthetic Hinglish Dataset 86.48 - - - - - Hinglish Embeddings Projection -

Note: English model has been directly taken from fast.ai

Effect of using Transfer Learning + Paraphrases from iNLTK

Language Repository Dataset used for Classification Results on using
complete training set
Percentage Decrease
in Training set size
Results on using
reduced training set
without Paraphrases
Results on using
reduced training set
with Paraphrases
Hindi NLP for Hindi IIT Patna Movie Reviews Accuracy: 57.74

MCC: 37.23
80% (2480 -> 496) Accuracy: 47.74

MCC: 20.50
Accuracy: 56.13

MCC: 34.39
Bengali NLP for Bengali Bengali News Articles (Soham Articles) Accuracy: 90.71

MCC: 87.92
99% (11284 -> 112) Accuracy: 69.88

MCC: 61.56
Accuracy: 74.06

MCC: 65.08
Gujarati NLP for Gujarati iNLTK Headlines Corpus - Gujarati Accuracy: 91.05

MCC: 86.09
90% (5269 -> 526) Accuracy: 80.88

MCC: 70.18
Accuracy: 81.03

MCC: 70.44
Malayalam NLP for Malayalam iNLTK Headlines Corpus - Malayalam Accuracy: 95.56

MCC: 93.29
90% (5036 -> 503) Accuracy: 82.38

MCC: 73.47
Accuracy: 84.29

MCC: 76.36
Marathi NLP for Marathi iNLTK Headlines Corpus - Marathi Accuracy: 92.40

MCC: 85.23
95% (9672 -> 483) Accuracy: 84.13

MCC: 68.59
Accuracy: 84.55

MCC: 69.11
Tamil NLP for Tamil iNLTK Headlines Corpus - Tamil Accuracy: 95.22

MCC: 92.70
95% (5346 -> 267) Accuracy: 86.25

MCC: 79.42
Accuracy: 89.84

MCC: 84.63

For more details around implementation or to reproduce results, checkout respective repositories.

Contributing

Add a new language support

If you would like to add support for language of your own choice to iNLTK, please start with checking/raising a issue here

Please checkout the steps I'd mentioned here for Telugu to begin with. They should be almost similar for other languages as well.

Improving models/using models for your own research

If you would like to take iNLTK's models and refine them with your own dataset or build your own custom models on top of it, please check out the repositories in the above table for the language of your choice. The repositories above contain links to datasets, pretrained models, classifiers and all of the code for that.

Add new functionality

If you wish for a particular functionality in iNLTK - Start by checking/raising a issue here

What's next

..and being worked upon

Shout out if you want to help :)

..and NOT being worked upon

Shout out if you want to lead :)

iNLTK's Appreciation

Citation

If you use this library in your research, please consider citing:

@inproceedings{arora-2020-inltk,
    title = "i{NLTK}: Natural Language Toolkit for Indic Languages",
    author = "Arora, Gaurav",
    booktitle = "Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS)",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.nlposs-1.10",
    doi = "10.18653/v1/2020.nlposs-1.10",
    pages = "66--71",
    abstract = "We present iNLTK, an open-source NLP library consisting of pre-trained language models and out-of-the-box support for Data Augmentation, Textual Similarity, Sentence Embeddings, Word Embeddings, Tokenization and Text Generation in 13 Indic Languages. By using pre-trained models from iNLTK for text classification on publicly available datasets, we significantly outperform previously reported results. On these datasets, we also show that by using pre-trained models and data augmentation from iNLTK, we can achieve more than 95{\%} of the previous best performance by using less than 10{\%} of the training data. iNLTK is already being widely used by the community and has 40,000+ downloads, 600+ stars and 100+ forks on GitHub. The library is available at https://github.com/goru001/inltk.",
}