• 设为首页
  • 点击收藏
  • 手机版
    手机扫一扫访问
    迪恩网络手机版
  • 关注官方公众号
    微信扫一扫关注
    公众号

anoopkunchukuttan/indic_nlp_library: Resources and tools for Indian language Nat ...

原作者: [db:作者] 来自: 网络 收藏 邀请

开源软件名称(OpenSource Name):

anoopkunchukuttan/indic_nlp_library

开源软件地址(OpenSource Url):

https://github.com/anoopkunchukuttan/indic_nlp_library

开源编程语言(OpenSource Language):

Python 80.4%

开源软件介绍(OpenSource Introduction):

Indic NLP Library

The goal of the Indic NLP Library is to build Python based libraries for common text processing and Natural Language Processing in Indian languages. Indian languages share a lot of similarity in terms of script, phonology, language syntax, etc. and this library is an attempt to provide a general solution to very commonly required toolsets for Indian language text.

The library provides the following functionalities:

  • Text Normalization
  • Script Information
  • Word Tokenization and Detokenization
  • Sentence Splitting
  • Word Segmentation
  • Syllabification
  • Script Conversion
  • Romanization
  • Indicization
  • Transliteration
  • Translation

The data resources required by the Indic NLP Library are hosted in a different repository. These resources are required for some modules. You can download from the Indic NLP Resources project.

If you are interested in Indian language NLP resources, you should check the Indic NLP Catalog for pointers.

Pre-requisites

  • Python 3.x
    • (For Python 2.x version check the tag PYTHON_2.7_FINAL_JAN_2019. Not actively supporting Python 2.x anymore, but will try to maintain as much compatibility as possible)
  • Indic NLP Resources
  • Urduhack: Needed only if Urdu normalization is required. It has other dependencies like Tensorflow.
  • Other dependencies are listed in setup.py

Configuration

  • Installation from pip:

    pip install indic-nlp-library

  • If you want to use the project from the github repo, add the project to the Python Path:

    • Clone this repository
    • Install dependencies: pip install -r requirements.txt
    • Run: export PYTHONPATH=$PYTHONPATH:<project base directory>
  • In either case, export the path to the Indic NLP Resources directory

    Run: export INDIC_RESOURCES_PATH=<path to Indic NLP resources>

Usage

You can use the Python API to access all the features of the library. Many of the most common operations are also accessible via a unified commandline API.

Getting Started

Check this IPython Notebook for examples to use the Python API.

  • You can find the Python 2.x Notebook here

Documentation

You can find detailed documentation HERE

This documents the Python API as well as the commandline reference.

Citing

If you use this library, please include the following citation:

@misc{kunchukuttan2020indicnlp,
author = "Anoop Kunchukuttan",
title = "{The IndicNLP Library}",
year = "2020",
howpublished={\url{https://github.com/anoopkunchukuttan/indic_nlp_library/blob/master/docs/indicnlp.pdf}}
}

You can find the document HERE

Website

http://anoopkunchukuttan.github.io/indic_nlp_library

Author

Anoop Kunchukuttan ([email protected])

Companies, Organizations, Projects using IndicNLP Library

Revision Log

0.81 : 26 May 2021

- Bug fix in version number extraction

0.80 : 24 May 2021

- Improved sentence splitting
- Bug fixes
- Support for Urdu Normalizer

0.71 : 03 Sep 2020

- Improved documentation
- Bug fixes

0.7 : 02 Apr 2020:

- Unified commandline 
- Improved documentation
- Added setup.py

0.6 : 16 Dec 2019:

- New romanizer and indicizer
- Script Unifiers
- Improved script normalizers
- Added contrib directory for sample uses
- changed to MIT license 

0.5 : 03 Jun 2019:

- Improved word tokenizer to handle dates and numbers. 
- Added sentence splitter that can handle common prefixes/honorofics and uses some heuristics.
- Added detokenizer
- Added acronym transliterator that can convert English acronyms to Brahmi-derived scripts

0.4 : 28 Jan 2019: Ported to Python 3, and lots of feature additions since last release; primarily around script information, script similarity and syllabification.

0.3 : 21 Oct 2014: Supports morph-analysis between Indian languages

0.2 : 13 Jun 2014: Supports transliteration between Indian languages and tokenization of Indian languages

0.1 : 12 Mar 2014: Initial version. Supports text normalization.

LICENSE

Indic NLP Library is released under the MIT license




鲜花

握手

雷人

路过

鸡蛋
该文章已有0人参与评论

请发表评论

全部评论

专题导读
上一篇:
center-for-threat-informed-defense/adversary_emulation_library: An open library ...发布时间:2022-08-15
下一篇:
AnikSarker/ACM-Library发布时间:2022-08-15
热门推荐
阅读排行榜

扫描微信二维码

查看手机版网站

随时了解更新最新资讯

139-2527-9053

在线客服(服务时间 9:00~18:00)

在线QQ客服
地址:深圳市南山区西丽大学城创智工业园
电邮:jeky_zhao#qq.com
移动电话:139-2527-9053

Powered by 互联科技 X3.4© 2001-2213 极客世界.|Sitemap