A model library for exploring state-of-the-art deep learning topologies and techniques for optimizing Natural Language Processing neural networks

Overview



A Deep Learning NLP/NLU library by Intel® AI Lab

GitHub Website GitHub release

Overview | Models | Installation | Examples | Documentation | Tutorials | Contributing

NLP Architect is an open source Python library for exploring state-of-the-art deep learning topologies and techniques for optimizing Natural Language Processing and Natural Language Understanding Neural Networks.

Overview

NLP Architect is an NLP library designed to be flexible, easy to extend, allow for easy and rapid integration of NLP models in applications and to showcase optimized models.

Features:

  • Core NLP models used in many NLP tasks and useful in many NLP applications

  • Novel NLU models showcasing novel topologies and techniques

  • Optimized NLP/NLU models showcasing different optimization algorithms on neural NLP/NLU models

  • Model-oriented design:

    • Train and run models from command-line.
    • API for using models for inference in python.
    • Procedures to define custom processes for training, inference or anything related to processing.
    • CLI sub-system for running procedures
  • Based on optimized Deep Learning frameworks:

  • Essential utilities for working with NLP models - Text/String pre-processing, IO, data-manipulation, metrics, embeddings.

Installing NLP Architect

We recommend to install NLP Architect in a new python environment, to use python 3.6+ with up-to-date pip, setuptools and h5py.

Install using pip

Install core library only

pip install nlp-architect

Install from source (Github)

Includes core library, examples, solutions and tutorials:

git clone https://github.com/IntelLabs/nlp-architect.git
cd nlp-architect
pip install -e .  # install in developer mode

Running Examples and Solutions

To run provided examples and solutions please install the library with [all] flag which will install extra packages required. (requires installation from source)

pip install .[all]

Models

NLP models that provide best (or near) in class performance:

Natural Language Understanding (NLU) models that address semantic understanding:

Optimizing NLP/NLU models and misc. optimization techniques:

Solutions (End-to-end applications) using one or more models:

Documentation

Full library documentation of NLP models, algorithms, solutions and instructions on how to run each model can be found on our website.

NLP Architect library design philosophy

NLP Architect is a model-oriented library designed to showcase novel and different neural network optimizations. The library contains NLP/NLU related models per task, different neural network topologies (which are used in models), procedures for simplifying workflows in the library, pre-defined data processors and dataset loaders and misc utilities. The library is designed to be a tool for model development: data pre-process, build model, train, validate, infer, save or load a model.

The main design guidelines are:

  • Deep Learning framework agnostic
  • NLP/NLU models per task
  • Different topologies used in models
  • Showcase End-to-End applications (Solutions) utilizing one or more NLP Architect model
  • Generic dataset loaders, textual data processing utilities, and miscellaneous utilities that support NLP model development (loaders, text processors, io, metrics, etc.)
  • Procedures for defining processes for training, inference, optimization or any kind of elaborate script.
  • Pythonic API for using models for inference
  • Extensive model documentation and tutorials

Note

NLP Architect is an active space of research and development; Throughout future releases new models, solutions, topologies and framework additions and changes will be made. We aim to make sure all models run with Python 3.6+. We encourage researchers and developers to contribute their work into the library.

Citing

If you use NLP Architect in your research, please use the following citation:

@misc{izsak_peter_2018_1477518,
  title        = {NLP Architect by Intel AI Lab},
  month        = nov,
  year         = 2018,
  doi          = {10.5281/zenodo.1477518},
  url          = {https://doi.org/10.5281/zenodo.1477518}
}

Disclaimer

The NLP Architect is released as reference code for research purposes. It is not an official Intel product, and the level of quality and support may not be as expected from an official product. NLP Architect is intended to be used locally and has not been designed, developed or evaluated for production usage or web-deployment. Additional algorithms and environments are planned to be added to the framework. Feedback and contributions from the open source and NLP research communities are more than welcome.

Contact

Contact the NLP Architect development team through Github issues or email: [email protected]

Comments
  • Runtime errors when running with the ABSA model

    Runtime errors when running with the ABSA model

    I am trying to run the ABSA solution on my local Ubuntu 18.04 even with the provided train reviews dataset but getting the following error (using the exact instructions + virtualenv): 'file_name': ['tripadvisor_co_uk-travel_restaurant_reviews_sample_2000_train.csv']}}], 'references': []}: OSError('Unable to load model. Filepath is not an hdf5 file (or h5py is not available) or SavedModel.',) When I am trying to load a bunch of reviews, I get the following error:

    'references': []}: ParserError('Expected 2 fields in line 2, saw 3')

    Please advise how to fix it.

    bug 
    opened by stevesolun 10
  • Pip install failing from master

    Pip install failing from master

    Target objective:

    pip Install .

    Steps to objective:

    Successfully built nlp-architect spacy 2.0.18 has requirement numpy>=1.15.0, but you'll have numpy 1.14.5 which is incompatible. Installing collected packages: spacy, nlp-architect Killed

    Pull-Request related:

    opened by aribornstein 8
  • bug: Error exporting QuantizedBert to ONNX

    bug: Error exporting QuantizedBert to ONNX

    When attempting to export a Q8BERT model to ONNX, TorchScript reports a runtime error.

    Since the quantized forward methods embed Tensors with grads from layer instance attributes rather than input parameters, exporting the model has to be done with explicit torch script tracing; however that creates a traced model with lacks fidelity with the python model, as branching behavior is not captured.

    However, attempting to create a script model with TorchScript, which should encode that behavior results in a missing config attribute on the BertModel superclass.

    To Reproduce Steps to reproduce the behavior:

    1. Train a Q8Bert model:
     nlp-train transformer_glue \
                --task_name mrpc \
                --model_name_or_path bert-base-uncased \
                --model_type quant_bert \
                --learning_rate 2e-5 \
                --output_dir $DATA_DIR \
                --data_dir $GLUE_DIR  \
                --evaluate_during_training \
                --do_lower_case \
                --per_gpu_train_batch_size 32 \
                --per_gpu_eval_batch_size 32 \
                --max_seq_length 128
    
    1. Load the 8bit model for inference
    import torch
    from nlp_architect.models.transformers.quantized_bert import QuantizedBertForSequenceClassification
    
    model = QuantizedBertForSequenceClassification.from_pretrained(configs.data_dir, from_8bit=True)
    device = torch.device("cuda")
    model.to(device)
    model.eval()
    
    
    1. Attempt to create the script model
    script_model = torch.jit.script(model)
    
    1. This produces the error:
    Traceback (most recent call last):
      File "convert-torch-q8bert-via-onnx-to-tensorflow.py", line 105, in <module>
        script_model = torch.jit.script(model)
      File "/home/krhys/.local/share/virtualenvs/q8bert-_S0V57VU/lib/python3.7/site-packages/torch/jit/__init__.py", line 1203, in script
        return torch.jit.torch.jit._recursive.recursive_script(obj)
      File "/home/krhys/.local/share/virtualenvs/q8bert-_S0V57VU/lib/python3.7/site-packages/torch/jit/_recursive.py", line 173, in recursive_script
        return copy_to_script_module(mod, overload_stubs + stubs)
      File "/home/krhys/.local/share/virtualenvs/q8bert-_S0V57VU/lib/python3.7/site-packages/torch/jit/_recursive.py", line 95, in copy_to_script_module
        torch.jit._create_methods_from_stubs(script_module, stubs)
      File "/home/krhys/.local/share/virtualenvs/q8bert-_S0V57VU/lib/python3.7/site-packages/torch/jit/__init__.py", line 1423, in _create_methods_from_stubs
        self._c._create_methods(self, defs, rcbs, defaults)
      File "/home/krhys/.local/share/virtualenvs/q8bert-_S0V57VU/lib/python3.7/site-packages/torch/jit/_recursive.py", line 195, in make_strong_submodule
        new_strong_submodule = recursive_script(module)
      File "/home/krhys/.local/share/virtualenvs/q8bert-_S0V57VU/lib/python3.7/site-packages/torch/jit/_recursive.py", line 173, in recursive_script
        return copy_to_script_module(mod, overload_stubs + stubs)
      File "/home/krhys/.local/share/virtualenvs/q8bert-_S0V57VU/lib/python3.7/site-packages/torch/jit/_recursive.py", line 95, in copy_to_script_module
        torch.jit._create_methods_from_stubs(script_module, stubs)
      File "/home/krhys/.local/share/virtualenvs/q8bert-_S0V57VU/lib/python3.7/site-packages/torch/jit/__init__.py", line 1423, in _create_methods_from_stubs
        self._c._create_methods(self, defs, rcbs, defaults)
    RuntimeError:
    module has no attribute 'config':
    at /home/krhys/.local/share/virtualenvs/q8bert-_S0V57VU/lib/python3.7/site-packages/transformers/modeling_bert.py:675:15
            # We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length]
            # ourselves in which case we just need to make it broadcastable to all heads.
            if attention_mask.dim() == 3:
                extended_attention_mask = attention_mask[:, None, :, :]
    
            # Provided a padding mask of dimensions [batch_size, seq_length]
            # - if the model is a decoder, apply a causal mask in addition to the padding mask
            # - if the model is an encoder, make the mask broadcastable to [batch_size, num_heads, seq_length, seq_length]
            if attention_mask.dim() == 2:
                if self.config.is_decoder:
                   ~~~~~~~~~~~ <--- HERE
                    batch_size, seq_length = input_shape
                    seq_ids = torch.arange(seq_length, device=device)
                    causal_mask = seq_ids[None, None, :].repeat(batch_size, seq_length, 1) <= seq_ids[None, :, None]
                    extended_attention_mask = causal_mask[:, None, :, :] * attention_mask[:, None, None, :]
                else:
                    extended_attention_mask = attention_mask[:, None, None, :]
    
            # Since attention_mask is 1.0 for positions we want to attend and 0.0 for
            # masked positions, this operation will create a tensor which is 0.0 for
    '__torch__.nlp_architect.models.transformers.quantized_bert.QuantizedBertModel.forward' is being compiled since it was called from '__torch__.nlp_architect.models.transformers.quantized_bert.QuantizedBertForSequenceClassification.forward'
    at /home/krhys/.local/share/virtualenvs/q8bert-_S0V57VU/lib/python3.7/site-packages/transformers/modeling_bert.py:1014:8
        def forward(self, input_ids=None, attention_mask=None, token_type_ids=None,
                    position_ids=None, head_mask=None, inputs_embeds=None, labels=None):
    
            outputs = self.bert(input_ids,
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~...  <--- HERE
                                attention_mask=attention_mask,
                                token_type_ids=token_type_ids,
                                position_ids=position_ids,
                                head_mask=head_mask,
                                inputs_embeds=inputs_embeds)
    
            pooled_output = outputs[1]
    
            pooled_output = self.dropout(pooled_output)
    

    Expected behavior

    It should produce a script model without error

    **Environment setup: **

    • OS (Linux/Mac OS): Ubuntu Xenial
    • Python version: 3.7

    Additional context

    It may be possible to create pure helper quantized forward functions with explicit arguments which have the @torch.jit.script annotations which are called by the quantized layer forward methods

    Note that the HuggingFace transformers Bert model can be exported to ONNX.

    bug Stale 
    opened by kirstin-rhys 7
  • Any readme for BERT example?

    Any readme for BERT example?

    Target objective:

    Want help to run https://github.com/NervanaSystems/nlp-architect/blob/master/nlp_architect/procedures/transformers/glue.py

    Hi, I want to run the bert example newly added to the repo. Wondering is there any readme for running the example or any documentation for bert/quantized-bert related features? For instance, where to download model and 'labels.txt'?

    Thanks so much for your help in advance!

    opened by hkvision 7
  • nlp_architect command fails if dev req aren't installed

    nlp_architect command fails if dev req aren't installed

    Target objective:

    nlp_architect command fails if dev req aren't installed detect is libraries required are install and print a msg that states the error + how to install

    Steps to objective:

    • [ ] fix cmd.py
    • [ ]

    Pull-Request related:

    bug 
    opened by peteriz 7
  • GPU training support

    GPU training support

    This is more of a question than issue but I wasn't able to find a forum to ask the question.

    I see installation instruction on enabling GPU supports. Yet, many algorithms (e.g. ner) do not specifically have a parameter for GPU enabling. Is it an issue easy to get around with or is it going to be a feature supported in the future?

    question 
    opened by DihuiLai 7
  • Problem running bist FileNotFoundError: [Errno 2] No such file or directory:  (...) params.json

    Problem running bist FileNotFoundError: [Errno 2] No such file or directory: (...) params.json

    I tried to run bist in your interface using the command :

    python3 server/serve.py --name bist

    I get the following error :

    25MB [00:28,  1.16s/MB]                                                                                                                                                                                     
    Download Complete
    Unzipping...
    Done.
    Traceback (most recent call last):
      File "serve.py", line 306, in <module>
        set_server_properties(app, args.name)
      File "serve.py", line 287, in set_server_properties
        service = Service(service_name)
      File "serve.py", line 143, in __init__
        self.service = self.load_service(service_name)
      File "serve.py", line 259, in load_service
        upload_service.load_model()
      File "/home/catalina/nlp-architect/nlp_architect/api/bist_parser_api.py", line 32, in load_model
        self.model = SpacyBISTParser()
      File "/home/catalina/nlp-architect/nlp_architect/pipelines/spacy_bist.py", line 48, in __init__
        self.bist_parser.load(bist_model if bist_model else SpacyBISTParser.pretrained)
      File "/home/catalina/nlp-architect/nlp_architect/models/bist_parser.py", line 117, in load
        with open(os.path.join(os.path.dirname(path), 'params.json'), 'r') as file:
    FileNotFoundError: [Errno 2] No such file or directory: '/home/catalina/nlp-architect/nlp_architect/pipelines/bist-pretrained/params.json'
    
    opened by Catadanna 7
  • How to improve bist model accuracy? :Bistmodel

    How to improve bist model accuracy? :Bistmodel

    I'm using spacybistparser with my own spacy and bist model trained using universal dependencies v2.0 dataset. My training follows the doc. I want to improve the accuracy of bistmodel. Currently, with default parameters, I train the bist model for 20iterations. The evaluation uas score is 87.8 percent. How can I improve it to 93.8 as mentioned in the docs? Is it possible with this english ud 2.0 dataset? Please help.

    question 
    opened by ravishpankar 6
  • question about Quantize BERT?

    question about Quantize BERT?

    opened by RyanHuangNLP 6
  • Question about Seq2SeqIntentModel

    Question about Seq2SeqIntentModel

    This is used for slot tagging, and in the example you provide, the dataset is SNIPS and it is used for NER. Wondering why it is placed under intent_extraction and come with the name IntentModel?

    question Stale 
    opened by hkvision 6
  • Getting ValueError('Length  must be less than 1000') error

    Getting ValueError('Length must be less than 1000') error

    I am trying to run ABSA with the a CSV and getting this error. I have sent the file to @danielkorat. It contains 1000 rows and it fails with the above error.

    Please advise how can it be solved. Until now 500K reviews (in one file) have been sent and it worked.

    bug 
    opened by stevesolun 5
  • Performance issue in the definition of _inference, examples/memn2n_dialogue/memn2n_dialogue.py(P1)

    Performance issue in the definition of _inference, examples/memn2n_dialogue/memn2n_dialogue.py(P1)

    Hello, I found a performance issue in the difinition of_inference, examples/memn2n_dialogue/memn2n_dialogue.py, tf.nn.embedding_lookup will be calculated repeately during the program execution, resulting in reduced efficiency. So I think it should be created before the loop.

    The same issue exist in tf.reduce_sum in line 187 and 200.

    Looking forward to your reply. Btw, I am very glad to create a PR to fix it if you are too busy.

    bug 
    opened by DLPerf 2
  • bug: ERROR: ResolutionImpossible

    bug: ERROR: ResolutionImpossible

    Describe the bug Installation error Model/procedure: $ pip install nlp-architect

    To Reproduce Steps to reproduce the behavior: $ pip install nlp-architect

    Expected behavior No Errors

    **Environment setup: **

    • OS (Linux/Mac OS): GNU/Linux
    • Python version: Python 3.9.6
    • Backend: pip

    Log:

    Collecting nlp-architect
      Using cached nlp_architect-0.5.4-py3-none-any.whl (308 kB)
    Collecting h5py
      Using cached h5py-3.3.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (4.5 MB)
    Collecting requests
      Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
         |████████████████████████████████| 62 kB 56 kB/s 
    Collecting gensim
      Using cached gensim-4.0.1-cp39-cp39-linux_x86_64.whl
    Collecting scipy
      Downloading scipy-1.7.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl (28.4 MB)
         |████████████████████████████████| 28.4 MB 871 kB/s 
    Collecting six
      Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
    Collecting numpy==1.16.4
      Using cached numpy-1.16.4.zip (5.1 MB)
    Collecting dynet==2.1
      Using cached dyNET-2.1.tar.gz (454 kB)
        ERROR: Command errored out with exit status 1:
         command: /home/anks/STEALTH/ENV/absa_env/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-832t4ncx/dynet_ec03a5a10dc14ba6bb53d279dc810a9e/setup.py'"'"'; __file__='"'"'/tmp/pip-install-832t4ncx/dynet_ec03a5a10dc14ba6bb53d279dc810a9e/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-pj_foilz
             cwd: /tmp/pip-install-832t4ncx/dynet_ec03a5a10dc14ba6bb53d279dc810a9e/
        Complete output (5 lines):
        Traceback (most recent call last):
          File "<string>", line 1, in <module>
          File "/tmp/pip-install-832t4ncx/dynet_ec03a5a10dc14ba6bb53d279dc810a9e/setup.py", line 17, in <module>
            from Cython.Distutils import build_ext as _build_ext
        ModuleNotFoundError: No module named 'Cython'
        ----------------------------------------
    WARNING: Discarding https://files.pythonhosted.org/packages/22/8e/148ecf0ec4ffce667e84abd6aa134bbff1f73cbb99c92c198149f122889e/dyNET-2.1.tar.gz#sha256=d0f58aaf3926da24baba6e3e76cb3d090c8b6d359196ce138b11faa291b2ec07 (from https://pypi.org/simple/dynet/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    Collecting nlp-architect
      Using cached nlp_architect-0.5.3-py3-none-any.whl (301 kB)
      Using cached nlp_architect-0.5.2-py3-none-any.whl (978 kB)
    Collecting tqdm
      Using cached tqdm-4.61.2-py2.py3-none-any.whl (76 kB)
    Collecting wordfreq
      Downloading wordfreq-2.5.0.tar.gz (56.8 MB)
         |████████████████████████████████| 56.8 MB 50 kB/s 
    Collecting future
      Downloading future-0.18.2.tar.gz (829 kB)
         |████████████████████████████████| 829 kB 71 kB/s 
    Collecting python-dateutil<2.8.1
      Downloading python_dateutil-2.8.0-py2.py3-none-any.whl (226 kB)
         |████████████████████████████████| 226 kB 945 kB/s 
    Collecting ftfy
      Downloading ftfy-6.0.3.tar.gz (64 kB)
         |████████████████████████████████| 64 kB 226 kB/s 
    Collecting termcolor
      Downloading termcolor-1.1.0.tar.gz (3.9 kB)
    Collecting numpy<1.17
      Using cached numpy-1.16.6.zip (5.1 MB)
    Collecting hyperopt
      Downloading hyperopt-0.2.5-py2.py3-none-any.whl (965 kB)
         |████████████████████████████████| 965 kB 112 kB/s 
    Collecting pywikibot
      Downloading pywikibot-6.4.0.tar.gz (506 kB)
         |████████████████████████████████| 506 kB 422 kB/s 
    Collecting nltk
      Downloading nltk-3.6.2-py3-none-any.whl (1.5 MB)
         |████████████████████████████████| 1.5 MB 696 kB/s 
    Collecting tensorflow-hub
      Downloading tensorflow_hub-0.12.0-py2.py3-none-any.whl (108 kB)
         |████████████████████████████████| 108 kB 330 kB/s 
    Collecting nlp-architect
      Using cached nlp_architect-0.5.1-py3-none-any.whl (1.6 MB)
    Collecting pillow
      Downloading Pillow-8.3.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl (3.0 MB)
         |████████████████████████████████| 3.0 MB 182 kB/s 
    Collecting regex==2018.02.21
      Downloading regex-2018.02.21.tar.gz (620 kB)
         |████████████████████████████████| 620 kB 475 kB/s 
    Collecting nlp-architect
      Downloading nlp_architect-0.5-py3-none-any.whl (1.2 MB)
         |████████████████████████████████| 1.2 MB 107 kB/s 
      Downloading nlp_architect-0.4.post3-py3-none-any.whl (1.1 MB)
         |████████████████████████████████| 1.1 MB 102 kB/s 
    Collecting spacy
      Downloading spacy-3.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.5 MB)
         |████████████████████████████████| 6.5 MB 857 kB/s 
    Collecting nlp-architect
      Downloading nlp_architect-0.4.post2-py3-none-any.whl (1.1 MB)
         |████████████████████████████████| 1.1 MB 70 kB/s 
    Collecting dynet==2.0.2
      Downloading dyNET-2.0.2.tar.gz (388 kB)
         |████████████████████████████████| 388 kB 557 kB/s 
        ERROR: Command errored out with exit status 1:
         command: /home/anks/STEALTH/ENV/absa_env/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-832t4ncx/dynet_eb19eec4e56248fdb3fef59af0805ab5/setup.py'"'"'; __file__='"'"'/tmp/pip-install-832t4ncx/dynet_eb19eec4e56248fdb3fef59af0805ab5/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-etz16c1n
             cwd: /tmp/pip-install-832t4ncx/dynet_eb19eec4e56248fdb3fef59af0805ab5/
        Complete output (5 lines):
        Traceback (most recent call last):
          File "<string>", line 1, in <module>
          File "/tmp/pip-install-832t4ncx/dynet_eb19eec4e56248fdb3fef59af0805ab5/setup.py", line 17, in <module>
            from Cython.Distutils import build_ext as _build_ext
        ModuleNotFoundError: No module named 'Cython'
        ----------------------------------------
    WARNING: Discarding https://files.pythonhosted.org/packages/01/10/cfc00197733dd7cf52d9e00c42017ebe7e818653980bccfe6f241ea5b79a/dyNET-2.0.2.tar.gz#sha256=d09f3799e3ad19572e9adec2784b0330a5d96bf6914a20858cd14b71e1482213 (from https://pypi.org/simple/dynet/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    Collecting nlp-architect
      Downloading nlp_architect-0.4.post1-py3-none-any.whl (1.1 MB)
         |████████████████████████████████| 1.1 MB 611 kB/s 
      Downloading nlp_architect-0.4-py3-none-any.whl (1.1 MB)
         |████████████████████████████████| 1.1 MB 190 kB/s 
      Downloading nlp_architect-0.3.1-py3-none-any.whl (1.0 MB)
         |████████████████████████████████| 1.0 MB 598 kB/s 
    Collecting pytest-mock
      Downloading pytest_mock-3.6.1-py3-none-any.whl (12 kB)
    ERROR: Cannot install nlp-architect==0.3.1, nlp-architect==0.4, nlp-architect==0.4.post1, nlp-architect==0.4.post2, nlp-architect==0.4.post3, nlp-architect==0.5, nlp-architect==0.5.1, nlp-architect==0.5.2, nlp-architect==0.5.3 and nlp-architect==0.5.4 because these package versions have conflicting dependencies.
    
    The conflict is caused by:
        nlp-architect 0.5.4 depends on dynet==2.1
        nlp-architect 0.5.3 depends on tensorflow==1.15.0
        nlp-architect 0.5.2 depends on dynet==2.1
        nlp-architect 0.5.1 depends on dynet==2.1
        nlp-architect 0.5 depends on dynet==2.1
        nlp-architect 0.4.post3 depends on dynet==2.1
        nlp-architect 0.4.post2 depends on dynet==2.0.2
        nlp-architect 0.4.post1 depends on dynet==2.0.2
        nlp-architect 0.4 depends on dynet==2.0.2
        nlp-architect 0.3.1 depends on dynet==2.0.2
    
    To fix this you could try to:
    1. loosen the range of package versions you've specified
    2. remove package versions to allow pip attempt to solve the dependency conflict
    
    ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
    
    
    bug 
    opened by AnkS4 0
  • question: [Quantization] Which files to change to make inference faster for Q8BERT?

    question: [Quantization] Which files to change to make inference faster for Q8BERT?

    I know from previous issues it is mentioned that that Q8BERT was just an experiment to measure the accuracy of quantized BERT model. But, given that the accuracy is good, what changed would need to be made to torch.nn.quantization file to replace the FP32 operations by INT8 operations?

    Replacing the FP32 Linear layers with the torch.nn.quantized.Linear should theoretically work since it will have optimized operations, but it doesn't. Same for other layers.

    If someone could just point out how to improve the inference speed (hints, tips, directions, code, anything), it would be helpful since the model's accuracy is really good and I would like to use it for downstream tasks. I don't mind even creating a PR once those changes are done so that it merges with the main repo.

    Thank you!

    question 
    opened by sarthaklangde 1
  • question: [Q8Bert experiment Setting]

    question: [Q8Bert experiment Setting]

    Hello, I read the Q8Bert paper and have tried to reproduce the experiment results. But, on some GLUE tasks ( e.g cola, mrpc ), the differences between the fp32 results and quantized ones are much larger than the differences reported in the paper. I tried sweeping initial learning rate but still the result was still far from the reported results.

    image

    So, I want to ask you if the experiment on Q8bert was done with default parameters set inside nlp-architect code as below.

    image

    If not, could you tell me the experiment setting.

    question 
    opened by daumkh402 2
  • improvement: distillation for TransformerSequenceClassifier models for GLUE tasks

    improvement: distillation for TransformerSequenceClassifier models for GLUE tasks

    Hi,

    I'm wondering if it would be easy to add support for knowledge distillation for the Transformers on GLUE tasks (i.e. the TransformerSequenceClassifier module).

    I see that the distillation loss has been implemented, and it's an option for the NeuralTagger which uses the TransformerTokenClassifier. Would it be easy to add distillation support for the GLUE models?

    Here's how I was envisioning implementing it, modeling off of the distillation implementation for the tagger models:

    1. TransformerSequenceClassifier’s train calls the base transformer model’s _train(). I would need to add a distiller argument to this function. This function would then handle distillation by loading in the teacher and the relevant arguments, just like the NeuralTagger
    2. in procedures/transformers/glue.py, i would need to add a do_kd_training function that adds distilation args. This function would need to create a teacher model from these args (loading in weights from the passed-in path), create a TeacherStudentDistill instance, and pass in this object as the distiller argument into the new _train() function.

    Does this seem about right? Are there any roadblocks you'd envision / why wasn't distillation implemented for the sequence classifier models to begin with?

    opened by rmovva 0
Releases(v0.5.5.1)
  • v0.5.5.1(Nov 17, 2020)

  • v0.5.4.1(Sep 16, 2020)

  • v0.5.4(Apr 12, 2020)

    • Improved ID-CNN and LSTM sequence tagger models accuracy
    • Added support for BILOU format for tagging tasks
    • Updated Transformer token tagging and neural taggers to save best and support BILOU format
    • Refactor pseudo/distillation procedures to reproduce results according to our paper
    Source code(tar.gz)
    Source code(zip)
  • v0.5(Aug 28, 2019)

    General features

    • Added PyTorch as a backend
    • New nlp_architect CLI for training/running/process models and scripts
    • New nlp_architect.procedures package for creating procedures/scripts for doing scripted work with NLP architect.
    • S3 URL caching of pre-trained models
    • Many additions/refactors/base classes reflecting new API (WIP)

    Transformer models

    • Integrated pytorch-transformers library into NLP Architect's model package.
    • Created a base transformer class for training transformer-based models
    • Added sequence/token classification models

    Optimization related models

    • Added a base transformer model for training quantized (8bit) BERT models
    • Added distillation process from Transformer models into CNN/LSTM based token classification models

    NLP related additions

    • Added GLUE benchmark tasks
    • Added CNN-LSTM and ID-CNN token classification models (pytorch backend)

    Documentation

    • Updated documentation website to reflect 0.5 release
    • Updated website with logo and matching style
    Source code(tar.gz)
    Source code(zip)
  • v0.4(Apr 2, 2019)

  • v0.3.1(Dec 17, 2018)

    Improvements

    • Installation process refactored and supports pip install nlp-architect (much more simple)
    • Moved server and solutions into main package
    • Many Cross-doc coreference model fixes and additions
    • Updates and fixed lots of documentation files

    Demos

    • Improved NLP Architect demos with a new design
    • Renamed nlp_architect demo to nlp_architect server
    Source code(tar.gz)
    Source code(zip)
  • v0.3(Nov 1, 2018)

    New Solution

    • Topics and Trend Analysis - extract topics and compare two temporal versions a corpus, highlighting hot and cold trends.

    New models

    • Sparse GNMT - A Tensorflow implementation of the GNMT model with sparsity and quantization operations integrated.
    • Semantic Relation Identification - Extract semantic relation types of two words or phrases using external resources.
    • Sieve-based Cross Document Coreference - A seive-based model for finding similar entities or events across different documents from the same domain.

    Improvements

    • Reading comprehension - added inference mode.
    • Sequential models - updated NER, IE, Chunker models to use tf.keras and added CNN-character based feature extractors and improved accuracy of all models.
    • CRF Layer - added native Tensorflow based CRF layer.
    • Word Sense Disambiguation - model updated to use tf.keras.
    • Demo UI - updated demo UI using AngularJS.
    • Installation - improved installation process and added support for CPU/MKL/GPU backends for Tensorflow.
    • NLP Architect cmd - added nlp_architect - a simple command initiator to handle maintenance tasks, see nlp_architect -h for the list of commands.
    • Lots of bug fixes and refactoring.
    Source code(tar.gz)
    Source code(zip)
  • v0.2(Aug 15, 2018)

    Release v0.2

    New Solution

    • Term Set Expansion - the task of expanding a given partial set of terms into a more complete set of terms that belong to the same semantic class. This solution demonstrates the usage of NLP Architect models (Word Chunker and NP2Vec) used in an application solution.

    New models

    • Unsupervised Crosslingual Embeddings model using a GAN to learn a mapping between languages - implemented in Tensorflow
    • Language Model (LM) using Temporal Convolution Network (TCN) - implemented in Tensorflow
    • Supervised Sentiment Classification - implemented in Keras

    Model improvements

    • Reading comprehension - refactored to use Tensorflow
    • End-to-end Memory Network for Goal Oriented Dialogue - refactored to use Tensorflow
    • Word Chunker - refactored to use tf.keras and use state-of-art model
    • NP semantic segmentation - refactored to use tf.keras
    • Updated CONLL2000, Amazon_Review, PTB, Fasttext, Wikitext-103 and Wikipedia-dump dataset loaders.

    New features

    • REST server refactored to use hug, new streamlined the UI and improved documentation. See updated documentation for further details.
    • Noun Phrase annotator plug-in for spaCy pipeline
    • Publications page with relevant material demonstrating the usage of NLP Architect
    • Tutorials page with a collection of Jupyter notebook tutorials using NLP Architect models
    Source code(tar.gz)
    Source code(zip)
  • 0.1(May 29, 2018)

    The current version of NLP Architect includes these features that we found interesting from both research perspectives and practical applications:

    • NLP core models that allow robust extraction of linguistic features for NLP workflow: for example, dependency parser (BIST) and NP chunker
    • NLU modules that provide best in class performance: for example, intent extraction (IE), name entity recognition (NER)
    • Modules that address semantic understanding: for example, colocations, most common word sense, NP embedding representation (e.g. NP2V)
    • Components instrumental for conversational AI: for example, ChatBot applications, including dialog system, sequence chunking and IE
    • End-to-end DL applications using new topologies: for example, Q&A, machine reading comprehension
    Source code(tar.gz)
    Source code(zip)
Official PyTorch implementation of Time-aware Large Kernel (TaLK) Convolutions (ICML 2020)

Time-aware Large Kernel (TaLK) Convolutions (Lioutas et al., 2020) This repository contains the source code, pre-trained models, as well as instructio

Vasileios Lioutas 28 Dec 07, 2022
PyKaldi is a Python scripting layer for the Kaldi speech recognition toolkit.

PyKaldi is a Python scripting layer for the Kaldi speech recognition toolkit. It provides easy-to-use, low-overhead, first-class Python wrappers for t

922 Dec 31, 2022
In this project, we compared Spanish BERT and Multilingual BERT in the Sentiment Analysis task.

Applying BERT Fine Tuning to Sentiment Classification on Amazon Reviews Abstract Sentiment analysis has made great progress in recent years, due to th

Alexander Leonardo Lique Lamas 5 Jan 03, 2022
Random-Word-Generator - Generates meaningful words from dictionary with given no. of letters and words.

Random Word Generator Generates meaningful words from dictionary with given no. of letters and words. This might be useful for generating short links

Mohammed Rabil 1 Jan 01, 2022
Code for EMNLP'21 paper "Types of Out-of-Distribution Texts and How to Detect Them"

Code for EMNLP'21 paper "Types of Out-of-Distribution Texts and How to Detect Them"

Udit Arora 19 Oct 28, 2022
Python wrapper for Stanford CoreNLP tools v3.4.1

Python interface to Stanford Core NLP tools v3.4.1 This is a Python wrapper for Stanford University's NLP group's Java-based CoreNLP tools. It can eit

Dustin Smith 610 Sep 07, 2022
SASE : Self-Adaptive noise distribution network for Speech Enhancement with heterogeneous data of Cross-Silo Federated learning

SASE : Self-Adaptive noise distribution network for Speech Enhancement with heterogeneous data of Cross-Silo Federated learning We propose a SASE mode

Tower 1 Nov 20, 2021
Prompt-learning is the latest paradigm to adapt pre-trained language models (PLMs) to downstream NLP tasks

Prompt-learning is the latest paradigm to adapt pre-trained language models (PLMs) to downstream NLP tasks, which modifies the input text with a textual template and directly uses PLMs to conduct pre

THUNLP 2.3k Jan 08, 2023
Understanding the Difficulty of Training Transformers

Admin Understanding the Difficulty of Training Transformers Guided by our analyses, we propose Adaptive Model Initialization (Admin), which successful

Liyuan Liu 300 Dec 29, 2022
The (extremely) naive sentiment classification function based on NBSVM trained on wisesight_sentiment

thai_sentiment The naive sentiment classification function based on NBSVM trained on wisesight_sentiment วิธีติดตั้ง pip install thai_sentiment==0.1.3

Charin 7 Dec 08, 2022
T‘rex Park is a Youzan sponsored project. Offering Chinese NLP and image models pretrained from E-commerce datasets

T‘rex Park is a Youzan sponsored project. Offering Chinese NLP and image models pretrained from E-commerce datasets (product titles, images, comments, etc.).

55 Nov 22, 2022
Pytorch-version BERT-flow: One can apply BERT-flow to any PLM within Pytorch framework.

Pytorch-version BERT-flow: One can apply BERT-flow to any PLM within Pytorch framework.

Ubiquitous Knowledge Processing Lab 59 Dec 01, 2022
NLP techniques such as named entity recognition, sentiment analysis, topic modeling, text classification with Python to predict sentiment and rating of drug from user reviews.

This file contains the following documents sumbited for Baruch CIS9665 group 9 fall 2021. 1. Dataset: drug_reviews.csv 2. python codes for text classi

Aarif Munwar Jahan 2 Jan 04, 2023
Contains analysis of trends from Fitbit Dataset (source: Kaggle) to see how the trends can be applied to Bellabeat customers and Bellabeat products

Contains analysis of trends from Fitbit Dataset (source: Kaggle) to see how the trends can be applied to Bellabeat customers and Bellabeat products.

Leah Pathan Khan 2 Jan 12, 2022
Creating a Feed of MISP Events from ThreatFox (by abuse.ch)

ThreatFox2Misp Creating a Feed of MISP Events from ThreatFox (by abuse.ch) What will it do? This will fetch IOCs from ThreatFox by Abuse.ch, convert t

17 Nov 22, 2022
LCG T-TEST USING EUCLIDEAN METHOD

This project has been created for statistical usage, purposing for determining ATL takers and nontakers using LCG ttest and Euclidean Method, especially for internal business case in Telkomsel.

2 Jan 21, 2022
RoNER is a Named Entity Recognition model based on a pre-trained BERT transformer model trained on RONECv2

RoNER RoNER is a Named Entity Recognition model based on a pre-trained BERT transformer model trained on RONECv2. It is meant to be an easy to use, hi

Stefan Dumitrescu 9 Nov 07, 2022
A highly sophisticated sequence-to-sequence model for code generation

CoderX A proof-of-concept AI system by Graham Neubig (June 30, 2021). About CoderX CoderX is a retrieval-based code generation AI system reminiscent o

Graham Neubig 39 Aug 03, 2021
Creating a python chatbot that Starbucks users can text to place an order + help cut wait time of a normal coffee.

Creating a python chatbot that Starbucks users can text to place an order + help cut wait time of a normal coffee.

2 Jan 20, 2022
Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

Simplemma: a simple multilingual lemmatizer for Python Purpose Lemmatization is the process of grouping together the inflected forms of a word so they

Adrien Barbaresi 70 Dec 29, 2022