Hi. I have many questions.... :-)

I'm a beginner for python. Is there any method to run the code standalone ?

e.g. I trained my data. And I'd like to see the scores on terminal by classifier.score('apple') . The word 'apple' can be changed.

Thank you regards,

import shorttext

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-5-cb09b3381050> in <module>()
----> 1 import shorttext

/usr/local/lib/python2.7/dist-packages/shorttext/__init__.py in <module>()
      5 sys.path.append(thisdir)
      6 
----> 7 from . import utils
      8 from . import data
      9 from . import classifiers

/usr/local/lib/python2.7/dist-packages/shorttext/utils/__init__.py in <module>()
      4 from . import textpreprocessing
      5 from .wordembed import load_word2vec_model
----> 6 from . import compactmodel_io
      7 
      8 from .textpreprocessing import spacy_tokenize as tokenize

/usr/local/lib/python2.7/dist-packages/shorttext/utils/compactmodel_io.py in <module>()
     13 from functools import partial
     14 
---> 15 import utils.classification_exceptions as e
     16 
     17 def removedir(dir):

ImportError: No module named classification_exceptions

Hi, I got the following error when i import shorttext, how shall i resolve?

Using TensorFlow backend.

I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so.7.5 locally I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so.5 locally I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so.7.5 locally I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so.7.5 locally Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.7/dist-packages/shorttext/init.py", line 7, in from . import utils File "/usr/local/lib/python2.7/dist-packages/shorttext/utils/init.py", line 3, in from . import gensim_corpora File "/usr/local/lib/python2.7/dist-packages/shorttext/utils/gensim_corpora.py", line 2, in from .textpreprocessing import spacy_tokenize as tokenize File "/usr/local/lib/python2.7/dist-packages/shorttext/utils/textpreprocessing.py", line 5, in import spacy File "/usr/local/lib/python2.7/dist-packages/spacy/init.py", line 8, in from . import en, de, zh, es, it, hu, fr, pt, nl, sv, fi, bn, he File "/usr/local/lib/python2.7/dist-packages/spacy/en/init.py", line 4, in from ..language import Language File "/usr/local/lib/python2.7/dist-packages/spacy/language.py", line 12, in from .syntax.parser import get_templates ImportError: dlopen: cannot load any more object with static TLS

Currently, score takes only a single input and as a result, the method is very slow if you are trying to classify thousands of examples. Is there a way you can generate scores for 10K+ samples at the same time.

I am experimenting with the library for the first time. The installation was successful and didn't need any extra steps. however when I started importing the library I got the following error related to keras:

/usr/local/lib/python3.7/dist-packages/shorttext/generators/bow/AutoEncodingTopicModeling.py in () 8 from gensim.corpora import Dictionary 9 from keras import Input ---> 10 from keras.engine import Model 11 from keras.layers import Dense 12 from scipy.spatial.distance import cosine

ImportError: cannot import name 'Model' from 'keras.engine' (/usr/local/lib/python3.7/dist-packages/keras/engine/init.py)

I tried to install keras separately but no improvement. any suggestions would be appreciated.

Code: trainclassdict = shorttext.data.nihreports(sample_size=None) topicmodeler = shorttext.generators.LDAModeler() topicmodeler.train(trainclassdict, 128) Error message: /lib/python2.7/site-packages/gensim/models/ldamodel.py:535: RuntimeWarning: overflow encountered in exp2 perwordbound, np.exp2(-perwordbound), len(chunk), corpus_words

Then the results are variable for topicmodeler.retrieve_topicvec('stem cell research')

I noticed that stopwords.txt includes negation terms such as "no" and "not". These terms revert the meaning of a word or a sentence, so they should be preserved in the text data. For example, "not a good idea" would become "good idea" after stopword removal. Therefore, I recommend removing negation terms from the stopword list. Thanks!

I was wondering what should be the format of data as input for:

shorttext.generators.LDAModeler() topicmodeler.train(data, 100)

Can I feed it with a pandas column? Or it should be in a dictionary format? If a dictionary, what should be the keys? I have a large set of tweets.

seems to be maxent is a fancy word for regression or you do have something special in your maxent? https://www.quora.com/What-is-the-relationship-between-Log-Linear-model-MaxEnt-model-and-Logistic-Regression or https://en.wikipedia.org/wiki/Multinomial_logistic_regression

Multinomial logistic regression is known by a variety of other names, including polytomous LR,[2][3] multiclass LR, softmax regression, multinomial logit, the maximum entropy (MaxEnt) classifier, and the conditional maximum entropy model.[4]

SciPy 1.6.0 drops Python 3.6 support https://scipy.github.io/devdocs/release.1.6.0.html So maybe scipy<1.6.0 should be specified in the requirements.txt and setup_requirements.txt files.

Some datasets are not available anymore.

For example the following: nihtraindata = shorttext.data.nihreports(sample_size=None)

Error message:

Downloading...
Source:  http://storage.googleapis.com/pyshorttext/nih_grant_public/nih_full.csv.zip
Failure to download file!
(<class 'urllib.error.HTTPError'>, <HTTPError 404: 'Not Found'>, <traceback object at 0x7f09063ed788>)

Python error:

HTTPError: HTTP Error 404: Not Found

During handling of the above exception, another exception occurred:

When opening the link the same error appears: