Hi,

I was playing with HDP models and I wanted to visualise them with pyLDAvis. Unfortunately it wasn't natively supported, so I made the few fixes to make it.

I am simply looking for attributes lda_alpha and lda_beta as they are specific to the HDP model. I also changed the __num_dist_rows__ function because even if the matrix was normalized, the doc_topic_dists made the asserts crying (maybe NaN values ?). I didn't look too much into it but this fix is working.

I am not sure if the lda_beta parameter is exactly the same as state.get_lambda() parameter but it needed the topic-term distribution so I though it was ok...

I am using it and it's working. Tell me if the PR seems right to you ! :)

I run my commands in the IPython notebook. when I use command pyLDAvis.show(LDAvis_prepared),there's only a red out[], and doesn't show anything!

then I use command pyLDAvis.show(LDAvis_prepared), it shows the answer. but, it advises me to use pyLDAvis.display()

Hi there, I'm using gensim to do LDA on a collection of novels (using just 40 for testing, I have several hundreds). Building the corpus and dictionary seems to work fine, as does the modeling process itself. I can also inspect the resulting model (topics in documents and words in topics, for example). However, when attempting to use pyLDAvis, I run into a KeyError.

I'm on Linux (Ubuntu 14.04) and using Python 3.4 and the following versions of relevant modules: pyLDAvis 1.2.0 numpy 1.9.2 gensim 0.11.1-1

This is my code (loading corpus, dictionary and model from previous step):

def gensim_output(modelfile, corpusfile, dictionaryfile): 
    """Displaying gensim topic models"""
    ## Load files from "gensim_modeling"
    corpus = corpora.MmCorpus(corpusfile)
    dictionary = corpora.Dictionary.load(dictionaryfile) # for pyLDAvis
    myldamodel = models.ldamodel.LdaModel.load(modelfile)    

    ## Interactive visualisation
    import pyLDAvis.gensim
    vis = pyLDAvis.gensim.prepare(myldamodel, corpus, dictionary)
    pyLDAvis.display(vis)

This is the output I get:

Traceback (most recent call last):

  File "<ipython-input-79-940daa51d8a9>", line 1, in <module>
    runfile('/home/[PATH]/an5/mygensim.py', wdir='/home/christof/Dropbox/0-Analysen/2015/rp_Sydney/an5')

  File "/usr/lib/python3/dist-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 586, in runfile
    execfile(filename, namespace)

  File "/usr/lib/python3/dist-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 48, in execfile
    exec(compile(open(filename, 'rb').read(), filename, 'exec'), namespace)

  File "/home/[PATH]/an5/mygensim.py", line 84, in <module>
    main("./5_lemmata/*.txt", "gensim_corpus.dict", "gensim_corpus.mm", "gensim_modelfile.gensim")

  File "/home/[PATH]/an5/mygensim.py", line 82, in main
    gensim_output(modelfile, corpusfile, dictionaryfile)

  File "/home/[PATH]/an5/mygensim.py", line 75, in gensim_output
    vis = pyLDAvis.gensim.prepare(myldamodel, corpus, dictionary)

  File "/usr/local/lib/python3.4/dist-packages/pyLDAvis/gensim.py", line 61, in prepare
    return vis_prepare(**_extract_data(topic_model, corpus, dictionary))

  File "/usr/local/lib/python3.4/dist-packages/pyLDAvis/gensim.py", line 24, in _extract_data
    term_freqs = [term_freqs_dict[id] for id in xrange(N)]

  File "/usr/local/lib/python3.4/dist-packages/pyLDAvis/gensim.py", line 24, in <listcomp>
    term_freqs = [term_freqs_dict[id] for id in xrange(N)]

KeyError: 6

Not sure whether this is a bug or bad usage of the module. Any help would be very much appreciated.

Preparing a gensim lda model does not work for me (Linux, Python 3.4) because of the following error: `` pyLDAvis.gensim.prepare(lda, corpus, dictionary)

TypeError Traceback (most recent call last) in () ----> 1 pyLDAvis.gensim.prepare(lda, corpus, dictionary)

/home/methodds/anaconda3/lib/python3.4/site-packages/pyLDAvis/gensim.py in prepare(topic_model, corpus, dictionary, **kargs) 64 http://nbviewer.ipython.org/github/bmabey/pyLDAvis/blob/master/notebooks/Gensim%20Newsgroup.ipynb 65 """ ---> 66 opts = fp.merge(_extract_data(topic_model, corpus, dictionary), kargs) 67 return vis_prepare(**opts)

/home/methodds/anaconda3/lib/python3.4/site-packages/pyLDAvis/gensim.py in _extract_data(topic_model, corpus, dictionary) 30 31 topics = topic_model.show_topics(formatted=False, num_words=len(vocab), num_topics=topic_model.num_topics) ---> 32 topics_df = pd.DataFrame([dict((y,x) for x, y in tuples) for tuples in topics])[vocab] 33 topic_term_dists = topics_df.values 34

/home/methodds/anaconda3/lib/python3.4/site-packages/pyLDAvis/gensim.py in (.0) 30 31 topics = topic_model.show_topics(formatted=False, num_words=len(vocab), num_topics=topic_model.num_topics) ---> 32 topics_df = pd.DataFrame([dict((y,x) for x, y in tuples) for tuples in topics])[vocab] 33 topic_term_dists = topics_df.values 34

/home/methodds/anaconda3/lib/python3.4/site-packages/pyLDAvis/gensim.py in (.0) 30 31 topics = topic_model.show_topics(formatted=False, num_words=len(vocab), num_topics=topic_model.num_topics) ---> 32 topics_df = pd.DataFrame([dict((y,x) for x, y in tuples) for tuples in topics])[vocab] 33 topic_term_dists = topics_df.values 34

TypeError: 'int' object is not iterable `` Any idea what is going on here?

Hello,

I am running into a visualization issue when running pyLDAvis.display() with any lda visualization from pyLDAvis.gensim.prepare().

Here is an example output from running this notebook http://nbviewer.ipython.org/github/bmabey/pyLDAvis/blob/master/notebooks/Gensim%20Newsgroup.ipynb

I have installed pyLDAvis 3.2.0 via pip. My OS is MacOS Big Sur v 11.1 and I am running this on python 3.8.5.

If I can provide any additional details to help please let me know!

When passing a GSDMM short text clustering model to pyLDAvis for visualisation, I sometimes get 'divide by zero' warnings even though the visualisation is created successfully. How can these be resolved? Is it because of a small corpus? I am usually building these models on around 100 documents containing 10-15 tokens each. Screenshot attached, would appreciate help on this!

I am using Python 3.7 on MacOS Catalina version 10.15.3.

I propose to remove the dependencies on scikit-bio.

scikit-bio has recently undergo incompatible changes to the API, especially with regards to the pcoa() function. In addition, related to issue 57, it is still incompatible with Windows machines.

After going through the codes, I see that only the pcoa() and DistanceMatrix() functions are used from the scikit-bio package. These can be reimplemented with functions from scikit-learn only. Given the maturity of the sklearn package, it should be a good idea.

I can try to implement these portions if necessary.

Enhancement for https://github.com/bmabey/pyLDAvis/issues/89

Please review.

Features:

• Init Button: Initial the topics with default valuues. eg: topic-0
• Load Button: Loads a json file with array of strings of size K
• Save Button: Downloads the current topic modelling (array of string of size K) to a json file.

I keep getting AttributeError when attempting to run:

import pyLDAvis

# ... creating LDA model, corpus, and dictionary in here
pyLDAvis.gensim.prepare(ldamodel, corpus, dictionary)

Here's the traceback:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-21-950ae09ed90b> in <module>()
----> 1 pyLDAvis.gensim.prepare(ldamodel, corpus, dictionary)

AttributeError: module 'pyLDAvis' has no attribute 'gensim'

However, the code runs fine when importing gensim explicitly:

from pyLDAvis import gensim

# ... same as above
gensim.prepare(ldamodel, corpus, dictionary)

Unsure why that would be the case? Is it possible that gensim would need to be added to __init__.py?

First and foremost wanted to thank everyone for helping me get this far. I am able to generate a gensim model, run it in IPython notebook, and get to see some results - but not the beautiful graphic we all were hoping for. I'm running WinPython 3.4 QT5 (latest I believe) and I installed both the genism and pyLDAvis also today, so everything is fresh. Here is what my output looks like:

In[9]: pyLDAvis.enable_notebook() In[10]: pyLDAvis.gensim.prepare(lda, corpus, dictionary) C:\WinPython\python-3.4.3.amd64\lib\site-packages\skbio\stats\ordination_principal_coordinate_analysis.py:109: RuntimeWarning: The result contains negative eigenvalues. Please compare their magnitude with the magnitude of some of the largest positive eigenvalues. If the negative ones are smaller, it's probably safe to ignore them, but if they are large in magnitude, the results won't be useful. See the Notes section for more details. The smallest eigenvalue is -0.009952346420900118 and the largest is 0.034359155356682575. RuntimeWarning Out[10]: PreparedData(topic_coordinates= Freq cluster topics x y topic
24 20.249823 1 1 0.055595 0.006318 3 18.859849 1 2 0.003016 0.038028 17 16.519686 1 3 -0.117297 0.009020 18 9.578099 1 4 -0.014738 0.006581 .... 13 0.000586 1 24 0.003268 0.007722 9 0.000586 1 25 -0.008638 -0.009829, topic_info= Category Freq Term Total loglift logprob 2300 Default 1280.000000 gladia 1280 30.0000 30.0000 5920 Default 984.000000 giskard 984 29.0000 29.0000 1512 Default 676.000000 amadiro 676 28.0000 28.0000 ... 2252 Topic25 0.000562 anacreon 117 -0.3992 -6.2565 9440 Topic25 0.000626 madam 372 -1.5745 -6.2751

[1929 rows x 6 columns], token_table= Topic Freq Term term
3268 1 0.181818 abilities 3268 2 0.318182 abilities ... 1155 10 0.019608 york

[1984 rows x 3 columns], R=30, lambda_step=0.01, plot_opts={'ylab': 'PC2', 'xlab': 'PC1'}, topic_order=[25, 4, 18, 19, 21, 11, 23, 17, 22, 8, 12, 24, 15, 13, 2, 7, 20, 9, 16, 3, 6, 5, 1, 14, 10])

CHANGED scale.linear(...) TO scaleLinear(...)

CHANGED .axis(...).scale(...).orient(...) TO axisLeft (y)(...), axisTop(...) (x) or axisBottom(...) (scaleX)

CHANGED d3.scaleordinal(...).domain(...).rangeRoundBands(...) TO d3.scaleBand().domain(...).rangeBands(...).padding(...)

CHANGED y.rangeBand(...) TO y.bandwidth(...)

REPLACED d3.v3.js with d3.v5.js

ADDED: optional function parameters that let user specify primary (color1) and secondary (color2) colors. (Default colors are originals)

This should also be compatible with d3.v4 but I havent tested it yet.

Error when using scikit-learn >= 1.2.0

pyLDAvis.sklearn.prepare raises an error due to a missing method get_feature_names() for the vectorizer argument.

AttributeError: 'CountVectorizer' object has no attribute 'get_feature_names'

Using the documentation of sklearn.feature_extraction.text.CountVectorizer as an example. It can be seen this function was deprecated in 1.0 docs, and removed in 1.2 docs. The same is true for the other vectorizer that can be used TfidfVectorizer.

The recommendation in those docs is to use get_feature_names_out() as a replacement.

Instead of returning a list of feature names, this now returns an ndarray of them. Though both being iterable types it makes no difference for the use case, where reference is only required to array-like.

This fix would also be backwards compatible to at least scikit-learn 1.0.

Tested on a fresh conda environment with Python==3.10.8, and gives expected behaviour.

When running the same steps with the same data on colab I get pretty good results with tsne, but locally (probably because of the Python version) I'm not able to run pyLDAvis.sklearn.prepare as I get ValueError: perplexity must be less than n_samples.

I know that colab is running 3.7 and locally I got 3.10 . I also know that both use pyLDAvis version 3.3.1 so its probably broken because of a Scikit update.

I was able to get it to work by manually setting a perplexity value in the TSNE object initialization under pyLDAvis/_prepare.py but it sure isn´t optimal.

Traceback (most recent call last):
  File "/home/isinyaaa/projects/foss-gpgpu-stack/analyze.py", line 418, in <module>
    main(args)
  File "/home/isinyaaa/projects/foss-gpgpu-stack/analyze.py", line 347, in main
    args.workers).run(vectorizer, processed_data)
  File "/home/isinyaaa/projects/foss-gpgpu-stack/analyze.py", line 128, in run
    self.save_result_as_html(model, data, vectorizer)
  File "/home/isinyaaa/projects/foss-gpgpu-stack/analyze.py", line 148, in save_result_as_html
    super().save_result_as_html(prepare, model, data, vectorizer, mds='tsne')
  File "/home/isinyaaa/projects/foss-gpgpu-stack/analyze.py", line 111, in save_result_as_html
    LDAvis_prepared = prepare(*args, **kwargs)
  File "/home/isinyaaa/.local/lib/python3.10/site-packages/pyLDAvis/sklearn.py", line 95, in prepare
    return pyLDAvis.prepare(**opts)
  File "/home/isinyaaa/.local/lib/python3.10/site-packages/pyLDAvis/_prepare.py", line 443, in prepare
    topic_coordinates = _topic_coordinates(mds, topic_term_dists, topic_proportion, start_index)
  File "/home/isinyaaa/.local/lib/python3.10/site-packages/pyLDAvis/_prepare.py", line 192, in _topic_coordinates
    mds_res = mds(topic_term_dists)
  File "/home/isinyaaa/.local/lib/python3.10/site-packages/pyLDAvis/_prepare.py", line 167, in js_TSNE
    return model.fit_transform(dist_matrix)
  File "/home/isinyaaa/.local/lib/python3.10/site-packages/sklearn/manifold/_t_sne.py", line 1122, in fit_transform
    self._check_params_vs_input(X)
  File "/home/isinyaaa/.local/lib/python3.10/site-packages/sklearn/manifold/_t_sne.py", line 793, in _check_params_vs_input
    raise ValueError("perplexity must be less than n_samples")
ValueError: perplexity must be less than n_samples

Bumps joblib from 1.0.1 to 1.2.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

In pyLDAvis_overview.ipynb, when it says "from the original R project and this presentation (slides, video)", the link to the Youtube video doesn't work anymore :(

Any backup?

When number of topics is less than the default perplexity for TSNE, an error is thrown. This reduces perplexity to be always smaller than the number of topics.