Accompanying source code to the paper "Matrix Factorization for Collaborative Filtering is just Solving an Adjoint Latent Dirichlet Allocation Model After All" by Florian Wilhelm. The preprint can be found here along with the following statement:
"Š Florian Wilhelm 2021. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive version was published in RecSys '21: Fifteenth ACM Conference on Recommender Systems Proceedings, https://doi.org/10.1145/3460231.3474266."
Installation
In order to set up the necessary environment:
- review and uncomment what you need in
environment.yml
and create an environmentlda4rec
with the help of conda:conda env create -f environment.yml
- activate the new environment with:
conda activate lda4rec
- (optionally) get a free neptune.ai account for experiment tracking and save the api token under
~/.neptune_api_token
(default).
Running Experiments
First check out and adapt the default experiment config configs/default.yaml
and run it with:
lda4rec -c configs/default.yaml run
A config like configs/default.yaml
can also be used as a template to create an experiment set with:
lda4rec -c configs/default.yaml create -ds movielens-100k
using the Movielens-100k dataset. Check out cli.py
for more details.
Cloud Setup
Commands for setting up an Ubuntu 20.10 VM with at least 20 GiB of HD on e.g. a GCP c2-standard-30 instance:
tmux
sudo apt-get install -y build-essential
curl https://sh.rustup.rs -sSf | sh
source $HOME/.cargo/env
cargo install pueue
curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O
sh Miniconda3-latest-Linux-x86_64.sh
source ~/.bashrc
git clone https://github.com/FlorianWilhelm/lda4rec.git
cd lda4rec
conda env create -f environment.yml
conda activate lda4rec
vim ~/.neptune_api_token # and copy it over
Then create and run all experiments for full control over parallelism with pueue:
pueued -d # only once to start the daemon
pueue parallel 10
export OMP_NUM_THREADS=4 # to limit then number of threads per model
lda4rec -c configs/default.yaml create # to create the config files
find ./configs -maxdepth 1 -name "exp_*.yaml" -exec pueue add "lda4rec -c {} run" \; -exec sleep 30 \;
Remark: -exec sleep 30
avoids race condition when reading datasets if parallelism is too high.
Dependency Management & Reproducibility
- Always keep your abstract (unpinned) dependencies updated in
environment.yml
and eventually insetup.cfg
if you want to ship and install your package viapip
later on. - Create concrete dependencies as
environment.lock.yml
for the exact reproduction of your environment with:conda env export -n lda4rec -f environment.lock.yml
--no-builds
during the export. - Update your current environment with respect to a new
environment.lock.yml
using:conda env update -f environment.lock.yml --prune
Project Organization
âââ AUTHORS.md <- List of developers and maintainers.
âââ CHANGELOG.md <- Changelog to keep track of new features and fixes.
âââ LICENSE.txt <- License as chosen on the command-line.
âââ README.md <- The top-level README for developers.
âââ configs <- Directory for configurations of model & application.
âââ data <- Downloaded datasets will be stored here.
âââ docs <- Directory for Sphinx documentation in rst or md.
âââ environment.yml <- The conda environment file for reproducibility.
âââ notebooks <- Jupyter notebooks. Naming convention is a number (for
â ordering), the creator's initials and a description,
â e.g. `1.0-fw-initial-data-exploration`.
âââ logs <- Generated logs are collected here.
âââ results <- Results as exported from neptune.ai.
âââ setup.cfg <- Declarative configuration of your project.
âââ setup.py <- Use `python setup.py develop` to install for development or
â or create a distribution with `python setup.py bdist_wheel`.
âââ src
â âââ lda4rec <- Actual Python package where the main functionality goes.
âââ tests <- Unit tests which can be run with `py.test`.
âââ .coveragerc <- Configuration for coverage reports of unit tests.
âââ .isort.cfg <- Configuration for git hook that sorts imports.
âââ .pre-commit-config.yaml <- Configuration of pre-commit git hooks.
How to Cite
Please cite LDA4Rec if it helps your research. You can use the following BibTeX entry:
@inproceedings{wilhelm2021lda4rec,
author = {Wilhelm, Florian},
title = {Matrix Factorization for Collaborative Filtering Is Just Solving an Adjoint Latent Dirichlet Allocation Model After All},
year = {2021},
month = sep,
isbn = {978-1-4503-8458-2/21/09},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3460231.3474266},
doi = {10.1145/3460231.3474266},
booktitle = {Fifteenth ACM Conference on Recommender Systems},
numpages = {8},
location = {Amsterdam, Netherlands},
series = {RecSys '21}
}
License
This sourcecode is AGPL-3-only licensed. If you require a more permissive licence, e.g. for commercial reasons, contact me to obtain a licence for your business.
Acknowledgement
Special thanks goes to Du Phan and Fritz Obermeyer from the (Num)Pyro project for their kind help and helpful comments on my code.
Note
This project has been set up using PyScaffold 4.0 and the dsproject extension 0.6. Some source code was taken from Spotlight (MIT-licensed) by Maciej Kula as well as lrann (MIT-licensed) by Florian Wilhelm and Marcel Kurovski.