Reddit Data Collector
Reddit Data Collector is a Python package that allows a user to collect post and comment data from Reddit. It is built on top of the Python module PRAW, which stands for "The Python Reddit API Wrapper". It aims to make it very simple for a user to collect data from Reddit for further analysis (e.g. Natural Language Processing), without having to learn the inner workings of PRAW or the Reddit API.
The main functionalities provided by the package currently include:
-
Ability to collect a sample of post data and comment data from Reddit by simply providing the subreddit names that you wish to collect data from.
-
Ability to convert that data into a pandas
DataFramein order to inspect it and save it for further use. -
Ability to seamlessly update an existing .csv file that contains some sample data collected with the package in the past, with some new sample data that is also collected with the package.
It is currently maintained by Nico Van den Hooff.
Installation
Dependencies
Reddit Data Collector requires Python and:
- pandas (>=1.3.5)
- praw (>=7.5.0)
- tqdm (>=4.62.3)
User installation
The recommended way to install Reddit Data Collector is using pip:
pip install reddit-data-collector
How to Use Reddit Data Collector
Please see the examples directory for step by step instructions on how to use Reddit Data Collector.
Development
Important links
- Official source code repo: https://github.com/nicovandenhooff/reddit-data-collector
- Downloaded releases: https://pypi.org/project/reddit-data-collector/
- Issue tracker: https://github.com/nicovandenhooff/reddit-data-collector/issues
Source code
You can check the latest sources with the command:
git clone https://github.com/nicovandenhooff/reddit-data-collector.git
Contributing
To learn more about making a contribution to Reddit Data Collector, please see the contributing file.
Potential Ideas for Contribution
- Add ability to collect images from Reddit posts that contain them.
- Add author information to post and comment data, currently the Reddit API is inconsistent with suspended and deleted author data, so this functionality has not been built in yet.
Testing
After installation, you can launch the test suite, which is contained in the tests/tests.py. Note that you will have to have pytest >= 6.2.5 installed. You can launch the test suite by following these steps from the projects root directory:
- Open up
tests.pywith the following command:
open tests/tests.py
Comment out lines 24 to 30. Change the values in DataCollector() in line 32 to your Reddit credentials.
- Run the following command:
pytest tests/test.py
Project History
The project was started in January 2022 by Nico Van den Hooff as a side project while he was completing the UBC Master of Data Science Project. Nico wanted to obtain a sample of posts and comments from Reddit, but noticed that while PRAW existed and provided seamless access to Reddit's API, there was no package available that allowed for a simple method to collect this data.
Inspiration
Certain sections of this README file was inspired by the scikit-learn README.
6 Jun 8, 2022
32 Dec 28, 2022
4 Sep 29, 2022
2 Dec 20, 2021
39 Dec 31, 2022
3 Dec 18, 2021
345 Dec 17, 2022
2 Sep 14, 2021
1 Jan 5, 2022
4 Jul 09, 2022
23 Nov 12, 2022
2 Feb 11, 2022
437 Dec 29, 2022
22 Dec 26, 2022
123 Jan 01, 2023
304 Jan 01, 2023
3 Nov 18, 2021
1 Dec 21, 2021
477 Dec 31, 2022
1 Dec 18, 2021
10 Mar 16, 2022
1.8k Jan 09, 2023
17 Nov 13, 2022
921 Jan 08, 2023
6 Apr 29, 2022
4 Jan 03, 2022
149 Jun 30, 2022
5 Aug 23, 2022
12 Nov 12, 2022