A Python package that can be used to download post and comment data from Reddit.

Related tags

Third-party APIs Wrappersreddit-data-collector
Overview

Reddit Data Collector

Reddit Data Collector is a Python package that allows a user to collect post and comment data from Reddit. It is built on top of the Python module PRAW, which stands for "The Python Reddit API Wrapper". It aims to make it very simple for a user to collect data from Reddit for further analysis (e.g. Natural Language Processing), without having to learn the inner workings of PRAW or the Reddit API.

The main functionalities provided by the package currently include:

  1. Ability to collect a sample of post data and comment data from Reddit by simply providing the subreddit names that you wish to collect data from.

  2. Ability to convert that data into a pandas DataFrame in order to inspect it and save it for further use.

  3. Ability to seamlessly update an existing .csv file that contains some sample data collected with the package in the past, with some new sample data that is also collected with the package.

It is currently maintained by Nico Van den Hooff.

Installation

Dependencies

Reddit Data Collector requires Python and:

  • pandas (>=1.3.5)
  • praw (>=7.5.0)
  • tqdm (>=4.62.3)

User installation

The recommended way to install Reddit Data Collector is using pip:

pip install reddit-data-collector

How to Use Reddit Data Collector

Please see the examples directory for step by step instructions on how to use Reddit Data Collector.

Development

Important links

Source code

You can check the latest sources with the command:

git clone https://github.com/nicovandenhooff/reddit-data-collector.git

Contributing

To learn more about making a contribution to Reddit Data Collector, please see the contributing file.

Potential Ideas for Contribution

  • Add ability to collect images from Reddit posts that contain them.
  • Add author information to post and comment data, currently the Reddit API is inconsistent with suspended and deleted author data, so this functionality has not been built in yet.

Testing

After installation, you can launch the test suite, which is contained in the tests/tests.py. Note that you will have to have pytest >= 6.2.5 installed. You can launch the test suite by following these steps from the projects root directory:

  1. Open up tests.py with the following command:
open tests/tests.py

Comment out lines 24 to 30. Change the values in DataCollector() in line 32 to your Reddit credentials.

  1. Run the following command:
pytest tests/test.py

Project History

The project was started in January 2022 by Nico Van den Hooff as a side project while he was completing the UBC Master of Data Science Project. Nico wanted to obtain a sample of posts and comments from Reddit, but noticed that while PRAW existed and provided seamless access to Reddit's API, there was no package available that allowed for a simple method to collect this data.

Inspiration

Certain sections of this README file was inspired by the scikit-learn README.

You might also like...
Auto Join: A GitHub action script to automatically invite everyone to the organization who comment at the issue page.

Auto Invite To Org By Issue Comment A GitHub action script to automatically invite everyone to the organization who comment at the issue page. What is

Max Base 6 Jun 8, 2022
Auto Liker, Auto Reaction, Auto Comment, Auto Follower Tool. RajeLiker Credit Hacker.
Auto Liker, Auto Reaction, Auto Comment, Auto Follower Tool. RajeLiker Credit Hacker.

Auto Liker, Auto Reaction, Auto Comment, Auto Follower Tool. RajeLiker Credit Hacker. Unlimited RajeLiker Credit Hack. Thanks To RajeLiker.

Md. Mehedi Hasan 32 Dec 28, 2022
A simple Discord bot that can fetch definitions and post them in chat.
A simple Discord bot that can fetch definitions and post them in chat.

A simple Discord bot that can fetch definitions and post them in chat. If you are connected to a voice channel, the bot will also read out the definition to you.

Tycho Bellers 4 Sep 29, 2022
A simple fun discord bot using discord.py that can post memes

A simple fun discord bot using discord.py * * Commands $commands - to see all commands $meme - for a random meme from the internet $cry - to make the

Dice Flip 2 Dec 20, 2021
One version package to rule them all, One version package to find them, One version package to bring them all, and in the darkness bind them.

AwesomeVersion One version package to rule them all, One version package to find them, One version package to bring them all, and in the darkness bind

Joakim Sørensen 39 Dec 31, 2022
A simple script & container to pull COVID data from covidlive.com.au and post a summary to a slack channel
A simple script & container to pull COVID data from covidlive.com.au and post a summary to a slack channel

CovidLive AU Summary Slackbot This bot is a very simple slackbot that pulls data, summarises and posts up to date AU COVID stats to a provided slack c

James 3 Dec 18, 2021
Track live sentiment for stocks from Reddit and Twitter and identify growing stocks
Track live sentiment for stocks from Reddit and Twitter and identify growing stocks

Market Sentiment About This repository can mainly be used for two things. a. Tracking the live sentiment of stocks from Reddit and Twitter b. Tracking

Market Sentiment 345 Dec 17, 2022
A reddit.com bot that will return reference links from official python documentation site for the standard library.

Python Docs Bot A reddit.com bot that will return documentation links for the library and language reference sections of the python docs website. The

Trevor Miller 2 Sep 14, 2021
A Python bot that uses the Reddit API to send users inspiring messages.

AnonBot By Edric Antoine A Python bot that uses the Reddit API to send users inspiring messages. When a message includes 'What would Anon do?', the bo

null 1 Jan 5, 2022
Releases(v1.1.0)

  • v1.1.0(Mar 14, 2022)

    [1.1.0] - 2022-03-13

    Changed

    • Changed get_data in reddit_data_collector.py to return pandas DataFrame by default
    • Updated tests for the above
    Source code(tar.gz)
    Source code(zip)

  • v1.0.2(Jan 15, 2022)

    [1.0.2] - 2022-01-14

    Fixed

    • Updated _check_subreddit_exists in reddit_data_collector.py to check both names as .lower()
    • Updated tests for the above

    Changed

    • Updated README to include instructions on coverage tests
    Source code(tar.gz)
    Source code(zip)

  • v1.0.1(Jan 12, 2022)

    [1.0.1] - 2022-01-12

    Fixed

    • Spelling error of separate argument in to_pandas function of reddit_data_collector.io.py, previously it was spelt like seperate

    Changed

    • Update example use and move to /examples
    • Update PyPi link in docs to working link
    • Add new potential ideas for contribution
    Source code(tar.gz)
    Source code(zip)

  • v1.0.0(Jan 7, 2022)

Owner
Nico Van den Hooff
UBC Master of Data Science
Nico Van den Hooff
GitHub Repository
Bot Maker For Discord - Python Edition

BMFD-PE Bot Maker For Discord - Python Edition BMFD-PE is a new version of BMFD write in Python The Version of BMFD-PE is : alpha0.1 Longer support :

Téo 2 Dec 22, 2021
An Async Bot/API wrapper for Twitch made in Python.

TwitchIO is an asynchronous Python wrapper around the Twitch API and IRC, with a powerful command extension for creating Twitch Chat Bots. TwitchIO co

TwitchIO 590 Jan 03, 2023
Slack bot to automatically delete yubisneeze / accidental yubikey presses

YubiSnooze Slack bot to automatically delete yubisneeze / accidental yubikey presses. It will search using the regex "[cbdefghijklnrtuv]{44}" and if t

Andrew MacPherson 3 Feb 09, 2022
A new coin listing alert bot using Python, Flask, MongoDB, Telegram API and Binance API

Bzzmans New Coin Listing Detection Bot Architecture About Project Work in progress. This bot basically gets new coin listings from Binance using Binan

Eyüp Barlas 21 May 31, 2022
a discord libary that use to make discord bot with low efficiency and bad performance because I don't know how to manage the project

Aircord 🛩️ a discord libary that use to make discord bot with low efficiency and bad performance because I don't know how to manage the project Examp

Aircord 2 Oct 24, 2021
go-cqhttp API typing annoations, return data models and utils for nonebot

go-cqhttp API typing annoations, return data models and utils for nonebot

风屿 6 Jan 04, 2023
Bender: A Markov Babbler Slack Bot

See the Digital Ocean tutorial for instructions on how to get the basic bot structure in place. Once you have that, set the gunicorn command to run as

Andrew Howard 1 Dec 04, 2021
Telegram Bot to Filter posts in Bot Inline search

Inline-Filter-Bot A Telegram Bot for filter in Inline Features Unlimited Filters Supports all type of filters Supports Alert Button Using Common Marku

Code X Botz 67 Dec 26, 2022
Web3 Pancakeswap Sniper & honeypot detector Take Profit/StopLose bot written in python3, For ANDROID WIN MAC & LINUX

Web3 Pancakeswap Sniper & honeypot detector Take Profit/StopLose bot written in python3, For ANDROID WIN MAC & LINUX

HYDRA 3 Dec 27, 2021
A Python script that exports users from one Telegram group to another using one or more concurrent user bots.

ExportTelegramUsers A Python script that exports users from one Telegram group to another using one or more concurrent user bots. Make sure to set all

Fasil Minale 17 Jun 26, 2022
This software's intent is to automate all activities related to manage Axie Infinity Scholars. It is specially aimed to mangers with large scholar roasters.

Axie Scholars Utilities This software's intent is to automate all activities related to manage Scholars. It is specially aimed to mangers with large s

Ferran Marin 153 Nov 16, 2022
A solution designed to extract, transform and load Chicago crime data from an RDS instance to other services in AWS.

This project is intended to implement a solution designed to extract, transform and load Chicago crime data from an RDS instance to other services in AWS.

Yesaswi Avula 1 Feb 04, 2022
Experimental bridges between Telegram calls and other platforms.

Bridges by Calls Music Experimental bridges between Telegram calls and other platforms. Current bridges Bridge 1 (YouTube, Twitch, Facebook, etc...) B

Calls Music 14 Oct 08, 2022
A simple library for interacting with Amazon S3.

BucketStore is a very simple Amazon S3 client, written in Python. It aims to be much more straight-forward to use than boto3, and specializes only in

Jacobi Petrucciani 219 Oct 03, 2022
Python library for the Stripe API.

Stripe Python Library The Stripe Python library provides convenient access to the Stripe API from applications written in the Python language. It incl

Stripe 1.3k Jan 03, 2023
The gPodder podcast client.

___ _ _ ____ __ _| _ \___ __| |__| |___ _ _ |__ / / _` | _/ _ \/ _` / _` / -_) '_| |_ \ \__, |_| \___/\__,_\__,_\___|_| |_

gPodder and related projects 1.1k Jan 04, 2023
Tracks twitter spaces and sends it to a discord webhook.

Tracks twitter spaces and sends it to a discord webhook. Uses the twitter api to find twitter spaces and then the m3u8 url for the space is found using selenium and will have it posted using a discor

Sam Phung 20 Dec 17, 2022
Python client for the iNaturalist APIs

pyinaturalist Introduction iNaturalist is a community science platform that helps people get involved in the natural world by observing and identifyin

Nicolas Noé 79 Dec 22, 2022
The easiest way to deploy this Bot

How To Host The easiest way to deploy this Bot Update Channe

Isekai Reszz 1 Jan 23, 2022
A tool for exporting Telegram group chats into static websites, preserving chat history like mailing list archives.

tg-archive is a tool for exporting Telegram group chats into static websites, preserving chat history like mailing list archives. Preview The @fossuni

Kailash Nadh 400 Dec 27, 2022