A Python package that can be used to download post and comment data from Reddit.

Related tags

Third-party APIs Wrappersreddit-data-collector
Overview

Reddit Data Collector

Reddit Data Collector is a Python package that allows a user to collect post and comment data from Reddit. It is built on top of the Python module PRAW, which stands for "The Python Reddit API Wrapper". It aims to make it very simple for a user to collect data from Reddit for further analysis (e.g. Natural Language Processing), without having to learn the inner workings of PRAW or the Reddit API.

The main functionalities provided by the package currently include:

  1. Ability to collect a sample of post data and comment data from Reddit by simply providing the subreddit names that you wish to collect data from.

  2. Ability to convert that data into a pandas DataFrame in order to inspect it and save it for further use.

  3. Ability to seamlessly update an existing .csv file that contains some sample data collected with the package in the past, with some new sample data that is also collected with the package.

It is currently maintained by Nico Van den Hooff.

Installation

Dependencies

Reddit Data Collector requires Python and:

  • pandas (>=1.3.5)
  • praw (>=7.5.0)
  • tqdm (>=4.62.3)

User installation

The recommended way to install Reddit Data Collector is using pip:

pip install reddit-data-collector

How to Use Reddit Data Collector

Please see the examples directory for step by step instructions on how to use Reddit Data Collector.

Development

Important links

Source code

You can check the latest sources with the command:

git clone https://github.com/nicovandenhooff/reddit-data-collector.git

Contributing

To learn more about making a contribution to Reddit Data Collector, please see the contributing file.

Potential Ideas for Contribution

  • Add ability to collect images from Reddit posts that contain them.
  • Add author information to post and comment data, currently the Reddit API is inconsistent with suspended and deleted author data, so this functionality has not been built in yet.

Testing

After installation, you can launch the test suite, which is contained in the tests/tests.py. Note that you will have to have pytest >= 6.2.5 installed. You can launch the test suite by following these steps from the projects root directory:

  1. Open up tests.py with the following command:
open tests/tests.py

Comment out lines 24 to 30. Change the values in DataCollector() in line 32 to your Reddit credentials.

  1. Run the following command:
pytest tests/test.py

Project History

The project was started in January 2022 by Nico Van den Hooff as a side project while he was completing the UBC Master of Data Science Project. Nico wanted to obtain a sample of posts and comments from Reddit, but noticed that while PRAW existed and provided seamless access to Reddit's API, there was no package available that allowed for a simple method to collect this data.

Inspiration

Certain sections of this README file was inspired by the scikit-learn README.

You might also like...
Auto Join: A GitHub action script to automatically invite everyone to the organization who comment at the issue page.

Auto Invite To Org By Issue Comment A GitHub action script to automatically invite everyone to the organization who comment at the issue page. What is

Max Base 6 Jun 8, 2022
Auto Liker, Auto Reaction, Auto Comment, Auto Follower Tool. RajeLiker Credit Hacker.
Auto Liker, Auto Reaction, Auto Comment, Auto Follower Tool. RajeLiker Credit Hacker.

Auto Liker, Auto Reaction, Auto Comment, Auto Follower Tool. RajeLiker Credit Hacker. Unlimited RajeLiker Credit Hack. Thanks To RajeLiker.

Md. Mehedi Hasan 32 Dec 28, 2022
A simple Discord bot that can fetch definitions and post them in chat.
A simple Discord bot that can fetch definitions and post them in chat.

A simple Discord bot that can fetch definitions and post them in chat. If you are connected to a voice channel, the bot will also read out the definition to you.

Tycho Bellers 4 Sep 29, 2022
A simple fun discord bot using discord.py that can post memes

A simple fun discord bot using discord.py * * Commands $commands - to see all commands $meme - for a random meme from the internet $cry - to make the

Dice Flip 2 Dec 20, 2021
One version package to rule them all, One version package to find them, One version package to bring them all, and in the darkness bind them.

AwesomeVersion One version package to rule them all, One version package to find them, One version package to bring them all, and in the darkness bind

Joakim Sørensen 39 Dec 31, 2022
A simple script & container to pull COVID data from covidlive.com.au and post a summary to a slack channel
A simple script & container to pull COVID data from covidlive.com.au and post a summary to a slack channel

CovidLive AU Summary Slackbot This bot is a very simple slackbot that pulls data, summarises and posts up to date AU COVID stats to a provided slack c

James 3 Dec 18, 2021
Track live sentiment for stocks from Reddit and Twitter and identify growing stocks
Track live sentiment for stocks from Reddit and Twitter and identify growing stocks

Market Sentiment About This repository can mainly be used for two things. a. Tracking the live sentiment of stocks from Reddit and Twitter b. Tracking

Market Sentiment 345 Dec 17, 2022
A reddit.com bot that will return reference links from official python documentation site for the standard library.

Python Docs Bot A reddit.com bot that will return documentation links for the library and language reference sections of the python docs website. The

Trevor Miller 2 Sep 14, 2021
A Python bot that uses the Reddit API to send users inspiring messages.

AnonBot By Edric Antoine A Python bot that uses the Reddit API to send users inspiring messages. When a message includes 'What would Anon do?', the bo

null 1 Jan 5, 2022
Releases(v1.1.0)

  • v1.1.0(Mar 14, 2022)

    [1.1.0] - 2022-03-13

    Changed

    • Changed get_data in reddit_data_collector.py to return pandas DataFrame by default
    • Updated tests for the above
    Source code(tar.gz)
    Source code(zip)

  • v1.0.2(Jan 15, 2022)

    [1.0.2] - 2022-01-14

    Fixed

    • Updated _check_subreddit_exists in reddit_data_collector.py to check both names as .lower()
    • Updated tests for the above

    Changed

    • Updated README to include instructions on coverage tests
    Source code(tar.gz)
    Source code(zip)

  • v1.0.1(Jan 12, 2022)

    [1.0.1] - 2022-01-12

    Fixed

    • Spelling error of separate argument in to_pandas function of reddit_data_collector.io.py, previously it was spelt like seperate

    Changed

    • Update example use and move to /examples
    • Update PyPi link in docs to working link
    • Add new potential ideas for contribution
    Source code(tar.gz)
    Source code(zip)

  • v1.0.0(Jan 7, 2022)

Owner
Nico Van den Hooff
UBC Master of Data Science
Nico Van den Hooff
GitHub Repository
Python Library to Extract youtube video Tags without Youtube API

YoutubeTags Python Library to Extract youtube video Tags without Youtube API Installation pip install YoutubeTags Example import YoutubeTags from Yout

Nuhman Pk 17 Nov 12, 2022
Filters to block and remove copycat-websites from DuckDuckGo and Google. Specific to dev websites like StackOverflow or GitHub.

uBlock-Origin-dev-filter Filters to block and remove copycat-websites from DuckDuckGo and Google. Specific to dev websites like StackOverflow or GitHu

1.7k Dec 30, 2022
A github actions + python code to extract URLs to code repositories to put into standard form, starting with github

A github actions + python code to extract URLs to code repositories to put into standard form, starting with github ---- NOTE: JUS

Justin Gosses 2 Nov 15, 2021
A Tᴇʟᴇɢʀᴀᴍ Vɪᴅᴇᴏ Pʟᴀʏᴇʀ Bᴏᴛ Tᴏ Pʟᴀʏ YT Vɪᴅᴇᴏs & Lɪᴠᴇ Sᴛʀᴇᴀᴍ.

Tuktuky_Music Telegram bot to stream videos in telegram voicechat for both groups and channels. Supports live strams, YouTube videos and telegram medi

TᑌKTᑌKY ᖇᗩᕼᗰᗩᑎ 3 Sep 14, 2021
An EmbedBuilder for Discord bots in Python.

An EmbedBuilder for Discord bots in Python. You need discord.py to use this module.

6 Jan 13, 2022
A bot which is a ghost and you can make friends with it

This is a bot which is a ghost and you can make friends with it. It will haunt your friends. Explore and test the bot in replit !

Siwan SR 0 Oct 06, 2022
汪汪Bot是一个Telegram Bot，用于帮助你管理一台服务器上的Docker

WangWangBot 汪汪Bot是一个Telegram Bot，用于帮助你管理一台服务器上的Docker运行的Bot。这是使用视频： 部署说明 安装运行 准备 local.env 普通版本 BOT_TOKEN=你的BOT_TOKEN ADMINS=使用,分隔的管理员ID列表 如果你使用doppl

老房东的代码练习册 56 Aug 23, 2022
OpenAPI Generator allows generation of API client libraries (SDK generation), server stubs, documentation and configuration automatically given an OpenAPI Spec (v2, v3)

OpenAPI Generator Master (5.3.1): 5.4.x (5.4.x): 6.0.x (6.0.x): ⭐ ⭐ ⭐ If you would like to contribute, please refer to guidelines and a list of open t

OpenAPI Tools 14.8k Jan 04, 2023
Data from popular CS:GO website hltv.org

Welcome to hltv-data 👋 🎮 Data from popular CS:GO website hltv.org Install pip install hltv-data Usage The public methods can be reached using HLTVCl

Dariusz Choruży 28 Dec 23, 2022
A bot can play all variants, but standard are abit weak, so if you need strongest you can change fsf instead of stockfish_14_Dev

MAINTAINERS Drdisrespect1 and drrespectable lichess-bot Engine communication code taken from https://github.com/ShailChoksi/lichess-bot by ShailChoksi

RPNS Nimsilu 1 Dec 12, 2021
A Wrapper for ScarletAPI

ScarletAPI A Wrapper for ScarletAPI still a work in progress Docs these are the

Amashi 0 Mar 24, 2022
A Telegram bot that can stream Telegram files to users over HTTP.

T.ME_FILE_TO_LINK Hi iam a file to link bot....best Bot telegram Telegram File To Link Generation Bot A Telegram bot that can stream Telegram files to

1 Oct 24, 2021
universal messaging & notifications api

Pronounced "boat-shahft" What is botschaft? Botschaft is unified messaging & notifications appliance. Want to text yourself when a long-running task c

Tyler M. Kontra 25 Aug 16, 2022
A Telegram bot to index Chinese and Japanese group contents, works with @lilydjwg/luoxu.

luoxu-bot luoxu-bot 是类似于 luoxu-web 的 CJK 友好的 Telegram Bot，依赖于 luoxu 所创建的后端。 测试环境 Python 3.7.9 pip 21.1.2 开发中使用到的 Telethon 需要 Python 3+ 配置 前往 luoxu 根据相

TigerBeanst 10 Nov 18, 2022
Use GitHub Actions to create a serverless service.

ActionServerless - Use GitHub Actions to create a serverless service ActionServerless is an action to do some computing and then generate a string/JSO

107 Oct 28, 2022
Open API to list Viet Nam administrative divisions

Viet Nam province API Homepage: https://provinces.open-api.vn This is online tool to let my VietnamProvinces library reach more users. VietnamProvince

Nguyễn Hồng Quân 52 Dec 05, 2022
An API that uses NLP and AI to let you predict possible diseases and symptoms based on a prompt of what you're feeling.

Disease detection API for MediSearch An API that uses NLP and AI to let you predict possible diseases and symptoms based on a prompt of what you're fe

Sebastian Ponce 1 Jan 15, 2022
A Discord bot to play bluffing games like Dobbins or Bobbins

Usage: pip install -r requirements.txt python3 bot.py DISCORD_BOT_TOKEN Gameplay: All commands are case-insensitive, with trailing punctuation and spa

4 May 27, 2022
A Python script that wraps the gitleaks tool to enable scanning of multiple repositories in parallel

mpgitleaks A Python script that wraps the gitleaks tool to enable scanning of multiple repositories in parallel. The motivation behind writing this sc

Emilio Reyes 7 Dec 29, 2022
Generate discord nitro codes and check them

Discord Nitro Generator and Checker A discord nitro generator and checker for all your nitro needs Explore the docs » Report Bug · Request Feature · J

509 Jan 02, 2023