Example of scraping a paginated API endpoint and dumping the data into a DB.
# install python dependencies
pipenv install
# copy and update env variables
cp .env.example .env
# initialize venv
pipenv shell
# execute script
python scrape_providers.py
QuickInfo PYPI link : https://pypi.org/project/quickinfo/ This is the library that you've all been searching for, it's built for developers and allows
This is a webscraper for a specific website. It is tuned to extract the headlines of that website. With some little adjustments the webscraper is able to extract any part of the website.
TarkovScrappy A nifty little bot that lets you know if a queried item might be required for a quest at some point in the land of Tarkov! Hideout items
pastepwn - Paste-Scraping Python Framework Pastebin is a very helpful tool to store or rather share ascii encoded data online. In the world of OSINT,
fetch_comments A simple code to fetch comments below an Instagram post and save them to a csv file usage First you have to enter your username and pas
Photon Incredibly fast crawler designed for OSINT. Photon Wiki • How To Use • Compatibility • Photon Library • Contribution • Roadmap Key Features Dat
WebScrapper Este web scrapper esta desarrollado en python 3.10.0 para buscar en la pagina de cyber puerta articulos dentro del catalogo. El programa t
Parsel Parsel is a BSD-licensed Python library to extract and remove data from HTML and XML using XPath and CSS selectors, optionally combined with re
项目简介 学习强国自动化脚本，解放你的时间！ 使用Selenium、requests、mitmpoxy、百度智能云文字识别开发而成 使用说明 注：Chrome版本 驱动会自动下载 首次使用会生成数据库文件db.db，用于提高文章、视频任务效率。 依赖安装 pip install -r require
Lexile-Atos-Scraper Quick Project made to help scrape Lexile and Atos(AR) levels from ISBN You will need to install the chrome webdriver if you have n
command line document download made easy Highlights list available documents in json format or download them filter documents using string matching re
This is a simple website crawler which asks for a website link from the user to crawl and find specific data from the given website address.
Crawl the information of a given keyword on Google search engine
crawler_for_university 用python爬取江苏几大高校的就业网站，并提供3种方式通知给用户，分别是通过微信发送、命令行直接输出、windows气泡通知。 环境依赖 wxpy,requests,bs4等库 功能描述 该项目基于python，通过爬虫爬各高校的就业信息网，爬取招聘信
A small library for extracting rich content from urls. what does it do? micawber supplies a few methods for retrieving rich metadata about a variety o
WEB-INFO- This tool can be used to extract information from any website Install Termux and run the command --- $ apt-get update $ apt-get upgrade $ pk
HAUT-checkin 河南工业大学自动校外打卡 由于github actions存在明显延迟，建议直接使用腾讯云函数 特点 多人打卡 使用简单，仅需账号密码以及用于微信推送的uid 自动获取上一次打卡信息用于打卡 向所有成员微信单独推送打卡状态 完美校园服务器繁忙时造成打卡失败会自动重新打卡
Arxiv daily 速览 功能：按关键词筛选arxiv每日最新paper，自动获取摘要，自动截取文中表格和图片。 1 测试环境 Ubuntu 16+ Python3.7 torch 1.9 Colab GPU 2 使用演示 首先下载权重baiduyun 提取码:il87，放置于code/Pars
Google Maps Crawler using Selenium Built as part of the Antifragile Dev Project Selenium crawler that browses Google Maps as a regular user and stores
Python3-based Douban Movie/Douban Book Scarpy crawler for cover downloading + data crawling + review entry.
2 Dec 13, 2021
1 Dec 13, 2021
2 Apr 11, 2022
105 Dec 29, 2022
2 Jul 14, 2022
9.3k Jan 02, 2023
1 Oct 27, 2021
859 Dec 29, 2022
359 Dec 30, 2022
1 Feb 11, 2022
16 Dec 26, 2022
1 Jan 10, 2022
4 Nov 09, 2022
8 Aug 16, 2021
588 Dec 27, 2022
1 Oct 24, 2021
36 Oct 27, 2022
20 Jul 30, 2022
46 Dec 16, 2022
1 Dec 03, 2022