Automatically download and crop key information from the arxiv daily paper. (cpu version)

Related tags

DownloaderFocusAX
Overview

FocusAX

按关键词筛选arxiv每日最新paper或从arxiv搜索。

  • 自动下载、获取摘要、自动截取文中表格和图片。

安装必要的环境

  • 安装 paddle
# GPU安装
python3 -m pip install paddlepaddle-gpu==2.1.1 -i https://mirror.baidu.com/pypi/simple

# CPU安装
 python3 -m pip install paddlepaddle==2.1.1 -i https://mirror.baidu.com/pypi/simple
  • 安装 Layout-Parser
=2.2"">
pip3 install -U https://paddleocr.bj.bcebos.com/whl/layoutparser-0.0.0-py3-none-any.whl
pip install "paddleocr>=2.2"
  • 按照其他必要的包
pip3 install -r requirements.txt
  • 下载模型权重
  • PubLayNet 下载解压后放置在paperparse目录下。目录结构如下
FocusAX
    - paperparse
        - ppyolov2_r50vd_dcn_365e_publaynet
            - inference.pdiparams
            - inference.pdiparams.info
            - inference.pdmodel
        - ...
    - downloader
        - ...
    - utils
        - ...
    - configs.py
    - focus_daily.py
    - focus_search.py
    - README.py
    - ...

使用教程

  • configs.py :程序参数配置文件
# =============== 网络代理 ================
# proxy = None # 不使用代理
proxy = {"http": "socks5://127.0.0.1:8080", "https": "socks5://127.0.0.1:8080"}
# =============== 保存文件根目录 ================
root_path = "./arxiv"
# =============== DNN模型推理配置信息 ================
threshold = 0.5
enable_mkldnn = True
enforce_cpu = True
thread_num = 4
  • focus_daily.py :按关键字过滤arxiv daily上的文章(仅当日)
if __name__ == '__main__':
    key_words = ['GAN'] # 要包含的关键词
    subject_words = ['ML', 'CV', 'AI']  # 要包含的类别
    start_parse(key_words, subject_words, needPDF=True, needZip=False)
  • focus_search.py :按关键字在arxiv检索
start_parse('Keyword')
  • root_path 目录中将创建新的文件夹保存结果

效果图

每个文件夹中的abs.md文件保留的是当前pdf的介绍,使用Typora等markdown编辑器打开。 image

image

ps:论文排版不规范会导致截图混乱。

其他

Owner
HeoLis
Interesting in generate methods.
HeoLis
A toolkit to automatically crawl the paper list and download paper pdfs of ACL Ahthology.

ACL-Anthology-Crawler A toolkit to automatically crawl the paper list and download paper pdfs of ACL Anthology

Ray GG 9 Oct 09, 2022
Youtube-music - Youtube music with python

youtube-music fzf on https://github.com/junegunn/fzf python3 ytb.py [no/yes] yes

direskyfer 0 Feb 03, 2022
Download candlestick data fast & easy for analysis

crypto-candlesticks 📈 The goal behind this project is to facilitate downloading cryptocurrency candlestick data fast & simple. Currently only the Bit

Pedro Torres 31 Dec 11, 2022
ComicDownloader - Downloads Comics from readcomiconline.li

ComicDownloader Downloads Comics from readcomiconline.li To use this script from

2 Nov 08, 2022
A prometheus exporter for torrent downloader like qbittorrent/transmission/deluge

downloader-exporter A prometheus exporter for qBitorrent/Transmission/Deluge. Get metrics from multiple servers and offers them in a prometheus format

Lei Shi 41 Nov 18, 2022
Tool To Get Downloads up to 4k from Paramount+

Paramount 4K Downloader Tool To Get Downloads up to 4k from Paramount+ 😄 Hello Fellow Developers/ ! Hi! My name is WVDUMP. I am Leaking the script

2 Dec 25, 2021
Download minecraft head or skin, allows TLauncher accounts

Download minecraft head or skin, allows TLauncher accounts

1 Dec 30, 2021
Jocomol 16 Dec 12, 2022
Yahoo! Finance next gen python 3 / pandas market data downloader

Yahoo! Finance-ng python3 / pandas market data downloader Ever since Yahoo! finance decommissioned their historical data API, many programs that relie

Pedro Larroy 7 Dec 09, 2022
Python library to download bulk of images from Bing.com

Python library to download bulk of images form Bing.com. This package uses async url, which makes it very fast while downloading.

Guru Prasad Singh 105 Dec 14, 2022
pubmex.py - a script to get a fancy paper title based on given DOI or PMID

pubmex.py is a script to get a fancy paper title based on given DOI or PMID (can be also combined with macOS Finder)

Marcin Magnus 13 Nov 20, 2022
Python utility to download jobs at seek.com.au

Job Seeker job_seeker is an utility to download data of a job search from seek.com.au into a csv file for data analysis and exploration Install using

PyBites 3 May 14, 2022
Ebook downloader built using python

ebook-downloader Getting Started Open a terminal and run the following commands. git clone github.com/georgemunyoro/ebook-downloader cd ./ebook-downlo

George Munyoro 1 Oct 19, 2021
Pypixiv - A fully-typed, asynchronous api wrapper for pixiv

pypixiv this library is a fully-typed, asynchronous api wrapper for pixiv. featu

DeltaLaboratory 2 Nov 16, 2022
New York Times Front Page Downloader.

TIMETRAVELER New York Times Front Page Downloader. Usage python3 timetraveler.py All data will be saved at ~/timetraveler/ Goals To keep a historica

Daeshon Jones 0 Oct 31, 2021
Youtube Downloader is a Graphic User Interface(GUI) that lets users download a Youtube Video or Audio through a URL

Youtube Downloader This Python and Tkinter based GUI allows users to directly download the Best Resolution Videos and Audios from Youtube. Pa-fy Insta

Samarth Kumar 2 Jun 25, 2022
Organize your downloads easily with DownloadOrganizer

DownloadOrganizer Organize your downloads organize your downloads easily with DownloadOrganizer Instilation how to install DownloadOrganizer Method 1:

1 Dec 02, 2021
Using Youtube downloader is the fast and easy way to download and save any YouTube video.

Youtube video downloader using Django Using Django as a backend along with pytube module to create Youtbue Video Downloader. https://yt-videos-downloa

Suman Raj Khanal 10 Jun 18, 2022
This is a python based web scraping bot for windows to download all ACCEPTED submissions of any user on Codeforces

CODEFORCES DOWNLOADER This is a python based web scraping bot for windows to download all ACCEPTED submissions of any user on Codeforces Requirements

Mohak 6 Dec 29, 2022
命令行版本的HLS/DASH流下载器,支持标准AES-128-CBC解密

XstreamDL-CLI 基于python 3.7.4+的,命令行版本的,HLS/DASH流下载器,支持标准AES-128-CBC解密 使用 首先安装必要的库

xhlove 239 Dec 31, 2022