Utility for downloading fanfiction in bulk from the Archive of Our Own

Overview

What is this?

This is a program intended to help you download fanfiction from the Archive of Our Own in bulk. This program is primarily intended to work with links to the Archive of Our Own itself, but has a secondary function of downloading any Pinboard bookmarks that link to the Archive of Our Own. You can ignore the Pinboard functionality if you don't know what Pinboard is or don't use Pinboard. This program is lightly tested and is currently very likely to have bugs.

Instructions

  • install python
    • make sure to choose the option "add to PATH" when you are installing python. if you do not do this the program is even less likely to work correctly than it already was.
  • clone (or download and unzip) the repository. the "repository" means the folder containing the code. you can download the repository by clicking on the "Code" button in github and selecting "Download ZIP"
  • windows: double-click on "ao3downloader.cmd"
  • other platforms: ao3downloader should work on any platform that supports python, however, you will need to do your own research into how to run python programs on your system.

Menu Options Explanation

  • 'download from ao3 link' - this works for most links to ao3. for example, you can use this to download a single work, a series, or any ao3 page that contains links to works or series (such as your bookmarks or an author's works). the program will download multiple pages automatically without the need to enter the next page link manually.
  • 'download latest version of incomplete fics (ao3 epub files only)' - you can use this to check a folder on your computer (and any subfolders) for epub files downloaded from ao3 that are incomplete works. for each incomplete fic found, the program will check ao3 to see if there are any new chapters, and if so, will download the new version to the downloads folder. apologies but this does not work for filetypes other than epub.
  • 'download pinboard xml document' - this is the first step in downloading your ao3 bookmarks from pinboard. ignore this if you don't use pinboard. to get the api token go to settings -> password on the pinboard website.
  • 'download bookmarks from pinboard xml document' - this is the second step in downloading your ao3 bookmarks from pinboard. ignore this if you don't use pinboard or if you haven't yet downloaded the pinboard xml document.
  • 'convert logfile into interactable html' - all downloads from ao3 (and some other actions) are logged in a file called log.jsonl in the downloads folder, along with information such as whether or not the download was successful, details about errors encountered, and so on. this option converts log.jsonl into a much more human-readable, searchable and sortable html file that can be opened in any browser. the file is saved in the downloads folder and is called 'logvisualization.html'

Notes

  • The purpose of entering your ao3 login information is to download archive-locked works or anything else that is not visible when you are not logged in. If you don't care about that, there is no need to enter your login information.
  • Try to keep your ao3 browsing to a minimum while the script is running. It won't break anything, but it may cause you to hit ao3's limit on how many hits to the site you are allowed within a certain time frame. This limit is per user, or per IP if you are not logged in. If this happens, the script will pause for 5 minutes to let the limit reset, and you may see a "Retry later" message when you try to open an ao3 page during that time. Don't be alarmed by this, just wait it out.
  • You should be able to guess the approximate runtime in seconds by taking the number of works to be downloaded times five. This is a very rough estimate as many factors can affect the total runtime.
  • If the script encounters a work that is part of a series, it will also download the entire series that the work is a part of.
  • For multi-page downloads from ao3, a message will be printed to the console each time a new page starts downloading. If you need to stop the download in the middle, take note of the last page downloaded before you close the window. When you restart, enter the link to that specific page instead of the first page, to avoid repeating downloads as much as possible. Note that pinboard bookmarks are not paginated in the same way, so this will not work if you are downloading bookmarks from pinboard.
  • IMPORTANT: some of your input choices are saved in settings.json. In some cases you will not be able to change these choices unless you clear your settings by deleting settings.json (or editing it, if you are comfortable with json). In addition, please note that saved settings include passwords and keys and are saved in plain text. Use appropriate caution with this file.

Troubleshooting

  • First, if you are able to create logvisualization.html (menu option 'v'), take a look through the logs to see if there are any helpful error messages.
  • If there are no logs or the logs are unhelpful, look for a folder called "venv" inside the repository. Delete "venv" and try re-running the script.
  • If deleting venv doesn't work, try deleting the entire repository and re-downloading from github (but remember to save your existing downloads if you have any!)
  • If re-downloading the repository doesn't work, try reinstalling python. Make sure to choose the option "add to PATH" during the installation.
  • If reinstalling python doesn't work, see this stackoverflow answer.
  • If you have tried all of the above and it still doesn't work, see below for how to send me a bug report.

Questions? Comments? Bug reports?

Feel free to email me at [email protected]. Please include "ao3downloader" in the subject line. If you are reporting a bug, please describe exactly what you did to make the bug happen to the best of your ability. (More is more! Be as detailed as possible.) Optionally when reporting bugs, it is also helpful if you include log.jsonl in the email as an attachment.

(Please note that while I will absolutely do my best to get back to you, I can't make any promises - I have a job, etc.)

Analog clock that shows the weather instead of the actual numerical hour it points to.

Eli's weatherClock An digital analog clock but instead of showing the hours, the clock shows the weather at that hour of the day. So instead of showin

Kovin 154 Dec 01, 2022
Webservice that notifies users on Slack when a change in GitLab concern them.

Gitlab Slack Notifier Webservice that notifies users on Slack when a change in GitLab concern them. Setup Slack Create a Slack app, go to "OAuth & Per

Heuritech 2 Nov 04, 2021
This is a small package to interact with the OpenLigaDB API.

OpenLigaDB This is a small package to interact with the OpenLigaDB API. Installation Run the following to install: pip install openligadb Usage from o

1 Dec 31, 2021
Growtopia server_data.php reader with bypass method, using discord bot

Server_data.php-reader Growtopia server_data.php reader with bypass method, using discord bot How to use 1 install python 2 change your bot token

7 Jul 16, 2022
Telegram Bot for saving and sharing personal and group notes

EZ Notes Bot (ezNotesBot) Telegram Bot for saving and sharing personal and group notes. Usage Personal notes: reply to any message in PM to save it as

Dash Eclipse 8 Nov 07, 2022
OpenSea Bulk Uploader And Trader 100000 NFTs (MAC WINDOWS ANDROID LINUX) Automatically and massively upload and sell your non-fungible tokens on OpenSea using Python Selenium

OpenSea Bulk Uploader And Trader 100000 NFTs (MAC WINDOWS ANDROID LINUX) Automatically and massively upload and sell your non-fungible tokens on OpenS

ERC-7211 3 Mar 24, 2022
toldium is a modular, fast, reliable and customizable multiplatform bot library for your communities

toldium The easy multiplatform bot toldium is a modular, fast, reliable and customizable multiplatform bot library for your communities, from a commun

Stockdroid Fans 5 Nov 03, 2021
Cities bot - A simple example of using aiogram and the wikipedia package

Cities game A simple example of using aiogram and the wikipedia package. The bot

Artem Meller 2 Jan 29, 2022
A file-based quote bot written in Python

Let's Write a Python Quote Bot! This repository will get you started with building a quote bot in Python. It's meant to be used along with the Learnin

A . S . M . RADWAN 2 Apr 03, 2022
SUPPORTS 500 GROUPS NO NEED OF BOT ๐Ÿ˜‰

LOVELY RADIO SUPPORTS 500 GROUPS NO NEED OF BOT ๐Ÿ˜‰ Requirements Telegram API_ID , API_HASH and SESSION_NAME HEROKU Get YouTube live stream link instal

6 Nov 24, 2021
This is a simple code for discord bot !

Discord bot dice roller this is a simple code for discord bot it can roll 1d4, 1d6, 1d8, 1d10, 1d12, 1d20, 1d100 for you in your discord server. Actua

Mostafa Koolabadi 0 Jan 02, 2022
Python API Client for Twitter API v2

๐Ÿ Python Client For Twitter API v2 ๐Ÿš€ Why Twitter Stream ? Twitter-Stream.py a python API client for Twitter API v2 now supports FilteredStream, Samp

Twitivity 31 Nov 19, 2022
A bot written in python that send prefilled Google Forms. It supports multithreading for faster execution time.

GoogleFormsBot https://flassy.xyz https://github.com/Shawey/GoogleFormsBot Requirements: os (Default) ast (Default) threading (Default) configparser (

Shawey 1 Jul 10, 2022
A visualization of people a user follows on Twitter

Twitter-Map This software allows the user to create maps of Twitter accounts. Installation git clone Oliver Greenwood 12 Jul 20, 2022

An Inline Telegram YouTube Downloader bot with custom, permanent thumbnail support and cancel upload facility. Make your fork now.

Inline-Tube-Mate (YouTube Downloader) An Inline Telegram bot that can download YouTube videos with permanent thumbnail support Bot need to be in Inlin

Renjith Mangal 41 Dec 14, 2022
Asynchronous Python Wrapper for the Ufile API

Ufile.io Asynchronous Python Wrapper for the Ufile API (Unofficial).

Gautam Kumar 16 Aug 31, 2022
The Discord bot framework for Python

Pycordia โš ๏ธ Note! As of now, this package is under early development so functionalities are bound to change drastically. We don't recommend you curren

รngel Carias 24 Jan 01, 2023
The best Fortnite all-in-one lobby bot!

Recommended to use on Python v3.8 stable for bot. FLB The best free Fortnite lobby bot experience! Discord server: PDennSploit Softworks LLC Getting S

Payson Holmes 2 May 11, 2022
Bot developed in python, 100% open-source, compatible with Windows and Linux.

Bombcrypto Bot [Family JOW] Bot desenvolvido em python, 100% do cรณdigo รฉ aberto, para aqueles que tenham conhecimento validarem que nรฃo existe nenhum

Renato Maia 71 Dec 20, 2022
PyLyrics Is An [Open-Source] Bot That Can Help You Get Song Lyrics

PyLyrics-Bot Telegram Bot To Search Song Lyrics From Genuis. ๐Ÿค– Demo: ๐Ÿ‘จโ€๐Ÿ’ป Deploy: โค Deploy Your Own Bot : Star ๐ŸŒŸ Fork ๐Ÿด & Deploy -Easy Way -Self-h

DAMIEN 12 Nov 12, 2022