Utility for downloading fanfiction in bulk from the Archive of Our Own

Overview

What is this?

This is a program intended to help you download fanfiction from the Archive of Our Own in bulk. This program is primarily intended to work with links to the Archive of Our Own itself, but has a secondary function of downloading any Pinboard bookmarks that link to the Archive of Our Own. You can ignore the Pinboard functionality if you don't know what Pinboard is or don't use Pinboard. This program is lightly tested and is currently very likely to have bugs.

Instructions

  • install python
    • make sure to choose the option "add to PATH" when you are installing python. if you do not do this the program is even less likely to work correctly than it already was.
  • clone (or download and unzip) the repository. the "repository" means the folder containing the code. you can download the repository by clicking on the "Code" button in github and selecting "Download ZIP"
  • windows: double-click on "ao3downloader.cmd"
  • other platforms: ao3downloader should work on any platform that supports python, however, you will need to do your own research into how to run python programs on your system.

Menu Options Explanation

  • 'download from ao3 link' - this works for most links to ao3. for example, you can use this to download a single work, a series, or any ao3 page that contains links to works or series (such as your bookmarks or an author's works). the program will download multiple pages automatically without the need to enter the next page link manually.
  • 'download latest version of incomplete fics (ao3 epub files only)' - you can use this to check a folder on your computer (and any subfolders) for epub files downloaded from ao3 that are incomplete works. for each incomplete fic found, the program will check ao3 to see if there are any new chapters, and if so, will download the new version to the downloads folder. apologies but this does not work for filetypes other than epub.
  • 'download pinboard xml document' - this is the first step in downloading your ao3 bookmarks from pinboard. ignore this if you don't use pinboard. to get the api token go to settings -> password on the pinboard website.
  • 'download bookmarks from pinboard xml document' - this is the second step in downloading your ao3 bookmarks from pinboard. ignore this if you don't use pinboard or if you haven't yet downloaded the pinboard xml document.
  • 'convert logfile into interactable html' - all downloads from ao3 (and some other actions) are logged in a file called log.jsonl in the downloads folder, along with information such as whether or not the download was successful, details about errors encountered, and so on. this option converts log.jsonl into a much more human-readable, searchable and sortable html file that can be opened in any browser. the file is saved in the downloads folder and is called 'logvisualization.html'

Notes

  • The purpose of entering your ao3 login information is to download archive-locked works or anything else that is not visible when you are not logged in. If you don't care about that, there is no need to enter your login information.
  • Try to keep your ao3 browsing to a minimum while the script is running. It won't break anything, but it may cause you to hit ao3's limit on how many hits to the site you are allowed within a certain time frame. This limit is per user, or per IP if you are not logged in. If this happens, the script will pause for 5 minutes to let the limit reset, and you may see a "Retry later" message when you try to open an ao3 page during that time. Don't be alarmed by this, just wait it out.
  • You should be able to guess the approximate runtime in seconds by taking the number of works to be downloaded times five. This is a very rough estimate as many factors can affect the total runtime.
  • If the script encounters a work that is part of a series, it will also download the entire series that the work is a part of.
  • For multi-page downloads from ao3, a message will be printed to the console each time a new page starts downloading. If you need to stop the download in the middle, take note of the last page downloaded before you close the window. When you restart, enter the link to that specific page instead of the first page, to avoid repeating downloads as much as possible. Note that pinboard bookmarks are not paginated in the same way, so this will not work if you are downloading bookmarks from pinboard.
  • IMPORTANT: some of your input choices are saved in settings.json. In some cases you will not be able to change these choices unless you clear your settings by deleting settings.json (or editing it, if you are comfortable with json). In addition, please note that saved settings include passwords and keys and are saved in plain text. Use appropriate caution with this file.

Troubleshooting

  • First, if you are able to create logvisualization.html (menu option 'v'), take a look through the logs to see if there are any helpful error messages.
  • If there are no logs or the logs are unhelpful, look for a folder called "venv" inside the repository. Delete "venv" and try re-running the script.
  • If deleting venv doesn't work, try deleting the entire repository and re-downloading from github (but remember to save your existing downloads if you have any!)
  • If re-downloading the repository doesn't work, try reinstalling python. Make sure to choose the option "add to PATH" during the installation.
  • If reinstalling python doesn't work, see this stackoverflow answer.
  • If you have tried all of the above and it still doesn't work, see below for how to send me a bug report.

Questions? Comments? Bug reports?

Feel free to email me at [email protected]. Please include "ao3downloader" in the subject line. If you are reporting a bug, please describe exactly what you did to make the bug happen to the best of your ability. (More is more! Be as detailed as possible.) Optionally when reporting bugs, it is also helpful if you include log.jsonl in the email as an attachment.

(Please note that while I will absolutely do my best to get back to you, I can't make any promises - I have a job, etc.)

A modern, easy to use, feature-rich, and async ready API wrapper for Discord written in Python.

disfork A modern, easy to use, feature-rich, and async ready API wrapper for Discord written in Python. Key Features Modern Pythonic API using async a

2 Feb 09, 2022
Dribble sign up screen built in python and kivy

Dribble sign up screen built in python and kivy contains Dribble icon with icon position and shadow animation.

1 Dec 06, 2021
:lock: Python 2.7/3.X client for HashiCorp Vault

hvac HashiCorp Vault API client for Python 3.x Tested against the latest release, HEAD ref, and 3 previous minor versions (counting back from the late

hvac 1k Dec 29, 2022
A Discord bot that enables using breakout rooms on a server

Discord Breakout Room Bot This bot enables you to use breakout rooms on your Discord server! Note This bot was thrown together within a few hours, so

Till Müller 2 Nov 23, 2021
A python script that changes our background based on current weather and time of the day.

Desktop background on Windows 10, based on current weather and time A python script that changes our background based on current weather and time of t

Maj Gaberšček 1 Nov 16, 2021
Auxiliator is telegram bot for basic web-application analysis

Auxiliator Auxiliator is telegram bot for basic web-application analysis What for? Sometimes there is no access to your main PC, where you can scan we

Revoltage 13 Dec 26, 2021
Some random bot for Discord which was created just for fun (Made with Discord.py library)

Ghosty Previously known as 'secondthunder-py-bot' This is repository of some random bot for Discord which was created just for fun and for some educat

Владислав 8 Oct 02, 2022
Brute Force Attack On Facebook Accounts

Brute Force Attack On Facebook Accounts For Install: pkg install update && pkg upgrade -y pkg install python pip install requests pip install mechani

MK X Shaon 1 Oct 30, 2021
Easily report Instagram pages and close the page

Program Features - 📌 Delete target post on Instagram. - 📌 Delete Media Target post on Instagram - 📌 Complete deletion of the target account on Inst

hack4lx 11 Nov 25, 2022
This asynchronous telegram bot sells books.

Selling_Books_Bot Description Say, you have a bunch of items you need no more and you want to sell it all out. That's where you're going to have to us

Roman 1 Oct 24, 2021
Der Dischkort Bot für Andiismus

AndreOS Der Dischkort Bot für Andiismus Wichtigger Bot für den hauseigenen Discord-Server Indoktrinationsmechanismusleitungsprogramm der andiistischen

Leon Bartle 3 Jan 13, 2022
This is to notify you via Discord whenever there is a new beacon.

BeaconNotifier-Discord This is to notify you via Discord whenever there is a new beacon. Make sure you have python3 installed Steps: Create a Discord

26 Dec 28, 2022
Asynchronous Guilded API wrapper for Python

Welcome to guilded.py, a discord.py-esque asynchronous Python wrapper for the Guilded API. If you know discord.py, you know guilded.py. Documentation

shay 115 Dec 30, 2022
A fun hangman style game to guess random movie names with a short summary about the movie.

hang-movie-man Hangman but for movies 😉 This is a fun hangman style game to guess random movie names from the local database and show some summary ab

Ankit Josh 10 Sep 07, 2022
Python client for Vektonn

Python client for Vektonn Installation Install the latest version: $ pip install vektonn Install specific version: $ pip install vektonn==1.2.3 Upgrad

Vektonn 16 Dec 09, 2022
GitHub Activity Generator - A script that helps you instantly generate a beautiful GitHub Contributions Graph for the last year.

GitHub Activity Generator A script that helps you instantly generate a beautiful GitHub Contributions Graph for the last year. Before 😐 😶 😒 After ?

1 Dec 30, 2021
Protect Discord server invite link

DiscordOauth2Join Protect discord server invite links! Setup I will not help setting up the discord application, but just python. First, install the r

ZEEE 4 Aug 12, 2021
You have 3 files: create mass groups, add mass members, rename all groups (only educational use!)

EDUCATIONAL ONLY! HOW TO INSTALL Edit config.json with your discord account token and the imagepath (if its in the same location as the all_together.p

46 Dec 27, 2022
SpotPlay2YouPlay - Converts new additions to a Spotify playlist to a matching Youtube playlist

SpotPlay2YouPlay - Converts new additions to a Spotify playlist to a matching Youtube playlist, can also be configured to converting whole playlists with the refresh fun

9 Mar 06, 2022
Wrapper for Gismeteo.ru.

pygismeteo Обёртка для Gismeteo.ru. Асинхронная версия здесь. Установка python -m pip install -U pygismeteo Документация https://pygismeteo.readthedoc

Almaz 7 Dec 26, 2022