AI Dungeon Catalog Archive Toolkit

Related tags

DownloaderAIDCAT
Overview

AI Dungeon Content Archive Toolkit (AID CAT)

AID CAT is a command-line utility that will allow you to download JSON backups of:

  1. Your private and published AI Dungeon scenarios, adventures, posts, bookmarks, worlds, friends, followers, and following.
  2. Any user's published scenarios, adventures, and posts, as well as their friends, followers, and following.
  3. Obfuscate all adventures' titles, descriptions, tags, memory, actions, undone actions, and world info with junk data.
    WARNING! Obfuscating adventures wipes all adventures on an account and CANNOT BE UNDONE!

Things AID CAT will NEVER do:

  1. Download another user's private content.
  2. Share your information with anyone else. All data transferred is strictly between your device and the aidungeon.io GraphQL backend.

Go check out KoboldAI!

Saved AI Dungeon games exported using AID CAT can now be imported and played with KoboldAI! KoboldAI is a browser front-end for playing with multiple local & remote AI models. KoboldAI supports interact with the AI models via Tensorflow and includes a easy-to-use automated install process. Currently supported models are GPT Neo 1.3B/2.7B, GPT-2 Med/Large/XL, Megatron (via InferKit API) as well as the ability to run your fine-tuned custom GPT-Neo (e.g., Neo-horni) and custom GPT-2 (e.g., CloverEdition) models.

Latest Release

2021-06-10: v0.7.0

Added the ability to disable Safe Mode
Hardened the obfuscation script against network instability
Optimized the obfuscation script to run faster
**WARNING! Obfuscating adventures wipes all adventures on an account and CANNOT BE UNDONE!**

Instructions below copied with edits from original script author's site (referenced files uploaded here for archival purposes):

The wAIfupocalypse is upon us

Let's save your stories!

Recently, Latitude has announced changes that will end AI cooms as we know them today. Going beyond just the explore page, employees may now ban you for content in your unpublished stories. For this reason, I have programmed a script that can automatically download all of the stories and scenarios in your account. Here are the instructions:

  1. Before you can use the script, you must download and install the Python 3 runtime. Make sure to select "Add Python 3.X to PATH" on the first page of the installer.
  2. The script needs your access token so it can access your private stories. The token will only be used for that purpose, and it will not be stored. See how to access it in Firefox or Chrome.
  3. Download the aidcat.py script here. Move it to where you want your stories saved and run it. (usually by double-clicking, or typing python aidcat.py or python3 aidcat.py in the command prompt while in the directory containing the script file).
  4. Enter your login token and press enter. While you wait, think back to the good times you had with your fictional friends.
  5. When the download is complete, a file called stories.json will be created containing all your stories.

I have also created a script that will turn that JSON file into HTML files so it's easier to read. Download it here and run it in the same folder as the JSON file. You might also want to get this stylesheet (put in same folder as html files) to make things look a little nicer.

For mobile users

You can run these scripts using Termux (Android) or Pythonista (iOS). Since it's difficult to access your login token on a mobile browser, here is a script to help you with that: login.py (1.1 KiB)

Termux Instructions

  1. pkg update && pkg install python && curl -O https://raw.githubusercontent.com/CuriousNekomimi/AIDCAT/main/aidcat.py && curl -O https://raw.githubusercontent.com/CuriousNekomimi/AIDCAT/main/login.py
  2. python login.py (To get your x-access-token)
  3. python aidcat.py (To run AIDCAT)

Changelog

2021-05-30: v0.6.9
Added the ability to obfuscate all adventures' title, description, tags, memory, actions, undone actions, and world info with junk data.
**WARNING! Obfuscating adventures wipes all adventures on an account and CANNOT BE UNDONE!**
Added an info page to sources documenting Latitude.
 
2021-05-11: v0.6.7
HUGE thanks to Eta for making these improvements and for refactoring the program to be object oriented:
- The phrase "any key" was changed to "Enter." (Actually waiting for any keypress is significantly more complicated).
- Option 1 of the auth menu previously said [1] Change your access access token, which was fixed.
- Added a missing error message on "Our Content" page.
- PEP 8 compliance:
  - The header docstring was changed to use """ instead of '''
  - Fixed whitespace around operators in code and method declarations
  - Fixed indentation in a couple places
  - Added whitespace between the # and the start of each comment
  - Added a newline at the end of the file
  - Added two lines of whitespace before and after function definitions
  - (Non-PEP 8) Changed menu choices to all be double quoted strings, for consistency
- Removed default values from actually-mandatory method parameters like save_json's content_type
- Separated the long as heck query strings from the rest of the code and moved them to the bottom of the file, just above if __name__ == '__main__', to improve readability
- Added a User class
  - This class encompasses the operations make_query (new), save_json, get_scenarios/subscenarios/adventures/posts/worlds/social, and get_saves (new, which calls all of the bookmarked operations together)
  - It holds the fields content_cache, and each of the query_s as instance variables (no longer global)
    - This means these can be maintained independently per user, which will help a lot with the batch download feature later on (it allows that to be multithreaded, for example)
- All of the menu code was changed to use User objects representing different users instead of passing around a username parameter
- The menu code was additionally refactored to cut down on duplicated code by a lot

2021-05-10: v0.5.7
Thanks to Eta for these suggestions:
UUID (access token) validation using python's built-in uuid library.
Cleaned up boolean checks.
Better file management. Saving files no longer risks a memory leak if the programmer forgets to call close().

2021-05-09: v0.5.6
Various improvements to the code in terms of efficiency and syntax, thanks Eta!

2021-05-09: v0.5.4
Changed aidcat.py to UTF-8 encoding. If you were seeing weird issues with screens, like corrupted text, this should fix that issue.
Thanks to Eta for the great suggestions for improving efficiency and helping with refactoring overall, including:
Menu headers are now all block quotes.
Reduced the number of escaped characters ( e.g. \' ).
More conventional print() statements.

Future Plans:
Proper GUID validation
HTML format export/converter. (Next priority)
Possibly add the ability to batch download all your friends' and followings' published content.
Ability to get the user's x-access-token from within the application. (Rather than digging around in the browser.)

2021-05-09: v0.5.2 (Completely new program written from scratch)
Command-line user interface with menus.
Download your scenarios, subscenarios, adventures, posts, and worlds (including official purchased worlds).
Download your saved scenarios, adventures, and posts (bookmarks).
Download a list of your friends, following, and followers.
Download other user's published scenarios, subscenarios, and posts.
Download a list of other user's friends, followers, and following.
Saved files tagged with ISO 8601 format times, so previous files aren't accidentally overwritten.
NOTE: Windows (Command prompt/PowerShell) doesn't display UTF-8 encoded characters. If someone has fancy UTF characters in their username, they will show up as empty boxes when pasted in, but the program will still work.

Future plans:
HTML format export/converter. (Next priority)
Possibly add the ability to batch download all your friends' and followings' published content.
Ability to get the user's x-access-token from within the application. (Rather than digging around in the browser.)

2021-05-02:
Added: Ability to save subscenarios (thanks to the original script author)!
Added: Ability to save bookmarks (thanks to the original script author)!
Added: Choice to save scenario bookmarks.
Added: Choice to save adventure bookmarks.

2021-05-02:
Added: Ability to choose whether or not to download scenarios, adventures, or worlds. Useful for users who have thousands of adventures and don't want to download them every time.
Added: More error catching and a reminder to report errors to https://github.com/CuriousNekomimi/AIDungeonRescue/issues.
Note: The original script author has released an update to download scenario options. I'll work on incorporating that code.

2021-05-02:
Added: Ability to save worlds!
Added: Ability to save x-access token for later use (token.txt).
Added: Scenarios, adventures, and worlds, now save to separate json files.
Added: Improved readability of saved scenario, adventure, and world json files with newlines and indents.
Added: Pause for user interaction after running so the user knows the program executed successfully.
Added: Notification that files saved successfully.
Added: Try/Except to main() for several methods.
Refactored: 'stories' to 'adventures'.
Comments
  • Downloading more than 50 actions

    Downloading more than 50 actions

    I ran this almost two weeks ago to export my stories, including one with over 15,000 actions (yes, I know). Everything was exported including that full story, resulting in a 3.88 MB file.

    Since you now have support for importing into KoboldAI, I figured I'd get the latest version of this tool and re-run it to download my AI Dungeon stories. This results in a much smaller file, only 293 KB. I checked and found that it now seems to cap out at the 50 most recent actions of the story.

    I still have the older version of this tool, so I ran it again to see if the change was on your end or AI Dungeon, and it downloaded the full 3.88 MB file again. So that tells me it's a change made to this tool.

    I'm guessing it's to keep things from getting to bloated before importing into KoboldAI or something, only keeping a certain number of relevant actions which would be potentially referenced by it? Is there an option somewhere now to download the full adventures again, or at least can you provide clarification on the reason for this?

    bug 
    opened by DagothRa 3
  • Tweaked and cleaned

    Tweaked and cleaned

    -Used requests sessions instead of urllib. -Put images in their own file. -Separated the UI from the program logic. -Created a module to transform .json to html (I used my tweaked version from Scriptanon's)

    I'm going to start making the automated tests.

    opened by Moist-Cat 2
  • Infinite Loop on Latitude corrupted backwards compatibility

    Infinite Loop on Latitude corrupted backwards compatibility

    Search the term "Latitude " (with the space), which will lead to an error message. It is inside a true while loop with a continue that triggers no advancement nor does it lead to a break, thus causing infinite looping.

    opened by YuliaS11 1
  • Tools to help migrate from AID to KAI

    Tools to help migrate from AID to KAI

    I created some simple tools to help integrate AIDCAT and KAI so people could migrate from AID faster. The scripts I made interpret AIDCAT's output in a "scenario explorer" that lets the person select their scenario, type the placeholders just like in AID and then save the generated story into a file ready to be used in KoboldAI. Same for my own scenarios (it has its own script) and another tool that converts AIDCAT's extracted stories from AID into something readable by KoboldAI.

    I plan to join these three tools into one, but for now they're separated. This might be useful for people using AIDCAT. Here's the link: https://github.com/thaalesalves/aid-scripts/tree/main/other

    opened by thaalesalves 1
  • Saving bookmarked scenarios fails if any sub-scenario fetch returns None.

    Saving bookmarked scenarios fails if any sub-scenario fetch returns None.

    For reasons unknown to me, certain sub-scenarios in my bookmarks return {Scenario: None}. This satisfies the if 'scenario' in result['data'] check but causes an exception when the script tries to progress and halts.

    I was able to do a quick fix locally by changing if 'data' in result and 'scenario' in result['data']: to if 'data' in result and 'scenario' in result['data'] and result['data']['scenario'] != None:

    opened by throwaway47323674 1
  • genhtml_edit.py throws an error when trying to open a file

    genhtml_edit.py throws an error when trying to open a file

    Welcome to the AIDCAT HTML Generator! First we'll ask for your Adventure file, press Enter to continue... Traceback (most recent call last): File "C:\Users\Me\Documents\AIDungeonRescue-main\aidcat\genhtml_edit.py", line 51, in infile = json.load(file) File "C:\Users\Me\AppData\Local\Programs\Python\Python37\lib\json_init_.py", line 293, in load return loads(fp.read(), UnicodeDecodeError: 'cp932' codec can't decode byte 0x9d in position 119195: illegal multibyte sequence

    opened by cjk137 1
  • Improved Token and Error Handling

    Improved Token and Error Handling

    This pull request improves token handling, adding autocorrection for missing hyphens and server validation to check if the token is associated with an account. It will also now display the account name associated with an entered token. Error handling is also improved, and a keyboard interrupt can now properly exit out of the program (without any error text, either).

    opened by Eta0 0
  • Broken with 'Walls Approach' update

    Broken with 'Walls Approach' update

    https://latitude.io/blog/the-walls-approach

    It seems they're now encrypting user stories. When I try to download them with AIDCAT, I get 400 Bad Request. I might have missed my chance, unless this script can be updated to decrypt the stories.

    opened by ImportTaste 16
Releases(v0.7.0)
  • v.0.6.9(May 30, 2021)

    Added the ability to obfuscate all adventures' title, description, tags, memory, actions, undone actions, and world info with junk data. WARNING: Obfuscating adventures wipes all adventures on an account and CANNOT BE UNDONE! Added an info page to sources documenting Latitude.

    Source code(tar.gz)
    Source code(zip)
  • v0.6.7(May 17, 2021)

  • v0.6.6(May 17, 2021)

    Descriptive exceptions and traceback for HTTP connections errors and file saving. Possible fix for Mac devices not saving files to the current directory.

    Source code(tar.gz)
    Source code(zip)
  • v0.6.5(May 16, 2021)

    Re-refactored actionWindow back to actions . It turns out that actionWindow only returns the last 50 actions. I sincerely apologize for any inconvenience this may have caused!

    Source code(tar.gz)
    Source code(zip)
  • v0.6.2_html_utf8(May 15, 2021)

  • v0.6.2_html(May 14, 2021)

  • v0.6.2(May 13, 2021)

    Eta0: This pull request improves token handling, adding autocorrection for missing hyphens and server validation to check if the token is associated with an account. It will also now display the account name associated with an entered token. Error handling is also improved, and a keyboard interrupt can now properly exit out of the program (without any error text, either).

    Source code(tar.gz)
    Source code(zip)
  • v0.6.0(May 11, 2021)

    HUGE thanks to Eta0 for making these improvements and for refactoring the program to be object oriented:

    • The phrase "any key" was changed to "Enter." (Actually waiting for any keypress is significantly more complicated).
    • Option 1 of the auth menu previously said [1] Change your access access token, which was fixed.
    • Added a missing error message on "Our Content" page.
    • PEP 8 compliance:
      • The header docstring was changed to use """ instead of '''
      • Fixed whitespace around operators in code and method declarations
      • Fixed indentation in a couple places
      • Added whitespace between the # and the start of each comment
      • Added a newline at the end of the file
      • Added two lines of whitespace before and after function definitions
      • (Non-PEP 8) Changed menu choices to all be double quoted strings, for consistency
    • Removed default values from actually-mandatory method parameters like save_json's content_type
    • Separated the long as heck query strings from the rest of the code and moved them to the bottom of the file, just above if name == 'main', to improve readability
    • Added a User class
      • This class encompasses the operations make_query (new), save_json, get_scenarios/subscenarios/adventures/posts/worlds/social, and get_saves (new, which calls all of the bookmarked operations together)
      • It holds the fields content_cache, and each of the query_s as instance variables (no longer global)
        • This means these can be maintained independently per user, which will help a lot with the batch download feature later on (it allows that to be multithreaded, for example)
    • All of the menu code was changed to use User objects representing different users instead of passing around a username parameter
    • The menu code was additionally refactored to cut down on duplicated code by a lot
    Source code(tar.gz)
    Source code(zip)
  • v0.5.7(May 10, 2021)

Owner
Mimi
Mimi
a simple ehentai downloader with jpg 2 pdf

Simple_Ehentai_DownLoader a simple ehentai downloader with jpg 2 pdf 中文介绍 Environment python3.8 How to use before you start,there are some tips. the q

Hibian 6 Dec 11, 2022
Organize your downloads easily with DownloadOrganizer

DownloadOrganizer Organize your downloads organize your downloads easily with DownloadOrganizer Instilation how to install DownloadOrganizer Method 1:

1 Dec 02, 2021
This Program helps you download songs from the Spotify track's link you give in.

Spotify-Downloader-GUI This Program helps you download songs from the Spotify track's link you give in. It uses yt-dlp to download songs from Youtube.

Harish 12 Jun 14, 2022
A growing collection of search plugins for the qBittorrent, an awesome and opensource torrent client

qBittorrent Search Plugins This is a still growing collection of search plugins for qBittorent, an amazing and open source torrent client, maintained

Alessio Tudisco 59 Dec 26, 2022
Automatically download and crop key information from the arxiv daily paper. (cpu version)

Automatically download and crop key information from the arxiv daily paper. (cpu version)

HeoLis 4 Jul 30, 2022
Pantheon - The fastest YouTube downloader.

A Youtube downloader written in Python3, using HTTP requests and an API.

Billy 38 Nov 21, 2022
Implementation of Cross-category Video Highlight Detection via Set-based Learning (ICCV 2021).

Cross-category Video Highlight Detection via Set-based Learning Introduction This project is an implementation of ``Cross-category Video Highlight Det

Minghao (Alan) Xu 49 Dec 17, 2022
YoutubeDownloader - Download any public Playlist from Youtube

YoutubeDownloader Download any public Youtube Channel / Playlist Features Bulk d

17 Nov 12, 2022
Python based Telegram bot. Search and download YouTube video or audio.

Python-Telegram-Youtube-Media-Bot Python based Telegram bot. Search and download YouTube video or audio. Just change settings.py and start TelegramBot

Ahmet Bohur 2 Oct 02, 2022
Download YouTube videos that are available in the given playlist

Youtube-Playlist-Downloader Download YouTube videos that are available in the given playlist Project assets: music downloaded music folder. (will be g

Sultan Aljaberi 1 Dec 22, 2021
Youtube video downloader and info extractor for python.

tube_dl Tube_dl is a Simple Youtube video downloader for Python. A Modular approach to bypass and download Youtube Videos and Playlist from Youtube us

Shekhar Chander 16 Jul 09, 2022
ASF Sentinel-1 Metadata Download tool

ASF Sentinel-1 Metadata Download tool Copyright: 2021-2022 Antonio Valentino Small Python tool (asfsmd) that allows to download XML files containing S

Antonio Valentino 9 Dec 04, 2022
DYA ( Ditch YouTube API ) is a package created to power the user with YouTube Data API functionality without any API Key

Ditch YouTubeAPI (BETA) DYA ( Ditch YouTube API ) is a package created to power the user with YouTube Data API functionality without any API Key Detai

Sougata Jana 23 Dec 22, 2022
Python script to download (TCR) genes from IMGT/GENE-DB

IMGTgeneDL 0.1.0 Jamie Heather | CCR @ MGH | 2021 This script provides an alternative way to access TCR and IG genes stored in IMGT/GENE-DB. It's prim

Jamie Heather 1 Mar 30, 2022
Python software to download videos from Tiktok without rights

download-video-tiktok Python software to download videos from Tiktok without rights to install pip install requests Follow us telegram : https://t.me

muntazir halim 1 Oct 28, 2021
Python script to automate youtube-dl downloads

Automated Download Tool !! Project status I am writing a new version of this program, which will solve several errors. The new version only supports G

Devil64-Dev 21 Sep 22, 2022
Can automatically download mods from a Curseforge modpack

Curseforge-Modpack-Downloader A Python script which automatically downloads mods from a Curseforge modpack. Installing Dependencies ⚠ Make sure you ha

Rayr 1 Sep 20, 2022
Download your Spotify playlists and songs along with album art and metadata

spotDL Download your Spotify playlists and songs along with album art and metadata The fastest, easiest, and most accurate command-line music download

10.6k Jan 03, 2023
GTK4 + Python tutorial with code examples

Taiko's GTK4 Python tutorial Wanna make apps for Linux but not sure how to start with GTK? This guide will hopefully help! The intent is to show you h

190 Jan 08, 2023
This is a python based web scraping bot for windows to download all ACCEPTED submissions of any user on Codeforces

CODEFORCES DOWNLOADER This is a python based web scraping bot for windows to download all ACCEPTED submissions of any user on Codeforces Requirements

Mohak 6 Dec 29, 2022