Tool for installing and updating MiSTer cores and other files

Overview

MiSTer Downloader

This tool installs and updates all the cores and other extra files for your MiSTer. It also updates the menu core, the MiSTer firmware and the Linux system. The source for all downloads is the MiSTer Distribution repository.

The MiSTer Downloader is a substitute for the MiSTer Updater, and is meant to offer a more safe and robust experience, while being much faster.

As a drawback, the Downloader is not backwards compatible with the old INI files that were configured for the MiSTer Updater. In fact, as of today, this tool doesn't implement many fine-grained features that allow you to customize the updating process in depth. In case you value these features, consider to keep using the MiSTer Updater as usual. Both tools will coexist in the near future.

Setup and Usage

Download this ZIP file and extract downloader.sh to your /Scripts folder on your primary SD card (create that folder if it doesn't exist). You only need to perform this operation once, since this tool self-updates itself.

To use it, on your MiSTer main menu, go to the Scripts screen, and select downloader.

Options

You may create a /media/fat/downloader.ini file to tweak some parameters.

Here you can see the default parameters and the options that you may change:

[MiSTer]
; base_path is where most files will be installed
;   Useful for setups with USB storage, for example: '/media/usb0/'
base_path = '/media/fat/'

; base_system_path is where system files such as 'MiSTer' and 'menu.rbf' will be installed.
;   Warning: It is recommended to NOT change this setting regardless of your setup.
base_system_path = '/media/fat/'

; allow_delete options:
;   0 -> Don't allow this tool to delete anything at all.
;   1 -> Allow this tool to delete any old file from previous updates.
;   2 -> Allow this tool to delete only old cores that receive a new version.
allow_delete = 1

; allow_reboot options:
;   0 -> Don't allow this tool to ever reboot automatically.
;   1 -> Allow this tool to reboot the system after any system file has been updated.
;   2 -> Allow this tool to reboot the system only after Linux has been updated.
allow_reboot = 1

; update_linux options:
;   true -> Updates Linux when there is a new update (very recommended).
;   false -> Doesn't update Linux.
update_linux = true

; parallel_update options:
;   true -> Tries to more than one file simultaneously.
;   false -> Will only download one file at a time.
parallel_update = true

; downloader_timeout: Can be tweaked to increase the timeout time in seconds
;   It is useful to increase this value for users with slow connections.
downloader_timeout = 300

; downloader_retries: Can be tweaked to increase the retries per failed download
;   It is useful to increase this value for users with very unstable connections.
downloader_retries = 3

Roadmap

  • Initial Release
  • Cheats fetching
  • First-run optimisations
  • Configurable custom download filters
  • Handle duplicated games folders through symlinks (GBA <-> GBA2P, and GAMEBOY <-> GAMEBOY2P)
  • Integration with MiSTer binary
Comments
  • mra_alternatives don't seem to get updated

    mra_alternatives don't seem to get updated

    If the hash for the mra_alternatives summary file doesn't match the newest, mra_alternatives_summary.json.zip gets downloaded and the hashes are updated in the store (summary and files) but the files themselves aren't updated.

    opened by lagomorph 12
  • Unable to update

    Unable to update

    When I try to run aupdate from the MiSTer menu the part where it updates the CA-certs work. But after that it fails.

    The script will run normally now:
    Running MiSTer Downloader
    
    bash: line 122: BASH_SOURCE[0]: unbound variable
    Press any key to continue
    
    opened by Dankoman 6
  • error: invalid zip file with overlapped components (possible zip bomb)

    error: invalid zip file with overlapped components (possible zip bomb)

    I started receiving errors when trying to update today. Unzip is throwing an exception because it is recognizing the downloaded file as malicious. I am running the latest version of download.sh. The same issue occurs with any update script which downloads zipped files. Previously had no issues updating.

    /root# cat /media/fat/Scripts/.config/downloader/downloader.log START!

    Reading file: /media/fat/downloader.ini Reading 'distribution_mister' db section Reading 'jtcores' db section env: { "DOWNLOADER_LAUNCHER_PATH": "/media/fat/Scripts/downloader.sh", "DOWNLOADER_INI_PATH": null, "CURL_SSL": "--cacert /etc/ssl/certs/cacert.pem", "COMMIT": "856b6d5", "ALLOW_REBOOT": null, "UPDATE_LINUX": "true", "DEFAULT_DB_URL": "https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip", "DEFAULT_DB_ID": "distribution_mister", "DEFAULT_BASE_PATH": null, "DEBUG": "false", "FAIL_ON_FILE_ERROR": "false" } config: { "databases": { "distribution_mister": { "db_url": "https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip", "section": "distribution_mister" }, "jtcores": { "db_url": "https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip", "section": "jtcores" } }, "config_path": "/media/fat/downloader.ini", "base_path": "/media/fat", "base_system_path": "/media/fat", "storage_priority": "prefer_sd", "allow_delete": 1, "allow_reboot": 1, "update_linux": true, "parallel_update": true, "downloader_size_mb_limit": 100, "downloader_process_limit": 300, "downloader_timeout": 300, "downloader_retries": 3, "zip_file_count_threshold": 60, "zip_accumulated_mb_threshold": 100, "filter": null, "verbose": false, "debug": false, "default_db_id": "distribution_mister", "start_time": 1654702379.9457989, "user_defined_options": [], "curl_ssl": "--cacert /etc/ssl/certs/cacert.pem", "update_linux_environment": 0, "fail_on_file_error": false, "commit": "856b6d5" } cacert file at "/etc/ssl/certs/cacert.pem" seems to be fine. Loading db from url: https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip Loading db from url: https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip Downloading 2 files: /tmp/temp_filet7etzsji /tmp/temp_filex45tz6id *..

    Checking hashes... ++ subprocess.run unzip -p /tmp/temp_filet7etzsji Return Code was '12'

    error: invalid zip file with overlapped components (possible zip bomb)

    Could not load json from "https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip" subprocess.run unzip -p /tmp/temp_filex45tz6id Return Code was '12'

    error: invalid zip file with overlapped components (possible zip bomb)

    Could not load json from "https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip" Skipping local_store saving...

    =========================== Downloader 1.5 (856b6d5) by theypsilon. Run time: 0:00:02.29s Log: /media/fat/Scripts/.config/downloader/downloader.log

    Installed: none.

    Errors: https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip, https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip

    linux_descriptions_count: 0 Length of failed_dbs: 2

    opened by LuoJohnT 5
  • Question: Save updates to /media folder without appending _Consoles/_Arcade/etc folders

    Question: Save updates to /media folder without appending _Consoles/_Arcade/etc folders

    Hi there,

    Is there any way to set the downloader to update and install the files I want straight into the /media folder instead of adding the associated _Console folder or any other similar folder? I don't use a ton of cores so I'd love to just run the script and keep everything clean. This way, I can just boot the MiSTer, and immediately see the cores in the list without entering additional directories.

    (Unless this is already possible)

    opened by Chromagram 3
  • downloader failed:

    downloader failed: "Could not load storage"

    Hello,

    I am attempting to run update_all.sh script with the downloader enabled. The latest jtbeta.zip file is in my 'mame' folder and all of the necessary options are enabled, as far as I can tell, in the update_all options menu (i.e. Install Premium Cores, Install Unofficial Cores etc.) I am recieving the following error:


    Reading file: /media/fat/download.ini Reading 'distribution_mister' db section Reading 'jtcores' db section Reading 'theypsilon_unofficial_distribution" db section Could not load storage Could not load json from "https://raw.githubusercontent.com/MiSTer--devel/Distribution_MiSTer/main/db.json.zip" Could not load json from "https://raw.githubusercontent.com/jotego/jtpremium/main/jtbindb.json.zip" Could not load json from "https://raw.githubusercontent.com/theypsilon/Distribution_Unofficial_MiSTer/unofficialdb.json.zip"

    There were some errors in the Updaters. Therefore, MiSTer hasn't been fully updated.

    Check these logs from the Updaters that failed:

    • /media/fat/Scripts/ .config/downloader/downloader1.log
    • BIOS-GETTER
    • /media/fat/Scripts/ .config/downloader/downloader2.log

    I receive the same error when running just the downloader.sh script. All official cores were downloaded and installed correctly using update_all.

    Thank you!

    opened by eebaser 3
  • Issue wit pc downloader

    Issue wit pc downloader

    Hi,

    I get the following error from https://github.com/MiSTer-devel/Downloader_MiSTer/releases/download/latest/downloader.zip

    {code} Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in run_code File "c:\TEMP\roka\2\tmp2bdrzqig.zip_main.py", line 49, in NameError: name 'exit' is not defined {code}

    sys.exit instead of exit is needed

    opened by RolandKaechele 2
  • Refactor downloader scripts

    Refactor downloader scripts

    While I was reading the scripts (to understand how they work) I've started putting some comments at some places. Later I've started introducing some "echo" statement as well. Also, I've noticed /etc/ssl/certs/cacert.pem is used in many places as a string literal, so I've created a constant to pass this around. In the end, I've tried to run this outside of MiSTer and I got hit by some missing binaries, so I added a function to check for those.

    Nothing significant here, mostly cosmetic changes I would say. Feel free to approve or reject this.

    opened by drpaneas 2
  • Error unpacking BIOS zips

    Error unpacking BIOS zips

    I'm on a fresh install and I am getting this error during the BiosDB step (in update_all):

    Extracting NeoGeo UniBios from http://unibios.free.fr
    checkdir:  cannot create extraction directory: /tmp/q48as6j6/neogeo_unibios
               No such file or directory
    Traceback (most recent call last):
      File "/tmp/dont_download.zip/downloader/main.py", line 37, in main
        exit_code = execute_full_run(
      File "/tmp/dont_download.zip/downloader/main.py", line 57, in execute_full_run
        exit_code = runner.full_run()
      File "/tmp/dont_download.zip/downloader/full_run_service.py", line 59, in full_run
        result = self._full_run_impl()
      File "/tmp/dont_download.zip/downloader/full_run_service.py", line 121, in _full_run_impl
        self._online_importer.download_dbs_contents(importer_command, full_resync)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 95, in download_dbs_contents
        db_importer.process_changed_files(changed_files, needed_zips, filtered_zip_data)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 586, in process_changed_files
        self._import_zip_contents(needed_zips, filtered_zip_data, file_downloader)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 672, in _import_zip_contents
        self._file_system.unzip_contents(temp_zip, tmp_path, list(zipped_files['files']))
      File "/tmp/dont_download.zip/downloader/file_system.py", line 335, in unzip_contents
        raise Exception("Could not unzip %s: %s" % (file, result.returncode))
    Exception: Could not unzip /tmp/hir0qr7h_neogeo_unibios_contents.zip: 2
    

    I think the parent tmp directory isn't being created after the name is generated?

    https://github.com/MiSTer-devel/Downloader_MiSTer/blob/0060984366a335546a7152ff83065c729d962d83/src/downloader/online_importer.py#L671

    opened by wizzomafizzo 2
  • Failure to remove directory when using retronas and MiSTer

    Failure to remove directory when using retronas and MiSTer

    I opened an issue for this over in the retronas repo, but it might be better suited here.

    Copy/paste from the original issue:


    Hi!

    Just getting started with retronas, and connected my MiSTer setup to it over CIFS. When I run downloader.sh, it tries to remove the empty folder for ATARI2600, but ends up failing:

    Deleting empty folder /media/fat/games/ATARI2600
    Traceback (most recent call last):
      File "/tmp/dont_download.zip/downloader/main.py", line 37, in main
        exit_code = execute_full_run(
      File "/tmp/dont_download.zip/downloader/main.py", line 57, in execute_full_run
        exit_code = runner.full_run()
      File "/tmp/dont_download.zip/downloader/full_run_service.py", line 59, in full_run
        result = self._full_run_impl()
      File "/tmp/dont_download.zip/downloader/full_run_service.py", line 121, in _full_run_impl
        self._online_importer.download_dbs_contents(importer_command, full_resync)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 108, in download_dbs_contents
        self._remove_folders(importer_command)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 210, in _remove_folders
        system_file_system.remove_folder(full_folder_path)
      File "/tmp/dont_download.zip/downloader/file_system.py", line 239, in remove_folder
        os.rmdir(self._path(path))
    NotADirectoryError: [Errno 20] Not a directory: '/media/fat/games/ATARI2600'
    

    This makes the rest of the update/download fail as well. Creating an empty file in the ATARI2600 folder removes this failure point.

    opened by jonasrosland 2
  • Won't read from /media/fat/downloader.ini

    Won't read from /media/fat/downloader.ini

    Readme says to use this ini to make changes, but it never reads from it. It's only looking at this one according to my log: Reading file: /media/fat/Update/MiSTer_Downloader/downloader.ini

    My ini: https://pastebin.com/DNPrxXEJ

    Log: https://pastebin.com/1HwS6gh8

    I set verbose to true and log says it's false Also tried moving the ini to "/media/fat/Update/MiSTer_Downloader/downloader.ini" and it still doesn't read it.

    opened by djvj1 2
  • Updates Not Downloading After Adding downloader.ini

    Updates Not Downloading After Adding downloader.ini

    So I created downloader.ini, copy/pasted the defaults and saved it into my root directory (where MiSTer.ini) lives. I noticed that when I run the downloader script now, nothing updates. After doing some research, I found that the downloader.ini needs to point to the default MiSTer [distribution_mister] database.

    If this is the case, then shouldn't this be documented and added to the list of default ini settings?

    opened by mjj03301977 2
  •  Invalid cross-device link on update

    Invalid cross-device link on update

    Copying new MiSTer binary:
    Traceback (most recent call last):
      File "/tmp/dont_download.zip/downloader/main.py", line 37, in main
        exit_code = execute_full_run(
      File "/tmp/dont_download.zip/downloader/main.py", line 57, in execute_full_run
        exit_code = runner.full_run()
      File "/tmp/dont_download.zip/downloader/full_run_service.py", line 59, in full_run
        result = self._full_run_impl()
      File "/tmp/dont_download.zip/downloader/full_run_service.py", line 121, in _full_run_impl
        self._online_importer.download_dbs_contents(importer_command, full_resync)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 95, in download_dbs_contents
        db_importer.process_changed_files(changed_files, needed_zips, filtered_zip_data)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 588, in process_changed_files
        file_downloader.download_files(self._is_first_run())
      File "/tmp/dont_download.zip/downloader/file_downloader.py", line 121, in download_files
        self._file_system.move(FILE_MiSTer_new, FILE_MiSTer)
      File "/tmp/dont_download.zip/downloader/file_system.py", line 189, in move
        os.replace(self._path(source), self._path(target))
    OSError: [Errno 18] Invalid cross-device link: '/media/usb0/MiSTer.new' -> '/media/fat/MiSTer'
    
    Downloader failed!
    

    Here is my download.ini

    [MiSTer]
    base_path = '/media/usb0'
    
    [distribution_mister]
    db_url = https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip
    
    [jtcores]
    db_url = https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip
    
    [bios_db]
    db_url = https://raw.githubusercontent.com/theypsilon/BiosDB_MiSTer/db/bios_db.json
    
    opened by KClough 2
  • Delete old config/DIP switch files when needed

    Delete old config/DIP switch files when needed

    A common source of problems is when the core logic changes leaving the saved config/DIP files unusable and possibly breaking some functionality if used with old files.

    For DIP switches, it is easy to detect a change by comparing the dip switch section inside the MRA with the previous MRA file. If a change in the DIP sw. section is detected, the downloader should delete the old file in the config/dips folder. This would apply to any MRA of any arcade core.

    For config file changes, I don't think it's easy to detect automatically because the configuration information shown in the OSD is embedded in the RBF file and has gone through some transformations. So it isn't plain ASCII. Maybe the downloader could read some metadata in the repositories. I could, for instance, make the config string available as plain text in the version.log file of each core. Currently, this file points to the source code git commit used to produce a specific RBF in JTBIN.

    Please consider implementing these improvements.

    opened by jotego 5
VITS: Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech

VITS: Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech Jaehyeon Kim, Jungil Kong, and Juhee Son In our rece

Jaehyeon Kim 1.7k Jan 08, 2023
a delightful machine learning tool that allows you to train, test and use models without writing code

igel A delightful machine learning tool that allows you to train/fit, test and use models without writing code Note I'm also working on a GUI desktop

Nidhal Baccouri 3k Jan 05, 2023
code for Fast Point Cloud Registration with Optimal Transport

robot This is the repository for the paper "Accurate Point Cloud Registration with Robust Optimal Transport". We are in the process of refactoring the

28 Jan 04, 2023
Image Fusion Transformer

Image-Fusion-Transformer Platform Python 3.7 Pytorch =1.0 Training Dataset MS-COCO 2014 (T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ram

Vibashan VS 68 Dec 23, 2022
The official implementation for ACL 2021 "Challenges in Information Seeking QA: Unanswerable Questions and Paragraph Retrieval".

Code for "Challenges in Information Seeking QA: Unanswerable Questions and Paragraph Retrieval" (ACL 2021, Long) This is the repository for baseline m

Akari Asai 25 Oct 30, 2022
Implement the Pareto Optimizer and pcgrad to make a self-adaptive loss for multi-task

multi-task_losses_optimizer Implement the Pareto Optimizer and pcgrad to make a self-adaptive loss for multi-task 已经实验过了,不会有cuda out of memory情况 ##Par

14 Dec 25, 2022
Unofficial PyTorch implementation of Masked Autoencoders Are Scalable Vision Learners

Unofficial PyTorch implementation of Masked Autoencoders Are Scalable Vision Learners This repository is built upon BEiT, thanks very much! Now, we on

Zhiliang Peng 2.3k Jan 04, 2023
Code for ICDM2020 full paper: "Sub-graph Contrast for Scalable Self-Supervised Graph Representation Learning"

Subg-Con Sub-graph Contrast for Scalable Self-Supervised Graph Representation Learning (Jiao et al., ICDM 2020): https://arxiv.org/abs/2009.10273 Over

34 Jul 06, 2022
CAST: Character labeling in Animation using Self-supervision by Tracking

CAST: Character labeling in Animation using Self-supervision by Tracking (Published as a conference paper at EuroGraphics 2022) Note: The CAST paper c

15 Nov 18, 2022
DeepSTD: Mining Spatio-temporal Disturbances of Multiple Context Factors for Citywide Traffic Flow Prediction

DeepSTD: Mining Spatio-temporal Disturbances of Multiple Context Factors for Citywide Traffic Flow Prediction This is the implementation of DeepSTD in

5 Sep 26, 2022
Cockpit is a visual and statistical debugger specifically designed for deep learning.

Cockpit: A Practical Debugging Tool for Training Deep Neural Networks

Felix Dangel 421 Dec 29, 2022
Codebase for Attentive Neural Hawkes Process (A-NHP) and Attentive Neural Datalog Through Time (A-NDTT)

Introduction Codebase for the paper Transformer Embeddings of Irregularly Spaced Events and Their Participants. This codebase contains two packages: a

Alan Yang 28 Dec 12, 2022
STMTrack: Template-free Visual Tracking with Space-time Memory Networks

STMTrack This is the official implementation of the paper: STMTrack: Template-free Visual Tracking with Space-time Memory Networks. Setup Prepare Anac

Zhihong Fu 62 Dec 21, 2022
A Simulation Environment to train Robots in Large Realistic Interactive Scenes

iGibson: A Simulation Environment to train Robots in Large Realistic Interactive Scenes iGibson is a simulation environment providing fast visual rend

Stanford Vision and Learning Lab 493 Jan 04, 2023
DVG-Face: Dual Variational Generation for Heterogeneous Face Recognition, TPAMI 2021

DVG-Face: Dual Variational Generation for HFR This repo is a PyTorch implementation of DVG-Face: Dual Variational Generation for Heterogeneous Face Re

52 Dec 30, 2022
This repository contains PyTorch code for Robust Vision Transformers.

This repository contains PyTorch code for Robust Vision Transformers.

117 Dec 07, 2022
PyTorch implementation of paper "StarEnhancer: Learning Real-Time and Style-Aware Image Enhancement" (ICCV 2021 Oral)

StarEnhancer StarEnhancer: Learning Real-Time and Style-Aware Image Enhancement (ICCV 2021 Oral) Abstract: Image enhancement is a subjective process w

IDKiro 133 Dec 28, 2022
Code for "Unsupervised Layered Image Decomposition into Object Prototypes" paper

DTI-Sprites Pytorch implementation of "Unsupervised Layered Image Decomposition into Object Prototypes" paper Check out our paper and webpage for deta

40 Dec 22, 2022
EdiBERT is a generative model based on a bi-directional transformer, suited for image manipulation

EdiBERT, a generative model for image editing EdiBERT is a generative model based on a bi-directional transformer, suited for image manipulation. The

16 Dec 07, 2022
Alpha-IoU: A Family of Power Intersection over Union Losses for Bounding Box Regression

Alpha-IoU: A Family of Power Intersection over Union Losses for Bounding Box Regression YOLOv5 with alpha-IoU losses implemented in PyTorch. Example r

Jacobi(Jiabo He) 147 Dec 05, 2022