Bulk download tool for the MyMedia platform

Overview

MyMedia Bulk Content Downloader

This is a bulk download tool for the MyMedia platform.

USE ONLY WHERE ALLOWED BY THE COPYRIGHT OWNER. NOT AFFILIATED WITH ANYTHING/ANYONE/ANY ORGANISATION.

Usage

0. Prerequisites

To use this tool, you will have to have all of the following installed.

As long as they are installed correctly, you can put them wherever you want, I have not hardcoded any default locations.

a. Python 3

This tool is written in Python and you need to have Python installed for it to work.

You can get Python from its official website. Download the installer, then finish the install before continuing.

b. The Selenium Python package

This is what I use to interact with Google Chrome. It comes with its own Chrome installation, so no need to install it seperately, and it shouldn't change anything on your regular Chrome.

Open your favorite terminal emulator (see the FAQ for how to do this) and type the following commands

py -m ensurepip --upgrade

pip3 install selenium

c. ffmpeg

MyMedia stores videos in a way that makes them difficult to download and use. ffmpeg is a tool that is used (among other things) to convert video formats. This tool uses it to convert that format to mp4 files.

You can download and install ffmpeg from its official website. Please make sure it is in your "path". See the FAQ if you don't know what that means, or if you don't know how to put it in your path.

1. Download the tool

Secondary (right) click here then select "Save Link As" to download it. You can put it wherever you want, but if you put it in your Downloads folder, you can just go off the examples in this guide.

If you're using Mac and don't know what a secondary click is, just click the link normally, then save the resulting page with ⌘+S

2. Get a list of all the videos you want to download

Create a text file (not a Word document) of all the MyMedia URLs (the thingy in the address bar when you're watching the video) you want to get the videos for, one URL per line. Make a note of where you saved this file

3. Create a folder where all your downloaded videos will be put in

Again, make a note of where this folder is

4. Open a terminal emulator and navigate to where you downloaded the tool

Please see the FAQs if you don't know how to do this

5. Run the tool

On Windows (Using cmd)

Type .\mymedia-downloader.py (path to the list) (path to the folder to put all the videos with the trailing \)

Replace the bits in the brackets with the paths. You can drag-and-drop the file and the folder to the terminal if you don't know how to find the path to a file, you will just need to add a \ after the second one. (This not a regular slash, it is a backslah. You type it by pressing the key to the left of the shift key on most English keyboards.)

On Mac or Linux

Type ./mymedia-downloader.py (path to the list) (path to the folder to put all the videos with the trailing /)

Replace the bits in the brackets with the paths. You can probably drag-and-drop the file and the folder to the terminal if you don't know how to find the path to a file, you might need to add a / after the second one.

The reason I'm saying probably is because Linux comes in a variety of flavours, and yours might not support this. I assume if you're using Linux, you know what a path is and how to find one. Otherwise google "find absolute path in (name of your Linux distro)".

6. Log in with your UTORid and password

Wait for the Chrome window to pop up, then click Log In and authenticate. I do not ever see your UTORid and password, all the communication happens directly between you and the university. Please see the source code if you don't believe me.

7. Wait until the screen says "Done."

After that, you should see all the videos you wanted to download in the folder you provided. Unfortunately I have not implemented naming these files correctly, so they are just numbered from 0 onwards. I might come back at this and fix that at some point.

Frequently Asked Querstions

Why should I trust you to enter my UTORid and password?

Great question. In theory, you shouldn't. However, I have provided the source code and you can see if you speak Python that the login process is identical to what would happen if you used Google Chrome yourself. After you log in, I hide the Chrome window to make things more efficient and create less of an eye sore, and download all the videos. If you don't speak Python, please feel free to show this to a friend who does. (Hint: Anyone who studies anything adjacent to computer science, data science or engineering should know Python after their first semester.)

Are you affiliated with (inster organisation here)?

No. I am not. I created this tool on my own, all errors are mine only and I do not claim to represent anyone. This tool is not "official" in any way, shape or form.

I am affiliated with (insert organisation here) and want to yell at you for creating this.

I have made my absolute best to make sure I'm not violating any EULA's or trademarks etc. It does not exploit any vulnerabilities, it simply automates a task that would be very tedious to do by hand. If you disagree and think that I'm either breaking some law/agreement/etc. or if you think this tool will create harm, please contact me. I'm sure you can find my UofT email, email me there. If you can't, please create an issue here.

What is a terminal emulator and how do I open/use one?

Using a terminal is like having a text message conversation with your computer. You tell it (in a very specific format) to do things, press enter, and it does them for you, and replies with the result.

On Windows (cmd)

Open the start menu, then type cmd.exe and press enter. This will open "Command Prompt". You can type cd then where you want to go to, to navigate to a folder. Again, cou can drag-and-drop folders to the command prompt to place avoid having to type their "paths" out. For example, to go to your Downloads folder, you can type cd Downloads right after launching command prompt. You will notice the bit before where you type your command change from C:\Users\yourusername to C:\Users\yourusername\Downloads. If you have changed the location of your Downloads folder, this will not work. I trust if you know how to do that, you know how to use command prompt and can figure this out on your own (or just drag-and-drop the folder.)

On Mac

Read this. You can use the cd gommand to move around. For example, you can type cd ~/Downloads to go to your downloads folder

On Linux

If you run Linux, you probably know how to do this. If you don't, google "(your distribution name) how to use terminal".

How do I put something on my path?

Your path is where your computer looks for things to run.

On Mac/Linux

Run export (path to the thing you want to add to your path) in your terminal each time

On Windows

Read this and do it once. You can remove what you added when you're done if you'd like.

Owner
Ege Feyzioglu
Ege Feyzioglu
A tool can scrape product in aliexpress: Title, Price, and URL Product.

Scrape-Product-Aliexpress A tool can scrape product in aliexpress: Title, Price, and URL Product. Usage: 1. Install Python 3.8 3.9 padahal halaman ins

Rahul Joshua Damanik 1 Dec 30, 2021
A Python web scraper to scrape latest posts from official Coinbase's Blog.

Coinbase Blog Scraper A Python web scraper to scrape latest posts from official Coinbase's Blog. IDEA It scrapes up latest blog posts from https://blo

Lucas Villela 3 Feb 18, 2022
Automatically download and crop key information from the arxiv daily paper.

Arxiv daily 速览 功能:按关键词筛选arxiv每日最新paper,自动获取摘要,自动截取文中表格和图片。 1 测试环境 Ubuntu 16+ Python3.7 torch 1.9 Colab GPU 2 使用演示 首先下载权重baiduyun 提取码:il87,放置于code/Pars

HeoLis 20 Jul 30, 2022
Scrapes mcc-mnc.com and outputs 3 files with the data (JSON, CSV & XLSX)

mcc-mnc.com-webscraper Scrapes mcc-mnc.com and outputs 3 files with the data (JSON, CSV & XLSX) A Python script for web scraping mcc-mnc.com Link: mcc

Anton Ivarsson 1 Nov 07, 2021
Crawler job that scrapes comments from social media posts and saves them in a S3 bucket.

Toxicity comments crawler Crawler job that scrapes comments from social media posts and saves them in a S3 bucket. Twitter Tweets and replies are scra

Douglas Trajano 2 Jan 24, 2022
Python script to check if there is any differences in responses of an application when the request comes from a search engine's crawler.

crawlersuseragents This Python script can be used to check if there is any differences in responses of an application when the request comes from a se

Podalirius 13 Dec 27, 2022
A Happy and lightweight Python Package that searches Google News RSS Feed and returns a usable JSON response and scrap complete article - No need to write scrappers for articles fetching anymore

GNews 🚩 A Happy and lightweight Python Package that searches Google News RSS Feed and returns a usable JSON response 🚩 As well as you can fetch full

Muhammad Abdullah 273 Dec 31, 2022
Web-scraping - A bot using Python with BeautifulSoup that scraps IRS website by form number and returns the results as json

Web-scraping - A bot using Python with BeautifulSoup that scraps IRS website (prior form publication) by form number and returns the results as json. It provides the option to download pdfs over a ra

1 Jan 04, 2022
Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

Parsel Parsel is a BSD-licensed Python library to extract and remove data from HTML and XML using XPath and CSS selectors, optionally combined with re

Scrapy project 859 Dec 29, 2022
WebScraper - A script that prints out a list of all EXTERNAL references in the HTML response to an HTTP/S request

Project A: WebScraper A script that prints out a list of all EXTERNAL references

2 Apr 26, 2022
A web Scraper for CSrankings.com that scrapes University and Faculty list for a particular country

A look into what we're building Demo.mp4 Prerequisites Python 3 Node v16+ Steps to run Create a virtual environment. Activate the virtual environment.

2 Jun 06, 2022
A scalable frontier for web crawlers

Frontera Overview Frontera is a web crawling framework consisting of crawl frontier, and distribution/scaling primitives, allowing to build a large sc

Scrapinghub 1.2k Jan 02, 2023
Scraping Top Repositories for Topics on GitHub,

0.-Webscrapping-using-python Scraping Top Repositories for Topics on GitHub, Web scraping is the process of extracting and parsing data from websites

Dev Aravind D Satprem 2 Mar 18, 2022
A module for CME that spiders hashes across the domain with a given hash.

hash_spider A module for CME that spiders hashes across the domain with a given hash. Installation Simply copy hash_spider.py to your CME module folde

37 Sep 08, 2022
Docker containerized Python Flask API that uses selenium to scrape and interact with websites

Docker containerized Python Flask API that uses selenium to scrape and interact with websites

Christian Gracia 0 Jan 22, 2022
Scrapes Every Email Address of Every Society in Every University

society-email-scrape Site Live at https://kcsoc.github.io/society-email-scrape/ How to automatically generate new data Go to unis.yml Add your uni Cre

Krishna Consciousness Society 18 Dec 14, 2022
This script is intended to crawl license information of repositories through the GitHub API.

GithubLicenseCrawler This script is intended to crawl license information of repositories through the GitHub API. Taking a csv file with requirements.

schutera 4 Oct 25, 2022
crypto currency scraping

SCRYPTO What ? Crypto currencies scraping (At the moment, only bitcoin and ethereum crypto currencies are supported) How ? A python script is running

15 Sep 01, 2022
A simple code to fetch comments below an Instagram post and save them to a csv file

fetch_comments A simple code to fetch comments below an Instagram post and save them to a csv file usage First you have to enter your username and pas

2 Jul 14, 2022
Facebook Group Scraping Using Beautiful Soup & Selenium

Extract Facebook group posts that are related to a specific topic and write them to a .json file.

Fatima Ghadieh 14 Aug 12, 2022