Pyspark sam - Analyze Big Sequence Alignments with PySpark in AWS EMR

Related tags

Third-party APIs Wrapperspyspark_sam
Overview

pyspark_sam

This repo hosts my code for the article "Analyze Big Sequence Alignments with PySpark in AWS EMR".

Prerequisite

  1. Spark

  2. AWS CLI

  3. AWS Account

Run

Follow the instruction in the article. Once you have uploaded the files into your S3 bucket, run

aws emr create-cluster --name "Spark_step_pip" \
    --release-label emr-6.5.0 \
    --applications Name=Spark \
    --log-uri s3://[your_S3_bucket]/logs/ \
    --instance-type m5.xlarge \
    --instance-count 3 \
    --bootstrap-actions Path=s3://[your_S3_bucket]/emr_bootstrap.sh \
    --use-default-roles --auto-terminate \
    --steps "Type=Spark,Name=SparkProgram,ActionOnFailure=CONTINUE,Args=[--deploy-mode,cluster,--master,yarn,--py-files,s3://[your_S3_bucket]/helper_function.py,s3://[your_S3_bucket]/spark_3mer.py,s3://[your_S3_bucket]/test.sam,[your_S3_bucket],sankey.json]"

When the job finishes, download the sankey.json. And run this command to visualize:

python sankey.py sankey.json

Authors

  • Sixing Huang - Concept and Coding

License

This project is licensed under the MIT License - see the LICENSE file for details

Owner
Sixing Huang
A triple Neo4j certified data scientist. I am currently working at BGI in Shenzhen.
Sixing Huang
GitHub Repository
allow windows programs to call dssp/mkdssp command from wsl; rework biopython on windows (PDB -> dssp -> fasta)

dssp-wsl Converting PDB (Protein Data Bank) file format to DSSP file format is required for generating datasets of peptides and their secondary struct

Taine Zhao 1 Feb 23, 2022
A modified Sequential and NLP based Bot

A modified Sequential and NLP based Bot I improvised this bot a bit with some implementations as a part of my own hobby project :) Note: I do not own

Jay Desale 2 Jan 07, 2022
Select random winners for a Twitter giveaway

twitter_picker Select random winners for a Twitter giveaway Once the Twitter giveaway (or airdrop) is closed, assign a number to each participant. The

Michael Rawner 1 Dec 11, 2021
This checks that your credit card is valid or not

Credit_card_Validator This checks that your credit card is valid or not. Where is the app ? main.exe is the application to run and main.py is the file

Ritik Ranjan 1 Dec 21, 2021
Nowadays we don't have time to listen to each and every song that we come across in a playlist.

Nowadays we don't have time to listen to each and every song that we come across in a playlist. so, this project helps you. we used Spotify API for collecting the dataset information and able to do E

3 Apr 26, 2022
Using multiple API sources, create an app that allows users to filter through random locations based on their temperature range choices.

World_weather_analysis Overview Using multiple API sources, create an app that allows users to filter through random locations based on their temperat

Jason Boyer 2 Sep 16, 2022
Yes, it's true :two_hearts: This repository has 316 stars.

Yes, it's true! Inspired by a similar repository from @RealPeha, but implemented using a webhook on AWS Lambda and API Gateway, so it's serverless! If

510 Dec 28, 2022
This wrapper now has async support, its basically the same except it uses asyncio

This is a python wrapper for my api api_url = "https://api.dhravya.me/" This wrapper now has async support, its basically the same except it uses asyn

Dhravya Shah 5 Mar 10, 2022
A Python 2.7/3.x module for Amcrest Cameras using the SDK HTTP API.

A Python 2.7/3.x module for Amcrest Cameras using the SDK HTTP API. Amcrest and Dahua devices share similar firmwares. Dahua Cameras and NVRs also work with this module.

Marcelo Moreira de Mello 176 Dec 21, 2022
A customizable, multilanguage Telegram shop bot with Telegram Payments support

Greed A customizable, multilanguage Telegram shop bot with Telegram Payments support! Demo Send a message to @greedtestbot on Telegram to view a demo

Stefano Pigozzi 328 Dec 29, 2022
A GitHub Action that automatically reports your Advent of Code progress in a table in your README

Advent README Stars This action adds and maintains a stars report in your README based on your Advent of Code progress. Example Table 2021 Results Day

Kevin Duff 36 Dec 30, 2022
Sukoshi is a proof-of-concept Python implant that leverages the MQTT protocol for C2 and uses AWS IoT Core as infrastructure.

Sukoshi | 少し Overview Sukoshi is a proof-of-concept Python implant that leverages the MQTT protocol for C2 and uses AWS IoT Core as infrastructure. It

Steven Patterson 37 Oct 29, 2022
A script to generate the m3u playlist containing direct streamable file (.mpd or MPEG-DASH or DASH) based on the channels that the user has subscribed on the Tata Sky portal. You just have to login using your password or otp that's it .

Tata Sky IPTV Script generator A script to generate the m3u playlist containing direct streamable file (.mpd or MPEG-DASH or DASH) based on the channe

Techie Sneh 19 Dec 03, 2021
A collective list of free APIs for use in software and web development.

Public APIs A collective list of free APIs for use in software and web development. A public API for this project can be found here! For information o

222.5k Jan 02, 2023
A small and fun Discord Bot that is written in Python and discord-interactions (with discord.py)

Articuno (discord-interactions) A small and fun Discord Bot that is written in Python and discord-interactions (with discord.py) Get started If you wa

Blue 8 Dec 26, 2022
Salmanul Farisx Bot With Python

Salman_Farisx_Bot How To Deploy Video Subscribe YouTube Channel Added Features Imdb posters for autofilter. Imdb rating for autofilter. Custom caption

1 Dec 23, 2021
A Python implementation of a Youtube Subscription manager & feed viewer, also does thumbnails

BUILDING Building requires python3.10, and the build package, which can be installed via pip: python3.10 -m pip install build To install, run python3.

2 Feb 28, 2022
A Telegram bot to upload files from Telegram or Direct links to Google Drive.

Google Drive Uploader Telegram Bot A Telegram bot to upload files from Telegram or Direct links to Google Drive. Features Telegram files support. Dire

IDNCoderX 21 Dec 05, 2022
Music cog for discord bots. Supports YouTube, YoutubeMusic, SoundCloud and Spotify.

dismusic Music cog for discord bots. Supports YouTube, YoutubeMusic, SoundCloud and Spotify. Installation python3 -m pip install dismusic Usage from d

Md Shahriyar Alam 59 Jan 08, 2023
A Tool to scrape URLs for a given domain from wayback machine, Commoncrawl and OTX Alienvault

Mr_URL Mr.URL fetches known URLs for a given domain from Wayback Machine, Commoncrawl and OTX Alienvault. It also finds old versions of any given URL

Stinger 9 Sep 05, 2022