pyspark_sam
This repo hosts my code for the article "Analyze Big Sequence Alignments with PySpark in AWS EMR".
Prerequisite
-
Spark
-
AWS CLI
-
AWS Account
Run
Follow the instruction in the article. Once you have uploaded the files into your S3 bucket, run
aws emr create-cluster --name "Spark_step_pip" \
--release-label emr-6.5.0 \
--applications Name=Spark \
--log-uri s3://[your_S3_bucket]/logs/ \
--instance-type m5.xlarge \
--instance-count 3 \
--bootstrap-actions Path=s3://[your_S3_bucket]/emr_bootstrap.sh \
--use-default-roles --auto-terminate \
--steps "Type=Spark,Name=SparkProgram,ActionOnFailure=CONTINUE,Args=[--deploy-mode,cluster,--master,yarn,--py-files,s3://[your_S3_bucket]/helper_function.py,s3://[your_S3_bucket]/spark_3mer.py,s3://[your_S3_bucket]/test.sam,[your_S3_bucket],sankey.json]"
When the job finishes, download the sankey.json. And run this command to visualize:
python sankey.py sankey.json
Authors
- Sixing Huang - Concept and Coding
License
This project is licensed under the MIT License - see the LICENSE file for details
700 Dec 26, 2022
5 Jun 16, 2022
4.7k Jan 02, 2023
11 Nov 10, 2022
3 Jan 21, 2022
3 Mar 14, 2022
88 Dec 15, 2022
1 Nov 17, 2022
2 Dec 14, 2022
1 Jan 10, 2022
1 Nov 01, 2021
42 Nov 01, 2022
1 Nov 30, 2021
38.2k Jan 01, 2023
16 Oct 21, 2022
72 Nov 18, 2022
682 Dec 28, 2022
4 Dec 05, 2022
2 Nov 06, 2022
6 Nov 27, 2022