Exercise to teach a newcomer to the CLSP grid to set up their environment and run jobs

Overview

Onboarding Exercise

You will make a list of the most common URLs in all the tweets in /home/aadelucia/files/minerva/raw_tweets_deduplicated/tweets/.

Goals of this exercise

  • Learn how to submit a job to the scheduler
  • Get comfortable with Tweet JSON format
  • Learn strategies for working with large datasets (map-reduce style setup)
  • Refresh on git commands

0. Set up your workspace

You will need to be on the grid for this exercise. You can access the grid with ssh @login.clsp.jhu.edu .

Set up your .bashrc

In order to make the grid really feel like home, we need to personalize it. We can do this by setting environment variables, having specific programs run on login, etc. Different OS have different places to save your preferences. On the grid, your preferences immediately get loaded upon login from ~/.profile. This is dumb because most other automated installed edit your ~/.bashrc to modify environment variables such as your PATH.

We need to edit your ~/.profile to source your .bashrc upon login. It's some weird thing where most programs edit your .bashrc but the system doesn't look at it upon login, so this will fix that. Open your ~/.profile with your favorite editor (I use vim) and add the following

# Code from Steven at CLSP help to read bashrc on login
if [ -n "BASH_VERSION" ]; then
    # include .bashrc if it exists
    if [ -f "$HOME/.bashrc" ]; then
    . "$HOME/.bashrc"
    fi
fi

Now you can add whatever you want to your ~/.bashrc and it will load on startup. For example, mine saves project directories as environment variables and loads a specific conda environment on login:

export PROJECT=/path/to/project
conda activate fav-env

Set up SSH keys

Okay we will need to set up multiple SSH keys in order to avoid entering our passwords over and over: (1) from your local machine to the grid (2) from the grid to GitHub (3) from your grid home directory to other nodes on the cluster.

  1. On your local machine, check if you have a key already. You can do this with ls ~/.ssh/id_rsa.pub. If that file exists, then you already have a key and skip to step 3.
  2. Generate a new private/public key pair with ssh-keygen -C local machine and accept the defaults. (-C is just a comment, this lets you know what the key is for)
  3. Copy your public key to the grid with ssh-copy-id -i ~/.ssh/id_rsa.pub @login.clsp.jhu.edu
  4. Make sure it works by ssh-ing to the grid. You should not have to type your password anymore. For debugging, contact me.
  5. On the grid, generate a new key for acessing GitHub with ssh-keygen -C CLSP grid and accept the defaults
  6. Copy the contents of ~/.ssh/id_rsa.pub and add it to your GitHub SSH keys (under Settings > SSH and GPG Keys)
  7. Add your Grid SSH key for easy node access with cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

CLSP Grid 101

Think of the grid, also referred to as a cluster or a "supercomputer" on large scales, as a community of well-organized citizens. Each citizen (node) has its role to play in the cluster society. When you log onto the cluster, the login nodes greet you. Their names are login and login2, and are accessed with ssh @login.clsp.jhu.edu . Their job is to welcome you to the grid and so some SCPing for files. That is all, everything else is above their paygrade. Now there are the compute nodes. They are the real heavy lifters, the real workers! They are identified by a letter and a number, such as b01 and c12. To see a full list of the nodes, run qconf -sel. These nodes are where you do all your file editing and debugging. They are accessed by first logging onto the grid and then running ssh like ssh b10. Working only on the compute nodes frees up the login node resources to welcome others to the grid.

It's easy to forget to immediately move to a compute node, so a modified ssh script can be used to access a compute node directly. From your local machine, run

ssh [email protected] -t ssh b10

This accesses the grid through the login node and then immediately ssh's onto a compute node.

Set up Anaconda

The grid has Python 3 but in order to make setting up our own environments easier, we will install the package manager Anaconda. Or, more specifically, Miniconda, which is just a lightweight version of Anaconda (doesn't come with all the packages installed).

Log on to the grid and run the following in your home directory:

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
chmod u+x Miniconda3-latest-Linux-x86_64.sh
./Miniconda3-latest-Linux-x86_64.sh

This downloads the Miniconda installation file and begins the installation process. I recommend installing it in your home directory, but you could also put it in the folder you created on a compute node.

After installation is over, run source ~/.bashrc to update your PATH.

To set up an environment for this project, go to the Minerva project root directory and run:

conda env create --file environment.yml
conda activate onboarding

Note: You can add conda activate onboarding to the end of your ~/.bashrc to make this environment load upon login.

Clone this repo (if you haven't already)

You will have to re-clone the repo if you already cloned using HTTPS and would like to use SSH.

  1. Pick a pretty node name that you like (list node names with ls /export/. I'm on /export/fs04/a13) and create a folder with your name in /export/ . The /home directory is small so do not store large datasets or repos there, or it could impact other users.
  2. Clone this repo with git clone [email protected]:AADeLucia/CLSP-grid-onboarding.git

1. Fill in the code

Remember from above, edit on a compute node!

analyze_tweet_urls.py has the starter code. The task is to complete the script so that it does the following:

Loops through the input files, counts the unique URLs, and saves the counts in the output directory in an easily parsible format (one output file per input file)

As I've been messing with tweets I created a small package (littlebird) for the processes I do over and over again (like iterating over GZIP'd Twitter files). It is already installed in the onboarding environment. An example of how to use littlebird is in token_count_features.py. I highly recommend looking at this script for a guide on how to do this exercise and how to use littlebird.

2. Submit the job to run

Okay your script is done, now we want to run it! There are about 86K Twitter files so we need to be smart about this or it will take 5ever. Luckily, we already wrote our script to take in a list of files, so the plan is to use a "job array" to submit multiple small jobs as part of one big job. A starter script that you need to fill in is batch_count_urls.sh.

  1. Edit batch_count_urls.sh to work with your script. I suggest starting with -t 1-2 to make sure your script works.
  2. Submit the job with qsub batch_count_urls.sh
  3. Check the progress of your job by looking at your log file or qstat -u

If your job status is Eqw this means there was an error in the job submission. To read the errors run qstat -j .

Note: when working with a job array, each job gets submitted to run with the most recent version of your code so be careful about editing your code while waiting for it to end.

3. Aggregate the counts

Create a method to aggregate the counts (in analyze_tweet_urls.py) that:

  • Loops over all the files in a given directory
  • Adds up the counts for the unique URLs
  • Saves the output (either a pickle dump or to a tab-separated (TSV) file)
  • Can be called with a command-line flag (e.g. --aggregate)

4. Make a pretty visualization

You have your results, yay! Now I want you to do something pretty with it. Make a word cloud, do a basic plot, whatever you want! This part is mostly to teach you how to use a jupyter notebook on the grid.

  1. Edit the paths in forward_notebook.sh to match your desired directory (home or where your files are). Also, change the PORT variable to a unique number (preferably above 5000).
  2. From the login node (NOT a compute node like b10. To get from a compute node to login run exit) run ~/.forward_notebook.sh to start the Jupyter Notebook. This job will stay running, so you only have to run this script again if something goes wrong.
  3. On your local machine run ssh -L {PORT}:localhost:{PORT} @login.clsp.jhu.edu (where PORT is from the earlier step) and navigate to localhost:{PORT} in your browser. If you are prompted for a token or password, open the notebook_log directory (specified in forward_notebook.sh) and take the notebook URL from the most recent log file.
  4. Make a visualization

5. Bask in the glory

Congratulations, you finished the exercise! Go reward yourself with some ice cream or send me a message so I can compliment you.

Owner
Alexandra
Computer Science graduate student at Johns Hopkins. My work is in natural language processing, data science, and machine learning.
Alexandra
Convert .1pux to .csv

1PasswordConverter Convert .1pux to .csv 1Password uses this new export format .1pux, I assume stands for 1 Password User eXport. As of right now, 1Pa

Shayne Hartford 7 Dec 16, 2022
Python - Aprendendo Python na ByLearn

PYTHON Identação Escopo Pai Escopo filho Escopo neto Variaveis

Italo Rafael 3 May 31, 2022
A Python program that generates a maze that solves itself using DFS

Maze Generator And Solver Program Purpose: Generates a maze that then solves itself Language: Python and Pygame Algorithm: Randomized DFS / Floodfill

Joshua Liu 1 Jul 25, 2022
Inverted-pendulum-with-fuzzy-control - Inverted pendulum with fuzzy control

Fuzzy Inverted Pendulum Basically, this project consists of an inverted pendulum

Mahan Ahmadvand 1 Aug 25, 2022
Install packages with pip as if you were in the past!

A PyPI time machine Do you wish you could just install packages with pip as if you were at some fixed date in the past? If so, the PyPI time machine i

Thomas Robitaille 51 Jan 09, 2023
A PowSyBl and Python integration based on GraalVM native image

PyPowSyBl The PyPowSyBl project gives access PowSyBl Java framework to Python developers. This Python integration relies on GraalVM to compile Java co

powsybl 23 Dec 14, 2022
JD扫码获取Cookie 本地版

JD扫码获取Cookie 本地版 请无视手机上的提示升级京东版本的提示! 下载链接 https://github.com/Zy143L/jd_cookie/releases 使用Python实现 代码很烂 没有做任何异常捕捉 但是能用 请不要将获取到的Cookie发送给任何陌生人 如果打开闪退 请使

Zy143L 420 Dec 11, 2022
Programa principal de la Silla C.D.P.

Silla CDP Página Web Contáctenos Lista de contenidos: Información del proyecto. Licencias. Contacto. Información del proyecto Silla CDP, o Silla Corre

Silla Control de Postura 1 Dec 02, 2021
Auto-ropper is a tool that aims to automate the exploitation of ROP.

Auto-ropper is a tool that aims to automate the exploitation of ROP. Its goal is to become a tool that no longer requires user interaction.

Zerotistic 16 Nov 13, 2022
An app that mirrors your phone to your compute and maps controller input to the screen

What is 'Dragalia Control'? An app that mirrors your phone to your compute and maps controller input to the screen. Inputs are mapped specifically for

1 May 03, 2022
A passive recon suite designed for fetching the information about web application

FREAK Suite designed for passive recon Usage: python3 setup.py python3 freak.py warning This tool will throw error if you doesn't provide valid api ke

toxic v3nom 7 Feb 17, 2022
Tools for analyzing Java JVM gc log files

gc_log This package consists of two separate utilities useful for : gc_log_visualizer.py regionsize.py GC Log Visualizer This was updated to run under

Brad Schoening 0 Jan 04, 2022
Fused multiply-add (with a single rounding) for Python.

pyfma Fused multiply-add for Python. Fused multiply-add computes (x*y) + z with a single rounding. Useful for dot products, matrix multiplications, po

Nico Schlömer 18 Nov 08, 2022
Keep your company's passwords behind the firewall

TeamVault TeamVault is an open-source web-based shared password manager for behind-the-firewall installation. It requires Python 3.3+ and Postgres (wi

//SEIBERT/MEDIA GmbH 38 Feb 20, 2022
A "multiclipboards" script for an efficient way to improve the original clipboards which are only able to save one string at a time

A "multiclipboards" script for an efficient way to improve the original clipboards which are only able to save one string at a time. Works on both Windows and Linux.

1 Jan 24, 2022
A small Blender addon for changing an object's local orientation while in edit mode

A small Blender addon for changing an object's local orientation while in edit mode.

Jonathan Lampel 50 Jan 06, 2023
Course materials for a 3-day seminar "Machine Learning and NLP: Advances and Applications" at New College of Florida

Machine Learning and NLP: Advances and Applications This repository hosts the course materials used for a 3-day seminar "Machine Learning and NLP: Adv

Yoshi Suhara 11 Jun 22, 2022
APRS Track Direct is a collection of tools that can be used to run an APRS website

APRS Track Direct APRS Track Direct is a collection of tools that can be used to run an APRS website. You can use data from APRS-IS, CWOP-IS, OGN, HUB

Per Qvarforth 42 Dec 29, 2022
This is a practice on Airflow, which is building virtual env, installing Airflow and constructing data pipeline (DAGs)

airflow-test This is a practice on Airflow, which is Builing virtualbox env and setting Airflow on that env Installing Airflow using python virtual en

Jaeyoung 1 Nov 01, 2021
I³ Tracker for Essential Open Innovation Datasets

I³ Tracker for Essential Open Innovation Datasets This repository is set up to track, version, and contribute updates to the I³ Essential Open Innovat

1 Feb 08, 2022