shpc_condo_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating a PBS Script for SHPC Condo\n",
    "**Emily Costa**\n",
    "08/01/2019\n",
    "***\n",
    "This tutorial is meant to introduce a CADES user to creating a PBS script in order to submit and run jobs on the SHPC Condo at Oak Ridge National Laboratory. In order to properly complete this tutorial, please open JupyterLab in the Condo."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's write some code to run on the cluster.We will be writing python and PBS scripts and creating new files to submit on the Condo.\n",
    "***\n",
    "The following is an example of how scripts can be written from a Jupyter Notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing hello_world.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile hello_world.py\n",
    "print(\"Hello world\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we are going to ***write a simple 'Hello World' program for MPI.***\n",
    "\n",
    "Using mpi4py, find and print of the size, rank, and processor name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mpi4py import MPI\n",
    "import sys\n",
    "\n",
    "# Your code\n",
    "\n",
    "sys.stdout.write(\n",
    "    \"Hello, World! I am process %d of %d on %s.\\n\"\n",
    "    % (rank, size, name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now **check your answer**. Make sure to run this cell after loading the answer, as we want to save the script to a new file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load solutions/PBS_file00"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a basic Python script to run on the Condo, we need to create the PBS script. There are two main components to creating a PBS script:\n",
    "1. Create and fill out flags for submitting the job.\n",
    "2. Write script to prepare for and run your python script.\n",
    "\n",
    "Let us begin."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Flags for Submitting a Job on the Condo\n",
    "\n",
    "The first part of the PBS script needs to contain all the flags needed to run your Python script. They will be written to the script in the following format:\n",
    "\n",
    "> `#PBS -flag argument`\n",
    "***\n",
    "\n",
    "> `-N job_name`, set the job name. Your output files will share this name.\n",
    "\n",
    "> `-M email_address`, errors will be emailed to this address.\n",
    "\n",
    "> `-l nodes=##:ppn=##`, node specs (number of node:processors per node)\n",
    "\n",
    "> `-l walltime=D:HH:MM:SS`, anticipated runtime for the job\n",
    "\n",
    "> `-W group_list=LDAP_group_list`, ex. cades-birthright.\n",
    "\n",
    "> `-A account_type`, ex. birthright.\n",
    "\n",
    "> `-l qos=quality_of_service`, ex. burst or std."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Write Main Program of PBS Script\n",
    "\n",
    "> `module purge`, remove old modules to ensure a clean slate.\n",
    "\n",
    "> `module load ...` or `source /path/to/script`, load modules to set up your programming environment.\n",
    "\n",
    "> `script_path /path/to/files` or `data_path /path/to/data`, set paths that are needed to copy scripts or data to current path. This is optional and only if the files are not in your current directory. For the hello_world example, we will not need to set paths.\n",
    "\n",
    "> `cp $script_path/script .` or `cp $data_path/data .`, copy the need scripts and data to your current directory. Again, this is not necessary if what you need is already there.\n",
    "\n",
    "> `mpirun ... python ...` or `mpiexec ... python ...`, run the Python script.\n",
    "\n",
    "> Done!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let us create a PBS file to run our \"Hello World\" python script. You will need to fill out the flags with your information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile hello_world.pbs\n",
    "#!/bin/bash\n",
    "\n",
    "#PBS -N HelloWorld\n",
    "#PBS -M your_email@ornl.gov\n",
    "#PBS -l nodes=1:ppn=4\n",
    "#PBS -l walltime=0:00:01:00\n",
    "#PBS -W group_list=cades-birthright\n",
    "#PBS -A birthright\n",
    "#PBS -l qos=burst\n",
    "\n",
    "module purge\n",
    "\n",
    "module load python/3.6.3\n",
    "\n",
    "mpiexec --map-by ppr:1:node python hello_world_mpi.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we want to submit the job. In the same directory as your PBS and Python scripts, run the following Linux command:\n",
    "\n",
    "> `qsub hello_world.pbs`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following are useful qsub arguments:\n",
    "\n",
    "> `checkjob job_number`, check status of jobs.\n",
    "\n",
    "> `qdel job_number`, cancel pending job.\n",
    "\n",
    "> `more job_name.ojob_number`, contains the results of the executed binary.\n",
    "\n",
    "> `more job_name.ejob_number`, contains any errors that occurred during execution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more information:\n",
    "\n",
    "https://cades.ornl.gov/service-suite/scalable-hpc/"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}