Skip to content

Commit

Permalink
Merge pull request #3 from prisms-center/4.X
Browse files Browse the repository at this point in the history
4.0.2
  • Loading branch information
bpuchala authored Apr 18, 2019
2 parents e9e19b3 + 7d3f33c commit 9a22178
Show file tree
Hide file tree
Showing 10 changed files with 198 additions and 177 deletions.
60 changes: 29 additions & 31 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,23 @@
prisms-jobs
===========

A Python package and set of scripts used for submitting and managing
cluster jobs via PBS (TORQUE) or Slurm.
A Python package and set of scripts used for submitting and managing
cluster jobs via PBS (TORQUE) or Slurm.

This code is developed by the PRedictive Integrated Structural Materials Science Center (PRISMS), at the University of Michigan, which is supported by the U.S. Department of Energy, Office of Basic Energy Sciences, Division of Materials Sciences and Engineering under Award #DE-SC0008637, and the Van der Ven group, originally at the University of Michigan and
This code is developed by the PRedictive Integrated Structural Materials Science Center (PRISMS), at the University of Michigan, which is supported by the U.S. Department of Energy, Office of Basic Energy Sciences, Division of Materials Sciences and Engineering under Award #DE-SC0008637, and the Van der Ven group, originally at the University of Michigan and
currently at the University of California Santa Barbara.


## Overview

When submitted through the ``prisms_jobs`` Python package or the included scripts,
cluster jobs are stored in a SQLite jobs database. This allows for convenient
monitoring and searching of submitted jobs.
When submitted through the ``prisms_jobs`` Python package or the included scripts,
cluster jobs are stored in a SQLite jobs database. This allows for convenient
monitoring and searching of submitted jobs.

It is often necessary to submit multiple jobs until a particular task is complete,
whether due to walltime or other limitations. ``prisms_jobs`` distinguishes and
whether due to walltime or other limitations. ``prisms_jobs`` distinguishes and
tracks both individual "jobstatus" ('R', 'Q', 'C', 'E', etc.) and "taskstatus".
Jobs marked as 'auto' can be automatically or easily resubmitted until the
Jobs marked as 'auto' can be automatically or easily resubmitted until the
"taskstatus" is "Complete".

Possible values for "taskstatus" are:
Expand All @@ -32,12 +32,12 @@ Possible values for "taskstatus" are:
|"Aborted" |The job and task have been aborted. |


Jobs are marked 'auto' either by submitting through the python class ``prisms_jobs.Job``
with the attribute ``auto=True``, or by submitting a script which contains
Jobs are marked 'auto' either by submitting through the python class ``prisms_jobs.Job``
with the attribute ``auto=True``, or by submitting a script which contains
the line ``#auto=True`` using the included ``psub`` command line program.

Jobs can be monitored using the command line program ``pstat``. All 'auto' jobs
which have stopped can be resubmitted using ``pstat --continue``. In this case,
Jobs can be monitored using the command line program ``pstat``. All 'auto' jobs
which have stopped can be resubmitted using ``pstat --continue``. In this case,
'continuation_jobid' is set with the jobid for the next job in the series of jobs
comprising a task.

Expand All @@ -58,15 +58,15 @@ Example screen shot:
------------ ------------------------ ----- ----- ------------ - ------------ ------------------------ - ------------
11791026 taskmaster 1 1 0:01:00:00 W 0:01:00:00 Untracked 0 -

Additionally, when scheduling periodic jobs is not allowed other ways, the
``taskmaster`` script can fully automate this process. ``taskmaster`` executes
Additionally, when scheduling periodic jobs is not allowed other ways, the
``taskmaster`` script can fully automate this process. ``taskmaster`` executes
``pstat --continue`` and then resubmits itself to execute again periodically.

A script marked 'auto' should check itself for completion and when reached execute
``pstat --complete $JOBID --force`` in bash, or ``prisms_jobs.complete_job()`` in Python. If an 'auto' job script does not set its taskstatus to "Complete" it may continue
A script marked 'auto' should check itself for completion and when reached execute
``pstat --complete $JOBID --force`` in bash, or ``prisms_jobs.complete_job()`` in Python. If an 'auto' job script does not set its taskstatus to "Complete" it may continue
to be resubmitted indefinitely.

Jobs not marked 'auto' are shown with the status "Check" in ``pstat`` until the user
Jobs not marked 'auto' are shown with the status "Check" in ``pstat`` until the user
marks them as "Complete".


Expand All @@ -77,11 +77,11 @@ Using ``pip``:
pip install prisms-jobs

or, to install in your user directory:

pip install --user prisms-jobs

If installing to a user directory, you may need to set your PATH to find the installed scripts. This can be done using:

export PATH=$PATH:`python -m site --user-base`/bin


Expand All @@ -99,20 +99,20 @@ If installing to a user directory, you may need to set your PATH to find the ins
git clone https://github.com/prisms-center/prisms_jobs.git
cd prisms_jobs

2. Checkout the branch/tag containing the version you wish to install. Latest is ``v4.0.1``:
2. Checkout the branch/tag containing the version you wish to install. Latest is ``v4.0.2``:

git checkout v4.0.1
git checkout v4.0.2

2. From the root directory of the repository:

pip install .

or, to install in your user directory:

pip install --user .

If installing to a user directory, you may need to set your PATH to find the installed scripts. This can be done using:

export PATH=$PATH:`python -m site --user-base`/bin


Expand All @@ -123,15 +123,15 @@ See the [docs](https://prisms-center.github.io/prisms_jobs_docs/).

## License

This directory contains the prisms_jobs Python package and related scripts developed
This directory contains the prisms_jobs Python package and related scripts developed
by the PRISMS Center at the University of Michigan, Ann Arbor, USA, and
the Van der Ven group, originally at the University of Michigan and
the Van der Ven group, originally at the University of Michigan and
currently at the University of California Santa Barbara.

(c) 2014 The Regents of the University of Michigan
(c) 2015 The Regents of the University of California

PRISMS Center http://prisms.engin.umich.edu
PRISMS Center http://prisms.engin.umich.edu
contact: [email protected]

This code is a free software; you can use it, redistribute it,
Expand All @@ -140,5 +140,3 @@ License as published by the Free Software Foundation; either version
2.1 of the License, or (at your option) any later version.

Please see the file LICENSE for details.


41 changes: 26 additions & 15 deletions build_conda.sh
Original file line number Diff line number Diff line change
@@ -1,21 +1,32 @@
# begin
anaconda login
# function to allow passing conda build command line arguments
build_conda () {
# build, get location of result, upload
conda build $1 conda-recipes/prisms_jobs > conda-recipes/tmp.out
LOCATION=$(grep 'conda upload' conda-recipes/tmp.out | cut -f3 -d ' ')
if [ -z $LOCATION ]; then
LOCATION=$(grep 'Nothing to test for' conda-recipes/tmp.out | cut -f5 -d ' ')
fi
anaconda upload --user prisms-center $LOCATION

# build, get location of result, upload
conda build conda-recipes/prisms_jobs > conda-recipes/tmp.out
LOCATION=$(grep 'conda upload' conda-recipes/tmp.out | cut -f3 -d ' ')
if [ -z $LOCATION ]; then
LOCATION=$(grep 'Nothing to test for' conda-recipes/tmp.out | cut -f5 -d ' ')
fi
anaconda upload --user prisms-center $LOCATION
if [[ "$OSTYPE" == "linux-gnu" ]]; then
OTHER=osx-64
elif [[ "$OSTYPE" == "darwin"* ]]; then
OTHER=linux-64
fi

# if on linux, convert to osx
conda convert --platform $OTHER $LOCATION -o conda-recipes

FILE=$(basename $LOCATION)
anaconda upload --user prisms-center conda-recipes/$OTHER/$FILE

}

# if on linux, convert to osx
conda convert --platform osx-64 $LOCATION -o conda-recipes
anaconda upload --user prisms-center conda-recipes/osx-64/prisms-jobs*
# begin
anaconda login

# if on osx, convert to linux
conda convert --platform linux-64 $LOCATION -o conda-recipes
anaconda upload --user prisms-center conda-recipes/linux-64/prisms-jobs*
build_conda "--python 2.7"
build_conda "--python 3.6"

# finish
anaconda logout
4 changes: 2 additions & 2 deletions conda-recipes/prisms_jobs/meta.yaml
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
package:
name: prisms-jobs
version: "4.0.1"
version: "4.0.2"

source:
git_rev: v4.0.1
git_rev: v4.0.2
git_url: https://github.com/prisms-center/prisms_jobs.git

requirements:
Expand Down
5 changes: 1 addition & 4 deletions doc/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@
# The short X.Y version.
version = u'4.0'
# The full version, including alpha/beta/rc tags.
release = u'4.0.1'
release = u'4.0.2'

# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
Expand Down Expand Up @@ -196,6 +196,3 @@
author, 'prisms_jobs', 'One line description of project.',
'Miscellaneous'),
]



19 changes: 8 additions & 11 deletions doc/source/install.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@ or, to install in your user directory:
::

pip install --user prisms-jobs
If installing to a user directory, you may need to set your PATH to find the

If installing to a user directory, you may need to set your PATH to find the
installed scripts. This can be done using:

::
Expand Down Expand Up @@ -45,30 +45,27 @@ Install from source
git clone https://github.com/prisms-center/prisms_jobs.git
cd prisms_jobs

2. Checkout the branch/tag containing the version you wish to install. Latest is ``v4.0.1``:
2. Checkout the branch/tag containing the version you wish to install. Latest is ``v4.0.2``:

::

git checkout v4.0.1
git checkout v4.0.2

3. From the root directory of the repository:

::

pip install .

or, to install in your user directory:

::

pip install --user .
If installing to a user directory, you may need to set your ``PATH`` to find the

If installing to a user directory, you may need to set your ``PATH`` to find the
installed scripts. This can be done using:

::
::

export PATH=$PATH:`python -m site --user-base`/bin



6 changes: 3 additions & 3 deletions prisms_jobs/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,17 +9,17 @@ def __init__(self, jobid, msg):

def __str__(self):
return self.jobid + ": " + self.msg

# import into 'prisms_jobs'
from prisms_jobs.job import Job
from prisms_jobs.jobdb import JobDB, JobDBError, EligibilityError, complete_job, error_job

__version__ = '4.0.1'
__version__ = '4.0.2'
__all__ = [
'Job',
'JobDB',
'JobsError',
'JobDBError',
'EligibilityError',
'EligibilityError',
'complete_job',
'error_job']
Loading

0 comments on commit 9a22178

Please sign in to comment.