-
Notifications
You must be signed in to change notification settings - Fork 8
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #3 from prisms-center/4.X
4.0.2
- Loading branch information
Showing
10 changed files
with
198 additions
and
177 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,23 +1,23 @@ | ||
prisms-jobs | ||
=========== | ||
|
||
A Python package and set of scripts used for submitting and managing | ||
cluster jobs via PBS (TORQUE) or Slurm. | ||
A Python package and set of scripts used for submitting and managing | ||
cluster jobs via PBS (TORQUE) or Slurm. | ||
|
||
This code is developed by the PRedictive Integrated Structural Materials Science Center (PRISMS), at the University of Michigan, which is supported by the U.S. Department of Energy, Office of Basic Energy Sciences, Division of Materials Sciences and Engineering under Award #DE-SC0008637, and the Van der Ven group, originally at the University of Michigan and | ||
This code is developed by the PRedictive Integrated Structural Materials Science Center (PRISMS), at the University of Michigan, which is supported by the U.S. Department of Energy, Office of Basic Energy Sciences, Division of Materials Sciences and Engineering under Award #DE-SC0008637, and the Van der Ven group, originally at the University of Michigan and | ||
currently at the University of California Santa Barbara. | ||
|
||
|
||
## Overview | ||
|
||
When submitted through the ``prisms_jobs`` Python package or the included scripts, | ||
cluster jobs are stored in a SQLite jobs database. This allows for convenient | ||
monitoring and searching of submitted jobs. | ||
When submitted through the ``prisms_jobs`` Python package or the included scripts, | ||
cluster jobs are stored in a SQLite jobs database. This allows for convenient | ||
monitoring and searching of submitted jobs. | ||
|
||
It is often necessary to submit multiple jobs until a particular task is complete, | ||
whether due to walltime or other limitations. ``prisms_jobs`` distinguishes and | ||
whether due to walltime or other limitations. ``prisms_jobs`` distinguishes and | ||
tracks both individual "jobstatus" ('R', 'Q', 'C', 'E', etc.) and "taskstatus". | ||
Jobs marked as 'auto' can be automatically or easily resubmitted until the | ||
Jobs marked as 'auto' can be automatically or easily resubmitted until the | ||
"taskstatus" is "Complete". | ||
|
||
Possible values for "taskstatus" are: | ||
|
@@ -32,12 +32,12 @@ Possible values for "taskstatus" are: | |
|"Aborted" |The job and task have been aborted. | | ||
|
||
|
||
Jobs are marked 'auto' either by submitting through the python class ``prisms_jobs.Job`` | ||
with the attribute ``auto=True``, or by submitting a script which contains | ||
Jobs are marked 'auto' either by submitting through the python class ``prisms_jobs.Job`` | ||
with the attribute ``auto=True``, or by submitting a script which contains | ||
the line ``#auto=True`` using the included ``psub`` command line program. | ||
|
||
Jobs can be monitored using the command line program ``pstat``. All 'auto' jobs | ||
which have stopped can be resubmitted using ``pstat --continue``. In this case, | ||
Jobs can be monitored using the command line program ``pstat``. All 'auto' jobs | ||
which have stopped can be resubmitted using ``pstat --continue``. In this case, | ||
'continuation_jobid' is set with the jobid for the next job in the series of jobs | ||
comprising a task. | ||
|
||
|
@@ -58,15 +58,15 @@ Example screen shot: | |
------------ ------------------------ ----- ----- ------------ - ------------ ------------------------ - ------------ | ||
11791026 taskmaster 1 1 0:01:00:00 W 0:01:00:00 Untracked 0 - | ||
|
||
Additionally, when scheduling periodic jobs is not allowed other ways, the | ||
``taskmaster`` script can fully automate this process. ``taskmaster`` executes | ||
Additionally, when scheduling periodic jobs is not allowed other ways, the | ||
``taskmaster`` script can fully automate this process. ``taskmaster`` executes | ||
``pstat --continue`` and then resubmits itself to execute again periodically. | ||
|
||
A script marked 'auto' should check itself for completion and when reached execute | ||
``pstat --complete $JOBID --force`` in bash, or ``prisms_jobs.complete_job()`` in Python. If an 'auto' job script does not set its taskstatus to "Complete" it may continue | ||
A script marked 'auto' should check itself for completion and when reached execute | ||
``pstat --complete $JOBID --force`` in bash, or ``prisms_jobs.complete_job()`` in Python. If an 'auto' job script does not set its taskstatus to "Complete" it may continue | ||
to be resubmitted indefinitely. | ||
|
||
Jobs not marked 'auto' are shown with the status "Check" in ``pstat`` until the user | ||
Jobs not marked 'auto' are shown with the status "Check" in ``pstat`` until the user | ||
marks them as "Complete". | ||
|
||
|
||
|
@@ -77,11 +77,11 @@ Using ``pip``: | |
pip install prisms-jobs | ||
|
||
or, to install in your user directory: | ||
|
||
pip install --user prisms-jobs | ||
|
||
If installing to a user directory, you may need to set your PATH to find the installed scripts. This can be done using: | ||
|
||
export PATH=$PATH:`python -m site --user-base`/bin | ||
|
||
|
||
|
@@ -99,20 +99,20 @@ If installing to a user directory, you may need to set your PATH to find the ins | |
git clone https://github.com/prisms-center/prisms_jobs.git | ||
cd prisms_jobs | ||
|
||
2. Checkout the branch/tag containing the version you wish to install. Latest is ``v4.0.1``: | ||
2. Checkout the branch/tag containing the version you wish to install. Latest is ``v4.0.2``: | ||
|
||
git checkout v4.0.1 | ||
git checkout v4.0.2 | ||
|
||
2. From the root directory of the repository: | ||
|
||
pip install . | ||
|
||
or, to install in your user directory: | ||
|
||
pip install --user . | ||
|
||
If installing to a user directory, you may need to set your PATH to find the installed scripts. This can be done using: | ||
|
||
export PATH=$PATH:`python -m site --user-base`/bin | ||
|
||
|
||
|
@@ -123,15 +123,15 @@ See the [docs](https://prisms-center.github.io/prisms_jobs_docs/). | |
|
||
## License | ||
|
||
This directory contains the prisms_jobs Python package and related scripts developed | ||
This directory contains the prisms_jobs Python package and related scripts developed | ||
by the PRISMS Center at the University of Michigan, Ann Arbor, USA, and | ||
the Van der Ven group, originally at the University of Michigan and | ||
the Van der Ven group, originally at the University of Michigan and | ||
currently at the University of California Santa Barbara. | ||
|
||
(c) 2014 The Regents of the University of Michigan | ||
(c) 2015 The Regents of the University of California | ||
|
||
PRISMS Center http://prisms.engin.umich.edu | ||
PRISMS Center http://prisms.engin.umich.edu | ||
contact: [email protected] | ||
|
||
This code is a free software; you can use it, redistribute it, | ||
|
@@ -140,5 +140,3 @@ License as published by the Free Software Foundation; either version | |
2.1 of the License, or (at your option) any later version. | ||
|
||
Please see the file LICENSE for details. | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,21 +1,32 @@ | ||
# begin | ||
anaconda login | ||
# function to allow passing conda build command line arguments | ||
build_conda () { | ||
# build, get location of result, upload | ||
conda build $1 conda-recipes/prisms_jobs > conda-recipes/tmp.out | ||
LOCATION=$(grep 'conda upload' conda-recipes/tmp.out | cut -f3 -d ' ') | ||
if [ -z $LOCATION ]; then | ||
LOCATION=$(grep 'Nothing to test for' conda-recipes/tmp.out | cut -f5 -d ' ') | ||
fi | ||
anaconda upload --user prisms-center $LOCATION | ||
|
||
# build, get location of result, upload | ||
conda build conda-recipes/prisms_jobs > conda-recipes/tmp.out | ||
LOCATION=$(grep 'conda upload' conda-recipes/tmp.out | cut -f3 -d ' ') | ||
if [ -z $LOCATION ]; then | ||
LOCATION=$(grep 'Nothing to test for' conda-recipes/tmp.out | cut -f5 -d ' ') | ||
fi | ||
anaconda upload --user prisms-center $LOCATION | ||
if [[ "$OSTYPE" == "linux-gnu" ]]; then | ||
OTHER=osx-64 | ||
elif [[ "$OSTYPE" == "darwin"* ]]; then | ||
OTHER=linux-64 | ||
fi | ||
|
||
# if on linux, convert to osx | ||
conda convert --platform $OTHER $LOCATION -o conda-recipes | ||
|
||
FILE=$(basename $LOCATION) | ||
anaconda upload --user prisms-center conda-recipes/$OTHER/$FILE | ||
|
||
} | ||
|
||
# if on linux, convert to osx | ||
conda convert --platform osx-64 $LOCATION -o conda-recipes | ||
anaconda upload --user prisms-center conda-recipes/osx-64/prisms-jobs* | ||
# begin | ||
anaconda login | ||
|
||
# if on osx, convert to linux | ||
conda convert --platform linux-64 $LOCATION -o conda-recipes | ||
anaconda upload --user prisms-center conda-recipes/linux-64/prisms-jobs* | ||
build_conda "--python 2.7" | ||
build_conda "--python 3.6" | ||
|
||
# finish | ||
anaconda logout |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.