README.md

# wait_notify

# WHAT IT DOES
Python script that runs as cron job and notifies HPC users about 'non-runable'
jobs, that is pending jobs that will never run.


# HOW IT WORKS
The script evaluates the 'reason' code for each pending job.  

Currently, it only notifies users whose jobs have a reason of
'PartitionTimeLimit', which means that the user set a time limit for their job
which exceeds that allowed for the partition.

Any user with one or more 'non-runable' jobs are sent one email that lists all
the non-runable jobs, and a record of the email is stored in an SQLITE3
database.  Even those the cron job runs daily, only one email is sent per week.


# USAGE

   Usage: wait_notify [-Icx] [-n N] [-t ADMIN_EMAIL]

   Read Slurm's sinfo output to determine which pending jobs are in a stuck
   state that will not run, and email the jobs's owner so they can
   cancel them and re-run if desired.

   Each email is logged in an SQLITE3 database, and emails will not be sent
   if an email has already gone out in the previous week.

   OPTIONS
     -I              Run the first time to initialize Sqlite3 database
                     that records emails.
     -x              Send email to users with stuck jobs.
   OPTIONS USEFUL FOR TESTING
     -c              Check only.  List jobs that are in a cancelled state
     -t ADMIN_EMAIL  Useful for testing.  Use only with -x.  Emails will be 
                        sent not to users but to the ADMIN_EMAIL instead.
     -n N            Only email the first N users.
     -f              Force email, even if one was sent in past week


# CONFIGURATION

See files
   user_notify_config.py
   wait_notify_config.py