Handle potentially large numbers of concurrent jobs

[emlys]

Investigate and handle potential bottlenecks as the number of jobs increases:
- slurm maximum concurent running jobs?
- slurm maximum total jobs including pending, finished?
- concurrent connections to the slurm database?
- number of monitor threads (currently one per job not yet completed)? @phargogh pointed out the strigger command - this kind of callback design could be an improvement