parallel: killall will kill the family, too.

doc/release_new_version

@@ -4,7 +4,7 @@
 
 Check that documentation is updated (compare to web):
 
-Fixet for 20150422
+Fixet for 20150522
 git diff last-release-commit
 Unmodified beta since last version => production
 Unmodified alpha since last version => beta
@@ -209,64 +209,30 @@ cc:Tim Cuthbertson <tim3d.junk@gmail.com>,
    Ryoichiro Suzuki <ryoichiro.suzuki@gmail.com>,
    Jesse Alama <jesse.alama@gmail.com>
 
-Subject: GNU Parallel 20150522 ('Nepal') released
+Subject: GNU Parallel 20150622 ('<<>>') released
 
-GNU Parallel 20150522 ('Nepal') has been released. It is available for download at: http://ftp.gnu.org/gnu/parallel/
+GNU Parallel 20150622 ('<<>>') has been released. It is available for download at: http://ftp.gnu.org/gnu/parallel/
 
 Haiku of the month:
 
-  Many jobs to run.
+  Programs very slow.
-  Idle cores is not your thing.
+  Multiple can run at once.
   Use GNU Parallel.
       -- Ole Tange
 
 New in this release:
 
-* Security: The security issue for --sshlogin + --fifo/--cat has been fixed. Thereby all issues with http://lists.gnu.org/archive/html/parallel/2015-04/msg00045.html have been fixed.
+* <<afventer opdatering>> CIDER: a pipeline for detecting waves of coordinated transcriptional regulation in gene expression time-course data http://biorxiv.org/content/biorxiv/early/2015/03/17/012518.full.pdf
-
-* Security: After further security analysis the issue fixed in 20150422 also fixed the problem for --tmux.
 
 * <<Har angiveligt submittet ny version - afventer opdatering>> GNU Parallel was used (unfortunately without citation) in: MUGBAS: a species free gene-based programme suite for post-GWAS analysis http://www.ncbi.nlm.nih.gov/pubmed/25765345
 
-taxator-tk http://algbio.cs.uni-duesseldorf.de/webapps/wa-download/ (check it)
-
 * <<Afventer updateret publisering>> GNU Parallel was used in: Large Scale Author Name Disambiguation in Digital Libraries http://ieeexplore.ieee.org/xpl/abstractReferences.jsp?tp=&arnumber=7004487&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D7004487
 
-* << Update forventet juni >> GNU Parallel was used in: SISRS: Site Identification from Short Read Sequences https://github.com/rachelss/SISRS/
-
 * <<kontaktet 2015-05-09>> GNU Parallel was used (unfortunately with wrong citation) in: TADSim: Discrete Event-Based Performance Prediction for Temperature-Accelerated Dynamics http://vruehle.de/publications/2015c.pdf
 
-* GNU Parallel was cited in: CIDER: a pipeline for detecting waves of coordinated transcriptional regulation in gene expression time-course data http://biorxiv.org/content/biorxiv/early/2015/03/17/012518.full.pdf
+* taxator-tk http://algbio.cs.uni-duesseldorf.de/webapps/wa-download/ (check it)
 
-* GNU Parallel was cited in: Building Genomic Analysis Pipelines in a Hackathon Setting with Bioinformatician Teams: DNA-seq, Epigenomics, Metagenomics and RNA-seq http://biorxiv.org/content/biorxiv/early/2015/05/05/018085.full.pdf
+* << Update forventet juni Rachel har lige svaret >> GNU Parallel was used in: SISRS: Site Identification from Short Read Sequences https://github.com/rachelss/SISRS/
-
-* GNU Parallel was cited in: Toward Enhanced Metadata Quality of Large-Scale Digital Libraries: Estimating Volume Time Range https://www.ideals.illinois.edu/bitstream/handle/2142/73656/186_ready.pdf
-
-* GNU Parallel was cited in: Sequencing the cap-snatching repertoire of H1N1 influenza provides insight into the mechanism of viral transcription initiation http://nar.oxfordjournals.org/content/early/2015/04/20/nar.gkv333.full.pdf
-
-* GNU Parallel was cited in: Genome assembly using Nanopore-guided long and error-free DNA reads http://www.biomedcentral.com/content/pdf/s12864-015-1519-z.pdf
-
-* GNU Parallel was cited in: Contrasting regional architectures of schizophrenia and other complex diseases using fast variance components analysis http://biorxiv.org/content/biorxiv/early/2015/03/13/016527.full.pdf
-
-* GNU Parallel was used (unfortunately with wrong citation) in: Comparing the CarbonTracker and TM5-4DVar data assimilation systems for CO2 surface flux inversions http://www.atmos-chem-phys-discuss.net/15/8883/2015/acpd-15-8883-2015-discussion.html
-
-* GNU Parallel was used in: Gene Set Omic Analysis (GSOA) method https://bitbucket.org/srp33/gsoa
-
-* A Quick and Neat :) Orchestrator using GNU Parallel http://www.elsotanillo.net/2015/05/a-quick-and-neat-orchestrator-using-gnu-parallel/
-
-* Execute commands on multiple computers using GNU Parallel (setting up a cluster on the cheap) https://spectraldifferences.wordpress.com/2015/04/26/execute-commands-on-multiple-computers-using-gnu-parallel-setting-up-a-cluster-on-the-cheap/
-
-* Functions and GNU parallel for effective cluster load management http://genomespot.blogspot.dk/2015/04/functions-and-gnu-parallel-for.html
-
-* Use parallel processing to save time importing databases http://drupalsun.com/node/41854
-
-* Run multiple ssh commands in parallel with GNU Parallel http://www.ameir.net/blog/archives/380-run-multiple-ssh-commands-in-parallel-with-gnu-parallel.html
-
-* Importing huge databases faster https://www.lullabot.com/blog/article/importing-huge-databases-faster
-
-* Run multiple ssh commands in parallel with GNU Parallel https://www.ameir.net/blog/archives/380-run-multiple-ssh-commands-in-parallel-with-gnu-parallel.html/comment-page-1
-
-* Parallel? Gnu parallel! https://debian.pro/1834
 
 * Bug fixes and man page updates.
 

src/parallel

@@ -1075,7 +1075,7 @@ sub parse_options {
 
 sub init_globals {
     # Defaults:
-    $Global::version = 20150521;
+    $Global::version = 20150523;
     $Global::progname = 'parallel';
     $Global::infinity = 2**31;
     $Global::debug = 0;
@@ -3141,60 +3141,48 @@ sub __USAGE__ {}
 
 sub killall {
     # Kill all jobs
-    # Send all jobs TERM
+    # Send all (grand*)children TERM,TERM,KILL
-    # Wait
+    # Wait up to 200 ms between each
-    # Send all jobs TERM
-    # Wait
-    # Send all jobs KILL
-    # Send all (grand*)children KILL
 
     $Global::start_no_new_jobs ||= 1;
+    $Global::killall = 1;
     # pids of the all children and (grand*)children
     # before we start the blood bath
     my @family_pids = family_pids(keys %Global::running);
-    # Send jobs TERM
+    my @pids = @family_pids;
-    ::debug("kill","TERM ", join(' ',keys %Global::running),"\n");
+    # Send jobs TERM,TERM,KILL
-    kill "TERM", keys %Global::running;
+    my @term_seq = ("TERM",200,"TERM",200,"KILL",200);
-    # Wait up to 200 ms
+    while(@term_seq) {
+	@pids = kill_sleep(shift @term_seq, shift @term_seq, @pids);
+    }
+    if(kill 0, @family_pids) {
+	my @fpids = grep { kill( 0, $_) } @family_pids;
+	::die_bug("Pids still alive @fpids (Parents: @pids)");
+    }
+}
+
+sub kill_sleep {
+    my ($signal, $sleep_max, @pids) = @_;
+    ::debug("kill","kill_sleep $signal @pids\n");
+    kill $signal, @pids;
     my $sleepsum = 0;
     my $sleep = 0.001;
-    for (; kill(0, keys %Global::running) and $sleepsum < 200;
+    for (; kill(0, @pids) and $sleepsum < $sleep_max;
 	 $sleepsum += $sleep) {
-	# This can change %Global::running
+	if($Global::killall and (waitpid(-1, &WNOHANG) > 0)
-	    ::debug("kill","Slept $sleepsum\n");
+	   # Killall => dont run reaper
-	if((my $stiff = waitpid(-1, &WNOHANG)) > 0) {
+	   or
-	    ::debug("kill","reaped $stiff\n");
+	   reaper()) {
-
+	    # Remove (grand)*children that are dead
-	    $Global::total_running--;
+	    @pids = grep { kill( 0, $_) } @pids;
-	    delete $Global::running{$stiff};
 	    $sleep = $sleep/2+0.001;
+	} else {
+	    $sleep *= 1.1;
+	    ::usleep($sleep);
 	}
-	$sleep *= 1.1;
-	::usleep($sleep);
     }
-    # Send jobs TERM (again)
+    @pids = grep { kill( 0, $_) } @pids;
-    ::debug("kill","TERM ", join(' ',keys %Global::running),"\n");
+    return @pids;
-    kill "TERM", keys %Global::running;
-    # Wait up to 200 ms
-     $sleepsum = 0;
-    $sleep = 0.001;
-    for (; kill(0, keys %Global::running) and $sleepsum < 200;
-	 $sleepsum += $sleep) {
-	# This can change %Global::running
-	if((my $stiff = waitpid(-1, &WNOHANG)) > 0) {
-	    $Global::total_running--;
-	    delete $Global::running{$stiff};
-	    $sleep = $sleep/2+0.001;
-	}
-	$sleep *= 1.1;
-	::usleep($sleep);
-    }
-    # Send jobs KILL
-    ::debug("kill","KILL ", join(' ',keys %Global::running),"\n");
-    kill "KILL", keys %Global::running;
-    # Send all (grand*)children KILL (if there are any left)
-    ::debug("kill","KILL @family_pids\n");
-    kill "KILL", @family_pids;
 }
 
 sub family_pids {
@@ -3230,10 +3218,11 @@ sub wait_and_exit {
     my $error = shift;
     unlink keys %Global::unlink;
     if($error) {
-	# Kill all without printing
+	# Kill all jobs without printing
 	killall();
     }
     for (keys %Global::unkilled_children) {
+	# Kill any (non-jobs) children
         kill 9, $_;
         waitpid($_,0);
         delete $Global::unkilled_children{$_};
@@ -4668,7 +4657,7 @@ sub compute_number_of_processes {
 	    # The child takes one process slot
 	    # It will be killed later
 	    $SIG{'TERM'} = $Global::original_sig{'TERM'};
-            sleep 10000000;
+            sleep 10101010;
             exit(0);
 	} else {
 	    # Failed to spawn

src/parallel.pod

@@ -627,7 +627,7 @@ If I<eof-str> is omitted, there is no end of file string.  If neither
 B<-E> nor B<-e> is used, no end of file string is used.
 
 
-=item B<--env> I<var> (beta testing)
+=item B<--env> I<var>
 
 Copy environment variable I<var>. This will copy I<var> to the
 environment that the command is run in. This is especially useful for
@@ -1173,7 +1173,7 @@ performance is important use B<--pipepart>.
 See also: B<--recstart>, B<--recend>, B<--fifo>, B<--cat>, B<--pipepart>.
 
 
-=item B<--pipepart> (beta testing)
+=item B<--pipepart>
 
 Pipe parts of a physical file. B<--pipepart> works similar to
 B<--pipe>, but is much faster. It has a few limitations:

src/parallel_design.pod

@@ -152,7 +152,7 @@ system "mkfifo", $f;
 $pid = fork || exec $s, "-c", $c;
 open($o,">",$f) || die $!;
 while(sysread(STDIN,$buf,32768)){
-syswrite $o, $buf;
+  syswrite $o, $buf;
 }
 close $o;
 waitpid $pid,0;
@@ -387,9 +387,9 @@ shell that runs the function definition followed by the actual
 command.
 
 B<env_parallel> (mentioned in the man page) copies the full current
-environment into the environment variable
+environment into the environment variable B<PARALLEL_ENV>. This
-B<parallel_bash_environment>. This variable is picked up by GNU
+variable is picked up by GNU B<parallel> and used to create the Perl
-B<parallel> and used to create the Perl script mentioned above.
+script mentioned above.
 
 
 =head2 Base64 encoded bzip2
@@ -666,6 +666,46 @@ As remote calculation of load can be slow, a process is spawned to run
 B<ps> and put the result in a file, which is then used next time.
 
 
+=head2 Killing jobs
+
+B<--memfree>, B<--halt> and when GNU B<parallel> meets a condition
+from which it cannot recover, jobs are killed. This is done by finding
+the (grand)*children of the jobs and killing those processes.
+
+More specifically GNU B<parallel> maintains a list of processes to be
+killed, sends a signal to all processes (first round this is a TERM).
+It weeds out the processes that exited from the list then waits a
+while and weeds out again. It does that until all processes are dead
+or 200 ms passed. Then it does another round with TERM, and finally a
+round with KILL.
+
+  pids = family_pids(jobs)
+  for signal in TERM, TERM, KILL:
+    for pid in pids:
+      kill signal, pid
+    while kill 0, pids and slept < 200 ms:
+      sleep sleeptime
+      pids = kill 0, pids
+      slept += sleeptime
+      sleeptime = sleeptime * 1.1
+
+By doing so there is a tiny risk, that GNU B<parallel> will kill
+processes that are not started from GNU B<parallel>. It, however,
+requires all of these to be true:
+
+* Process A is sent a signal
+* It dies during a I<sleep sleeptime> cycle
+* A new process B is spawned (by an unrelated process)
+* This is done during the same I<sleep sleeptime> cycle
+* B is owned by the same user
+* B reuses the pid of the A
+
+It is considered unlikely to ever happen due to:
+
+* The longest I<sleep sleeptime> sleeps is 10 ms
+* Re-use of a dead pid rarely happens within a few seconds
+
+
 =head1 Ideas for new design
 
 =head2 Multiple processes working together

testsuite/Makefile

@@ -7,7 +7,7 @@ testsuite: 3
 	make stopvm
 
 1: ../src/parallel tests-to-run/* wanted-results/* prereqlocal startdb prereqremote
-	TRIES=1 time bash Start.sh - mem || true
+	TRIES=1 time bash Start.sh - 'mem|polarhome' || true
 	touch ~/.parallel/will-cite
 	make stopvm
 

testsuite/Start.sh

@@ -41,7 +41,7 @@ echo forever pstree -lp $$ >/tmp/monitor
 chmod 755 /tmp/monitor
 date
 mkdir -p actual-results
-ls -t tests-to-run/*${1}*.sh | grep -v ${2} |
+ls -t tests-to-run/*${1}*.sh | egrep -v "${2}" |
   stdout parallel --tty -tj1 run_test | tee testsuite.log
 # If testsuite.log contains @@ then there is a diff
 if grep -q '@@' testsuite.log ; then

testsuite/tests-to-run/parallel-local-3s.sh

@@ -76,8 +76,8 @@ echo '**'
 
 testhalt() { 
   echo '### testhalt --halt '$1; 
-  (yes 0 | head -n 10; seq 10) | stdout parallel -kj4 --halt $1 'sleep {= $_*=0.3 =}; exit {}'; echo $?; 
+  (yes 0 | head -n 10; seq 10) | stdout parallel -kj4 --halt $1 'sleep {= $_=$_*0.3+1 =}; exit {}'; echo $?; 
-  (seq 10; yes 0 | head -n 10) | stdout parallel -kj4 --halt $1 'sleep {= $_*=0.3 =}; exit {}'; echo $?; 
+  (seq 10; yes 0 | head -n 10) | stdout parallel -kj4 --halt $1 'sleep {= $_=$_*0.3+1 =}; exit {}'; echo $?; 
 }; 
 export -f testhalt; 
   parallel -kj0 testhalt ::: now,fail=0 now,fail=1 now,fail=2 now,fail=30%  now,fail=70% 

testsuite/wanted-results/parallel-local-3s

@@ -157,498 +157,515 @@ parallel: This job succeeded:
 perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ not\ shift 1
 echo '**'
 **
-testhalt() {   echo '### testhalt --halt '$1;   (yes 0 | head -n 10; seq 10) | stdout parallel -kj4 --halt $1 'sleep {= $_*=0.3 =}; exit {}'; echo $?;   (seq 10; yes 0 | head -n 10) | stdout parallel -kj4 --halt $1 'sleep {= $_*=0.3 =}; exit {}'; echo $?; }; export -f testhalt;   parallel -kj0 testhalt ::: now,fail=0 now,fail=1 now,fail=2 now,fail=30%  now,fail=70%     soon,fail=0 soon,fail=1 soon,fail=2 soon,fail=30% soon,fail=70%     now,success=0 now,success=1 now,success=2 now,success=30% now,success=70%     soon,success=0 soon,success=1 soon,success=2 soon,success=30% now,success=70%
+testhalt() {   echo '### testhalt --halt '$1;   (yes 0 | head -n 10; seq 10) | stdout parallel -kj4 --halt $1 'sleep {= $_=$_*0.3+1 =}; exit {}'; echo $?;   (seq 10; yes 0 | head -n 10) | stdout parallel -kj4 --halt $1 'sleep {= $_=$_*0.3+1 =}; exit {}'; echo $?; }; export -f testhalt;   parallel -kj0 testhalt ::: now,fail=0 now,fail=1 now,fail=2 now,fail=30%  now,fail=70%     soon,fail=0 soon,fail=1 soon,fail=2 soon,fail=30% soon,fail=70%     now,success=0 now,success=1 now,success=2 now,success=30% now,success=70%     soon,success=0 soon,success=1 soon,success=2 soon,success=30% now,success=70%
 ### testhalt --halt now,fail=0
 parallel: This job failed:
-sleep 0.3; exit 1
+sleep 1.3; exit 1
 0
 parallel: This job failed:
-sleep 0.3; exit 1
+sleep 1.3; exit 1
 0
 ### testhalt --halt now,fail=1
 parallel: This job failed:
-sleep 0.3; exit 1
+sleep 1.3; exit 1
 1
 parallel: This job failed:
-sleep 0.3; exit 1
+sleep 1.3; exit 1
 1
 ### testhalt --halt now,fail=2
 parallel: This job failed:
-sleep 0.3; exit 1
+sleep 1.3; exit 1
 parallel: This job failed:
-sleep 0.6; exit 2
+sleep 1.6; exit 2
 2
 parallel: This job failed:
-sleep 0.3; exit 1
+sleep 1.3; exit 1
 parallel: This job failed:
-sleep 0.6; exit 2
+sleep 1.6; exit 2
 2
 ### testhalt --halt now,fail=30%
 parallel: This job failed:
-sleep 0.3; exit 1
+sleep 1.3; exit 1
 parallel: This job failed:
-sleep 0.6; exit 2
+sleep 1.6; exit 2
 parallel: This job failed:
-sleep 0.9; exit 3
+sleep 1.9; exit 3
 parallel: This job failed:
-sleep 1.2; exit 4
+sleep 2.2; exit 4
 parallel: This job failed:
-sleep 1.5; exit 5
+sleep 2.5; exit 5
 parallel: This job failed:
-sleep 1.8; exit 6
+sleep 2.8; exit 6
 30
 parallel: This job failed:
-sleep 0.3; exit 1
+sleep 1.3; exit 1
 parallel: This job failed:
-sleep 0.6; exit 2
+sleep 1.6; exit 2
 parallel: This job failed:
-sleep 0.9; exit 3
+sleep 1.9; exit 3
 parallel: This job failed:
-sleep 1.2; exit 4
+sleep 2.2; exit 4
 parallel: This job failed:
-sleep 1.5; exit 5
+sleep 2.5; exit 5
 parallel: This job failed:
-sleep 1.8; exit 6
+sleep 2.8; exit 6
 30
 ### testhalt --halt now,fail=70%
 parallel: This job failed:
-sleep 0.3; exit 1
+sleep 1.3; exit 1
 parallel: This job failed:
-sleep 0.6; exit 2
+sleep 1.6; exit 2
 parallel: This job failed:
-sleep 0.9; exit 3
+sleep 1.9; exit 3
 parallel: This job failed:
-sleep 1.2; exit 4
+sleep 2.2; exit 4
 parallel: This job failed:
-sleep 1.5; exit 5
+sleep 2.5; exit 5
 parallel: This job failed:
-sleep 1.8; exit 6
+sleep 2.8; exit 6
 parallel: This job failed:
-sleep 2.1; exit 7
+sleep 3.1; exit 7
 parallel: This job failed:
-sleep 2.4; exit 8
+sleep 3.4; exit 8
 parallel: This job failed:
-sleep 2.7; exit 9
+sleep 3.7; exit 9
 parallel: This job failed:
-sleep 3; exit 10
+sleep 4; exit 10
 50
 parallel: This job failed:
-sleep 0.3; exit 1
+sleep 1.3; exit 1
 parallel: This job failed:
-sleep 0.6; exit 2
+sleep 1.6; exit 2
 parallel: This job failed:
-sleep 0.9; exit 3
+sleep 1.9; exit 3
 parallel: This job failed:
-sleep 1.2; exit 4
+sleep 2.2; exit 4
 parallel: This job failed:
-sleep 1.5; exit 5
+sleep 2.5; exit 5
 parallel: This job failed:
-sleep 1.8; exit 6
+sleep 2.8; exit 6
 parallel: This job failed:
-sleep 2.1; exit 7
+sleep 3.1; exit 7
 parallel: This job failed:
-sleep 2.4; exit 8
+sleep 3.4; exit 8
 parallel: This job failed:
-sleep 2.7; exit 9
+sleep 3.7; exit 9
 parallel: This job failed:
-sleep 3; exit 10
+sleep 4; exit 10
 50
 ### testhalt --halt soon,fail=0
 parallel: This job failed:
-sleep 0.3; exit 1
+sleep 1.3; exit 1
 parallel: Starting no more jobs. Waiting for 3 jobs to finish.
 parallel: This job failed:
-sleep 0.6; exit 2
+sleep 1.6; exit 2
 parallel: Starting no more jobs. Waiting for 2 jobs to finish.
 parallel: This job failed:
-sleep 0.9; exit 3
+sleep 1.9; exit 3
 parallel: Starting no more jobs. Waiting for 1 jobs to finish.
 parallel: This job failed:
-sleep 1.2; exit 4
+sleep 2.2; exit 4
 0
 parallel: This job failed:
-sleep 0.3; exit 1
+sleep 1.3; exit 1
 parallel: Starting no more jobs. Waiting for 3 jobs to finish.
 parallel: This job failed:
-sleep 0.6; exit 2
+sleep 1.6; exit 2
 parallel: Starting no more jobs. Waiting for 2 jobs to finish.
 parallel: This job failed:
-sleep 0.9; exit 3
+sleep 1.9; exit 3
 parallel: Starting no more jobs. Waiting for 1 jobs to finish.
 parallel: This job failed:
-sleep 1.2; exit 4
+sleep 2.2; exit 4
 0
 ### testhalt --halt soon,fail=1
 parallel: This job failed:
-sleep 0.3; exit 1
+sleep 1.3; exit 1
 parallel: Starting no more jobs. Waiting for 3 jobs to finish.
 parallel: This job failed:
-sleep 0.6; exit 2
+sleep 1.6; exit 2
 parallel: Starting no more jobs. Waiting for 2 jobs to finish.
 parallel: This job failed:
-sleep 0.9; exit 3
+sleep 1.9; exit 3
 parallel: Starting no more jobs. Waiting for 1 jobs to finish.
 parallel: This job failed:
-sleep 1.2; exit 4
+sleep 2.2; exit 4
 1
 parallel: This job failed:
-sleep 0.3; exit 1
+sleep 1.3; exit 1
 parallel: Starting no more jobs. Waiting for 3 jobs to finish.
 parallel: This job failed:
-sleep 0.6; exit 2
+sleep 1.6; exit 2
 parallel: Starting no more jobs. Waiting for 2 jobs to finish.
 parallel: This job failed:
-sleep 0.9; exit 3
+sleep 1.9; exit 3
 parallel: Starting no more jobs. Waiting for 1 jobs to finish.
 parallel: This job failed:
-sleep 1.2; exit 4
+sleep 2.2; exit 4
 1
 ### testhalt --halt soon,fail=2
 parallel: This job failed:
-sleep 0.3; exit 1
+sleep 1.3; exit 1
 parallel: This job failed:
-sleep 0.6; exit 2
+sleep 1.6; exit 2
 parallel: Starting no more jobs. Waiting for 3 jobs to finish.
 parallel: This job failed:
-sleep 0.9; exit 3
+sleep 1.9; exit 3
 parallel: Starting no more jobs. Waiting for 2 jobs to finish.
 parallel: This job failed:
-sleep 1.2; exit 4
+sleep 2.2; exit 4
 parallel: Starting no more jobs. Waiting for 1 jobs to finish.
 parallel: This job failed:
-sleep 1.5; exit 5
+sleep 2.5; exit 5
 2
 parallel: This job failed:
-sleep 0.3; exit 1
+sleep 1.3; exit 1
 parallel: This job failed:
-sleep 0.6; exit 2
+sleep 1.6; exit 2
 parallel: Starting no more jobs. Waiting for 3 jobs to finish.
 parallel: This job failed:
-sleep 0.9; exit 3
+sleep 1.9; exit 3
 parallel: Starting no more jobs. Waiting for 2 jobs to finish.
 parallel: This job failed:
-sleep 1.2; exit 4
+sleep 2.2; exit 4
 parallel: Starting no more jobs. Waiting for 1 jobs to finish.
 parallel: This job failed:
-sleep 1.5; exit 5
+sleep 2.5; exit 5
 2
 ### testhalt --halt soon,fail=30%
 parallel: This job failed:
-sleep 0.3; exit 1
+sleep 1.3; exit 1
 parallel: This job failed:
-sleep 0.6; exit 2
+sleep 1.6; exit 2
 parallel: This job failed:
-sleep 0.9; exit 3
+sleep 1.9; exit 3
 parallel: This job failed:
-sleep 1.2; exit 4
+sleep 2.2; exit 4
 parallel: This job failed:
-sleep 1.5; exit 5
+sleep 2.5; exit 5
 parallel: This job failed:
-sleep 1.8; exit 6
+sleep 2.8; exit 6
 parallel: Starting no more jobs. Waiting for 3 jobs to finish.
 parallel: This job failed:
-sleep 2.1; exit 7
+sleep 3.1; exit 7
 parallel: Starting no more jobs. Waiting for 2 jobs to finish.
 parallel: This job failed:
-sleep 2.4; exit 8
+sleep 3.4; exit 8
 parallel: Starting no more jobs. Waiting for 1 jobs to finish.
 parallel: This job failed:
-sleep 2.7; exit 9
+sleep 3.7; exit 9
 30
 parallel: This job failed:
-sleep 0.3; exit 1
+sleep 1.3; exit 1
 parallel: This job failed:
-sleep 0.6; exit 2
+sleep 1.6; exit 2
 parallel: This job failed:
-sleep 0.9; exit 3
+sleep 1.9; exit 3
 parallel: This job failed:
-sleep 1.2; exit 4
+sleep 2.2; exit 4
 parallel: This job failed:
-sleep 1.5; exit 5
+sleep 2.5; exit 5
 parallel: This job failed:
-sleep 1.8; exit 6
+sleep 2.8; exit 6
 parallel: Starting no more jobs. Waiting for 3 jobs to finish.
 parallel: This job failed:
-sleep 2.1; exit 7
+sleep 3.1; exit 7
 parallel: Starting no more jobs. Waiting for 2 jobs to finish.
 parallel: This job failed:
-sleep 2.4; exit 8
+sleep 3.4; exit 8
 parallel: Starting no more jobs. Waiting for 1 jobs to finish.
 parallel: This job failed:
-sleep 2.7; exit 9
+sleep 3.7; exit 9
 30
 ### testhalt --halt soon,fail=70%
 parallel: This job failed:
-sleep 0.3; exit 1
+sleep 1.3; exit 1
 parallel: This job failed:
-sleep 0.6; exit 2
+sleep 1.6; exit 2
 parallel: This job failed:
-sleep 0.9; exit 3
+sleep 1.9; exit 3
 parallel: This job failed:
-sleep 1.2; exit 4
+sleep 2.2; exit 4
 parallel: This job failed:
-sleep 1.5; exit 5
+sleep 2.5; exit 5
 parallel: This job failed:
-sleep 1.8; exit 6
+sleep 2.8; exit 6
 parallel: This job failed:
-sleep 2.1; exit 7
+sleep 3.1; exit 7
 parallel: This job failed:
-sleep 2.4; exit 8
+sleep 3.4; exit 8
 parallel: This job failed:
-sleep 2.7; exit 9
+sleep 3.7; exit 9
 parallel: This job failed:
-sleep 3; exit 10
+sleep 4; exit 10
 50
 parallel: This job failed:
-sleep 0.3; exit 1
+sleep 1.3; exit 1
 parallel: This job failed:
-sleep 0.6; exit 2
+sleep 1.6; exit 2
 parallel: This job failed:
-sleep 0.9; exit 3
+sleep 1.9; exit 3
 parallel: This job failed:
-sleep 1.2; exit 4
+sleep 2.2; exit 4
 parallel: This job failed:
-sleep 1.5; exit 5
+sleep 2.5; exit 5
 parallel: This job failed:
-sleep 1.8; exit 6
+sleep 2.8; exit 6
 parallel: This job failed:
-sleep 2.1; exit 7
+sleep 3.1; exit 7
 parallel: This job failed:
-sleep 2.4; exit 8
+sleep 3.4; exit 8
 parallel: This job failed:
-sleep 2.7; exit 9
+sleep 3.7; exit 9
 parallel: This job failed:
-sleep 3; exit 10
+sleep 4; exit 10
 50
 ### testhalt --halt now,success=0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 0
 ### testhalt --halt now,success=1
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 0
 ### testhalt --halt now,success=2
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 0
 ### testhalt --halt now,success=30%
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 0
 ### testhalt --halt now,success=70%
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 50
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 50
 ### testhalt --halt soon,success=0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: Starting no more jobs. Waiting for 3 jobs to finish.
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: Starting no more jobs. Waiting for 2 jobs to finish.
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: Starting no more jobs. Waiting for 1 jobs to finish.
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: Starting no more jobs. Waiting for 3 jobs to finish.
+parallel: This job succeeded:
+sleep 1; exit 0
+parallel: Starting no more jobs. Waiting for 2 jobs to finish.
 0
 ### testhalt --halt soon,success=1
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: Starting no more jobs. Waiting for 3 jobs to finish.
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: Starting no more jobs. Waiting for 2 jobs to finish.
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: Starting no more jobs. Waiting for 1 jobs to finish.
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: Starting no more jobs. Waiting for 3 jobs to finish.
+parallel: This job succeeded:
+sleep 1; exit 0
+parallel: Starting no more jobs. Waiting for 2 jobs to finish.
 0
 ### testhalt --halt soon,success=2
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: Starting no more jobs. Waiting for 3 jobs to finish.
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: Starting no more jobs. Waiting for 2 jobs to finish.
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: Starting no more jobs. Waiting for 1 jobs to finish.
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: Starting no more jobs. Waiting for 3 jobs to finish.
+parallel: This job succeeded:
+sleep 1; exit 0
+parallel: Starting no more jobs. Waiting for 2 jobs to finish.
 0
 ### testhalt --halt soon,success=30%
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: Starting no more jobs. Waiting for 3 jobs to finish.
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: Starting no more jobs. Waiting for 2 jobs to finish.
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: Starting no more jobs. Waiting for 1 jobs to finish.
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: Starting no more jobs. Waiting for 3 jobs to finish.
+parallel: This job succeeded:
+sleep 1; exit 0
+parallel: Starting no more jobs. Waiting for 2 jobs to finish.
+parallel: This job succeeded:
+sleep 1; exit 0
+parallel: Starting no more jobs. Waiting for 1 jobs to finish.
+parallel: This job succeeded:
+sleep 1; exit 0
 0
 ### testhalt --halt now,success=70%
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 50
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 parallel: This job succeeded:
-sleep 0; exit 0
+sleep 1; exit 0
 50
 echo '**'
 **