tange/parallel
1
0
Fork 0
mirror of https://git.savannah.gnu.org/git/parallel.git synced 2024-11-22 14:07:55 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge branch 'master' of ssh://git.sv.gnu.org/srv/git/parallel into stable

Browse source 
Conflicts:
	doc/release_new_version
	src/parallel
	testsuite/wanted-results/parallel-tutorial
This commit is contained in:
Ole Tange 2015-05-22 00:41:02 +02:00
parent 39ff23c10d 03a69b38ce
commit 1b0c5d4ca7
29 changed files with 1193 additions and 198 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
CREDITS
View file

@ -1,3 +1,5 @@
Rasmus Villemoes: Code snips for signal processing.
Martin d'Anjou: Code snips for signal processing.
rici@stackoverflow.com: Documentation on exporting arrays using --env.
Malcolm Cook: The idea to use a general perl expression as replacement strings.
Ævar Arnfjörð Bjarmason: Reading the whole code.

2
Makefile.am
View file

@ -59,4 +59,4 @@ upload:
pushd; \
sudo cp /usr/local/bin/parallel /usr/local/bin/parallel-$(YYYYMMDD)
EXTRA_DIST = CITATION
EXTRA_DIST = CITATION CREDITS

2
Makefile.in
View file

@ -263,7 +263,7 @@ top_build_prefix = @top_build_prefix@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
SUBDIRS = src
EXTRA_DIST = CITATION
EXTRA_DIST = CITATION CREDITS
all: config.h
$(MAKE) $(AM_MAKEFLAGS) all-recursive

176
src/parallel.pod
View file

@ -84,7 +84,21 @@ If it is a Bash function you need to B<export -f> the function
first. To use aliases copy the full environment as described under
B<--env> and use B<env_parallel> instead of B<parallel>.
If it is a Ksh function you can encode the function in a variable:
foo() {
echo $*;
}
export fun=`typeset -f foo`; parallel 'eval "$fun";' foo ::: works
To export all functions and make them available when running remote:
export fun=`typeset -f`; parallel --env fun 'eval "$fun";' foo ::: works
=cut
# ssh ksh@lo 'foo() { echo $* ; }; export fun="`typeset -f`"; parallel -S ksh@lo --env fun "eval \"\$fun\";"foo ::: works'
# ssh zsh@lo 'foo() { echo $* ; }; export fun="`typeset -f`"; parallel -S zsh@lo --env fun "eval \"\$fun\";"foo ::: works'
# If it is a zsh function you will need to use this helper function
# B<exportf> to export and to set $PARALLEL_SHELL to bash:
#
@ -422,9 +436,19 @@ Implies B<--semaphore>.
=item B<--bibtex>
Print the BibTeX entry for GNU B<parallel> and disable citation
Print the BibTeX entry for GNU B<parallel> and silence citation
notice.
If it is impossible for you to run B<--bibtex> you can use
B<--will-cite>.
If you use B<--will-cite> in scripts to be run by others you are
making it harder for others to see the citation notice. The
development of GNU B<parallel> is indirectly financed through
citations, so if users do not know they should cite then that makes it
harder to finance development. However, if you pay 10000 EUR, you
should feel free to use B<--will-cite>.
=item B<--block> I<size>
@ -451,6 +475,8 @@ will give data to the program on stdin (standard input). With B<--cat>
GNU B<parallel> will create a temporary file with the name in {}, so
you can do: B<parallel --pipe --cat wc {}>.
Implies B<--pipe> unless B<--pipepart> is used.
See also B<--fifo>.
@ -494,9 +520,9 @@ Compress temporary files. If the output is big and very compressible
this will take up less disk space in $TMPDIR and possibly be faster
due to less disk I/O.
GNU B<parallel> will try B<lzop>, B<pigz>, B<gzip>, B<pbzip2>,
B<plzip>, B<bzip2>, B<lzma>, B<lzip>, B<xz> in that order, and use the
first available.
GNU B<parallel> will try B<lz4>, B<pigz>, B<lzop>, B<plzip>,
B<pbzip2>, B<pxz>, B<gzip>, B<lzma>, B<xz>, B<bzip2>, B<lzip> in that
order, and use the first available.
=item B<--compress-program> I<prg>
@ -644,6 +670,8 @@ with the name in {}, so you can do: B<parallel --pipe --fifo wc {}>.
Beware: If data is not read from the fifo, the job will block forever.
Implies B<--pipe> unless B<--pipepart> is used.
See also B<--cat>.
@ -687,46 +715,84 @@ See also: B<--line-buffer> B<--ungroup>
Print a summary of the options to GNU B<parallel> and exit.
=item B<--halt-on-error> I<val>
=item B<--halt-on-error> I<val> (alpha testing)
=item B<--halt> I<val>
=item B<--halt> I<val> (alpha testing)
How should GNU B<parallel> terminate?
When should GNU B<parallel> terminate? In some situations it makes no
sense to run all jobs. GNU B<parallel> should simply give up as soon
as a condition is met.
I<val> defaults to B<never>, which runs all jobs no matter what.
I<val> can also take on the form of I<when>,I<why>.
I<when> can be 'now' which means kill all running jobs and halt
immediately, or it can be 'soon' which means wait for all running jobs
to complete, but start no new jobs.
I<why> can be 'fail=X', 'fail=Y%', 'success=X', or 'success=Y%' where
X is the number of jobs that has to fail or succeed before halting,
and Y is the percentage of jobs that has to fail or succeed before
halting.
Example:
=over 23
=item Z<> --halt now,fail=1
exit when the first job fails. Kill running jobs.
=item Z<> --halt soon,fail=3
exit when 3 jobs fail, but wait for running jobs to complete.
=item Z<> --halt soon,fail=3%
exit when 3% of the jobs have failed, but wait for running jobs to complete.
=item Z<> --halt now,success=1
exit when a job succeeds. Kill running jobs.
=item Z<> --halt soon,success=3
exit when 3 jobs succeeds, but wait for running jobs to complete.
=item Z<> --halt now,success=3%
exit when 3% of the jobs have succeeded. Kill running jobs.
=back
For backwards compability these also work:
=over 7
=item Z<>0
Do not halt if a job fails. Exit status will be the number of jobs
failed. This is the default.
never
=item Z<>1
Do not start new jobs if a job fails, but complete the running jobs
including cleanup. The exit status will be the exit status from the
last failing job.
soon,fail=1
=item Z<>2
Kill off all jobs immediately and exit without cleanup. The exit
status will be the exit status from the failing job.
now,fail=1
=item Z<>-1
Do not start new jobs if a job succeeds, but complete the running jobs
including cleanup. The exit status will be the exit status from the
last failing job if any.
soon,success=1
=item Z<>-2
Kill off all jobs immediately and exit without cleanup. The exit
status will be 0.
now,success=1
=item Z<>1-99%
If I<val>% of the jobs fail and minimum 3: Do not start new jobs, but
complete the running jobs including cleanup. The exit status will be
the exit status from the last failing job.
soon,fail=1-99%
=back
@ -809,10 +875,11 @@ See also B<--resume> B<--resume-failed>.
=item B<-P> I<N>
Number of jobslots. Run up to N jobs in parallel. 0 means as many as
possible. Default is 100% which will run one job per CPU core.
Number of jobslots on each machine. Run up to N jobs in parallel. 0
means as many as possible. Default is 100% which will run one job per
CPU core on each machine.
If B<--semaphore> is set default is 1 thus making a mutex.
If B<--semaphore> is set, the default is 1 thus making a mutex.
=item B<--jobs> I<+N>
@ -917,6 +984,8 @@ Implies B<-X> unless B<-m>, B<--xargs>, or B<--pipe> is set.
=item B<--line-buffer>
=item B<--lb>
Buffer output on line basis. B<--group> will keep the output together
for a whole job. B<--ungroup> allows output to mixup with half a line
coming from one job and half a line coming from another
@ -1065,6 +1134,9 @@ defaults to '\n'. To have no record separator use B<--recend "">.
B<--files> is often used with B<--pipe>.
B<--pipe> maxes out at around 1 GB/s input, and 100 MB/s output. If
performance is important use B<--pipepart>.
See also: B<--recstart>, B<--recend>, B<--fifo>, B<--cat>, B<--pipepart>.
@ -1077,7 +1149,8 @@ B<--pipe>, but is much faster. It has a few limitations:
=item Z<>*
The file must be a physical (seekable) file and must be given using B<-a> or B<::::>.
The file must be a physical (seekable) file (not a stream) and must be
given using B<-a> or B<::::>.
=item Z<>*
@ -1384,6 +1457,17 @@ commands.
See also B<--joblog>, B<--resume>.
=item B<--retry-failed> (alpha testing)
Retry all failed jobs in joblog. By reading B<--joblog> GNU
B<parallel> will figure out the failed jobs and run those again.
B<--retry-failed> ignore the command and arguments: It only looks at
the joblog.
See also B<--joblog>, B<--resume>, B<--resume-failed>.
=item B<--retries> I<n>
If a job fails, retry it on another computer on which it has not
@ -1498,7 +1582,7 @@ to do anything.
Work as a counting semaphore. B<--semaphore> will cause GNU
B<parallel> to start I<command> in the background. When the number of
simultaneous jobs is reached, GNU B<parallel> will wait for one of
jobs given by B<--jobs> is reached, GNU B<parallel> will wait for one of
these to complete before starting another command.
B<--semaphore> implies B<--bg> unless B<--fg> is specified.
@ -1616,7 +1700,7 @@ Does not run the command but quotes it. Useful for making quoted
composed commands for GNU B<parallel>.
=item B<--shuf> (beta testing)
=item B<--shuf> (alpha testing)
Shuffle jobs. When having multiple input sources it is hard to
randomize jobs. --shuf will generate all jobs, and shuffle them before
@ -1630,6 +1714,13 @@ Do not use the first line of input (used by GNU B<parallel> itself
when called with B<--shebang>).
=item B<--ssh> I<sshcommand> (alpha testing)
GNU B<parallel> defaults to using B<ssh> for remote access. This can
be overridden with B<--ssh>. It can also be set on a per server
basis (see B<--sshlogin>).
=item B<--sshdelay> I<secs>
Delay starting next ssh by I<secs> seconds. GNU B<parallel> will pause
@ -3390,6 +3481,14 @@ $SHELL. If undefined use:
=back
=item $PARALLEL_SSH (alpha testing)
GNU B<parallel> defaults to using B<ssh> for remote access. This can
be overridden with $PARALLEL_SSH, which again can be overridden with
B<--ssh>. It can also be set on a per server basis (see
B<--sshlogin>).
=item $PARALLEL_SEQ
$PARALLEL_SEQ will be set to the sequence number of the job
@ -3481,21 +3580,25 @@ remote computers:
=head1 EXIT STATUS
If B<--halt-on-error> 0 or not specified:
Exit status depends on B<--halt-on-error> if one of these are used:
success=X, success=Y%, fail=Y%.
=over 6
=item Z<>0
All jobs ran without error.
All jobs ran without error. If success=X is used: X jobs ran without
error. If success=Y% is used: Y% of the jobs ran without error.
=item Z<>1-253
=item Z<>1-100
Some of the jobs failed. The exit status gives the number of failed jobs
Some of the jobs failed. The exit status gives the number of failed
jobs. If Y% is used the exit status is the percentage of jobs that
failed.
=item Z<>254
=item Z<>101
More than 253 jobs failed.
More than 100 jobs failed.
=item Z<>255
@ -3503,7 +3606,8 @@ Other error.
=back
If B<--halt-on-error> 1 or 2: Exit status of the failing job.
If fail=1 is used, the exit status will be the exit status of the
failing job.
=head1 DIFFERENCES BETWEEN GNU Parallel AND ALTERNATIVES
@ -3532,7 +3636,7 @@ Manipulation of input
M3. Arguments can be put anywhere in the execution line
M4. Multiple arguments can be put anywhere in the execution line
M5. Arguments can be replaced with context
M6. Input can be treated as complete execution line
M6. Input can be treated as the complete command line
Outputs
O1. Grouping output so output from different jobs do not mix

30
src/parallel_design.pod
View file

@ -141,19 +141,29 @@ command.
{unlink;rmdir;} if($bash=~s/h//) {exit$bash;} exit$csh;' "$?h"
"$status" {});
{} is really just a tmpfile. The Perl script saves the exit value,
unlinks the tmpfile, and returns the exit value - no matter if the
shell is B<bash> (using $?) or B<*csh> (using $status).
{} is set to $PARALLEL_TMP which is a tmpfile. The Perl script saves
the exit value, unlinks the tmpfile, and returns the exit value - no
matter if the shell is B<bash> (using $?) or B<*csh> (using $status).
=item --fifo
(mkfifo {};
(<<input>> {};) & _PID=$!; cat > {}; wait $_PID; perl -e '$bash=shift; $csh=shift; for(@ARGV)
{unlink;rmdir;} if($bash=~s/h//) {exit$bash;} exit$csh;' "$?h"
"$status" {});
perl -e '($s,$c,$f) = @ARGV;
system "mkfifo", $f;
$pid = fork || exec $s, "-c", $c;
open($o,">",$f) || die $!;
while(sysread(STDIN,$buf,32768)){
syswrite $o, $buf;
}
close $o;
waitpid $pid,0;
unlink $f;
exit $?/256;' $shell <<input>> $PARALLEL_TMP
B<wait $_PID> makes sure the exit value is from that PID. This makes it
incompatible with B<*csh>. The Perl script is the same as from B<--cat>.
This is an elaborate way of: mkfifo {}; run <<input>> in the
background using $shell; copying STDIN to {}; waiting for background
to complete; remove {} and exit with the exit code from <<input>>.
It is made this way to be compatible with B<*csh>.
=item --sshlogin I<sln>
@ -249,8 +259,6 @@ For B<tmux 1.8> 17000 can be lowered to 2100.
The interesting areas are title 0..1000 with (title + whole command)
in 996..1127 and 9331..9636.
=back
The ordering of the wrapping is important:

114
src/parallel_tutorial.html
View file

@ -55,6 +55,7 @@
<li><a href="#Control-the-execution">Control the execution</a>
<ul>
<li><a href="#Number-of-simultaneous-jobs">Number of simultaneous jobs</a></li>
<li><a href="#Shuffle-job-order">Shuffle job order</a></li>
<li><a href="#Interactivity">Interactivity</a></li>
<li><a href="#A-terminal-for-every-job">A terminal for every job</a></li>
<li><a href="#Timing">Timing</a></li>
@ -93,6 +94,7 @@
<li><a href="#Semaphore">Semaphore</a>
<ul>
<li><a href="#Counting-semaphore">Counting semaphore</a></li>
<li><a href="#Timeout">Timeout</a></li>
</ul>
</li>
<li><a href="#Informational">Informational</a></li>
@ -1068,6 +1070,16 @@
<pre><code> parallel --use-cpus-instead-of-cores -N0 sleep 1 :::: num8</code></pre>
<h2 id="Shuffle-job-order">Shuffle job order</h2>
<p>If you have many jobs (e.g. by multiple combinations of input sources), it can be handy to shuffle the jobs, so you get different values run.</p>
<pre><code> parallel --shuf echo ::: 1 2 3 ::: a b c ::: A B C</code></pre>
<p>Output:</p>
<pre><code> All combinations but different order for each run.</code></pre>
<h2 id="Interactivity">Interactivity</h2>
<p>GNU Parallel can ask the user if a command should be run using --interactive:</p>
@ -1098,7 +1110,7 @@
<p>This will tell you to run something similar to:</p>
<pre><code> tmux -S /tmp/paroRLCx.tms attach</code></pre>
<pre><code> tmux -S /tmp/tmsrPrO0 attach</code></pre>
<p>Using normal tmux keystrokes (CTRL-b n or CTRL-b p) you can cycle between windows of the running jobs. When a job is finished it will pause for 10 seconds before closing the window.</p>
@ -1233,9 +1245,9 @@
<h2 id="Termination">Termination</h2>
<p>For certain jobs there is no need to continue if one of the jobs fails and has an exit code != 0. GNU Parallel will stop spawning new jobs with --halt 1:</p>
<p>For certain jobs there is no need to continue if one of the jobs fails and has an exit code != 0. GNU Parallel will stop spawning new jobs with --halt soon,fail=1:</p>
<pre><code> parallel -j2 --halt 1 echo {}\; exit {} ::: 0 0 1 2 3</code></pre>
<pre><code> parallel -j2 --halt soon,fail=1 echo {}\; exit {} ::: 0 0 1 2 3</code></pre>
<p>Output:</p>
@ -1248,9 +1260,9 @@
parallel: Starting no more jobs. Waiting for 1 jobs to finish. This job failed:
echo 2; exit 2</code></pre>
<p>With --halt 2 the running jobs will be killed immediately:</p>
<p>With --halt now,fail=1 the running jobs will be killed immediately:</p>
<pre><code> parallel -j2 --halt 2 echo {}\; exit {} ::: 0 0 1 2 3</code></pre>
<pre><code> parallel -j2 --halt now,fail=1 echo {}\; exit {} ::: 0 0 1 2 3</code></pre>
<p>Output:</p>
@ -1260,23 +1272,38 @@
parallel: This job failed:
echo 1; exit 1</code></pre>
<p>If --halt is given a percentage this percentage of the jobs must fail (though minimum 3) before GNU Parallel stops spawning more jobs:</p>
<p>If --halt is given a percentage this percentage of the jobs must fail before GNU Parallel stops spawning more jobs:</p>
<pre><code> parallel -j2 --halt 20% echo {}\; exit {} ::: 0 0 1 2 3 4 5 6 7</code></pre>
<pre><code> parallel -j2 --halt soon,fail=20% echo {}\; exit {} ::: 0 1 2 3 4 5 6 7 8 9</code></pre>
<p>Output:</p>
<pre><code> 0
0
1
parallel: This job failed:
echo 1; exit 1
2
parallel: This job failed:
echo 2; exit 2
parallel: Starting no more jobs. Waiting for 1 jobs to finish.
3
parallel: This job failed:
echo 3; exit 3</code></pre>
<p>If you are looking for success instead of failures, you can use success:</p>
<pre><code> parallel -j2 --halt soon,success=1 echo {}\; exit {} ::: 1 2 3 0 4 5 6</code></pre>
<p>Output:</p>
<pre><code> 1
2
3
4
parallel: Starting no more jobs. Waiting for 2 jobs to finish. This job failed:
echo 4; exit 4
5
parallel: Starting no more jobs. Waiting for 1 jobs to finish. This job failed:
echo 5; exit 5</code></pre>
0
parallel: This job succeeded:
echo 0; exit 0
parallel: Starting no more jobs. Waiting for 1 jobs to finish.
4</code></pre>
<p>GNU Parallel can retry the command with --retries. This is useful if a command fails for unknown reasons now and then.</p>
@ -1382,6 +1409,12 @@
<p>Output: Same as above.</p>
<p>Or newline:</p>
<pre><code> # This gives a \n between $SERVER1 and $SERVER2
SERVERS=&quot;`echo $SERVER1; echo $SERVER2`&quot;
parallel -S &quot;$SERVERS&quot; echo ::: running on more hosts</code></pre>
<p>The can also be read from a file (replace user@ with the user on $SERVER2):</p>
<pre><code> echo $SERVER1 &gt; nodefile
@ -1401,6 +1434,24 @@
<pre><code> force 4 cpus on server</code></pre>
<p>Servers can be put into groups by prepending &#39;@groupname&#39; to the server and the group can then be selected by appending &#39;@groupname&#39; to the argument if using &#39;--hostgrp&#39;.</p>
<pre><code> parallel --hostgrp -S @grp1/$SERVER1 -S @grp2/SERVER2 echo {} ::: run_on_grp1@grp1 run_on_grp2@grp2</code></pre>
<p>Output:</p>
<pre><code> run_on_grp1
run_on_grp2</code></pre>
<p>A host can be in multiple groups by separating groups with &#39;+&#39;, and you can force GNU <b>parallel</b> to limit the groups on which the command can be run with &#39;-S @groupname&#39;:</p>
<pre><code> parallel -S @grp1 -S @grp1+grp2/$SERVER1 -S @grp2/SERVER2 echo {} ::: run_on_grp1 also_grp1</code></pre>
<p>Output:</p>
<pre><code> run_on_grp1
also_grp1</code></pre>
<h2 id="Transferring-files">Transferring files</h2>
<p>GNU Parallel can transfer the files to be processed to the remote host. It does that using rsync.</p>
@ -1969,7 +2020,19 @@
<p>GNU Parallel can work as a counting semaphore. This is slower and less efficient than its normal mode.</p>
<p>An alias for &#39;parallel --semaphore&#39; is &#39;sem&#39;. The default is to allow only one program to run at a time (technically called a mutex). The program is started in the background. Use --wait for all &#39;sem&#39;s to finish:</p>
<p>A counting semaphore is like a row of toilets. People needing a toilet can use any toilet, but if there are more people than toilets, they will have to wait for one of the toilets to be available.</p>
<p>An alias for &#39;parallel --semaphore&#39; is &#39;sem&#39;.</p>
<p>&#39;sem&#39; will follow a person to the toilets, wait until a toilet is available, leave the person in the toilet and exit.</p>
<p>&#39;sem --fg&#39; will follow a person to the toilets, wait until a toilet is available, stay with the person in the toilet and exit when the person exits.</p>
<p>&#39;sem --wait&#39; will wait for all persons to leave the toilets.</p>
<p>&#39;sem&#39; does not have a queue discipline, so the next person is chosen randomly.</p>
<p>-j sets the number of toilets. The default is to have only one toilet (technically this is called a mutex). The program is started in the background and &#39;sem&#39; exits immediately. Use --wait to wait for all &#39;sem&#39;s to finish:</p>
<pre><code> sem &#39;sleep 1; echo The first finished&#39; &amp;&amp;
echo The first is now running in the background &amp;&amp;
@ -1984,7 +2047,7 @@
The second is now running in the background
The second finished</code></pre>
<p>The command can be run in the foreground with --fg:</p>
<p>The command can be run in the foreground with --fg, which will only exit when the command completes:</p>
<pre><code> sem --fg &#39;sleep 1; echo The first finished&#39; &amp;&amp;
echo The first finished running in the foreground &amp;&amp;
@ -2027,6 +2090,25 @@
The third finished
The fourth finished</code></pre>
<h2 id="Timeout">Timeout</h2>
<p>With --semaphoretimeout you can force running the command anyway after a period (postive number) or give up (negative number):</p>
<pre><code> sem --id foo -u &#39;echo Slow started; sleep 5; echo Slow ended&#39; &amp;&amp;
sem --id foo --semaphoretimeout 1 &#39;echo Force this running after 1 sec&#39; &amp;&amp;
sem --id foo --semaphoretimeout -2 &#39;echo Give up after 1 sec&#39;
sem --id foo --wait</code></pre>
<p>Output:</p>
<pre><code> Slow started
parallel: Warning: Semaphore timed out. Stealing the semaphore.
Force this running after 1 sec
Slow ended
parallel: Warning: Semaphore timed out. Exiting.</code></pre>
<p>Note how the &#39;Give up&#39; was not run.</p>
<h1 id="Informational">Informational</h1>
<p>GNU Parallel has some options to give short information about the configuration.</p>

129
src/parallel_tutorial.pod
View file

@ -996,6 +996,18 @@ GNU Parallel can base it on the number of CPUs:
parallel --use-cpus-instead-of-cores -N0 sleep 1 :::: num8
=head2 Shuffle job order
If you have many jobs (e.g. by multiple combinations of input
sources), it can be handy to shuffle the jobs, so you get different
values run.
parallel --shuf echo ::: 1 2 3 ::: a b c ::: A B C
Output:
All combinations but different order for each run.
=head2 Interactivity
GNU Parallel can ask the user if a command should be run using --interactive:
@ -1027,7 +1039,7 @@ Using tmux GNU Parallel can start a terminal for every job run:
This will tell you to run something similar to:
tmux -S /tmp/paroRLCx.tms attach
tmux -S /tmp/tmsrPrO0 attach
Using normal tmux keystrokes (CTRL-b n or CTRL-b p) you can cycle
between windows of the running jobs. When a job is finished it will
@ -1178,9 +1190,9 @@ Note how seq 1 2 3 have been repeated because they had exit value != 0.
For certain jobs there is no need to continue if one of the jobs fails
and has an exit code != 0. GNU Parallel will stop spawning new jobs
with --halt 1:
with --halt soon,fail=1:
parallel -j2 --halt 1 echo {}\; exit {} ::: 0 0 1 2 3
parallel -j2 --halt soon,fail=1 echo {}\; exit {} ::: 0 0 1 2 3
Output:
@ -1193,9 +1205,9 @@ Output:
parallel: Starting no more jobs. Waiting for 1 jobs to finish. This job failed:
echo 2; exit 2
With --halt 2 the running jobs will be killed immediately:
With --halt now,fail=1 the running jobs will be killed immediately:
parallel -j2 --halt 2 echo {}\; exit {} ::: 0 0 1 2 3
parallel -j2 --halt now,fail=1 echo {}\; exit {} ::: 0 0 1 2 3
Output:
@ -1206,23 +1218,38 @@ Output:
echo 1; exit 1
If --halt is given a percentage this percentage of the jobs must fail
(though minimum 3) before GNU Parallel stops spawning more jobs:
before GNU Parallel stops spawning more jobs:
parallel -j2 --halt 20% echo {}\; exit {} ::: 0 0 1 2 3 4 5 6 7
parallel -j2 --halt soon,fail=20% echo {}\; exit {} ::: 0 1 2 3 4 5 6 7 8 9
Output:
0
0
1
parallel: This job failed:
echo 1; exit 1
2
parallel: This job failed:
echo 2; exit 2
parallel: Starting no more jobs. Waiting for 1 jobs to finish.
3
parallel: This job failed:
echo 3; exit 3
If you are looking for success instead of failures, you can use success:
parallel -j2 --halt soon,success=1 echo {}\; exit {} ::: 1 2 3 0 4 5 6
Output:
1
2
3
0
parallel: This job succeeded:
echo 0; exit 0
parallel: Starting no more jobs. Waiting for 1 jobs to finish.
4
parallel: Starting no more jobs. Waiting for 2 jobs to finish. This job failed:
echo 4; exit 4
5
parallel: Starting no more jobs. Waiting for 1 jobs to finish. This job failed:
echo 5; exit 5
GNU Parallel can retry the command with --retries. This is useful if a
command fails for unknown reasons now and then.
@ -1332,6 +1359,12 @@ Or they can be separated by ,:
Output: Same as above.
Or newline:
# This gives a \n between $SERVER1 and $SERVER2
SERVERS="`echo $SERVER1; echo $SERVER2`"
parallel -S "$SERVERS" echo ::: running on more hosts
The can also be read from a file (replace user@ with the user on $SERVER2):
echo $SERVER1 > nodefile
@ -1352,6 +1385,28 @@ Output:
force 4 cpus on server
Servers can be put into groups by prepending '@groupname' to the
server and the group can then be selected by appending '@groupname' to
the argument if using '--hostgrp'.
parallel --hostgrp -S @grp1/$SERVER1 -S @grp2/SERVER2 echo {} ::: run_on_grp1@grp1 run_on_grp2@grp2
Output:
run_on_grp1
run_on_grp2
A host can be in multiple groups by separating groups with '+', and
you can force GNU B<parallel> to limit the groups on which the command
can be run with '-S @groupname':
parallel -S @grp1 -S @grp1+grp2/$SERVER1 -S @grp2/SERVER2 echo {} ::: run_on_grp1 also_grp1
Output:
run_on_grp1
also_grp1
=head2 Transferring files
GNU Parallel can transfer the files to be processed to the remote
@ -1965,10 +2020,28 @@ This technique can be used for:
GNU Parallel can work as a counting semaphore. This is slower and less
efficient than its normal mode.
An alias for 'parallel --semaphore' is 'sem'. The default is to allow
only one program to run at a time (technically called a mutex). The
program is started in the background. Use --wait for all 'sem's to
finish:
A counting semaphore is like a row of toilets. People needing a toilet
can use any toilet, but if there are more people than toilets, they
will have to wait for one of the toilets to be available.
An alias for 'parallel --semaphore' is 'sem'.
'sem' will follow a person to the toilets, wait until a toilet is
available, leave the person in the toilet and exit.
'sem --fg' will follow a person to the toilets, wait until a toilet is
available, stay with the person in the toilet and exit when the person
exits.
'sem --wait' will wait for all persons to leave the toilets.
'sem' does not have a queue discipline, so the next person is chosen
randomly.
-j sets the number of toilets. The default is to have only one toilet
(technically this is called a mutex). The program is started in the
background and 'sem' exits immediately. Use --wait to wait for all
'sem's to finish:
sem 'sleep 1; echo The first finished' &&
echo The first is now running in the background &&
@ -1983,7 +2056,8 @@ Output:
The second is now running in the background
The second finished
The command can be run in the foreground with --fg:
The command can be run in the foreground with --fg, which will only
exit when the command completes:
sem --fg 'sleep 1; echo The first finished' &&
echo The first finished running in the foreground &&
@ -2033,6 +2107,25 @@ Output:
The third finished
The fourth finished
=head2 Timeout
With --semaphoretimeout you can force running the command anyway after
a period (postive number) or give up (negative number):
sem --id foo -u 'echo Slow started; sleep 5; echo Slow ended' &&
sem --id foo --semaphoretimeout 1 'echo Force this running after 1 sec' &&
sem --id foo --semaphoretimeout -2 'echo Give up after 1 sec'
sem --id foo --wait
Output:
Slow started
parallel: Warning: Semaphore timed out. Stealing the semaphore.
Force this running after 1 sec
Slow ended
parallel: Warning: Semaphore timed out. Exiting.
Note how the 'Give up' was not run.
=head1 Informational

106
src/sem.pod
View file

@ -12,11 +12,14 @@ B<sem> [--fg] [--id <id>] [--semaphoretimeout <secs>] [-j <num>] [--wait] comman
GNU B<sem> is an alias for GNU B<parallel --semaphore>.
It works as a tool for executing shell commands in parallel. GNU
B<sem> acts as a counting semaphore. When GNU B<sem> is called with
command it will start the command in the background. When I<num>
number of commands are running in the background, GNU B<sem> will wait
for one of these to complete before starting another command.
GNU B<sem> acts as a counting semaphore. When GNU B<sem> is called
with command it starts the command in the background. When I<num>
number of commands are running in the background, GNU B<sem> waits for
one of these to complete before starting the command.
GNU B<sem> does not read any arguments to build the command (no -a,
:::, and ::::). It simply waits for a semaphore to become available
and then runs the command given.
Before looking at the options you may want to check out the examples
after the list of options. That will give you an idea of what GNU
@ -28,23 +31,21 @@ B<sem> is capable of.
=item I<command>
Command to execute. The command may be followed by arguments for the command.
Command to execute. The command may be followed by arguments for the
command.
=item B<--bg>
Run command in background thus GNU B<parallel> will not wait for
Run command in background thus GNU B<sem> will not wait for
completion of the command before exiting. This is the default.
In toilet analogy: GNU B<sem> waits for a toilet to be available,
gives the toilet to a person, and exits immediately.
See also: B<--fg>
=item B<-j> I<N>
Run up to N commands in parallel. Default is 1 thus acting like a
mutex.
=item B<--jobs> I<N>
=item B<-j> I<N>
@ -56,6 +57,8 @@ mutex.
Run up to N commands in parallel. Default is 1 thus acting like a
mutex.
In toilet analogy: B<-j> is the number of toilets.
=item B<--jobs> I<+N>
@ -122,29 +125,70 @@ are often a good value.
The semaphore is stored in ~/.parallel/semaphores/
In toilet analogy the name corresponds to different types of toilets:
e.g. male, female, customer, staff.
=item B<--fg>
Do not put command in background.
In toilet analogy: GNU B<sem> waits for a toilet to be available,
takes a person to the toilet, waits for the person to finish, and
exits.
=item B<--semaphoretimeout> I<secs> (alpha testing)
=item B<--st> I<secs> (alpha testing)
If I<secs> > 0: If the semaphore is not released within I<secs> seconds, take it anyway.
If I<secs> > 0: If the semaphore is not released within I<secs>
seconds, take it anyway.
If I<secs> < 0: If the semaphore is not released within I<secs> seconds, exit.
If I<secs> < 0: If the semaphore is not released within I<secs>
seconds, exit.
In toilet analogy: I<secs> > 0: If no toilet becomes available within
I<secs> seconds, pee on the floor. I<secs> < 0: If no toilet becomes
available within I<secs> seconds, exit without doing anything.
=item B<--wait>
=item B<-w>
Wait for all commands to complete.
In toilet analogy: Wait until all toilets are empty, then exit.
=back
=head1 UNDERSTANDING A SEMAPHORE
Try the following example:
sem -j 2 'sleep 1;echo 1 finished'; echo sem 1 exited
sem -j 2 'sleep 2;echo 2 finished'; echo sem 2 exited
sem -j 2 'sleep 3;echo 3 finished'; echo sem 3 exited
sem -j 2 'sleep 4;echo 4 finished'; echo sem 4 exited
sem --wait; echo sem --wait done
In toilet analogy this uses 2 toilets (B<-j 2>). GNU B<sem> takes '1'
to a toilet, and exits immediately. While '1' is sleeping, another GNU
B<sem> takes '2' to a toilet, and exits immediately.
While '1' and '2' are sleeping, another GNU B<sem> waits for a free
toilet. When '1' finishes, a toilet becomes available, and this GNU
B<sem> stops waiting, and takes '3' to a toilet, and exits
immediately.
While '2' and '3' are sleeping, another GNU B<sem> waits for a free
toilet. When '2' finishes, a toilet becomes available, and this GNU
B<sem> stops waiting, and takes '4' to a toilet, and exits
immediately.
Finally another GNU B<sem> waits for all toilets to become free.
=head1 EXAMPLE: Gzipping *.log
Run one gzip process per CPU core. Block until a CPU core becomes
@ -161,12 +205,32 @@ available.
pod2html creates two files: pod2htmd.tmp and pod2htmi.tmp which it
does not clean up. It uses these two files for a short time. But if
you run multiple pod2html in parallel (e.g. in a Makefile with make
-j) you need to protect pod2html from running twice at the same
time. B<sem> running as a mutex will do just that:
-j) there is a risk that two different instances of pod2html will
write to the files at the same time:
sem --fg --id pod2html pod2html foo.pod > foo.html
sem --fg --id pod2html rm -f pod2htmd.tmp pod2htmi.tmp
# This may fail due to shared pod2htmd.tmp/pod2htmi.tmp files
foo.html:
pod2html foo.pod --outfile foo.html
bar.html:
pod2html bar.pod --outfile bar.html
$ make -j foo.html bar.html
You need to protect pod2html from running twice at the same time.
B<sem> running as a mutex will make sure only one runs:
foo.html:
sem --id pod2html pod2html foo.pod --outfile foo.html
bar.html:
sem --id pod2html pod2html bar.pod --outfile bar.html
clean: foo.html bar.html
sem --id pod2html --wait
rm -f pod2htmd.tmp pod2htmi.tmp
$ make -j foo.html bar.html clean
=head1 BUGS

23
testsuite/Start.sh
View file

@ -10,21 +10,22 @@ export TIMEOUT=$MAX_SEC_PER_TEST
run_test() {
script="$1"
base=`basename "$script" .sh`
export TMPDIR=/tmp/$base
export TMPDIR=/tmp/"$base"
mkdir -p "$TMPDIR"
if [ "$TRIES" = "3" ] ; then
# Try 3 times
bash $script > actual-results/$base
diff -Naur wanted-results/$base actual-results/$base >/dev/null ||
bash $script > actual-results/$base
diff -Naur wanted-results/$base actual-results/$base >/dev/null ||
bash $script > actual-results/$base
diff -Naur wanted-results/$base actual-results/$base ||
(touch $script && echo touch $script)
bash "$script" > actual-results/"$base"
diff -Naur wanted-results/"$base" actual-results/"$base" >/dev/null ||
bash "$script" > actual-results/"$base"
diff -Naur wanted-results/"$base" actual-results/"$base" >/dev/null ||
bash "$script" > actual-results/"$base"
diff -Naur wanted-results/"$base" actual-results/"$base" ||
(touch "$script" && echo touch "$script")
else
# Run only once
bash $script > actual-results/$base
diff -Naur wanted-results/$base actual-results/$base ||
(touch $script && echo touch $script)
bash "$script" > actual-results/"$base"
diff -Naur wanted-results/"$base" actual-results/"$base" ||
(touch "$script" && echo touch "$script")
fi
# Check if it was cleaned up

10
testsuite/tests-to-run/parallel-local-0.3s.sh
View file

@ -148,6 +148,16 @@ echo '**'
parallel --halt 2 ::: 'sleep 1' burnP6 false; killall burnP6 && echo ERROR: burnP6 should be killed
parallel --halt -2 ::: 'sleep 1' burnP5 true; killall burnP5 && echo ERROR: burnP5 should be killed
parallel --halt error echo ::: should not print
parallel --halt soon echo ::: should not print
parallel --halt now echo ::: should not print
echo '**'
echo '### bug #44995: parallel echo {#} ::: 1 2 ::: 1 2'
parallel -k echo {#} ::: 1 2 ::: 1 2
EOF
echo '### 1 .par file from --files expected'
find /tmp{/*,}/*.{par,tms,tmx} 2>/dev/null -mmin -10 | wc -l

10
testsuite/tests-to-run/parallel-local-1s.sh Normal file
View file

@ -0,0 +1,10 @@
#!/bin/bash
# Simple jobs that never fails
# Each should be taking 1-3s and be possible to run in parallel
# I.e.: No race conditions, no logins
cat <<'EOF' | sed -e 's/;$/; /;s/$SERVER1/'$SERVER1'/;s/$SERVER2/'$SERVER2'/' | stdout parallel -vj0 -k --joblog /tmp/jl-`basename $0` -L1
echo '### Test --fifo under csh'
csh -c "seq 3000000 | parallel -k --pipe --fifo 'sleep .{#};cat {}|wc -c ; false; echo \$status; false'"; echo exit $?
EOF

34
testsuite/tests-to-run/parallel-local-3s.sh
View file

@ -14,6 +14,7 @@ echo '**'
echo '### Test --halt-on-error 0';
(echo "sleep 1;true"; echo "sleep 2;false";echo "sleep 3;true") | parallel -j10 --halt-on-error 0;
echo $?;
(echo "sleep 1;true"; echo "sleep 2;false";echo "sleep 3;true";echo "sleep 4; non_exist") | parallel -j10 --halt 0;
echo $?
@ -22,7 +23,8 @@ echo '**'
echo '### Test --halt-on-error 1';
(echo "sleep 1;true"; echo "sleep 2;false";echo "sleep 3;true") | parallel -j10 --halt-on-error 1;
echo $?;
(echo "sleep 1;true"; echo "sleep 2;false";echo "sleep 3;true";echo "sleep 4; non_exist") | parallel -j10 --halt 1;
(echo "sleep 1;true"; echo "sleep 2; non_exist";echo "sleep 3;true";echo "sleep 4; false") | parallel -j10 --halt 1;
echo $?
echo '**'
@ -30,6 +32,7 @@ echo '**'
echo '### Test --halt-on-error 2';
(echo "sleep 1;true"; echo "sleep 2;false";echo "sleep 3;true") | parallel -j10 --halt-on-error 2;
echo $?;
(echo "sleep 1;true"; echo "sleep 2;false";echo "sleep 3;true";echo "sleep 4; non_exist") | parallel -j10 --halt 2;
echo $?
@ -38,6 +41,7 @@ echo '**'
echo '### Test --halt -1';
(echo "sleep 1;false"; echo "sleep 2;true";echo "sleep 3;false") | parallel -j10 --halt-on-error -1;
echo $?;
(echo "sleep 1;false"; echo "sleep 2;true";echo "sleep 3;false";echo "sleep 4; non_exist") | parallel -j10 --halt -1;
echo $?
@ -46,29 +50,43 @@ echo '**'
echo '### Test --halt -2';
(echo "sleep 1;false"; echo "sleep 2;true";echo "sleep 3;false") | parallel -j10 --halt-on-error -2;
echo $?;
(echo "sleep 1;false"; echo "sleep 2;true";echo "sleep 3;false";echo "sleep 4; non_exist") | parallel -j10 --halt -2;
echo $?
echo '**'
echo '### Test last dying print --halt-on-error 1';
(seq 0 8;echo 0; echo 9) | parallel -j10 -kq --halt 1 perl -e 'sleep $ARGV[0];print STDERR @ARGV,"\n"; exit shift';
echo '### Test first dying print --halt-on-error 1';
(echo 0; echo 3; seq 0 7;echo 0; echo 8) | parallel -j10 -kq --halt 1 perl -e 'sleep $ARGV[0];print STDERR @ARGV,"\n"; exit shift';
echo exit code $?
echo '### Test last dying print --halt-on-error 2';
(seq 0 8;echo 0; echo 9) | parallel -j10 -kq --halt 2 perl -e 'sleep $ARGV[0];print STDERR @ARGV,"\n"; exit shift';
(echo 0; echo 3; seq 0 7;echo 0; echo 8) | parallel -j10 -kq --halt 2 perl -e 'sleep $ARGV[0];print STDERR @ARGV,"\n"; exit shift';
echo exit code $?
echo '### Test last dying print --halt-on-error -1';
(seq 0 8;echo 0; echo 9) | parallel -j10 -kq --halt -1 perl -e 'sleep $ARGV[0];print STDERR @ARGV,"\n"; exit not shift';
(echo 0; echo 3; seq 0 7;echo 0; echo 8) | parallel -j10 -kq --halt -1 perl -e 'sleep $ARGV[0];print STDERR @ARGV,"\n"; exit not shift';
echo exit code $?
echo '### Test last dying print --halt-on-error -2';
(seq 0 8;echo 0; echo 9) | parallel -j10 -kq --halt -2 perl -e 'sleep $ARGV[0];print STDERR @ARGV,"\n"; exit not shift';
(echo 0; echo 3; seq 0 7;echo 0; echo 8) | parallel -j10 -kq --halt -2 perl -e 'sleep $ARGV[0];print STDERR @ARGV,"\n"; exit not shift';
echo exit code $?
echo '**'
testhalt() {
echo '### testhalt --halt '$1;
(yes 0 | head -n 10; seq 10) | stdout parallel -kj4 --halt $1 'sleep {= $_*=0.3 =}; exit {}'; echo $?;
(seq 10; yes 0 | head -n 10) | stdout parallel -kj4 --halt $1 'sleep {= $_*=0.3 =}; exit {}'; echo $?;
};
export -f testhalt;
parallel -kj0 testhalt ::: now,fail=0 now,fail=1 now,fail=2 now,fail=30% now,fail=70%
soon,fail=0 soon,fail=1 soon,fail=2 soon,fail=30% soon,fail=70%
now,success=0 now,success=1 now,success=2 now,success=30% now,success=70%
soon,success=0 soon,success=1 soon,success=2 soon,success=30% now,success=70%
echo '**'
echo '### Test slow arguments generation - https://savannah.gnu.org/bugs/?32834';
seq 1 3 | parallel -j1 "sleep 2; echo {}" | parallel -kj2 echo
@ -78,7 +96,7 @@ echo '### Are children killed if GNU Parallel receives TERM twice? There should
parallel -q bash -c 'sleep 120 & pid=$!; wait $pid' ::: 1 &
T=$!;
sleep 1;
sleep 2;
pstree $$;
kill -TERM $T;
sleep 1;
@ -93,7 +111,7 @@ echo '### Are children killed if GNU Parallel receives INT twice? There should b
parallel -q bash -c 'sleep 120 & pid=$!; wait $pid' ::: 1 &
T=$!;
sleep 1;
sleep 2;
pstree $$;
kill -INT $T;
sleep 1;

10
testsuite/tests-to-run/parallel-local-ssh1.sh
View file

@ -61,6 +61,16 @@ echo '### bug #40001: --joblog and --nonall seem not to work together:'
echo '### bug #40132: FreeBSD: --workdir . gives warning if . == $HOME'
cd && parallel --workdir . -S lo pwd ::: ""
echo '### use function as $PARALLEL_SSH'
foossh() { echo "FOOSSH" >&2; ssh "$@"; };
export -f foossh;
PARALLEL_SSH=foossh parallel -S 1/lo echo ::: 'Run through FOOSSH?'
echo '### use --ssh'
barssh() { echo "BARSSH" >&2; ssh "$@"; };
export -f barssh;
parallel --ssh barssh -S 1/lo echo ::: 'Run through BARSSH?'
echo '### test filename :'
echo content-of-: > :;
echo : | parallel -j1 --trc {}.{.} -S parallel@lo '(echo remote-{}.{.};cat {}) > {}.{.}';

2
testsuite/tests-to-run/parallel-local-ssh2.sh
View file

@ -3,7 +3,7 @@
# /tmp/parallel-local-ssh2 will by default be owned by me and should be writable by *@localhost
chmod 777 "$TMPDIR" 2>/dev/null
cat <<'EOF' | sed -e s/\$SERVER1/$SERVER1/\;s/\$SERVER2/$SERVER2/ | stdout parallel -vj5 -k --joblog /tmp/jl-`basename $0` -L1
cat <<'EOF' | sed -e s/\$SERVER1/$SERVER1/\;s/\$SERVER2/$SERVER2/ | stdout parallel -vj5 --retries 3 -k --joblog /tmp/jl-`basename $0` -L1
echo "### bug #43518: GNU Parallel doesn't obey servers' jobs capacity when an ssh login file is reloaded"
# Pre-20141106 Would reset the number of jobs run on all sshlogin if --slf changed
# Thus must take at least 25 sec to run

2
testsuite/tests-to-run/parallel-local22.sh
View file

@ -72,7 +72,7 @@ echo '### bug #42892: parallel -a nonexiting --pipepart'
echo '### bug #42913: Dont use $SHELL but the shell currently running'
echo '## Unknown shell => $SHELL (bash)'
parallel -j1 "cp \`which {}\` /tmp/SHELL; /tmp/SHELL -c 'parallel -Dinit echo ::: 1' | grep which;"
::: ash bash csh dash fdsh fish fizsh ksh ksh93 mksh pdksh posh rbash rush rzsh sash sh static-sh tcsh yash zsh;
::: ash bash csh dash fish fizsh ksh ksh93 mksh pdksh posh rbash rush rzsh sash sh static-sh tcsh yash zsh;
rm -f /tmp/SHELL /tmp/par*.par
echo '## Known shells -c'

2
testsuite/tests-to-run/parallel-local4.sh
View file

@ -2,7 +2,7 @@
cat <<'EOF' | sed -e 's/;$/; /;s/$SERVER1/'$SERVER1'/;s/$SERVER2/'$SERVER2'/' | stdout parallel -vj0 -k --joblog /tmp/jl-`basename $0` -L1
echo '### bug #36595: silent loss of input with --pipe and --sshlogin'
seq 10000 | xargs | parallel --pipe -S 10/localhost cat | wc
seq 10000 | xargs | parallel --pipe -S 10/localhost cat 2>/dev/null | wc
echo 'bug #36707: --controlmaster eats jobs'
seq 2 | parallel -k --controlmaster --sshlogin localhost echo OK{}

2
testsuite/tests-to-run/parallel-local7.sh
View file

@ -7,7 +7,7 @@ par_tmux_filter() {
export -f par_tmux_filter
par_tmux() {
(stdout parallel --timeout 3 --tmux --delay .3 echo '{}{=$_="\\"x$_=}'; echo $?) | par_tmux_filter
(stdout parallel --timeout 3 --tmux --delay .4 echo '{}{=$_="\\"x$_=}'; echo $?) | par_tmux_filter
}
export -f par_tmux
cat <<'EOF' | sed -e 's/;$/; /;s/$SERVER1/'$SERVER1'/;s/$SERVER2/'$SERVER2'/' | stdout parallel -vj0 --timeout 60 --retries 2 -k --joblog /tmp/jl-`basename $0` -L1

4
testsuite/tests-to-run/parallel-tutorial.sh
View file

@ -33,6 +33,10 @@ perl -ne '$/="\n\n"; /^Output/../^[^O]\S/ and next; /^ / and print;' ../../src/
s/... ... .. ..:..:.. \D+ ..../DATE OUTPUT/;
# Timestamp from --joblog
s/\d{10}.\d{3}\s+..\d+/TIMESTAMP\t9.999/g;
# Version
s/201\d{5}/VERSION/g;
# [123] [abc] [ABC]
s/^[123] [abc] [ABC]$/123 abc ABC/g;
# Remote script
s/(PARALLEL_PID\D+)\d+/${1}000000/g;
# /usr/bin/time -f %e

2
testsuite/wanted-results/parallel-freebsd
View file

@ -58,7 +58,7 @@ bash -c 'echo bug \#43358: shellshock breaks exporting functions using --env _;
bug #43358: shellshock breaks exporting functions using --env _
Non-shellshock-hardened to non-shellshock-hardened
Function non-shellshock-hardened
parallel: Warning: Shell functions may not be supported in bash
parallel: Warning: Shell functions may not be supported in bash.
bash -c 'echo bug \#43358: shellshock breaks exporting functions using --env _; echo Non-shellshock-hardened to shellshock-hardened; funky() { echo Function $1; }; export -f funky; parallel --env funky -S parallel@192.168.1.72 funky ::: shellshock-hardened'
bug #43358: shellshock breaks exporting functions using --env _
Non-shellshock-hardened to shellshock-hardened

15
testsuite/wanted-results/parallel-local-0.3s
View file

@ -317,5 +317,20 @@ parallel --halt -2 ::: 'sleep 1' burnP5 true; killall burnP5 && echo ERROR: burn
parallel: This job succeeded:
true
burnP5: no process found
parallel --halt error echo ::: should not print
parallel: Error: --halt must have 'never', 'soon', or 'now'.
parallel --halt soon echo ::: should not print
parallel: Error: --halt soon must be followed by ,success or ,fail.
parallel --halt now echo ::: should not print
parallel: Error: --halt now must be followed by ,success or ,fail.
echo '**'
**
echo '### bug #44995: parallel echo {#} ::: 1 2 ::: 1 2'
### bug #44995: parallel echo {#} ::: 1 2 ::: 1 2
parallel -k echo {#} ::: 1 2 ::: 1 2
1
2
3
4
### 1 .par file from --files expected
1

3
testsuite/wanted-results/parallel-local-100s
View file

@ -9,7 +9,8 @@ echo "### Test Force outside the file handle limit, 2009-02-17 Gave fork error"
### Test Force outside the file handle limit, 2009-02-17 Gave fork error
(echo echo Start; seq 1 20000 | perl -pe 's/^/true /'; echo echo end) | stdout parallel -uj 0 | egrep -v 'processes took|adjusting'
parallel: Warning: Only enough file handles to run 252 jobs in parallel.
Running 'parallel -j0 -N252 --pipe parallel -j0' or raising ulimit -n or /etc/security/limits.conf may help.
parallel: Warning: Running 'parallel -j0 -N 252 --pipe parallel -j0' or
parallel: Warning: raising ulimit -n or /etc/security/limits.conf may help.
Start
end
echo '**'

48
testsuite/wanted-results/parallel-local-1s Normal file
View file

@ -0,0 +1,48 @@
echo '### Test --fifo under csh'
### Test --fifo under csh
csh -c "seq 3000000 | parallel -k --pipe --fifo 'sleep .{#};cat {}|wc -c ; false; echo \$status; false'"; echo exit $?
1048571
1
1048579
1
1048572
1
1048579
1
1048579
1
1048572
1
1048580
1
1048576
1
1048576
1
1048576
1
1048576
1
1048576
1
1048576
1
1048576
1
1048576
1
1048576
1
1048576
1
1048576
1
1048576
1
1048576
1
1048576
1
868800
1
exit 22

629
testsuite/wanted-results/parallel-local-3s
View file

@ -13,17 +13,19 @@ echo '### Test --halt-on-error 0'; (echo "sleep 1;true"; echo "sleep 2;false";
/bin/bash: non_exist: command not found
echo '**'
**
echo '### Test --halt-on-error 1'; (echo "sleep 1;true"; echo "sleep 2;false";echo "sleep 3;true") | parallel -j10 --halt-on-error 1; echo $?; (echo "sleep 1;true"; echo "sleep 2;false";echo "sleep 3;true";echo "sleep 4; non_exist") | parallel -j10 --halt 1; echo $?
echo '### Test --halt-on-error 1'; (echo "sleep 1;true"; echo "sleep 2;false";echo "sleep 3;true") | parallel -j10 --halt-on-error 1; echo $?; (echo "sleep 1;true"; echo "sleep 2; non_exist";echo "sleep 3;true";echo "sleep 4; false") | parallel -j10 --halt 1; echo $?
### Test --halt-on-error 1
1
127
parallel: Starting no more jobs. Waiting for 2 jobs to finish. This job failed:
sleep 2;false
parallel: Starting no more jobs. Waiting for 3 jobs to finish. This job failed:
parallel: This job failed:
sleep 2;false
parallel: Starting no more jobs. Waiting for 1 jobs to finish.
/bin/bash: non_exist: command not found
parallel: Starting no more jobs. Waiting for 1 jobs to finish. This job failed:
sleep 4; non_exist
parallel: This job failed:
sleep 2; non_exist
parallel: Starting no more jobs. Waiting for 2 jobs to finish.
parallel: This job failed:
sleep 4; false
echo '**'
**
echo '### Test --halt-on-error 2'; (echo "sleep 1;true"; echo "sleep 2;false";echo "sleep 3;true") | parallel -j10 --halt-on-error 2; echo $?; (echo "sleep 1;true"; echo "sleep 2;false";echo "sleep 3;true";echo "sleep 4; non_exist") | parallel -j10 --halt 2; echo $?
@ -40,10 +42,12 @@ echo '### Test --halt -1'; (echo "sleep 1;false"; echo "sleep 2;true";echo "sl
### Test --halt -1
0
0
parallel: Starting no more jobs. Waiting for 2 jobs to finish. This job succeeded:
parallel: This job succeeded:
sleep 2;true
parallel: Starting no more jobs. Waiting for 3 jobs to finish. This job succeeded:
parallel: Starting no more jobs. Waiting for 1 jobs to finish.
parallel: This job succeeded:
sleep 2;true
parallel: Starting no more jobs. Waiting for 2 jobs to finish.
/bin/bash: non_exist: command not found
echo '**'
**
@ -57,86 +61,597 @@ parallel: This job succeeded:
sleep 2;true
echo '**'
**
echo '### Test last dying print --halt-on-error 1'; (seq 0 8;echo 0; echo 9) | parallel -j10 -kq --halt 1 perl -e 'sleep $ARGV[0];print STDERR @ARGV,"\n"; exit shift'; echo exit code $?
### Test last dying print --halt-on-error 1
exit code 9
echo '### Test first dying print --halt-on-error 1'; (echo 0; echo 3; seq 0 7;echo 0; echo 8) | parallel -j10 -kq --halt 1 perl -e 'sleep $ARGV[0];print STDERR @ARGV,"\n"; exit shift'; echo exit code $?
### Test first dying print --halt-on-error 1
exit code 1
0
parallel: This job failed:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ shift 1
parallel: Starting no more jobs. Waiting for 8 jobs to finish.
parallel: This job failed:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ shift 2
parallel: Starting no more jobs. Waiting for 7 jobs to finish.
3
0
1
parallel: Starting no more jobs. Waiting for 9 jobs to finish. This job failed:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ shift 1
2
parallel: Starting no more jobs. Waiting for 8 jobs to finish. This job failed:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ shift 2
3
parallel: Starting no more jobs. Waiting for 7 jobs to finish. This job failed:
parallel: This job failed:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ shift 3
parallel: Starting no more jobs. Waiting for 6 jobs to finish.
3
parallel: This job failed:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ shift 3
parallel: Starting no more jobs. Waiting for 5 jobs to finish.
4
parallel: Starting no more jobs. Waiting for 6 jobs to finish. This job failed:
parallel: This job failed:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ shift 4
parallel: Starting no more jobs. Waiting for 4 jobs to finish.
5
parallel: Starting no more jobs. Waiting for 5 jobs to finish. This job failed:
parallel: This job failed:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ shift 5
parallel: Starting no more jobs. Waiting for 3 jobs to finish.
6
parallel: Starting no more jobs. Waiting for 4 jobs to finish. This job failed:
parallel: This job failed:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ shift 6
parallel: Starting no more jobs. Waiting for 2 jobs to finish.
7
parallel: Starting no more jobs. Waiting for 3 jobs to finish. This job failed:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ shift 7
8
0
parallel: Starting no more jobs. Waiting for 2 jobs to finish. This job failed:
parallel: This job failed:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ shift 7
parallel: Starting no more jobs. Waiting for 1 jobs to finish.
8
parallel: This job failed:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ shift 8
9
parallel: Starting no more jobs. Waiting for 1 jobs to finish. This job failed:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ shift 9
echo '### Test last dying print --halt-on-error 2'; (seq 0 8;echo 0; echo 9) | parallel -j10 -kq --halt 2 perl -e 'sleep $ARGV[0];print STDERR @ARGV,"\n"; exit shift'; echo exit code $?
echo '### Test last dying print --halt-on-error 2'; (echo 0; echo 3; seq 0 7;echo 0; echo 8) | parallel -j10 -kq --halt 2 perl -e 'sleep $ARGV[0];print STDERR @ARGV,"\n"; exit shift'; echo exit code $?
### Test last dying print --halt-on-error 2
exit code 1
0
1
parallel: This job failed:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ shift 1
echo '### Test last dying print --halt-on-error -1'; (seq 0 8;echo 0; echo 9) | parallel -j10 -kq --halt -1 perl -e 'sleep $ARGV[0];print STDERR @ARGV,"\n"; exit not shift'; echo exit code $?
echo '### Test last dying print --halt-on-error -1'; (echo 0; echo 3; seq 0 7;echo 0; echo 8) | parallel -j10 -kq --halt -1 perl -e 'sleep $ARGV[0];print STDERR @ARGV,"\n"; exit not shift'; echo exit code $?
### Test last dying print --halt-on-error -1
exit code 0
0
1
parallel: Starting no more jobs. Waiting for 9 jobs to finish. This job succeeded:
parallel: This job succeeded:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ not\ shift 1
2
parallel: Starting no more jobs. Waiting for 8 jobs to finish. This job succeeded:
parallel: Starting no more jobs. Waiting for 8 jobs to finish.
parallel: This job succeeded:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ not\ shift 2
parallel: Starting no more jobs. Waiting for 7 jobs to finish.
3
parallel: Starting no more jobs. Waiting for 7 jobs to finish. This job succeeded:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ not\ shift 3
4
parallel: Starting no more jobs. Waiting for 6 jobs to finish. This job succeeded:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ not\ shift 4
5
parallel: Starting no more jobs. Waiting for 5 jobs to finish. This job succeeded:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ not\ shift 5
6
parallel: Starting no more jobs. Waiting for 4 jobs to finish. This job succeeded:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ not\ shift 6
7
parallel: Starting no more jobs. Waiting for 3 jobs to finish. This job succeeded:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ not\ shift 7
8
0
parallel: Starting no more jobs. Waiting for 2 jobs to finish. This job succeeded:
1
2
parallel: This job succeeded:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ not\ shift 3
parallel: Starting no more jobs. Waiting for 6 jobs to finish.
3
parallel: This job succeeded:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ not\ shift 3
parallel: Starting no more jobs. Waiting for 5 jobs to finish.
4
parallel: This job succeeded:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ not\ shift 4
parallel: Starting no more jobs. Waiting for 4 jobs to finish.
5
parallel: This job succeeded:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ not\ shift 5
parallel: Starting no more jobs. Waiting for 3 jobs to finish.
6
parallel: This job succeeded:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ not\ shift 6
parallel: Starting no more jobs. Waiting for 2 jobs to finish.
7
0
parallel: This job succeeded:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ not\ shift 7
parallel: Starting no more jobs. Waiting for 1 jobs to finish.
8
parallel: This job succeeded:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ not\ shift 8
9
parallel: Starting no more jobs. Waiting for 1 jobs to finish. This job succeeded:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ not\ shift 9
echo '### Test last dying print --halt-on-error -2'; (seq 0 8;echo 0; echo 9) | parallel -j10 -kq --halt -2 perl -e 'sleep $ARGV[0];print STDERR @ARGV,"\n"; exit not shift'; echo exit code $?
echo '### Test last dying print --halt-on-error -2'; (echo 0; echo 3; seq 0 7;echo 0; echo 8) | parallel -j10 -kq --halt -2 perl -e 'sleep $ARGV[0];print STDERR @ARGV,"\n"; exit not shift'; echo exit code $?
### Test last dying print --halt-on-error -2
exit code 0
0
1
parallel: This job succeeded:
perl -e sleep\ \$ARGV\[0\]\;print\ STDERR\ @ARGV,\"\\n\"\;\ exit\ not\ shift 1
echo '**'
**
testhalt() { echo '### testhalt --halt '$1; (yes 0 | head -n 10; seq 10) | stdout parallel -kj4 --halt $1 'sleep {= $_*=0.3 =}; exit {}'; echo $?; (seq 10; yes 0 | head -n 10) | stdout parallel -kj4 --halt $1 'sleep {= $_*=0.3 =}; exit {}'; echo $?; }; export -f testhalt; parallel -kj0 testhalt ::: now,fail=0 now,fail=1 now,fail=2 now,fail=30% now,fail=70% soon,fail=0 soon,fail=1 soon,fail=2 soon,fail=30% soon,fail=70% now,success=0 now,success=1 now,success=2 now,success=30% now,success=70% soon,success=0 soon,success=1 soon,success=2 soon,success=30% now,success=70%
### testhalt --halt now,fail=0
parallel: This job failed:
sleep 0.3; exit 1
0
parallel: This job failed:
sleep 0.3; exit 1
0
### testhalt --halt now,fail=1
parallel: This job failed:
sleep 0.3; exit 1
1
parallel: This job failed:
sleep 0.3; exit 1
1
### testhalt --halt now,fail=2
parallel: This job failed:
sleep 0.3; exit 1
parallel: This job failed:
sleep 0.6; exit 2
2
parallel: This job failed:
sleep 0.3; exit 1
parallel: This job failed:
sleep 0.6; exit 2
2
### testhalt --halt now,fail=30%
parallel: This job failed:
sleep 0.3; exit 1
parallel: This job failed:
sleep 0.6; exit 2
parallel: This job failed:
sleep 0.9; exit 3
parallel: This job failed:
sleep 1.2; exit 4
parallel: This job failed:
sleep 1.5; exit 5
parallel: This job failed:
sleep 1.8; exit 6
30
parallel: This job failed:
sleep 0.3; exit 1
parallel: This job failed:
sleep 0.6; exit 2
parallel: This job failed:
sleep 0.9; exit 3
parallel: This job failed:
sleep 1.2; exit 4
parallel: This job failed:
sleep 1.5; exit 5
parallel: This job failed:
sleep 1.8; exit 6
30
### testhalt --halt now,fail=70%
parallel: This job failed:
sleep 0.3; exit 1
parallel: This job failed:
sleep 0.6; exit 2
parallel: This job failed:
sleep 0.9; exit 3
parallel: This job failed:
sleep 1.2; exit 4
parallel: This job failed:
sleep 1.5; exit 5
parallel: This job failed:
sleep 1.8; exit 6
parallel: This job failed:
sleep 2.1; exit 7
parallel: This job failed:
sleep 2.4; exit 8
parallel: This job failed:
sleep 2.7; exit 9
parallel: This job failed:
sleep 3; exit 10
50
parallel: This job failed:
sleep 0.3; exit 1
parallel: This job failed:
sleep 0.6; exit 2
parallel: This job failed:
sleep 0.9; exit 3
parallel: This job failed:
sleep 1.2; exit 4
parallel: This job failed:
sleep 1.5; exit 5
parallel: This job failed:
sleep 1.8; exit 6
parallel: This job failed:
sleep 2.1; exit 7
parallel: This job failed:
sleep 2.4; exit 8
parallel: This job failed:
sleep 2.7; exit 9
parallel: This job failed:
sleep 3; exit 10
50
### testhalt --halt soon,fail=0
parallel: This job failed:
sleep 0.3; exit 1
parallel: Starting no more jobs. Waiting for 3 jobs to finish.
parallel: This job failed:
sleep 0.6; exit 2
parallel: Starting no more jobs. Waiting for 2 jobs to finish.
parallel: This job failed:
sleep 0.9; exit 3
parallel: Starting no more jobs. Waiting for 1 jobs to finish.
parallel: This job failed:
sleep 1.2; exit 4
0
parallel: This job failed:
sleep 0.3; exit 1
parallel: Starting no more jobs. Waiting for 3 jobs to finish.
parallel: This job failed:
sleep 0.6; exit 2
parallel: Starting no more jobs. Waiting for 2 jobs to finish.
parallel: This job failed:
sleep 0.9; exit 3
parallel: Starting no more jobs. Waiting for 1 jobs to finish.
parallel: This job failed:
sleep 1.2; exit 4
0
### testhalt --halt soon,fail=1
parallel: This job failed:
sleep 0.3; exit 1
parallel: Starting no more jobs. Waiting for 3 jobs to finish.
parallel: This job failed:
sleep 0.6; exit 2
parallel: Starting no more jobs. Waiting for 2 jobs to finish.
parallel: This job failed:
sleep 0.9; exit 3
parallel: Starting no more jobs. Waiting for 1 jobs to finish.
parallel: This job failed:
sleep 1.2; exit 4
1
parallel: This job failed:
sleep 0.3; exit 1
parallel: Starting no more jobs. Waiting for 3 jobs to finish.
parallel: This job failed:
sleep 0.6; exit 2
parallel: Starting no more jobs. Waiting for 2 jobs to finish.
parallel: This job failed:
sleep 0.9; exit 3
parallel: Starting no more jobs. Waiting for 1 jobs to finish.
parallel: This job failed:
sleep 1.2; exit 4
1
### testhalt --halt soon,fail=2
parallel: This job failed:
sleep 0.3; exit 1
parallel: This job failed:
sleep 0.6; exit 2
parallel: Starting no more jobs. Waiting for 3 jobs to finish.
parallel: This job failed:
sleep 0.9; exit 3
parallel: Starting no more jobs. Waiting for 2 jobs to finish.
parallel: This job failed:
sleep 1.2; exit 4
parallel: Starting no more jobs. Waiting for 1 jobs to finish.
parallel: This job failed:
sleep 1.5; exit 5
2
parallel: This job failed:
sleep 0.3; exit 1
parallel: This job failed:
sleep 0.6; exit 2
parallel: Starting no more jobs. Waiting for 3 jobs to finish.
parallel: This job failed:
sleep 0.9; exit 3
parallel: Starting no more jobs. Waiting for 2 jobs to finish.
parallel: This job failed:
sleep 1.2; exit 4
parallel: Starting no more jobs. Waiting for 1 jobs to finish.
parallel: This job failed:
sleep 1.5; exit 5
2
### testhalt --halt soon,fail=30%
parallel: This job failed:
sleep 0.3; exit 1
parallel: This job failed:
sleep 0.6; exit 2
parallel: This job failed:
sleep 0.9; exit 3
parallel: This job failed:
sleep 1.2; exit 4
parallel: This job failed:
sleep 1.5; exit 5
parallel: This job failed:
sleep 1.8; exit 6
parallel: Starting no more jobs. Waiting for 3 jobs to finish.
parallel: This job failed:
sleep 2.1; exit 7
parallel: Starting no more jobs. Waiting for 2 jobs to finish.
parallel: This job failed:
sleep 2.4; exit 8
parallel: Starting no more jobs. Waiting for 1 jobs to finish.
parallel: This job failed:
sleep 2.7; exit 9
30
parallel: This job failed:
sleep 0.3; exit 1
parallel: This job failed:
sleep 0.6; exit 2
parallel: This job failed:
sleep 0.9; exit 3
parallel: This job failed:
sleep 1.2; exit 4
parallel: This job failed:
sleep 1.5; exit 5
parallel: This job failed:
sleep 1.8; exit 6
parallel: Starting no more jobs. Waiting for 3 jobs to finish.
parallel: This job failed:
sleep 2.1; exit 7
parallel: Starting no more jobs. Waiting for 2 jobs to finish.
parallel: This job failed:
sleep 2.4; exit 8
parallel: Starting no more jobs. Waiting for 1 jobs to finish.
parallel: This job failed:
sleep 2.7; exit 9
30
### testhalt --halt soon,fail=70%
parallel: This job failed:
sleep 0.3; exit 1
parallel: This job failed:
sleep 0.6; exit 2
parallel: This job failed:
sleep 0.9; exit 3
parallel: This job failed:
sleep 1.2; exit 4
parallel: This job failed:
sleep 1.5; exit 5
parallel: This job failed:
sleep 1.8; exit 6
parallel: This job failed:
sleep 2.1; exit 7
parallel: This job failed:
sleep 2.4; exit 8
parallel: This job failed:
sleep 2.7; exit 9
parallel: This job failed:
sleep 3; exit 10
50
parallel: This job failed:
sleep 0.3; exit 1
parallel: This job failed:
sleep 0.6; exit 2
parallel: This job failed:
sleep 0.9; exit 3
parallel: This job failed:
sleep 1.2; exit 4
parallel: This job failed:
sleep 1.5; exit 5
parallel: This job failed:
sleep 1.8; exit 6
parallel: This job failed:
sleep 2.1; exit 7
parallel: This job failed:
sleep 2.4; exit 8
parallel: This job failed:
sleep 2.7; exit 9
parallel: This job failed:
sleep 3; exit 10
50
### testhalt --halt now,success=0
parallel: This job succeeded:
sleep 0; exit 0
0
parallel: This job succeeded:
sleep 0; exit 0
0
### testhalt --halt now,success=1
parallel: This job succeeded:
sleep 0; exit 0
0
parallel: This job succeeded:
sleep 0; exit 0
0
### testhalt --halt now,success=2
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
0
### testhalt --halt now,success=30%
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
0
### testhalt --halt now,success=70%
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
50
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
50
### testhalt --halt soon,success=0
parallel: This job succeeded:
sleep 0; exit 0
parallel: Starting no more jobs. Waiting for 3 jobs to finish.
parallel: This job succeeded:
sleep 0; exit 0
parallel: Starting no more jobs. Waiting for 2 jobs to finish.
parallel: This job succeeded:
sleep 0; exit 0
parallel: Starting no more jobs. Waiting for 1 jobs to finish.
parallel: This job succeeded:
sleep 0; exit 0
0
parallel: This job succeeded:
sleep 0; exit 0
parallel: Starting no more jobs. Waiting for 3 jobs to finish.
0
### testhalt --halt soon,success=1
parallel: This job succeeded:
sleep 0; exit 0
parallel: Starting no more jobs. Waiting for 3 jobs to finish.
parallel: This job succeeded:
sleep 0; exit 0
parallel: Starting no more jobs. Waiting for 2 jobs to finish.
parallel: This job succeeded:
sleep 0; exit 0
parallel: Starting no more jobs. Waiting for 1 jobs to finish.
parallel: This job succeeded:
sleep 0; exit 0
0
parallel: This job succeeded:
sleep 0; exit 0
parallel: Starting no more jobs. Waiting for 3 jobs to finish.
0
### testhalt --halt soon,success=2
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: Starting no more jobs. Waiting for 3 jobs to finish.
parallel: This job succeeded:
sleep 0; exit 0
parallel: Starting no more jobs. Waiting for 2 jobs to finish.
parallel: This job succeeded:
sleep 0; exit 0
parallel: Starting no more jobs. Waiting for 1 jobs to finish.
parallel: This job succeeded:
sleep 0; exit 0
0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: Starting no more jobs. Waiting for 3 jobs to finish.
0
### testhalt --halt soon,success=30%
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: Starting no more jobs. Waiting for 3 jobs to finish.
parallel: This job succeeded:
sleep 0; exit 0
parallel: Starting no more jobs. Waiting for 2 jobs to finish.
parallel: This job succeeded:
sleep 0; exit 0
parallel: Starting no more jobs. Waiting for 1 jobs to finish.
parallel: This job succeeded:
sleep 0; exit 0
0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: Starting no more jobs. Waiting for 3 jobs to finish.
0
### testhalt --halt now,success=70%
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
50
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
parallel: This job succeeded:
sleep 0; exit 0
50
echo '**'
**
echo '### Test slow arguments generation - https://savannah.gnu.org/bugs/?32834'; seq 1 3 | parallel -j1 "sleep 2; echo {}" | parallel -kj2 echo
### Test slow arguments generation - https://savannah.gnu.org/bugs/?32834
1
@ -146,7 +661,7 @@ echo '**'
**
echo '### Are children killed if GNU Parallel receives TERM twice? There should be no sleep at the end'
### Are children killed if GNU Parallel receives TERM twice? There should be no sleep at the end
parallel -q bash -c 'sleep 120 & pid=$!; wait $pid' ::: 1 & T=$!; sleep 1; pstree $$; kill -TERM $T; sleep 1; pstree $$; kill -TERM $T; sleep 1; pstree $$; echo '**'
parallel -q bash -c 'sleep 120 & pid=$!; wait $pid' ::: 1 & T=$!; sleep 2; pstree $$; kill -TERM $T; sleep 1; pstree $$; kill -TERM $T; sleep 1; pstree $$; echo '**'
bash-+-perl---bash---sleep
`-pstree
bash-+-perl---bash---sleep
@ -158,7 +673,7 @@ parallel: Waiting for these 1 jobs to finish. Send SIGTERM again to stop now.
parallel: bash -c sleep\ 120\ \&\ pid=\$\!\;\ wait\ \$pid 1
echo '### Are children killed if GNU Parallel receives INT twice? There should be no sleep at the end'
### Are children killed if GNU Parallel receives INT twice? There should be no sleep at the end
parallel -q bash -c 'sleep 120 & pid=$!; wait $pid' ::: 1 & T=$!; sleep 1; pstree $$; kill -INT $T; sleep 1; pstree $$;
parallel -q bash -c 'sleep 120 & pid=$!; wait $pid' ::: 1 & T=$!; sleep 2; pstree $$; kill -INT $T; sleep 1; pstree $$;
bash-+-perl---bash---sleep
`-pstree
bash---pstree

12
testsuite/wanted-results/parallel-local-ssh1
View file

@ -1,7 +1,7 @@
echo '### Stop if all hosts are filtered and there are no hosts left to run on'
### Stop if all hosts are filtered and there are no hosts left to run on
stdout parallel --filter-hosts -S no-such.host echo ::: 1
parallel: Warning: Removed no-such.host
parallel: Warning: Removed no-such.host.
parallel: Error: There are no hosts left to run on.
echo '### Can csh propagate a variable containing \n'; export A=$(seq 3); parallel -S csh@localhost --env A bash -c \''echo "$A"'\' ::: dummy
### Can csh propagate a variable containing \n
@ -84,6 +84,16 @@ echo '### bug #40132: FreeBSD: --workdir . gives warning if . == $HOME'
### bug #40132: FreeBSD: --workdir . gives warning if . == $HOME
cd && parallel --workdir . -S lo pwd ::: ""
/home/tange
echo '### use function as $PARALLEL_SSH'
### use function as $PARALLEL_SSH
foossh() { echo "FOOSSH" >&2; ssh "$@"; }; export -f foossh; PARALLEL_SSH=foossh parallel -S 1/lo echo ::: 'Run through FOOSSH?'
Run through FOOSSH?
FOOSSH
echo '### use --ssh'
### use --ssh
barssh() { echo "BARSSH" >&2; ssh "$@"; }; export -f barssh; parallel --ssh barssh -S 1/lo echo ::: 'Run through BARSSH?'
Run through BARSSH?
BARSSH
echo '### test filename :'
### test filename :
echo content-of-: > :; echo : | parallel -j1 --trc {}.{.} -S parallel@lo '(echo remote-{}.{.};cat {}) > {}.{.}'; cat :.:; rm : :.:

3
testsuite/wanted-results/parallel-local164
View file

@ -344,7 +344,8 @@ bug #38439: "open files" with --files --pipe blocks after a while
19 of 20
20 of 20
parallel: Warning: Only enough file handles to run 1 jobs in parallel.
Running 'parallel -j0 -N1 --pipe parallel -j0' or raising ulimit -n or /etc/security/limits.conf may help.
parallel: Warning: Running 'parallel -j0 -N 1 --pipe parallel -j0' or
parallel: Warning: raising ulimit -n or /etc/security/limits.conf may help.
echo 'bug #34241: --pipe should not spawn unneeded processes - part 2'
bug #34241: --pipe should not spawn unneeded processes - part 2
seq 500 | parallel --tmpdir . -j10 --pipe --block 1k --files wc >/dev/null; ls *.par | wc -l; rm *.par; seq 500 | parallel --tmpdir . -j10 --pipe --block 1k --files --dry-run wc >/dev/null; echo No .par should exist; stdout ls *.par

3
testsuite/wanted-results/parallel-local22
View file

@ -95,7 +95,7 @@ echo '### bug #42913: Dont use $SHELL but the shell currently running'
### bug #42913: Dont use $SHELL but the shell currently running
echo '## Unknown shell => $SHELL (bash)'
## Unknown shell => $SHELL (bash)
parallel -j1 "cp \`which {}\` /tmp/SHELL; /tmp/SHELL -c 'parallel -Dinit echo ::: 1' | grep which;" ::: ash bash csh dash fdsh fish fizsh ksh ksh93 mksh pdksh posh rbash rush rzsh sash sh static-sh tcsh yash zsh; rm -f /tmp/SHELL /tmp/par*.par
parallel -j1 "cp \`which {}\` /tmp/SHELL; /tmp/SHELL -c 'parallel -Dinit echo ::: 1' | grep which;" ::: ash bash csh dash fish fizsh ksh ksh93 mksh pdksh posh rbash rush rzsh sash sh static-sh tcsh yash zsh; rm -f /tmp/SHELL /tmp/par*.par
shell? /bin/bash -c cp `which ash` /tmp/SHELL; /tmp/SHELL -c 'parallel -Dinit echo ::: 1' | grep which;
which bash => shell path /bin/bash
shell? /bin/bash -c cp `which bash` /tmp/SHELL; /tmp/SHELL -c 'parallel -Dinit echo ::: 1' | grep which;
@ -131,7 +131,6 @@ shell? /bin/bash -c cp `which yash` /tmp/SHELL; /tmp/SHELL -c 'parallel -Dinit e
which bash => shell path /bin/bash
shell? /bin/bash -c cp `which zsh` /tmp/SHELL; /tmp/SHELL -c 'parallel -Dinit echo ::: 1' | grep which;
which bash => shell path /bin/bash
/tmp/SHELL: -c: bad option(s)
Local configuration error occurred.
Contact the systems administrator for further assistance.
SHELL: applet not found

4
testsuite/wanted-results/parallel-local4
View file

@ -1,6 +1,6 @@
echo '### bug #36595: silent loss of input with --pipe and --sshlogin'
### bug #36595: silent loss of input with --pipe and --sshlogin
seq 10000 | xargs | parallel --pipe -S 10/localhost cat | wc
seq 10000 | xargs | parallel --pipe -S 10/localhost cat 2>/dev/null | wc
1 10000 48894
echo 'bug #36707: --controlmaster eats jobs'
bug #36707: --controlmaster eats jobs
@ -101,7 +101,7 @@ X : XXXXXXXXXX.XXX X.XXX X X X X true X
echo '### How do we deal with missing $HOME'
### How do we deal with missing $HOME
unset HOME; stdout perl -w $(which parallel) -k echo ::: 1 2 3
parallel: Warning: $HOME not set. Using /tmp
parallel: Warning: $HOME not set. Using /tmp.
1
2
3

8
testsuite/wanted-results/parallel-local5
View file

@ -1,7 +1,7 @@
### Test --pipe
echo '### Test 200M records with too small block'; ( echo start; seq 1 44 | parallel -uj1 cat /tmp/blocktest\;true; echo end; echo start; seq 1 44 | parallel -uj1 cat /tmp/blocktest\;true; echo end; echo start; seq 1 44 | parallel -uj1 cat /tmp/blocktest\;true; echo end; ) | stdout parallel -k --block 200m -j2 --pipe --recend 'end\n' wc -c | egrep -v '^0$'
### Test 200M records with too small block
parallel: Warning: A record was longer than 200000000. Increasing to --blocksize 260000001
parallel: Warning: A record was longer than 200000000. Increasing to --blocksize 260000001.
303111434
303111434
303111434
@ -148,10 +148,10 @@ echo -n 01a02a0a0a12a34a45a6a | parallel -k -j1 --blocksize 100 --pipe --recen
2>0a12a34a
3>45a6a
echo -n 01a02a0a0a12a34a45a6a | stdout parallel -k -j1 --blocksize 1 --pipe --recend a -N 3 'echo -n "$PARALLEL_SEQ>"; cat; echo; sleep 0.1'
parallel: Warning: A record was longer than 1. Increasing to --blocksize 3
parallel: Warning: A record was longer than 3. Increasing to --blocksize 5
parallel: Warning: A record was longer than 1. Increasing to --blocksize 3.
parallel: Warning: A record was longer than 3. Increasing to --blocksize 5.
parallel: Warning: A record was longer than 5. Increasing to --blocksize 8.
1>01a02a0a
parallel: Warning: A record was longer than 5. Increasing to --blocksize 8
2>0a12a34a
3>45a6a
echo '### Test 10M records with too big block'; ( echo start; seq 1 1 | parallel -uj1 cat /tmp/blocktest\;true; echo end; echo start; seq 1 1 | parallel -uj1 cat /tmp/blocktest\;true; echo end; echo start; seq 1 1 | parallel -uj1 cat /tmp/blocktest\;true; echo end; ) | stdout parallel -k --block 10M -j2 --pipe --recstart 'start\n' wc -c | egrep -v '^0$'

2
testsuite/wanted-results/parallel-local9
View file

@ -141,6 +141,7 @@ echo "# --recend '' --files --halt-on-error"
350eda13a37912d755c9d733d149bdaf -
echo '### Test of -j filename - non-existent file'; nice stdout parallel -j no_such_file echo ::: 1
### Test of -j filename - non-existent file
parallel: Error: Parsing of --jobs/-j/--max-procs/-P failed.
Usage:
parallel [options] [command [arguments]] < list_of_arguments
@ -178,7 +179,6 @@ please cite:
This helps funding further development; and it won't cost you a cent.
If you pay 10000 EUR you should feel free to use GNU Parallel without citing.
parallel: Error: Parsing of --jobs/-j/--max-procs/-P failed.
echo '### Test of -j filename'; echo 3 >/tmp/jobs_to_run1; parallel -j /tmp/jobs_to_run1 -v sleep {} ::: 10 8 6 5 4; # Should give 6 8 10 5 4
### Test of -j filename
sleep 6