mirror of
https://git.savannah.gnu.org/git/parallel.git
synced 2024-11-22 22:17:54 +00:00
Man page updated
This commit is contained in:
parent
fd44e4c793
commit
bbb1662979
106
src/parallel
106
src/parallel
|
@ -6,8 +6,7 @@ parallel - build and execute shell command lines from standard input in parallel
|
||||||
|
|
||||||
=head1 SYNOPSIS
|
=head1 SYNOPSIS
|
||||||
|
|
||||||
B<parallel> [-0cdEfghiIkmnpqrtuUvVX] [B<-I> str] [B<-j> num] [--silent]
|
B<parallel> [options] [I<command> [arguments]] [< list_of_arguments]
|
||||||
[command [arguments]] [< list_of_arguments]
|
|
||||||
|
|
||||||
=head1 DESCRIPTION
|
=head1 DESCRIPTION
|
||||||
|
|
||||||
|
@ -27,23 +26,21 @@ GNU B<parallel> makes sure output from the commands is the same output as
|
||||||
you would get had you run the commands sequentially. This makes it
|
you would get had you run the commands sequentially. This makes it
|
||||||
possible to use output from GNU B<parallel> as input for other programs.
|
possible to use output from GNU B<parallel> as input for other programs.
|
||||||
|
|
||||||
For each line of input GNU B<parallel> will execute B<command> with the
|
For each line of input GNU B<parallel> will execute I<command> with
|
||||||
line as arguments. If no B<command> is given the line of input is
|
the line as arguments. If no I<command> is given, the line of input is
|
||||||
executed. GNU B<parallel> can often be used as a substitute for B<xargs>
|
executed. Several lines will be run in parallel. GNU B<parallel> can
|
||||||
or B<cat | sh>.
|
often be used as a substitute for B<xargs> or B<cat | sh>.
|
||||||
|
|
||||||
Several lines will be run in parallel.
|
|
||||||
|
|
||||||
=over 9
|
=over 9
|
||||||
|
|
||||||
=item I<command>
|
=item I<command>
|
||||||
|
|
||||||
Command to execute. If B<command> or the following arguments contain
|
Command to execute. If I<command> or the following arguments contain
|
||||||
{} every instance will be substituted with the input line. Setting a
|
{} every instance will be substituted with the input line. Setting a
|
||||||
command also invokes B<-f>.
|
command also invokes B<-f>.
|
||||||
|
|
||||||
If B<command> is given, GNU B<parallel> will behave similar to B<xargs>. If
|
If I<command> is given, GNU B<parallel> will behave similar to B<xargs>. If
|
||||||
B<command> is not given GNU B<parallel> will behave similar to B<cat | sh>.
|
I<command> is not given GNU B<parallel> will behave similar to B<cat | sh>.
|
||||||
|
|
||||||
|
|
||||||
=item B<{}>
|
=item B<{}>
|
||||||
|
@ -56,12 +53,15 @@ changed with B<-I>.
|
||||||
=item B<{.}>
|
=item B<{.}>
|
||||||
|
|
||||||
Input line without extension. This is a specialized replacement string
|
Input line without extension. This is a specialized replacement string
|
||||||
with the extension removed. It will remove from the last B<.> till the
|
with the extension removed. If the input line contains B<.> after the
|
||||||
end of line of each input line and replace {.} with the
|
last B</> the last B<.> till the end of the string will be removed and
|
||||||
remaining. E.g. I<foo.jpg> becomes I<foo>. If the input line does
|
B<{.}> will be replaced with the remaining. E.g. I<foo.jpg> becomes
|
||||||
not contain B<.> it will remain unchanged.
|
I<foo>, I<subdir/foo.jpg> becomes I<subdir/foo>, I<sub.dir/foo.jpg>
|
||||||
|
becomes I<sub.dir/foo>, I<sub.dir/bar> remains I<sub.dir/bar>. If the
|
||||||
|
input line does not contain B<.> it will remain unchanged.
|
||||||
|
|
||||||
{.} can be used the same places as {}.
|
B<{.}> can be used the same places as B<{}>. The replacement string
|
||||||
|
B<{.}> can be changed with B<-U>.
|
||||||
|
|
||||||
|
|
||||||
=item B<--null>
|
=item B<--null>
|
||||||
|
@ -84,18 +84,18 @@ Otherwise, stdin is redirected from /dev/null.
|
||||||
|
|
||||||
=item B<--cleanup>
|
=item B<--cleanup>
|
||||||
|
|
||||||
Remove transfered files. B<--cleanup> will remove the transfered files
|
Remove transferred files. B<--cleanup> will remove the transferred files
|
||||||
on the remote server after processing is done.
|
on the remote server after processing is done.
|
||||||
|
|
||||||
find log -name '*gz' | parallel \
|
find log -name '*gz' | parallel \
|
||||||
--sshlogin server.example.com --transfer --return {.}.bz2 \
|
--sshlogin server.example.com --transfer --return {.}.bz2 \
|
||||||
--cleanup "zcat {} | bzip -9 >{.}.bz2"
|
--cleanup "zcat {} | bzip -9 >{.}.bz2"
|
||||||
|
|
||||||
With B<--transfer> the file transfered to the remote server will be
|
With B<--transfer> the file transferred to the remote server will be
|
||||||
removed on the remote server. Directories created will not be removed
|
removed on the remote server. Directories created will not be removed
|
||||||
- even if they are empty.
|
- even if they are empty.
|
||||||
|
|
||||||
With B<--return> the file transfered from the remote server will be
|
With B<--return> the file transferred from the remote server will be
|
||||||
removed on the remote server. Directories created will not be removed
|
removed on the remote server. Directories created will not be removed
|
||||||
- even if they are empty.
|
- even if they are empty.
|
||||||
|
|
||||||
|
@ -108,7 +108,7 @@ B<--cleanup> is ignored when not used with B<--transfer> or B<--return>.
|
||||||
|
|
||||||
Line is a command. The input line contains more than one argument or
|
Line is a command. The input line contains more than one argument or
|
||||||
the input line needs to be evaluated by the shell. This is the default
|
the input line needs to be evaluated by the shell. This is the default
|
||||||
if B<command> is not set. Can be reversed with B<-f>.
|
if I<command> is not set. Can be reversed with B<-f>.
|
||||||
|
|
||||||
|
|
||||||
=item B<--delimiter> I<delim>
|
=item B<--delimiter> I<delim>
|
||||||
|
@ -151,7 +151,7 @@ B<-E> nor B<-e> is used, no end of file string is used.
|
||||||
|
|
||||||
Line is a filename. The input line contains a filename that will be
|
Line is a filename. The input line contains a filename that will be
|
||||||
quoted so it is not evaluated by the shell. This is the default if
|
quoted so it is not evaluated by the shell. This is the default if
|
||||||
B<command> is set. Can be reversed with B<-c>.
|
I<command> is set. Can be reversed with B<-c>.
|
||||||
|
|
||||||
|
|
||||||
=item B<--group>
|
=item B<--group>
|
||||||
|
@ -205,7 +205,7 @@ Run up to N jobs in parallel. 0 means as many as possible. Default is 9.
|
||||||
Add N to the number of CPU cores. Run this many jobs in parallel. For
|
Add N to the number of CPU cores. Run this many jobs in parallel. For
|
||||||
compute intensive jobs B<-j> +0 is useful as it will run
|
compute intensive jobs B<-j> +0 is useful as it will run
|
||||||
number-of-cpu-cores jobs in parallel. See also
|
number-of-cpu-cores jobs in parallel. See also
|
||||||
--use-cpus-instead-of-cores.
|
B<--use-cpus-instead-of-cores>.
|
||||||
|
|
||||||
|
|
||||||
=item B<--jobs> I<-N>
|
=item B<--jobs> I<-N>
|
||||||
|
@ -218,7 +218,7 @@ number-of-cpu-cores jobs in parallel. See also
|
||||||
|
|
||||||
Subtract N from the number of CPU cores. Run this many jobs in parallel.
|
Subtract N from the number of CPU cores. Run this many jobs in parallel.
|
||||||
If the evaluated number is less than 1 then 1 will be used. See also
|
If the evaluated number is less than 1 then 1 will be used. See also
|
||||||
--use-cpus-instead-of-cores.
|
B<--use-cpus-instead-of-cores>.
|
||||||
|
|
||||||
|
|
||||||
=item B<--jobs> I<N>%
|
=item B<--jobs> I<N>%
|
||||||
|
@ -231,7 +231,7 @@ If the evaluated number is less than 1 then 1 will be used. See also
|
||||||
|
|
||||||
Multiply N% with the number of CPU cores. Run this many jobs in parallel.
|
Multiply N% with the number of CPU cores. Run this many jobs in parallel.
|
||||||
If the evaluated number is less than 1 then 1 will be used. See also
|
If the evaluated number is less than 1 then 1 will be used. See also
|
||||||
--use-cpus-instead-of-cores.
|
B<--use-cpus-instead-of-cores>.
|
||||||
|
|
||||||
|
|
||||||
=item B<--keeporder>
|
=item B<--keeporder>
|
||||||
|
@ -267,7 +267,7 @@ Print the number of physical CPUs and exit (used by GNU B<parallel>
|
||||||
itself to determine the number of physical CPUs on remote machines).
|
itself to determine the number of physical CPUs on remote machines).
|
||||||
|
|
||||||
|
|
||||||
=item B<--number-of-cores> (not implemented)
|
=item B<--number-of-cores>
|
||||||
|
|
||||||
Print the number of cores and exit (used by GNU B<parallel> itself to determine the
|
Print the number of cores and exit (used by GNU B<parallel> itself to determine the
|
||||||
number of cores on remote machines).
|
number of cores on remote machines).
|
||||||
|
@ -286,7 +286,7 @@ with 'y' or 'Y'. Implies B<-t>.
|
||||||
|
|
||||||
=item B<-q>
|
=item B<-q>
|
||||||
|
|
||||||
Quote B<command>. This will quote the command line so special
|
Quote I<command>. This will quote the command line so special
|
||||||
characters are not interpreted by the shell. See the section
|
characters are not interpreted by the shell. See the section
|
||||||
QUOTING. Most people will never need this. Quoting is disabled by
|
QUOTING. Most people will never need this. Quoting is disabled by
|
||||||
default.
|
default.
|
||||||
|
@ -302,7 +302,7 @@ If the standard input only contains whitespace, do not run the command.
|
||||||
|
|
||||||
Transfer files from remote servers. B<--return> is used with
|
Transfer files from remote servers. B<--return> is used with
|
||||||
B<--sshlogin> when the arguments are files on the remote servers. When
|
B<--sshlogin> when the arguments are files on the remote servers. When
|
||||||
processing is done the file I<filename> will be transfered
|
processing is done the file I<filename> will be transferred
|
||||||
from the remote server using B<rsync> and will be put relative to
|
from the remote server using B<rsync> and will be put relative to
|
||||||
the default login dir. E.g.
|
the default login dir. E.g.
|
||||||
|
|
||||||
|
@ -320,7 +320,7 @@ This will transfer the file I</tmp/foo/bar.out> from the server
|
||||||
I<server.example.com> to the file I</tmp/foo/bar.out> after running
|
I<server.example.com> to the file I</tmp/foo/bar.out> after running
|
||||||
B<touch /tmp/foo/bar.out> on I<server.example.com>.
|
B<touch /tmp/foo/bar.out> on I<server.example.com>.
|
||||||
|
|
||||||
Multiple files can be transfered by repeating the options multiple
|
Multiple files can be transferred by repeating the options multiple
|
||||||
times:
|
times:
|
||||||
|
|
||||||
echo /tmp/foo/bar.txt | \
|
echo /tmp/foo/bar.txt | \
|
||||||
|
@ -378,6 +378,8 @@ For examples: see B<--sshloginfile>.
|
||||||
|
|
||||||
The remote host must have GNU B<parallel> installed.
|
The remote host must have GNU B<parallel> installed.
|
||||||
|
|
||||||
|
B<--sshlogin> is known to cause problems with B<-m> and B<-X>.
|
||||||
|
|
||||||
|
|
||||||
=item B<--sshloginfile> I<filename> (beta testing)
|
=item B<--sshloginfile> I<filename> (beta testing)
|
||||||
|
|
||||||
|
@ -393,7 +395,9 @@ lines. Empty lines and lines starting with '#' are ignored. Example:
|
||||||
4/ssh -p 2222 quadserver.example.net
|
4/ssh -p 2222 quadserver.example.net
|
||||||
# Use a different ssh program
|
# Use a different ssh program
|
||||||
myssh -p 2222 -l compute hexacpu.example.net
|
myssh -p 2222 -l compute hexacpu.example.net
|
||||||
/usr/local/bin/myssh -p 2222 -l compute hexacpu.example.net
|
# Use a different ssh program with default number of cores
|
||||||
|
//usr/local/bin/myssh -p 2222 -l compute hexacpu.example.net
|
||||||
|
# Use a different ssh program with 6 cores
|
||||||
6//usr/local/bin/myssh -p 2222 -l compute hexacpu.example.net
|
6//usr/local/bin/myssh -p 2222 -l compute hexacpu.example.net
|
||||||
# Assume 16 cores on the local machine
|
# Assume 16 cores on the local machine
|
||||||
16/:
|
16/:
|
||||||
|
@ -418,8 +422,8 @@ See also B<-v>.
|
||||||
=item B<--transfer>
|
=item B<--transfer>
|
||||||
|
|
||||||
Transfer files to remote servers. B<--transfer> is used with
|
Transfer files to remote servers. B<--transfer> is used with
|
||||||
B<--sshlogin> when the arguments are files and should be transfered to
|
B<--sshlogin> when the arguments are files and should be transferred to
|
||||||
the remote servers. The files will be transfered using B<rsync> and
|
the remote servers. The files will be transferred using B<rsync> and
|
||||||
will be put relative to the default login dir. E.g.
|
will be put relative to the default login dir. E.g.
|
||||||
|
|
||||||
echo foo/bar.txt | parallel \
|
echo foo/bar.txt | parallel \
|
||||||
|
@ -445,7 +449,7 @@ B<--transfer> is ignored when used with B<--sshlogin :> or when not used with B<
|
||||||
|
|
||||||
Transfer, Return, Cleanup. Short hand for:
|
Transfer, Return, Cleanup. Short hand for:
|
||||||
|
|
||||||
--transfer --return filename --cleanup
|
B<--transfer> B<--return> I<filename> B<--cleanup>
|
||||||
|
|
||||||
|
|
||||||
=item B<--ungroup>
|
=item B<--ungroup>
|
||||||
|
@ -494,19 +498,19 @@ Print the version GNU B<parallel> and exit.
|
||||||
=item B<-m>
|
=item B<-m>
|
||||||
|
|
||||||
Multiple. Insert as many arguments as the command line length
|
Multiple. Insert as many arguments as the command line length
|
||||||
permits. If {} is not used the arguments will be appended to the line.
|
permits. If B<{}> is not used the arguments will be appended to the
|
||||||
If {} is used multiple times each {} will be replaced with all the
|
line. If B<{}> is used multiple times each B<{}> will be replaced
|
||||||
arguments.
|
with all the arguments.
|
||||||
|
|
||||||
Support for B<-m> with B<--sshlogin> is limited and may fail.
|
Support for B<-m> with B<--sshlogin> is limited and may fail.
|
||||||
|
|
||||||
|
|
||||||
=item B<-X>
|
=item B<-X>
|
||||||
|
|
||||||
xargs with context replace. This works like B<-m> except if {} is part
|
xargs with context replace. This works like B<-m> except if B<{}> is part
|
||||||
of a word (like I<pic{}.jpg>) then the whole word will be
|
of a word (like I<pic{}.jpg>) then the whole word will be
|
||||||
repeated. Normally B<-X> will do the right thing, whereas B<-m> can
|
repeated. Normally B<-X> will do the right thing, whereas B<-m> can
|
||||||
give surprising results if {} is used as part of a word.
|
give surprising results if B<{}> is used as part of a word.
|
||||||
|
|
||||||
Support for B<-X> with B<--sshlogin> is limited and may fail.
|
Support for B<-X> with B<--sshlogin> is limited and may fail.
|
||||||
|
|
||||||
|
@ -577,7 +581,7 @@ To do it recursively use B<find>:
|
||||||
|
|
||||||
B<find . -name '*.jpg' | parallel -j +0 convert -geometry 120 {} {}_thumb.jpg>
|
B<find . -name '*.jpg' | parallel -j +0 convert -geometry 120 {} {}_thumb.jpg>
|
||||||
|
|
||||||
Notice how the argument has to start with {} as {} will include path
|
Notice how the argument has to start with B<{}> as B<{}> will include path
|
||||||
(e.g. running B<convert -geometry 120 ./foo/bar.jpg
|
(e.g. running B<convert -geometry 120 ./foo/bar.jpg
|
||||||
thumb_./foo/bar.jpg> would clearly be wrong). The command will
|
thumb_./foo/bar.jpg> would clearly be wrong). The command will
|
||||||
generate files like ./foo/bar.jpg_thumb.jpg.
|
generate files like ./foo/bar.jpg_thumb.jpg.
|
||||||
|
@ -617,15 +621,15 @@ B<ls | parallel '(echo -n {}" "; ls {}|wc -l) >> B<{}.dir'>
|
||||||
|
|
||||||
=head1 EXAMPLE: Removing file extension when processing files
|
=head1 EXAMPLE: Removing file extension when processing files
|
||||||
|
|
||||||
When processing files removing the file extension using {.} is often
|
When processing files removing the file extension using B<{.}> is
|
||||||
useful.
|
often useful.
|
||||||
|
|
||||||
Create a directory for each zip-file and unzip it in that dir:
|
Create a directory for each zip-file and unzip it in that dir:
|
||||||
|
|
||||||
B<ls *zip | parallel 'mkdir {.}; cd {.}; unzip ../{}'>
|
B<ls *zip | parallel 'mkdir {.}; cd {.}; unzip ../{}'>
|
||||||
|
|
||||||
Recompress all .gz files in current directory using B<bzip2> running 1
|
Recompress all .gz files in current directory using B<bzip2> running 1
|
||||||
job per CPU in parallel:
|
job per CPU core in parallel:
|
||||||
|
|
||||||
B<ls *.gz | parallel -j+0 "zcat {} | bzip2 >>B<{.}.bz2 && rm {}">
|
B<ls *.gz | parallel -j+0 "zcat {} | bzip2 >>B<{.}.bz2 && rm {}">
|
||||||
|
|
||||||
|
@ -691,7 +695,7 @@ B<(echo foss.org.my; echo debian.org; echo freenetproject.org) | parallel -u tra
|
||||||
Normally the output of a job will be printed as soon as it
|
Normally the output of a job will be printed as soon as it
|
||||||
completes. Sometimes you want the order of the output to remain the
|
completes. Sometimes you want the order of the output to remain the
|
||||||
same as the order of the input. This is often important, if the output
|
same as the order of the input. This is often important, if the output
|
||||||
is used for input for another system. B<-k> will make sure the order of
|
is used as input for another system. B<-k> will make sure the order of
|
||||||
output will be in the same order as input even if later jobs end
|
output will be in the same order as input even if later jobs end
|
||||||
before earlier jobs.
|
before earlier jobs.
|
||||||
|
|
||||||
|
@ -780,25 +784,25 @@ I<$HOME/logs>. On I<server.example.com> the file will be recompressed
|
||||||
using B<zcat> and B<bzip2> resulting in the corresponding file with
|
using B<zcat> and B<bzip2> resulting in the corresponding file with
|
||||||
I<.gz> replaced with I<.bz2>.
|
I<.gz> replaced with I<.bz2>.
|
||||||
|
|
||||||
If you want the file to be transfered back to the local machine add
|
If you want the file to be transferred back to the local machine add
|
||||||
I<--return {.}.bz2>:
|
I<--return {.}.bz2>:
|
||||||
|
|
||||||
find logs/ -name '*.gz' | \
|
find logs/ -name '*.gz' | \
|
||||||
parallel --sshlogin server.example.com \
|
parallel --sshlogin server.example.com \
|
||||||
--transfer --return {.}.bz2 "zcat {} | bzip2 -9 >{.}.bz2"
|
--transfer --return {.}.bz2 "zcat {} | bzip2 -9 >{.}.bz2"
|
||||||
|
|
||||||
After the recompressing is done the I<.bz2>-file is transfered back to
|
After the recompressing is done the I<.bz2>-file is transferred back to
|
||||||
the local machine and put next to the original I<.gz>-file.
|
the local machine and put next to the original I<.gz>-file.
|
||||||
|
|
||||||
If you want to delete the transfered files on the remote machine add
|
If you want to delete the transferred files on the remote machine add
|
||||||
I<--cleanup>. This will remove both the file transfered to the remote
|
I<--cleanup>. This will remove both the file transferred to the remote
|
||||||
machine and the files transfered from the remote machine:
|
machine and the files transferred from the remote machine:
|
||||||
|
|
||||||
find logs/ -name '*.gz' | \
|
find logs/ -name '*.gz' | \
|
||||||
parallel --sshlogin server.example.com \
|
parallel --sshlogin server.example.com \
|
||||||
--transfer --return {.}.bz2 --cleanup "zcat {} | bzip2 -9 >{.}.bz2"
|
--transfer --return {.}.bz2 --cleanup "zcat {} | bzip2 -9 >{.}.bz2"
|
||||||
|
|
||||||
If you want run one several servers add the servers to I<--sshlogin>
|
If you want run on several servers add the servers to I<--sshlogin>
|
||||||
either using ',' or multiple I<--sshlogin>:
|
either using ',' or multiple I<--sshlogin>:
|
||||||
|
|
||||||
find logs/ -name '*.gz' | \
|
find logs/ -name '*.gz' | \
|
||||||
|
@ -852,7 +856,7 @@ To run 100 processes simultaneously do:
|
||||||
|
|
||||||
B<parallel -j 100 < jobs_to_run>
|
B<parallel -j 100 < jobs_to_run>
|
||||||
|
|
||||||
As there is not a B<command> the option B<-c> is default because the
|
As there is not a I<command> the option B<-c> is default because the
|
||||||
jobs needs to be evaluated by the shell.
|
jobs needs to be evaluated by the shell.
|
||||||
|
|
||||||
|
|
||||||
|
@ -888,7 +892,7 @@ B<sh: -c: line 0: syntax error near unexpected token>
|
||||||
then you might try using B<-q>.
|
then you might try using B<-q>.
|
||||||
|
|
||||||
If you are using B<bash> process substitution like B<<(cat foo)> then
|
If you are using B<bash> process substitution like B<<(cat foo)> then
|
||||||
you may try B<-q> and prepending B<command> with B<bash -c>:
|
you may try B<-q> and prepending I<command> with B<bash -c>:
|
||||||
|
|
||||||
B<ls | parallel -q bash -c 'wc -c <(echo {})'>
|
B<ls | parallel -q bash -c 'wc -c <(echo {})'>
|
||||||
|
|
||||||
|
@ -962,6 +966,8 @@ B<xargs> has no support for keeping the order of the output, therefore
|
||||||
if running jobs in parallel using B<xargs> the output of the second
|
if running jobs in parallel using B<xargs> the output of the second
|
||||||
job cannot be postponed till the first job is done.
|
job cannot be postponed till the first job is done.
|
||||||
|
|
||||||
|
B<xargs> has no support for running jobs on remote machines.
|
||||||
|
|
||||||
B<xargs> has no support for context replace, so you will have to create the
|
B<xargs> has no support for context replace, so you will have to create the
|
||||||
arguments.
|
arguments.
|
||||||
|
|
||||||
|
|
Loading…
Reference in a new issue