parallel: Use unit prefix for more arguments that take a number.

This commit is contained in:
Ole Tange 2021-08-03 22:50:03 +02:00
parent 80544cbfc8
commit 42aec9a964
21 changed files with 328 additions and 56 deletions

View file

@ -7,6 +7,9 @@ SUBDIRS = src
test: test:
cd testsuite; make 1 cd testsuite; make 1
web:
cd src; make web
upload: upload:
eval `gpg-agent --daemon` eval `gpg-agent --daemon`
# make sure $YYYYMMDD is set # make sure $YYYYMMDD is set

View file

@ -758,6 +758,9 @@ uninstall-am:
test: test:
cd testsuite; make 1 cd testsuite; make 1
web:
cd src; make web
upload: upload:
eval `gpg-agent --daemon` eval `gpg-agent --daemon`
# make sure $YYYYMMDD is set # make sure $YYYYMMDD is set

View file

@ -4,6 +4,19 @@
Quote of the month: Quote of the month:
Have you heard of our lord and saviour GNU parallel? https://gnu.org/software/pa
Have you heard of our lord and saviour GNU parallel?
-- kxyne @Kxyne@twitter
GNU parallel is a severely underrated tool for just spawning a
shitload of tasks. I use it every day and fucking love it.
-- Jane @serialexpjane@twitter
Installing GNU parallel is the second thing I do on any new Linux
machine, right after uninstalling pulseaudio.
-- Andy Cooper @integgroll@twitter
Parallel is amazing! Parallel is amazing!
-- fatboy93@reddit -- fatboy93@reddit

View file

@ -144,6 +144,7 @@ http://www.gnu.org/software/parallel/parset.html
http://www.gnu.org/software/parallel/parsort.html http://www.gnu.org/software/parallel/parsort.html
http://www.gnu.org/software/parallel/niceload.html http://www.gnu.org/software/parallel/niceload.html
make web
cp -r src/*.pdf src/sphinx/build/html/* ../parallel-web/parallel cp -r src/*.pdf src/sphinx/build/html/* ../parallel-web/parallel
cp ../parallel-web/parallel/parallel.html ../parallel-web/parallel/man.html cp ../parallel-web/parallel/parallel.html ../parallel-web/parallel/man.html
@ -228,8 +229,10 @@ http://freshmeat.net/projects/parallel/releases/new
https://joindiaspora.com/stream https://joindiaspora.com/stream
New release of #GNU Parallel pi.dk/0 New in this release pi.dk/2 See the intro videos pi.dk/1 New release of #GNU Parallel pi.dk/0
10 secs installation: wget -O - pi.dk/3|bash New in this release pi.dk/2
See the intro videos pi.dk/1
10 secs installation: https://git.savannah.gnu.org/cgit/parallel.git/tree/README
[x] Twitter [x] Twitter
Aspect: Public Aspect: Public
@ -252,9 +255,9 @@ from:tange@gnu.org
to:parallel@gnu.org, bug-parallel@gnu.org to:parallel@gnu.org, bug-parallel@gnu.org
stable-bcc: Jesse Alama <jessealama@fastmail.fm> stable-bcc: Jesse Alama <jessealama@fastmail.fm>
Subject: GNU Parallel 20210722 ('Blue Unity') released <<[stable]>> Subject: GNU Parallel 20210822 ('turkish fire/greek fire/tysk syndflod/Tunesia') released <<[stable]>>
GNU Parallel 20210722 ('Blue Unity') <<[stable]>> has been released. It is available for download at: lbry://@GnuParallel:4 GNU Parallel 20210822 ('') <<[stable]>> has been released. It is available for download at: lbry://@GnuParallel:4
<<No new functionality was introduced so this is a good candidate for a stable release.>> <<No new functionality was introduced so this is a good candidate for a stable release.>>
@ -271,6 +274,12 @@ New in this release:
News about GNU Parallel: News about GNU Parallel:
<<>> <<>>
https://cdmana.com/2021/07/20210728132344693t.html
https://github.com/gibbslab/biobash uses GNU Parallel
https://net2.com/how-to-execute-commands-in-parallel-in-linux/
Get the book: GNU Parallel 2018 http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html Get the book: GNU Parallel 2018 http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html

View file

@ -34,6 +34,12 @@ doc_DATA = parallel.html env_parallel.html sem.html sql.html \
parallel_cheat_bw.pdf parallel_cheat_bw.pdf
endif endif
web: sphinx
true
sphinx: *.rst
cd sphinx && make && cd ..
# Build documentation file if the tool to build exists. # Build documentation file if the tool to build exists.
# Otherwise: Use the distributed version # Otherwise: Use the distributed version
parallel.1: parallel.pod parallel.1: parallel.pod
@ -342,12 +348,6 @@ parallel_cheat_bw.pdf: parallel_cheat_bw.fodt
sem: parallel sem: parallel
ln -fs parallel sem ln -fs parallel sem
sphinx: parallel.rst env_parallel.rst sem.rst sql.rst \
niceload.rst parallel_tutorial.rst parallel_book.rst \
parallel_design.rst parallel_alternatives.rst parcat.rst \
parset.rst parsort.rst
cd sphinx && make
DISTCLEANFILES = parallel.1 env_parallel.1 sem.1 sql.1 niceload.1 \ DISTCLEANFILES = parallel.1 env_parallel.1 sem.1 sql.1 niceload.1 \
parallel_tutorial.7 parallel_book.7 parallel_design.7 \ parallel_tutorial.7 parallel_book.7 parallel_design.7 \
parallel_alternatives.7 parcat.1 parset.1 parsort.1 \ parallel_alternatives.7 parcat.1 parset.1 parsort.1 \

View file

@ -385,7 +385,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210722 (GNU parallel `parallel --minversion 1`)" echo "parset 20210723 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -384,7 +384,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210722 (GNU parallel `parallel --minversion 1`)" echo "parset 20210723 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -385,7 +385,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210722 (GNU parallel `parallel --minversion 1`)" echo "parset 20210723 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -363,7 +363,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210722 (GNU parallel `parallel --minversion 1`)" echo "parset 20210723 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -365,7 +365,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210722 (GNU parallel `parallel --minversion 1`)" echo "parset 20210723 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -390,7 +390,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210722 (GNU parallel `parallel --minversion 1`)" echo "parset 20210723 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -355,7 +355,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210722 (GNU parallel `parallel --minversion 1`)" echo "parset 20210723 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -26,7 +26,7 @@
use strict; use strict;
use Getopt::Long; use Getopt::Long;
$Global::progname="niceload"; $Global::progname="niceload";
$Global::version = 20210722; $Global::version = 20210723;
Getopt::Long::Configure("bundling","require_order"); Getopt::Long::Configure("bundling","require_order");
get_options_from_array(\@ARGV) || die_usage(); get_options_from_array(\@ARGV) || die_usage();
if($opt::version) { if($opt::version) {

View file

@ -1617,18 +1617,18 @@ sub options_hash() {
# xargs-compatibility - implemented, man, testsuite # xargs-compatibility - implemented, man, testsuite
"max-procs|P=s" => \$opt::jobs, "max-procs|P=s" => \$opt::jobs,
"delimiter|d=s" => \$opt::d, "delimiter|d=s" => \$opt::d,
"max-chars|s=i" => \$opt::max_chars, "max-chars|s=s" => \$opt::max_chars,
"arg-file|a=s" => \@opt::a, "arg-file|a=s" => \@opt::a,
"no-run-if-empty|r" => \$opt::r, "no-run-if-empty|r" => \$opt::r,
"replace|i:s" => \$opt::i, "replace|i:s" => \$opt::i,
"E=s" => \$opt::eof, "E=s" => \$opt::eof,
"eof|e:s" => \$opt::eof, "eof|e:s" => \$opt::eof,
"max-args|maxargs|n=i" => \$opt::max_args, "max-args|maxargs|n=s" => \$opt::max_args,
"max-replace-args|N=i" => \$opt::max_replace_args, "max-replace-args|N=s" => \$opt::max_replace_args,
"colsep|col-sep|C=s" => \$opt::colsep, "colsep|col-sep|C=s" => \$opt::colsep,
"csv"=> \$opt::csv, "csv"=> \$opt::csv,
"help|h" => \$opt::help, "help|h" => \$opt::help,
"L=f" => \$opt::L, "L=s" => \$opt::L,
"max-lines|l:f" => \$opt::max_lines, "max-lines|l:f" => \$opt::max_lines,
"interactive|p" => \$opt::interactive, "interactive|p" => \$opt::interactive,
"verbose|t" => \$opt::verbose, "verbose|t" => \$opt::verbose,
@ -1801,6 +1801,7 @@ sub parse_options(@) {
if(defined $opt::verbose) { $Global::stderr_verbose = 1; } if(defined $opt::verbose) { $Global::stderr_verbose = 1; }
if(defined $opt::eof) { $Global::end_of_file_string = $opt::eof; } if(defined $opt::eof) { $Global::end_of_file_string = $opt::eof; }
if(defined $opt::max_args) { if(defined $opt::max_args) {
$opt::max_args = multiply_binary_prefix($opt::max_args);
$Global::max_number_of_args = $opt::max_args; $Global::max_number_of_args = $opt::max_args;
} }
if(defined $opt::blocktimeout) { if(defined $opt::blocktimeout) {
@ -1856,6 +1857,9 @@ sub parse_options(@) {
if(defined $opt::max_line_length_allowed) { if(defined $opt::max_line_length_allowed) {
print Limits::Command::real_max_length(),"\n"; wait_and_exit(0); print Limits::Command::real_max_length(),"\n"; wait_and_exit(0);
} }
if(defined $opt::max_chars) {
$opt::max_chars = multiply_binary_prefix($opt::max_chars);
}
if(defined $opt::version) { version(); wait_and_exit(0); } if(defined $opt::version) { version(); wait_and_exit(0); }
if(defined $opt::record_env) { record_env(); wait_and_exit(0); } if(defined $opt::record_env) { record_env(); wait_and_exit(0); }
if(defined $opt::show_limits) { show_limits(); } if(defined $opt::show_limits) { show_limits(); }
@ -1989,10 +1993,14 @@ sub parse_options(@) {
$opt::max_lines = 1; $opt::max_lines = 1;
$opt::null = 1; $opt::null = 1;
$/ = "\0"; $/ = "\0";
} elsif ($opt::max_lines == 0) { } else {
$opt::max_lines = multiply_binary_prefix($opt::max_lines);
if ($opt::max_lines == 0) {
# If not given (or if 0 is given) => 1 # If not given (or if 0 is given) => 1
$opt::max_lines = 1; $opt::max_lines = 1;
} }
}
$Global::max_lines = $opt::max_lines; $Global::max_lines = $opt::max_lines;
if(not $opt::pipe) { if(not $opt::pipe) {
# --pipe -L means length of record - not max_number_of_args # --pipe -L means length of record - not max_number_of_args
@ -2002,6 +2010,7 @@ sub parse_options(@) {
# Read more than one arg at a time (-L, -N) # Read more than one arg at a time (-L, -N)
if(defined $opt::L) { if(defined $opt::L) {
$opt::L = multiply_binary_prefix($opt::L);
$Global::max_lines = $opt::L; $Global::max_lines = $opt::L;
if(not $opt::pipe) { if(not $opt::pipe) {
# --pipe -L means length of record - not max_number_of_args # --pipe -L means length of record - not max_number_of_args
@ -2009,6 +2018,7 @@ sub parse_options(@) {
} }
} }
if(defined $opt::max_replace_args) { if(defined $opt::max_replace_args) {
$opt::max_replace_args = multiply_binary_prefix($opt::max_replace_args);
$Global::max_number_of_args = $opt::max_replace_args; $Global::max_number_of_args = $opt::max_replace_args;
$Global::ContextReplace = 1; $Global::ContextReplace = 1;
} }
@ -2223,7 +2233,7 @@ sub check_invalid_option_combinations() {
sub init_globals() { sub init_globals() {
# Defaults: # Defaults:
$Global::version = 20210722; $Global::version = 20210723;
$Global::progname = 'parallel'; $Global::progname = 'parallel';
$::name = "GNU Parallel"; $::name = "GNU Parallel";
$Global::infinity = 2**31; $Global::infinity = 2**31;
@ -5361,6 +5371,8 @@ env_parallel --session
env_parallel -k echo ::: Put your code here env_parallel -k echo ::: Put your code here
parset p,y,c,h -k echo ::: Put your code here parset p,y,c,h -k echo ::: Put your code here
echo $p $y $c $h echo $p $y $c $h
echo You can also activate GNU Parallel for interactive use by:
echo . "$0"
!; !;
} else { } else {
::error("Cannot open $0"); ::error("Cannot open $0");

View file

@ -608,10 +608,7 @@ alternatives in B<man parallel_alternatives>.
=item B<--block-size> I<size> =item B<--block-size> I<size>
Size of block in bytes to read at a time. The I<size> can be postfixed Size of block in bytes to read at a time. The I<size> can be postfixed
with K, M, G, T, P, E, k, m, g, t, p, or e which would multiply the with K, M, G, T, P, k, m, g, t, or p (see UNIT PREFIX).
size with 1024, 1048576, 1073741824, 1099511627776, 1125899906842624,
1152921504606846976, 1000, 1000000, 1000000000, 1000000000000,
1000000000000000, or 1000000000000000000 respectively.
GNU B<parallel> tries to meet the block size but can be off by the GNU B<parallel> tries to meet the block size but can be off by the
length of one record. For performance reasons I<size> should be bigger length of one record. For performance reasons I<size> should be bigger
@ -1283,6 +1280,9 @@ continued on the next input line.
B<-L 0> means read one line, but insert 0 arguments on the command B<-L 0> means read one line, but insert 0 arguments on the command
line. line.
I<recsize> can be postfixed with K, M, G, T, P, k, m, g, t, or p (see
UNIT PREFIX).
Implies B<-X> unless B<-m>, B<--xargs>, or B<--pipe> is set. Implies B<-X> unless B<-m>, B<--xargs>, or B<--pipe> is set.
@ -1437,10 +1437,7 @@ most likely do what is needed.
=item B<--memfree> I<size> =item B<--memfree> I<size>
Minimum memory free when starting another job. The I<size> can be Minimum memory free when starting another job. The I<size> can be
postfixed with K, M, G, T, P, k, m, g, t, or p which would multiply postfixed with K, M, G, T, P, k, m, g, t, or p (see UNIT PREFIX).
the size with 1024, 1048576, 1073741824, 1099511627776,
1125899906842624, 1000, 1000000, 1000000000, 1000000000000, or
1000000000000000, respectively.
If the jobs take up very different amount of RAM, GNU B<parallel> will If the jobs take up very different amount of RAM, GNU B<parallel> will
only start as many as there is memory for. If less than I<size> bytes only start as many as there is memory for. If less than I<size> bytes
@ -1457,10 +1454,8 @@ See also: B<--memsuspend>
=item B<--memsuspend> I<size> =item B<--memsuspend> I<size>
Suspend jobs when there is less than 2 * I<size> memory free. The Suspend jobs when there is less than 2 * I<size> memory free. The
I<size> can be postfixed with K, M, G, T, P, k, m, g, t, or p which I<size> can be postfixed with K, M, G, T, P, k, m, g, t, or p (see
would multiply the size with 1024, 1048576, 1073741824, 1099511627776, UNIT PREFIX).
1125899906842624, 1000, 1000000, 1000000000, 1000000000000, or
1000000000000000, respectively.
If the available memory falls below 2 * I<size>, GNU B<parallel> If the available memory falls below 2 * I<size>, GNU B<parallel>
will suspend some of the running jobs. If the available memory falls will suspend some of the running jobs. If the available memory falls
@ -1502,6 +1497,9 @@ GNU B<parallel> will exit.
B<-n 0> means read one argument, but insert 0 arguments on the command B<-n 0> means read one argument, but insert 0 arguments on the command
line. line.
I<max-args> can be postfixed with K, M, G, T, P, k, m, g, t, or p (see
UNIT PREFIX).
Implies B<-X> unless B<-m> is set. Implies B<-X> unless B<-m> is set.
@ -1523,6 +1521,9 @@ This will set the owner of the homedir to the user:
Implies B<-X> unless B<-m> or B<--pipe> is set. Implies B<-X> unless B<-m> or B<--pipe> is set.
I<max-args> can be postfixed with K, M, G, T, P, k, m, g, t, or p (see
UNIT PREFIX).
When used with B<--pipe> B<-N> is the number of records to read. This When used with B<--pipe> B<-N> is the number of records to read. This
is somewhat slower than B<--block>. is somewhat slower than B<--block>.
@ -2254,6 +2255,9 @@ the argument strings. The largest allowed value is system-dependent,
and is calculated as the argument length limit for exec, less the size and is calculated as the argument length limit for exec, less the size
of your environment. The default value is the maximum. of your environment. The default value is the maximum.
I<max-chars> can be postfixed with K, M, G, T, P, k, m, g, t, or p
(see UNIT PREFIX).
Implies B<-X> unless B<-m> is set. Implies B<-X> unless B<-m> is set.
@ -4858,6 +4862,18 @@ chunk border.
B<--group-by> can be combined with B<--round-robin> or B<--pipe-part>. B<--group-by> can be combined with B<--round-robin> or B<--pipe-part>.
=head1 UNIT PREFIX
Many numerical arguments in GNU B<parallel> can be postfixed with K,
M, G, T, P, k, m, g, t, or p which would multiply the number with
1024, 1048576, 1073741824, 1099511627776, 1125899906842624, 1000,
1000000, 1000000000, 1000000000000, or 1000000000000000, respectively.
You can even give it as a math expression. E.g. 1000000 can be written
as 1M-12*2.024*2k.
=head1 QUOTING =head1 QUOTING
GNU B<parallel> is very liberal in quoting. You only need to quote GNU B<parallel> is very liberal in quoting. You only need to quote
@ -5453,6 +5469,8 @@ section in the man page (man parallel - search for EXAMPLE:).
=back =back
=head2 Bug dependent on environment
If you suspect the error is dependent on your environment or If you suspect the error is dependent on your environment or
distribution, please see if you can reproduce the error on one of distribution, please see if you can reproduce the error on one of
these VirtualBox images: these VirtualBox images:

View file

@ -692,8 +692,7 @@ using GNU B<parallel>:
11$ parallel '[ -f {} ] && echo {}' < List | ... 11$ parallel '[ -f {} ] && echo {}' < List | ...
https://web.archive.org/web/20160702211113/ https://www.databits.net/~ksb/msrc/local/bin/xapply/xapply.html
http://carrera.databits.net/~ksb/msrc/local/bin/xapply/xapply.html
=head2 DIFFERENCES BETWEEN AIX apply AND GNU Parallel =head2 DIFFERENCES BETWEEN AIX apply AND GNU Parallel
@ -3226,13 +3225,131 @@ https://github.com/themattrix/bash-concurrent
(Last checked: 2021-02) (Last checked: 2021-02)
=head2 Todo =head2 DIFFERENCES BETWEEN spawntool AND GNU Parallel
Summary (see legend above):
=over
=item I1 - - - - - -
=item M1 - - - - M6
=item - O2 O3 - O5 O6 - x x O10
=item E1 - - - - - -
=item - - - - - - - - -
=item - -
=back
B<spawn> reads a full command line from stdin which it executes in
parallel.
http://code.google.com/p/spawntool/ http://code.google.com/p/spawntool/
(Last checked: 2021-07)
http://code.google.com/p/push/
=head2 DIFFERENCES BETWEEN go-pssh AND GNU Parallel
Summary (see legend above):
=over
=item - - - - - - -
=item M1 - - - - -
=item O1 - - - - - - x x O10
=item E1 - - - - - -
=item R1 R2 - - - R6 - - -
=item - -
=back
B<go-pssh> does B<ssh> in parallel to multiple machines. It runs the
same command on multiple machines similar to B<--nonall>.
The hostnames must be given as IP-addresses (not as hostnames).
Output is sent to stdout (standard output) if command is successful,
and to stderr (standard error) if the command fails.
=head3 EXAMPLES FROM go-pssh
1$ go-pssh -l <ip>,<ip> -u <user> -p <port> -P <passwd> -c "<command>"
1$ parallel -S 'sshpass -p <passwd> ssh -p <port> <user>@<ip>' \
--nonall "<command>"
2$ go-pssh scp -f host.txt -u <user> -p <port> -P <password> \
-s /local/file_or_directory -d /remote/directory
2$ parallel --nonall --slf host.txt \
--basefile /local/file_or_directory/./ --wd /remote/directory
--ssh 'sshpass -p <password> ssh -p <port> -l <user>' true
3$ go-pssh scp -l <ip>,<ip> -u <user> -p <port> -P <password> \
-s /local/file_or_directory -d /remote/directory
3$ parallel --nonall -S <ip>,<ip> \
--basefile /local/file_or_directory/./ --wd /remote/directory
--ssh 'sshpass -p <password> ssh -p <port> -l <user>' true
https://github.com/xuchenCN/go-pssh
(Last checked: 2021-07)
=head2 DIFFERENCES BETWEEN go-parallel AND GNU Parallel
Summary (see legend above):
=over
=item I1 I2 - - - - I7
=item - - M3 - - M6
=item - O2 O3 - O5 - - x x - O10
=item E1 - - E4 - - -
=item - - - - - - - - -
=item - -
=back
B<go-parallel> uses Go templates for replacement strings. Quite
similar to the I<{= perl expr =}> replacement string.
=head3 EXAMPLES FROM go-parallel
1$ go-parallel -a ./files.txt -t 'cp {{.Input}} {{.Input | dirname | dirname}}'
1$ parallel -a ./files.txt cp {} '{= $_=::dirname(::dirname($_)) =}'
2$ go-parallel -a ./files.txt -t 'mkdir -p {{.Input}} {{noExt .Input}}'
2$ parallel -a ./files.txt echo mkdir -p {} {.}
3$ go-parallel -a ./files.txt -t 'mkdir -p {{.Input}} {{.Input | basename | noExt}}'
3$ parallel -a ./files.txt echo mkdir -p {} {/.}
https://github.com/mylanconnolly/parallel https://github.com/mylanconnolly/parallel
(Last checked: 2021-07)
=head2 Todo
http://code.google.com/p/push/ (cannot compile)
https://github.com/krashanoff/parallel https://github.com/krashanoff/parallel
@ -3250,8 +3367,6 @@ https://github.com/benoror/better-npm-run - not obvious how to use
https://github.com/bahmutov/with-package https://github.com/bahmutov/with-package
https://github.com/xuchenCN/go-pssh
https://github.com/flesler/parallel https://github.com/flesler/parallel
https://github.com/Julian/Verge https://github.com/Julian/Verge
@ -3262,7 +3377,6 @@ https://vicerveza.homeunix.net/~viric/soft/ts/
https://github.com/chapmanjacobd/que https://github.com/chapmanjacobd/que
https://github.com/Overv/outrun#outrun
=head1 TESTING OTHER TOOLS =head1 TESTING OTHER TOOLS

View file

@ -122,7 +122,7 @@ GetOptions(
"help" => \$opt::dummy, "help" => \$opt::dummy,
) || exit(255); ) || exit(255);
$Global::progname = ($0 =~ m:(^|/)([^/]+)$:)[1]; $Global::progname = ($0 =~ m:(^|/)([^/]+)$:)[1];
$Global::version = 20210722; $Global::version = 20210723;
if($opt::version) { version(); exit 0; } if($opt::version) { version(); exit 0; }
@Global::sortoptions = @Global::sortoptions =
shell_quote(@ARGV_before[0..($#ARGV_before-$#ARGV-1)]); shell_quote(@ARGV_before[0..($#ARGV_before-$#ARGV-1)]);

View file

@ -600,7 +600,7 @@ $Global::Initfile && unlink $Global::Initfile;
exit ($err); exit ($err);
sub parse_options { sub parse_options {
$Global::version = 20210722; $Global::version = 20210723;
$Global::progname = 'sql'; $Global::progname = 'sql';
# This must be done first as this may exec myself # This must be done first as this may exec myself

View file

@ -8,6 +8,16 @@
# Each should be taking 3-10s and be possible to run in parallel # Each should be taking 3-10s and be possible to run in parallel
# I.e.: No race conditions, no logins # I.e.: No race conditions, no logins
par_prefix_for_L_n_N_s() {
echo Must give xxx000 args
seq 10000 | parallel -N 1k 'echo {} | wc -w' | sort
seq 10000 | parallel -n 1k 'echo {} | wc -w' | sort
echo Must give xxx000 lines
seq 1000000 | parallel -L 1k --pipe wc -l | sort
echo Must give max 1000 chars per line
seq 10000 | parallel -mj1 -s 1k 'echo {} | wc -w' | sort
}
par_parset_assoc_arr() { par_parset_assoc_arr() {
mytest=$(cat <<'EOF' mytest=$(cat <<'EOF'
mytest() { mytest() {

View file

@ -401,6 +401,86 @@ par_pipe_regexp_non_quoted Start
par_pipe_regexp_non_quoted Start this line is a false Start line par_pipe_regexp_non_quoted Start this line is a false Start line
par_pipe_regexp_non_quoted End this line is a false End line par_pipe_regexp_non_quoted End this line is a false End line
par_pipe_regexp_non_quoted End par_pipe_regexp_non_quoted End
par_prefix_for_L_n_N_s Must give xxx000 args
par_prefix_for_L_n_N_s 1000
par_prefix_for_L_n_N_s 1000
par_prefix_for_L_n_N_s 1000
par_prefix_for_L_n_N_s 1000
par_prefix_for_L_n_N_s 1000
par_prefix_for_L_n_N_s 1000
par_prefix_for_L_n_N_s 1000
par_prefix_for_L_n_N_s 1000
par_prefix_for_L_n_N_s 1000
par_prefix_for_L_n_N_s 1000
par_prefix_for_L_n_N_s 1000
par_prefix_for_L_n_N_s 1000
par_prefix_for_L_n_N_s 1000
par_prefix_for_L_n_N_s 1000
par_prefix_for_L_n_N_s 1000
par_prefix_for_L_n_N_s 1000
par_prefix_for_L_n_N_s 1000
par_prefix_for_L_n_N_s 1000
par_prefix_for_L_n_N_s 1000
par_prefix_for_L_n_N_s 1000
par_prefix_for_L_n_N_s Must give xxx000 lines
par_prefix_for_L_n_N_s 149000
par_prefix_for_L_n_N_s 149000
par_prefix_for_L_n_N_s 149000
par_prefix_for_L_n_N_s 149000
par_prefix_for_L_n_N_s 149000
par_prefix_for_L_n_N_s 165000
par_prefix_for_L_n_N_s 90000
par_prefix_for_L_n_N_s Must give max 1000 chars per line
par_prefix_for_L_n_N_s 126
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 197
par_prefix_for_L_n_N_s 244
par_prefix_for_L_n_N_s 246
par_prefix_for_L_n_N_s 246
par_prefix_for_L_n_N_s 273
par_progress ### Test of --progress par_progress ### Test of --progress
par_progress 16 par_progress 16
par_progress ### Test of --progress with no jobs par_progress ### Test of --progress with no jobs

View file

@ -23,6 +23,8 @@ par_zsh_embed your
par_zsh_embed code par_zsh_embed code
par_zsh_embed here par_zsh_embed here
par_zsh_embed Put your code here par_zsh_embed Put your code here
par_zsh_embed You can also activate GNU Parallel for interactive use by:
par_zsh_embed . ./parallel-embed
par_tcsh_embed Not implemented par_tcsh_embed Not implemented
par_sh_embed --embed par_sh_embed --embed
par_sh_embed Redirect the output to a file and add your changes at the end: par_sh_embed Redirect the output to a file and add your changes at the end:
@ -40,6 +42,8 @@ par_sh_embed your
par_sh_embed code par_sh_embed code
par_sh_embed here par_sh_embed here
par_sh_embed Put your code here par_sh_embed Put your code here
par_sh_embed You can also activate GNU Parallel for interactive use by:
par_sh_embed . ./parallel-embed
par_propagate_env ### bug #41805: Idea: propagate --env for parallel --number-of-cores par_propagate_env ### bug #41805: Idea: propagate --env for parallel --number-of-cores
par_propagate_env ** test_zsh par_propagate_env ** test_zsh
par_propagate_env FOO=test_zsh par_propagate_env FOO=test_zsh
@ -90,6 +94,8 @@ par_ksh_embed your
par_ksh_embed code par_ksh_embed code
par_ksh_embed here par_ksh_embed here
par_ksh_embed Put your code here par_ksh_embed Put your code here
par_ksh_embed You can also activate GNU Parallel for interactive use by:
par_ksh_embed . ./parallel-embed
par_fish_embed Not implemented par_fish_embed Not implemented
par_env_parallel_big_env ### bug #54128: command too long when exporting big env par_env_parallel_big_env ### bug #54128: command too long when exporting big env
par_env_parallel_big_env should not fail par_env_parallel_big_env should not fail
@ -120,6 +126,8 @@ par_bash_embed your
par_bash_embed code par_bash_embed code
par_bash_embed here par_bash_embed here
par_bash_embed Put your code here par_bash_embed Put your code here
par_bash_embed You can also activate GNU Parallel for interactive use by:
par_bash_embed . ./parallel-embed
par_ash_embed --embed par_ash_embed --embed
par_ash_embed Redirect the output to a file and add your changes at the end: par_ash_embed Redirect the output to a file and add your changes at the end:
par_ash_embed /usr/local/bin/parallel --embed > new_script par_ash_embed /usr/local/bin/parallel --embed > new_script
@ -136,6 +144,8 @@ par_ash_embed your
par_ash_embed code par_ash_embed code
par_ash_embed here par_ash_embed here
par_ash_embed Put your code here par_ash_embed Put your code here
par_ash_embed You can also activate GNU Parallel for interactive use by:
par_ash_embed . ./parallel-embed
par_PARALLEL_SSHLOGIN_SSHHOST ### bug #56554: Introduce $PARALLEL_SSHLOGIN $PARALLEL_SSHHOST par_PARALLEL_SSHLOGIN_SSHHOST ### bug #56554: Introduce $PARALLEL_SSHLOGIN $PARALLEL_SSHHOST
par_PARALLEL_SSHLOGIN_SSHHOST /usr/bin/ssh csh@lo /usr/bin/ssh csh@lo lo par_PARALLEL_SSHLOGIN_SSHHOST /usr/bin/ssh csh@lo /usr/bin/ssh csh@lo lo
par_PARALLEL_SSHLOGIN_SSHHOST /usr/bin/ssh csh@lo csh par_PARALLEL_SSHLOGIN_SSHHOST /usr/bin/ssh csh@lo csh