tange/parallel
1
0
Fork 0
mirror of https://git.savannah.gnu.org/git/parallel.git synced 2024-11-24 06:57:55 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Released as 20200522 ('Kraftwerk')

Browse source
This commit is contained in:
Ole Tange 2020-05-23 20:04:09 +02:00
parent 5bba5c922d
commit 1e7da025c4
16 changed files with 270 additions and 178 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

34
NEWS
View file

@ -1,3 +1,37 @@
20200522
New in this release:
* While running a job $PARALLEL_JOBSLOT is the jobslot of the job. It
is equal to {%} unless the job is being retried. See {%} for
details.
* While running a job $PARALLEL_SSHLOGIN is the sshlogin line with
number of cores removed. E.g. '4//usr/bin/specialssh user@host'
becomes: '/usr/bin/specialssh user@host'
* While running a job $PARALLEL_SSHHOST is the host part of an
sshlogin line. E.g. '4//usr/bin/specialssh user@host' becomes:
'host'
* --plus activates the replacement strings {slot} = $PARALLEL_JOBSLOT,
{sshlogin} = $PARALLEL_SSHLOGIN, {host} = $PARALLEL_SSHHOST
* Bug fixes and man page updates.
News about GNU Parallel:
* Portable Batch System (PBS) & GNU Parallel - Running a Program
Multiple Times in Parallel
https://www.youtube.com/watch?v=6ccbWu6Befo
* GNU Parallel przykład https://www.youtube.com/watch?v=gs_wG4Kt2G4
* Demo of LINUX APP - GNU PARALLEL - running multiple Gstreamer webcam
.sh scripts with only 1 command
https://www.youtube.com/watch?v=trQuA_wmWjg
20200422 20200422
This release celebrates GNU Parallel's 10 years as a GNU tool: This release celebrates GNU Parallel's 10 years as a GNU tool:

24
README
View file

@ -57,11 +57,11 @@ document.
Full installation of GNU Parallel is as simple as: Full installation of GNU Parallel is as simple as:
wget https://ftpmirror.gnu.org/parallel/parallel-20200422.tar.bz2 wget https://ftpmirror.gnu.org/parallel/parallel-20200522.tar.bz2
wget https://ftpmirror.gnu.org/parallel/parallel-20200422.tar.bz2.sig wget https://ftpmirror.gnu.org/parallel/parallel-20200522.tar.bz2.sig
gpg parallel-20200422.tar.bz2.sig gpg parallel-20200522.tar.bz2.sig
bzip2 -dc parallel-20200422.tar.bz2 | tar xvf - bzip2 -dc parallel-20200522.tar.bz2 | tar xvf -
cd parallel-20200422 cd parallel-20200522
./configure && make && sudo make install ./configure && make && sudo make install
@ -70,11 +70,11 @@ Full installation of GNU Parallel is as simple as:
If you are not root you can add ~/bin to your path and install in If you are not root you can add ~/bin to your path and install in
~/bin and ~/share: ~/bin and ~/share:
wget https://ftpmirror.gnu.org/parallel/parallel-20200422.tar.bz2 wget https://ftpmirror.gnu.org/parallel/parallel-20200522.tar.bz2
wget https://ftpmirror.gnu.org/parallel/parallel-20200422.tar.bz2.sig wget https://ftpmirror.gnu.org/parallel/parallel-20200522.tar.bz2.sig
gpg parallel-20200422.tar.bz2.sig gpg parallel-20200522.tar.bz2.sig
bzip2 -dc parallel-20200422.tar.bz2 | tar xvf - bzip2 -dc parallel-20200522.tar.bz2 | tar xvf -
cd parallel-20200422 cd parallel-20200522
./configure --prefix=$HOME && make && make install ./configure --prefix=$HOME && make && make install
Or if your system lacks 'make' you can simply copy src/parallel Or if your system lacks 'make' you can simply copy src/parallel
@ -122,8 +122,8 @@ will love you for it.
When using programs that use GNU Parallel to process data for When using programs that use GNU Parallel to process data for
publication please cite: publication please cite:
O. Tange (2018): GNU Parallel 2018, Mar 2018, ISBN 9781387509881, Tange, O. (2020, May 22). GNU Parallel 20200522 ('Kraftwerk').
DOI https://doi.org/10.5281/zenodo.1146014 Zenodo. https://doi.org/10.5281/zenodo.3840974
= New versions = = New versions =

20
configure vendored
View file

@ -1,6 +1,6 @@
#! /bin/sh #! /bin/sh
# Guess values for system-dependent variables and create Makefiles. # Guess values for system-dependent variables and create Makefiles.
# Generated by GNU Autoconf 2.69 for parallel 20200422. # Generated by GNU Autoconf 2.69 for parallel 20200522.
# #
# Report bugs to <bug-parallel@gnu.org>. # Report bugs to <bug-parallel@gnu.org>.
# #
@ -579,8 +579,8 @@ MAKEFLAGS=
# Identity of this package. # Identity of this package.
PACKAGE_NAME='parallel' PACKAGE_NAME='parallel'
PACKAGE_TARNAME='parallel' PACKAGE_TARNAME='parallel'
PACKAGE_VERSION='20200422' PACKAGE_VERSION='20200522'
PACKAGE_STRING='parallel 20200422' PACKAGE_STRING='parallel 20200522'
PACKAGE_BUGREPORT='bug-parallel@gnu.org' PACKAGE_BUGREPORT='bug-parallel@gnu.org'
PACKAGE_URL='' PACKAGE_URL=''
@ -1214,7 +1214,7 @@ if test "$ac_init_help" = "long"; then
# Omit some internal or obsolete options to make the list less imposing. # Omit some internal or obsolete options to make the list less imposing.
# This message is too long to be a string in the A/UX 3.1 sh. # This message is too long to be a string in the A/UX 3.1 sh.
cat <<_ACEOF cat <<_ACEOF
\`configure' configures parallel 20200422 to adapt to many kinds of systems. \`configure' configures parallel 20200522 to adapt to many kinds of systems.
Usage: $0 [OPTION]... [VAR=VALUE]... Usage: $0 [OPTION]... [VAR=VALUE]...
@ -1281,7 +1281,7 @@ fi
if test -n "$ac_init_help"; then if test -n "$ac_init_help"; then
case $ac_init_help in case $ac_init_help in
short | recursive ) echo "Configuration of parallel 20200422:";; short | recursive ) echo "Configuration of parallel 20200522:";;
esac esac
cat <<\_ACEOF cat <<\_ACEOF
@ -1357,7 +1357,7 @@ fi
test -n "$ac_init_help" && exit $ac_status test -n "$ac_init_help" && exit $ac_status
if $ac_init_version; then if $ac_init_version; then
cat <<\_ACEOF cat <<\_ACEOF
parallel configure 20200422 parallel configure 20200522
generated by GNU Autoconf 2.69 generated by GNU Autoconf 2.69
Copyright (C) 2012 Free Software Foundation, Inc. Copyright (C) 2012 Free Software Foundation, Inc.
@ -1374,7 +1374,7 @@ cat >config.log <<_ACEOF
This file contains any messages produced by compilers while This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake. running configure, to aid debugging if configure makes a mistake.
It was created by parallel $as_me 20200422, which was It was created by parallel $as_me 20200522, which was
generated by GNU Autoconf 2.69. Invocation command line was generated by GNU Autoconf 2.69. Invocation command line was
$ $0 $@ $ $0 $@
@ -2237,7 +2237,7 @@ fi
# Define the identity of the package. # Define the identity of the package.
PACKAGE='parallel' PACKAGE='parallel'
VERSION='20200422' VERSION='20200522'
cat >>confdefs.h <<_ACEOF cat >>confdefs.h <<_ACEOF
@ -2880,7 +2880,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
# report actual input values of CONFIG_FILES etc. instead of their # report actual input values of CONFIG_FILES etc. instead of their
# values after options handling. # values after options handling.
ac_log=" ac_log="
This file was extended by parallel $as_me 20200422, which was This file was extended by parallel $as_me 20200522, which was
generated by GNU Autoconf 2.69. Invocation command line was generated by GNU Autoconf 2.69. Invocation command line was
CONFIG_FILES = $CONFIG_FILES CONFIG_FILES = $CONFIG_FILES
@ -2942,7 +2942,7 @@ _ACEOF
cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`" ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`"
ac_cs_version="\\ ac_cs_version="\\
parallel config.status 20200422 parallel config.status 20200522
configured by $0, generated by GNU Autoconf 2.69, configured by $0, generated by GNU Autoconf 2.69,
with options \\"\$ac_cs_config\\" with options \\"\$ac_cs_config\\"

2
configure.ac
View file

@ -1,4 +1,4 @@
AC_INIT([parallel], [20200422], [bug-parallel@gnu.org]) AC_INIT([parallel], [20200522], [bug-parallel@gnu.org])
AM_INIT_AUTOMAKE([-Wall -Werror foreign]) AM_INIT_AUTOMAKE([-Wall -Werror foreign])
AC_CONFIG_HEADERS([config.h]) AC_CONFIG_HEADERS([config.h])
AC_CONFIG_FILES([ AC_CONFIG_FILES([

64
doc/release_new_version
View file

@ -12,19 +12,58 @@ Modified => alpha
== Update version == == Update version ==
Get DOI:
https://zenodo.org/deposit/new (Reserve DOI)
configure.ac: AC_INIT([parallel], [20100422], [bug-parallel@gnu.org]) configure.ac: AC_INIT([parallel], [20100422], [bug-parallel@gnu.org])
src/parallel: $Global::version = 20100422; src/parallel: $Global::version = 20100422;
README: parallel-20130222 README: parallel-20130222
DOINO=3840974
TAG=Kraftwerk
YYYYMMDD=$(echo `yyyymmdd`-1 | bc) YYYYMMDD=$(echo `yyyymmdd`-1 | bc)
YYYYMMDD=$(echo `yyyymmdd`+1 | bc) YYYYMMDD=$(echo `yyyymmdd`+1 | bc)
YYYYMMDD=`yyyymmdd` YYYYMMDD=`yyyymmdd`
echo $YYYYMMDD
perl -i -pe "s/20\d\d\d\d\d\d/$YYYYMMDD/" configure.ac updater() {
perl -i -pe "/version/ and s/20\d\d\d\d\d\d/$YYYYMMDD/" src/parallel export DOINO
perl -i -pe "/version/ and s/20\d\d\d\d\d\d/$YYYYMMDD/" src/sql export TAG
perl -i -pe "/version/ and s/20\d\d\d\d\d\d/$YYYYMMDD/" src/niceload export YYYYMMDD
perl -i -pe "s/parallel-20\d\d\d\d\d\d/parallel-$YYYYMMDD/" README export DOI=10.5281/zenodo.$DOINO
export YYYY=${YYYYMMDD:0:4}
export MON=`date +%b`
export MONTH=`date +%B`
echo Tag=$TAG Date:$YYYYMMDD Year:$YYYY Mon:$MON Month:$MONTH DOI:$DOI
export TITLE="GNU Parallel $YYYYMMDD ('$TAG')"
perl -i -pe "s/20\d\d\d\d\d\d/$YYYYMMDD/" configure.ac
perl -i -pe "/version/ and s/20\d\d\d\d\d\d/$YYYYMMDD/" src/sql
perl -i -pe "/version/ and s/20\d\d\d\d\d\d/$YYYYMMDD/" src/niceload
perl -i -pe "s/parallel-20\d\d\d\d\d\d/parallel-$YYYYMMDD/" README
perl -i -pe '
# Update version 20209999
/version/ and s/20\d\d\d\d\d\d/$ENV{YYYYMMDD}/;
# Update: 10.5281/zenodo.1146014
s:10.5281/zenodo.\d+:$ENV{DOI}:;
# Update "@software{tange_2015_16303,"
s:tange_\d+_\d+:tange_$ENV{YYYY}_$ENV{DOINO}:;
# Update month = mar,
s/(month\s+=\s+)\S+,",/$1$ENV{MON},",/;
# Update title = {GNU Parallel 20200522 ('Kraftwerk')},",
/ title\s+= / and s/\{.*\}/{$ENV{TITLE}}/;
# Tange, O. (2020, May 22). GNU Parallel 20200522 ('Kraftwerk').
s/(Tange, O. .).*(.. )(GNU.*[)])/$1$ENV{YYYY}, $ENV{MONTH} 22$2$ENV{TITLE}/;
' src/parallel README
(
ppar --help
ppar --citation
grep -i 'zenodo|tange' README
) 2>&1 | grep -E '^ |^}|tange'
mv ~/.parallel/will-cite ~/.parallel/will-cite.
ppar ::: true
mv ~/.parallel/will-cite. ~/.parallel/will-cite
}
updater
=== Autoconf/automake === === Autoconf/automake ===
@ -223,12 +262,23 @@ Quote of the month:
New in this release: New in this release:
* While running a job $PARALLEL_JOBSLOT is the jobslot of the job. It is equal to {%} unless the job is being retried. See {%} for details.
* While running a job $PARALLEL_SSHLOGIN is the sshlogin line with number of cores removed. E.g. '4//usr/bin/specialssh user@host' becomes: '/usr/bin/specialssh user@host'
* While running a job $PARALLEL_SSHHOST is the host part of an sshlogin line. E.g. '4//usr/bin/specialssh user@host' becomes: 'host'
* --plus activates the replacement strings {slot} = $PARALLEL_JOBSLOT, {sshlogin} = $PARALLEL_SSHLOGIN, {host} = $PARALLEL_SSHHOST
* Bug fixes and man page updates. * Bug fixes and man page updates.
News about GNU Parallel: News about GNU Parallel:
* * Portable Batch System (PBS) & GNU Parallel - Running a Program Multiple Times in Parallel https://www.youtube.com/watch?v=6ccbWu6Befo
* GNU Parallel przykład https://www.youtube.com/watch?v=gs_wG4Kt2G4
* demo of LINUX APP - GNU PARALLEL - running multiple Gstreamer webcam .sh scripts with only 1 command https://www.youtube.com/watch?v=trQuA_wmWjg
Get the book: GNU Parallel 2018 http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html Get the book: GNU Parallel 2018 http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html

2
src/niceload
View file

@ -23,7 +23,7 @@
use strict; use strict;
use Getopt::Long; use Getopt::Long;
$Global::progname="niceload"; $Global::progname="niceload";
$Global::version = 20200423; $Global::version = 20200522;
Getopt::Long::Configure("bundling","require_order"); Getopt::Long::Configure("bundling","require_order");
get_options_from_array(\@ARGV) || die_usage(); get_options_from_array(\@ARGV) || die_usage();
if($opt::version) { if($opt::version) {

151
src/parallel_alternatives.pod
View file

@ -114,11 +114,6 @@ E1 - - - E5 -
- - - -
xjobs, prll, dxargs, mdm/middelman, xapply, paexec, ladon, jobflow,
ClusterSSH: TODO - Please file a bug-report if you know what features
they support (See REPORTING BUGS).
=head2 DIFFERENCES BETWEEN xargs AND GNU Parallel =head2 DIFFERENCES BETWEEN xargs AND GNU Parallel
Summary table (see legend above): Summary table (see legend above):
@ -520,9 +515,11 @@ using GNU B<parallel>:
4$ parallel indent ::: *.c 4$ parallel indent ::: *.c
5$ find ~ksb/bin -type f ! -perm -111 -print | xapply -f -v 'chmod a+x' - 5$ find ~ksb/bin -type f ! -perm -111 -print | \
xapply -f -v 'chmod a+x' -
5$ find ~ksb/bin -type f ! -perm -111 -print | parallel -v chmod a+x 5$ find ~ksb/bin -type f ! -perm -111 -print | \
parallel -v chmod a+x
6$ find */ -... | fmt 960 1024 | xapply -f -i /dev/tty 'vi' - 6$ find */ -... | fmt 960 1024 | xapply -f -i /dev/tty 'vi' -
@ -532,15 +529,16 @@ using GNU B<parallel>:
7$ find ... | xapply -f -5 -i /dev/tty 'vi' - - - - - 7$ find ... | xapply -f -5 -i /dev/tty 'vi' - - - - -
7$ sh <(find ... |parallel -n5 echo vi) 7$ sh <(find ... | parallel -n5 echo vi)
7$ find ... |parallel -n5 -uj1 vi 7$ find ... | parallel -n5 -uj1 vi
8$ xapply -fn "" /etc/passwd 8$ xapply -fn "" /etc/passwd
8$ parallel -k echo < /etc/passwd 8$ parallel -k echo < /etc/passwd
9$ tr ':' '\012' < /etc/passwd | xapply -7 -nf 'chown %1 %6' - - - - - - - 9$ tr ':' '\012' < /etc/passwd | \
xapply -7 -nf 'chown %1 %6' - - - - - - -
9$ tr ':' '\012' < /etc/passwd | parallel -N7 chown {1} {6} 9$ tr ':' '\012' < /etc/passwd | parallel -N7 chown {1} {6}
@ -563,7 +561,7 @@ very much like GNU B<parallel>. B<apply> does not run jobs in
parallel. B<apply> does not use an argument separator (like B<:::>); parallel. B<apply> does not use an argument separator (like B<:::>);
instead the template must be the first argument. instead the template must be the first argument.
=head3 EXAMPLES FROM IBM's KNOWLEDGE CENTER =head3 EXAMPLES FROM IBM's KNOWLEDGE CENTER
Here are the examples from IBM's Knowledge Center and the Here are the examples from IBM's Knowledge Center and the
corresponding command using GNU B<parallel>: corresponding command using GNU B<parallel>:
@ -649,65 +647,47 @@ https://github.com/cheusov/paexec
=head2 DIFFERENCES BETWEEN map(sitaramc) AND GNU Parallel =head2 DIFFERENCES BETWEEN map(sitaramc) AND GNU Parallel
Summary table (see legend above): Summary table (see legend above):
I1 - - I4 - - - I1 - - I4 - - (I7)
M1 (M2) M3 M4 M5 - M1 (M2) M3 (M4) M5 M6
- O2 O3 - O5 - - N/A N/A O10 - O2 O3 - O5 - - N/A N/A O10
E1 - - - - - - E1 - - - - - -
- - - - - - - - - - - - - - - - - -
- - - -
(M2): Only if there is a single replacement string. (I7): Only under special circumstances. See below.
B<map> sees it as a feature to have less features and in doing so it (M2+M4): Only if there is a single replacement string.
also handles corner cases incorrectly. A lot of GNU B<parallel>'s code
is to handle corner cases correctly on every platform, so you will not
get a nasty surprise if a user, for example, saves a file called: I<My
brother's 12" records.txt>
B<map>'s example showing how to deal with special characters fails on B<map> rejects input with special characters:
special characters:
echo "The Cure" > My\ brother\'s\ 12\"\ records echo "The Cure" > My\ brother\'s\ 12\"\ records
ls | \ ls | map 'echo %; wc %'
map 'echo -n `gzip < "%" | wc -c`; echo -n '*100/'; wc -c < "%"' |
bc
It works with GNU B<parallel>: It works with GNU B<parallel>:
ls | \ ls | parallel 'echo {}; wc {}'
parallel \
'echo -n `gzip < {} | wc -c`; echo -n '*100/'; wc -c < {}' | bc
And you can even get the file name prepended: Under some circumstances it also works with B<map>:
ls | \ ls | map 'echo % works %'
parallel --tag \
'(echo -n `gzip < {} | wc -c`'*100/'; wc -c < {}) | bc'
B<map> has no support for grouping. So this gives the wrong results But tiny changes make it reject the input with special characters:
without any warnings:
parallel perl -e '\$a=\"1{}\"x10000000\;print\ \$a,\"\\n\"' '>' {} \ ls | map 'echo % does not work "%"'
::: a b c d e f
ls -l a b c d e f
parallel -kP4 -n1 grep 1 > out.par ::: a b c d e f
map -p 4 'grep 1' a b c d e f > out.map-unbuf
map -p 4 'grep --line-buffered 1' a b c d e f > out.map-linebuf
map -p 1 'grep --line-buffered 1' a b c d e f > out.map-serial
ls -l out*
md5sum out*
The documentation shows a workaround, but not only does that mix This means that many UTF-8 characters will be rejected. This is by
stdout (standard output) with stderr (standard error) it also fails design. From the web page: "As such, programs that I<quietly handle
completely for certain jobs (and may even be considered less readable): them, with no warnings at all,> are doing their users a disservice."
parallel echo -n {} ::: 1 2 3 B<map> delays each job by 0.01 s. This can be emulated by using
B<parallel --delay 0.01>.
map -p 4 'echo -n % 2>&1 | sed -e "s/^/$$:/"' 1 2 3 | \ B<map> prints '+' on stderr when a job starts, and '-' when a job
sort | cut -f2- -d: finishes. This cannot be disabled. B<parallel> has B<--bar> if you
need to see progress.
B<map>s replacement strings (% %D %B %E) can be simulated in GNU B<map>'s replacement strings (% %D %B %E) can be simulated in GNU
B<parallel> by putting this in B<~/.parallel/config>: B<parallel> by putting this in B<~/.parallel/config>:
--rpl '%' --rpl '%'
@ -730,25 +710,58 @@ context replace:
map "echo 'BEGIN{'%'}END'" 1 2 3 map "echo 'BEGIN{'%'}END'" 1 2 3
B<map> requires Perl v5.10.0 making it harder to use on old systems. B<map> has no support for grouping. So this gives the wrong results:
To put a % in the command line B<map> requires you to use %%: parallel perl -e '\$a=\"1{}\"x10000000\;print\ \$a,\"\\n\"' '>' {} \
::: a b c d e f
ls -l a b c d e f
parallel -kP4 -n1 grep 1 ::: a b c d e f > out.par
map -n1 -p 4 'grep 1' a b c d e f > out.map-unbuf
map -n1 -p 4 'grep --line-buffered 1' a b c d e f > out.map-linebuf
map -n1 -p 1 'grep --line-buffered 1' a b c d e f > out.map-serial
ls -l out*
md5sum out*
seq 10 | map -n1 echo % +10%% {} =head3 EXAMPLES FROM map's WEBSITE
seq 10 | parallel -I ,, echo ,, +10% {} Here are the examples from B<map>'s web page with the equivalent using
GNU B<parallel>:
GNU B<parallel> has -I to specify another replacement string than 1$ ls *.gif | map convert % %B.png # default max-args: 1
B<{}> if you use B<{}> in the command template.
By design B<map> is option incompatible with B<xargs>, it does not 1$ ls *.gif | parallel convert {} {.}.png
have remote job execution, a structured way of saving results,
multiple input sources, progress indicator, configurable record
delimiter (only field delimiter), logging of jobs run with possibility
to resume, keeping the output in the same order as input, --pipe
processing, and dynamically timeouts.
https://github.com/sitaramc/map (Last checked: 2020-04) 2$ map "mkdir %B; tar -C %B -xf %" *.tgz # default max-args: 1
2$ parallel 'mkdir {.}; tar -C {.} -xf {}' ::: *.tgz
3$ ls *.gif | map cp % /tmp # default max-args: 100
3$ ls *.gif | parallel -X cp {} /tmp
4$ ls *.tar | map -n 1 tar -xf %
4$ ls *.tar | parallel tar -xf
5$ map "cp % /tmp" *.tgz
5$ parallel cp {} /tmp ::: *.tgz
6$ map "du -sm /home/%/mail" alice bob carol
6$ parallel "du -sm /home/{}/mail" ::: alice bob carol
or if you prefer running a single job with multiple args:
6$ parallel -Xj1 "du -sm /home/{}/mail" ::: alice bob carol
7$ cat /etc/passwd | map -d: 'echo user %1 has shell %7'
7$ cat /etc/passwd | parallel --colsep : 'echo user {1} has shell {7}'
8$ export MAP_MAX_PROCS=$(( `nproc` / 2 ))
8$ export PARALLEL=-j50%
https://github.com/sitaramc/map (Last checked: 2020-05)
=head2 DIFFERENCES BETWEEN ladon AND GNU Parallel =head2 DIFFERENCES BETWEEN ladon AND GNU Parallel
@ -849,9 +862,11 @@ jobs. This can be emulated by GNU B<parallel> using B<bash>'s B<ulimit>:
3$ cat urls.txt | parallel -j32 wget {} 3$ cat urls.txt | parallel -j32 wget {}
4$ find . -name '*.bmp' | jobflow -threads=8 -exec bmp2jpeg {.}.bmp {.}.jpg 4$ find . -name '*.bmp' | \
jobflow -threads=8 -exec bmp2jpeg {.}.bmp {.}.jpg
4$ find . -name '*.bmp' | parallel -j8 bmp2jpeg {.}.bmp {.}.jpg 4$ find . -name '*.bmp' | \
parallel -j8 bmp2jpeg {.}.bmp {.}.jpg
https://github.com/rofl0r/jobflow https://github.com/rofl0r/jobflow
@ -877,9 +892,11 @@ Output to stderr (standard error) is changed if the command fails.
1$ seq 12 -1 1 | parallel -P 4 -n 3 "sleep {1}; echo {2} {3}" 1$ seq 12 -1 1 | parallel -P 4 -n 3 "sleep {1}; echo {2} {3}"
2$ cat t.txt | gargs --sep "\s+" -p 2 "echo '{0}:{1}-{2}' full-line: \'{}\'" 2$ cat t.txt | gargs --sep "\s+" \
-p 2 "echo '{0}:{1}-{2}' full-line: \'{}\'"
2$ cat t.txt | parallel --colsep "\\s+" -P 2 "echo '{1}:{2}-{3}' full-line: \'{}\'" 2$ cat t.txt | parallel --colsep "\\s+" \
-P 2 "echo '{1}:{2}-{3}' full-line: \'{}\'"
https://github.com/brentp/gargs https://github.com/brentp/gargs
@ -2633,7 +2650,7 @@ https://github.com/ExpectationMax/simple_gpu_scheduler
seq 3 | parallel echo true >> gpu.queue seq 3 | parallel echo true >> gpu.queue
# Add jobs # Add jobs
echo "my_command_with | and stuff > logfile" >> gpu.queue echo "my_command_with | and stuff > logfile" >> gpu.queue
# Needed to flush output from completed jobs # Needed to flush output from completed jobs
seq 3 | parallel echo true >> gpu.queue seq 3 | parallel echo true >> gpu.queue

2
src/sql
View file

@ -574,7 +574,7 @@ $Global::Initfile && unlink $Global::Initfile;
exit ($err); exit ($err);
sub parse_options { sub parse_options {
$Global::version = 20200423; $Global::version = 20200522;
$Global::progname = 'sql'; $Global::progname = 'sql';
# This must be done first as this may exec myself # This must be done first as this may exec myself

9
testsuite/tests-to-run/parallel-local-10s.sh
View file

@ -507,6 +507,15 @@ par_tmp_full() {
stdout parallel -j1 --tmpdir $SHM cat /dev/zero ::: dummy stdout parallel -j1 --tmpdir $SHM cat /dev/zero ::: dummy
} }
par_jobs_file() {
echo '### Test of -j filename - non-existent file'
stdout parallel -j no_such_file echo ::: 1
echo '### Test of -j filename'
echo 3 >/tmp/jobs_to_run1
parallel -j /tmp/jobs_to_run1 -v sleep {} ::: 10 8 6 5 4
# Should give 6 8 10 5 4
}
export -f $(compgen -A function | grep par_) export -f $(compgen -A function | grep par_)
compgen -A function | grep par_ | LC_ALL=C sort | compgen -A function | grep par_ | LC_ALL=C sort |

4
testsuite/tests-to-run/parallel-local-ssh7.sh
View file

@ -3317,4 +3317,6 @@ compgen -A function | grep par_ | LC_ALL=C sort -r |
s/\d+ >= \d+/XXX >= XXX/; s/\d+ >= \d+/XXX >= XXX/;
s/sh:? \d?\d\d:/sh: XXX:/; s/sh:? \d?\d\d:/sh: XXX:/;
s/:\d?\d\d:/:XXX:/; s/:\d?\d\d:/:XXX:/;
s/sh\[\d+\]/sh[XXX]/;' s/sh\[\d+\]/sh[XXX]/;
s/.*(tange|zenodo).*//i;
'

11
testsuite/tests-to-run/parallel-local12.sh
View file

@ -22,7 +22,9 @@ cat >/tmp/parallel-script-for-script2 <<EOF
echo no output |parallel --tty -kv ::: 'echo a' 'cat' echo no output |parallel --tty -kv ::: 'echo a' 'cat'
EOF EOF
chmod 755 /tmp/parallel-script-for-script2 chmod 755 /tmp/parallel-script-for-script2
echo via pseudotty | script -q -f -c /tmp/parallel-script-for-script2 /dev/null echo via pseudotty |
script -q -f -c /tmp/parallel-script-for-script2 /dev/null |
perl -ne '/tange| .*/ or print'
sleep 2 sleep 2
rm /tmp/parallel-script-for-script2 rm /tmp/parallel-script-for-script2
@ -34,11 +36,14 @@ cat >/tmp/parallel-script-for-script3 <<EOF
seq 10 | parallel --tty -X vi file{} seq 10 | parallel --tty -X vi file{}
EOF EOF
chmod 755 /tmp/parallel-script-for-script3 chmod 755 /tmp/parallel-script-for-script3
echo ZZZZ | script -q -f -c /tmp/parallel-script-for-script3 /dev/null echo ZZZZ |
script -q -f -c /tmp/parallel-script-for-script3 /dev/null |
perl -ne '/tange| .*/ or print'
sleep 2 sleep 2
rm /tmp/parallel-script-for-script3 rm /tmp/parallel-script-for-script3
stdout parallel --citation < /dev/null stdout parallel --citation < /dev/null |
perl -ne '/tange| .*/ or print'
touch ~/.parallel/will-cite touch ~/.parallel/will-cite
echo 1 > ~/.parallel/runs-without-willing-to-cite echo 1 > ~/.parallel/runs-without-willing-to-cite

8
testsuite/tests-to-run/parallel-local9.sh
View file

@ -100,14 +100,6 @@ echo "# --recend '' --files --halt-on-error"
nice nice perl -e '@x=1 .. 17000; for(1..100) { print "@x\n"}' | pv -qL 1000000 | nice nice perl -e '@x=1 .. 17000; for(1..100) { print "@x\n"}' | pv -qL 1000000 |
$PAR --recend '' --files --halt-on-error 2 cat | parallel -Xj1 cat {} ';' rm {} | md5sum $PAR --recend '' --files --halt-on-error 2 cat | parallel -Xj1 cat {} ';' rm {} | md5sum
echo '### Test of -j filename - non-existent file';
nice stdout parallel -j no_such_file echo ::: 1
echo '### Test of -j filename';
echo 3 >/tmp/jobs_to_run1;
parallel -j /tmp/jobs_to_run1 -v sleep {} ::: 10 8 6 5 4;
# Should give 6 8 10 5 4
echo '### Test ::::' echo '### Test ::::'
echo '### Change --arg-file-sep' echo '### Change --arg-file-sep'
$XAP --arg-file-sep :::: -k echo {1} {2} :::: <(seq 1 10) <(seq 5 15) $XAP --arg-file-sep :::: -k echo {1} {2} :::: <(seq 1 10) <(seq 5 15)

45
testsuite/wanted-results/parallel-local-10s
View file

@ -205,6 +205,51 @@ par_fifo_under_csh 1
par_fifo_under_csh 868832 par_fifo_under_csh 868832
par_fifo_under_csh 1 par_fifo_under_csh 1
par_fifo_under_csh exit 22 par_fifo_under_csh exit 22
par_jobs_file ### Test of -j filename - non-existent file
par_jobs_file parallel: Error: Parsing of --jobs/-j/--max-procs/-P failed.
par_jobs_file Usage:
par_jobs_file
par_jobs_file parallel [options] [command [arguments]] < list_of_arguments
par_jobs_file parallel [options] [command [arguments]] (::: arguments|:::: argfile(s))...
par_jobs_file cat ... | parallel --pipe [options] [command [arguments]]
par_jobs_file
par_jobs_file -j n Run n jobs in parallel
par_jobs_file -k Keep same order
par_jobs_file -X Multiple arguments with context replace
par_jobs_file --colsep regexp Split input on regexp for positional replacements
par_jobs_file {} {.} {/} {/.} {#} {%} {= perl code =} Replacement strings
par_jobs_file {3} {3.} {3/} {3/.} {=3 perl code =} Positional replacement strings
par_jobs_file With --plus: {} = {+/}/{/} = {.}.{+.} = {+/}/{/.}.{+.} = {..}.{+..} =
par_jobs_file {+/}/{/..}.{+..} = {...}.{+...} = {+/}/{/...}.{+...}
par_jobs_file
par_jobs_file -S sshlogin Example: foo@server.example.com
par_jobs_file --slf .. Use ~/.parallel/sshloginfile as the list of sshlogins
par_jobs_file --trc {}.bar Shorthand for --transfer --return {}.bar --cleanup
par_jobs_file --onall Run the given command with argument on all sshlogins
par_jobs_file --nonall Run the given command with no arguments on all sshlogins
par_jobs_file
par_jobs_file --pipe Split stdin (standard input) to multiple jobs.
par_jobs_file --recend str Record end separator for --pipe.
par_jobs_file --recstart str Record start separator for --pipe.
par_jobs_file
par_jobs_file See 'man parallel' for details
par_jobs_file
par_jobs_file Academic tradition requires you to cite works you base your article on.
par_jobs_file If you use programs that use GNU Parallel to process data for an article in a
par_jobs_file scientific publication, please cite:
par_jobs_file
par_jobs_file Tange, O. (2020, May 22). GNU Parallel 20200522 ('Kraftwerk').
par_jobs_file Zenodo. https://doi.org/10.5281/zenodo.3840974
par_jobs_file
par_jobs_file This helps funding further development; AND IT WON'T COST YOU A CENT.
par_jobs_file If you pay 10000 EUR you should feel free to use GNU Parallel without citing.
par_jobs_file
par_jobs_file ### Test of -j filename
par_jobs_file sleep 6
par_jobs_file sleep 8
par_jobs_file sleep 10
par_jobs_file sleep 5
par_jobs_file sleep 4
par_k ### Test -k par_k ### Test -k
par_k parallel: Warning: Only enough file handles to run X jobs in parallel. par_k parallel: Warning: Only enough file handles to run X jobs in parallel.
par_k parallel: Warning: Running 'parallel -jX -N X --pipe parallel -jX' or par_k parallel: Warning: Running 'parallel -jX -N X --pipe parallel -jX' or

8
testsuite/wanted-results/parallel-local-ssh7
View file

@ -445,8 +445,8 @@ par_tcsh_man Academic tradition requires you to cite works you base your article
par_tcsh_man If you use programs that use GNU Parallel to process data for an article in a par_tcsh_man par_tcsh_man If you use programs that use GNU Parallel to process data for an article in a par_tcsh_man
par_tcsh_man scientific publication, please cite: par_tcsh_man par_tcsh_man scientific publication, please cite: par_tcsh_man
par_tcsh_man par_tcsh_man par_tcsh_man par_tcsh_man
par_tcsh_man O. Tange (2018): GNU Parallel 2018, Mar 2018, ISBN 9781387509881, par_tcsh_man
par_tcsh_man DOI https://doi.org/10.5281/zenodo.1146014 par_tcsh_man
par_tcsh_man par_tcsh_man par_tcsh_man par_tcsh_man
par_tcsh_man This helps funding further development; AND IT WON'T COST YOU A CENT. par_tcsh_man par_tcsh_man This helps funding further development; AND IT WON'T COST YOU A CENT. par_tcsh_man
par_tcsh_man If you pay 10000 EUR you should feel free to use GNU Parallel without citing. par_tcsh_man par_tcsh_man If you pay 10000 EUR you should feel free to use GNU Parallel without citing. par_tcsh_man
@ -1839,8 +1839,8 @@ par_csh_man
par_csh_man par_csh_man
par_csh_man par_csh_man
par_csh_man {+/}/{/..}.{+..} = {...}.{+...} = {+/}/{/...}.{+...} par_csh_man {+/}/{/..}.{+..} = {...}.{+...} = {+/}/{/...}.{+...}
par_csh_man DOI https://doi.org/10.5281/zenodo.1146014
par_csh_man O. Tange (2018): GNU Parallel 2018, Mar 2018, ISBN 9781387509881,
par_csh_man ### From man env_parallel par_csh_man ### From man env_parallel
par_csh_man --colsep regexp Split input on regexp for positional replacements par_csh_man --colsep regexp Split input on regexp for positional replacements
par_csh_man --nonall Run the given command with no arguments on all sshlogins par_csh_man --nonall Run the given command with no arguments on all sshlogins

17
testsuite/wanted-results/parallel-local12
View file

@ -4,8 +4,6 @@ Academic tradition requires you to cite works you base your article on.
If you use programs that use GNU Parallel to process data for an article in a If you use programs that use GNU Parallel to process data for an article in a
scientific publication, please cite: scientific publication, please cite:
O. Tange (2018): GNU Parallel 2018, Mar 2018, ISBN 9781387509881,
DOI https://doi.org/10.5281/zenodo.1146014
This helps funding further development; AND IT WON'T COST YOU A CENT. This helps funding further development; AND IT WON'T COST YOU A CENT.
If you pay 10000 EUR you should feel free to use GNU Parallel without citing. If you pay 10000 EUR you should feel free to use GNU Parallel without citing.
@ -27,8 +25,6 @@ Academic tradition requires you to cite works you base your article on.
If you use programs that use GNU Parallel to process data for an article in a If you use programs that use GNU Parallel to process data for an article in a
scientific publication, please cite: scientific publication, please cite:
O. Tange (2018): GNU Parallel 2018, Mar 2018, ISBN 9781387509881,
DOI https://doi.org/10.5281/zenodo.1146014
This helps funding further development; AND IT WON'T COST YOU A CENT. This helps funding further development; AND IT WON'T COST YOU A CENT.
If you pay 10000 EUR you should feel free to use GNU Parallel without citing. If you pay 10000 EUR you should feel free to use GNU Parallel without citing.
@ -39,22 +35,12 @@ https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice
To silence this citation notice: run 'parallel --citation' once. To silence this citation notice: run 'parallel --citation' once.
10 files to edit 10 files to edit
[?2004h[?1049h[?1h=[?2004h[?12h[?12l[?25l"file1" [New File]½ [>c]10;?]11;?~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ 0,0-1All[?25h[?25lE173: 9 more files to edit[?2004h0,0-1All[?25h[?2004l[?2004l[?1l>[?1049lAcademic tradition requires you to cite works you base your article on. Academic tradition requires you to cite works you base your article on.
If you use programs that use GNU Parallel to process data for an article in a If you use programs that use GNU Parallel to process data for an article in a
scientific publication, please cite: scientific publication, please cite:
@book{tange_ole_2018_1146014,
author = {Tange, Ole},
title = {GNU Parallel 2018},
publisher = {Ole Tange},
month = Mar,
year = 2018,
ISBN = {9781387509881},
doi = {10.5281/zenodo.1146014},
url = {https://doi.org/10.5281/zenodo.1146014}
} }
(Feel free to use \nocite{tange_ole_2018_1146014})
This helps funding further development; AND IT WON'T COST YOU A CENT. This helps funding further development; AND IT WON'T COST YOU A CENT.
If you pay 10000 EUR you should feel free to use GNU Parallel without citing. If you pay 10000 EUR you should feel free to use GNU Parallel without citing.
@ -64,7 +50,6 @@ https://lists.gnu.org/archive/html/parallel/2013-11/msg00006.html
https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice
https://git.savannah.gnu.org/cgit/parallel.git/tree/doc/citation-notice-faq.txt https://git.savannah.gnu.org/cgit/parallel.git/tree/doc/citation-notice-faq.txt
If you send a copy of your published article to tange@gnu.org, it will be
mentioned in the release notes of next version of GNU Parallel. mentioned in the release notes of next version of GNU Parallel.

47
testsuite/wanted-results/parallel-local9
View file

@ -126,53 +126,6 @@ echo "# --recend '' --files --halt-on-error"
# --recend '' --files --halt-on-error # --recend '' --files --halt-on-error
nice nice perl -e '@x=1 .. 17000; for(1..100) { print "@x\n"}' | pv -qL 1000000 | $PAR --recend '' --files --halt-on-error 2 cat | parallel -Xj1 cat {} ';' rm {} | md5sum nice nice perl -e '@x=1 .. 17000; for(1..100) { print "@x\n"}' | pv -qL 1000000 | $PAR --recend '' --files --halt-on-error 2 cat | parallel -Xj1 cat {} ';' rm {} | md5sum
350eda13a37912d755c9d733d149bdaf - 350eda13a37912d755c9d733d149bdaf -
echo '### Test of -j filename - non-existent file'; nice stdout parallel -j no_such_file echo ::: 1
### Test of -j filename - non-existent file
parallel: Error: Parsing of --jobs/-j/--max-procs/-P failed.
Usage:
parallel [options] [command [arguments]] < list_of_arguments
parallel [options] [command [arguments]] (::: arguments|:::: argfile(s))...
cat ... | parallel --pipe [options] [command [arguments]]
-j n Run n jobs in parallel
-k Keep same order
-X Multiple arguments with context replace
--colsep regexp Split input on regexp for positional replacements
{} {.} {/} {/.} {#} {%} {= perl code =} Replacement strings
{3} {3.} {3/} {3/.} {=3 perl code =} Positional replacement strings
With --plus: {} = {+/}/{/} = {.}.{+.} = {+/}/{/.}.{+.} = {..}.{+..} =
{+/}/{/..}.{+..} = {...}.{+...} = {+/}/{/...}.{+...}
-S sshlogin Example: foo@server.example.com
--slf .. Use ~/.parallel/sshloginfile as the list of sshlogins
--trc {}.bar Shorthand for --transfer --return {}.bar --cleanup
--onall Run the given command with argument on all sshlogins
--nonall Run the given command with no arguments on all sshlogins
--pipe Split stdin (standard input) to multiple jobs.
--recend str Record end separator for --pipe.
--recstart str Record start separator for --pipe.
See 'man parallel' for details
Academic tradition requires you to cite works you base your article on.
If you use programs that use GNU Parallel to process data for an article in a
scientific publication, please cite:
O. Tange (2018): GNU Parallel 2018, Mar 2018, ISBN 9781387509881,
DOI https://doi.org/10.5281/zenodo.1146014
This helps funding further development; AND IT WON'T COST YOU A CENT.
If you pay 10000 EUR you should feel free to use GNU Parallel without citing.
echo '### Test of -j filename'; echo 3 >/tmp/jobs_to_run1; parallel -j /tmp/jobs_to_run1 -v sleep {} ::: 10 8 6 5 4; # Should give 6 8 10 5 4
### Test of -j filename
sleep 6
sleep 8
sleep 10
sleep 5
sleep 4
echo '### Test ::::' echo '### Test ::::'
### Test :::: ### Test ::::
echo '### Change --arg-file-sep' echo '### Change --arg-file-sep'