Released as 20190122 ('Shutdown')

This commit is contained in:
Ole Tange 2019-01-21 03:16:59 +01:00
parent 3cdfcc1926
commit 62e34af892
20 changed files with 153 additions and 123 deletions

13
NEWS
View file

@ -1,3 +1,16 @@
20190122
* 'env_parallel --end-session' makes it possible to nest 'env_parallel
--session'
* Using GNU Parallel in pentesting https://youtu.be/mZ0OUJmkIlA?t=873
* Supercomputer Workshop: Feb 1&7
http://www.ece.unm.edu/featured-students/supercomputer-workshop-feb-17.html
* Bug fixes and man page updates.
20181222 20181222
* Better /proc/cpuinfo parser. * Better /proc/cpuinfo parser.

12
README
View file

@ -44,9 +44,9 @@ document.
Full installation of GNU Parallel is as simple as: Full installation of GNU Parallel is as simple as:
wget https://ftpmirror.gnu.org/parallel/parallel-20181222.tar.bz2 wget https://ftpmirror.gnu.org/parallel/parallel-20190122.tar.bz2
bzip2 -dc parallel-20181222.tar.bz2 | tar xvf - bzip2 -dc parallel-20190122.tar.bz2 | tar xvf -
cd parallel-20181222 cd parallel-20190122
./configure && make && sudo make install ./configure && make && sudo make install
@ -55,9 +55,9 @@ Full installation of GNU Parallel is as simple as:
If you are not root you can add ~/bin to your path and install in If you are not root you can add ~/bin to your path and install in
~/bin and ~/share: ~/bin and ~/share:
wget https://ftpmirror.gnu.org/parallel/parallel-20181222.tar.bz2 wget https://ftpmirror.gnu.org/parallel/parallel-20190122.tar.bz2
bzip2 -dc parallel-20181222.tar.bz2 | tar xvf - bzip2 -dc parallel-20190122.tar.bz2 | tar xvf -
cd parallel-20181222 cd parallel-20190122
./configure --prefix=$HOME && make && make install ./configure --prefix=$HOME && make && make install
Or if your system lacks 'make' you can simply copy src/parallel Or if your system lacks 'make' you can simply copy src/parallel

20
configure vendored
View file

@ -1,6 +1,6 @@
#! /bin/sh #! /bin/sh
# Guess values for system-dependent variables and create Makefiles. # Guess values for system-dependent variables and create Makefiles.
# Generated by GNU Autoconf 2.69 for parallel 20181222. # Generated by GNU Autoconf 2.69 for parallel 20190122.
# #
# Report bugs to <bug-parallel@gnu.org>. # Report bugs to <bug-parallel@gnu.org>.
# #
@ -579,8 +579,8 @@ MAKEFLAGS=
# Identity of this package. # Identity of this package.
PACKAGE_NAME='parallel' PACKAGE_NAME='parallel'
PACKAGE_TARNAME='parallel' PACKAGE_TARNAME='parallel'
PACKAGE_VERSION='20181222' PACKAGE_VERSION='20190122'
PACKAGE_STRING='parallel 20181222' PACKAGE_STRING='parallel 20190122'
PACKAGE_BUGREPORT='bug-parallel@gnu.org' PACKAGE_BUGREPORT='bug-parallel@gnu.org'
PACKAGE_URL='' PACKAGE_URL=''
@ -1214,7 +1214,7 @@ if test "$ac_init_help" = "long"; then
# Omit some internal or obsolete options to make the list less imposing. # Omit some internal or obsolete options to make the list less imposing.
# This message is too long to be a string in the A/UX 3.1 sh. # This message is too long to be a string in the A/UX 3.1 sh.
cat <<_ACEOF cat <<_ACEOF
\`configure' configures parallel 20181222 to adapt to many kinds of systems. \`configure' configures parallel 20190122 to adapt to many kinds of systems.
Usage: $0 [OPTION]... [VAR=VALUE]... Usage: $0 [OPTION]... [VAR=VALUE]...
@ -1281,7 +1281,7 @@ fi
if test -n "$ac_init_help"; then if test -n "$ac_init_help"; then
case $ac_init_help in case $ac_init_help in
short | recursive ) echo "Configuration of parallel 20181222:";; short | recursive ) echo "Configuration of parallel 20190122:";;
esac esac
cat <<\_ACEOF cat <<\_ACEOF
@ -1357,7 +1357,7 @@ fi
test -n "$ac_init_help" && exit $ac_status test -n "$ac_init_help" && exit $ac_status
if $ac_init_version; then if $ac_init_version; then
cat <<\_ACEOF cat <<\_ACEOF
parallel configure 20181222 parallel configure 20190122
generated by GNU Autoconf 2.69 generated by GNU Autoconf 2.69
Copyright (C) 2012 Free Software Foundation, Inc. Copyright (C) 2012 Free Software Foundation, Inc.
@ -1374,7 +1374,7 @@ cat >config.log <<_ACEOF
This file contains any messages produced by compilers while This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake. running configure, to aid debugging if configure makes a mistake.
It was created by parallel $as_me 20181222, which was It was created by parallel $as_me 20190122, which was
generated by GNU Autoconf 2.69. Invocation command line was generated by GNU Autoconf 2.69. Invocation command line was
$ $0 $@ $ $0 $@
@ -2237,7 +2237,7 @@ fi
# Define the identity of the package. # Define the identity of the package.
PACKAGE='parallel' PACKAGE='parallel'
VERSION='20181222' VERSION='20190122'
cat >>confdefs.h <<_ACEOF cat >>confdefs.h <<_ACEOF
@ -2880,7 +2880,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
# report actual input values of CONFIG_FILES etc. instead of their # report actual input values of CONFIG_FILES etc. instead of their
# values after options handling. # values after options handling.
ac_log=" ac_log="
This file was extended by parallel $as_me 20181222, which was This file was extended by parallel $as_me 20190122, which was
generated by GNU Autoconf 2.69. Invocation command line was generated by GNU Autoconf 2.69. Invocation command line was
CONFIG_FILES = $CONFIG_FILES CONFIG_FILES = $CONFIG_FILES
@ -2942,7 +2942,7 @@ _ACEOF
cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`" ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`"
ac_cs_version="\\ ac_cs_version="\\
parallel config.status 20181222 parallel config.status 20190122
configured by $0, generated by GNU Autoconf 2.69, configured by $0, generated by GNU Autoconf 2.69,
with options \\"\$ac_cs_config\\" with options \\"\$ac_cs_config\\"

View file

@ -1,4 +1,4 @@
AC_INIT([parallel], [20181222], [bug-parallel@gnu.org]) AC_INIT([parallel], [20190122], [bug-parallel@gnu.org])
AM_INIT_AUTOMAKE([-Wall -Werror foreign]) AM_INIT_AUTOMAKE([-Wall -Werror foreign])
AC_CONFIG_HEADERS([config.h]) AC_CONFIG_HEADERS([config.h])
AC_CONFIG_FILES([ AC_CONFIG_FILES([

View file

@ -32,7 +32,7 @@ It is even optional to run this, as GNU Parallel will work without having 'paral
> I do not write scientific articles. Does the notice apply to me? > I do not write scientific articles. Does the notice apply to me?
No. The notice only applies if you write scientific articles. The notice is only relevant if you write scientific articles.
> What shows citing software is an academic tradition? > What shows citing software is an academic tradition?
@ -51,7 +51,7 @@ These links say: Yes, you should cite software, and if the author suggests a way
If you feel the benefit from using GNU Parallel is too small to warrant a citation, then prove that by simply using another tool. If you replace your use of GNU Parallel with another tool, you obviously do not have to cite GNU Parallel. If it is too much work replacing the use of GNU Parallel, then it is a good indication that the benefit is big enough to warrant a citation. If you feel the benefit from using GNU Parallel is too small to warrant a citation, then prove that by simply using another tool. If you replace your use of GNU Parallel with another tool, you obviously do not have to cite GNU Parallel. If it is too much work replacing the use of GNU Parallel, then it is a good indication that the benefit is big enough to warrant a citation.
> Do other software show how to cite? > Do other software tools show how to cite?
Here are other examples of software showing how to cite. Some of these refer to peer-reviewed articles - others do not: Here are other examples of software showing how to cite. Some of these refer to peer-reviewed articles - others do not:
@ -77,7 +77,7 @@ This principle has even been tested in court:
http://www.inta.org/INTABulletin/Pages/GERMANYGeneralPublicLicenseDoesNotPermitUseofThird-PartyTrademarksforAdvertisingModifiedVersionsofOpen-SourceSoftware.aspx http://www.inta.org/INTABulletin/Pages/GERMANYGeneralPublicLicenseDoesNotPermitUseofThird-PartyTrademarksforAdvertisingModifiedVersionsofOpen-SourceSoftware.aspx
https://www.admody.com/urteilsdatenbank/cafe6fdaeed3/OLG-Duesseldorf_Urteil_vom_28-September-2010_Az_I-20-U-41-09 https://www.admody.com/urteilsdatenbank/cafe6fdaeed3/OLG-Duesseldorf_Urteil_vom_28-September-2010_Az_I-20-U-41-09
Also know that if you fork GNU Parallel and remove the notice, you are not helping to fund further develpment. So if you like GNU Parallel and want it to be maintained in the future, then this is a bad idea, as it will give less funding. Also know that if you fork GNU Parallel and remove the notice, you are not helping to fund further develpment. So if you like GNU Parallel and want it to be maintained in the future, then this is a bad idea, as it will lead to less funding.
> How important is the notice for the survival of GNU Parallel? > How important is the notice for the survival of GNU Parallel?

View file

@ -213,10 +213,14 @@ GNU Parallel 20190122 ('') <<[stable]>> has been released. It is available for d
Quote of the month: Quote of the month:
<<>> Ok! GNU Parallel is one of the best things out there. Almost as good as vanilla ice cream.
-- @coffe@mastodon.art
New in this release: New in this release:
* 'env_parallel --end-session' makes it possible to nest 'env_parallel --session'
* Using GNU Parallel in pentesting https://youtu.be/mZ0OUJmkIlA?t=873 * Using GNU Parallel in pentesting https://youtu.be/mZ0OUJmkIlA?t=873
* Supercomputer Workshop: Feb 1&7 http://www.ece.unm.edu/featured-students/supercomputer-workshop-feb-17.html * Supercomputer Workshop: Feb 1&7 http://www.ece.unm.edu/featured-students/supercomputer-workshop-feb-17.html

View file

@ -1,6 +1,6 @@
<directory name="parallel" rev="228" srcmd5="6b5a77aa91c7b5b1222e49ab36f563df" vrev="1"> <directory name="parallel" rev="230" srcmd5="dae05e9a043866038c83e6f5a5e56c3e" vrev="2">
<entry md5="04ebc06103bdf5ce70c360742adbcbd8" mtime="1545440269" name="parallel-20181222.tar.bz2" size="1799481" /> <entry md5="01a210240252d9618a5c17b12c585d54" mtime="1548036048" name="parallel-20190122.tar.bz2" size="1793620" />
<entry md5="68cf0f3d678a67b8317122c1f4c45afc" mtime="1545440269" name="parallel.spec" size="4667" /> <entry md5="13c250f5c33508b6d2f44544c38215ab" mtime="1548035924" name="parallel.spec" size="4667" />
<entry md5="0ee51c455e89f82f7c8f0dfe9310c250" mtime="1545440269" name="parallel_20181222.dsc" size="556" /> <entry md5="ec9efb3ebd9f192e9104ebd37269f579" mtime="1548035925" name="parallel_20190122.dsc" size="556" />
<entry md5="ddaa3054a0ab04e30ad38d2b32617a4a" mtime="1545440270" name="parallel_20181222.tar.gz" size="1995147" /> <entry md5="f74e15e4b9c93b4fc0793b979faf3200" mtime="1548035961" name="parallel_20190122.tar.gz" size="1997144" />
</directory> </directory>

View file

@ -1,7 +1,7 @@
Summary: Shell tool for executing jobs in parallel Summary: Shell tool for executing jobs in parallel
Name: parallel Name: parallel
Version: 20181222 Version: 20190122
Release: 1.2 Release: 1.2
License: GPL License: GPL
Group: Productivity/File utilities Group: Productivity/File utilities

View file

@ -1,7 +1,7 @@
Summary: Shell tool for executing jobs in parallel Summary: Shell tool for executing jobs in parallel
Name: parallel Name: parallel
Version: 20181222 Version: 20190122
Release: 1.2 Release: 1.2
License: GPL License: GPL
Group: Productivity/File utilities Group: Productivity/File utilities

View file

@ -99,6 +99,7 @@ Same as GNU B<parallel> in addition to these:
Undo last B<--session> Undo last B<--session>
=item B<--record-env> =item B<--record-env>
Record all names currently defined to be ignored every time running Record all names currently defined to be ignored every time running

View file

@ -23,7 +23,7 @@
use strict; use strict;
use Getopt::Long; use Getopt::Long;
$Global::progname="niceload"; $Global::progname="niceload";
$Global::version = 20181223; $Global::version = 20190122;
Getopt::Long::Configure("bundling","require_order"); Getopt::Long::Configure("bundling","require_order");
get_options_from_array(\@ARGV) || die_usage(); get_options_from_array(\@ARGV) || die_usage();
if($opt::version) { if($opt::version) {

View file

@ -1389,7 +1389,7 @@ Output can be buffered by using B<-d>. Output is buffered in memory,
so big output can cause swapping and therefore be terrible slow or so big output can cause swapping and therefore be terrible slow or
even cause out of memory. even cause out of memory.
https://github.com/gdm85/coshell https://github.com/gdm85/coshell (Last checked: 2019-01)
=head2 DIFFERENCES BETWEEN spread AND GNU Parallel =head2 DIFFERENCES BETWEEN spread AND GNU Parallel
@ -1416,8 +1416,7 @@ B<pyargs> deals badly with input containing spaces. It buffers stdout,
but not stderr. It buffers in RAM. {} does not work as replacement but not stderr. It buffers in RAM. {} does not work as replacement
string. It does not support running functions. string. It does not support running functions.
B<pyargs> does not support composed commands if run with B<--lines>, B<pyargs> does not support composed commands if run with B<--lines>.
and fails on B<pyargs traceroute gnu.org fsf.org>.
=head3 Examples =head3 Examples
@ -1450,7 +1449,7 @@ and fails on B<pyargs traceroute gnu.org fsf.org>.
# Similar, but not exactly the same # Similar, but not exactly the same
parallel seq ::: 1 2 3 4 5 6 parallel seq ::: 1 2 3 4 5 6
https://github.com/robertblackwell/pyargs https://github.com/robertblackwell/pyargs (Last checked: 2019-01)
=head2 DIFFERENCES BETWEEN concurrently AND GNU Parallel =head2 DIFFERENCES BETWEEN concurrently AND GNU Parallel
@ -1718,18 +1717,32 @@ dependency graph described in a file, so this is similar to B<make>.
https://github.com/cetra3/lorikeet (Last checked: 2018-10) https://github.com/cetra3/lorikeet (Last checked: 2018-10)
=head2 DIFFERENCES BETWEEN spp AND GNU Parallel =head2 DIFFERENCES BETWEEN spp AND GNU Parallel
B<spp> can run jobs in parallel. B<spp> does not use a command B<spp> can run jobs in parallel. B<spp> does not use a command
template to generate the jobs, but requires jobs to be in a template to generate the jobs, but requires jobs to be in a
file. Output from the jobs mix. file. Output from the jobs mix.
https://github.com/john01dav/spp https://github.com/john01dav/spp (Last checked: 2019-01)
=head2 Todo =head2 Todo
Url for spread Url for spread
https://github.com/reggi/pkgrun
https://github.com/benoror/better-npm-run
https://github.com/bahmutov/with-package
https://github.com/spion/npm-parallel
https://github.com/royriojas/shell-executor
https://github.com/darkguy2008/parallelshell
https://github.com/xuchenCN/go-pssh https://github.com/xuchenCN/go-pssh
https://github.com/amritb/with-this.git https://github.com/amritb/with-this.git
@ -1751,8 +1764,6 @@ https://github.com/amattn/paral
https://github.com/mmstick/concurr https://github.com/mmstick/concurr
pyargs
=head1 TESTING OTHER TOOLS =head1 TESTING OTHER TOOLS

View file

@ -574,7 +574,7 @@ $Global::Initfile && unlink $Global::Initfile;
exit ($err); exit ($err);
sub parse_options { sub parse_options {
$Global::version = 20181223; $Global::version = 20190122;
$Global::progname = 'sql'; $Global::progname = 'sql';
# This must be done first as this may exec myself # This must be done first as this may exec myself

View file

@ -82,7 +82,7 @@ linebuffer_matters() {
# Ignore certain warnings # Ignore certain warnings
# parallel: Warning: Starting 11 processes took > 2 sec. # parallel: Warning: Starting 11 processes took > 2 sec.
# parallel: Warning: Consider adjusting -j. Press CTRL-C to stop. # parallel: Warning: Consider adjusting -j. Press CTRL-C to stop.
grep -v '^parallel: Warning: (Starting|Consider)' grep -v '^parallel: Warning: (Starting|Consider)' >&2
} }
parallel -j0 $linebuffer --compress $TAG \ parallel -j0 $linebuffer --compress $TAG \
@ -171,7 +171,7 @@ par_test_detected_shell() {
} }
export -f test_known_shell_pipe export -f test_known_shell_pipe
stdout parallel -j0 --tag -k \ stdout parallel -j2 --tag -k \
::: test_unknown_shell test_known_shell_c test_known_shell_pipe \ ::: test_unknown_shell test_known_shell_c test_known_shell_pipe \
::: $shells | ::: $shells |
grep -Ev 'parallel: Warning: (Starting .* processes took|Consider adjusting)' grep -Ev 'parallel: Warning: (Starting .* processes took|Consider adjusting)'
@ -195,7 +195,7 @@ par_linebuffer_files() {
rm -rf "/tmp/par48658-$compress" rm -rf "/tmp/par48658-$compress"
} }
export -f doit export -f doit
parallel --tag -k doit ::: zstd pzstd clzip lz4 lzop pigz pxz gzip plzip pbzip2 lzma xz lzip bzip2 lbzip2 lrz parallel -j1 --tag -k doit ::: zstd pzstd clzip lz4 lzop pigz pxz gzip plzip pbzip2 lzma xz lzip bzip2 lbzip2 lrz
} }
par_no_newline_compress() { par_no_newline_compress() {
@ -217,7 +217,7 @@ par_no_newline_compress() {
echo "K" echo "K"
} }
export -f nopipe_doit export -f nopipe_doit
parallel -qk --header : {pipe}_doit {tagstring} {compress} \ parallel -j1 -qk --header : {pipe}_doit {tagstring} {compress} \
::: tagstring '--tagstring {#}' -k \ ::: tagstring '--tagstring {#}' -k \
::: compress --compress -k \ ::: compress --compress -k \
::: pipe pipe nopipe ::: pipe pipe nopipe

View file

@ -1,5 +1,47 @@
#!/bin/bash #!/bin/bash
par_parcat_mixing() {
echo 'parcat output should mix: a b a b'
mktmpfifo() {
tmp=$(tempfile)
rm $tmp
mkfifo $tmp
echo $tmp
}
slow_output() {
string=$1
perl -e 'print "'$string'"x9000,"start\n"'
sleep 2
perl -e 'print "'$string'"x9000,"end\n"'
}
tmp1=$(mktmpfifo)
tmp2=$(mktmpfifo)
slow_output a > $tmp1 &
sleep 1
slow_output b > $tmp2 &
parcat $tmp1 $tmp2 | tr -s ab
}
par_tmux_termination() {
echo '### --tmux test - check termination'
doit() {
perl -e 'map {printf "$_%o%c\n",$_,$_}1..255' |
stdout parallel --tmux 'sleep 0.2;echo {}' :::: - ::: a b |
perl -pe 's:(/tmp\S*/tms).....:$1XXXXX:;'
}
export -f doit
stdout parallel --timeout 120 doit ::: 1
}
par_linebuffer_tag_slow_output() {
echo "Test output tag with mixing halflines"
halfline() {
perl -e '$| = 1; map { print $ARGV[0]; sleep(1); print "$_\n" } split //, "Half\n"' $1
}
export -f halfline
parallel --delay 0.5 -j0 --tag --line-buffer halfline ::: a b
}
par_distribute_input_by_ability() { par_distribute_input_by_ability() {
echo "### bug #48290: round-robin does not distribute data based on business" echo "### bug #48290: round-robin does not distribute data based on business"
echo "### Distribute input to jobs that are ready" echo "### Distribute input to jobs that are ready"

View file

@ -29,50 +29,6 @@
echo 1=OK $?' | grep -v '\[1\]' | grep -v 'SHA256' echo 1=OK $?' | grep -v '\[1\]' | grep -v 'SHA256'
#} #}
par_parcat_mixing() {
echo 'parcat output should mix: a b a b'
mktmpfifo() {
tmp=$(tempfile)
rm $tmp
mkfifo $tmp
echo $tmp
}
slow_output() {
string=$1
perl -e 'print "'$string'"x9000,"start\n"'
sleep 2
perl -e 'print "'$string'"x9000,"end\n"'
}
tmp1=$(mktmpfifo)
tmp2=$(mktmpfifo)
slow_output a > $tmp1 &
sleep 1
slow_output b > $tmp2 &
parcat $tmp1 $tmp2 | tr -s ab
}
par_testhalt() {
testhalt_false() {
echo '### testhalt --halt '$1;
(yes 0 | head -n 10; seq 10) |
stdout parallel -kj4 --delay 0.27 --halt $1 \
'echo job {#}; sleep {= $_=0.3*($_+1+seq()) =}; exit {}'; echo $?;
}
testhalt_true() {
echo '### testhalt --halt '$1;
(seq 10; yes 0 | head -n 10) |
stdout parallel -kj4 --delay 0.17 --halt $1 \
'echo job {#}; sleep {= $_=0.3*($_+1+seq()) =}; exit {}'; echo $?;
};
export -f testhalt_false;
export -f testhalt_true;
stdout parallel -kj0 --delay 0.11 --tag testhalt_{4} {1},{2}={3} \
::: now soon ::: fail success done ::: 0 1 2 30% 70% ::: true false |
# Remove lines that only show up now and then
perl -ne '/Starting no more jobs./ or print'
}
par_hostgroup() { par_hostgroup() {
echo '### --hostgroup force ncpu' echo '### --hostgroup force ncpu'
parallel --delay 0.1 --hgrp -S @g1/1/parallel@lo -S @g2/3/lo whoami\;sleep 0.4{} ::: {1..8} | sort parallel --delay 0.1 --hgrp -S @g1/1/parallel@lo -S @g2/3/lo whoami\;sleep 0.4{} ::: {1..8} | sort
@ -105,24 +61,26 @@ par_hostgroup() {
parallel -S @g1+g2 -S @g1/1/tcsh@lo -S @g1/1/localhost -S @g2/1/parallel@lo whoami\;true ::: {1..6} | sort parallel -S @g1+g2 -S @g1/1/tcsh@lo -S @g1/1/localhost -S @g2/1/parallel@lo whoami\;true ::: {1..6} | sort
} }
par_tmux_termination() { par_testhalt() {
echo '### --tmux test - check termination' testhalt_false() {
doit() { echo '### testhalt --halt '$1;
perl -e 'map {printf "$_%o%c\n",$_,$_}1..255' | (yes 0 | head -n 10; seq 10) |
stdout parallel --tmux 'sleep 0.2;echo {}' :::: - ::: a b | stdout parallel -kj4 --delay 0.27 --halt $1 \
perl -pe 's:(/tmp\S*/tms).....:$1XXXXX:;' 'echo job {#}; sleep {= $_=0.3*($_+1+seq()) =}; exit {}'; echo $?;
} }
export -f doit testhalt_true() {
stdout parallel --timeout 120 doit ::: 1 echo '### testhalt --halt '$1;
} (seq 10; yes 0 | head -n 10) |
stdout parallel -kj4 --delay 0.17 --halt $1 \
'echo job {#}; sleep {= $_=0.3*($_+1+seq()) =}; exit {}'; echo $?;
};
export -f testhalt_false;
export -f testhalt_true;
par_linebuffer_tag_slow_output() { stdout parallel -kj0 --delay 0.11 --tag testhalt_{4} {1},{2}={3} \
echo "Test output tag with mixing halflines" ::: now soon ::: fail success done ::: 0 1 2 30% 70% ::: true false |
halfline() { # Remove lines that only show up now and then
perl -e '$| = 1; map { print $ARGV[0]; sleep(1); print "$_\n" } split //, "Half\n"' $1 perl -ne '/Starting no more jobs./ or print'
}
export -f halfline
parallel --delay 0.5 -j0 --tag --line-buffer halfline ::: a b
} }
par_continuous_output() { par_continuous_output() {

View file

@ -148,5 +148,5 @@ export -f $(compgen -A function | egrep 'p_|par_')
# Tested that -j0 in parallel is fastest (up to 15 jobs) # Tested that -j0 in parallel is fastest (up to 15 jobs)
# more than 3 jobs: sqlite locks # more than 3 jobs: sqlite locks
compgen -A function | grep par_ | LC_ALL=C sort | compgen -A function | grep par_ | LC_ALL=C sort |
stdout parallel --timeout 60 -vj50% -k --tag --joblog /tmp/jl-`basename $0` p_wrapper \ stdout parallel --timeout 100 -vj50% -k --tag --joblog /tmp/jl-`basename $0` p_wrapper \
:::: - ::: \$MYSQL \$PG \$SQLITE :::: - ::: \$MYSQL \$PG \$SQLITE

View file

@ -173,7 +173,7 @@ par_no_route_to_host() {
perl -ne 's/ssh:.* host (\d+\.\d+\.\d+\.\d+) .* No route .*/$1/ and print; $|=1' perl -ne 's/ssh:.* host (\d+\.\d+\.\d+\.\d+) .* No route .*/$1/ and print; $|=1'
} }
# Retry if really fails this fast # Retry if the hosts really fails this fast
filterhosts() { filterhosts() {
stdout parallel --timeout 2 -j5 ssh {} echo | stdout parallel --timeout 2 -j5 ssh {} echo |
perl -ne 's/ssh:.* host (\d+\.\d+\.\d+\.\d+) .* No route .*/$1/ and print; $|=1' perl -ne 's/ssh:.* host (\d+\.\d+\.\d+\.\d+) .* No route .*/$1/ and print; $|=1'
@ -181,6 +181,7 @@ par_no_route_to_host() {
( (
# Cache a list of hosts that fail fast with 'No route' # Cache a list of hosts that fail fast with 'No route'
# Filter the list 4 times to make sure to get good hosts
renice 10 -p $$ >/dev/null renice 10 -p $$ >/dev/null
findhosts | filterhosts | filterhosts | findhosts | filterhosts | filterhosts |
filterhosts | filterhosts | head > /tmp/filtered.$$ filterhosts | filterhosts | head > /tmp/filtered.$$

View file

@ -8,6 +8,24 @@ par_distribute_input_by_ability 4
par_distribute_input_by_ability 5 par_distribute_input_by_ability 5
par_distribute_input_by_ability 6 par_distribute_input_by_ability 6
par_distribute_input_by_ability 7 par_distribute_input_by_ability 7
par_linebuffer_tag_slow_output Test output tag with mixing halflines
par_linebuffer_tag_slow_output a aH
par_linebuffer_tag_slow_output b bH
par_linebuffer_tag_slow_output a aa
par_linebuffer_tag_slow_output b ba
par_linebuffer_tag_slow_output a al
par_linebuffer_tag_slow_output b bl
par_linebuffer_tag_slow_output a af
par_linebuffer_tag_slow_output b bf
par_linebuffer_tag_slow_output a a
par_linebuffer_tag_slow_output a
par_linebuffer_tag_slow_output b b
par_linebuffer_tag_slow_output b
par_parcat_mixing parcat output should mix: a b a b
par_parcat_mixing astart
par_parcat_mixing bstart
par_parcat_mixing aend
par_parcat_mixing bend
par_print_before_halt_on_error ### What is printed before the jobs are killed par_print_before_halt_on_error ### What is printed before the jobs are killed
par_print_before_halt_on_error -2 exit code 0 par_print_before_halt_on_error -2 exit code 0
par_print_before_halt_on_error -2 0.1 0.1 par_print_before_halt_on_error -2 0.1 0.1
@ -61,3 +79,5 @@ par_print_before_halt_on_error 2 exit code 1
par_print_before_halt_on_error 2 0.1 0.1 par_print_before_halt_on_error 2 0.1 0.1
par_print_before_halt_on_error 2 parallel: This job failed: par_print_before_halt_on_error 2 parallel: This job failed:
par_print_before_halt_on_error 2 perl -e 'sleep 1; sleep $ARGV[0]; print STDERR "",@ARGV,"\n"; 2 > 0 ? exit shift : exit not shift;' 1 par_print_before_halt_on_error 2 perl -e 'sleep 1; sleep $ARGV[0]; print STDERR "",@ARGV,"\n"; 2 > 0 ? exit shift : exit not shift;' 1
par_tmux_termination ### --tmux test - check termination
par_tmux_termination See output with: tmux -S /tmp/parallel-local-race01-tmpdir/tmsXXXXX attach

View file

@ -104,24 +104,6 @@ par_hostgroup tange
par_hostgroup tange par_hostgroup tange
par_hostgroup tcsh par_hostgroup tcsh
par_hostgroup tcsh par_hostgroup tcsh
par_linebuffer_tag_slow_output Test output tag with mixing halflines
par_linebuffer_tag_slow_output a aH
par_linebuffer_tag_slow_output b bH
par_linebuffer_tag_slow_output a aa
par_linebuffer_tag_slow_output b ba
par_linebuffer_tag_slow_output a al
par_linebuffer_tag_slow_output b bl
par_linebuffer_tag_slow_output a af
par_linebuffer_tag_slow_output b bf
par_linebuffer_tag_slow_output a a
par_linebuffer_tag_slow_output a
par_linebuffer_tag_slow_output b b
par_linebuffer_tag_slow_output b
par_parcat_mixing parcat output should mix: a b a b
par_parcat_mixing astart
par_parcat_mixing bstart
par_parcat_mixing aend
par_parcat_mixing bend
par_testhalt now fail 0 true ### testhalt --halt now,fail=0 par_testhalt now fail 0 true ### testhalt --halt now,fail=0
par_testhalt now fail 0 true job 1 par_testhalt now fail 0 true job 1
par_testhalt now fail 0 true parallel: This job failed: par_testhalt now fail 0 true parallel: This job failed:
@ -1462,5 +1444,3 @@ par_testhalt soon done 70% false job 17
par_testhalt soon done 70% false parallel: This job finished: par_testhalt soon done 70% false parallel: This job finished:
par_testhalt soon done 70% false echo job 17; sleep 7.5; exit 7 par_testhalt soon done 70% false echo job 17; sleep 7.5; exit 7
par_testhalt soon done 70% false 20 par_testhalt soon done 70% false 20
par_tmux_termination ### --tmux test - check termination
par_tmux_termination See output with: tmux -S /tmp/parallel-local-race02-tmpdir/tmsXXXXX attach