parallel: Don't postpone output until $jobslots jobs have been started.

This commit is contained in:
Ole Tange 2021-10-03 20:31:55 +02:00
parent 123126f678
commit 917c1795f0
29 changed files with 179 additions and 127 deletions

View file

@ -234,7 +234,6 @@ New in this release pi.dk/2
See the intro videos pi.dk/1 See the intro videos pi.dk/1
10 secs installation: https://git.savannah.gnu.org/cgit/parallel.git/tree/README 10 secs installation: https://git.savannah.gnu.org/cgit/parallel.git/tree/README
[x] Twitter
Aspect: Public Aspect: Public
GNU Parallel - for people who live life in the parallel lane. GNU Parallel - for people who live life in the parallel lane.
@ -255,31 +254,32 @@ from:tange@gnu.org
to:parallel@gnu.org, bug-parallel@gnu.org to:parallel@gnu.org, bug-parallel@gnu.org
stable-bcc: Jesse Alama <jessealama@fastmail.fm> stable-bcc: Jesse Alama <jessealama@fastmail.fm>
Subject: GNU Parallel 20210922 ('<<>>Vindelev/Chris Donahue/Ida') released <<[stable]>> Subject: GNU Parallel 20211022 ('<<>>Scholz') released <<[stable]>>
GNU Parallel 20210922 ('Vindelev') [stable] has been released. It is available for download at: lbry://@GnuParallel:4 GNU Parallel 20211022 ('<<>>') <<[stable]>> has been released. It is available for download at: lbry://@GnuParallel:4
No new functionality was introduced so this is a good candidate for a <<No new functionality was introduced so this is a good candidate for a
stable release. stable release.>>
Quote of the month: Quote of the month:
Take a look at GNU Parallel. Its magnificent. <<>>
-- Siira@ycombinator.com
New in this release: New in this release:
* Bug fixes and man page updates. * Bug fixes and man page updates.
News about GNU Parallel: News about GNU Parallel:
https://medium.com/codex/3-programs-that-will-level-up-your-bash-game-bf14e17e2291
* 3 Programs that will Level Up your Bash Game https://medium.com/codex/3-programs-that-will-level-up-your-bash-game-bf14e17e2291 https://towardsdatascience.com/i-summarized-100-command-line-tools-for-data-scientists-in-15-minutes-976def544ffe
https://www.incredibuild.com/blog/speed-up-your-builds-by-parallelizing
* I Summarized 100+ Command-Line Tools for Data Scientists in 15 Minutes https://towardsdatascience.com/i-summarized-100-command-line-tools-for-data-scientists-in-15-minutes-976def544ffe https://gist.github.com/dinh/93b936a55de9a331199d6582bc2921d2
* Speed up Your Builds by Parallelizing https://www.incredibuild.com/blog/speed-up-your-builds-by-parallelizing
* Bash: GNU Parallel with Curl https://gist.github.com/dinh/93b936a55de9a331199d6582bc2921d2
Get the book: GNU Parallel 2018 http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html Get the book: GNU Parallel 2018 http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html

View file

@ -385,7 +385,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210922 (GNU parallel `parallel --minversion 1`)" echo "parset 20210923 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -384,7 +384,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210922 (GNU parallel `parallel --minversion 1`)" echo "parset 20210923 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -385,7 +385,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210922 (GNU parallel `parallel --minversion 1`)" echo "parset 20210923 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -363,7 +363,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210922 (GNU parallel `parallel --minversion 1`)" echo "parset 20210923 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -365,7 +365,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210922 (GNU parallel `parallel --minversion 1`)" echo "parset 20210923 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -390,7 +390,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210922 (GNU parallel `parallel --minversion 1`)" echo "parset 20210923 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -355,7 +355,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210922 (GNU parallel `parallel --minversion 1`)" echo "parset 20210923 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -26,7 +26,7 @@
use strict; use strict;
use Getopt::Long; use Getopt::Long;
$Global::progname="niceload"; $Global::progname="niceload";
$Global::version = 20210922; $Global::version = 20210923;
Getopt::Long::Configure("bundling","require_order"); Getopt::Long::Configure("bundling","require_order");
get_options_from_array(\@ARGV) || die_usage(); get_options_from_array(\@ARGV) || die_usage();
if($opt::version) { if($opt::version) {

View file

@ -2244,7 +2244,7 @@ sub check_invalid_option_combinations() {
sub init_globals() { sub init_globals() {
# Defaults: # Defaults:
$Global::version = 20210922; $Global::version = 20210923;
$Global::progname = 'parallel'; $Global::progname = 'parallel';
$::name = "GNU Parallel"; $::name = "GNU Parallel";
$Global::infinity = 2**31; $Global::infinity = 2**31;
@ -7263,6 +7263,9 @@ sub compute_number_of_processes($) {
$count_jobs_already_read = $Global::JobQueue->next_seq(); $count_jobs_already_read = $Global::JobQueue->next_seq();
my $wait_time_for_getting_args = 0; my $wait_time_for_getting_args = 0;
my $start_time = time; my $start_time = time;
if($wanted_processes < $Global::infinity) {
$Global::dummy_jobs = 1;
}
while(1) { while(1) {
$system_limit >= $wanted_processes and last; $system_limit >= $wanted_processes and last;
not $more_filehandles and last; not $more_filehandles and last;
@ -11577,7 +11580,6 @@ sub len($) {
# Worst case is BASE64 encoding 3 bytes -> 4 bytes # Worst case is BASE64 encoding 3 bytes -> 4 bytes
$len = int($len*4/3); $len = int($len*4/3);
} }
return $len; return $len;
} }

View file

@ -1197,8 +1197,17 @@ See also: B<--resume> B<--resume-failed>.
=item B<-P> I<N> =item B<-P> I<N>
Number of jobslots on each machine. Run up to N jobs in parallel. 0 Number of jobslots on each machine. Run up to N jobs in parallel. 0
means as many as possible. Default is 100% which will run one job per means as many as possible (this can take a while to
CPU on each machine. determine). Default is 100% which will run one job per CPU on each
machine.
Due to a bug B<-j 0> will also evaluate replacement strings twice up
to the number of joblots:
# This will not count from 1 but from number-of-jobslots
seq 10000 | parallel -j0 echo '{= $_ = $foo++; =}' | head
# This will count from 1
seq 10000 | parallel -j100 echo '{= $_ = $foo++; =}' | head
If B<--semaphore> is set, the default is 1 thus making a mutex. If B<--semaphore> is set, the default is 1 thus making a mutex.
@ -4773,6 +4782,10 @@ computers:
true >jobqueue; tail -n+0 -f jobqueue | parallel -S .. true >jobqueue; tail -n+0 -f jobqueue | parallel -S ..
Output only will be printed when reading the next input after a job
has finished: So you need to submit a job after the first has finished
to see the output from the first job.
If you keep this running for a long time, jobqueue will grow. A way of If you keep this running for a long time, jobqueue will grow. A way of
removing the jobs already run is by making GNU B<parallel> stop when removing the jobs already run is by making GNU B<parallel> stop when
it hits a special value and then restart. To use B<--eof> to make GNU it hits a special value and then restart. To use B<--eof> to make GNU
@ -4802,20 +4815,6 @@ In some cases you can run on more CPUs and computers during the night:
GNU B<parallel> discovers if B<jobfile> or B<~/.parallel/sshloginfile> GNU B<parallel> discovers if B<jobfile> or B<~/.parallel/sshloginfile>
changes. changes.
There is a a small issue when using GNU B<parallel> as queue
system/batch manager: You have to submit JobSlot number of jobs before
they will start, and after that you can submit one at a time, and job
will start immediately if free slots are available.
Output from the running or completed jobs are held back and will only
be printed when the next job is started (unless you use --ungroup or
--line-buffer, in which case the output from the jobs are printed
immediately).
E.g. if you have 10 jobslots then the output from the first completed
job will only be printed when job 11 has started, and the output of
second completed job will only be printed when job 12 has started.
=head2 EXAMPLE: GNU Parallel as dir processor =head2 EXAMPLE: GNU Parallel as dir processor

View file

@ -122,7 +122,7 @@ GetOptions(
"help" => \$opt::dummy, "help" => \$opt::dummy,
) || exit(255); ) || exit(255);
$Global::progname = ($0 =~ m:(^|/)([^/]+)$:)[1]; $Global::progname = ($0 =~ m:(^|/)([^/]+)$:)[1];
$Global::version = 20210922; $Global::version = 20210923;
if($opt::version) { version(); exit 0; } if($opt::version) { version(); exit 0; }
@Global::sortoptions = @Global::sortoptions =
shell_quote(@ARGV_before[0..($#ARGV_before-$#ARGV-1)]); shell_quote(@ARGV_before[0..($#ARGV_before-$#ARGV-1)]);

View file

@ -600,7 +600,7 @@ $Global::Initfile && unlink $Global::Initfile;
exit ($err); exit ($err);
sub parse_options { sub parse_options {
$Global::version = 20210922; $Global::version = 20210923;
$Global::progname = 'sql'; $Global::progname = 'sql';
# This must be done first as this may exec myself # This must be done first as this may exec myself

View file

@ -20,10 +20,12 @@ stdout ping -w 1 -c 1 centos3 >/dev/null || (
cd testsuite/vagrant/tange/centos3/ 2>/dev/null cd testsuite/vagrant/tange/centos3/ 2>/dev/null
cd vagrant/tange/centos3/ 2>/dev/null cd vagrant/tange/centos3/ 2>/dev/null
cd ../vagrant/tange/centos3/ 2>/dev/null cd ../vagrant/tange/centos3/ 2>/dev/null
(
stdout vagrant up >/dev/null stdout vagrant up >/dev/null
vagrant ssh -c 'sudo ifconfig eth1 172.27.27.3' | vagrant ssh -c 'sudo ifconfig eth1 172.27.27.3' |
# Ignore empty ^M line # Ignore empty ^M line
grep .. grep ..
) &
) )
( (
# Copy binaries to server # Copy binaries to server

View file

@ -16,6 +16,23 @@ export -f stdsort
# Test amount of parallelization # Test amount of parallelization
# parallel --shuf --jl /tmp/myjl -j1 'export JOBS={1};'bash tests-to-run/parallel-local-0.3s.sh ::: {1..16} ::: {1..5} # parallel --shuf --jl /tmp/myjl -j1 'export JOBS={1};'bash tests-to-run/parallel-local-0.3s.sh ::: {1..16} ::: {1..5}
par_long_input() {
echo '### Long input lines should not fail if they are not used'
perl -e 'map { print "$_\t"."X"x100000 ."\t".uc($_)."\n" } (a..c)' |
parallel --colsep '\t' echo {1}
perl -e 'map { print "$_\t"."X"x100000 ."\t".uc($_)."\n" } (a..c)' |
parallel --colsep '\t' echo {3}
perl -e 'map { print "$_\t"."X"x100000 ."\t".uc($_)."\n" } (a..c)' |
parallel --colsep '\t' echo {1} {3}
perl -e 'map { print "$_\t"."X"x1000000 ."\t".uc($_)."\n" } (a..c)' |
parallel --colsep '\t' echo {1}
perl -e 'map { print "$_\t"."X"x1000000 ."\t".uc($_)."\n" } (a..c)' |
parallel --colsep '\t' echo {3}
perl -e 'map { print "$_\t"."X"x1000000 ."\t".uc($_)."\n" } (a..c)' |
parallel --colsep '\t' echo {1} {3}
}
par_ctagstring() { par_ctagstring() {
echo '### --ctag --ctagstring should be different from --tag --tagstring' echo '### --ctag --ctagstring should be different from --tag --tagstring'
parallel --tag echo ::: 1 ::: a| wc -c parallel --tag echo ::: 1 ::: a| wc -c
@ -92,7 +109,7 @@ EOF
--tmpl "$tmp2"=/tmp/tmpl-{x}-{y}.t2 \ --tmpl "$tmp2"=/tmp/tmpl-{x}-{y}.t2 \
myprog {#}.t1 /tmp/tmpl-{x}-{y}.t2 \ myprog {#}.t1 /tmp/tmpl-{x}-{y}.t2 \
::: x 1.1 2.22 3.333 ::: y 111.111 222.222 333.333 | ::: x 1.1 2.22 3.333 ::: y 111.111 222.222 333.333 |
perl -pe 's/0.\d{13,}/0.RANDOM_NUMBER/' | perl -pe 's/0.\d{12,}/0.RANDOM_NUMBER/' |
perl -pe 's/Slot: \d/Slot: X/' perl -pe 's/Slot: \d/Slot: X/'
rm "$tmp1" "$tmp2" rm "$tmp1" "$tmp2"
} }

View file

@ -10,9 +10,9 @@
par_plus_slot_replacement() { par_plus_slot_replacement() {
echo '### show {slot} {0%} {0#}' echo '### show {slot} {0%} {0#}'
parallel -k --plus echo '{slot}=$PARALLEL_JOBSLOT={%}' ::: A B C parallel -k --plus 'sleep 0.{%};echo {slot}=$PARALLEL_JOBSLOT={%}' ::: A B C
parallel -j15 -k --plus 'echo Seq: {0#} {#}' ::: {1..100} | sort parallel -j15 -k --plus 'echo Seq: {0#} {#}' ::: {1..100} | sort
parallel -j15 -k --plus 'sleep 0.0{}; echo Slot: {0%} {%}' ::: {1..100} | parallel -j15 -k --plus 'sleep 0.{}; echo Slot: {0%} {%}' ::: {1..100} |
sort -u sort -u
} }

View file

@ -338,21 +338,21 @@ par_totaljob_repl() {
parallel -k -N7 --plus echo {#} {##} ::: {1..14} parallel -k -N7 --plus echo {#} {##} ::: {1..14}
parallel -k -N7 --plus echo {#} {##} ::: {1..15} parallel -k -N7 --plus echo {#} {##} ::: {1..15}
parallel -k -S 8/: -X --plus echo {#} {##} ::: {1..15} parallel -k -S 8/: -X --plus echo {#} {##} ::: {1..15}
parallel -k --plus --delay 0.1 -j 10 'echo {0#}/{##}:{0%}' ::: {1..5} ::: {1..4} parallel -k --plus --delay 0.1 -j 10 'sleep 1; echo {0#}/{##}:{0%}' ::: {1..5} ::: {1..4}
} }
par_jobslot_repl() { par_jobslot_repl() {
echo 'bug #46232: {%} with --bar/--eta/--shuf or --halt xx% broken' echo 'bug #46232: {%} with --bar/--eta/--shuf or --halt xx% broken'
parallel --bar -kj2 --delay 0.1 echo {%} ::: a b ::: c d e 2>/dev/null parallel -kj2 --delay 0.1 --bar 'sleep 0.2;echo {%}' ::: a b ::: c d e 2>/dev/null
parallel --halt now,fail=10% -kj2 --delay 0.1 echo {%} ::: a b ::: c d e parallel -kj2 --delay 0.1 --eta 'sleep 0.2;echo {%}' ::: a b ::: c d e 2>/dev/null
parallel --eta -kj2 --delay 0.1 echo {%} ::: a b ::: c d e 2>/dev/null parallel -kj2 --delay 0.1 --shuf 'sleep 0.2;echo {%}' ::: a b ::: c d e 2>/dev/null
parallel --shuf -kj2 --delay 0.1 echo {%} ::: a b ::: c d e 2>/dev/null parallel -kj2 --delay 0.1 --halt now,fail=10% 'sleep 0.2;echo {%}' ::: a b ::: c d e
echo 'bug #46231: {%} with --pipepart broken. Should give 1+2' echo 'bug #46231: {%} with --pipepart broken. Should give 1+2'
seq 10000 > /tmp/num10000 seq 10000 > /tmp/num10000
parallel -k --pipepart -ka /tmp/num10000 --block 10k -j2 --delay 0.05 echo {%} parallel -k --pipepart -ka /tmp/num10000 --block 10k -j2 --delay 0.05 'sleep 0.1; echo {%}'
rm /tmp/num10000 rm /tmp/num10000
} }

View file

@ -6,7 +6,7 @@
# These fail regularly # These fail regularly
#par_ctrlz_should_suspend_children() { ctrlz_should_suspend_children() {
echo 'bug #46120: Suspend should suspend (at least local) children' echo 'bug #46120: Suspend should suspend (at least local) children'
echo 'it should burn 1.9 CPU seconds, but no more than that' echo 'it should burn 1.9 CPU seconds, but no more than that'
echo 'The 5 second sleep will make it be killed by timeout when it fgs' echo 'The 5 second sleep will make it be killed by timeout when it fgs'
@ -31,7 +31,14 @@
sleep 5; sleep 5;
fg; fg;
echo 1=OK $?' | grep -v '\[1\]' | grep -v 'SHA256' echo 1=OK $?' | grep -v '\[1\]' | grep -v 'SHA256'
#} }
ctrlz_should_suspend_children
par_more_than_9_relative_sshlogin() {
echo '### Check more than 9(relative) simultaneous sshlogins'
seq 1 11 | stdout parallel -k -j10000% -S "ssh lo" echo |
grep -v 'parallel: Warning:'
}
par_sql_CSV() { par_sql_CSV() {
echo '### CSV write to the right place' echo '### CSV write to the right place'

View file

@ -54,5 +54,7 @@ compgen -A function | grep par_ | LC_ALL=C sort |
grep -v 'If you believe this message is in error, please check the process' | grep -v 'If you believe this message is in error, please check the process' |
grep -v 'listing for any "ruby" or "vagrant" processes and kill them. Then' | grep -v 'listing for any "ruby" or "vagrant" processes and kill them. Then' |
grep -v 'try again.' | grep -v 'try again.' |
grep -v 'A new version of Vagrant is available:' |
grep -v 'To upgrade visit: ' |
grep . grep .
) )

View file

@ -54,10 +54,7 @@ echo '### Check forced number of CPUs being respected'
seq 1 20 | stdout parallel -k -j+0 -S 1/:,7/$SSHLOGIN1 "hostname; echo {} >/dev/null" | sort seq 1 20 | stdout parallel -k -j+0 -S 1/:,7/$SSHLOGIN1 "hostname; echo {} >/dev/null" | sort
echo '### Check more than 9 simultaneous sshlogins' echo '### Check more than 9 simultaneous sshlogins'
seq 1 11 | parallel -k -j0 -S "ssh $SSHLOGIN1" echo seq 1 11 | parallel -k -j0 -S "ssh lo" echo
echo '### Check more than 9(relative) simultaneous sshlogins'
seq 1 11 | parallel -k -j10000% -S "ssh $SSHLOGIN1" echo
echo '### Check -S syntax' echo '### Check -S syntax'
seq 1 11 | parallel -k -j100% -S : echo seq 1 11 | parallel -k -j100% -S : echo

View file

@ -492,6 +492,17 @@ par_long_cmd_mem_use perl -e '$r=rand(shift); for($f = 0; $f < $r; $f++){ $a = "
par_long_cmd_mem_use perl -e '$r=rand(shift); for($f = 0; $f < $r; $f++){ $a = "a"x100 } print shift,"\n"' 10000 97 par_long_cmd_mem_use perl -e '$r=rand(shift); for($f = 0; $f < $r; $f++){ $a = "a"x100 } print shift,"\n"' 10000 97
par_long_cmd_mem_use perl -e '$r=rand(shift); for($f = 0; $f < $r; $f++){ $a = "a"x100 } print shift,"\n"' 10000 98 par_long_cmd_mem_use perl -e '$r=rand(shift); for($f = 0; $f < $r; $f++){ $a = "a"x100 } print shift,"\n"' 10000 98
par_long_cmd_mem_use perl -e '$r=rand(shift); for($f = 0; $f < $r; $f++){ $a = "a"x100 } print shift,"\n"' 10000 99 par_long_cmd_mem_use perl -e '$r=rand(shift); for($f = 0; $f < $r; $f++){ $a = "a"x100 } print shift,"\n"' 10000 99
par_long_input ### Long input lines should not fail if they are not used
par_long_input a
par_long_input c
par_long_input b
par_long_input A
par_long_input C
par_long_input B
par_long_input parallel: Error: Command line too long (200014 >= 131063) at input 0: a XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...
par_long_input parallel: Error: Command line too long (1000009 >= 131063) at input 0: a XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...
par_long_input parallel: Error: Command line too long (1000009 >= 131063) at input 0: a XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...
par_long_input parallel: Error: Command line too long (2000014 >= 131063) at input 0: a XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...
par_maxchar_s ### Test --max-chars and -s: Max number of chars in a line par_maxchar_s ### Test --max-chars and -s: Max number of chars in a line
par_maxchar_s line 1 line 1 par_maxchar_s line 1 line 1
par_maxchar_s line 2 par_maxchar_s line 2

View file

@ -698,38 +698,38 @@ par_perlexpr_repl {=a
par_perlexpr_repl {= a par_perlexpr_repl {= a
par_perlexpr_repl {= a =} par_perlexpr_repl {= a =}
par_perlexpr_repl ### bug #45842: Do not evaluate {= =} twice par_perlexpr_repl ### bug #45842: Do not evaluate {= =} twice
par_perlexpr_repl 5 par_perlexpr_repl 1
par_perlexpr_repl 6 par_perlexpr_repl 2
par_perlexpr_repl 7 par_perlexpr_repl 3
par_perlexpr_repl 8 par_perlexpr_repl 4
par_perlexpr_repl 5 par_perlexpr_repl 1
par_perlexpr_repl 6 par_perlexpr_repl 2
par_perlexpr_repl 7 par_perlexpr_repl 3
par_perlexpr_repl 8 par_perlexpr_repl 4
par_perlexpr_repl 1 2
par_perlexpr_repl 3 4
par_perlexpr_repl 5 6
par_perlexpr_repl 7 8
par_perlexpr_repl 9 10
par_perlexpr_repl 11 12
par_perlexpr_repl 13 14
par_perlexpr_repl 15 16
par_perlexpr_repl 17 18 par_perlexpr_repl 17 18
par_perlexpr_repl 19 20 par_perlexpr_repl 19 20
par_perlexpr_repl 21 22 par_perlexpr_repl 21 22
par_perlexpr_repl 23 24 par_perlexpr_repl 23 24
par_perlexpr_repl 25 26 par_perlexpr_repl 1
par_perlexpr_repl 27 28 par_perlexpr_repl 2
par_perlexpr_repl 29 30 par_perlexpr_repl 3
par_perlexpr_repl 31 32 par_perlexpr_repl 4
par_perlexpr_repl 33 34 par_perlexpr_repl 5
par_perlexpr_repl 35 36 par_perlexpr_repl 6
par_perlexpr_repl 37 38 par_perlexpr_repl 7
par_perlexpr_repl 39 40 par_perlexpr_repl 8
par_perlexpr_repl 9 par_perlexpr_repl 9
par_perlexpr_repl 10 par_perlexpr_repl 10
par_perlexpr_repl 11 par_perlexpr_repl 11
par_perlexpr_repl 12 par_perlexpr_repl 12
par_perlexpr_repl 13
par_perlexpr_repl 14
par_perlexpr_repl 15
par_perlexpr_repl 16
par_perlexpr_repl 17
par_perlexpr_repl 18
par_perlexpr_repl 19
par_perlexpr_repl 20
par_perlexpr_repl ### bug #45939: {2} in {= =} fails par_perlexpr_repl ### bug #45939: {2} in {= =} fails
par_perlexpr_repl OK par_perlexpr_repl OK
par_perlexpr_repl OK-OK par_perlexpr_repl OK-OK

View file

@ -119,7 +119,7 @@ par_eta 16
par_eta ### Test of --eta with no jobs par_eta ### Test of --eta with no jobs
par_eta par_eta
par_eta Computers / CPU cores / Max jobs to run par_eta Computers / CPU cores / Max jobs to run
par_eta 1:local / 8 / 1 par_eta 1:local / 8 / 8
par_eta par_eta ETA: 0s Left: 0 AVG: 0.00s 0 par_eta par_eta ETA: 0s Left: 0 AVG: 0.00s 0
par_exitval_signal ### Test --joblog with exitval and Test --joblog with signal -- timing dependent par_exitval_signal ### Test --joblog with exitval and Test --joblog with signal -- timing dependent
par_exitval_signal exitval=128+6 OK par_exitval_signal exitval=128+6 OK
@ -486,7 +486,7 @@ par_progress 16
par_progress ### Test of --progress with no jobs par_progress ### Test of --progress with no jobs
par_progress par_progress
par_progress Computers / CPU cores / Max jobs to run par_progress Computers / CPU cores / Max jobs to run
par_progress 1:local / 8 / 1 par_progress 1:local / 8 / 8
par_progress par_progress 0 par_progress par_progress 0
par_replacement_slashslash ### Test {//} par_replacement_slashslash ### Test {//}
par_replacement_slashslash . a par_replacement_slashslash . a
@ -674,10 +674,10 @@ par_totaljob_repl 10
par_totaljob_repl 10 par_totaljob_repl 10
par_totaljob_repl 10 par_totaljob_repl 10
par_totaljob_repl 10 par_totaljob_repl 10
par_totaljob_repl 10 par_totaljob_repl 1
par_totaljob_repl 10 par_totaljob_repl 2
par_totaljob_repl 10 par_totaljob_repl 3
par_totaljob_repl 10 par_totaljob_repl 4
par_totaljob_repl 10 par_totaljob_repl 10
par_totaljob_repl 10 par_totaljob_repl 10
par_totaljob_repl 10 par_totaljob_repl 10
@ -725,17 +725,17 @@ par_wrong_slot_rpl_resume 4 3
par_wrong_slot_rpl_resume 1 4 par_wrong_slot_rpl_resume 1 4
par_wrong_slot_rpl_resume 2 5 par_wrong_slot_rpl_resume 2 5
par_wrong_slot_rpl_resume 3 6 par_wrong_slot_rpl_resume 3 6
par_wrong_slot_rpl_resume 4 7 par_wrong_slot_rpl_resume 1 7
par_wrong_slot_rpl_resume 1 8 par_wrong_slot_rpl_resume 2 8
par_wrong_slot_rpl_resume 2 9 par_wrong_slot_rpl_resume 3 9
par_wrong_slot_rpl_resume 3 10 par_wrong_slot_rpl_resume 4 10
par_wrong_slot_rpl_resume 4 11 par_wrong_slot_rpl_resume 1 11
par_wrong_slot_rpl_resume 1 12 par_wrong_slot_rpl_resume 2 12
par_wrong_slot_rpl_resume 2 13 par_wrong_slot_rpl_resume 3 13
par_wrong_slot_rpl_resume 3 14 par_wrong_slot_rpl_resume 4 14
par_wrong_slot_rpl_resume 4 15 par_wrong_slot_rpl_resume 1 15
par_wrong_slot_rpl_resume 1 16 par_wrong_slot_rpl_resume 2 16
par_wrong_slot_rpl_resume 2 17 par_wrong_slot_rpl_resume 3 17
par_wrong_slot_rpl_resume 3 18 par_wrong_slot_rpl_resume 4 18
par_wrong_slot_rpl_resume 4 19 par_wrong_slot_rpl_resume 1 19
par_wrong_slot_rpl_resume 1 20 par_wrong_slot_rpl_resume 2 20

View file

@ -15,10 +15,10 @@ par_PARALLEL_RSYNC_OPTS ### test rsync opts
par_PARALLEL_RSYNC_OPTS rsync --protocol 30 -rlDzRRRRrsync --protocol 30 --rsync-path='cd ././.; rsync' -rlDzRRRRrsync --protocol 30 -zzrrllddRRRRrsync --protocol 30 --rsync-path='cd ././.; rsync' -zzrrllddRRRR par_PARALLEL_RSYNC_OPTS rsync --protocol 30 -rlDzRRRRrsync --protocol 30 --rsync-path='cd ././.; rsync' -rlDzRRRRrsync --protocol 30 -zzrrllddRRRRrsync --protocol 30 --rsync-path='cd ././.; rsync' -zzrrllddRRRR
par_continuous_output Test delayed output with '' par_continuous_output Test delayed output with ''
par_continuous_output -u is optimal but hard to reach, due to non-mixing par_continuous_output -u is optimal but hard to reach, due to non-mixing
par_continuous_output 6 par_continuous_output 1
par_continuous_output 0 par_continuous_output 1
par_continuous_output 0 par_continuous_output 1
par_continuous_output 0 par_continuous_output 1
par_continuous_output 1 par_continuous_output 1
par_continuous_output 1 par_continuous_output 1
par_continuous_output 1 par_continuous_output 1
@ -27,9 +27,6 @@ par_continuous_output 1
par_continuous_output 0 par_continuous_output 0
par_continuous_output Test delayed output with '-u' par_continuous_output Test delayed output with '-u'
par_continuous_output -u is optimal but hard to reach, due to non-mixing par_continuous_output -u is optimal but hard to reach, due to non-mixing
par_continuous_output 5
par_continuous_output 0
par_continuous_output 0
par_continuous_output 0 par_continuous_output 0
par_continuous_output 1 par_continuous_output 1
par_continuous_output 1 par_continuous_output 1
@ -37,6 +34,9 @@ par_continuous_output 1
par_continuous_output 1 par_continuous_output 1
par_continuous_output 1 par_continuous_output 1
par_continuous_output 1 par_continuous_output 1
par_continuous_output 1
par_continuous_output 1
par_continuous_output 1
par_hostgroup ### --hostgroup force ncpu par_hostgroup ### --hostgroup force ncpu
par_hostgroup parallel par_hostgroup parallel
par_hostgroup parallel par_hostgroup parallel
@ -114,6 +114,18 @@ par_kill_hup parallel: Waiting for these 2 jobs to finish. Send SIGTERM to stop
par_kill_hup parallel: bash -c 'sleep 3 & pid=$!; wait $pid' par_kill_hup parallel: bash -c 'sleep 3 & pid=$!; wait $pid'
par_kill_hup parallel: bash -c 'sleep 3 & pid=$!; wait $pid' par_kill_hup parallel: bash -c 'sleep 3 & pid=$!; wait $pid'
par_kill_hup bash---pstree par_kill_hup bash---pstree
par_more_than_9_relative_sshlogin ### Check more than 9(relative) simultaneous sshlogins
par_more_than_9_relative_sshlogin 1
par_more_than_9_relative_sshlogin 2
par_more_than_9_relative_sshlogin 3
par_more_than_9_relative_sshlogin 4
par_more_than_9_relative_sshlogin 5
par_more_than_9_relative_sshlogin 6
par_more_than_9_relative_sshlogin 7
par_more_than_9_relative_sshlogin 8
par_more_than_9_relative_sshlogin 9
par_more_than_9_relative_sshlogin 10
par_more_than_9_relative_sshlogin 11
par_resume_failed_k ### bug #38299: --resume-failed -k par_resume_failed_k ### bug #38299: --resume-failed -k
par_resume_failed_k job1 val 0 par_resume_failed_k job1 val 0
par_resume_failed_k job2 val 1 par_resume_failed_k job2 val 1

View file

@ -11,7 +11,7 @@ par_nonall_should_not_block ### bug #47608: parallel --nonall -S lo 'echo ::: '
par_nonall_should_not_block ::: par_nonall_should_not_block :::
par_progress_text_max_jobs_to_run ### bug #49404: "Max jobs to run" does not equal the number of jobs specified when using GNU Parallel on remote server? par_progress_text_max_jobs_to_run ### bug #49404: "Max jobs to run" does not equal the number of jobs specified when using GNU Parallel on remote server?
par_progress_text_max_jobs_to_run should give 10 running jobs par_progress_text_max_jobs_to_run should give 10 running jobs
par_progress_text_max_jobs_to_run 1:lo / 16 / 10 par_progress_text_max_jobs_to_run 1:lo / 16 / 16
par_quoting_for_onall ### bug #35427: quoting of {2} broken for --onall par_quoting_for_onall ### bug #35427: quoting of {2} broken for --onall
par_quoting_for_onall /bin/ls par_quoting_for_onall /bin/ls
par_remote_function_nice ### functions and --nice par_remote_function_nice ### functions and --nice

View file

@ -998,7 +998,7 @@ echo '### -s6 echo < files.xi'
stdout xargs -s6 echo < files.xi stdout xargs -s6 echo < files.xi
xargs: argument line too long xargs: argument line too long
stdout parallel -k -X -s6 echo < files.xi stdout parallel -k -X -s6 echo < files.xi
parallel: Error: Command line too long (27 >= 6) at input -7: /src/gnu/autoconf-1.11 parallel: Error: Command line too long (27 >= 6) at input 0: /src/gnu/autoconf-1.11
echo '### -iARG -s86 echo ARG is xARGx < files.xi' echo '### -iARG -s86 echo ARG is xARGx < files.xi'
### -iARG -s86 echo ARG is xARGx < files.xi ### -iARG -s86 echo ARG is xARGx < files.xi
stdout xargs -iARG -s86 echo ARG is xARGx < files.xi stdout xargs -iARG -s86 echo ARG is xARGx < files.xi

View file

@ -171,6 +171,11 @@ ls | parallel -j500 'sleep 1; find {} -type f | perl -ne "END{print $..\" "{=$_=
4 ab 4 ab
4 b 4 b
6 a 6 a
parallel: Warning: Only enough file handles to run 252 jobs in parallel.
parallel: Warning: Try running 'parallel -j0 -N 252 --pipe parallel -j0'
parallel: Warning: or increasing 'ulimit -n' (try: ulimit -n `ulimit -Hn`)
parallel: Warning: or increasing 'nofile' in /etc/security/limits.conf
parallel: Warning: or increasing /proc/sys/fs/file-max
ls | parallel --group -j500 'sleep 1; find {} -type f | perl -ne "END{print $..\" "{=$_=pQ($_)=}"\n\"}"' | LC_ALL=C sort ls | parallel --group -j500 'sleep 1; find {} -type f | perl -ne "END{print $..\" "{=$_=pQ($_)=}"\n\"}"' | LC_ALL=C sort
1 1-col.txt 1 1-col.txt
1 1-col.txt.diff 1 1-col.txt.diff
@ -280,6 +285,11 @@ ls | parallel --group -j500 'sleep 1; find {} -type f | perl -ne "END{print $..\
4 ab 4 ab
4 b 4 b
6 a 6 a
parallel: Warning: Only enough file handles to run 252 jobs in parallel.
parallel: Warning: Try running 'parallel -j0 -N 252 --pipe parallel -j0'
parallel: Warning: or increasing 'ulimit -n' (try: ulimit -n `ulimit -Hn`)
parallel: Warning: or increasing 'nofile' in /etc/security/limits.conf
parallel: Warning: or increasing /proc/sys/fs/file-max
find . -type f | parallel --group "perl -ne '/^\\S+\\s+\\S+$/ and print \$ARGV,\"\\n\"'" | LC_ALL=C sort find . -type f | parallel --group "perl -ne '/^\\S+\\s+\\S+$/ and print \$ARGV,\"\\n\"'" | LC_ALL=C sort
./ ab /c' d/ ef"g.diff ./ ab /c' d/ ef"g.diff
./ ab /c' d/ ef"g.diff ./ ab /c' d/ ef"g.diff

View file

@ -499,6 +499,11 @@ echo C
sleep 1 sleep 1
echo 0 > my_jobs echo 0 > my_jobs
wait wait
parallel: Warning: Only enough file handles to run 248 jobs in parallel.
parallel: Warning: Try running 'parallel -j0 -N 248 --pipe parallel -j0'
parallel: Warning: or increasing 'ulimit -n' (try: ulimit -n `ulimit -Hn`)
parallel: Warning: or increasing 'nofile' in /etc/security/limits.conf
parallel: Warning: or increasing /proc/sys/fs/file-max
9 9
parallel --use-cpus-instead-of-cores -N0 sleep 1 :::: num8 parallel --use-cpus-instead-of-cores -N0 sleep 1 :::: num8
parallel --shuf echo ::: 1 2 3 ::: a b c ::: A B C parallel --shuf echo ::: 1 2 3 ::: a b c ::: A B C

View file

@ -110,18 +110,6 @@ centos8.localdomain
9 9
10 10
11 11
### Check more than 9(relative) simultaneous sshlogins
1
2
3
4
5
6
7
8
9
10
11
### Check -S syntax ### Check -S syntax
1 1
2 2