tange/parallel
1
0
Fork 0
mirror of https://git.savannah.gnu.org/git/parallel.git synced 2024-11-26 07:57:58 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

niceload: --recheck was not respected. More test suite cases.

Browse source 
alpha/beta testing in man page updated.
This commit is contained in:
Ole Tange 2011-07-22 10:47:14 +02:00
parent e93df253fc
commit 8d13ddebbe
7 changed files with 84 additions and 71 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
src/niceload
View file

@ -379,7 +379,7 @@ sub new {
'startnoswap' => $startnoswap, 'startnoswap' => $startnoswap,
'factor' => $::opt_factor || 1, 'factor' => $::opt_factor || 1,
'recheck' => $::opt_recheck || 1, 'recheck' => $::opt_recheck || 1,
'runtime' => 1, 'runtime' => $::opt_recheck || 1,
'over_run_limit' => 1, 'over_run_limit' => 1,
'over_start_limit' => 1, 'over_start_limit' => 1,
'verbose' => $::opt_verbose, 'verbose' => $::opt_verbose,
@ -485,6 +485,7 @@ sub sleep_for_recheck {
print "Sleeping $self->{'recheck'}s\n"; print "Sleeping $self->{'recheck'}s\n";
} }
::debug("recheck in $self->{'recheck'}s\n"); ::debug("recheck in $self->{'recheck'}s\n");
::usleep(1); # For some reason this gets interrupted
::usleep(1000*$self->{'recheck'}); ::usleep(1000*$self->{'recheck'});
} }

76
src/niceload.pod
View file

@ -28,25 +28,25 @@ run 1 second, suspend (3.00-1.00) seconds, run 1 second, suspend
=over 9 =over 9
=item B<-f> I<FACTOR> =item B<-f> I<FACTOR> (alpha testing)
=item B<--factor> I<FACTOR> =item B<--factor> I<FACTOR> (alpha testing)
Suspend time factor. Dynamically set B<-s> as amount over limit * Suspend time factor. Dynamically set B<-s> as amount over limit *
factor. Default is 1. factor. Default is 1.
=item B<-H> =item B<-H> (alpha testing)
=item B<--hard> =item B<--hard> (alpha testing)
Hard limit. B<--hard> will suspend the process until the system is Hard limit. B<--hard> will suspend the process until the system is
under the limits. The default is B<--soft>. under the limits. The default is B<--soft>.
=item B<--io> I<iolimit> =item B<--io> I<iolimit> (alpha testing)
=item B<-I> I<iolimit> =item B<-I> I<iolimit> (alpha testing)
Limit for I/O. The amount of disk I/O will be computed as a value 0 - Limit for I/O. The amount of disk I/O will be computed as a value 0 -
10, where 0 is no I/O and 10 is at least one disk is 100% satuated. 10, where 0 is no I/O and 10 is at least one disk is 100% satuated.
@ -54,18 +54,18 @@ Limit for I/O. The amount of disk I/O will be computed as a value 0 -
B<--io> will set both B<--start-io> and B<run-io>. B<--io> will set both B<--start-io> and B<run-io>.
=item B<--load> I<loadlimit> =item B<--load> I<loadlimit> (alpha testing)
=item B<-L> I<loadlimit> =item B<-L> I<loadlimit> (alpha testing)
Limit for load average. Limit for load average.
B<--load> will set both B<--start-load> and B<run-load>. B<--load> will set both B<--start-load> and B<run-load>.
=item B<--mem> I<memlimit> =item B<--mem> I<memlimit> (alpha testing)
=item B<-M> I<memlimit> =item B<-M> I<memlimit> (alpha testing)
Limit for free memory. This is the amount of bytes available as free Limit for free memory. This is the amount of bytes available as free
+ cache. This limit is treated opposite other limits: If the system + cache. This limit is treated opposite other limits: If the system
@ -79,9 +79,9 @@ respectively.
B<--mem> will set both B<--start-mem> and B<run-mem>. B<--mem> will set both B<--start-mem> and B<run-mem>.
=item B<--noswap> =item B<--noswap> (alpha testing)
=item B<-N> =item B<-N> (alpha testing)
No swapping. If the system is swapping both in and out it is a good No swapping. If the system is swapping both in and out it is a good
indication that the system is memory stressed. indication that the system is memory stressed.
@ -91,88 +91,88 @@ B<--noswap> is over limit if the system is swapping both in and out.
B<--noswap> will set both B<--start-noswap> and B<run-noswap>. B<--noswap> will set both B<--start-noswap> and B<run-noswap>.
=item B<-n> I<niceness> =item B<-n> I<niceness> (alpha testing)
=item B<--nice> I<niceness> =item B<--nice> I<niceness> (alpha testing)
Sets niceness. See B<nice>(1). Sets niceness. See B<nice>(1).
=item B<-p> I<PID> =item B<-p> I<PID> (alpha testing)
=item B<--pid> I<PID> =item B<--pid> I<PID> (alpha testing)
Process ID of process to suspend. Process ID of process to suspend.
=item B<--quote> =item B<--quote> (alpha testing)
=item B<-q> =item B<-q> (alpha testing)
Quote the command line. Useful if the command contains chars like *, Quote the command line. Useful if the command contains chars like *,
$, >, and " that should not be interpreted by the shell. $, >, and " that should not be interpreted by the shell.
=item B<--run-io> I<iolimit> =item B<--run-io> I<iolimit> (alpha testing)
=item B<--ri> I<iolimit> =item B<--ri> I<iolimit> (alpha testing)
=item B<--run-load> I<loadlimit> =item B<--run-load> I<loadlimit> (alpha testing)
=item B<--rl> I<loadlimit> =item B<--rl> I<loadlimit> (alpha testing)
=item B<--run-mem> I<memlimit> =item B<--run-mem> I<memlimit> (alpha testing)
=item B<--rm> I<memlimit> =item B<--rm> I<memlimit> (alpha testing)
Run limit. The running program will be slowed down if the system is Run limit. The running program will be slowed down if the system is
above the limit. See: B<--io>, B<--load>, B<--mem>, B<--noswap>. above the limit. See: B<--io>, B<--load>, B<--mem>, B<--noswap>.
=item B<--start-io> I<iolimit> =item B<--start-io> I<iolimit> (alpha testing)
=item B<--si> I<iolimit> =item B<--si> I<iolimit> (alpha testing)
=item B<--start-load> I<loadlimit> =item B<--start-load> I<loadlimit> (alpha testing)
=item B<--sl> I<loadlimit> =item B<--sl> I<loadlimit> (alpha testing)
=item B<--start-mem> I<memlimit> =item B<--start-mem> I<memlimit> (alpha testing)
=item B<--sm> I<memlimit> =item B<--sm> I<memlimit> (alpha testing)
Start limit. The program will not start until the system is below the Start limit. The program will not start until the system is below the
limit. See: B<--io>, B<--load>, B<--mem>, B<--noswap>. limit. See: B<--io>, B<--load>, B<--mem>, B<--noswap>.
=item B<--soft> =item B<--soft> (alpha testing)
=item B<-S> =item B<-S> (alpha testing)
Soft limit. B<niceload> will suspend a process for a while and then Soft limit. B<niceload> will suspend a process for a while and then
let it run for a second thus only slowing down a process while the let it run for a second thus only slowing down a process while the
system is over one of the given limits. This is the default. system is over one of the given limits. This is the default.
=item B<--suspend> I<SEC> =item B<--suspend> I<SEC> (alpha testing)
=item B<-s> I<SEC> =item B<-s> I<SEC> (alpha testing)
Suspend time. Suspend the command this many seconds when the max load Suspend time. Suspend the command this many seconds when the max load
average is reached. average is reached.
=item B<--recheck> I<SEC> =item B<--recheck> I<SEC> (alpha testing)
=item B<-t> I<SEC> =item B<-t> I<SEC> (alpha testing)
Recheck load time. Sleep SEC seconds before checking load Recheck load time. Sleep SEC seconds before checking load
again. Default is 1 second. again. Default is 1 second.
=item B<--verbose> =item B<--verbose> (alpha testing)
=item B<-v> =item B<-v> (alpha testing)
Verbose. Print some extra output on what is happening. Use B<-v> until Verbose. Print some extra output on what is happening. Use B<-v> until
you know what your are doing. you know what your are doing.

50
src/parallel.pod
View file

@ -71,7 +71,7 @@ alias or a function will not work (see why
http://www.perlmonks.org/index.pl?node_id=484296). http://www.perlmonks.org/index.pl?node_id=484296).
=item B<{}> (still alpha testing) =item B<{}> (alpha testing)
Input line. This replacement string will be replaced by a full line Input line. This replacement string will be replaced by a full line
read from the input source. The input source is normally stdin read from the input source. The input source is normally stdin
@ -84,7 +84,7 @@ If the command line contains no replacement strings then B<{}> will be
appended to the command line. appended to the command line.
=item B<{.}> (still alpha testing) =item B<{.}> (alpha testing)
Input line without extension. This replacement string will be replaced Input line without extension. This replacement string will be replaced
by the input with the extension removed. If the input line contains by the input with the extension removed. If the input line contains
@ -100,7 +100,7 @@ The replacement string B<{.}> can be changed with B<-U>.
To understand replacement strings see B<{}>. To understand replacement strings see B<{}>.
=item B<{/}> (still alpha testing) =item B<{/}> (alpha testing)
Basename of input line. This replacement string will be replaced by Basename of input line. This replacement string will be replaced by
the input with the directory part removed. the input with the directory part removed.
@ -111,7 +111,7 @@ B<--basenamereplace>.
To understand replacement strings see B<{}>. To understand replacement strings see B<{}>.
=item B<{//}> (still alpha testing) =item B<{//}> (alpha testing)
Dirname of input line. This replacement string will be replaced by the Dirname of input line. This replacement string will be replaced by the
dir of the input line. See B<dirname>(1). dir of the input line. See B<dirname>(1).
@ -122,7 +122,7 @@ B<--dirnamereplace>.
To understand replacement strings see B<{}>. To understand replacement strings see B<{}>.
=item B<{/.}> (still alpha testing) =item B<{/.}> (alpha testing)
Basename of input line without extension. This replacement string will Basename of input line without extension. This replacement string will
be replaced by the input with the directory and extension part be replaced by the input with the directory and extension part
@ -134,7 +134,7 @@ B<--basenameextensionreplace>.
To understand replacement strings see B<{}>. To understand replacement strings see B<{}>.
=item B<{#}> (still alpha testing) =item B<{#}> (alpha testing)
Sequence number of the job to run. This replacement string will be Sequence number of the job to run. This replacement string will be
replaced by the sequence number of the job being run. It contains the replaced by the sequence number of the job being run. It contains the
@ -145,7 +145,7 @@ The replacement string B<{#}> can be changed with B<--seqreplace>.
To understand replacement strings see B<{}>. To understand replacement strings see B<{}>.
=item B<{>I<n>B<}> (still alpha testing) =item B<{>I<n>B<}> (alpha testing)
Argument from input source I<n> or the I<n>'th argument. This Argument from input source I<n> or the I<n>'th argument. This
positional replacement string will be replaced by the input from input positional replacement string will be replaced by the input from input
@ -155,7 +155,7 @@ argument (when used with B<-N>).
To understand replacement strings see B<{}>. To understand replacement strings see B<{}>.
=item B<{>I<n>.B<}> (still alpha testing) =item B<{>I<n>.B<}> (alpha testing)
Argument from input source I<n> or the I<n>'th argument without Argument from input source I<n> or the I<n>'th argument without
extension. It is a combination of B<{>I<n>B<}> and B<{.}>. extension. It is a combination of B<{>I<n>B<}> and B<{.}>.
@ -168,7 +168,7 @@ extension removed.
To understand positional replacement strings see B<{>I<n>B<}>. To understand positional replacement strings see B<{>I<n>B<}>.
=item B<{>I<n>/B<}> (still alpha testing) =item B<{>I<n>/B<}> (alpha testing)
Basename of argument from input source I<n> or the I<n>'th argument. Basename of argument from input source I<n> or the I<n>'th argument.
It is a combination of B<{>I<n>B<}> and B<{/}>. It is a combination of B<{>I<n>B<}> and B<{/}>.
@ -181,7 +181,7 @@ directory (if any) removed.
To understand positional replacement strings see B<{>I<n>B<}>. To understand positional replacement strings see B<{>I<n>B<}>.
=item B<{>I<n>/.B<}> (still alpha testing) =item B<{>I<n>/.B<}> (alpha testing)
Basename of argument from input source I<n> or the I<n>'th argument Basename of argument from input source I<n> or the I<n>'th argument
without extension. It is a combination of B<{>I<n>B<}>, B<{/}>, and without extension. It is a combination of B<{>I<n>B<}>, B<{/}>, and
@ -195,7 +195,7 @@ directory (if any) and extension removed.
To understand positional replacement strings see B<{>I<n>B<}>. To understand positional replacement strings see B<{>I<n>B<}>.
=item B<:::> I<arguments> (alpha testing) =item B<:::> I<arguments> (beta testing)
Use arguments from the command line as input source instead of stdin Use arguments from the command line as input source instead of stdin
(standard input). Unlike other options for GNU B<parallel> B<:::> is (standard input). Unlike other options for GNU B<parallel> B<:::> is
@ -231,7 +231,7 @@ B<:::> and B<::::> can be mixed. So these are equivalent:
seq 4 5 | parallel echo {1} {2} {3} :::: <(seq 6 7) - ::: 1 2 3 seq 4 5 | parallel echo {1} {2} {3} :::: <(seq 6 7) - ::: 1 2 3
=item B<::::> I<argfiles> (alpha testing) =item B<::::> I<argfiles> (beta testing)
Another way to write B<-a> I<argfile1> B<-a> I<argfile2> ... Another way to write B<-a> I<argfile1> B<-a> I<argfile2> ...
@ -390,9 +390,9 @@ as \n, or an octal or hexadecimal escape code. Octal and
hexadecimal escape codes are understood as for the printf command. hexadecimal escape codes are understood as for the printf command.
Multibyte characters are not supported. Multibyte characters are not supported.
=item B<--dirnamereplace> I<replace-str> (alpha testing) =item B<--dirnamereplace> I<replace-str> (beta testing)
=item B<--dnr> I<replace-str> (alpha testing) =item B<--dnr> I<replace-str> (beta testing)
Use the replacement string I<replace-str> instead of B<{//}> for Use the replacement string I<replace-str> instead of B<{//}> for
dirname of input line. dirname of input line.
@ -625,7 +625,7 @@ B<-l 0> is an alias for B<-l 1>.
Implies B<-X> unless B<-m> is set. Implies B<-X> unless B<-m> is set.
=item B<--load> I<max-load> (alpha testing) =item B<--load> I<max-load> (beta testing)
Do not start new jobs on a given computer unless the load is less than Do not start new jobs on a given computer unless the load is less than
I<max-load>. I<max-load> uses the same syntax as B<--jobs>, so I<100%> I<max-load>. I<max-load> uses the same syntax as B<--jobs>, so I<100%>
@ -675,7 +675,7 @@ See also B<-X> for context replace. If in doubt use B<-X> as that will
most likely do what is needed. most likely do what is needed.
=item B<--minversion> I<version> (alpha testing) =item B<--minversion> I<version> (beta testing)
Print the version GNU B<parallel> and exit. If the current version of Print the version GNU B<parallel> and exit. If the current version of
GNU B<parallel> is less than I<version> the exit code is GNU B<parallel> is less than I<version> the exit code is
@ -685,7 +685,7 @@ This is useful for scripts that depend on features only available from
a certain version of GNU B<parallel>. a certain version of GNU B<parallel>.
=item B<--nonall> (alpha testing) =item B<--nonall> (beta testing)
B<--onall> with no arguments. Run the command on all computers given B<--onall> with no arguments. Run the command on all computers given
with B<--sshlogin> but take no arguments. GNU B<parallel> will log with B<--sshlogin> but take no arguments. GNU B<parallel> will log
@ -696,7 +696,7 @@ This is useful for running the same command (e.g. uptime) on a list of
servers. servers.
=item B<--onall> (alpha testing) =item B<--onall> (beta testing)
Run all the jobs on all computers given with B<--sshlogin>. GNU Run all the jobs on all computers given with B<--sshlogin>. GNU
B<parallel> will log into B<--jobs> number of computers in parallel B<parallel> will log into B<--jobs> number of computers in parallel
@ -718,9 +718,9 @@ Instead of printing the output to stdout (standard output) the output
of each job is saved in a file and the filename is then printed. of each job is saved in a file and the filename is then printed.
=item B<--pipe> (alpha testing) =item B<--pipe> (beta testing)
=item B<--spreadstdin> (alpha testing) =item B<--spreadstdin> (beta testing)
Spread input to jobs on stdin (standard input). Read a block of data Spread input to jobs on stdin (standard input). Read a block of data
from stdin (standard input) and give one block of data as input to one from stdin (standard input) and give one block of data as input to one
@ -1028,9 +1028,9 @@ Do not use the first line of input (used by GNU B<parallel> itself
when called with B<--shebang>). when called with B<--shebang>).
=item B<-S> I<[ncpu/]sshlogin[,[ncpu/]sshlogin[,...]]> (alpha testing) =item B<-S> I<[ncpu/]sshlogin[,[ncpu/]sshlogin[,...]]> (beta testing)
=item B<--sshlogin> I<[ncpu/]sshlogin[,[ncpu/]sshlogin[,...]]> (alpha testing) =item B<--sshlogin> I<[ncpu/]sshlogin[,[ncpu/]sshlogin[,...]]> (beta testing)
Distribute jobs to remote computers. The jobs will be run on a list of Distribute jobs to remote computers. The jobs will be run on a list of
remote computers. GNU B<parallel> will determine the number of CPU remote computers. GNU B<parallel> will determine the number of CPU
@ -1066,7 +1066,7 @@ B<--sshlogin> is often used with B<--transfer>, B<--return>,
B<--cleanup>, and B<--trc>. B<--cleanup>, and B<--trc>.
=item B<--sshloginfile> I<filename> (alpha testing) =item B<--sshloginfile> I<filename> (beta testing)
File with sshlogins. The file consists of sshlogins on separate File with sshlogins. The file consists of sshlogins on separate
lines. Empty lines and lines starting with '#' are ignored. Example: lines. Empty lines and lines starting with '#' are ignored. Example:
@ -1095,7 +1095,7 @@ The sshloginfile '..' is special, it read sshlogins from
The sshloginfile '-' is special, too, it read sshlogins from stdin The sshloginfile '-' is special, too, it read sshlogins from stdin
(standard input). (standard input).
=item B<--noswap> (alpha testing) =item B<--noswap> (beta testing)
Do not start new jobs on a given computer if there is both swap-in and Do not start new jobs on a given computer if there is both swap-in and
swap-out activity. swap-out activity.
@ -1299,7 +1299,7 @@ See also B<-m>.
Exit if the size (see the B<-s> option) is exceeded. Exit if the size (see the B<-s> option) is exceeded.
=item B<--xapply> (alpha testing) =item B<--xapply> (beta testing)
Read multiple files like B<xapply>. If multiple B<-a> are given, one Read multiple files like B<xapply>. If multiple B<-a> are given, one
line will be read from each of the files. The arguments can be line will be read from each of the files. The arguments can be

2
testsuite/Makefile
View file

@ -36,7 +36,7 @@ portable:
time bash Portable.sh time bash Portable.sh
timings: tests-to-run/* timings: tests-to-run/*
ls tests-to-run/*3.sh | xargs -n1 echo /usr/bin/time -f %e bash >/tmp/timing.script ls tests-to-run/*.sh | xargs -n1 echo /usr/bin/time -f %e bash >/tmp/timing.script
stdout bash -x /tmp/timing.script >/tmp/timing.out stdout bash -x /tmp/timing.script >/tmp/timing.out
echo usr.bin.time_END >>/tmp/timing.out echo usr.bin.time_END >>/tmp/timing.out
perl -ne '/usr.bin.time/ and do { print $$last.$$h; $$h=$$_ }; chomp; $$last = $$_' /tmp/timing.out | sort -n >timings perl -ne '/usr.bin.time/ and do { print $$last.$$h; $$h=$$_ }; chomp; $$last = $$_' /tmp/timing.out | sort -n >timings

13
testsuite/tests-to-run/niceload01.sh
View file

@ -12,12 +12,12 @@ while [ $MEMAVAIL -gt 1000000 ] ; do
sleep 2; sleep 2;
MEMAVAIL=$(free | perl -ane '/buffers.cache:/ and print $F[3]') MEMAVAIL=$(free | perl -ane '/buffers.cache:/ and print $F[3]')
done done
# free
echo '### --rm and --runmem' echo '### --rm and --runmem'
niceload -H --rm 1g free -g | perl -ane '/buffers.cache:/ and print $F[3],"\n"' | grep '[1-9]' >/dev/null && echo OK & niceload -H --rm 1g free -g | perl -ane '/buffers.cache:/ and print $F[3],"\n"' | grep '[1-9]' >/dev/null && echo OK &
niceload -H --runmem 1.2g free -g | perl -ane '/buffers.cache:/ and print $F[3],"\n"' | grep '[1-9]' >/dev/null && echo OK & niceload -H --runmem 1.2g free -g | perl -ane '/buffers.cache:/ and print $F[3],"\n"' | grep '[1-9]' >/dev/null && echo OK &
wait wait
# free
# Force swapping # Force swapping
MEMAVAIL=$(free | perl -ane '/buffers.cache:/ and print $F[3]') MEMAVAIL=$(free | perl -ane '/buffers.cache:/ and print $F[3]')
@ -33,6 +33,13 @@ niceload -H -N vmstat 1 2 | tail -n1 | awk '{print $7*$8}' &
niceload -H --noswap vmstat 1 2 | tail -n1 | awk '{print $7*$8}' & niceload -H --noswap vmstat 1 2 | tail -n1 | awk '{print $7*$8}' &
wait wait
# force load > 10
while uptime | grep -v age:.[1-9][0-9].[0-9][0-9] >/dev/null ; do (timeout 5 nice burnP6 2>/dev/null &) done
echo '### --soft -f and test if child is actually suspended and thus takes longer'
niceload --soft -t 0.2 -f 0.5 'seq 1000000 | wc;echo This should finish last' &
(sleep 1; seq 1000000 | wc;echo This should finish first) &
wait
# force load > 10 # force load > 10
while uptime | grep -v age:.[1-9][0-9].[0-9][0-9] >/dev/null ; do (timeout 5 nice burnP6 2>/dev/null &) done while uptime | grep -v age:.[1-9][0-9].[0-9][0-9] >/dev/null ; do (timeout 5 nice burnP6 2>/dev/null &) done
@ -43,7 +50,7 @@ niceload --hard -l 9 uptime | grep '[1-9][0-9]\.[0-9][0-9],' || echo OK
echo '### -f and --factor' echo '### -f and --factor'
niceload -H -f 0.1 -l6 echo f 0.1 first & niceload -H -f 0.1 -l6 echo f 0.1 first &
niceload -H --factor 10 -l6 echo factor last & niceload -H --factor 10 -l6 echo factor 10 last &
wait wait

4
testsuite/tests-to-run/test48.sh
View file

@ -16,7 +16,7 @@ echo '### Test -N is not broken by distribution - two lines'
seq 19 | parallel -k -N 10 echo seq 19 | parallel -k -N 10 echo
echo '### Test -N context replace' echo '### Test -N context replace'
seq 19 | parallel -N 10 echo a{}b seq 19 | parallel -k -N 10 echo a{}b
echo '### Test -L context replace' echo '### Test -L context replace'
seq 19 | parallel -L 10 echo a{}b seq 19 | parallel -k -L 10 echo a{}b

7
testsuite/wanted-results/niceload01
View file

@ -6,9 +6,14 @@ OK
### -N and --noswap ### -N and --noswap
0 0
0 0
### --soft -f and test if child is actually suspended and thus takes longer
1000000 1000000 6888896
This should finish first
1000000 1000000 6888896
This should finish last
### -H and --hard ### -H and --hard
OK OK
OK OK
### -f and --factor ### -f and --factor
f 0.1 first f 0.1 first
factor last factor 10 last