tange/parallel
1
0
Fork 0
mirror of https://git.savannah.gnu.org/git/parallel.git synced 2024-11-25 23:47:53 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

parallel: Cache max command line length.

Browse source
This commit is contained in:
Ole Tange 2013-09-27 00:33:20 +02:00
parent a6eb7a78c6
commit 46eb082dc5
4 changed files with 38 additions and 48 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

21
doc/release_new_version
View file

@ -201,30 +201,13 @@ cc:Sandro Cazzaniga <kharec@mandriva.org>,
Ryoichiro Suzuki <ryoichiro.suzuki@gmail.com>, Ryoichiro Suzuki <ryoichiro.suzuki@gmail.com>,
Jesse Alama <jesse.alama@gmail.com> Jesse Alama <jesse.alama@gmail.com>
Subject: GNU Parallel 20130922 ('Manning') released Subject: GNU Parallel 20131022 ('Westgate') released
GNU Parallel 20130922 ('Manning') has been released. It is GNU Parallel 20131022 ('Westgate') has been released. It is
available for download at: http://ftp.gnu.org/gnu/parallel/ available for download at: http://ftp.gnu.org/gnu/parallel/
This release has no new features, and very few changes, making it a
good candidate for a stable release.
New in this release: New in this release:
* PDF-files of documentation is now included.
* Cloning at warp speed 100 VMs in 1 minute using one command line
http://www.beebotech.com.au/tag/command-line/
* Calcul Quebec's introduction to GNU Parallel
https://wiki.calculquebec.ca/w/GNU_parallel
* Brincando com o GNU parallel
http://blog.tadeucruz.com/2013/09/brincando-com-o-gnu-parallel/
* GNU Parallelで並列処理を行う
http://qiita.com/PENGUINANA_/items/af27a833e835fe17f09b
* Bug fixes and man page updates. * Bug fixes and man page updates.

27
src/parallel
View file

@ -5456,21 +5456,28 @@ sub max_length {
# Returns: # Returns:
# number of chars on the longest command line allowed # number of chars on the longest command line allowed
if(not $Limits::Command::line_max_len) { if(not $Limits::Command::line_max_len) {
# Disk cache of max command line length
my $len_cache = $ENV{'HOME'} . "/.parallel/tmp/linelen-" . ::hostname();
my $cached_limit;
if(-e $len_cache) {
open(my $fh, "<", $len_cache) || ::die_bug("Cannot read $len_cache");
$cached_limit = <$fh>;
close $fh;
} else {
$cached_limit = real_max_length();
Semaphore::mkdir_or_die($ENV{'HOME'} . "/.parallel/tmp");
open(my $fh, ">", $len_cache) || ::die_bug("Cannot write $len_cache");
print $fh $cached_limit;
close $fh;
}
$Limits::Command::line_max_len = $cached_limit;
if($opt::max_chars) { if($opt::max_chars) {
if(is_acceptable_command_line_length($opt::max_chars)) { if($opt::max_chars <= $cached_limit) {
$Limits::Command::line_max_len = $opt::max_chars;
} else {
# -s is too long: Find the correct
$Limits::Command::line_max_len = binary_find_max_length(0,$opt::max_chars);
}
if($opt::max_chars <= $Limits::Command::line_max_len) {
$Limits::Command::line_max_len = $opt::max_chars; $Limits::Command::line_max_len = $opt::max_chars;
} else { } else {
::warning("Value for -s option ", ::warning("Value for -s option ",
"should be < $Limits::Command::line_max_len.\n"); "should be < $cached_limit.\n");
} }
} else {
$Limits::Command::line_max_len = real_max_length();
} }
} }
return $Limits::Command::line_max_len; return $Limits::Command::line_max_len;

BIN
src/parallel.pdf
View file

Binary file not shown.

38
src/parallel.pod
View file

@ -430,7 +430,7 @@ occurs as a line of input, the rest of the input is ignored. If
neither B<-E> nor B<-e> is used, no end of file string is used. neither B<-E> nor B<-e> is used, no end of file string is used.
=item B<--delay> I<secs> (alpha testing) =item B<--delay> I<secs> (beta testing)
Delay starting next job I<secs> seconds. GNU B<parallel> will pause Delay starting next job I<secs> seconds. GNU B<parallel> will pause
I<secs> seconds after starting each job. I<secs> can be less than 1 I<secs> seconds after starting each job. I<secs> can be less than 1
@ -456,7 +456,7 @@ If I<eof-str> is omitted, there is no end of file string. If neither
B<-E> nor B<-e> is used, no end of file string is used. B<-E> nor B<-e> is used, no end of file string is used.
=item B<--env> I<var> (alpha testing) =item B<--env> I<var> (beta testing)
Copy environment variable I<var>. This will copy I<var> to the Copy environment variable I<var>. This will copy I<var> to the
environment that the command is run in. This is especially useful for environment that the command is run in. This is especially useful for
@ -489,7 +489,7 @@ See also B<--bg>, B<man sem>.
Implies B<--semaphore>. Implies B<--semaphore>.
=item B<--filter-hosts> (alpha testing) =item B<--filter-hosts> (beta testing)
Remove down hosts. For each remote host: check that login through ssh Remove down hosts. For each remote host: check that login through ssh
works. If not: do not use this host. works. If not: do not use this host.
@ -551,7 +551,7 @@ status will be the exit status from the failing job.
=back =back
=item B<--header> I<regexp> (alpha testing) =item B<--header> I<regexp> (beta testing)
Use regexp as header. For normal usage the matched header (typically Use regexp as header. For normal usage the matched header (typically
the first line: B<--header '.*\n'>) will be split using B<--colsep> the first line: B<--header '.*\n'>) will be split using B<--colsep>
@ -705,7 +705,7 @@ B<-l 0> is an alias for B<-l 1>.
Implies B<-X> unless B<-m>, B<--xargs>, or B<--pipe> is set. Implies B<-X> unless B<-m>, B<--xargs>, or B<--pipe> is set.
=item B<--line-buffer> (alpha testing) =item B<--line-buffer> (beta testing)
Buffer output on line basis. B<--group> will keep the output together Buffer output on line basis. B<--group> will keep the output together
for a whole job. B<--ungroup> allows output to mixup with half a line for a whole job. B<--ungroup> allows output to mixup with half a line
@ -774,7 +774,7 @@ This is useful for scripts that depend on features only available from
a certain version of GNU B<parallel>. a certain version of GNU B<parallel>.
=item B<--nonall> =item B<--nonall> (alpha testing)
B<--onall> with no arguments. Run the command on all computers given B<--onall> with no arguments. Run the command on all computers given
with B<--sshlogin> but take no arguments. GNU B<parallel> will log with B<--sshlogin> but take no arguments. GNU B<parallel> will log
@ -785,7 +785,7 @@ This is useful for running the same command (e.g. uptime) on a list of
servers. servers.
=item B<--onall> =item B<--onall> (alpha testing)
Run all the jobs on all computers given with B<--sshlogin>. GNU Run all the jobs on all computers given with B<--sshlogin>. GNU
B<parallel> will log into B<--jobs> number of computers in parallel B<parallel> will log into B<--jobs> number of computers in parallel
@ -807,9 +807,9 @@ Instead of printing the output to stdout (standard output) the output
of each job is saved in a file and the filename is then printed. of each job is saved in a file and the filename is then printed.
=item B<--pipe> (alpha testing) =item B<--pipe> (beta testing)
=item B<--spreadstdin> (alpha testing) =item B<--spreadstdin> (beta testing)
Spread input to jobs on stdin (standard input). Read a block of data Spread input to jobs on stdin (standard input). Read a block of data
from stdin (standard input) and give one block of data as input to one from stdin (standard input) and give one block of data as input to one
@ -967,7 +967,7 @@ If the stdin (standard input) only contains whitespace, do not run the command.
If used with B<--pipe> this is slow. If used with B<--pipe> this is slow.
=item B<--record-env> (alpha testing) =item B<--record-env> (beta testing)
Record current environment variables in ~/.parallel/ignored_vars. This Record current environment variables in ~/.parallel/ignored_vars. This
is useful before using B<--env _>. is useful before using B<--env _>.
@ -1015,9 +1015,9 @@ it to the command.
Only used with B<--pipe>. Only used with B<--pipe>.
=item B<--results> I<prefix> (alpha testing) =item B<--results> I<prefix> (beta testing)
=item B<--res> I<prefix> (alpha testing) =item B<--res> I<prefix> (beta testing)
Save the output into files. The files will be stored in a directory tree Save the output into files. The files will be stored in a directory tree
rooted at I<prefix>. Within this directory tree, each command will result rooted at I<prefix>. Within this directory tree, each command will result
@ -1126,9 +1126,9 @@ B<--return> is ignored when used with B<--sshlogin :> or when not used
with B<--sshlogin>. with B<--sshlogin>.
=item B<--round-robin> (alpha testing) =item B<--round-robin> (beta testing)
=item B<--round> (alpha testing) =item B<--round> (beta testing)
Normally B<--pipe> will give a single block to each instance of the Normally B<--pipe> will give a single block to each instance of the
command. With B<--round-robin> all blocks will at random be written to command. With B<--round-robin> all blocks will at random be written to
@ -1318,9 +1318,9 @@ B<--sshlogin> is often used with B<--transfer>, B<--return>,
B<--cleanup>, and B<--trc>. B<--cleanup>, and B<--trc>.
=item B<--sshloginfile> I<filename> (alpha testing) =item B<--sshloginfile> I<filename> (beta testing)
=item B<--slf> I<filename> (alpha testing) =item B<--slf> I<filename> (beta testing)
File with sshlogins. The file consists of sshlogins on separate File with sshlogins. The file consists of sshlogins on separate
lines. Empty lines and lines starting with '#' are ignored. Example: lines. Empty lines and lines starting with '#' are ignored. Example:
@ -1412,7 +1412,7 @@ different dir for the files. Setting B<--tmpdir> is equivalent to
setting $TMPDIR. setting $TMPDIR.
=item B<--timeout> I<val> (alpha testing) =item B<--timeout> I<val> (beta testing)
Time out for command. If the command runs for longer than I<val> Time out for command. If the command runs for longer than I<val>
seconds it will get killed with SIGTERM, followed by SIGTERM 200 ms seconds it will get killed with SIGTERM, followed by SIGTERM 200 ms
@ -1557,9 +1557,9 @@ Use B<-v> B<-v> to print the wrapping ssh command when running remotely.
Print the version GNU B<parallel> and exit. Print the version GNU B<parallel> and exit.
=item B<--workdir> I<mydir> (alpha testing) =item B<--workdir> I<mydir> (beta testing)
=item B<--wd> I<mydir> (alpha testing) =item B<--wd> I<mydir> (beta testing)
Files transferred using B<--transfer> and B<--return> will be relative Files transferred using B<--transfer> and B<--return> will be relative
to I<mydir> on remote computers, and the command will be executed in to I<mydir> on remote computers, and the command will be executed in