-L, -l, --max-lines, and -x pass unittest.

This commit is contained in:
Ole Tange 2010-06-13 01:24:25 +02:00
parent 47153f2bae
commit 847e972912
20 changed files with 885 additions and 346 deletions

View file

@ -1,11 +1,97 @@
Ved første ssh til hosten: =head1 YouTube video
if not -B kopieret:
kopier GNU Parallel is a tool with lots of uses in shell. Every time you use
tilføj til cleanup xargs or a for-loop GNU Parallel can probably do that faster, safer
and more readable.
If you have access to more computers through ssh, GNU Parallel makes
it easy to distribute jobs to these.
terminal2: ssh parallel@vh2.pi.dk
ssh parallel@vh2.pi.dk
PS1="\e[7mGNU Parallel:\[\033[01;34m\]\w\[\033[00m\]\e[27m$ "
gunzip logs/*gz
rm logs/*bz2
rm -rf zip/*[^p]
# GET GNU PARALLEL
wget ftp://ftp.gnu.org/gnu/parallel/parallel-20100601.tar.bz2
tar xvjf parallel-20100601.tar.bz2
cd parallel-20100601
./configure && make ##
su
make install
exit
cd
# YOUR FIRST PARALLEL JOBS
cd logs
du
/usr/bin/time gzip *
/usr/bin/time gunzip *
ls | time parallel gzip
ls | time parallel gunzip
# RECOMPRESS gz TO bz2
ls | time parallel gzip
ls *.gz | time parallel 'zcat {} | bzip2 > {.}.bz2'
## vis top local
# RECOMPRESS gz TO bz2 USING local(:) AND REMOTE server1
ls *.gz | time parallel -S server1,: --trc {.}.bz2 \
'zcat {} | bzip2 > {.}.bz2'
## vis top local
## vis top remote
# MAKING SMALL SCRIPTS
cd ../zip
ls -l
ls *.zip | parallel 'mkdir {.} && cd {.} && unzip ../{}' ###
ls -l
# GROUP OUTPUT
traceroute debian.org
traceroute debian.org & traceroute freenetproject.org ###
(echo debian.org; echo freenetproject.org) \
| parallel traceroute
# KEEP ORDER
(echo debian.org; echo freenetproject.org) \
| parallel -k traceroute
# RUN MANY JOBS. USE OUTPUT
# Find the number of hosts responding to ping
seq 1 255 | parallel -j255 ping -c 1 178.63.11.{} 2>&1 \
| grep '64 bytes' | wc -l
seq 1 255 | parallel -j0 ping -c 1 178.63.11.{} 2>&1 \
| grep '64 bytes' | wc -l
# MULTIPLE ARGUMENTS
# make dir: (1-20000).dir
cd ../dirs
rm -rf *; sync
seq 1 20000 | time parallel mkdir {}.dir
rm -rf *; sync
seq 1 20000 | time parallel -X mkdir {}.dir
# CALLING GNU PARALLEL FROM GNU PARALLEL
# make dir: top-(1-100)/sub-(1-300)
rm -rf *; sync
seq 1 100 | time parallel -I /// \
'mkdir top-///;cd top-///; seq 1 300 | parallel -X mkdir sub-{}'
find | wc -l
# Thanks for watching
# Find GNU Parallel at http://www.gnu.org/software/parallel/
# GIVE ME THE FIRST RESULT
(echo foss.org.my; echo debian.org; echo freenetproject.org) | parallel -H2 traceroute {}";false"
find . -type f | parallel -k -j150% -n 1000 -m grep -H -n STRING {}
(echo foss.org.my; echo debian.org; echo freenetproject.org) | parallel traceroute
Ved afslutning:
if cleanup:
for each cleanup: cleanup
=head1 IDEAS =head1 IDEAS
@ -14,8 +100,6 @@ En ssh med 20% loss og 900 ms delay, så kan login nås på 15 sek.
Test if -0 works on filenames ending in '\n' Test if -0 works on filenames ending in '\n'
Transfer scriptfile before first job. Remove it when last job done.
monitor to see which jobs are currently running monitor to see which jobs are currently running
http://code.google.com/p/ppss/ http://code.google.com/p/ppss/

View file

@ -48,6 +48,8 @@ echo put parallel-$YYYYMMDD.tar.bz2{,.sig,*asc} | ncftp ftp://ftp-upload.gnu.org
== Update Savannah == == Update Savannah ==
https://savannah.gnu.org/news/submit.php?group=parallel
doc/pod2savannah_publicinfo src/parallel | klipper-stdin doc/pod2savannah_publicinfo src/parallel | klipper-stdin
https://savannah.gnu.org/project/admin/editgroupinfo.php?group=parallel https://savannah.gnu.org/project/admin/editgroupinfo.php?group=parallel

View file

@ -89,9 +89,9 @@ you use this option, stdin is given to the first process run.
Otherwise, stdin is redirected from /dev/null. Otherwise, stdin is redirected from /dev/null.
=item B<--basefile> I<file> =item B<--basefile> I<file> (beta testing)
=item B<-B> I<file> =item B<-B> I<file> (beta testing)
I<file> will be transferred to each sshlogin before a jobs is I<file> will be transferred to each sshlogin before a jobs is
started. It will be removed if B<--cleanup> is active. The file may be started. It will be removed if B<--cleanup> is active. The file may be
@ -185,15 +185,16 @@ B<-g> is the default. Can be reversed with B<-u>.
Print a summary of the options to GNU B<parallel> and exit. Print a summary of the options to GNU B<parallel> and exit.
=item B<--halt-on-error> <0|1|2> =item B<--halt-on-error> <0|1|2> (beta testing)
=item B<-H> <0|1|2> =item B<-H> <0|1|2> (beta testing)
=over 3 =over 3
=item 0 =item 0
Do not halt if a job fails. This is the default. Do not halt if a job fails. Exit status will be the number of jobs
failed. This is the default.
=item 1 =item 1
@ -282,6 +283,23 @@ Keep sequence of output same as the order of input. If jobs 1 2 3 4
end in the sequence 3 1 4 2 the output will still be 1 2 3 4. end in the sequence 3 1 4 2 the output will still be 1 2 3 4.
=item B<-L> I<max-lines>
Use at most I<max-lines> nonblank input lines per command line.
Trailing blanks cause an input line to be logically continued on the
next input line. Implies B<-x>.
=item B<--max-lines>[=I<max-lines>]
=item B<-l>[I<max-lines>]
Synonym for the B<-L> option. Unlike B<-L>, the I<max-lines> argument
is optional. If I<max-lines> is not specified, it defaults to one.
The B<-l> option is deprecated since the POSIX standard specifies
B<-L> instead.
=item B<--controlmaster> (experimental) =item B<--controlmaster> (experimental)
=item B<-M> (experimental) =item B<-M> (experimental)
@ -565,6 +583,14 @@ give surprising results if B<{}> is used as part of a word.
Support for B<-X> with B<--sshlogin> is limited and may fail. Support for B<-X> with B<--sshlogin> is limited and may fail.
=item B<--exit>
=item B<-x>
Exit if the size (see the B<-s> option) is exceeded.
=back =back
=head1 EXAMPLE: Working as xargs -n1. Argument appending =head1 EXAMPLE: Working as xargs -n1. Argument appending
@ -781,6 +807,17 @@ B<(echo foss.org.my; echo debian.org; echo freenetproject.org) | parallel -k tra
This will make sure the traceroute to foss.org.my will be printed This will make sure the traceroute to foss.org.my will be printed
first. first.
=head1 EXAMPLE: Parallel grep
B<grep -r> greps recursively through directories. On multicore CPUs
GNU B<parallel> can often speed this up.
find . -type f | parallel -k -j150% -n 1000 -m grep -H -n STRING {}
This will run 1.5 job per core, and give 1000 arguments to B<grep>.
=head1 EXAMPLE: Using remote computers =head1 EXAMPLE: Using remote computers
To run commands on a remote computer SSH needs to be set up and you To run commands on a remote computer SSH needs to be set up and you
@ -1105,7 +1142,9 @@ support for running number-of-cpu-cores jobs in parallel.
B<xargs> has no support for grouping the output, therefore output may B<xargs> has no support for grouping the output, therefore output may
run together, e.g. the first half of a line is from one process and run together, e.g. the first half of a line is from one process and
the last half of the line is from another process. the last half of the line is from another process. The example
B<Parallel grep> cannot be done reliably with B<xargs> because of
this.
B<xargs> has no support for keeping the order of the output, therefore B<xargs> has no support for keeping the order of the output, therefore
if running jobs in parallel using B<xargs> the output of the second if running jobs in parallel using B<xargs> the output of the second
@ -1152,6 +1191,12 @@ B<make -j> can run jobs in parallel, but requires a crafted Makefile
to do this. That results in extra quoting to get filename containing to do this. That results in extra quoting to get filename containing
newline to work correctly. newline to work correctly.
B<make -j> has no support for grouping the output, therefore output
may run together, e.g. the first half of a line is from one process
and the last half of the line is from another process. The example
B<Parallel grep> cannot be done reliably with B<make -j> because of
this.
(Very early versions of GNU Parallel was coincidently implemented (Very early versions of GNU Parallel was coincidently implemented
using B<make -j>). using B<make -j>).
@ -1177,7 +1222,8 @@ B<ppss> again it may cause nothing to happen as B<ppss> thinks the
task is already done. GNU B<parallel> will normally not need cleaning task is already done. GNU B<parallel> will normally not need cleaning
up if running locally and will only need cleaning up if stopped up if running locally and will only need cleaning up if stopped
abnormally and running remote (B<--cleanup> may not complete if abnormally and running remote (B<--cleanup> may not complete if
stopped abnormally). stopped abnormally). The example B<Parallel grep> would require extra
postprocessing if written using B<ppss>.
=head3 EXAMPLES FROM ppss MANUAL =head3 EXAMPLES FROM ppss MANUAL
@ -1606,18 +1652,15 @@ sub parse_options {
"eof|e:s" => \$::opt_E, "eof|e:s" => \$::opt_E,
"max-args|n=i" => \$::opt_n, "max-args|n=i" => \$::opt_n,
"help|h" => \$::opt_help, "help|h" => \$::opt_help,
"L=i" => \$::opt_L,
"max-lines|l:i" => \$::opt_l,
"verbose|t" => \$::opt_verbose, "verbose|t" => \$::opt_verbose,
"version|V" => \$::opt_version, "version|V" => \$::opt_version,
"show-limits" => \$::opt_show_limits, "show-limits" => \$::opt_show_limits,
"exit|x" => \$::opt_x,
## xargs-compatibility - implemented, man - unittest missing ## xargs-compatibility - implemented, man - unittest missing
"interactive|p" => \$::opt_p, "interactive|p" => \$::opt_p,
## How to unittest? tty skal emuleres ## How to unittest? tty must be emulated
# xargs-compatibility - unimplemented
"L=i" => \$::opt_L,
"max-lines|l:i" => \$::opt_l,
## (echo a b;echo c) | xargs -l1 echo
## (echo a b' ';echo c) | xargs -l1 echo
"exit|x" => \$::opt_x,
) || die_usage(); ) || die_usage();
$Global::debug = (defined $::opt_D); $Global::debug = (defined $::opt_D);
$Global::input_is_filename = (@ARGV); $Global::input_is_filename = (@ARGV);
@ -1640,6 +1683,10 @@ sub parse_options {
if(defined $::opt_U) { $Global::replace_no_ext = $::opt_U; } if(defined $::opt_U) { $Global::replace_no_ext = $::opt_U; }
if(defined $::opt_i and $::opt_i) { $Global::replacestring = $::opt_i; } if(defined $::opt_i and $::opt_i) { $Global::replacestring = $::opt_i; }
if(defined $::opt_E and $::opt_E) { $Global::end_of_file_string = $::opt_E; } if(defined $::opt_E and $::opt_E) { $Global::end_of_file_string = $::opt_E; }
if(defined $::opt_L and $::opt_L or defined $::opt_l) {
$Global::max_lines = $::opt_l || $::opt_L || 1;
$Global::max_number_of_args = $Global::max_lines;
}
if(defined $::opt_n and $::opt_n) { $Global::max_number_of_args = $::opt_n; } if(defined $::opt_n and $::opt_n) { $Global::max_number_of_args = $::opt_n; }
if(defined $::opt_help) { die_usage(); } if(defined $::opt_help) { die_usage(); }
if(defined $::opt_number_of_cpus) { print no_of_cpus(),"\n"; exit(0); } if(defined $::opt_number_of_cpus) { print no_of_cpus(),"\n"; exit(0); }
@ -1658,7 +1705,9 @@ sub parse_options {
if(defined $::opt_a) { if(defined $::opt_a) {
if(not open(ARGFILE,"<",$::opt_a)) { if(not open(ARGFILE,"<",$::opt_a)) {
print STDERR "$Global::progname: Cannot open input file `$::opt_a': No such file or directory\n"; print STDERR "$Global::progname: ".
"Cannot open input file `$::opt_a': ".
"No such file or directory\n";
exit(255); exit(255);
} }
$Global::argfile = *ARGFILE; $Global::argfile = *ARGFILE;
@ -1687,7 +1736,8 @@ sub parse_options {
} }
} else { } else {
for my $sshlogin (keys %Global::host) { for my $sshlogin (keys %Global::host) {
$Global::host{$sshlogin}{'max_no_of_running'} = $Global::default_simultaneous_sshlogins; $Global::host{$sshlogin}{'max_no_of_running'} =
$Global::default_simultaneous_sshlogins;
} }
} }
$Global::job_end_sequence=1; $Global::job_end_sequence=1;
@ -1713,60 +1763,64 @@ sub generate_command_line {
my $command = shift; my $command = shift;
my ($job_line,$last_good); my ($job_line,$last_good);
my ($next_arg,@quoted_args,@quoted_args_no_ext,$arg_length); my ($next_arg,@quoted_args,@quoted_args_no_ext,$arg_length);
my ($number_of_substitution,$number_of_substitution_no_ext,$length_of_context,$length_of_command_no_args,$spaces); my ($number_of_substitution,
if($Global::xargs or $Global::Xargs) { $number_of_substitution_no_ext,$spaces,
($number_of_substitution, $number_of_substitution_no_ext,$spaces, $length_of_command_no_args,$length_of_context) =
$length_of_command_no_args,$length_of_context) = xargs_computations($command); xargs_computations($command);
}
my $number_of_args = 0; my $number_of_args = 0;
# max number of lines (-L) =
# number_of_read_lines = 0
while (defined($next_arg = get_next_arg())) { while (defined($next_arg = get_next_arg())) {
my $next_arg_no_ext = no_extension($next_arg); my $next_arg_no_ext = no_extension($next_arg);
# if defined max_number_of_lines
# number_of_read_lines++
# if $next_arg =~ /\w$/ then number_of_read_lines--
# Trailing blanks cause an
# input line to be logically continued on the next input line.
# if number_of_read_lines > max_number_of_lines
# last
push (@quoted_args, $next_arg); push (@quoted_args, $next_arg);
push (@quoted_args_no_ext, $next_arg_no_ext); push (@quoted_args_no_ext, $next_arg_no_ext);
$number_of_args++; $number_of_args++;
if(not $Global::xargs and not $Global::Xargs) {
# No xargs-mode: Just one argument per line
last;
} else {
# Emulate xargs if there is a command and -x or -X is set # Emulate xargs if there is a command and -x or -X is set
my $next_arg_len = $number_of_substitution * (length ($next_arg) + $spaces) + my $next_arg_len =
$number_of_substitution * (length ($next_arg) + $spaces)
+ $number_of_substitution_no_ext * (length ($next_arg_no_ext) + $spaces) + $number_of_substitution_no_ext * (length ($next_arg_no_ext) + $spaces)
+ $length_of_context; + $length_of_context;
$arg_length += $next_arg_len; $arg_length += $next_arg_len;
my $job_line_length = $length_of_command_no_args + $arg_length; my $job_line_length = $length_of_command_no_args + $arg_length;
if($job_line_length >= max_length_of_command_line()) { if($job_line_length >= max_length_of_command_line()) {
unget_arg(pop @quoted_args); unget_arg(pop @quoted_args);
if($::opt_x and $length_of_command_no_args + $next_arg_len
>= max_length_of_command_line()) {
# To be compatible with xargs -x
print STDERR ("Command line too long ($job_line_length >= "
. max_length_of_command_line() .
") at number $number_of_args: ".
(substr($next_arg,0,50))."...\n");
exit(255);
}
if(defined $quoted_args[0]) { if(defined $quoted_args[0]) {
last; last;
} else { } else {
die ("Command line too long ($job_line_length >= " print STDERR ("Command line too long ($job_line_length >= "
. max_length_of_command_line() . ") at number $number_of_args: $next_arg"); . max_length_of_command_line() .
") at number $number_of_args: ".
(substr($next_arg,0,50))."...\n");
exit(255);
} }
} }
if($Global::max_number_of_args and $number_of_args >= $Global::max_number_of_args) { if($Global::max_number_of_args and
$number_of_args >= $Global::max_number_of_args) {
last; last;
} }
if(not $Global::xargs and not $Global::Xargs) {
# No xargs-mode: Just one argument per line
last;
} }
} }
if(@quoted_args) { if(@quoted_args) {
$job_line = $command; $job_line = $command;
if(defined $job_line and if(defined $job_line and
($job_line =~/\Q$Global::replacestring\E/o or $job_line =~/\Q$Global::replace_no_ext\E/o)) { ($job_line =~/\Q$Global::replacestring\E/o or
$job_line =~/\Q$Global::replace_no_ext\E/o)) {
# substitute {} and {.} with args # substitute {} and {.} with args
if($Global::Xargs) { if($Global::Xargs) {
# Context sensitive replace (foo{}bar with fooargsbar) # Context sensitive replace (foo{}bar with fooargsbar)
$job_line = context_replace($job_line, \@quoted_args, \@quoted_args_no_ext); $job_line =
context_replace($job_line, \@quoted_args, \@quoted_args_no_ext);
} else { } else {
# Normal replace {} with args and {.} with args without extension # Normal replace {} with args and {.} with args without extension
my $arg=join(" ",@quoted_args); my $arg=join(" ",@quoted_args);
@ -1793,23 +1847,20 @@ sub generate_command_line {
sub xargs_computations { sub xargs_computations {
my $command = shift; my $command = shift;
if(not @Calculated::xargs_computations) { if(not @Calculated::xargs_computations) {
my ($length_of_command_no_args, $length_of_context, $spaces); my ($number_of_substitution, $number_of_substitution_no_ext,
$spaces,$length_of_command_no_args,$length_of_context)
= (1,0,0,0,0);
if($command) {
# Count number of {}'s on the command line # Count number of {}'s on the command line
my $no_of_replace = ($command =~ s/\Q$Global::replacestring\E/$Global::replacestring/go); my $no_of_replace =
my $number_of_substitution = $no_of_replace || 1; ($command =~ s/\Q$Global::replacestring\E/$Global::replacestring/go);
$number_of_substitution = $no_of_replace || 1;
# Count number of {.}'s on the command line # Count number of {.}'s on the command line
my $no_of_no_ext = ($command =~ s/\Q$Global::replace_no_ext\E/$Global::replace_no_ext/go); my $no_of_no_ext =
my $number_of_substitution_no_ext = $no_of_no_ext || 0; ($command =~ s/\Q$Global::replace_no_ext\E/$Global::replace_no_ext/go);
$number_of_substitution_no_ext = $no_of_no_ext || 0;
# Count # Count
my $c = $command; my $c = $command;
if($Global::xargs) {
# remove all {}s
$c =~ s/\Q$Global::replacestring\E|\Q$Global::replace_no_ext\E//og;
$length_of_command_no_args = length($c) - $no_of_replace - $no_of_no_ext;
$length_of_context = 0;
$spaces = 1;
}
if($Global::Xargs) { if($Global::Xargs) {
$c =~ s/\S*\Q$Global::replacestring\E\S*//go; $c =~ s/\S*\Q$Global::replacestring\E\S*//go;
$c =~ s/\S*\Q$Global::replace_no_ext\E\S*//go; $c =~ s/\S*\Q$Global::replace_no_ext\E\S*//go;
@ -1818,8 +1869,15 @@ sub xargs_computations {
- $no_of_replace * length($Global::replacestring) - $no_of_replace * length($Global::replacestring)
- $no_of_no_ext * length($Global::replace_no_ext); - $no_of_no_ext * length($Global::replace_no_ext);
$spaces = 0; $spaces = 0;
} else {
# remove all {}s
$c =~ s/\Q$Global::replacestring\E|\Q$Global::replace_no_ext\E//og;
$length_of_command_no_args = length($c) -
$no_of_replace - $no_of_no_ext;
$length_of_context = 0;
$spaces = 1;
}
} }
@Calculated::xargs_computations = @Calculated::xargs_computations =
($number_of_substitution, $number_of_substitution_no_ext, ($number_of_substitution, $number_of_substitution_no_ext,
$spaces,$length_of_command_no_args,$length_of_context); $spaces,$length_of_command_no_args,$length_of_context);
@ -1944,7 +2002,8 @@ sub compute_number_of_processes {
my $sshlogin = shift; my $sshlogin = shift;
my $wanted_processes = user_requested_processes($opt_P,$sshlogin); my $wanted_processes = user_requested_processes($opt_P,$sshlogin);
debug("Wanted procs: $wanted_processes\n"); debug("Wanted procs: $wanted_processes\n");
my $system_limit = processes_available_by_system_limit($wanted_processes,$sshlogin); my $system_limit =
processes_available_by_system_limit($wanted_processes,$sshlogin);
debug("Limited to procs: $system_limit\n"); debug("Limited to procs: $system_limit\n");
return $system_limit; return $system_limit;
} }
@ -2038,8 +2097,10 @@ sub processes_available_by_system_limit {
wait(); wait();
# Cleanup: Unget the command_lines (and args_refs) # Cleanup: Unget the command_lines (and args_refs)
unget_command_line(@command_lines); unget_command_line(@command_lines);
if($sshlogin ne ":" and $system_limit > $Global::default_simultaneous_sshlogins) { if($sshlogin ne ":" and
$system_limit = simultaneous_sshlogin_limit($sshlogin,$system_limit); $system_limit > $Global::default_simultaneous_sshlogins) {
$system_limit =
simultaneous_sshlogin_limit($sshlogin,$system_limit);
} }
return $system_limit; return $system_limit;
} }
@ -2067,12 +2128,17 @@ sub simultaneous_sshlogin_limit {
my ($sshcmd,$serverlogin) = sshcommand_of_sshlogin($sshlogin); my ($sshcmd,$serverlogin) = sshcommand_of_sshlogin($sshlogin);
# Try twice because it guesses wrong sometimes # Try twice because it guesses wrong sometimes
# Choose the minimal # Choose the minimal
my $ssh_limit = min(simultaneous_sshlogin($sshlogin,$wanted_processes), my $ssh_limit =
min(simultaneous_sshlogin($sshlogin,$wanted_processes),
simultaneous_sshlogin($sshlogin,$wanted_processes)); simultaneous_sshlogin($sshlogin,$wanted_processes));
if($ssh_limit < $wanted_processes) { if($ssh_limit < $wanted_processes) {
print STDERR ("Warning: ssh to $serverlogin only allows for $ssh_limit simultaneous logins.\n", print STDERR
"You may raise this by changing /etc/ssh/sshd_config:MaxStartup on $serverlogin\n", ("Warning: ssh to $serverlogin only allows ",
"Using only ",$ssh_limit-1," connections to avoid race conditions\n"); "for $ssh_limit simultaneous logins.\n",
"You may raise this by changing ",
"/etc/ssh/sshd_config:MaxStartup on $serverlogin\n",
"Using only ",$ssh_limit-1," connections ",
"to avoid race conditions\n");
} }
# Race condition can cause problem if using all sshs. # Race condition can cause problem if using all sshs.
if($ssh_limit > 1) { $ssh_limit -= 1; } if($ssh_limit > 1) { $ssh_limit -= 1; }
@ -2106,14 +2172,17 @@ sub user_requested_processes {
if($opt_P =~ /^\+(\d+)$/) { if($opt_P =~ /^\+(\d+)$/) {
# E.g. -P +2 # E.g. -P +2
my $j = $1; my $j = $1;
$processes = $j + no_of_processing_units_sshlogin($sshlogin); $processes =
$j + no_of_processing_units_sshlogin($sshlogin);
} elsif ($opt_P =~ /^-(\d+)$/) { } elsif ($opt_P =~ /^-(\d+)$/) {
# E.g. -P -2 # E.g. -P -2
my $j = $1; my $j = $1;
$processes = no_of_processing_units_sshlogin($sshlogin) - $j; $processes =
no_of_processing_units_sshlogin($sshlogin) - $j;
} elsif ($opt_P =~ /^(\d+)\%$/) { } elsif ($opt_P =~ /^(\d+)\%$/) {
my $j = $1; my $j = $1;
$processes = no_of_processing_units_sshlogin($sshlogin) * $j / 100; $processes =
no_of_processing_units_sshlogin($sshlogin) * $j / 100;
} elsif ($opt_P =~ /^(\d+)$/) { } elsif ($opt_P =~ /^(\d+)$/) {
$processes = $1; $processes = $1;
if($processes == 0) { if($processes == 0) {
@ -2153,7 +2222,8 @@ sub no_of_processing_units_sshlogin {
if($ncpu =~ /^[0-9]+$/) { if($ncpu =~ /^[0-9]+$/) {
$Global::host{$sshlogin}{'ncpus'} = $ncpu; $Global::host{$sshlogin}{'ncpus'} = $ncpu;
} else { } else {
print STDERR ("Warning: Could not figure out number of cpus on $serverlogin. Using 1"); print STDERR ("Warning: Could not figure out ",
"number of cpus on $serverlogin. Using 1");
$Global::host{$sshlogin}{'ncpus'} = 1; $Global::host{$sshlogin}{'ncpus'} = 1;
} }
} }
@ -2164,8 +2234,7 @@ sub no_of_processing_units_sshlogin {
sub no_of_cpus { sub no_of_cpus {
if(not $Global::no_of_cpus) { if(not $Global::no_of_cpus) {
local $/="\n"; # If delimiter is set, then $/ will be wrong local $/="\n"; # If delimiter is set, then $/ will be wrong
my $no_of_cpus = (0 my $no_of_cpus = (no_of_cpus_freebsd()
|| no_of_cpus_freebsd()
|| no_of_cpus_darwin() || no_of_cpus_darwin()
|| no_of_cpus_solaris() || no_of_cpus_solaris()
|| no_of_cpus_gnu_linux() || no_of_cpus_gnu_linux()
@ -2183,8 +2252,7 @@ sub no_of_cpus {
sub no_of_cores { sub no_of_cores {
if(not $Global::no_of_cores) { if(not $Global::no_of_cores) {
local $/="\n"; # If delimiter is set, then $/ will be wrong local $/="\n"; # If delimiter is set, then $/ will be wrong
my $no_of_cores = (0 my $no_of_cores = (no_of_cores_freebsd()
|| no_of_cores_freebsd()
|| no_of_cores_darwin() || no_of_cores_darwin()
|| no_of_cores_solaris() || no_of_cores_solaris()
|| no_of_cores_gnu_linux() || no_of_cores_gnu_linux()
@ -2303,7 +2371,7 @@ sub min {
# $Global::running{$pid}{sshlogin} = server to run on # $Global::running{$pid}{sshlogin} = server to run on
# $Global::running{$pid}{'exitstatus'} = exit status # $Global::running{$pid}{'exitstatus'} = exit status
# $Global::host{$sshlogin}{'no_of_running'} = number of currently running jobs # $Global::host{$sshlogin}{'no_of_running'} = number of currently running jobs
# $Global::host{$sshlogin}{'ncpus'} = number of cpus # $Global::host{$sshlogin}{'ncpus'} = number of CPUs (or CPU cores)
# $Global::host{$sshlogin}{'maxlength'} = max line length (currently buggy for remote) # $Global::host{$sshlogin}{'maxlength'} = max line length (currently buggy for remote)
# $Global::host{$sshlogin}{'max_no_of_running'} = number of currently running jobs # $Global::host{$sshlogin}{'max_no_of_running'} = number of currently running jobs
# $Global::host{$sshlogin}{'sshcmd'} = command to use as ssh # $Global::host{$sshlogin}{'sshcmd'} = command to use as ssh
@ -2388,7 +2456,8 @@ sub get_next_arg {
} }
$arg = <$Global::argfile>; $arg = <$Global::argfile>;
chomp $arg; chomp $arg;
if($Global::end_of_file_string and $arg eq $Global::end_of_file_string) { if($Global::end_of_file_string and
$arg eq $Global::end_of_file_string) {
# Ignore the rest of STDIN # Ignore the rest of STDIN
while (<$Global::argfile>) {} while (<$Global::argfile>) {}
return undef; return undef;
@ -2398,6 +2467,12 @@ sub get_next_arg {
return get_next_arg(); return get_next_arg();
} }
} }
if($Global::max_lines) {
if($arg =~ /\s$/) {
# Trailing space => continued on next line
$arg .= get_next_arg();
}
}
if($Global::input_is_filename) { if($Global::input_is_filename) {
$arg = shell_quote($arg); $arg = shell_quote($arg);
} }
@ -2494,8 +2569,10 @@ sub start_job {
close TTY; close TTY;
my $run_yes = ($answer =~ /^\s*y/i); my $run_yes = ($answer =~ /^\s*y/i);
if (not $run_yes) { if (not $run_yes) {
open STDOUT, ">&", $Global::original_stdout or die "Can't dup \$oldout: $!"; open STDOUT, ">&", $Global::original_stdout
open STDERR, ">&", $Global::original_stderr or die "Can't dup \$oldout: $!"; or die "Can't dup \$oldout: $!";
open STDERR, ">&", $Global::original_stderr
or die "Can't dup \$oldout: $!";
return; return;
} }
} else { } else {
@ -2515,14 +2592,17 @@ sub start_job {
$pid = open3("<&STDIN", ">&STDOUT", ">&STDERR", $command) || $pid = open3("<&STDIN", ">&STDOUT", ">&STDERR", $command) ||
die("open3 failed. Report a bug to <bug-parallel\@gnu.org>\n"); die("open3 failed. Report a bug to <bug-parallel\@gnu.org>\n");
# Re-open to avoid complaining # Re-open to avoid complaining
open STDIN, "<&", $Global::original_stdin or die "Can't dup \$Global::original_stdin: $!"; open STDIN, "<&", $Global::original_stdin
or die "Can't dup \$Global::original_stdin: $!";
} else { } else {
$pid = open3(gensym, ">&STDOUT", ">&STDERR", $command) || $pid = open3(gensym, ">&STDOUT", ">&STDERR", $command) ||
die("open3 failed. Report a bug to <bug-parallel\@gnu.org>\n"); die("open3 failed. Report a bug to <bug-parallel\@gnu.org>\n");
} }
debug("started: $command\n"); debug("started: $command\n");
open STDOUT, ">&", $Global::original_stdout or die "Can't dup \$Global::original_stdout: $!"; open STDOUT, ">&", $Global::original_stdout
open STDERR, ">&", $Global::original_stderr or die "Can't dup \$Global::original_stderr: $!"; or die "Can't dup \$Global::original_stdout: $!";
open STDERR, ">&", $Global::original_stderr
or die "Can't dup \$Global::original_stderr: $!";
if($Global::grouped) { if($Global::grouped) {
return ("seq" => $Global::job_start_sequence, return ("seq" => $Global::job_start_sequence,
@ -2702,8 +2782,8 @@ sub sshtransferreturn {
# Use different subdirs depending on abs or rel path # Use different subdirs depending on abs or rel path
my $rsync_destdir = ($relpath ? "./" : "/"); my $rsync_destdir = ($relpath ? "./" : "/");
if($transfer) { if($transfer) {
# Abs path: rsync -rlDzR /home/tange/dir/subdir/file.gz server:/ # Abs path: rsync -rlDzRE /home/tange/dir/subdir/file.gz server:/
# Rel path: rsync -rlDzR ./subdir/file.gz server:./ # Rel path: rsync -rlDzRE ./subdir/file.gz server:./
if(-r shell_unquote($file)) { if(-r shell_unquote($file)) {
return "rsync $rsync_opt $file $serverlogin:$rsync_destdir"; return "rsync $rsync_opt $file $serverlogin:$rsync_destdir";
} else { } else {
@ -2720,8 +2800,8 @@ sub sshtransferreturn {
my $remove = $removesource ? "--remove-source-files" : ""; my $remove = $removesource ? "--remove-source-files" : "";
my $replaced = context_replace($ret_file,[$file],[$noext]); my $replaced = context_replace($ret_file,[$file],[$noext]);
# --return # --return
# Abs path: rsync -rlDzR server:/home/tange/dir/subdir/file.gz / # Abs path: rsync -rlDzRE server:/home/tange/dir/subdir/file.gz /
# Rel path: rsync -rlDzR server:./subsir/file.gz ./ # Rel path: rsync -rlDzRE server:./subsir/file.gz ./
push(@cmd, "rsync $rsync_opt $remove $serverlogin:" push(@cmd, "rsync $rsync_opt $remove $serverlogin:"
.shell_quote($replaced)." ".$rsync_destdir); .shell_quote($replaced)." ".$rsync_destdir);
} }
@ -2779,7 +2859,8 @@ sub ListRunningJobs {
} }
sub StartNoNewJobs { sub StartNoNewJobs {
print STDERR ("$Global::progname: SIGTERM received. No new jobs will be started.\n", print STDERR
("$Global::progname: SIGTERM received. No new jobs will be started.\n",
"$Global::progname: Waiting for these ", scalar(keys %Global::running), "$Global::progname: Waiting for these ", scalar(keys %Global::running),
" jobs to finish. Send SIGTERM again to stop now.\n"); " jobs to finish. Send SIGTERM again to stop now.\n");
ListRunningJobs(); ListRunningJobs();
@ -2826,7 +2907,8 @@ sub Reaper {
my $print_now = ($Global::running{$stiff}{'exitstatus'} and my $print_now = ($Global::running{$stiff}{'exitstatus'} and
$::opt_halt_on_error and $::opt_halt_on_error == 2); $::opt_halt_on_error and $::opt_halt_on_error == 2);
if($Global::keeporder and not $print_now) { if($Global::keeporder and not $print_now) {
$Global::print_later{$Global::running{$stiff}{"seq"}} = $Global::running{$stiff}; $Global::print_later{$Global::running{$stiff}{"seq"}} =
$Global::running{$stiff};
while($Global::print_later{$Global::job_end_sequence}) { while($Global::print_later{$Global::job_end_sequence}) {
debug("Found job end $Global::job_end_sequence"); debug("Found job end $Global::job_end_sequence");
print_job($Global::print_later{$Global::job_end_sequence}); print_job($Global::print_later{$Global::job_end_sequence});

View file

@ -18,12 +18,12 @@ ls b
1 1 10 1 1 10
1 1 15 1 1 15
1 1 15 1 1 15
4 4 26
6 6 41
1-col.txt 1-col.txt
1-col.txt.diff 1-col.txt.diff
2-col.txt 2-col.txt
2-col.txt.diff 2-col.txt.diff
4 4 26
6 6 41
a a
b b
ls 1-col.txt.diff|wc;echo 1-col.txt.diff ls 1-col.txt.diff|wc;echo 1-col.txt.diff
@ -32,6 +32,7 @@ ls 2-col.txt.diff|wc;echo 2-col.txt.diff
ls 2-col.txt|wc;echo 2-col.txt ls 2-col.txt|wc;echo 2-col.txt
ls a|wc;echo a ls a|wc;echo a
ls b|wc;echo b ls b|wc;echo b
### Check that we can have more input than max procs (-j 0)
touch more_than_5000-9990 touch more_than_5000-9990
touch more_than_5000-9991 touch more_than_5000-9991
touch more_than_5000-9992 touch more_than_5000-9992

View file

@ -1,5 +1,7 @@
### This causes problems if we kill child processes
33bf8b2986551515cdaff5e860618098 - 33bf8b2986551515cdaff5e860618098 -
960bedf6398273937e314fb49c7b3ffa - d7fb96d6a56d4347bc24930a395c431a -
### Test of xargs -m and -X
b35d8e49be8d94899b719c40d3f1f4bb - b35d8e49be8d94899b719c40d3f1f4bb -
3 60000 348894 3 60000 348894
1foo bar2foo bar3 Afoo barBfoo barC 1foo bar2foo bar3 Afoo barBfoo barC
@ -8,5 +10,6 @@ b35d8e49be8d94899b719c40d3f1f4bb -
6 119994 697800 6 119994 697800
31d9274be5fdc2de59487cb05ba57776 - 31d9274be5fdc2de59487cb05ba57776 -
Chars per line: 116300 Chars per line: 116300
### Bug before 2009-08-26 causing regexp compile error or infinite loop
'a' 'a'
'a' 'a'

View file

@ -1,3 +1,94 @@
### Test -L -l and --max-lines
a_b c
a_b c
a_b
c
a_b
c
a_b c
d
a_b c
d
a_b c d
e
a_b c d
e
a_b c
d
e
a_b c
d
e
a_b c d
e
a_b c d
e
a_b c
d
e
a_b c
d
e
a_b c d
e
a_b c d
e
a_b c
d
e
a_b c
d
e
### test too long args
Command line too long (1000005 >= 131071) at number 1: zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz...
xargs: argument line too long
1 2
3 4
5 6
7 8
Command line too long (1000007 >= 10) at number 1: zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz...
9 10
1 2
3 4
5 6
7 8
xargs: argument line too long
9 10
### Test -x
1 2
3 4
5 6
Command line too long (15 >= 10) at number 3: 12345...
7 8
1 2
3 4
5 6
xargs: argument line too long
7 8
1 2
3 4
5 6
7 8
9 10
1234
12
13
14
15
1 2
3 4
5 6
7 8
9 10
1234
12
13
14
15
### Test bugfix if no command given
Command line too long (1000002 >= 10) at number 1: zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz...
1 2 3 4 5
### Test -a and --arg-file: Read input from file instead of stdin
1 1
2 2
3 3
@ -18,12 +109,23 @@
8 8
9 9
10 10
xargs Expect: 3 1 2
3 3
1 1
2 2
parallel Expect: 3 1 2
3
1
2
xargs Expect: 1 3 2
1 1
3 3
2 2
parallel Expect: 1 3 2
1
3
2
### Test -i and --replace: Replace with argument
replace replace
replace replace
replace replace
@ -33,12 +135,15 @@ replace
replace replace
replace replace
replace replace
### Test -E: Artificial end-of-file
include this
include this
### Test -e and --eof: Artificial end-of-file
include this include this
include this include this
include this include this
include this include this
include this ### Test -n and --max-args: Max number of args per line (only with -X and -m)
include this
line 1 line 1
line 2 line 2
line 3 line 3
@ -59,6 +164,7 @@ line 1 line 1
line 2 line 2
line 1 line 1 line 1 line 1
line 2 line 2
### Test --max-procs and -P: Number of processes
max proc 1 max proc 1
max proc 2 max proc 2
max proc 3 max proc 3
@ -79,6 +185,8 @@ max proc 10
200% proc 8 200% proc 8
200% proc 9 200% proc 9
200% proc 10 200% proc 10
### Test --delimiter and -d: Delimiter instead of newline
# Yes there is supposed to be an extra newline for -d N
This is line 1 This is line 1
This is line 2 This is line 2
This is line 3 This is line 3
@ -93,20 +201,26 @@ line 3
delimiter TAB line 1 delimiter TAB line 1
line 2 line 2
line 3 line 3
### Test --max-chars and -s: Max number of chars in a line
line 1 line 1 line 1 line 1
line 2 line 2
line 1 line 1 line 1 line 1
line 2 line 2
### Test --no-run-if-empty and -r: This should give no output
### Test --help and -h: Help output (just check we get the same amount of lines)
Output from -h and --help Output from -h and --help
4 4
4 4
### Test --version: Version output (just check we get the same amount of lines)
7 7
### Test --verbose and -t
echo bar echo bar
echo car echo car
echo far echo far
echo bar echo bar
echo car echo car
echo far echo far
### Test --show-limits
Maximal size of command: 131071 Maximal size of command: 131071
Maximal used size of command: 131071 Maximal used size of command: 131071

View file

@ -1,4 +1,5 @@
### Test --transfer --return --cleanup ### Test --transfer --return --cleanup
# Create some weirdly files in /tmp
### --transfer - abspath ### --transfer - abspath
file1 file1
file2 file2
@ -93,6 +94,9 @@ good if no file
ls: cannot access tmp/parallel.file*: No such file or directory ls: cannot access tmp/parallel.file*: No such file or directory
OK OK
### --return - abspath ### --return - abspath
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.out
/tmp/parallel.file1.out
/tmp/parallel.file10.out /tmp/parallel.file10.out
/tmp/parallel.file13.out /tmp/parallel.file13.out
/tmp/parallel.file14.out /tmp/parallel.file14.out
@ -101,9 +105,8 @@ OK
/tmp/parallel.file17.out /tmp/parallel.file17.out
/tmp/parallel.file18.out /tmp/parallel.file18.out
/tmp/parallel.file19.out /tmp/parallel.file19.out
/tmp/parallel.file1.out
/tmp/parallel.file20.out
/tmp/parallel.file2.out /tmp/parallel.file2.out
/tmp/parallel.file20.out
/tmp/parallel.file3.out /tmp/parallel.file3.out
/tmp/parallel.file5.out /tmp/parallel.file5.out
/tmp/parallel.file6.out /tmp/parallel.file6.out
@ -111,9 +114,10 @@ OK
/tmp/parallel.file8.out /tmp/parallel.file8.out
/tmp/parallel.file9.out /tmp/parallel.file9.out
/tmp/parallel.file>fire.out /tmp/parallel.file>fire.out
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.out
### --return - relpath ### --return - relpath
tmp/parallel.file : & ) \n*.jpg.out
tmp/parallel.file/sub dir.out
tmp/parallel.file1.out
tmp/parallel.file10.out tmp/parallel.file10.out
tmp/parallel.file13.out tmp/parallel.file13.out
tmp/parallel.file14.out tmp/parallel.file14.out
@ -122,9 +126,8 @@ tmp/parallel.file16.out
tmp/parallel.file17.out tmp/parallel.file17.out
tmp/parallel.file18.out tmp/parallel.file18.out
tmp/parallel.file19.out tmp/parallel.file19.out
tmp/parallel.file1.out
tmp/parallel.file20.out
tmp/parallel.file2.out tmp/parallel.file2.out
tmp/parallel.file20.out
tmp/parallel.file3.out tmp/parallel.file3.out
tmp/parallel.file5.out tmp/parallel.file5.out
tmp/parallel.file6.out tmp/parallel.file6.out
@ -132,9 +135,13 @@ tmp/parallel.file7.out
tmp/parallel.file8.out tmp/parallel.file8.out
tmp/parallel.file9.out tmp/parallel.file9.out
tmp/parallel.file>fire.out tmp/parallel.file>fire.out
tmp/parallel.file : & ) \n*.jpg.out
tmp/parallel.file/sub dir.out
### --return - multiple files ### --return - multiple files
tmp/parallel.file : & ) \n*.jpg.file.done
tmp/parallel.file : & ) \n*.jpg.out
tmp/parallel.file/sub dir.file.done
tmp/parallel.file/sub dir.out
tmp/parallel.file1.file.done
tmp/parallel.file1.out
tmp/parallel.file10.file.done tmp/parallel.file10.file.done
tmp/parallel.file10.out tmp/parallel.file10.out
tmp/parallel.file13.file.done tmp/parallel.file13.file.done
@ -151,12 +158,10 @@ tmp/parallel.file18.file.done
tmp/parallel.file18.out tmp/parallel.file18.out
tmp/parallel.file19.file.done tmp/parallel.file19.file.done
tmp/parallel.file19.out tmp/parallel.file19.out
tmp/parallel.file1.file.done
tmp/parallel.file1.out
tmp/parallel.file20.file.done
tmp/parallel.file20.out
tmp/parallel.file2.file.done tmp/parallel.file2.file.done
tmp/parallel.file2.out tmp/parallel.file2.out
tmp/parallel.file20.file.done
tmp/parallel.file20.out
tmp/parallel.file3.file.done tmp/parallel.file3.file.done
tmp/parallel.file3.out tmp/parallel.file3.out
tmp/parallel.file5.file.done tmp/parallel.file5.file.done
@ -171,11 +176,13 @@ tmp/parallel.file9.file.done
tmp/parallel.file9.out tmp/parallel.file9.out
tmp/parallel.file>fire.file.done tmp/parallel.file>fire.file.done
tmp/parallel.file>fire.out tmp/parallel.file>fire.out
tmp/parallel.file : & ) \n*.jpg.file.done
tmp/parallel.file : & ) \n*.jpg.out
tmp/parallel.file/sub dir.file.done
tmp/parallel.file/sub dir.out
### --return --cleanup - abspath ### --return --cleanup - abspath
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file10.file.done /tmp/parallel.file10.file.done
/tmp/parallel.file10.out /tmp/parallel.file10.out
/tmp/parallel.file13.file.done /tmp/parallel.file13.file.done
@ -192,12 +199,10 @@ tmp/parallel.file/sub dir.out
/tmp/parallel.file18.out /tmp/parallel.file18.out
/tmp/parallel.file19.file.done /tmp/parallel.file19.file.done
/tmp/parallel.file19.out /tmp/parallel.file19.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file2.file.done /tmp/parallel.file2.file.done
/tmp/parallel.file2.out /tmp/parallel.file2.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file3.file.done /tmp/parallel.file3.file.done
/tmp/parallel.file3.out /tmp/parallel.file3.out
/tmp/parallel.file5.file.done /tmp/parallel.file5.file.done
@ -212,14 +217,16 @@ tmp/parallel.file/sub dir.out
/tmp/parallel.file9.out /tmp/parallel.file9.out
/tmp/parallel.file>fire.file.done /tmp/parallel.file>fire.file.done
/tmp/parallel.file>fire.out /tmp/parallel.file>fire.out
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
good if no file good if no file
ls: cannot access /tmp/parallel.file*: No such file or directory ls: cannot access /tmp/parallel.file*: No such file or directory
OK OK
### --return --cleanup - relpath ### --return --cleanup - relpath
tmp/parallel.file : & ) \n*.jpg.file.done
tmp/parallel.file : & ) \n*.jpg.out
tmp/parallel.file/sub dir.file.done
tmp/parallel.file/sub dir.out
tmp/parallel.file1.file.done
tmp/parallel.file1.out
tmp/parallel.file10.file.done tmp/parallel.file10.file.done
tmp/parallel.file10.out tmp/parallel.file10.out
tmp/parallel.file13.file.done tmp/parallel.file13.file.done
@ -236,12 +243,10 @@ tmp/parallel.file18.file.done
tmp/parallel.file18.out tmp/parallel.file18.out
tmp/parallel.file19.file.done tmp/parallel.file19.file.done
tmp/parallel.file19.out tmp/parallel.file19.out
tmp/parallel.file1.file.done
tmp/parallel.file1.out
tmp/parallel.file20.file.done
tmp/parallel.file20.out
tmp/parallel.file2.file.done tmp/parallel.file2.file.done
tmp/parallel.file2.out tmp/parallel.file2.out
tmp/parallel.file20.file.done
tmp/parallel.file20.out
tmp/parallel.file3.file.done tmp/parallel.file3.file.done
tmp/parallel.file3.out tmp/parallel.file3.out
tmp/parallel.file5.file.done tmp/parallel.file5.file.done
@ -256,14 +261,16 @@ tmp/parallel.file9.file.done
tmp/parallel.file9.out tmp/parallel.file9.out
tmp/parallel.file>fire.file.done tmp/parallel.file>fire.file.done
tmp/parallel.file>fire.out tmp/parallel.file>fire.out
tmp/parallel.file : & ) \n*.jpg.file.done
tmp/parallel.file : & ) \n*.jpg.out
tmp/parallel.file/sub dir.file.done
tmp/parallel.file/sub dir.out
good if no file good if no file
ls: cannot access tmp/parallel.file*: No such file or directory ls: cannot access tmp/parallel.file*: No such file or directory
OK OK
### --return --cleanup - multiple returns ### --return --cleanup - multiple returns
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file10.file.done /tmp/parallel.file10.file.done
/tmp/parallel.file10.out /tmp/parallel.file10.out
/tmp/parallel.file13.file.done /tmp/parallel.file13.file.done
@ -280,12 +287,10 @@ OK
/tmp/parallel.file18.out /tmp/parallel.file18.out
/tmp/parallel.file19.file.done /tmp/parallel.file19.file.done
/tmp/parallel.file19.out /tmp/parallel.file19.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file2.file.done /tmp/parallel.file2.file.done
/tmp/parallel.file2.out /tmp/parallel.file2.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file3.file.done /tmp/parallel.file3.file.done
/tmp/parallel.file3.out /tmp/parallel.file3.out
/tmp/parallel.file5.file.done /tmp/parallel.file5.file.done
@ -300,14 +305,16 @@ OK
/tmp/parallel.file9.out /tmp/parallel.file9.out
/tmp/parallel.file>fire.file.done /tmp/parallel.file>fire.file.done
/tmp/parallel.file>fire.out /tmp/parallel.file>fire.out
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
good if no file good if no file
ls: cannot access tmp/parallel.file*: No such file or directory ls: cannot access tmp/parallel.file*: No such file or directory
OK OK
### --transfer --return --cleanup - abspath ### --transfer --return --cleanup - abspath
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file10.file.done /tmp/parallel.file10.file.done
/tmp/parallel.file10.out /tmp/parallel.file10.out
/tmp/parallel.file13.file.done /tmp/parallel.file13.file.done
@ -324,12 +331,10 @@ OK
/tmp/parallel.file18.out /tmp/parallel.file18.out
/tmp/parallel.file19.file.done /tmp/parallel.file19.file.done
/tmp/parallel.file19.out /tmp/parallel.file19.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file2.file.done /tmp/parallel.file2.file.done
/tmp/parallel.file2.out /tmp/parallel.file2.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file3.file.done /tmp/parallel.file3.file.done
/tmp/parallel.file3.out /tmp/parallel.file3.out
/tmp/parallel.file5.file.done /tmp/parallel.file5.file.done
@ -344,14 +349,16 @@ OK
/tmp/parallel.file9.out /tmp/parallel.file9.out
/tmp/parallel.file>fire.file.done /tmp/parallel.file>fire.file.done
/tmp/parallel.file>fire.out /tmp/parallel.file>fire.out
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
good if no file good if no file
ls: cannot access /tmp/parallel.file*: No such file or directory ls: cannot access /tmp/parallel.file*: No such file or directory
OK OK
### --transfer --return --cleanup - relpath ### --transfer --return --cleanup - relpath
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file10.file.done /tmp/parallel.file10.file.done
/tmp/parallel.file10.out /tmp/parallel.file10.out
/tmp/parallel.file13.file.done /tmp/parallel.file13.file.done
@ -368,12 +375,10 @@ OK
/tmp/parallel.file18.out /tmp/parallel.file18.out
/tmp/parallel.file19.file.done /tmp/parallel.file19.file.done
/tmp/parallel.file19.out /tmp/parallel.file19.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file2.file.done /tmp/parallel.file2.file.done
/tmp/parallel.file2.out /tmp/parallel.file2.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file3.file.done /tmp/parallel.file3.file.done
/tmp/parallel.file3.out /tmp/parallel.file3.out
/tmp/parallel.file5.file.done /tmp/parallel.file5.file.done
@ -388,14 +393,16 @@ OK
/tmp/parallel.file9.out /tmp/parallel.file9.out
/tmp/parallel.file>fire.file.done /tmp/parallel.file>fire.file.done
/tmp/parallel.file>fire.out /tmp/parallel.file>fire.out
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
good if no file good if no file
ls: cannot access tmp/parallel.file*: No such file or directory ls: cannot access tmp/parallel.file*: No such file or directory
OK OK
### --transfer --return --cleanup - multiple files ### --transfer --return --cleanup - multiple files
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file10.file.done /tmp/parallel.file10.file.done
/tmp/parallel.file10.out /tmp/parallel.file10.out
/tmp/parallel.file13.file.done /tmp/parallel.file13.file.done
@ -412,12 +419,10 @@ OK
/tmp/parallel.file18.out /tmp/parallel.file18.out
/tmp/parallel.file19.file.done /tmp/parallel.file19.file.done
/tmp/parallel.file19.out /tmp/parallel.file19.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file2.file.done /tmp/parallel.file2.file.done
/tmp/parallel.file2.out /tmp/parallel.file2.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file3.file.done /tmp/parallel.file3.file.done
/tmp/parallel.file3.out /tmp/parallel.file3.out
/tmp/parallel.file5.file.done /tmp/parallel.file5.file.done
@ -432,13 +437,15 @@ OK
/tmp/parallel.file9.out /tmp/parallel.file9.out
/tmp/parallel.file>fire.file.done /tmp/parallel.file>fire.file.done
/tmp/parallel.file>fire.out /tmp/parallel.file>fire.out
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
ls: cannot access tmp/parallel.file*: No such file or directory ls: cannot access tmp/parallel.file*: No such file or directory
OK OK
### --trc - abspath ### --trc - abspath
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file10.file.done /tmp/parallel.file10.file.done
/tmp/parallel.file10.out /tmp/parallel.file10.out
/tmp/parallel.file13.file.done /tmp/parallel.file13.file.done
@ -455,12 +462,10 @@ OK
/tmp/parallel.file18.out /tmp/parallel.file18.out
/tmp/parallel.file19.file.done /tmp/parallel.file19.file.done
/tmp/parallel.file19.out /tmp/parallel.file19.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file2.file.done /tmp/parallel.file2.file.done
/tmp/parallel.file2.out /tmp/parallel.file2.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file3.file.done /tmp/parallel.file3.file.done
/tmp/parallel.file3.out /tmp/parallel.file3.out
/tmp/parallel.file5.file.done /tmp/parallel.file5.file.done
@ -475,14 +480,16 @@ OK
/tmp/parallel.file9.out /tmp/parallel.file9.out
/tmp/parallel.file>fire.file.done /tmp/parallel.file>fire.file.done
/tmp/parallel.file>fire.out /tmp/parallel.file>fire.out
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
good if no file good if no file
ls: cannot access /tmp/parallel.file*: No such file or directory ls: cannot access /tmp/parallel.file*: No such file or directory
OK OK
### --trc - relpath ### --trc - relpath
tmp/parallel.file : & ) \n*.jpg.file.done
tmp/parallel.file : & ) \n*.jpg.out
tmp/parallel.file/sub dir.file.done
tmp/parallel.file/sub dir.out
tmp/parallel.file1.file.done
tmp/parallel.file1.out
tmp/parallel.file10.file.done tmp/parallel.file10.file.done
tmp/parallel.file10.out tmp/parallel.file10.out
tmp/parallel.file13.file.done tmp/parallel.file13.file.done
@ -499,12 +506,10 @@ tmp/parallel.file18.file.done
tmp/parallel.file18.out tmp/parallel.file18.out
tmp/parallel.file19.file.done tmp/parallel.file19.file.done
tmp/parallel.file19.out tmp/parallel.file19.out
tmp/parallel.file1.file.done
tmp/parallel.file1.out
tmp/parallel.file20.file.done
tmp/parallel.file20.out
tmp/parallel.file2.file.done tmp/parallel.file2.file.done
tmp/parallel.file2.out tmp/parallel.file2.out
tmp/parallel.file20.file.done
tmp/parallel.file20.out
tmp/parallel.file3.file.done tmp/parallel.file3.file.done
tmp/parallel.file3.out tmp/parallel.file3.out
tmp/parallel.file5.file.done tmp/parallel.file5.file.done
@ -519,14 +524,16 @@ tmp/parallel.file9.file.done
tmp/parallel.file9.out tmp/parallel.file9.out
tmp/parallel.file>fire.file.done tmp/parallel.file>fire.file.done
tmp/parallel.file>fire.out tmp/parallel.file>fire.out
tmp/parallel.file : & ) \n*.jpg.file.done
tmp/parallel.file : & ) \n*.jpg.out
tmp/parallel.file/sub dir.file.done
tmp/parallel.file/sub dir.out
good if no file good if no file
ls: cannot access tmp/parallel.file*: No such file or directory ls: cannot access tmp/parallel.file*: No such file or directory
OK OK
### --trc - multiple files ### --trc - multiple files
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file10.file.done /tmp/parallel.file10.file.done
/tmp/parallel.file10.out /tmp/parallel.file10.out
/tmp/parallel.file13.file.done /tmp/parallel.file13.file.done
@ -543,12 +550,10 @@ OK
/tmp/parallel.file18.out /tmp/parallel.file18.out
/tmp/parallel.file19.file.done /tmp/parallel.file19.file.done
/tmp/parallel.file19.out /tmp/parallel.file19.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file2.file.done /tmp/parallel.file2.file.done
/tmp/parallel.file2.out /tmp/parallel.file2.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file3.file.done /tmp/parallel.file3.file.done
/tmp/parallel.file3.out /tmp/parallel.file3.out
/tmp/parallel.file5.file.done /tmp/parallel.file5.file.done
@ -563,10 +568,6 @@ OK
/tmp/parallel.file9.out /tmp/parallel.file9.out
/tmp/parallel.file>fire.file.done /tmp/parallel.file>fire.file.done
/tmp/parallel.file>fire.out /tmp/parallel.file>fire.out
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
good if no file good if no file
ls: cannot access /tmp/parallel.file*: No such file or directory ls: cannot access /tmp/parallel.file*: No such file or directory
OK OK

View file

@ -75,7 +75,7 @@ OK
ls: cannot access tmp/parallel.file*: No such file or directory ls: cannot access tmp/parallel.file*: No such file or directory
OK OK
Input for ssh Input for ssh
parallel-server1 rsync --server -lDrRze.iLsf . ./ parallel-server1 rsync --server -lDErRze.iLsf . ./
parallel-server1 cat tmp/parallel.file.' parallel-server1 cat tmp/parallel.file.'
'newline2 > tmp/parallel.file.' 'newline2 > tmp/parallel.file.'
'newline2.out;cat tmp/parallel.file.' 'newline2.out;cat tmp/parallel.file.'
@ -87,7 +87,7 @@ parallel-server1 rsync --server --sender -lDrRze.iLsf --remove-source-files . ./
'newline2.out2 'newline2.out2
parallel-server1 rm -f tmp/parallel.file.' parallel-server1 rm -f tmp/parallel.file.'
'newline2 'newline2
-l parallel parallel-server2 rsync --server -lDrRze.iLsf . ./ -l parallel parallel-server2 rsync --server -lDErRze.iLsf . ./
parallel@parallel-server2 cat tmp/parallel.file.' parallel@parallel-server2 cat tmp/parallel.file.'
'newline1 > tmp/parallel.file.' 'newline1 > tmp/parallel.file.'
'newline1.out;cat tmp/parallel.file.' 'newline1.out;cat tmp/parallel.file.'

View file

@ -0,0 +1,36 @@
### Test --basefile + --cleanup + permissions
scriptrun 1
scriptrun 2
scriptrun 3
scriptrun 4
scriptrun 5
good if no file
ls: cannot access script: No such file or directory
OK
### Test --basefile + --sshlogin :
1
10
11
12
13
2
3
4
5
6
7
8
9
1
10
11
12
13
2
3
4
5
6
7
8
9

View file

@ -14,7 +14,7 @@ ls | $PAR echo ls | sort
ls | $PAR -j 1 echo ls | sort ls | $PAR -j 1 echo ls | sort
find -type f | $PAR diff {} a/foo ">"{}.diff | sort find -type f | $PAR diff {} a/foo ">"{}.diff | sort
ls | $PAR -vg "ls {}|wc;echo {}" | sort ls | $PAR -vg "ls {}|wc;echo {}" | sort
# Check that we can have more input than max procs (-j 0) echo '### Check that we can have more input than max procs (-j 0)'
perl -e 'print map {"more_than_5000-$_\n" } (4000..9999)' | $PAR -vj 0 touch | sort | tail perl -e 'print map {"more_than_5000-$_\n" } (4000..9999)' | $PAR -vj 0 touch | sort | tail
perl -e 'print map {"more_than_5000-$_\n" } (4000..9900)' | $PAR -j 0 rm | sort perl -e 'print map {"more_than_5000-$_\n" } (4000..9900)' | $PAR -j 0 rm | sort
ls | $PAR -j500 'sleep 1; ls {} | perl -ne "END{print $..\" {}\n\"}"' | sort ls | $PAR -j500 'sleep 1; ls {} | perl -ne "END{print $..\" {}\n\"}"' | sort

View file

@ -1,23 +1,21 @@
#!/bin/bash #!/bin/bash
PAR=parallel echo '### This causes problems if we kill child processes'
seq 1 40 | parallel -j 0 seq 1 10 | sort |md5sum
seq 1 40 | parallel -j 0 seq 1 10 '| parallel -j 3 echo' | sort |md5sum
# This causes problems if we kill child processes echo '### Test of xargs -m and -X'
seq 1 40 | $PAR -j 0 seq 1 10 | sort |md5sum seq 1 60000 | parallel -m echo | mop -d 4 "|sort |md5sum" "| wc"
seq 1 40 | $PAR -j 0 seq 1 10 '| '$PAR' -j 3 echo' | sort |md5sum (echo foo;echo bar) | parallel -m echo 1{}2{}3 A{}B{}C
(echo foo;echo bar) | parallel -X echo 1{}2{}3 A{}B{}C
# Test of xargs seq 1 60000 | parallel -m echo a{}b{}c | mop -d 4 "|sort |md5sum" "| wc"
seq 1 60000 | $PAR -m echo | mop -d 4 "|sort |md5sum" "| wc" seq 1 60000 | parallel -m echo a{}b{}c | \
(echo foo;echo bar) | $PAR -m echo 1{}2{}3 A{}B{}C
(echo foo;echo bar) | $PAR -X echo 1{}2{}3 A{}B{}C
seq 1 60000 | $PAR -m echo a{}b{}c | mop -d 4 "|sort |md5sum" "| wc"
seq 1 60000 | $PAR -m echo a{}b{}c | \
mop -q "|sort |md5sum" :par mop -q "|sort |md5sum" :par
echo -n "Chars per line: " echo -n "Chars per line: "
CHAR=$(cat ~/.mop/:par | wc -c) CHAR=$(cat ~/.mop/:par | wc -c)
LINES=$(cat ~/.mop/:par | wc -l) LINES=$(cat ~/.mop/:par | wc -l)
echo "$CHAR/$LINES" | bc echo "$CHAR/$LINES" | bc
# Bug before 2009-08-26 causing regexp compile error or infinite loop echo '### Bug before 2009-08-26 causing regexp compile error or infinite loop'
echo a | $PAR -qX echo "'"{}"' " echo a | parallel -qX echo "'"{}"' "
echo a | $PAR -qX echo "'{}'" echo a | parallel -qX echo "'{}'"

View file

@ -4,23 +4,58 @@
PAR=parallel PAR=parallel
# Test -a and --arg-file: Read input from file instead of stdin echo '### Test -L -l and --max-lines'
(echo a_b;echo c) | parallel -km -L2 echo
(echo a_b;echo c) | xargs -L2 echo
(echo a_b;echo c) | parallel -km -L1 echo
(echo a_b;echo c) | xargs -L1 echo
(echo a_b' ';echo c;echo d) | parallel -km -L1 echo
(echo a_b' ';echo c;echo d) | xargs -L1 echo
(echo a_b' ';echo c;echo d;echo e) | parallel -km -L2 echo
(echo a_b' ';echo c;echo d;echo e) | xargs -L2 echo
(echo a_b' ';echo c;echo d;echo e) | parallel -km -l echo
(echo a_b' ';echo c;echo d;echo e) | xargs -l echo
(echo a_b' ';echo c;echo d;echo e) | parallel -km -l2 echo
(echo a_b' ';echo c;echo d;echo e) | xargs -l2 echo
(echo a_b' ';echo c;echo d;echo e) | parallel -km -l1 echo
(echo a_b' ';echo c;echo d;echo e) | xargs -l1 echo
(echo a_b' ';echo c;echo d;echo e) | parallel -km --max-lines 2 echo
(echo a_b' ';echo c;echo d;echo e) | xargs --max-lines=2 echo
(echo a_b' ';echo c;echo d;echo e) | parallel -km --max-lines echo
(echo a_b' ';echo c;echo d;echo e) | xargs --max-lines echo
echo '### test too long args'
perl -e 'print "z"x1000000' | parallel echo 2>&1
perl -e 'print "z"x1000000' | xargs echo 2>&1
(seq 1 10; perl -e 'print "z"x1000000'; seq 12 15) | stdout parallel -j1 -km -s 10 echo
(seq 1 10; perl -e 'print "z"x1000000'; seq 12 15) | stdout xargs -s 10 echo
echo '### Test -x'
(seq 1 10; echo 12345; seq 12 15) | stdout parallel -j1 -km -s 10 -x echo
(seq 1 10; echo 12345; seq 12 15) | stdout xargs -s 10 -x echo
(seq 1 10; echo 1234; seq 12 15) | stdout parallel -j1 -km -s 10 -x echo
(seq 1 10; echo 1234; seq 12 15) | stdout xargs -s 10 -x echo
echo '### Test bugfix if no command given'
(echo echo; seq 1 5; perl -e 'print "z"x1000000'; seq 12 15) | stdout parallel -j1 -km -s 10
echo '### Test -a and --arg-file: Read input from file instead of stdin'
seq 1 10 >/tmp/$$ seq 1 10 >/tmp/$$
$PAR -k -a /tmp/$$ echo $PAR -k -a /tmp/$$ echo
$PAR -k --arg-file /tmp/$$ echo $PAR -k --arg-file /tmp/$$ echo
cd input-files/test15 cd input-files/test15
# echo 3 | xargs -P 2 -n 1 -a files cat - echo 'xargs Expect: 3 1 2'
echo 3 | xargs -P 1 -n 1 -a files cat -
echo 'parallel Expect: 3 1 2'
echo 3 | parallel -k -P 2 -n 1 -a files cat - echo 3 | parallel -k -P 2 -n 1 -a files cat -
# echo 3 | xargs -I {} -P 2 -n 1 -a files cat {} - echo 'xargs Expect: 1 3 2'
# Should give: echo 3 | xargs -I {} -P 1 -n 1 -a files cat {} -
# 3 echo 'parallel Expect: 1 3 2'
# 1 echo 3 | parallel -k -I {} -P 1 -n 1 -a files cat {} -
# 2
echo 3 | parallel -k -I {} -P 2 -n 1 -a files cat {} -
# Test -i and --replace: Replace with argument echo '### Test -i and --replace: Replace with argument'
(echo a; echo END; echo b) | $PAR -k -i -eEND echo repl{}ce (echo a; echo END; echo b) | $PAR -k -i -eEND echo repl{}ce
(echo a; echo END; echo b) | $PAR -k --replace -eEND echo repl{}ce (echo a; echo END; echo b) | $PAR -k --replace -eEND echo repl{}ce
(echo a; echo END; echo b) | $PAR -k -i+ -eEND echo repl+ce (echo a; echo END; echo b) | $PAR -k -i+ -eEND echo repl+ce
@ -31,17 +66,17 @@ echo 3 | parallel -k -I {} -P 2 -n 1 -a files cat {} -
(echo a; echo END; echo b) | $PAR -k --replace=^ -eEND echo repl^ce (echo a; echo END; echo b) | $PAR -k --replace=^ -eEND echo repl^ce
(echo a; echo END; echo b) | $PAR -k -I^ -eEND echo repl^ce (echo a; echo END; echo b) | $PAR -k -I^ -eEND echo repl^ce
# Test -E: Artificial end-of-file echo '### Test -E: Artificial end-of-file'
(echo include this; echo END; echo not this) | $PAR -k -E END echo (echo include this; echo END; echo not this) | $PAR -k -E END echo
(echo include this; echo END; echo not this) | $PAR -k -EEND echo (echo include this; echo END; echo not this) | $PAR -k -EEND echo
# Test -e and --eof: Artificial end-of-file echo '### Test -e and --eof: Artificial end-of-file'
(echo include this; echo END; echo not this) | $PAR -k -e END echo (echo include this; echo END; echo not this) | $PAR -k -e END echo
(echo include this; echo END; echo not this) | $PAR -k -eEND echo (echo include this; echo END; echo not this) | $PAR -k -eEND echo
(echo include this; echo END; echo not this) | $PAR -k --eof=END echo (echo include this; echo END; echo not this) | $PAR -k --eof=END echo
(echo include this; echo END; echo not this) | $PAR -k --eof END echo (echo include this; echo END; echo not this) | $PAR -k --eof END echo
# Test -n and --max-args: Max number of args per line (only with -X and -m) echo '### Test -n and --max-args: Max number of args per line (only with -X and -m)'
(echo line 1;echo line 2;echo line 3) | $PAR -k -n1 -m echo (echo line 1;echo line 2;echo line 3) | $PAR -k -n1 -m echo
(echo line 1;echo line 1;echo line 2) | $PAR -k -n2 -m echo (echo line 1;echo line 1;echo line 2) | $PAR -k -n2 -m echo
(echo line 1;echo line 2;echo line 3) | $PAR -k -n1 -X echo (echo line 1;echo line 2;echo line 3) | $PAR -k -n1 -X echo
@ -51,37 +86,37 @@ echo 3 | parallel -k -I {} -P 2 -n 1 -a files cat {} -
(echo line 1;echo line 1;echo line 2) | $PAR -k --max-args=2 -X echo (echo line 1;echo line 1;echo line 2) | $PAR -k --max-args=2 -X echo
(echo line 1;echo line 1;echo line 2) | $PAR -k --max-args 2 -X echo (echo line 1;echo line 1;echo line 2) | $PAR -k --max-args 2 -X echo
# Test --max-procs and -P: Number of processes echo '### Test --max-procs and -P: Number of processes'
seq 1 10 | $PAR -k --max-procs +0 echo max proc seq 1 10 | $PAR -k --max-procs +0 echo max proc
seq 1 10 | $PAR -k -P 200% echo 200% proc seq 1 10 | $PAR -k -P 200% echo 200% proc
# Test --delimiter and -d: Delimiter instead of newline echo '### Test --delimiter and -d: Delimiter instead of newline'
# Yes there is supposed to be an extra newline for -d N echo '# Yes there is supposed to be an extra newline for -d N'
echo line 1Nline 2Nline 3 | $PAR -k -d N echo This is echo line 1Nline 2Nline 3 | $PAR -k -d N echo This is
echo line 1Nline 2Nline 3 | $PAR -k --delimiter N echo This is echo line 1Nline 2Nline 3 | $PAR -k --delimiter N echo This is
printf "delimiter NUL line 1\0line 2\0line 3" | $PAR -k -d '\0' echo printf "delimiter NUL line 1\0line 2\0line 3" | $PAR -k -d '\0' echo
printf "delimiter TAB line 1\tline 2\tline 3" | $PAR -k --delimiter '\t' echo printf "delimiter TAB line 1\tline 2\tline 3" | $PAR -k --delimiter '\t' echo
# Test --max-chars and -s: Max number of chars in a line echo '### Test --max-chars and -s: Max number of chars in a line'
(echo line 1;echo line 1;echo line 2) | $PAR -k --max-chars 25 -X echo (echo line 1;echo line 1;echo line 2) | $PAR -k --max-chars 25 -X echo
(echo line 1;echo line 1;echo line 2) | $PAR -k -s 25 -X echo (echo line 1;echo line 1;echo line 2) | $PAR -k -s 25 -X echo
# Test --no-run-if-empty and -r: This should give no output echo '### Test --no-run-if-empty and -r: This should give no output'
echo " " | $PAR -r echo echo " " | $PAR -r echo
echo " " | $PAR --no-run-if-empty echo echo " " | $PAR --no-run-if-empty echo
# Test --help and -h: Help output (just check we get the same amount of lines) echo '### Test --help and -h: Help output (just check we get the same amount of lines)'
echo Output from -h and --help echo Output from -h and --help
$PAR -h | wc -l $PAR -h | wc -l
$PAR --help | wc -l $PAR --help | wc -l
# Test --version: Version output (just check we get the same amount of lines) echo '### Test --version: Version output (just check we get the same amount of lines)'
$PAR --version | wc -l $PAR --version | wc -l
# Test --verbose and -t echo '### Test --verbose and -t'
(echo b; echo c; echo f) | $PAR -k -t echo {}ar 2>&1 >/dev/null (echo b; echo c; echo f) | $PAR -k -t echo {}ar 2>&1 >/dev/null
(echo b; echo c; echo f) | $PAR -k --verbose echo {}ar 2>&1 >/dev/null (echo b; echo c; echo f) | $PAR -k --verbose echo {}ar 2>&1 >/dev/null
# Test --show-limits echo '### Test --show-limits'
(echo b; echo c; echo f) | $PAR -k --show-limits echo {}ar (echo b; echo c; echo f) | $PAR -k --show-limits echo {}ar
(echo b; echo c; echo f) | $PAR -k --show-limits -s 100 echo {}ar (echo b; echo c; echo f) | $PAR -k --show-limits -s 100 echo {}ar

View file

@ -5,13 +5,16 @@ PAR=parallel
SERVER1=parallel-server1 SERVER1=parallel-server1
SERVER2=parallel-server2 SERVER2=parallel-server2
# Make sure sort order is the same
export LANG=C
echo '### Test --transfer --return --cleanup' echo '### Test --transfer --return --cleanup'
rm -rf /tmp/parallel.file* rm -rf /tmp/parallel.file*
stdout ssh $SERVER1 rm -rf 'tmp/parallel.file*' '/tmp/parallel.file*' stdout ssh $SERVER1 rm -rf 'tmp/parallel.file*' '/tmp/parallel.file*'
stdout ssh parallel@$SERVER2 rm -rf 'tmp/parallel.file*' '/tmp/parallel.file*' stdout ssh parallel@$SERVER2 rm -rf 'tmp/parallel.file*' '/tmp/parallel.file*'
(seq 1 3;echo '>fire';seq 5 10; echo ' : & ) \n*.jpg'; echo '/./sub dir'; seq 13 20) >/tmp/test17 (seq 1 3;echo '>fire';seq 5 10; echo ' : & ) \n*.jpg'; echo '/./sub dir'; seq 13 20) >/tmp/test17
# Create some weirdly files in /tmp echo '# Create some weirdly files in /tmp'
mkdir -p /tmp/parallel.file mkdir -p /tmp/parallel.file
cat /tmp/test17 | $PAR -k echo file{} '>'/tmp/parallel.file{}.file cat /tmp/test17 | $PAR -k echo file{} '>'/tmp/parallel.file{}.file
cat /tmp/test17 | $PAR -k echo /tmp/parallel.file{}.file >/tmp/test17abs cat /tmp/test17 | $PAR -k echo /tmp/parallel.file{}.file >/tmp/test17abs

View file

@ -0,0 +1,25 @@
#!/bin/bash
PAR=parallel
SERVER1=parallel-server1
SERVER2=parallel-server2
cd /tmp
(
echo '### Test --basefile + --cleanup + permissions'
echo echo scriptrun '"$@"' > script
chmod 755 script
seq 1 5 | parallel -kS $SERVER1 --cleanup -B script ./script
echo good if no file
stdout ssh $SERVER1 ls 'script' || echo OK
echo '### Test --basefile + --sshlogin :'
echo cat '"$@"' > my_script
chmod 755 my_script
rm -f parallel_*.test parallel_*.out
seq 1 13 | parallel echo {} '>' parallel_{}.test
ls parallel_*.test | parallel -j+0 --trc {.}.out -B my_script \
-S parallel-server1,parallel@parallel-server2,: "./my_script {} > {.}.out"
cat parallel_*.test parallel_*.out
)

View file

@ -18,12 +18,12 @@ ls b
1 1 10 1 1 10
1 1 15 1 1 15
1 1 15 1 1 15
4 4 26
6 6 41
1-col.txt 1-col.txt
1-col.txt.diff 1-col.txt.diff
2-col.txt 2-col.txt
2-col.txt.diff 2-col.txt.diff
4 4 26
6 6 41
a a
b b
ls 1-col.txt.diff|wc;echo 1-col.txt.diff ls 1-col.txt.diff|wc;echo 1-col.txt.diff
@ -32,6 +32,7 @@ ls 2-col.txt.diff|wc;echo 2-col.txt.diff
ls 2-col.txt|wc;echo 2-col.txt ls 2-col.txt|wc;echo 2-col.txt
ls a|wc;echo a ls a|wc;echo a
ls b|wc;echo b ls b|wc;echo b
### Check that we can have more input than max procs (-j 0)
touch more_than_5000-9990 touch more_than_5000-9990
touch more_than_5000-9991 touch more_than_5000-9991
touch more_than_5000-9992 touch more_than_5000-9992

View file

@ -1,5 +1,7 @@
### This causes problems if we kill child processes
33bf8b2986551515cdaff5e860618098 - 33bf8b2986551515cdaff5e860618098 -
960bedf6398273937e314fb49c7b3ffa - d7fb96d6a56d4347bc24930a395c431a -
### Test of xargs -m and -X
b35d8e49be8d94899b719c40d3f1f4bb - b35d8e49be8d94899b719c40d3f1f4bb -
3 60000 348894 3 60000 348894
1foo bar2foo bar3 Afoo barBfoo barC 1foo bar2foo bar3 Afoo barBfoo barC
@ -8,5 +10,6 @@ b35d8e49be8d94899b719c40d3f1f4bb -
6 119994 697800 6 119994 697800
31d9274be5fdc2de59487cb05ba57776 - 31d9274be5fdc2de59487cb05ba57776 -
Chars per line: 116300 Chars per line: 116300
### Bug before 2009-08-26 causing regexp compile error or infinite loop
'a' 'a'
'a' 'a'

View file

@ -1,3 +1,94 @@
### Test -L -l and --max-lines
a_b c
a_b c
a_b
c
a_b
c
a_b c
d
a_b c
d
a_b c d
e
a_b c d
e
a_b c
d
e
a_b c
d
e
a_b c d
e
a_b c d
e
a_b c
d
e
a_b c
d
e
a_b c d
e
a_b c d
e
a_b c
d
e
a_b c
d
e
### test too long args
Command line too long (1000005 >= 131071) at number 1: zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz...
xargs: argument line too long
1 2
3 4
5 6
7 8
Command line too long (1000007 >= 10) at number 1: zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz...
9 10
1 2
3 4
5 6
7 8
xargs: argument line too long
9 10
### Test -x
1 2
3 4
5 6
Command line too long (15 >= 10) at number 3: 12345...
7 8
1 2
3 4
5 6
xargs: argument line too long
7 8
1 2
3 4
5 6
7 8
9 10
1234
12
13
14
15
1 2
3 4
5 6
7 8
9 10
1234
12
13
14
15
### Test bugfix if no command given
Command line too long (1000002 >= 10) at number 1: zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz...
1 2 3 4 5
### Test -a and --arg-file: Read input from file instead of stdin
1 1
2 2
3 3
@ -18,12 +109,23 @@
8 8
9 9
10 10
xargs Expect: 3 1 2
3 3
1 1
2 2
parallel Expect: 3 1 2
3
1
2
xargs Expect: 1 3 2
1 1
3 3
2 2
parallel Expect: 1 3 2
1
3
2
### Test -i and --replace: Replace with argument
replace replace
replace replace
replace replace
@ -33,12 +135,15 @@ replace
replace replace
replace replace
replace replace
### Test -E: Artificial end-of-file
include this
include this
### Test -e and --eof: Artificial end-of-file
include this include this
include this include this
include this include this
include this include this
include this ### Test -n and --max-args: Max number of args per line (only with -X and -m)
include this
line 1 line 1
line 2 line 2
line 3 line 3
@ -59,6 +164,7 @@ line 1 line 1
line 2 line 2
line 1 line 1 line 1 line 1
line 2 line 2
### Test --max-procs and -P: Number of processes
max proc 1 max proc 1
max proc 2 max proc 2
max proc 3 max proc 3
@ -79,6 +185,8 @@ max proc 10
200% proc 8 200% proc 8
200% proc 9 200% proc 9
200% proc 10 200% proc 10
### Test --delimiter and -d: Delimiter instead of newline
# Yes there is supposed to be an extra newline for -d N
This is line 1 This is line 1
This is line 2 This is line 2
This is line 3 This is line 3
@ -93,20 +201,26 @@ line 3
delimiter TAB line 1 delimiter TAB line 1
line 2 line 2
line 3 line 3
### Test --max-chars and -s: Max number of chars in a line
line 1 line 1 line 1 line 1
line 2 line 2
line 1 line 1 line 1 line 1
line 2 line 2
### Test --no-run-if-empty and -r: This should give no output
### Test --help and -h: Help output (just check we get the same amount of lines)
Output from -h and --help Output from -h and --help
4 4
4 4
### Test --version: Version output (just check we get the same amount of lines)
7 7
### Test --verbose and -t
echo bar echo bar
echo car echo car
echo far echo far
echo bar echo bar
echo car echo car
echo far echo far
### Test --show-limits
Maximal size of command: 131071 Maximal size of command: 131071
Maximal used size of command: 131071 Maximal used size of command: 131071

View file

@ -1,4 +1,5 @@
### Test --transfer --return --cleanup ### Test --transfer --return --cleanup
# Create some weirdly files in /tmp
### --transfer - abspath ### --transfer - abspath
file1 file1
file2 file2
@ -93,6 +94,9 @@ good if no file
ls: cannot access tmp/parallel.file*: No such file or directory ls: cannot access tmp/parallel.file*: No such file or directory
OK OK
### --return - abspath ### --return - abspath
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.out
/tmp/parallel.file1.out
/tmp/parallel.file10.out /tmp/parallel.file10.out
/tmp/parallel.file13.out /tmp/parallel.file13.out
/tmp/parallel.file14.out /tmp/parallel.file14.out
@ -101,9 +105,8 @@ OK
/tmp/parallel.file17.out /tmp/parallel.file17.out
/tmp/parallel.file18.out /tmp/parallel.file18.out
/tmp/parallel.file19.out /tmp/parallel.file19.out
/tmp/parallel.file1.out
/tmp/parallel.file20.out
/tmp/parallel.file2.out /tmp/parallel.file2.out
/tmp/parallel.file20.out
/tmp/parallel.file3.out /tmp/parallel.file3.out
/tmp/parallel.file5.out /tmp/parallel.file5.out
/tmp/parallel.file6.out /tmp/parallel.file6.out
@ -111,9 +114,10 @@ OK
/tmp/parallel.file8.out /tmp/parallel.file8.out
/tmp/parallel.file9.out /tmp/parallel.file9.out
/tmp/parallel.file>fire.out /tmp/parallel.file>fire.out
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.out
### --return - relpath ### --return - relpath
tmp/parallel.file : & ) \n*.jpg.out
tmp/parallel.file/sub dir.out
tmp/parallel.file1.out
tmp/parallel.file10.out tmp/parallel.file10.out
tmp/parallel.file13.out tmp/parallel.file13.out
tmp/parallel.file14.out tmp/parallel.file14.out
@ -122,9 +126,8 @@ tmp/parallel.file16.out
tmp/parallel.file17.out tmp/parallel.file17.out
tmp/parallel.file18.out tmp/parallel.file18.out
tmp/parallel.file19.out tmp/parallel.file19.out
tmp/parallel.file1.out
tmp/parallel.file20.out
tmp/parallel.file2.out tmp/parallel.file2.out
tmp/parallel.file20.out
tmp/parallel.file3.out tmp/parallel.file3.out
tmp/parallel.file5.out tmp/parallel.file5.out
tmp/parallel.file6.out tmp/parallel.file6.out
@ -132,9 +135,13 @@ tmp/parallel.file7.out
tmp/parallel.file8.out tmp/parallel.file8.out
tmp/parallel.file9.out tmp/parallel.file9.out
tmp/parallel.file>fire.out tmp/parallel.file>fire.out
tmp/parallel.file : & ) \n*.jpg.out
tmp/parallel.file/sub dir.out
### --return - multiple files ### --return - multiple files
tmp/parallel.file : & ) \n*.jpg.file.done
tmp/parallel.file : & ) \n*.jpg.out
tmp/parallel.file/sub dir.file.done
tmp/parallel.file/sub dir.out
tmp/parallel.file1.file.done
tmp/parallel.file1.out
tmp/parallel.file10.file.done tmp/parallel.file10.file.done
tmp/parallel.file10.out tmp/parallel.file10.out
tmp/parallel.file13.file.done tmp/parallel.file13.file.done
@ -151,12 +158,10 @@ tmp/parallel.file18.file.done
tmp/parallel.file18.out tmp/parallel.file18.out
tmp/parallel.file19.file.done tmp/parallel.file19.file.done
tmp/parallel.file19.out tmp/parallel.file19.out
tmp/parallel.file1.file.done
tmp/parallel.file1.out
tmp/parallel.file20.file.done
tmp/parallel.file20.out
tmp/parallel.file2.file.done tmp/parallel.file2.file.done
tmp/parallel.file2.out tmp/parallel.file2.out
tmp/parallel.file20.file.done
tmp/parallel.file20.out
tmp/parallel.file3.file.done tmp/parallel.file3.file.done
tmp/parallel.file3.out tmp/parallel.file3.out
tmp/parallel.file5.file.done tmp/parallel.file5.file.done
@ -171,11 +176,13 @@ tmp/parallel.file9.file.done
tmp/parallel.file9.out tmp/parallel.file9.out
tmp/parallel.file>fire.file.done tmp/parallel.file>fire.file.done
tmp/parallel.file>fire.out tmp/parallel.file>fire.out
tmp/parallel.file : & ) \n*.jpg.file.done
tmp/parallel.file : & ) \n*.jpg.out
tmp/parallel.file/sub dir.file.done
tmp/parallel.file/sub dir.out
### --return --cleanup - abspath ### --return --cleanup - abspath
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file10.file.done /tmp/parallel.file10.file.done
/tmp/parallel.file10.out /tmp/parallel.file10.out
/tmp/parallel.file13.file.done /tmp/parallel.file13.file.done
@ -192,12 +199,10 @@ tmp/parallel.file/sub dir.out
/tmp/parallel.file18.out /tmp/parallel.file18.out
/tmp/parallel.file19.file.done /tmp/parallel.file19.file.done
/tmp/parallel.file19.out /tmp/parallel.file19.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file2.file.done /tmp/parallel.file2.file.done
/tmp/parallel.file2.out /tmp/parallel.file2.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file3.file.done /tmp/parallel.file3.file.done
/tmp/parallel.file3.out /tmp/parallel.file3.out
/tmp/parallel.file5.file.done /tmp/parallel.file5.file.done
@ -212,14 +217,16 @@ tmp/parallel.file/sub dir.out
/tmp/parallel.file9.out /tmp/parallel.file9.out
/tmp/parallel.file>fire.file.done /tmp/parallel.file>fire.file.done
/tmp/parallel.file>fire.out /tmp/parallel.file>fire.out
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
good if no file good if no file
ls: cannot access /tmp/parallel.file*: No such file or directory ls: cannot access /tmp/parallel.file*: No such file or directory
OK OK
### --return --cleanup - relpath ### --return --cleanup - relpath
tmp/parallel.file : & ) \n*.jpg.file.done
tmp/parallel.file : & ) \n*.jpg.out
tmp/parallel.file/sub dir.file.done
tmp/parallel.file/sub dir.out
tmp/parallel.file1.file.done
tmp/parallel.file1.out
tmp/parallel.file10.file.done tmp/parallel.file10.file.done
tmp/parallel.file10.out tmp/parallel.file10.out
tmp/parallel.file13.file.done tmp/parallel.file13.file.done
@ -236,12 +243,10 @@ tmp/parallel.file18.file.done
tmp/parallel.file18.out tmp/parallel.file18.out
tmp/parallel.file19.file.done tmp/parallel.file19.file.done
tmp/parallel.file19.out tmp/parallel.file19.out
tmp/parallel.file1.file.done
tmp/parallel.file1.out
tmp/parallel.file20.file.done
tmp/parallel.file20.out
tmp/parallel.file2.file.done tmp/parallel.file2.file.done
tmp/parallel.file2.out tmp/parallel.file2.out
tmp/parallel.file20.file.done
tmp/parallel.file20.out
tmp/parallel.file3.file.done tmp/parallel.file3.file.done
tmp/parallel.file3.out tmp/parallel.file3.out
tmp/parallel.file5.file.done tmp/parallel.file5.file.done
@ -256,14 +261,16 @@ tmp/parallel.file9.file.done
tmp/parallel.file9.out tmp/parallel.file9.out
tmp/parallel.file>fire.file.done tmp/parallel.file>fire.file.done
tmp/parallel.file>fire.out tmp/parallel.file>fire.out
tmp/parallel.file : & ) \n*.jpg.file.done
tmp/parallel.file : & ) \n*.jpg.out
tmp/parallel.file/sub dir.file.done
tmp/parallel.file/sub dir.out
good if no file good if no file
ls: cannot access tmp/parallel.file*: No such file or directory ls: cannot access tmp/parallel.file*: No such file or directory
OK OK
### --return --cleanup - multiple returns ### --return --cleanup - multiple returns
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file10.file.done /tmp/parallel.file10.file.done
/tmp/parallel.file10.out /tmp/parallel.file10.out
/tmp/parallel.file13.file.done /tmp/parallel.file13.file.done
@ -280,12 +287,10 @@ OK
/tmp/parallel.file18.out /tmp/parallel.file18.out
/tmp/parallel.file19.file.done /tmp/parallel.file19.file.done
/tmp/parallel.file19.out /tmp/parallel.file19.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file2.file.done /tmp/parallel.file2.file.done
/tmp/parallel.file2.out /tmp/parallel.file2.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file3.file.done /tmp/parallel.file3.file.done
/tmp/parallel.file3.out /tmp/parallel.file3.out
/tmp/parallel.file5.file.done /tmp/parallel.file5.file.done
@ -300,14 +305,16 @@ OK
/tmp/parallel.file9.out /tmp/parallel.file9.out
/tmp/parallel.file>fire.file.done /tmp/parallel.file>fire.file.done
/tmp/parallel.file>fire.out /tmp/parallel.file>fire.out
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
good if no file good if no file
ls: cannot access tmp/parallel.file*: No such file or directory ls: cannot access tmp/parallel.file*: No such file or directory
OK OK
### --transfer --return --cleanup - abspath ### --transfer --return --cleanup - abspath
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file10.file.done /tmp/parallel.file10.file.done
/tmp/parallel.file10.out /tmp/parallel.file10.out
/tmp/parallel.file13.file.done /tmp/parallel.file13.file.done
@ -324,12 +331,10 @@ OK
/tmp/parallel.file18.out /tmp/parallel.file18.out
/tmp/parallel.file19.file.done /tmp/parallel.file19.file.done
/tmp/parallel.file19.out /tmp/parallel.file19.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file2.file.done /tmp/parallel.file2.file.done
/tmp/parallel.file2.out /tmp/parallel.file2.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file3.file.done /tmp/parallel.file3.file.done
/tmp/parallel.file3.out /tmp/parallel.file3.out
/tmp/parallel.file5.file.done /tmp/parallel.file5.file.done
@ -344,14 +349,16 @@ OK
/tmp/parallel.file9.out /tmp/parallel.file9.out
/tmp/parallel.file>fire.file.done /tmp/parallel.file>fire.file.done
/tmp/parallel.file>fire.out /tmp/parallel.file>fire.out
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
good if no file good if no file
ls: cannot access /tmp/parallel.file*: No such file or directory ls: cannot access /tmp/parallel.file*: No such file or directory
OK OK
### --transfer --return --cleanup - relpath ### --transfer --return --cleanup - relpath
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file10.file.done /tmp/parallel.file10.file.done
/tmp/parallel.file10.out /tmp/parallel.file10.out
/tmp/parallel.file13.file.done /tmp/parallel.file13.file.done
@ -368,12 +375,10 @@ OK
/tmp/parallel.file18.out /tmp/parallel.file18.out
/tmp/parallel.file19.file.done /tmp/parallel.file19.file.done
/tmp/parallel.file19.out /tmp/parallel.file19.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file2.file.done /tmp/parallel.file2.file.done
/tmp/parallel.file2.out /tmp/parallel.file2.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file3.file.done /tmp/parallel.file3.file.done
/tmp/parallel.file3.out /tmp/parallel.file3.out
/tmp/parallel.file5.file.done /tmp/parallel.file5.file.done
@ -388,14 +393,16 @@ OK
/tmp/parallel.file9.out /tmp/parallel.file9.out
/tmp/parallel.file>fire.file.done /tmp/parallel.file>fire.file.done
/tmp/parallel.file>fire.out /tmp/parallel.file>fire.out
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
good if no file good if no file
ls: cannot access tmp/parallel.file*: No such file or directory ls: cannot access tmp/parallel.file*: No such file or directory
OK OK
### --transfer --return --cleanup - multiple files ### --transfer --return --cleanup - multiple files
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file10.file.done /tmp/parallel.file10.file.done
/tmp/parallel.file10.out /tmp/parallel.file10.out
/tmp/parallel.file13.file.done /tmp/parallel.file13.file.done
@ -412,12 +419,10 @@ OK
/tmp/parallel.file18.out /tmp/parallel.file18.out
/tmp/parallel.file19.file.done /tmp/parallel.file19.file.done
/tmp/parallel.file19.out /tmp/parallel.file19.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file2.file.done /tmp/parallel.file2.file.done
/tmp/parallel.file2.out /tmp/parallel.file2.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file3.file.done /tmp/parallel.file3.file.done
/tmp/parallel.file3.out /tmp/parallel.file3.out
/tmp/parallel.file5.file.done /tmp/parallel.file5.file.done
@ -432,13 +437,15 @@ OK
/tmp/parallel.file9.out /tmp/parallel.file9.out
/tmp/parallel.file>fire.file.done /tmp/parallel.file>fire.file.done
/tmp/parallel.file>fire.out /tmp/parallel.file>fire.out
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
ls: cannot access tmp/parallel.file*: No such file or directory ls: cannot access tmp/parallel.file*: No such file or directory
OK OK
### --trc - abspath ### --trc - abspath
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file10.file.done /tmp/parallel.file10.file.done
/tmp/parallel.file10.out /tmp/parallel.file10.out
/tmp/parallel.file13.file.done /tmp/parallel.file13.file.done
@ -455,12 +462,10 @@ OK
/tmp/parallel.file18.out /tmp/parallel.file18.out
/tmp/parallel.file19.file.done /tmp/parallel.file19.file.done
/tmp/parallel.file19.out /tmp/parallel.file19.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file2.file.done /tmp/parallel.file2.file.done
/tmp/parallel.file2.out /tmp/parallel.file2.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file3.file.done /tmp/parallel.file3.file.done
/tmp/parallel.file3.out /tmp/parallel.file3.out
/tmp/parallel.file5.file.done /tmp/parallel.file5.file.done
@ -475,14 +480,16 @@ OK
/tmp/parallel.file9.out /tmp/parallel.file9.out
/tmp/parallel.file>fire.file.done /tmp/parallel.file>fire.file.done
/tmp/parallel.file>fire.out /tmp/parallel.file>fire.out
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
good if no file good if no file
ls: cannot access /tmp/parallel.file*: No such file or directory ls: cannot access /tmp/parallel.file*: No such file or directory
OK OK
### --trc - relpath ### --trc - relpath
tmp/parallel.file : & ) \n*.jpg.file.done
tmp/parallel.file : & ) \n*.jpg.out
tmp/parallel.file/sub dir.file.done
tmp/parallel.file/sub dir.out
tmp/parallel.file1.file.done
tmp/parallel.file1.out
tmp/parallel.file10.file.done tmp/parallel.file10.file.done
tmp/parallel.file10.out tmp/parallel.file10.out
tmp/parallel.file13.file.done tmp/parallel.file13.file.done
@ -499,12 +506,10 @@ tmp/parallel.file18.file.done
tmp/parallel.file18.out tmp/parallel.file18.out
tmp/parallel.file19.file.done tmp/parallel.file19.file.done
tmp/parallel.file19.out tmp/parallel.file19.out
tmp/parallel.file1.file.done
tmp/parallel.file1.out
tmp/parallel.file20.file.done
tmp/parallel.file20.out
tmp/parallel.file2.file.done tmp/parallel.file2.file.done
tmp/parallel.file2.out tmp/parallel.file2.out
tmp/parallel.file20.file.done
tmp/parallel.file20.out
tmp/parallel.file3.file.done tmp/parallel.file3.file.done
tmp/parallel.file3.out tmp/parallel.file3.out
tmp/parallel.file5.file.done tmp/parallel.file5.file.done
@ -519,14 +524,16 @@ tmp/parallel.file9.file.done
tmp/parallel.file9.out tmp/parallel.file9.out
tmp/parallel.file>fire.file.done tmp/parallel.file>fire.file.done
tmp/parallel.file>fire.out tmp/parallel.file>fire.out
tmp/parallel.file : & ) \n*.jpg.file.done
tmp/parallel.file : & ) \n*.jpg.out
tmp/parallel.file/sub dir.file.done
tmp/parallel.file/sub dir.out
good if no file good if no file
ls: cannot access tmp/parallel.file*: No such file or directory ls: cannot access tmp/parallel.file*: No such file or directory
OK OK
### --trc - multiple files ### --trc - multiple files
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file10.file.done /tmp/parallel.file10.file.done
/tmp/parallel.file10.out /tmp/parallel.file10.out
/tmp/parallel.file13.file.done /tmp/parallel.file13.file.done
@ -543,12 +550,10 @@ OK
/tmp/parallel.file18.out /tmp/parallel.file18.out
/tmp/parallel.file19.file.done /tmp/parallel.file19.file.done
/tmp/parallel.file19.out /tmp/parallel.file19.out
/tmp/parallel.file1.file.done
/tmp/parallel.file1.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file2.file.done /tmp/parallel.file2.file.done
/tmp/parallel.file2.out /tmp/parallel.file2.out
/tmp/parallel.file20.file.done
/tmp/parallel.file20.out
/tmp/parallel.file3.file.done /tmp/parallel.file3.file.done
/tmp/parallel.file3.out /tmp/parallel.file3.out
/tmp/parallel.file5.file.done /tmp/parallel.file5.file.done
@ -563,10 +568,6 @@ OK
/tmp/parallel.file9.out /tmp/parallel.file9.out
/tmp/parallel.file>fire.file.done /tmp/parallel.file>fire.file.done
/tmp/parallel.file>fire.out /tmp/parallel.file>fire.out
/tmp/parallel.file : & ) \n*.jpg.file.done
/tmp/parallel.file : & ) \n*.jpg.out
/tmp/parallel.file/sub dir.file.done
/tmp/parallel.file/sub dir.out
good if no file good if no file
ls: cannot access /tmp/parallel.file*: No such file or directory ls: cannot access /tmp/parallel.file*: No such file or directory
OK OK

View file

@ -75,7 +75,7 @@ OK
ls: cannot access tmp/parallel.file*: No such file or directory ls: cannot access tmp/parallel.file*: No such file or directory
OK OK
Input for ssh Input for ssh
parallel-server1 rsync --server -lDrRze.iLsf . ./ parallel-server1 rsync --server -lDErRze.iLsf . ./
parallel-server1 cat tmp/parallel.file.' parallel-server1 cat tmp/parallel.file.'
'newline2 > tmp/parallel.file.' 'newline2 > tmp/parallel.file.'
'newline2.out;cat tmp/parallel.file.' 'newline2.out;cat tmp/parallel.file.'
@ -87,7 +87,7 @@ parallel-server1 rsync --server --sender -lDrRze.iLsf --remove-source-files . ./
'newline2.out2 'newline2.out2
parallel-server1 rm -f tmp/parallel.file.' parallel-server1 rm -f tmp/parallel.file.'
'newline2 'newline2
-l parallel parallel-server2 rsync --server -lDrRze.iLsf . ./ -l parallel parallel-server2 rsync --server -lDErRze.iLsf . ./
parallel@parallel-server2 cat tmp/parallel.file.' parallel@parallel-server2 cat tmp/parallel.file.'
'newline1 > tmp/parallel.file.' 'newline1 > tmp/parallel.file.'
'newline1.out;cat tmp/parallel.file.' 'newline1.out;cat tmp/parallel.file.'

View file

@ -0,0 +1,36 @@
### Test --basefile + --cleanup + permissions
scriptrun 1
scriptrun 2
scriptrun 3
scriptrun 4
scriptrun 5
good if no file
ls: cannot access script: No such file or directory
OK
### Test --basefile + --sshlogin :
1
10
11
12
13
2
3
4
5
6
7
8
9
1
10
11
12
13
2
3
4
5
6
7
8
9