Implemented ::::.

FIXED BUG: Dependent quoting of arguments after :::.

doc/FUTURE_IDEAS

@@ -1,41 +1,8 @@
-FIXED BUG: time parallel -j+0 --eta -Sserver1,server2,server3,: \
+FIXED BUG: Dependent quoting of arguments after :::
---transfer --return {.}.bz2 --cleanup 'zcat {} | bzip2 -9 > {.}.bz2' ::: *gz
+FIXED BUG: parallel echo {1} {2} :::: <(echo '>') <(echo b)
-Time is negative
+Implemented ::::.
 
-FIXED BUG: echo "foo,bar,baz" | parallel -d, -L 1 echo
+multiple -a == ::::
-Gave warning.
-
-xapply compatability
-
-xapply -p/dev/null -f 'diff %1 %2' manifest1 checklist1
-BUG: parallel diff {1} {2} :::: manifest1 checklist1
-
-tr ':' '\012' < /etc/passwd | xapply -7 -nf 'chown %1 %6' - - - - - - -
-BUG: tr ':' '\012' < /etc/passwd | parallel -N7 chown {1} {6}
-
-xapply '(cd %1 && make all)' */
-parallel 'cd {} && make all' ::: *
-
-xapply -f 'diff %1 ../version5/%1' manifest | more
-parallel diff {} ../version5/{} < manifest | more
-
-xapply 'indent' *.c
-parallel indent ::: *.c
-
-find ~ksb/bin -type f ! -perm -111 -print | xapply -f -v 'chmod a+x' -
-find ~ksb/bin -type f ! -perm -111 -print | parallel -v chmod a+x
-
-find ... | xapply -f -5 -i /dev/tty 'vi' - - - - -
-sh <(find ... |parallel -n5 echo vi)
-
-xapply -fn "" /etc/passwd
-parallel -k echo /etc/passwd
-
-xapply '[ -d %1/RCS ] || echo %1' */
-parallel '[ -d {}/RCS ] || echo {}' ::: *
-
-xapply -f '[ -f %1 ] && echo %1' List | ...
-parallel '[ -f {} ] && echo {}' < List
 
 # Hvordan udregnes system limits på remote systems hvis jeg ikke ved, hvormange
 # argumenter, der er? Lav system limits lokalt og lad det være max

src/parallel

@@ -10,6 +10,8 @@ B<parallel> [options] [I<command> [arguments]] < list_of_arguments
 
 B<parallel> [options] [I<command> [arguments]] B<:::> arguments
 
+B<parallel> [options] [I<command> [arguments]] B<::::> argfile(s)
+
 =head1 DESCRIPTION
 
 GNU B<parallel> is a shell tool for executing jobs in parallel locally
@@ -78,7 +80,15 @@ B<{.}> can be used the same places as B<{}>. The replacement string
 B<{.}> can be changed with B<-U>.
 
 
-=item B<:::> (beta testing)
+=item B<{>I<n>B<}>
+
+Argument from argument file I<n> or the I<n>'th argument. See B<::::>
+and B<-N>.
+
+B<{>I<n>B<}> can be used the same places as B<{}>.
+
+
+=item B<:::> I<arguments> (beta testing)
 
 Use arguments from the command line as input instead of from stdin
 (standard input). Unlike other options for GNU B<parallel> B<:::> is
@@ -101,12 +111,13 @@ stdin (standard input) will be passed to the first process run.
 If B<--arg-file> is set arguments from that file will be appended.
 
 
-=item B<::::> (unimplemented)
+=item B<::::> I<argfiles> (beta test)
 
-Use arguments from the command line as files for input. One line will
+Use arguments from the command line as arguments files for input. One
-be read from each of the files. The arguments can be access in the
+line will be read from each of the files. The arguments can be
-command as {1} .. {n}, so {1} will be a line from the first file, and
+accessed in the command as B<{1}> .. B<{>I<n>B<}>, so B<{1}> will be a
-{6} will refer to the corresponding line from the 6th file.
+line from the first file, and B<{6}> will refer to the line with the
+same line number from the 6th file.
 
 
 =item B<--null>
@@ -398,13 +409,14 @@ GNU B<parallel> will exit.
 Implies B<-X> unless B<-m> is set.
 
 
-=item B<--max-replace-args>=I<max-args> (unimplemented)
+=item B<--max-replace-args>=I<max-args> (beta test)
 
-=item B<-N> I<max-args> (unimplemented)
+=item B<-N> I<max-args> (beta test)
 
 Use at most I<max-args> arguments per command line. Like B<-n> but
-also makes replacement strings {1} .. {I<max-args>} that represents
+also makes replacement strings B<{1}> .. B<{>I<max-args>B<}> that
-argument 1 .. I<max-args>. If too few args the {number} will be empty.
+represents argument 1 .. I<max-args>. If too few args the {I<n>} will
+be empty.
 
 This will set the owner of the homedir to the user:
 
@@ -1406,7 +1418,7 @@ killall -SIGUSR1 parallel # Not quite equivalent: Only shows the currently runni
 B<pexec> is also a tool for running jobs in parallel.
 
 Here are the examples from B<pexec>'s info page with the equivalent
-using parallel:
+using GNU B<parallel>:
 
 pexec -o sqrt-%s.dat -p "$(seq 10)" -e NUM -n 4 -c -- \
   'echo "scale=10000;sqrt($NUM)" | bc'
@@ -1539,6 +1551,58 @@ B<seq 1 19 | parallel -j+0 buffon -o - | sort -n >>B< result>
 B<cat files | parallel -j+0 cmd>
 
 
+=head2 DIFFERENCES BETWEEN xapply AND GNU Parallel
+
+B<xapply> can run jobs in parallel on the local computer.
+
+Here are the examples from B<xapply>'s man page with the equivalent
+using GNU B<parallel>:
+
+xapply '(cd %1 && make all)' */
+
+parallel 'cd {} && make all' ::: */
+
+xapply -f 'diff %1 ../version5/%1' manifest | more
+
+parallel diff {} ../version5/{} < manifest | more
+
+xapply -p/dev/null -f 'diff %1 %2' manifest1 checklist1
+
+parallel diff {1} {2} :::: manifest1 checklist1
+
+xapply 'indent' *.c
+
+parallel indent ::: *.c
+
+find ~ksb/bin -type f ! -perm -111 -print | xapply -f -v 'chmod a+x' -
+
+find ~ksb/bin -type f ! -perm -111 -print | parallel -v chmod a+x
+
+find */ -... | fmt 960 1024 | xapply -f -i /dev/tty 'vi' -
+
+sh <(find */ -... | parallel -s 1024 echo vi)
+
+find ... | xapply -f -5 -i /dev/tty 'vi' - - - - -
+
+sh <(find ... |parallel -n5 echo vi)
+
+xapply -fn "" /etc/passwd
+
+parallel -k echo < /etc/passwd
+
+tr ':' '\012' < /etc/passwd | xapply -7 -nf 'chown %1 %6' - - - - - - -
+
+tr ':' '\012' < /etc/passwd | parallel -N7 chown {1} {6}
+
+xapply '[ -d %1/RCS ] || echo %1' */
+
+parallel '[ -d {}/RCS ] || echo {}' ::: */
+
+xapply -f '[ -f %1 ] && echo %1' List | ...
+
+parallel '[ -f {} ] && echo {}' < List | ...
+
+
 =head2 DIFFERENCES BETWEEN ClusterSSH AND GNU Parallel
 
 ClusterSSH solves a different problem than GNU B<parallel>.
@@ -1549,7 +1613,7 @@ several machines that are almost identical.
 
 GNU B<parallel> runs the same (or different) commands with different
 arguments in parallel possibly using remote machines to help
-computing. If more than one machine is listed in -S GNU B<parallel> may
+computing. If more than one machine is listed in B<-S> GNU B<parallel> may
 only use one of these (e.g. if there are 8 jobs to be run and one
 machine has 8 cores).
 
@@ -1563,6 +1627,14 @@ B<cat hostlist | parallel ssh {} do_stuff>
 Filenames beginning with '-' can cause some commands to give
 unexpected results, as it will often be interpreted as an option.
 
+Because the way newline is quoted this will not work:
+
+echo 1,2,3 | parallel -vkd, "echo 'a{}'"
+
+However, this will work:
+
+echo 1,2,3 | parallel -vkd, echo a{}
+
 
 =head1 REPORTING BUGS
 
@@ -1804,6 +1876,7 @@ sub parse_options {
 	       "E=s" => \$::opt_E,
 	       "eof|e:s" => \$::opt_E,
 	       "max-args|n=i" => \$::opt_n,
+	       "max-replace-args|N=i" => \$::opt_N,
 	       "help|h" => \$::opt_help,
 	       "L=i" => \$::opt_L,
 	       "max-lines|l:i" => \$::opt_l,
@@ -1814,7 +1887,6 @@ sub parse_options {
 	       "exit|x" => \$::opt_x,
 	) || die_usage();
     $Global::debug = (defined $::opt_D);
-    $Global::input_is_filename = (@ARGV);
     if(defined $::opt_m) { $Global::xargs = 1; }
     if(defined $::opt_X) { $Global::Xargs = 1; }
     if(defined $::opt_v) { $Global::verbose = 1; }
@@ -1839,6 +1911,7 @@ sub parse_options {
 	$Global::max_number_of_args = $Global::max_lines;
     }
     if(defined $::opt_n and $::opt_n) { $Global::max_number_of_args = $::opt_n; }
+    if(defined $::opt_N and $::opt_N) { $Global::max_number_of_args = $::opt_N; }
     if(defined $::opt_help) { die_usage(); }
     if(defined $::opt_arg_sep) { $Global::arg_sep = $::opt_arg_sep; }
     if(defined $::opt_number_of_cpus) { print no_of_cpus(),"\n"; exit(0); }
@@ -1855,13 +1928,6 @@ sub parse_options {
 	$::opt_cleanup = 1;
     }
 
-    if(($::opt_l || $::opt_L || $::opt_n || $::opt_s) and not
-       ($::opt_m or $::opt_X)) {
-	# These --max-line, -l, -L, --max-args, -n, --max-chars, -s
-	# do not make sense without -X or -m 
-	# so default to -X
-	$Global::Xargs = 1;
-    }
     if(grep /^$Global::arg_sep$/o, @ARGV) {
 	# Arguments on the command line.
 	# Ignore STDIN by reading from /dev/null
@@ -1873,7 +1939,12 @@ sub parse_options {
 	while(@ARGV) {
 	    my $arg = shift @ARGV;
 	    if($arg eq $Global::arg_sep) {
-		unget_arg(@ARGV);
+		$Global::input_is_filename = (@new_argv);
+		if($Global::input_is_filename) {
+		    unget_arg(shell_quote(@ARGV));
+		} else {
+		    unget_arg(@ARGV);
+		}
 		$Global::total_jobs += @ARGV;
 		@ARGV=();
 	    } else {
@@ -1884,6 +1955,68 @@ sub parse_options {
 	@ARGV=@new_argv;
     }
 
+    if(grep /^::::$/o, @ARGV) {
+	# max-replace-args
+	$::opt_a ||= "/dev/null";
+
+	# read all the files and merge them
+	my @new_argv = ();
+	my @argument_files;
+	while(@ARGV) {
+	    my $arg = shift @ARGV;
+	    if($arg eq "::::") {
+		@argument_files = @ARGV;
+		@ARGV=();
+	    } else {
+		push @new_argv, $arg;
+	    }
+	}
+	# Output: @ARGV = command option
+	@ARGV=@new_argv;
+	$Global::input_is_filename ||= (@ARGV);
+
+	$::opt_N = $#argument_files+1;
+	$Global::max_number_of_args = $#argument_files+1;
+
+	# read the files
+	my @content;
+	my $max_lineno = 0;
+	my $in_fh = gensym;
+	for (my $fileno = 0; $fileno <= $#argument_files; $fileno++) {
+	    if(not open ($in_fh, $argument_files[$fileno])) {
+		print STDERR ("$Global::progname: Cannot open $argument_files[$fileno]\n");
+		    exit (255);
+	    }
+	    for (my $lineno=0; $content[$fileno][$lineno] = get_next_arg_from_fh($in_fh); $lineno++) {
+		$max_lineno = max($max_lineno,$lineno);
+	    }
+	    close $in_fh;
+	}
+	for (my $lineno=0; $lineno <= $max_lineno; $lineno++) {
+	    for (my $fileno = 0; $fileno <= $#argument_files; $fileno++) {
+		my $arg = $content[$fileno][$lineno];
+		if(defined $arg) {
+		    unget_arg($arg);
+		} else {
+		    unget_arg("");
+		}
+	    }
+	}
+	$Global::total_jobs += $max_lineno;
+    }
+
+    # must be done after ::: and :::: because they mess with @ARGV
+    $Global::input_is_filename ||= (@ARGV);
+
+    if(($::opt_l || $::opt_L || $::opt_n || $::opt_N || $::opt_s) and not
+       ($::opt_m or $::opt_X)) {
+	# These --max-line, -l, -L, --max-args, -n, --max-chars, -s
+	# do not make sense without -X or -m
+	# so default to -X
+	# Needs to be done after ::::, as that can set $::opt_N
+	$Global::Xargs = 1;
+    }
+
     if(defined $::opt_a) {
 	# must be done after opt_arg_sep
 	if(not open(ARGFILE,"<",$::opt_a)) {
@@ -1961,6 +2094,7 @@ sub generate_command_line {
     my ($job_line,$last_good);
     my ($quoted_args,$quoted_args_no_ext) =
 	get_multiple_args($command,max_length_of_command_line(),0);
+    my $is_substituted = 0;
 
     if(@$quoted_args) {
 	$job_line = $command;
@@ -1979,7 +2113,23 @@ sub generate_command_line {
 		$job_line =~ s/\Q$Global::replacestring\E/$arg/go;
 		$job_line =~ s/\Q$Global::replace_no_ext\E/$arg_no_ext/go;
 	    }
-	} else {
+	    $is_substituted = 1;
+	}
+	if(defined $job_line and $::opt_N and $job_line =~/\{\d+\}/o) {
+	    # substitute {#} with args
+	    my $argno=1;
+	    for my $argno (1..$::opt_N) {
+		my $arg = $quoted_args->[$argno-1];
+		if(defined $arg) {
+		    $job_line =~ s/\{$argno\}/$arg/g;
+		} else {
+		    $job_line =~ s/\{$argno\}//g;
+		}
+		$argno++;
+	    }
+	    $is_substituted = 1;
+	}
+	if (not $is_substituted) {
 	    # append args
 	    my $arg=join(" ",@$quoted_args);
 	    if($job_line) {
@@ -2776,7 +2926,42 @@ sub unget_command_line {
 sub more_arguments {
     # Returns:
     #   whether there are more arguments to be processed or not
-    return (@Global::unget_arg or not eof $Global::argfile);
+    my $fh = shift || $Global::argfile;
+    return (@Global::unget_arg or not eof $fh);
+}
+
+sub get_next_arg_from_fh {
+    # Returns:
+    #   next argument from file handle
+    #   undef if end of file
+    my $fh = shift;
+    if(not more_arguments($fh)) {
+	return undef;
+    }
+    my $arg = <$fh>;
+    # Remove delimiter
+    $arg =~ s:$/$::;
+    if($Global::end_of_file_string and
+       $arg eq $Global::end_of_file_string) {
+	# Ignore the rest of input file
+	while (<$fh>) {}
+	return undef;
+    }
+    if($Global::ignore_empty) {
+	if($arg =~ /^\s*$/) {
+	    return get_next_arg_from_fh($fh);
+	}
+    }
+    if($Global::max_lines and more_arguments()) {
+	if($arg =~ /\s$/) {
+	    # Trailing space => continued on next line
+	    $arg .= get_next_arg_from_fh($fh);
+	}
+    }
+    if($Global::input_is_filename) {
+	$arg = shell_quote($arg);
+    }
+    return $arg;
 }
 
 sub get_next_arg {
@@ -2787,34 +2972,11 @@ sub get_next_arg {
     if(@Global::unget_arg) {
 	$arg = shift @Global::unget_arg;
     } else {
-	if(not more_arguments()) {
+	$arg = get_next_arg_from_fh($Global::argfile);
-	    return undef;
+	if(defined $arg) {
+	    $Global::total_jobs++;
 	}
-	$arg = <$Global::argfile>;
-	chomp $arg;
-	if($Global::end_of_file_string and
-	   $arg eq $Global::end_of_file_string) {
-	    # Ignore the rest of STDIN
-	    while (<$Global::argfile>) {}
-	    return undef;
-	}
-	if($Global::ignore_empty) {
-	    if($arg =~ /^\s*$/) {
-		return get_next_arg();
-	    }
-	}
-	if($Global::max_lines and more_arguments()) {
-	    if($arg =~ /\s$/) {
-		# Trailing space => continued on next line
-		$arg .= get_next_arg();
-	    }
-	}
-	if($Global::input_is_filename) {
-	    $arg = shell_quote($arg);
-	}
-	$Global::total_jobs++;
     }
-    debug("Next arg: !".$arg."!\n");
     return $arg;
 }
 
@@ -3542,6 +3704,7 @@ sub usage {
     print "Usage:\n";
     print "$Global::progname [options] [command [arguments]] < list_of_arguments)\n";
     print "$Global::progname [options] [command [arguments]] ::: arguments\n";
+    print "$Global::progname [options] [command [arguments]] :::: argfile(s)\n";
     print "\n";
     print "See 'man $Global::progname' for the options\n";
 }

unittest/Start.sh

@@ -4,7 +4,7 @@ export LANG=C
 SHFILE=/tmp/unittest-parallel.sh
 
 ls -t tests-to-run/test*.sh \
-| perl -pe 's:(.*/(.*)).sh:sh $1.sh > actual-results/$2; diff -Naur wanted-results/$2 actual-results/$2:' \
+| perl -pe 's:(.*/(.*)).sh:bash $1.sh > actual-results/$2; diff -Naur wanted-results/$2 actual-results/$2:' \
 >$SHFILE
 
 mkdir -p actual-results

unittest/actual-results/test25

@@ -40,3 +40,22 @@ cat
 1
 2
 3
+### Bug did not quote
+echo \>
+>
+echo \>
+>
+echo \> 2
+> 2
+> 2
+### Must not quote
+echo | wc -l
+1
+echo | wc -l
+1
+echo a b c | wc -w
+3
+echo a b c | wc -w
+3
+echo a b | wc -w
+2

unittest/tests-to-run/test13.sh

@@ -1,11 +1,9 @@
 #!/bin/bash
 
-PAR=parallel
-
 echo '### Test -k'
 ulimit -n 50
-(echo "sleep 3; echo begin"; seq 1 30 | $PAR -kq echo "sleep 1; echo {}"; echo "echo end") \
+(echo "sleep 3; echo begin"; seq 1 30 | parallel -kq echo "sleep 1; echo {}"; echo "echo end") \
-| $PAR -k -j0
+| parallel -k -j0
 
 echo '### Test SIGTERM'
-(sleep 5; killall $PAR -TERM) & seq 1 100 | $PAR -k sleep 3';' echo
+(sleep 5; killall parallel -TERM) & seq 1 100 | parallel -k sleep 3';' echo

unittest/tests-to-run/test25.sh

@@ -17,3 +17,13 @@ echo '### Bug made 4 5 go before 1 2 3'
 parallel -k ::: "sleep 1; echo 1" "echo 2" "echo 3" "echo 4" "echo 5"
 echo '### Bug made 3 go before 1 2'
 parallel -kj 1 ::: "sleep 1; echo 1" "echo 2" "echo 3"
+echo '### Bug did not quote'
+echo '>' | parallel -v echo
+parallel -v echo ::: '>'
+(echo '>'; echo  2) | parallel -vX echo
+parallel -X echo ::: '>' 2
+echo '### Must not quote'
+echo 'echo | wc -l' | parallel -v
+parallel -v ::: 'echo | wc -l'
+echo 'echo a b c | wc -w' | parallel -v
+parallel -kv ::: 'echo a b c | wc -w' 'echo a b | wc -w'

unittest/tests-to-run/test26.sh

@@ -0,0 +1,34 @@
+#!/bin/bash
+
+echo '### Test xapply --max-replace-args'
+seq 0 7 | parallel -k --max-replace-args=3 echo {3} {2} {1}
+echo '### Test -N'
+seq 1 5 | parallel -kN3 echo {1} {2} {3}
+echo '### Test -N with 0'
+seq 0 7 | parallel -kN3 echo {1} {2} {3}
+echo '### Test :::: on nonexistent'
+stdout parallel -k echo {1} {2} {3} :::: nonexistent
+echo '### Test :::: two files'
+parallel -k echo {1} {2} :::: <(seq 1 10) <(seq 5 15)
+echo '### Test -d, ::::'
+parallel -kd, 'echo a{1} {2}b' :::: <(echo 1,2,3,) <(echo 5,6,7,8)
+echo '### Test -d, :::: one file too much'
+parallel -kd, echo 'a{1}' '{2}b' :::: <(echo 1,2,3,) <(echo 5,6,7,8) <(echo 9,0)
+echo '### Bug: did not quote'
+parallel echo {1} {2} :::: <(echo '>') <(echo b)
+echo '### Quote test triplet 1'
+parallel -kv :::: <(echo 'echo a'; echo 'echo b')
+parallel -kv -a <(echo 'echo a'; echo 'echo b')
+(echo 'echo a'; echo 'echo b') | parallel -kv
+echo '### Quote test triplet 2'
+parallel -kv echo :::: <(echo 'echo a'; echo 'echo b')
+parallel -kv -a <(echo 'echo a'; echo 'echo b') echo
+(echo 'echo a'; echo 'echo b') | parallel -kv echo
+echo '### Quoting if there is a command and 2 arg files'
+parallel -kv echo :::: <(echo 'echo a') <(echo 'echo b')
+echo '### Quoting if there is a command and 2 arg files of uneven length'
+parallel -kv echo :::: <(echo 'echo a';echo a1) <(echo 'echo b')
+echo '### Quoting if there is no command and 2 arg files'
+parallel -kv :::: <(echo 'echo a') <(echo 'echo b')
+echo '### Quoting if there is no command and 2 arg files of uneven length'
+parallel -kv :::: <(echo 'echo a';echo echo a1) <(echo 'echo b')

unittest/wanted-results/test15

@@ -272,8 +272,8 @@ line 2
 ### Test --no-run-if-empty and -r: This should give no output
 ### Test --help and -h: Help output (just check we get the same amount of lines)
 Output from -h and --help
-4
+6
-4
+6
 ### Test --version: Version output (just check we get the same amount of lines)
 7
 ### Test --verbose and -t

unittest/wanted-results/test25

@@ -40,3 +40,22 @@ cat
 1
 2
 3
+### Bug did not quote
+echo \>
+>
+echo \>
+>
+echo \> 2
+> 2
+> 2
+### Must not quote
+echo | wc -l
+1
+echo | wc -l
+1
+echo a b c | wc -w
+3
+echo a b c | wc -w
+3
+echo a b | wc -w
+2

unittest/wanted-results/test26

@@ -0,0 +1,83 @@
+### Test xapply --max-replace-args
+2 1 0
+5 4 3
+7 6
+### Test -N
+1 2 3
+4 5
+### Test -N with 0
+0 1 2
+3 4 5
+6 7
+### Test :::: on nonexistent
+parallel: Cannot open nonexistent
+### Test :::: two files
+1 5
+2 6
+3 7
+4 8
+5 9
+6 10
+7 11
+8 12
+9 13
+10 14
+15
+### Test -d, ::::
+a1 5b
+a2 6b
+a3 7b
+a
+ 8
+b
+### Test -d, :::: one file too much
+a1 5b
+a2 6b
+a3 7b
+a
+ 8
+b
+### Bug: did not quote
+> b
+### Quote test triplet 1
+echo a
+a
+echo b
+b
+echo a
+a
+echo b
+b
+echo a
+a
+echo b
+b
+### Quote test triplet 2
+echo echo\ a
+echo a
+echo echo\ b
+echo b
+echo echo\ a
+echo a
+echo echo\ b
+echo b
+echo echo\ a
+echo a
+echo echo\ b
+echo b
+### Quoting if there is a command and 2 arg files
+echo echo\ a echo\ b
+echo a echo b
+### Quoting if there is a command and 2 arg files of uneven length
+echo echo\ a echo\ b
+echo a echo b
+echo a1 
+a1
+### Quoting if there is no command and 2 arg files
+echo a echo b
+a echo b
+### Quoting if there is no command and 2 arg files of uneven length
+echo a echo b
+a echo b
+echo a1 
+a1