-j 1 ran 2 jobs. Should of course only run 1 job
This commit is contained in:
Ole Tange 2009-04-04 11:33:42 +02:00
parent d78f8539f9
commit b60b16779e
2 changed files with 54 additions and 8 deletions

View file

@ -6,7 +6,7 @@ parallel - build and execute command lines from standard input in parallel
=head1 SYNOPSIS =head1 SYNOPSIS
B<parallel> [-0cfgqsuvx] [-j num] [command [arguments]] < list_of_arguments B<parallel> [-0cfgqsuvxX] [-j num] [command [arguments]] < list_of_arguments
=head1 DESCRIPTION =head1 DESCRIPTION
@ -176,6 +176,7 @@ is a better solution:
find . -name '*.jpg' | parallel -j +0 convert -geometry 120 {} {}_thumb.jpg find . -name '*.jpg' | parallel -j +0 convert -geometry 120 {} {}_thumb.jpg
find . -name '*_thumb.jpg' | ren 's:/([^/]+)_thumb.jpg$:/thumb_$1:' find . -name '*_thumb.jpg' | ren 's:/([^/]+)_thumb.jpg$:/thumb_$1:'
=head1 EXAMPLE 4: Substitution and redirection =head1 EXAMPLE 4: Substitution and redirection
This will compare all files in the dir to the file foo and save the This will compare all files in the dir to the file foo and save the
@ -200,6 +201,29 @@ To put the output in a file called <name>.dir:
B<ls | parallel '(echo -n {}" "; ls {}|wc -l) >>B< {}.dir'> B<ls | parallel '(echo -n {}" "; ls {}|wc -l) >>B< {}.dir'>
=head1 EXAMPLE 6: Context replace
To remove the files I<pict1000.jpg> .. I<pict9999.jpg> you could do:
B<seq 1000 9999 | parallel rm pict{}.jpg>
You could also do:
B<seq 1000 9999 | perl -pe 's/(.*)/pict$1.jpg/' | parallel -x rm>
The first will run B<rm> 8999 times, while the last will only run
B<rm> as many times needed to keep the command line length short
enough.
You could also run:
B<seq 1000 9999 | parallel -X rm pict{}.jpg>
This will also only run B<rm> as many times needed to keep the command
line length short enough.
=head1 QUOTING =head1 QUOTING
For more advanced use quoting may be an issue. The following will For more advanced use quoting may be an issue. The following will
@ -531,7 +555,9 @@ sub compute_number_of_processes {
# Number of processes wanted and limited by system ressources # Number of processes wanted and limited by system ressources
my $opt_j = shift; my $opt_j = shift;
my $wanted_processes = user_requested_processes($opt_j); my $wanted_processes = user_requested_processes($opt_j);
debug("Wanted procs: $wanted_processes\n");
my $system_limit = processes_available_by_system_limit($wanted_processes); my $system_limit = processes_available_by_system_limit($wanted_processes);
debug("Limited to procs: $system_limit\n");
return $system_limit; return $system_limit;
} }
@ -594,21 +620,21 @@ sub processes_available_by_system_limit {
$system_limit = int ($system_limit * 0.9)+1; $system_limit = int ($system_limit * 0.9)+1;
$spawning_too_slow = 1; $spawning_too_slow = 1;
} }
} while($system_limit <= $wanted_processes } while($system_limit < $wanted_processes
and defined $next_command_line and defined $next_command_line
and $more_filehandles and $more_filehandles
and not $max_system_proc_reached and not $max_system_proc_reached
and not $spawning_too_slow); and not $spawning_too_slow);
if($system_limit <= $wanted_processes and not $more_filehandles) { if($system_limit < $wanted_processes and not $more_filehandles) {
print STDERR ("Warning: Only enough filehandles to run ", print STDERR ("Warning: Only enough filehandles to run ",
$system_limit, " jobs in parallel. ", $system_limit, " jobs in parallel. ",
"Raising ulimit -n may help\n"); "Raising ulimit -n may help\n");
} }
if($system_limit <= $wanted_processes and $max_system_proc_reached) { if($system_limit < $wanted_processes and $max_system_proc_reached) {
print STDERR ("Warning: Only enough available processes to run ", print STDERR ("Warning: Only enough available processes to run ",
$system_limit, " jobs in parallel.\n"); $system_limit, " jobs in parallel.\n");
} }
if($system_limit <= $wanted_processes and $spawning_too_slow) { if($system_limit < $wanted_processes and $spawning_too_slow) {
print STDERR ("Warning: Starting 10 extra processes takes > 1 sec.\n", print STDERR ("Warning: Starting 10 extra processes takes > 1 sec.\n",
"Limiting to ", $system_limit, " jobs in parallel.\n"); "Limiting to ", $system_limit, " jobs in parallel.\n");
} }
@ -894,7 +920,7 @@ sub die_usage {
sub usage { sub usage {
print "Usage:\n"; print "Usage:\n";
print "parallel [-0cfgqsuvx] [command [arguments]] < list_of_arguments\n"; print "parallel [-0cfgqsuvxX] [-j num] [command [arguments]] < list_of_arguments\n";
} }
# #

View file

@ -124,7 +124,7 @@
.\" ======================================================================== .\" ========================================================================
.\" .\"
.IX Title "PARALLEL 1" .IX Title "PARALLEL 1"
.TH PARALLEL 1 "2009-03-16" "perl v5.10.0" "User Contributed Perl Documentation" .TH PARALLEL 1 "2009-04-04" "perl v5.10.0" "User Contributed Perl Documentation"
.\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" For nroff, turn off justification. Always turn off hyphenation; it makes
.\" way too many mistakes in technical documents. .\" way too many mistakes in technical documents.
.if n .ad l .if n .ad l
@ -133,7 +133,7 @@
parallel \- build and execute command lines from standard input in parallel parallel \- build and execute command lines from standard input in parallel
.SH "SYNOPSIS" .SH "SYNOPSIS"
.IX Header "SYNOPSIS" .IX Header "SYNOPSIS"
\&\fBparallel\fR [\-0cfgqsuvx] [\-j num] [command [arguments]] < list_of_arguments \&\fBparallel\fR [\-0cfgqsuvxX] [\-j num] [command [arguments]] < list_of_arguments
.SH "DESCRIPTION" .SH "DESCRIPTION"
.IX Header "DESCRIPTION" .IX Header "DESCRIPTION"
For each line of input \fBparallel\fR will execute \fBcommand\fR with the For each line of input \fBparallel\fR will execute \fBcommand\fR with the
@ -306,6 +306,26 @@ files in each directory:
To put the output in a file called <name>.dir: To put the output in a file called <name>.dir:
.PP .PP
\&\fBls | parallel '(echo \-n {}\*(L" \*(R"; ls {}|wc \-l) \fR>\fB {}.dir'\fR \&\fBls | parallel '(echo \-n {}\*(L" \*(R"; ls {}|wc \-l) \fR>\fB {}.dir'\fR
.SH "EXAMPLE 6: Context replace"
.IX Header "EXAMPLE 6: Context replace"
To remove the files \fIpict1000.jpg\fR .. \fIpict9999.jpg\fR you could do:
.PP
\&\fBseq 1000 9999 | parallel rm pict{}.jpg\fR
.PP
You could also do:
.PP
\&\fBseq 1000 9999 | perl \-pe 's/(.*)/pict$1.jpg/' | parallel \-x rm\fR
.PP
The first will run \fBrm\fR 8999 times, while the last will only run
\&\fBrm\fR as many times needed to keep the command line length short
enough.
.PP
You could also run:
.PP
\&\fBseq 1000 9999 | parallel \-X rm pict{}.jpg\fR
.PP
This will also only run \fBrm\fR as many times needed to keep the command
line length short enough.
.SH "QUOTING" .SH "QUOTING"
.IX Header "QUOTING" .IX Header "QUOTING"
For more advanced use quoting may be an issue. The following will For more advanced use quoting may be an issue. The following will