parallel: --joblog contains exitstatus

This commit is contained in:
Ole Tange 2011-04-07 21:54:02 +02:00
parent e1b8521487
commit a14fabad36
2 changed files with 45 additions and 9 deletions

View file

@ -565,7 +565,7 @@ sub parse_options {
} else { } else {
print $Global::joblog print $Global::joblog
join("\t", "Seq", "Host", "Starttime", "Runtime", join("\t", "Seq", "Host", "Starttime", "Runtime",
"Send", "Receive", "Command" "Send", "Receive", "Status", "Command"
). "\n"; ). "\n";
} }
} }
@ -3037,8 +3037,10 @@ sub print {
} }
printf $Global::joblog printf $Global::joblog
join("\t", $self->seq(), $self->sshlogin()->string(), join("\t", $self->seq(), $self->sshlogin()->string(),
$self->starttime(), $self->runtime(), $self->transfersize(), $self->returnsize(), $cmd $self->starttime(), $self->runtime(),
). "\n"; $self->transfersize(), $self->returnsize(),
$self->exitstatus(), $cmd
). "\n";
flush $Global::joblog; flush $Global::joblog;
} }

View file

@ -4,6 +4,7 @@
parallel - build and execute shell command lines from standard input in parallel parallel - build and execute shell command lines from standard input in parallel
=head1 SYNOPSIS =head1 SYNOPSIS
B<parallel> [options] [I<command> [arguments]] < list_of_arguments B<parallel> [options] [I<command> [arguments]] < list_of_arguments
@ -16,6 +17,7 @@ B<parallel> --semaphore [options] I<command>
B<#!/usr/bin/parallel> --shebang [options] [I<command> [arguments]] B<#!/usr/bin/parallel> --shebang [options] [I<command> [arguments]]
=head1 DESCRIPTION =head1 DESCRIPTION
GNU B<parallel> is a shell tool for executing jobs in parallel using one GNU B<parallel> is a shell tool for executing jobs in parallel using one
@ -50,6 +52,7 @@ http://www.youtube.com/watch?v=OpaiGYxkSuQ or at
http://tinyogg.com/watch/TORaR/ http://tinyogg.com/watch/hfxKj/ and http://tinyogg.com/watch/TORaR/ http://tinyogg.com/watch/hfxKj/ and
http://tinyogg.com/watch/YQuXd/ http://tinyogg.com/watch/YQuXd/
=head1 OPTIONS =head1 OPTIONS
=over 9 =over 9
@ -421,7 +424,8 @@ use B<-I> instead.
Logfile for executed jobs. Saved a list of the executed jobs to Logfile for executed jobs. Saved a list of the executed jobs to
I<logfile> in the following TAB separated format: sequence number, I<logfile> in the following TAB separated format: sequence number,
sshlogin, start time as seconds since epoch, run time in seconds, sshlogin, start time as seconds since epoch, run time in seconds,
bytes in files transfered, bytes in files returned, command run. bytes in files transfered, bytes in files returned, exit status,
and command run.
To convert the times into ISO-8601 strict do: To convert the times into ISO-8601 strict do:
@ -1133,6 +1137,7 @@ For this to work B<--shebang> or B<-Y> must be set as the first option.
=back =back
=head1 EXAMPLE: Working as xargs -n1. Argument appending =head1 EXAMPLE: Working as xargs -n1. Argument appending
GNU B<parallel> can work similar to B<xargs -n1>. GNU B<parallel> can work similar to B<xargs -n1>.
@ -1244,6 +1249,7 @@ Other special shell charaters (such as * ; $ > < | >> <<) also needs
to be put in quotes, as they may otherwise be interpreted by the shell to be put in quotes, as they may otherwise be interpreted by the shell
and not given to GNU B<parallel>. and not given to GNU B<parallel>.
=head1 EXAMPLE: Composed commands =head1 EXAMPLE: Composed commands
A job can consist of several commands. This will print the number of A job can consist of several commands. This will print the number of
@ -1271,6 +1277,7 @@ symlinks are empty files.
B<cp -rs /the/source/dir mirror_dir; find mirror_dir -type l | parallel -m rm {} '&&' touch {}> B<cp -rs /the/source/dir mirror_dir; find mirror_dir -type l | parallel -m rm {} '&&' touch {}>
=head1 EXAMPLE: Removing file extension when processing files =head1 EXAMPLE: Removing file extension when processing files
When processing files removing the file extension using B<{.}> is When processing files removing the file extension using B<{.}> is
@ -1293,6 +1300,7 @@ Put all converted in the same directory:
B<find sounddir -type f -name '*.wav' | parallel lame {} -o mydir/{/.}.mp3> B<find sounddir -type f -name '*.wav' | parallel lame {} -o mydir/{/.}.mp3>
=head1 EXAMPLE: Removing two file extensions when processing files and =head1 EXAMPLE: Removing two file extensions when processing files and
calling GNU Parallel from itself calling GNU Parallel from itself
@ -1302,6 +1310,7 @@ foo) you can do:
B<ls *.tar.gz| parallel -U {tar} 'echo {tar}|parallel "mkdir -p {.} ; tar -C {.} -xf {.}.tar.gz"'> B<ls *.tar.gz| parallel -U {tar} 'echo {tar}|parallel "mkdir -p {.} ; tar -C {.} -xf {.}.tar.gz"'>
=head1 EXAMPLE: Download 10 images for each of the past 30 days =head1 EXAMPLE: Download 10 images for each of the past 30 days
Let us assume a website stores images like: Let us assume a website stores images like:
@ -1318,6 +1327,19 @@ Based on this we can let GNU B<parallel> generate 10 B<wget>s per day:
I<the above> B<| parallel -I {o} seq -w 10 "|" parallel wget I<the above> B<| parallel -I {o} seq -w 10 "|" parallel wget
http://www.example.com/path/to/{o}_{}.jpg> http://www.example.com/path/to/{o}_{}.jpg>
=head1 EXAMPLE: Process files from a tar file while unpacking
If the files to be processed are in a tar file then unpacking one file
and processing it immediately may be faster than first unpacking all
files.
B<tar xvf foo.tgz | perl -ne 'print $l;$l=$_;END{print $l}' |
parallel echo>
The Perl one-liner is needed to avoid race condition.
=head1 EXAMPLE: Rewriting a for-loop and a while-read-loop =head1 EXAMPLE: Rewriting a for-loop and a while-read-loop
for-loops like this: for-loops like this:
@ -1421,6 +1443,7 @@ combined in the correct order.
B<seq 0 99 | parallel -k curl -r \ B<seq 0 99 | parallel -k curl -r \
{}0000000-{}9999999 http://example.com/the/big/file> > B<file> {}0000000-{}9999999 http://example.com/the/big/file> > B<file>
=head1 EXAMPLE: Parallel grep =head1 EXAMPLE: Parallel grep
B<grep -r> greps recursively through directories. On multicore CPUs B<grep -r> greps recursively through directories. On multicore CPUs
@ -1554,6 +1577,7 @@ the special short hand I<-S ..> can be used:
find logs/ -name '*.gz' | parallel -S .. \ find logs/ -name '*.gz' | parallel -S .. \
--trc {.}.bz2 "zcat {} | bzip2 -9 >{.}.bz2" --trc {.}.bz2 "zcat {} | bzip2 -9 >{.}.bz2"
=head1 EXAMPLE: Distributing work to local and remote computers =head1 EXAMPLE: Distributing work to local and remote computers
Convert *.mp3 to *.ogg running one process per CPU core on local computer and server2: Convert *.mp3 to *.ogg running one process per CPU core on local computer and server2:
@ -1561,6 +1585,7 @@ Convert *.mp3 to *.ogg running one process per CPU core on local computer and se
parallel --trc {.}.ogg -S server2,: \ parallel --trc {.}.ogg -S server2,: \
'mpg321 -w - {} | oggenc -q0 - -o {.}.ogg' ::: *.mp3 'mpg321 -w - {} | oggenc -q0 - -o {.}.ogg' ::: *.mp3
=head1 EXAMPLE: Use multiple inputs in one command =head1 EXAMPLE: Use multiple inputs in one command
Copy files like foo.es.ext to foo.ext: Copy files like foo.es.ext to foo.ext:
@ -1750,6 +1775,7 @@ printed when job 11 is started.
=back =back
=head1 EXAMPLE: GNU Parallel as dir processor =head1 EXAMPLE: GNU Parallel as dir processor
If you have a dir in which users drop files that needs to be processed If you have a dir in which users drop files that needs to be processed
@ -1771,6 +1797,10 @@ computers:
B<inotifywait -q -m -r -e CLOSE_WRITE --format %w%f my_dir | parallel -S .. B<inotifywait -q -m -r -e CLOSE_WRITE --format %w%f my_dir | parallel -S ..
-u echo> -u echo>
If the files to be processed are in a tar file then unpacking one file
and processing it immediately may be faster than first unpacking all
files. Set up the dir processor as above and unpack into the dir.
=head1 QUOTING =head1 QUOTING
@ -1973,6 +2003,7 @@ must be one argument.
=back =back
=head1 DEFAULT PROFILE (CONFIG FILE) =head1 DEFAULT PROFILE (CONFIG FILE)
The file ~/.parallel/config (formerly known as .parallelrc) will be The file ~/.parallel/config (formerly known as .parallelrc) will be
@ -1984,6 +2015,7 @@ Options on the command line takes precedence over the environment
variable $PARALLEL which takes precedence over the file variable $PARALLEL which takes precedence over the file
~/.parallel/config. ~/.parallel/config.
=head1 PROFILE FILES =head1 PROFILE FILES
If B<--profile> set, GNU B<parallel> will read the profile from that file instead of If B<--profile> set, GNU B<parallel> will read the profile from that file instead of
@ -2281,8 +2313,8 @@ The argument replace string ($ITEM) cannot be changed. Arguments must
be quoted - thus arguments containing special characters (space '"&!*) be quoted - thus arguments containing special characters (space '"&!*)
may cause problems. More than one argument is not supported. File may cause problems. More than one argument is not supported. File
names containing newlines are not processed correctly. When reading names containing newlines are not processed correctly. When reading
input from a file null cannot be used terminator. B<ppss> needs to input from a file null cannot be used as a terminator. B<ppss> needs
read the whole input file before starting any jobs. to read the whole input file before starting any jobs.
Output and status information is stored in ppss_dir and thus requires Output and status information is stored in ppss_dir and thus requires
cleanup when completed. If the dir is not removed before running cleanup when completed. If the dir is not removed before running
@ -2796,16 +2828,18 @@ license terms of this work.
A copy of the full license is included in the file as cc-by-sa.txt. A copy of the full license is included in the file as cc-by-sa.txt.
=head1 DEPENDENCIES =head1 DEPENDENCIES
GNU B<parallel> uses Perl, and the Perl modules Getopt::Long, GNU B<parallel> uses Perl, and the Perl modules Getopt::Long,
IPC::Open3, Symbol, IO::File, POSIX, and File::Temp. For remote usage IPC::Open3, Symbol, IO::File, POSIX, and File::Temp. For remote usage
it also uses Rsync with Ssh. it also uses rsync with ssh.
=head1 SEE ALSO =head1 SEE ALSO
B<find>(1), B<xargs>(1), B<make>(1), B<pexec>(1), B<ppss>(1), B<ssh>(1), B<rsync>(1), B<find>(1), B<xargs>(1), B<make>(1),
B<xjobs>(1), B<prll>(1), B<dxargs>(1), B<mdm>(1) B<pexec>(1), B<ppss>(1), B<xjobs>(1), B<prll>(1), B<dxargs>(1),
B<mdm>(1),
=cut =cut