parallel: /tmp/*.arg was not removed

This commit is contained in:
Ole Tange 2011-05-21 01:07:45 +02:00
parent 8fb1966254
commit 10ac95d29a
5 changed files with 82 additions and 38 deletions

29
README
View file

@ -5,22 +5,23 @@ Please send problems and feedback to bug-parallel@gnu.org.
= Presentation of GNU Parallel =
GNU Parallel is a shell tool for executing jobs in parallel using one
or more computers. A job is typically a single command or a small
script that has to be run for each of the lines in the input. The
typical input is a list of files, a list of hosts, a list of users, or
a list of tables.
GNU parallel is a shell tool for executing jobs in parallel using one
or more computers. A job is can be a single command or a small script
that has to be run for each of the lines in the input. The typical
input is a list of files, a list of hosts, a list of users, a list of
URLs, or a list of tables. A job can also be a command that reads from
a pipe. GNU parallel can then split the input and pipe it into
commands in parallel.
If you use xargs today you will find GNU Parallel very easy to use. If
you write loops in shell, you will find GNU Parallel may be able to
replace most of the loops and make them run faster by running jobs in
parallel. If you use ppss or pexec you will find GNU Parallel will
often make the command easier to read.
If you use xargs and tee today you will find GNU parallel very easy to
use as GNU parallel is written to have the same options as xargs. If
you write loops in shell, you will find GNU parallel may be able to
replace most of the loops and make them run faster by running several
jobs in parallel.
GNU Parallel also makes sure output from the commands is the same
output as you would get had you run the commands sequentially. This
makes it possible to use output from GNU Parallel as input for other
programs.
GNU parallel makes sure output from the commands is the same output as
you would get had you run the commands sequentially. This makes it
possible to use output from GNU parallel as input for other programs.
= Installation =

View file

@ -1,72 +1,106 @@
Video
Video 30. 36. 41. 48
# GNU Parallel 20110522 - The Pakistan Release
I am Ole Tange. I am the author of GNU Parallel.
So far GNU Parallel has been focused on replacing a single
for-loop. The Pakistan release introduces a way to replace nested
loops.
As example I will use the image manipulation program 'convert' from
ImageMagick. This will convert foo.png to jpg with a size of 800 and
JPEG-quality of 95.
# NESTED FOR LOOPS
convert -size 800 -quality 95 foo.png foo_800_q95.jpg
As example I will use the image manipulation program 'convert' from
ImageMagick. This command will convert foo.png to jpg with a size of
800 and JPEG-quality of 95.
convert -resize 800 -quality 95 foo.png foo_800_q95.jpg
With a for-loop it can be done on a list of files:
time \
for file in *.png ; do
convert -size 800 -quality 95 $file ${file%.*}_800_q95.jpg
convert -resize 800 -quality 95 $file ${file%.*}_800_q95.jpg
done
Using GNU Parallel it looks like this:
This is the kind of loops GNU Parallel is good at replacing:
time parallel convert -size 800 -quality 95 {} {.}_800_q95.jpg ::: *.png
parallel convert -resize 800 -quality 95 {} {.}_800_q95.jpg ::: *.png
To get the images in 3 different JPEG-qualities you can use a nested for-loop:
time \
for qual in 25 50 95 ; do
for file in *.png ; do
convert -size 800 -quality $qual $file ${file%.*}_800_q${qual}.jpg
convert -resize 800 -quality $qual $file ${file%.*}_800_q${qual}.jpg
done
done
With GNU Parallel 'Pakistan' you can do this:
time parallel convert -size 800 -quality 95 {1} {1.}_800_q{2}.jpg ::: *.png ::: 25 50 95
parallel convert -resize 800 -quality {2} {1} {1.}_800_q{2}.jpg ::: *.png ::: 25 50 95
The new is that you can use the ::: multiple times. GNU Parallel will
then generate all the combinations and execute the command with these.
The {1} and {2} will be replaced by the relevant input source.
To get the 3 different JPEG-qualities in 2 different sizes you can
nest the for-loop even further:
time \
for size in 800 30 ; do
for qual in 25 50 95 ; do
for file in *.png ; do
convert -size $size -quality $qual $file ${file%.*}_${size}_q${qual}.jpg
convert -resize $size -quality $qual $file ${file%.*}_${size}_q${qual}.jpg
done
done
done
With GNU Parallel 'Pakistan' you can do this:
time parallel convert -size {3} -quality {2} {1} {1.}_{3}_q{2}.jpg ::: *.png ::: 25 50 95 ::: 800 30
parallel convert -resize {3} -quality {2} {1} {1.}_{3}_q{2}.jpg ::: *.png ::: 25 50 95 ::: 800 30
GNU Parallel will again generate all the combinations of the input
sources and run the jobs in parallel.
You can also provide the arguments in a file. This will do the same as above:
(echo 25; echo 50; echo 95) > qualities
ls *.png > png-files
(echo 800; echo 30) > sizes
time parallel convert -size {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: png-files qualities sizes
parallel convert -resize {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: png-files qualities sizes
It is also possible to mix triple and quad colon. These will do the same as above:
But you can even mix triple and quad colon. These will do the same:
time parallel convert -size {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: png-files ::: 25 50 95 ::: 800 30
parallel convert -resize {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: png-files ::: 25 50 95 ::: 800 30
time parallel convert -size {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: png-files ::: 25 50 95 :::: sizes
parallel convert -resize {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: png-files ::: 25 50 95 :::: sizes
The special file '-' reads from standard input. This will do the same as above:
ls *.png | time parallel convert -size {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: - ::: 25 50 95 :::: sizes
ls *.png | parallel convert -resize {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: - ::: 25 50 95 :::: sizes
This is probably one of the ways you will use this feature as that can easily be combined with find:
find . -name '*.png' | \
parallel convert -resize {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: - ::: 25 50 95 :::: sizes
# Thank you for watching
#
# If you like GNU Parallel:
# * Post this video on forums/blogs/Twitter/Facebook/Linkedin
# * Join the mailing list http://lists.gnu.org/mailman/listinfo/parallel
# * Request or write a review for your favourite blog or magazine
# * Request or build a package for your favourite distribution
# * Invite me for your next conference (Contact http://ole.tange.dk)
#
# If GNU Parallel saves you money:
# * (Have your company) donate to FSF https://my.fsf.org/donate/
#
# If you use GNU Parallel for a publication please cite:
# O. Tange (2011): GNU Parallel - The Command-Line Power Tool, ;login:
# The USENIX Magazine, February 2011:42-47.
#
# Find GNU Parallel at http://www.gnu.org/software/parallel/
Postkort:

View file

@ -188,6 +188,9 @@ New in this release:
* Implemented {//} for the input line with the basename removed (dirname).
* GNU Parallel now has a logo.
http://www.gnu.org/software/parallel/logo.png
* Article about GNU SQL in USENIX Magazine ;login: (print)
http://www.usenix.org/publications/login/2011-04/
@ -209,6 +212,9 @@ New in this release:
* Short article about using GNU Parallel with lame:
http://loopkid.net/articles/2011/04/30/accelerate-lame-mp3-conversion
* Using GNU Parallel to run tail -f in Japanese. Thanks to Clouder.
http://blog.clouder.jp/archives/001140.html
* BBC Research & Development uses GNU Parallel:
http://www.bbc.co.uk/blogs/researchanddevelopment/2010/11/prototyping-weeknotes-41-26112.shtml
@ -267,6 +273,5 @@ will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different
Databases Using DBURLs, ;login: The USENIX Magazine, April
2011.
Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
>>>>>

View file

@ -687,7 +687,8 @@ sub read_args_from_command_line {
# Group of arguments on the command line.
# Put them into a file.
# Create argfile
my ($outfh,$name) = ::tempfile(SUFFIX => ".arg", DELETE => 1);
my ($outfh,$name) = ::tempfile(SUFFIX => ".arg");
unlink($name);
# Put args into argfile
print $outfh map { $_,$/ } @group;
seek $outfh, 0, 0;

View file

@ -997,7 +997,10 @@ sub version {
"This is free software: you are free to change and redistribute it.",
"GNU $Global::progname comes with no warranty.",
"",
"Web site: http://www.gnu.org/software/${Global::progname}\n"
"Web site: http://www.gnu.org/software/${Global::progname}\n",
"When using GNU $Global::progname for a publication please cite:\n",
"O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different",
"Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.\n"
);
}