parallel: /tmp/*.arg was not removed

This commit is contained in:
Ole Tange 2011-05-21 01:07:45 +02:00
parent 8fb1966254
commit 10ac95d29a
5 changed files with 82 additions and 38 deletions

29
README
View file

@ -5,22 +5,23 @@ Please send problems and feedback to bug-parallel@gnu.org.
= Presentation of GNU Parallel = = Presentation of GNU Parallel =
GNU Parallel is a shell tool for executing jobs in parallel using one GNU parallel is a shell tool for executing jobs in parallel using one
or more computers. A job is typically a single command or a small or more computers. A job is can be a single command or a small script
script that has to be run for each of the lines in the input. The that has to be run for each of the lines in the input. The typical
typical input is a list of files, a list of hosts, a list of users, or input is a list of files, a list of hosts, a list of users, a list of
a list of tables. URLs, or a list of tables. A job can also be a command that reads from
a pipe. GNU parallel can then split the input and pipe it into
commands in parallel.
If you use xargs today you will find GNU Parallel very easy to use. If If you use xargs and tee today you will find GNU parallel very easy to
you write loops in shell, you will find GNU Parallel may be able to use as GNU parallel is written to have the same options as xargs. If
replace most of the loops and make them run faster by running jobs in you write loops in shell, you will find GNU parallel may be able to
parallel. If you use ppss or pexec you will find GNU Parallel will replace most of the loops and make them run faster by running several
often make the command easier to read. jobs in parallel.
GNU Parallel also makes sure output from the commands is the same GNU parallel makes sure output from the commands is the same output as
output as you would get had you run the commands sequentially. This you would get had you run the commands sequentially. This makes it
makes it possible to use output from GNU Parallel as input for other possible to use output from GNU parallel as input for other programs.
programs.
= Installation = = Installation =

View file

@ -1,72 +1,106 @@
Video Video 30. 36. 41. 48
# GNU Parallel 20110522 - The Pakistan Release
I am Ole Tange. I am the author of GNU Parallel.
So far GNU Parallel has been focused on replacing a single So far GNU Parallel has been focused on replacing a single
for-loop. The Pakistan release introduces a way to replace nested for-loop. The Pakistan release introduces a way to replace nested
loops. loops.
As example I will use the image manipulation program 'convert' from # NESTED FOR LOOPS
ImageMagick. This will convert foo.png to jpg with a size of 800 and
JPEG-quality of 95.
convert -size 800 -quality 95 foo.png foo_800_q95.jpg As example I will use the image manipulation program 'convert' from
ImageMagick. This command will convert foo.png to jpg with a size of
800 and JPEG-quality of 95.
convert -resize 800 -quality 95 foo.png foo_800_q95.jpg
With a for-loop it can be done on a list of files: With a for-loop it can be done on a list of files:
time \
for file in *.png ; do for file in *.png ; do
convert -size 800 -quality 95 $file ${file%.*}_800_q95.jpg convert -resize 800 -quality 95 $file ${file%.*}_800_q95.jpg
done done
Using GNU Parallel it looks like this: This is the kind of loops GNU Parallel is good at replacing:
time parallel convert -size 800 -quality 95 {} {.}_800_q95.jpg ::: *.png parallel convert -resize 800 -quality 95 {} {.}_800_q95.jpg ::: *.png
To get the images in 3 different JPEG-qualities you can use a nested for-loop: To get the images in 3 different JPEG-qualities you can use a nested for-loop:
time \
for qual in 25 50 95 ; do for qual in 25 50 95 ; do
for file in *.png ; do for file in *.png ; do
convert -size 800 -quality $qual $file ${file%.*}_800_q${qual}.jpg convert -resize 800 -quality $qual $file ${file%.*}_800_q${qual}.jpg
done done
done done
With GNU Parallel 'Pakistan' you can do this: With GNU Parallel 'Pakistan' you can do this:
time parallel convert -size 800 -quality 95 {1} {1.}_800_q{2}.jpg ::: *.png ::: 25 50 95 parallel convert -resize 800 -quality {2} {1} {1.}_800_q{2}.jpg ::: *.png ::: 25 50 95
The new is that you can use the ::: multiple times. GNU Parallel will
then generate all the combinations and execute the command with these.
The {1} and {2} will be replaced by the relevant input source.
To get the 3 different JPEG-qualities in 2 different sizes you can To get the 3 different JPEG-qualities in 2 different sizes you can
nest the for-loop even further: nest the for-loop even further:
time \
for size in 800 30 ; do for size in 800 30 ; do
for qual in 25 50 95 ; do for qual in 25 50 95 ; do
for file in *.png ; do for file in *.png ; do
convert -size $size -quality $qual $file ${file%.*}_${size}_q${qual}.jpg convert -resize $size -quality $qual $file ${file%.*}_${size}_q${qual}.jpg
done done
done done
done done
With GNU Parallel 'Pakistan' you can do this: With GNU Parallel 'Pakistan' you can do this:
time parallel convert -size {3} -quality {2} {1} {1.}_{3}_q{2}.jpg ::: *.png ::: 25 50 95 ::: 800 30 parallel convert -resize {3} -quality {2} {1} {1.}_{3}_q{2}.jpg ::: *.png ::: 25 50 95 ::: 800 30
GNU Parallel will again generate all the combinations of the input
sources and run the jobs in parallel.
You can also provide the arguments in a file. This will do the same as above: You can also provide the arguments in a file. This will do the same as above:
(echo 25; echo 50; echo 95) > qualities (echo 25; echo 50; echo 95) > qualities
ls *.png > png-files ls *.png > png-files
(echo 800; echo 30) > sizes (echo 800; echo 30) > sizes
time parallel convert -size {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: png-files qualities sizes parallel convert -resize {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: png-files qualities sizes
It is also possible to mix triple and quad colon. These will do the same as above: But you can even mix triple and quad colon. These will do the same:
time parallel convert -size {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: png-files ::: 25 50 95 ::: 800 30 parallel convert -resize {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: png-files ::: 25 50 95 ::: 800 30
time parallel convert -size {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: png-files ::: 25 50 95 :::: sizes parallel convert -resize {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: png-files ::: 25 50 95 :::: sizes
The special file '-' reads from standard input. This will do the same as above: The special file '-' reads from standard input. This will do the same as above:
ls *.png | time parallel convert -size {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: - ::: 25 50 95 :::: sizes ls *.png | parallel convert -resize {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: - ::: 25 50 95 :::: sizes
This is probably one of the ways you will use this feature as that can easily be combined with find:
find . -name '*.png' | \
parallel convert -resize {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: - ::: 25 50 95 :::: sizes
# Thank you for watching
#
# If you like GNU Parallel:
# * Post this video on forums/blogs/Twitter/Facebook/Linkedin
# * Join the mailing list http://lists.gnu.org/mailman/listinfo/parallel
# * Request or write a review for your favourite blog or magazine
# * Request or build a package for your favourite distribution
# * Invite me for your next conference (Contact http://ole.tange.dk)
#
# If GNU Parallel saves you money:
# * (Have your company) donate to FSF https://my.fsf.org/donate/
#
# If you use GNU Parallel for a publication please cite:
# O. Tange (2011): GNU Parallel - The Command-Line Power Tool, ;login:
# The USENIX Magazine, February 2011:42-47.
#
# Find GNU Parallel at http://www.gnu.org/software/parallel/
Postkort: Postkort:

View file

@ -188,6 +188,9 @@ New in this release:
* Implemented {//} for the input line with the basename removed (dirname). * Implemented {//} for the input line with the basename removed (dirname).
* GNU Parallel now has a logo.
http://www.gnu.org/software/parallel/logo.png
* Article about GNU SQL in USENIX Magazine ;login: (print) * Article about GNU SQL in USENIX Magazine ;login: (print)
http://www.usenix.org/publications/login/2011-04/ http://www.usenix.org/publications/login/2011-04/
@ -209,6 +212,9 @@ New in this release:
* Short article about using GNU Parallel with lame: * Short article about using GNU Parallel with lame:
http://loopkid.net/articles/2011/04/30/accelerate-lame-mp3-conversion http://loopkid.net/articles/2011/04/30/accelerate-lame-mp3-conversion
* Using GNU Parallel to run tail -f in Japanese. Thanks to Clouder.
http://blog.clouder.jp/archives/001140.html
* BBC Research & Development uses GNU Parallel: * BBC Research & Development uses GNU Parallel:
http://www.bbc.co.uk/blogs/researchanddevelopment/2010/11/prototyping-weeknotes-41-26112.shtml http://www.bbc.co.uk/blogs/researchanddevelopment/2010/11/prototyping-weeknotes-41-26112.shtml
@ -267,6 +273,5 @@ will get that database's interactive shell.
When using GNU SQL for a publication please cite: When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different
Databases Using DBURLs, ;login: The USENIX Magazine, April Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
2011.
>>>>> >>>>>

View file

@ -687,7 +687,8 @@ sub read_args_from_command_line {
# Group of arguments on the command line. # Group of arguments on the command line.
# Put them into a file. # Put them into a file.
# Create argfile # Create argfile
my ($outfh,$name) = ::tempfile(SUFFIX => ".arg", DELETE => 1); my ($outfh,$name) = ::tempfile(SUFFIX => ".arg");
unlink($name);
# Put args into argfile # Put args into argfile
print $outfh map { $_,$/ } @group; print $outfh map { $_,$/ } @group;
seek $outfh, 0, 0; seek $outfh, 0, 0;

View file

@ -997,7 +997,10 @@ sub version {
"This is free software: you are free to change and redistribute it.", "This is free software: you are free to change and redistribute it.",
"GNU $Global::progname comes with no warranty.", "GNU $Global::progname comes with no warranty.",
"", "",
"Web site: http://www.gnu.org/software/${Global::progname}\n" "Web site: http://www.gnu.org/software/${Global::progname}\n",
"When using GNU $Global::progname for a publication please cite:\n",
"O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different",
"Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.\n"
); );
} }