diff --git a/README b/README index 849f1e84..7ef70e25 100644 --- a/README +++ b/README @@ -5,22 +5,23 @@ Please send problems and feedback to bug-parallel@gnu.org. = Presentation of GNU Parallel = -GNU Parallel is a shell tool for executing jobs in parallel using one -or more computers. A job is typically a single command or a small -script that has to be run for each of the lines in the input. The -typical input is a list of files, a list of hosts, a list of users, or -a list of tables. +GNU parallel is a shell tool for executing jobs in parallel using one +or more computers. A job is can be a single command or a small script +that has to be run for each of the lines in the input. The typical +input is a list of files, a list of hosts, a list of users, a list of +URLs, or a list of tables. A job can also be a command that reads from +a pipe. GNU parallel can then split the input and pipe it into +commands in parallel. -If you use xargs today you will find GNU Parallel very easy to use. If -you write loops in shell, you will find GNU Parallel may be able to -replace most of the loops and make them run faster by running jobs in -parallel. If you use ppss or pexec you will find GNU Parallel will -often make the command easier to read. +If you use xargs and tee today you will find GNU parallel very easy to +use as GNU parallel is written to have the same options as xargs. If +you write loops in shell, you will find GNU parallel may be able to +replace most of the loops and make them run faster by running several +jobs in parallel. -GNU Parallel also makes sure output from the commands is the same -output as you would get had you run the commands sequentially. This -makes it possible to use output from GNU Parallel as input for other -programs. +GNU parallel makes sure output from the commands is the same output as +you would get had you run the commands sequentially. This makes it +possible to use output from GNU parallel as input for other programs. = Installation = diff --git a/doc/FUTURE_IDEAS b/doc/FUTURE_IDEAS index d6f8e205..6f874407 100644 --- a/doc/FUTURE_IDEAS +++ b/doc/FUTURE_IDEAS @@ -1,72 +1,106 @@ -Video +Video 30. 36. 41. 48 + + +# GNU Parallel 20110522 - The Pakistan Release + +I am Ole Tange. I am the author of GNU Parallel. So far GNU Parallel has been focused on replacing a single for-loop. The Pakistan release introduces a way to replace nested loops. -As example I will use the image manipulation program 'convert' from -ImageMagick. This will convert foo.png to jpg with a size of 800 and -JPEG-quality of 95. +# NESTED FOR LOOPS -convert -size 800 -quality 95 foo.png foo_800_q95.jpg +As example I will use the image manipulation program 'convert' from +ImageMagick. This command will convert foo.png to jpg with a size of +800 and JPEG-quality of 95. + +convert -resize 800 -quality 95 foo.png foo_800_q95.jpg With a for-loop it can be done on a list of files: -time \ for file in *.png ; do - convert -size 800 -quality 95 $file ${file%.*}_800_q95.jpg + convert -resize 800 -quality 95 $file ${file%.*}_800_q95.jpg done -Using GNU Parallel it looks like this: +This is the kind of loops GNU Parallel is good at replacing: -time parallel convert -size 800 -quality 95 {} {.}_800_q95.jpg ::: *.png +parallel convert -resize 800 -quality 95 {} {.}_800_q95.jpg ::: *.png To get the images in 3 different JPEG-qualities you can use a nested for-loop: -time \ for qual in 25 50 95 ; do for file in *.png ; do - convert -size 800 -quality $qual $file ${file%.*}_800_q${qual}.jpg + convert -resize 800 -quality $qual $file ${file%.*}_800_q${qual}.jpg done done With GNU Parallel 'Pakistan' you can do this: -time parallel convert -size 800 -quality 95 {1} {1.}_800_q{2}.jpg ::: *.png ::: 25 50 95 +parallel convert -resize 800 -quality {2} {1} {1.}_800_q{2}.jpg ::: *.png ::: 25 50 95 + +The new is that you can use the ::: multiple times. GNU Parallel will +then generate all the combinations and execute the command with these. +The {1} and {2} will be replaced by the relevant input source. To get the 3 different JPEG-qualities in 2 different sizes you can nest the for-loop even further: -time \ for size in 800 30 ; do for qual in 25 50 95 ; do for file in *.png ; do - convert -size $size -quality $qual $file ${file%.*}_${size}_q${qual}.jpg + convert -resize $size -quality $qual $file ${file%.*}_${size}_q${qual}.jpg done done done With GNU Parallel 'Pakistan' you can do this: -time parallel convert -size {3} -quality {2} {1} {1.}_{3}_q{2}.jpg ::: *.png ::: 25 50 95 ::: 800 30 +parallel convert -resize {3} -quality {2} {1} {1.}_{3}_q{2}.jpg ::: *.png ::: 25 50 95 ::: 800 30 + +GNU Parallel will again generate all the combinations of the input +sources and run the jobs in parallel. You can also provide the arguments in a file. This will do the same as above: (echo 25; echo 50; echo 95) > qualities ls *.png > png-files (echo 800; echo 30) > sizes -time parallel convert -size {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: png-files qualities sizes +parallel convert -resize {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: png-files qualities sizes -It is also possible to mix triple and quad colon. These will do the same as above: +But you can even mix triple and quad colon. These will do the same: -time parallel convert -size {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: png-files ::: 25 50 95 ::: 800 30 +parallel convert -resize {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: png-files ::: 25 50 95 ::: 800 30 -time parallel convert -size {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: png-files ::: 25 50 95 :::: sizes +parallel convert -resize {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: png-files ::: 25 50 95 :::: sizes The special file '-' reads from standard input. This will do the same as above: -ls *.png | time parallel convert -size {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: - ::: 25 50 95 :::: sizes +ls *.png | parallel convert -resize {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: - ::: 25 50 95 :::: sizes + +This is probably one of the ways you will use this feature as that can easily be combined with find: + +find . -name '*.png' | \ +parallel convert -resize {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: - ::: 25 50 95 :::: sizes + +# Thank you for watching +# +# If you like GNU Parallel: +# * Post this video on forums/blogs/Twitter/Facebook/Linkedin +# * Join the mailing list http://lists.gnu.org/mailman/listinfo/parallel +# * Request or write a review for your favourite blog or magazine +# * Request or build a package for your favourite distribution +# * Invite me for your next conference (Contact http://ole.tange.dk) +# +# If GNU Parallel saves you money: +# * (Have your company) donate to FSF https://my.fsf.org/donate/ +# +# If you use GNU Parallel for a publication please cite: +# O. Tange (2011): GNU Parallel - The Command-Line Power Tool, ;login: +# The USENIX Magazine, February 2011:42-47. +# +# Find GNU Parallel at http://www.gnu.org/software/parallel/ Postkort: diff --git a/doc/release_new_version b/doc/release_new_version index 442d8589..919d935f 100644 --- a/doc/release_new_version +++ b/doc/release_new_version @@ -188,6 +188,9 @@ New in this release: * Implemented {//} for the input line with the basename removed (dirname). +* GNU Parallel now has a logo. + http://www.gnu.org/software/parallel/logo.png + * Article about GNU SQL in USENIX Magazine ;login: (print) http://www.usenix.org/publications/login/2011-04/ @@ -209,6 +212,9 @@ New in this release: * Short article about using GNU Parallel with lame: http://loopkid.net/articles/2011/04/30/accelerate-lame-mp3-conversion +* Using GNU Parallel to run tail -f in Japanese. Thanks to Clouder. + http://blog.clouder.jp/archives/001140.html + * BBC Research & Development uses GNU Parallel: http://www.bbc.co.uk/blogs/researchanddevelopment/2010/11/prototyping-weeknotes-41-26112.shtml @@ -267,6 +273,5 @@ will get that database's interactive shell. When using GNU SQL for a publication please cite: O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different -Databases Using DBURLs, ;login: The USENIX Magazine, April -2011. +Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32. >>>>> diff --git a/src/parallel b/src/parallel index b489f8ea..041f99d6 100755 --- a/src/parallel +++ b/src/parallel @@ -687,7 +687,8 @@ sub read_args_from_command_line { # Group of arguments on the command line. # Put them into a file. # Create argfile - my ($outfh,$name) = ::tempfile(SUFFIX => ".arg", DELETE => 1); + my ($outfh,$name) = ::tempfile(SUFFIX => ".arg"); + unlink($name); # Put args into argfile print $outfh map { $_,$/ } @group; seek $outfh, 0, 0; diff --git a/src/sql b/src/sql index 72af6cef..d142003d 100755 --- a/src/sql +++ b/src/sql @@ -997,7 +997,10 @@ sub version { "This is free software: you are free to change and redistribute it.", "GNU $Global::progname comes with no warranty.", "", - "Web site: http://www.gnu.org/software/${Global::progname}\n" + "Web site: http://www.gnu.org/software/${Global::progname}\n", + "When using GNU $Global::progname for a publication please cite:\n", + "O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different", + "Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.\n" ); }