From b0a723415266759e5c8b3ac8b803a41bb5ca58bc Mon Sep 17 00:00:00 2001 From: Ole Tange Date: Fri, 13 May 2011 14:02:03 +0200 Subject: [PATCH] parallel: test -E for ::: and :::: --- doc/FUTURE_IDEAS | 33 ++++++---------------- doc/release_new_version | 47 ++++++++++++++++++++++++++----- src/parallel.pod | 48 +++++++++++++++++--------------- testsuite/tests-to-run/test58.sh | 12 ++++++++ 4 files changed, 85 insertions(+), 55 deletions(-) diff --git a/doc/FUTURE_IDEAS b/doc/FUTURE_IDEAS index d6f26e59..d6f8e205 100644 --- a/doc/FUTURE_IDEAS +++ b/doc/FUTURE_IDEAS @@ -4,8 +4,9 @@ So far GNU Parallel has been focused on replacing a single for-loop. The Pakistan release introduces a way to replace nested loops. -As example I will use the image manipulation program 'convert'. This -will convert foo.png to jpg with a size of 800 and JPEG-quality of 95. +As example I will use the image manipulation program 'convert' from +ImageMagick. This will convert foo.png to jpg with a size of 800 and +JPEG-quality of 95. convert -size 800 -quality 95 foo.png foo_800_q95.jpg @@ -13,7 +14,7 @@ With a for-loop it can be done on a list of files: time \ for file in *.png ; do - convert -size 800 -quality 95 $file ${file##.JPG}_800_q95.jpg + convert -size 800 -quality 95 $file ${file%.*}_800_q95.jpg done Using GNU Parallel it looks like this: @@ -26,7 +27,7 @@ To get the images in 3 different JPEG-qualities you can use a nested for-loop: time \ for qual in 25 50 95 ; do for file in *.png ; do - convert -size 800 -quality $qual $file ${file##.JPG}_800_q${qual}.jpg + convert -size 800 -quality $qual $file ${file%.*}_800_q${qual}.jpg done done @@ -34,14 +35,14 @@ With GNU Parallel 'Pakistan' you can do this: time parallel convert -size 800 -quality 95 {1} {1.}_800_q{2}.jpg ::: *.png ::: 25 50 95 -To get the 3 different JPEG-qualities in 2 different sizes you can use -a nest the for-loop even further: +To get the 3 different JPEG-qualities in 2 different sizes you can +nest the for-loop even further: time \ for size in 800 30 ; do for qual in 25 50 95 ; do for file in *.png ; do - convert -size $size -quality $qual $file ${file##.JPG}_${size}_q${qual}.jpg + convert -size $size -quality $qual $file ${file%.*}_${size}_q${qual}.jpg done done done @@ -68,24 +69,6 @@ The special file '-' reads from standard input. This will do the same as above: ls *.png | time parallel convert -size {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: - ::: 25 50 95 :::: sizes -Test with first argument == END_OF_FILE string - -unittest til xapply og uden. - -parallel echo {1} {2} {3} ::: a b c :::: myfile ::: X Y - -parallel oggenc -q {1} {2} {2.}_{1}.ogg ::: 1 5 10 :::: wavfiles - -or - -parallel oggenc -q {1} {2} {2.}_{1}.ogg ::: 1 5 10 ::: *.wav - -or - -ls *.wav | parallel oggenc -q {1} {2} {2.}_{1}.ogg ::: 1 5 10 :::: - - - - Postkort: - Forside kun figur - Bagside: diff --git a/doc/release_new_version b/doc/release_new_version index 346f95a6..442d8589 100644 --- a/doc/release_new_version +++ b/doc/release_new_version @@ -167,24 +167,30 @@ This is a major release. So far GNU Parallel has been focused on replacing a single for-loop. The Pakistan release introduces ways to replace nested loops. +If you are using the {1} {2} syntax for multiple input sources, then you +need to read about --xapply as the function has changed. + New in this release: * Multiple ::: can be put on the command line and will be treated similar to contents from multiple -a's. * ::: and :::: can now be mixed on the command line. Use {1} .. {n} to - refer to the arguments from the different arguments. + refer to inputs from the different input sources. * --xapply is now needed to get xapply's behaviour of reading one line - from arguments from each of multiple input files. + from each of the input sources. -* Multiple input source will cause all combinations of arguments from - the sources to be generated. E.g. ::: 1 2 ::: a b c will result in - the combinations (1,a) (1,b) (1,c) (2,a) (2,b) (2,c). This can often - replace nested loops. +* Multiple input sources will cause all combinations of arguments from + the sources to be generated. E.g. 'parallel echo {1}+{2} ::: 1 2 ::: + a b c' will print 1+a 1+b 1+c 2+a 2+b 2+c. This can often replace + nested loops. * Implemented {//} for the input line with the basename removed (dirname). +* Article about GNU SQL in USENIX Magazine ;login: (print) + http://www.usenix.org/publications/login/2011-04/ + * Using GNU Parallel with EC2. Thanks to Kevin Wu. http://blog.kevinformatics.com/post/4970574713/interested-in-your-experience-using-gnu-parallel-in @@ -203,6 +209,15 @@ New in this release: * Short article about using GNU Parallel with lame: http://loopkid.net/articles/2011/04/30/accelerate-lame-mp3-conversion +* BBC Research & Development uses GNU Parallel: + http://www.bbc.co.uk/blogs/researchanddevelopment/2010/11/prototyping-weeknotes-41-26112.shtml + +* Short article about using GNU Parallel on RHEL. Thanks to Rishi Deshpande. + http://nuclear-imaging.info/site_content/2011/05/11/gnu-parallel/ + +* Using GNU Parallel for FLAC->MP3 conversion. Thanks to Derek Marcotte. + http://derek.chezmarcotte.ca/?p=286 + * Bug fixes and man page updates. @@ -220,7 +235,7 @@ If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several -jobs in parallel. +jobs in parallel. GNU Parallel can even replace nested loops. GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it @@ -236,4 +251,22 @@ When using GNU Parallel for a publication please cite: O. Tange (2011): GNU Parallel - The Command-Line Power Tool, ;login: The USENIX Magazine, February 2011:42-47. + + += About GNU SQL = + +GNU sql aims to give a simple, unified interface for accessing +databases through all the different databases' command line +clients. So far the focus has been on giving a common way to specify +login information (protocol, username, password, hostname, and port +number), size (database and table size), and running queries. + +The database is addressed using a DBURL. If commands are left out you +will get that database's interactive shell. + +When using GNU SQL for a publication please cite: + +O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different +Databases Using DBURLs, ;login: The USENIX Magazine, April +2011. >>>>> diff --git a/src/parallel.pod b/src/parallel.pod index c3227082..a71e4a7f 100644 --- a/src/parallel.pod +++ b/src/parallel.pod @@ -100,7 +100,7 @@ B<{/}> can be used the same places as B<{}>. The replacement string B<{/}> can be changed with B<--basenamereplace>. -=item B<{//}> +=item B<{//}> (alpha testing) Dirname of input line. This is a specialized replacement string containing the dir of the input. See B(1). @@ -128,7 +128,7 @@ The replacement string B<{#}> can be changed with B<--seqreplace>. =item B<{>IB<}> -Argument from argument file I or the I'th argument. See B<-a> +Argument from input source I or the I'th argument. See B<-a> and B<-N>. B<{>IB<}> can be used the same places as B<{}>. @@ -136,7 +136,7 @@ B<{>IB<}> can be used the same places as B<{}>. =item B<{>I.B<}> -Argument from argument file I or the I'th argument without +Argument from input source I or the I'th argument without extension. It is a combination of B<{>IB<}> and B<{.}>. B<{>I.B<}> can be used the same places as B<{>IB<}>. @@ -144,7 +144,7 @@ B<{>I.B<}> can be used the same places as B<{>IB<}>. =item B<{>I/B<}> -Basename of argument from argument file I or the I'th argument. +Basename of argument from input source I or the I'th argument. It is a combination of B<{>IB<}> and B<{/}>. See B<-a> and B<-N>. B<{>I/B<}> can be used the same places as B<{>IB<}>. @@ -152,7 +152,7 @@ B<{>I/B<}> can be used the same places as B<{>IB<}>. =item B<{>I/.B<}> -Basename of argument from argument file I or the I'th argument +Basename of argument from input source I or the I'th argument without extension. It is a combination of B<{>IB<}>, B<{/}>, and B<{.}>. See B<-a> and B<-N>. @@ -161,9 +161,9 @@ B<{>I/.B<}> can be used the same places as B<{>IB<}>. =item B<:::> I (alpha testing) -Use arguments from the command line as input instead of from stdin -(standard input). Unlike other options for GNU B B<:::> is -placed after the I and before the arguments. +Use arguments from the command line as input source instead of from +stdin (standard input). Unlike other options for GNU B +B<:::> is placed after the I and before the arguments. The following are equivalent: @@ -179,10 +179,11 @@ argument separator to something else. See also B<--arg-sep>. stdin (standard input) will be passed to the first process run. -If multiple B<:::> are given, all combinations of group of arguments -will be generated. E.g. ::: 1 2 ::: a b c will result in the -combinations (1,a) (1,b) (1,c) (2,a) (2,b) (2,c). This is useful for -replacing nested for-loops. +If multiple B<:::> are given, each group will be treated as an input +source, and all combinations of input sources will be +generated. E.g. ::: 1 2 ::: a b c will result in the combinations +(1,a) (1,b) (1,c) (2,a) (2,b) (2,c). This is useful for replacing +nested for-loops. B<:::> and B<::::> can be mixed. So these are equivalent: @@ -198,6 +199,8 @@ B<:::> and B<::::> can be mixed. So these are equivalent: Another way to write B<-a> I B<-a> I ... +B<:::> and B<::::> can be mixed. + See B<-a>. @@ -214,21 +217,20 @@ for processing arguments that may contain \n (newline). =item B<-a> I -Read items from the file I instead of stdin (standard input). If -you use this option, stdin is given to the first process run. -Otherwise, stdin is redirected from /dev/null. +Use I as input source. If you use this option, stdin is +given to the first process run. Otherwise, stdin is redirected from +/dev/null. -If multiple B<-a> are given, all combinations of all lines will be -generated. E.g. 1 2 and a b c will result in the combinations (1,a) -(1,b) (1,c) (2,a) (2,b) (2,c). This is useful for replacing nested -for-loops. - -The arguments can be accessed in the command as B<{1}> -.. B<{>IB<}>, so B<{1}> will be a line from the first file, and -B<{6}> will refer to a line from the 6th file. +If multiple B<-a> are given, each I will be treated as an +input source, and all combinations of input sources will be +generated. E.g. The file B contains B<1 2>, the file B +contains B. B<-a foo> B<-a bar> will result in the combinations +(1,a) (1,b) (1,c) (2,a) (2,b) (2,c). This is useful for replacing +nested for-loops. See also B<--xapply> + =item B<--arg-file-sep> I Use I instead of B<::::> as separator string between command diff --git a/testsuite/tests-to-run/test58.sh b/testsuite/tests-to-run/test58.sh index 705f8240..a0e3d67b 100644 --- a/testsuite/tests-to-run/test58.sh +++ b/testsuite/tests-to-run/test58.sh @@ -18,3 +18,15 @@ seq 6 7 | parallel -k -a - -a <(seq 4 5) echo {1} {2} {3} ::: 1 2 3 echo '### Test :::: < - :::' seq 4 5 | parallel -k echo {1} {2} {3} :::: <(seq 6 7) - ::: 1 2 3 + +echo '### Test -E' +seq 1 100 | parallel -k -E 5 echo :::: - ::: 2 3 4 5 6 7 8 9 10 :::: <(seq 3 11) + +echo '### Test -E one empty' +seq 1 100 | parallel -k -E 3 echo :::: - ::: 2 3 4 5 6 7 8 9 10 :::: <(seq 3 11) + +echo '### Test -E 2 empty' +seq 1 100 | parallel -k -E 3 echo :::: - ::: 3 4 5 6 7 8 9 10 :::: <(seq 3 11) + +echo '### Test -E all empty' +seq 3 100 | parallel -k -E 3 echo :::: - ::: 3 4 5 6 7 8 9 10 :::: <(seq 3 11)