tange/parallel
1
0
Fork 0
mirror of https://git.savannah.gnu.org/git/parallel.git synced 2024-11-21 21:47:54 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

parallel: test -E for ::: and ::::

Browse source
This commit is contained in:
Ole Tange 2011-05-13 14:02:03 +02:00
parent 24dc6e55c4
commit b0a7234152
4 changed files with 85 additions and 55 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

33
doc/FUTURE_IDEAS
View file

@ -4,8 +4,9 @@ So far GNU Parallel has been focused on replacing a single
for-loop. The Pakistan release introduces a way to replace nested
loops.
As example I will use the image manipulation program 'convert'. This
will convert foo.png to jpg with a size of 800 and JPEG-quality of 95.
As example I will use the image manipulation program 'convert' from
ImageMagick. This will convert foo.png to jpg with a size of 800 and
JPEG-quality of 95.
convert -size 800 -quality 95 foo.png foo_800_q95.jpg
@ -13,7 +14,7 @@ With a for-loop it can be done on a list of files:
time \
for file in *.png ; do
convert -size 800 -quality 95 $file ${file##.JPG}_800_q95.jpg
convert -size 800 -quality 95 $file ${file%.*}_800_q95.jpg
done
Using GNU Parallel it looks like this:
@ -26,7 +27,7 @@ To get the images in 3 different JPEG-qualities you can use a nested for-loop:
time \
for qual in 25 50 95 ; do
for file in *.png ; do
convert -size 800 -quality $qual $file ${file##.JPG}_800_q${qual}.jpg
convert -size 800 -quality $qual $file ${file%.*}_800_q${qual}.jpg
done
done
@ -34,14 +35,14 @@ With GNU Parallel 'Pakistan' you can do this:
time parallel convert -size 800 -quality 95 {1} {1.}_800_q{2}.jpg ::: *.png ::: 25 50 95
To get the 3 different JPEG-qualities in 2 different sizes you can use
a nest the for-loop even further:
To get the 3 different JPEG-qualities in 2 different sizes you can
nest the for-loop even further:
time \
for size in 800 30 ; do
for qual in 25 50 95 ; do
for file in *.png ; do
convert -size $size -quality $qual $file ${file##.JPG}_${size}_q${qual}.jpg
convert -size $size -quality $qual $file ${file%.*}_${size}_q${qual}.jpg
done
done
done
@ -68,24 +69,6 @@ The special file '-' reads from standard input. This will do the same as above:
ls *.png | time parallel convert -size {3} -quality {2} {1} {1.}_{3}_q{2}.jpg :::: - ::: 25 50 95 :::: sizes
Test with first argument == END_OF_FILE string
unittest til xapply og uden.
parallel echo {1} {2} {3} ::: a b c :::: myfile ::: X Y
parallel oggenc -q {1} {2} {2.}_{1}.ogg ::: 1 5 10 :::: wavfiles
or
parallel oggenc -q {1} {2} {2.}_{1}.ogg ::: 1 5 10 ::: *.wav
or
ls *.wav | parallel oggenc -q {1} {2} {2.}_{1}.ogg ::: 1 5 10 :::: -
Postkort:
- Forside kun figur
- Bagside:

47
doc/release_new_version
View file

@ -167,24 +167,30 @@ This is a major release. So far GNU Parallel has been focused on
replacing a single for-loop. The Pakistan release introduces ways to
replace nested loops.
If you are using the {1} {2} syntax for multiple input sources, then you
need to read about --xapply as the function has changed.
New in this release:
* Multiple ::: can be put on the command line and will be treated
similar to contents from multiple -a's.
* ::: and :::: can now be mixed on the command line. Use {1} .. {n} to
refer to the arguments from the different arguments.
refer to inputs from the different input sources.
* --xapply is now needed to get xapply's behaviour of reading one line
from arguments from each of multiple input files.
from each of the input sources.
* Multiple input source will cause all combinations of arguments from
the sources to be generated. E.g. ::: 1 2 ::: a b c will result in
the combinations (1,a) (1,b) (1,c) (2,a) (2,b) (2,c). This can often
replace nested loops.
* Multiple input sources will cause all combinations of arguments from
the sources to be generated. E.g. 'parallel echo {1}+{2} ::: 1 2 :::
a b c' will print 1+a 1+b 1+c 2+a 2+b 2+c. This can often replace
nested loops.
* Implemented {//} for the input line with the basename removed (dirname).
* Article about GNU SQL in USENIX Magazine ;login: (print)
http://www.usenix.org/publications/login/2011-04/
* Using GNU Parallel with EC2. Thanks to Kevin Wu.
http://blog.kevinformatics.com/post/4970574713/interested-in-your-experience-using-gnu-parallel-in
@ -203,6 +209,15 @@ New in this release:
* Short article about using GNU Parallel with lame:
http://loopkid.net/articles/2011/04/30/accelerate-lame-mp3-conversion
* BBC Research & Development uses GNU Parallel:
http://www.bbc.co.uk/blogs/researchanddevelopment/2010/11/prototyping-weeknotes-41-26112.shtml
* Short article about using GNU Parallel on RHEL. Thanks to Rishi Deshpande.
http://nuclear-imaging.info/site_content/2011/05/11/gnu-parallel/
* Using GNU Parallel for FLAC->MP3 conversion. Thanks to Derek Marcotte.
http://derek.chezmarcotte.ca/?p=286
* Bug fixes and man page updates.
@ -220,7 +235,7 @@ If you use xargs and tee today you will find GNU Parallel very easy to
use as GNU Parallel is written to have the same options as xargs. If
you write loops in shell, you will find GNU Parallel may be able to
replace most of the loops and make them run faster by running several
jobs in parallel.
jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as
you would get had you run the commands sequentially. This makes it
@ -236,4 +251,22 @@ When using GNU Parallel for a publication please cite:
O. Tange (2011): GNU Parallel - The Command-Line Power Tool, ;login:
The USENIX Magazine, February 2011:42-47.
= About GNU SQL =
GNU sql aims to give a simple, unified interface for accessing
databases through all the different databases' command line
clients. So far the focus has been on giving a common way to specify
login information (protocol, username, password, hostname, and port
number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you
will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different
Databases Using DBURLs, ;login: The USENIX Magazine, April
2011.
>>>>>

48
src/parallel.pod
View file

@ -100,7 +100,7 @@ B<{/}> can be used the same places as B<{}>. The replacement string
B<{/}> can be changed with B<--basenamereplace>.
=item B<{//}>
=item B<{//}> (alpha testing)
Dirname of input line. This is a specialized replacement string
containing the dir of the input. See B<dirname>(1).
@ -128,7 +128,7 @@ The replacement string B<{#}> can be changed with B<--seqreplace>.
=item B<{>I<n>B<}>
Argument from argument file I<n> or the I<n>'th argument. See B<-a>
Argument from input source I<n> or the I<n>'th argument. See B<-a>
and B<-N>.
B<{>I<n>B<}> can be used the same places as B<{}>.
@ -136,7 +136,7 @@ B<{>I<n>B<}> can be used the same places as B<{}>.
=item B<{>I<n>.B<}>
Argument from argument file I<n> or the I<n>'th argument without
Argument from input source I<n> or the I<n>'th argument without
extension. It is a combination of B<{>I<n>B<}> and B<{.}>.
B<{>I<n>.B<}> can be used the same places as B<{>I<n>B<}>.
@ -144,7 +144,7 @@ B<{>I<n>.B<}> can be used the same places as B<{>I<n>B<}>.
=item B<{>I<n>/B<}>
Basename of argument from argument file I<n> or the I<n>'th argument.
Basename of argument from input source I<n> or the I<n>'th argument.
It is a combination of B<{>I<n>B<}> and B<{/}>. See B<-a> and B<-N>.
B<{>I<n>/B<}> can be used the same places as B<{>I<n>B<}>.
@ -152,7 +152,7 @@ B<{>I<n>/B<}> can be used the same places as B<{>I<n>B<}>.
=item B<{>I<n>/.B<}>
Basename of argument from argument file I<n> or the I<n>'th argument
Basename of argument from input source I<n> or the I<n>'th argument
without extension. It is a combination of B<{>I<n>B<}>, B<{/}>, and
B<{.}>. See B<-a> and B<-N>.
@ -161,9 +161,9 @@ B<{>I<n>/.B<}> can be used the same places as B<{>I<n>B<}>.
=item B<:::> I<arguments> (alpha testing)
Use arguments from the command line as input instead of from stdin
(standard input). Unlike other options for GNU B<parallel> B<:::> is
placed after the I<command> and before the arguments.
Use arguments from the command line as input source instead of from
stdin (standard input). Unlike other options for GNU B<parallel>
B<:::> is placed after the I<command> and before the arguments.
The following are equivalent:
@ -179,10 +179,11 @@ argument separator to something else. See also B<--arg-sep>.
stdin (standard input) will be passed to the first process run.
If multiple B<:::> are given, all combinations of group of arguments
will be generated. E.g. ::: 1 2 ::: a b c will result in the
combinations (1,a) (1,b) (1,c) (2,a) (2,b) (2,c). This is useful for
replacing nested for-loops.
If multiple B<:::> are given, each group will be treated as an input
source, and all combinations of input sources will be
generated. E.g. ::: 1 2 ::: a b c will result in the combinations
(1,a) (1,b) (1,c) (2,a) (2,b) (2,c). This is useful for replacing
nested for-loops.
B<:::> and B<::::> can be mixed. So these are equivalent:
@ -198,6 +199,8 @@ B<:::> and B<::::> can be mixed. So these are equivalent:
Another way to write B<-a> I<argfile1> B<-a> I<argfile2> ...
B<:::> and B<::::> can be mixed.
See B<-a>.
@ -214,21 +217,20 @@ for processing arguments that may contain \n (newline).
=item B<-a> I<input-file>
Read items from the file I<input-file> instead of stdin (standard input). If
you use this option, stdin is given to the first process run.
Otherwise, stdin is redirected from /dev/null.
Use I<input-file> as input source. If you use this option, stdin is
given to the first process run. Otherwise, stdin is redirected from
/dev/null.
If multiple B<-a> are given, all combinations of all lines will be
generated. E.g. 1 2 and a b c will result in the combinations (1,a)
(1,b) (1,c) (2,a) (2,b) (2,c). This is useful for replacing nested
for-loops.
The arguments can be accessed in the command as B<{1}>
.. B<{>I<n>B<}>, so B<{1}> will be a line from the first file, and
B<{6}> will refer to a line from the 6th file.
If multiple B<-a> are given, each I<input-file> will be treated as an
input source, and all combinations of input sources will be
generated. E.g. The file B<foo> contains B<1 2>, the file B<bar>
contains B<a b c>. B<-a foo> B<-a bar> will result in the combinations
(1,a) (1,b) (1,c) (2,a) (2,b) (2,c). This is useful for replacing
nested for-loops.
See also B<--xapply>
=item B<--arg-file-sep> I<sep-str>
Use I<sep-str> instead of B<::::> as separator string between command

12
testsuite/tests-to-run/test58.sh
View file

@ -18,3 +18,15 @@ seq 6 7 | parallel -k -a - -a <(seq 4 5) echo {1} {2} {3} ::: 1 2 3
echo '### Test :::: < - :::'
seq 4 5 | parallel -k echo {1} {2} {3} :::: <(seq 6 7) - ::: 1 2 3
echo '### Test -E'
seq 1 100 | parallel -k -E 5 echo :::: - ::: 2 3 4 5 6 7 8 9 10 :::: <(seq 3 11)
echo '### Test -E one empty'
seq 1 100 | parallel -k -E 3 echo :::: - ::: 2 3 4 5 6 7 8 9 10 :::: <(seq 3 11)
echo '### Test -E 2 empty'
seq 1 100 | parallel -k -E 3 echo :::: - ::: 3 4 5 6 7 8 9 10 :::: <(seq 3 11)
echo '### Test -E all empty'
seq 3 100 | parallel -k -E 3 echo :::: - ::: 3 4 5 6 7 8 9 10 :::: <(seq 3 11)