Added basic xargs-test. Passes unittest

This commit is contained in:
Ole Tange 2010-07-18 04:17:49 +02:00
parent 757dddaf6c
commit 611e7db260
51 changed files with 515 additions and 21 deletions

3
.gitignore vendored
View file

@ -5,3 +5,6 @@ config.log
config.status config.status
parallel-*.bz2* parallel-*.bz2*
stamp-h1 stamp-h1
actual-results
tmp
video

51
NEWS
View file

@ -1,3 +1,54 @@
2010XXXX
* Arguments can now be given on the command line. Example:
parallel bzip2 ::: *.txt
* xapply like functionality. Example:
parallel diff {1} {2} :::: filelist1 filelist2
parallel diff {1} {2} ">"{1.}-{2.}.diff :::: filelist1 filelist2
* Arch Linux package. Thanks to Peter Simons <simons at cryp dot to>
* Mandriva package. Thanks to Sandro Cazzaniga <kharec at mandriva dot
org>
* -L -l -n -s now implies -X unless -m is set
20100620
* New video showing the new options.
http://www.youtube.com/watch?v=OpaiGYxkSuQ or at
http://tinyogg.com/watch/TORaR/ and http://tinyogg.com/watch/hfxKj/
* 100% options complete with xargs. All options for xargs can now be
used in GNU Parallel - even the more exotic.
* --basefile for transferring basedata. When running jobs on remote
computers --basefile will transfer files before the first jobs is
run. It can be used to transfer data that remains the same for each
job such as scripts or lookup tables.
* --progress shows progress. To see how many jobs is running on each
server use --progress. It can be turned on even after GNU Parallel
is started.
* --eta shows estimated time left in seconds.
* --halt-on-error stops if an error occurs. GNU Parallel will default
to run all jobs - even if some of them fail. With --halt-on-error
GNU Parallel can ignore errors, wait for the currently running jobs
to finish, or stop immediately when an error occurs.
20100601
* GNU Parallel now has support for running jobs on remote machines
* Intro video http://www.youtube.com/watch?v=LlXDtd_pRaY
http://openvideo.dailymotion.com/video/xdmqwz_gnu-parallel-script-processing-and_tech
20100424 20100424
* GNU Parallel is now official GNU software * GNU Parallel is now official GNU software

6
README
View file

@ -31,11 +31,15 @@ Installation of GNU Parallel is a simple as:
After this you should be able to do: After this you should be able to do:
(echo foss.org.my; echo gnu.org; echo freenetproject.org) | parallel traceroute parallel traceroute ::: foss.org.my gnu.org freenetproject.org
This will run traceroute to 3 different hosts in parallel and print This will run traceroute to 3 different hosts in parallel and print
the output when the traceroutes are done. the output when the traceroutes are done.
Watch the intro video for a quick introduction:
http://www.youtube.com/watch?v=OpaiGYxkSuQ or at
http://tinyogg.com/watch/TORaR/ and http://tinyogg.com/watch/hfxKj/
= New versions = = New versions =

View file

@ -1,3 +1,20 @@
No: too many differences Unittests from xargs findutils-4.4.2/xargs/testsuite
Import sql
inputfile tabel, Split colonner til {n}
sql :foo 'select * from bar' | parallel --colsep '\s+\|\s+' do_stuff {4} {1}
parallel -a table_file --colsep '\s+' do_stuff {4} {1}
--autocolsep: Læs alle linjer.
Prøv fastlængde: Find tegn, som står i alle linjer på de samme pladser. Risiko for falske pos
Prøv fieldsep: Find eet tegn, som optræder det samme antal gange i alle linjer (tab sep)
Prøv klyngesep: Find den samme klynge tegn, som står samme antal gange i alle linjer (' | ' sep)
Fjern whitespace før og efter colonne
--trc combined with {#}
# Hvordan udregnes system limits på remote systems hvis jeg ikke ved, hvormange # Hvordan udregnes system limits på remote systems hvis jeg ikke ved, hvormange
# argumenter, der er? Lav system limits lokalt og lad det være max # argumenter, der er? Lav system limits lokalt og lad det være max

View file

@ -27,6 +27,10 @@ make && sudo make install
cd unittest; make unittest cd unittest; make unittest
== Update NEWS ==
With the same things that goes in the announce mail
== Package == == Package ==
./configure ./configure
@ -80,28 +84,18 @@ download at: http://ftp.gnu.org/gnu/parallel/
New in this release: New in this release:
* New video showing the new options. * Arguments can now be given on the command line. Example:
http://www.youtube.com/watch?v=OpaiGYxkSuQ parallel bzip2 ::: *.txt
* 100% options complete with xargs. All options for xargs can now * xapply like functionality. Example:
be used in GNU Parallel - even the more exotic. parallel diff {1} {2} :::: filelist1 filelist2
* --basefile for transferring basedata. When running jobs on remote * Arch Linux package. Thanks to Peter Simons <simons at cryp dot to>
computers --basefile will transfer files before the first jobs is
run. It can be used to transfer data that remains the same for each
job such as scripts or lookup tables.
* --progress shows progress. To see how many jobs is running on each * Mandriva package. Thanks to Sandro Cazzaniga <kharec at mandriva dot
server use --progress. It can be turned on even after GNU Parallel org>
is started.
* --eta shows estimated time left in seconds.
* --halt-on-error stops if an error occurs. GNU Parallel will default
to run all jobs - even if some of them fail. With --halt-on-error
GNU Parallel can ignore errors, wait for the currently running jobs
to finish, or stop immediately when an error occurs.
* -L -l -n -s now implies -X unless -m is set
= About GNU Parallel = = About GNU Parallel =

View file

@ -1813,7 +1813,7 @@ reap_if_needed();
drain_job_queue(); drain_job_queue();
cleanup(); cleanup();
if($::opt_halt_on_error) { if($::opt_halt_on_error) {
wait_and_exit $Global::halt_on_error_exitstatus; wait_and_exit($Global::halt_on_error_exitstatus);
} else { } else {
wait_and_exit(min($Global::exitstatus,254)); wait_and_exit(min($Global::exitstatus,254));
} }
@ -1840,7 +1840,7 @@ sub parse_options {
$Global::halt_on_error_exitstatus = 0; $Global::halt_on_error_exitstatus = 0;
$Global::total_jobs = 0; $Global::total_jobs = 0;
$Global::arg_sep = ":::"; $Global::arg_sep = ":::";
$Global::arg_file_sep = ":::"; $Global::arg_file_sep = "::::";
Getopt::Long::Configure ("bundling","require_order"); Getopt::Long::Configure ("bundling","require_order");
# Add options from .parallelrc # Add options from .parallelrc

View file

@ -15,6 +15,14 @@ echo a
a a
echo b echo b
b b
echo a
a
echo b
b
echo a
a
echo b
b
### Test stdin goes to first command only ### Test stdin goes to first command only
cat - cat -
via first cat via first cat

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

View file

@ -0,0 +1,5 @@
one
two
EOF
three
four

View file

@ -0,0 +1,5 @@
one
two
bEOF
three
four

View file

@ -0,0 +1,5 @@
one
two
EOFe
three
four

View file

@ -0,0 +1,3 @@
sleep 2 && echo one
sleep 1 && echo two
echo three

View file

@ -0,0 +1,3 @@

View file

@ -0,0 +1,3 @@
foo '' bar
baz ''
'' fnord

View file

@ -0,0 +1,5 @@
firstline
secondline
_
thirdline
fourthline

Binary file not shown.

View file

@ -0,0 +1,5 @@
one
two
_
three
four

View file

@ -0,0 +1,5 @@
firstline
secondline
EOF
thirdline
fourthline

View file

@ -0,0 +1,3 @@
exit 255
echo 1
echo 2

View file

@ -0,0 +1,22 @@
/src/gnu/autoconf-1.11
/src/gnu/autoconf-1.11/README
/src/gnu/autoconf-1.11/Makefile.in
/src/gnu/autoconf-1.11/INSTALL
/src/gnu/autoconf-1.11/NEWS
/src/gnu/autoconf-1.11/COPYING
/src/gnu/autoconf-1.11/ChangeLog
/src/gnu/autoconf-1.11/autoconf.texi
/src/gnu/autoconf-1.11/acconfig.h
/src/gnu/autoconf-1.11/autoconf.sh
/src/gnu/autoconf-1.11/acgeneral.m4
/src/gnu/autoconf-1.11/acspecific.m4
/src/gnu/autoconf-1.11/configure
/src/gnu/autoconf-1.11/configure.in
/src/gnu/autoconf-1.11/autoheader.sh
/src/gnu/autoconf-1.11/mkinstalldirs
/src/gnu/autoconf-1.11/install.sh
/src/gnu/autoconf-1.11/autoconf.info
/src/gnu/autoconf-1.11/standards.texi
/src/gnu/autoconf-1.11/make-stds.texi
/src/gnu/autoconf-1.11/standards.info
/src/gnu/autoconf-1.11/texinfo.tex

Binary file not shown.

View file

@ -0,0 +1,3 @@
foo bar
baz
ugh

View file

@ -0,0 +1,2 @@
1 2 3 4
a b c d

View file

@ -0,0 +1,3 @@
false
echo 1
echo 2

View file

@ -0,0 +1 @@
hello world

Binary file not shown.

View file

@ -0,0 +1,8 @@
dumb
s s
f f
r r
t t
v v

Binary file not shown.

View file

@ -0,0 +1,9 @@
1 22 333 4444
55555 666666
7777777
88888888
999999999 1 22
333 4444 55555
666666 7777777
88888888
999999999

Binary file not shown.

View file

@ -0,0 +1,10 @@
1 22 333 4444
55555 666666
7777777
88888888
999999999 1 22
333 4444 55555
666666 7777777
88888888
999999999

View file

@ -0,0 +1,6 @@
first nonblank line
second nonblank line
third nonblank line
fourth nonblank line

Binary file not shown.

View file

@ -0,0 +1,2 @@
first
second

View file

@ -0,0 +1,5 @@
this is
"quoted stuff"
and \
an embedded newline
with 'single quotes' as well.

View file

@ -0,0 +1,2 @@

Binary file not shown.

View file

@ -0,0 +1,18 @@
1
22
333
4444
55555
666666
7777777
88888888
999999999
1
22
333
4444
55555
666666
7777777
88888888
999999999

Binary file not shown.

View file

@ -0,0 +1,18 @@
999999999
88888888
7777777
666666
55555
4444
333
22
1
999999999
88888888
7777777
666666
55555
4444
333
22
1

View file

@ -0,0 +1,3 @@
kill $$
echo 1
echo 2

View file

@ -0,0 +1,3 @@
1
2
3

View file

@ -0,0 +1,3 @@
one
"two
three

View file

@ -0,0 +1,2 @@
one
"two

View file

@ -0,0 +1 @@
q w e r 6 7 8 9

View file

@ -7,6 +7,8 @@ parallel -kv ::: 'echo a' 'echo b'
echo '### Change --arg-sep' echo '### Change --arg-sep'
parallel --arg-sep ::: -kv ::: 'echo a' 'echo b' parallel --arg-sep ::: -kv ::: 'echo a' 'echo b'
parallel --arg-sep .--- -kv .--- 'echo a' 'echo b' parallel --arg-sep .--- -kv .--- 'echo a' 'echo b'
parallel --argsep ::: -kv ::: 'echo a' 'echo b'
parallel --argsep .--- -kv .--- 'echo a' 'echo b'
echo '### Test stdin goes to first command only' echo '### Test stdin goes to first command only'
echo via first cat |parallel -kv cat ::: - - echo via first cat |parallel -kv cat ::: - -
echo via cat |parallel --arg-sep .--- -kv .--- 'cat' 'echo b' echo via cat |parallel --arg-sep .--- -kv .--- 'cat' 'echo b'

View file

@ -1,5 +1,12 @@
#!/bin/bash #!/bin/bash
echo '### Test ::::'
echo '### Change --arg-file-sep'
parallel --arg-file-sep :::: -k echo {1} {2} :::: <(seq 1 10) <(seq 5 15)
parallel --arg-file-sep .--- -k echo {1} {2} .--- <(seq 1 10) <(seq 5 15)
parallel --argfilesep :::: -k echo {1} {2} :::: <(seq 1 10) <(seq 5 15)
parallel --argfilesep .--- -k echo {1} {2} .--- <(seq 1 10) <(seq 5 15)
echo '### Test xapply --max-replace-args' echo '### Test xapply --max-replace-args'
seq 0 7 | parallel -k --max-replace-args=3 echo {3} {2} {1} seq 0 7 | parallel -k --max-replace-args=3 echo {3} {2} {1}
echo '### Test -N' echo '### Test -N'

View file

@ -0,0 +1,186 @@
#!/bin/bash -x
echo '### Tests from xargs'
# Largely untested
rsync -Ha --delete input-files/xargs-inputs/ tmp/
cd tmp
stdout xargs -0 -n3 echo < files0.xi
stdout parallel -k -0 -n3 echo < files0.xi
stdout xargs -d o -n1 echo < helloworld.xi
stdout parallel -k -d o -n1 echo < helloworld.xi
stdout xargs -E_ -0 echo < eof_-0.xi
stdout parallel -k -E_ -0 echo < eof_-0.xi
stdout xargs -i -0 echo from \{\} to x{}y < items-0.xi
stdout parallel -k -i -0 echo from \{\} to x{}y < items-0.xi
stdout xargs -i -s26 -0 echo from \{\} to x{}y < items-0.xi
stdout parallel -k -i -s26 -0 echo from \{\} to x{}y < items-0.xi
stdout xargs -l -0 echo < ldata-0.xi
stdout parallel -k -l -0 echo < ldata-0.xi
stdout xargs -l -0 echo < ldatab-0.xi
stdout parallel -k -l -0 echo < ldatab-0.xi
stdout xargs -L2 -0 echo < ldata-0.xi
stdout parallel -k -L2 -0 echo < ldata-0.xi
stdout xargs -L2 -0 echo < ldatab-0.xi
stdout parallel -k -L2 -0 echo < ldatab-0.xi
stdout xargs -L3 -0 echo < ldata-0.xi
stdout parallel -k -L3 -0 echo < ldata-0.xi
stdout xargs -n1 -0 echo < stairs-0.xi
stdout parallel -k -n1 -0 echo < stairs-0.xi
stdout xargs -n2 -0 echo < stairs-0.xi
stdout parallel -k -n2 -0 echo < stairs-0.xi
stdout xargs -n2 -s26 -0 echo < stairs-0.xi
stdout parallel -k -n2 -s26 -0 echo < stairs-0.xi
stdout xargs -n2 -s26 -x -0 echo < stairs-0.xi
stdout parallel -k -n2 -s26 -x -0 echo < stairs-0.xi
stdout xargs -n3 -0 echo < stairs2-0.xi
stdout parallel -k -n3 -0 echo < stairs2-0.xi
stdout xargs -n3 -s36 -0 echo < stairs2-0.xi
stdout parallel -k -n3 -s36 -0 echo < stairs2-0.xi
stdout xargs -0 echo < noeof-0.xi
stdout parallel -k -0 echo < noeof-0.xi
stdout xargs -P3 -n1 -IARG sh -c ARG < Pdata.xi
stdout parallel -k -P3 -n1 -IARG sh -c ARG < Pdata.xi
stdout xargs -r echo this plus that < blank.xi
stdout parallel -k -r echo this plus that < blank.xi
stdout xargs -0 -s118 echo < stairs-0.xi
stdout parallel -k -0 -s118 echo < stairs-0.xi
stdout xargs -0 -s19 echo < stairs-0.xi
stdout parallel -k -0 -s19 echo < stairs-0.xi
stdout xargs -0 -s19 echo < stairs2-0.xi
stdout parallel -k -0 -s19 echo < stairs2-0.xi
stdout xargs -0 -s20 echo < stairs-0.xi
stdout parallel -k -0 -s20 echo < stairs-0.xi
stdout xargs -0 -s30 echo < stairs-0.xi
stdout parallel -k -0 -s30 echo < stairs-0.xi
stdout xargs -0 echo this plus that < space.xi
stdout parallel -k -0 echo this plus that < space.xi
stdout xargs -r echo this plus that < space.xi
stdout parallel -k -r echo this plus that < space.xi
stdout xargs -t -0 echo this plus that < space.xi
stdout parallel -k -t -0 echo this plus that < space.xi
stdout xargs true < 32767-ys.xi
stdout parallel -k true < 32767-ys.xi
stdout xargs true < 16383-ys.xi
stdout parallel -k true < 16383-ys.xi
stdout xargs -E EOF echo < EOFb.xi
stdout parallel -k -E EOF echo < EOFb.xi
stdout xargs -E EOF echo < EOFe.xi
stdout parallel -k -E EOF echo < EOFe.xi
stdout xargs -E EOF echo < EOF.xi
stdout parallel -k -E EOF echo < EOF.xi
stdout xargs -E_ echo < eof_.xi
stdout parallel -k -E_ echo < eof_.xi
stdout xargs -E_ -IARG echo from ARG to xARGy < eof_.xi
stdout parallel -k -E_ -IARG echo from ARG to xARGy < eof_.xi
stdout xargs -s470 echo hi there < files.xi
stdout parallel -k -s470 echo hi there < files.xi
stdout xargs -IARG echo from ARG to xARGy -E_ < eof_.xi
stdout parallel -k -IARG echo from ARG to xARGy -E_ < eof_.xi
stdout xargs -IARG echo from ARG to xARGy < items.xi
stdout parallel -k -IARG echo from ARG to xARGy < items.xi
stdout xargs -IARG -s15 echo ARG < stairs.xi
stdout parallel -k -IARG -s15 echo ARG < stairs.xi
stdout xargs -L2 echo < ldatab.xi
stdout parallel -k -L2 echo < ldatab.xi
stdout xargs -L2 -n2 echo < ldata.xi
stdout parallel -k -L2 -n2 echo < ldata.xi
stdout xargs -L3 echo < ldata.xi
stdout parallel -k -L3 echo < ldata.xi
stdout xargs -n1 < stairs.xi
stdout parallel -k -n1 echo < stairs.xi
stdout xargs -n2 echo < stairs.xi
stdout parallel -k -n2 echo < stairs.xi
stdout xargs -n2 -s26 echo < stairs.xi
stdout parallel -k -n2 -s26 echo < stairs.xi
stdout xargs -n2 -s26 -x echo < stairs.xi
stdout parallel -k -n2 -s26 -x echo < stairs.xi
stdout xargs -n3 echo < files.xi
stdout parallel -k -n3 echo < files.xi
stdout xargs -n3 -s36 echo < stairs2.xi
stdout parallel -k -n3 -s36 echo < stairs2.xi
stdout xargs echo < noeof.xi
stdout parallel -k echo < noeof.xi
stdout xargs echo < quotes.xi
stdout parallel -k echo < quotes.xi
stdout xargs -s118 echo < stairs.xi
stdout parallel -k -s118 echo < stairs.xi
stdout xargs -s19 echo < stairs2.xi
stdout parallel -k -s19 echo < stairs2.xi
stdout xargs -s19 echo < stairs.xi
stdout parallel -k -s19 echo < stairs.xi
stdout xargs -s20 echo < stairs.xi
stdout parallel -k -s20 echo < stairs.xi
stdout xargs -s30 echo < stairs.xi
stdout parallel -k -s30 echo < stairs.xi
stdout xargs -s470 echo < files.xi
stdout parallel -k -s470 echo < files.xi
stdout xargs -s47 echo < files.xi
stdout parallel -k -s47 echo < files.xi
stdout xargs -s48 echo < files.xi
stdout parallel -k -s48 echo < files.xi
stdout xargs -s6 echo < files.xi
stdout parallel -k -s6 echo < files.xi
stdout xargs -iARG -s86 echo ARG is xARGx < files.xi
stdout parallel -k -iARG -s86 echo ARG is xARGx < files.xi
stdout xargs echo this plus that < space.xi
stdout parallel -k echo this plus that < space.xi
stdout xargs -IARG echo from ARG to xARGy < space.xi
stdout parallel -k -IARG echo from ARG to xARGy < space.xi
stdout xargs printf "\[%s\]\n" < verticaltabs.xi
stdout parallel -k printf "\[%s\]\n" < verticaltabs.xi
stdout xargs printf "\[%s\]\n" < formfeeds.xi
stdout parallel -k printf "\[%s\]\n" < formfeeds.xi
stdout xargs -L2 echo < ldata.xi
stdout parallel -k -L2 echo < ldata.xi
stdout xargs echo < unmatched2.xi
stdout parallel -k echo < unmatched2.xi
stdout xargs echo < unmatched.xi
stdout parallel -k echo < unmatched.xi
stdout xargs -n2 -x < unmatched.xi
stdout parallel -k -n2 -x < unmatched.xi
stdout xargs -eEOF < eofstr.xi
stdout parallel -k -eEOF < eofstr.xi
stdout xargs -e < eof_.xi
stdout parallel -k -e < eof_.xi
stdout xargs -E_ < eof1.xi
stdout parallel -k -E_ < eof1.xi
stdout xargs -iARG echo ARG is xARGx < files.xi
stdout parallel -k -iARG echo ARG is xARGx < files.xi
stdout xargs -i echo from \{\} to x{}y < items.xi
stdout parallel -k -i echo from \{\} to x{}y < items.xi
stdout xargs -i -s26 echo from \{\} to x{}y < items.xi
stdout parallel -k -i -s26 echo from \{\} to x{}y < items.xi
stdout xargs -i__ echo FIRST __ IS OK < quotes.xi
stdout parallel -k -i__ echo FIRST __ IS OK < quotes.xi
stdout xargs -l < ldatab.xi
stdout parallel -k -l < ldatab.xi
stdout xargs -l < ldata.xi
stdout parallel -k -l < ldata.xi
stdout xargs -l1 -n4 < files.xi
stdout parallel -k -l1 -n4 < files.xi
stdout xargs -l2 < files.xi
stdout parallel -k -l2 < files.xi
stdout xargs -s30 -t < stairs.xi
stdout parallel -k -s30 -t < stairs.xi
stdout xargs -t echo this plus that < space.xi
stdout parallel -k -t echo this plus that < space.xi
stdout xargs -n1 printf "@%s@\n" < empty.xi
stdout parallel -k -n1 printf "@%s@\n" < empty.xi
stdout xargs -n2 -t echo < foobar.xi
stdout parallel -k -n2 -t echo < foobar.xi
#
# xargs_start 123 {-n1 -IARG sh -c ARG} ftt.xi
# xargs_start 124 {-n1 -IARG sh -c ARG} ett.xi
# xargs_start 125 {-n1 -IARG sh -c ARG} stt.xi
# xargs_start 123 { false } files.xi
# xargs_start p {-t echo this plus that}
# xargs_start p "" sv-bug-20273.xi "" "sh -c \{$XARGS $XARGSFLAGS -E2; cat\}"
# xargs_start p {echo this plus that}
# xargs_start p {}
# xargs_start p {-r}
# xargs_start p {-r echo this plus that}
# xargs_start p {echo this plus that} {}
# xargs_start p {-t}

View file

@ -15,6 +15,14 @@ echo a
a a
echo b echo b
b b
echo a
a
echo b
b
echo a
a
echo b
b
### Test stdin goes to first command only ### Test stdin goes to first command only
cat - cat -
via first cat via first cat

View file

@ -1,3 +1,49 @@
### Test ::::
### Change --arg-file-sep
1 5
2 6
3 7
4 8
5 9
6 10
7 11
8 12
9 13
10 14
15
1 5
2 6
3 7
4 8
5 9
6 10
7 11
8 12
9 13
10 14
15
1 5
2 6
3 7
4 8
5 9
6 10
7 11
8 12
9 13
10 14
15
1 5
2 6
3 7
4 8
5 9
6 10
7 11
8 12
9 13
10 14
15
### Test xapply --max-replace-args ### Test xapply --max-replace-args
2 1 0 2 1 0
5 4 3 5 4 3
@ -94,3 +140,6 @@ b a
### Multiple -a: nonexistent ### Multiple -a: nonexistent
parallel: Cannot open input file `nonexist': No such file or directory parallel: Cannot open input file `nonexist': No such file or directory
parallel: Cannot open input file `nonexist': No such file or directory parallel: Cannot open input file `nonexist': No such file or directory
### Test {#.}
echo c-ext b-withext a-noext
c-ext b-withext a-noext