=head1 search terms

run commands in parallel

Parallel shell loops

multi threading in bash xargs

# TAGS: parallel | parallel processing | multicore | multiprocessor | Clustering/Distributed Networks
# job control | multiple jobs | parallelization | text processing | cluster | filters
# Clustering Tools | Command Line Tools | Utilities | System Administration
# Bash parallel

GNU parallel execution shell bash script simultaneous concurrent linux
scripting run xargs ppss code.google.com/p/ppss/

@vvuksan @ychaker @ncb000gt 
xargs can lead to nasty surprises caused by the separator problem
http://nd.gd/0t GNU Parallel http://nd.gd/0s may be better.

Comments:

http://nd.gd/0u http://www.gnu.org/software/parallel/
http://nd.gd/0s http://www.youtube.com/watch?v=OpaiGYxkSuQ
http://nd.gd/0t http://en.wikipedia.org/wiki/Xargs#The_separator_problem
http://nd.gd/3k http://www.gnu.org/software/parallel/man.html#differences_between_xargs_and_gnu_parallel

Watch the intro video for GNU Parallel: http://nd.gd/0s

If your input file names are generated by users, you need to deal with
surprising file names containing space, ', or " in the filename. 

xargs can give nasty surprises due to the separator problem
http://en.wikipedia.org/wiki/Xargs#The_separator_problem
http://nd.gd/0t

GNU Parallel http://nd.gd/0s may be better.

@jaylyerly @stevenf xargs will bite you if file names contain
space http://nd.gd/0t. Use GNU Parallel instead: http://nd.gd/0s

Please repay by spreading the word about GNU Parallel to your
contacts/blog/facebook/linkedin/mailing lists/user group

Your use of xargs can lead to nasty surprises because of the separator
problem http://en.wikipedia.org/wiki/Xargs#The_separator_problem

GNU Parallel http://www.gnu.org/software/parallel/ does not have that
problem.

Watch the intro video for GNU Parallel:
http://www.youtube.com/watch?v=OpaiGYxkSuQ

If you have GNU Parallel http://www.gnu.org/software/parallel/ installed you can do this:

Watch the intro video for GNU Parallel to learn more:
http://www.youtube.com/watch?v=OpaiGYxkSuQ


GNU Parallel also makes it possible to run small scripts. Try this:

ls *.zip | parallel ‘mkdir {.}; cd {.}; unzip ../{}’

= Other people says =

fawxtin‎: Using GNU Parallel to run things simultaneously, neat! ;)
cowsandmilk‎: if you haven't quit xargs for gnu parallel yet, you're really missing out
afader‎: GNU Parallel is cool
ecyrd‎: Okay, why hasn't anyone told me of GNU parallel before?
cashion‎: very cool! Did not know about GNU parallel.
toorghezi‎: Writing shell scripts? You should know about GNU Parallel!
Racecar564 : GNU Parallel rocks!
arclight : Very nice! I especially like the {} -> {.}.bz2 syntax.
nwp8861 : I surprised that it's so comfortable!