=head1 search terms

run commands in parallel

Parallel shell loops

multi threading in bash xargs

# TAGS: parallel | parallel processing | multicore | multiprocessor | Clustering/Distributed Networks
# job control | multiple jobs | parallelization | text processing | cluster | filters
# Clustering Tools | Command Line Tools | Utilities | System Administration
# Bash parallel

GNU parallel execution shell bash script simultaneous concurrent linux
scripting run xargs ppss code.google.com/p/ppss/

@vvuksan @ychaker @ncb000gt 
xargs can lead to nasty surprises caused by the separator problem
http://nd.gd/0t GNU Parallel http://nd.gd/0s may be better.

Comments:

http://nd.gd/0u http://www.gnu.org/software/parallel/
http://nd.gd/0s http://www.youtube.com/watch?v=OpaiGYxkSuQ
http://nd.gd/0t http://en.wikipedia.org/wiki/Xargs#The_separator_problem
http://nd.gd/3k http://www.gnu.org/software/parallel/man.html#differences_between_xargs_and_gnu_parallel
http://nd.gd/po http://www.gnu.org/software/parallel/man.html#example__distributing_work_to_local_and_remote_computers
http://nd.gd/039 http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

If you like xargs you may love GNU Parallel: http://nd.gd/0s

With GNU Parallel (http://nd.gd/0s) you can do:
ls | grep jpeg | parallel mv {} {.}.jpg

Watch the intro video for GNU Parallel: http://nd.gd/0s

If your input file names are generated by users, you need to deal with
surprising file names containing space, ', or " in the filename. 

xargs can give nasty surprises due to the separator problem
http://en.wikipedia.org/wiki/Xargs#The_separator_problem
http://nd.gd/0t

GNU Parallel http://nd.gd/0s may be better.

@jaylyerly @stevenf xargs will bite you if file names contain
space http://nd.gd/0t. Use GNU Parallel instead: http://nd.gd/0s

Please repay by spreading the word about GNU Parallel to your
contacts/blog/facebook/linkedin/mailing lists/user group

Your use of xargs can lead to nasty surprises because of the separator
problem http://en.wikipedia.org/wiki/Xargs#The_separator_problem

GNU Parallel http://www.gnu.org/software/parallel/ does not have that
problem.

If you have GNU Parallel http://www.gnu.org/software/parallel/ installed you can do this:


You can install GNU Parallel simply by:

    wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
    chmod 755 parallel
    cp parallel sem

Watch the intro videos for GNU Parallel to learn more: http://nd.gd/039


GNU Parallel also makes it possible to run small scripts. Try this:

ls *.zip | parallel 'mkdir {.}; cd {.}; unzip ../{}'