=head1 search terms

run commands in parallel

Parallel shell loops

multi threading in bash xargs

# TAGS: parallel | parallel processing | multicore | multiprocessor | Clustering/Distributed Networks
# job control | multiple jobs | parallelization | text processing | cluster | filters
# Clustering Tools | Command Line Tools | Utilities | System Administration
# Bash parallel

GNU parallel execution shell bash script simultaneous concurrent linux
scripting run xargs ppss code.google.com/p/ppss/

@vvuksan @ychaker @ncb000gt 
xargs can lead to nasty surprises caused by the separator problem
http://nd.gd/0t GNU Parallel http://nd.gd/0s may be better.

Comments:

http://pi.dk/0 https://www.gnu.org/software/parallel/
http://pi.dk/1 https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
http://pi.dk/2 https://savannah.gnu.org/news/?group=parallel

http://pi.dk/5 https://en.wikipedia.org/wiki/Xargs#The_separator_problem
http://pi.dk/6 https://www.gnu.org/software/parallel/man.html#differences_between_xargs_and_gnu_parallel
http://pi.dk/7 https://www.gnu.org/software/parallel/man.html#example__distributing_work_to_local_and_remote_computers

If you like xargs you may love GNU Parallel: http://pi.dk/1

With GNU Parallel (http://pi.dk/0) you can do:
ls | grep jpeg | parallel mv {} {.}.jpg

Watch the intro video for GNU Parallel: http://pi.dk/1

If your input file names are generated by users, you need to deal with
surprising file names containing space, ', or " in the filename. 

xargs can give nasty surprises due to the separator problem
http://pi.dk/5

@jaylyerly @stevenf xargs will bite you if file names contain
space http://pi.dk/5. Use GNU Parallel instead: http://pi.dk/0

Please repay by spreading the word about GNU Parallel to your
contacts/blog/facebook/linkedin/mailing lists/user group

Your use of xargs can lead to nasty surprises because of the separator
problem http://en.wikipedia.org/wiki/Xargs#The_separator_problem

GNU Parallel http://www.gnu.org/software/parallel/ does not have that
problem.

If you have GNU Parallel http://www.gnu.org/software/parallel/ installed you can do this:


You can install GNU Parallel simply by:

    wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
    chmod 755 parallel
    cp parallel sem

Watch the intro videos for GNU Parallel to learn more:
https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1


GNU Parallel also makes it possible to run small scripts. Try this:

ls *.zip | parallel 'mkdir {.}; cd {.}; unzip ../{}'