2010-12-02 20:58:10 +00:00
|
|
|
=head1 search terms
|
|
|
|
|
|
|
|
|
|
run commands in parallel
|
|
|
|
|
|
|
|
|
|
Parallel shell loops
|
|
|
|
|
|
|
|
|
|
multi threading in bash xargs
|
|
|
|
|
|
|
|
|
|
# TAGS: parallel | parallel processing | multicore | multiprocessor | Clustering/Distributed Networks
|
|
|
|
|
# job control | multiple jobs | parallelization | text processing | cluster | filters
|
|
|
|
|
# Clustering Tools | Command Line Tools | Utilities | System Administration
|
|
|
|
|
# Bash parallel
|
|
|
|
|
|
|
|
|
|
GNU parallel execution shell bash script simultaneous concurrent linux
|
|
|
|
|
scripting run xargs ppss code.google.com/p/ppss/
|
|
|
|
|
|
|
|
|
|
@vvuksan @ychaker @ncb000gt
|
|
|
|
|
xargs can lead to nasty surprises caused by the separator problem
|
|
|
|
|
http://nd.gd/0t GNU Parallel http://nd.gd/0s may be better.
|
|
|
|
|
|
|
|
|
|
Comments:
|
|
|
|
|
|
2012-01-22 14:06:19 +00:00
|
|
|
http://pi.dk/0 https://www.gnu.org/software/parallel/
|
|
|
|
|
http://pi.dk/1 https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
|
|
|
|
|
http://pi.dk/2 https://savannah.gnu.org/news/?group=parallel
|
2010-12-02 20:58:10 +00:00
|
|
|
|
2012-01-22 14:06:19 +00:00
|
|
|
http://pi.dk/5 https://en.wikipedia.org/wiki/Xargs#The_separator_problem
|
|
|
|
|
http://pi.dk/6 https://www.gnu.org/software/parallel/man.html#differences_between_xargs_and_gnu_parallel
|
|
|
|
|
http://pi.dk/7 https://www.gnu.org/software/parallel/man.html#example__distributing_work_to_local_and_remote_computers
|
2010-12-02 23:04:12 +00:00
|
|
|
|
2012-01-22 14:06:19 +00:00
|
|
|
If you like xargs you may love GNU Parallel: http://pi.dk/1
|
|
|
|
|
|
|
|
|
|
With GNU Parallel (http://pi.dk/0) you can do:
|
2010-12-02 23:04:12 +00:00
|
|
|
ls | grep jpeg | parallel mv {} {.}.jpg
|
|
|
|
|
|
2012-01-22 14:06:19 +00:00
|
|
|
Watch the intro video for GNU Parallel: http://pi.dk/1
|
2010-12-02 20:58:10 +00:00
|
|
|
|
|
|
|
|
If your input file names are generated by users, you need to deal with
|
|
|
|
|
surprising file names containing space, ', or " in the filename.
|
|
|
|
|
|
|
|
|
|
xargs can give nasty surprises due to the separator problem
|
2012-01-22 14:06:19 +00:00
|
|
|
http://pi.dk/5
|
2010-12-02 20:58:10 +00:00
|
|
|
|
|
|
|
|
@jaylyerly @stevenf xargs will bite you if file names contain
|
2012-01-22 14:06:19 +00:00
|
|
|
space http://pi.dk/5. Use GNU Parallel instead: http://pi.dk/0
|
2010-12-02 20:58:10 +00:00
|
|
|
|
|
|
|
|
Please repay by spreading the word about GNU Parallel to your
|
|
|
|
|
contacts/blog/facebook/linkedin/mailing lists/user group
|
|
|
|
|
|
|
|
|
|
Your use of xargs can lead to nasty surprises because of the separator
|
|
|
|
|
problem http://en.wikipedia.org/wiki/Xargs#The_separator_problem
|
|
|
|
|
|
|
|
|
|
GNU Parallel http://www.gnu.org/software/parallel/ does not have that
|
|
|
|
|
problem.
|
|
|
|
|
|
|
|
|
|
If you have GNU Parallel http://www.gnu.org/software/parallel/ installed you can do this:
|
|
|
|
|
|
|
|
|
|
|
2011-12-09 22:25:20 +00:00
|
|
|
You can install GNU Parallel simply by:
|
2010-12-02 20:58:10 +00:00
|
|
|
|
2011-12-09 22:25:20 +00:00
|
|
|
wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
|
|
|
|
|
chmod 755 parallel
|
|
|
|
|
cp parallel sem
|
2010-12-02 20:58:10 +00:00
|
|
|
|
2012-01-22 03:42:05 +00:00
|
|
|
Watch the intro videos for GNU Parallel to learn more:
|
|
|
|
|
https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
|
2011-12-09 22:25:20 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
GNU Parallel also makes it possible to run small scripts. Try this:
|
2010-12-02 20:58:10 +00:00
|
|
|
|
2011-12-09 22:25:20 +00:00
|
|
|
ls *.zip | parallel 'mkdir {.}; cd {.}; unzip ../{}'
|
2010-12-02 20:58:10 +00:00
|
|
|
|
