2010-12-02 20:58:10 +00:00
|
|
|
|
=head1 search terms
|
|
|
|
|
|
|
|
|
|
|
|
run commands in parallel
|
|
|
|
|
|
|
|
|
|
|
|
Parallel shell loops
|
|
|
|
|
|
|
|
|
|
|
|
multi threading in bash xargs
|
|
|
|
|
|
|
|
|
|
|
|
# TAGS: parallel | parallel processing | multicore | multiprocessor | Clustering/Distributed Networks
|
|
|
|
|
|
# job control | multiple jobs | parallelization | text processing | cluster | filters
|
|
|
|
|
|
# Clustering Tools | Command Line Tools | Utilities | System Administration
|
|
|
|
|
|
# Bash parallel
|
|
|
|
|
|
|
|
|
|
|
|
GNU parallel execution shell bash script simultaneous concurrent linux
|
|
|
|
|
|
scripting run xargs ppss code.google.com/p/ppss/
|
|
|
|
|
|
|
|
|
|
|
|
@vvuksan @ychaker @ncb000gt
|
|
|
|
|
|
xargs can lead to nasty surprises caused by the separator problem
|
|
|
|
|
|
http://nd.gd/0t GNU Parallel http://nd.gd/0s may be better.
|
|
|
|
|
|
|
|
|
|
|
|
Comments:
|
|
|
|
|
|
|
|
|
|
|
|
http://nd.gd/0u http://www.gnu.org/software/parallel/
|
|
|
|
|
|
http://nd.gd/0s http://www.youtube.com/watch?v=OpaiGYxkSuQ
|
|
|
|
|
|
http://nd.gd/0t http://en.wikipedia.org/wiki/Xargs#The_separator_problem
|
|
|
|
|
|
http://nd.gd/3k http://www.gnu.org/software/parallel/man.html#differences_between_xargs_and_gnu_parallel
|
2011-06-16 12:47:55 +00:00
|
|
|
|
http://nd.gd/po http://www.gnu.org/software/parallel/man.html#example__distributing_work_to_local_and_remote_computers
|
2010-12-02 20:58:10 +00:00
|
|
|
|
|
2010-12-02 23:04:12 +00:00
|
|
|
|
If you like xargs you may love GNU Parallel: http://nd.gd/0s
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
With GNU Parallel (http://nd.gd/0s) you can do:
|
|
|
|
|
|
ls | grep jpeg | parallel mv {} {.}.jpg
|
|
|
|
|
|
|
2010-12-02 20:58:10 +00:00
|
|
|
|
Watch the intro video for GNU Parallel: http://nd.gd/0s
|
|
|
|
|
|
|
|
|
|
|
|
If your input file names are generated by users, you need to deal with
|
|
|
|
|
|
surprising file names containing space, ', or " in the filename.
|
|
|
|
|
|
|
|
|
|
|
|
xargs can give nasty surprises due to the separator problem
|
|
|
|
|
|
http://en.wikipedia.org/wiki/Xargs#The_separator_problem
|
|
|
|
|
|
http://nd.gd/0t
|
|
|
|
|
|
|
|
|
|
|
|
GNU Parallel http://nd.gd/0s may be better.
|
|
|
|
|
|
|
|
|
|
|
|
@jaylyerly @stevenf xargs will bite you if file names contain
|
|
|
|
|
|
space http://nd.gd/0t. Use GNU Parallel instead: http://nd.gd/0s
|
|
|
|
|
|
|
|
|
|
|
|
Please repay by spreading the word about GNU Parallel to your
|
|
|
|
|
|
contacts/blog/facebook/linkedin/mailing lists/user group
|
|
|
|
|
|
|
|
|
|
|
|
Your use of xargs can lead to nasty surprises because of the separator
|
|
|
|
|
|
problem http://en.wikipedia.org/wiki/Xargs#The_separator_problem
|
|
|
|
|
|
|
|
|
|
|
|
GNU Parallel http://www.gnu.org/software/parallel/ does not have that
|
|
|
|
|
|
problem.
|
|
|
|
|
|
|
|
|
|
|
|
Watch the intro video for GNU Parallel:
|
|
|
|
|
|
http://www.youtube.com/watch?v=OpaiGYxkSuQ
|
|
|
|
|
|
|
|
|
|
|
|
If you have GNU Parallel http://www.gnu.org/software/parallel/ installed you can do this:
|
|
|
|
|
|
|
|
|
|
|
|
Watch the intro video for GNU Parallel to learn more:
|
|
|
|
|
|
http://www.youtube.com/watch?v=OpaiGYxkSuQ
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
GNU Parallel also makes it possible to run small scripts. Try this:
|
|
|
|
|
|
|
|
|
|
|
|
ls *.zip | parallel ‘mkdir {.}; cd {.}; unzip ../{}’
|
|
|
|
|
|
|
|
|
|
|
|
= Other people says =
|
|
|
|
|
|
|
|
|
|
|
|
fawxtin: Using GNU Parallel to run things simultaneously, neat! ;)
|
|
|
|
|
|
cowsandmilk: if you haven't quit xargs for gnu parallel yet, you're really missing out
|
|
|
|
|
|
afader: GNU Parallel is cool
|
|
|
|
|
|
ecyrd: Okay, why hasn't anyone told me of GNU parallel before?
|
|
|
|
|
|
cashion: very cool! Did not know about GNU parallel.
|
|
|
|
|
|
toorghezi: Writing shell scripts? You should know about GNU Parallel!
|
|
|
|
|
|
Racecar564 : GNU Parallel rocks!
|
|
|
|
|
|
arclight : Very nice! I especially like the {} -> {.}.bz2 syntax.
|
|
|
|
|
|
nwp8861 : I surprised that it's so comfortable!
|
