parallel: EXAMPLE: Start editor with filenames from stdin (standard input).

parallel: EXAMPLE: GNU Parallel as queue system/batch manager.
This commit is contained in:
Ole Tange 2010-10-05 22:22:52 +02:00
parent a8d4668fb6
commit 692bb88453
3 changed files with 85 additions and 28 deletions

View file

@ -75,6 +75,38 @@ find music-files -type f | parallel -j+0 -S :,server1,server2 \
# sem
# --retry
(echo a1.txt; echo b1.txt; echo c1.txt; echo a2.txt; echo b2.txt; echo c2.txt;)| \
parallel -X -N 3 my-program --file={}
(echo a1.txt; echo b1.txt; echo c1.txt; echo d1.txt; echo e1.txt; echo f1.txt;)| \
parallel -X my-program --file={}
# First job controls the tty
# -u needed because output should not be saved for later
find . -type f | parallel -uXj1 vim
find . -type f | parallel -uXj1 emacs
# If you have 1000 files only one contains 'foobar'
# stop when this one is found
find . -type f | parallel grep -l foobar | head -1
# To test a list of hosts are up and pingable save this
# to a file called machinesup
#!/usr/bin/parallel --shebang --no-run-if-empty ping -c 3 {} >/dev/null 2>&1
google.com
yahoo.com
nowhere.gone
# Then:
# chmod 755 machinesup
# ./machinesup || echo Some machines are down
=head1 YouTube video

View file

@ -116,38 +116,26 @@ cc:Peter Simons <simons@cryp.to>, Sandro Cazzaniga <kharec@mandriva.org>,
Tim Cuthbertson <tim3d.junk@gmail.com>, Ludovic Courtès <ludo@gnu.org>,
Markus Ammer <mkmm@gmx-topmail.de>, Pavel Nuzhdin <pnzhdin@gmail.com>,
Phil Sung <psung@alum.mit.edu>, Michael Shigorin <mike@altlinux.org>,
Andrew McFague <amcfague@wgen.net>, Steven M. Christensen <sunfreeware@gmail.com>
Andrew McFague <amcfague@wgen.net>, Steven M. Christensen <sunfreeware@gmail.com>,
Chris Howey <howeyc@gmail.com>
Subject: GNU Parallel 20100922 released
Subject: GNU Parallel 2010XXXX released
GNU Parallel 20100922 has been released. It is available for
GNU Parallel 2010XXXX has been released. It is available for
download at: http://ftp.gnu.org/gnu/parallel/
New in this release:
* FreeBSD port. Thanks to Chris Howey <howeyc at gmail dot com>
* First review on print:
http://www.linux-magazine.com/Issues/2010 Nov 2010
Old in this release:
* See GNU Parallel live at FSCONS 2010-11-07:
http://www.fscons.org/fs/gnu-parallel
* Untested Debian and xUbuntu packages available through OpenSUSE
build service:
https://build.opensuse.org/package/show?package=parallel&project=home%3Atange
* Using --retries a job will be retried on another computer if it
fails. This is useful if some jobs fail for no apparent reason (such
as network failure).
* BSD xargs -o (open /dev/tty) is now default for the job running in
foreground. Useful for interactive commands like:
ls | parallel -Xuj1 vi
* GNU sql now supports SQLite.
* Renamed .dburl.aliases to .sql/aliases and /etc/sql/aliases.
* GNU sql now support --list-tables
* Alias for DBURL can contain '?query' part with %-quoting.
= About GNU Parallel =

View file

@ -462,7 +462,8 @@ with all the arguments.
Support for B<-m> with B<--sshlogin> is limited and may fail.
See also B<-X> for context replace.
See also B<-X> for context replace. If in doubt use B<-X> as that will
most likely do what is needed.
=item B<--progress>
@ -843,13 +844,19 @@ Implies B<--semaphore>.
=item B<-X>
xargs with context replace. This works like B<-m> except if B<{}> is part
of a word (like I<pic{}.jpg>) then the whole word will be
repeated. Normally B<-X> will do the right thing, whereas B<-m> can
give surprising results if B<{}> is used as part of a word.
Multiple arguments with context replace. Insert as many arguments as
the command line length permits. If B<{}> is not used the arguments
will be appended to the line. If B<{}> is used as part of a word
(like I<pic{}.jpg>) then the whole word will be repeated. If B<{}> is
used multiple times each B<{}> will be replaced with the arguments.
Normally B<-X> will do the right thing, whereas B<-m> can give
unexpected results if B<{}> is used as part of a word.
Support for B<-X> with B<--sshlogin> is limited and may fail.
See also B<-m>.
=item B<--exit>
@ -1370,6 +1377,36 @@ same time:
seq 1 3 | parallel sem --id mymutex sed -i -e 'i{}' myfile
=head1 EXAMPLE: Start editor with filenames from stdin (standard input)
You can use GNU Parallel to start interactive programs like emacs or vi:
B<cat filelist | parallel -uXj1 emacs>
B<cat filelist | parallel -uXj1 vi>
If there are more files than will fit on a single command line, the
editor will be started again with the remaining files.
=head1 EXAMPLE: GNU Parallel as queue system/batch manager
GNU Parallel can work as a simple job queue system or batch manager.
The idea is to put the jobs into a file and have GNU Parallel read
from that continuously. As GNU Parallel will stop at end of file we
use tail to continue reading:
B<echo >>B<jobqueue>; B<tail -f jobqueue | parallel>
To submit your jobs to the queue:
B<echo my_command my_arg >>>B< jobqueue>
You can of course use B<-S> to distribute the jobs to remote
computers:
B<echo >>B<jobqueue>; B<tail -f jobqueue | parallel -S ..>
=head1 QUOTING
For more advanced use quoting may be an issue. The following will