mirror of
https://git.savannah.gnu.org/git/parallel.git
synced 2024-11-22 05:57:54 +00:00
370 lines
8.7 KiB
Plaintext
370 lines
8.7 KiB
Plaintext
#!/usr/bin/perl -w
|
|
|
|
=head1 NAME
|
|
|
|
sem - semaphore for executing shell command lines in parallel
|
|
|
|
=head1 SYNOPSIS
|
|
|
|
B<sem> [--fg] [--id <id>] [--semaphoretimeout <secs>] [-j <num>] [--wait] command
|
|
|
|
=head1 DESCRIPTION
|
|
|
|
GNU B<sem> is an alias for GNU B<parallel --semaphore>.
|
|
|
|
GNU B<sem> acts as a counting semaphore. When GNU B<sem> is called
|
|
with command it starts the command in the background. When I<num>
|
|
number of commands are running in the background, GNU B<sem> waits for
|
|
one of these to complete before starting the command.
|
|
|
|
GNU B<sem> does not read any arguments to build the command (no -a,
|
|
:::, and ::::). It simply waits for a semaphore to become available
|
|
and then runs the command given.
|
|
|
|
Before looking at the options you may want to check out the examples
|
|
after the list of options. That will give you an idea of what GNU
|
|
B<sem> is capable of.
|
|
|
|
=head1 OPTIONS
|
|
|
|
=over 9
|
|
|
|
=item I<command>
|
|
|
|
Command to execute. The command may be followed by arguments for the
|
|
command.
|
|
|
|
|
|
=item B<--bg>
|
|
|
|
Run command in background thus GNU B<sem> will not wait for
|
|
completion of the command before exiting. This is the default.
|
|
|
|
In toilet analogy: GNU B<sem> waits for a toilet to be available,
|
|
gives the toilet to a person, and exits immediately.
|
|
|
|
See also: B<--fg>
|
|
|
|
|
|
=item B<--jobs> I<N>
|
|
|
|
=item B<-j> I<N>
|
|
|
|
=item B<--max-procs> I<N>
|
|
|
|
=item B<-P> I<N>
|
|
|
|
Run up to N commands in parallel. Default is 1 thus acting like a
|
|
mutex.
|
|
|
|
In toilet analogy: B<-j> is the number of toilets.
|
|
|
|
|
|
=item B<--jobs> I<+N>
|
|
|
|
=item B<-j> I<+N>
|
|
|
|
=item B<--max-procs> I<+N>
|
|
|
|
=item B<-P> I<+N>
|
|
|
|
Add N to the number of CPU cores. Run up to this many jobs in
|
|
parallel. For compute intensive jobs B<-j> +0 is useful as it will run
|
|
number-of-cpu-cores jobs simultaneously.
|
|
|
|
|
|
=item B<--jobs> I<-N>
|
|
|
|
=item B<-j> I<-N>
|
|
|
|
=item B<--max-procs> I<-N>
|
|
|
|
=item B<-P> I<-N>
|
|
|
|
Subtract N from the number of CPU cores. Run up to this many jobs in
|
|
parallel. If the evaluated number is less than 1 then 1 will be used.
|
|
See also B<--use-cpus-instead-of-cores>.
|
|
|
|
|
|
=item B<--jobs> I<N>%
|
|
|
|
=item B<-j> I<N>%
|
|
|
|
=item B<--max-procs> I<N>%
|
|
|
|
=item B<-P> I<N>%
|
|
|
|
Multiply N% with the number of CPU cores. Run up to this many jobs in
|
|
parallel. If the evaluated number is less than 1 then 1 will be used.
|
|
See also B<--use-cpus-instead-of-cores>.
|
|
|
|
|
|
=item B<--jobs> I<procfile>
|
|
|
|
=item B<-j> I<procfile>
|
|
|
|
=item B<--max-procs> I<procfile>
|
|
|
|
=item B<-P> I<procfile>
|
|
|
|
Read parameter from file. Use the content of I<procfile> as parameter
|
|
for I<-j>. E.g. I<procfile> could contain the string 100% or +2 or
|
|
10.
|
|
|
|
|
|
=item B<--semaphorename> I<name>
|
|
|
|
=item B<--id> I<name>
|
|
|
|
Use B<name> as the name of the semaphore. Default is the name of the
|
|
controlling tty (output from B<tty>).
|
|
|
|
The default normally works as expected when used interactively, but
|
|
when used in a script I<name> should be set. I<$$> or I<my_task_name>
|
|
are often a good value.
|
|
|
|
The semaphore is stored in ~/.parallel/semaphores/
|
|
|
|
In toilet analogy the name corresponds to different types of toilets:
|
|
e.g. male, female, customer, staff.
|
|
|
|
|
|
=item B<--fg>
|
|
|
|
Do not put command in background.
|
|
|
|
In toilet analogy: GNU B<sem> waits for a toilet to be available,
|
|
takes a person to the toilet, waits for the person to finish, and
|
|
exits.
|
|
|
|
|
|
=item B<--semaphoretimeout> I<secs>
|
|
|
|
=item B<--st> I<secs>
|
|
|
|
If I<secs> > 0: If the semaphore is not released within I<secs>
|
|
seconds, take it anyway.
|
|
|
|
If I<secs> < 0: If the semaphore is not released within I<secs>
|
|
seconds, exit.
|
|
|
|
In toilet analogy: I<secs> > 0: If no toilet becomes available within
|
|
I<secs> seconds, pee on the floor. I<secs> < 0: If no toilet becomes
|
|
available within I<secs> seconds, exit without doing anything.
|
|
|
|
|
|
=item B<--wait>
|
|
|
|
Wait for all commands to complete.
|
|
|
|
In toilet analogy: Wait until all toilets are empty, then exit.
|
|
|
|
|
|
=back
|
|
|
|
=head1 UNDERSTANDING A SEMAPHORE
|
|
|
|
Try the following example:
|
|
|
|
sem -j 2 'sleep 1;echo 1 finished'; echo sem 1 exited
|
|
sem -j 2 'sleep 2;echo 2 finished'; echo sem 2 exited
|
|
sem -j 2 'sleep 3;echo 3 finished'; echo sem 3 exited
|
|
sem -j 2 'sleep 4;echo 4 finished'; echo sem 4 exited
|
|
sem --wait; echo sem --wait done
|
|
|
|
In toilet analogy this uses 2 toilets (B<-j 2>). GNU B<sem> takes '1'
|
|
to a toilet, and exits immediately. While '1' is sleeping, another GNU
|
|
B<sem> takes '2' to a toilet, and exits immediately.
|
|
|
|
While '1' and '2' are sleeping, another GNU B<sem> waits for a free
|
|
toilet. When '1' finishes, a toilet becomes available, and this GNU
|
|
B<sem> stops waiting, and takes '3' to a toilet, and exits
|
|
immediately.
|
|
|
|
While '2' and '3' are sleeping, another GNU B<sem> waits for a free
|
|
toilet. When '2' finishes, a toilet becomes available, and this GNU
|
|
B<sem> stops waiting, and takes '4' to a toilet, and exits
|
|
immediately.
|
|
|
|
Finally another GNU B<sem> waits for all toilets to become free.
|
|
|
|
|
|
=head1 EXAMPLE: Gzipping *.log
|
|
|
|
Run one gzip process per CPU core. Block until a CPU core becomes
|
|
available.
|
|
|
|
for i in *.log ; do
|
|
echo $i
|
|
sem -j+0 gzip $i ";" echo done
|
|
done
|
|
sem --wait
|
|
|
|
=head1 EXAMPLE: Protecting pod2html from itself
|
|
|
|
pod2html creates two files: pod2htmd.tmp and pod2htmi.tmp which it
|
|
does not clean up. It uses these two files for a short time. But if
|
|
you run multiple pod2html in parallel (e.g. in a Makefile with make
|
|
-j) there is a risk that two different instances of pod2html will
|
|
write to the files at the same time:
|
|
|
|
# This may fail due to shared pod2htmd.tmp/pod2htmi.tmp files
|
|
foo.html:
|
|
pod2html foo.pod --outfile foo.html
|
|
|
|
bar.html:
|
|
pod2html bar.pod --outfile bar.html
|
|
|
|
$ make -j foo.html bar.html
|
|
|
|
You need to protect pod2html from running twice at the same time.
|
|
B<sem> running as a mutex will make sure only one runs:
|
|
|
|
foo.html:
|
|
sem --id pod2html pod2html foo.pod --outfile foo.html
|
|
|
|
bar.html:
|
|
sem --id pod2html pod2html bar.pod --outfile bar.html
|
|
|
|
clean: foo.html bar.html
|
|
sem --id pod2html --wait
|
|
rm -f pod2htmd.tmp pod2htmi.tmp
|
|
|
|
$ make -j foo.html bar.html clean
|
|
|
|
=head1 BUGS
|
|
|
|
None known.
|
|
|
|
|
|
=head1 REPORTING BUGS
|
|
|
|
Report bugs to <bug-parallel@gnu.org>.
|
|
|
|
|
|
=head1 AUTHOR
|
|
|
|
Copyright (C) 2010,2011,2012,2013,2014,2015,2016,2017 Ole Tange,
|
|
http://ole.tange.dk and Free Software Foundation, Inc.
|
|
|
|
|
|
=head1 LICENSE
|
|
|
|
Copyright (C) 2010,2011,2012,2013 Free Software Foundation, Inc.
|
|
|
|
This program is free software; you can redistribute it and/or modify
|
|
it under the terms of the GNU General Public License as published by
|
|
the Free Software Foundation; either version 3 of the License, or
|
|
at your option any later version.
|
|
|
|
This program is distributed in the hope that it will be useful,
|
|
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
GNU General Public License for more details.
|
|
|
|
You should have received a copy of the GNU General Public License
|
|
along with this program. If not, see <http://www.gnu.org/licenses/>.
|
|
|
|
=head2 Documentation license I
|
|
|
|
Permission is granted to copy, distribute and/or modify this documentation
|
|
under the terms of the GNU Free Documentation License, Version 1.3 or
|
|
any later version published by the Free Software Foundation; with no
|
|
Invariant Sections, with no Front-Cover Texts, and with no Back-Cover
|
|
Texts. A copy of the license is included in the file fdl.txt.
|
|
|
|
=head2 Documentation license II
|
|
|
|
You are free:
|
|
|
|
=over 9
|
|
|
|
=item B<to Share>
|
|
|
|
to copy, distribute and transmit the work
|
|
|
|
=item B<to Remix>
|
|
|
|
to adapt the work
|
|
|
|
=back
|
|
|
|
Under the following conditions:
|
|
|
|
=over 9
|
|
|
|
=item B<Attribution>
|
|
|
|
You must attribute the work in the manner specified by the author or
|
|
licensor (but not in any way that suggests that they endorse you or
|
|
your use of the work).
|
|
|
|
=item B<Share Alike>
|
|
|
|
If you alter, transform, or build upon this work, you may distribute
|
|
the resulting work only under the same, similar or a compatible
|
|
license.
|
|
|
|
=back
|
|
|
|
With the understanding that:
|
|
|
|
=over 9
|
|
|
|
=item B<Waiver>
|
|
|
|
Any of the above conditions can be waived if you get permission from
|
|
the copyright holder.
|
|
|
|
=item B<Public Domain>
|
|
|
|
Where the work or any of its elements is in the public domain under
|
|
applicable law, that status is in no way affected by the license.
|
|
|
|
=item B<Other Rights>
|
|
|
|
In no way are any of the following rights affected by the license:
|
|
|
|
=over 2
|
|
|
|
=item *
|
|
|
|
Your fair dealing or fair use rights, or other applicable
|
|
copyright exceptions and limitations;
|
|
|
|
=item *
|
|
|
|
The author's moral rights;
|
|
|
|
=item *
|
|
|
|
Rights other persons may have either in the work itself or in
|
|
how the work is used, such as publicity or privacy rights.
|
|
|
|
=back
|
|
|
|
=back
|
|
|
|
=over 9
|
|
|
|
=item B<Notice>
|
|
|
|
For any reuse or distribution, you must make clear to others the
|
|
license terms of this work.
|
|
|
|
=back
|
|
|
|
A copy of the full license is included in the file as cc-by-sa.txt.
|
|
|
|
=head1 DEPENDENCIES
|
|
|
|
GNU B<sem> uses Perl, and the Perl modules Getopt::Long,
|
|
Symbol, Fcntl.
|
|
|
|
|
|
=head1 SEE ALSO
|
|
|
|
B<parallel>(1)
|
|
|
|
=cut
|