tange/parallel
1
0
Fork 0
mirror of https://git.savannah.gnu.org/git/parallel.git synced 2024-12-22 20:57:53 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

parallel: --tmux works for remote running jobs.

Browse source
This commit is contained in:
Ole Tange 2014-07-18 16:18:44 +02:00
parent 1a1c0ebe0f
commit 81ff568ce7
6 changed files with 250 additions and 33 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
doc/release_new_version
View file

@ -241,6 +241,8 @@ New in this release:
* The parenthesis around {= perl expression =} can be changed with --parens.
* --tmux will direct the output to a tmux session instead of files.
* GNU Parallel was cited in: bammds: A tool for assessing the ancestry of low depth whole genome data using multidimensional scaling (MDS) http://bioinformatics.oxfordjournals.org/content/early/2014/06/28/bioinformatics.btu410.abstract
* GNU Parallel was cited in: Molecular ferroelectric contributions to anomalous hysteresis in hybrid perovskite solar cells http://people.bath.ac.uk/aw558/publications/2014/arxiv_hysteresis_14.pdf
@ -251,10 +253,14 @@ New in this release:
* Webcast at 2014-08-20 covering GNU Parallel: Data Science at the Command Line http://www.oreilly.com/pub/e/3115
* GNU Parallel all the things! http://longwayaround.org.uk/notes/gnu-parallel-all-the-things/
* Shell command composition and dispatch http://lukeluo.blogspot.dk/2014/07/linux-virtual-console6-shell-command.html
* Parallelising plink (or anything else) the easy way http://chrisladroue.com/2012/03/parallelising-plink-or-anything-else-the-easy-way/
* Easy and cheap cluster building on AWS https://grapeot.me/easy-and-cheap-cluster-building-on-aws.html
* Paralelizace běžných činností v konzoli pomocí GNU Parallel http://www.abclinuxu.cz/clanky/paralelizace-beznych-cinnosti-v-konzoli-pomoci-gnu-parallel
* [原] Ubuntu 下使用 parallel 命令的注意事项 http://blog.ailms.me/2014/06/28/ubuntu-with-parallel.html

35
src/parallel
View file

@ -5220,8 +5220,9 @@ sub start {
(cat $tmpfile; rm $tmpfile; cat - ) | } .
"($command);";
if($opt::tmux) {
$command = tmux_wrap($command);
$command = tmux_wrap($command,$job->replaced());
}
# The eval is needed to catch exception from open3
eval {
$pid = ::open3($stdin_fh, ">&OUT", ">&ERR", $ENV{SHELL}, "-c", $command) ||
@ -5236,7 +5237,7 @@ sub start {
*IN = *STDIN;
# The eval is needed to catch exception from open3
if($opt::tmux) {
$command = tmux_wrap($command);
$command = tmux_wrap($command,$job->replaced());
}
eval {
$pid = ::open3("<&IN", ">&OUT", ">&ERR", $ENV{SHELL}, "-c", $command) ||
@ -5252,7 +5253,7 @@ sub start {
*IN = $devtty_fh;
# The eval is needed to catch exception from open3
if($opt::tmux) {
$command = tmux_wrap($command);
$command = tmux_wrap($command,$job->replaced());
}
eval {
$pid = ::open3("<&IN", ">&OUT", ">&ERR", $ENV{SHELL}, "-c", $command) ||
@ -5263,7 +5264,7 @@ sub start {
};
} else {
if($opt::tmux) {
$command = tmux_wrap($command);
$command = tmux_wrap($command,$job->replaced());
}
eval {
$pid = ::open3(::gensym, ">&OUT", ">&ERR", $ENV{SHELL}, "-c", $command) ||
@ -5292,21 +5293,33 @@ sub start {
}
sub tmux_wrap {
my $command = shift;
# Start tmux seesion as pPID
# Wrap command with tmux for session pPID
# Input:
# $actual_command = the actual command being run (incl ssh wrap)
# $visual_command = the command the user wants run (= -v)
my $actual_command = shift;
my $visual_command = shift;
# Temporary file name. Used for fifo to communicate exit val
my ($fh, $tmpfile) = ::tempfile(SUFFIX => ".tmx");
close $fh;
unlink $tmpfile;
my $c = $visual_command;
# tmux does not like some chars (e.g. ;).
$c =~ s/[^-<>(),:_+=#a-z0-9 `\\\$\|]//g;
my $tmux;
if($Global::total_running == 0) {
$tmux = "tmux new-session -s p$$ -d";
if($Global::total_running == 0) {
$tmux = "tmux new-session -s p$$ -d -n ".
::shell_quote_scalar($c);
print $Global::original_stderr "See output with: tmux attach -t p$$\n";
} else {
$tmux = "tmux new-window -t p$$";
$tmux = "tmux new-window -t p$$ -n ".::shell_quote_scalar($c);
}
return "mkfifo $tmpfile; $tmux ".
::shell_quote_scalar("(".$command.');echo $?$status >'.$tmpfile).
"; ( exit `cat < $tmpfile;rm $tmpfile` )";
::shell_quote_scalar("(".$actual_command.');echo $?$status >'.$tmpfile.";".
"echo ".::shell_quote_scalar($visual_command).";".
"echo \007Job finished at: `date`;sleep 60").
"; ( exit `cat < $tmpfile;rm $tmpfile` )";
}
sub is_already_in_results {

101
src/parallel_tutorial.1
View file

@ -124,7 +124,7 @@
.\" ========================================================================
.\"
.IX Title "PARALLEL_TUTORIAL 1"
.TH PARALLEL_TUTORIAL 1 "2014-07-15" "20140711" "parallel"
.TH PARALLEL_TUTORIAL 1 "2014-07-18" "20140711" "parallel"
.\" For nroff, turn off justification. Always turn off hyphenation; it makes
.\" way too many mistakes in technical documents.
.if n .ad l
@ -483,8 +483,8 @@ Output (the order may be different):
.Ve
.SS "Replacement strings"
.IX Subsection "Replacement strings"
\fIThe 7 replacement strings\fR
.IX Subsection "The 7 replacement strings"
\fIThe 7 predefined replacement strings\fR
.IX Subsection "The 7 predefined replacement strings"
.PP
\&\s-1GNU\s0 Parallel has several replacement strings. If no replacement
strings are used the default is to append {}:
@ -587,8 +587,10 @@ Output (the order may be different):
\& 2
\& 1
.Ve
.SS "Changing the replacement strings"
.PP
\fIChanging the replacement strings\fR
.IX Subsection "Changing the replacement strings"
.PP
The replacement string {} can be changed with \-I:
.PP
.Vb 1
@ -676,8 +678,64 @@ Output (the order may be different):
\& 2
\& 1
.Ve
.SS "Positional replacement strings"
.PP
\fIPerl expression replacement string\fR
.IX Subsection "Perl expression replacement string"
.PP
When predefined replacement strings are not flexible enough a perl
expression can be used instead. One example is to remove two
extensions: foo.tar.gz \-> foo
.PP
.Vb 1
\& parallel echo \*(Aq{= s:\e.[^.]+$::;s:\e.[^.]+$::; =}\*(Aq ::: foo.tar.gz
.Ve
.PP
Output:
.PP
.Vb 1
\& foo
.Ve
.PP
If the strings \fB{=\fR and \fB=}\fR cause problems they can be replaced with \-\-parens:
.PP
.Vb 1
\& parallel \-\-parens ,,,, echo \*(Aq,, s:\e.[^.]+$::;s:\e.[^.]+$::; ,,\*(Aq ::: foo.tar.gz
.Ve
.PP
Output: Same as above.
.PP
To define a short hand replacement string use \fB\-\-rpl\fR:
.PP
.Vb 1
\& parallel \-\-rpl \*(Aq.. s:\e.[^.]+$::;s:\e.[^.]+$::;\*(Aq echo \*(Aq..\*(Aq ::: foo.tar.gz
.Ve
.PP
Output: Same as above.
.PP
If the short hand starts with '{' it can be used as a positional
replacement string, too:
.PP
.Vb 1
\& parallel \-\-rpl \*(Aq{..} s:\e.[^.]+$::;s:\e.[^.]+$::;\*(Aq echo \*(Aq{..}\*(Aq ::: foo.tar.gz
.Ve
.PP
Output: Same as above.
.PP
\&\s-1GNU\s0 \fBparallel\fR's 7 replacement strings are implemented as:
.PP
.Vb 7
\& \-\-rpl \*(Aq{} \*(Aq
\& \-\-rpl \*(Aq{#} $_=$job\->seq()\*(Aq
\& \-\-rpl \*(Aq{%} $_=$job\->slot()\*(Aq
\& \-\-rpl \*(Aq{/} s:.*/::\*(Aq
\& \-\-rpl \*(Aq{//} $Global::use{"File::Basename"} ||= eval "use File::Basename; 1;"; $_ = dirname($_);\*(Aq
\& \-\-rpl \*(Aq{/.} s:.*/::; s:\e.[^/.]+$::;\*(Aq
\& \-\-rpl \*(Aq{.} s:\e.[^/.]+$::\*(Aq
.Ve
.PP
\fIPositional replacement strings\fR
.IX Subsection "Positional replacement strings"
.PP
With multiple input sources the argument from the individual input
sources can be access with {number}:
.PP
@ -726,8 +784,35 @@ Output (the order may be different):
\& 1=B 2=D 3=E \-1=E \-2=D \-3=B
\& 1=B 2=D 3=F \-1=F \-2=D \-3=B
.Ve
.SS "Input from columns"
.PP
\fIPositional perl expression replacement string\fR
.IX Subsection "Positional perl expression replacement string"
.PP
To use a perl expression as a positional replacement string simply
prepend the perl expression with number and space:
.PP
.Vb 1
\& parallel echo \*(Aq{=2 s:\e.[^.]+$::;s:\e.[^.]+$::; =} {1}\*(Aq ::: bar ::: foo.tar.gz
.Ve
.PP
Output:
.PP
.Vb 1
\& foo bar
.Ve
.PP
If a defined short hand starts with '{' it can be used as a positional
replacement string, too:
.PP
.Vb 1
\& parallel \-\-rpl \*(Aq{..} s:\e.[^.]+$::;s:\e.[^.]+$::;\*(Aq echo \*(Aq{2..} {1}\*(Aq ::: bar ::: foo.tar.gz
.Ve
.PP
Output: Same as above.
.PP
\fIInput from columns\fR
.IX Subsection "Input from columns"
.PP
The columns in a file can be bound to positional replacement strings
using \-\-colsep. Here the columns are separated with \s-1TAB\s0 (\et):
.PP
@ -742,8 +827,10 @@ Output (the order may be different):
\& 1=A 2=B
\& 1=C 2=D
.Ve
.SS "Header defined replacement strings"
.PP
\fIHeader defined replacement strings\fR
.IX Subsection "Header defined replacement strings"
.PP
With \-\-header \s-1GNU\s0 Parallel will use the first value of the input
source as the name of the replacement string. Only the non-modified
version {} is supported:

70
src/parallel_tutorial.html
View file

@ -41,13 +41,15 @@
<li><a href="#replacement_strings">Replacement strings</a></li>
<ul>
<li><a href="#the_7_replacement_strings">The 7 replacement strings</a></li>
<li><a href="#the_7_predefined_replacement_strings">The 7 predefined replacement strings</a></li>
<li><a href="#changing_the_replacement_strings">Changing the replacement strings</a></li>
<li><a href="#perl_expression_replacement_string">Perl expression replacement string</a></li>
<li><a href="#positional_replacement_strings">Positional replacement strings</a></li>
<li><a href="#positional_perl_expression_replacement_string">Positional perl expression replacement string</a></li>
<li><a href="#input_from_columns">Input from columns</a></li>
<li><a href="#header_defined_replacement_strings">Header defined replacement strings</a></li>
</ul>
<li><a href="#changing_the_replacement_strings">Changing the replacement strings</a></li>
<li><a href="#positional_replacement_strings">Positional replacement strings</a></li>
<li><a href="#input_from_columns">Input from columns</a></li>
<li><a href="#header_defined_replacement_strings">Header defined replacement strings</a></li>
<li><a href="#more_than_one_argument">More than one argument</a></li>
<li><a href="#quoting">Quoting</a></li>
<li><a href="#trimming_space">Trimming space</a></li>
@ -390,7 +392,7 @@ exported using 'export -f':</p>
<h2><a name="replacement_strings">Replacement strings</a></h2>
<p>
</p>
<h3><a name="the_7_replacement_strings">The 7 replacement strings</a></h3>
<h3><a name="the_7_predefined_replacement_strings">The 7 predefined replacement strings</a></h3>
<p>GNU Parallel has several replacement strings. If no replacement
strings are used the default is to append {}:</p>
<pre>
@ -447,7 +449,7 @@ number of jobs to run in parallel):</p>
1</pre>
<p>
</p>
<h2><a name="changing_the_replacement_strings">Changing the replacement strings</a></h2>
<h3><a name="changing_the_replacement_strings">Changing the replacement strings</a></h3>
<p>The replacement string {} can be changed with -I:</p>
<pre>
parallel -I ,, echo ,, ::: A/B.C</pre>
@ -496,7 +498,40 @@ number of jobs to run in parallel):</p>
1</pre>
<p>
</p>
<h2><a name="positional_replacement_strings">Positional replacement strings</a></h2>
<h3><a name="perl_expression_replacement_string">Perl expression replacement string</a></h3>
<p>When predefined replacement strings are not flexible enough a perl
expression can be used instead. One example is to remove two
extensions: foo.tar.gz -&gt; foo</p>
<pre>
parallel echo '{= s:\.[^.]+$::;s:\.[^.]+$::; =}' ::: foo.tar.gz</pre>
<p>Output:</p>
<pre>
foo</pre>
<p>If the strings <strong>{=</strong> and <strong>=}</strong> cause problems they can be replaced with --parens:</p>
<pre>
parallel --parens ,,,, echo ',, s:\.[^.]+$::;s:\.[^.]+$::; ,,' ::: foo.tar.gz</pre>
<p>Output: Same as above.</p>
<p>To define a short hand replacement string use <strong>--rpl</strong>:</p>
<pre>
parallel --rpl '.. s:\.[^.]+$::;s:\.[^.]+$::;' echo '..' ::: foo.tar.gz</pre>
<p>Output: Same as above.</p>
<p>If the short hand starts with '{' it can be used as a positional
replacement string, too:</p>
<pre>
parallel --rpl '{..} s:\.[^.]+$::;s:\.[^.]+$::;' echo '{..}' ::: foo.tar.gz</pre>
<p>Output: Same as above.</p>
<p>GNU <strong>parallel</strong>'s 7 replacement strings are implemented as:</p>
<pre>
--rpl '{} '
--rpl '{#} $_=$job-&gt;seq()'
--rpl '{%} $_=$job-&gt;slot()'
--rpl '{/} s:.*/::'
--rpl '{//} $Global::use{&quot;File::Basename&quot;} ||= eval &quot;use File::Basename; 1;&quot;; $_ = dirname($_);'
--rpl '{/.} s:.*/::; s:\.[^/.]+$::;'
--rpl '{.} s:\.[^/.]+$::'</pre>
<p>
</p>
<h3><a name="positional_replacement_strings">Positional replacement strings</a></h3>
<p>With multiple input sources the argument from the individual input
sources can be access with {number}:</p>
<pre>
@ -530,7 +565,22 @@ from behind:</p>
1=B 2=D 3=F -1=F -2=D -3=B</pre>
<p>
</p>
<h2><a name="input_from_columns">Input from columns</a></h2>
<h3><a name="positional_perl_expression_replacement_string">Positional perl expression replacement string</a></h3>
<p>To use a perl expression as a positional replacement string simply
prepend the perl expression with number and space:</p>
<pre>
parallel echo '{=2 s:\.[^.]+$::;s:\.[^.]+$::; =} {1}' ::: bar ::: foo.tar.gz</pre>
<p>Output:</p>
<pre>
foo bar</pre>
<p>If a defined short hand starts with '{' it can be used as a positional
replacement string, too:</p>
<pre>
parallel --rpl '{..} s:\.[^.]+$::;s:\.[^.]+$::;' echo '{2..} {1}' ::: bar ::: foo.tar.gz</pre>
<p>Output: Same as above.</p>
<p>
</p>
<h3><a name="input_from_columns">Input from columns</a></h3>
<p>The columns in a file can be bound to positional replacement strings
using --colsep. Here the columns are separated with TAB (\t):</p>
<pre>
@ -542,7 +592,7 @@ using --colsep. Here the columns are separated with TAB (\t):</p>
1=C 2=D</pre>
<p>
</p>
<h2><a name="header_defined_replacement_strings">Header defined replacement strings</a></h2>
<h3><a name="header_defined_replacement_strings">Header defined replacement strings</a></h3>
<p>With --header GNU Parallel will use the first value of the input
source as the name of the replacement string. Only the non-modified
version {} is supported:</p>

BIN
src/parallel_tutorial.pdf
View file

Binary file not shown.

71
src/parallel_tutorial.pod
View file

@ -300,7 +300,7 @@ Output (the order may be different):
=head2 Replacement strings
=head3 The 7 replacement strings
=head3 The 7 predefined replacement strings
GNU Parallel has several replacement strings. If no replacement
strings are used the default is to append {}:
@ -372,7 +372,7 @@ Output (the order may be different):
2
1
=head2 Changing the replacement strings
=head3 Changing the replacement strings
The replacement string {} can be changed with -I:
@ -434,7 +434,48 @@ Output (the order may be different):
2
1
=head2 Positional replacement strings
=head3 Perl expression replacement string
When predefined replacement strings are not flexible enough a perl
expression can be used instead. One example is to remove two
extensions: foo.tar.gz -> foo
parallel echo '{= s:\.[^.]+$::;s:\.[^.]+$::; =}' ::: foo.tar.gz
Output:
foo
If the strings B<{=> and B<=}> cause problems they can be replaced with --parens:
parallel --parens ,,,, echo ',, s:\.[^.]+$::;s:\.[^.]+$::; ,,' ::: foo.tar.gz
Output: Same as above.
To define a short hand replacement string use B<--rpl>:
parallel --rpl '.. s:\.[^.]+$::;s:\.[^.]+$::;' echo '..' ::: foo.tar.gz
Output: Same as above.
If the short hand starts with '{' it can be used as a positional
replacement string, too:
parallel --rpl '{..} s:\.[^.]+$::;s:\.[^.]+$::;' echo '{..}' ::: foo.tar.gz
Output: Same as above.
GNU B<parallel>'s 7 replacement strings are implemented as:
--rpl '{} '
--rpl '{#} $_=$job->seq()'
--rpl '{%} $_=$job->slot()'
--rpl '{/} s:.*/::'
--rpl '{//} $Global::use{"File::Basename"} ||= eval "use File::Basename; 1;"; $_ = dirname($_);'
--rpl '{/.} s:.*/::; s:\.[^/.]+$::;'
--rpl '{.} s:\.[^/.]+$::'
=head3 Positional replacement strings
With multiple input sources the argument from the individual input
sources can be access with {number}:
@ -473,7 +514,27 @@ Output (the order may be different):
1=B 2=D 3=E -1=E -2=D -3=B
1=B 2=D 3=F -1=F -2=D -3=B
=head2 Input from columns
=head3 Positional perl expression replacement string
To use a perl expression as a positional replacement string simply
prepend the perl expression with number and space:
parallel echo '{=2 s:\.[^.]+$::;s:\.[^.]+$::; =} {1}' ::: bar ::: foo.tar.gz
Output:
foo bar
If a defined short hand starts with '{' it can be used as a positional
replacement string, too:
parallel --rpl '{..} s:\.[^.]+$::;s:\.[^.]+$::;' echo '{2..} {1}' ::: bar ::: foo.tar.gz
Output: Same as above.
=head3 Input from columns
The columns in a file can be bound to positional replacement strings
using --colsep. Here the columns are separated with TAB (\t):
@ -486,7 +547,7 @@ Output (the order may be different):
1=A 2=B
1=C 2=D
=head2 Header defined replacement strings
=head3 Header defined replacement strings
With --header GNU Parallel will use the first value of the input
source as the name of the replacement string. Only the non-modified