tange/parallel
1
0
Fork 0
mirror of https://git.savannah.gnu.org/git/parallel.git synced 2024-12-25 06:07:54 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Released as 20140822 ('Williams').

Browse source
This commit is contained in:
Ole Tange 2014-08-23 00:39:59 +02:00
parent 6df2491b81
commit 82550070ee
10 changed files with 123 additions and 69 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

40
NEWS
View file

@ -1,3 +1,43 @@
20140822
* GNU Parallel now uses the same shell it was started from as the
command shell for local jobs. So if GNU Parallel is started from
tcsh it will use tcsh as its shell even if the login $SHELL is
different. For remote jobs the login $SHELL will be used.
* The whole current environment in bash can be copied by using a shell
wrapper function (Search manual for env_parallel).
* --plus adds the replacement strings {+/} {+.} {+..} {+...} {..}
{...} {/..} {/...}. The idea being that '+foo' matches the opposite
of 'foo' and {} = {+/}/{/} = {.}.{+.} = {+/}/{/.}.{+.} = {..}.{+..}
= {+/}/{/..}.{+..} = {...}.{+...} = {+/}/{/...}.{+...}
* GNU Parallel now deals correctly with the combination
rsync-3.1.X-client and rsync-2.5.7-server
* GNU Parallel was cited in: A Web Service for Scholarly Big Data
Information Extraction
http://patshih.ist.psu.edu/publications/Williams-CiteSeerExtractor-ICWS14.pdf
* Comparison of the speed of different GNU Parallel versions:
http://lists.gnu.org/archive/html/parallel/2014-08/msg00030.html
* GNU Parallel was covered in the webcast 2014-08-20: Data Science at
the Command Line http://www.oreilly.com/pub/e/3115
* Distributed processing with GNU parallel
http://kazjote.eu/2014/08/11/distributed-processing-with-gnu-parallel
* A Peek into GNU Parallel
http://blog.dataweave.in/post/94238943763/a-peek-into-gnu-parallel
* Сборка GNU parallel для CentOS/RHEL
http://www.stableit.ru/2014/07/gnu-parallel-centosrhel.html
* Bug fixes and man page updates.
20140722
* {= perl expression =} can be used as replacement string. The

12
README
View file

@ -40,9 +40,9 @@ document.
Full installation of GNU Parallel is as simple as:
wget http://ftpmirror.gnu.org/parallel/parallel-20140815.tar.bz2
bzip2 -dc parallel-20140815.tar.bz2 | tar xvf -
cd parallel-20140815
wget http://ftpmirror.gnu.org/parallel/parallel-20140822.tar.bz2
bzip2 -dc parallel-20140822.tar.bz2 | tar xvf -
cd parallel-20140822
./configure && make && make install
@ -51,9 +51,9 @@ Full installation of GNU Parallel is as simple as:
If you are not root you can add ~/bin to your path and install in
~/bin and ~/share:
wget http://ftpmirror.gnu.org/parallel/parallel-20140815.tar.bz2
bzip2 -dc parallel-20140815.tar.bz2 | tar xvf -
cd parallel-20140815
wget http://ftpmirror.gnu.org/parallel/parallel-20140822.tar.bz2
bzip2 -dc parallel-20140822.tar.bz2 | tar xvf -
cd parallel-20140822
./configure --prefix=$HOME && make && make install
Or if your system lacks 'make' you can simply copy src/parallel

20
configure vendored
View file

@ -1,6 +1,6 @@
#! /bin/sh
# Guess values for system-dependent variables and create Makefiles.
# Generated by GNU Autoconf 2.69 for parallel 20140815.
# Generated by GNU Autoconf 2.69 for parallel 20140822.
#
# Report bugs to <bug-parallel@gnu.org>.
#
@ -579,8 +579,8 @@ MAKEFLAGS=
# Identity of this package.
PACKAGE_NAME='parallel'
PACKAGE_TARNAME='parallel'
PACKAGE_VERSION='20140815'
PACKAGE_STRING='parallel 20140815'
PACKAGE_VERSION='20140822'
PACKAGE_STRING='parallel 20140822'
PACKAGE_BUGREPORT='bug-parallel@gnu.org'
PACKAGE_URL=''
@ -1203,7 +1203,7 @@ if test "$ac_init_help" = "long"; then
# Omit some internal or obsolete options to make the list less imposing.
# This message is too long to be a string in the A/UX 3.1 sh.
cat <<_ACEOF
\`configure' configures parallel 20140815 to adapt to many kinds of systems.
\`configure' configures parallel 20140822 to adapt to many kinds of systems.
Usage: $0 [OPTION]... [VAR=VALUE]...
@ -1269,7 +1269,7 @@ fi
if test -n "$ac_init_help"; then
case $ac_init_help in
short | recursive ) echo "Configuration of parallel 20140815:";;
short | recursive ) echo "Configuration of parallel 20140822:";;
esac
cat <<\_ACEOF
@ -1345,7 +1345,7 @@ fi
test -n "$ac_init_help" && exit $ac_status
if $ac_init_version; then
cat <<\_ACEOF
parallel configure 20140815
parallel configure 20140822
generated by GNU Autoconf 2.69
Copyright (C) 2012 Free Software Foundation, Inc.
@ -1362,7 +1362,7 @@ cat >config.log <<_ACEOF
This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.
It was created by parallel $as_me 20140815, which was
It was created by parallel $as_me 20140822, which was
generated by GNU Autoconf 2.69. Invocation command line was
$ $0 $@
@ -2225,7 +2225,7 @@ fi
# Define the identity of the package.
PACKAGE='parallel'
VERSION='20140815'
VERSION='20140822'
cat >>confdefs.h <<_ACEOF
@ -2867,7 +2867,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
# report actual input values of CONFIG_FILES etc. instead of their
# values after options handling.
ac_log="
This file was extended by parallel $as_me 20140815, which was
This file was extended by parallel $as_me 20140822, which was
generated by GNU Autoconf 2.69. Invocation command line was
CONFIG_FILES = $CONFIG_FILES
@ -2929,7 +2929,7 @@ _ACEOF
cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`"
ac_cs_version="\\
parallel config.status 20140815
parallel config.status 20140822
configured by $0, generated by GNU Autoconf 2.69,
with options \\"\$ac_cs_config\\"

2
configure.ac
View file

@ -1,4 +1,4 @@
AC_INIT([parallel], [20140815], [bug-parallel@gnu.org])
AC_INIT([parallel], [20140822], [bug-parallel@gnu.org])
AM_INIT_AUTOMAKE([-Wall -Werror foreign])
AC_CONFIG_HEADERS([config.h])
AC_CONFIG_FILES([

6
doc/release_new_version
View file

@ -234,10 +234,12 @@ New in this release:
* The whole current environment in bash can be copied by using a shell wrapper function (Search manual for env_parallel).
* GNU Parallel was cited in: A Web Service for Scholarly Big Data Information Extraction http://patshih.ist.psu.edu/publications/Williams-CiteSeerExtractor-ICWS14.pdf
* --plus adds the replacement strings {+/} {+.} {+..} {+...} {..} {...} {/..} {/...}. The idea being that '+foo' matches the opposite of 'foo' and {} = {+/}/{/} = {.}.{+.} = {+/}/{/.}.{+.} = {..}.{+..} = {+/}/{/..}.{+..} = {...}.{+...} = {+/}/{/...}.{+...}
* GNU Parallel now deals correctly with the combination rsync-3.1.X-client and rsync-2.5.7-server
* GNU Parallel was cited in: A Web Service for Scholarly Big Data Information Extraction http://patshih.ist.psu.edu/publications/Williams-CiteSeerExtractor-ICWS14.pdf
* Comparison of the speed of different GNU Parallel versions: http://lists.gnu.org/archive/html/parallel/2014-08/msg00030.html
* GNU Parallel was covered in the webcast 2014-08-20: Data Science at the Command Line http://www.oreilly.com/pub/e/3115

4
packager/obs/home:tange/parallel/parallel.spec
View file

@ -1,11 +1,11 @@
Summary: Shell tool for executing jobs in parallel
Name: parallel
Version: 20140722
Version: 20140822
Release: 1
License: GPL
Group: Productivity/File utilities
URL: ftp://ftp.gnu.org/gnu/parallel
Source0: %{name}-%{version}.tar.bz2
Source0: %{name}_%{version}.tar.gz
BuildArch: noarch
BuildArchitectures: noarch
BuildRoot: %{_tmppath}/%{name}-%{version}-buildroot

2
src/niceload
View file

@ -24,7 +24,7 @@
use strict;
use Getopt::Long;
$Global::progname="niceload";
$Global::version = 20140815;
$Global::version = 20140822;
Getopt::Long::Configure("bundling","require_order");
get_options_from_array(\@ARGV) || die_usage();
if($opt::version) {

2
src/parallel
View file

@ -721,7 +721,7 @@ sub get_options_from_array {
sub parse_options {
# Returns: N/A
# Defaults:
$Global::version = 20140815;
$Global::version = 20140822;
$Global::progname = 'parallel';
$Global::infinity = 2**31;
$Global::debug = 0;

102
src/parallel.pod
View file

@ -97,7 +97,7 @@ B<exportf> to export and to set $SHELL to bash:
The command cannot contain the character \257 (macron: ¯).
=item B<{}> (beta testing)
=item B<{}> (alpha testing)
Input line. This replacement string will be replaced by a full line
read from the input source. The input source is normally stdin
@ -110,7 +110,7 @@ If the command line contains no replacement strings then B<{}> will be
appended to the command line.
=item B<{.}> (beta testing)
=item B<{.}> (alpha testing)
Input line without extension. This replacement string will be replaced
by the input with the extension removed. If the input line contains
@ -126,7 +126,7 @@ The replacement string B<{.}> can be changed with B<--er>.
To understand replacement strings see B<{}>.
=item B<{/}> (beta testing)
=item B<{/}> (alpha testing)
Basename of input line. This replacement string will be replaced by
the input with the directory part removed.
@ -137,7 +137,7 @@ B<--basenamereplace>.
To understand replacement strings see B<{}>.
=item B<{//}> (beta testing)
=item B<{//}> (alpha testing)
Dirname of input line. This replacement string will be replaced by the
dir of the input line. See B<dirname>(1).
@ -148,7 +148,7 @@ B<--dirnamereplace>.
To understand replacement strings see B<{}>.
=item B<{/.}> (beta testing)
=item B<{/.}> (alpha testing)
Basename of input line without extension. This replacement string will
be replaced by the input with the directory and extension part
@ -160,7 +160,7 @@ B<--basenameextensionreplace>.
To understand replacement strings see B<{}>.
=item B<{#}> (beta testing)
=item B<{#}> (alpha testing)
Sequence number of the job to run. This replacement string will be
replaced by the sequence number of the job being run. It contains the
@ -171,7 +171,7 @@ The replacement string B<{#}> can be changed with B<--seqreplace>.
To understand replacement strings see B<{}>.
=item B<{%}> (beta testing)
=item B<{%}> (alpha testing)
Job slot number. This replacement string will be replaced by the job's
slot number between 1 and number of jobs to run in parallel. There
@ -183,7 +183,7 @@ The replacement string B<{%}> can be changed with B<--slotreplace>.
To understand replacement strings see B<{}>.
=item B<{>I<n>B<}> (beta testing)
=item B<{>I<n>B<}> (alpha testing)
Argument from input source I<n> or the I<n>'th argument. This
positional replacement string will be replaced by the input from input
@ -194,7 +194,7 @@ I<n>'th last argument.
To understand replacement strings see B<{}>.
=item B<{>I<n>.B<}> (beta testing)
=item B<{>I<n>.B<}> (alpha testing)
Argument from input source I<n> or the I<n>'th argument without
extension. It is a combination of B<{>I<n>B<}> and B<{.}>.
@ -207,7 +207,7 @@ extension removed.
To understand positional replacement strings see B<{>I<n>B<}>.
=item B<{>I<n>/B<}> (beta testing)
=item B<{>I<n>/B<}> (alpha testing)
Basename of argument from input source I<n> or the I<n>'th argument.
It is a combination of B<{>I<n>B<}> and B<{/}>.
@ -220,7 +220,7 @@ directory (if any) removed.
To understand positional replacement strings see B<{>I<n>B<}>.
=item B<{>I<n>//B<}> (beta testing)
=item B<{>I<n>//B<}> (alpha testing)
Dirname of argument from input source I<n> or the I<n>'th argument.
It is a combination of B<{>I<n>B<}> and B<{//}>.
@ -232,7 +232,7 @@ the I<n>'th argument (when used with B<-N>). See B<dirname>(1).
To understand positional replacement strings see B<{>I<n>B<}>.
=item B<{>I<n>/.B<}> (beta testing)
=item B<{>I<n>/.B<}> (alpha testing)
Basename of argument from input source I<n> or the I<n>'th argument
without extension. It is a combination of B<{>I<n>B<}>, B<{/}>, and
@ -246,7 +246,7 @@ directory (if any) and extension removed.
To understand positional replacement strings see B<{>I<n>B<}>.
=item B<{=>I<perl expression>B<=}> (beta testing)
=item B<{=>I<perl expression>B<=}> (alpha testing)
Replace with calculated I<perl expression>. B<$_> will contain the
same as B<{}>. After evaluating I<perl expression> B<$_> will be used
@ -259,7 +259,7 @@ The B<{=>I<perl expression>B<=}> must be given as a single string.
See also: B<--rpl> B<--parens>
=item B<{=>I<n> I<perl expression>B<=}> (beta testing)
=item B<{=>I<n> I<perl expression>B<=}> (alpha testing)
Positional equivalent to B<{= perl expression =}>. To understand
positional replacement strings see B<{>I<n>B<}>.
@ -321,9 +321,9 @@ Use NUL as delimiter. Normally input lines will end in \n
for processing arguments that may contain \n (newline).
=item B<--arg-file> I<input-file> (alpha testing)
=item B<--arg-file> I<input-file> (beta testing)
=item B<-a> I<input-file> (alpha testing)
=item B<-a> I<input-file> (beta testing)
Use I<input-file> as input source. If you use this option, stdin
(standard input) is given to the first process run. Otherwise, stdin
@ -370,9 +370,9 @@ It is compatible with B<zenity>:
seq 1000 | parallel -j30 --bar '(echo {};sleep 0.1)' 2> >(zenity --progress --auto-kill) | wc
=item B<--basefile> I<file>
=item B<--basefile> I<file> (alpha testing)
=item B<--bf> I<file>
=item B<--bf> I<file> (alpha testing)
I<file> will be transferred to each sshlogin before a jobs is
started. It will be removed if B<--cleanup> is active. The file may be
@ -413,9 +413,9 @@ Print the BibTeX entry for GNU B<parallel> and disable citation
notice.
=item B<--block> I<size> (alpha testing)
=item B<--block> I<size> (beta testing)
=item B<--block-size> I<size> (alpha testing)
=item B<--block-size> I<size> (beta testing)
Size of block in bytes. The size can be postfixed with K, M, G, T, P,
k, m, g, t, or p which would multiply the size with 1024, 1048576,
@ -523,7 +523,7 @@ Use the replacement string I<replace-str> instead of B<{//}> for
dirname of input line.
=item B<-E> I<eof-str>
=item B<-E> I<eof-str> (alpha testing)
Set the end of file string to eof-str. If the end of file string
occurs as a line of input, the rest of the input is ignored. If
@ -556,7 +556,7 @@ If I<eof-str> is omitted, there is no end of file string. If neither
B<-E> nor B<-e> is used, no end of file string is used.
=item B<--env> I<var>
=item B<--env> I<var> (alpha testing)
Copy environment variable I<var>. This will copy I<var> to the
environment that the command is run in. This is especially useful for
@ -588,10 +588,23 @@ cannot be exported:
parallel --env my_importer \
'my_importer; echo "{}" "${indexed[{}]}" "${assoc[${indexed[{}]}]}"' ::: "${!indexed[@]}"
To copy the full environment use this function:
env_parallel() {
export parallel_bash_environment='() {
'"$(echo "shopt -s expand_aliases 2>/dev/null"; alias;typeset -p | grep -vFf <(readonly; echo GROUPS; echo FUNCNAME; echo DIRSTACK; echo _; echo PIPESTATUS; echo USERNAME) | grep -v BASH_;typeset -f)"'
}'
# Run as: env_parallel parallel_bash_environment "2>/dev/null;" ...
`which parallel` "$@"
unset parallel_bash_environment
}
# call as:
env_parallel ...
See also: B<--record-env>.
=item B<--eta> (alpha testing)
=item B<--eta> (beta testing)
Show the estimated number of seconds before finishing. This forces GNU
B<parallel> to read all jobs before starting to find the number of
@ -661,9 +674,9 @@ See also: B<--line-buffer> B<--ungroup>
Print a summary of the options to GNU B<parallel> and exit.
=item B<--halt-on-error> I<val> (beta testing)
=item B<--halt-on-error> I<val> (alpha testing)
=item B<--halt> I<val> (beta testing)
=item B<--halt> I<val> (alpha testing)
How should GNU B<parallel> terminate if one of more jobs fail?
@ -713,9 +726,9 @@ If I<regexp> is a number, it will match that many lines.
Use the replacement string I<replace-str> instead of {}.
=item B<--replace>[=I<replace-str>]
=item B<--replace>[=I<replace-str>] (alpha testing)
=item B<-i>[I<replace-str>]
=item B<-i>[I<replace-str>] (alpha testing)
This option is a synonym for B<-I>I<replace-str> if I<replace-str> is
specified, and for B<-I>{} otherwise. This option is deprecated;
@ -784,9 +797,8 @@ B<--use-cpus-instead-of-cores>.
=item B<-P> I<N>% (alpha testing)
Multiply N% with the number of CPU cores. Run this many jobs in parallel.
If the evaluated number is less than 1 then 1 will be used. See also
B<--use-cpus-instead-of-cores>.
Multiply N% with the number of CPU cores. Run this many jobs in
parallel. See also B<--use-cpus-instead-of-cores>.
=item B<--jobs> I<procfile>
@ -875,9 +887,9 @@ the same syntax as B<--jobs>, so I<100%> for one per CPU is a valid
setting. Only difference is 0 which is interpreted as 0.01.
=item B<--controlmaster>
=item B<--controlmaster> (alpha testing)
=item B<-M>
=item B<-M> (alpha testing)
Use ssh's ControlMaster to make ssh connections faster. Useful if jobs
run remote and are very fast to run. This is disabled for sshlogins
@ -987,7 +999,7 @@ B<--files> is often used with B<--pipe>.
See also: B<--recstart>, B<--recend>, B<--fifo>, B<--cat>, B<--pipepart>.
=item B<--pipepart> (alpha testing)
=item B<--pipepart> (beta testing)
Pipe parts of a physical file. B<--pipepart> works similar to
B<--pipe>, but is much faster. It has a few limitations:
@ -1118,7 +1130,7 @@ from the terminal. Only run the command line if the response starts
with 'y' or 'Y'. Implies B<-t>.
=item B<--parens> I<parensstring> (beta testing)
=item B<--parens> I<parensstring>
Use to define start and end parenthesis for B<{= perl expression =}>. The
left and the right parenthesis can be multiple characters and are
@ -1308,7 +1320,7 @@ useful if some jobs fail for no apparent reason (such as network
failure).
=item B<--return> I<filename>
=item B<--return> I<filename> (alpha testing)
Transfer files from remote computers. B<--return> is used with
B<--sshlogin> when the arguments are files on the remote computers. When
@ -1356,7 +1368,7 @@ B<--keep-order> will not work with B<--round-robin> as it is
impossible to track which input block corresponds to which output.
=item B<--rpl> 'I<tag> I<perl expression>' (beta testing)
=item B<--rpl> 'I<tag> I<perl expression>' (alpha testing)
Use I<tag> as a replacement string for I<perl expression>. This makes
it possible to define your own replacement strings. GNU B<parallel>'s
@ -1461,9 +1473,9 @@ Use the replacement string I<replace-str> instead of B<{#}> for
job sequence number.
=item B<--shebang> (beta testing)
=item B<--shebang>
=item B<--hashbang> (beta testing)
=item B<--hashbang>
GNU B<parallel> can be called as a shebang (#!) command as the first
line of a script. The content of the file will be treated as
@ -1488,7 +1500,7 @@ On FreeBSD B<env> is needed:
freenetproject.org
=item B<--shebang-wrap> (beta testing)
=item B<--shebang-wrap>
GNU B<parallel> can parallelize scripts by wrapping the shebang
line. If the program can be run like this:
@ -1537,9 +1549,9 @@ I<secs> seconds after starting each ssh. I<secs> can be less than 1
seconds.
=item B<-S> I<[ncpu/]sshlogin[,[ncpu/]sshlogin[,...]]>
=item B<-S> I<[ncpu/]sshlogin[,[ncpu/]sshlogin[,...]]> (alpha testing)
=item B<--sshlogin> I<[ncpu/]sshlogin[,[ncpu/]sshlogin[,...]]>
=item B<--sshlogin> I<[ncpu/]sshlogin[,[ncpu/]sshlogin[,...]]> (alpha testing)
Distribute jobs to remote computers. The jobs will be run on a list of
remote computers. GNU B<parallel> will determine the number of CPU
@ -1575,9 +1587,9 @@ B<--sshlogin> is often used with B<--transfer>, B<--return>,
B<--cleanup>, and B<--trc>.
=item B<--sshloginfile> I<filename>
=item B<--sshloginfile> I<filename> (alpha testing)
=item B<--slf> I<filename>
=item B<--slf> I<filename> (alpha testing)
File with sshlogins. The file consists of sshlogins on separate
lines. Empty lines and lines starting with '#' are ignored. Example:
@ -1662,7 +1674,7 @@ different dir for the files. Setting B<--tmpdir> is equivalent to
setting $TMPDIR.
=item B<--tmux> (alpha testing)
=item B<--tmux> (beta testing)
Use B<tmux> for output. Start a B<tmux> session and run each job in a
window in that session. No other output will be produced.
@ -1688,7 +1700,7 @@ Print the job to be run on stderr (standard error).
See also B<-v>, B<-p>.
=item B<--transfer>
=item B<--transfer> (alpha testing)
Transfer files to remote computers. B<--transfer> is used with
B<--sshlogin> when the arguments are files and should be transferred

2
src/sql
View file

@ -566,7 +566,7 @@ $Global::Initfile && unlink $Global::Initfile;
exit ($err);
sub parse_options {
$Global::version = 20140815;
$Global::version = 20140822;
$Global::progname = 'sql';
# This must be done first as this may exec myself