Released as 20161222 ('Castro')

This commit is contained in:
Ole Tange 2016-12-22 14:27:00 +01:00
parent 9b4fd8b33b
commit a92ca3bf92
12 changed files with 372 additions and 282 deletions

66
NEWS
View file

@ -1,3 +1,68 @@
20161222
* --results foo.csv will save output as a CSV-file. Can be directly
imported into R or LibreCalc.
* When using --pipepart a negative --block-size is not interpreted as
a block size but as the number of blocks each jobslot should have.
* --sqlmaster/--sqlandworker will append jobs to the DBURL if the
DBURL is prepended with +.
* GNU Parallel was cited in: A cloud-based workflow to quantify
transcript-expression levels in public cancer compendia
http://www.nature.com/articles/srep39259
* GNU Parallel was cited in: Learning string distance with smoothing
for OCR spelling correction
http://paperity.org/p/78557440/learning-string-distance-with-smoothing-for-ocr-spelling-correction
* GNU Parallel was cited in: Transient compute clustering with GNU
Parallel and sshfs
https://gist.github.com/Brainiarc7/24c966c8a001061ee86cc4bc05826bf4
* GNU Parallel was cited in: Determination of crystal structures of
proteins of unknown identity using a marathon molecular replacement
procedure: Structure of Stenotrophomonas maltophilia
phosphate-binding protein
https://www.researchgate.net/publication/308186413_Determination_of_crystal_structures_of_proteins_of_unknown_identity_using_a_marathon_molecular_replacement_procedure_Structure_of_Stenotrophomonas_maltophilia_phosphate-binding_protein
* GNU Parallel was cited in: The Outer Solar System Origins Survey:
I. Design and First-Quarter Discoveries
https://arxiv.org/pdf/1511.02895v2.pdf
* GNU Parallel was cited in: Large-scale benchmarking reveals false
discoveries and count transformation sensitivity in 16S rRNA gene
amplicon data analysis methods used in microbiome studies
http://microbiomejournal.biomedcentral.com/articles/10.1186/s40168-016-0208-8
* GNU Parallel was cited in: Decomposing Images into Layers via
RGB-space Geometry
https://cs.gmu.edu/~ygingold/singleimage/Decomposing%20Images%20into%20Layers%20via%20RGB-space%20Geometry%20(Tan%20et%20al%202016%20TOG)%20small.pdf
* 4 Ways to Batch Convert Your PNG to JPG and Vice-Versa
http://www.tecmint.com/linux-image-conversion-tools/
* All's Fair in Love and Distributed Storage
http://cohesity.com/blog/alls-fair-love-distributed-storage/
* How can I use GNU Parallel to run a lot of commands in parallel
https://www.msi.umn.edu/support/faq/how-can-i-use-gnu-parallel-run-lot-commands-parallel
* 정해영의 블로그 - JEONG Haeyoung's blog
http://blog.genoglobe.com/2016/11/gnu-parallel.html
* 在Linux下将PNG和JPG批量互转的四种方法
http://os.51cto.com/art/201612/524182.htm
* Running in parallel http://tomkimpson.com/howto/gnuparallel/
* Taco Bell Parallel Programming
https://giorgos.sealabs.net/taco-bell-parallel-programming.html
* Bug fixes and man page updates.
20161122
* --record-env can now be used with env_parallel for bash, ksh, pdksh,
@ -16,6 +81,7 @@
* Tutorial: Parallel web scraping with CasperJS and GNU Parallel
http://g-liu.com/blog/2016/10/tutorial-parallel-web-scraping-with-casperjs-and-gnu-parallel/
* Bug fixes and man page updates.

12
README
View file

@ -44,9 +44,9 @@ document.
Full installation of GNU Parallel is as simple as:
wget http://ftpmirror.gnu.org/parallel/parallel-20161122.tar.bz2
bzip2 -dc parallel-20161122.tar.bz2 | tar xvf -
cd parallel-20161122
wget http://ftpmirror.gnu.org/parallel/parallel-20161222.tar.bz2
bzip2 -dc parallel-20161222.tar.bz2 | tar xvf -
cd parallel-20161222
./configure && make && sudo make install
@ -55,9 +55,9 @@ Full installation of GNU Parallel is as simple as:
If you are not root you can add ~/bin to your path and install in
~/bin and ~/share:
wget http://ftpmirror.gnu.org/parallel/parallel-20161122.tar.bz2
bzip2 -dc parallel-20161122.tar.bz2 | tar xvf -
cd parallel-20161122
wget http://ftpmirror.gnu.org/parallel/parallel-20161222.tar.bz2
bzip2 -dc parallel-20161222.tar.bz2 | tar xvf -
cd parallel-20161222
./configure --prefix=$HOME && make && make install
Or if your system lacks 'make' you can simply copy src/parallel

20
configure vendored
View file

@ -1,6 +1,6 @@
#! /bin/sh
# Guess values for system-dependent variables and create Makefiles.
# Generated by GNU Autoconf 2.69 for parallel 20161122.
# Generated by GNU Autoconf 2.69 for parallel 20161222.
#
# Report bugs to <bug-parallel@gnu.org>.
#
@ -579,8 +579,8 @@ MAKEFLAGS=
# Identity of this package.
PACKAGE_NAME='parallel'
PACKAGE_TARNAME='parallel'
PACKAGE_VERSION='20161122'
PACKAGE_STRING='parallel 20161122'
PACKAGE_VERSION='20161222'
PACKAGE_STRING='parallel 20161222'
PACKAGE_BUGREPORT='bug-parallel@gnu.org'
PACKAGE_URL=''
@ -1203,7 +1203,7 @@ if test "$ac_init_help" = "long"; then
# Omit some internal or obsolete options to make the list less imposing.
# This message is too long to be a string in the A/UX 3.1 sh.
cat <<_ACEOF
\`configure' configures parallel 20161122 to adapt to many kinds of systems.
\`configure' configures parallel 20161222 to adapt to many kinds of systems.
Usage: $0 [OPTION]... [VAR=VALUE]...
@ -1269,7 +1269,7 @@ fi
if test -n "$ac_init_help"; then
case $ac_init_help in
short | recursive ) echo "Configuration of parallel 20161122:";;
short | recursive ) echo "Configuration of parallel 20161222:";;
esac
cat <<\_ACEOF
@ -1345,7 +1345,7 @@ fi
test -n "$ac_init_help" && exit $ac_status
if $ac_init_version; then
cat <<\_ACEOF
parallel configure 20161122
parallel configure 20161222
generated by GNU Autoconf 2.69
Copyright (C) 2012 Free Software Foundation, Inc.
@ -1362,7 +1362,7 @@ cat >config.log <<_ACEOF
This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.
It was created by parallel $as_me 20161122, which was
It was created by parallel $as_me 20161222, which was
generated by GNU Autoconf 2.69. Invocation command line was
$ $0 $@
@ -2225,7 +2225,7 @@ fi
# Define the identity of the package.
PACKAGE='parallel'
VERSION='20161122'
VERSION='20161222'
cat >>confdefs.h <<_ACEOF
@ -2867,7 +2867,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
# report actual input values of CONFIG_FILES etc. instead of their
# values after options handling.
ac_log="
This file was extended by parallel $as_me 20161122, which was
This file was extended by parallel $as_me 20161222, which was
generated by GNU Autoconf 2.69. Invocation command line was
CONFIG_FILES = $CONFIG_FILES
@ -2929,7 +2929,7 @@ _ACEOF
cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`"
ac_cs_version="\\
parallel config.status 20161122
parallel config.status 20161222
configured by $0, generated by GNU Autoconf 2.69,
with options \\"\$ac_cs_config\\"

View file

@ -1,4 +1,4 @@
AC_INIT([parallel], [20161122], [bug-parallel@gnu.org])
AC_INIT([parallel], [20161222], [bug-parallel@gnu.org])
AM_INIT_AUTOMAKE([-Wall -Werror foreign])
AC_CONFIG_HEADERS([config.h])
AC_CONFIG_FILES([

View file

@ -1,7 +1,7 @@
#!/bin/sh
# install - install a program, script, or datafile
scriptversion=2014-09-12.12; # UTC
scriptversion=2011-11-20.07; # UTC
# This originates from X11R5 (mit/util/scripts/install.sh), which was
# later released in X11R6 (xc/config/util/install.sh) with the
@ -41,15 +41,19 @@ scriptversion=2014-09-12.12; # UTC
# This script is compatible with the BSD install script, but was written
# from scratch.
tab=' '
nl='
'
IFS=" $tab$nl"
IFS=" "" $nl"
# Set DOITPROG to "echo" to test this script.
# set DOITPROG to echo to test this script
# Don't use :- since 4.3BSD and earlier shells don't like it.
doit=${DOITPROG-}
doit_exec=${doit:-exec}
if test -z "$doit"; then
doit_exec=exec
else
doit_exec=$doit
fi
# Put in absolute file names if you don't have them in your path;
# or use environment vars.
@ -64,6 +68,17 @@ mvprog=${MVPROG-mv}
rmprog=${RMPROG-rm}
stripprog=${STRIPPROG-strip}
posix_glob='?'
initialize_posix_glob='
test "$posix_glob" != "?" || {
if (set -f) 2>/dev/null; then
posix_glob=
else
posix_glob=:
fi
}
'
posix_mkdir=
# Desired mode of installed file.
@ -82,7 +97,7 @@ dir_arg=
dst_arg=
copy_on_change=false
is_target_a_directory=possibly
no_target_directory=
usage="\
Usage: $0 [OPTION]... [-T] SRCFILE DSTFILE
@ -122,57 +137,46 @@ while test $# -ne 0; do
-d) dir_arg=true;;
-g) chgrpcmd="$chgrpprog $2"
shift;;
shift;;
--help) echo "$usage"; exit $?;;
-m) mode=$2
case $mode in
*' '* | *"$tab"* | *"$nl"* | *'*'* | *'?'* | *'['*)
echo "$0: invalid mode: $mode" >&2
exit 1;;
esac
shift;;
case $mode in
*' '* | *' '* | *'
'* | *'*'* | *'?'* | *'['*)
echo "$0: invalid mode: $mode" >&2
exit 1;;
esac
shift;;
-o) chowncmd="$chownprog $2"
shift;;
shift;;
-s) stripcmd=$stripprog;;
-t)
is_target_a_directory=always
dst_arg=$2
# Protect names problematic for 'test' and other utilities.
case $dst_arg in
-* | [=\(\)!]) dst_arg=./$dst_arg;;
esac
shift;;
-t) dst_arg=$2
# Protect names problematic for 'test' and other utilities.
case $dst_arg in
-* | [=\(\)!]) dst_arg=./$dst_arg;;
esac
shift;;
-T) is_target_a_directory=never;;
-T) no_target_directory=true;;
--version) echo "$0 $scriptversion"; exit $?;;
--) shift
break;;
--) shift
break;;
-*) echo "$0: invalid option: $1" >&2
exit 1;;
-*) echo "$0: invalid option: $1" >&2
exit 1;;
*) break;;
esac
shift
done
# We allow the use of options -d and -T together, by making -d
# take the precedence; this is for compatibility with GNU install.
if test -n "$dir_arg"; then
if test -n "$dst_arg"; then
echo "$0: target directory not allowed when installing a directory." >&2
exit 1
fi
fi
if test $# -ne 0 && test -z "$dir_arg$dst_arg"; then
# When -d is used, all remaining arguments are directories to create.
# When -t is used, the destination is already specified.
@ -203,15 +207,6 @@ if test $# -eq 0; then
exit 0
fi
if test -z "$dir_arg"; then
if test $# -gt 1 || test "$is_target_a_directory" = always; then
if test ! -d "$dst_arg"; then
echo "$0: $dst_arg: Is not a directory." >&2
exit 1
fi
fi
fi
if test -z "$dir_arg"; then
do_exit='(exit $ret); exit $ret'
trap "ret=129; $do_exit" 1
@ -228,16 +223,16 @@ if test -z "$dir_arg"; then
*[0-7])
if test -z "$stripcmd"; then
u_plus_rw=
u_plus_rw=
else
u_plus_rw='% 200'
u_plus_rw='% 200'
fi
cp_umask=`expr '(' 777 - $mode % 1000 ')' $u_plus_rw`;;
*)
if test -z "$stripcmd"; then
u_plus_rw=
u_plus_rw=
else
u_plus_rw=,u+rw
u_plus_rw=,u+rw
fi
cp_umask=$mode$u_plus_rw;;
esac
@ -274,15 +269,41 @@ do
# If destination is a directory, append the input filename; won't work
# if double slashes aren't ignored.
if test -d "$dst"; then
if test "$is_target_a_directory" = never; then
echo "$0: $dst_arg: Is a directory" >&2
exit 1
if test -n "$no_target_directory"; then
echo "$0: $dst_arg: Is a directory" >&2
exit 1
fi
dstdir=$dst
dst=$dstdir/`basename "$src"`
dstdir_status=0
else
dstdir=`dirname "$dst"`
# Prefer dirname, but fall back on a substitute if dirname fails.
dstdir=`
(dirname "$dst") 2>/dev/null ||
expr X"$dst" : 'X\(.*[^/]\)//*[^/][^/]*/*$' \| \
X"$dst" : 'X\(//\)[^/]' \| \
X"$dst" : 'X\(//\)$' \| \
X"$dst" : 'X\(/\)' \| . 2>/dev/null ||
echo X"$dst" |
sed '/^X\(.*[^/]\)\/\/*[^/][^/]*\/*$/{
s//\1/
q
}
/^X\(\/\/\)[^/].*/{
s//\1/
q
}
/^X\(\/\/\)$/{
s//\1/
q
}
/^X\(\/\).*/{
s//\1/
q
}
s/.*/./; q'
`
test -d "$dstdir"
dstdir_status=$?
fi
@ -293,81 +314,74 @@ do
if test $dstdir_status != 0; then
case $posix_mkdir in
'')
# Create intermediate dirs using mode 755 as modified by the umask.
# This is like FreeBSD 'install' as of 1997-10-28.
umask=`umask`
case $stripcmd.$umask in
# Optimize common cases.
*[2367][2367]) mkdir_umask=$umask;;
.*0[02][02] | .[02][02] | .[02]) mkdir_umask=22;;
# Create intermediate dirs using mode 755 as modified by the umask.
# This is like FreeBSD 'install' as of 1997-10-28.
umask=`umask`
case $stripcmd.$umask in
# Optimize common cases.
*[2367][2367]) mkdir_umask=$umask;;
.*0[02][02] | .[02][02] | .[02]) mkdir_umask=22;;
*[0-7])
mkdir_umask=`expr $umask + 22 \
- $umask % 100 % 40 + $umask % 20 \
- $umask % 10 % 4 + $umask % 2
`;;
*) mkdir_umask=$umask,go-w;;
esac
*[0-7])
mkdir_umask=`expr $umask + 22 \
- $umask % 100 % 40 + $umask % 20 \
- $umask % 10 % 4 + $umask % 2
`;;
*) mkdir_umask=$umask,go-w;;
esac
# With -d, create the new directory with the user-specified mode.
# Otherwise, rely on $mkdir_umask.
if test -n "$dir_arg"; then
mkdir_mode=-m$mode
else
mkdir_mode=
fi
# With -d, create the new directory with the user-specified mode.
# Otherwise, rely on $mkdir_umask.
if test -n "$dir_arg"; then
mkdir_mode=-m$mode
else
mkdir_mode=
fi
posix_mkdir=false
case $umask in
*[123567][0-7][0-7])
# POSIX mkdir -p sets u+wx bits regardless of umask, which
# is incompatible with FreeBSD 'install' when (umask & 300) != 0.
;;
*)
# $RANDOM is not portable (e.g. dash); use it when possible to
# lower collision chance
tmpdir=${TMPDIR-/tmp}/ins$RANDOM-$$
trap 'ret=$?; rmdir "$tmpdir/a/b" "$tmpdir/a" "$tmpdir" 2>/dev/null; exit $ret' 0
posix_mkdir=false
case $umask in
*[123567][0-7][0-7])
# POSIX mkdir -p sets u+wx bits regardless of umask, which
# is incompatible with FreeBSD 'install' when (umask & 300) != 0.
;;
*)
tmpdir=${TMPDIR-/tmp}/ins$RANDOM-$$
trap 'ret=$?; rmdir "$tmpdir/d" "$tmpdir" 2>/dev/null; exit $ret' 0
# As "mkdir -p" follows symlinks and we work in /tmp possibly; so
# create the $tmpdir first (and fail if unsuccessful) to make sure
# that nobody tries to guess the $tmpdir name.
if (umask $mkdir_umask &&
$mkdirprog $mkdir_mode "$tmpdir" &&
exec $mkdirprog $mkdir_mode -p -- "$tmpdir/a/b") >/dev/null 2>&1
then
if test -z "$dir_arg" || {
# Check for POSIX incompatibilities with -m.
# HP-UX 11.23 and IRIX 6.5 mkdir -m -p sets group- or
# other-writable bit of parent directory when it shouldn't.
# FreeBSD 6.1 mkdir -m -p sets mode of existing directory.
test_tmpdir="$tmpdir/a"
ls_ld_tmpdir=`ls -ld "$test_tmpdir"`
case $ls_ld_tmpdir in
d????-?r-*) different_mode=700;;
d????-?--*) different_mode=755;;
*) false;;
esac &&
$mkdirprog -m$different_mode -p -- "$test_tmpdir" && {
ls_ld_tmpdir_1=`ls -ld "$test_tmpdir"`
test "$ls_ld_tmpdir" = "$ls_ld_tmpdir_1"
}
}
then posix_mkdir=:
fi
rmdir "$tmpdir/a/b" "$tmpdir/a" "$tmpdir"
else
# Remove any dirs left behind by ancient mkdir implementations.
rmdir ./$mkdir_mode ./-p ./-- "$tmpdir" 2>/dev/null
fi
trap '' 0;;
esac;;
if (umask $mkdir_umask &&
exec $mkdirprog $mkdir_mode -p -- "$tmpdir/d") >/dev/null 2>&1
then
if test -z "$dir_arg" || {
# Check for POSIX incompatibilities with -m.
# HP-UX 11.23 and IRIX 6.5 mkdir -m -p sets group- or
# other-writable bit of parent directory when it shouldn't.
# FreeBSD 6.1 mkdir -m -p sets mode of existing directory.
ls_ld_tmpdir=`ls -ld "$tmpdir"`
case $ls_ld_tmpdir in
d????-?r-*) different_mode=700;;
d????-?--*) different_mode=755;;
*) false;;
esac &&
$mkdirprog -m$different_mode -p -- "$tmpdir" && {
ls_ld_tmpdir_1=`ls -ld "$tmpdir"`
test "$ls_ld_tmpdir" = "$ls_ld_tmpdir_1"
}
}
then posix_mkdir=:
fi
rmdir "$tmpdir/d" "$tmpdir"
else
# Remove any dirs left behind by ancient mkdir implementations.
rmdir ./$mkdir_mode ./-p ./-- 2>/dev/null
fi
trap '' 0;;
esac;;
esac
if
$posix_mkdir && (
umask $mkdir_umask &&
$doit_exec $mkdirprog $mkdir_mode -p -- "$dstdir"
umask $mkdir_umask &&
$doit_exec $mkdirprog $mkdir_mode -p -- "$dstdir"
)
then :
else
@ -377,51 +391,53 @@ do
# directory the slow way, step by step, checking for races as we go.
case $dstdir in
/*) prefix='/';;
[-=\(\)!]*) prefix='./';;
*) prefix='';;
/*) prefix='/';;
[-=\(\)!]*) prefix='./';;
*) prefix='';;
esac
eval "$initialize_posix_glob"
oIFS=$IFS
IFS=/
set -f
$posix_glob set -f
set fnord $dstdir
shift
set +f
$posix_glob set +f
IFS=$oIFS
prefixes=
for d
do
test X"$d" = X && continue
test X"$d" = X && continue
prefix=$prefix$d
if test -d "$prefix"; then
prefixes=
else
if $posix_mkdir; then
(umask=$mkdir_umask &&
$doit_exec $mkdirprog $mkdir_mode -p -- "$dstdir") && break
# Don't fail if two instances are running concurrently.
test -d "$prefix" || exit 1
else
case $prefix in
*\'*) qprefix=`echo "$prefix" | sed "s/'/'\\\\\\\\''/g"`;;
*) qprefix=$prefix;;
esac
prefixes="$prefixes '$qprefix'"
fi
fi
prefix=$prefix/
prefix=$prefix$d
if test -d "$prefix"; then
prefixes=
else
if $posix_mkdir; then
(umask=$mkdir_umask &&
$doit_exec $mkdirprog $mkdir_mode -p -- "$dstdir") && break
# Don't fail if two instances are running concurrently.
test -d "$prefix" || exit 1
else
case $prefix in
*\'*) qprefix=`echo "$prefix" | sed "s/'/'\\\\\\\\''/g"`;;
*) qprefix=$prefix;;
esac
prefixes="$prefixes '$qprefix'"
fi
fi
prefix=$prefix/
done
if test -n "$prefixes"; then
# Don't fail if two instances are running concurrently.
(umask $mkdir_umask &&
eval "\$doit_exec \$mkdirprog $prefixes") ||
test -d "$dstdir" || exit 1
obsolete_mkdir_used=true
# Don't fail if two instances are running concurrently.
(umask $mkdir_umask &&
eval "\$doit_exec \$mkdirprog $prefixes") ||
test -d "$dstdir" || exit 1
obsolete_mkdir_used=true
fi
fi
fi
@ -456,12 +472,15 @@ do
# If -C, don't bother to copy if it wouldn't change the file.
if $copy_on_change &&
old=`LC_ALL=C ls -dlL "$dst" 2>/dev/null` &&
new=`LC_ALL=C ls -dlL "$dsttmp" 2>/dev/null` &&
set -f &&
old=`LC_ALL=C ls -dlL "$dst" 2>/dev/null` &&
new=`LC_ALL=C ls -dlL "$dsttmp" 2>/dev/null` &&
eval "$initialize_posix_glob" &&
$posix_glob set -f &&
set X $old && old=:$2:$4:$5:$6 &&
set X $new && new=:$2:$4:$5:$6 &&
set +f &&
$posix_glob set +f &&
test "$old" = "$new" &&
$cmpprog "$dst" "$dsttmp" >/dev/null 2>&1
then
@ -474,24 +493,24 @@ do
# to itself, or perhaps because mv is so ancient that it does not
# support -f.
{
# Now remove or move aside any old file at destination location.
# We try this two ways since rm can't unlink itself on some
# systems and the destination file might be busy for other
# reasons. In this case, the final cleanup might fail but the new
# file should still install successfully.
{
test ! -f "$dst" ||
$doit $rmcmd -f "$dst" 2>/dev/null ||
{ $doit $mvcmd -f "$dst" "$rmtmp" 2>/dev/null &&
{ $doit $rmcmd -f "$rmtmp" 2>/dev/null; :; }
} ||
{ echo "$0: cannot unlink or rename $dst" >&2
(exit 1); exit 1
}
} &&
# Now remove or move aside any old file at destination location.
# We try this two ways since rm can't unlink itself on some
# systems and the destination file might be busy for other
# reasons. In this case, the final cleanup might fail but the new
# file should still install successfully.
{
test ! -f "$dst" ||
$doit $rmcmd -f "$dst" 2>/dev/null ||
{ $doit $mvcmd -f "$dst" "$rmtmp" 2>/dev/null &&
{ $doit $rmcmd -f "$rmtmp" 2>/dev/null; :; }
} ||
{ echo "$0: cannot unlink or rename $dst" >&2
(exit 1); exit 1
}
} &&
# Now rename the file to the real destination.
$doit $mvcmd "$dsttmp" "$dst"
# Now rename the file to the real destination.
$doit $mvcmd "$dsttmp" "$dst"
}
fi || exit 1

View file

@ -3,7 +3,7 @@
scriptversion=2013-10-28.13; # UTC
# Copyright (C) 1996-2014 Free Software Foundation, Inc.
# Copyright (C) 1996-2013 Free Software Foundation, Inc.
# Originally written by Fran,cois Pinard <pinard@iro.umontreal.ca>, 1996.
# This program is free software; you can redistribute it and/or modify

View file

@ -1,6 +1,6 @@
Summary: Shell tool for executing jobs in parallel
Name: parallel
Version: 20161122
Version: 20161222
Release: 1.1
License: GPL
Group: Productivity/File utilities

View file

@ -24,7 +24,7 @@
use strict;
use Getopt::Long;
$Global::progname="niceload";
$Global::version = 20161123;
$Global::version = 20161222;
Getopt::Long::Configure("bundling","require_order");
get_options_from_array(\@ARGV) || die_usage();
if($opt::version) {

View file

@ -227,8 +227,10 @@ if($opt::halt and $Global::halt_when ne "never") {
wait_and_exit(min(undef_as_zero($Global::exitstatus),101));
}
sub __PIPE_MODE__ {}
sub pipe_part_files {
# Input:
# $file = the file to read
@ -769,8 +771,10 @@ sub write_record_to_pipe {
return 1;
}
sub __SEM_MODE__ {}
sub acquire_semaphore {
# Acquires semaphore. If needed: spawns to the background
# Uses:
@ -793,8 +797,10 @@ sub acquire_semaphore {
return $sem;
}
sub __PARSE_OPTIONS__ {}
sub options_hash {
# Returns:
# %hash = the GetOptions config
@ -1078,10 +1084,13 @@ sub parse_options {
}
if(defined $opt::results) {
# Is the output a dir or CSV-file?
if($opt::results =~ /\.csv$/i) {
# CSV with , as separator
$Global::csv = ",";
$Global::membuffer ||= 1;
} elsif($opt::results =~ /\.tsv$/i) {
# CSV with TAB as separator
$Global::csv = "\t";
$Global::membuffer ||= 1;
}
@ -1250,7 +1259,7 @@ sub check_invalid_option_combinations {
sub init_globals {
# Defaults:
$Global::version = 20161208;
$Global::version = 20161222;
$Global::progname = 'parallel';
$Global::infinity = 2**31;
$Global::debug = 0;
@ -1636,7 +1645,7 @@ sub find_compression_program {
# parallel --shuf -j1 --joblog jl-m --arg-sep , parallel --compress-program \'{3}" "-{2}\' cat ::: 1gb '>'/dev/null , 1 2 3 , {1..3} , $multithread
# sort -nk4 jl-?
# 1-core:
# 2-cores:
# 2-cores: lz4 lzop zstd gzip bzip2 lzma xz clzip pigz lbzip2 pbzip2 lzip lrz plzip pxz pzstd
# 4-cores:
# 8-cores: pzstd lz4 zstd pigz lzop lbzip2 pbzip2 gzip lzip lrz plzip pxz bzip2 lzma xz clzip
# 16-cores: pzstd lz4 pigz lzop lbzip2 pbzip2 plzip lzip lrz pxz gzip lzma xz bzip2
@ -1653,7 +1662,6 @@ sub find_compression_program {
return ("cat","cat");
}
sub read_options {
# Read options from command line, profile and $PARALLEL
# Uses:
@ -1889,8 +1897,10 @@ sub cleanup {
}
}
sub __QUOTING_ARGUMENTS_FOR_SHELL__ {}
sub shell_quote {
# Input:
# @strings = strings to be quoted
@ -2038,6 +2048,7 @@ sub unquote_printf {
return $_;
}
sub __FILEHANDLES__ {}
@ -2154,6 +2165,7 @@ sub set_fh_non_blocking {
sub __RUNNING_THE_JOBS_AND_PRINTING_PROGRESS__ {}
# Variable structure:
#
# $Global::running{$pid} = Pointer to Job-object
@ -2870,8 +2882,10 @@ sub get_job_with_sshlogin {
return $job;
}
sub __REMOTE_SSH__ {}
sub read_sshloginfiles {
# Read a list of --slf's
# Input:
@ -3106,7 +3120,7 @@ sub cleanup_basefile {
sub run_parallel {
my ($stdin,@args) = @_;
my $cmd = join "",map { " $_ & " } split /\n/, $stdin;
print $Global::original_stderr ` $cmd wait` ;
print $Global::original_stderr ` $cmd wait` ;
return 0
}
@ -3507,8 +3521,10 @@ sub onall {
}
}
sub __SIGNAL_HANDLING__ {}
sub tstp {
# Send TSTP signal (Ctrl-Z) to all children process groups
# Uses:
@ -3520,7 +3536,6 @@ sub tstp {
kill "TSTP", $$;
}
sub save_original_signal_handler {
# Remember the original signal handler
# Uses:
@ -3662,8 +3677,10 @@ sub reaper {
return wantarray ? @pids_reaped : $children_reaped;
}
sub __USAGE__ {}
sub killall {
# Kill all jobs by killing their process groups
@ -3790,7 +3807,6 @@ sub usage {
"",);
}
sub citation_notice {
# if --will-cite or --plain: do nothing
# if stderr redirected: do nothing
@ -3953,8 +3969,10 @@ sub show_limits {
"press CTRL-D or CTRL-C\n");
}
sub __GENERIC_COMMON_FUNCTION__ {}
sub mkdir_or_die {
# If dir is not executable: die
my $dir = shift;
@ -4003,13 +4021,13 @@ sub tmpname {
return $tmpname;
}
sub tmpfifo {
# Find an unused name and mkfifo on it
use POSIX qw(mkfifo);
my $tmpfifo = tmpname("fif",@_);
mkfifo($tmpfifo,0600);
return $tmpfifo;
}
#sub tmpfifo {
# # Find an unused name and mkfifo on it
# use POSIX qw(mkfifo);
# my $tmpfifo = tmpname("fif",@_);
# mkfifo($tmpfifo,0600);
# return $tmpfifo;
#}
sub rm {
# Remove file and remove it from %Global::unlink
@ -4484,8 +4502,10 @@ sub kill_youngster_if_not_enough_mem {
}
}
sub __DEBUGGING__ {}
sub debug {
# Uses:
# $Global::debug
@ -4578,8 +4598,10 @@ sub my_carp {
carp(@_);
}
sub __OBJECT_ORIENTED_PARTS__ {}
package SSHLogin;
sub new {
@ -7382,7 +7404,9 @@ sub sshlogin_wrap {
my @vars = vars_to_export();
my $csh_friendly = not grep { /\n/ } @ENV{@vars};
my @bash_functions = grep { substr($ENV{$_},0,4) eq "() {" } @vars;
my @non_functions = grep { substr($ENV{$_},0,4) ne "() {" } @vars;
my @non_functions = (grep { !/PARALLEL_ENV/ }
grep { substr($ENV{$_},0,4) ne "() {" } @vars);
# eval of @envset will set %ENV
my $envset = join"", map {
'$ENV{"'.::perl_quote_scalar($_).'"}="'.
@ -8245,7 +8269,7 @@ sub print {
# Memory optimization: Overwrite with the joined output
$self->{'output'}{1} = join("", @{$self->{'output'}{1}});
$self->{'output'}{2} = join("", @{$self->{'output'}{2}});
print $Global::csv_fh
print $Global::csv_fh
(map { $$_ }
combine_ref
($self->seq(),
@ -8296,7 +8320,7 @@ sub combine_ref {
$must_be_quoted ||=1;
}
if($must_be_quoted){
push \@out, \$sep, \$quot, $column, \$quot;
push @out, \$sep, \$quot, $column, \$quot;
} else {
push @out, \$sep, $column;
}
@ -8306,46 +8330,6 @@ sub combine_ref {
return @out;
}
sub combine {
# Inspired by Text::CSV_PP::_combine (by Makamaka Hannyaharamitu)
my @part = @_;
my $sep = $Global::csv;
my $quot = '"';
my $must_be_quoted;
for my $column (@part) {
# Memory optimization: Content transferred as reference
if(ref $column ne "SCALAR") {
($column,$$column) = ($$column,"");
}
if(not defined $column) {
$column = '';
next;
}
$must_be_quoted = 0;
if($column =~ s/$quot/$quot$quot/go){
# Replace " => ""
$must_be_quoted ||=1;
}
if($column =~ /[\s\Q$sep\E]/o){
# Put quotes around if the column contains ,
$must_be_quoted ||=1;
}
$Global::use{"bytes"} ||= eval "use bytes; 1;";
if ($column =~ /\0/) {
# Contains \0 => put quotes around
$must_be_quoted ||=1;
}
if($must_be_quoted){
$column = $quot . $column . $quot;
}
}
return join($sep, @part) . "\n";
}
sub print_files {
# Print the name of the file containing stdout on stdout
# Uses:
@ -8702,6 +8686,7 @@ sub set_exitsignal {
}
}
package CommandLine;
sub new {

View file

@ -499,7 +499,7 @@ If you use B<-N>, B<--block-size> should be bigger than N+1 records.
I<size> defaults to 1M.
When using B<--pipepart> negative a block size is not interpreted as a
When using B<--pipepart> a negative block size is not interpreted as a
blocksize but as the number of blocks each jobslot should have. So
this will run 10*5 = 50 jobs in total:
@ -1670,9 +1670,9 @@ B<--return> is ignored when used with B<--sshlogin :> or when not used
with B<--sshlogin>.
=item B<--round-robin>
=item B<--round-robin> (alpha testing)
=item B<--round>
=item B<--round> (alpha testing)
Normally B<--pipe> will give a single block to each instance of the
command. With B<--round-robin> all blocks will at random be written to
@ -3427,7 +3427,7 @@ Note: The default for GNU B<parallel> is to remove the spaces around the columns
parallel -a table_file.tsv --trim n --colsep '\t' cmd -o {2} -i {1}
=head1 EXAMPLE: Output as CSV-file
=head1 EXAMPLE: Output to database
GNU B<parallel> can output to a database table and a CSV-file:
@ -3464,6 +3464,18 @@ Or MySQL:
perl -pe 's/"/""/g;s/\t/","/g;s/^/"/;s/$/"/;s/\\\\/\\/g;s/\\t/\t/g;s/\\n/\n/g;' mytable.tsv
=head1 EXAMPLE: Output to CSV-file for R
If you have no need for the advanced job distribution control that a
database provides, but you simply want output into a CSV file that you
can read into R or LibreCalc, then you can use B<--results>:
parallel --results my.csv seq ::: 10 20 30
R
> mydf <- read.csv("my.csv");
> print(mydf[2,])
> write(as.character(mydf[2,c("Stdout")]),'')
=head1 EXAMPLE: Run the same command 10 times
If you want to run the same command with the same arguments 10 times

View file

@ -69,12 +69,6 @@ The only way to have access to the environment is directly from the
shell, so the program must be written in a shell script that will be
sourced and there has to deal with the dialect of the relevant shell.
Dumping the environment into a format that can be activated again is
easy in Bash: B<typeset> and B<alias> basically give you the commands
you must run to re-create the environment. It is much harder in B<csh>
where there is no easy way to convert an environment into commands
that generate the environment.
=head3 env_parallel.*
@ -183,10 +177,10 @@ compresses the buffered data. This is a bit tricky because there
should be no files to clean up if GNU B<parallel> is killed by a power
outage.
GNU B<parallel> first selects a compression program. If the user has not
selected one, the first of these that is in $PATH is used: B<lz4 pigz
lzop plzip pbzip2 pxz gzip lzma xz bzip2 lzip>. They are sorted by
speed on a 16 core machine.
GNU B<parallel> first selects a compression program. If the user has
not selected one, the first of these that is in $PATH is used: B<pzstd
lbzip2 pbzip2 zstd pigz lz4 lzop plzip lzip lrz gzip pxz lzma bzip2 xz
clzip>. They are sorted by speed on a 32 core machine.
Schematically the setup is as follows:
@ -449,23 +443,6 @@ that correctly for all corner cases is next to impossible to do by
hand.
=head2 --block-size adjustment
Every time GNU B<parallel> detects a record bigger than
B<--block-size> it increases the block size by 30%. A small
B<--block-size> gives very poor performance; by exponentially
increasing the block size performance will not suffer.
GNU B<parallel> will waste CPU power if B<--block-size> does not
contain a full record, because it tries to find a full record and will
fail to do so. The recommendation is therefore to use a
B<--block-size> > 2 records, so you always get at least one full
record when you read one block.
If you use B<-N> then B<--block-size> should be big enough to contain
N+1 records.
=head2 Shell shock
The shell shock bug in B<bash> did not affect GNU B<parallel>, but the
@ -736,6 +713,36 @@ data. See this for a surprising example:
yes | dd bs=1024k count=10 | wc
=head2 --block-size adjustment
Every time GNU B<parallel> detects a record bigger than
B<--block-size> it increases the block size by 30%. A small
B<--block-size> gives very poor performance; by exponentially
increasing the block size performance will not suffer.
GNU B<parallel> will waste CPU power if B<--block-size> does not
contain a full record, because it tries to find a full record and will
fail to do so. The recommendation is therefore to use a
B<--block-size> > 2 records, so you always get at least one full
record when you read one block.
If you use B<-N> then B<--block-size> should be big enough to contain
N+1 records.
=head2 Automatic --block-size computation
With B<--pipepart> GNU B<parallel> can compute the B<--block-size>
automatically. A B<--block-size> of B<-1> will use a block size so
that each jobslot will receive approximately 1 block. B<--block -2>
will pass 2 blocks to each jobslot and B<-I<n>> will pass I<n> blocks
to each jobslot.
This can be done because B<--pipepart> reads from files, and we can
compute the total size of the input.
=head2 --jobs and --onall
When running the same commands on many servers what should B<--jobs>
@ -968,11 +975,12 @@ single string.
The DBURL must point to a table name. The table will be dropped and
created. The reason for not reusing an exising table is that the user
may have added more input sources which would require more columns in
the table.
the table. By prepending '+' to the DBURL the table will not be
dropped.
The table columns are similar to joblog with the addition of B<V1>
.. B<Vn> which are values from the input sources, and stdout and
stderr which are the output from standard output and standard error,
.. B<Vn> which are values from the input sources, and Stdout and
Stderr which are the output from standard output and standard error,
respectively.
The Signal column has been renamed to _Signal due to Signal being a

View file

@ -566,7 +566,7 @@ $Global::Initfile && unlink $Global::Initfile;
exit ($err);
sub parse_options {
$Global::version = 20161123;
$Global::version = 20161222;
$Global::progname = 'sql';
# This must be done first as this may exec myself