mirror of
https://git.savannah.gnu.org/git/parallel.git
synced 2024-11-21 21:47:54 +00:00
Released as 20161222 ('Castro')
This commit is contained in:
Ole Tange
parent
9b4fd8b33b
commit
a92ca3bf92
66
NEWS
66
NEWS
|
|
@ -1,3 +1,68 @@
|
|||
20161222
|
||||
|
||||
* --results foo.csv will save output as a CSV-file. Can be directly
|
||||
imported into R or LibreCalc.
|
||||
|
||||
* When using --pipepart a negative --block-size is not interpreted as
|
||||
a block size but as the number of blocks each jobslot should have.
|
||||
|
||||
* --sqlmaster/--sqlandworker will append jobs to the DBURL if the
|
||||
DBURL is prepended with +.
|
||||
|
||||
* GNU Parallel was cited in: A cloud-based workflow to quantify
|
||||
transcript-expression levels in public cancer compendia
|
||||
http://www.nature.com/articles/srep39259
|
||||
|
||||
* GNU Parallel was cited in: Learning string distance with smoothing
|
||||
for OCR spelling correction
|
||||
http://paperity.org/p/78557440/learning-string-distance-with-smoothing-for-ocr-spelling-correction
|
||||
|
||||
* GNU Parallel was cited in: Transient compute clustering with GNU
|
||||
Parallel and sshfs
|
||||
https://gist.github.com/Brainiarc7/24c966c8a001061ee86cc4bc05826bf4
|
||||
|
||||
* GNU Parallel was cited in: Determination of crystal structures of
|
||||
proteins of unknown identity using a marathon molecular replacement
|
||||
procedure: Structure of Stenotrophomonas maltophilia
|
||||
phosphate-binding protein
|
||||
https://www.researchgate.net/publication/308186413_Determination_of_crystal_structures_of_proteins_of_unknown_identity_using_a_marathon_molecular_replacement_procedure_Structure_of_Stenotrophomonas_maltophilia_phosphate-binding_protein
|
||||
|
||||
* GNU Parallel was cited in: The Outer Solar System Origins Survey:
|
||||
I. Design and First-Quarter Discoveries
|
||||
https://arxiv.org/pdf/1511.02895v2.pdf
|
||||
|
||||
* GNU Parallel was cited in: Large-scale benchmarking reveals false
|
||||
discoveries and count transformation sensitivity in 16S rRNA gene
|
||||
amplicon data analysis methods used in microbiome studies
|
||||
http://microbiomejournal.biomedcentral.com/articles/10.1186/s40168-016-0208-8
|
||||
|
||||
* GNU Parallel was cited in: Decomposing Images into Layers via
|
||||
RGB-space Geometry
|
||||
https://cs.gmu.edu/~ygingold/singleimage/Decomposing%20Images%20into%20Layers%20via%20RGB-space%20Geometry%20(Tan%20et%20al%202016%20TOG)%20small.pdf
|
||||
|
||||
* 4 Ways to Batch Convert Your PNG to JPG and Vice-Versa
|
||||
http://www.tecmint.com/linux-image-conversion-tools/
|
||||
|
||||
* All's Fair in Love and Distributed Storage
|
||||
http://cohesity.com/blog/alls-fair-love-distributed-storage/
|
||||
|
||||
* How can I use GNU Parallel to run a lot of commands in parallel
|
||||
https://www.msi.umn.edu/support/faq/how-can-i-use-gnu-parallel-run-lot-commands-parallel
|
||||
|
||||
* 정해영의 블로그 - JEONG Haeyoung's blog
|
||||
http://blog.genoglobe.com/2016/11/gnu-parallel.html
|
||||
|
||||
* 在Linux下将PNG和JPG批量互转的四种方法
|
||||
http://os.51cto.com/art/201612/524182.htm
|
||||
|
||||
* Running in parallel http://tomkimpson.com/howto/gnuparallel/
|
||||
|
||||
* Taco Bell Parallel Programming
|
||||
https://giorgos.sealabs.net/taco-bell-parallel-programming.html
|
||||
|
||||
* Bug fixes and man page updates.
|
||||
|
||||
|
||||
20161122
|
||||
|
||||
* --record-env can now be used with env_parallel for bash, ksh, pdksh,
|
||||
|
|
@ -16,6 +81,7 @@
|
|||
|
||||
* Tutorial: Parallel web scraping with CasperJS and GNU Parallel
|
||||
http://g-liu.com/blog/2016/10/tutorial-parallel-web-scraping-with-casperjs-and-gnu-parallel/
|
||||
|
||||
* Bug fixes and man page updates.
|
||||
|
||||
|
||||
|
|
|
|||
12
README
12
README
|
|
@ -44,9 +44,9 @@ document.
|
|||
|
||||
Full installation of GNU Parallel is as simple as:
|
||||
|
||||
wget http://ftpmirror.gnu.org/parallel/parallel-20161122.tar.bz2
|
||||
bzip2 -dc parallel-20161122.tar.bz2 | tar xvf -
|
||||
cd parallel-20161122
|
||||
wget http://ftpmirror.gnu.org/parallel/parallel-20161222.tar.bz2
|
||||
bzip2 -dc parallel-20161222.tar.bz2 | tar xvf -
|
||||
cd parallel-20161222
|
||||
./configure && make && sudo make install
|
||||
|
||||
|
||||
|
|
@ -55,9 +55,9 @@ Full installation of GNU Parallel is as simple as:
|
|||
If you are not root you can add ~/bin to your path and install in
|
||||
~/bin and ~/share:
|
||||
|
||||
wget http://ftpmirror.gnu.org/parallel/parallel-20161122.tar.bz2
|
||||
bzip2 -dc parallel-20161122.tar.bz2 | tar xvf -
|
||||
cd parallel-20161122
|
||||
wget http://ftpmirror.gnu.org/parallel/parallel-20161222.tar.bz2
|
||||
bzip2 -dc parallel-20161222.tar.bz2 | tar xvf -
|
||||
cd parallel-20161222
|
||||
./configure --prefix=$HOME && make && make install
|
||||
|
||||
Or if your system lacks 'make' you can simply copy src/parallel
|
||||
|
|
|
|||
20
configure
vendored
20
configure
vendored
|
|
@ -1,6 +1,6 @@
|
|||
#! /bin/sh
|
||||
# Guess values for system-dependent variables and create Makefiles.
|
||||
# Generated by GNU Autoconf 2.69 for parallel 20161122.
|
||||
# Generated by GNU Autoconf 2.69 for parallel 20161222.
|
||||
#
|
||||
# Report bugs to <bug-parallel@gnu.org>.
|
||||
#
|
||||
|
|
@ -579,8 +579,8 @@ MAKEFLAGS=
|
|||
# Identity of this package.
|
||||
PACKAGE_NAME='parallel'
|
||||
PACKAGE_TARNAME='parallel'
|
||||
PACKAGE_VERSION='20161122'
|
||||
PACKAGE_STRING='parallel 20161122'
|
||||
PACKAGE_VERSION='20161222'
|
||||
PACKAGE_STRING='parallel 20161222'
|
||||
PACKAGE_BUGREPORT='bug-parallel@gnu.org'
|
||||
PACKAGE_URL=''
|
||||
|
||||
|
|
@ -1203,7 +1203,7 @@ if test "$ac_init_help" = "long"; then
|
|||
# Omit some internal or obsolete options to make the list less imposing.
|
||||
# This message is too long to be a string in the A/UX 3.1 sh.
|
||||
cat <<_ACEOF
|
||||
\`configure' configures parallel 20161122 to adapt to many kinds of systems.
|
||||
\`configure' configures parallel 20161222 to adapt to many kinds of systems.
|
||||
|
||||
Usage: $0 [OPTION]... [VAR=VALUE]...
|
||||
|
||||
|
|
@ -1269,7 +1269,7 @@ fi
|
|||
|
||||
if test -n "$ac_init_help"; then
|
||||
case $ac_init_help in
|
||||
short | recursive ) echo "Configuration of parallel 20161122:";;
|
||||
short | recursive ) echo "Configuration of parallel 20161222:";;
|
||||
esac
|
||||
cat <<\_ACEOF
|
||||
|
||||
|
|
@ -1345,7 +1345,7 @@ fi
|
|||
test -n "$ac_init_help" && exit $ac_status
|
||||
if $ac_init_version; then
|
||||
cat <<\_ACEOF
|
||||
parallel configure 20161122
|
||||
parallel configure 20161222
|
||||
generated by GNU Autoconf 2.69
|
||||
|
||||
Copyright (C) 2012 Free Software Foundation, Inc.
|
||||
|
|
@ -1362,7 +1362,7 @@ cat >config.log <<_ACEOF
|
|||
This file contains any messages produced by compilers while
|
||||
running configure, to aid debugging if configure makes a mistake.
|
||||
|
||||
It was created by parallel $as_me 20161122, which was
|
||||
It was created by parallel $as_me 20161222, which was
|
||||
generated by GNU Autoconf 2.69. Invocation command line was
|
||||
|
||||
$ $0 $@
|
||||
|
|
@ -2225,7 +2225,7 @@ fi
|
|||
|
||||
# Define the identity of the package.
|
||||
PACKAGE='parallel'
|
||||
VERSION='20161122'
|
||||
VERSION='20161222'
|
||||
|
||||
|
||||
cat >>confdefs.h <<_ACEOF
|
||||
|
|
@ -2867,7 +2867,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
|
|||
# report actual input values of CONFIG_FILES etc. instead of their
|
||||
# values after options handling.
|
||||
ac_log="
|
||||
This file was extended by parallel $as_me 20161122, which was
|
||||
This file was extended by parallel $as_me 20161222, which was
|
||||
generated by GNU Autoconf 2.69. Invocation command line was
|
||||
|
||||
CONFIG_FILES = $CONFIG_FILES
|
||||
|
|
@ -2929,7 +2929,7 @@ _ACEOF
|
|||
cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
|
||||
ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`"
|
||||
ac_cs_version="\\
|
||||
parallel config.status 20161122
|
||||
parallel config.status 20161222
|
||||
configured by $0, generated by GNU Autoconf 2.69,
|
||||
with options \\"\$ac_cs_config\\"
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
AC_INIT([parallel], [20161122], [bug-parallel@gnu.org])
|
||||
AC_INIT([parallel], [20161222], [bug-parallel@gnu.org])
|
||||
AM_INIT_AUTOMAKE([-Wall -Werror foreign])
|
||||
AC_CONFIG_HEADERS([config.h])
|
||||
AC_CONFIG_FILES([
|
||||
|
|
|
|||
119
install-sh
119
install-sh
|
|
@ -1,7 +1,7 @@
|
|||
#!/bin/sh
|
||||
# install - install a program, script, or datafile
|
||||
|
||||
scriptversion=2014-09-12.12; # UTC
|
||||
scriptversion=2011-11-20.07; # UTC
|
||||
|
||||
# This originates from X11R5 (mit/util/scripts/install.sh), which was
|
||||
# later released in X11R6 (xc/config/util/install.sh) with the
|
||||
|
|
@ -41,15 +41,19 @@ scriptversion=2014-09-12.12; # UTC
|
|||
# This script is compatible with the BSD install script, but was written
|
||||
# from scratch.
|
||||
|
||||
tab=' '
|
||||
nl='
|
||||
'
|
||||
IFS=" $tab$nl"
|
||||
IFS=" "" $nl"
|
||||
|
||||
# Set DOITPROG to "echo" to test this script.
|
||||
# set DOITPROG to echo to test this script
|
||||
|
||||
# Don't use :- since 4.3BSD and earlier shells don't like it.
|
||||
doit=${DOITPROG-}
|
||||
doit_exec=${doit:-exec}
|
||||
if test -z "$doit"; then
|
||||
doit_exec=exec
|
||||
else
|
||||
doit_exec=$doit
|
||||
fi
|
||||
|
||||
# Put in absolute file names if you don't have them in your path;
|
||||
# or use environment vars.
|
||||
|
|
@ -64,6 +68,17 @@ mvprog=${MVPROG-mv}
|
|||
rmprog=${RMPROG-rm}
|
||||
stripprog=${STRIPPROG-strip}
|
||||
|
||||
posix_glob='?'
|
||||
initialize_posix_glob='
|
||||
test "$posix_glob" != "?" || {
|
||||
if (set -f) 2>/dev/null; then
|
||||
posix_glob=
|
||||
else
|
||||
posix_glob=:
|
||||
fi
|
||||
}
|
||||
'
|
||||
|
||||
posix_mkdir=
|
||||
|
||||
# Desired mode of installed file.
|
||||
|
|
@ -82,7 +97,7 @@ dir_arg=
|
|||
dst_arg=
|
||||
|
||||
copy_on_change=false
|
||||
is_target_a_directory=possibly
|
||||
no_target_directory=
|
||||
|
||||
usage="\
|
||||
Usage: $0 [OPTION]... [-T] SRCFILE DSTFILE
|
||||
|
|
@ -128,7 +143,8 @@ while test $# -ne 0; do
|
|||
|
||||
-m) mode=$2
|
||||
case $mode in
|
||||
*' '* | *"$tab"* | *"$nl"* | *'*'* | *'?'* | *'['*)
|
||||
*' '* | *' '* | *'
|
||||
'* | *'*'* | *'?'* | *'['*)
|
||||
echo "$0: invalid mode: $mode" >&2
|
||||
exit 1;;
|
||||
esac
|
||||
|
|
@ -139,16 +155,14 @@ while test $# -ne 0; do
|
|||
|
||||
-s) stripcmd=$stripprog;;
|
||||
|
||||
-t)
|
||||
is_target_a_directory=always
|
||||
dst_arg=$2
|
||||
-t) dst_arg=$2
|
||||
# Protect names problematic for 'test' and other utilities.
|
||||
case $dst_arg in
|
||||
-* | [=\(\)!]) dst_arg=./$dst_arg;;
|
||||
esac
|
||||
shift;;
|
||||
|
||||
-T) is_target_a_directory=never;;
|
||||
-T) no_target_directory=true;;
|
||||
|
||||
--version) echo "$0 $scriptversion"; exit $?;;
|
||||
|
||||
|
|
@ -163,16 +177,6 @@ while test $# -ne 0; do
|
|||
shift
|
||||
done
|
||||
|
||||
# We allow the use of options -d and -T together, by making -d
|
||||
# take the precedence; this is for compatibility with GNU install.
|
||||
|
||||
if test -n "$dir_arg"; then
|
||||
if test -n "$dst_arg"; then
|
||||
echo "$0: target directory not allowed when installing a directory." >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
if test $# -ne 0 && test -z "$dir_arg$dst_arg"; then
|
||||
# When -d is used, all remaining arguments are directories to create.
|
||||
# When -t is used, the destination is already specified.
|
||||
|
|
@ -203,15 +207,6 @@ if test $# -eq 0; then
|
|||
exit 0
|
||||
fi
|
||||
|
||||
if test -z "$dir_arg"; then
|
||||
if test $# -gt 1 || test "$is_target_a_directory" = always; then
|
||||
if test ! -d "$dst_arg"; then
|
||||
echo "$0: $dst_arg: Is not a directory." >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
if test -z "$dir_arg"; then
|
||||
do_exit='(exit $ret); exit $ret'
|
||||
trap "ret=129; $do_exit" 1
|
||||
|
|
@ -274,7 +269,7 @@ do
|
|||
# If destination is a directory, append the input filename; won't work
|
||||
# if double slashes aren't ignored.
|
||||
if test -d "$dst"; then
|
||||
if test "$is_target_a_directory" = never; then
|
||||
if test -n "$no_target_directory"; then
|
||||
echo "$0: $dst_arg: Is a directory" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
|
@ -282,7 +277,33 @@ do
|
|||
dst=$dstdir/`basename "$src"`
|
||||
dstdir_status=0
|
||||
else
|
||||
dstdir=`dirname "$dst"`
|
||||
# Prefer dirname, but fall back on a substitute if dirname fails.
|
||||
dstdir=`
|
||||
(dirname "$dst") 2>/dev/null ||
|
||||
expr X"$dst" : 'X\(.*[^/]\)//*[^/][^/]*/*$' \| \
|
||||
X"$dst" : 'X\(//\)[^/]' \| \
|
||||
X"$dst" : 'X\(//\)$' \| \
|
||||
X"$dst" : 'X\(/\)' \| . 2>/dev/null ||
|
||||
echo X"$dst" |
|
||||
sed '/^X\(.*[^/]\)\/\/*[^/][^/]*\/*$/{
|
||||
s//\1/
|
||||
q
|
||||
}
|
||||
/^X\(\/\/\)[^/].*/{
|
||||
s//\1/
|
||||
q
|
||||
}
|
||||
/^X\(\/\/\)$/{
|
||||
s//\1/
|
||||
q
|
||||
}
|
||||
/^X\(\/\).*/{
|
||||
s//\1/
|
||||
q
|
||||
}
|
||||
s/.*/./; q'
|
||||
`
|
||||
|
||||
test -d "$dstdir"
|
||||
dstdir_status=$?
|
||||
fi
|
||||
|
|
@ -324,41 +345,34 @@ do
|
|||
# is incompatible with FreeBSD 'install' when (umask & 300) != 0.
|
||||
;;
|
||||
*)
|
||||
# $RANDOM is not portable (e.g. dash); use it when possible to
|
||||
# lower collision chance
|
||||
tmpdir=${TMPDIR-/tmp}/ins$RANDOM-$$
|
||||
trap 'ret=$?; rmdir "$tmpdir/a/b" "$tmpdir/a" "$tmpdir" 2>/dev/null; exit $ret' 0
|
||||
trap 'ret=$?; rmdir "$tmpdir/d" "$tmpdir" 2>/dev/null; exit $ret' 0
|
||||
|
||||
# As "mkdir -p" follows symlinks and we work in /tmp possibly; so
|
||||
# create the $tmpdir first (and fail if unsuccessful) to make sure
|
||||
# that nobody tries to guess the $tmpdir name.
|
||||
if (umask $mkdir_umask &&
|
||||
$mkdirprog $mkdir_mode "$tmpdir" &&
|
||||
exec $mkdirprog $mkdir_mode -p -- "$tmpdir/a/b") >/dev/null 2>&1
|
||||
exec $mkdirprog $mkdir_mode -p -- "$tmpdir/d") >/dev/null 2>&1
|
||||
then
|
||||
if test -z "$dir_arg" || {
|
||||
# Check for POSIX incompatibilities with -m.
|
||||
# HP-UX 11.23 and IRIX 6.5 mkdir -m -p sets group- or
|
||||
# other-writable bit of parent directory when it shouldn't.
|
||||
# FreeBSD 6.1 mkdir -m -p sets mode of existing directory.
|
||||
test_tmpdir="$tmpdir/a"
|
||||
ls_ld_tmpdir=`ls -ld "$test_tmpdir"`
|
||||
ls_ld_tmpdir=`ls -ld "$tmpdir"`
|
||||
case $ls_ld_tmpdir in
|
||||
d????-?r-*) different_mode=700;;
|
||||
d????-?--*) different_mode=755;;
|
||||
*) false;;
|
||||
esac &&
|
||||
$mkdirprog -m$different_mode -p -- "$test_tmpdir" && {
|
||||
ls_ld_tmpdir_1=`ls -ld "$test_tmpdir"`
|
||||
$mkdirprog -m$different_mode -p -- "$tmpdir" && {
|
||||
ls_ld_tmpdir_1=`ls -ld "$tmpdir"`
|
||||
test "$ls_ld_tmpdir" = "$ls_ld_tmpdir_1"
|
||||
}
|
||||
}
|
||||
then posix_mkdir=:
|
||||
fi
|
||||
rmdir "$tmpdir/a/b" "$tmpdir/a" "$tmpdir"
|
||||
rmdir "$tmpdir/d" "$tmpdir"
|
||||
else
|
||||
# Remove any dirs left behind by ancient mkdir implementations.
|
||||
rmdir ./$mkdir_mode ./-p ./-- "$tmpdir" 2>/dev/null
|
||||
rmdir ./$mkdir_mode ./-p ./-- 2>/dev/null
|
||||
fi
|
||||
trap '' 0;;
|
||||
esac;;
|
||||
|
|
@ -382,12 +396,14 @@ do
|
|||
*) prefix='';;
|
||||
esac
|
||||
|
||||
eval "$initialize_posix_glob"
|
||||
|
||||
oIFS=$IFS
|
||||
IFS=/
|
||||
set -f
|
||||
$posix_glob set -f
|
||||
set fnord $dstdir
|
||||
shift
|
||||
set +f
|
||||
$posix_glob set +f
|
||||
IFS=$oIFS
|
||||
|
||||
prefixes=
|
||||
|
|
@ -458,10 +474,13 @@ do
|
|||
if $copy_on_change &&
|
||||
old=`LC_ALL=C ls -dlL "$dst" 2>/dev/null` &&
|
||||
new=`LC_ALL=C ls -dlL "$dsttmp" 2>/dev/null` &&
|
||||
set -f &&
|
||||
|
||||
eval "$initialize_posix_glob" &&
|
||||
$posix_glob set -f &&
|
||||
set X $old && old=:$2:$4:$5:$6 &&
|
||||
set X $new && new=:$2:$4:$5:$6 &&
|
||||
set +f &&
|
||||
$posix_glob set +f &&
|
||||
|
||||
test "$old" = "$new" &&
|
||||
$cmpprog "$dst" "$dsttmp" >/dev/null 2>&1
|
||||
then
|
||||
|
|
|
|||
2
missing
2
missing
|
|
@ -3,7 +3,7 @@
|
|||
|
||||
scriptversion=2013-10-28.13; # UTC
|
||||
|
||||
# Copyright (C) 1996-2014 Free Software Foundation, Inc.
|
||||
# Copyright (C) 1996-2013 Free Software Foundation, Inc.
|
||||
# Originally written by Fran,cois Pinard <pinard@iro.umontreal.ca>, 1996.
|
||||
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
Summary: Shell tool for executing jobs in parallel
|
||||
Name: parallel
|
||||
Version: 20161122
|
||||
Version: 20161222
|
||||
Release: 1.1
|
||||
License: GPL
|
||||
Group: Productivity/File utilities
|
||||
|
|
|
|||
|
|
@ -24,7 +24,7 @@
|
|||
use strict;
|
||||
use Getopt::Long;
|
||||
$Global::progname="niceload";
|
||||
$Global::version = 20161123;
|
||||
$Global::version = 20161222;
|
||||
Getopt::Long::Configure("bundling","require_order");
|
||||
get_options_from_array(\@ARGV) || die_usage();
|
||||
if($opt::version) {
|
||||
|
|
|
|||
93
src/parallel
93
src/parallel
|
|
@ -227,8 +227,10 @@ if($opt::halt and $Global::halt_when ne "never") {
|
|||
wait_and_exit(min(undef_as_zero($Global::exitstatus),101));
|
||||
}
|
||||
|
||||
|
||||
sub __PIPE_MODE__ {}
|
||||
|
||||
|
||||
sub pipe_part_files {
|
||||
# Input:
|
||||
# $file = the file to read
|
||||
|
|
@ -769,8 +771,10 @@ sub write_record_to_pipe {
|
|||
return 1;
|
||||
}
|
||||
|
||||
|
||||
sub __SEM_MODE__ {}
|
||||
|
||||
|
||||
sub acquire_semaphore {
|
||||
# Acquires semaphore. If needed: spawns to the background
|
||||
# Uses:
|
||||
|
|
@ -793,8 +797,10 @@ sub acquire_semaphore {
|
|||
return $sem;
|
||||
}
|
||||
|
||||
|
||||
sub __PARSE_OPTIONS__ {}
|
||||
|
||||
|
||||
sub options_hash {
|
||||
# Returns:
|
||||
# %hash = the GetOptions config
|
||||
|
|
@ -1078,10 +1084,13 @@ sub parse_options {
|
|||
}
|
||||
|
||||
if(defined $opt::results) {
|
||||
# Is the output a dir or CSV-file?
|
||||
if($opt::results =~ /\.csv$/i) {
|
||||
# CSV with , as separator
|
||||
$Global::csv = ",";
|
||||
$Global::membuffer ||= 1;
|
||||
} elsif($opt::results =~ /\.tsv$/i) {
|
||||
# CSV with TAB as separator
|
||||
$Global::csv = "\t";
|
||||
$Global::membuffer ||= 1;
|
||||
}
|
||||
|
|
@ -1250,7 +1259,7 @@ sub check_invalid_option_combinations {
|
|||
|
||||
sub init_globals {
|
||||
# Defaults:
|
||||
$Global::version = 20161208;
|
||||
$Global::version = 20161222;
|
||||
$Global::progname = 'parallel';
|
||||
$Global::infinity = 2**31;
|
||||
$Global::debug = 0;
|
||||
|
|
@ -1636,7 +1645,7 @@ sub find_compression_program {
|
|||
# parallel --shuf -j1 --joblog jl-m --arg-sep , parallel --compress-program \'{3}" "-{2}\' cat ::: 1gb '>'/dev/null , 1 2 3 , {1..3} , $multithread
|
||||
# sort -nk4 jl-?
|
||||
# 1-core:
|
||||
# 2-cores:
|
||||
# 2-cores: lz4 lzop zstd gzip bzip2 lzma xz clzip pigz lbzip2 pbzip2 lzip lrz plzip pxz pzstd
|
||||
# 4-cores:
|
||||
# 8-cores: pzstd lz4 zstd pigz lzop lbzip2 pbzip2 gzip lzip lrz plzip pxz bzip2 lzma xz clzip
|
||||
# 16-cores: pzstd lz4 pigz lzop lbzip2 pbzip2 plzip lzip lrz pxz gzip lzma xz bzip2
|
||||
|
|
@ -1653,7 +1662,6 @@ sub find_compression_program {
|
|||
return ("cat","cat");
|
||||
}
|
||||
|
||||
|
||||
sub read_options {
|
||||
# Read options from command line, profile and $PARALLEL
|
||||
# Uses:
|
||||
|
|
@ -1889,8 +1897,10 @@ sub cleanup {
|
|||
}
|
||||
}
|
||||
|
||||
|
||||
sub __QUOTING_ARGUMENTS_FOR_SHELL__ {}
|
||||
|
||||
|
||||
sub shell_quote {
|
||||
# Input:
|
||||
# @strings = strings to be quoted
|
||||
|
|
@ -2038,6 +2048,7 @@ sub unquote_printf {
|
|||
return $_;
|
||||
}
|
||||
|
||||
|
||||
sub __FILEHANDLES__ {}
|
||||
|
||||
|
||||
|
|
@ -2154,6 +2165,7 @@ sub set_fh_non_blocking {
|
|||
|
||||
sub __RUNNING_THE_JOBS_AND_PRINTING_PROGRESS__ {}
|
||||
|
||||
|
||||
# Variable structure:
|
||||
#
|
||||
# $Global::running{$pid} = Pointer to Job-object
|
||||
|
|
@ -2870,8 +2882,10 @@ sub get_job_with_sshlogin {
|
|||
return $job;
|
||||
}
|
||||
|
||||
|
||||
sub __REMOTE_SSH__ {}
|
||||
|
||||
|
||||
sub read_sshloginfiles {
|
||||
# Read a list of --slf's
|
||||
# Input:
|
||||
|
|
@ -3507,8 +3521,10 @@ sub onall {
|
|||
}
|
||||
}
|
||||
|
||||
|
||||
sub __SIGNAL_HANDLING__ {}
|
||||
|
||||
|
||||
sub tstp {
|
||||
# Send TSTP signal (Ctrl-Z) to all children process groups
|
||||
# Uses:
|
||||
|
|
@ -3520,7 +3536,6 @@ sub tstp {
|
|||
kill "TSTP", $$;
|
||||
}
|
||||
|
||||
|
||||
sub save_original_signal_handler {
|
||||
# Remember the original signal handler
|
||||
# Uses:
|
||||
|
|
@ -3662,8 +3677,10 @@ sub reaper {
|
|||
return wantarray ? @pids_reaped : $children_reaped;
|
||||
}
|
||||
|
||||
|
||||
sub __USAGE__ {}
|
||||
|
||||
|
||||
sub killall {
|
||||
# Kill all jobs by killing their process groups
|
||||
|
||||
|
|
@ -3790,7 +3807,6 @@ sub usage {
|
|||
"",);
|
||||
}
|
||||
|
||||
|
||||
sub citation_notice {
|
||||
# if --will-cite or --plain: do nothing
|
||||
# if stderr redirected: do nothing
|
||||
|
|
@ -3953,8 +3969,10 @@ sub show_limits {
|
|||
"press CTRL-D or CTRL-C\n");
|
||||
}
|
||||
|
||||
|
||||
sub __GENERIC_COMMON_FUNCTION__ {}
|
||||
|
||||
|
||||
sub mkdir_or_die {
|
||||
# If dir is not executable: die
|
||||
my $dir = shift;
|
||||
|
|
@ -4003,13 +4021,13 @@ sub tmpname {
|
|||
return $tmpname;
|
||||
}
|
||||
|
||||
sub tmpfifo {
|
||||
# Find an unused name and mkfifo on it
|
||||
use POSIX qw(mkfifo);
|
||||
my $tmpfifo = tmpname("fif",@_);
|
||||
mkfifo($tmpfifo,0600);
|
||||
return $tmpfifo;
|
||||
}
|
||||
#sub tmpfifo {
|
||||
# # Find an unused name and mkfifo on it
|
||||
# use POSIX qw(mkfifo);
|
||||
# my $tmpfifo = tmpname("fif",@_);
|
||||
# mkfifo($tmpfifo,0600);
|
||||
# return $tmpfifo;
|
||||
#}
|
||||
|
||||
sub rm {
|
||||
# Remove file and remove it from %Global::unlink
|
||||
|
|
@ -4484,8 +4502,10 @@ sub kill_youngster_if_not_enough_mem {
|
|||
}
|
||||
}
|
||||
|
||||
|
||||
sub __DEBUGGING__ {}
|
||||
|
||||
|
||||
sub debug {
|
||||
# Uses:
|
||||
# $Global::debug
|
||||
|
|
@ -4578,8 +4598,10 @@ sub my_carp {
|
|||
carp(@_);
|
||||
}
|
||||
|
||||
|
||||
sub __OBJECT_ORIENTED_PARTS__ {}
|
||||
|
||||
|
||||
package SSHLogin;
|
||||
|
||||
sub new {
|
||||
|
|
@ -7382,7 +7404,9 @@ sub sshlogin_wrap {
|
|||
my @vars = vars_to_export();
|
||||
my $csh_friendly = not grep { /\n/ } @ENV{@vars};
|
||||
my @bash_functions = grep { substr($ENV{$_},0,4) eq "() {" } @vars;
|
||||
my @non_functions = grep { substr($ENV{$_},0,4) ne "() {" } @vars;
|
||||
my @non_functions = (grep { !/PARALLEL_ENV/ }
|
||||
grep { substr($ENV{$_},0,4) ne "() {" } @vars);
|
||||
|
||||
# eval of @envset will set %ENV
|
||||
my $envset = join"", map {
|
||||
'$ENV{"'.::perl_quote_scalar($_).'"}="'.
|
||||
|
|
@ -8296,7 +8320,7 @@ sub combine_ref {
|
|||
$must_be_quoted ||=1;
|
||||
}
|
||||
if($must_be_quoted){
|
||||
push \@out, \$sep, \$quot, $column, \$quot;
|
||||
push @out, \$sep, \$quot, $column, \$quot;
|
||||
} else {
|
||||
push @out, \$sep, $column;
|
||||
}
|
||||
|
|
@ -8306,46 +8330,6 @@ sub combine_ref {
|
|||
return @out;
|
||||
}
|
||||
|
||||
sub combine {
|
||||
# Inspired by Text::CSV_PP::_combine (by Makamaka Hannyaharamitu)
|
||||
my @part = @_;
|
||||
my $sep = $Global::csv;
|
||||
my $quot = '"';
|
||||
|
||||
my $must_be_quoted;
|
||||
for my $column (@part) {
|
||||
# Memory optimization: Content transferred as reference
|
||||
if(ref $column ne "SCALAR") {
|
||||
($column,$$column) = ($$column,"");
|
||||
}
|
||||
if(not defined $column) {
|
||||
$column = '';
|
||||
next;
|
||||
}
|
||||
|
||||
$must_be_quoted = 0;
|
||||
|
||||
if($column =~ s/$quot/$quot$quot/go){
|
||||
# Replace " => ""
|
||||
$must_be_quoted ||=1;
|
||||
}
|
||||
if($column =~ /[\s\Q$sep\E]/o){
|
||||
# Put quotes around if the column contains ,
|
||||
$must_be_quoted ||=1;
|
||||
}
|
||||
|
||||
$Global::use{"bytes"} ||= eval "use bytes; 1;";
|
||||
if ($column =~ /\0/) {
|
||||
# Contains \0 => put quotes around
|
||||
$must_be_quoted ||=1;
|
||||
}
|
||||
if($must_be_quoted){
|
||||
$column = $quot . $column . $quot;
|
||||
}
|
||||
}
|
||||
return join($sep, @part) . "\n";
|
||||
}
|
||||
|
||||
sub print_files {
|
||||
# Print the name of the file containing stdout on stdout
|
||||
# Uses:
|
||||
|
|
@ -8702,6 +8686,7 @@ sub set_exitsignal {
|
|||
}
|
||||
}
|
||||
|
||||
|
||||
package CommandLine;
|
||||
|
||||
sub new {
|
||||
|
|
|
|||
|
|
@ -499,7 +499,7 @@ If you use B<-N>, B<--block-size> should be bigger than N+1 records.
|
|||
|
||||
I<size> defaults to 1M.
|
||||
|
||||
When using B<--pipepart> negative a block size is not interpreted as a
|
||||
When using B<--pipepart> a negative block size is not interpreted as a
|
||||
blocksize but as the number of blocks each jobslot should have. So
|
||||
this will run 10*5 = 50 jobs in total:
|
||||
|
||||
|
|
@ -1670,9 +1670,9 @@ B<--return> is ignored when used with B<--sshlogin :> or when not used
|
|||
with B<--sshlogin>.
|
||||
|
||||
|
||||
=item B<--round-robin>
|
||||
=item B<--round-robin> (alpha testing)
|
||||
|
||||
=item B<--round>
|
||||
=item B<--round> (alpha testing)
|
||||
|
||||
Normally B<--pipe> will give a single block to each instance of the
|
||||
command. With B<--round-robin> all blocks will at random be written to
|
||||
|
|
@ -3427,7 +3427,7 @@ Note: The default for GNU B<parallel> is to remove the spaces around the columns
|
|||
parallel -a table_file.tsv --trim n --colsep '\t' cmd -o {2} -i {1}
|
||||
|
||||
|
||||
=head1 EXAMPLE: Output as CSV-file
|
||||
=head1 EXAMPLE: Output to database
|
||||
|
||||
GNU B<parallel> can output to a database table and a CSV-file:
|
||||
|
||||
|
|
@ -3464,6 +3464,18 @@ Or MySQL:
|
|||
perl -pe 's/"/""/g;s/\t/","/g;s/^/"/;s/$/"/;s/\\\\/\\/g;s/\\t/\t/g;s/\\n/\n/g;' mytable.tsv
|
||||
|
||||
|
||||
=head1 EXAMPLE: Output to CSV-file for R
|
||||
|
||||
If you have no need for the advanced job distribution control that a
|
||||
database provides, but you simply want output into a CSV file that you
|
||||
can read into R or LibreCalc, then you can use B<--results>:
|
||||
|
||||
parallel --results my.csv seq ::: 10 20 30
|
||||
R
|
||||
> mydf <- read.csv("my.csv");
|
||||
> print(mydf[2,])
|
||||
> write(as.character(mydf[2,c("Stdout")]),'')
|
||||
|
||||
=head1 EXAMPLE: Run the same command 10 times
|
||||
|
||||
If you want to run the same command with the same arguments 10 times
|
||||
|
|
|
|||
|
|
@ -69,12 +69,6 @@ The only way to have access to the environment is directly from the
|
|||
shell, so the program must be written in a shell script that will be
|
||||
sourced and there has to deal with the dialect of the relevant shell.
|
||||
|
||||
Dumping the environment into a format that can be activated again is
|
||||
easy in Bash: B<typeset> and B<alias> basically give you the commands
|
||||
you must run to re-create the environment. It is much harder in B<csh>
|
||||
where there is no easy way to convert an environment into commands
|
||||
that generate the environment.
|
||||
|
||||
|
||||
=head3 env_parallel.*
|
||||
|
||||
|
|
@ -183,10 +177,10 @@ compresses the buffered data. This is a bit tricky because there
|
|||
should be no files to clean up if GNU B<parallel> is killed by a power
|
||||
outage.
|
||||
|
||||
GNU B<parallel> first selects a compression program. If the user has not
|
||||
selected one, the first of these that is in $PATH is used: B<lz4 pigz
|
||||
lzop plzip pbzip2 pxz gzip lzma xz bzip2 lzip>. They are sorted by
|
||||
speed on a 16 core machine.
|
||||
GNU B<parallel> first selects a compression program. If the user has
|
||||
not selected one, the first of these that is in $PATH is used: B<pzstd
|
||||
lbzip2 pbzip2 zstd pigz lz4 lzop plzip lzip lrz gzip pxz lzma bzip2 xz
|
||||
clzip>. They are sorted by speed on a 32 core machine.
|
||||
|
||||
Schematically the setup is as follows:
|
||||
|
||||
|
|
@ -449,23 +443,6 @@ that correctly for all corner cases is next to impossible to do by
|
|||
hand.
|
||||
|
||||
|
||||
=head2 --block-size adjustment
|
||||
|
||||
Every time GNU B<parallel> detects a record bigger than
|
||||
B<--block-size> it increases the block size by 30%. A small
|
||||
B<--block-size> gives very poor performance; by exponentially
|
||||
increasing the block size performance will not suffer.
|
||||
|
||||
GNU B<parallel> will waste CPU power if B<--block-size> does not
|
||||
contain a full record, because it tries to find a full record and will
|
||||
fail to do so. The recommendation is therefore to use a
|
||||
B<--block-size> > 2 records, so you always get at least one full
|
||||
record when you read one block.
|
||||
|
||||
If you use B<-N> then B<--block-size> should be big enough to contain
|
||||
N+1 records.
|
||||
|
||||
|
||||
=head2 Shell shock
|
||||
|
||||
The shell shock bug in B<bash> did not affect GNU B<parallel>, but the
|
||||
|
|
@ -736,6 +713,36 @@ data. See this for a surprising example:
|
|||
|
||||
yes | dd bs=1024k count=10 | wc
|
||||
|
||||
|
||||
=head2 --block-size adjustment
|
||||
|
||||
Every time GNU B<parallel> detects a record bigger than
|
||||
B<--block-size> it increases the block size by 30%. A small
|
||||
B<--block-size> gives very poor performance; by exponentially
|
||||
increasing the block size performance will not suffer.
|
||||
|
||||
GNU B<parallel> will waste CPU power if B<--block-size> does not
|
||||
contain a full record, because it tries to find a full record and will
|
||||
fail to do so. The recommendation is therefore to use a
|
||||
B<--block-size> > 2 records, so you always get at least one full
|
||||
record when you read one block.
|
||||
|
||||
If you use B<-N> then B<--block-size> should be big enough to contain
|
||||
N+1 records.
|
||||
|
||||
|
||||
=head2 Automatic --block-size computation
|
||||
|
||||
With B<--pipepart> GNU B<parallel> can compute the B<--block-size>
|
||||
automatically. A B<--block-size> of B<-1> will use a block size so
|
||||
that each jobslot will receive approximately 1 block. B<--block -2>
|
||||
will pass 2 blocks to each jobslot and B<-I<n>> will pass I<n> blocks
|
||||
to each jobslot.
|
||||
|
||||
This can be done because B<--pipepart> reads from files, and we can
|
||||
compute the total size of the input.
|
||||
|
||||
|
||||
=head2 --jobs and --onall
|
||||
|
||||
When running the same commands on many servers what should B<--jobs>
|
||||
|
|
@ -968,11 +975,12 @@ single string.
|
|||
The DBURL must point to a table name. The table will be dropped and
|
||||
created. The reason for not reusing an exising table is that the user
|
||||
may have added more input sources which would require more columns in
|
||||
the table.
|
||||
the table. By prepending '+' to the DBURL the table will not be
|
||||
dropped.
|
||||
|
||||
The table columns are similar to joblog with the addition of B<V1>
|
||||
.. B<Vn> which are values from the input sources, and stdout and
|
||||
stderr which are the output from standard output and standard error,
|
||||
.. B<Vn> which are values from the input sources, and Stdout and
|
||||
Stderr which are the output from standard output and standard error,
|
||||
respectively.
|
||||
|
||||
The Signal column has been renamed to _Signal due to Signal being a
|
||||
|
|
|
|||
Loading…
Reference in a new issue