env_parallel.fish: Converted to '-quoting.

This commit is contained in:
Ole Tange 2018-12-28 12:47:25 +01:00
parent 1501d93eea
commit a51c54dafe
43 changed files with 2167 additions and 282 deletions

View file

@ -15,7 +15,7 @@
# If that fails, it copies to $HOME/bin # If that fails, it copies to $HOME/bin
# #
# You can download and run the script directly by: # You can download and run the script directly by:
# (wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash # (lynx -source pi.dk/3 || wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash
run() { run() {
# tail on openindiana must be /usr/xpg4/bin/tail # tail on openindiana must be /usr/xpg4/bin/tail
@ -24,12 +24,13 @@ run() {
# grep on openindiana must be /usr/xpg4/bin/grep # grep on openindiana must be /usr/xpg4/bin/grep
GREP=$(echo | grep -vE . 2>/dev/null && echo grep || GREP=$(echo | grep -vE . 2>/dev/null && echo grep ||
(echo | /usr/xpg4/bin/grep -vE . && echo /usr/xpg4/bin/grep)) (echo | /usr/xpg4/bin/grep -vE . && echo /usr/xpg4/bin/grep))
# FreeBSD prefers 'fetch', MacOS prefers 'curl', Linux prefers 'wget' # FreeBSD prefers 'fetch', MacOS prefers 'curl', Linux prefers 'wget'
GET=$( GET=$(
(fetch -o /dev/null file:///bin/sh && echo fetch -o -) || (lynx -source /dev/null && echo lynx -source) ||
(fetch -o /dev/null file:///bin/sh && echo fetch -o -) ||
(curl -h >/dev/null && echo curl -L) || (curl -h >/dev/null && echo curl -L) ||
(wget -h >/dev/null && echo wget -qO -) || (wget -h >/dev/null && echo wget -qO -) ||
echo 'No wget, curl, fetch: Please inform parallel@gnu.org what you use for downloading URLs' >&2 echo 'No lynx, wget, curl, fetch: Please inform parallel@gnu.org what you use for downloading URLs' >&2
) )
if test "$GET" = ""; then if test "$GET" = ""; then
exit 1 exit 1

26
NEWS
View file

@ -1,5 +1,20 @@
<<<<<<< HEAD 20181222
=======
* Better /proc/cpuinfo parser.
* Simpler job scheduling flow.
* MPGI Workshop: Job Parallelization with Task Arrays and GNU Parallel https://www.msi.umn.edu/tutorials/mpgi-workshop-module-4
* CONCOCT uses GNU Parallel https://media.readthedocs.org/pdf/concoct/latest/concoct.pdf
* If you use SLURM read https://wiki.hpc.uconn.edu/index.php/Advanced_SLURM
* すぐ忘れてしまう、仕事で使う技 https://qiita.com/hana_shin/items/53c3c78525c9c758ae7c
* Bug fixes and man page updates.
20181122 20181122
* Experimental simpler job flow control. * Experimental simpler job flow control.
@ -55,7 +70,6 @@
* Bug fixes and man page updates. * Bug fixes and man page updates.
>>>>>>> 8286e771ec57e60ee17aee87741fe96790880db7
20180722 20180722
* The quoting engine has been changed. Instead of using \-quoting GNU * The quoting engine has been changed. Instead of using \-quoting GNU
@ -82,11 +96,8 @@
* Parallel download genomic data with GNU-Parallel * Parallel download genomic data with GNU-Parallel
https://digibio.blogspot.com/search/label/GNU-Parallel https://digibio.blogspot.com/search/label/GNU-Parallel
<<<<<<< HEAD
=======
* Bug fixes and man page updates. * Bug fixes and man page updates.
>>>>>>> 8286e771ec57e60ee17aee87741fe96790880db7
20180622 20180622
@ -293,7 +304,8 @@
* Initial support for XDG Base Directory $XDG_* environment variables. * Initial support for XDG Base Directory $XDG_* environment variables.
* Validating half a million TIFF files. Part Two * Validating half a million TIFF files. Part
Two
https://www.dpoc.ac.uk/2017/08/17/validating-half-a-million-tiff-files-part-two/ https://www.dpoc.ac.uk/2017/08/17/validating-half-a-million-tiff-files-part-two/
* Turing data science class: Cluster computing * Turing data science class: Cluster computing

12
README
View file

@ -44,9 +44,9 @@ document.
Full installation of GNU Parallel is as simple as: Full installation of GNU Parallel is as simple as:
wget https://ftpmirror.gnu.org/parallel/parallel-20181122.tar.bz2 wget https://ftpmirror.gnu.org/parallel/parallel-20181222.tar.bz2
bzip2 -dc parallel-20181122.tar.bz2 | tar xvf - bzip2 -dc parallel-20181222.tar.bz2 | tar xvf -
cd parallel-20181122 cd parallel-20181222
./configure && make && sudo make install ./configure && make && sudo make install
@ -55,9 +55,9 @@ Full installation of GNU Parallel is as simple as:
If you are not root you can add ~/bin to your path and install in If you are not root you can add ~/bin to your path and install in
~/bin and ~/share: ~/bin and ~/share:
wget https://ftpmirror.gnu.org/parallel/parallel-20181122.tar.bz2 wget https://ftpmirror.gnu.org/parallel/parallel-20181222.tar.bz2
bzip2 -dc parallel-20181122.tar.bz2 | tar xvf - bzip2 -dc parallel-20181222.tar.bz2 | tar xvf -
cd parallel-20181122 cd parallel-20181222
./configure --prefix=$HOME && make && make install ./configure --prefix=$HOME && make && make install
Or if your system lacks 'make' you can simply copy src/parallel Or if your system lacks 'make' you can simply copy src/parallel

20
configure vendored
View file

@ -1,6 +1,6 @@
#! /bin/sh #! /bin/sh
# Guess values for system-dependent variables and create Makefiles. # Guess values for system-dependent variables and create Makefiles.
# Generated by GNU Autoconf 2.69 for parallel 20181122. # Generated by GNU Autoconf 2.69 for parallel 20181222.
# #
# Report bugs to <bug-parallel@gnu.org>. # Report bugs to <bug-parallel@gnu.org>.
# #
@ -579,8 +579,8 @@ MAKEFLAGS=
# Identity of this package. # Identity of this package.
PACKAGE_NAME='parallel' PACKAGE_NAME='parallel'
PACKAGE_TARNAME='parallel' PACKAGE_TARNAME='parallel'
PACKAGE_VERSION='20181122' PACKAGE_VERSION='20181222'
PACKAGE_STRING='parallel 20181122' PACKAGE_STRING='parallel 20181222'
PACKAGE_BUGREPORT='bug-parallel@gnu.org' PACKAGE_BUGREPORT='bug-parallel@gnu.org'
PACKAGE_URL='' PACKAGE_URL=''
@ -1214,7 +1214,7 @@ if test "$ac_init_help" = "long"; then
# Omit some internal or obsolete options to make the list less imposing. # Omit some internal or obsolete options to make the list less imposing.
# This message is too long to be a string in the A/UX 3.1 sh. # This message is too long to be a string in the A/UX 3.1 sh.
cat <<_ACEOF cat <<_ACEOF
\`configure' configures parallel 20181122 to adapt to many kinds of systems. \`configure' configures parallel 20181222 to adapt to many kinds of systems.
Usage: $0 [OPTION]... [VAR=VALUE]... Usage: $0 [OPTION]... [VAR=VALUE]...
@ -1281,7 +1281,7 @@ fi
if test -n "$ac_init_help"; then if test -n "$ac_init_help"; then
case $ac_init_help in case $ac_init_help in
short | recursive ) echo "Configuration of parallel 20181122:";; short | recursive ) echo "Configuration of parallel 20181222:";;
esac esac
cat <<\_ACEOF cat <<\_ACEOF
@ -1357,7 +1357,7 @@ fi
test -n "$ac_init_help" && exit $ac_status test -n "$ac_init_help" && exit $ac_status
if $ac_init_version; then if $ac_init_version; then
cat <<\_ACEOF cat <<\_ACEOF
parallel configure 20181122 parallel configure 20181222
generated by GNU Autoconf 2.69 generated by GNU Autoconf 2.69
Copyright (C) 2012 Free Software Foundation, Inc. Copyright (C) 2012 Free Software Foundation, Inc.
@ -1374,7 +1374,7 @@ cat >config.log <<_ACEOF
This file contains any messages produced by compilers while This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake. running configure, to aid debugging if configure makes a mistake.
It was created by parallel $as_me 20181122, which was It was created by parallel $as_me 20181222, which was
generated by GNU Autoconf 2.69. Invocation command line was generated by GNU Autoconf 2.69. Invocation command line was
$ $0 $@ $ $0 $@
@ -2237,7 +2237,7 @@ fi
# Define the identity of the package. # Define the identity of the package.
PACKAGE='parallel' PACKAGE='parallel'
VERSION='20181122' VERSION='20181222'
cat >>confdefs.h <<_ACEOF cat >>confdefs.h <<_ACEOF
@ -2880,7 +2880,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
# report actual input values of CONFIG_FILES etc. instead of their # report actual input values of CONFIG_FILES etc. instead of their
# values after options handling. # values after options handling.
ac_log=" ac_log="
This file was extended by parallel $as_me 20181122, which was This file was extended by parallel $as_me 20181222, which was
generated by GNU Autoconf 2.69. Invocation command line was generated by GNU Autoconf 2.69. Invocation command line was
CONFIG_FILES = $CONFIG_FILES CONFIG_FILES = $CONFIG_FILES
@ -2942,7 +2942,7 @@ _ACEOF
cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`" ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`"
ac_cs_version="\\ ac_cs_version="\\
parallel config.status 20181122 parallel config.status 20181222
configured by $0, generated by GNU Autoconf 2.69, configured by $0, generated by GNU Autoconf 2.69,
with options \\"\$ac_cs_config\\" with options \\"\$ac_cs_config\\"

View file

@ -1,4 +1,4 @@
AC_INIT([parallel], [20181122], [bug-parallel@gnu.org]) AC_INIT([parallel], [20181222], [bug-parallel@gnu.org])
AM_INIT_AUTOMAKE([-Wall -Werror foreign]) AM_INIT_AUTOMAKE([-Wall -Werror foreign])
AC_CONFIG_HEADERS([config.h]) AC_CONFIG_HEADERS([config.h])
AC_CONFIG_FILES([ AC_CONFIG_FILES([

View file

@ -206,17 +206,19 @@ from:tange@gnu.org
to:parallel@gnu.org, bug-parallel@gnu.org to:parallel@gnu.org, bug-parallel@gnu.org
stable-bcc: Jesse Alama <jessealama@fastmail.fm> stable-bcc: Jesse Alama <jessealama@fastmail.fm>
Subject: GNU Parallel 20181222 ('Kilogram/brexitdeal/Stan Lee/Cesar Sayoc/Tree of Life/Iran') released <<[stable]>> Subject: GNU Parallel 20190122 ('Indonesia=Tanjung Lesung=Sunda ') released <<[stable]>>
GNU Parallel 20181222 ('') <<[stable]>> has been released. It is available for download at: http://ftpmirror.gnu.org/parallel/ GNU Parallel 20190122 ('') <<[stable]>> has been released. It is available for download at: http://ftpmirror.gnu.org/parallel/
<<No new functionality was introduced so this is a good candidate for a stable release.>> <<No new functionality was introduced so this is a good candidate for a stable release.>>
Quote of the month: Quote of the month:
<<>>
New in this release: New in this release:
* https://qiita.com/hana_shin/items/53c3c78525c9c758ae7c *
* Bug fixes and man page updates. * Bug fixes and man page updates.
@ -235,11 +237,12 @@ GNU Parallel makes sure output from the commands is the same output as you would
You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/ You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/
You can install GNU Parallel in just 10 seconds with: (wget -O - pi.dk/3 || curl pi.dk/3/) | bash You can install GNU Parallel in just 10 seconds with:
(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash
Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1 Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your commandline will love you for it. Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
When using programs that use GNU Parallel to process data for publication please cite: When using programs that use GNU Parallel to process data for publication please cite:

View file

@ -1,6 +1,6 @@
<directory name="parallel" rev="227" srcmd5="21d1e9e6d757ee09c239531d094bac7b" vrev="1"> <directory name="parallel" rev="228" srcmd5="6b5a77aa91c7b5b1222e49ab36f563df" vrev="1">
<entry md5="7f377215f4bfec95ae667a756cdda905" mtime="1542929947" name="parallel-20181122.tar.bz2" size="1797510" /> <entry md5="04ebc06103bdf5ce70c360742adbcbd8" mtime="1545440269" name="parallel-20181222.tar.bz2" size="1799481" />
<entry md5="21c2c593abc38419b337648cd12518ae" mtime="1542929947" name="parallel.spec" size="4667" /> <entry md5="68cf0f3d678a67b8317122c1f4c45afc" mtime="1545440269" name="parallel.spec" size="4667" />
<entry md5="30d6a27fae9852f24b415b6e00feb43f" mtime="1542929947" name="parallel_20181122.dsc" size="556" /> <entry md5="0ee51c455e89f82f7c8f0dfe9310c250" mtime="1545440269" name="parallel_20181222.dsc" size="556" />
<entry md5="4e0cdca7d064c5174b0858b8e2acb1f5" mtime="1542929947" name="parallel_20181122.tar.gz" size="1991811" /> <entry md5="ddaa3054a0ab04e30ad38d2b32617a4a" mtime="1545440270" name="parallel_20181222.tar.gz" size="1995147" />
</directory> </directory>

View file

@ -1,7 +1,7 @@
Summary: Shell tool for executing jobs in parallel Summary: Shell tool for executing jobs in parallel
Name: parallel Name: parallel
Version: 20181122 Version: 20181222
Release: 1.2 Release: 1.2
License: GPL License: GPL
Group: Productivity/File utilities Group: Productivity/File utilities

View file

@ -1,7 +1,7 @@
Summary: Shell tool for executing jobs in parallel Summary: Shell tool for executing jobs in parallel
Name: parallel Name: parallel
Version: 20181122 Version: 20181222
Release: 1.2 Release: 1.2
License: GPL License: GPL
Group: Productivity/File utilities Group: Productivity/File utilities

View file

@ -7,7 +7,7 @@
# after which 'env_parallel' works # after which 'env_parallel' works
# #
# #
# Copyright (C) 2016,2017,2018 # Copyright (C) 2016-2019
# Ole Tange and Free Software Foundation, Inc. # Ole Tange and Free Software Foundation, Inc.
# #
# This program is free software; you can redistribute it and/or modify # This program is free software; you can redistribute it and/or modify

View file

@ -7,7 +7,7 @@
# after which 'env_parallel' works # after which 'env_parallel' works
# #
# #
# Copyright (C) 2016-2018 # Copyright (C) 2016-2019
# Ole Tange and Free Software Foundation, Inc. # Ole Tange and Free Software Foundation, Inc.
# #
# This program is free software; you can redistribute it and/or modify # This program is free software; you can redistribute it and/or modify

View file

@ -7,7 +7,7 @@
# after which 'env_parallel' works # after which 'env_parallel' works
# #
# #
# Copyright (C) 2016-2018 # Copyright (C) 2016-2019
# Ole Tange and Free Software Foundation, Inc. # Ole Tange and Free Software Foundation, Inc.
# #
# This program is free software; you can redistribute it and/or modify # This program is free software; you can redistribute it and/or modify

View file

@ -7,7 +7,7 @@
# after which 'env_parallel' works # after which 'env_parallel' works
# #
# #
# Copyright (C) 2016,2017,2018 # Copyright (C) 2016-2019
# Ole Tange and Free Software Foundation, Inc. # Ole Tange and Free Software Foundation, Inc.
# #
# This program is free software; you can redistribute it and/or modify # This program is free software; you can redistribute it and/or modify

View file

@ -7,7 +7,7 @@
# after which 'env_parallel' works # after which 'env_parallel' works
# #
# #
# Copyright (C) 2016,2017,2018 # Copyright (C) 2016-2019
# Ole Tange and Free Software Foundation, Inc. # Ole Tange and Free Software Foundation, Inc.
# #
# This program is free software; you can redistribute it and/or modify # This program is free software; you can redistribute it and/or modify
@ -136,7 +136,7 @@ function env_parallel
# #
begin; begin;
for v in (set -n | \ for v in (set -n | \
grep -Ev '^(PARALLEL_TMP|PARALLEL_ENV)$' | \ grep -Ev '^(PARALLEL_TMP)$' | \
grep -E "^$_grep_REGEXP\$" | grep -vE "^$_ignore_UNDERSCORE\$"); grep -E "^$_grep_REGEXP\$" | grep -vE "^$_ignore_UNDERSCORE\$");
# Separate variables with the string: \000 # Separate variables with the string: \000
# array_name1 val1\0 # array_name1 val1\0
@ -158,15 +158,15 @@ function env_parallel
# Ignore read-only vars # Ignore read-only vars
$name=~/^(HOME|USER|COLUMNS|FISH_VERSION|LINES|PWD|SHLVL|_| $name=~/^(HOME|USER|COLUMNS|FISH_VERSION|LINES|PWD|SHLVL|_|
history|status|version)$/x and next; history|status|version)$/x and next;
# Quote $val # Single quote $val
$val=~s/[\002-\011\013-\032\\\#\?\`\(\)\{\}\[\]\^\*\<\=\>\~ if($val =~ /[^-_.+a-z0-9\/]/i) {
\|\; \"\!\$\&\202-\377]/\\\$&/gox; $val =~ s/\047/\047"\047"\047/g; # "-quote single quotes
# Quote single quote $val = "\047$val\047"; # single-quote entire string
$val=~s/'"'"'/\\\$&/go; $val =~ s/^\047\047|\047\047$//g; # Remove unneeded '' at ends
# Quote newline as '\n' } elsif ($val eq "") {
$val =~ s/[\n]/\\\n/go; $val = "\047\047";
# Empty value => 2 single quotes = \047\047 }
$val=~s/^$/\047\047/o;
if($name ne $last and $last) { if($name ne $last and $last) {
# The $name is different, so this is a new variable. # The $name is different, so this is a new variable.
# Print the last one. # Print the last one.
@ -183,16 +183,8 @@ function env_parallel
s/\n/\001/g' s/\n/\001/g'
end; end;
) )
# --session # If --record-env: exit
perl -e 'exit grep { /^--session$/ } @ARGV' -- $argv; or begin; perl -e 'exit grep { /^--record-env$/ } @ARGV' -- $argv; and parallel $argv;
setenv PARALLEL_IGNORED_NAMES (
functions -n | perl -ne 's/,/\n/g; /^(env_parallel)$/ and next; print';
set -n;
)
end;
# If --record-env or --session: exit
perl -e 'exit grep { /^(--record-env|--session)$/ } @ARGV' -- $argv; and parallel $argv;
set _parallel_exit_CODE $status set _parallel_exit_CODE $status
set -e PARALLEL_ENV set -e PARALLEL_ENV
return $_parallel_exit_CODE return $_parallel_exit_CODE

View file

@ -7,7 +7,7 @@
# after which 'env_parallel' works # after which 'env_parallel' works
# #
# #
# Copyright (C) 2016,2017,2018 # Copyright (C) 2016-2019
# Ole Tange and Free Software Foundation, Inc. # Ole Tange and Free Software Foundation, Inc.
# #
# This program is free software; you can redistribute it and/or modify # This program is free software; you can redistribute it and/or modify

View file

@ -7,7 +7,7 @@
# after which 'env_parallel' works # after which 'env_parallel' works
# #
# #
# Copyright (C) 2016,2017,2018 # Copyright (C) 2016-2019
# Ole Tange and Free Software Foundation, Inc. # Ole Tange and Free Software Foundation, Inc.
# #
# This program is free software; you can redistribute it and/or modify # This program is free software; you can redistribute it and/or modify

View file

@ -7,7 +7,7 @@
# after which 'env_parallel' works # after which 'env_parallel' works
# #
# #
# Copyright (C) 2016,2017,2018 # Copyright (C) 2016-2019
# Ole Tange and Free Software Foundation, Inc. # Ole Tange and Free Software Foundation, Inc.
# #
# This program is free software; you can redistribute it and/or modify # This program is free software; you can redistribute it and/or modify

View file

@ -743,7 +743,7 @@ Copyright (C) 2007-10-18 Ole Tange, http://ole.tange.dk
Copyright (C) 2008-2010 Ole Tange, http://ole.tange.dk Copyright (C) 2008-2010 Ole Tange, http://ole.tange.dk
Copyright (C) 2010-2018 Ole Tange, http://ole.tange.dk and Free Copyright (C) 2010-2019 Ole Tange, http://ole.tange.dk and Free
Software Foundation, Inc. Software Foundation, Inc.

View file

@ -7,7 +7,7 @@
# after which 'env_parallel' works # after which 'env_parallel' works
# #
# #
# Copyright (C) 2016-2018 # Copyright (C) 2016-2019
# Ole Tange and Free Software Foundation, Inc. # Ole Tange and Free Software Foundation, Inc.
# #
# This program is free software; you can redistribute it and/or modify # This program is free software; you can redistribute it and/or modify

View file

@ -7,7 +7,7 @@
# after which 'env_parallel' works # after which 'env_parallel' works
# #
# #
# Copyright (C) 2016-2018 # Copyright (C) 2016-2019
# Ole Tange and Free Software Foundation, Inc. # Ole Tange and Free Software Foundation, Inc.
# #
# This program is free software; you can redistribute it and/or modify # This program is free software; you can redistribute it and/or modify

View file

@ -7,7 +7,7 @@
# after which 'env_parallel' works # after which 'env_parallel' works
# #
# #
# Copyright (C) 2016,2017,2018 # Copyright (C) 2016-2019
# Ole Tange and Free Software Foundation, Inc. # Ole Tange and Free Software Foundation, Inc.
# #
# This program is free software; you can redistribute it and/or modify # This program is free software; you can redistribute it and/or modify

View file

@ -1,10 +1,9 @@
#!/usr/bin/perl -w #!/usr/bin/perl -w
# Copyright (C) 2004,2005,2006,2006,2008,2009,2010 Ole Tange, # Copyright (C) 2004-2010 Ole Tange, http://ole.tange.dk
# http://ole.tange.dk
# #
# Copyright (C) 2010,2011,2012,2013,2014,2015,2016,2017,2018 Ole # Copyright (C) 2010-2019 Ole Tange, http://ole.tange.dk and
# Tange, http://ole.tange.dk and Free Software Foundation, Inc. # Free Software Foundation, Inc.
# #
# This program is free software; you can redistribute it and/or modify # This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by # it under the terms of the GNU General Public License as published by
@ -24,11 +23,7 @@
use strict; use strict;
use Getopt::Long; use Getopt::Long;
$Global::progname="niceload"; $Global::progname="niceload";
<<<<<<< HEAD $Global::version = 20181223;
$Global::version = 20180723;
=======
$Global::version = 20181124;
>>>>>>> 8286e771ec57e60ee17aee87741fe96790880db7
Getopt::Long::Configure("bundling","require_order"); Getopt::Long::Configure("bundling","require_order");
get_options_from_array(\@ARGV) || die_usage(); get_options_from_array(\@ARGV) || die_usage();
if($opt::version) { if($opt::version) {

View file

@ -305,13 +305,11 @@ Copyright (C) 2004-11-19 Ole Tange, http://ole.tange.dk
Copyright (C) 2005-2010 Ole Tange, http://ole.tange.dk Copyright (C) 2005-2010 Ole Tange, http://ole.tange.dk
Copyright (C) 2010-2018 Ole Tange, http://ole.tange.dk and Free Copyright (C) 2010-2019 Ole Tange, http://ole.tange.dk and Free
Software Foundation, Inc. Software Foundation, Inc.
=head1 LICENSE =head1 LICENSE
Copyright (C) 2010-2018 Free Software Foundation, Inc.
This program is free software; you can redistribute it and/or modify This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3 of the License, or the Free Software Foundation; either version 3 of the License, or

View file

@ -1,6 +1,6 @@
#!/usr/bin/env perl #!/usr/bin/env perl
# Copyright (C) 2007-2018 Ole Tange and Free Software Foundation, Inc. # Copyright (C) 2007-2019 Ole Tange and Free Software Foundation, Inc.
# #
# This program is free software; you can redistribute it and/or modify # This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by # it under the terms of the GNU General Public License as published by
@ -526,10 +526,16 @@ sub spreadstdin {
# (?!negative lookahead) is needed to avoid backtracking # (?!negative lookahead) is needed to avoid backtracking
# See: https://unix.stackexchange.com/questions/439356/ # See: https://unix.stackexchange.com/questions/439356/
while($buf =~ while($buf =~
/# From start up till recend /(
^((?:(?!$recend$recstart).)*?$recend # Either recstart or at least one char from start
^(?: $recstart | .)
# followed something
(?:(?!$recend$recstart).)*?
# and then recend
$recend
# Then n-1 times recstart.*recend # Then n-1 times recstart.*recend
(?:$recstart(?:(?!$recend$recstart).)*?$recend){$read_n_lines}) (?:$recstart(?:(?!$recend$recstart).)*?$recend){$read_n_lines}
)
# Followed by recstart # Followed by recstart
(?=$recstart)/osx) { (?=$recstart)/osx) {
$anything_written += $anything_written +=
@ -1066,7 +1072,7 @@ sub options_hash {
"plain" => \$opt::plain, "plain" => \$opt::plain,
"profile|J=s" => \@opt::profile, "profile|J=s" => \@opt::profile,
"pipe|spreadstdin" => \$opt::pipe, "pipe|spreadstdin" => \$opt::pipe,
"rr|robin|round-robin|roundrobin" => \$opt::roundrobin, "robin|round-robin|roundrobin" => \$opt::roundrobin,
"recstart=s" => \$opt::recstart, "recstart=s" => \$opt::recstart,
"recend=s" => \$opt::recend, "recend=s" => \$opt::recend,
"regexp|regex" => \$opt::regexp, "regexp|regex" => \$opt::regexp,
@ -1077,6 +1083,8 @@ sub options_hash {
"gnu" => \$opt::gnu, "gnu" => \$opt::gnu,
"link|xapply" => \$opt::link, "link|xapply" => \$opt::link,
"linkinputsource|xapplyinputsource=i" => \@opt::linkinputsource, "linkinputsource|xapplyinputsource=i" => \@opt::linkinputsource,
# Before changing this line, please read
# https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice
"bibtex|citation" => \$opt::citation, "bibtex|citation" => \$opt::citation,
"wc|willcite|will-cite|nn|nonotice|no-notice" => \$opt::willcite, "wc|willcite|will-cite|nn|nonotice|no-notice" => \$opt::willcite,
# Termination and retries # Termination and retries
@ -1180,6 +1188,8 @@ sub parse_options {
my @argv_before = @ARGV; my @argv_before = @ARGV;
@ARGV = read_options(); @ARGV = read_options();
# Before changing this line, please read
# https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice
if(defined $opt::citation) { if(defined $opt::citation) {
citation(\@argv_before,\@ARGV); citation(\@argv_before,\@ARGV);
wait_and_exit(0); wait_and_exit(0);
@ -1283,11 +1293,8 @@ sub parse_options {
} }
$opt::memfree = multiply_binary_prefix($opt::memfree); $opt::memfree = multiply_binary_prefix($opt::memfree);
check_invalid_option_combinations(); check_invalid_option_combinations();
if((defined $opt::fifo or defined $opt::cat if((defined $opt::fifo or defined $opt::cat)
or
defined $opt::roundrobin)
and not $opt::pipepart) { and not $opt::pipepart) {
# --fifo, --cat, and --round-robin => --pipe if not --pipepart
$opt::pipe = 1; $opt::pipe = 1;
} }
if(defined $opt::minversion) { if(defined $opt::minversion) {
@ -1421,14 +1428,7 @@ sub parse_options {
# #
# This problem has been covered by others - though no solution has # This problem has been covered by others - though no solution has
# been found: # been found:
#
# "Is it alright to compromise or even deliberately ignore the
# happiness of the maintainers so that we can enjoy free and open
# source software?"
# Slide 8 from:
# https://www.slideshare.net/NadiaEghbal/consider-the-maintainer # https://www.slideshare.net/NadiaEghbal/consider-the-maintainer
#
# NumPy has long not had funding:
# https://www.numfocus.org/blog/why-is-numpy-only-now-getting-funded/ # https://www.numfocus.org/blog/why-is-numpy-only-now-getting-funded/
# #
# Before implementing the citation notice it was discussed with # Before implementing the citation notice it was discussed with
@ -1469,20 +1469,19 @@ sub parse_options {
# runs. If that is too much trouble for you, why not use one of # runs. If that is too much trouble for you, why not use one of
# the alternatives instead? # the alternatives instead?
# See a list in: 'man parallel_alternatives' # See a list in: 'man parallel_alternatives'
#
# Please read the above before changing this line.
citation_notice(); citation_notice();
parse_halt(); parse_halt();
if(defined $ENV{'PARALLEL_ENV'}) { if($ENV{'PARALLEL_ENV'}) {
# Read environment and set $Global::parallel_env # Read environment and set $Global::parallel_env
# Must be done before is_acceptable_command_line_length() # Must be done before is_acceptable_command_line_length()
my $penv = $ENV{'PARALLEL_ENV'}; my $penv = $ENV{'PARALLEL_ENV'};
if(not ($opt::onall or $opt::nonall)) { # unset $PARALLEL_ENV: It should not be given to children
# unset $PARALLEL_ENV: It should not be given to children # because it takes up a lot of env space
# because it takes up a lot of env space delete $ENV{'PARALLEL_ENV'};
# Except when --(n)onall/nonall is set
# delete $ENV{'PARALLEL_ENV'};
}
if(-e $penv) { if(-e $penv) {
# This is a file/fifo: Replace envvar with content of file # This is a file/fifo: Replace envvar with content of file
open(my $parallel_env, "<", $penv) || open(my $parallel_env, "<", $penv) ||
@ -1528,7 +1527,7 @@ sub check_invalid_option_combinations {
::wait_and_exit(255); ::wait_and_exit(255);
} }
if(defined $opt::retries and defined $opt::roundrobin) { if(defined $opt::retries and defined $opt::roundrobin) {
::error("--retries cannot be combined with --round-robin."); ::error("--retries cannot be combined with --roundrobin.");
::wait_and_exit(255); ::wait_and_exit(255);
} }
if(defined $opt::pipepart and if(defined $opt::pipepart and
@ -1572,11 +1571,7 @@ sub check_invalid_option_combinations {
sub init_globals { sub init_globals {
# Defaults: # Defaults:
<<<<<<< HEAD $Global::version = 20181223;
$Global::version = 20180723;
=======
$Global::version = 20181124;
>>>>>>> 8286e771ec57e60ee17aee87741fe96790880db7
$Global::progname = 'parallel'; $Global::progname = 'parallel';
$Global::infinity = 2**31; $Global::infinity = 2**31;
$Global::debug = 0; $Global::debug = 0;
@ -2180,6 +2175,8 @@ sub arrayindex {
# Input: # Input:
# $arr_ref1 = ref to @array1 to search in # $arr_ref1 = ref to @array1 to search in
# $arr_ref2 = ref to @array2 to search for # $arr_ref2 = ref to @array2 to search for
# Returns:
# $pos = position of @array1 in @array2, -1 if not found
my ($arr_ref1,$arr_ref2) = @_; my ($arr_ref1,$arr_ref2) = @_;
my $array1_as_string = join "", map { "\0".$_ } @$arr_ref1; my $array1_as_string = join "", map { "\0".$_ } @$arr_ref1;
my $array2_as_string = join "", map { "\0".$_ } @$arr_ref2; my $array2_as_string = join "", map { "\0".$_ } @$arr_ref2;
@ -2299,8 +2296,8 @@ sub __QUOTING_ARGUMENTS_FOR_SHELL__ {}
sub shell_quote { sub shell_quote {
# Input: # Input:
# @strings = strings to be quoted # @strings = strings to be quoted
# Output: # Returns:
# @shell_quoted_strings = string quoted with \ as needed by the shell # @shell_quoted_strings = string quoted as needed by the shell
return wantarray ? return wantarray ?
(map { Q($_) } @_) (map { Q($_) } @_)
: (join" ",map { Q($_) } @_); : (join" ",map { Q($_) } @_);
@ -2347,7 +2344,11 @@ sub shell_quote_scalar_csh {
sub shell_quote_scalar_default { sub shell_quote_scalar_default {
# Quote for other shells (Bourne compatibles) # Quote for other shells (Bourne compatibles)
my ($par) = $_[0]; # Inputs:
# $string = string to be quoted
# Returns:
# $shell_quoted = string quoted as needed by the shell
my $par = $_[0];
if($par =~ /[^-_.+a-z0-9\/]/i) { if($par =~ /[^-_.+a-z0-9\/]/i) {
$par =~ s/'/'"'"'/g; # "-quote single quotes $par =~ s/'/'"'"'/g; # "-quote single quotes
$par = "'$par'"; # '-quote entire string $par = "'$par'"; # '-quote entire string
@ -2441,6 +2442,10 @@ sub pQ {
sub unquote_printf { sub unquote_printf {
# Convert \t \n \r \000 \0 # Convert \t \n \r \000 \0
# Inputs:
# $string = string with \t \n \r \num \0
# Returns:
# $replaced = string with TAB NEWLINE CR <ascii-num> NUL
$_ = shift; $_ = shift;
s/\\t/\t/g; s/\\t/\t/g;
s/\\n/\n/g; s/\\n/\n/g;
@ -4276,6 +4281,8 @@ sub usage {
" O. Tange (2018): GNU Parallel 2018, Mar 2018, ISBN 9781387509881,", " O. Tange (2018): GNU Parallel 2018, Mar 2018, ISBN 9781387509881,",
" DOI https://doi.org/10.5281/zenodo.1146014", " DOI https://doi.org/10.5281/zenodo.1146014",
"", "",
# Before changing this line, please read
# https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice
"This helps funding further development; AND IT WON'T COST YOU A CENT.", "This helps funding further development; AND IT WON'T COST YOU A CENT.",
"If you pay 10000 EUR you should feel free to use GNU Parallel without citing.", "If you pay 10000 EUR you should feel free to use GNU Parallel without citing.",
"", "",
@ -4304,6 +4311,8 @@ sub citation_notice {
" O. Tange (2018): GNU Parallel 2018, Mar 2018, ISBN 9781387509881,", " O. Tange (2018): GNU Parallel 2018, Mar 2018, ISBN 9781387509881,",
" DOI https://doi.org/10.5281/zenodo.1146014", " DOI https://doi.org/10.5281/zenodo.1146014",
"", "",
# Before changing this line, please read
# https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice
"This helps funding further development; AND IT WON'T COST YOU A CENT.", "This helps funding further development; AND IT WON'T COST YOU A CENT.",
"If you pay 10000 EUR you should feel free to use GNU Parallel without citing.", "If you pay 10000 EUR you should feel free to use GNU Parallel without citing.",
"", "",
@ -4384,7 +4393,7 @@ sub version {
print join print join
("\n", ("\n",
"GNU $Global::progname $Global::version", "GNU $Global::progname $Global::version",
"Copyright (C) 2007-2018 Ole Tange and Free Software Foundation, Inc.", "Copyright (C) 2007-2019 Ole Tange and Free Software Foundation, Inc.",
"License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>", "License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>",
"This is free software: you are free to change and redistribute it.", "This is free software: you are free to change and redistribute it.",
"GNU $Global::progname comes with no warranty.", "GNU $Global::progname comes with no warranty.",
@ -4424,6 +4433,8 @@ sub citation {
"", "",
"(Feel free to use \\nocite{tange_ole_2018_1146014})", "(Feel free to use \\nocite{tange_ole_2018_1146014})",
"", "",
# Before changing this line, please read
# https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice
"This helps funding further development; AND IT WON'T COST YOU A CENT.", "This helps funding further development; AND IT WON'T COST YOU A CENT.",
"If you pay 10000 EUR you should feel free to use GNU Parallel without citing.", "If you pay 10000 EUR you should feel free to use GNU Parallel without citing.",
"", "",
@ -4504,7 +4515,7 @@ sub embed {
} }
print "#!$Global::shell print "#!$Global::shell
# Copyright (C) 2007-2018 $user, Ole Tange and Free Software # Copyright (C) 2007-2019 $user, Ole Tange and Free Software
# Foundation, Inc. # Foundation, Inc.
# #
# This program is free software; you can redistribute it and/or modify # This program is free software; you can redistribute it and/or modify
@ -6582,28 +6593,38 @@ sub sct_gnu_linux() {
# 'active' => #taskset_threads } # 'active' => #taskset_threads }
my $cpu; my $cpu;
local $/ = "\n"; # If delimiter is set, then $/ will be wrong local $/ = "\n"; # If delimiter is set, then $/ will be wrong
if(-e "/proc/cpuinfo") { if($ENV{'PARALLEL_CPUINFO'} or -e "/proc/cpuinfo") {
$cpu->{'sockets'} = 0; $cpu->{'sockets'} = 0;
$cpu->{'cores'} = 0; $cpu->{'cores'} = 0;
$cpu->{'threads'} = 0; $cpu->{'threads'} = 0;
my %seen; my %seen;
my %phy_seen; my %phy_seen;
my @cpuinfo;
my $physicalid;
if(open(my $in_fh, "<", "/proc/cpuinfo")) { if(open(my $in_fh, "<", "/proc/cpuinfo")) {
while(<$in_fh>) { @cpuinfo = <$in_fh>;
if(/^physical id.*[:](.*)/ and not $phy_seen{$1}++) { close $in_fh;
}
if($ENV{'PARALLEL_CPUINFO'}) {
# Use CPUINFO from environment - used for testing only
@cpuinfo = split/(?<=\n)/,$ENV{'PARALLEL_CPUINFO'};
}
for(@cpuinfo) {
if(/^physical id.*[:](.*)/) {
$physicalid=$1;
if(not $phy_seen{$1}++) {
$cpu->{'sockets'}++; $cpu->{'sockets'}++;
} }
if(/^core id.*[:](.*)/ and not $seen{$1}++) {
$cpu->{'cores'}++;
}
/^processor.*[:]/i and $cpu->{'threads'}++;
} }
close $in_fh; if(/^core id.*[:](.*)/ and not $seen{$physicalid,$1}++) {
$cpu->{'cores'}++;
}
/^processor.*[:]/i and $cpu->{'threads'}++;
} }
$cpu->{'sockets'} ||= 1; $cpu->{'sockets'} ||= 1;
$cpu->{'cores'} ||= $cpu->{'threads'}; $cpu->{'cores'} ||= $cpu->{'threads'};
} }
if(-e "/proc/self/status") { if(-e "/proc/self/status" and not $ENV{'PARALLEL_CPUINFO'}) {
# if 'taskset' is used to limit number of threads # if 'taskset' is used to limit number of threads
if(open(my $in_fh, "<", "/proc/self/status")) { if(open(my $in_fh, "<", "/proc/self/status")) {
while(<$in_fh>) { while(<$in_fh>) {
@ -8010,12 +8031,10 @@ sub wrapped {
# --pipepart: prepend: # --pipepart: prepend:
# < /tmp/foo perl -e 'while(@ARGV) { # < /tmp/foo perl -e 'while(@ARGV) {
# sysseek(STDIN,shift,0) || die; $left = shift; # sysseek(STDIN,shift,0) || die; $left = shift;
# while($read = # while($read = sysread(STDIN,$buf, ($left > 131072 ? 131072 : $left))){
# sysread(STDIN,$buf,
# ($left > 131072 ? 131072 : $left))) {
# $left -= $read; syswrite(STDOUT,$buf); # $left -= $read; syswrite(STDOUT,$buf);
# } # }
# }' 0 0 0 11 | # }' 0 0 0 11 |
# #
# --pipepart --tee: prepend: # --pipepart --tee: prepend:
# < dash-a-file # < dash-a-file
@ -8142,7 +8161,7 @@ sub base64_zip_eval {
} }
wait; wait;
$SIG{CHLD} = $chld; $SIG{CHLD} = $chld;
eval $eval; eval $eval;
}); });
::debug("base64",$script,"\n"); ::debug("base64",$script,"\n");
return $script; return $script;
@ -8203,16 +8222,14 @@ sub sshlogin_wrap {
if(not $monitor_parent_sshd_script) { if(not $monitor_parent_sshd_script) {
$monitor_parent_sshd_script = $monitor_parent_sshd_script =
# This will be packed in ', so only use " # This will be packed in ', so only use "
::spacefree(0,'$shell = "'.($ENV{'PARALLEL_SHELL'} ::spacefree(0,'$shell = "'.($ENV{'PARALLEL_SHELL'} || '$ENV{SHELL}').'";'.
|| '$ENV{SHELL}').'";'. '$tmpdir = "'.::perl_quote_scalar($ENV{'TMPDIR'}).'";'.
'$tmpdir = "'.::pQ($ENV{'TMPDIR'}).'";'.
'$nice = '.$opt::nice.';'. '$nice = '.$opt::nice.';'.
q{ q{
# Set $PARALLEL_TMP to a non-existent file name in $TMPDIR # Set $PARALLEL_TMP to a non-existent file name in $TMPDIR
do { do {
$ENV{PARALLEL_TMP} = $tmpdir."/par". $ENV{PARALLEL_TMP} = $tmpdir."/par".
join"", map join"", map { (0..9,"a".."z","A".."Z")[rand(62)] } (1..5);
{ (0..9,"a".."z","A".."Z")[rand(62)] } (1..5);
} while(-e $ENV{PARALLEL_TMP}); } while(-e $ENV{PARALLEL_TMP});
$SIG{CHLD} = sub { $done = 1; }; $SIG{CHLD} = sub { $done = 1; };
$pid = fork; $pid = fork;
@ -8259,8 +8276,7 @@ sub sshlogin_wrap {
local $/ = "\n"; local $/ = "\n";
# --env _ # --env _
# Include all vars that are not in a clean environment # Include all vars that are not in a clean environment
if(open(my $vars_fh, if(open(my $vars_fh, "<", $Global::config_dir . "/ignored_vars")) {
"<", $Global::config_dir . "/ignored_vars")) {
my @ignore = <$vars_fh>; my @ignore = <$vars_fh>;
chomp @ignore; chomp @ignore;
my %ignore; my %ignore;
@ -8274,10 +8290,8 @@ sub sshlogin_wrap {
::wait_and_exit(255); ::wait_and_exit(255);
} }
} }
# Duplicate vars as BASH functions to include # Duplicate vars as BASH functions to include post-shellshock functions (v1+v2)
# post-shellshock functions (v1+v2) # So --env myfunc should look for BASH_FUNC_myfunc() and BASH_FUNC_myfunc%%
# So --env myfunc should look for
# BASH_FUNC_myfunc() and BASH_FUNC_myfunc%%
push(@vars, "PARALLEL_PID", "PARALLEL_SEQ", push(@vars, "PARALLEL_PID", "PARALLEL_SEQ",
map { ("BASH_FUNC_$_()", "BASH_FUNC_$_%%") } @vars); map { ("BASH_FUNC_$_()", "BASH_FUNC_$_%%") } @vars);
# Keep only defined variables # Keep only defined variables
@ -8308,19 +8322,17 @@ sub sshlogin_wrap {
if(@bashfunc) { if(@bashfunc) {
# Functions are not supported for all shells # Functions are not supported for all shells
if($Global::shell !~ m:(bash|rbash|zsh|rzsh|dash|ksh):) { if($Global::shell !~ m:(bash|rbash|zsh|rzsh|dash|ksh):) {
::warning("Shell functions may not be supported in ". ::warning("Shell functions may not be supported in $Global::shell.");
"$Global::shell.");
} }
$bashfuncset = $bashfuncset =
'@bash_functions=qw('."@bash_functions".");". '@bash_functions=qw('."@bash_functions".");".
::spacefree(1,'$shell="'.($ENV{'PARALLEL_SHELL'} || ::spacefree(1,'$shell="'.($ENV{'PARALLEL_SHELL'} || '$ENV{SHELL}').'";'.q{
'$ENV{SHELL}').'";'.q{
if($shell=~/csh/) { if($shell=~/csh/) {
print STDERR "CSH/TCSH DO NOT SUPPORT newlines IN VARIABLES/FUNCTIONS. Unset @bash_functions\n"; print STDERR "CSH/TCSH DO NOT SUPPORT newlines IN VARIABLES/FUNCTIONS. Unset @bash_functions\n";
exec "false"; exec "false";
} }
}). }).
"\n".'$bashfunc = "'.::pQ("@bashfunc").'";'; "\n".'$bashfunc = "'.::perl_quote_scalar("@bashfunc").'";';
} else { } else {
$bashfuncset = '$bashfunc = "";' $bashfuncset = '$bashfunc = "";'
} }
@ -8352,13 +8364,11 @@ sub sshlogin_wrap {
$command = "cd ".::Q($wd)." || exit 255; ".$command; $command = "cd ".::Q($wd)." || exit 255; ".$command;
} }
if(@opt::env) { if(@opt::env) {
# Prepend with environment setter, # Prepend with environment setter, which sets functions in zsh
# which sets functions in zsh
my ($csh_friendly,$envset,$bashfuncset) = env_as_eval(); my ($csh_friendly,$envset,$bashfuncset) = env_as_eval();
my $perl_code = $envset.$bashfuncset. my $perl_code = $envset.$bashfuncset.
('@ARGV="'.::pQ($command).'";'. '@ARGV="'.::perl_quote_scalar($command).'";'.
'exec"'.$Global::shell.'",'. "exec\"$Global::shell\",\"-c\",\(\$bashfunc.\"\@ARGV\"\)\;die\"exec:\$\!\\n\"\;";
'"-c",($bashfunc."@ARGV");die"exec:$!\n";');
if(length $perl_code > 999 if(length $perl_code > 999
or or
not $csh_friendly not $csh_friendly
@ -8368,11 +8378,7 @@ sub sshlogin_wrap {
# csh does not deal well with $ENV with \n # csh does not deal well with $ENV with \n
$self->{'sshlogin_wrap'}{$command} = base64_wrap($perl_code); $self->{'sshlogin_wrap'}{$command} = base64_wrap($perl_code);
} else { } else {
<<<<<<< HEAD
$self->{'sshlogin_wrap'} = "perl -e ".::Q($perl_code);
=======
$self->{'sshlogin_wrap'}{$command} = "perl -e ".::Q($perl_code); $self->{'sshlogin_wrap'}{$command} = "perl -e ".::Q($perl_code);
>>>>>>> 8286e771ec57e60ee17aee87741fe96790880db7
} }
} else { } else {
$self->{'sshlogin_wrap'}{$command} = $command; $self->{'sshlogin_wrap'}{$command} = $command;
@ -8383,17 +8389,12 @@ sub sshlogin_wrap {
# Create remote workdir if needed. Then cd to it. # Create remote workdir if needed. Then cd to it.
my $wd = ::pQ($self->workdir()); my $wd = ::pQ($self->workdir());
$pwd = qq{system("mkdir","-p","--","$wd"); chdir "$wd" ||}. $pwd = qq{system("mkdir","-p","--","$wd"); chdir "$wd" ||}.
qq{print(STDERR "parallel: Cannot chdir to $wd\\n") &&}. qq{print(STDERR "parallel: Cannot chdir to $wd\\n") && exit 255;};
qq{exit 255;};
} }
my ($csh_friendly,$envset,$bashfuncset) = env_as_eval(); my ($csh_friendly,$envset,$bashfuncset) = env_as_eval();
my $remote_command = $pwd.$envset.$bashfuncset. my $remote_command = $pwd.$envset.$bashfuncset.
<<<<<<< HEAD
'@ARGV="'.::pQ($command).'";'. monitor_parent_sshd_script();
=======
'@ARGV="'.::perl_quote_scalar($command).'";'. '@ARGV="'.::perl_quote_scalar($command).'";'.
monitor_parent_sshd_script(); monitor_parent_sshd_script();
>>>>>>> 8286e771ec57e60ee17aee87741fe96790880db7
$quoted_remote_command = "perl -e ". ::Q($remote_command); $quoted_remote_command = "perl -e ". ::Q($remote_command);
my $dq_remote_command = ::Q($quoted_remote_command); my $dq_remote_command = ::Q($quoted_remote_command);
if(length $dq_remote_command > 999 if(length $dq_remote_command > 999
@ -8405,11 +8406,7 @@ sub sshlogin_wrap {
# csh does not deal well with $ENV with \n # csh does not deal well with $ENV with \n
$quoted_remote_command = $quoted_remote_command =
"perl -e ". ::Q(::Q(base64_zip_eval()))." ". "perl -e ". ::Q(::Q(base64_zip_eval()))." ".
<<<<<<< HEAD
join" ", ::shell_quote(::shell_quote(string_zip_base64($remote_command)));
=======
join" ",::shell_quote(::shell_quote(string_zip_base64($remote_command))); join" ",::shell_quote(::shell_quote(string_zip_base64($remote_command)));
>>>>>>> 8286e771ec57e60ee17aee87741fe96790880db7
} else { } else {
$quoted_remote_command = $dq_remote_command; $quoted_remote_command = $dq_remote_command;
} }
@ -8523,24 +8520,17 @@ sub sshreturn {
my $sshlogin = $self->sshlogin(); my $sshlogin = $self->sshlogin();
my $sshcmd = $sshlogin->sshcommand(); my $sshcmd = $sshlogin->sshcommand();
my $serverlogin = $sshlogin->serverlogin(); my $serverlogin = $sshlogin->serverlogin();
<<<<<<< HEAD
my $rsync_opts = $ENV{'PARALLEL_RSYNC_OPTS'}." -e".::Q($sshcmd);
=======
my $rsync_opts = $ENV{'PARALLEL_RSYNC_OPTS'}. " -e". ::Q($sshcmd); my $rsync_opts = $ENV{'PARALLEL_RSYNC_OPTS'}. " -e". ::Q($sshcmd);
>>>>>>> 8286e771ec57e60ee17aee87741fe96790880db7
my $pre = ""; my $pre = "";
for my $file ($self->return()) { for my $file ($self->return()) {
# Remove ./ if any $file =~ s:^\./::g; # Remove ./ if any
$file =~ s:^\./::g; my $relpath = ($file !~ m:^/:) || ($file =~ m:/\./:); # Is the path relative or /./?
# Is the path relative or /./?
my $relpath = ($file !~ m:^/:) || ($file =~ m:/\./:);
my $cd = ""; my $cd = "";
my $wd = ""; my $wd = "";
if($relpath) { if($relpath) {
# rsync -avR /foo/./bar/baz.c remote:/tmp/ # rsync -avR /foo/./bar/baz.c remote:/tmp/
# == (on old systems) # == (on old systems)
# rsync -avR --rsync-path="cd /foo; rsync" # rsync -avR --rsync-path="cd /foo; rsync" remote:bar/baz.c /tmp/
# remote:bar/baz.c /tmp/
$wd = ::shell_quote_file($self->workdir()."/"); $wd = ::shell_quote_file($self->workdir()."/");
} }
# Only load File::Basename if actually needed # Only load File::Basename if actually needed
@ -8552,12 +8542,7 @@ sub sshreturn {
$nobasedir =~ s:.*/\./::; $nobasedir =~ s:.*/\./::;
$cd = ::shell_quote_file(::dirname($nobasedir)); $cd = ::shell_quote_file(::dirname($nobasedir));
my $rsync_cd = '--rsync-path='.::Q("cd $wd$cd; rsync"); my $rsync_cd = '--rsync-path='.::Q("cd $wd$cd; rsync");
<<<<<<< HEAD
my $basename =
::shell_quote_scalar(::shell_quote_file(::basename($file)));
=======
my $basename = ::Q(::shell_quote_file(::basename($file))); my $basename = ::Q(::shell_quote_file(::basename($file)));
>>>>>>> 8286e771ec57e60ee17aee87741fe96790880db7
# --return # --return
# mkdir -p /home/tange/dir/subdir/; # mkdir -p /home/tange/dir/subdir/;
# rsync (--protocol 30) -rlDzR # rsync (--protocol 30) -rlDzR
@ -8582,7 +8567,6 @@ sub sshcleanup {
my $cleancmd = ""; my $cleancmd = "";
for my $file ($self->remote_cleanup()) { for my $file ($self->remote_cleanup()) {
# TODO @subworkdirs not used?
my @subworkdirs = parentdirs_of($file); my @subworkdirs = parentdirs_of($file);
$cleancmd .= $sshlogin->cleanup_cmd($file,$workdir).";"; $cleancmd .= $sshlogin->cleanup_cmd($file,$workdir).";";
} }
@ -8631,9 +8615,7 @@ sub workdir {
$part eq "" and next; $part eq "" and next;
$parent .= "/".$part; $parent .= "/".$part;
my ($parent_dev, $parent_ino) = (stat($parent))[0,1]; my ($parent_dev, $parent_ino) = (stat($parent))[0,1];
if($parent_dev == $home_dev if($parent_dev == $home_dev and $parent_ino == $home_ino) {
and
$parent_ino == $home_ino) {
# dev and ino is the same: We found the homedir. # dev and ino is the same: We found the homedir.
$workdir = join("/",@dir_parts); $workdir = join("/",@dir_parts);
last; last;
@ -8698,8 +8680,7 @@ sub start {
# The eval is needed to catch exception from open3 # The eval is needed to catch exception from open3
eval { eval {
if(not $pid = ::open3($stdin_fh, ">&OUT", ">&ERR", "-")) { if(not $pid = ::open3($stdin_fh, ">&OUT", ">&ERR", "-")) {
# Each child gets its own process group # Each child gets its own process group to make it safe to killall
# to make it safe to killall
eval{ setpgrp(0,0) }; eval{ setpgrp(0,0) };
eval{ setpriority(0,0,$opt::nice) }; eval{ setpriority(0,0,$opt::nice) };
exec($Global::shell,"-c",$command) exec($Global::shell,"-c",$command)
@ -8731,8 +8712,7 @@ sub start {
"exec '$Global::shell', '-c', \@ARGV"); "exec '$Global::shell', '-c', \@ARGV");
# The eval is needed to catch exception from open3 # The eval is needed to catch exception from open3
eval { eval {
$pid = ::open3($stdin_fh, ">&OUT", ">&ERR", $pid = ::open3($stdin_fh, ">&OUT", ">&ERR", @setpgrp_wrap, $command)
@setpgrp_wrap, $command)
|| ::die_bug("open3-$stdin_fh"); || ::die_bug("open3-$stdin_fh");
1; 1;
}; };
@ -9075,7 +9055,7 @@ sub print {
if($Global::joblog and defined $self->{'exitstatus'}) { if($Global::joblog and defined $self->{'exitstatus'}) {
# Add to joblog when finished # Add to joblog when finished
$self->print_joblog(); $self->print_joblog();
# Printing is only relevant for grouped/--line-buffer output # Printing is only relevant for grouped/--line-buffer output.
$opt::ungroup and return; $opt::ungroup and return;
} }
} }
@ -9096,8 +9076,7 @@ sub print {
} }
next; next;
} }
::debug("print", ::debug("print", "File descriptor $fdno (", $self->fh($fdno,"name"), "):\n");
"File descriptor $fdno (", $self->fh($fdno,"name"), "):\n");
if($opt::linebuffer) { if($opt::linebuffer) {
# Line buffered print out # Line buffered print out
$self->print_linebuffer($fdno,$in_fh,$out_fd); $self->print_linebuffer($fdno,$in_fh,$out_fd);
@ -9158,8 +9137,7 @@ sub print {
print $Global::csv_fh print $Global::csv_fh
(map { $$_ } (map { $$_ }
combine_ref("Seq", "Host", "Starttime", "JobRuntime", combine_ref("Seq", "Host", "Starttime", "JobRuntime",
"Send", "Receive", "Exitval", "Signal", "Send", "Receive", "Exitval", "Signal", "Command",
"Command",
@V, @V,
"Stdout","Stderr" "Stdout","Stderr"
)),"\n"; )),"\n";
@ -9565,8 +9543,7 @@ sub set_exitsignal {
# --halt now,fail=X% or soon,fail=X% # --halt now,fail=X% or soon,fail=X%
# --halt now,done=X% or soon,done=X% # --halt now,done=X% or soon,done=X%
$Global::halt_exitstatus = $Global::halt_exitstatus =
::ceil($Global::total_failed / $total_jobs ::ceil($Global::total_failed / $total_jobs * 100);
* 100);
} elsif($Global::halt_count) { } elsif($Global::halt_count) {
# --halt now,fail=X or soon,fail=X # --halt now,fail=X or soon,fail=X
# --halt now,done=X or soon,done=X # --halt now,done=X or soon,done=X
@ -9580,8 +9557,7 @@ sub set_exitsignal {
$Global::halt_exitstatus = $Global::halt_exitstatus =
($job->exitstatus() ($job->exitstatus()
or or
$job->exitsignal() ? $job->exitsignal() ? $job->exitsignal() + 128 : 0);
$job->exitsignal() + 128 : 0);
} }
} }
::debug("halt","Pct: ",$Global::halt_pct, ::debug("halt","Pct: ",$Global::halt_pct,
@ -9731,8 +9707,7 @@ sub slot {
} }
if(defined $self->{'max_number_of_args'}) { if(defined $self->{'max_number_of_args'}) {
if($self->number_of_args() >= if($self->number_of_args() >= $self->{'max_number_of_args'}) {
$self->{'max_number_of_args'}) {
last; last;
} }
} }
@ -9744,13 +9719,11 @@ sub slot {
$already_spread ||= 1; $already_spread ||= 1;
if($self->number_of_args() > 1) { if($self->number_of_args() > 1) {
$self->{'max_number_of_args'} = $self->{'max_number_of_args'} =
::ceil($self->number_of_args() / ::ceil($self->number_of_args()/$Global::max_jobs_running);
$Global::max_jobs_running);
$Global::JobQueue->{'commandlinequeue'}->{'max_number_of_args'} = $Global::JobQueue->{'commandlinequeue'}->{'max_number_of_args'} =
$self->{'max_number_of_args'}; $self->{'max_number_of_args'};
$self->{'arg_queue'}->unget($self->pop_all()); $self->{'arg_queue'}->unget($self->pop_all());
while($self->number_of_args() < while($self->number_of_args() < $self->{'max_number_of_args'}) {
$self->{'max_number_of_args'}) {
$self->push($self->{'arg_queue'}->get()); $self->push($self->{'arg_queue'}->get());
} }
} }
@ -9758,8 +9731,7 @@ sub slot {
} }
if($opt::sqlmaster) { if($opt::sqlmaster) {
# Insert the V1..Vn for this $seq in SQL table # Insert the V1..Vn for this $seq in SQL table instead of generating one
# instead of generating one
$Global::sql->insert_records($self->seq(), $self->{'command'}, $Global::sql->insert_records($self->seq(), $self->{'command'},
$self->{'arg_list_flat_orig'}); $self->{'arg_list_flat_orig'});
} }
@ -10119,8 +10091,8 @@ sub replace_placeholders {
# \257<...\257> or space # \257<...\257> or space
$_ $_
} else { } else {
# Split each space into a token # Split each space/tab into a token
split /(?= )|(?<= )/ split /(?=\s)|(?<=\s)/
} }
} }
# Split \257< ... \257> into own token # Split \257< ... \257> into own token
@ -10157,7 +10129,7 @@ sub replace_placeholders {
my @ctxgroup; my @ctxgroup;
for my $t (@tokens,"\0end") { for my $t (@tokens,"\0end") {
# \0end = last token was end of tokens. # \0end = last token was end of tokens.
if($t eq " " or $t eq "\0end" or $t eq "\0spc") { if($t eq "\t" or $t eq " " or $t eq "\0end" or $t eq "\0spc") {
# Context group complete: Replace in it # Context group complete: Replace in it
if(grep { /^\257</ } @ctxgroup) { if(grep { /^\257</ } @ctxgroup) {
# Context group contains a replacement string: # Context group contains a replacement string:

View file

@ -1054,7 +1054,7 @@ to see the difference:
If used with B<--onall> or B<--nonall> the output will grouped by If used with B<--onall> or B<--nonall> the output will grouped by
sshlogin in sorted order. sshlogin in sorted order.
If used with B<--pipe --round-robin> and the same input, the jobslots If used with B<--pipe --roundrobin> and the same input, the jobslots
will get the same blocks in the same order in every run. will get the same blocks in the same order in every run.
@ -1460,32 +1460,20 @@ on remote computers).
Print the number of physical CPU cores and exit. Print the number of physical CPU cores and exit.
<<<<<<< HEAD
=item B<--number-of-cores> (beta testing)
=======
=item B<--number-of-cores> =item B<--number-of-cores>
>>>>>>> 8286e771ec57e60ee17aee87741fe96790880db7
Print the number of physical CPU cores and exit (used by GNU B<parallel> itself Print the number of physical CPU cores and exit (used by GNU B<parallel> itself
to determine the number of physical CPU cores on remote computers). to determine the number of physical CPU cores on remote computers).
<<<<<<< HEAD
=item B<--number-of-sockets> (beta testing)
=======
=item B<--number-of-sockets> =item B<--number-of-sockets>
>>>>>>> 8286e771ec57e60ee17aee87741fe96790880db7
Print the number of filled CPU sockets and exit (used by GNU Print the number of filled CPU sockets and exit (used by GNU
B<parallel> itself to determine the number of filled CPU sockets on B<parallel> itself to determine the number of filled CPU sockets on
remote computers). remote computers).
<<<<<<< HEAD
=item B<--number-of-threads> (beta testing)
=======
=item B<--number-of-threads> =item B<--number-of-threads>
>>>>>>> 8286e771ec57e60ee17aee87741fe96790880db7
Print the number of hyperthreaded CPU cores and exit (used by GNU Print the number of hyperthreaded CPU cores and exit (used by GNU
B<parallel> itself to determine the number of hyperthreaded CPU cores B<parallel> itself to determine the number of hyperthreaded CPU cores
@ -1897,8 +1885,6 @@ with B<--sshlogin>.
=item B<--round> =item B<--round>
=item B<--rr>
Normally B<--pipe> will give a single block to each instance of the Normally B<--pipe> will give a single block to each instance of the
command. With B<--round-robin> all blocks will at random be written to command. With B<--round-robin> all blocks will at random be written to
commands already running. This is useful if the command takes a long commands already running. This is useful if the command takes a long
@ -2590,15 +2576,9 @@ Use the replacement string I<replace-str> instead of B<{.}> for input
line without extension. line without extension.
<<<<<<< HEAD
=item B<--use-sockets-instead-of-threads> (beta testing)
=item B<--use-cores-instead-of-threads> (beta testing)
=======
=item B<--use-sockets-instead-of-threads> =item B<--use-sockets-instead-of-threads>
=item B<--use-cores-instead-of-threads> =item B<--use-cores-instead-of-threads>
>>>>>>> 8286e771ec57e60ee17aee87741fe96790880db7
=item B<--use-cpus-instead-of-cores> (obsolete) =item B<--use-cpus-instead-of-cores> (obsolete)
@ -4897,7 +4877,7 @@ Copyright (C) 2007-10-18 Ole Tange, http://ole.tange.dk
Copyright (C) 2008-2010 Ole Tange, http://ole.tange.dk Copyright (C) 2008-2010 Ole Tange, http://ole.tange.dk
Copyright (C) 2010-2018 Ole Tange, Copyright (C) 2010-2019 Ole Tange,
http://ole.tange.dk and Free Software Foundation, Inc. http://ole.tange.dk and Free Software Foundation, Inc.
Parts of the manual concerning B<xargs> compatibility is inspired by Parts of the manual concerning B<xargs> compatibility is inspired by
@ -4906,8 +4886,6 @@ the manual of B<xargs> from GNU findutils 4.4.2.
=head1 LICENSE =head1 LICENSE
Copyright (C) 2007-2018 Free Software Foundation, Inc.
This program is free software; you can redistribute it and/or modify This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3 of the License, or the Free Software Foundation; either version 3 of the License, or

View file

@ -1728,7 +1728,7 @@ https://github.com/john01dav/spp
Url for spread Url for spread
https://github.com/xuchenCN/go-pssh
https://github.com/amritb/with-this.git https://github.com/amritb/with-this.git
@ -1951,7 +1951,7 @@ Copyright (C) 2007-10-18 Ole Tange, http://ole.tange.dk
Copyright (C) 2008-2010 Ole Tange, http://ole.tange.dk Copyright (C) 2008-2010 Ole Tange, http://ole.tange.dk
Copyright (C) 2010-2018 Ole Tange, http://ole.tange.dk and Free Copyright (C) 2010-2019 Ole Tange, http://ole.tange.dk and Free
Software Foundation, Inc. Software Foundation, Inc.
Parts of the manual concerning B<xargs> compatibility is inspired by Parts of the manual concerning B<xargs> compatibility is inspired by
@ -1960,8 +1960,6 @@ the manual of B<xargs> from GNU findutils 4.4.2.
=head1 LICENSE =head1 LICENSE
Copyright (C) 2007-2018 Free Software Foundation, Inc.
This program is free software; you can redistribute it and/or modify This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3 of the License, or the Free Software Foundation; either version 3 of the License, or

View file

@ -1235,6 +1235,11 @@ saying "Do not show this again". But if that is too much trouble for
you, why not use one of the alternatives instead? See a list in: you, why not use one of the alternatives instead? See a list in:
B<man parallel_alternatives>. B<man parallel_alternatives>.
As the request for citation is not a legal requirement this is
acceptable under GPLv3 and cleared with Richard M. Stallman
himself. Thus it does not fall under this:
https://www.gnu.org/licenses/gpl-faq.en.html#RequireCitation
=head1 Ideas for new design =head1 Ideas for new design

View file

@ -3139,7 +3139,7 @@ https://my.fsf.org/donate/
=back =back
(C) 2013-2018 Ole Tange, GPLv3 (C) 2013-2019 Ole Tange, FDLv1.3 (See fdl.txt)
=cut =cut

View file

@ -60,13 +60,11 @@ GNU B<parcat> is part of GNU B<parallel>. Report bugs to
=head1 AUTHOR =head1 AUTHOR
Copyright (C) 2016,2017,2018 Ole Tange, http://ole.tange.dk and Free Copyright (C) 2016-2019 Ole Tange, http://ole.tange.dk and Free
Software Foundation, Inc. Software Foundation, Inc.
=head1 LICENSE =head1 LICENSE
Copyright (C) 2007,2008,2009,2010,2011 Free Software Foundation, Inc.
This program is free software; you can redistribute it and/or modify This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3 of the License, or the Free Software Foundation; either version 3 of the License, or

View file

@ -1,7 +1,7 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Copyright (C) 2016,2017,2018 # Copyright (C) 2016-2019 Ole Tange, http://ole.tange.dk and
# Ole Tange and Free Software Foundation, Inc. # Free Software Foundation, Inc.
# #
# This program is free software; you can redistribute it and/or modify # This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by # it under the terms of the GNU General Public License as published by

View file

@ -182,14 +182,12 @@ Copyright (C) 2007-10-18 Ole Tange, http://ole.tange.dk
Copyright (C) 2008-2010 Ole Tange, http://ole.tange.dk Copyright (C) 2008-2010 Ole Tange, http://ole.tange.dk
Copyright (C) 2010-2018 Ole Tange, Copyright (C) 2010-2019 Ole Tange, http://ole.tange.dk and Free
http://ole.tange.dk and Free Software Foundation, Inc. Software Foundation, Inc.
=head1 LICENSE =head1 LICENSE
Copyright (C) 2016 Ole Tange and Free Software Foundation, Inc.
This program is free software; you can redistribute it and/or modify This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3 of the License, or the Free Software Foundation; either version 3 of the License, or

View file

@ -244,14 +244,12 @@ Report bugs to <bug-parallel@gnu.org>.
=head1 AUTHOR =head1 AUTHOR
Copyright (C) 2010-2018 Ole Tange, Copyright (C) 2010-2019 Ole Tange, http://ole.tange.dk and Free
http://ole.tange.dk and Free Software Foundation, Inc. Software Foundation, Inc.
=head1 LICENSE =head1 LICENSE
Copyright (C) 2010,2011,2012,2013 Free Software Foundation, Inc.
This program is free software; you can redistribute it and/or modify This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3 of the License, or the Free Software Foundation; either version 3 of the License, or

14
src/sql
View file

@ -339,15 +339,13 @@ When using GNU B<sql> for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different
Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32. Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
Copyright (C) 2008,2009,2010 Ole Tange http://ole.tange.dk Copyright (C) 2008-2010 Ole Tange http://ole.tange.dk
Copyright (C) 2010,2011,2012,2013,2014,2015,2016,2017,2018 Ole Tange, Copyright (C) 2010-2019 Ole Tange, http://ole.tange.dk and Free
http://ole.tange.dk and Free Software Foundation, Inc. Software Foundation, Inc.
=head1 LICENSE =head1 LICENSE
Copyright (C) 2007,2008,2009,2010,2011 Free Software Foundation, Inc.
This program is free software; you can redistribute it and/or modify This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3 of the License, or the Free Software Foundation; either version 3 of the License, or
@ -576,11 +574,7 @@ $Global::Initfile && unlink $Global::Initfile;
exit ($err); exit ($err);
sub parse_options { sub parse_options {
<<<<<<< HEAD $Global::version = 20181223;
$Global::version = 20180723;
=======
$Global::version = 20181124;
>>>>>>> 8286e771ec57e60ee17aee87741fe96790880db7
$Global::progname = 'sql'; $Global::progname = 'sql';
# This must be done first as this may exec myself # This must be done first as this may exec myself

View file

@ -664,7 +664,9 @@ par_X_eta_div_zero() {
# We do not care how long it took # We do not care how long it took
seq 2 | stdout parallel -X --eta echo | seq 2 | stdout parallel -X --eta echo |
grep -E -v 'ETA:.*AVG' | grep -E -v 'ETA:.*AVG' |
perl -pe 's/\d+/0/g' perl -pe 's/\d+/0/g' |
perl -pe 's/Comp.* to complete//' |
perl -ne '/../ and print'
} }
par_parcat_args_stdin() { par_parcat_args_stdin() {
@ -777,7 +779,12 @@ par_slow_pipe_regexp() {
echo "### bug #53718: --pipe --regexp -N blocks" echo "### bug #53718: --pipe --regexp -N blocks"
echo This should take a few ms, but took more than 2 hours echo This should take a few ms, but took more than 2 hours
seq 54000 80000 | seq 54000 80000 |
timeout -k 1 60 parallel -N1000 --regexp --pipe --recstart 4 --recend 5 -k wc parallel -N1000 --regexp --pipe --recstart 4 --recend 5 -k wc
echo "### These should give same output"
seq 54000 80000 |
parallel -N1000 --regexp --pipe --recstart 4 --recend 5 -k cat |
md5sum
seq 54000 80000 | md5sum
} }
par_results() { par_results() {
@ -865,6 +872,15 @@ par_space_envvar() {
export PARALLEL=" -v" && parallel echo ::: 'space in envvar OK' export PARALLEL=" -v" && parallel echo ::: 'space in envvar OK'
} }
par_pipe_N1_regexp() {
echo 'bug #55131: --regexp --recstart hangs'
echo "These should give the same"
printf 'begin\n%send\n' '' a b c |
parallel -kN1 --recstart 'begin\n' --pipe --regexp echo JOB{#}\;cat\;echo END
printf 'begin\n%send\n' '' a b c |
parallel -kN1 --recstart 'begin\n' --pipe echo JOB{#}\;cat\;echo END
}
export -f $(compgen -A function | grep par_) export -f $(compgen -A function | grep par_)
compgen -A function | grep par_ | LC_ALL=C sort | compgen -A function | grep par_ | LC_ALL=C sort |
parallel -j6 --tag -k --joblog +/tmp/jl-`basename $0` '{} 2>&1' parallel --timeout 20 -j6 --tag -k --joblog +/tmp/jl-`basename $0` '{} 2>&1'

File diff suppressed because it is too large Load diff

View file

@ -147,11 +147,6 @@ par_no_table() {
export -f $(compgen -A function | egrep 'p_|par_') export -f $(compgen -A function | egrep 'p_|par_')
# Tested that -j0 in parallel is fastest (up to 15 jobs) # Tested that -j0 in parallel is fastest (up to 15 jobs)
# more than 3 jobs: sqlite locks # more than 3 jobs: sqlite locks
<<<<<<< HEAD
compgen -A function | grep par_ | sort |
stdout parallel --timeout 250 -vj40% -k --tag --joblog /tmp/jl-`basename $0` p_wrapper \
=======
compgen -A function | grep par_ | LC_ALL=C sort | compgen -A function | grep par_ | LC_ALL=C sort |
stdout parallel --timeout 200 -vj3 -k --tag --joblog /tmp/jl-`basename $0` p_wrapper \ stdout parallel --timeout 200 -vj3 -k --tag --joblog /tmp/jl-`basename $0` p_wrapper \
>>>>>>> 8286e771ec57e60ee17aee87741fe96790880db7
:::: - ::: \$MYSQL \$PG \$SQLITE :::: - ::: \$MYSQL \$PG \$SQLITE

View file

@ -791,6 +791,7 @@ par_fish_underscore() {
echo "### Testing of --env _" echo "### Testing of --env _"
# . `which env_parallel.fish`; # . `which env_parallel.fish`;
true > ~/.parallel/ignored_vars;
alias not_copied_alias="echo BAD" alias not_copied_alias="echo BAD"
function not_copied_func function not_copied_func
@ -818,7 +819,7 @@ par_fish_underscore() {
env_parallel --env _ -S server echo \$not_copied_var ::: error=OK; env_parallel --env _ -S server echo \$not_copied_var ::: error=OK;
env_parallel --env _ -S server echo \$not_copied_array ::: error=OK; env_parallel --env _ -S server echo \$not_copied_array ::: error=OK;
echo myvar >> ~/.parallel/ignored_vars; echo myvar > ~/.parallel/ignored_vars;
env_parallel --env _ myfunc ::: work; env_parallel --env _ myfunc ::: work;
env_parallel --env _ -S server myfunc ::: work; env_parallel --env _ -S server myfunc ::: work;
echo myarray >> ~/.parallel/ignored_vars; echo myarray >> ~/.parallel/ignored_vars;
@ -1213,13 +1214,13 @@ _EOF
par_fish_funky() { par_fish_funky() {
myscript=$(cat <<'_EOF' myscript=$(cat <<'_EOF'
echo "Fish is broken"
env_parallel --session env_parallel --session
set myvar "myvar works" set myvar "myvar works"
setenv myenvvar "myenvvar works" setenv myenvvar "myenvvar works"
set funky (perl -e "print pack \"c*\", 1..255") set funky (perl -e "print pack \"c*\", 1..255")
setenv funkyenv (perl -e "print pack \"c*\", 1..255") # 10 and 30 cause problems
setenv funkyenv (perl -e "print pack \"c*\", 1..9,11..29,31..255")
set myarray "" array_val2 3 "" 5 " space 6 " set myarray "" array_val2 3 "" 5 " space 6 "
@ -1511,7 +1512,6 @@ _EOF
par_fish_env_parallel() { par_fish_env_parallel() {
myscript=$(cat <<'_EOF' myscript=$(cat <<'_EOF'
echo "Fish is broken"
echo 'bug #50435: Remote fifo broke in 20150522' echo 'bug #50435: Remote fifo broke in 20150522'
# Due to $PARALLEL_TMP being transferred # Due to $PARALLEL_TMP being transferred
set OK OK set OK OK

View file

@ -81,6 +81,7 @@ perl -ne '$/="\n\n"; /^Output/../^[^O]\S/ and next; /^ / and print;' ../../src/
s{rsync: link_stat ".*/home/parallel/input_file.out" .*\n}{}; s{rsync: link_stat ".*/home/parallel/input_file.out" .*\n}{};
s{rsync error: some files/attrs were not transferred .*\n}{}; s{rsync error: some files/attrs were not transferred .*\n}{};
s{.* GtkDialog .*\n}{}; s{.* GtkDialog .*\n}{};
s{tried 1}{};
' | ' |
perl -ne '/GTK2_RC_FILES/ and next; perl -ne '/GTK2_RC_FILES/ and next;
/GTK_RC_FILES/ and next; /GTK_RC_FILES/ and next;

View file

@ -1251,7 +1251,6 @@ par_PARALLEL_ENV OK as file
par_PARALLEL_ENV ### PARALLEL_ENV as fifo par_PARALLEL_ENV ### PARALLEL_ENV as fifo
par_PARALLEL_ENV OK as fifo par_PARALLEL_ENV OK as fifo
par_X_eta_div_zero ### bug #34422: parallel -X --eta crashes with div by zero par_X_eta_div_zero ### bug #34422: parallel -X --eta crashes with div by zero
par_X_eta_div_zero 0
par_X_eta_div_zero Computers / CPU cores / Max jobs to run par_X_eta_div_zero Computers / CPU cores / Max jobs to run
par_X_eta_div_zero 0:local / 0 / 0 par_X_eta_div_zero 0:local / 0 / 0
par_append_joblog ### can you append to a joblog using + par_append_joblog ### can you append to a joblog using +
@ -1391,6 +1390,40 @@ par_parcat_rm bug #51691: parcat --rm remove fifo when opened
par_parcat_rm OK1 par_parcat_rm OK1
par_parcat_rm OK file removed par_parcat_rm OK file removed
par_perlexpr_with_newline Perl expression spanning 2 lines par_perlexpr_with_newline Perl expression spanning 2 lines
par_pipe_N1_regexp bug #55131: --regexp --recstart hangs
par_pipe_N1_regexp These should give the same
par_pipe_N1_regexp JOB1
par_pipe_N1_regexp begin
par_pipe_N1_regexp end
par_pipe_N1_regexp END
par_pipe_N1_regexp JOB2
par_pipe_N1_regexp begin
par_pipe_N1_regexp aend
par_pipe_N1_regexp END
par_pipe_N1_regexp JOB3
par_pipe_N1_regexp begin
par_pipe_N1_regexp bend
par_pipe_N1_regexp END
par_pipe_N1_regexp JOB4
par_pipe_N1_regexp begin
par_pipe_N1_regexp cend
par_pipe_N1_regexp END
par_pipe_N1_regexp JOB1
par_pipe_N1_regexp begin
par_pipe_N1_regexp end
par_pipe_N1_regexp END
par_pipe_N1_regexp JOB2
par_pipe_N1_regexp begin
par_pipe_N1_regexp aend
par_pipe_N1_regexp END
par_pipe_N1_regexp JOB3
par_pipe_N1_regexp begin
par_pipe_N1_regexp bend
par_pipe_N1_regexp END
par_pipe_N1_regexp JOB4
par_pipe_N1_regexp begin
par_pipe_N1_regexp cend
par_pipe_N1_regexp END
par_pipe_no_command ### --pipe without command par_pipe_no_command ### --pipe without command
par_pipe_no_command parallel: Error: --pipe/--pipepart must have a command to pipe into (e.g. 'cat'). par_pipe_no_command parallel: Error: --pipe/--pipepart must have a command to pipe into (e.g. 'cat').
par_pipe_recend bug #54328: --pipe --recend blocks par_pipe_recend bug #54328: --pipe --recend blocks
@ -1481,8 +1514,11 @@ par_retries_replacement_string 33
par_retries_replacement_string 33 par_retries_replacement_string 33
par_slow_pipe_regexp ### bug #53718: --pipe --regexp -N blocks par_slow_pipe_regexp ### bug #53718: --pipe --regexp -N blocks
par_slow_pipe_regexp This should take a few ms, but took more than 2 hours par_slow_pipe_regexp This should take a few ms, but took more than 2 hours
par_slow_pipe_regexp 979 980 5875 par_slow_pipe_regexp 980 981 5881
par_slow_pipe_regexp 25022 25022 150131 par_slow_pipe_regexp 25021 25021 150125
par_slow_pipe_regexp ### These should give same output
par_slow_pipe_regexp 6e72d7f86f6a423b9a7fa97630587815 -
par_slow_pipe_regexp 6e72d7f86f6a423b9a7fa97630587815 -
par_space_envvar ### bug: --gnu was ignored if env var started with space: PARALLEL=' --gnu' par_space_envvar ### bug: --gnu was ignored if env var started with space: PARALLEL=' --gnu'
par_space_envvar echo 'space in envvar OK' par_space_envvar echo 'space in envvar OK'
par_space_envvar space in envvar OK par_space_envvar space in envvar OK

View file

@ -619,6 +619,12 @@ par_test_XI_mI a7 b1 2 3 4 5 6 7
par_test_XI_mI a8 b1 2 3 4 5 6 7 8 par_test_XI_mI a8 b1 2 3 4 5 6 7 8
par_test_XI_mI a9 b1 2 3 4 5 6 7 8 9 par_test_XI_mI a9 b1 2 3 4 5 6 7 8 9
par_test_XI_mI a10 b1 2 3 4 5 6 7 8 9 10 par_test_XI_mI a10 b1 2 3 4 5 6 7 8 9 10
par_test_cpu_detection 2-8-8-8 2 8 8 8
par_test_cpu_detection 1-4-8-4 1 4 8 4
par_test_cpu_detection 1-2-4-2 1 2 4 2
par_test_cpu_detection 1-2-2-2 1 2 2 2
par_test_cpu_detection 2-24-48-24 2 24 48 24
par_test_cpu_detection 1-2-2-2 1 2 2 2
par_too_long_line_X bug #54869: Long lines break par_too_long_line_X bug #54869: Long lines break
par_too_long_line_X 1 29302 131012 par_too_long_line_X 1 29302 131012
par_too_long_line_X 1 12698 63490 par_too_long_line_X 1 12698 63490

View file

@ -4,7 +4,7 @@ par_pipe_retries 165668 165668 1048571
par_pipe_retries localhost-: par_pipe_retries localhost-:
par_pipe_retries 134362 134362 940534 par_pipe_retries 134362 134362 940534
par_pipe_retries localhost-: par_pipe_retries localhost-:
par_pipe_retries parallel: Error: --retries cannot be combined with --round-robin. par_pipe_retries parallel: Error: --retries cannot be combined with --roundrobin.
par_lsh ### --ssh lsh par_lsh ### --ssh lsh
par_lsh parallel: Warning: Could not figure out number of cpus on lo (). Using 1. par_lsh parallel: Warning: Could not figure out number of cpus on lo (). Using 1.
par_lsh OK par_lsh OK

View file

@ -61,7 +61,7 @@ echo '### Check that 4 processes are really used'
echo '### --version must have higher priority than retired options' echo '### --version must have higher priority than retired options'
### --version must have higher priority than retired options ### --version must have higher priority than retired options
$NICEPAR --version -g -Y -U -W -T | tail $NICEPAR --version -g -Y -U -W -T | tail
GNU parallel 20181124 GNU parallel 20181202
Copyright (C) 2007-2018 Ole Tange and Free Software Foundation, Inc. Copyright (C) 2007-2018 Ole Tange and Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it. This is free software: you are free to change and redistribute it.

View file

@ -654,9 +654,9 @@ echo X; exit X
completed 1 completed 1
completed 2 completed 2
completed 0 completed 0
tried 1
tried 0 tried 0
tried 1
show_signals() { show_signals() {
perl -e 'for(keys %SIG) { perl -e 'for(keys %SIG) {
$SIG{$_} = eval "sub { print \"Got $_\\n\"; }"; $SIG{$_} = eval "sub { print \"Got $_\\n\"; }";
@ -1440,8 +1440,8 @@ Start 4
Slow started Slow started
Forced running after 1 sec Forced running after 1 sec
Slow ended Slow ended
Give up after 2 secs
parallel: Warning: Semaphore timed out. Stealing the semaphore. parallel: Warning: Semaphore timed out. Stealing the semaphore.
parallel: Warning: Semaphore timed out. Exiting.
parallel --help parallel --help
Usage: Usage: