tange/parallel
1
0
Fork 0
mirror of https://git.savannah.gnu.org/git/parallel.git synced 2024-11-22 05:57:54 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Released as 20210122 ('CapitolRiots')

Browse source
This commit is contained in:
Ole Tange 2021-01-21 21:01:30 +01:00
parent e81a0eba05
commit 0b44a9ceb2
29 changed files with 364 additions and 294 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

30
NEWS
View file

@ -1,3 +1,33 @@
20210122
* --memsuspend suspends jobs when there is little memory free. This
way you can run jobs in parallel that added up potentially will take
up more RAM than is available.
* $PARALLEL_ARGHOSTGROUPS and the replacement string {agrp} will give
the hostgroup given on the argument when using --hostgroup.
* Handy time functions for {= =}: yyyy_mm_dd_hh_mm_ss()
yyyy_mm_dd_hh_mm() yyyy_mm_dd() yyyymmddhhmmss() yyyymmddhhmm()
yyyymmdd() hash($str)
* Bug fixes and man page updates.
News about GNU Parallel:
* Software Engineering For Scientists
https://canvas.stanford.edu/courses/133091
* Optional Individual Submission 4 Job Handling - GNU Parallel
https://www.youtube.com/watch?v=-tX0bFAYAxM
* 关于gnu parallel：GnuParallel：在集群上并行化脚本，脚本将文件写入主
节点https://www.codenong.com/25172209/
* すぐ忘れてしまう、仕事で使う技
https://qiita.com/hana_shin/items/53c3c78525c9c758ae7c
20201222 20201222
* --pipe engine changed making --pipe alpha quality. * --pipe engine changed making --pipe alpha quality.

24
README
View file

@ -57,11 +57,11 @@ document.
Full installation of GNU Parallel is as simple as: Full installation of GNU Parallel is as simple as:
wget https://ftpmirror.gnu.org/parallel/parallel-20201222.tar.bz2 wget https://ftpmirror.gnu.org/parallel/parallel-20210122.tar.bz2
wget https://ftpmirror.gnu.org/parallel/parallel-20201222.tar.bz2.sig wget https://ftpmirror.gnu.org/parallel/parallel-20210122.tar.bz2.sig
gpg parallel-20201222.tar.bz2.sig gpg parallel-20210122.tar.bz2.sig
bzip2 -dc parallel-20201222.tar.bz2 | tar xvf - bzip2 -dc parallel-20210122.tar.bz2 | tar xvf -
cd parallel-20201222 cd parallel-20210122
./configure && make && sudo make install ./configure && make && sudo make install
@ -70,11 +70,11 @@ Full installation of GNU Parallel is as simple as:
If you are not root you can add ~/bin to your path and install in If you are not root you can add ~/bin to your path and install in
~/bin and ~/share: ~/bin and ~/share:
wget https://ftpmirror.gnu.org/parallel/parallel-20201222.tar.bz2 wget https://ftpmirror.gnu.org/parallel/parallel-20210122.tar.bz2
wget https://ftpmirror.gnu.org/parallel/parallel-20201222.tar.bz2.sig wget https://ftpmirror.gnu.org/parallel/parallel-20210122.tar.bz2.sig
gpg parallel-20201222.tar.bz2.sig gpg parallel-20210122.tar.bz2.sig
bzip2 -dc parallel-20201222.tar.bz2 | tar xvf - bzip2 -dc parallel-20210122.tar.bz2 | tar xvf -
cd parallel-20201222 cd parallel-20210122
./configure --prefix=$HOME && make && make install ./configure --prefix=$HOME && make && make install
Or if your system lacks 'make' you can simply copy src/parallel Or if your system lacks 'make' you can simply copy src/parallel
@ -122,8 +122,8 @@ will love you for it.
When using programs that use GNU Parallel to process data for When using programs that use GNU Parallel to process data for
publication please cite: publication please cite:
Tange, O. (2020, December 22). GNU Parallel 20201222 ('Vaccine'). Tange, O. (2021, January 22). GNU Parallel 20210122 ('Capitol Riots').
Zenodo. https://doi.org/10.5281/zenodo.4381888 Zenodo. https://doi.org/10.5281/zenodo.4454976
Copyright (C) 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, Copyright (C) 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015,
2016, 2017, 2018, 2019, 2020 Ole Tange, http://ole.tange.dk and Free 2016, 2017, 2018, 2019, 2020 Ole Tange, http://ole.tange.dk and Free

20
configure vendored
View file

@ -1,6 +1,6 @@
#! /bin/sh #! /bin/sh
# Guess values for system-dependent variables and create Makefiles. # Guess values for system-dependent variables and create Makefiles.
# Generated by GNU Autoconf 2.69 for parallel 20201222. # Generated by GNU Autoconf 2.69 for parallel 20210122.
# #
# Report bugs to <bug-parallel@gnu.org>. # Report bugs to <bug-parallel@gnu.org>.
# #
@ -579,8 +579,8 @@ MAKEFLAGS=
# Identity of this package. # Identity of this package.
PACKAGE_NAME='parallel' PACKAGE_NAME='parallel'
PACKAGE_TARNAME='parallel' PACKAGE_TARNAME='parallel'
PACKAGE_VERSION='20201222' PACKAGE_VERSION='20210122'
PACKAGE_STRING='parallel 20201222' PACKAGE_STRING='parallel 20210122'
PACKAGE_BUGREPORT='bug-parallel@gnu.org' PACKAGE_BUGREPORT='bug-parallel@gnu.org'
PACKAGE_URL='' PACKAGE_URL=''
@ -1214,7 +1214,7 @@ if test "$ac_init_help" = "long"; then
# Omit some internal or obsolete options to make the list less imposing. # Omit some internal or obsolete options to make the list less imposing.
# This message is too long to be a string in the A/UX 3.1 sh. # This message is too long to be a string in the A/UX 3.1 sh.
cat <<_ACEOF cat <<_ACEOF
\`configure' configures parallel 20201222 to adapt to many kinds of systems. \`configure' configures parallel 20210122 to adapt to many kinds of systems.
Usage: $0 [OPTION]... [VAR=VALUE]... Usage: $0 [OPTION]... [VAR=VALUE]...
@ -1281,7 +1281,7 @@ fi
if test -n "$ac_init_help"; then if test -n "$ac_init_help"; then
case $ac_init_help in case $ac_init_help in
short | recursive ) echo "Configuration of parallel 20201222:";; short | recursive ) echo "Configuration of parallel 20210122:";;
esac esac
cat <<\_ACEOF cat <<\_ACEOF
@ -1357,7 +1357,7 @@ fi
test -n "$ac_init_help" && exit $ac_status test -n "$ac_init_help" && exit $ac_status
if $ac_init_version; then if $ac_init_version; then
cat <<\_ACEOF cat <<\_ACEOF
parallel configure 20201222 parallel configure 20210122
generated by GNU Autoconf 2.69 generated by GNU Autoconf 2.69
Copyright (C) 2012 Free Software Foundation, Inc. Copyright (C) 2012 Free Software Foundation, Inc.
@ -1374,7 +1374,7 @@ cat >config.log <<_ACEOF
This file contains any messages produced by compilers while This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake. running configure, to aid debugging if configure makes a mistake.
It was created by parallel $as_me 20201222, which was It was created by parallel $as_me 20210122, which was
generated by GNU Autoconf 2.69. Invocation command line was generated by GNU Autoconf 2.69. Invocation command line was
$ $0 $@ $ $0 $@
@ -2237,7 +2237,7 @@ fi
# Define the identity of the package. # Define the identity of the package.
PACKAGE='parallel' PACKAGE='parallel'
VERSION='20201222' VERSION='20210122'
cat >>confdefs.h <<_ACEOF cat >>confdefs.h <<_ACEOF
@ -2880,7 +2880,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
# report actual input values of CONFIG_FILES etc. instead of their # report actual input values of CONFIG_FILES etc. instead of their
# values after options handling. # values after options handling.
ac_log=" ac_log="
This file was extended by parallel $as_me 20201222, which was This file was extended by parallel $as_me 20210122, which was
generated by GNU Autoconf 2.69. Invocation command line was generated by GNU Autoconf 2.69. Invocation command line was
CONFIG_FILES = $CONFIG_FILES CONFIG_FILES = $CONFIG_FILES
@ -2942,7 +2942,7 @@ _ACEOF
cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`" ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`"
ac_cs_version="\\ ac_cs_version="\\
parallel config.status 20201222 parallel config.status 20210122
configured by $0, generated by GNU Autoconf 2.69, configured by $0, generated by GNU Autoconf 2.69,
with options \\"\$ac_cs_config\\" with options \\"\$ac_cs_config\\"

2
configure.ac
View file

@ -1,4 +1,4 @@
AC_INIT([parallel], [20201222], [bug-parallel@gnu.org]) AC_INIT([parallel], [20210122], [bug-parallel@gnu.org])
AM_INIT_AUTOMAKE([-Wall -Werror foreign]) AM_INIT_AUTOMAKE([-Wall -Werror foreign])
AC_CONFIG_HEADERS([config.h]) AC_CONFIG_HEADERS([config.h])
AC_CONFIG_FILES([ AC_CONFIG_FILES([

10
doc/haikus
View file

@ -1,5 +1,11 @@
Quote of the month: Quote of the month:
GNU Parallel and ripgrep would be your friend here. Ripgrep is fast, really fast.
-- CENAPT @cenaptech@twitter
GNU Parallel is a priceless timesaver
-- June "Wakalix" @TWakalix@twitter
GNU Parallel is great GNU Parallel is great
-- Newton's Flaming Laser Sword @swordgoesfwoosh@twitter -- Newton's Flaming Laser Sword @swordgoesfwoosh@twitter
@ -88,6 +94,10 @@ https://negfeedback.blogspot.com/2020/05/indispensable-command-line-tools.html
=== Used === === Used ===
I think many people would be surprised to learn that GNU parallel is
"just" a single Perl script.
-- Peter Menzel @ptr_menzel@twitter
GNU Parallel is my single favourite tool for batch processing data GNU Parallel is my single favourite tool for batch processing data
from the command line. from the command line.
-- Jeff Wintersinger @jwintersinger -- Jeff Wintersinger @jwintersinger

21
doc/release_new_version
View file

@ -16,6 +16,8 @@ With the same things that goes in the announce mail
== Testsuite == == Testsuite ==
Start termux on Android
Start: wssh mac ping fi.dk
cd testsuite; make mem; make cd testsuite; make mem; make
== Update version == == Update version ==
@ -203,28 +205,31 @@ Please help spreading GNU Parallel by making a testimonial video like Juan Sierr
It does not have to be as detailed as Juan's. It is perfectly fine if you just say your name, and what field you are using GNU Parallel for. It does not have to be as detailed as Juan's. It is perfectly fine if you just say your name, and what field you are using GNU Parallel for.
Quote of the month: Quote of the month:
<<>>
I think many people would be surprised to learn that GNU parallel is "just" a single Perl script.
-- Peter Menzel @ptr_menzel@twitter
New in this release: New in this release:
* --memsuspend suspends jobs when there is little memory free. This way you can run jobs in parallel that added up potentially will take up more RAM than is available.
* $PARALLEL_ARGHOSTGROUPS and the replacement string {agrp} will give the hostgroup given on the argument when using --hostgroup. * $PARALLEL_ARGHOSTGROUPS and the replacement string {agrp} will give the hostgroup given on the argument when using --hostgroup.
* Handy time functions for {= =}: yyyy_mm_dd_hh_mm_ss() yyyy_mm_dd_hh_mm() yyyy_mm_dd() yyyymmddhhmmss() yyyymmddhhmm() yyyymmdd() hash($str) * Handy time functions for {= =}: yyyy_mm_dd_hh_mm_ss() yyyy_mm_dd_hh_mm() yyyy_mm_dd() yyyymmddhhmmss() yyyymmddhhmm() yyyymmdd() hash($str)
<<>>
* Bug fixes and man page updates. * Bug fixes and man page updates.
News about GNU Parallel: News about GNU Parallel:
https://www.codenong.com/25172209/ * Software Engineering For Scientists https://canvas.stanford.edu/courses/133091
https://qiita.com/hana_shin/items/53c3c78525c9c758ae7c * Optional Individual Submission 4 Job Handling - GNU Parallel https://www.youtube.com/watch?v=-tX0bFAYAxM
https://canvas.stanford.edu/courses/133091 * 关于gnu parallel：GnuParallel：在集群上并行化脚本，脚本将文件写入主节点https://www.codenong.com/25172209/
* すぐ忘れてしまう、仕事で使う技 https://qiita.com/hana_shin/items/53c3c78525c9c758ae7c
https://www.youtube.com/watch?v=-tX0bFAYAxM
<<>> <<>>

12
packager/obs/home:tange/parallel/.osc/_files
View file

@ -1,7 +1,7 @@
<directory name="parallel" rev="294" vrev="1" srcmd5="dfdf794f7ba13a4411521b55c133131b"> <directory name="parallel" rev="295" vrev="1" srcmd5="3d5753d873ec7dd1c5e728f166982dd0">
<entry name="PKGBUILD" md5="17b8210106cf6e27530746de13848c6a" size="936" mtime="1608555894" /> <entry name="PKGBUILD" md5="84008e34fb54d73054e039551aa5a1b1" size="936" mtime="1611258951" />
<entry name="parallel-20201222.tar.bz2" md5="e959152aaecf2eed217fee96a939f073" size="2115063" mtime="1608555895" /> <entry name="parallel-20210122.tar.bz2" md5="6799fa098597b5a32e87f82f8ef4d8ad" size="2129364" mtime="1611258952" />
<entry name="parallel.spec" md5="03891ffdf3417d97649b974131c1cb28" size="4876" mtime="1608555895" /> <entry name="parallel.spec" md5="73c304015393921bb524310ecc3f68da" size="4876" mtime="1611258952" />
<entry name="parallel_20201222.dsc" md5="9ef49ddb84177a25a7a5b2134f5119ca" size="556" mtime="1608555895" /> <entry name="parallel_20210122.dsc" md5="327c5825cefbbf7674e1a668e57ef499" size="556" mtime="1611258953" />
<entry name="parallel_20201222.tar.gz" md5="19b2939db032b974d84703f166cf69c8" size="2309827" mtime="1608555896" /> <entry name="parallel_20210122.tar.gz" md5="f0b8b80919399ef704eb6d00d03f3a68" size="2319784" mtime="1611258953" />
</directory> </directory>

2
packager/obs/home:tange/parallel/.osc/parallel.spec
View file

@ -1,7 +1,7 @@
Summary: Shell tool for executing jobs in parallel Summary: Shell tool for executing jobs in parallel
Name: parallel Name: parallel
Version: 20201222 Version: 20210122
Release: 1.3 Release: 1.3
License: GPL-3.0-or-later License: GPL-3.0-or-later
Group: Productivity/File utilities Group: Productivity/File utilities

2
src/env_parallel.ash
View file

@ -382,7 +382,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20201223 (GNU parallel `parallel --minversion 1`)" echo "parset 20210122 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

2
src/env_parallel.bash
View file

@ -384,7 +384,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20201223 (GNU parallel `parallel --minversion 1`)" echo "parset 20210122 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

2
src/env_parallel.dash
View file

@ -382,7 +382,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20201223 (GNU parallel `parallel --minversion 1`)" echo "parset 20210122 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

2
src/env_parallel.ksh
View file

@ -365,7 +365,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20201223 (GNU parallel `parallel --minversion 1`)" echo "parset 20210122 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

2
src/env_parallel.mksh
View file

@ -368,7 +368,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20201223 (GNU parallel `parallel --minversion 1`)" echo "parset 20210122 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

2
src/env_parallel.sh
View file

@ -382,7 +382,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20201223 (GNU parallel `parallel --minversion 1`)" echo "parset 20210122 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

2
src/env_parallel.zsh
View file

@ -359,7 +359,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20201223 (GNU parallel `parallel --minversion 1`)" echo "parset 20210122 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

2
src/niceload
View file

@ -23,7 +23,7 @@
use strict; use strict;
use Getopt::Long; use Getopt::Long;
$Global::progname="niceload"; $Global::progname="niceload";
$Global::version = 20201223; $Global::version = 20210122;
Getopt::Long::Configure("bundling","require_order"); Getopt::Long::Configure("bundling","require_order");
get_options_from_array(\@ARGV) || die_usage(); get_options_from_array(\@ARGV) || die_usage();
if($opt::version) { if($opt::version) {

26
src/parallel
View file

@ -2151,7 +2151,7 @@ sub check_invalid_option_combinations() {
sub init_globals() { sub init_globals() {
# Defaults: # Defaults:
$Global::version = 20210113; $Global::version = 20210122;
$Global::progname = 'parallel'; $Global::progname = 'parallel';
$::name = "GNU Parallel"; $::name = "GNU Parallel";
$Global::infinity = 2**31; $Global::infinity = 2**31;
@ -4933,8 +4933,8 @@ sub usage() {
"If you use programs that use GNU Parallel to process data for an article in a", "If you use programs that use GNU Parallel to process data for an article in a",
"scientific publication, please cite:", "scientific publication, please cite:",
"", "",
" Tange, O. (2020, December 22). GNU Parallel 20201222 ('Vaccine').", " Tange, O. (2021, January 22). GNU Parallel 20210122 ('Capitol Riots').",
" Zenodo. https://doi.org/10.5281/zenodo.4381888", " Zenodo. https://doi.org/10.5281/zenodo.4454976",
"", "",
# Before changing this line, please read # Before changing this line, please read
# https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice # https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice
@ -4964,8 +4964,8 @@ sub citation_notice() {
"If you use programs that use GNU Parallel to process data for an article in a", "If you use programs that use GNU Parallel to process data for an article in a",
"scientific publication, please cite:", "scientific publication, please cite:",
"", "",
" Tange, O. (2020, December 22). GNU Parallel 20201222 ('Vaccine').", " Tange, O. (2021, January 22). GNU Parallel 20210122 ('Capitol Riots').",
" Zenodo. https://doi.org/10.5281/zenodo.4381888", " Zenodo. https://doi.org/10.5281/zenodo.4454976",
"", "",
# Before changing this line, please read # Before changing this line, please read
# https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and # https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and
@ -5078,20 +5078,20 @@ sub citation() {
"If you use programs that use GNU Parallel to process data for an article in a", "If you use programs that use GNU Parallel to process data for an article in a",
"scientific publication, please cite:", "scientific publication, please cite:",
"", "",
"\@software{tange_2020_4381888,", "\@software{tange_2021_4454976,",
" author = {Tange, Ole},", " author = {Tange, Ole},",
" title = {GNU Parallel 20201222 ('Vaccine')},", " title = {GNU Parallel 20210122 ('Capitol Riots')},",
" month = Dec,", " month = Jan,",
" year = 2020,", " year = 2020,",
" note = {{GNU Parallel is a general parallelizer to run", " note = {{GNU Parallel is a general parallelizer to run",
" multiple serial command line programs in parallel", " multiple serial command line programs in parallel",
" without changing them.}},", " without changing them.}},",
" publisher = {Zenodo},", " publisher = {Zenodo},",
" doi = {10.5281/zenodo.4381888},", " doi = {10.5281/zenodo.4454976},",
" url = {https://doi.org/10.5281/zenodo.4381888}", " url = {https://doi.org/10.5281/zenodo.4454976}",
"}", "}",
"", "",
"(Feel free to use \\nocite{tange_2020_4381888})", "(Feel free to use \\nocite{tange_2021_4454976})",
"", "",
# Before changing this line, please read # Before changing this line, please read
# https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and # https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and
@ -9280,12 +9280,12 @@ sub sshlogin_wrap($) {
my @bashfunc = map { my @bashfunc = map {
my $v=$_; my $v=$_;
s/BASH_FUNC_(.*)(\(\)|%%)/$1/; s/BASH_FUNC_(.*)(\(\)|%%)/$1/;
"$_$ENV{$v};export -f $_ >/dev/null;" } @bash_functions; "$_$ENV{$v};\nexport -f $_ 2> /dev/null;\n" } @bash_functions;
# eval $bashfuncset will set $bashfunc # eval $bashfuncset will set $bashfunc
my $bashfuncset; my $bashfuncset;
if(@bashfunc) { if(@bashfunc) {
# Functions are not supported for all shells # Functions are not supported for all shells
if($Global::shell !~ m:(bash|rbash|zsh|rzsh|dash|ksh):) { if($Global::shell !~ m:(^|/)(ash|bash|rbash|zsh|rzsh|dash|ksh):) {
::warning("Shell functions may not be supported in $Global::shell."); ::warning("Shell functions may not be supported in $Global::shell.");
} }
$bashfuncset = $bashfuncset =

72
src/parallel.pod
View file

@ -467,7 +467,7 @@ contains B<a b c>. B<-a foo> B<-a bar> will result in the combinations
(1,a) (1,b) (1,c) (2,a) (2,b) (2,c). This is useful for replacing (1,a) (1,b) (1,c) (2,a) (2,b) (2,c). This is useful for replacing
nested for-loops. nested for-loops.
See also B<--link> and B<{>I<n>B<}>. See also: B<--link> and B<{>I<n>B<}>.
=item B<--arg-file-sep> I<sep-str> =item B<--arg-file-sep> I<sep-str>
@ -549,7 +549,7 @@ I<bincol> is small (<10), slower if it is big (>100).
B<--bin> requires B<--pipe> and a fixed numeric value for B<--jobs>. B<--bin> requires B<--pipe> and a fixed numeric value for B<--jobs>.
See also B<--shard>, B<--group-by>, B<--roundrobin>. See also: B<--shard>, B<--group-by>, B<--roundrobin>.
=item B<--bg> =item B<--bg>
@ -645,7 +645,7 @@ you can do: B<parallel --pipe --cat wc {}>.
Implies B<--pipe> unless B<--pipepart> is used. Implies B<--pipe> unless B<--pipepart> is used.
See also B<--fifo>. See also: B<--fifo>.
=item B<--cleanup> =item B<--cleanup>
@ -846,7 +846,7 @@ foreground (opposite B<--bg>), and wait for completion of the command
before exiting. before exiting.
See also B<--bg>, B<man sem>. See also: B<--bg>, B<man sem>.
=item B<--fifo> =item B<--fifo>
@ -860,7 +860,7 @@ Beware: If data is not read from the fifo, the job will block forever.
Implies B<--pipe> unless B<--pipepart> is used. Implies B<--pipe> unless B<--pipepart> is used.
See also B<--cat>. See also: B<--cat>.
=item B<--filter-hosts> =item B<--filter-hosts>
@ -971,7 +971,7 @@ UserID when grouping:
cat table.csv | parallel --pipe --colsep , --header : \ cat table.csv | parallel --pipe --colsep , --header : \
--group-by 'UserID s/\D//g' -kN1 wc --group-by 'UserID s/\D//g' -kN1 wc
See also B<--shard>, B<--roundrobin>. See also: B<--shard>, B<--roundrobin>.
=item B<--help> =item B<--help>
@ -1151,7 +1151,7 @@ If the host is long, you can use B<column -t> to pretty print it:
cat joblog | column -t cat joblog | column -t
See also B<--resume> B<--resume-failed>. See also: B<--resume> B<--resume-failed>.
=item B<--jobs> I<N> =item B<--jobs> I<N>
@ -1427,6 +1427,34 @@ to be run later.
B<--retries> must be set to determine how many times GNU B<parallel> B<--retries> must be set to determine how many times GNU B<parallel>
should retry a given job. should retry a given job.
See also: B<--memsuspend>
=item B<--memsuspend> I<size> (alpha testing)
Suspend jobs when there is less than 2 * I<size> memory free. The
I<size> can be postfixed with K, M, G, T, P, k, m, g, t, or p which
would multiply the size with 1024, 1048576, 1073741824, 1099511627776,
1125899906842624, 1000, 1000000, 1000000000, 1000000000000, or
1000000000000000, respectively.
If the available memory falls below 2 * I<size>, GNU B<parallel>
will suspend some of the running jobs. If the available memory falls
below I<size>, only one job will be running.
If a single job takes up at most I<size> RAM, all jobs will complete
without running out of memory. If you have swap available, you can
usually lower I<size> to around half the size of a single job - with
the slight risk of swapping a little.
Jobs will be resumed when more RAM is available - typically when the
oldest job completes.
B<--memsuspend> only works on local jobs because there is no obvious
way to suspend remote jobs.
See also: B<--memfree>
=item B<--minversion> I<version> =item B<--minversion> I<version>
@ -1629,7 +1657,7 @@ everything at startup.
By sending GNU B<parallel> SIGUSR2 you can toggle turning on/off By sending GNU B<parallel> SIGUSR2 you can toggle turning on/off
B<--progress> on a running GNU B<parallel> process. B<--progress> on a running GNU B<parallel> process.
See also B<--eta> and B<--bar>. See also: B<--eta> and B<--bar>.
=item B<--max-line-length-allowed> =item B<--max-line-length-allowed>
@ -1767,7 +1795,7 @@ B<--memfree> may give better results, so try using that first.
Record current environment variables in ~/.parallel/ignored_vars. This Record current environment variables in ~/.parallel/ignored_vars. This
is useful before using B<--env _>. is useful before using B<--env _>.
See also B<--env>, B<--session>. See also: B<--env>, B<--session>.
=item B<--recstart> I<startstring> =item B<--recstart> I<startstring>
@ -1933,7 +1961,7 @@ will generate the files:
my_foo/stderr my_foo/stderr
my_foo/stdout my_foo/stdout
See also B<--files>, B<--tag>, B<--header>, B<--joblog>. See also: B<--files>, B<--tag>, B<--header>, B<--joblog>.
=item B<--resume> =item B<--resume>
@ -1945,7 +1973,7 @@ sequence numbers in B<--joblog> then the input, the command, and
B<--joblog> all have to remain unchanged; otherwise GNU B<parallel> B<--joblog> all have to remain unchanged; otherwise GNU B<parallel>
may run wrong commands. may run wrong commands.
See also B<--joblog>, B<--results>, B<--resume-failed>, B<--retries>. See also: B<--joblog>, B<--results>, B<--resume-failed>, B<--retries>.
=item B<--resume-failed> =item B<--resume-failed>
@ -1958,7 +1986,7 @@ numbers in B<--joblog> then the input, the command, and B<--joblog>
all have to remain unchanged; otherwise GNU B<parallel> may run wrong all have to remain unchanged; otherwise GNU B<parallel> may run wrong
commands. commands.
See also B<--joblog>, B<--resume>, B<--retry-failed>, B<--retries>. See also: B<--joblog>, B<--resume>, B<--retry-failed>, B<--retries>.
=item B<--retry-failed> =item B<--retry-failed>
@ -2028,7 +2056,7 @@ line:
6 [...] 2 0 echo 5;sleep .5; exit 2 6 [...] 2 0 echo 5;sleep .5; exit 2
4 [...] 1 0 echo 7;sleep .7; exit 1 4 [...] 1 0 echo 7;sleep .7; exit 1
See also B<--joblog>, B<--resume>, B<--resume-failed>, B<--retries>. See also: B<--joblog>, B<--resume>, B<--resume-failed>, B<--retries>.
=item B<--retries> I<n> =item B<--retries> I<n>
@ -2095,7 +2123,7 @@ impossible to track which input block corresponds to which output.
B<--roundrobin> implies B<--pipe>, except if B<--pipepart> is given. B<--roundrobin> implies B<--pipe>, except if B<--pipepart> is given.
See also B<--group-by>, B<--shard>. See also: B<--group-by>, B<--shard>.
=item B<--rpl> 'I<tag> I<perl expression>' =item B<--rpl> 'I<tag> I<perl expression>'
@ -2220,7 +2248,7 @@ Used with B<--fg>, B<--wait>, and B<--semaphorename>.
The command B<sem> is an alias for B<parallel --semaphore>. The command B<sem> is an alias for B<parallel --semaphore>.
See also B<man sem>. See also: B<man sem>.
=item B<--semaphorename> I<name> =item B<--semaphorename> I<name>
@ -2238,7 +2266,7 @@ The semaphore is stored in ~/.parallel/semaphores/
Implies B<--semaphore>. Implies B<--semaphore>.
See also B<man sem>. See also: B<man sem>.
=item B<--semaphoretimeout> I<secs> =item B<--semaphoretimeout> I<secs>
@ -2251,7 +2279,7 @@ If I<secs> < 0: If the semaphore is not released within I<secs> seconds, exit.
Implies B<--semaphore>. Implies B<--semaphore>.
See also B<man sem>. See also: B<man sem>.
=item B<--seqreplace> I<replace-str> =item B<--seqreplace> I<replace-str>
@ -2268,7 +2296,7 @@ variables with names in B<$PARALLEL_IGNORED_NAMES> will not be copied.
Only supported in B<Ash, Bash, Dash, Ksh, Sh, and Zsh>. Only supported in B<Ash, Bash, Dash, Ksh, Sh, and Zsh>.
See also B<--env>, B<--record-env>. See also: B<--env>, B<--record-env>.
=item B<--shard> I<shardexpr> =item B<--shard> I<shardexpr>
@ -2290,7 +2318,7 @@ I<shardcol> is small (<10), slower if it is big (>100).
B<--shard> requires B<--pipe> and a fixed numeric value for B<--jobs>. B<--shard> requires B<--pipe> and a fixed numeric value for B<--jobs>.
See also B<--bin>, B<--group-by>, B<--roundrobin>. See also: B<--bin>, B<--group-by>, B<--roundrobin>.
=item B<--shebang> =item B<--shebang>
@ -2695,7 +2723,7 @@ B<--timeout 1d3.5h16.6m4s>.
Print the job to be run on stderr (standard error). Print the job to be run on stderr (standard error).
See also B<-v>, B<-p>. See also: B<-v>, B<-p>.
=item B<--transfer> =item B<--transfer>
@ -2900,7 +2928,7 @@ Wait for all commands to complete.
Used with B<--semaphore> or B<--sqlmaster>. Used with B<--semaphore> or B<--sqlmaster>.
See also B<man sem>. See also: B<man sem>.
=item B<-X> =item B<-X>
@ -2920,7 +2948,7 @@ unexpected results if B<{}> is used as part of a word.
Support for B<-X> with B<--sshlogin> is limited and may fail. Support for B<-X> with B<--sshlogin> is limited and may fail.
See also B<-m>. See also: B<-m>.
=item B<--exit> =item B<--exit>

6
src/parallel_design.pod
View file

@ -63,9 +63,9 @@ around 100 machines - depending on the duration of each job.
GNU B<parallel> busy waits. This is because the reason why a job is GNU B<parallel> busy waits. This is because the reason why a job is
not started may be due to load average (when using B<--load>), and not started may be due to load average (when using B<--load>), and
thus it will not make sense to wait for a job to finish. Instead the thus it will not make sense to just wait for a job to finish. Instead
load average must be checked again. Load average is not the only the load average must be rechecked regularly. Load average is not the
reason: B<--timeout> has a similar problem. only reason: B<--timeout> has a similar problem.
To not burn up too much CPU GNU B<parallel> sleeps exponentially To not burn up too much CPU GNU B<parallel> sleeps exponentially
longer and longer if nothing happens, maxing out at 1 second. longer and longer if nothing happens, maxing out at 1 second.

2
src/parsort
View file

@ -118,7 +118,7 @@ GetOptions(
"help" => \$opt::dummy, "help" => \$opt::dummy,
) || exit(255); ) || exit(255);
$Global::progname = ($0 =~ m:(^|/)([^/]+)$:)[1]; $Global::progname = ($0 =~ m:(^|/)([^/]+)$:)[1];
$Global::version = 20210102; $Global::version = 20210122;
if($opt::version) { version(); exit 0; } if($opt::version) { version(); exit 0; }
@Global::sortoptions = @Global::sortoptions =
shell_quote(@ARGV_before[0..($#ARGV_before-$#ARGV-1)]); shell_quote(@ARGV_before[0..($#ARGV_before-$#ARGV-1)]);

2
src/sql
View file

@ -574,7 +574,7 @@ $Global::Initfile && unlink $Global::Initfile;
exit ($err); exit ($err);
sub parse_options { sub parse_options {
$Global::version = 20201223; $Global::version = 20210122;
$Global::progname = 'sql'; $Global::progname = 'sql';
# This must be done first as this may exec myself # This must be done first as this may exec myself

2
testsuite/tests-to-run/parallel-local-0.3s.sh
View file

@ -952,7 +952,7 @@ par_hash_and_time_functions() {
yyyymmdd()) =}' ::: 1 | yyyymmdd()) =}' ::: 1 |
perl -pe 's/\d/9/g' perl -pe 's/\d/9/g'
parallel echo '{= $_=hash($_) =}' ::: 1 | parallel echo '{= $_=hash($_) =}' ::: 1 |
perl -pe 's/[a-f0-9]/X/g' perl -pe 's/[a-f0-9]+/X/g'
} }
export -f $(compgen -A function | grep par_) export -f $(compgen -A function | grep par_)

10
testsuite/tests-to-run/parallel-local-1s.sh
View file

@ -921,16 +921,6 @@ par_sql_colsep() {
rm /tmp/parallel-sql-colsep-$$ rm /tmp/parallel-sql-colsep-$$
} }
par_sql_CSV() {
echo '### CSV write to the right place'
rm -rf /tmp/parallel-CSV
mkdir /tmp/parallel-CSV
parallel --sqlandworker csv:///%2Ftmp%2Fparallel-CSV/OK echo ::: 'ran OK'
ls /tmp/parallel-CSV
stdout parallel --sqlandworker csv:///%2Fmust%2Ffail/fail echo ::: 1 |
perl -pe 's/\d/0/g'
}
export -f $(compgen -A function | grep par_) export -f $(compgen -A function | grep par_)
compgen -A function | grep par_ | LC_ALL=C sort | compgen -A function | grep par_ | LC_ALL=C sort |
parallel --timeout 1000% -j6 --tag -k --joblog /tmp/jl-`basename $0` '{} 2>&1' | parallel --timeout 1000% -j6 --tag -k --joblog /tmp/jl-`basename $0` '{} 2>&1' |

10
testsuite/tests-to-run/parallel-local-race02.sh
View file

@ -29,6 +29,16 @@
echo 1=OK $?' | grep -v '\[1\]' | grep -v 'SHA256' echo 1=OK $?' | grep -v '\[1\]' | grep -v 'SHA256'
#} #}
par_sql_CSV() {
echo '### CSV write to the right place'
rm -rf /tmp/parallel-CSV
mkdir /tmp/parallel-CSV
parallel --sqlandworker csv:///%2Ftmp%2Fparallel-CSV/OK echo ::: 'ran OK'
ls /tmp/parallel-CSV
stdout parallel --sqlandworker csv:///%2Fmust%2Ffail/fail echo ::: 1 |
perl -pe 's/\d/0/g'
}
par_hostgroup() { par_hostgroup() {
echo '### --hostgroup force ncpu' echo '### --hostgroup force ncpu'
parallel --delay 0.1 --hgrp -S @g1/1/parallel@lo -S @g2/3/lo whoami\;sleep 0.4{} ::: {1..8} | sort parallel --delay 0.1 --hgrp -S @g1/1/parallel@lo -S @g2/3/lo whoami\;sleep 0.4{} ::: {1..8} | sort

84
testsuite/tests-to-run/parallel-local-ssh4.sh
View file

@ -4,36 +4,51 @@ unset run_test
unset run_once unset run_once
# SSH only allowed to localhost/lo # SSH only allowed to localhost/lo
# --retries if ssh dies
cat <<'EOF' | sed -e s/\$SERVER1/$SERVER1/\;s/\$SERVER2/$SERVER2/ | parallel -vj4 --retries 2 -k --joblog /tmp/jl-`basename $0` -L1 -r par_env_underscore() {
echo '### --env _' echo '### --env _'
fUbAr="OK FUBAR" parallel -S parallel@lo --env _ echo '$fUbAr $DEBEMAIL' ::: test echo ignored_var >> ~/.parallel/ignored_vars
fUbAr="OK FUBAR" parallel -S csh@lo --env _ echo '$fUbAr $DEBEMAIL' ::: test unset $(compgen -A function | grep par_)
ignored_var="should not be copied"
export ignored_var
fUbAr="OK FUBAR" parallel -S parallel@lo --env _ echo '$fUbAr $ignored_var' ::: test
echo 'In csh this may fail with ignored_var: Undefined variable.'
fUbAr="OK FUBAR" parallel -S csh@lo --env _ echo '$fUbAr $ignored_var' ::: test 2>&1
echo '### --env _ with explicit mentioning of normally ignored var $DEBEMAIL' echo '### --env _ with explicit mentioning of normally ignored var $ignored_var'
fUbAr="OK FUBAR" parallel -S parallel@lo --env DEBEMAIL,_ echo '$fUbAr $DEBEMAIL' ::: test ignored_var="should be copied"
fUbAr="OK FUBAR" parallel -S csh@lo --env DEBEMAIL,_ echo '$fUbAr $DEBEMAIL' ::: test fUbAr="OK FUBAR" parallel -S parallel@lo --env ignored_var,_ echo '$fUbAr $ignored_var' ::: test
fUbAr="OK FUBAR" parallel -S csh@lo --env ignored_var,_ echo '$fUbAr $ignored_var' ::: test 2>&1
}
par_warn_when_exporting_func() {
echo 'bug #40137: SHELL not bash: Warning when exporting funcs' echo 'bug #40137: SHELL not bash: Warning when exporting funcs'
. <(printf 'myfunc() {\necho $1\n}'); export -f myfunc; parallel --env myfunc -S lo myfunc ::: no_warning myrun() {
. <(printf 'myfunc() {\necho $1\n}'); export -f myfunc; SHELL=/bin/sh parallel --env myfunc -S lo myfunc ::: warning . <(printf 'myfunc() {\necho Function run: $1\n}')
export -f myfunc
echo "Run function in $1"
PARALLEL_SHELL=$1 parallel --env myfunc -S lo myfunc ::: OK
}
export -f myrun
parallel -k --tag myrun ::: /bin/{sh,bash} /usr/bin/{ash,csh,dash,ksh,tcsh,zsh}
}
par_exporting_in_zsh() {
echo '### zsh' echo '### zsh'
echo 'env in zsh' echo 'env in zsh'
echo 'Normal variable export' echo 'Normal variable export'
export B=\'; export B=\'" Var with quote"
PARALLEL_SHELL=/usr/bin/zsh parallel --env B echo '$B' ::: a PARALLEL_SHELL=/usr/bin/zsh parallel --env B echo '$B' ::: OK
echo 'Function export as variable' echo 'Function export as variable'
export myfuncvar="() { echo myfuncvar \$*; }"; export myfuncvar="() { echo myfuncvar as var \$*; }"
PARALLEL_SHELL=/usr/bin/zsh parallel --env myfuncvar myfuncvar ::: a PARALLEL_SHELL=/usr/bin/zsh parallel --env myfuncvar myfuncvar ::: OK
echo 'Function export as function' echo 'Function export as function'
myfunc() { echo myfunc $*; }; myfunc() { echo myfunc ran $*; }
export -f myfunc; export -f myfunc
PARALLEL_SHELL=/usr/bin/zsh parallel --env myfunc myfunc ::: a PARALLEL_SHELL=/usr/bin/zsh parallel --env myfunc myfunc ::: OK
ssh zsh@lo 'fun="() { echo function from zsh to zsh \$*; }"; ssh zsh@lo 'fun="() { echo function from zsh to zsh \$*; }";
export fun; export fun;
@ -42,7 +57,9 @@ echo 'env in zsh'
ssh zsh@lo 'fun="() { echo function from zsh to bash \$*; }"; ssh zsh@lo 'fun="() { echo function from zsh to bash \$*; }";
export fun; export fun;
parallel -S parallel@lo --env fun fun ::: OK' parallel -S parallel@lo --env fun fun ::: OK'
}
par_bigvar_csh() {
echo '### csh' echo '### csh'
echo "3 big vars run remotely - length(base64) > 1000" echo "3 big vars run remotely - length(base64) > 1000"
stdout ssh csh@lo 'setenv A `seq 200|xargs`; stdout ssh csh@lo 'setenv A `seq 200|xargs`;
@ -55,8 +72,9 @@ echo '### csh2'
setenv B `seq 200 -1 1|xargs`; setenv B `seq 200 -1 1|xargs`;
setenv C `seq 300 -2 1|xargs`; setenv C `seq 300 -2 1|xargs`;
parallel --env A,B,C -k echo \$\{\}\|wc ::: A B C' parallel --env A,B,C -k echo \$\{\}\|wc ::: A B C'
}
par_bigvar_rc() {
echo '### rc' echo '### rc'
echo "3 big vars run remotely - length(base64) > 1000" echo "3 big vars run remotely - length(base64) > 1000"
stdout ssh rc@lo 'A=`{seq 200}; stdout ssh rc@lo 'A=`{seq 200};
@ -70,10 +88,14 @@ echo '### rc2'
B=`{seq 200 -1 1}; B=`{seq 200 -1 1};
C=`{seq 300 -2 1}; C=`{seq 300 -2 1};
parallel --env A,B,C -k echo '"'"'${}|wc'"'"' ::: A B C' parallel --env A,B,C -k echo '"'"'${}|wc'"'"' ::: A B C'
}
par_tmux_different_shells() {
echo '### Test tmux works on different shells' echo '### Test tmux works on different shells'
(stdout parallel -Scsh@lo,tcsh@lo,parallel@lo,zsh@lo --tmux echo ::: 1 2 3 4; echo $?) | grep -v 'See output'; (stdout parallel -Scsh@lo,tcsh@lo,parallel@lo,zsh@lo --tmux echo ::: 1 2 3 4; echo $?) |
(stdout parallel -Scsh@lo,tcsh@lo,parallel@lo,zsh@lo --tmux false ::: 1 2 3 4; echo $?) | grep -v 'See output'; grep -v 'See output';
(stdout parallel -Scsh@lo,tcsh@lo,parallel@lo,zsh@lo --tmux false ::: 1 2 3 4; echo $?) |
grep -v 'See output';
export PARTMUX='parallel -Scsh@lo,tcsh@lo,parallel@lo,zsh@lo --tmux '; export PARTMUX='parallel -Scsh@lo,tcsh@lo,parallel@lo,zsh@lo --tmux ';
stdout ssh zsh@lo "$PARTMUX" 'true ::: 1 2 3 4; echo $status' | grep -v 'See output'; stdout ssh zsh@lo "$PARTMUX" 'true ::: 1 2 3 4; echo $status' | grep -v 'See output';
@ -85,7 +107,9 @@ echo '### Test tmux works on different shells'
echo "# command is currently too long for csh. Maybe it can be fixed?"; echo "# command is currently too long for csh. Maybe it can be fixed?";
stdout ssh csh@lo "$PARTMUX" 'true ::: 1 2 3 4; echo $status' | grep -v 'See output'; stdout ssh csh@lo "$PARTMUX" 'true ::: 1 2 3 4; echo $status' | grep -v 'See output';
stdout ssh csh@lo "$PARTMUX" 'false ::: 1 2 3 4; echo $status' | grep -v 'See output' stdout ssh csh@lo "$PARTMUX" 'false ::: 1 2 3 4; echo $status' | grep -v 'See output'
}
par_tmux_length() {
echo '### works' echo '### works'
stdout parallel -Sparallel@lo --tmux echo ::: \\\\\\\"\\\\\\\"\\\;\@ | grep -v 'See output' stdout parallel -Sparallel@lo --tmux echo ::: \\\\\\\"\\\\\\\"\\\;\@ | grep -v 'See output'
stdout parallel -Sparallel@lo --tmux echo ::: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx | grep -v 'See output' stdout parallel -Sparallel@lo --tmux echo ::: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx | grep -v 'See output'
@ -96,20 +120,30 @@ echo '### These blocked due to length'
stdout parallel -Stcsh@lo --tmux echo ::: \\\\\\\"\\\\\\\"\\\;\@ | grep -v 'See output' stdout parallel -Stcsh@lo --tmux echo ::: \\\\\\\"\\\\\\\"\\\;\@ | grep -v 'See output'
stdout parallel -Szsh@lo --tmux echo ::: \\\\\\\"\\\\\\\"\\\;\@ | grep -v 'See output' stdout parallel -Szsh@lo --tmux echo ::: \\\\\\\"\\\\\\\"\\\;\@ | grep -v 'See output'
stdout parallel -Scsh@lo --tmux echo ::: 111111111111111111111111111111111111111111111111111111111 | grep -v 'See output' stdout parallel -Scsh@lo --tmux echo ::: 111111111111111111111111111111111111111111111111111111111 | grep -v 'See output'
}
par_transfer_return_multiple_inputs() {
echo '### bug #43746: --transfer and --return of multiple inputs {1} and {2}' echo '### bug #43746: --transfer and --return of multiple inputs {1} and {2}'
echo '### and:' echo '### and:'
echo '### bug #44371: --trc with csh complains' echo '### bug #44371: --trc with csh complains'
cd /tmp; echo 1 > file1; echo 2 > file2; cd /tmp; echo file1 output line 1 > file1; echo file2 output line 3 > file2
parallel -Scsh@lo --transferfile {1} --transferfile {2} --trc {1}.a --trc {2}.b parallel -Scsh@lo --transferfile {1} --transferfile {2} --trc {1}.a --trc {2}.b \
'(cat {1}; echo A {1}) > {1}.a; (cat {2};echo B {2}) > {2}.b' ::: file1 ::: file2; '(cat {1}; echo A {1} output line 2) > {1}.a; (cat {2};echo B {2} output line 4) > {2}.b' ::: file1 ::: file2
cat file1.a file2.b; cat file1.a file2.b
rm /tmp/file1 /tmp/file2 /tmp/file1.a /tmp/file2.b rm /tmp/file1 /tmp/file2 /tmp/file1.a /tmp/file2.b
}
par_csh_nice() {
echo '### bug #44143: csh and nice' echo '### bug #44143: csh and nice'
parallel --nice 1 -S csh@lo setenv B {}\; echo '$B' ::: OK parallel --nice 1 -S csh@lo setenv B {}\; echo '$B' ::: OK
}
par_multiple_hosts_repeat_arg() {
echo '### bug #45575: -m and multiple hosts repeats first args' echo '### bug #45575: -m and multiple hosts repeats first args'
seq 1 3 | parallel -X -S 2/lo,2/: -k echo seq 1 3 | parallel -X -S 2/lo,2/: -k echo
}
EOF export -f $(compgen -A function | grep par_)
compgen -A function | grep par_ | LC_ALL=C sort |
parallel --timeout 3000% -j6 --tag -k --joblog /tmp/jl-`basename $0` '{} 2>&1' |
perl -pe 's:/usr/bin:/bin:g;'

2
testsuite/wanted-results/parallel-local-0.3s
View file

@ -199,7 +199,7 @@ par_halt_one_job parallel: This job failed:
par_halt_one_job echo 1;exit 1 par_halt_one_job echo 1;exit 1
par_hash_and_time_functions ### Functions for replacement string par_hash_and_time_functions ### Functions for replacement string
par_hash_and_time_functions 9999-99-99T99:99:99 9999-99-99T99:99 9999-99-99 99999999999999 999999999999 99999999 par_hash_and_time_functions 9999-99-99T99:99:99 9999-99-99T99:99 9999-99-99 99999999999999 999999999999 99999999
par_hash_and_time_functions XxXXXXXXXX par_hash_and_time_functions XxX
par_help ### Test --help and -h: Help output (just check we get the same amount of lines) par_help ### Test --help and -h: Help output (just check we get the same amount of lines)
par_help Output from -h and --help par_help Output from -h and --help
par_help 37 par_help 37

4
testsuite/wanted-results/parallel-local-1s
View file

@ -713,10 +713,6 @@ par_seqreplace_long_line ### Test --seqreplace and line too long
par_seqreplace_long_line 9 1 1 101 par_seqreplace_long_line 9 1 1 101
par_seqreplace_long_line 90 1 1 201 par_seqreplace_long_line 90 1 1 201
par_seqreplace_long_line 1 parallel: Error: Command line too long (309 >= 210) at input 0: 100 par_seqreplace_long_line 1 parallel: Error: Command line too long (309 >= 210) at input 0: 100
par_sql_CSV ### CSV write to the right place
par_sql_CSV ran OK
par_sql_CSV ok
par_sql_CSV parallel: Error: /must/fail is not a directory.
par_sql_colsep ### SQL should add Vn columns for --colsep par_sql_colsep ### SQL should add Vn columns for --colsep
par_sql_colsep /a/A/1/11/ par_sql_colsep /a/A/1/11/
par_sql_colsep /a/A/2/22/ par_sql_colsep /a/A/2/22/

4
testsuite/wanted-results/parallel-local-race02
View file

@ -147,6 +147,10 @@ par_retries_bug_from_2010 4
par_retries_bug_from_2010 4 par_retries_bug_from_2010 4
par_retries_bug_from_2010 4 par_retries_bug_from_2010 4
par_retries_bug_from_2010 2 par_retries_bug_from_2010 2
par_sql_CSV ### CSV write to the right place
par_sql_CSV ran OK
par_sql_CSV ok
par_sql_CSV parallel: Error: /must/fail is not a directory.
par_testhalt now fail 0 true ### testhalt --halt now,fail=0 par_testhalt now fail 0 true ### testhalt --halt now,fail=0
par_testhalt now fail 0 true job 1 par_testhalt now fail 0 true job 1
par_testhalt now fail 0 true parallel: This job failed: par_testhalt now fail 0 true parallel: This job failed:

199
testsuite/wanted-results/parallel-local-ssh4
View file

@ -1,118 +1,81 @@
echo '### --env _' par_bigvar_csh ### csh
### --env _ par_bigvar_csh 3 big vars run remotely - length(base64) > 1000
fUbAr="OK FUBAR" parallel -S parallel@lo --env _ echo '$fUbAr $DEBEMAIL' ::: test par_bigvar_csh 1 200 692
OK FUBAR test par_bigvar_csh 1 200 692
fUbAr="OK FUBAR" parallel -S csh@lo --env _ echo '$fUbAr $DEBEMAIL' ::: test par_bigvar_csh 1 150 547
OK FUBAR test par_bigvar_csh ### csh2
echo '### --env _ with explicit mentioning of normally ignored var $DEBEMAIL' par_bigvar_csh 3 big vars run locally
### --env _ with explicit mentioning of normally ignored var $DEBEMAIL par_bigvar_csh 1 200 692
fUbAr="OK FUBAR" parallel -S parallel@lo --env DEBEMAIL,_ echo '$fUbAr $DEBEMAIL' ::: test par_bigvar_csh 1 200 692
OK FUBAR ole@tange.dk test par_bigvar_csh 1 150 547
fUbAr="OK FUBAR" parallel -S csh@lo --env DEBEMAIL,_ echo '$fUbAr $DEBEMAIL' ::: test par_bigvar_rc ### rc
OK FUBAR ole@tange.dk test par_bigvar_rc 3 big vars run remotely - length(base64) > 1000
echo 'bug #40137: SHELL not bash: Warning when exporting funcs' par_bigvar_rc 1 200 692
bug #40137: SHELL not bash: Warning when exporting funcs par_bigvar_rc 1 200 692
. <(printf 'myfunc() {\necho $1\n}'); export -f myfunc; parallel --env myfunc -S lo myfunc ::: no_warning par_bigvar_rc 1 150 547
no_warning par_bigvar_rc ### rc2
. <(printf 'myfunc() {\necho $1\n}'); export -f myfunc; SHELL=/bin/sh parallel --env myfunc -S lo myfunc ::: warning par_bigvar_rc 3 big vars run locally
warning par_bigvar_rc 1 200 692
echo '### zsh' par_bigvar_rc 1 200 692
### zsh par_bigvar_rc 1 150 547
echo 'env in zsh' par_csh_nice ### bug #44143: csh and nice
env in zsh par_csh_nice OK
echo 'Normal variable export' par_env_underscore ### --env _
Normal variable export par_env_underscore OK FUBAR test
export B=\'; PARALLEL_SHELL=/usr/bin/zsh parallel --env B echo '$B' ::: a par_env_underscore In csh this may fail with ignored_var: Undefined variable.
' a par_env_underscore ignored_var: Undefined variable.
echo 'Function export as variable' par_env_underscore ### --env _ with explicit mentioning of normally ignored var $ignored_var
Function export as variable par_env_underscore OK FUBAR should be copied test
export myfuncvar="() { echo myfuncvar \$*; }"; PARALLEL_SHELL=/usr/bin/zsh parallel --env myfuncvar myfuncvar ::: a par_env_underscore OK FUBAR should be copied test
myfuncvar a par_exporting_in_zsh ### zsh
echo 'Function export as function' par_exporting_in_zsh env in zsh
Function export as function par_exporting_in_zsh Normal variable export
myfunc() { echo myfunc $*; }; export -f myfunc; PARALLEL_SHELL=/usr/bin/zsh parallel --env myfunc myfunc ::: a par_exporting_in_zsh ' Var with quote OK
myfunc a par_exporting_in_zsh Function export as variable
ssh zsh@lo 'fun="() { echo function from zsh to zsh \$*; }"; export fun; parallel --env fun fun ::: OK' par_exporting_in_zsh myfuncvar as var OK
function from zsh to zsh OK par_exporting_in_zsh Function export as function
ssh zsh@lo 'fun="() { echo function from zsh to bash \$*; }"; export fun; parallel -S parallel@lo --env fun fun ::: OK' par_exporting_in_zsh myfunc ran OK
function from zsh to bash OK par_exporting_in_zsh function from zsh to zsh OK
echo '### csh' par_exporting_in_zsh function from zsh to bash OK
### csh par_multiple_hosts_repeat_arg ### bug #45575: -m and multiple hosts repeats first args
echo "3 big vars run remotely - length(base64) > 1000" par_multiple_hosts_repeat_arg parallel: Warning: Using -X or -m with --sshlogin may fail.
3 big vars run remotely - length(base64) > 1000 par_multiple_hosts_repeat_arg 1
stdout ssh csh@lo 'setenv A `seq 200|xargs`; setenv B `seq 200 -1 1|xargs`; setenv C `seq 300 -2 1|xargs`; parallel -Scsh@lo --env A,B,C -k echo \$\{\}\|wc ::: A B C' par_multiple_hosts_repeat_arg 2
1 200 692 par_multiple_hosts_repeat_arg 3
1 200 692 par_tmux_different_shells ### Test tmux works on different shells
1 150 547 par_tmux_different_shells 0
echo '### csh2' par_tmux_different_shells 4
### csh2 par_tmux_different_shells 0
echo "3 big vars run locally" par_tmux_different_shells 4
3 big vars run locally par_tmux_different_shells 0
stdout ssh csh@lo 'setenv A `seq 200|xargs`; setenv B `seq 200 -1 1|xargs`; setenv C `seq 300 -2 1|xargs`; parallel --env A,B,C -k echo \$\{\}\|wc ::: A B C' par_tmux_different_shells 4
1 200 692 par_tmux_different_shells 0
1 200 692 par_tmux_different_shells 4
1 150 547 par_tmux_different_shells # command is currently too long for csh. Maybe it can be fixed?
echo '### rc' par_tmux_different_shells 0
### rc par_tmux_different_shells 4
echo "3 big vars run remotely - length(base64) > 1000" par_tmux_length ### works
3 big vars run remotely - length(base64) > 1000 par_tmux_length ### These blocked due to length
stdout ssh rc@lo 'A=`{seq 200}; B=`{seq 200 -1 1}; C=`{seq 300 -2 1}; parallel -Src@lo --env A,B,C -k echo '"'"'${}|wc'"'"' ::: A B C' par_transfer_return_multiple_inputs ### bug #43746: --transfer and --return of multiple inputs {1} and {2}
1 200 692 par_transfer_return_multiple_inputs ### and:
1 200 692 par_transfer_return_multiple_inputs ### bug #44371: --trc with csh complains
1 150 547 par_transfer_return_multiple_inputs file1 output line 1
echo '### rc2' par_transfer_return_multiple_inputs A file1 output line 2
### rc2 par_transfer_return_multiple_inputs file2 output line 3
echo "3 big vars run locally" par_transfer_return_multiple_inputs B file2 output line 4
3 big vars run locally par_warn_when_exporting_func bug #40137: SHELL not bash: Warning when exporting funcs
stdout ssh rc@lo 'A=`{seq 200}; B=`{seq 200 -1 1}; C=`{seq 300 -2 1}; parallel --env A,B,C -k echo '"'"'${}|wc'"'"' ::: A B C' par_warn_when_exporting_func /bin/sh Run function in /bin/sh
1 200 692 par_warn_when_exporting_func /bin/sh parallel: Warning: Shell functions may not be supported in /bin/sh.
1 200 692 par_warn_when_exporting_func /bin/bash Run function in /bin/bash
1 150 547 par_warn_when_exporting_func /bin/bash Function run: OK
echo '### Test tmux works on different shells' par_warn_when_exporting_func /bin/ash Run function in /bin/ash
### Test tmux works on different shells par_warn_when_exporting_func /bin/csh Run function in /bin/csh
(stdout parallel -Scsh@lo,tcsh@lo,parallel@lo,zsh@lo --tmux echo ::: 1 2 3 4; echo $?) | grep -v 'See output'; par_warn_when_exporting_func /bin/csh parallel: Warning: Shell functions may not be supported in /bin/csh.
0 par_warn_when_exporting_func /bin/csh CSH/TCSH DO NOT SUPPORT newlines IN VARIABLES/FUNCTIONS. Unset myfunc
(stdout parallel -Scsh@lo,tcsh@lo,parallel@lo,zsh@lo --tmux false ::: 1 2 3 4; echo $?) | grep -v 'See output'; par_warn_when_exporting_func /bin/dash Run function in /bin/dash
4 par_warn_when_exporting_func /bin/ksh Run function in /bin/ksh
export PARTMUX='parallel -Scsh@lo,tcsh@lo,parallel@lo,zsh@lo --tmux '; stdout ssh zsh@lo "$PARTMUX" 'true ::: 1 2 3 4; echo $status' | grep -v 'See output'; stdout ssh zsh@lo "$PARTMUX" 'false ::: 1 2 3 4; echo $status' | grep -v 'See output'; stdout ssh parallel@lo "$PARTMUX" 'true ::: 1 2 3 4; echo $?' | grep -v 'See output'; stdout ssh parallel@lo "$PARTMUX" 'false ::: 1 2 3 4; echo $?' | grep -v 'See output'; stdout ssh tcsh@lo "$PARTMUX" 'true ::: 1 2 3 4; echo $status' | grep -v 'See output'; stdout ssh tcsh@lo "$PARTMUX" 'false ::: 1 2 3 4; echo $status' | grep -v 'See output'; echo "# command is currently too long for csh. Maybe it can be fixed?"; stdout ssh csh@lo "$PARTMUX" 'true ::: 1 2 3 4; echo $status' | grep -v 'See output'; stdout ssh csh@lo "$PARTMUX" 'false ::: 1 2 3 4; echo $status' | grep -v 'See output' par_warn_when_exporting_func /bin/tcsh Run function in /bin/tcsh
0 par_warn_when_exporting_func /bin/tcsh parallel: Warning: Shell functions may not be supported in /bin/tcsh.
4 par_warn_when_exporting_func /bin/tcsh CSH/TCSH DO NOT SUPPORT newlines IN VARIABLES/FUNCTIONS. Unset myfunc
0 par_warn_when_exporting_func /bin/zsh Run function in /bin/zsh
4 par_warn_when_exporting_func /bin/zsh Function run: OK
0
4
# command is currently too long for csh. Maybe it can be fixed?
0
4
echo '### works'
### works
stdout parallel -Sparallel@lo --tmux echo ::: \\\\\\\"\\\\\\\"\\\;\@ | grep -v 'See output'
stdout parallel -Sparallel@lo --tmux echo ::: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx | grep -v 'See output'
echo '### These blocked due to length'
### These blocked due to length
stdout parallel -Slo --tmux echo ::: \\\\\\\"\\\\\\\"\\\;\@ | grep -v 'See output'
stdout parallel -Scsh@lo --tmux echo ::: \\\\\\\"\\\\\\\"\\\;\@ | grep -v 'See output'
stdout parallel -Stcsh@lo --tmux echo ::: \\\\\\\"\\\\\\\"\\\;\@ | grep -v 'See output'
stdout parallel -Szsh@lo --tmux echo ::: \\\\\\\"\\\\\\\"\\\;\@ | grep -v 'See output'
stdout parallel -Scsh@lo --tmux echo ::: 111111111111111111111111111111111111111111111111111111111 | grep -v 'See output'
echo '### bug #43746: --transfer and --return of multiple inputs {1} and {2}'
### bug #43746: --transfer and --return of multiple inputs {1} and {2}
echo '### and:'
### and:
echo '### bug #44371: --trc with csh complains'
### bug #44371: --trc with csh complains
cd /tmp; echo 1 > file1; echo 2 > file2; parallel -Scsh@lo --transferfile {1} --transferfile {2} --trc {1}.a --trc {2}.b '(cat {1}; echo A {1}) > {1}.a; (cat {2};echo B {2}) > {2}.b' ::: file1 ::: file2; cat file1.a file2.b; rm /tmp/file1 /tmp/file2 /tmp/file1.a /tmp/file2.b
1
A file1
2
B file2
echo '### bug #44143: csh and nice'
### bug #44143: csh and nice
parallel --nice 1 -S csh@lo setenv B {}\; echo '$B' ::: OK
OK
echo '### bug #45575: -m and multiple hosts repeats first args'
### bug #45575: -m and multiple hosts repeats first args
seq 1 3 | parallel -X -S 2/lo,2/: -k echo
1
2
3