Released as 20211222 ('Støjberg')

This commit is contained in:
Ole Tange 2021-12-22 22:27:34 +01:00
parent 0fa2bbc099
commit ed9ddd0ddd
27 changed files with 276 additions and 248 deletions

23
NEWS
View file

@ -1,3 +1,26 @@
20211222
New in this release:
* Bug fixes and man page updates.
News about GNU Parallel:
* The first version of Parallel will be 20 years on 2022-01-06
* HerrComp Intro HPC gnu parallel
https://www.youtube.com/watch?v=tcgjKWfgWQY
* 多数のファイルを処理する場合の工夫
https://qiita.com/toramili/items/44a770cfb9ef1edc9e26
* How To Install parallel on Ubuntu 20.04
https://installati.one/ubuntu/20.04/parallel/
* Distributing embarrassingly parallel tasks GNU Parallel
https://ulhpc-tutorials.readthedocs.io/en/latest/sequential/gnu-parallel/
20211122 20211122
New in this release: New in this release:

24
README
View file

@ -57,11 +57,11 @@ document.
Full installation of GNU Parallel is as simple as: Full installation of GNU Parallel is as simple as:
wget https://ftpmirror.gnu.org/parallel/parallel-20211122.tar.bz2 wget https://ftpmirror.gnu.org/parallel/parallel-20211222.tar.bz2
wget https://ftpmirror.gnu.org/parallel/parallel-20211122.tar.bz2.sig wget https://ftpmirror.gnu.org/parallel/parallel-20211222.tar.bz2.sig
gpg parallel-20211122.tar.bz2.sig gpg parallel-20211222.tar.bz2.sig
bzip2 -dc parallel-20211122.tar.bz2 | tar xvf - bzip2 -dc parallel-20211222.tar.bz2 | tar xvf -
cd parallel-20211122 cd parallel-20211222
./configure && make && sudo make install ./configure && make && sudo make install
@ -70,11 +70,11 @@ Full installation of GNU Parallel is as simple as:
If you are not root you can add ~/bin to your path and install in If you are not root you can add ~/bin to your path and install in
~/bin and ~/share: ~/bin and ~/share:
wget https://ftpmirror.gnu.org/parallel/parallel-20211122.tar.bz2 wget https://ftpmirror.gnu.org/parallel/parallel-20211222.tar.bz2
wget https://ftpmirror.gnu.org/parallel/parallel-20211122.tar.bz2.sig wget https://ftpmirror.gnu.org/parallel/parallel-20211222.tar.bz2.sig
gpg parallel-20211122.tar.bz2.sig gpg parallel-20211222.tar.bz2.sig
bzip2 -dc parallel-20211122.tar.bz2 | tar xvf - bzip2 -dc parallel-20211222.tar.bz2 | tar xvf -
cd parallel-20211122 cd parallel-20211222
./configure --prefix=$HOME && make && make install ./configure --prefix=$HOME && make && make install
Or if your system lacks 'make' you can simply copy src/parallel Or if your system lacks 'make' you can simply copy src/parallel
@ -122,8 +122,8 @@ will love you for it.
When using programs that use GNU Parallel to process data for When using programs that use GNU Parallel to process data for
publication please cite: publication please cite:
Tange, O. (2021, November 22). GNU Parallel 20211122 ('Peng Shuai'). Tange, O. (2021, December 22). GNU Parallel 20211222 ('Støjberg').
Zenodo. https://doi.org/10.5281/zenodo.5719513 Zenodo. https://doi.org/10.5281/zenodo.5797028
Copyright (C) 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, Copyright (C) 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015,
2016, 2017, 2018, 2019, 2020, 2021 Ole Tange, http://ole.tange.dk and 2016, 2017, 2018, 2019, 2020, 2021 Ole Tange, http://ole.tange.dk and

20
configure vendored
View file

@ -1,6 +1,6 @@
#! /bin/sh #! /bin/sh
# Guess values for system-dependent variables and create Makefiles. # Guess values for system-dependent variables and create Makefiles.
# Generated by GNU Autoconf 2.69 for parallel 20211122. # Generated by GNU Autoconf 2.69 for parallel 20211222.
# #
# Report bugs to <bug-parallel@gnu.org>. # Report bugs to <bug-parallel@gnu.org>.
# #
@ -579,8 +579,8 @@ MAKEFLAGS=
# Identity of this package. # Identity of this package.
PACKAGE_NAME='parallel' PACKAGE_NAME='parallel'
PACKAGE_TARNAME='parallel' PACKAGE_TARNAME='parallel'
PACKAGE_VERSION='20211122' PACKAGE_VERSION='20211222'
PACKAGE_STRING='parallel 20211122' PACKAGE_STRING='parallel 20211222'
PACKAGE_BUGREPORT='bug-parallel@gnu.org' PACKAGE_BUGREPORT='bug-parallel@gnu.org'
PACKAGE_URL='' PACKAGE_URL=''
@ -1214,7 +1214,7 @@ if test "$ac_init_help" = "long"; then
# Omit some internal or obsolete options to make the list less imposing. # Omit some internal or obsolete options to make the list less imposing.
# This message is too long to be a string in the A/UX 3.1 sh. # This message is too long to be a string in the A/UX 3.1 sh.
cat <<_ACEOF cat <<_ACEOF
\`configure' configures parallel 20211122 to adapt to many kinds of systems. \`configure' configures parallel 20211222 to adapt to many kinds of systems.
Usage: $0 [OPTION]... [VAR=VALUE]... Usage: $0 [OPTION]... [VAR=VALUE]...
@ -1281,7 +1281,7 @@ fi
if test -n "$ac_init_help"; then if test -n "$ac_init_help"; then
case $ac_init_help in case $ac_init_help in
short | recursive ) echo "Configuration of parallel 20211122:";; short | recursive ) echo "Configuration of parallel 20211222:";;
esac esac
cat <<\_ACEOF cat <<\_ACEOF
@ -1357,7 +1357,7 @@ fi
test -n "$ac_init_help" && exit $ac_status test -n "$ac_init_help" && exit $ac_status
if $ac_init_version; then if $ac_init_version; then
cat <<\_ACEOF cat <<\_ACEOF
parallel configure 20211122 parallel configure 20211222
generated by GNU Autoconf 2.69 generated by GNU Autoconf 2.69
Copyright (C) 2012 Free Software Foundation, Inc. Copyright (C) 2012 Free Software Foundation, Inc.
@ -1374,7 +1374,7 @@ cat >config.log <<_ACEOF
This file contains any messages produced by compilers while This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake. running configure, to aid debugging if configure makes a mistake.
It was created by parallel $as_me 20211122, which was It was created by parallel $as_me 20211222, which was
generated by GNU Autoconf 2.69. Invocation command line was generated by GNU Autoconf 2.69. Invocation command line was
$ $0 $@ $ $0 $@
@ -2237,7 +2237,7 @@ fi
# Define the identity of the package. # Define the identity of the package.
PACKAGE='parallel' PACKAGE='parallel'
VERSION='20211122' VERSION='20211222'
cat >>confdefs.h <<_ACEOF cat >>confdefs.h <<_ACEOF
@ -2880,7 +2880,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
# report actual input values of CONFIG_FILES etc. instead of their # report actual input values of CONFIG_FILES etc. instead of their
# values after options handling. # values after options handling.
ac_log=" ac_log="
This file was extended by parallel $as_me 20211122, which was This file was extended by parallel $as_me 20211222, which was
generated by GNU Autoconf 2.69. Invocation command line was generated by GNU Autoconf 2.69. Invocation command line was
CONFIG_FILES = $CONFIG_FILES CONFIG_FILES = $CONFIG_FILES
@ -2942,7 +2942,7 @@ _ACEOF
cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`" ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`"
ac_cs_version="\\ ac_cs_version="\\
parallel config.status 20211122 parallel config.status 20211222
configured by $0, generated by GNU Autoconf 2.69, configured by $0, generated by GNU Autoconf 2.69,
with options \\"\$ac_cs_config\\" with options \\"\$ac_cs_config\\"

View file

@ -1,4 +1,4 @@
AC_INIT([parallel], [20211122], [bug-parallel@gnu.org]) AC_INIT([parallel], [20211222], [bug-parallel@gnu.org])
AM_INIT_AUTOMAKE([-Wall -Werror foreign]) AM_INIT_AUTOMAKE([-Wall -Werror foreign])
AC_CONFIG_HEADERS([config.h]) AC_CONFIG_HEADERS([config.h])
AC_CONFIG_FILES([ AC_CONFIG_FILES([

View file

@ -53,7 +53,7 @@ Quote of the month:
This is a fantastic tool, and I wish I had upgraded from xargs years ago! This is a fantastic tool, and I wish I had upgraded from xargs years ago!
-- Stuart Anderson -- Stuart Anderson
GNU Parallel and ripgrep would be your friend here. Ripgrep is fast, really fast. GNU Parallel and ripgrep would be your friend here.
-- CENAPT @cenaptech@twitter -- CENAPT @cenaptech@twitter
GNU Parallel is great GNU Parallel is great
@ -82,9 +82,6 @@ Quote of the month:
involved involved
-- rrees @rrees@twitter -- rrees @rrees@twitter
My favorite man page is that of GNU parallel.
-- Jeroen Janssens @jeroenhjanssens@twitter
I have gotten a *ton* of mileage out of jq, awk, and GNU parallel, I have gotten a *ton* of mileage out of jq, awk, and GNU parallel,
even at multi-GB sizes. even at multi-GB sizes.
-- Eric Wolak @ericthewolak@twitter -- Eric Wolak @ericthewolak@twitter
@ -144,6 +141,9 @@ https://negfeedback.blogspot.com/2020/05/indispensable-command-line-tools.html
=== Used === === Used ===
My favorite man page is that of GNU parallel.
-- Jeroen Janssens @jeroenhjanssens@twitter
GNU parallel 便利すぎ GNU parallel 便利すぎ
-- @butagannen@twitter 豚顔面 -- @butagannen@twitter 豚顔面

View file

@ -254,16 +254,16 @@ from:tange@gnu.org
to:parallel@gnu.org, bug-parallel@gnu.org to:parallel@gnu.org, bug-parallel@gnu.org
stable-bcc: Jesse Alama <jessealama@fastmail.fm> stable-bcc: Jesse Alama <jessealama@fastmail.fm>
Subject: GNU Parallel 20211122 ('Peng Shuai') released [stable] Subject: GNU Parallel 20211222 ('Støjberg') released [stable]
GNU Parallel 20211122 ('Peng Shuai') [stable] has been released. It is available for download at: lbry://@GnuParallel:4 GNU Parallel 20211222 ('Støjberg') [stable] has been released. It is available for download at: lbry://@GnuParallel:4
No new functionality was introduced so this is a good candidate for a stable release. No new functionality was introduced so this is a good candidate for a stable release.
Quote of the month: Quote of the month:
GNU parallel 便利すぎ My favorite man page is that of GNU parallel.
-- @butagannen@twitter 豚顔面 -- Jeroen Janssens @jeroenhjanssens@twitter
New in this release: New in this release:
@ -271,9 +271,16 @@ New in this release:
News about GNU Parallel: News about GNU Parallel:
* Otimizando o seu Hacking com o GNU Parallel - Mateus Buogo https://www.youtube.com/watch?v=xIpPPZXDKGU * The first version of Parallel will be 20 years on 2022-01-06
* HerrComp Intro HPC gnu parallel https://www.youtube.com/watch?v=tcgjKWfgWQY
* 多数のファイルを処理する場合の工夫 https://qiita.com/toramili/items/44a770cfb9ef1edc9e26
* How To Install parallel on Ubuntu 20.04 https://installati.one/ubuntu/20.04/parallel/
* Distributing embarrassingly parallel tasks GNU Parallel https://ulhpc-tutorials.readthedocs.io/en/latest/sequential/gnu-parallel/
* SMACK 12 - Do it faster! Simple ways to use all those cores (and GPUs) efficiently. https://www.youtube.com/watch?v=lXDGY7NvOYk
Get the book: GNU Parallel 2018 http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html Get the book: GNU Parallel 2018 http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html

View file

@ -1,6 +1,7 @@
<directory name="parallel" rev="315" vrev="1" srcmd5="33b4d80c573d095c4ca4caa666375fea"> <directory name="parallel" rev="317" vrev="1" srcmd5="e846e607bc463efd4dcef96a779f85a1">
<entry name="PKGBUILD" md5="75a71ab9576a431055b4ac5b0a9e90ad" size="936" mtime="1637624640" /> <entry name="PKGBUILD" md5="b96ad8edb325e80e5aba8b32d5254f78" size="936" mtime="1640129207" />
<entry name="parallel-20211122.tar.bz2" md5="0006cbd95988cd917544fcecc573a0e7" size="2267602" mtime="1637624641" /> <entry name="parallel-20211222.tar.bz2" md5="b309c72c1397097ef7a70e1389d5f747" size="2267915" mtime="1640129208" />
<entry name="parallel.spec" md5="a6ed860e4299d7f7ff347bf27271ead1" size="5630" mtime="1637624641" /> <entry name="parallel.spec" md5="d52fefe84eec835ff08d7e3643f74e0b" size="5630" mtime="1640129208" />
<entry name="parallel_20211122.tar.gz" md5="755f684b26232778e596109491e64c09" size="2512204" mtime="1637624641" /> <entry name="parallel_20211222.dsc" md5="4438eb9e04d278e2d900438461ac0139" size="556" mtime="1640129208" />
<entry name="parallel_20211222.tar.gz" md5="75f015315f2b351a62f31c9b59795f41" size="2512216" mtime="1640129208" />
</directory> </directory>

View file

@ -1,7 +1,7 @@
Summary: Shell tool for executing jobs in parallel Summary: Shell tool for executing jobs in parallel
Name: parallel Name: parallel
Version: 20211122 Version: 20211222
Release: 1.3 Release: 1.3
License: GPL-3.0-or-later License: GPL-3.0-or-later
Group: Productivity/File utilities Group: Productivity/File utilities

View file

@ -30,10 +30,10 @@ installer() {
source="$1" source="$1"
script="$2" script="$2"
into="$3" into="$3"
if grepq $script $into; then if grepq $script "$into"; then
true already installed true already installed
else else
echo $source \`which $script\` >> $into echo $source \`which $script\` >> "$into"
fi fi
} }
@ -42,32 +42,32 @@ while test $# -gt 0; do
case $key in case $key in
-i|--install) -i|--install)
installer . env_parallel.bash $HOME/.bashrc installer . env_parallel.bash "$HOME"/.bashrc
installer . env_parallel.sh $HOME/.shrc installer . env_parallel.sh "$HOME"/.shrc
installer . env_parallel.zsh $HOME/.zshenv installer . env_parallel.zsh "$HOME"/.zshenv
installer source env_parallel.ksh $HOME/.kshrc installer source env_parallel.ksh "$HOME"/.kshrc
installer source env_parallel.mksh $HOME/.mkshrc installer source env_parallel.mksh "$HOME"/.mkshrc
echo $SHELL | grepq /pdksh && echo $SHELL | grepq /pdksh &&
installer . env_parallel.pdksh $HOME/.profile installer . env_parallel.pdksh "$HOME"/.profile
echo $SHELL | grepq /ash && echo $SHELL | grepq /ash &&
installer . env_parallel.ash $HOME/.profile installer . env_parallel.ash "$HOME"/.profile
echo $SHELL | grepq /dash && echo $SHELL | grepq /dash &&
installer . env_parallel.dash $HOME/.profile installer . env_parallel.dash "$HOME"/.profile
installer source env_parallel.csh $HOME/.cshrc installer source env_parallel.csh "$HOME"/.cshrc
installer source env_parallel.tcsh $HOME/.tcshrc installer source env_parallel.tcsh "$HOME"/.tcshrc
mkdir -p $HOME/.config/fish mkdir -p "$HOME"/.config/fish
grepq env_parallel.fish $HOME/.config/fish/config.fish || grepq env_parallel.fish "$HOME"/.config/fish/config.fish ||
echo '. (which env_parallel.fish)' >> $HOME/.config/fish/config.fish echo '. (which env_parallel.fish)' >> "$HOME"/.config/fish/config.fish
echo 'Installed env_parallel in:' echo 'Installed env_parallel in:'
echo " " $HOME/.bashrc echo " " "$HOME"/.bashrc
echo " " $HOME/.shrc echo " " "$HOME"/.shrc
echo " " $HOME/.zshenv echo " " "$HOME"/.zshenv
echo " " $HOME/.config/fish/config.fish echo " " "$HOME"/.config/fish/config.fish
echo " " $HOME/.kshrc echo " " "$HOME"/.kshrc
echo " " $HOME/.mkshrc echo " " "$HOME"/.mkshrc
echo " " $HOME/.profile echo " " "$HOME"/.profile
echo " " $HOME/.cshrc echo " " "$HOME"/.cshrc
echo " " $HOME/.tcshrc echo " " "$HOME"/.tcshrc
exit exit
;; ;;
*) *)

View file

@ -385,7 +385,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20211122 (GNU parallel `parallel --minversion 1`)" echo "parset 20211222 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -384,7 +384,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20211122 (GNU parallel `parallel --minversion 1`)" echo "parset 20211222 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -385,7 +385,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20211122 (GNU parallel `parallel --minversion 1`)" echo "parset 20211222 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -363,7 +363,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20211122 (GNU parallel `parallel --minversion 1`)" echo "parset 20211222 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -365,7 +365,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20211122 (GNU parallel `parallel --minversion 1`)" echo "parset 20211222 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -390,7 +390,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20211122 (GNU parallel `parallel --minversion 1`)" echo "parset 20211222 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -355,7 +355,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20211122 (GNU parallel `parallel --minversion 1`)" echo "parset 20211222 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -26,7 +26,7 @@
use strict; use strict;
use Getopt::Long; use Getopt::Long;
$Global::progname="niceload"; $Global::progname="niceload";
$Global::version = 20211122; $Global::version = 20211222;
Getopt::Long::Configure("bundling","require_order"); Getopt::Long::Configure("bundling","require_order");
get_options_from_array(\@ARGV) || die_usage(); get_options_from_array(\@ARGV) || die_usage();
if($opt::version) { if($opt::version) {

View file

@ -924,7 +924,7 @@ sub group_by_loop($$) {
return $loop; return $loop;
} }
sub group_by_stdin_filter() { sub pipe_group_by_setup() {
# Record separator with 119 bit random value # Record separator with 119 bit random value
$opt::recend = ''; $opt::recend = '';
$opt::recstart = $opt::recstart =
@ -2264,7 +2264,7 @@ sub check_invalid_option_combinations() {
sub init_globals() { sub init_globals() {
# Defaults: # Defaults:
$Global::version = 20211122; $Global::version = 20211222;
$Global::progname = 'parallel'; $Global::progname = 'parallel';
$::name = "GNU Parallel"; $::name = "GNU Parallel";
$Global::infinity = 2**31; $Global::infinity = 2**31;
@ -5077,8 +5077,8 @@ sub usage() {
"If you use programs that use GNU Parallel to process data for an article in a", "If you use programs that use GNU Parallel to process data for an article in a",
"scientific publication, please cite:", "scientific publication, please cite:",
"", "",
" Tange, O. (2021, November 22). GNU Parallel 20211122 ('Peng Shuai').", " Tange, O. (2021, December 22). GNU Parallel 20211222 ('Støjberg').",
" Zenodo. https://doi.org/10.5281/zenodo.5719513", " Zenodo. https://doi.org/10.5281/zenodo.5797028",
"", "",
# Before changing these lines, please read # Before changing these lines, please read
# https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice # https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice
@ -5110,8 +5110,8 @@ sub citation_notice() {
"If you use programs that use GNU Parallel to process data for an article in a", "If you use programs that use GNU Parallel to process data for an article in a",
"scientific publication, please cite:", "scientific publication, please cite:",
"", "",
" Tange, O. (2021, November 22). GNU Parallel 20211122 ('Peng Shuai').", " Tange, O. (2021, December 22). GNU Parallel 20211222 ('Støjberg').",
" Zenodo. https://doi.org/10.5281/zenodo.5719513", " Zenodo. https://doi.org/10.5281/zenodo.5797028",
"", "",
# Before changing these line, please read # Before changing these line, please read
# https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and # https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and
@ -5236,20 +5236,20 @@ sub citation() {
"If you use programs that use GNU Parallel to process data for an article in a", "If you use programs that use GNU Parallel to process data for an article in a",
"scientific publication, please cite:", "scientific publication, please cite:",
"", "",
"\@software{tange_2021_5719513,", "\@software{tange_2021_5797028,",
" author = {Tange, Ole},", " author = {Tange, Ole},",
" title = {GNU Parallel 20211122 ('Peng Shuai')},", " title = {GNU Parallel 20211222 ('Støjberg')},",
" month = Nov,", " month = Dec,",
" year = 2021,", " year = 2021,",
" note = {{GNU Parallel is a general parallelizer to run", " note = {{GNU Parallel is a general parallelizer to run",
" multiple serial command line programs in parallel", " multiple serial command line programs in parallel",
" without changing them.}},", " without changing them.}},",
" publisher = {Zenodo},", " publisher = {Zenodo},",
" doi = {10.5281/zenodo.5719513},", " doi = {10.5281/zenodo.5797028},",
" url = {https://doi.org/10.5281/zenodo.5719513}", " url = {https://doi.org/10.5281/zenodo.5797028}",
"}", "}",
"", "",
"(Feel free to use \\nocite{tange_2021_5719513})", "(Feel free to use \\nocite{tange_2021_5797028})",
"", "",
# Before changing these lines, please read # Before changing these lines, please read
# https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and # https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and
@ -13991,15 +13991,16 @@ sub main() {
if($opt::pipepart) { if($opt::pipepart) {
pipepart_setup(); pipepart_setup();
} elsif($opt::pipe and $opt::tee) { } elsif($opt::pipe) {
if($opt::tee) {
pipe_tee_setup(); pipe_tee_setup();
} elsif($opt::pipe and $opt::shard or $opt::bin) { } elsif($opt::shard or $opt::bin) {
pipe_shard_setup(); pipe_shard_setup();
} elsif($opt::groupby) {
pipe_group_by_setup();
}
} }
if(not $opt::pipepart and $opt::groupby) {
group_by_stdin_filter();
}
if($opt::eta or $opt::bar or $opt::shuf or $Global::halt_pct) { if($opt::eta or $opt::bar or $opt::shuf or $Global::halt_pct) {
# Count the number of jobs or shuffle all jobs # Count the number of jobs or shuffle all jobs
# before starting any. # before starting any.

View file

@ -122,7 +122,7 @@ GetOptions(
"help" => \$opt::dummy, "help" => \$opt::dummy,
) || exit(255); ) || exit(255);
$Global::progname = ($0 =~ m:(^|/)([^/]+)$:)[1]; $Global::progname = ($0 =~ m:(^|/)([^/]+)$:)[1];
$Global::version = 20211122; $Global::version = 20211222;
if($opt::version) { version(); exit 0; } if($opt::version) { version(); exit 0; }
@Global::sortoptions = grep { ! /^-D$/ } @Global::sortoptions = grep { ! /^-D$/ }
shell_quote(@ARGV_before[0..($#ARGV_before-$#ARGV-1)]); shell_quote(@ARGV_before[0..($#ARGV_before-$#ARGV-1)]);

View file

@ -600,7 +600,7 @@ $Global::Initfile && unlink $Global::Initfile;
exit ($err); exit ($err);
sub parse_options { sub parse_options {
$Global::version = 20211122; $Global::version = 20211222;
$Global::progname = 'sql'; $Global::progname = 'sql';
# This must be done first as this may exec myself # This must be done first as this may exec myself

View file

@ -8,61 +8,6 @@
# Each should be taking 10-30s and be possible to run in parallel # Each should be taking 10-30s and be possible to run in parallel
# I.e.: No race conditions, no logins # I.e.: No race conditions, no logins
par_shard() {
echo '### --shard'
# Each of the 5 lines should match:
# ##### ##### ######
seq 100000 | parallel --pipe --shard 1 -j5 wc |
perl -pe 's/(.*\d{5,}){3}/OK/'
# Data should be sharded to all processes
shard_on_col() {
col=$1
seq 10 99 | shuf | perl -pe 's/(.)/$1\t/g' |
parallel --pipe --shard $col -j2 --colsep "\t" sort -k$col |
field $col | sort | uniq -c
}
shard_on_col 1
shard_on_col 2
shard_on_col_name() {
colname=$1
col=$2
(echo AB; seq 10 99 | shuf) | perl -pe 's/(.)/$1\t/g' |
parallel --header : --pipe --shard $colname -j2 --colsep "\t" sort -k$col |
field $col | sort | uniq -c
}
shard_on_col_name A 1
shard_on_col_name B 2
shard_on_col_expr() {
colexpr="$1"
col=$2
(seq 10 99 | shuf) | perl -pe 's/(.)/$1\t/g' |
parallel --pipe --shard "$colexpr" -j2 --colsep "\t" "sort -k$col; echo c1 c2" |
field $col | sort | uniq -c
}
shard_on_col_expr '1 $_%=3' 1
shard_on_col_expr '2 $_%=3' 2
shard_on_col_name_expr() {
colexpr="$1"
col=$2
(echo AB; seq 10 99 | shuf) | perl -pe 's/(.)/$1\t/g' |
parallel --header : --pipe --shard "$colexpr" -j2 --colsep "\t" "sort -k$col; echo c1 c2" |
field $col | sort | uniq -c
}
shard_on_col_name_expr 'A $_%=3' 1
shard_on_col_name_expr 'B $_%=3' 2
echo '*** broken'
# Shorthand for --pipe -j+0
seq 100000 | parallel --shard 1 wc |
perl -pe 's/(.*\d{5,}){3}/OK/'
# Combine with arguments
seq 100000 | parallel --shard 1 echo {}\;wc ::: {1..5} ::: a b |
perl -pe 's/(.*\d{5,}){3}/OK/'
}
par_bin() { par_bin() {
echo '### Test --bin' echo '### Test --bin'
seq 10 | parallel --pipe --bin 1 -j4 wc | sort seq 10 | parallel --pipe --bin 1 -j4 wc | sort

View file

@ -8,6 +8,61 @@
# Each should be taking 30-100s and be possible to run in parallel # Each should be taking 30-100s and be possible to run in parallel
# I.e.: No race conditions, no logins # I.e.: No race conditions, no logins
par_shard() {
echo '### --shard'
# Each of the 5 lines should match:
# ##### ##### ######
seq 100000 | parallel --pipe --shard 1 -j5 wc |
perl -pe 's/(.*\d{5,}){3}/OK/'
# Data should be sharded to all processes
shard_on_col() {
col=$1
seq 10 99 | shuf | perl -pe 's/(.)/$1\t/g' |
parallel --pipe --shard $col -j2 --colsep "\t" sort -k$col |
field $col | sort | uniq -c
}
shard_on_col 1
shard_on_col 2
shard_on_col_name() {
colname=$1
col=$2
(echo AB; seq 10 99 | shuf) | perl -pe 's/(.)/$1\t/g' |
parallel --header : --pipe --shard $colname -j2 --colsep "\t" sort -k$col |
field $col | sort | uniq -c
}
shard_on_col_name A 1
shard_on_col_name B 2
shard_on_col_expr() {
colexpr="$1"
col=$2
(seq 10 99 | shuf) | perl -pe 's/(.)/$1\t/g' |
parallel --pipe --shard "$colexpr" -j2 --colsep "\t" "sort -k$col; echo c1 c2" |
field $col | sort | uniq -c
}
shard_on_col_expr '1 $_%=3' 1
shard_on_col_expr '2 $_%=3' 2
shard_on_col_name_expr() {
colexpr="$1"
col=$2
(echo AB; seq 10 99 | shuf) | perl -pe 's/(.)/$1\t/g' |
parallel --header : --pipe --shard "$colexpr" -j2 --colsep "\t" "sort -k$col; echo c1 c2" |
field $col | sort | uniq -c
}
shard_on_col_name_expr 'A $_%=3' 1
shard_on_col_name_expr 'B $_%=3' 2
echo '*** broken'
# Shorthand for --pipe -j+0
seq 200000 | parallel --pipe --shard 1 wc |
perl -pe 's/(.*\d{5,}){3}/OK/'
# Combine with arguments
seq 200000 | parallel --pipe --shard 1 echo {}\;wc ::: {1..5} ::: a b |
perl -pe 's/(.*\d{5,}){3}/OK/'
}
par_test_diff_roundrobin_k() { par_test_diff_roundrobin_k() {
echo '### test there is difference on -k' echo '### test there is difference on -k'
. $(which env_parallel.bash) . $(which env_parallel.bash)

View file

@ -936,97 +936,6 @@ par_semaphore job3a 4
par_semaphore job2b 5 par_semaphore job2b 5
par_semaphore job3b 6 par_semaphore job3b 6
par_semaphore done par_semaphore done
par_shard ### --shard
par_shard OK
par_shard OK
par_shard OK
par_shard OK
par_shard OK
par_shard 10 1
par_shard 10 2
par_shard 10 3
par_shard 10 4
par_shard 10 5
par_shard 10 6
par_shard 10 7
par_shard 10 8
par_shard 10 9
par_shard 9 0
par_shard 9 1
par_shard 9 2
par_shard 9 3
par_shard 9 4
par_shard 9 5
par_shard 9 6
par_shard 9 7
par_shard 9 8
par_shard 9 9
par_shard 10 1
par_shard 10 2
par_shard 10 3
par_shard 10 4
par_shard 10 5
par_shard 10 6
par_shard 10 7
par_shard 10 8
par_shard 10 9
par_shard 9 0
par_shard 9 1
par_shard 9 2
par_shard 9 3
par_shard 9 4
par_shard 9 5
par_shard 9 6
par_shard 9 7
par_shard 9 8
par_shard 9 9
par_shard 10 1
par_shard 10 2
par_shard 10 3
par_shard 10 4
par_shard 10 5
par_shard 10 6
par_shard 10 7
par_shard 10 8
par_shard 10 9
par_shard 2 c1
par_shard 9 0
par_shard 9 1
par_shard 9 2
par_shard 9 3
par_shard 9 4
par_shard 9 5
par_shard 9 6
par_shard 9 7
par_shard 9 8
par_shard 9 9
par_shard 2 c2
par_shard 10 1
par_shard 10 2
par_shard 10 3
par_shard 10 4
par_shard 10 5
par_shard 10 6
par_shard 10 7
par_shard 10 8
par_shard 10 9
par_shard 2 c1
par_shard 9 0
par_shard 9 1
par_shard 9 2
par_shard 9 3
par_shard 9 4
par_shard 9 5
par_shard 9 6
par_shard 9 7
par_shard 9 8
par_shard 9 9
par_shard 2 c2
par_shard *** broken
par_shard parallel: Error: --shard requires --jobs to be higher than the number of
par_shard parallel: Error: arguments. Increase --jobs.
par_shard parallel: Error: --shard requires --jobs to be higher than the number of
par_shard parallel: Error: arguments. Increase --jobs.
par_shellquote ### Test --shellquote in all shells par_shellquote ### Test --shellquote in all shells
par_shellquote ash ' par_shellquote ash '
par_shellquote ash par_shellquote ash  !"#$%&'"'"'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~€亗儎厗噲墛媽崕彁憭摂晼棙櫄洔潪煚、¥ウЖ┆<D096><E29486><EFBFBD>辈炒刀犯购患骄坷谅媚牌侨墒颂臀闲岩釉罩棕仝圮蒉哙徕沅彐玷殛腱眍镳耱篝貊鼬<E8B28A><E9BCAC><EFBFBD><EFBFBD>' par_shellquote ash par_shellquote ash  !"#$%&'"'"'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~€亗儎厗噲墛媽崕彁憭摂晼棙櫄洔潪煚、¥ウЖ┆<D096><E29486><EFBFBD>辈炒刀犯购患骄坷谅媚牌侨墒颂臀闲岩釉罩棕仝圮蒉哙徕沅彐玷殛腱眍镳耱篝貊鼬<E8B28A><E9BCAC><EFBFBD><EFBFBD>'

View file

@ -1640,6 +1640,93 @@ par_race_condition1 9
par_race_condition1 10 par_race_condition1 10
par_round_robin_blocks bug #49664: --round-robin does not complete par_round_robin_blocks bug #49664: --round-robin does not complete
par_round_robin_blocks 8 par_round_robin_blocks 8
par_shard ### --shard
par_shard OK
par_shard OK
par_shard OK
par_shard OK
par_shard OK
par_shard 10 1
par_shard 10 2
par_shard 10 3
par_shard 10 4
par_shard 10 5
par_shard 10 6
par_shard 10 7
par_shard 10 8
par_shard 10 9
par_shard 9 0
par_shard 9 1
par_shard 9 2
par_shard 9 3
par_shard 9 4
par_shard 9 5
par_shard 9 6
par_shard 9 7
par_shard 9 8
par_shard 9 9
par_shard 10 1
par_shard 10 2
par_shard 10 3
par_shard 10 4
par_shard 10 5
par_shard 10 6
par_shard 10 7
par_shard 10 8
par_shard 10 9
par_shard 9 0
par_shard 9 1
par_shard 9 2
par_shard 9 3
par_shard 9 4
par_shard 9 5
par_shard 9 6
par_shard 9 7
par_shard 9 8
par_shard 9 9
par_shard 10 1
par_shard 10 2
par_shard 10 3
par_shard 10 4
par_shard 10 5
par_shard 10 6
par_shard 10 7
par_shard 10 8
par_shard 10 9
par_shard 2 c1
par_shard 9 0
par_shard 9 1
par_shard 9 2
par_shard 9 3
par_shard 9 4
par_shard 9 5
par_shard 9 6
par_shard 9 7
par_shard 9 8
par_shard 9 9
par_shard 2 c2
par_shard 10 1
par_shard 10 2
par_shard 10 3
par_shard 10 4
par_shard 10 5
par_shard 10 6
par_shard 10 7
par_shard 10 8
par_shard 10 9
par_shard 2 c1
par_shard 9 0
par_shard 9 1
par_shard 9 2
par_shard 9 3
par_shard 9 4
par_shard 9 5
par_shard 9 6
par_shard 9 7
par_shard 9 8
par_shard 9 9
par_shard 2 c2
par_shard *** broken
par_sighup ### Test SIGHUP par_sighup ### Test SIGHUP
par_sighup 1 par_sighup 1
par_sighup 10 par_sighup 10

View file

@ -171,8 +171,8 @@ ls | parallel -j500 'sleep 1; find {} -type f | perl -ne "END{print $..\" "{=$_=
4 ab 4 ab
4 b 4 b
6 a 6 a
parallel: Warning: Only enough file handles to run 251 jobs in parallel. parallel: Warning: Only enough file handles to run 252 jobs in parallel.
parallel: Warning: Try running 'parallel -j0 -N 251 --pipe parallel -j0' parallel: Warning: Try running 'parallel -j0 -N 252 --pipe parallel -j0'
parallel: Warning: or increasing 'ulimit -n' (try: ulimit -n `ulimit -Hn`) parallel: Warning: or increasing 'ulimit -n' (try: ulimit -n `ulimit -Hn`)
parallel: Warning: or increasing 'nofile' in /etc/security/limits.conf parallel: Warning: or increasing 'nofile' in /etc/security/limits.conf
parallel: Warning: or increasing /proc/sys/fs/file-max parallel: Warning: or increasing /proc/sys/fs/file-max
@ -285,8 +285,8 @@ ls | parallel --group -j500 'sleep 1; find {} -type f | perl -ne "END{print $..\
4 ab 4 ab
4 b 4 b
6 a 6 a
parallel: Warning: Only enough file handles to run 251 jobs in parallel. parallel: Warning: Only enough file handles to run 252 jobs in parallel.
parallel: Warning: Try running 'parallel -j0 -N 251 --pipe parallel -j0' parallel: Warning: Try running 'parallel -j0 -N 252 --pipe parallel -j0'
parallel: Warning: or increasing 'ulimit -n' (try: ulimit -n `ulimit -Hn`) parallel: Warning: or increasing 'ulimit -n' (try: ulimit -n `ulimit -Hn`)
parallel: Warning: or increasing 'nofile' in /etc/security/limits.conf parallel: Warning: or increasing 'nofile' in /etc/security/limits.conf
parallel: Warning: or increasing /proc/sys/fs/file-max parallel: Warning: or increasing /proc/sys/fs/file-max

View file

@ -1,12 +1,12 @@
par_big_func 1 7968 191232 par_big_func 1 7964 191136
par_big_func 1 5368 128772 par_big_func 1 5372 128868
par_big_func_name 1 4960 119040 par_big_func_name 1 4960 119040
par_big_func_name 1 40 960 par_big_func_name 1 40 960
par_big_var_func_name 1 4956 118944 par_big_var_func_name 1 4956 118944
par_big_var_func_name 1 4956 118944 par_big_var_func_name 1 4956 118944
par_big_var_func_name 1 3424 82116 par_big_var_func_name 1 3424 82116
par_many_args 1 16400 32800 par_many_args 1 16396 32792
par_many_args 1 3600 7200 par_many_args 1 3604 7208
par_many_func 1 4372 104928 par_many_func 1 4372 104928
par_many_func 1 2296 55076 par_many_func 1 2296 55076
par_many_var 1 5124 122976 par_many_var 1 5124 122976

View file

@ -499,8 +499,8 @@ echo C
sleep 1 sleep 1
echo 0 > my_jobs echo 0 > my_jobs
wait wait
parallel: Warning: Only enough file handles to run 247 jobs in parallel. parallel: Warning: Only enough file handles to run 248 jobs in parallel.
parallel: Warning: Try running 'parallel -j0 -N 247 --pipe parallel -j0' parallel: Warning: Try running 'parallel -j0 -N 248 --pipe parallel -j0'
parallel: Warning: or increasing 'ulimit -n' (try: ulimit -n `ulimit -Hn`) parallel: Warning: or increasing 'ulimit -n' (try: ulimit -n `ulimit -Hn`)
parallel: Warning: or increasing 'nofile' in /etc/security/limits.conf parallel: Warning: or increasing 'nofile' in /etc/security/limits.conf
parallel: Warning: or increasing /proc/sys/fs/file-max parallel: Warning: or increasing /proc/sys/fs/file-max