Released as 20211222 ('Støjberg')

This commit is contained in:
Ole Tange 2021-12-22 22:27:34 +01:00
parent 0fa2bbc099
commit ed9ddd0ddd
27 changed files with 276 additions and 248 deletions

23
NEWS
View file

@ -1,3 +1,26 @@
20211222
New in this release:
* Bug fixes and man page updates.
News about GNU Parallel:
* The first version of Parallel will be 20 years on 2022-01-06
* HerrComp Intro HPC gnu parallel
https://www.youtube.com/watch?v=tcgjKWfgWQY
* 多数のファイルを処理する場合の工夫
https://qiita.com/toramili/items/44a770cfb9ef1edc9e26
* How To Install parallel on Ubuntu 20.04
https://installati.one/ubuntu/20.04/parallel/
* Distributing embarrassingly parallel tasks GNU Parallel
https://ulhpc-tutorials.readthedocs.io/en/latest/sequential/gnu-parallel/
20211122
New in this release:

24
README
View file

@ -57,11 +57,11 @@ document.
Full installation of GNU Parallel is as simple as:
wget https://ftpmirror.gnu.org/parallel/parallel-20211122.tar.bz2
wget https://ftpmirror.gnu.org/parallel/parallel-20211122.tar.bz2.sig
gpg parallel-20211122.tar.bz2.sig
bzip2 -dc parallel-20211122.tar.bz2 | tar xvf -
cd parallel-20211122
wget https://ftpmirror.gnu.org/parallel/parallel-20211222.tar.bz2
wget https://ftpmirror.gnu.org/parallel/parallel-20211222.tar.bz2.sig
gpg parallel-20211222.tar.bz2.sig
bzip2 -dc parallel-20211222.tar.bz2 | tar xvf -
cd parallel-20211222
./configure && make && sudo make install
@ -70,11 +70,11 @@ Full installation of GNU Parallel is as simple as:
If you are not root you can add ~/bin to your path and install in
~/bin and ~/share:
wget https://ftpmirror.gnu.org/parallel/parallel-20211122.tar.bz2
wget https://ftpmirror.gnu.org/parallel/parallel-20211122.tar.bz2.sig
gpg parallel-20211122.tar.bz2.sig
bzip2 -dc parallel-20211122.tar.bz2 | tar xvf -
cd parallel-20211122
wget https://ftpmirror.gnu.org/parallel/parallel-20211222.tar.bz2
wget https://ftpmirror.gnu.org/parallel/parallel-20211222.tar.bz2.sig
gpg parallel-20211222.tar.bz2.sig
bzip2 -dc parallel-20211222.tar.bz2 | tar xvf -
cd parallel-20211222
./configure --prefix=$HOME && make && make install
Or if your system lacks 'make' you can simply copy src/parallel
@ -122,8 +122,8 @@ will love you for it.
When using programs that use GNU Parallel to process data for
publication please cite:
Tange, O. (2021, November 22). GNU Parallel 20211122 ('Peng Shuai').
Zenodo. https://doi.org/10.5281/zenodo.5719513
Tange, O. (2021, December 22). GNU Parallel 20211222 ('Støjberg').
Zenodo. https://doi.org/10.5281/zenodo.5797028
Copyright (C) 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015,
2016, 2017, 2018, 2019, 2020, 2021 Ole Tange, http://ole.tange.dk and

20
configure vendored
View file

@ -1,6 +1,6 @@
#! /bin/sh
# Guess values for system-dependent variables and create Makefiles.
# Generated by GNU Autoconf 2.69 for parallel 20211122.
# Generated by GNU Autoconf 2.69 for parallel 20211222.
#
# Report bugs to <bug-parallel@gnu.org>.
#
@ -579,8 +579,8 @@ MAKEFLAGS=
# Identity of this package.
PACKAGE_NAME='parallel'
PACKAGE_TARNAME='parallel'
PACKAGE_VERSION='20211122'
PACKAGE_STRING='parallel 20211122'
PACKAGE_VERSION='20211222'
PACKAGE_STRING='parallel 20211222'
PACKAGE_BUGREPORT='bug-parallel@gnu.org'
PACKAGE_URL=''
@ -1214,7 +1214,7 @@ if test "$ac_init_help" = "long"; then
# Omit some internal or obsolete options to make the list less imposing.
# This message is too long to be a string in the A/UX 3.1 sh.
cat <<_ACEOF
\`configure' configures parallel 20211122 to adapt to many kinds of systems.
\`configure' configures parallel 20211222 to adapt to many kinds of systems.
Usage: $0 [OPTION]... [VAR=VALUE]...
@ -1281,7 +1281,7 @@ fi
if test -n "$ac_init_help"; then
case $ac_init_help in
short | recursive ) echo "Configuration of parallel 20211122:";;
short | recursive ) echo "Configuration of parallel 20211222:";;
esac
cat <<\_ACEOF
@ -1357,7 +1357,7 @@ fi
test -n "$ac_init_help" && exit $ac_status
if $ac_init_version; then
cat <<\_ACEOF
parallel configure 20211122
parallel configure 20211222
generated by GNU Autoconf 2.69
Copyright (C) 2012 Free Software Foundation, Inc.
@ -1374,7 +1374,7 @@ cat >config.log <<_ACEOF
This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.
It was created by parallel $as_me 20211122, which was
It was created by parallel $as_me 20211222, which was
generated by GNU Autoconf 2.69. Invocation command line was
$ $0 $@
@ -2237,7 +2237,7 @@ fi
# Define the identity of the package.
PACKAGE='parallel'
VERSION='20211122'
VERSION='20211222'
cat >>confdefs.h <<_ACEOF
@ -2880,7 +2880,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
# report actual input values of CONFIG_FILES etc. instead of their
# values after options handling.
ac_log="
This file was extended by parallel $as_me 20211122, which was
This file was extended by parallel $as_me 20211222, which was
generated by GNU Autoconf 2.69. Invocation command line was
CONFIG_FILES = $CONFIG_FILES
@ -2942,7 +2942,7 @@ _ACEOF
cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`"
ac_cs_version="\\
parallel config.status 20211122
parallel config.status 20211222
configured by $0, generated by GNU Autoconf 2.69,
with options \\"\$ac_cs_config\\"

View file

@ -1,4 +1,4 @@
AC_INIT([parallel], [20211122], [bug-parallel@gnu.org])
AC_INIT([parallel], [20211222], [bug-parallel@gnu.org])
AM_INIT_AUTOMAKE([-Wall -Werror foreign])
AC_CONFIG_HEADERS([config.h])
AC_CONFIG_FILES([

View file

@ -53,7 +53,7 @@ Quote of the month:
This is a fantastic tool, and I wish I had upgraded from xargs years ago!
-- Stuart Anderson
GNU Parallel and ripgrep would be your friend here. Ripgrep is fast, really fast.
GNU Parallel and ripgrep would be your friend here.
-- CENAPT @cenaptech@twitter
GNU Parallel is great
@ -82,9 +82,6 @@ Quote of the month:
involved
-- rrees @rrees@twitter
My favorite man page is that of GNU parallel.
-- Jeroen Janssens @jeroenhjanssens@twitter
I have gotten a *ton* of mileage out of jq, awk, and GNU parallel,
even at multi-GB sizes.
-- Eric Wolak @ericthewolak@twitter
@ -144,6 +141,9 @@ https://negfeedback.blogspot.com/2020/05/indispensable-command-line-tools.html
=== Used ===
My favorite man page is that of GNU parallel.
-- Jeroen Janssens @jeroenhjanssens@twitter
GNU parallel 便利すぎ
-- @butagannen@twitter 豚顔面

View file

@ -254,16 +254,16 @@ from:tange@gnu.org
to:parallel@gnu.org, bug-parallel@gnu.org
stable-bcc: Jesse Alama <jessealama@fastmail.fm>
Subject: GNU Parallel 20211122 ('Peng Shuai') released [stable]
Subject: GNU Parallel 20211222 ('Støjberg') released [stable]
GNU Parallel 20211122 ('Peng Shuai') [stable] has been released. It is available for download at: lbry://@GnuParallel:4
GNU Parallel 20211222 ('Støjberg') [stable] has been released. It is available for download at: lbry://@GnuParallel:4
No new functionality was introduced so this is a good candidate for a stable release.
Quote of the month:
GNU parallel 便利すぎ
-- @butagannen@twitter 豚顔面
My favorite man page is that of GNU parallel.
-- Jeroen Janssens @jeroenhjanssens@twitter
New in this release:
@ -271,9 +271,16 @@ New in this release:
News about GNU Parallel:
* Otimizando o seu Hacking com o GNU Parallel - Mateus Buogo https://www.youtube.com/watch?v=xIpPPZXDKGU
* The first version of Parallel will be 20 years on 2022-01-06
* HerrComp Intro HPC gnu parallel https://www.youtube.com/watch?v=tcgjKWfgWQY
* 多数のファイルを処理する場合の工夫 https://qiita.com/toramili/items/44a770cfb9ef1edc9e26
* How To Install parallel on Ubuntu 20.04 https://installati.one/ubuntu/20.04/parallel/
* Distributing embarrassingly parallel tasks GNU Parallel https://ulhpc-tutorials.readthedocs.io/en/latest/sequential/gnu-parallel/
* SMACK 12 - Do it faster! Simple ways to use all those cores (and GPUs) efficiently. https://www.youtube.com/watch?v=lXDGY7NvOYk
Get the book: GNU Parallel 2018 http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html

View file

@ -1,6 +1,7 @@
<directory name="parallel" rev="315" vrev="1" srcmd5="33b4d80c573d095c4ca4caa666375fea">
<entry name="PKGBUILD" md5="75a71ab9576a431055b4ac5b0a9e90ad" size="936" mtime="1637624640" />
<entry name="parallel-20211122.tar.bz2" md5="0006cbd95988cd917544fcecc573a0e7" size="2267602" mtime="1637624641" />
<entry name="parallel.spec" md5="a6ed860e4299d7f7ff347bf27271ead1" size="5630" mtime="1637624641" />
<entry name="parallel_20211122.tar.gz" md5="755f684b26232778e596109491e64c09" size="2512204" mtime="1637624641" />
<directory name="parallel" rev="317" vrev="1" srcmd5="e846e607bc463efd4dcef96a779f85a1">
<entry name="PKGBUILD" md5="b96ad8edb325e80e5aba8b32d5254f78" size="936" mtime="1640129207" />
<entry name="parallel-20211222.tar.bz2" md5="b309c72c1397097ef7a70e1389d5f747" size="2267915" mtime="1640129208" />
<entry name="parallel.spec" md5="d52fefe84eec835ff08d7e3643f74e0b" size="5630" mtime="1640129208" />
<entry name="parallel_20211222.dsc" md5="4438eb9e04d278e2d900438461ac0139" size="556" mtime="1640129208" />
<entry name="parallel_20211222.tar.gz" md5="75f015315f2b351a62f31c9b59795f41" size="2512216" mtime="1640129208" />
</directory>

View file

@ -1,7 +1,7 @@
Summary: Shell tool for executing jobs in parallel
Name: parallel
Version: 20211122
Version: 20211222
Release: 1.3
License: GPL-3.0-or-later
Group: Productivity/File utilities

View file

@ -30,10 +30,10 @@ installer() {
source="$1"
script="$2"
into="$3"
if grepq $script $into; then
if grepq $script "$into"; then
true already installed
else
echo $source \`which $script\` >> $into
echo $source \`which $script\` >> "$into"
fi
}
@ -42,32 +42,32 @@ while test $# -gt 0; do
case $key in
-i|--install)
installer . env_parallel.bash $HOME/.bashrc
installer . env_parallel.sh $HOME/.shrc
installer . env_parallel.zsh $HOME/.zshenv
installer source env_parallel.ksh $HOME/.kshrc
installer source env_parallel.mksh $HOME/.mkshrc
installer . env_parallel.bash "$HOME"/.bashrc
installer . env_parallel.sh "$HOME"/.shrc
installer . env_parallel.zsh "$HOME"/.zshenv
installer source env_parallel.ksh "$HOME"/.kshrc
installer source env_parallel.mksh "$HOME"/.mkshrc
echo $SHELL | grepq /pdksh &&
installer . env_parallel.pdksh $HOME/.profile
installer . env_parallel.pdksh "$HOME"/.profile
echo $SHELL | grepq /ash &&
installer . env_parallel.ash $HOME/.profile
installer . env_parallel.ash "$HOME"/.profile
echo $SHELL | grepq /dash &&
installer . env_parallel.dash $HOME/.profile
installer source env_parallel.csh $HOME/.cshrc
installer source env_parallel.tcsh $HOME/.tcshrc
mkdir -p $HOME/.config/fish
grepq env_parallel.fish $HOME/.config/fish/config.fish ||
echo '. (which env_parallel.fish)' >> $HOME/.config/fish/config.fish
installer . env_parallel.dash "$HOME"/.profile
installer source env_parallel.csh "$HOME"/.cshrc
installer source env_parallel.tcsh "$HOME"/.tcshrc
mkdir -p "$HOME"/.config/fish
grepq env_parallel.fish "$HOME"/.config/fish/config.fish ||
echo '. (which env_parallel.fish)' >> "$HOME"/.config/fish/config.fish
echo 'Installed env_parallel in:'
echo " " $HOME/.bashrc
echo " " $HOME/.shrc
echo " " $HOME/.zshenv
echo " " $HOME/.config/fish/config.fish
echo " " $HOME/.kshrc
echo " " $HOME/.mkshrc
echo " " $HOME/.profile
echo " " $HOME/.cshrc
echo " " $HOME/.tcshrc
echo " " "$HOME"/.bashrc
echo " " "$HOME"/.shrc
echo " " "$HOME"/.zshenv
echo " " "$HOME"/.config/fish/config.fish
echo " " "$HOME"/.kshrc
echo " " "$HOME"/.mkshrc
echo " " "$HOME"/.profile
echo " " "$HOME"/.cshrc
echo " " "$HOME"/.tcshrc
exit
;;
*)

View file

@ -385,7 +385,7 @@ _parset_main() {
return 255
fi
if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20211122 (GNU parallel `parallel --minversion 1`)"
echo "parset 20211222 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -384,7 +384,7 @@ _parset_main() {
return 255
fi
if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20211122 (GNU parallel `parallel --minversion 1`)"
echo "parset 20211222 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -385,7 +385,7 @@ _parset_main() {
return 255
fi
if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20211122 (GNU parallel `parallel --minversion 1`)"
echo "parset 20211222 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -363,7 +363,7 @@ _parset_main() {
return 255
fi
if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20211122 (GNU parallel `parallel --minversion 1`)"
echo "parset 20211222 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -365,7 +365,7 @@ _parset_main() {
return 255
fi
if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20211122 (GNU parallel `parallel --minversion 1`)"
echo "parset 20211222 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -390,7 +390,7 @@ _parset_main() {
return 255
fi
if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20211122 (GNU parallel `parallel --minversion 1`)"
echo "parset 20211222 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -355,7 +355,7 @@ _parset_main() {
return 255
fi
if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20211122 (GNU parallel `parallel --minversion 1`)"
echo "parset 20211222 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -26,7 +26,7 @@
use strict;
use Getopt::Long;
$Global::progname="niceload";
$Global::version = 20211122;
$Global::version = 20211222;
Getopt::Long::Configure("bundling","require_order");
get_options_from_array(\@ARGV) || die_usage();
if($opt::version) {

View file

@ -924,7 +924,7 @@ sub group_by_loop($$) {
return $loop;
}
sub group_by_stdin_filter() {
sub pipe_group_by_setup() {
# Record separator with 119 bit random value
$opt::recend = '';
$opt::recstart =
@ -2264,7 +2264,7 @@ sub check_invalid_option_combinations() {
sub init_globals() {
# Defaults:
$Global::version = 20211122;
$Global::version = 20211222;
$Global::progname = 'parallel';
$::name = "GNU Parallel";
$Global::infinity = 2**31;
@ -5077,8 +5077,8 @@ sub usage() {
"If you use programs that use GNU Parallel to process data for an article in a",
"scientific publication, please cite:",
"",
" Tange, O. (2021, November 22). GNU Parallel 20211122 ('Peng Shuai').",
" Zenodo. https://doi.org/10.5281/zenodo.5719513",
" Tange, O. (2021, December 22). GNU Parallel 20211222 ('Støjberg').",
" Zenodo. https://doi.org/10.5281/zenodo.5797028",
"",
# Before changing these lines, please read
# https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice
@ -5110,8 +5110,8 @@ sub citation_notice() {
"If you use programs that use GNU Parallel to process data for an article in a",
"scientific publication, please cite:",
"",
" Tange, O. (2021, November 22). GNU Parallel 20211122 ('Peng Shuai').",
" Zenodo. https://doi.org/10.5281/zenodo.5719513",
" Tange, O. (2021, December 22). GNU Parallel 20211222 ('Støjberg').",
" Zenodo. https://doi.org/10.5281/zenodo.5797028",
"",
# Before changing these line, please read
# https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and
@ -5236,20 +5236,20 @@ sub citation() {
"If you use programs that use GNU Parallel to process data for an article in a",
"scientific publication, please cite:",
"",
"\@software{tange_2021_5719513,",
"\@software{tange_2021_5797028,",
" author = {Tange, Ole},",
" title = {GNU Parallel 20211122 ('Peng Shuai')},",
" month = Nov,",
" title = {GNU Parallel 20211222 ('Støjberg')},",
" month = Dec,",
" year = 2021,",
" note = {{GNU Parallel is a general parallelizer to run",
" multiple serial command line programs in parallel",
" without changing them.}},",
" publisher = {Zenodo},",
" doi = {10.5281/zenodo.5719513},",
" url = {https://doi.org/10.5281/zenodo.5719513}",
" doi = {10.5281/zenodo.5797028},",
" url = {https://doi.org/10.5281/zenodo.5797028}",
"}",
"",
"(Feel free to use \\nocite{tange_2021_5719513})",
"(Feel free to use \\nocite{tange_2021_5797028})",
"",
# Before changing these lines, please read
# https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and
@ -13991,15 +13991,16 @@ sub main() {
if($opt::pipepart) {
pipepart_setup();
} elsif($opt::pipe and $opt::tee) {
} elsif($opt::pipe) {
if($opt::tee) {
pipe_tee_setup();
} elsif($opt::pipe and $opt::shard or $opt::bin) {
} elsif($opt::shard or $opt::bin) {
pipe_shard_setup();
} elsif($opt::groupby) {
pipe_group_by_setup();
}
}
if(not $opt::pipepart and $opt::groupby) {
group_by_stdin_filter();
}
if($opt::eta or $opt::bar or $opt::shuf or $Global::halt_pct) {
# Count the number of jobs or shuffle all jobs
# before starting any.

View file

@ -122,7 +122,7 @@ GetOptions(
"help" => \$opt::dummy,
) || exit(255);
$Global::progname = ($0 =~ m:(^|/)([^/]+)$:)[1];
$Global::version = 20211122;
$Global::version = 20211222;
if($opt::version) { version(); exit 0; }
@Global::sortoptions = grep { ! /^-D$/ }
shell_quote(@ARGV_before[0..($#ARGV_before-$#ARGV-1)]);

View file

@ -600,7 +600,7 @@ $Global::Initfile && unlink $Global::Initfile;
exit ($err);
sub parse_options {
$Global::version = 20211122;
$Global::version = 20211222;
$Global::progname = 'sql';
# This must be done first as this may exec myself

View file

@ -8,61 +8,6 @@
# Each should be taking 10-30s and be possible to run in parallel
# I.e.: No race conditions, no logins
par_shard() {
echo '### --shard'
# Each of the 5 lines should match:
# ##### ##### ######
seq 100000 | parallel --pipe --shard 1 -j5 wc |
perl -pe 's/(.*\d{5,}){3}/OK/'
# Data should be sharded to all processes
shard_on_col() {
col=$1
seq 10 99 | shuf | perl -pe 's/(.)/$1\t/g' |
parallel --pipe --shard $col -j2 --colsep "\t" sort -k$col |
field $col | sort | uniq -c
}
shard_on_col 1
shard_on_col 2
shard_on_col_name() {
colname=$1
col=$2
(echo AB; seq 10 99 | shuf) | perl -pe 's/(.)/$1\t/g' |
parallel --header : --pipe --shard $colname -j2 --colsep "\t" sort -k$col |
field $col | sort | uniq -c
}
shard_on_col_name A 1
shard_on_col_name B 2
shard_on_col_expr() {
colexpr="$1"
col=$2
(seq 10 99 | shuf) | perl -pe 's/(.)/$1\t/g' |
parallel --pipe --shard "$colexpr" -j2 --colsep "\t" "sort -k$col; echo c1 c2" |
field $col | sort | uniq -c
}
shard_on_col_expr '1 $_%=3' 1
shard_on_col_expr '2 $_%=3' 2
shard_on_col_name_expr() {
colexpr="$1"
col=$2
(echo AB; seq 10 99 | shuf) | perl -pe 's/(.)/$1\t/g' |
parallel --header : --pipe --shard "$colexpr" -j2 --colsep "\t" "sort -k$col; echo c1 c2" |
field $col | sort | uniq -c
}
shard_on_col_name_expr 'A $_%=3' 1
shard_on_col_name_expr 'B $_%=3' 2
echo '*** broken'
# Shorthand for --pipe -j+0
seq 100000 | parallel --shard 1 wc |
perl -pe 's/(.*\d{5,}){3}/OK/'
# Combine with arguments
seq 100000 | parallel --shard 1 echo {}\;wc ::: {1..5} ::: a b |
perl -pe 's/(.*\d{5,}){3}/OK/'
}
par_bin() {
echo '### Test --bin'
seq 10 | parallel --pipe --bin 1 -j4 wc | sort

View file

@ -8,6 +8,61 @@
# Each should be taking 30-100s and be possible to run in parallel
# I.e.: No race conditions, no logins
par_shard() {
echo '### --shard'
# Each of the 5 lines should match:
# ##### ##### ######
seq 100000 | parallel --pipe --shard 1 -j5 wc |
perl -pe 's/(.*\d{5,}){3}/OK/'
# Data should be sharded to all processes
shard_on_col() {
col=$1
seq 10 99 | shuf | perl -pe 's/(.)/$1\t/g' |
parallel --pipe --shard $col -j2 --colsep "\t" sort -k$col |
field $col | sort | uniq -c
}
shard_on_col 1
shard_on_col 2
shard_on_col_name() {
colname=$1
col=$2
(echo AB; seq 10 99 | shuf) | perl -pe 's/(.)/$1\t/g' |
parallel --header : --pipe --shard $colname -j2 --colsep "\t" sort -k$col |
field $col | sort | uniq -c
}
shard_on_col_name A 1
shard_on_col_name B 2
shard_on_col_expr() {
colexpr="$1"
col=$2
(seq 10 99 | shuf) | perl -pe 's/(.)/$1\t/g' |
parallel --pipe --shard "$colexpr" -j2 --colsep "\t" "sort -k$col; echo c1 c2" |
field $col | sort | uniq -c
}
shard_on_col_expr '1 $_%=3' 1
shard_on_col_expr '2 $_%=3' 2
shard_on_col_name_expr() {
colexpr="$1"
col=$2
(echo AB; seq 10 99 | shuf) | perl -pe 's/(.)/$1\t/g' |
parallel --header : --pipe --shard "$colexpr" -j2 --colsep "\t" "sort -k$col; echo c1 c2" |
field $col | sort | uniq -c
}
shard_on_col_name_expr 'A $_%=3' 1
shard_on_col_name_expr 'B $_%=3' 2
echo '*** broken'
# Shorthand for --pipe -j+0
seq 200000 | parallel --pipe --shard 1 wc |
perl -pe 's/(.*\d{5,}){3}/OK/'
# Combine with arguments
seq 200000 | parallel --pipe --shard 1 echo {}\;wc ::: {1..5} ::: a b |
perl -pe 's/(.*\d{5,}){3}/OK/'
}
par_test_diff_roundrobin_k() {
echo '### test there is difference on -k'
. $(which env_parallel.bash)

View file

@ -936,97 +936,6 @@ par_semaphore job3a 4
par_semaphore job2b 5
par_semaphore job3b 6
par_semaphore done
par_shard ### --shard
par_shard OK
par_shard OK
par_shard OK
par_shard OK
par_shard OK
par_shard 10 1
par_shard 10 2
par_shard 10 3
par_shard 10 4
par_shard 10 5
par_shard 10 6
par_shard 10 7
par_shard 10 8
par_shard 10 9
par_shard 9 0
par_shard 9 1
par_shard 9 2
par_shard 9 3
par_shard 9 4
par_shard 9 5
par_shard 9 6
par_shard 9 7
par_shard 9 8
par_shard 9 9
par_shard 10 1
par_shard 10 2
par_shard 10 3
par_shard 10 4
par_shard 10 5
par_shard 10 6
par_shard 10 7
par_shard 10 8
par_shard 10 9
par_shard 9 0
par_shard 9 1
par_shard 9 2
par_shard 9 3
par_shard 9 4
par_shard 9 5
par_shard 9 6
par_shard 9 7
par_shard 9 8
par_shard 9 9
par_shard 10 1
par_shard 10 2
par_shard 10 3
par_shard 10 4
par_shard 10 5
par_shard 10 6
par_shard 10 7
par_shard 10 8
par_shard 10 9
par_shard 2 c1
par_shard 9 0
par_shard 9 1
par_shard 9 2
par_shard 9 3
par_shard 9 4
par_shard 9 5
par_shard 9 6
par_shard 9 7
par_shard 9 8
par_shard 9 9
par_shard 2 c2
par_shard 10 1
par_shard 10 2
par_shard 10 3
par_shard 10 4
par_shard 10 5
par_shard 10 6
par_shard 10 7
par_shard 10 8
par_shard 10 9
par_shard 2 c1
par_shard 9 0
par_shard 9 1
par_shard 9 2
par_shard 9 3
par_shard 9 4
par_shard 9 5
par_shard 9 6
par_shard 9 7
par_shard 9 8
par_shard 9 9
par_shard 2 c2
par_shard *** broken
par_shard parallel: Error: --shard requires --jobs to be higher than the number of
par_shard parallel: Error: arguments. Increase --jobs.
par_shard parallel: Error: --shard requires --jobs to be higher than the number of
par_shard parallel: Error: arguments. Increase --jobs.
par_shellquote ### Test --shellquote in all shells
par_shellquote ash '
par_shellquote ash par_shellquote ash  !"#$%&'"'"'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~€亗儎厗噲墛媽崕彁憭摂晼棙櫄洔潪煚、¥ウЖ┆<D096><E29486><EFBFBD>辈炒刀犯购患骄坷谅媚牌侨墒颂臀闲岩釉罩棕仝圮蒉哙徕沅彐玷殛腱眍镳耱篝貊鼬<E8B28A><E9BCAC><EFBFBD><EFBFBD>'

View file

@ -1640,6 +1640,93 @@ par_race_condition1 9
par_race_condition1 10
par_round_robin_blocks bug #49664: --round-robin does not complete
par_round_robin_blocks 8
par_shard ### --shard
par_shard OK
par_shard OK
par_shard OK
par_shard OK
par_shard OK
par_shard 10 1
par_shard 10 2
par_shard 10 3
par_shard 10 4
par_shard 10 5
par_shard 10 6
par_shard 10 7
par_shard 10 8
par_shard 10 9
par_shard 9 0
par_shard 9 1
par_shard 9 2
par_shard 9 3
par_shard 9 4
par_shard 9 5
par_shard 9 6
par_shard 9 7
par_shard 9 8
par_shard 9 9
par_shard 10 1
par_shard 10 2
par_shard 10 3
par_shard 10 4
par_shard 10 5
par_shard 10 6
par_shard 10 7
par_shard 10 8
par_shard 10 9
par_shard 9 0
par_shard 9 1
par_shard 9 2
par_shard 9 3
par_shard 9 4
par_shard 9 5
par_shard 9 6
par_shard 9 7
par_shard 9 8
par_shard 9 9
par_shard 10 1
par_shard 10 2
par_shard 10 3
par_shard 10 4
par_shard 10 5
par_shard 10 6
par_shard 10 7
par_shard 10 8
par_shard 10 9
par_shard 2 c1
par_shard 9 0
par_shard 9 1
par_shard 9 2
par_shard 9 3
par_shard 9 4
par_shard 9 5
par_shard 9 6
par_shard 9 7
par_shard 9 8
par_shard 9 9
par_shard 2 c2
par_shard 10 1
par_shard 10 2
par_shard 10 3
par_shard 10 4
par_shard 10 5
par_shard 10 6
par_shard 10 7
par_shard 10 8
par_shard 10 9
par_shard 2 c1
par_shard 9 0
par_shard 9 1
par_shard 9 2
par_shard 9 3
par_shard 9 4
par_shard 9 5
par_shard 9 6
par_shard 9 7
par_shard 9 8
par_shard 9 9
par_shard 2 c2
par_shard *** broken
par_sighup ### Test SIGHUP
par_sighup 1
par_sighup 10

View file

@ -171,8 +171,8 @@ ls | parallel -j500 'sleep 1; find {} -type f | perl -ne "END{print $..\" "{=$_=
4 ab
4 b
6 a
parallel: Warning: Only enough file handles to run 251 jobs in parallel.
parallel: Warning: Try running 'parallel -j0 -N 251 --pipe parallel -j0'
parallel: Warning: Only enough file handles to run 252 jobs in parallel.
parallel: Warning: Try running 'parallel -j0 -N 252 --pipe parallel -j0'
parallel: Warning: or increasing 'ulimit -n' (try: ulimit -n `ulimit -Hn`)
parallel: Warning: or increasing 'nofile' in /etc/security/limits.conf
parallel: Warning: or increasing /proc/sys/fs/file-max
@ -285,8 +285,8 @@ ls | parallel --group -j500 'sleep 1; find {} -type f | perl -ne "END{print $..\
4 ab
4 b
6 a
parallel: Warning: Only enough file handles to run 251 jobs in parallel.
parallel: Warning: Try running 'parallel -j0 -N 251 --pipe parallel -j0'
parallel: Warning: Only enough file handles to run 252 jobs in parallel.
parallel: Warning: Try running 'parallel -j0 -N 252 --pipe parallel -j0'
parallel: Warning: or increasing 'ulimit -n' (try: ulimit -n `ulimit -Hn`)
parallel: Warning: or increasing 'nofile' in /etc/security/limits.conf
parallel: Warning: or increasing /proc/sys/fs/file-max

View file

@ -1,12 +1,12 @@
par_big_func 1 7968 191232
par_big_func 1 5368 128772
par_big_func 1 7964 191136
par_big_func 1 5372 128868
par_big_func_name 1 4960 119040
par_big_func_name 1 40 960
par_big_var_func_name 1 4956 118944
par_big_var_func_name 1 4956 118944
par_big_var_func_name 1 3424 82116
par_many_args 1 16400 32800
par_many_args 1 3600 7200
par_many_args 1 16396 32792
par_many_args 1 3604 7208
par_many_func 1 4372 104928
par_many_func 1 2296 55076
par_many_var 1 5124 122976

View file

@ -499,8 +499,8 @@ echo C
sleep 1
echo 0 > my_jobs
wait
parallel: Warning: Only enough file handles to run 247 jobs in parallel.
parallel: Warning: Try running 'parallel -j0 -N 247 --pipe parallel -j0'
parallel: Warning: Only enough file handles to run 248 jobs in parallel.
parallel: Warning: Try running 'parallel -j0 -N 248 --pipe parallel -j0'
parallel: Warning: or increasing 'ulimit -n' (try: ulimit -n `ulimit -Hn`)
parallel: Warning: or increasing 'nofile' in /etc/security/limits.conf
parallel: Warning: or increasing /proc/sys/fs/file-max