Compare commits

..

3 commits

26 changed files with 493 additions and 109 deletions

15
NEWS
View file

@ -2,11 +2,24 @@
New in this release: New in this release:
* No new features. This is a candidate for a stable release.
* Bug fixes and man page updates. * Bug fixes and man page updates.
News about GNU Parallel: News about GNU Parallel:
* <<>> * Separate arguments with a custom separator in GNU Parallel
https://boxofcuriosities.co.uk/post/separate-arguments-with-a-custom-separator-in-gnu-parallel
* GNU parallel is underrated
https://amontalenti.com/2021/11/10/parallel
* Unlocking the Power of Supercomputers: My HPC Adventure with 2800
Cores and GNU Parallel
https://augalip.com/2024/03/10/unlocking-the-power-of-supercomputers-my-hpc-adventure-with-2800-cores-and-gnu-parallel/
* Converting WebP Images to PNG Using parallel and dwebp
https://bytefreaks.net/gnulinux/bash/converting-webp-images-to-png-using-parallel-and-dwebp
20240922 20240922

24
README
View file

@ -57,11 +57,11 @@ document.
Full installation of GNU Parallel is as simple as: Full installation of GNU Parallel is as simple as:
wget https://ftpmirror.gnu.org/parallel/parallel-20240922.tar.bz2 wget https://ftpmirror.gnu.org/parallel/parallel-20241022.tar.bz2
wget https://ftpmirror.gnu.org/parallel/parallel-20240922.tar.bz2.sig wget https://ftpmirror.gnu.org/parallel/parallel-20241022.tar.bz2.sig
gpg parallel-20240922.tar.bz2.sig gpg parallel-20241022.tar.bz2.sig
bzip2 -dc parallel-20240922.tar.bz2 | tar xvf - bzip2 -dc parallel-20241022.tar.bz2 | tar xvf -
cd parallel-20240922 cd parallel-20241022
./configure && make && sudo make install ./configure && make && sudo make install
@ -70,11 +70,11 @@ Full installation of GNU Parallel is as simple as:
If you are not root you can add ~/bin to your path and install in If you are not root you can add ~/bin to your path and install in
~/bin and ~/share: ~/bin and ~/share:
wget https://ftpmirror.gnu.org/parallel/parallel-20240922.tar.bz2 wget https://ftpmirror.gnu.org/parallel/parallel-20241022.tar.bz2
wget https://ftpmirror.gnu.org/parallel/parallel-20240922.tar.bz2.sig wget https://ftpmirror.gnu.org/parallel/parallel-20241022.tar.bz2.sig
gpg parallel-20240922.tar.bz2.sig gpg parallel-20241022.tar.bz2.sig
bzip2 -dc parallel-20240922.tar.bz2 | tar xvf - bzip2 -dc parallel-20241022.tar.bz2 | tar xvf -
cd parallel-20240922 cd parallel-20241022
./configure --prefix=$HOME && make && make install ./configure --prefix=$HOME && make && make install
Or if your system lacks 'make' you can simply copy src/parallel Or if your system lacks 'make' you can simply copy src/parallel
@ -122,8 +122,8 @@ will love you for it.
When using programs that use GNU Parallel to process data for When using programs that use GNU Parallel to process data for
publication please cite: publication please cite:
Tange, O. (2024, September 22). GNU Parallel 20240922 ('Gold Apollo AR924'). Tange, O. (2024, October 22). GNU Parallel 20241022 ('Sinwar Nasrallah').
Zenodo. https://doi.org/10.5281/zenodo.13826092 Zenodo. https://doi.org/10.5281/zenodo.13957646
Copyright (C) 2007-2022 Ole Tange, http://ole.tange.dk and Free Copyright (C) 2007-2022 Ole Tange, http://ole.tange.dk and Free
Software Foundation, Inc. Software Foundation, Inc.

20
configure vendored
View file

@ -1,6 +1,6 @@
#! /bin/sh #! /bin/sh
# Guess values for system-dependent variables and create Makefiles. # Guess values for system-dependent variables and create Makefiles.
# Generated by GNU Autoconf 2.71 for parallel 20240922. # Generated by GNU Autoconf 2.71 for parallel 20241022.
# #
# Report bugs to <bug-parallel@gnu.org>. # Report bugs to <bug-parallel@gnu.org>.
# #
@ -610,8 +610,8 @@ MAKEFLAGS=
# Identity of this package. # Identity of this package.
PACKAGE_NAME='parallel' PACKAGE_NAME='parallel'
PACKAGE_TARNAME='parallel' PACKAGE_TARNAME='parallel'
PACKAGE_VERSION='20240922' PACKAGE_VERSION='20241022'
PACKAGE_STRING='parallel 20240922' PACKAGE_STRING='parallel 20241022'
PACKAGE_BUGREPORT='bug-parallel@gnu.org' PACKAGE_BUGREPORT='bug-parallel@gnu.org'
PACKAGE_URL='' PACKAGE_URL=''
@ -1246,7 +1246,7 @@ if test "$ac_init_help" = "long"; then
# Omit some internal or obsolete options to make the list less imposing. # Omit some internal or obsolete options to make the list less imposing.
# This message is too long to be a string in the A/UX 3.1 sh. # This message is too long to be a string in the A/UX 3.1 sh.
cat <<_ACEOF cat <<_ACEOF
\`configure' configures parallel 20240922 to adapt to many kinds of systems. \`configure' configures parallel 20241022 to adapt to many kinds of systems.
Usage: $0 [OPTION]... [VAR=VALUE]... Usage: $0 [OPTION]... [VAR=VALUE]...
@ -1313,7 +1313,7 @@ fi
if test -n "$ac_init_help"; then if test -n "$ac_init_help"; then
case $ac_init_help in case $ac_init_help in
short | recursive ) echo "Configuration of parallel 20240922:";; short | recursive ) echo "Configuration of parallel 20241022:";;
esac esac
cat <<\_ACEOF cat <<\_ACEOF
@ -1390,7 +1390,7 @@ fi
test -n "$ac_init_help" && exit $ac_status test -n "$ac_init_help" && exit $ac_status
if $ac_init_version; then if $ac_init_version; then
cat <<\_ACEOF cat <<\_ACEOF
parallel configure 20240922 parallel configure 20241022
generated by GNU Autoconf 2.71 generated by GNU Autoconf 2.71
Copyright (C) 2021 Free Software Foundation, Inc. Copyright (C) 2021 Free Software Foundation, Inc.
@ -1427,7 +1427,7 @@ cat >config.log <<_ACEOF
This file contains any messages produced by compilers while This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake. running configure, to aid debugging if configure makes a mistake.
It was created by parallel $as_me 20240922, which was It was created by parallel $as_me 20241022, which was
generated by GNU Autoconf 2.71. Invocation command line was generated by GNU Autoconf 2.71. Invocation command line was
$ $0$ac_configure_args_raw $ $0$ac_configure_args_raw
@ -2379,7 +2379,7 @@ fi
# Define the identity of the package. # Define the identity of the package.
PACKAGE='parallel' PACKAGE='parallel'
VERSION='20240922' VERSION='20241022'
printf "%s\n" "#define PACKAGE \"$PACKAGE\"" >>confdefs.h printf "%s\n" "#define PACKAGE \"$PACKAGE\"" >>confdefs.h
@ -3026,7 +3026,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
# report actual input values of CONFIG_FILES etc. instead of their # report actual input values of CONFIG_FILES etc. instead of their
# values after options handling. # values after options handling.
ac_log=" ac_log="
This file was extended by parallel $as_me 20240922, which was This file was extended by parallel $as_me 20241022, which was
generated by GNU Autoconf 2.71. Invocation command line was generated by GNU Autoconf 2.71. Invocation command line was
CONFIG_FILES = $CONFIG_FILES CONFIG_FILES = $CONFIG_FILES
@ -3090,7 +3090,7 @@ ac_cs_config_escaped=`printf "%s\n" "$ac_cs_config" | sed "s/^ //; s/'/'\\\\\\\\
cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
ac_cs_config='$ac_cs_config_escaped' ac_cs_config='$ac_cs_config_escaped'
ac_cs_version="\\ ac_cs_version="\\
parallel config.status 20240922 parallel config.status 20241022
configured by $0, generated by GNU Autoconf 2.71, configured by $0, generated by GNU Autoconf 2.71,
with options \\"\$ac_cs_config\\" with options \\"\$ac_cs_config\\"

View file

@ -1,4 +1,4 @@
AC_INIT([parallel],[20240922],[bug-parallel@gnu.org]) AC_INIT([parallel],[20241022],[bug-parallel@gnu.org])
AM_INIT_AUTOMAKE([-Wall -Werror foreign]) AM_INIT_AUTOMAKE([-Wall -Werror foreign])
AC_CONFIG_HEADERS([config.h]) AC_CONFIG_HEADERS([config.h])
AC_CONFIG_FILES([ AC_CONFIG_FILES([

View file

@ -134,9 +134,6 @@ write a haiku about gnu parallel and the :::+ option
Using [GNU Parallel is] super easy if you use xargs, and it is magic for making things multi-process. Then adding in the ssh magic it can do it is an incredible tool that is completely underutilized. Using [GNU Parallel is] super easy if you use xargs, and it is magic for making things multi-process. Then adding in the ssh magic it can do it is an incredible tool that is completely underutilized.
-- Ancients @Ancients@twitter -- Ancients @Ancients@twitter
GNU Parallel is one of the most helpful tools I've been using recently, and it's just something like: parallel -j4 'gzip {}' ::: folder/*.csv
-- Milton Pividori @miltondp@twitter
This is a fantastic tool, and I wish I had upgraded from xargs years ago! This is a fantastic tool, and I wish I had upgraded from xargs years ago!
-- Stuart Anderson -- Stuart Anderson
@ -224,6 +221,9 @@ https://negfeedback.blogspot.com/2020/05/indispensable-command-line-tools.html
=== Used === === Used ===
GNU Parallel is one of the most helpful tools I've been using recently, and it's just something like: parallel -j4 'gzip {}' ::: folder/*.csv
-- Milton Pividori @miltondp@twitter
Recently executed a flawless live data migration of ~2.4pb using GNU parallel for scale and bash scripts. Recently executed a flawless live data migration of ~2.4pb using GNU parallel for scale and bash scripts.
-- @mechanicker@twitter Dhruva -- @mechanicker@twitter Dhruva

View file

@ -195,6 +195,7 @@ git branch premaster
Add tar.bz2 [Start upload] and [Publish]. Add tar.bz2 [Start upload] and [Publish].
== Update documentation == == Update documentation ==
Update version number + 1 Update version number + 1
@ -217,6 +218,7 @@ Modified => alpha
make testurls make testurls
== Check REUSE == == Check REUSE ==
make reuse make reuse
@ -226,7 +228,7 @@ https://api.reuse.software/status/git.savannah.gnu.org/git/parallel
== Update Savannah == == Update Savannah ==
file:///home/tange/privat/parallel/doc/release_new_version perl -ne '/^Subject:/..0 and print' < doc/release_new_version | clipboard
Content from release mail: Content from release mail:
@ -236,10 +238,6 @@ Test URLs.
https://savannah.gnu.org/news/approve.php?group=parallel https://savannah.gnu.org/news/approve.php?group=parallel
== Update Freshmeat ==
// FreeCode is dying
http://freshmeat.net/projects/parallel/releases/new
== Update Mastodon Twitter == == Update Mastodon Twitter ==
@ -252,6 +250,7 @@ See the intro videos pi.dk/1
GNU Parallel - for people who live life in the parallel lane. GNU Parallel - for people who live life in the parallel lane.
== Send announce == == Send announce ==
http://groups.google.com/group/comp.unix.shell/post http://groups.google.com/group/comp.unix.shell/post
@ -268,25 +267,30 @@ from:tange@gnu.org
to:parallel@gnu.org, bug-parallel@gnu.org to:parallel@gnu.org, bug-parallel@gnu.org
stable-bcc: Jesse Alama <jessealama@fastmail.fm> stable-bcc: Jesse Alama <jessealama@fastmail.fm>
Subject: GNU Parallel 20240922 ('Gold Apollo AR924') released Subject: GNU Parallel 20241022 ('Sinwar Nasrallah') released
GNU Parallel 20240922 ('Gold Apollo AR924') has been released. It is available for download at: lbry://@GnuParallel:4 GNU Parallel 20241022 ('Sinwar Nasrallah') has been released. It is available for download at: lbry://@GnuParallel:4
Quote of the month: Quote of the month:
Recently executed a flawless live data migration of ~2.4pb using GNU parallel for scale and bash scripts. GNU Parallel is one of the most helpful tools I've been using recently, and it's just something like: parallel -j4 'gzip {}' ::: folder/*.csv
-- @mechanicker@twitter Dhruva -- Milton Pividori @miltondp@twitter
New in this release: New in this release:
* --fast disables a lot of functionality to speed up running jobs. * No new features. This is a candidate for a stable release.
* Bug fixes and man page updates. * Bug fixes and man page updates.
News about GNU Parallel: News about GNU Parallel:
* Job requiring GNU Parallel knowledge * Separate arguments with a custom separator in GNU Parallel https://boxofcuriosities.co.uk/post/separate-arguments-with-a-custom-separator-in-gnu-parallel
https://www.capgemini.com/ca-en/jobs/Id6D4pEBZ6aB2WPS2aAJ/systems-engineer/
* GNU parallel is underrated https://amontalenti.com/2021/11/10/parallel
* Unlocking the Power of Supercomputers: My HPC Adventure with 2800 Cores and GNU Parallel https://augalip.com/2024/03/10/unlocking-the-power-of-supercomputers-my-hpc-adventure-with-2800-cores-and-gnu-parallel/
* Converting WebP Images to PNG Using parallel and dwebp https://bytefreaks.net/gnulinux/bash/converting-webp-images-to-png-using-parallel-and-dwebp
GNU Parallel - For people who live life in the parallel lane. GNU Parallel - For people who live life in the parallel lane.

View file

@ -1,9 +1,10 @@
<directory name="parallel" rev="400" vrev="9" srcmd5="0a74278ca0d9d295fddd6ce59053a55d"> <directory name="parallel" rev="402" vrev="3" srcmd5="eabde6986e84a2ee89d3573226212c2b">
<entry name="Makefile" md5="d62bf3bdf277f9787ce915e6d1309317" size="1254" mtime="1727031498" /> <entry name="Makefile" md5="d62bf3bdf277f9787ce915e6d1309317" size="1254" mtime="1727031498" />
<entry name="PKGBUILD" md5="5de38e88564fd456a4a955e1fd7a9e99" size="795" mtime="1727031498" /> <entry name="PKGBUILD" md5="d0637068b5eb4d647b3e281a32c4a383" size="795" mtime="1729438421" />
<entry name="parallel-20240922.tar.bz2" md5="ede9c532b8a381cca11b8756b6f45146" size="2675527" mtime="1727031499" /> <entry name="parallel-20241022.tar.bz2" md5="d6b236cd452959c5f8e1740f9d6bd52f" size="2677432" mtime="1729438421" />
<entry name="parallel-20240922.tar.bz2.sig" md5="03e191a290803ab56fa983b6bf3f8f34" size="2080" mtime="1727031499" /> <entry name="parallel-20241022.tar.bz2.sig" md5="310b1d88f6413d6f3126be2eb55514ba" size="2080" mtime="1729438421" />
<entry name="parallel.spec" md5="c08379d0ce10a09a1b00a76ecdd2108c" size="6224" mtime="1727029032" /> <entry name="parallel.spec" md5="f723f1c49a4f85e96ee2cd0bbf668e77" size="6224" mtime="1729437802" />
<entry name="parallel_20240922.dsc" md5="ea350c690a20c6f0b8f735dd895f0e6c" size="557" mtime="1727030980" /> <entry name="parallel_20240922.tar.gz" md5="e0845f75badfde97790129aa5b793563" size="2917995" mtime="1729437803" />
<entry name="parallel_20240922.tar.gz" md5="b07dcee6799cb75f34ec91683d32f85c" size="2917962" mtime="1727030981" /> <entry name="parallel_20241022.dsc" md5="3f822685e8ab5c249191f2afad054250" size="557" mtime="1729438422" />
<entry name="parallel_20241022.tar.gz" md5="c4c614919c48a7f9718c6de7e00ae681" size="2918710" mtime="1729438422" />
</directory> </directory>

View file

@ -1,7 +1,7 @@
Summary: Shell tool for executing jobs in parallel Summary: Shell tool for executing jobs in parallel
Name: parallel Name: parallel
Version: 20240922 Version: 20241022
Release: 2.1 Release: 2.1
License: GPL-3.0-or-later License: GPL-3.0-or-later
Group: Productivity/File utilities Group: Productivity/File utilities

View file

@ -601,7 +601,7 @@ _parset_main() {
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
# shellcheck disable=SC2006 # shellcheck disable=SC2006
echo "parset 20240922 (GNU parallel `parallel --minversion 1`)" echo "parset 20241022 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2024 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2024 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -605,7 +605,7 @@ _parset_main() {
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
# shellcheck disable=SC2006 # shellcheck disable=SC2006
echo "parset 20240922 (GNU parallel `parallel --minversion 1`)" echo "parset 20241022 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2024 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2024 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -601,7 +601,7 @@ _parset_main() {
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
# shellcheck disable=SC2006 # shellcheck disable=SC2006
echo "parset 20240922 (GNU parallel `parallel --minversion 1`)" echo "parset 20241022 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2024 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2024 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -601,7 +601,7 @@ _parset_main() {
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
# shellcheck disable=SC2006 # shellcheck disable=SC2006
echo "parset 20240922 (GNU parallel `parallel --minversion 1`)" echo "parset 20241022 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2024 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2024 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -570,7 +570,7 @@ _parset_main() {
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
# shellcheck disable=SC2006 # shellcheck disable=SC2006
echo "parset 20240922 (GNU parallel `parallel --minversion 1`)" echo "parset 20241022 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2024 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2024 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -601,7 +601,7 @@ _parset_main() {
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
# shellcheck disable=SC2006 # shellcheck disable=SC2006
echo "parset 20240922 (GNU parallel `parallel --minversion 1`)" echo "parset 20241022 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2024 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2024 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -601,7 +601,7 @@ _parset_main() {
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
# shellcheck disable=SC2006 # shellcheck disable=SC2006
echo "parset 20240922 (GNU parallel `parallel --minversion 1`)" echo "parset 20241022 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2024 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2024 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -26,7 +26,7 @@
use strict; use strict;
use Getopt::Long; use Getopt::Long;
$Global::progname="niceload"; $Global::progname="niceload";
$Global::version = 20240922; $Global::version = 20241022;
Getopt::Long::Configure("bundling","require_order"); Getopt::Long::Configure("bundling","require_order");
get_options_from_array(\@ARGV) || die_usage(); get_options_from_array(\@ARGV) || die_usage();
if($opt::version) { if($opt::version) {

View file

@ -2888,7 +2888,7 @@ sub check_invalid_option_combinations() {
sub init_globals() { sub init_globals() {
# Defaults: # Defaults:
$Global::version = 20240922; $Global::version = 20241022;
$Global::progname = 'parallel'; $Global::progname = 'parallel';
$::name = "GNU Parallel"; $::name = "GNU Parallel";
$Global::infinity = 2**31; $Global::infinity = 2**31;
@ -5946,8 +5946,8 @@ sub usage() {
"If you use programs that use GNU Parallel to process data for an article in a", "If you use programs that use GNU Parallel to process data for an article in a",
"scientific publication, please cite:", "scientific publication, please cite:",
"", "",
" Tange, O. (2024, September 22). GNU Parallel 20240922 ('Gold Apollo AR924').", " Tange, O. (2024, October 22). GNU Parallel 20241022 ('Sinwar Nasrallah').",
" Zenodo. https://doi.org/10.5281/zenodo.13826092", " Zenodo. https://doi.org/10.5281/zenodo.13957646",
"", "",
# Before changing these lines, please read # Before changing these lines, please read
# https://www.gnu.org/software/parallel/parallel_design.html#citation-notice # https://www.gnu.org/software/parallel/parallel_design.html#citation-notice
@ -5979,8 +5979,8 @@ sub citation_notice() {
"If you use programs that use GNU Parallel to process data for an article in a", "If you use programs that use GNU Parallel to process data for an article in a",
"scientific publication, please cite:", "scientific publication, please cite:",
"", "",
" Tange, O. (2024, September 22). GNU Parallel 20240922 ('Gold Apollo AR924').", " Tange, O. (2024, October 22). GNU Parallel 20241022 ('Sinwar Nasrallah').",
" Zenodo. https://doi.org/10.5281/zenodo.13826092", " Zenodo. https://doi.org/10.5281/zenodo.13957646",
"", "",
# Before changing these line, please read # Before changing these line, please read
# https://www.gnu.org/software/parallel/parallel_design.html#citation-notice and # https://www.gnu.org/software/parallel/parallel_design.html#citation-notice and
@ -6106,20 +6106,20 @@ sub citation() {
"If you use programs that use GNU Parallel to process data for an article in a", "If you use programs that use GNU Parallel to process data for an article in a",
"scientific publication, please cite:", "scientific publication, please cite:",
"", "",
"\@software{tange_2024_13826092,", "\@software{tange_2024_13957646,",
" author = {Tange, Ole},", " author = {Tange, Ole},",
" title = {GNU Parallel 20240922 ('Gold Apollo AR924')},", " title = {GNU Parallel 20241022 ('Sinwar Nasrallah')},",
" month = Sep,", " month = Oct,",
" year = 2024,", " year = 2024,",
" note = {{GNU Parallel is a general parallelizer to run", " note = {{GNU Parallel is a general parallelizer to run",
" multiple serial command line programs in parallel", " multiple serial command line programs in parallel",
" without changing them.}},", " without changing them.}},",
" publisher = {Zenodo},", " publisher = {Zenodo},",
" doi = {10.5281/zenodo.13826092},", " doi = {10.5281/zenodo.13957646},",
" url = {https://doi.org/10.5281/zenodo.13826092}", " url = {https://doi.org/10.5281/zenodo.13957646}",
"}", "}",
"", "",
"(Feel free to use \\nocite{tange_2024_13826092})", "(Feel free to use \\nocite{tange_2024_13957646})",
"", "",
# Before changing these lines, please read # Before changing these lines, please read
# https://www.gnu.org/software/parallel/parallel_design.html#citation-notice and # https://www.gnu.org/software/parallel/parallel_design.html#citation-notice and
@ -7376,26 +7376,90 @@ sub new($$) {
my $local; my $local;
my $string; my $string;
# SSHLogins can have these formats: # SSHLogins can have these formats:
# @grp+grp/ncpu//usr/bin/ssh user@server # @grp1+grp2/ => use only servers in @grp1+grp2
# ncpu//usr/bin/ssh user@server # [@grp+grp/][ncpu/][ssh command ][[user][:password]@][host[:port]]
# /usr/bin/ssh user@server #
# user@server # Here most combinations are generated:
# ncpu/user@server # grp=grp1+grp2
# @grp+grp/user@server # ncpu=4
# above with: user:password@server # ssh=/usr/bin/ssh
# above with: user@server:port # user=user
# So: # pass=pass
# [@grp+grp][ncpu/][ssh command ][[user][:password]@][server[:port]] # host=host
# port=port
# parallel -k echo {1}{2}{3}{4}{5}{=1'$_ = ($arg[4]||$arg[5]) ? "\@" : ""' =}$host{6} ::: '' @$grp/ ::: '' $ncpu/ ::: '' $ssh' ' ::: '' $user ::: '' :$pass ::: '' :$port
# host
# host:port
# :pass@host
# :pass@host:port
# user@host
# user@host:port
# user:pass@host
# user:pass@host:port
# /usr/bin/ssh host
# /usr/bin/ssh host:port
# /usr/bin/ssh :pass@host
# /usr/bin/ssh :pass@host:port
# /usr/bin/ssh user@host
# /usr/bin/ssh user@host:port
# /usr/bin/ssh user:pass@host
# /usr/bin/ssh user:pass@host:port
# ncpu/host
# ncpu/host:port
# ncpu/:pass@host
# ncpu/:pass@host:port
# ncpu/user@host
# ncpu/user@host:port
# ncpu/user:pass@host
# ncpu/user:pass@host:port
# ncpu//usr/bin/ssh host
# ncpu//usr/bin/ssh host:port
# ncpu//usr/bin/ssh :pass@host
# ncpu//usr/bin/ssh :pass@host:port
# ncpu//usr/bin/ssh user@host
# ncpu//usr/bin/ssh user@host:port
# ncpu//usr/bin/ssh user:pass@host
# ncpu//usr/bin/ssh user:pass@host:port
# @grp1+grp2/host
# @grp1+grp2/host:port
# @grp1+grp2/:pass@host
# @grp1+grp2/:pass@host:port
# @grp1+grp2/user@host
# @grp1+grp2/user@host:port
# @grp1+grp2/user:pass@host
# @grp1+grp2/user:pass@host:port
# @grp1+grp2//usr/bin/ssh host
# @grp1+grp2//usr/bin/ssh host:port
# @grp1+grp2//usr/bin/ssh :pass@host
# @grp1+grp2//usr/bin/ssh :pass@host:port
# @grp1+grp2//usr/bin/ssh user@host
# @grp1+grp2//usr/bin/ssh user@host:port
# @grp1+grp2//usr/bin/ssh user:pass@host
# @grp1+grp2//usr/bin/ssh user:pass@host:port
# @grp1+grp2/ncpu/host
# @grp1+grp2/ncpu/host:port
# @grp1+grp2/ncpu/:pass@host
# @grp1+grp2/ncpu/:pass@host:port
# @grp1+grp2/ncpu/user@host
# @grp1+grp2/ncpu/user@host:port
# @grp1+grp2/ncpu/user:pass@host
# @grp1+grp2/ncpu/user:pass@host:port
# @grp1+grp2/ncpu//usr/bin/ssh host
# @grp1+grp2/ncpu//usr/bin/ssh host:port
# @grp1+grp2/ncpu//usr/bin/ssh :pass@host
# @grp1+grp2/ncpu//usr/bin/ssh :pass@host:port
# @grp1+grp2/ncpu//usr/bin/ssh user@host
# @grp1+grp2/ncpu//usr/bin/ssh user@host:port
# @grp1+grp2/ncpu//usr/bin/ssh user:pass@host
# @grp1+grp2/ncpu//usr/bin/ssh user:pass@host:port
# [@grp+grp]/ncpu//usr/bin/ssh user:pass@server:port # [@grp+grp/][ncpu/][ssh command ][[user][:password]@][host[:port]]
if($s =~ s:^\@([^/]+)/?::) { if($s =~ s:^\@([^/]+)/::) {
# Look for SSHLogin hostgroups # Look for SSHLogin hostgroups
%hostgroups = map { $_ => 1 } split(/\+|,/, $1); %hostgroups = map { $_ => 1 } split(/\+|,/, $1);
} }
# An SSHLogin is always in the hostgroup of its "numcpu/host"
$hostgroups{$s} = 1;
# [ncpu/]/usr/bin/ssh user:pass@server:port # [ncpu/][ssh command ][[user][:password]@][host[:port]]
if ($s =~ s:^(\d+)/::) { $ncpus = $1; } if ($s =~ s:^(\d+)/::) { $ncpus = $1; }
# Why disallow space in password? # Why disallow space in password?
@ -7407,11 +7471,13 @@ sub new($$) {
# cmd 'C:/bin/ssh' user 'user' with password 'C:/bin/ssh' # cmd 'C:/bin/ssh' user 'user' with password 'C:/bin/ssh'
# This is impossible to determine. # This is impossible to determine.
# With space forbidden in password it uniquely parses as the 2nd. # With space forbidden in password it uniquely parses as the 2nd.
# [/usr/bin/ssh ]user:pass@server:port # [ssh command ][[user][:password]@][host[:port]]
if($s =~ s/^(.*) //) { $sshcommand = $1; } if($s =~ s/^(.*) //) { $sshcommand = $1; }
# [user:pass@]server:port # [[user][:password]@][host[:port]]
if($s =~ s/^(.+)@//) { # An SSHLogin is always in the hostgroup of its "user:pass@host:port"
$hostgroups{$s} = 1;
if($s =~ s/^(.*)@//) {
my $userpw = $1; my $userpw = $1;
# user[:pass] # user[:pass]
if($userpw =~ s/:(.*)//) { if($userpw =~ s/:(.*)//) {
@ -7422,9 +7488,25 @@ sub new($$) {
::wait_and_exit(255); ::wait_and_exit(255);
} }
} }
# This might be empty - that is OK
$user = $userpw; $user = $userpw;
} }
# [server]:port # [host[:port]]
# host can have these formats:
# hostname (: as local is dealt with later)
# IPv4 i.p.n.o
# with port
# * :22
# * :ssh
# IPv6:
# * \[b11010000011101]
# * \[o64072/14]
# * \[xd074/14]
# * \[208.116.0.0/14]
# with port
# * .22
# * p22
# * #22
if(not $s =~ /:.*:/ if(not $s =~ /:.*:/
and and
$s =~ s/^([-a-z0-9._]+)//i) { $s =~ s/^([-a-z0-9._]+)//i) {
@ -7446,7 +7528,6 @@ sub new($$) {
# 2001:db8::1 port 80 - not supported # 2001:db8::1 port 80 - not supported
$host = $1; $host = $1;
} }
# [:port] # [:port]
if($s =~ s/^:(\w+)//i) { if($s =~ s/^:(\w+)//i) {
$port = $1; $port = $1;
@ -7468,10 +7549,11 @@ sub new($$) {
($user && $user."@"). ($user && $user."@").
($host && $host). ($host && $host).
($port && ":$port"); ($port && ":$port");
my $userpassword = ($user && $user).($password && ":".$password);
my $pwstring = my $pwstring =
# Only include the sshcommand in $string if it is set by user # Only include the sshcommand in $string if it is set by user
($sshcommand && $sshcommand." "). ($sshcommand && $sshcommand." ").
($user && $user. ($password && ":".$password)."@"). ($userpassword && $userpassword."@").
($host && $host). ($host && $host).
($port && ":$port"); ($port && ":$port");
if($host eq ':') { if($host eq ':') {
@ -8871,10 +8953,20 @@ sub sct_gnu_linux($) {
my $threads_per_core; my $threads_per_core;
my $cores_per_socket; my $cores_per_socket;
for(@lscpu) { for(@lscpu) {
# lscpu
/^CPU.s.:\s*(\d+)/ and $cpu->{'threads'} = $1; /^CPU.s.:\s*(\d+)/ and $cpu->{'threads'} = $1;
/^Thread.s. per core:\s*(\d+)/ and $threads_per_core = $1; /^Thread.s. per core:\s*(\d+)/ and $threads_per_core = $1;
/^Core.s. per socket:\s*(\d+)/ and $cores_per_socket = $1; /^Core.s. per socket:\s*(\d+)/ and $cores_per_socket = $1;
/^(CPU )?Socket.s.:\s*(\d+)/i and $cpu->{'sockets'} = $2; /^(CPU )?Socket.s.:\s*(\d+)/i and $cpu->{'sockets'} = $2;
# lscpu --all --extended
# 3 0 0 1 1:1:1:0 yes 2900.0000 800.0000 1995.6210
# 1 0 0 1 1:1:1 yes 1300.0000 800.0000 800.0000
/^\s+(\d+)\s+\d+\s+(\d+)\s+(\d+)\s+\d+:\d+:\S+\s/ and do {
# Really only the last line matters
$cpu->{'threads'} = $1+1;
$cpu->{'sockets'} = $2+1;
$cpu->{'cores'} = $3+1;
};
} }
if($cores_per_socket and $cpu->{'sockets'}) { if($cores_per_socket and $cpu->{'sockets'}) {
$cpu->{'cores'} = $cores_per_socket * $cpu->{'sockets'}; $cpu->{'cores'} = $cores_per_socket * $cpu->{'sockets'};
@ -8896,7 +8988,7 @@ sub sct_gnu_linux($) {
read_cpuinfo(split/(?<=\n)/,$ENV{'PARALLEL_CPUINFO'}); read_cpuinfo(split/(?<=\n)/,$ENV{'PARALLEL_CPUINFO'});
} elsif($ENV{'PARALLEL_LSCPU'}) { } elsif($ENV{'PARALLEL_LSCPU'}) {
# Use LSCPU from environment - used for testing only # Use LSCPU from environment - used for testing only
read_lscpu(split/(?<=\n)/,$ENV{'PARALLEL_LSCPU'}); read_lscpu(split/\n/,$ENV{'PARALLEL_LSCPU'});
} elsif(-r "$ENV{'PARALLEL_CPUPREFIX'}/cpu0/topology/thread_siblings") { } elsif(-r "$ENV{'PARALLEL_CPUPREFIX'}/cpu0/topology/thread_siblings") {
# Use CPUPREFIX from environment - used for testing only # Use CPUPREFIX from environment - used for testing only
read_topology($ENV{'PARALLEL_CPUPREFIX'}); read_topology($ENV{'PARALLEL_CPUPREFIX'});
@ -8906,7 +8998,7 @@ sub sct_gnu_linux($) {
# Not debugging: Look at this computer # Not debugging: Look at this computer
if(!($cpu->{'sockets'} and $cpu->{'cores'} and $cpu->{'threads'}) if(!($cpu->{'sockets'} and $cpu->{'cores'} and $cpu->{'threads'})
and and
open(my $in_fh, "-|", "lscpu")) { open(my $in_fh, "-|", "lscpu --all --extended")) {
# Parse output from lscpu # Parse output from lscpu
read_lscpu(<$in_fh>); read_lscpu(<$in_fh>);
close $in_fh; close $in_fh;

View file

@ -3190,11 +3190,11 @@ See also: TIME POSTFIXES B<--sshlogin> B<--delay>
=item B<--sshlogin> I<[@hostgroups/][ncpus/][[user][:[password]]@]host[:port][,...]> =item B<--sshlogin> I<[@hostgroups/][ncpus/][[user][:[password]]@]host[:port][,...]>
=item B<--sshlogin> I<@hostgroup> =item B<--sshlogin> I<@hostgroup>/
=item B<-S> I<[@hostgroups/][ncpus/][ssh command][[user][:[password]]@]host[:port][,...]> =item B<-S> I<[@hostgroups/][ncpus/][ssh command][[user][:[password]]@]host[:port][,...]>
=item B<-S> I<@hostgroup> =item B<-S> I<@hostgroup>/
Distribute jobs to remote computers. Distribute jobs to remote computers.

View file

@ -137,7 +137,7 @@ GetOptions(
"help" => \$opt::dummy, "help" => \$opt::dummy,
) || exit(255); ) || exit(255);
$Global::progname = ($0 =~ m:(^|/)([^/]+)$:)[1]; $Global::progname = ($0 =~ m:(^|/)([^/]+)$:)[1];
$Global::version = 20240922; $Global::version = 20241022;
if($opt::version) { version(); exit 0; } if($opt::version) { version(); exit 0; }
# Remove -D and --parallel=N # Remove -D and --parallel=N
my @s = (grep { ! /^-D$|^--parallel=\S+$/ } my @s = (grep { ! /^-D$|^--parallel=\S+$/ }

View file

@ -670,7 +670,7 @@ $Global::Initfile && unlink $Global::Initfile;
exit ($err); exit ($err);
sub parse_options { sub parse_options {
$Global::version = 20240922; $Global::version = 20241022;
$Global::progname = 'sql'; $Global::progname = 'sql';
# This must be done first as this may exec myself # This must be done first as this may exec myself

View file

@ -8,6 +8,43 @@
# Each should be taking 30-100s and be possible to run in parallel # Each should be taking 30-100s and be possible to run in parallel
# I.e.: No race conditions, no logins # I.e.: No race conditions, no logins
par_sshlogin_parsing() {
echo '### Generate sshlogins to test parsing'
sudo sshd -p 22222
gen_sshlogin() {
grp=grp1+grp2
ncpu=4
ssh=/usr/bin/ssh
user=parallel
userpass=withpassword
pass="$withpassword"
host=lo
port=22222
# no pass
parallel -k echo \
{1}{2}{3}{4}{5}{=1'$_ = ($arg[4]||$arg[5]) ? "\@" : ""' =}$host{6} \
::: '' @$grp/ ::: '' $ncpu/ ::: '' $ssh' ' \
::: '' $user ::: '' ::: '' :$port
# pass
parallel -k echo \
{1}{2}{3}{4}{5}{=1'$_ = ($arg[4]||$arg[5]) ? "\@" : ""' =}$host{6} \
::: '' @$grp/ ::: '' $ncpu/ ::: '' $ssh' ' \
::: '' $userpass ::: :"$pass" ::: '' :$port
}
doit() {
if parallel -S "$1" {} '$SSH_CLIENT|field 3;whoami' ::: echo ; then
: echo OK
else
echo Fail
fi
}
export -f doit
gen_sshlogin | parallel --tag --timeout 20 -k doit
}
par__print_in_blocks() { par__print_in_blocks() {
echo '### bug #41565: Print happens in blocks - not after each job complete' echo '### bug #41565: Print happens in blocks - not after each job complete'
median() { perl -e '@a=sort {$a<=>$b} <>;print $a[$#a/2]';} median() { perl -e '@a=sort {$a<=>$b} <>;print $a[$#a/2]';}

View file

@ -1369,6 +1369,94 @@ par__test_cpu_detection_lscpu() {
compgen -A function | grep ^cpu | sort | parallel -j0 -k test_one compgen -A function | grep ^cpu | sort | parallel -j0 -k test_one
rm ~/.parallel/tmp/sshlogin/*/cpuspec 2>/dev/null rm ~/.parallel/tmp/sshlogin/*/cpuspec 2>/dev/null
} }
par__test_cpu_detection_lscpu_--all_--extended() {
pack() { zstd -19 | mmencode; }
unpack() { mmencode -u | zstd -d; }
export -f unpack
# ssh server lscpu --all --extended | pack
cpu3() {
echo '1-2-4-2 Core i5-2410M laptop firewall'
echo '
KLUv/QRoBQQA0gYWGHCtDnzPzw76rSjXJHV5twOYLUlfqv43WWq5mbDAYo/P3Rr/alsIFuAc
t+/adX0Jmv63//6KGvO/U8xmmP8fqSFpPZOv5YtYvoflY1Lhl3jAqjwSlZ5BYBEg8ALb2pEO
kGvVfEIAircDrMryswQmQCOci78EmFUDeEAyiFCTmkD1
' | unpack
}
cpu13() {
echo '1-2-2-2 AMD Neo N36L Dual-Core Processor'
echo '
KLUv/QRoTQMAgoURFpBNY2syao8D/vfeGkqOTGuz1NTY0pTEmmc+kEmMAeKQU1L1io8POKT9
0NGeov2vQ5h7C0ynZBole5e3bjZZj3mN53aUFQ4ggALUaxAYHlqAHBGpjAmBGaAzHIBXAmbV
ATDAGAgoKVhdvw==
' | unpack
}
cpu19() {
echo '1-2-4-2 ThinkPad A475 AMD PRO A12-8830B R7 (64g)'
echo '
KLUv/QRoJQQA8kYXGXCtDnzPzw76rSgRSLE7rjbEE0IrrPrfZAGrmTB062XdsOHjM1vXgoH/
xa0fELRjjb/ArsBOSdUffNDxv/+iB2QkD5Fc0mPSejbhxBNGPC+XGt9IJFbmsqj0jhIDEiBg
w3QNIJFcgPBwhAEJIBYx/SYGA8TZTgGYFYADN9xCOVCeG3FI2w==
' | unpack
}
cpu21() {
echo '2-12-24-12 Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz (kreta)'
echo '
KLUv/QRojQYA8gkdGHBvDlztQ2HSU/8ZXvMutLS0iaTrVP01X4yZl5XlxoX71m1btty7ravb
8lJ2fIwND+H2+9vX7s51e2vNzljzmXu/+wN7VdVAldkt6vpbf/0B5ZCMJBKS/y+UYyrL2Wye
Xe3saeeLZhn+FMiwOC+IRTnEDAE4qOFUMuCwz74BUE8tazGNESBEhBMr/mcCUbkH8CBRaVnR
Utmi0lKbdTtxAhKCFxDgq4qrB4fssyd+hzTUliaPXXMlXeivPoZcFv1AtK+bJxn3SVtkgXx6
tCoL/r+Y
' | unpack
}
cpu22() {
echo '1-16-24-16 12900K https://stackoverflow.com/a/71282744/363028'
echo '
KLUv/QRoHQcAcgwiGHBvDlztQ2HSU/8Yv0AX1nZ3bwlO1V/vBfALb+je992vua+73ee2b7ub
tn9/f+rK1a/eejl3egrTrbSMcAULz210bF9oqJ/7/KSwzi5CuM3tub3fbgPSqqqBqhCNWZm9
aqsNMMhimzqWaVhs22AQhWnslG3KXle2umLjaQhHRkI0kI2iYSw5RDaosXQnWFJDZdk3gE2r
AaEIERhChBNZNPGPBKPKPHiLEhWHkWF4HJyLA2JJed1oAHlHMUYzijHSQmXEO4oxmlEtDApN
QezCFCBrSYSKt5G7ixaw9Ml2VwFqvgJs
' | unpack
}
cpu23() {
echo '1-14-20-14 https://github.com/nodejs/node/issues/52233#issuecomment-2023768771'
echo '
KLUv/QRoJQcAMo0jGnBNc3zPzw76LWIb4GzTPebS0iYy0Qqr/ps4mMdssstf1ov5zEWlvvzl
HiblK5eh5Cd3HeQjVyXK9OMfNxnVQT//XHTIzhM+Yc9S97zxxmZC/vqqT+LPT9HYApB7gbt1
KoFM2l97LYApHNo6CIeG4NC2C1NoTNNsd3sE2x/YuiQapYvFoDSKLoOGKQOhBC6ggXgpkEA0
VO13kKWWA5kQ3YTsJmQz37VJpE2QblKQYrhcIOthvksBHXYsUkQpM6SgX0iEoJ4ZIWhvPggs
iRBrnLZwBFTkfvsVAbMDGgiSg60Cro5aVQ==
' | unpack
}
cpu24() {
echo '1-24-32-24 https://in.mathworks.com/matlabcentral/answers/2100291-matlab-parallel-computing-beyond-8-cores-causing-os-to-crash#comment_3113976'
echo '
KLUv/QRoHQsAttY4GoBNOszTAR2eDUbFsPkwkPbee8skydcXNmkPOgAuAC0AeC1g3amamS1m
6ZQiqbS3ttYCEIVDWwjh8Agc2pZBFBpEom6OzSfXPHJNmUVC0AZjECQYZQYJRB0KQU15PkRP
fGIR3rnhDXvslqE8uqNrqp6XleyS3cc7Xo1sdfDqqphGCxcvu3s+ewE9xnRLHjvlKzcx+cn1
rSMfuWh4/OPeSj9dO5ULLafs7NtdKVMdfXQVOukKrzApn7I+ozzKorkA5FsQ2Z38yd66e+Yz
a6icd3YlzGNWoctfNh273S1vWZTLXw5NqIFYSgIJGFnYyr4BcEsrjQEyCSCIAGQIhiBS4Y+E
TSFiA9wB0VQdkzMaTs9TzcD9lry8sZBCrX3I76Ie/D7KiJraeWFJGDH4wPBHl05EiXgccoB2
OzxYwCGuPxjIfC63kHF+GajpnoHBk33Vzo8heiJL/27jCygcJJ+kCrJLTb4=
' | unpack
}
export -f $(compgen -A function | grep ^cpu)
test_one() {
eval $1 | head -n1
export PARALLEL_LSCPU="$(eval $1 | tail -n +2)"
echo $(parallel --number-of-sockets) \
$(parallel --number-of-cores) \
$(parallel --number-of-threads) \
$(parallel --number-of-cpus)
}
export -f test_one
compgen -A function | grep ^cpu | sort | parallel -j0 -k test_one
rm ~/.parallel/tmp/sshlogin/*/cpuspec 2>/dev/null
}
par_combineexec() { par_combineexec() {
combineexec() { combineexec() {

View file

@ -99,18 +99,21 @@ par_sql_CSV() {
} }
par_hostgroup() { par_hostgroup() {
echo '### --hostgroup force ncpu' echo '### --hostgroup force ncpu - 2x parallel, 6x me'
parallel --delay 0.1 --hgrp -S @g1/1/parallel@lo -S @g2/3/lo whoami\;sleep 0.4{} ::: {1..8} | sort -u parallel --delay 0.1 --hgrp -S @g1/1/parallel@lo -S @g2/3/lo \
'whoami;sleep 0.4{}' ::: {1..8} | sort
echo '### --hostgroup two group arg' echo '### --hostgroup two group arg - 2x parallel, 6x me'
parallel -k --sshdelay 0.1 --hgrp -S @g1/1/parallel@lo -S @g2/3/lo whoami\;sleep 0.3{} ::: {1..8}@g1+g2 | sort parallel -k --sshdelay 0.1 --hgrp -S @g1/1/parallel@lo -S @g2/3/lo \
'whoami;sleep 0.3{}' ::: {1..8}@g1+g2 | sort
echo '### --hostgroup one group arg' echo '### --hostgroup one group arg - 8x me'
parallel --delay 0.2 --hgrp -S @g1/1/parallel@lo -S @g2/3/lo whoami\;sleep 0.4{} ::: {1..8}@g2 parallel --delay 0.2 --hgrp -S @g1/1/parallel@lo -S @g2/3/lo \
'whoami;sleep 0.4{}' ::: {1..8}@g2
echo '### --hostgroup multiple group arg + unused group' echo '### --hostgroup multiple group arg + unused group - 2x parallel, 6x me, 0x tcsh'
parallel --delay 0.2 --hgrp -S @g1/1/parallel@lo -S @g1/3/lo -S @g3/100/tcsh@lo whoami\;sleep 0.8{} ::: {1..8}@g1+g2 2>&1 | parallel --delay 0.2 --hgrp -S @g1/1/parallel@lo -S @g1/3/lo -S @g3/30/tcsh@lo \
sort -u | grep -v Warning 'whoami;sleep 0.8{}' ::: {1..8}@g1+g2 2>&1 | sort -u | grep -v Warning
echo '### --hostgroup two groups @' echo '### --hostgroup two groups @'
parallel -k --hgrp -S @g1/parallel@lo -S @g2/lo --tag whoami\;echo ::: parallel@g1 tange@g2 parallel -k --hgrp -S @g1/parallel@lo -S @g2/lo --tag whoami\;echo ::: parallel@g1 tange@g2
@ -121,14 +124,17 @@ par_hostgroup() {
echo '### --hostgroup --sshlogin with @' echo '### --hostgroup --sshlogin with @'
parallel -k --hostgroups -S parallel@lo echo ::: no_group implicit_group@parallel@lo parallel -k --hostgroups -S parallel@lo echo ::: no_group implicit_group@parallel@lo
echo '### --hostgroup -S @group' echo '### --hostgroup -S @group - bad if you get parallel@lo'
parallel -S @g1/ -S @g1/1/tcsh@lo -S @g1/1/localhost -S @g2/1/parallel@lo whoami\;true ::: {1..6} | sort -u parallel -S @g1/ -S @g1/1/tcsh@lo -S @g1/1/localhost -S @g2/1/parallel@lo \
'whoami;true' ::: {1..6} | sort -u
echo '### --hostgroup -S @group1 -Sgrp2' echo '### --hostgroup -S @group1 -Sgrp2 - get all twice'
parallel -S @g1/ -S @g2 -S @g1/1/tcsh@lo -S @g1/1/localhost -S @g2/1/parallel@lo whoami\;true ::: {1..6} | sort parallel -S @g1/ -S @g2/ -S @g1/1/tcsh@lo -S @g1/1/localhost -S @g2/1/parallel@lo \
'whoami;sleep 1;true' ::: {1..6} | sort
echo '### --hostgroup -S @group1+grp2' echo '### --hostgroup -S @group1+grp2 - get all twice'
parallel -S @g1+g2 -S @g1/1/tcsh@lo -S @g1/1/localhost -S @g2/1/parallel@lo whoami\;true ::: {1..6} | sort parallel -S @g1+g2/ -S @g1/1/tcsh@lo -S @g1/1/localhost -S @g2/1/parallel@lo \
'whoami;sleep 1;true' ::: {1..6} | sort
} }
par_PARALLEL_RSYNC_OPTS() { par_PARALLEL_RSYNC_OPTS() {

View file

@ -111,7 +111,7 @@ par_listproc() {
echo "### Test --listproc" echo "### Test --listproc"
sql --listproc :oraunittest | sql --listproc :oraunittest |
perl -ne '/select 1 from dual|user_objects|user_tablespaces|connect_by_filtering/ and next; perl -ne '/select 1 from dual|user_objects|user_tablespaces|connect_by_filtering/ and next;
s/[21 ]\.\d{5}/1.99999/; s/[21 ]\.\d{4,5}/1.99999/;
s/ +/ /g; s/ +/ /g;
print' print'
sql --listproc $MYSQL_TEST_DBURL | sql --listproc $MYSQL_TEST_DBURL |

View file

@ -3058,6 +3058,135 @@ par_slow_total_jobs b
par_slow_total_jobs 1 par_slow_total_jobs 1
par_slow_total_jobs 2 par_slow_total_jobs 2
par_slow_total_jobs parallel: Warning: Reading X arguments took longer than XX seconds. par_slow_total_jobs parallel: Warning: Reading X arguments took longer than XX seconds.
par_sshlogin_parsing ### Generate sshlogins to test parsing
par_sshlogin_parsing lo 22
par_sshlogin_parsing lo tange
par_sshlogin_parsing lo:22222 22222
par_sshlogin_parsing lo:22222 tange
par_sshlogin_parsing parallel@lo 22
par_sshlogin_parsing parallel@lo parallel
par_sshlogin_parsing parallel@lo:22222 22222
par_sshlogin_parsing parallel@lo:22222 parallel
par_sshlogin_parsing /usr/bin/ssh lo 22
par_sshlogin_parsing /usr/bin/ssh lo tange
par_sshlogin_parsing /usr/bin/ssh lo:22222 22222
par_sshlogin_parsing /usr/bin/ssh lo:22222 tange
par_sshlogin_parsing /usr/bin/ssh parallel@lo 22
par_sshlogin_parsing /usr/bin/ssh parallel@lo parallel
par_sshlogin_parsing /usr/bin/ssh parallel@lo:22222 22222
par_sshlogin_parsing /usr/bin/ssh parallel@lo:22222 parallel
par_sshlogin_parsing 4/lo 22
par_sshlogin_parsing 4/lo tange
par_sshlogin_parsing 4/lo:22222 22222
par_sshlogin_parsing 4/lo:22222 tange
par_sshlogin_parsing 4/parallel@lo 22
par_sshlogin_parsing 4/parallel@lo parallel
par_sshlogin_parsing 4/parallel@lo:22222 22222
par_sshlogin_parsing 4/parallel@lo:22222 parallel
par_sshlogin_parsing 4//usr/bin/ssh lo 22
par_sshlogin_parsing 4//usr/bin/ssh lo tange
par_sshlogin_parsing 4//usr/bin/ssh lo:22222 22222
par_sshlogin_parsing 4//usr/bin/ssh lo:22222 tange
par_sshlogin_parsing 4//usr/bin/ssh parallel@lo 22
par_sshlogin_parsing 4//usr/bin/ssh parallel@lo parallel
par_sshlogin_parsing 4//usr/bin/ssh parallel@lo:22222 22222
par_sshlogin_parsing 4//usr/bin/ssh parallel@lo:22222 parallel
par_sshlogin_parsing @grp1+grp2/lo 22
par_sshlogin_parsing @grp1+grp2/lo tange
par_sshlogin_parsing @grp1+grp2/lo:22222 22222
par_sshlogin_parsing @grp1+grp2/lo:22222 tange
par_sshlogin_parsing @grp1+grp2/parallel@lo 22
par_sshlogin_parsing @grp1+grp2/parallel@lo parallel
par_sshlogin_parsing @grp1+grp2/parallel@lo:22222 22222
par_sshlogin_parsing @grp1+grp2/parallel@lo:22222 parallel
par_sshlogin_parsing @grp1+grp2//usr/bin/ssh lo 22
par_sshlogin_parsing @grp1+grp2//usr/bin/ssh lo tange
par_sshlogin_parsing @grp1+grp2//usr/bin/ssh lo:22222 22222
par_sshlogin_parsing @grp1+grp2//usr/bin/ssh lo:22222 tange
par_sshlogin_parsing @grp1+grp2//usr/bin/ssh parallel@lo 22
par_sshlogin_parsing @grp1+grp2//usr/bin/ssh parallel@lo parallel
par_sshlogin_parsing @grp1+grp2//usr/bin/ssh parallel@lo:22222 22222
par_sshlogin_parsing @grp1+grp2//usr/bin/ssh parallel@lo:22222 parallel
par_sshlogin_parsing @grp1+grp2/4/lo 22
par_sshlogin_parsing @grp1+grp2/4/lo tange
par_sshlogin_parsing @grp1+grp2/4/lo:22222 22222
par_sshlogin_parsing @grp1+grp2/4/lo:22222 tange
par_sshlogin_parsing @grp1+grp2/4/parallel@lo 22
par_sshlogin_parsing @grp1+grp2/4/parallel@lo parallel
par_sshlogin_parsing @grp1+grp2/4/parallel@lo:22222 22222
par_sshlogin_parsing @grp1+grp2/4/parallel@lo:22222 parallel
par_sshlogin_parsing @grp1+grp2/4//usr/bin/ssh lo 22
par_sshlogin_parsing @grp1+grp2/4//usr/bin/ssh lo tange
par_sshlogin_parsing @grp1+grp2/4//usr/bin/ssh lo:22222 22222
par_sshlogin_parsing @grp1+grp2/4//usr/bin/ssh lo:22222 tange
par_sshlogin_parsing @grp1+grp2/4//usr/bin/ssh parallel@lo 22
par_sshlogin_parsing @grp1+grp2/4//usr/bin/ssh parallel@lo parallel
par_sshlogin_parsing @grp1+grp2/4//usr/bin/ssh parallel@lo:22222 22222
par_sshlogin_parsing @grp1+grp2/4//usr/bin/ssh parallel@lo:22222 parallel
par_sshlogin_parsing ::;`echo>/tmp/trap`;)(|<*&"'@lo 22
par_sshlogin_parsing ::;`echo>/tmp/trap`;)(|<*&"'@lo tange
par_sshlogin_parsing ::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 22222
par_sshlogin_parsing ::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 tange
par_sshlogin_parsing withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo 22
par_sshlogin_parsing withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo withpassword
par_sshlogin_parsing withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 22222
par_sshlogin_parsing withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 withpassword
par_sshlogin_parsing /usr/bin/ssh ::;`echo>/tmp/trap`;)(|<*&"'@lo 22
par_sshlogin_parsing /usr/bin/ssh ::;`echo>/tmp/trap`;)(|<*&"'@lo tange
par_sshlogin_parsing /usr/bin/ssh ::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 22222
par_sshlogin_parsing /usr/bin/ssh ::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 tange
par_sshlogin_parsing /usr/bin/ssh withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo 22
par_sshlogin_parsing /usr/bin/ssh withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo withpassword
par_sshlogin_parsing /usr/bin/ssh withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 22222
par_sshlogin_parsing /usr/bin/ssh withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 withpassword
par_sshlogin_parsing 4/::;`echo>/tmp/trap`;)(|<*&"'@lo 22
par_sshlogin_parsing 4/::;`echo>/tmp/trap`;)(|<*&"'@lo tange
par_sshlogin_parsing 4/::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 22222
par_sshlogin_parsing 4/::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 tange
par_sshlogin_parsing 4/withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo 22
par_sshlogin_parsing 4/withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo withpassword
par_sshlogin_parsing 4/withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 22222
par_sshlogin_parsing 4/withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 withpassword
par_sshlogin_parsing 4//usr/bin/ssh ::;`echo>/tmp/trap`;)(|<*&"'@lo 22
par_sshlogin_parsing 4//usr/bin/ssh ::;`echo>/tmp/trap`;)(|<*&"'@lo tange
par_sshlogin_parsing 4//usr/bin/ssh ::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 22222
par_sshlogin_parsing 4//usr/bin/ssh ::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 tange
par_sshlogin_parsing 4//usr/bin/ssh withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo 22
par_sshlogin_parsing 4//usr/bin/ssh withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo withpassword
par_sshlogin_parsing 4//usr/bin/ssh withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 22222
par_sshlogin_parsing 4//usr/bin/ssh withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 withpassword
par_sshlogin_parsing @grp1+grp2/::;`echo>/tmp/trap`;)(|<*&"'@lo 22
par_sshlogin_parsing @grp1+grp2/::;`echo>/tmp/trap`;)(|<*&"'@lo tange
par_sshlogin_parsing @grp1+grp2/::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 22222
par_sshlogin_parsing @grp1+grp2/::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 tange
par_sshlogin_parsing @grp1+grp2/withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo 22
par_sshlogin_parsing @grp1+grp2/withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo withpassword
par_sshlogin_parsing @grp1+grp2/withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 22222
par_sshlogin_parsing @grp1+grp2/withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 withpassword
par_sshlogin_parsing @grp1+grp2//usr/bin/ssh ::;`echo>/tmp/trap`;)(|<*&"'@lo 22
par_sshlogin_parsing @grp1+grp2//usr/bin/ssh ::;`echo>/tmp/trap`;)(|<*&"'@lo tange
par_sshlogin_parsing @grp1+grp2//usr/bin/ssh ::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 22222
par_sshlogin_parsing @grp1+grp2//usr/bin/ssh ::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 tange
par_sshlogin_parsing @grp1+grp2//usr/bin/ssh withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo 22
par_sshlogin_parsing @grp1+grp2//usr/bin/ssh withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo withpassword
par_sshlogin_parsing @grp1+grp2//usr/bin/ssh withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 22222
par_sshlogin_parsing @grp1+grp2//usr/bin/ssh withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 withpassword
par_sshlogin_parsing @grp1+grp2/4/::;`echo>/tmp/trap`;)(|<*&"'@lo 22
par_sshlogin_parsing @grp1+grp2/4/::;`echo>/tmp/trap`;)(|<*&"'@lo tange
par_sshlogin_parsing @grp1+grp2/4/::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 22222
par_sshlogin_parsing @grp1+grp2/4/::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 tange
par_sshlogin_parsing @grp1+grp2/4/withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo 22
par_sshlogin_parsing @grp1+grp2/4/withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo withpassword
par_sshlogin_parsing @grp1+grp2/4/withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 22222
par_sshlogin_parsing @grp1+grp2/4/withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 withpassword
par_sshlogin_parsing @grp1+grp2/4//usr/bin/ssh ::;`echo>/tmp/trap`;)(|<*&"'@lo 22
par_sshlogin_parsing @grp1+grp2/4//usr/bin/ssh ::;`echo>/tmp/trap`;)(|<*&"'@lo tange
par_sshlogin_parsing @grp1+grp2/4//usr/bin/ssh ::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 22222
par_sshlogin_parsing @grp1+grp2/4//usr/bin/ssh ::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 tange
par_sshlogin_parsing @grp1+grp2/4//usr/bin/ssh withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo 22
par_sshlogin_parsing @grp1+grp2/4//usr/bin/ssh withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo withpassword
par_sshlogin_parsing @grp1+grp2/4//usr/bin/ssh withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 22222
par_sshlogin_parsing @grp1+grp2/4//usr/bin/ssh withpassword::;`echo>/tmp/trap`;)(|<*&"'@lo:22222 withpassword
par_test_detected_shell ### bug #42913: Dont use $SHELL but the shell currently running par_test_detected_shell ### bug #42913: Dont use $SHELL but the shell currently running
par_test_detected_shell test_unknown_shell bash Global::shell /usr/bin/bash par_test_detected_shell test_unknown_shell bash Global::shell /usr/bin/bash
par_test_detected_shell test_unknown_shell csh Global::shell /usr/bin/bash par_test_detected_shell test_unknown_shell csh Global::shell /usr/bin/bash

View file

@ -672,6 +672,20 @@ par__test_cpu_detection_lscpu 1-8-8-8 Huawei P Smart Octa-core (4x2.36 GHz Corte
par__test_cpu_detection_lscpu 2 8 8 8 par__test_cpu_detection_lscpu 2 8 8 8
par__test_cpu_detection_lscpu 1-4-4-4 x96 quad-core Android TV-box par__test_cpu_detection_lscpu 1-4-4-4 x96 quad-core Android TV-box
par__test_cpu_detection_lscpu 1 1 1 1 par__test_cpu_detection_lscpu 1 1 1 1
par__test_cpu_detection_lscpu_--all_--extended 1-2-2-2 AMD Neo N36L Dual-Core Processor
par__test_cpu_detection_lscpu_--all_--extended 1 2 2 2
par__test_cpu_detection_lscpu_--all_--extended 1-2-4-2 ThinkPad A475 AMD PRO A12-8830B R7 (64g)
par__test_cpu_detection_lscpu_--all_--extended 1 2 4 2
par__test_cpu_detection_lscpu_--all_--extended 2-12-24-12 Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz (kreta)
par__test_cpu_detection_lscpu_--all_--extended 2 12 24 12
par__test_cpu_detection_lscpu_--all_--extended 1-16-24-16 12900K https://stackoverflow.com/a/71282744/363028
par__test_cpu_detection_lscpu_--all_--extended 1 16 24 16
par__test_cpu_detection_lscpu_--all_--extended 1-14-20-14 https://github.com/nodejs/node/issues/52233#issuecomment-2023768771
par__test_cpu_detection_lscpu_--all_--extended 1 14 20 14
par__test_cpu_detection_lscpu_--all_--extended 1-24-32-24 https://in.mathworks.com/matlabcentral/answers/2100291-matlab-parallel-computing-beyond-8-cores-causing-os-to-crash#comment_3113976
par__test_cpu_detection_lscpu_--all_--extended 1 24 32 24
par__test_cpu_detection_lscpu_--all_--extended 1-2-4-2 Core i5-2410M laptop firewall
par__test_cpu_detection_lscpu_--all_--extended 1 2 4 2
par__test_cpu_detection_topology 4-64-64-64 Dell R815 4 CPU 64-core par__test_cpu_detection_topology 4-64-64-64 Dell R815 4 CPU 64-core
par__test_cpu_detection_topology 4 8 8 8 par__test_cpu_detection_topology 4 8 8 8
par__test_cpu_detection_topology 1-2-2-2 AMD Neo N36L Dual-Core Processor par__test_cpu_detection_topology 1-2-2-2 AMD Neo N36L Dual-Core Processor