tange/parallel
1
0
Fork 0
mirror of https://git.savannah.gnu.org/git/parallel.git synced 2024-11-22 05:57:54 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Added *.rst format of documentation.

Browse source
This commit is contained in:
Ole Tange 2021-02-17 15:49:47 +01:00
parent 0b44a9ceb2
commit 0753d4348d
37 changed files with 917 additions and 328 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
README
View file

@ -96,7 +96,7 @@ system is old or Microsoft Windows):
If you are developing your script to run on a remote server, that does If you are developing your script to run on a remote server, that does
not have GNU Parallel installed, but you have it installed on you not have GNU Parallel installed, but you have it installed on you
development machine, the use can use `parallel --embed`. development machine, then you can use `parallel --embed`.
parallel --embed > newscript.sh parallel --embed > newscript.sh
@ -126,8 +126,8 @@ publication please cite:
Zenodo. https://doi.org/10.5281/zenodo.4454976 Zenodo. https://doi.org/10.5281/zenodo.4454976
Copyright (C) 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, Copyright (C) 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015,
2016, 2017, 2018, 2019, 2020 Ole Tange, http://ole.tange.dk and Free 2016, 2017, 2018, 2019, 2020, 2021 Ole Tange, http://ole.tange.dk and
Software Foundation, Inc. Free Software Foundation, Inc.
= New versions = = New versions =

6
doc/citation-notice-faq.txt
View file

@ -146,13 +146,13 @@ removed (and staying in main).
In other words: It is preferable having fewer users, who all know they In other words: It is preferable having fewer users, who all know they
should cite, over having many users, who do not know they should cite. should cite, over having many users, who do not know they should cite.
If the goal had been to get more users, then the license would have
been public domain.
This is because a long-term survival with funding is more important This is because a long-term survival with funding is more important
than short-term gains in popularity that can be achieved by being than short-term gains in popularity that can be achieved by being
distributed as part of a distribution. distributed as part of a distribution.
If the goal had been to get more users, then the license would have
been public domain.
> Is there another way I can get rid of the citation notice? > Is there another way I can get rid of the citation notice?

30
doc/haikus
View file

@ -1,5 +1,35 @@
Quote of the month: Quote of the month:
Parallel is amazing!
-- fatboy93@reddit
parallel is my new kink
-- python_noob_001@reddit
GNU Parallel makes my life so much easier.
I'm glad I don't have to implement multi-threaded Python scripts on the regular.
-- Fredrick Brennan @fr_brennan@twitter
If you work with lots of files at once
Take a good look at GNU parallel
Change your life for the better
-- French @notareverser@twitter
@GnuParallel the best thing ever and it's not up for debate #EOchat
-- Nathan Thomas @DrNASApants@twitter
Using [GNU Parallel is] super easy if you use xargs, and it is magic for making things multi-process. Then adding in the ssh magic it can do it is an incredible tool that is completely underutilized.
-- Ancients @Ancients@twitter
GNU Parallel is one of the most helpful tools I've been using recently, and it's just something like: parallel -j4 'gzip {}' ::: folder/*.csv
-- Milton Pividori @miltondp@twitter
We use gnu parallel now - and happier for it.
-- Ben Davies @benjamindavies@twitter
This is a fantastic tool, and I wish I had upgraded from xargs years ago!
-- Stuart Anderson
GNU Parallel and ripgrep would be your friend here. Ripgrep is fast, really fast. GNU Parallel and ripgrep would be your friend here. Ripgrep is fast, really fast.
-- CENAPT @cenaptech@twitter -- CENAPT @cenaptech@twitter

33
doc/release_new_version
View file

@ -194,9 +194,9 @@ from:tange@gnu.org
to:parallel@gnu.org, bug-parallel@gnu.org to:parallel@gnu.org, bug-parallel@gnu.org
stable-bcc: Jesse Alama <jessealama@fastmail.fm> stable-bcc: Jesse Alama <jessealama@fastmail.fm>
Subject: GNU Parallel 20210122 ('Capitol Riots') released <<[stable]>> Subject: GNU Parallel 20210222 ('AngSangSuKyi/Navalny/Håndbold/Larry King<<>>') released <<[stable]>>
GNU Parallel 20210122 ('') <<[stable]>> has been released. It is available for download at: http://ftpmirror.gnu.org/parallel/ GNU Parallel 20210222 ('') <<[stable]>> has been released. It is available for download at: http://ftpmirror.gnu.org/parallel/
<<No new functionality was introduced so this is a good candidate for a stable release.>> <<No new functionality was introduced so this is a good candidate for a stable release.>>
@ -206,32 +206,41 @@ It does not have to be as detailed as Juan's. It is perfectly fine if you just s
Quote of the month: Quote of the month:
I think many people would be surprised to learn that GNU parallel is "just" a single Perl script. <<>>
-- Peter Menzel @ptr_menzel@twitter
New in this release: New in this release:
* --memsuspend suspends jobs when there is little memory free. This way you can run jobs in parallel that added up potentially will take up more RAM than is available. * --filter filter out jobs to run based on a GNU Parallel expression.
* $PARALLEL_ARGHOSTGROUPS and the replacement string {agrp} will give the hostgroup given on the argument when using --hostgroup. * --tmpl <<>>
* --plus implements {0%} {0..0%} dynamic replacement string for slot and {0#} {0..0#} dynamic replacement string for seq
* crnl
* New scheduling code, which makes -S alpha quality.
* Handy time functions for {= =}: yyyy_mm_dd_hh_mm_ss() yyyy_mm_dd_hh_mm() yyyy_mm_dd() yyyymmddhhmmss() yyyymmddhhmm() yyyymmdd() hash($str)
* Bug fixes and man page updates. * Bug fixes and man page updates.
News about GNU Parallel: News about GNU Parallel:
* Software Engineering For Scientists https://canvas.stanford.edu/courses/133091 https://www.polarmicrobes.org/a-short-tutorial-on-gnu-parallel/
* Optional Individual Submission 4 Job Handling - GNU Parallel https://www.youtube.com/watch?v=-tX0bFAYAxM https://opensource.com/article/21/2/linux-programming
* 关于gnu parallel：GnuParallel：在集群上并行化脚本，脚本将文件写入主节点https://www.codenong.com/25172209/ https://medium.com/analytics-vidhya/simple-tutorial-to-install-use-gnu-parallel-79251120d618
* すぐ忘れてしまう、仕事で使う技 https://qiita.com/hana_shin/items/53c3c78525c9c758ae7c https://www.lido.tu-dortmund.de/cms/de/LiDO3/LiDO3_first_contact_handout.pdf
https://blog.jastix.biz/post/rill-stage-2-1-cli-data-analysis/
<<>> https://blog.knoldus.com/introduction-to-gnu-parallel/
https://www.hahwul.com/cullinan/parallel/
* <<>>
Get the book: GNU Parallel 2018 http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html Get the book: GNU Parallel 2018 http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html

19
packager/obs/Makefile
View file

@ -13,13 +13,14 @@ all:
cd home\:tange/parallel/ && osc up cd home\:tange/parallel/ && osc up
cd home\:tange/parallel/ && parallel osc add ::: *.spec *.dsc *.tar.gz *.tar.bz2 && echo Src added OK || true cd home\:tange/parallel/ && parallel osc add ::: *.spec *.dsc *.tar.gz *.tar.bz2 && echo Src added OK || true
cd home\:tange/parallel/ && osc ci -m "New release" cd home\:tange/parallel/ && osc ci -m "New release"
# wait for "building" state ### Wait for "building" state to complete
cd home\:tange/parallel/ && yes building | parallel -j1 --delay 10 --halt now,success=1 'osc results|G -E {}' echo '### Wait for "building" state'
# wait for "building" state to end cd home\:tange/parallel/ && yes building | parallel -vj1 --delay 10 --halt now,success=1 'osc results|G -E {}'
# Ignore RedHat_RHEL-6 and Univention_3.2 that are broken echo '### Wait for "building" state to end'
cd home\:tange/parallel/ && yes building | parallel -j1 --delay 10 --halt now,fail=1 'osc results|G -v RedHat_RHEL-6 -v Univention_3.2 -E {}' || true echo '### Ignore RedHat_RHEL-6 and Univention_3.2 that are broken'
# wait for "finished" state of .deb cd home\:tange/parallel/ && yes building | parallel -vj1 --delay 10 --halt now,fail=1 'osc results|G -v RedHat_RHEL-6 -v Univention_3.2 -E {}' || true
cd home\:tange/parallel/ && echo succeeded | parallel -j1 --retries 30 --delay 10 --halt now,success=1 'osc results|G -E "(Debian|Ubuntu).*{}"' echo '### Wait for "finished" state of .deb'
# wait for "finished" state of .rpm cd home\:tange/parallel/ && echo succeeded | parallel -vj1 --retries 30 --delay 10 --halt now,success=1 'osc results|G -E "(Debian|Ubuntu).*{}"'
cd home\:tange/parallel/ && echo succeeded | parallel -j1 --retries 30 --delay 10 --halt now,success=1 'osc results|G -E "(SUSE|SLE|Scientific|RHEL|Fedora|CentOS).*{}"' echo '### Wait for "finished" state of .rpm'
cd home\:tange/parallel/ && echo succeeded | parallel -vj1 --retries 30 --delay 10 --halt now,success=1 'osc results|G -E "(SUSE|SLE|Scientific|RHEL|Fedora|CentOS).*{}"'

6
packager/obs/home:tange/parallel/.osc/_files
View file

@ -1,6 +1,6 @@
<directory name="parallel" rev="295" vrev="1" srcmd5="3d5753d873ec7dd1c5e728f166982dd0"> <directory name="parallel" rev="297" vrev="3" srcmd5="d241350faf9509d6b0f6c2cd5e490318">
<entry name="PKGBUILD" md5="84008e34fb54d73054e039551aa5a1b1" size="936" mtime="1611258951" /> <entry name="PKGBUILD" md5="f078fe714342aa680f5babdf52815b63" size="936" mtime="1611260073" />
<entry name="parallel-20210122.tar.bz2" md5="6799fa098597b5a32e87f82f8ef4d8ad" size="2129364" mtime="1611258952" /> <entry name="parallel-20210122.tar.bz2" md5="86e78bbb2d820c2a23bcac06ec902204" size="2129334" mtime="1611260073" />
<entry name="parallel.spec" md5="73c304015393921bb524310ecc3f68da" size="4876" mtime="1611258952" /> <entry name="parallel.spec" md5="73c304015393921bb524310ecc3f68da" size="4876" mtime="1611258952" />
<entry name="parallel_20210122.dsc" md5="327c5825cefbbf7674e1a668e57ef499" size="556" mtime="1611258953" /> <entry name="parallel_20210122.dsc" md5="327c5825cefbbf7674e1a668e57ef499" size="556" mtime="1611258953" />
<entry name="parallel_20210122.tar.gz" md5="f0b8b80919399ef704eb6d00d03f3a68" size="2319784" mtime="1611258953" /> <entry name="parallel_20210122.tar.gz" md5="f0b8b80919399ef704eb6d00d03f3a68" size="2319784" mtime="1611258953" />

52
src/Makefile.am
View file

@ -20,6 +20,10 @@ doc_DATA = parallel.html env_parallel.html sem.html sql.html \
niceload.texi parallel_tutorial.texi parallel_book.texi \ niceload.texi parallel_tutorial.texi parallel_book.texi \
parallel_design.texi parallel_alternatives.texi parcat.texi \ parallel_design.texi parallel_alternatives.texi parcat.texi \
parset.texi parsort.texi \ parset.texi parsort.texi \
parallel.rst env_parallel.rst sem.rst sql.rst \
niceload.rst parallel_tutorial.rst parallel_book.rst \
parallel_design.rst parallel_alternatives.rst parcat.rst \
parset.rst parsort.rst \
parallel.pdf env_parallel.pdf sem.pdf sql.pdf niceload.pdf \ parallel.pdf env_parallel.pdf sem.pdf sql.pdf niceload.pdf \
parallel_tutorial.pdf parallel_book.pdf parallel_design.pdf \ parallel_tutorial.pdf parallel_book.pdf parallel_design.pdf \
parallel_alternatives.pdf parcat.pdf parset.pdf parsort.pdf \ parallel_alternatives.pdf parcat.pdf parset.pdf parsort.pdf \
@ -231,6 +235,54 @@ parsort.texi: parsort
pod2texi --output="$(srcdir)"/parsort.texi "$(srcdir)"/parsort \ pod2texi --output="$(srcdir)"/parsort.texi "$(srcdir)"/parsort \
|| echo "Warning: pod2texi not found. Using old parsort.texi" || echo "Warning: pod2texi not found. Using old parsort.texi"
parallel.rst: parallel.pod
pod2rst --outfile "$(srcdir)"/parallel.rst --infile="$(srcdir)"/parallel.pod \
|| echo "Warning: pod2rst not found. Using old parallel.rst"
env_parallel.rst: env_parallel.pod
pod2rst --outfile "$(srcdir)"/env_parallel.rst --infile="$(srcdir)"/env_parallel.pod \
|| echo "Warning: pod2rst not found. Using old env_parallel.rst"
parallel_tutorial.rst: parallel_tutorial.pod
pod2rst --outfile "$(srcdir)"/parallel_tutorial.rst --infile="$(srcdir)"/parallel_tutorial.pod \
|| echo "Warning: pod2rst not found. Using old parallel_tutorial.rst"
parallel_book.rst: parallel_book.pod
pod2rst --outfile "$(srcdir)"/parallel_book.rst --infile="$(srcdir)"/parallel_book.pod \
|| echo "Warning: pod2rst not found. Using old parallel_book.rst"
parallel_design.rst: parallel_design.pod
pod2rst --outfile "$(srcdir)"/parallel_design.rst --infile="$(srcdir)"/parallel_design.pod \
|| echo "Warning: pod2rst not found. Using old parallel_design.rst"
parallel_alternatives.rst: parallel_alternatives.pod
pod2rst --outfile "$(srcdir)"/parallel_alternatives.rst --infile="$(srcdir)"/parallel_alternatives.pod \
|| echo "Warning: pod2rst not found. Using old parallel_alternatives.rst"
sem.rst: sem.pod
pod2rst --outfile "$(srcdir)"/sem.rst --infile="$(srcdir)"/sem.pod \
|| echo "Warning: pod2rst not found. Using old sem.rst"
sql.rst: sql
pod2rst --outfile "$(srcdir)"/sql.rst --infile="$(srcdir)"/sql \
|| echo "Warning: pod2rst not found. Using old sql.rst"
niceload.rst: niceload.pod
pod2rst --outfile "$(srcdir)"/niceload.rst --infile="$(srcdir)"/niceload.pod \
|| echo "Warning: pod2rst not found. Using old niceload.rst"
parcat.rst: parcat.pod
pod2rst --outfile "$(srcdir)"/parcat.rst --infile="$(srcdir)"/parcat.pod \
|| echo "Warning: pod2rst not found. Using old parcat.rst"
parset.rst: parset.pod
pod2rst --outfile "$(srcdir)"/parset.rst --infile="$(srcdir)"/parset.pod \
|| echo "Warning: pod2rst not found. Using old parset.rst"
parsort.rst: parsort
pod2rst --outfile "$(srcdir)"/parsort.rst --infile="$(srcdir)"/parsort \
|| echo "Warning: pod2rst not found. Using old parsort.rst"
parallel.pdf: parallel.pod parallel.pdf: parallel.pod
pod2pdf --output-file "$(srcdir)"/parallel.pdf "$(srcdir)"/parallel.pod --title "GNU Parallel" \ pod2pdf --output-file "$(srcdir)"/parallel.pdf "$(srcdir)"/parallel.pod --title "GNU Parallel" \
|| echo "Warning: pod2pdf not found. Using old parallel.pdf" || echo "Warning: pod2pdf not found. Using old parallel.pdf"

52
src/Makefile.in
View file

@ -247,6 +247,10 @@ bin_SCRIPTS = parallel sql niceload parcat parset parsort \
@DOCUMENTATION_TRUE@ niceload.texi parallel_tutorial.texi parallel_book.texi \ @DOCUMENTATION_TRUE@ niceload.texi parallel_tutorial.texi parallel_book.texi \
@DOCUMENTATION_TRUE@ parallel_design.texi parallel_alternatives.texi parcat.texi \ @DOCUMENTATION_TRUE@ parallel_design.texi parallel_alternatives.texi parcat.texi \
@DOCUMENTATION_TRUE@ parset.texi parsort.texi \ @DOCUMENTATION_TRUE@ parset.texi parsort.texi \
@DOCUMENTATION_TRUE@ parallel.rst env_parallel.rst sem.rst sql.rst \
@DOCUMENTATION_TRUE@ niceload.rst parallel_tutorial.rst parallel_book.rst \
@DOCUMENTATION_TRUE@ parallel_design.rst parallel_alternatives.rst parcat.rst \
@DOCUMENTATION_TRUE@ parset.rst parsort.rst \
@DOCUMENTATION_TRUE@ parallel.pdf env_parallel.pdf sem.pdf sql.pdf niceload.pdf \ @DOCUMENTATION_TRUE@ parallel.pdf env_parallel.pdf sem.pdf sql.pdf niceload.pdf \
@DOCUMENTATION_TRUE@ parallel_tutorial.pdf parallel_book.pdf parallel_design.pdf \ @DOCUMENTATION_TRUE@ parallel_tutorial.pdf parallel_book.pdf parallel_design.pdf \
@DOCUMENTATION_TRUE@ parallel_alternatives.pdf parcat.pdf parset.pdf parsort.pdf \ @DOCUMENTATION_TRUE@ parallel_alternatives.pdf parcat.pdf parset.pdf parsort.pdf \
@ -826,6 +830,54 @@ parsort.texi: parsort
pod2texi --output="$(srcdir)"/parsort.texi "$(srcdir)"/parsort \ pod2texi --output="$(srcdir)"/parsort.texi "$(srcdir)"/parsort \
|| echo "Warning: pod2texi not found. Using old parsort.texi" || echo "Warning: pod2texi not found. Using old parsort.texi"
parallel.rst: parallel.pod
pod2rst --outfile "$(srcdir)"/parallel.rst --infile="$(srcdir)"/parallel.pod \
|| echo "Warning: pod2rst not found. Using old parallel.rst"
env_parallel.rst: env_parallel.pod
pod2rst --outfile "$(srcdir)"/env_parallel.rst --infile="$(srcdir)"/env_parallel.pod \
|| echo "Warning: pod2rst not found. Using old env_parallel.rst"
parallel_tutorial.rst: parallel_tutorial.pod
pod2rst --outfile "$(srcdir)"/parallel_tutorial.rst --infile="$(srcdir)"/parallel_tutorial.pod \
|| echo "Warning: pod2rst not found. Using old parallel_tutorial.rst"
parallel_book.rst: parallel_book.pod
pod2rst --outfile "$(srcdir)"/parallel_book.rst --infile="$(srcdir)"/parallel_book.pod \
|| echo "Warning: pod2rst not found. Using old parallel_book.rst"
parallel_design.rst: parallel_design.pod
pod2rst --outfile "$(srcdir)"/parallel_design.rst --infile="$(srcdir)"/parallel_design.pod \
|| echo "Warning: pod2rst not found. Using old parallel_design.rst"
parallel_alternatives.rst: parallel_alternatives.pod
pod2rst --outfile "$(srcdir)"/parallel_alternatives.rst --infile="$(srcdir)"/parallel_alternatives.pod \
|| echo "Warning: pod2rst not found. Using old parallel_alternatives.rst"
sem.rst: sem.pod
pod2rst --outfile "$(srcdir)"/sem.rst --infile="$(srcdir)"/sem.pod \
|| echo "Warning: pod2rst not found. Using old sem.rst"
sql.rst: sql
pod2rst --outfile "$(srcdir)"/sql.rst --infile="$(srcdir)"/sql \
|| echo "Warning: pod2rst not found. Using old sql.rst"
niceload.rst: niceload.pod
pod2rst --outfile "$(srcdir)"/niceload.rst --infile="$(srcdir)"/niceload.pod \
|| echo "Warning: pod2rst not found. Using old niceload.rst"
parcat.rst: parcat.pod
pod2rst --outfile "$(srcdir)"/parcat.rst --infile="$(srcdir)"/parcat.pod \
|| echo "Warning: pod2rst not found. Using old parcat.rst"
parset.rst: parset.pod
pod2rst --outfile "$(srcdir)"/parset.rst --infile="$(srcdir)"/parset.pod \
|| echo "Warning: pod2rst not found. Using old parset.rst"
parsort.rst: parsort
pod2rst --outfile "$(srcdir)"/parsort.rst --infile="$(srcdir)"/parsort \
|| echo "Warning: pod2rst not found. Using old parsort.rst"
parallel.pdf: parallel.pod parallel.pdf: parallel.pod
pod2pdf --output-file "$(srcdir)"/parallel.pdf "$(srcdir)"/parallel.pod --title "GNU Parallel" \ pod2pdf --output-file "$(srcdir)"/parallel.pdf "$(srcdir)"/parallel.pod --title "GNU Parallel" \
|| echo "Warning: pod2pdf not found. Using old parallel.pdf" || echo "Warning: pod2pdf not found. Using old parallel.pdf"

2
src/env_parallel.ash
View file

@ -382,7 +382,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210122 (GNU parallel `parallel --minversion 1`)" echo "parset 20210123 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

2
src/env_parallel.bash
View file

@ -384,7 +384,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210122 (GNU parallel `parallel --minversion 1`)" echo "parset 20210123 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

2
src/env_parallel.dash
View file

@ -382,7 +382,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210122 (GNU parallel `parallel --minversion 1`)" echo "parset 20210123 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

2
src/env_parallel.ksh
View file

@ -365,7 +365,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210122 (GNU parallel `parallel --minversion 1`)" echo "parset 20210123 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

2
src/env_parallel.mksh
View file

@ -368,7 +368,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210122 (GNU parallel `parallel --minversion 1`)" echo "parset 20210123 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

2
src/env_parallel.sh
View file

@ -382,7 +382,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210122 (GNU parallel `parallel --minversion 1`)" echo "parset 20210123 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

2
src/env_parallel.zsh
View file

@ -359,7 +359,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210122 (GNU parallel `parallel --minversion 1`)" echo "parset 20210123 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

2
src/niceload
View file

@ -23,7 +23,7 @@
use strict; use strict;
use Getopt::Long; use Getopt::Long;
$Global::progname="niceload"; $Global::progname="niceload";
$Global::version = 20210122; $Global::version = 20210123;
Getopt::Long::Configure("bundling","require_order"); Getopt::Long::Configure("bundling","require_order");
get_options_from_array(\@ARGV) || die_usage(); get_options_from_array(\@ARGV) || die_usage();
if($opt::version) { if($opt::version) {

191
src/parallel.pod
View file

@ -729,7 +729,7 @@ Even quoted newlines are parsed correctly:
When used with B<--pipe> only pass full CSV-records. When used with B<--pipe> only pass full CSV-records.
=item B<--delay> I<mytime> (beta testing) =item B<--delay> I<mytime>
Delay starting next job by I<mytime>. GNU B<parallel> will pause Delay starting next job by I<mytime>. GNU B<parallel> will pause
I<mytime> after starting each job. I<mytime> is normally in seconds, I<mytime> after starting each job. I<mytime> is normally in seconds,
@ -863,6 +863,16 @@ Implies B<--pipe> unless B<--pipepart> is used.
See also: B<--cat>. See also: B<--cat>.
=item B<--filter> I<filter> (alpha testing)
Only run jobs where I<filter> is true. I<filter> can contain
replacement strings and Perl code. Example:
parallel --filter '{1} < {2}' echo ::: {1..3} ::: {1..3}
Runs: 1,2 1,3 2,3
=item B<--filter-hosts> =item B<--filter-hosts>
Remove down hosts. For each remote host: check that login through ssh Remove down hosts. For each remote host: check that login through ssh
@ -901,7 +911,7 @@ B<--group> is the default. Can be reversed with B<-u>.
See also: B<--line-buffer> B<--ungroup> See also: B<--line-buffer> B<--ungroup>
=item B<--group-by> I<val> (beta testing) =item B<--group-by> I<val>
Group input by value. Combined with B<--pipe>/B<--pipepart> Group input by value. Combined with B<--pipe>/B<--pipepart>
B<--group-by> groups lines with the same value into a record. B<--group-by> groups lines with the same value into a record.
@ -1091,9 +1101,9 @@ B<--header :> is an alias for B<--header '.*\n'>.
If I<regexp> is a number, it is a fixed number of lines. If I<regexp> is a number, it is a fixed number of lines.
=item B<--hostgroups> (alpha testing) =item B<--hostgroups> (beta testing)
=item B<--hgrp> (alpha testing) =item B<--hgrp> (beta testing)
Enable hostgroups on arguments. If an argument contains '@' the string Enable hostgroups on arguments. If an argument contains '@' the string
after '@' will be removed and treated as a list of hostgroups on which after '@' will be removed and treated as a list of hostgroups on which
@ -1430,7 +1440,7 @@ should retry a given job.
See also: B<--memsuspend> See also: B<--memsuspend>
=item B<--memsuspend> I<size> (alpha testing) =item B<--memsuspend> I<size> (beta testing)
Suspend jobs when there is less than 2 * I<size> memory free. The Suspend jobs when there is less than 2 * I<size> memory free. The
I<size> can be postfixed with K, M, G, T, P, k, m, g, t, or p which I<size> can be postfixed with K, M, G, T, P, k, m, g, t, or p which
@ -1540,9 +1550,9 @@ of each job is saved in a file and the filename is then printed.
See also: B<--results> See also: B<--results>
=item B<--pipe> (beta testing) =item B<--pipe>
=item B<--spreadstdin> (beta testing) =item B<--spreadstdin>
Spread input to jobs on stdin (standard input). Read a block of data Spread input to jobs on stdin (standard input). Read a block of data
from stdin (standard input) and give one block of data as input to one from stdin (standard input) and give one block of data as input to one
@ -1615,6 +1625,12 @@ Activate additional replacement strings: {+/} {+.} {+..} {+...} {..}
B<{##}> is the total number of jobs to be run. It is incompatible with B<{##}> is the total number of jobs to be run. It is incompatible with
B<-X>/B<-m>/B<--xargs>. B<-X>/B<-m>/B<--xargs>.
B<{0%}>, B<{00%}>, B<{000%}>, ... B<{0..0%}> prefix jobslot with
zeros. (alpha testing)
B<{0#}>, B<{00#}>, B<{000#}>, ... B<{0..0#}> prefix sequence numbers
with zeros. (alpha testing)
B<{choose_k}> is inspired by n choose k: Given a list of n elements, B<{choose_k}> is inspired by n choose k: Given a list of n elements,
choose k. k is the number of input sources and n is the number of choose k. k is the number of input sources and n is the number of
arguments in an input source. The content of the input sources must arguments in an input source. The content of the input sources must
@ -1906,7 +1922,7 @@ B<-.csv>/B<-.tsv> are special: It will give the file on stdout
(standard output). (standard output).
B<JSON file output> (beta testing) B<JSON file output>
If I<name> ends in B<.json> the output will be a JSON-file If I<name> ends in B<.json> the output will be a JSON-file
named I<name>. named I<name>.
@ -1915,7 +1931,7 @@ B<-.json> is special: It will give the file on stdout (standard
output). output).
B<Replacement string output file> (beta testing) B<Replacement string output file>
If I<name> contains a replacement string and the replaced result does If I<name> contains a replacement string and the replaced result does
not end in /, then the standard output will be stored in a file named not end in /, then the standard output will be stored in a file named
@ -2477,7 +2493,7 @@ be overridden with B<--ssh>. It can also be set on a per server
basis (see B<--sshlogin>). basis (see B<--sshlogin>).
=item B<--sshdelay> I<mytime> (beta testing) =item B<--sshdelay> I<mytime>
Delay starting next ssh by I<mytime>. GNU B<parallel> will not start Delay starting next ssh by I<mytime>. GNU B<parallel> will not start
another ssh for the next I<mytime>. another ssh for the next I<mytime>.
@ -2614,6 +2630,15 @@ Silent. The job to be run will not be printed. This is the default.
Can be reversed with B<-v>. Can be reversed with B<-v>.
=item B<--template> I<file>=I<repl> (alpha testing)
=item B<--tmpl> I<file>=I<repl> (alpha testing)
Copy I<file> to I<repl>. All replacement strings in the contents of
I<file> will be replaced. All replacement strings in the name I<repl>
will be replaced.
=item B<--tty> =item B<--tty>
Open terminal tty. If GNU B<parallel> is used for starting a program Open terminal tty. If GNU B<parallel> is used for starting a program
@ -2975,7 +3000,9 @@ most likely do what is needed.
=back =back
=head1 EXAMPLE: Working as xargs -n1. Argument appending =head1 EXAMPLES
=head2 EXAMPLE: Working as xargs -n1. Argument appending
GNU B<parallel> can work similar to B<xargs -n1>. GNU B<parallel> can work similar to B<xargs -n1>.
@ -2992,7 +3019,7 @@ FUBAR in all files in this dir and subdirs:
Note B<-q> is needed because of the space in 'FOO BAR'. Note B<-q> is needed because of the space in 'FOO BAR'.
=head1 EXAMPLE: Simple network scanner =head2 EXAMPLE: Simple network scanner
B<prips> can generate IP-addresses from CIDR notation. With GNU B<prips> can generate IP-addresses from CIDR notation. With GNU
B<parallel> you can build a simple network scanner to see which B<parallel> you can build a simple network scanner to see which
@ -3003,7 +3030,7 @@ addresses respond to B<ping>:
'ping -c 1 {} >/dev/null && echo {}' 2>/dev/null 'ping -c 1 {} >/dev/null && echo {}' 2>/dev/null
=head1 EXAMPLE: Reading arguments from command line =head2 EXAMPLE: Reading arguments from command line
GNU B<parallel> can take the arguments from command line instead of GNU B<parallel> can take the arguments from command line instead of
stdin (standard input). To compress all html files in the current dir stdin (standard input). To compress all html files in the current dir
@ -3016,7 +3043,7 @@ To convert *.wav to *.mp3 using LAME running one process per CPU run:
parallel lame {} -o {.}.mp3 ::: *.wav parallel lame {} -o {.}.mp3 ::: *.wav
=head1 EXAMPLE: Inserting multiple arguments =head2 EXAMPLE: Inserting multiple arguments
When moving a lot of files like this: B<mv *.log destdir> you will When moving a lot of files like this: B<mv *.log destdir> you will
sometimes get the error: sometimes get the error:
@ -3037,7 +3064,7 @@ In many shells you can also use B<printf>:
printf '%s\0' *.log | parallel -0 -m mv {} destdir printf '%s\0' *.log | parallel -0 -m mv {} destdir
=head1 EXAMPLE: Context replace =head2 EXAMPLE: Context replace
To remove the files I<pict0000.jpg> .. I<pict9999.jpg> you could do: To remove the files I<pict0000.jpg> .. I<pict9999.jpg> you could do:
@ -3059,7 +3086,7 @@ This will also only run B<rm> as many times needed to keep the command
line length short enough. line length short enough.
=head1 EXAMPLE: Compute intensive jobs and substitution =head2 EXAMPLE: Compute intensive jobs and substitution
If ImageMagick is installed this will generate a thumbnail of a jpg If ImageMagick is installed this will generate a thumbnail of a jpg
file: file:
@ -3088,7 +3115,7 @@ make files like ./foo/bar_thumb.jpg:
parallel convert -geometry 120 {} {.}_thumb.jpg parallel convert -geometry 120 {} {.}_thumb.jpg
=head1 EXAMPLE: Substitution and redirection =head2 EXAMPLE: Substitution and redirection
This will generate an uncompressed version of .gz-files next to the .gz-file: This will generate an uncompressed version of .gz-files next to the .gz-file:
@ -3104,7 +3131,7 @@ to be put in quotes, as they may otherwise be interpreted by the shell
and not given to GNU B<parallel>. and not given to GNU B<parallel>.
=head1 EXAMPLE: Composed commands =head2 EXAMPLE: Composed commands
A job can consist of several commands. This will print the number of A job can consist of several commands. This will print the number of
files in each directory: files in each directory:
@ -3139,7 +3166,7 @@ Find the files in a list that do not exist
cat file_list | parallel 'if [ ! -e {} ] ; then echo {}; fi' cat file_list | parallel 'if [ ! -e {} ] ; then echo {}; fi'
=head1 EXAMPLE: Composed command with perl replacement string =head2 EXAMPLE: Composed command with perl replacement string
You have a bunch of file. You want them sorted into dirs. The dir of You have a bunch of file. You want them sorted into dirs. The dir of
each file should be named the first letter of the file name. each file should be named the first letter of the file name.
@ -3147,7 +3174,7 @@ each file should be named the first letter of the file name.
parallel 'mkdir -p {=s/(.).*/$1/=}; mv {} {=s/(.).*/$1/=}' ::: * parallel 'mkdir -p {=s/(.).*/$1/=}; mv {} {=s/(.).*/$1/=}' ::: *
=head1 EXAMPLE: Composed command with multiple input sources =head2 EXAMPLE: Composed command with multiple input sources
You have a dir with files named as 24 hours in 5 minute intervals: You have a dir with files named as 24 hours in 5 minute intervals:
00:00, 00:05, 00:10 .. 23:55. You want to find the files missing: 00:00, 00:05, 00:10 .. 23:55. You want to find the files missing:
@ -3156,7 +3183,7 @@ You have a dir with files named as 24 hours in 5 minute intervals:
::: {00..23} ::: {00..55..5} ::: {00..23} ::: {00..55..5}
=head1 EXAMPLE: Calling Bash functions =head2 EXAMPLE: Calling Bash functions
If the composed command is longer than a line, it becomes hard to If the composed command is longer than a line, it becomes hard to
read. In Bash you can use functions. Just remember to B<export -f> the read. In Bash you can use functions. Just remember to B<export -f> the
@ -3189,7 +3216,7 @@ can copy the full environment without having to B<export -f>
anything. See B<env_parallel>. anything. See B<env_parallel>.
=head1 EXAMPLE: Function tester =head2 EXAMPLE: Function tester
To test a program with different parameters: To test a program with different parameters:
@ -3208,7 +3235,7 @@ If B<my_program> fails a red FAIL will be printed followed by the failing
command; otherwise a green OK will be printed followed by the command. command; otherwise a green OK will be printed followed by the command.
=head1 EXAMPLE: Continously show the latest line of output =head2 EXAMPLE: Continously show the latest line of output
It can be useful to monitor the output of running jobs. It can be useful to monitor the output of running jobs.
@ -3219,7 +3246,7 @@ which the output of the job is printed in full:
3> >(perl -ne '$|=1;chomp;printf"%.'$COLUMNS's\r",$_." "x100') 3> >(perl -ne '$|=1;chomp;printf"%.'$COLUMNS's\r",$_." "x100')
=head1 EXAMPLE: Log rotate =head2 EXAMPLE: Log rotate
Log rotation renames a logfile to an extension with a higher number: Log rotation renames a logfile to an extension with a higher number:
log.1 becomes log.2, log.2 becomes log.3, and so on. The oldest log is log.1 becomes log.2, log.2 becomes log.3, and so on. The oldest log is
@ -3231,7 +3258,7 @@ the log:
mv log log.1 mv log log.1
=head1 EXAMPLE: Removing file extension when processing files =head2 EXAMPLE: Removing file extension when processing files
When processing files removing the file extension using B<{.}> is When processing files removing the file extension using B<{.}> is
often useful. often useful.
@ -3255,7 +3282,7 @@ Put all converted in the same directory:
parallel lame {} -o mydir/{/.}.mp3 parallel lame {} -o mydir/{/.}.mp3
=head1 EXAMPLE: Removing strings from the argument =head2 EXAMPLE: Removing strings from the argument
If you have directory with tar.gz files and want these extracted in If you have directory with tar.gz files and want these extracted in
the corresponding dir (e.g foo.tar.gz will be extracted in the dir the corresponding dir (e.g foo.tar.gz will be extracted in the dir
@ -3277,7 +3304,7 @@ To remove a string anywhere you can use regular expressions with
parallel --plus echo {/demo_/} ::: demo_mycode remove_demo_here parallel --plus echo {/demo_/} ::: demo_mycode remove_demo_here
=head1 EXAMPLE: Download 24 images for each of the past 30 days =head2 EXAMPLE: Download 24 images for each of the past 30 days
Let us assume a website stores images like: Let us assume a website stores images like:
@ -3299,7 +3326,7 @@ B<$(date -d "today -$1 days" +%Y%m%d)> will give the dates in
YYYYMMDD with B<$1> days subtracted. YYYYMMDD with B<$1> days subtracted.
=head1 EXAMPLE: Download world map from NASA =head2 EXAMPLE: Download world map from NASA
NASA provides tiles to download on earthdata.nasa.gov. Download tiles NASA provides tiles to download on earthdata.nasa.gov. Download tiles
for Blue Marble world map and create a 10240x20480 map. for Blue Marble world map and create a 10240x20480 map.
@ -3317,7 +3344,7 @@ for Blue Marble world map and create a 10240x20480 map.
convert -append line{0..19}.jpg world.jpg convert -append line{0..19}.jpg world.jpg
=head1 EXAMPLE: Download Apollo-11 images from NASA using jq =head2 EXAMPLE: Download Apollo-11 images from NASA using jq
Search NASA using their API to get JSON for images related to 'apollo Search NASA using their API to get JSON for images related to 'apollo
11' and has 'moon landing' in the description. 11' and has 'moon landing' in the description.
@ -3344,7 +3371,7 @@ B<parallel> finally uses B<wget> to fetch the images.
parallel wget parallel wget
=head1 EXAMPLE: Download video playlist in parallel =head2 EXAMPLE: Download video playlist in parallel
B<youtube-dl> is an excellent tool to download videos. It can, B<youtube-dl> is an excellent tool to download videos. It can,
however, not download videos in parallel. This takes a playlist and however, not download videos in parallel. This takes a playlist and
@ -3357,7 +3384,7 @@ downloads 10 videos in parallel.
youtube-dl --playlist-start {#} --playlist-end {#} '"https://$url"' youtube-dl --playlist-start {#} --playlist-end {#} '"https://$url"'
=head1 EXAMPLE: Prepend last modified date (ISO8601) to file name =head2 EXAMPLE: Prepend last modified date (ISO8601) to file name
parallel mv {} '{= $a=pQ($_); $b=$_;' \ parallel mv {} '{= $a=pQ($_); $b=$_;' \
'$_=qx{date -r "$a" +%FT%T}; chomp; $_="$_ $b" =}' ::: * '$_=qx{date -r "$a" +%FT%T}; chomp; $_="$_ $b" =}' ::: *
@ -3365,7 +3392,7 @@ downloads 10 videos in parallel.
B<{=> and B<=}> mark a perl expression. B<pQ> perl-quotes the B<{=> and B<=}> mark a perl expression. B<pQ> perl-quotes the
string. B<date +%FT%T> is the date in ISO8601 with time. string. B<date +%FT%T> is the date in ISO8601 with time.
=head1 EXAMPLE: Save output in ISO8601 dirs =head2 EXAMPLE: Save output in ISO8601 dirs
Save output from B<ps aux> every second into dirs named Save output from B<ps aux> every second into dirs named
yyyy-mm-ddThh:mm:ss+zz:zz. yyyy-mm-ddThh:mm:ss+zz:zz.
@ -3374,7 +3401,7 @@ yyyy-mm-ddThh:mm:ss+zz:zz.
--results '{= $_=`date -Isec`; chomp=}/' ps aux --results '{= $_=`date -Isec`; chomp=}/' ps aux
=head1 EXAMPLE: Digital clock with "blinking" : =head2 EXAMPLE: Digital clock with "blinking" :
The : in a digital clock blinks. To make every other line have a ':' The : in a digital clock blinks. To make every other line have a ':'
and the rest a ' ' a perl expression is used to look at the 3rd input and the rest a ' ' a perl expression is used to look at the 3rd input
@ -3384,7 +3411,7 @@ source. If the value modulo 2 is 1: Use ":" otherwise use " ":
::: {0..12} ::: {0..5} ::: {0..9} ::: {0..12} ::: {0..5} ::: {0..9}
=head1 EXAMPLE: Aggregating content of files =head2 EXAMPLE: Aggregating content of files
This: This:
@ -3404,7 +3431,7 @@ So you end up with x1z1 .. x5z5 each containing the content of all
values of y. values of y.
=head1 EXAMPLE: Breadth first parallel web crawler/mirrorer =head2 EXAMPLE: Breadth first parallel web crawler/mirrorer
This script below will crawl and mirror a URL in parallel. It This script below will crawl and mirror a URL in parallel. It
downloads first pages that are 1 click down, then 2 clicks down, then downloads first pages that are 1 click down, then 2 clicks down, then
@ -3451,7 +3478,7 @@ URLs and the process is started over until no unseen links are found.
rm -f $URLLIST $URLLIST2 $SEEN rm -f $URLLIST $URLLIST2 $SEEN
=head1 EXAMPLE: Process files from a tar file while unpacking =head2 EXAMPLE: Process files from a tar file while unpacking
If the files to be processed are in a tar file then unpacking one file If the files to be processed are in a tar file then unpacking one file
and processing it immediately may be faster than first unpacking all and processing it immediately may be faster than first unpacking all
@ -3464,7 +3491,7 @@ The Perl one-liner is needed to make sure the file is complete before
handing it to GNU B<parallel>. handing it to GNU B<parallel>.
=head1 EXAMPLE: Rewriting a for-loop and a while-read-loop =head2 EXAMPLE: Rewriting a for-loop and a while-read-loop
for-loops like this: for-loops like this:
@ -3529,7 +3556,7 @@ can both be rewritten as:
export -f doit export -f doit
cat list | parallel doit cat list | parallel doit
=head1 EXAMPLE: Rewriting nested for-loops =head2 EXAMPLE: Rewriting nested for-loops
Nested for-loops like this: Nested for-loops like this:
@ -3556,7 +3583,7 @@ can be written like this:
parallel echo {1} {2} ::: red green blue ::: S M L XL XXL | sort parallel echo {1} {2} ::: red green blue ::: S M L XL XXL | sort
=head1 EXAMPLE: Finding the lowest difference between files =head2 EXAMPLE: Finding the lowest difference between files
B<diff> is good for finding differences in text files. B<diff | wc -l> B<diff> is good for finding differences in text files. B<diff | wc -l>
gives an indication of the size of the difference. To find the gives an indication of the size of the difference. To find the
@ -3568,7 +3595,7 @@ This way it is possible to see if some files are closer to other
files. files.
=head1 EXAMPLE: for-loops with column names =head2 EXAMPLE: for-loops with column names
When doing multiple nested for-loops it can be easier to keep track of When doing multiple nested for-loops it can be easier to keep track of
the loop variable if is is named instead of just having a number. Use the loop variable if is is named instead of just having a number. Use
@ -3584,7 +3611,7 @@ This also works if the input file is a file with columns:
parallel --colsep '\t' --header : echo {Name} {E-mail address} parallel --colsep '\t' --header : echo {Name} {E-mail address}
=head1 EXAMPLE: All combinations in a list =head2 EXAMPLE: All combinations in a list
GNU B<parallel> makes all combinations when given two lists. GNU B<parallel> makes all combinations when given two lists.
@ -3600,7 +3627,7 @@ B<{choose_k}> works for any number of input sources:
parallel --plus echo {choose_k} ::: A B C D ::: A B C D ::: A B C D parallel --plus echo {choose_k} ::: A B C D ::: A B C D ::: A B C D
=head1 EXAMPLE: From a to b and b to c =head2 EXAMPLE: From a to b and b to c
Assume you have input like: Assume you have input like:
@ -3628,7 +3655,7 @@ If the input is in the array $a here are two solutions:
parallel echo {1} - {2} ::: "${a[@]::${#a[@]}-1}" :::+ "${a[@]:1}" parallel echo {1} - {2} ::: "${a[@]::${#a[@]}-1}" :::+ "${a[@]:1}"
=head1 EXAMPLE: Count the differences between all files in a dir =head2 EXAMPLE: Count the differences between all files in a dir
Using B<--results> the results are saved in /tmp/diffcount*. Using B<--results> the results are saved in /tmp/diffcount*.
@ -3639,7 +3666,7 @@ To see the difference between file A and file B look at the file
'/tmp/diffcount/1/A/2/B'. '/tmp/diffcount/1/A/2/B'.
=head1 EXAMPLE: Speeding up fast jobs =head2 EXAMPLE: Speeding up fast jobs
Starting a job on the local machine takes around 10 ms. This can be a Starting a job on the local machine takes around 10 ms. This can be a
big overhead if the job takes very few ms to run. Often you can group big overhead if the job takes very few ms to run. Often you can group
@ -3682,7 +3709,7 @@ The overhead is 100000 times smaller namely around 100 nanoseconds per
job. job.
=head1 EXAMPLE: Using shell variables =head2 EXAMPLE: Using shell variables
When using shell variables you need to quote them correctly as they When using shell variables you need to quote them correctly as they
may otherwise be interpreted by the shell. may otherwise be interpreted by the shell.
@ -3721,7 +3748,7 @@ If you use them in a function you just quote as you normally would do:
parallel myfunc ::: '!' parallel myfunc ::: '!'
=head1 EXAMPLE: Group output lines =head2 EXAMPLE: Group output lines
When running jobs that output data, you often do not want the output When running jobs that output data, you often do not want the output
of multiple jobs to run together. GNU B<parallel> defaults to grouping of multiple jobs to run together. GNU B<parallel> defaults to grouping
@ -3742,7 +3769,7 @@ Compare the output of:
https://ftpmirror.gnu.org/parallel/parallel-20{}0822.tar.bz2 \ https://ftpmirror.gnu.org/parallel/parallel-20{}0822.tar.bz2 \
::: {12..16} ::: {12..16}
=head1 EXAMPLE: Tag output lines =head2 EXAMPLE: Tag output lines
GNU B<parallel> groups the output lines, but it can be hard to see GNU B<parallel> groups the output lines, but it can be hard to see
where the different jobs begin. B<--tag> prepends the argument to make where the different jobs begin. B<--tag> prepends the argument to make
@ -3763,7 +3790,7 @@ Check the uptime of the servers in I<~/.parallel/sshloginfile>:
parallel --tag -S .. --nonall uptime parallel --tag -S .. --nonall uptime
=head1 EXAMPLE: Colorize output =head2 EXAMPLE: Colorize output
Give each job a new color. Most terminals support ANSI colors with the Give each job a new color. Most terminals support ANSI colors with the
escape code "\033[30;3Xm" where 0 <= X <= 7: escape code "\033[30;3Xm" where 0 <= X <= 7:
@ -3778,7 +3805,7 @@ To get rid of the initial \t (which comes from B<--tagstring>):
... | perl -pe 's/\t//' ... | perl -pe 's/\t//'
=head1 EXAMPLE: Keep order of output same as order of input =head2 EXAMPLE: Keep order of output same as order of input
Normally the output of a job will be printed as soon as it Normally the output of a job will be printed as soon as it
completes. Sometimes you want the order of the output to remain the completes. Sometimes you want the order of the output to remain the
@ -3826,7 +3853,7 @@ combined in the correct order.
{}0000000-{}9999999 http://example.com/the/big/file > file {}0000000-{}9999999 http://example.com/the/big/file > file
=head1 EXAMPLE: Parallel grep =head2 EXAMPLE: Parallel grep
B<grep -r> greps recursively through directories. On multicore CPUs B<grep -r> greps recursively through directories. On multicore CPUs
GNU B<parallel> can often speed this up. GNU B<parallel> can often speed this up.
@ -3836,7 +3863,7 @@ GNU B<parallel> can often speed this up.
This will run 1.5 job per CPU, and give 1000 arguments to B<grep>. This will run 1.5 job per CPU, and give 1000 arguments to B<grep>.
=head1 EXAMPLE: Grepping n lines for m regular expressions. =head2 EXAMPLE: Grepping n lines for m regular expressions.
The simplest solution to grep a big file for a lot of regexps is: The simplest solution to grep a big file for a lot of regexps is:
@ -3860,7 +3887,7 @@ on the disk system it may be faster or slower to parallelize. The only
way to know for certain is to test and measure. way to know for certain is to test and measure.
=head2 Limiting factor: RAM =head3 Limiting factor: RAM
The normal B<grep -f regexps.txt bigfile> works no matter the size of The normal B<grep -f regexps.txt bigfile> works no matter the size of
bigfile, but if regexps.txt is so big it cannot fit into memory, then bigfile, but if regexps.txt is so big it cannot fit into memory, then
@ -3911,7 +3938,7 @@ If you can live with duplicated lines and wrong order, it is faster to do:
parallel --pipepart -a regexps.txt --block $percpu --compress \ parallel --pipepart -a regexps.txt --block $percpu --compress \
grep -F -f - bigfile grep -F -f - bigfile
=head2 Limiting factor: CPU =head3 Limiting factor: CPU
If the CPU is the limiting factor parallelization should be done on If the CPU is the limiting factor parallelization should be done on
the regexps: the regexps:
@ -3941,13 +3968,13 @@ combine the two using B<--cat>:
If a line matches multiple regexps, the line may be duplicated. If a line matches multiple regexps, the line may be duplicated.
=head2 Bigger problem =head3 Bigger problem
If the problem is too big to be solved by this, you are probably ready If the problem is too big to be solved by this, you are probably ready
for Lucene. for Lucene.
=head1 EXAMPLE: Using remote computers =head2 EXAMPLE: Using remote computers
To run commands on a remote computer SSH needs to be set up and you To run commands on a remote computer SSH needs to be set up and you
must be able to login without entering a password (The commands must be able to login without entering a password (The commands
@ -4021,7 +4048,7 @@ has 8 CPUs.
seq 10 | parallel --sshlogin 8/server.example.com echo seq 10 | parallel --sshlogin 8/server.example.com echo
=head1 EXAMPLE: Transferring of files =head2 EXAMPLE: Transferring of files
To recompress gzipped files with B<bzip2> using a remote computer run: To recompress gzipped files with B<bzip2> using a remote computer run:
@ -4092,7 +4119,7 @@ the special short hand I<-S ..> can be used:
--trc {.}.bz2 "zcat {} | bzip2 -9 >{.}.bz2" --trc {.}.bz2 "zcat {} | bzip2 -9 >{.}.bz2"
=head1 EXAMPLE: Distributing work to local and remote computers =head2 EXAMPLE: Distributing work to local and remote computers
Convert *.mp3 to *.ogg running one process per CPU on local computer Convert *.mp3 to *.ogg running one process per CPU on local computer
and server2: and server2:
@ -4101,7 +4128,7 @@ and server2:
'mpg321 -w - {} | oggenc -q0 - -o {.}.ogg' ::: *.mp3 'mpg321 -w - {} | oggenc -q0 - -o {.}.ogg' ::: *.mp3
=head1 EXAMPLE: Running the same command on remote computers =head2 EXAMPLE: Running the same command on remote computers
To run the command B<uptime> on remote computers you can do: To run the command B<uptime> on remote computers you can do:
@ -4118,7 +4145,7 @@ output.
If you have a lot of hosts use '-j0' to access more hosts in parallel. If you have a lot of hosts use '-j0' to access more hosts in parallel.
=head1 EXAMPLE: Running 'sudo' on remote computers =head2 EXAMPLE: Running 'sudo' on remote computers
Put the password into passwordfile then run: Put the password into passwordfile then run:
@ -4126,7 +4153,7 @@ Put the password into passwordfile then run:
-S user@server1,user@server2 sudo -S ls -l /root -S user@server1,user@server2 sudo -S ls -l /root
=head1 EXAMPLE: Using remote computers behind NAT wall =head2 EXAMPLE: Using remote computers behind NAT wall
If the workers are behind a NAT wall, you need some trickery to get to If the workers are behind a NAT wall, you need some trickery to get to
them. them.
@ -4155,7 +4182,7 @@ can simply:
parallel -S host1,host2,host3 echo ::: This does work parallel -S host1,host2,host3 echo ::: This does work
=head2 No jumphost, but port forwards =head3 No jumphost, but port forwards
If there is no jumphost but each server has port 22 forwarded from the If there is no jumphost but each server has port 22 forwarded from the
firewall (e.g. the firewall's port 22001 = port 22 on host1, 22002 = host2, firewall (e.g. the firewall's port 22001 = port 22 on host1, 22002 = host2,
@ -4174,7 +4201,7 @@ And then use host{1..3}.v as normal hosts:
parallel -S host1.v,host2.v,host3.v echo ::: a b c parallel -S host1.v,host2.v,host3.v echo ::: a b c
=head2 No jumphost, no port forwards =head3 No jumphost, no port forwards
If ports cannot be forwarded, you need some sort of VPN to traverse If ports cannot be forwarded, you need some sort of VPN to traverse
the NAT-wall. TOR is one options for that, as it is very easy to get the NAT-wall. TOR is one options for that, as it is very easy to get
@ -4202,7 +4229,7 @@ If not all hosts are accessible through TOR:
See more B<ssh> tricks on https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Proxies_and_Jump_Hosts See more B<ssh> tricks on https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Proxies_and_Jump_Hosts
=head1 EXAMPLE: Parallelizing rsync =head2 EXAMPLE: Parallelizing rsync
B<rsync> is a great tool, but sometimes it will not fill up the B<rsync> is a great tool, but sometimes it will not fill up the
available bandwidth. Running multiple B<rsync> in parallel can fix available bandwidth. Running multiple B<rsync> in parallel can fix
@ -4228,7 +4255,7 @@ are called digits.png (e.g. 000000.png) you might be able to do:
seq -w 0 99 | parallel rsync -Havessh fooserver:src/*{}.png destdir/ seq -w 0 99 | parallel rsync -Havessh fooserver:src/*{}.png destdir/
=head1 EXAMPLE: Use multiple inputs in one command =head2 EXAMPLE: Use multiple inputs in one command
Copy files like foo.es.ext to foo.ext: Copy files like foo.es.ext to foo.ext:
@ -4257,7 +4284,7 @@ Alternative version:
find . -type f | sort | parallel convert {} {#}.png find . -type f | sort | parallel convert {} {#}.png
=head1 EXAMPLE: Use a table as input =head2 EXAMPLE: Use a table as input
Content of table_file.tsv: Content of table_file.tsv:
@ -4279,7 +4306,7 @@ the columns. To keep the spaces:
parallel -a table_file.tsv --trim n --colsep '\t' cmd -o {2} -i {1} parallel -a table_file.tsv --trim n --colsep '\t' cmd -o {2} -i {1}
=head1 EXAMPLE: Output to database =head2 EXAMPLE: Output to database
GNU B<parallel> can output to a database table and a CSV-file: GNU B<parallel> can output to a database table and a CSV-file:
@ -4319,7 +4346,7 @@ Or MySQL:
s/\\([\\tn])/$s{$1}/g;' mytable.tsv s/\\([\\tn])/$s{$1}/g;' mytable.tsv
=head1 EXAMPLE: Output to CSV-file for R =head2 EXAMPLE: Output to CSV-file for R
If you have no need for the advanced job distribution control that a If you have no need for the advanced job distribution control that a
database provides, but you simply want output into a CSV file that you database provides, but you simply want output into a CSV file that you
@ -4332,7 +4359,7 @@ can read into R or LibreCalc, then you can use B<--results>:
> write(as.character(mydf[2,c("Stdout")]),'') > write(as.character(mydf[2,c("Stdout")]),'')
=head1 EXAMPLE: Use XML as input =head2 EXAMPLE: Use XML as input
The show Aflyttet on Radio 24syv publishes an RSS feed with their audio The show Aflyttet on Radio 24syv publishes an RSS feed with their audio
podcasts on: http://arkiv.radio24syv.dk/audiopodcast/channel/4466232 podcasts on: http://arkiv.radio24syv.dk/audiopodcast/channel/4466232
@ -4345,7 +4372,7 @@ using GNU B<parallel>:
parallel -u wget '{= s/ url="//; s/"//; =}' parallel -u wget '{= s/ url="//; s/"//; =}'
=head1 EXAMPLE: Run the same command 10 times =head2 EXAMPLE: Run the same command 10 times
If you want to run the same command with the same arguments 10 times If you want to run the same command with the same arguments 10 times
in parallel you can do: in parallel you can do:
@ -4353,7 +4380,7 @@ in parallel you can do:
seq 10 | parallel -n0 my_command my_args seq 10 | parallel -n0 my_command my_args
=head1 EXAMPLE: Working as cat | sh. Resource inexpensive jobs and evaluation =head2 EXAMPLE: Working as cat | sh. Resource inexpensive jobs and evaluation
GNU B<parallel> can work similar to B<cat | sh>. GNU B<parallel> can work similar to B<cat | sh>.
@ -4378,7 +4405,7 @@ To run 100 processes simultaneously do:
As there is not a I<command> the jobs will be evaluated by the shell. As there is not a I<command> the jobs will be evaluated by the shell.
=head1 EXAMPLE: Call program with FASTA sequence =head2 EXAMPLE: Call program with FASTA sequence
FASTA files have the format: FASTA files have the format:
@ -4397,7 +4424,7 @@ To call B<myprog> with the sequence as argument run:
'read a; echo Name: "$a"; myprog $(tr -d "\n")' 'read a; echo Name: "$a"; myprog $(tr -d "\n")'
=head1 EXAMPLE: Processing a big file using more CPUs =head2 EXAMPLE: Processing a big file using more CPUs
To process a big file or some output you can use B<--pipe> to split up To process a big file or some output you can use B<--pipe> to split up
the data into blocks and pipe the blocks into the processing program. the data into blocks and pipe the blocks into the processing program.
@ -4434,7 +4461,7 @@ the parts directly to the program:
parallel -Xj1 sort -m {} ';' rm {} >bigfile.sort parallel -Xj1 sort -m {} ';' rm {} >bigfile.sort
=head1 EXAMPLE: Grouping input lines =head2 EXAMPLE: Grouping input lines
When processing with B<--pipe> you may have lines grouped by a When processing with B<--pipe> you may have lines grouped by a
value. Here is I<my.csv>: value. Here is I<my.csv>:
@ -4473,7 +4500,7 @@ If your program can process multiple customers replace B<-N1> with a
reasonable B<--blocksize>. reasonable B<--blocksize>.
=head1 EXAMPLE: Running more than 250 jobs workaround =head2 EXAMPLE: Running more than 250 jobs workaround
If you need to run a massive amount of jobs in parallel, then you will If you need to run a massive amount of jobs in parallel, then you will
likely hit the filehandle limit which is often around 250 jobs. If you likely hit the filehandle limit which is often around 250 jobs. If you
@ -4492,7 +4519,7 @@ RAM to do this, and you may need to increase /proc/sys/kernel/pid_max):
parallel --pipe -N 250 --roundrobin -j250 parallel -j250 your_prg parallel --pipe -N 250 --roundrobin -j250 parallel -j250 your_prg
=head1 EXAMPLE: Working as mutex and counting semaphore =head2 EXAMPLE: Working as mutex and counting semaphore
The command B<sem> is an alias for B<parallel --semaphore>. The command B<sem> is an alias for B<parallel --semaphore>.
@ -4525,7 +4552,7 @@ same time:
seq 3 | parallel sem --id mymutex sed -i -e '1i{}' myfile seq 3 | parallel sem --id mymutex sed -i -e '1i{}' myfile
=head1 EXAMPLE: Mutex for a script =head2 EXAMPLE: Mutex for a script
Assume a script is called from cron or from a web service, but only Assume a script is called from cron or from a web service, but only
one instance can be run at a time. With B<sem> and B<--shebang-wrap> one instance can be run at a time. With B<sem> and B<--shebang-wrap>
@ -4556,7 +4583,7 @@ Here B<python>:
print "exclusively"; print "exclusively";
=head1 EXAMPLE: Start editor with filenames from stdin (standard input) =head2 EXAMPLE: Start editor with filenames from stdin (standard input)
You can use GNU B<parallel> to start interactive programs like emacs or vi: You can use GNU B<parallel> to start interactive programs like emacs or vi:
@ -4567,7 +4594,7 @@ If there are more files than will fit on a single command line, the
editor will be started again with the remaining files. editor will be started again with the remaining files.
=head1 EXAMPLE: Running sudo =head2 EXAMPLE: Running sudo
B<sudo> requires a password to run a command as root. It caches the B<sudo> requires a password to run a command as root. It caches the
access, so you only need to enter the password again if you have not access, so you only need to enter the password again if you have not
@ -4590,7 +4617,7 @@ or:
This way you only have to enter the sudo password once. This way you only have to enter the sudo password once.
=head1 EXAMPLE: GNU Parallel as queue system/batch manager =head2 EXAMPLE: GNU Parallel as queue system/batch manager
GNU B<parallel> can work as a simple job queue system or batch manager. GNU B<parallel> can work as a simple job queue system or batch manager.
The idea is to put the jobs into a file and have GNU B<parallel> read The idea is to put the jobs into a file and have GNU B<parallel> read
@ -4650,7 +4677,7 @@ the output of second completed job will only be printed when job 12
has started. has started.
=head1 EXAMPLE: GNU Parallel as dir processor =head2 EXAMPLE: GNU Parallel as dir processor
If you have a dir in which users drop files that needs to be processed If you have a dir in which users drop files that needs to be processed
you can do this on GNU/Linux (If you know what B<inotifywait> is you can do this on GNU/Linux (If you know what B<inotifywait> is
@ -4676,7 +4703,7 @@ Using GNU B<parallel> as dir processor has the same limitations as
using GNU B<parallel> as queue system/batch manager. using GNU B<parallel> as queue system/batch manager.
=head1 EXAMPLE: Locate the missing package =head2 EXAMPLE: Locate the missing package
If you have downloaded source and tried compiling it, you may have seen: If you have downloaded source and tried compiling it, you may have seen:
@ -4902,7 +4929,7 @@ Dir where GNU B<parallel> stores config files, semaphores, and caches
information between invocations. Default: $HOME/.parallel. information between invocations. Default: $HOME/.parallel.
=item $PARALLEL_ARGHOSTGROUPS =item $PARALLEL_ARGHOSTGROUPS (beta testing)
When using B<--hostgroups> GNU B<parallel> sets this to the hostgroups When using B<--hostgroups> GNU B<parallel> sets this to the hostgroups
of the job. of the job.

62
src/parallel_alternatives.pod
View file

@ -2777,7 +2777,7 @@ https://gitlab.com/netikras/bthread (Last checked: 2021-01)
Summary (see legend above): Summary (see legend above):
I1 - - - - - I7 I1 - - - - - I7
M1 - - - - M6 M1 - - - - M6
- O2 O3 - - O6 - N/A N/A O10 - O2 O3 - - O6 - x x O10
E1 - - - - - - E1 - - - - - -
- - - - - - - - - - - - - - - - - -
- - - -
@ -2877,7 +2877,7 @@ with replacement strings. Such as:
that can be used like: that can be used like:
parallel --header : --tmpl my.tmpl {#}.t myprog {#}.t \ parallel --header : --tmpl my.tmpl={#}.t myprog {#}.t \
::: x 1 2 3 ::: y 1 2 3 ::: x 1 2 3 ::: y 1 2 3
Filtering may also be supported as: Filtering may also be supported as:
@ -2891,23 +2891,62 @@ which will basically do:
https://github.com/eviatarbach/parasweep (Last checked: 2021-01) https://github.com/eviatarbach/parasweep (Last checked: 2021-01)
=head2 Todo =head2 DIFFERENCES BETWEEN parallel-bash AND GNU Parallel
Summary (see legend above):
I1 I2 - - - - -
- - M3 - - M6
- O2 O3 - O5 O6 - O8 x O10
E1 - - - - - -
- - - - - - - - -
- -
B<parallel-bash> is written in pure bash. It is really fast (overhead
of ~0.05 ms/job compared to GNU B<parallel>'s ~3 ms/job). So if your
jobs are extremely short lived, and you can live with the quite
limited command, this may be useful.
B<parallel-bash> will not start the first job, until it has read all
input. The input can at most be 20935 lines and the lines cannot be
all be empty.
Ctrl-C does not stop spawning new jobs. Ctrl-Z does not suspend
running jobs.
=head3 EXAMPLES FROM parallel-bash
1$ some_input | parallel-bash -p 5 -c echo
1$ some_input | parallel -j 5 echo
2$ parallel-bash -p 5 -c echo < some_file
2$ parallel -j 5 echo < some_file
3$ parallel-bash -p 5 -c echo <<< 'some string'
3$ parallel -j 5 -c echo <<< 'some string'
4$ something | parallel-bash -p 5 -c echo {} {}
4$ something | parallel -j 5 echo {} {}
https://reposhub.com/python/command-line-tools/Akianonymus-parallel-bash.html https://reposhub.com/python/command-line-tools/Akianonymus-parallel-bash.html
(Last checked: 2021-02)
=head2 Todo
https://github.com/Nukesor/pueue https://github.com/Nukesor/pueue
PASH: Light-touch Data-Parallel Shell Processing
https://arxiv.org/pdf/2012.15443.pdf KumQuat https://arxiv.org/pdf/2012.15443.pdf KumQuat
https://arxiv.org/pdf/2007.09436.pdf https://arxiv.org/pdf/2007.09436.pdf PaSH: Light-touch Data-Parallel Shell Processing
https://github.com/JeiKeiLim/simple_distribute_job https://github.com/JeiKeiLim/simple_distribute_job
https://github.com/Akianonymus/parallel-bash https://github.com/reggi/pkgrun - not obvious how to use
https://github.com/reggi/pkgrun
https://github.com/benoror/better-npm-run - not obvious how to use https://github.com/benoror/better-npm-run - not obvious how to use
@ -2919,6 +2958,11 @@ https://github.com/flesler/parallel
https://github.com/Julian/Verge https://github.com/Julian/Verge
http://manpages.ubuntu.com/manpages/xenial/man1/tsp.1.html
http://vicerveza.homeunix.net/~viric/soft/ts/
=head1 TESTING OTHER TOOLS =head1 TESTING OTHER TOOLS

2
src/parallel_design.pod
View file

@ -1153,7 +1153,7 @@ multiple B<:::>.
When B<:::> was chosen, B<::::> came as a fairly natural extension. When B<:::> was chosen, B<::::> came as a fairly natural extension.
Linking input sources meant having to decide for some way to indicate Linking input sources meant having to decide for some way to indicate
linking of B<:::> and B<::::>. B<:::+> and B<::::+> was chosen, so linking of B<:::> and B<::::>. B<:::+> and B<::::+> were chosen, so
that they were similar to B<:::> and B<::::>. that they were similar to B<:::> and B<::::>.

2
src/parsort
View file

@ -118,7 +118,7 @@ GetOptions(
"help" => \$opt::dummy, "help" => \$opt::dummy,
) || exit(255); ) || exit(255);
$Global::progname = ($0 =~ m:(^|/)([^/]+)$:)[1]; $Global::progname = ($0 =~ m:(^|/)([^/]+)$:)[1];
$Global::version = 20210122; $Global::version = 20210123;
if($opt::version) { version(); exit 0; } if($opt::version) { version(); exit 0; }
@Global::sortoptions = @Global::sortoptions =
shell_quote(@ARGV_before[0..($#ARGV_before-$#ARGV-1)]); shell_quote(@ARGV_before[0..($#ARGV_before-$#ARGV-1)]);

2
src/sql
View file

@ -574,7 +574,7 @@ $Global::Initfile && unlink $Global::Initfile;
exit ($err); exit ($err);
sub parse_options { sub parse_options {
$Global::version = 20210122; $Global::version = 20210123;
$Global::progname = 'sql'; $Global::progname = 'sql';
# This must be done first as this may exec myself # This must be done first as this may exec myself

58
testsuite/tests-to-run/parallel-local-0.3s.sh
View file

@ -12,6 +12,53 @@ export -f stdsort
# Test amount of parallelization # Test amount of parallelization
# parallel --shuf --jl /tmp/myjl -j1 'export JOBS={1};'bash tests-to-run/parallel-local-0.3s.sh ::: {1..16} ::: {1..5} # parallel --shuf --jl /tmp/myjl -j1 'export JOBS={1};'bash tests-to-run/parallel-local-0.3s.sh ::: {1..16} ::: {1..5}
par_crnl() {
echo '### Give a warning if input is DOS-ascii'
printf "b\r\nc\r\nd\r\ne\r\nf\r\n" | stdout parallel -k echo {}a
echo This should give no warning because -d is set
printf "b\r\nc\r\nd\r\ne\r\nf\r\n" | parallel -k -d '\r\n' echo {}a
echo This should give no warning because line2 has newline only
printf "b\r\nc\nd\r\ne\r\nf\r\n" | parallel -k echo {}a
}
par_tmpl() {
tmp1=$(mktemp)
tmp2=$(mktemp)
cat <<'EOF' > "$tmp1"
Template1
Xval: {x}
Yval: {y}
FixedValue: 9
Seq: {#}
Slot: {%}
# x with 2 decimals
DecimalX: {=x $_=sprintf("%.2f",$_) =}
TenX: {=x $_=$_*10 =}
RandomVal: {=1 $_=rand() =}
EOF
cat <<'EOF' > "$tmp2"
Template2
X,Y: {x},{y}
val1,val2: {1},{2}
EOF
myprog() {
echo "$@"
cat "$@"
}
export -f myprog
parallel -k --header : --tmpl "$tmp1"={#}.t1 \
--tmpl "$tmp2"=/tmp/tmpl-{x}-{y}.t2 \
myprog {#}.t1 /tmp/tmpl-{x}-{y}.t2 \
::: x 1.1 2.22 3.333 ::: y 111.111 222.222 333.333 |
perl -pe 's/0.\d{13,}/0.RANDOM_NUMBER/' |
perl -pe 's/Slot: \d/Slot: X/'
rm "$tmp1" "$tmp2"
}
par_resume_k() { par_resume_k() {
echo '### --resume -k' echo '### --resume -k'
tmp=$(tempfile) tmp=$(tempfile)
@ -795,12 +842,6 @@ par_empty_input_on_stdin() {
true | stdout parallel --shuf echo true | stdout parallel --shuf echo
} }
par_tee_too_many_args() {
echo '### Fail if there are more arguments than --jobs'
seq 11 | parallel -k --tag --pipe -j4 --tee grep {} ::: {1..4}
seq 11 | parallel -k --tag --pipe -j4 --tee grep {} ::: {1..5}
}
par_space_envvar() { par_space_envvar() {
echo "### bug: --gnu was ignored if env var started with space: PARALLEL=' --gnu'" echo "### bug: --gnu was ignored if env var started with space: PARALLEL=' --gnu'"
export PARALLEL=" -v" && parallel echo ::: 'space in envvar OK' export PARALLEL=" -v" && parallel echo ::: 'space in envvar OK'
@ -909,11 +950,6 @@ par_cr_newline_header() {
parallel --colsep , --header : echo {foo} parallel --colsep , --header : echo {foo}
} }
par_plus_slot_replacement() {
echo '### show {slot}'
parallel -k --plus echo '{slot}=$PARALLEL_JOBSLOT={%}' ::: A B C
}
par_PARALLEL_HOME_with_+() { par_PARALLEL_HOME_with_+() {
echo 'bug #59453: PARALLEL_HOME with plus sign causes error: config not readable' echo 'bug #59453: PARALLEL_HOME with plus sign causes error: config not readable'
tmp=$(mktemp -d) tmp=$(mktemp -d)

2
testsuite/tests-to-run/parallel-local-10s.sh
View file

@ -539,7 +539,7 @@ par__pipepart_spawn() {
echo '### bug #46214: Using --pipepart doesnt spawn multiple jobs in version 20150922' echo '### bug #46214: Using --pipepart doesnt spawn multiple jobs in version 20150922'
seq 1000000 > /tmp/num1000000 seq 1000000 > /tmp/num1000000
stdout parallel --pipepart --progress -a /tmp/num1000000 --block 10k -j0 true | stdout parallel --pipepart --progress -a /tmp/num1000000 --block 10k -j0 true |
grep 1:local | perl -pe 's/\d\d\d/999/g; s/[2-9]/2+/g;' grep 1:local | perl -pe 's/\d\d\d/999/g; s/\d\d+|[2-9]/2+/g;'
} }
par__pipe_tee() { par__pipe_tee() {

15
testsuite/tests-to-run/parallel-local-1s.sh
View file

@ -4,6 +4,14 @@
# Each should be taking 1-3s and be possible to run in parallel # Each should be taking 1-3s and be possible to run in parallel
# I.e.: No race conditions, no logins # I.e.: No race conditions, no logins
par_plus_slot_replacement() {
echo '### show {slot} {0%} {0#}'
parallel -k --plus echo '{slot}=$PARALLEL_JOBSLOT={%}' ::: A B C
parallel -j15 -k --plus 'echo Seq: {000#} {00#} {0#}' ::: {1..100} | sort
parallel -j15 -k --plus 'sleep 0.0{}; echo Slot: {000%} {00%} {0%}' ::: {1..100} |
sort -u
}
par_recend_recstart_hash() { par_recend_recstart_hash() {
echo "### bug #59843: --regexp --recstart '#' fails" echo "### bug #59843: --regexp --recstart '#' fails"
(echo '#rec1'; echo 'bar'; echo '#rec2') | (echo '#rec1'; echo 'bar'; echo '#rec2') |
@ -222,7 +230,9 @@ par_test_job_number() {
par_seqreplace_long_line() { par_seqreplace_long_line() {
echo '### Test --seqreplace and line too long' echo '### Test --seqreplace and line too long'
seq 1 1000 | stdout parallel -j1 -s 210 -k --seqreplace I echo IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII \|wc | uniq -c seq 1 1000 |
stdout parallel -j1 -s 210 -k --seqreplace I echo IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII \|wc |
uniq -c
} }
par_bug37042() { par_bug37042() {
@ -238,7 +248,8 @@ par_bug37042() {
par_header() { par_header() {
echo "### Test --header with -N" echo "### Test --header with -N"
(echo h1; echo h2; echo 1a;echo 1b; echo 2a;echo 2b; echo 3a)| parallel -j1 --pipe -N2 -k --header '.*\n.*\n' echo Start\;cat \; echo Stop (echo h1; echo h2; echo 1a;echo 1b; echo 2a;echo 2b; echo 3a) |
parallel -j1 --pipe -N2 -k --header '.*\n.*\n' echo Start\;cat \; echo Stop
echo "### Test --header with --block 1k" echo "### Test --header with --block 1k"
(echo h1; echo h2; perl -e '$a="x"x110;for(1..22){print $_,$a,"\n"}') | (echo h1; echo h2; perl -e '$a="x"x110;for(1..22){print $_,$a,"\n"}') |

4
testsuite/tests-to-run/parallel-local-30s.sh
View file

@ -251,9 +251,9 @@ par_memory_leak() {
} }
export -f a_run export -f a_run
echo "### Test for memory leaks" echo "### Test for memory leaks"
echo "Of 100 runs of 1 job at least one should be bigger than a 3000 job run" echo "Of 300 runs of 1 job at least one should be bigger than a 3000 job run"
. `which env_parallel.bash` . `which env_parallel.bash`
parset small_max,big ::: 'seq 100 | parallel a_run 1 | jq -s max' 'a_run 3000' parset small_max,big ::: 'seq 300 | parallel a_run 1 | jq -s max' 'a_run 3000'
if [ $small_max -lt $big ] ; then if [ $small_max -lt $big ] ; then
echo "Bad: Memleak likely." echo "Bad: Memleak likely."
else else

6
testsuite/tests-to-run/parallel-local-3s.sh
View file

@ -134,6 +134,12 @@ par_tee_with_premature_close() {
fi fi
} }
par_tee_too_many_args() {
echo '### Fail if there are more arguments than --jobs'
seq 11 | stdout parallel -k --tag --pipe -j4 --tee grep {} ::: {1..4}
seq 11 | stdout parallel -k --tag --pipe -j0 --tee grep {} ::: {1..10000}
}
par_maxargs() { par_maxargs() {
echo '### Test -n and --max-args: Max number of args per line (only with -X and -m)' echo '### Test -n and --max-args: Max number of args per line (only with -X and -m)'

2
testsuite/tests-to-run/parallel-local-ssh3.sh
View file

@ -69,7 +69,7 @@ par_export_functions_csh() {
par_progress_text_max_jobs_to_run() { par_progress_text_max_jobs_to_run() {
echo '### bug #49404: "Max jobs to run" does not equal the number of jobs specified when using GNU Parallel on remote server?' echo '### bug #49404: "Max jobs to run" does not equal the number of jobs specified when using GNU Parallel on remote server?'
echo should give 10 running jobs echo should give 10 running jobs
stdout parallel -S 16/lo --progress true ::: {1..10} | grep /.10 stdout parallel -S 16/lo --progress true ::: {1..10} | grep /.16
} }
par_hgrp_rpl() { par_hgrp_rpl() {

8
testsuite/tests-to-run/parallel-local-ssh8.sh
View file

@ -27,8 +27,8 @@ par_path_remote_bash() {
export A="`seq 1000`" export A="`seq 1000`"
PATH=$PATH:/tmp PATH=$PATH:/tmp
. /usr/local/bin/env_parallel.bash . /usr/local/bin/env_parallel.bash
# --filter to see if $PATH with parallel is transferred # --filter-hosts to see if $PATH with parallel is transferred
env_parallel --filter --env A,PATH -Slo echo '$PATH' ::: OK env_parallel --filter-hosts --env A,PATH -Slo echo '$PATH' ::: OK
_EOS _EOS
stdout ssh nopathbash@lo -T | stdout ssh nopathbash@lo -T |
perl -ne '/StArT/..0 and print' | perl -ne '/StArT/..0 and print' |
@ -55,8 +55,8 @@ par_path_remote_csh() {
if ("`alias env_parallel`" == '') then if ("`alias env_parallel`" == '') then
source `which env_parallel.csh` source `which env_parallel.csh`
endif endif
# --filter to see if $PATH with parallel is transferred # --filter-hosts to see if $PATH with parallel is transferred
env_parallel --filter --env A,PATH -Slo echo '$PATH' ::: OK env_parallel --filter-hosts --env A,PATH -Slo echo '$PATH' ::: OK
# Sleep needed to avoid stderr/stdout mixing # Sleep needed to avoid stderr/stdout mixing
sleep 1 sleep 1
echo Done echo Done

46
testsuite/tests-to-run/parallel-remote1.sh
View file

@ -6,9 +6,9 @@ SERVER3=parallel-server3
SSHUSER1=vagrant SSHUSER1=vagrant
SSHUSER2=vagrant SSHUSER2=vagrant
SSHUSER3=vagrant SSHUSER3=vagrant
SSHLOGIN1=$SSHUSER1@$SERVER1 export SSHLOGIN1=$SSHUSER1@$SERVER1
SSHLOGIN2=$SSHUSER2@$SERVER2 export SSHLOGIN2=$SSHUSER2@$SERVER2
SSHLOGIN3=$SSHUSER3@$SERVER3 export SSHLOGIN3=$SSHUSER3@$SERVER3
#SERVER1=parallel-server1 #SERVER1=parallel-server1
#SERVER2=lo #SERVER2=lo
@ -16,31 +16,57 @@ SSHLOGIN3=$SSHUSER3@$SERVER3
#SSHLOGIN2=parallel@lo #SSHLOGIN2=parallel@lo
#SSHLOGIN3=parallel@parallel-server2 #SSHLOGIN3=parallel@parallel-server2
par_special_ssh() {
echo '### Test use special ssh' echo '### Test use special ssh'
echo 'TODO test ssh with > 9 simultaneous' echo 'TODO test ssh with > 9 simultaneous'
echo 'ssh "$@"; echo "$@" >>/tmp/myssh1-run' >/tmp/myssh1 echo 'ssh "$@"; echo "$@" >>/tmp/myssh1-run' >/tmp/myssh1
echo 'ssh "$@"; echo "$@" >>/tmp/myssh2-run' >/tmp/myssh2 echo 'ssh "$@"; echo "$@" >>/tmp/myssh2-run' >/tmp/myssh2
chmod 755 /tmp/myssh1 /tmp/myssh2 chmod 755 /tmp/myssh1 /tmp/myssh2
seq 1 100 | parallel --sshdelay 0.03 --retries 10 --sshlogin "/tmp/myssh1 $SSHLOGIN1,/tmp/myssh2 $SSHLOGIN2" -k echo seq 1 100 | parallel --sshdelay 0.03 --retries 10 --sshlogin "/tmp/myssh1 $SSHLOGIN1,/tmp/myssh2 $SSHLOGIN2" -k echo
}
cat <<'EOF' | sed -e s/\$SERVER1/$SERVER1/\;s/\$SERVER2/$SERVER2/\;s/\$SSHLOGIN1/$SSHLOGIN1/\;s/\$SSHLOGIN2/$SSHLOGIN2/\;s/\$SSHLOGIN3/$SSHLOGIN3/ | parallel -vj3 -k -L1 -r par_filter_hosts_different_errors() {
echo '### --filter-hosts - OK, non-such-user, connection refused, wrong host'
stdout parallel --nonall --filter-hosts -S localhost,NoUser@localhost,154.54.72.206,"ssh 5.5.5.5" hostname |
grep -v 'parallel: Warning: Removed'
}
par_timeout_retries() {
echo '### test --timeout --retries' echo '### test --timeout --retries'
parallel -j0 --timeout 5 --retries 3 -k ssh {} echo {} ::: 192.168.1.197 8.8.8.8 $SSHLOGIN1 $SSHLOGIN2 $SSHLOGIN3 parallel -j0 --timeout 5 --retries 3 -k ssh {} echo {} ::: 192.168.1.197 8.8.8.8 $SSHLOGIN1 $SSHLOGIN2 $SSHLOGIN3
}
par_filter_hosts_no_ssh_nxserver() {
echo '### test --filter-hosts with server w/o ssh, non-existing server' echo '### test --filter-hosts with server w/o ssh, non-existing server'
parallel -S 192.168.1.197,8.8.8.8,$SSHLOGIN1,$SSHLOGIN2,$SSHLOGIN3 --filter-hosts --nonall -k --tag echo stdout parallel -S 192.168.1.197,8.8.8.8,$SSHLOGIN1,$SSHLOGIN2,$SSHLOGIN3 --filter-hosts --nonall -k --tag echo |
grep -v 'parallel: Warning: Removed'
}
par_controlmaster_is_faster() {
echo '### bug #41964: --controlmaster not seems to reuse OpenSSH connections to the same host' echo '### bug #41964: --controlmaster not seems to reuse OpenSSH connections to the same host'
(parallel -S $SSHLOGIN1 true ::: {1..20}; echo No --controlmaster - finish last) & (parallel -S $SSHLOGIN1 true ::: {1..20};
(parallel -M -S $SSHLOGIN1 true ::: {1..20}; echo With --controlmaster - finish first) & echo No --controlmaster - finish last) &
(parallel -M -S $SSHLOGIN1 true ::: {1..20};
echo With --controlmaster - finish first) &
wait wait
}
echo '### --filter-hosts - OK, non-such-user, connection refused, wrong host' par_workdir_in_HOME() {
parallel --nonall --filter-hosts -S localhost,NoUser@localhost,154.54.72.206,"ssh 5.5.5.5" hostname
echo '### test --workdir . in $HOME' echo '### test --workdir . in $HOME'
cd && mkdir -p parallel-test && cd parallel-test && cd && mkdir -p parallel-test && cd parallel-test &&
echo OK > testfile && parallel --workdir . --transfer -S $SSHLOGIN1 cat {} ::: testfile echo OK > testfile && parallel --workdir . --transfer -S $SSHLOGIN1 cat {} ::: testfile
}
export -f $(compgen -A function | grep par_)
compgen -A function | grep par_ | LC_ALL=C sort |
parallel --timeout 1000% -j6 --tag -k --joblog /tmp/jl-`basename $0` '{} 2>&1' |
perl -pe 's:/usr/bin:/bin:g'
cat <<'EOF' | sed -e s/\$SERVER1/$SERVER1/\;s/\$SERVER2/$SERVER2/\;s/\$SSHLOGIN1/$SSHLOGIN1/\;s/\$SSHLOGIN2/$SSHLOGIN2/\;s/\$SSHLOGIN3/$SSHLOGIN3/ | parallel -vj3 -k -L1 -r
echo '### TODO: test --filter-hosts proxied through the one host' echo '### TODO: test --filter-hosts proxied through the one host'

175
testsuite/wanted-results/parallel-local-0.3s
View file

@ -68,6 +68,25 @@ par_compress_stdout_stderr ### Test compress - stderr
par_compress_stdout_stderr ls: cannot access '/OK-if-missing-file': No such file or directory par_compress_stdout_stderr ls: cannot access '/OK-if-missing-file': No such file or directory
par_cr_newline_header ### --header : should set named replacement string if input line ends in \r\n par_cr_newline_header ### --header : should set named replacement string if input line ends in \r\n
par_cr_newline_header bar par_cr_newline_header bar
par_crnl ### Give a warning if input is DOS-ascii
par_crnl parallel: Warning: The first three values end in CR-NL. Consider using -d "\r\n"
par_crnl b par_crnl a
par_crnl c par_crnl a
par_crnl d par_crnl a
par_crnl e par_crnl a
par_crnl f par_crnl a
par_crnl This should give no warning because -d is set
par_crnl ba
par_crnl ca
par_crnl da
par_crnl ea
par_crnl fa
par_crnl This should give no warning because line2 has newline only
par_crnl b par_crnl a
par_crnl ca
par_crnl d par_crnl a
par_crnl e par_crnl a
par_crnl f par_crnl a
par_csv col1"x3"-new par_csv col1"x3"-new
par_csv line col2-new2 par_csv line col2-new2
par_csv line col3-col 4 par_csv line col3-col 4
@ -692,10 +711,6 @@ par_pipepart_recend_recstart 9
par_pipepart_recend_recstart 10 par_pipepart_recend_recstart 10
par_pipepart_roundrobin ### bug #45769: --round-robin --pipepart gives wrong results par_pipepart_roundrobin ### bug #45769: --round-robin --pipepart gives wrong results
par_pipepart_roundrobin 2 par_pipepart_roundrobin 2
par_plus_slot_replacement ### show {slot}
par_plus_slot_replacement 1=1=1
par_plus_slot_replacement 2=2=2
par_plus_slot_replacement 3=3=3
par_profile ### Test -J profile, -J /dir/profile, -J ./profile par_profile ### Test -J profile, -J /dir/profile, -J ./profile
par_profile local local par_profile local local
par_profile abs abs par_profile abs abs
@ -936,14 +951,6 @@ par_tee 4 -l 122853
par_tee 4 -c 815290 par_tee 4 -c 815290
par_tee 5 -l 122853 par_tee 5 -l 122853
par_tee 5 -c 815290 par_tee 5 -c 815290
par_tee_too_many_args ### Fail if there are more arguments than --jobs
par_tee_too_many_args 1 1
par_tee_too_many_args 1 10
par_tee_too_many_args 1 11
par_tee_too_many_args 2 2
par_tee_too_many_args 3 3
par_tee_too_many_args 4 4
par_tee_too_many_args parallel: Error: --tee requires --jobs to be higher. Try --jobs 0.
par_test_L_context_replace ### Test -N context replace par_test_L_context_replace ### Test -N context replace
par_test_L_context_replace a1b a2b a3b a4b a5b a6b a7b a8b a9b a10b par_test_L_context_replace a1b a2b a3b a4b a5b a6b a7b a8b a9b a10b
par_test_L_context_replace a11b a12b a13b a14b a15b a16b a17b a18b a19b par_test_L_context_replace a11b a12b a13b a14b a15b a16b a17b a18b a19b
@ -1014,6 +1021,150 @@ par_testquote yash "#&/
par_testquote yash ()*=?' par_testquote yash ()*=?'
par_testquote zsh "#&/ par_testquote zsh "#&/
par_testquote zsh ()*=?' par_testquote zsh ()*=?'
par_tmpl 1.t1 /tmp/tmpl-1.1-111.111.t2
par_tmpl Template1
par_tmpl Xval: 1.1
par_tmpl Yval: 111.111
par_tmpl FixedValue: 9
par_tmpl Seq: 1
par_tmpl Slot: X
par_tmpl # x with 2 decimals
par_tmpl DecimalX: 1.10
par_tmpl TenX: 11
par_tmpl RandomVal: 0.RANDOM_NUMBER
par_tmpl
par_tmpl Template2
par_tmpl X,Y: 1.1,111.111
par_tmpl val1,val2: 1.1,111.111
par_tmpl
par_tmpl 2.t1 /tmp/tmpl-1.1-222.222.t2
par_tmpl Template1
par_tmpl Xval: 1.1
par_tmpl Yval: 222.222
par_tmpl FixedValue: 9
par_tmpl Seq: 2
par_tmpl Slot: X
par_tmpl # x with 2 decimals
par_tmpl DecimalX: 1.10
par_tmpl TenX: 11
par_tmpl RandomVal: 0.RANDOM_NUMBER
par_tmpl
par_tmpl Template2
par_tmpl X,Y: 1.1,222.222
par_tmpl val1,val2: 1.1,222.222
par_tmpl
par_tmpl 3.t1 /tmp/tmpl-1.1-333.333.t2
par_tmpl Template1
par_tmpl Xval: 1.1
par_tmpl Yval: 333.333
par_tmpl FixedValue: 9
par_tmpl Seq: 3
par_tmpl Slot: X
par_tmpl # x with 2 decimals
par_tmpl DecimalX: 1.10
par_tmpl TenX: 11
par_tmpl RandomVal: 0.RANDOM_NUMBER
par_tmpl
par_tmpl Template2
par_tmpl X,Y: 1.1,333.333
par_tmpl val1,val2: 1.1,333.333
par_tmpl
par_tmpl 4.t1 /tmp/tmpl-2.22-111.111.t2
par_tmpl Template1
par_tmpl Xval: 2.22
par_tmpl Yval: 111.111
par_tmpl FixedValue: 9
par_tmpl Seq: 4
par_tmpl Slot: X
par_tmpl # x with 2 decimals
par_tmpl DecimalX: 2.22
par_tmpl TenX: 22.2
par_tmpl RandomVal: 0.RANDOM_NUMBER
par_tmpl
par_tmpl Template2
par_tmpl X,Y: 2.22,111.111
par_tmpl val1,val2: 2.22,111.111
par_tmpl
par_tmpl 5.t1 /tmp/tmpl-2.22-222.222.t2
par_tmpl Template1
par_tmpl Xval: 2.22
par_tmpl Yval: 222.222
par_tmpl FixedValue: 9
par_tmpl Seq: 5
par_tmpl Slot: X
par_tmpl # x with 2 decimals
par_tmpl DecimalX: 2.22
par_tmpl TenX: 22.2
par_tmpl RandomVal: 0.RANDOM_NUMBER
par_tmpl
par_tmpl Template2
par_tmpl X,Y: 2.22,222.222
par_tmpl val1,val2: 2.22,222.222
par_tmpl
par_tmpl 6.t1 /tmp/tmpl-2.22-333.333.t2
par_tmpl Template1
par_tmpl Xval: 2.22
par_tmpl Yval: 333.333
par_tmpl FixedValue: 9
par_tmpl Seq: 6
par_tmpl Slot: X
par_tmpl # x with 2 decimals
par_tmpl DecimalX: 2.22
par_tmpl TenX: 22.2
par_tmpl RandomVal: 0.RANDOM_NUMBER
par_tmpl
par_tmpl Template2
par_tmpl X,Y: 2.22,333.333
par_tmpl val1,val2: 2.22,333.333
par_tmpl
par_tmpl 7.t1 /tmp/tmpl-3.333-111.111.t2
par_tmpl Template1
par_tmpl Xval: 3.333
par_tmpl Yval: 111.111
par_tmpl FixedValue: 9
par_tmpl Seq: 7
par_tmpl Slot: X
par_tmpl # x with 2 decimals
par_tmpl DecimalX: 3.33
par_tmpl TenX: 33.33
par_tmpl RandomVal: 0.RANDOM_NUMBER
par_tmpl
par_tmpl Template2
par_tmpl X,Y: 3.333,111.111
par_tmpl val1,val2: 3.333,111.111
par_tmpl
par_tmpl 8.t1 /tmp/tmpl-3.333-222.222.t2
par_tmpl Template1
par_tmpl Xval: 3.333
par_tmpl Yval: 222.222
par_tmpl FixedValue: 9
par_tmpl Seq: 8
par_tmpl Slot: X
par_tmpl # x with 2 decimals
par_tmpl DecimalX: 3.33
par_tmpl TenX: 33.33
par_tmpl RandomVal: 0.RANDOM_NUMBER
par_tmpl
par_tmpl Template2
par_tmpl X,Y: 3.333,222.222
par_tmpl val1,val2: 3.333,222.222
par_tmpl
par_tmpl 9.t1 /tmp/tmpl-3.333-333.333.t2
par_tmpl Template1
par_tmpl Xval: 3.333
par_tmpl Yval: 333.333
par_tmpl FixedValue: 9
par_tmpl Seq: 9
par_tmpl Slot: X
par_tmpl # x with 2 decimals
par_tmpl DecimalX: 3.33
par_tmpl TenX: 33.33
par_tmpl RandomVal: 0.RANDOM_NUMBER
par_tmpl
par_tmpl Template2
par_tmpl X,Y: 3.333,333.333
par_tmpl val1,val2: 3.333,333.333
par_tmpl
par_tmux_command_not_found ### PARALLEL_TMUX not found par_tmux_command_not_found ### PARALLEL_TMUX not found
par_tmux_command_not_found parallel: Error: not-existing not found in $PATH. par_tmux_command_not_found parallel: Error: not-existing not found in $PATH.
par_total_from_joblog bug #47086: [PATCH] Initialize total_completed from joblog par_total_from_joblog bug #47086: [PATCH] Initialize total_completed from joblog

6
testsuite/wanted-results/parallel-local-10s
View file

@ -20,7 +20,7 @@ par__pipe_tee bug #45479: --pipe/--pipepart --tee
par__pipe_tee --pipe --tee par__pipe_tee --pipe --tee
par__pipe_tee 314572800 par__pipe_tee 314572800
par__pipepart_spawn ### bug #46214: Using --pipepart doesnt spawn multiple jobs in version 20150922 par__pipepart_spawn ### bug #46214: Using --pipepart doesnt spawn multiple jobs in version 20150922
par__pipepart_spawn 1:local / 2+ / 2+2+2+ par__pipepart_spawn 1:local / 2+ / 2+
par__pipepart_tee bug #45479: --pipe/--pipepart --tee par__pipepart_tee bug #45479: --pipe/--pipepart --tee
par__pipepart_tee --pipepart --tee par__pipepart_tee --pipepart --tee
par__pipepart_tee 314572800 par__pipepart_tee 314572800
@ -300,6 +300,10 @@ par_k parallel: Warning: Try running 'parallel -jX -N X --pipe parallel -jX'
par_k parallel: Warning: or increasing 'ulimit -n' (try: ulimit -n `ulimit -Hn`) par_k parallel: Warning: or increasing 'ulimit -n' (try: ulimit -n `ulimit -Hn`)
par_k parallel: Warning: or increasing 'nofile' in /etc/security/limits.conf par_k parallel: Warning: or increasing 'nofile' in /etc/security/limits.conf
par_k parallel: Warning: or increasing /proc/sys/fs/file-max par_k parallel: Warning: or increasing /proc/sys/fs/file-max
par_k parallel: Warning: Try running 'parallel -jX -N XXX --pipe parallel -jX'
par_k parallel: Warning: or increasing 'ulimit -n' (try: ulimit -n `ulimit -Hn`)
par_k parallel: Warning: or increasing 'nofile' in /etc/security/limits.conf
par_k parallel: Warning: or increasing /proc/sys/fs/file-max
par_k begin par_k begin
par_k 1 par_k 1
par_k 2 par_k 2

123
testsuite/wanted-results/parallel-local-1s
View file

@ -270,6 +270,10 @@ par_open_files_blocks parallel: Warning: Try running 'parallel -j0 -N 2 --pipe p
par_open_files_blocks parallel: Warning: or increasing 'ulimit -n' (try: ulimit -n `ulimit -Hn`) par_open_files_blocks parallel: Warning: or increasing 'ulimit -n' (try: ulimit -n `ulimit -Hn`)
par_open_files_blocks parallel: Warning: or increasing 'nofile' in /etc/security/limits.conf par_open_files_blocks parallel: Warning: or increasing 'nofile' in /etc/security/limits.conf
par_open_files_blocks parallel: Warning: or increasing /proc/sys/fs/file-max par_open_files_blocks parallel: Warning: or increasing /proc/sys/fs/file-max
par_open_files_blocks parallel: Warning: Try running 'parallel -j0 -N 100 --pipe parallel -j0'
par_open_files_blocks parallel: Warning: or increasing 'ulimit -n' (try: ulimit -n `ulimit -Hn`)
par_open_files_blocks parallel: Warning: or increasing 'nofile' in /etc/security/limits.conf
par_open_files_blocks parallel: Warning: or increasing /proc/sys/fs/file-max
par_open_files_blocks 1 of 21 par_open_files_blocks 1 of 21
par_open_files_blocks 2 of 21 par_open_files_blocks 2 of 21
par_open_files_blocks 3 of 21 par_open_files_blocks 3 of 21
@ -375,6 +379,125 @@ par_pipepart_block 17-20
par_pipepart_block 18-20 par_pipepart_block 18-20
par_pipepart_block 19-20 par_pipepart_block 19-20
par_pipepart_block 20-20 par_pipepart_block 20-20
par_plus_slot_replacement ### show {slot} {0%} {0#}
par_plus_slot_replacement 1=1=1
par_plus_slot_replacement 2=2=2
par_plus_slot_replacement 3=3=3
par_plus_slot_replacement Seq: 001 01 1
par_plus_slot_replacement Seq: 002 02 2
par_plus_slot_replacement Seq: 003 03 3
par_plus_slot_replacement Seq: 004 04 4
par_plus_slot_replacement Seq: 005 05 5
par_plus_slot_replacement Seq: 006 06 6
par_plus_slot_replacement Seq: 007 07 7
par_plus_slot_replacement Seq: 008 08 8
par_plus_slot_replacement Seq: 009 09 9
par_plus_slot_replacement Seq: 010 10 10
par_plus_slot_replacement Seq: 011 11 11
par_plus_slot_replacement Seq: 012 12 12
par_plus_slot_replacement Seq: 013 13 13
par_plus_slot_replacement Seq: 014 14 14
par_plus_slot_replacement Seq: 015 15 15
par_plus_slot_replacement Seq: 016 16 16
par_plus_slot_replacement Seq: 017 17 17
par_plus_slot_replacement Seq: 018 18 18
par_plus_slot_replacement Seq: 019 19 19
par_plus_slot_replacement Seq: 020 20 20
par_plus_slot_replacement Seq: 021 21 21
par_plus_slot_replacement Seq: 022 22 22
par_plus_slot_replacement Seq: 023 23 23
par_plus_slot_replacement Seq: 024 24 24
par_plus_slot_replacement Seq: 025 25 25
par_plus_slot_replacement Seq: 026 26 26
par_plus_slot_replacement Seq: 027 27 27
par_plus_slot_replacement Seq: 028 28 28
par_plus_slot_replacement Seq: 029 29 29
par_plus_slot_replacement Seq: 030 30 30
par_plus_slot_replacement Seq: 031 31 31
par_plus_slot_replacement Seq: 032 32 32
par_plus_slot_replacement Seq: 033 33 33
par_plus_slot_replacement Seq: 034 34 34
par_plus_slot_replacement Seq: 035 35 35
par_plus_slot_replacement Seq: 036 36 36
par_plus_slot_replacement Seq: 037 37 37
par_plus_slot_replacement Seq: 038 38 38
par_plus_slot_replacement Seq: 039 39 39
par_plus_slot_replacement Seq: 040 40 40
par_plus_slot_replacement Seq: 041 41 41
par_plus_slot_replacement Seq: 042 42 42
par_plus_slot_replacement Seq: 043 43 43
par_plus_slot_replacement Seq: 044 44 44
par_plus_slot_replacement Seq: 045 45 45
par_plus_slot_replacement Seq: 046 46 46
par_plus_slot_replacement Seq: 047 47 47
par_plus_slot_replacement Seq: 048 48 48
par_plus_slot_replacement Seq: 049 49 49
par_plus_slot_replacement Seq: 050 50 50
par_plus_slot_replacement Seq: 051 51 51
par_plus_slot_replacement Seq: 052 52 52
par_plus_slot_replacement Seq: 053 53 53
par_plus_slot_replacement Seq: 054 54 54
par_plus_slot_replacement Seq: 055 55 55
par_plus_slot_replacement Seq: 056 56 56
par_plus_slot_replacement Seq: 057 57 57
par_plus_slot_replacement Seq: 058 58 58
par_plus_slot_replacement Seq: 059 59 59
par_plus_slot_replacement Seq: 060 60 60
par_plus_slot_replacement Seq: 061 61 61
par_plus_slot_replacement Seq: 062 62 62
par_plus_slot_replacement Seq: 063 63 63
par_plus_slot_replacement Seq: 064 64 64
par_plus_slot_replacement Seq: 065 65 65
par_plus_slot_replacement Seq: 066 66 66
par_plus_slot_replacement Seq: 067 67 67
par_plus_slot_replacement Seq: 068 68 68
par_plus_slot_replacement Seq: 069 69 69
par_plus_slot_replacement Seq: 070 70 70
par_plus_slot_replacement Seq: 071 71 71
par_plus_slot_replacement Seq: 072 72 72
par_plus_slot_replacement Seq: 073 73 73
par_plus_slot_replacement Seq: 074 74 74
par_plus_slot_replacement Seq: 075 75 75
par_plus_slot_replacement Seq: 076 76 76
par_plus_slot_replacement Seq: 077 77 77
par_plus_slot_replacement Seq: 078 78 78
par_plus_slot_replacement Seq: 079 79 79
par_plus_slot_replacement Seq: 080 80 80
par_plus_slot_replacement Seq: 081 81 81
par_plus_slot_replacement Seq: 082 82 82
par_plus_slot_replacement Seq: 083 83 83
par_plus_slot_replacement Seq: 084 84 84
par_plus_slot_replacement Seq: 085 85 85
par_plus_slot_replacement Seq: 086 86 86
par_plus_slot_replacement Seq: 087 87 87
par_plus_slot_replacement Seq: 088 88 88
par_plus_slot_replacement Seq: 089 89 89
par_plus_slot_replacement Seq: 090 90 90
par_plus_slot_replacement Seq: 091 91 91
par_plus_slot_replacement Seq: 092 92 92
par_plus_slot_replacement Seq: 093 93 93
par_plus_slot_replacement Seq: 094 94 94
par_plus_slot_replacement Seq: 095 95 95
par_plus_slot_replacement Seq: 096 96 96
par_plus_slot_replacement Seq: 097 97 97
par_plus_slot_replacement Seq: 098 98 98
par_plus_slot_replacement Seq: 099 99 99
par_plus_slot_replacement Seq: 100 100 100
par_plus_slot_replacement Slot: 001 01 1
par_plus_slot_replacement Slot: 002 02 2
par_plus_slot_replacement Slot: 003 03 3
par_plus_slot_replacement Slot: 004 04 4
par_plus_slot_replacement Slot: 005 05 5
par_plus_slot_replacement Slot: 006 06 6
par_plus_slot_replacement Slot: 007 07 7
par_plus_slot_replacement Slot: 008 08 8
par_plus_slot_replacement Slot: 009 09 9
par_plus_slot_replacement Slot: 010 10 10
par_plus_slot_replacement Slot: 011 11 11
par_plus_slot_replacement Slot: 012 12 12
par_plus_slot_replacement Slot: 013 13 13
par_plus_slot_replacement Slot: 014 14 14
par_plus_slot_replacement Slot: 015 15 15
par_profiles_with_space ### bug #42902: profiles containing arguments with space par_profiles_with_space ### bug #42902: profiles containing arguments with space
par_profiles_with_space /bin/bash=/bin/bash par_profiles_with_space /bin/bash=/bin/bash
par_profiles_with_space echo '/bin/bash=/bin/bash' par_profiles_with_space echo '/bin/bash=/bin/bash'

6
testsuite/wanted-results/parallel-local-300s
View file

@ -223,7 +223,8 @@ par_test_build_and_install rm "/tmp/parallel-install/bin"/sem || true
par_test_build_and_install ln -s parallel "/tmp/parallel-install/bin"/sem par_test_build_and_install ln -s parallel "/tmp/parallel-install/bin"/sem
par_test_build_and_install make[0]: Leaving directory '/tmp/parallel-00000000/src' par_test_build_and_install make[0]: Leaving directory '/tmp/parallel-00000000/src'
par_test_build_and_install /bin/mkdir -p '/tmp/parallel-install/share/doc/parallel' par_test_build_and_install /bin/mkdir -p '/tmp/parallel-install/share/doc/parallel'
par_test_build_and_install /bin/install -c -m 644 parallel.html env_parallel.html sem.html sql.html niceload.html parallel_tutorial.html parallel_book.html parallel_design.html parallel_alternatives.html parcat.html parset.html parsort.html parallel.texi env_parallel.texi sem.texi sql.texi niceload.texi parallel_tutorial.texi parallel_book.texi parallel_design.texi parallel_alternatives.texi parcat.texi parset.texi parsort.texi parallel.pdf env_parallel.pdf sem.pdf sql.pdf niceload.pdf parallel_tutorial.pdf parallel_book.pdf parallel_design.pdf parallel_alternatives.pdf parcat.pdf parset.pdf parsort.pdf parallel_cheat_bw.pdf '/tmp/parallel-install/share/doc/parallel' par_test_build_and_install /bin/install -c -m 644 parallel.html env_parallel.html sem.html sql.html niceload.html parallel_tutorial.html parallel_book.html parallel_design.html parallel_alternatives.html parcat.html parset.html parsort.html parallel.texi env_parallel.texi sem.texi sql.texi niceload.texi parallel_tutorial.texi parallel_book.texi parallel_design.texi parallel_alternatives.texi parcat.texi parset.texi parsort.texi parallel.rst env_parallel.rst sem.rst sql.rst niceload.rst parallel_tutorial.rst parallel_book.rst parallel_design.rst parallel_alternatives.rst parcat.rst parset.rst parsort.rst parallel.pdf env_parallel.pdf sem.pdf sql.pdf '/tmp/parallel-install/share/doc/parallel'
par_test_build_and_install /bin/install -c -m 644 niceload.pdf parallel_tutorial.pdf parallel_book.pdf parallel_design.pdf parallel_alternatives.pdf parcat.pdf parset.pdf parsort.pdf parallel_cheat_bw.pdf '/tmp/parallel-install/share/doc/parallel'
par_test_build_and_install /bin/mkdir -p '/tmp/parallel-install/share/man/man1' par_test_build_and_install /bin/mkdir -p '/tmp/parallel-install/share/man/man1'
par_test_build_and_install /bin/install -c -m 644 parallel.1 env_parallel.1 sem.1 sql.1 niceload.1 parcat.1 parset.1 parsort.1 '/tmp/parallel-install/share/man/man1' par_test_build_and_install /bin/install -c -m 644 parallel.1 env_parallel.1 sem.1 sql.1 niceload.1 parcat.1 parset.1 parsort.1 '/tmp/parallel-install/share/man/man1'
par_test_build_and_install /bin/mkdir -p '/tmp/parallel-install/share/man/man7' par_test_build_and_install /bin/mkdir -p '/tmp/parallel-install/share/man/man7'
@ -637,7 +638,8 @@ par_test_build_and_install || echo "Warning: pod2pdf not found. Using old parset
par_test_build_and_install /bin/bash: pod2pdf: command not found par_test_build_and_install /bin/bash: pod2pdf: command not found
par_test_build_and_install Warning: pod2pdf not found. Using old parset.pdf par_test_build_and_install Warning: pod2pdf not found. Using old parset.pdf
par_test_build_and_install /bin/mkdir -p '/tmp/parallel-install/share/doc/parallel' par_test_build_and_install /bin/mkdir -p '/tmp/parallel-install/share/doc/parallel'
par_test_build_and_install /bin/install -c -m 644 parallel.html env_parallel.html sem.html sql.html niceload.html parallel_tutorial.html parallel_book.html parallel_design.html parallel_alternatives.html parcat.html parset.html parsort.html parallel.texi env_parallel.texi sem.texi sql.texi niceload.texi parallel_tutorial.texi parallel_book.texi parallel_design.texi parallel_alternatives.texi parcat.texi parset.texi parsort.texi parallel.pdf env_parallel.pdf sem.pdf sql.pdf niceload.pdf parallel_tutorial.pdf parallel_book.pdf parallel_design.pdf parallel_alternatives.pdf parcat.pdf parset.pdf parsort.pdf parallel_cheat_bw.pdf '/tmp/parallel-install/share/doc/parallel' par_test_build_and_install /bin/install -c -m 644 parallel.html env_parallel.html sem.html sql.html niceload.html parallel_tutorial.html parallel_book.html parallel_design.html parallel_alternatives.html parcat.html parset.html parsort.html parallel.texi env_parallel.texi sem.texi sql.texi niceload.texi parallel_tutorial.texi parallel_book.texi parallel_design.texi parallel_alternatives.texi parcat.texi parset.texi parsort.texi parallel.rst env_parallel.rst sem.rst sql.rst niceload.rst parallel_tutorial.rst parallel_book.rst parallel_design.rst parallel_alternatives.rst parcat.rst parset.rst parsort.rst parallel.pdf env_parallel.pdf sem.pdf sql.pdf '/tmp/parallel-install/share/doc/parallel'
par_test_build_and_install /bin/install -c -m 644 niceload.pdf parallel_tutorial.pdf parallel_book.pdf parallel_design.pdf parallel_alternatives.pdf parcat.pdf parset.pdf parsort.pdf parallel_cheat_bw.pdf '/tmp/parallel-install/share/doc/parallel'
par_test_build_and_install pod2man --release='00000000' --center='parallel' \ par_test_build_and_install pod2man --release='00000000' --center='parallel' \
par_test_build_and_install --section=1 "."/parallel.pod > "."/parallel.1n \ par_test_build_and_install --section=1 "."/parallel.pod > "."/parallel.1n \
par_test_build_and_install && mv "."/parallel.1n "."/parallel.1 \ par_test_build_and_install && mv "."/parallel.1n "."/parallel.1 \

2
testsuite/wanted-results/parallel-local-30s
View file

@ -1544,7 +1544,7 @@ par_memfree Free mem: 1k
par_memfree parallel: Warning: This job was killed because it timed out: par_memfree parallel: Warning: This job was killed because it timed out:
par_memfree parallel: Warning: parallel --memfree 1t echo Free mem: ::: 1t par_memfree parallel: Warning: parallel --memfree 1t echo Free mem: ::: 1t
par_memory_leak ### Test for memory leaks par_memory_leak ### Test for memory leaks
par_memory_leak Of 100 runs of 1 job at least one should be bigger than a 3000 job run par_memory_leak Of 300 runs of 1 job at least one should be bigger than a 3000 job run
par_memory_leak Good: No memleak detected. par_memory_leak Good: No memleak detected.
par_no_newline_compress bug #41613: --compress --line-buffer - no newline par_no_newline_compress bug #41613: --compress --line-buffer - no newline
par_no_newline_compress tagstring=--tagstring {#} compress=--compress par_no_newline_compress tagstring=--tagstring {#} compress=--compress

22
testsuite/wanted-results/parallel-local-3s
View file

@ -119,7 +119,7 @@ par_eta 16
par_eta ### Test of --eta with no jobs par_eta ### Test of --eta with no jobs
par_eta par_eta
par_eta Computers / CPU cores / Max jobs to run par_eta Computers / CPU cores / Max jobs to run
par_eta 1:local / 8 / 1 par_eta 1:local / 8 / 8
par_eta par_eta ETA: 0s Left: 0 AVG: 0.00s 0 par_eta par_eta ETA: 0s Left: 0 AVG: 0.00s 0
par_exitval_signal ### Test --joblog with exitval and Test --joblog with signal -- timing dependent par_exitval_signal ### Test --joblog with exitval and Test --joblog with signal -- timing dependent
par_exitval_signal exitval=128+6 OK par_exitval_signal exitval=128+6 OK
@ -213,7 +213,7 @@ par_progress 16
par_progress ### Test of --progress with no jobs par_progress ### Test of --progress with no jobs
par_progress par_progress
par_progress Computers / CPU cores / Max jobs to run par_progress Computers / CPU cores / Max jobs to run
par_progress 1:local / 8 / 1 par_progress 1:local / 8 / 8
par_progress par_progress 0 par_progress par_progress 0
par_replacement_slashslash ### Test {//} par_replacement_slashslash ### Test {//}
par_replacement_slashslash . a par_replacement_slashslash . a
@ -291,6 +291,24 @@ par_sqlworker_hostname <hostname>
par_sqlworker_hostname <hostname> par_sqlworker_hostname <hostname>
par_sshdelay ### test --sshdelay par_sshdelay ### test --sshdelay
par_sshdelay OK par_sshdelay OK
par_tee_too_many_args ### Fail if there are more arguments than --jobs
par_tee_too_many_args 1 1
par_tee_too_many_args 1 10
par_tee_too_many_args 1 11
par_tee_too_many_args 2 2
par_tee_too_many_args 3 3
par_tee_too_many_args 4 4
par_tee_too_many_args parallel: Warning: Only enough file handles to run 251 jobs in parallel.
par_tee_too_many_args parallel: Warning: Try running 'parallel -j0 -N 251 --pipe parallel -j0'
par_tee_too_many_args parallel: Warning: or increasing 'ulimit -n' (try: ulimit -n `ulimit -Hn`)
par_tee_too_many_args parallel: Warning: or increasing 'nofile' in /etc/security/limits.conf
par_tee_too_many_args parallel: Warning: or increasing /proc/sys/fs/file-max
par_tee_too_many_args parallel: Warning: No more file handles.
par_tee_too_many_args parallel: Warning: Try running 'parallel -j0 -N 100 --pipe parallel -j0'
par_tee_too_many_args parallel: Warning: or increasing 'ulimit -n' (try: ulimit -n `ulimit -Hn`)
par_tee_too_many_args parallel: Warning: or increasing 'nofile' in /etc/security/limits.conf
par_tee_too_many_args parallel: Warning: or increasing /proc/sys/fs/file-max
par_tee_too_many_args parallel: Error: --tee requires --jobs to be higher. Try --jobs 0.
par_tee_with_premature_close --tee --pipe should send all data to all commands par_tee_with_premature_close --tee --pipe should send all data to all commands
par_tee_with_premature_close even if a command closes stdin before reading everything par_tee_with_premature_close even if a command closes stdin before reading everything
par_tee_with_premature_close tee with --output-error=warn-nopipe support par_tee_with_premature_close tee with --output-error=warn-nopipe support

2
testsuite/wanted-results/parallel-local-ssh3
View file

@ -11,7 +11,7 @@ par_nonall_should_not_block ### bug #47608: parallel --nonall -S lo 'echo ::: '
par_nonall_should_not_block ::: par_nonall_should_not_block :::
par_progress_text_max_jobs_to_run ### bug #49404: "Max jobs to run" does not equal the number of jobs specified when using GNU Parallel on remote server? par_progress_text_max_jobs_to_run ### bug #49404: "Max jobs to run" does not equal the number of jobs specified when using GNU Parallel on remote server?
par_progress_text_max_jobs_to_run should give 10 running jobs par_progress_text_max_jobs_to_run should give 10 running jobs
par_progress_text_max_jobs_to_run 1:lo / 16 / 10 par_progress_text_max_jobs_to_run 1:lo / 16 / 16
par_quoting_for_onall ### bug #35427: quoting of {2} broken for --onall par_quoting_for_onall ### bug #35427: quoting of {2} broken for --onall
par_quoting_for_onall /bin/ls par_quoting_for_onall /bin/ls
par_remote_function_nice ### functions and --nice par_remote_function_nice ### functions and --nice

251
testsuite/wanted-results/parallel-remote1
View file

@ -1,129 +1,126 @@
### Test use special ssh par_controlmaster_is_faster ### bug #41964: --controlmaster not seems to reuse OpenSSH connections to the same host
TODO test ssh with > 9 simultaneous par_controlmaster_is_faster With --controlmaster - finish first
1 par_controlmaster_is_faster No --controlmaster - finish last
2 par_filter_hosts_different_errors ### --filter-hosts - OK, non-such-user, connection refused, wrong host
3 par_filter_hosts_different_errors aspire
4 par_filter_hosts_no_ssh_nxserver ### test --filter-hosts with server w/o ssh, non-existing server
5 par_filter_hosts_no_ssh_nxserver vagrant@parallel-server1
6 par_filter_hosts_no_ssh_nxserver vagrant@parallel-server2
7 par_filter_hosts_no_ssh_nxserver vagrant@parallel-server3
8 par_special_ssh ### Test use special ssh
9 par_special_ssh TODO test ssh with > 9 simultaneous
10 par_special_ssh 1
11 par_special_ssh 2
12 par_special_ssh 3
13 par_special_ssh 4
14 par_special_ssh 5
15 par_special_ssh 6
16 par_special_ssh 7
17 par_special_ssh 8
18 par_special_ssh 9
19 par_special_ssh 10
20 par_special_ssh 11
21 par_special_ssh 12
22 par_special_ssh 13
23 par_special_ssh 14
24 par_special_ssh 15
25 par_special_ssh 16
26 par_special_ssh 17
27 par_special_ssh 18
28 par_special_ssh 19
29 par_special_ssh 20
30 par_special_ssh 21
31 par_special_ssh 22
32 par_special_ssh 23
33 par_special_ssh 24
34 par_special_ssh 25
35 par_special_ssh 26
36 par_special_ssh 27
37 par_special_ssh 28
38 par_special_ssh 29
39 par_special_ssh 30
40 par_special_ssh 31
41 par_special_ssh 32
42 par_special_ssh 33
43 par_special_ssh 34
44 par_special_ssh 35
45 par_special_ssh 36
46 par_special_ssh 37
47 par_special_ssh 38
48 par_special_ssh 39
49 par_special_ssh 40
50 par_special_ssh 41
51 par_special_ssh 42
52 par_special_ssh 43
53 par_special_ssh 44
54 par_special_ssh 45
55 par_special_ssh 46
56 par_special_ssh 47
57 par_special_ssh 48
58 par_special_ssh 49
59 par_special_ssh 50
60 par_special_ssh 51
61 par_special_ssh 52
62 par_special_ssh 53
63 par_special_ssh 54
64 par_special_ssh 55
65 par_special_ssh 56
66 par_special_ssh 57
67 par_special_ssh 58
68 par_special_ssh 59
69 par_special_ssh 60
70 par_special_ssh 61
71 par_special_ssh 62
72 par_special_ssh 63
73 par_special_ssh 64
74 par_special_ssh 65
75 par_special_ssh 66
76 par_special_ssh 67
77 par_special_ssh 68
78 par_special_ssh 69
79 par_special_ssh 70
80 par_special_ssh 71
81 par_special_ssh 72
82 par_special_ssh 73
83 par_special_ssh 74
84 par_special_ssh 75
85 par_special_ssh 76
86 par_special_ssh 77
87 par_special_ssh 78
88 par_special_ssh 79
89 par_special_ssh 80
90 par_special_ssh 81
91 par_special_ssh 82
92 par_special_ssh 83
93 par_special_ssh 84
94 par_special_ssh 85
95 par_special_ssh 86
96 par_special_ssh 87
97 par_special_ssh 88
98 par_special_ssh 89
99 par_special_ssh 90
100 par_special_ssh 91
echo '### test --timeout --retries' par_special_ssh 92
### test --timeout --retries par_special_ssh 93
parallel -j0 --timeout 5 --retries 3 -k ssh {} echo {} ::: 192.168.1.197 8.8.8.8 vagrant@parallel-server1 vagrant@parallel-server2 vagrant@parallel-server3 par_special_ssh 94
vagrant@parallel-server1 par_special_ssh 95
vagrant@parallel-server2 par_special_ssh 96
vagrant@parallel-server3 par_special_ssh 97
echo '### test --filter-hosts with server w/o ssh, non-existing server' par_special_ssh 98
### test --filter-hosts with server w/o ssh, non-existing server par_special_ssh 99
parallel -S 192.168.1.197,8.8.8.8,vagrant@parallel-server1,vagrant@parallel-server2,vagrant@parallel-server3 --filter-hosts --nonall -k --tag echo par_special_ssh 100
vagrant@parallel-server1 par_timeout_retries ### test --timeout --retries
vagrant@parallel-server2 par_timeout_retries ssh: connect to host 192.168.1.197 port 22: No route to host par_timeout_retries
vagrant@parallel-server3 par_timeout_retries parallel: Warning: This job was killed because it timed out:
echo '### bug #41964: --controlmaster not seems to reuse OpenSSH connections to the same host' par_timeout_retries parallel: Warning: ssh 8.8.8.8 echo 8.8.8.8
### bug #41964: --controlmaster not seems to reuse OpenSSH connections to the same host par_timeout_retries parallel: Warning: This job was killed because it timed out:
(parallel -S vagrant@parallel-server1 true ::: {1..20}; echo No --controlmaster - finish last) & (parallel -M -S vagrant@parallel-server1 true ::: {1..20}; echo With --controlmaster - finish first) & wait par_timeout_retries parallel: Warning: ssh 8.8.8.8 echo 8.8.8.8
With --controlmaster - finish first par_timeout_retries parallel: Warning: This job was killed because it timed out:
No --controlmaster - finish last par_timeout_retries parallel: Warning: ssh 8.8.8.8 echo 8.8.8.8
echo '### --filter-hosts - OK, non-such-user, connection refused, wrong host' par_timeout_retries vagrant@parallel-server1
### --filter-hosts - OK, non-such-user, connection refused, wrong host par_timeout_retries vagrant@parallel-server2
parallel --nonall --filter-hosts -S localhost,NoUser@localhost,154.54.72.206,"ssh 5.5.5.5" hostname par_timeout_retries vagrant@parallel-server3
aspire par_workdir_in_HOME ### test --workdir . in $HOME
echo '### test --workdir . in $HOME' par_workdir_in_HOME OK
### test --workdir . in $HOME
cd && mkdir -p parallel-test && cd parallel-test && echo OK > testfile && parallel --workdir . --transfer -S vagrant@parallel-server1 cat {} ::: testfile
OK
echo '### TODO: test --filter-hosts proxied through the one host' echo '### TODO: test --filter-hosts proxied through the one host'
### TODO: test --filter-hosts proxied through the one host ### TODO: test --filter-hosts proxied through the one host