Released as 20210422 ('Ever Given')

This commit is contained in:
Ole Tange 2021-04-22 18:20:41 +02:00
parent 22244d765a
commit 3cfcbc6920
83 changed files with 698 additions and 544 deletions

2
.gitignore vendored
View file

@ -1,3 +1,5 @@
# SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
# SPDX-License-Identifier: GPL-3.0-or-later
*~ *~
*.pdf *.pdf
Makefile Makefile

View file

@ -19,3 +19,7 @@ Or:
DOI https://doi.org/10.5281/zenodo.11460 DOI https://doi.org/10.5281/zenodo.11460
This helps funding further development. This helps funding further development.
SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
SPDX-License-Identifier: GFDL-1.3-or-later
SPDX-License-Identifier: CC-BY-SA-4.0

View file

@ -130,6 +130,7 @@ reuse:
(wd=`pwd`; \ (wd=`pwd`; \
cd /tmp/reuse; \ cd /tmp/reuse; \
git clone "$$wd"; \ git clone "$$wd"; \
cd parallel; \
reuse lint || \ reuse lint || \
(echo Use; \ (echo Use; \
echo ' reuse addheader --copyright="Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc." --license="GPL-3.0-or-later" source-files'; \ echo ' reuse addheader --copyright="Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc." --license="GPL-3.0-or-later" source-files'; \

View file

@ -881,6 +881,7 @@ reuse:
(wd=`pwd`; \ (wd=`pwd`; \
cd /tmp/reuse; \ cd /tmp/reuse; \
git clone "$$wd"; \ git clone "$$wd"; \
cd parallel; \
reuse lint || \ reuse lint || \
(echo Use; \ (echo Use; \
echo ' reuse addheader --copyright="Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc." --license="GPL-3.0-or-later" source-files'; \ echo ' reuse addheader --copyright="Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc." --license="GPL-3.0-or-later" source-files'; \

29
NEWS
View file

@ -1,3 +1,32 @@
20210422
New in this release:
* Bug fixes and man page updates.
News about GNU Parallel:
* Bioinformatics tutorials - linux and shell advanced - parallel
https://www.youtube.com/watch?v=5leL8pyl0XA
* GNU Parallel for quick gains
https://edbennett.github.io/high-performance-python/04-gnu-parallel/index.html
* Processing Linux Commands in Parallel
https://www.baeldung.com/linux/processing-commands-in-parallel
* GNU parallel
https://docs-research-it.berkeley.edu/services/high-performance-computing/user-guide/running-your-jobs/gnu-parallel/
* GNU Parallel 활용 가이드https://genoglobe.com/kribb/gnu_parallel
* Parallel Grep and Awk
https://www.highonscience.com/blog/2021/03/21/parallel-grep/
* Getting things done with shell scripting
https://doma.dev/blog/get-things-done-with-bash/
20210322 20210322
* Bug fixes and man page updates. * Bug fixes and man page updates.

27
README
View file

@ -57,11 +57,11 @@ document.
Full installation of GNU Parallel is as simple as: Full installation of GNU Parallel is as simple as:
wget https://ftpmirror.gnu.org/parallel/parallel-20210322.tar.bz2 wget https://ftpmirror.gnu.org/parallel/parallel-20210422.tar.bz2
wget https://ftpmirror.gnu.org/parallel/parallel-20210322.tar.bz2.sig wget https://ftpmirror.gnu.org/parallel/parallel-20210422.tar.bz2.sig
gpg parallel-20210322.tar.bz2.sig gpg parallel-20210422.tar.bz2.sig
bzip2 -dc parallel-20210322.tar.bz2 | tar xvf - bzip2 -dc parallel-20210422.tar.bz2 | tar xvf -
cd parallel-20210322 cd parallel-20210422
./configure && make && sudo make install ./configure && make && sudo make install
@ -70,11 +70,11 @@ Full installation of GNU Parallel is as simple as:
If you are not root you can add ~/bin to your path and install in If you are not root you can add ~/bin to your path and install in
~/bin and ~/share: ~/bin and ~/share:
wget https://ftpmirror.gnu.org/parallel/parallel-20210322.tar.bz2 wget https://ftpmirror.gnu.org/parallel/parallel-20210422.tar.bz2
wget https://ftpmirror.gnu.org/parallel/parallel-20210322.tar.bz2.sig wget https://ftpmirror.gnu.org/parallel/parallel-20210422.tar.bz2.sig
gpg parallel-20210322.tar.bz2.sig gpg parallel-20210422.tar.bz2.sig
bzip2 -dc parallel-20210322.tar.bz2 | tar xvf - bzip2 -dc parallel-20210422.tar.bz2 | tar xvf -
cd parallel-20210322 cd parallel-20210422
./configure --prefix=$HOME && make && make install ./configure --prefix=$HOME && make && make install
Or if your system lacks 'make' you can simply copy src/parallel Or if your system lacks 'make' you can simply copy src/parallel
@ -122,8 +122,8 @@ will love you for it.
When using programs that use GNU Parallel to process data for When using programs that use GNU Parallel to process data for
publication please cite: publication please cite:
Tange, O. (2021, March 22). GNU Parallel 20210322 ('2002-01-06'). Tange, O. (2021, April 22). GNU Parallel 20210422 ('Ever Given').
Zenodo. https://doi.org/10.5281/zenodo.4628277 Zenodo. https://doi.org/10.5281/zenodo.4710607
Copyright (C) 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, Copyright (C) 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015,
2016, 2017, 2018, 2019, 2020, 2021 Ole Tange, http://ole.tange.dk and 2016, 2017, 2018, 2019, 2020, 2021 Ole Tange, http://ole.tange.dk and
@ -142,3 +142,6 @@ if you system has split up Perl into multiple packages then these are
the important ones: the important ones:
opkg install perlbase-getopt perlbase-ipc procps-ng-ps perlbase-mime opkg install perlbase-getopt perlbase-ipc procps-ng-ps perlbase-mime
SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
SPDX-License-Identifier: GPL-3.0-or-later

View file

@ -1,5 +1,8 @@
#!/bin/bash #!/bin/bash
# I always forget how to make ./configure # SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
# SPDX-License-Identifier: GPL-3.0-or-later
autoreconf --install -W gnu autoreconf --install -W gnu &&
./configure &&
make -j

20
configure vendored
View file

@ -1,6 +1,6 @@
#! /bin/sh #! /bin/sh
# Guess values for system-dependent variables and create Makefiles. # Guess values for system-dependent variables and create Makefiles.
# Generated by GNU Autoconf 2.69 for parallel 20210322. # Generated by GNU Autoconf 2.69 for parallel 20210422.
# #
# Report bugs to <bug-parallel@gnu.org>. # Report bugs to <bug-parallel@gnu.org>.
# #
@ -579,8 +579,8 @@ MAKEFLAGS=
# Identity of this package. # Identity of this package.
PACKAGE_NAME='parallel' PACKAGE_NAME='parallel'
PACKAGE_TARNAME='parallel' PACKAGE_TARNAME='parallel'
PACKAGE_VERSION='20210322' PACKAGE_VERSION='20210422'
PACKAGE_STRING='parallel 20210322' PACKAGE_STRING='parallel 20210422'
PACKAGE_BUGREPORT='bug-parallel@gnu.org' PACKAGE_BUGREPORT='bug-parallel@gnu.org'
PACKAGE_URL='' PACKAGE_URL=''
@ -1214,7 +1214,7 @@ if test "$ac_init_help" = "long"; then
# Omit some internal or obsolete options to make the list less imposing. # Omit some internal or obsolete options to make the list less imposing.
# This message is too long to be a string in the A/UX 3.1 sh. # This message is too long to be a string in the A/UX 3.1 sh.
cat <<_ACEOF cat <<_ACEOF
\`configure' configures parallel 20210322 to adapt to many kinds of systems. \`configure' configures parallel 20210422 to adapt to many kinds of systems.
Usage: $0 [OPTION]... [VAR=VALUE]... Usage: $0 [OPTION]... [VAR=VALUE]...
@ -1281,7 +1281,7 @@ fi
if test -n "$ac_init_help"; then if test -n "$ac_init_help"; then
case $ac_init_help in case $ac_init_help in
short | recursive ) echo "Configuration of parallel 20210322:";; short | recursive ) echo "Configuration of parallel 20210422:";;
esac esac
cat <<\_ACEOF cat <<\_ACEOF
@ -1357,7 +1357,7 @@ fi
test -n "$ac_init_help" && exit $ac_status test -n "$ac_init_help" && exit $ac_status
if $ac_init_version; then if $ac_init_version; then
cat <<\_ACEOF cat <<\_ACEOF
parallel configure 20210322 parallel configure 20210422
generated by GNU Autoconf 2.69 generated by GNU Autoconf 2.69
Copyright (C) 2012 Free Software Foundation, Inc. Copyright (C) 2012 Free Software Foundation, Inc.
@ -1374,7 +1374,7 @@ cat >config.log <<_ACEOF
This file contains any messages produced by compilers while This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake. running configure, to aid debugging if configure makes a mistake.
It was created by parallel $as_me 20210322, which was It was created by parallel $as_me 20210422, which was
generated by GNU Autoconf 2.69. Invocation command line was generated by GNU Autoconf 2.69. Invocation command line was
$ $0 $@ $ $0 $@
@ -2237,7 +2237,7 @@ fi
# Define the identity of the package. # Define the identity of the package.
PACKAGE='parallel' PACKAGE='parallel'
VERSION='20210322' VERSION='20210422'
cat >>confdefs.h <<_ACEOF cat >>confdefs.h <<_ACEOF
@ -2880,7 +2880,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
# report actual input values of CONFIG_FILES etc. instead of their # report actual input values of CONFIG_FILES etc. instead of their
# values after options handling. # values after options handling.
ac_log=" ac_log="
This file was extended by parallel $as_me 20210322, which was This file was extended by parallel $as_me 20210422, which was
generated by GNU Autoconf 2.69. Invocation command line was generated by GNU Autoconf 2.69. Invocation command line was
CONFIG_FILES = $CONFIG_FILES CONFIG_FILES = $CONFIG_FILES
@ -2942,7 +2942,7 @@ _ACEOF
cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`" ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`"
ac_cs_version="\\ ac_cs_version="\\
parallel config.status 20210322 parallel config.status 20210422
configured by $0, generated by GNU Autoconf 2.69, configured by $0, generated by GNU Autoconf 2.69,
with options \\"\$ac_cs_config\\" with options \\"\$ac_cs_config\\"

View file

@ -1,4 +1,4 @@
AC_INIT([parallel], [20210322], [bug-parallel@gnu.org]) AC_INIT([parallel], [20210422], [bug-parallel@gnu.org])
AM_INIT_AUTOMAKE([-Wall -Werror foreign]) AM_INIT_AUTOMAKE([-Wall -Werror foreign])
AC_CONFIG_HEADERS([config.h]) AC_CONFIG_HEADERS([config.h])
AC_CONFIG_FILES([ AC_CONFIG_FILES([

View file

@ -1,8 +1,5 @@
Luk filen ved EOF - lad være med bare at læse videre. # SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
# SPDX-License-Identifier: GPL-3.0-or-later
> /tmp/ged; tail -f /tmp/ged| xargs -n1 -E eof & sleep 1; echo echo a >>/tmp/ged; echo eof >>/tmp/ged; seq 4 >>/tmp/ged; wait
> /tmp/ged; tail -f /tmp/ged| parallel -n1 -E eof & sleep 1; echo echo a >>/tmp/ged; echo eof >>/tmp/ged; seq 4 >>/tmp/ged; wait
niceload seeks last column: niceload seeks last column:
@ -10,17 +7,8 @@ iostat -x 1 2
niceload --start-condition niceload --start-condition
niceload should prioritize jobs and only unsuspend the highest
priority job. If the running job with lowest pri has run for 1 sec:
Consider unsuspending next pri job.
Til QUOTING: Til QUOTING:
cat <<'_EOF' | parallel -v echo
awk -v FS="\",\"" '{print $1, $3, $4, $5, $9, $14}' | grep -v "#" | sed -e '1d' -e 's/\"//g' -e 's/\/\/\//\t/g' | cut -f1-6,11 | sed -e 's/\/\//\t/g' -e 's/ /\t/g
_EOF
FN="two spaces" FN="two spaces"
echo 1 | parallel -q echo {} "$FN" echo 1 | parallel -q echo {} "$FN"
# Prints 2 spaces between 'two' and 'spaces' # Prints 2 spaces between 'two' and 'spaces'
@ -49,16 +37,6 @@ echo 1 | parallel echo {} $FN4 \; echo $FN4 {}
locate .gz | parallel -X find {} -size +1000 -size -2000 | parallel --workdir ... -S .. --trc {/}.bz2 'zcat {} | bzip2 > {/}.bz2' locate .gz | parallel -X find {} -size +1000 -size -2000 | parallel --workdir ... -S .. --trc {/}.bz2 'zcat {} | bzip2 > {/}.bz2'
== Compare ==
Unchanged since 2008 http://code.google.com/p/spawntool/
Unchanged since 2011 http://code.google.com/p/push/
== Bug? ==
.parallel/config with --long-options
== SQL == == SQL ==
Example with %0a as newline Example with %0a as newline
@ -66,12 +44,6 @@ sql :my_postgres?'\dt %0a SELECT * FROM users'
cat ~/.sql/aliases | parallel --colsep '\s' sql {1} '"select 0.14+3;" | grep -q 3.14 || (echo dead: {1}; exit 1)' cat ~/.sql/aliases | parallel --colsep '\s' sql {1} '"select 0.14+3;" | grep -q 3.14 || (echo dead: {1}; exit 1)'
== FEX ==
fex syntax for splitting fields
http://www.semicomplete.com/projects/fex/
sql :foo 'select * from bar' | parallel --fex '|{1,2}' do_stuff {2} {1}
--autocolsep: Læs alle linjer. --autocolsep: Læs alle linjer.
Prøv fastlængde: Find tegn, som står i alle linjer på de samme pladser. Risiko for falske pos Prøv fastlængde: Find tegn, som står i alle linjer på de samme pladser. Risiko for falske pos
@ -102,13 +74,17 @@ colsep = [sepchars]{no_of_sepchars}
Til inspiration. Til inspiration.
Hvis du stadig er ved at lave post- eller visitkort ting, så kunne du evt tilføje en QR code under frimærket. Med MECARD tagget kan flere tags gemmes i en og samme fil: Hvis du stadig er ved at lave post- eller visitkort ting, så kunne du
evt tilføje en QR code under frimærket. Med MECARD tagget kan flere
tags gemmes i en og samme fil:
qrencode -l L -o x.png "MECARD:N:GNU Parallel;EMAIL:parallel@gnu.org;URL:gnu.org/software/parallel;" qrencode -l L -o x.png "MECARD:N:GNU
Parallel;EMAIL:parallel@gnu.org;URL:gnu.org/software/parallel;"
Den ser OK ud i en Androide tlf. Den ser OK ud i en Androide tlf.
Husk at skrive indholdet under billedet, det er irreterende at skulle gætte. Husk at skrive indholdet under billedet, det er irreterende at skulle
gætte.
GNU parallel is a UNIX-tool for running commands in parallel. GNU parallel is a UNIX-tool for running commands in parallel.
@ -117,25 +93,6 @@ To gzip all files running one job per CPU write:
Watch the intro video to learn more: http://pi.dk/1 Watch the intro video to learn more: http://pi.dk/1
Or read more about GNU parallel: http://gnu.org/s/parallel Or read more about GNU parallel: http://gnu.org/s/parallel
job->start():
$jobslot = Global::jobslot->$sshlogin
sub get_jobslot {
my $sshlogin = shift;
my $jobslot_id = pop @Global::jobslots{$sshlogin};
if not defined $jobslot_id {
$jobslot_id = ++$Global::max_jobslot_id;
}
return $jobslot_id;
}
sub release_jobslot {
my $sshlogin = shift;
my $jobslot_id = shift;
push @Global::jobslots{$sshlogin}, $jobslot_id;
}
Test sshlogins in parallel. Assume parallel is in path
seq 1 10 | parallel -I {o} 'seq 1 255 | parallel echo ssh -oNoHostAuthenticationForLocalhost=true 127.0.{o}.{}' >/tmp/sshloginfile seq 1 10 | parallel -I {o} 'seq 1 255 | parallel echo ssh -oNoHostAuthenticationForLocalhost=true 127.0.{o}.{}' >/tmp/sshloginfile
seq 1 1000 | parallel --sshloginfile /tmp/sshloginfile echo seq 1 1000 | parallel --sshloginfile /tmp/sshloginfile echo
@ -720,50 +677,3 @@ parallel -a table_file.tsv --colsep '\t' mycmd -o {2} {3} -i {1}
# Run traceroute in parallel, but keep the output order the same # Run traceroute in parallel, but keep the output order the same
parallel -k traceroute ::: foss.org.my debian.org freenetproject.org parallel -k traceroute ::: foss.org.my debian.org freenetproject.org
Test of signal passing through ssh
#!/bin/bash
SERVER1=parallel-server3
SERVER2=parallel-server2
export BG_PROC
start_remote_sleep() {
parallel -D -u -S parallel@$SERVER2 sleep ::: 370 &
BG_PROC=$!
while ! ssh parallel@$SERVER2 ps -A -o cmd | grep -q '^sleep 370' ; do
sleep 0.3
done
}
stop_local_parallel() {
kill -9 $BG_PROC
}
check_and_stop_remote_sleep() {
ssh parallel@$SERVER2 ps -A -o cmd | grep '^sleep 370'
ssh parallel@$SERVER2 killall sleep
}
echo '### Test kill signals'
start_remote_sleep 2>/dev/null
kill -1 $BG_PROC
check_and_stop_remote_sleep
sub propagate_signal {
my $signal = shift;
# $signal = "KILL";
::debug("Sending $signal to ",keys %Global::running);
kill $signal, keys %Global::running;
if(defined $Global::original_sig{$signal}) {
&{$Global::original_sig{$signal}};
}
}
my %do_not_propagate = map { $_ => 1 } qw(TTOU TTIN CONT TSTP __WARN__ __DIE__);
for (keys %SIG) {
$do_not_propagate{$_} and next;
$SIG{$_} = eval 'sub { propagate_signal("'.$_.'"); };';
}

View file

@ -1,5 +1,8 @@
#!/bin/bash #!/bin/bash
# SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
# SPDX-License-Identifier: GPL-3.0-or-later
# Create a boxplot of running 1000 jobs 10 times on all released # Create a boxplot of running 1000 jobs 10 times on all released
# versions of GNU Parallel # versions of GNU Parallel

View file

@ -1,5 +1,10 @@
# SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
# SPDX-License-Identifier: GFDL-1.3-or-later
# SPDX-License-Identifier: CC-BY-SA-4.0
Quote of the month: Quote of the month:
Parallel is amazing! Parallel is amazing!
-- fatboy93@reddit -- fatboy93@reddit
@ -121,10 +126,14 @@ https://negfeedback.blogspot.com/2020/05/indispensable-command-line-tools.html
=== Used === === Used ===
GNU Parallel is your friend.
Can shorten that time by X cores.
-- iRODS @irods@twitter
GNU Parallel is my new favorite thing GNU Parallel is my new favorite thing
-- Will Tejeda @thewilltejeda -- Will Tejeda @thewilltejeda
GNU Parallel is a priceless timesaver GNU Parallel is a priceless timesaver
-- June "Wakalix" @TWakalix@twitter -- June "Wakalix" @TWakalix@twitter
I think many people would be surprised to learn that GNU parallel is I think many people would be surprised to learn that GNU parallel is

View file

@ -1,5 +1,8 @@
#!/usr/bin/perl #!/usr/bin/perl
# SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
# SPDX-License-Identifier: GPL-3.0-or-later
$line=""; $line="";
while(<>) { while(<>) {

View file

@ -1,3 +1,7 @@
# SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
# SPDX-License-Identifier: GFDL-1.3-or-later
# SPDX-License-Identifier: CC-BY-SA-4.0
=head1 GNU Parallel 10 year anniversery - 2020-04-22 =head1 GNU Parallel 10 year anniversery - 2020-04-22
Git log entry 2010-04-22: Git log entry 2010-04-22:

View file

@ -1,3 +1,7 @@
# SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
# SPDX-License-Identifier: GFDL-1.3-or-later
# SPDX-License-Identifier: CC-BY-SA-4.0
= Notes on how to release new version = = Notes on how to release new version =
== alpha/beta/production == == alpha/beta/production ==
@ -40,7 +44,7 @@ gpl v3
10.5281/zenodo.4381888 10.5281/zenodo.4381888
DOINO=3956817 DOINO=3956817
TAG='Privacy Shield' SPCTAG='Privacy Shield'
# Update version # Update version
. packager/releasescripts/updateversion . packager/releasescripts/updateversion
@ -50,6 +54,7 @@ TAG='Privacy Shield'
=== Reconfig autoconf/automake, build tar.bz2 and test it === === Reconfig autoconf/automake, build tar.bz2 and test it ===
. .last-doitag.txt
make reconf && make reconf &&
make pack_unpack_and_test_build make pack_unpack_and_test_build
@ -81,6 +86,41 @@ make
https://build.opensuse.org/package/show/home:tange/parallel https://build.opensuse.org/package/show/home:tange/parallel
# Check that one .deb (Debian 5.0) and one .rpm (CentOS) end with 'succeeded' # Check that one .deb (Debian 5.0) and one .rpm (CentOS) end with 'succeeded'
== Upload LBRY ==
lbry://@GnuParallel#4/parallel-20210322#4
An easy way to support GNU Parallel is to tip on LBRY.
:
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
Thumbnail: https://www.gnu.org/software/parallel/logo-gray+black10000.png
Tags: gnu parallel software
file_path="parallel-20210422.tar.bz2"
name="GNU Parallel $YYYYMMDD ('$SPCTAG')"
title="$name"
description="$description"
lbrynet publish \
--bid=0.01 \
--file_path="$file_path" \
--name="$name" \
--title="$title" \
--description="$description" \
--author="$author" \
$languages_opt \
$tags_opt \
--license="$license" \
--thumbnail_url="$thumbnail_url" \
--release_time="$release_time" \
--channel_name="$channel_name" \
== Update website == == Update website ==
http://www.gnu.org/software/parallel/ http://www.gnu.org/software/parallel/
@ -113,11 +153,11 @@ git diff
. .last-doitag.txt . .last-doitag.txt
echo "Released as $YYYYMMDD ('$TAG')." | grep MyTag && (STOP;STOP;STOP) echo "Released as $YYYYMMDD ('$TAG')." | grep MyTag && (STOP;STOP;STOP)
echo "$TAG" | grep ' ' && (STOP;STOP;STOP) echo "$TAG" | grep ' ' && (STOP;STOP;STOP)
echo "Released as $YYYYMMDD ('$TAG')." echo "Released as $YYYYMMDD ('$SPCTAG')."
git commit -a -m "Released as $YYYYMMDD ('$TAG')" git commit -a -m "Released as $YYYYMMDD ('$SPCTAG')"
git tag -s -u 88888888 -m "Released as $YYYYMMDD ('$TAG')" $TAG git tag -s -u 88888888 -m "Released as $YYYYMMDD ('$SPCTAG')" $TAG
git tag -s -u 88888888 -m "Released as $YYYYMMDD ('$TAG')" $YYYYMMDD git tag -s -u 88888888 -m "Released as $YYYYMMDD ('$SPCTAG')" $YYYYMMDD
torsocks git push torsocks git push
torsocks git push origin $TAG torsocks git push origin $TAG
@ -201,9 +241,9 @@ from:tange@gnu.org
to:parallel@gnu.org, bug-parallel@gnu.org to:parallel@gnu.org, bug-parallel@gnu.org
stable-bcc: Jesse Alama <jessealama@fastmail.fm> stable-bcc: Jesse Alama <jessealama@fastmail.fm>
Subject: GNU Parallel 20210422 ('<<>>') released <<[stable]>> Subject: GNU Parallel 20210422 ('Ever Given') released <<[stable]>>
GNU Parallel 20210322 ('<<>>') <<[stable]>> has been released. It is available for download at: http://ftpmirror.gnu.org/parallel/ GNU Parallel 20210322 ('Ever Given') <<[stable]>> has been released. It is available for download at: lbry://@GnuParallel:4
<<No new functionality was introduced so this is a good candidate for a stable release.>> <<No new functionality was introduced so this is a good candidate for a stable release.>>
@ -213,7 +253,9 @@ It does not have to be as detailed as Juan's. It is perfectly fine if you just s
Quote of the month: Quote of the month:
<<>> GNU Parallel is your friend.
Can shorten that time by X cores.
-- iRODS @irods@twitter
New in this release: New in this release:
@ -221,7 +263,19 @@ New in this release:
News about GNU Parallel: News about GNU Parallel:
<<>> * Bioinformatics tutorials - linux and shell advanced - parallel https://www.youtube.com/watch?v=5leL8pyl0XA
* GNU Parallel for quick gains https://edbennett.github.io/high-performance-python/04-gnu-parallel/index.html
* Processing Linux Commands in Parallel https://www.baeldung.com/linux/processing-commands-in-parallel
* GNU parallel https://docs-research-it.berkeley.edu/services/high-performance-computing/user-guide/running-your-jobs/gnu-parallel/
* GNU Parallel 활용 가이드https://genoglobe.com/kribb/gnu_parallel
* Parallel Grep and Awk https://www.highonscience.com/blog/2021/03/21/parallel-grep/
* Getting things done with shell scripting https://doma.dev/blog/get-things-done-with-bash/
Get the book: GNU Parallel 2018 http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html Get the book: GNU Parallel 2018 http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html

View file

@ -1,2 +1,6 @@
#!/bin/sh #!/bin/sh
# SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
# SPDX-License-Identifier: GPL-3.0-or-later
0launch http://0install.net/2007/interfaces/FeedLint.xml http://git.savannah.gnu.org/cgit/parallel.git/plain/packager/0install/parallel.xml 0launch http://0install.net/2007/interfaces/FeedLint.xml http://git.savannah.gnu.org/cgit/parallel.git/plain/packager/0install/parallel.xml

View file

@ -1,5 +1,7 @@
#!/bin/bash #!/bin/bash
# SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
# SPDX-License-Identifier: GPL-3.0-or-later
version=$1 version=$1
if [ -z "${version}" ]; then if [ -z "${version}" ]; then

View file

@ -17,6 +17,10 @@
# along with this program; if not, see <http://www.gnu.org/licenses/> # along with this program; if not, see <http://www.gnu.org/licenses/>
# or write to the Free Software Foundation, Inc., 51 Franklin St, # or write to the Free Software Foundation, Inc., 51 Franklin St,
# Fifth Floor, Boston, MA 02110-1301 USA # Fifth Floor, Boston, MA 02110-1301 USA
#
# SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
#
# SPDX-License-Identifier: GPL-3.0-or-later
if [ "$EUID" -ne 0 ] if [ "$EUID" -ne 0 ]
then echo "Please run as root" then echo "Please run as root"

View file

@ -1,3 +1,7 @@
# SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
#
# SPDX-License-Identifier: GPL-3.0-or-later
all: clean makedist unpack build all: clean makedist unpack build
unpack: unpack:

View file

@ -1,6 +1,12 @@
<?xml version="1.0" ?> <?xml version="1.0" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"> <html xmlns="http://www.w3.org/1999/xhtml">
<!--
SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
SPDX-License-Identifier: GPL-3.0-or-later
-->
<head> <head>
<title>Debian parallel package - build and execute shell command lines from standard input in parallel</title> <title>Debian parallel package - build and execute shell command lines from standard input in parallel</title>
<meta http-equiv="content-type" content="text/html; charset=utf-8" /> <meta http-equiv="content-type" content="text/html; charset=utf-8" />

View file

@ -1,7 +1,7 @@
<directory name="parallel" rev="307" vrev="1" srcmd5="9b7ed2eb331cb506e28ebf0566bf8cd2"> <directory name="parallel" rev="308" vrev="1" srcmd5="823c6225d48d631ed7390d29b4ae789a">
<entry name="PKGBUILD" md5="46a42ade195bbb01303cefe4ba5e3e75" size="936" mtime="1616445767" /> <entry name="PKGBUILD" md5="2a3ef4282bfa239677f68f7b6dc14090" size="936" mtime="1619106347" />
<entry name="parallel-20210322.tar.bz2" md5="0238fc6747294e2e572850f97b188ffa" size="2234128" mtime="1616445768" /> <entry name="parallel-20210422.tar.bz2" md5="2c467364d5b8d70f2cb272c987518dc7" size="2239130" mtime="1619106348" />
<entry name="parallel.spec" md5="dea85d7d49aff4dc2e10e140411ee053" size="5630" mtime="1616445768" /> <entry name="parallel.spec" md5="2720ff83f2292608800633f4b6437aa5" size="5630" mtime="1619106348" />
<entry name="parallel_20210322.dsc" md5="825f718e140c45e897dd75c3a88802fb" size="556" mtime="1616445769" /> <entry name="parallel_20210422.dsc" md5="d2560ad0bc0c2cdb461bd997d4bfcd89" size="556" mtime="1619106348" />
<entry name="parallel_20210322.tar.gz" md5="7bc192c70c212a14700042fcc52d4272" size="2478535" mtime="1616445769" /> <entry name="parallel_20210422.tar.gz" md5="9a9a0d863b1a6520d09a79c95a6ae7e5" size="2481859" mtime="1619106348" />
</directory> </directory>

View file

@ -1,7 +1,7 @@
Summary: Shell tool for executing jobs in parallel Summary: Shell tool for executing jobs in parallel
Name: parallel Name: parallel
Version: 20210322 Version: 20210422
Release: 1.3 Release: 1.3
License: GPL-3.0-or-later License: GPL-3.0-or-later
Group: Productivity/File utilities Group: Productivity/File utilities

View file

@ -1,72 +0,0 @@
Summary: Shell tool for executing jobs in parallel
Name: parallel
Version: 20100906
Release: 1
License: GPL
Group: Productivity/File utilities
URL: ftp://ftp.gnu.org/gnu/parallel
Source0: %{name}-%{version}.tar.bz2
BuildArch: noarch
BuildRoot: %{_tmppath}/%{name}-%{version}-buildroot
Requires: perl
Recommends: ssh rsync
%description
GNU Parallel is a shell tool for executing jobs in parallel using one
or more machines. A job is typically a single command or a small
script that has to be run for each of the lines in the input. The
typical input is a list of files, a list of hosts, a list of users, or
a list of tables.
If you use xargs today you will find GNU Parallel very easy to use. If
you write loops in shell, you will find GNU Parallel may be able to
replace most of the loops and make them run faster by running jobs in
parallel. If you use ppss or pexec you will find GNU Parallel will
often make the command easier to read.
GNU Parallel also makes sure output from the commands is the same
output as you would get had you run the commands sequentially. This
makes it possible to use output from GNU Parallel as input for other
programs.
%prep
if [ "${RPM_BUILD_ROOT}x" == "x" ]; then
echo "RPM_BUILD_ROOT empty, bad idea!"
exit 1
fi
if [ "${RPM_BUILD_ROOT}" == "/" ]; then
echo "RPM_BUILD_ROOT is set to "/", bad idea!"
exit 1
fi
%setup -q
%build
./configure
make
%install
rm -rf $RPM_BUILD_ROOT
make install prefix=$RPM_BUILD_ROOT%{_prefix} exec_prefix=$RPM_BUILD_ROOT%{_prefix} \
datarootdir=$RPM_BUILD_ROOT%{_prefix} docdir=$RPM_BUILD_ROOT%{_docdir} \
mandir=$RPM_BUILD_ROOT%{_mandir}
rm $RPM_BUILD_ROOT%{_docdir}/parallel.html
rm $RPM_BUILD_ROOT%{_docdir}/sem.html
rm $RPM_BUILD_ROOT%{_docdir}/sql.html
%clean
rm -rf $RPM_BUILD_ROOT
%files
%defattr(-,root,root,-)
/usr/bin/*
/usr/share/man/man1/*
%doc README NEWS src/parallel.html src/sem.html src/sql.html
%changelog
* Mon Sep 06 2010 Ole Tange
- Upgrade to current git-version of source. Tested on build.opensuse.org
* Fri Aug 27 2010 Ole Tange
- Untested upgrade to current git-version of source.
* Sat Aug 08 2010 Markus Ammer
- Initial package setup.

View file

@ -1,8 +1,12 @@
#!/bin/bash #!/bin/bash
# SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
# SPDX-License-Identifier: GPL-3.0-or-later
updater() { updater() {
export DOINO export DOINO
export TAG export TAG
export SPCTAG
export YYYYMMDD=`yyyymmdd` export YYYYMMDD=`yyyymmdd`
export DOI=10.5281/zenodo.$DOINO export DOI=10.5281/zenodo.$DOINO
export YYYY=${YYYYMMDD:0:4} export YYYY=${YYYYMMDD:0:4}
@ -10,8 +14,9 @@ updater() {
export YYYYMMDD=${YYYYMM}22 export YYYYMMDD=${YYYYMM}22
export MON=`date +%b` export MON=`date +%b`
export MONTH=`date +%B` export MONTH=`date +%B`
echo Tag=$TAG Date:$YYYYMMDD Year:$YYYY Mon:$MON Month:$MONTH DOI:$DOI TAG=$(echo $SPCTAG | perl -pe 's/ //g')
export TITLE="GNU Parallel $YYYYMMDD ('$TAG')" echo Tag=$TAG = "$SPCTAG" Date:$YYYYMMDD Year:$YYYY Mon:$MON Month:$MONTH DOI:$DOI
export TITLE="GNU Parallel $YYYYMMDD ('$SPCTAG')"
if [ -z "$DOINO" ] ; then if [ -z "$DOINO" ] ; then
echo '*** Set DOINO and try again ***' echo '*** Set DOINO and try again ***'
@ -21,14 +26,15 @@ updater() {
echo . .last-doitag.txt echo . .last-doitag.txt
return return
fi fi
if [ -z "$TAG" ] ; then if [ -z "$SPCTAG" ] ; then
echo '*** Set TAG and try again ***' echo '*** Set SPCTAG and try again ***'
cat .last-doitag.txt cat .last-doitag.txt
echo . .last-doitag.txt echo . .last-doitag.txt
return return
fi fi
echo "export DOINO=$DOINO" > .last-doitag.txt echo "export DOINO=$DOINO" > .last-doitag.txt
echo "export TAG=$TAG" >> .last-doitag.txt echo "export TAG='$TAG'" >> .last-doitag.txt
echo "export SPCTAG='$SPCTAG'" >> .last-doitag.txt
echo "export YYYYMMDD=$YYYYMMDD" >> .last-doitag.txt echo "export YYYYMMDD=$YYYYMMDD" >> .last-doitag.txt
echo 'echo "$DOINO $YYYYMMDD ($TAG)"' >> .last-doitag.txt echo 'echo "$DOINO $YYYYMMDD ($TAG)"' >> .last-doitag.txt

View file

@ -1,3 +1,7 @@
# SPDX-FileCopyrightText: 2002-2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
#
# SPDX-License-Identifier: GPL-3.0-or-later
bin_SCRIPTS = parallel sql niceload parcat parset parsort \ bin_SCRIPTS = parallel sql niceload parcat parset parsort \
env_parallel env_parallel.ash env_parallel.bash \ env_parallel env_parallel.ash env_parallel.bash \
env_parallel.csh env_parallel.dash env_parallel.fish \ env_parallel.csh env_parallel.dash env_parallel.fish \

View file

@ -14,6 +14,10 @@
@SET_MAKE@ @SET_MAKE@
# SPDX-FileCopyrightText: 2002-2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
#
# SPDX-License-Identifier: GPL-3.0-or-later
VPATH = @srcdir@ VPATH = @srcdir@
am__is_gnu_make = { \ am__is_gnu_make = { \

View file

@ -17,6 +17,9 @@
# along with this program; if not, see <http://www.gnu.org/licenses/> # along with this program; if not, see <http://www.gnu.org/licenses/>
# or write to the Free Software Foundation, Inc., 51 Franklin St, # or write to the Free Software Foundation, Inc., 51 Franklin St,
# Fifth Floor, Boston, MA 02110-1301 USA # Fifth Floor, Boston, MA 02110-1301 USA
#
# SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
# SPDX-License-Identifier: GPL-3.0-or-later
grepq() { grepq() {
# grep -q for systems without -q # grep -q for systems without -q

View file

@ -385,7 +385,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210323 (GNU parallel `parallel --minversion 1`)" echo "parset 20210422 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -37,8 +37,9 @@ env_parallel() {
} }
_bodies_of_ALIASES() { _bodies_of_ALIASES() {
local _i local _i
for _i in $@; do for _i in "$@"; do
if [ $(alias $_i | wc -l) == 1 ] ; then # shellcheck disable=SC2046
if [ $(alias "$_i" | wc -l) == 1 ] ; then
true Alias is a single line. Good. true Alias is a single line. Good.
else else
_warning_PAR "Alias '$_i' contains newline." _warning_PAR "Alias '$_i' contains newline."
@ -65,6 +66,7 @@ env_parallel() {
echo '(_|TIMEOUT|GROUPS|FUNCNAME|DIRSTACK|PIPESTATUS|USERNAME|BASHPID|BASH_[A-Z_]+)' echo '(_|TIMEOUT|GROUPS|FUNCNAME|DIRSTACK|PIPESTATUS|USERNAME|BASHPID|BASH_[A-Z_]+)'
} }
_ignore_READONLY() { _ignore_READONLY() {
# shellcheck disable=SC1078,SC1079,SC2026
readonly | perl -e '@r = map { readonly | perl -e '@r = map {
chomp; chomp;
# sh on UnixWare: readonly TIMEOUT # sh on UnixWare: readonly TIMEOUT
@ -196,15 +198,16 @@ env_parallel() {
END { exit not $exit }' END { exit not $exit }'
} }
_warning_PAR() { _warning_PAR() {
echo "env_parallel: Warning: $@" >&2 echo "env_parallel: Warning: $*" >&2
} }
_error_PAR() { _error_PAR() {
echo "env_parallel: Error: $@" >&2 echo "env_parallel: Error: $*" >&2
} }
# Bash is broken in version 3.2.25 and 4.2.39 # Bash is broken in version 3.2.25 and 4.2.39
# The crazy '[ "`...`" == "" ]' is needed for the same reason # The crazy '[ "`...`" == "" ]' is needed for the same reason
if [ "`_which_PAR parallel`" == "" ]; then if [ "`_which_PAR parallel`" == "" ]; then
# shellcheck disable=SC2016
_error_PAR 'parallel must be in $PATH.' _error_PAR 'parallel must be in $PATH.'
return 255 return 255
fi fi
@ -226,7 +229,7 @@ env_parallel() {
(_names_of_ALIASES; (_names_of_ALIASES;
_names_of_FUNCTIONS; _names_of_FUNCTIONS;
_names_of_VARIABLES) | _names_of_VARIABLES) |
cat > $HOME/.parallel/ignored_vars cat > "$HOME"/.parallel/ignored_vars
return 0 return 0
fi fi
@ -388,7 +391,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210323 (GNU parallel `parallel --minversion 1`)" echo "parset 20210422 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"
@ -420,12 +423,13 @@ _parset_main() {
if perl -e 'exit not grep /,| /, @ARGV' "$_parset_NAME" ; then if perl -e 'exit not grep /,| /, @ARGV' "$_parset_NAME" ; then
# $_parset_NAME contains , or space # $_parset_NAME contains , or space
# Split on , or space to get the names # Split on , or space to get the names
# shellcheck disable=SC2016,SC2046
eval "$( eval "$(
# Compute results into files # Compute results into files
($_parset_PARALLEL_PRG --files -k "$@"; echo $? > "$_exit_FILE") | ($_parset_PARALLEL_PRG --files -k "$@"; echo $? > "$_exit_FILE") |
# var1=`cat tmpfile1; rm tmpfile1` # var1=`cat tmpfile1; rm tmpfile1`
# var2=`cat tmpfile2; rm tmpfile2` # var2=`cat tmpfile2; rm tmpfile2`
parallel --plain -q echo {2}='`cat {1}; rm {1}`' :::: - :::+ $( parallel --plain -q echo '{2}=`cat {1}; rm {1}`' :::: - :::+ $(
echo "$_parset_NAME" | perl -pe 's/,/ /g' echo "$_parset_NAME" | perl -pe 's/,/ /g'
) )
); );

View file

@ -385,7 +385,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210323 (GNU parallel `parallel --minversion 1`)" echo "parset 20210422 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -368,7 +368,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210323 (GNU parallel `parallel --minversion 1`)" echo "parset 20210422 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -371,7 +371,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210323 (GNU parallel `parallel --minversion 1`)" echo "parset 20210422 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -27,6 +27,7 @@
# #
# SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc. # SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
# SPDX-License-Identifier: GPL-3.0-or-later # SPDX-License-Identifier: GPL-3.0-or-later
# shellcheck disable=SC2006
env_parallel() { env_parallel() {
# env_parallel.sh # env_parallel.sh
@ -36,8 +37,8 @@ env_parallel() {
for _i in `alias 2>/dev/null | perl -ne 's/^alias //;s/^(\S+)=.*/$1/ && print' 2>/dev/null`; do for _i in `alias 2>/dev/null | perl -ne 's/^alias //;s/^(\S+)=.*/$1/ && print' 2>/dev/null`; do
# Check if this name really is an alias # Check if this name really is an alias
# or just part of a multiline alias definition # or just part of a multiline alias definition
if alias $_i >/dev/null 2>/dev/null; then if alias "$_i" >/dev/null 2>/dev/null; then
echo $_i echo "$_i"
fi fi
done done
} }
@ -47,7 +48,7 @@ env_parallel() {
# alias myalias='definition' (FreeBSD ash) # alias myalias='definition' (FreeBSD ash)
# so remove 'alias ' from first line # so remove 'alias ' from first line
for _i in "$@"; do for _i in "$@"; do
echo 'alias '"`alias $_i | perl -pe '1..1 and s/^alias //'`" echo 'alias '"`alias "$_i" | perl -pe '1..1 and s/^alias //'`"
done done
} }
_names_of_maybe_FUNCTIONS() { _names_of_maybe_FUNCTIONS() {
@ -55,6 +56,7 @@ env_parallel() {
} }
_names_of_FUNCTIONS() { _names_of_FUNCTIONS() {
# myfunc is a function # myfunc is a function
# shellcheck disable=SC2046
LANG=C type `_names_of_maybe_FUNCTIONS` | LANG=C type `_names_of_maybe_FUNCTIONS` |
perl -ne '/^(\S+) is a function$/ and not $seen{$1}++ and print "$1\n"' perl -ne '/^(\S+) is a function$/ and not $seen{$1}++ and print "$1\n"'
} }
@ -70,7 +72,7 @@ env_parallel() {
for _i in "$@" for _i in "$@"
do do
perl -e 'print @ARGV' "$_i=" perl -e 'print @ARGV' "$_i="
eval echo \"\$$_i\" | perl -e '$/=undef; $a=<>; chop($a); print $a' | eval echo "\"\$$_i\"" | perl -e '$/=undef; $a=<>; chop($a); print $a' |
perl -pe 's/[\002-\011\013-\032\\\#\?\`\(\)\{\}\[\]\^\*\<\=\>\~\|\; \"\!\$\&\202-\377]/\\$&/go;'"s/'/\\\'/g; s/[\n]/'\\n'/go;"; perl -pe 's/[\002-\011\013-\032\\\#\?\`\(\)\{\}\[\]\^\*\<\=\>\~\|\; \"\!\$\&\202-\377]/\\$&/go;'"s/'/\\\'/g; s/[\n]/'\\n'/go;";
echo echo
done done
@ -80,6 +82,7 @@ env_parallel() {
echo '(_|TIMEOUT)' echo '(_|TIMEOUT)'
} }
_ignore_READONLY() { _ignore_READONLY() {
# shellcheck disable=SC1078,SC1079,SC2026
readonly | perl -e '@r = map { readonly | perl -e '@r = map {
chomp; chomp;
# sh on UnixWare: readonly TIMEOUT # sh on UnixWare: readonly TIMEOUT
@ -203,15 +206,16 @@ env_parallel() {
END { exit not $exit }' END { exit not $exit }'
} }
_warning_PAR() { _warning_PAR() {
echo "env_parallel: Warning: $@" >&2 echo "env_parallel: Warning: $*" >&2
} }
_error_PAR() { _error_PAR() {
echo "env_parallel: Error: $@" >&2 echo "env_parallel: Error: $*" >&2
} }
if _which_PAR parallel >/dev/null; then if _which_PAR parallel >/dev/null; then
true parallel found in path true parallel found in path
else else
# shellcheck disable=SC2016
_error_PAR 'parallel must be in $PATH.' _error_PAR 'parallel must be in $PATH.'
return 255 return 255
fi fi
@ -231,7 +235,7 @@ env_parallel() {
(_names_of_ALIASES; (_names_of_ALIASES;
_names_of_FUNCTIONS; _names_of_FUNCTIONS;
_names_of_VARIABLES) | _names_of_VARIABLES) |
cat > $HOME/.parallel/ignored_vars cat > "$HOME"/.parallel/ignored_vars
return 0 return 0
fi fi
@ -310,6 +314,7 @@ env_parallel() {
unset _remove_bad_NAMES _grep_REGEXP unset _remove_bad_NAMES _grep_REGEXP
unset _prefix_PARALLEL_ENV unset _prefix_PARALLEL_ENV
# Test if environment is too big # Test if environment is too big
# shellcheck disable=SC2092
if `_which_PAR true` >/dev/null 2>/dev/null ; then if `_which_PAR true` >/dev/null 2>/dev/null ; then
parallel "$@" parallel "$@"
_parallel_exit_CODE=$? _parallel_exit_CODE=$?
@ -385,7 +390,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210323 (GNU parallel `parallel --minversion 1`)" echo "parset 20210422 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"
@ -417,12 +422,13 @@ _parset_main() {
if perl -e 'exit not grep /,| /, @ARGV' "$_parset_NAME" ; then if perl -e 'exit not grep /,| /, @ARGV' "$_parset_NAME" ; then
# $_parset_NAME contains , or space # $_parset_NAME contains , or space
# Split on , or space to get the names # Split on , or space to get the names
# shellcheck disable=SC2016,SC2046
eval "` eval "`
# Compute results into files # Compute results into files
($_parset_PARALLEL_PRG --files -k "$@"; echo $? > "$_exit_FILE") | ($_parset_PARALLEL_PRG --files -k "$@"; echo $? > "$_exit_FILE") |
# var1= cat tmpfile1; rm tmpfile1 # var1= cat tmpfile1; rm tmpfile1
# var2= cat tmpfile2; rm tmpfile2 # var2= cat tmpfile2; rm tmpfile2
parallel -q echo {2}='\`cat {1}; rm {1}\`' :::: - :::+ \` parallel --plain -q echo '{2}=\`cat {1}; rm {1}\`' :::: - :::+ \`
echo "$_parset_NAME" | echo "$_parset_NAME" |
perl -pe 's/,/ /g' perl -pe 's/,/ /g'
\` \`

View file

@ -362,7 +362,7 @@ _parset_main() {
return 255 return 255
fi fi
if [ "$_parset_NAME" = "--version" ] ; then if [ "$_parset_NAME" = "--version" ] ; then
echo "parset 20210323 (GNU parallel `parallel --minversion 1`)" echo "parset 20210422 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc." echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"

View file

@ -26,7 +26,7 @@
use strict; use strict;
use Getopt::Long; use Getopt::Long;
$Global::progname="niceload"; $Global::progname="niceload";
$Global::version = 20210323; $Global::version = 20210422;
Getopt::Long::Configure("bundling","require_order"); Getopt::Long::Configure("bundling","require_order");
get_options_from_array(\@ARGV) || die_usage(); get_options_from_array(\@ARGV) || die_usage();
if($opt::version) { if($opt::version) {

View file

@ -14,7 +14,7 @@
# General Public License for more details. # General Public License for more details.
# #
# You should have received a copy of the GNU General Public License # You should have received a copy of the GNU General Public License
# along with this program; if not, see <http://www.gnu.org/licenses/> # along with this program; if not, see <https://www.gnu.org/licenses/>
# or write to the Free Software Foundation, Inc., 51 Franklin St, # or write to the Free Software Foundation, Inc., 51 Franklin St,
# Fifth Floor, Boston, MA 02110-1301 USA # Fifth Floor, Boston, MA 02110-1301 USA
# #
@ -1030,8 +1030,8 @@ sub spreadstdin() {
# Buf does not start with $recstart => There is garbage. # Buf does not start with $recstart => There is garbage.
# Make a single record of the garbage # Make a single record of the garbage
if($buf =~ if($buf =~
/(?s)^(?-s)( /(?s:^)(
(?:(?:(?!$recend$recstart)(?s).(?-s))*?$recend) (?:(?:(?!$recend$recstart)(?s:.))*?$recend)
) )
# Followed by recstart # Followed by recstart
(?=$recstart)/mox and length $1 > 0) { (?=$recstart)/mox and length $1 > 0) {
@ -1047,11 +1047,11 @@ sub spreadstdin() {
$Global::max_number_of_args * ($Global::max_lines || 1); $Global::max_number_of_args * ($Global::max_lines || 1);
# (?!negative lookahead) is needed to avoid backtracking # (?!negative lookahead) is needed to avoid backtracking
# See: https://unix.stackexchange.com/questions/439356/ # See: https://unix.stackexchange.com/questions/439356/
# (?s).(?-s) = (.|[\n]) but faster # (?s:.) = (.|[\n]) but faster
while($buf =~ while($buf =~
/(?s)^(?-s)( /(?s:^)(
# n more times recstart.*recend # n more times recstart.*recend
(?:$recstart(?:(?!$recend$recstart)(?s)(.)(?-s))*?$recend){$n_records} (?:$recstart(?:(?!$recend$recstart)(?s:.))*?$recend){$n_records}
) )
# Followed by recstart # Followed by recstart
(?=$recstart)/mox and length $1 > 0) { (?=$recstart)/mox and length $1 > 0) {
@ -1065,8 +1065,8 @@ sub spreadstdin() {
sub pass_regexp() { sub pass_regexp() {
# Find the last recend-recstart in $buf # Find the last recend-recstart in $buf
$eof and return; $eof and return;
# (?s).(?-s) = (.|[\n]) but faster # (?s:.) = (.|[\n]) but faster
if($buf =~ /^((?s).(?-s)*$recend)$recstart(?s).(?-s)*?$/mox) { if($buf =~ /^((?s:.)*$recend)$recstart(?s:.)*?$/mox) {
$anything_written += $anything_written +=
write_record_to_pipe($chunk_number++,\$header,\$buf, write_record_to_pipe($chunk_number++,\$header,\$buf,
$recstart,$recend,length $1); $recstart,$recend,length $1);
@ -1246,20 +1246,18 @@ sub recstartrecend() {
$recend = $opt::recend; $recend = $opt::recend;
if($opt::regexp and $recend eq '') { if($opt::regexp and $recend eq '') {
# --regexp --recend '' # --regexp --recend ''
$recend = '(?s).(?-s)'; $recend = '(?s:.)';
} }
} }
if($opt::regexp) { if($opt::regexp) {
# Do not allow /x comments - to avoid having to quote space
$recstart = "(?-x:".$recstart.")";
$recend = "(?-x:".$recend.")";
# If $recstart/$recend contains '|' # If $recstart/$recend contains '|'
# the | should only apply to the regexp # the | should only apply to the regexp
$recstart = "(?:".$recstart.")"; $recstart = "(?:".$recstart.")";
$recend = "(?:".$recend.")"; $recend = "(?:".$recend.")";
# Quote # and space
$recstart =~ s/#/\\#/g;
$recend =~ s/#/\\#/g;
$recstart =~ s/ /\\ /g;
$recend =~ s/ /\\ /g;
} else { } else {
# $recstart/$recend = printf strings (\n) # $recstart/$recend = printf strings (\n)
$recstart =~ s/\\([0rnt\'\"\\])/"qq|\\$1|"/gee; $recstart =~ s/\\([0rnt\'\"\\])/"qq|\\$1|"/gee;
@ -2189,7 +2187,7 @@ sub check_invalid_option_combinations() {
sub init_globals() { sub init_globals() {
# Defaults: # Defaults:
$Global::version = 20210323; $Global::version = 20210422;
$Global::progname = 'parallel'; $Global::progname = 'parallel';
$::name = "GNU Parallel"; $::name = "GNU Parallel";
$Global::infinity = 2**31; $Global::infinity = 2**31;
@ -2246,8 +2244,12 @@ sub init_globals() {
'{:(\d+?):(\d+?)}' => '$_ = substr($_,$$1,$$2);', '{:(\d+?):(\d+?)}' => '$_ = substr($_,$$1,$$2);',
# Bash ${a#bc} # Bash ${a#bc}
'{#([^#}][^}]*?)}' => 's/^$$1//;', '{#([^#}][^}]*?)}' => 's/^$$1//;',
# Bash ${a##bc}
'{##([^#}][^}]*?)}' => 's/^$$1//;',
# Bash ${a%def} # Bash ${a%def}
'{%([^}]+?)}' => 's/$$1$//;', '{%([^}]+?)}' => 's/$$1$//;',
# Bash ${a%%def}
'{%%([^}]+?)}' => 's/$$1$//;',
# Bash ${a/def/ghi} ${a/def/} # Bash ${a/def/ghi} ${a/def/}
'{/([^}]+?)/([^}]*?)}' => 's/$$1/$$2/;', '{/([^}]+?)/([^}]*?)}' => 's/$$1/$$2/;',
# Bash ${a^a} # Bash ${a^a}
@ -4984,8 +4986,8 @@ sub usage() {
"If you use programs that use GNU Parallel to process data for an article in a", "If you use programs that use GNU Parallel to process data for an article in a",
"scientific publication, please cite:", "scientific publication, please cite:",
"", "",
" Tange, O. (2021, March 22). GNU Parallel 20210322 ('2002-01-06').", " Tange, O. (2021, April 22). GNU Parallel 20210422 ('Ever Given').",
" Zenodo. https://doi.org/10.5281/zenodo.4628277", " Zenodo. https://doi.org/10.5281/zenodo.4710607",
"", "",
# Before changing this line, please read # Before changing this line, please read
# https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice # https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice
@ -5015,8 +5017,8 @@ sub citation_notice() {
"If you use programs that use GNU Parallel to process data for an article in a", "If you use programs that use GNU Parallel to process data for an article in a",
"scientific publication, please cite:", "scientific publication, please cite:",
"", "",
" Tange, O. (2021, March 22). GNU Parallel 20210322 ('2002-01-06').", " Tange, O. (2021, April 22). GNU Parallel 20210422 ('Ever Given').",
" Zenodo. https://doi.org/10.5281/zenodo.4628277", " Zenodo. https://doi.org/10.5281/zenodo.4710607",
"", "",
# Before changing this line, please read # Before changing this line, please read
# https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and # https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and
@ -5139,20 +5141,20 @@ sub citation() {
"If you use programs that use GNU Parallel to process data for an article in a", "If you use programs that use GNU Parallel to process data for an article in a",
"scientific publication, please cite:", "scientific publication, please cite:",
"", "",
"\@software{tange_2021_4628277,", "\@software{tange_2021_4710607,",
" author = {Tange, Ole},", " author = {Tange, Ole},",
" title = {GNU Parallel 20210322 ('2002-01-06')},", " title = {GNU Parallel 20210422 ('Ever Given')},",
" month = Mar,", " month = Apr,",
" year = 2020,", " year = 2021,",
" note = {{GNU Parallel is a general parallelizer to run", " note = {{GNU Parallel is a general parallelizer to run",
" multiple serial command line programs in parallel", " multiple serial command line programs in parallel",
" without changing them.}},", " without changing them.}},",
" publisher = {Zenodo},", " publisher = {Zenodo},",
" doi = {10.5281/zenodo.4628277},", " doi = {10.5281/zenodo.4710607},",
" url = {https://doi.org/10.5281/zenodo.4628277}", " url = {https://doi.org/10.5281/zenodo.4710607}",
"}", "}",
"", "",
"(Feel free to use \\nocite{tange_2021_4628277})", "(Feel free to use \\nocite{tange_2021_4710607})",
"", "",
# Before changing this line, please read # Before changing this line, please read
# https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and # https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and
@ -8020,15 +8022,15 @@ sub cleanup_cmd($$$) {
$dir .= $_."/"; $dir .= $_."/";
unshift @rmdir, ::shell_quote_file($dir); unshift @rmdir, ::shell_quote_file($dir);
} }
my $rmdir = @rmdir ? "sh -c ".::Q("rmdir @rmdir 2>/dev/null;") : ""; my $rmdir = @rmdir ? "rmdir @rmdir 2>/dev/null;" : "";
if(defined $opt::workdir and $opt::workdir eq "...") { if(defined $opt::workdir and $opt::workdir eq "...") {
$rmdir .= ::Q("rm -rf " . ::shell_quote_file($workdir).';'); $rmdir .= "rm -rf " . ::shell_quote_file($workdir).';';
} }
my $rmf = "sh -c ".
$f = ::shell_quote_file($f); ::Q("rm -f ".::shell_quote_file($f)." 2>/dev/null;".$rmdir);
my $sshcmd = $self->sshcommand(); my $sshcmd = $self->sshcommand();
my $serverlogin = $self->serverlogin(); my $serverlogin = $self->serverlogin();
return "$sshcmd $serverlogin -- ".::Q("rm -f $f; $rmdir"); return "$sshcmd $serverlogin -- ".::Q("$rmf");
} }
{ {
@ -9263,12 +9265,13 @@ sub sshlogin_wrap($) {
exec $shell, "-c", ($bashfunc."@ARGV"); exec $shell, "-c", ($bashfunc."@ARGV");
die "exec: $!\n"; die "exec: $!\n";
} }
my $parent = getppid;
do { do {
# Parent is not init (ppid=1), so sshd is alive # Parent pid is not changed, so sshd is alive
# Exponential sleep up to 1 sec # Exponential sleep up to 1 sec
$s = $s < 1 ? 0.001 + $s * 1.03 : $s; $s = $s < 1 ? 0.001 + $s * 1.03 : $s;
select(undef, undef, undef, $s); select(undef, undef, undef, $s);
} until ($done || getppid == 1); } until ($done || getppid != $parent);
if(not $done) { if(not $done) {
# Kill as per --termseq # Kill as per --termseq
my @term_seq = split/,/,$termseq; my @term_seq = split/,/,$termseq;
@ -12674,21 +12677,27 @@ sub total_jobs() {
$Global::unquote_arg = 1; $Global::unquote_arg = 1;
} }
sub yyyy_mm_dd_hh_mm_ss() { sub yyyy_mm_dd_hh_mm_ss() {
# ISO8601 2038-01-19T03:14:08
::strftime("%Y-%m-%dT%H:%M:%S", localtime(time())); ::strftime("%Y-%m-%dT%H:%M:%S", localtime(time()));
} }
sub yyyy_mm_dd_hh_mm() { sub yyyy_mm_dd_hh_mm() {
# ISO8601 2038-01-19T03:14
::strftime("%Y-%m-%dT%H:%M", localtime(time())); ::strftime("%Y-%m-%dT%H:%M", localtime(time()));
} }
sub yyyy_mm_dd() { sub yyyy_mm_dd() {
# ISO8601 2038-01-19
::strftime("%Y-%m-%d", localtime(time())); ::strftime("%Y-%m-%d", localtime(time()));
} }
sub yyyymmddhhmmss() { sub yyyymmddhhmmss() {
# ISO8601 20380119031408
::strftime("%Y%m%d%H%M%S", localtime(time())); ::strftime("%Y%m%d%H%M%S", localtime(time()));
} }
sub yyyymmddhhmm() { sub yyyymmddhhmm() {
# ISO8601 203801190314
::strftime("%Y%m%d%H%M", localtime(time())); ::strftime("%Y%m%d%H%M", localtime(time()));
} }
sub yyyymmdd() { sub yyyymmdd() {
# ISO8601 20380119
::strftime("%Y%m%d", localtime(time())); ::strftime("%Y%m%d", localtime(time()));
} }

View file

@ -67,12 +67,12 @@ how-to, reference and explanation.
=head3 Tutorial =head3 Tutorial
If you prefer reading a book buy B<GNU Parallel 2018> at If you prefer reading a book buy B<GNU Parallel 2018> at
http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html https://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html
or download it at: https://doi.org/10.5281/zenodo.1146014 Read at or download it at: https://doi.org/10.5281/zenodo.1146014 Read at
least chapter 1+2. It should take you less than 20 minutes. least chapter 1+2. It should take you less than 20 minutes.
Otherwise start by watching the intro videos for a quick introduction: Otherwise start by watching the intro videos for a quick introduction:
http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1 https://youtube.com/playlist?list=PL284C9FF2488BC6D1
If you want to dive deeper: spend a couple of hours walking through If you want to dive deeper: spend a couple of hours walking through
the tutorial (B<man parallel_tutorial>). Your command line will love the tutorial (B<man parallel_tutorial>). Your command line will love
@ -352,6 +352,10 @@ sequence number of job
the arguments the arguments
=item Z<> B<skip()>
skip this job (see also B<--filter>)
=item Z<> B<yyyy_mm_dd_hh_mm_ss()> =item Z<> B<yyyy_mm_dd_hh_mm_ss()>
=item Z<> B<yyyy_mm_dd_hh_mm()> =item Z<> B<yyyy_mm_dd_hh_mm()>
@ -668,12 +672,12 @@ files on the remote computer after processing is done.
--return {.}.bz2 --cleanup "zcat {} | bzip -9 >{.}.bz2" --return {.}.bz2 --cleanup "zcat {} | bzip -9 >{.}.bz2"
With B<--transferfile {}> the file transferred to the remote computer With B<--transferfile {}> the file transferred to the remote computer
will be removed on the remote computer. Directories created will not will be removed on the remote computer. Directories on the remote
be removed - even if they are empty. computer containing the file will be removed if they are empty.
With B<--return> the file transferred from the remote computer will be With B<--return> the file transferred from the remote computer will be
removed on the remote computer. Directories created will not be removed on the remote computer. Directories on the remote
removed - even if they are empty. computer containing the file will be removed if they are empty.
B<--cleanup> is ignored when not used with B<--transferfile> or B<--cleanup> is ignored when not used with B<--transferfile> or
B<--return>. B<--return>.
@ -697,7 +701,7 @@ B<--colsep> implies B<--trim rl>, which can be overridden with
B<--trim n>. B<--trim n>.
I<regexp> is a Perl Regular Expression: I<regexp> is a Perl Regular Expression:
http://perldoc.perl.org/perlre.html https://perldoc.perl.org/perlre.html
=item B<--compress> =item B<--compress>
@ -2132,6 +2136,8 @@ B<--return> is often used with B<--transferfile> and B<--cleanup>.
B<--return> is ignored when used with B<--sshlogin :> or when not used B<--return> is ignored when used with B<--sshlogin :> or when not used
with B<--sshlogin>. with B<--sshlogin>.
For details on transferring see B<--transferfile>.
=item B<--round-robin> =item B<--round-robin>
@ -2372,7 +2378,7 @@ On FreeBSD B<env> is needed:
https://ftpmirror.gnu.org/parallel/parallel-20140822.tar.bz2 https://ftpmirror.gnu.org/parallel/parallel-20140822.tar.bz2
There are many limitations of shebang (#!) depending on your operating There are many limitations of shebang (#!) depending on your operating
system. See details on http://www.in-ulm.de/~mascheck/various/shebang/ system. See details on https://www.in-ulm.de/~mascheck/various/shebang/
=item B<--shebang-wrap> =item B<--shebang-wrap>
@ -2789,29 +2795,41 @@ Transfer files to remote computers. Shorthand for: B<--transferfile {}>.
B<--transferfile> is used with B<--sshlogin> to transfer files to the B<--transferfile> is used with B<--sshlogin> to transfer files to the
remote computers. The files will be transferred using B<rsync> and remote computers. The files will be transferred using B<rsync> and
will be put relative to the default work dir. If the path contains /./ will be put relative to the work dir (see B<--workdir>).
the remaining path will be relative to the work dir. E.g.
The I<filename> will normally contain a replacement string.
If the path contains /./ the remaining path will be relative to the
work dir (for details: see B<rsync>). If the work dir is
B</home/user>, the transferring will be as follows:
/tmp/foo/bar => /tmp/foo/bar
tmp/foo/bar => /home/user/tmp/foo/bar
/tmp/./foo/bar => /home/user/foo/bar
tmp/./foo/bar => /home/user/foo/bar
I<Examples>
This will transfer the file I<foo/bar.txt> to the computer
I<server.example.com> to the file I<$HOME/foo/bar.txt> before running
B<wc foo/bar.txt> on I<server.example.com>:
echo foo/bar.txt | parallel --transferfile {} \ echo foo/bar.txt | parallel --transferfile {} \
--sshlogin server.example.com wc --sshlogin server.example.com wc
This will transfer the file I<foo/bar.txt> to the computer This will transfer the file I</tmp/foo/bar.txt> to the computer
I<server.example.com> to the file I<$HOME/foo/bar.txt> before running I<server.example.com> to the file I</tmp/foo/bar.txt> before running
B<wc foo/bar.txt> on I<server.example.com>. B<wc /tmp/foo/bar.txt> on I<server.example.com>:
echo /tmp/foo/bar.txt | parallel --transferfile {} \ echo /tmp/foo/bar.txt | parallel --transferfile {} \
--sshlogin server.example.com wc --sshlogin server.example.com wc
This will transfer the file I</tmp/foo/bar.txt> to the computer This will transfer the file I</tmp/foo/bar.txt> to the computer
I<server.example.com> to the file I</tmp/foo/bar.txt> before running I<server.example.com> to the file I<foo/bar.txt> before running
B<wc /tmp/foo/bar.txt> on I<server.example.com>. B<wc ./foo/bar.txt> on I<server.example.com>:
echo /tmp/./foo/bar.txt | parallel --transferfile {} \ echo /tmp/./foo/bar.txt | parallel --transferfile {} \
--sshlogin server.example.com wc {= s:.*/./:./: =} --sshlogin server.example.com wc {= s:.*/\./:./: =}
This will transfer the file I</tmp/foo/bar.txt> to the computer
I<server.example.com> to the file I<foo/bar.txt> before running
B<wc ./foo/bar.txt> on I<server.example.com>.
B<--transferfile> is often used with B<--return> and B<--cleanup>. A B<--transferfile> is often used with B<--return> and B<--cleanup>. A
shorthand for B<--transferfile {}> is B<--transfer>. shorthand for B<--transferfile {}> is B<--transfer>.
@ -2950,7 +2968,8 @@ Print the version GNU B<parallel> and exit.
=item B<--wd> I<mydir> =item B<--wd> I<mydir>
Jobs will be run in the dir I<mydir>. Jobs will be run in the dir I<mydir>. The default is the current dir
for the local machine, and the login dir for remote computers.
Files transferred using B<--transferfile> and B<--return> will be Files transferred using B<--transferfile> and B<--return> will be
relative to I<mydir> on remote computers. relative to I<mydir> on remote computers.
@ -3335,7 +3354,7 @@ To remove a string anywhere you can use regular expressions with
Let us assume a website stores images like: Let us assume a website stores images like:
http://www.example.com/path/to/YYYYMMDD_##.jpg https://www.example.com/path/to/YYYYMMDD_##.jpg
where YYYYMMDD is the date and ## is the number 01-24. This will where YYYYMMDD is the date and ## is the number 01-24. This will
download images for the past 30 days: download images for the past 30 days:
@ -3343,7 +3362,7 @@ download images for the past 30 days:
getit() { getit() {
date=$(date -d "today -$1 days" +%Y%m%d) date=$(date -d "today -$1 days" +%Y%m%d)
num=$2 num=$2
echo wget http://www.example.com/path/to/${date}_${num}.jpg echo wget https://www.example.com/path/to/${date}_${num}.jpg
} }
export -f getit export -f getit
@ -3871,13 +3890,13 @@ parallel.
To download byte 10000000-19999999 you can use B<curl>: To download byte 10000000-19999999 you can use B<curl>:
curl -r 10000000-19999999 http://example.com/the/big/file >file.part curl -r 10000000-19999999 https://example.com/the/big/file >file.part
To download a 1 GB file we need 100 10MB chunks downloaded and To download a 1 GB file we need 100 10MB chunks downloaded and
combined in the correct order. combined in the correct order.
seq 0 99 | parallel -k curl -r \ seq 0 99 | parallel -k curl -r \
{}0000000-{}9999999 http://example.com/the/big/file > file {}0000000-{}9999999 https://example.com/the/big/file > file
=head2 EXAMPLE: Parallel grep =head2 EXAMPLE: Parallel grep
@ -4146,6 +4165,15 @@ the special short hand I<-S ..> can be used:
--trc {.}.bz2 "zcat {} | bzip2 -9 >{.}.bz2" --trc {.}.bz2 "zcat {} | bzip2 -9 >{.}.bz2"
=head2 EXAMPLE: Advanced file transfer
Assume you have files in in/*, want them processed on server,
and transferred back into /other/dir:
parallel -S server --trc /other/dir/./{/}.out \
cp {/} {/}.out ::: in/./*
=head2 EXAMPLE: Distributing work to local and remote computers =head2 EXAMPLE: Distributing work to local and remote computers
Convert *.mp3 to *.ogg running one process per CPU on local computer Convert *.mp3 to *.ogg running one process per CPU on local computer
@ -5347,7 +5375,7 @@ this.
=item * =item *
The complete output of B<parallel --version>. If you are not running The complete output of B<parallel --version>. If you are not running
the latest released version (see http://ftp.gnu.org/gnu/parallel/) you the latest released version (see https://ftp.gnu.org/gnu/parallel/) you
should specify why you believe the problem is not fixed in that should specify why you believe the problem is not fixed in that
version. version.
@ -5384,7 +5412,7 @@ by others, the output might help them figure out the problem.
=item * =item *
Whether you have watched the intro videos Whether you have watched the intro videos
(http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1), walked (https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1), walked
through the tutorial (man parallel_tutorial), and read the EXAMPLE through the tutorial (man parallel_tutorial), and read the EXAMPLE
section in the man page (man parallel - search for EXAMPLE:). section in the man page (man parallel - search for EXAMPLE:).
@ -5393,8 +5421,8 @@ section in the man page (man parallel - search for EXAMPLE:).
If you suspect the error is dependent on your environment or If you suspect the error is dependent on your environment or
distribution, please see if you can reproduce the error on one of distribution, please see if you can reproduce the error on one of
these VirtualBox images: these VirtualBox images:
http://sourceforge.net/projects/virtualboximage/files/ https://sourceforge.net/projects/virtualboximage/files/
http://www.osboxes.org/virtualbox-images/ https://www.osboxes.org/virtualbox-images/
Specifying the name of your distribution is not enough as you may have Specifying the name of your distribution is not enough as you may have
installed software that is not in the VirtualBox images. installed software that is not in the VirtualBox images.
@ -5467,7 +5495,7 @@ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details. GNU General Public License for more details.
You should have received a copy of the GNU General Public License You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>. along with this program. If not, see <https://www.gnu.org/licenses/>.
=head2 Documentation license I =head2 Documentation license I

View file

@ -575,7 +575,7 @@ using GNU B<parallel>:
echo unzip 1.zip >> /var/run/my_named_pipe; echo unzip 1.zip >> /var/run/my_named_pipe;
echo tar cf /backup/myhome.tar /home/me >> /var/run/my_named_pipe echo tar cf /backup/myhome.tar /home/me >> /var/run/my_named_pipe
http://www.maier-komor.de/xjobs.html (Last checked: 2019-01) https://www.maier-komor.de/xjobs.html (Last checked: 2019-01)
=head2 DIFFERENCES BETWEEN prll AND GNU Parallel =head2 DIFFERENCES BETWEEN prll AND GNU Parallel
@ -1096,9 +1096,9 @@ https://github.com/reconquest/orgalorg
=head2 DIFFERENCES BETWEEN Rust parallel AND GNU Parallel =head2 DIFFERENCES BETWEEN Rust parallel AND GNU Parallel
Rust parallel focuses on speed. It is almost as fast as B<xargs>. It Rust parallel focuses on speed. It is almost as fast as B<xargs>, but
implements a few features from GNU B<parallel>, but lacks many not as fast as B<parallel-bash>. It implements a few features from GNU
functions. All these fail: B<parallel>, but lacks many functions. All these fail:
# Read arguments from file # Read arguments from file
parallel -a file echo parallel -a file echo
@ -3216,6 +3216,10 @@ https://github.com/themattrix/bash-concurrent
=head2 Todo =head2 Todo
http://code.google.com/p/spawntool/
http://code.google.com/p/push/
https://github.com/mylanconnolly/parallel https://github.com/mylanconnolly/parallel
https://github.com/krashanoff/parallel https://github.com/krashanoff/parallel
@ -3240,9 +3244,9 @@ https://github.com/flesler/parallel
https://github.com/Julian/Verge https://github.com/Julian/Verge
http://manpages.ubuntu.com/manpages/xenial/man1/tsp.1.html https://manpages.ubuntu.com/manpages/xenial/man1/tsp.1.html
http://vicerveza.homeunix.net/~viric/soft/ts/ https://vicerveza.homeunix.net/~viric/soft/ts/
https://github.com/chapmanjacobd/que https://github.com/chapmanjacobd/que
@ -3533,7 +3537,7 @@ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details. GNU General Public License for more details.
You should have received a copy of the GNU General Public License You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>. along with this program. If not, see <https://www.gnu.org/licenses/>.
=head2 Documentation license I =head2 Documentation license I

View file

@ -108,7 +108,7 @@ job slots with B<-j>. Give B<-j> the number of jobs to run in
parallel: parallel:
parallel -j50 \ parallel -j50 \
wget http://ftpmirror.gnu.org/parallel/parallel-{1}{2}22.tar.bz2 \ wget https://ftpmirror.gnu.org/parallel/parallel-{1}{2}22.tar.bz2 \
::: 2012 2013 2014 2015 2016 \ ::: 2012 2013 2014 2015 2016 \
::: 01 02 03 04 05 06 07 08 09 10 11 12 ::: 01 02 03 04 05 06 07 08 09 10 11 12

View file

@ -14,11 +14,11 @@ real world.
=head2 Reader's guide =head2 Reader's guide
If you prefer reading a book buy B<GNU Parallel 2018> at If you prefer reading a book buy B<GNU Parallel 2018> at
http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html https://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html
or download it at: https://doi.org/10.5281/zenodo.1146014 or download it at: https://doi.org/10.5281/zenodo.1146014
Otherwise start by watching the intro videos for a quick introduction: Otherwise start by watching the intro videos for a quick introduction:
http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1 https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Then browse through the B<EXAMPLE>s after the list of B<OPTIONS> in Then browse through the B<EXAMPLE>s after the list of B<OPTIONS> in
B<man parallel> (Use B<LESS=+/EXAMPLE: man parallel>). That will give B<man parallel> (Use B<LESS=+/EXAMPLE: man parallel>). That will give
@ -3006,11 +3006,11 @@ Output:
GNU parallel 20210122 GNU parallel 20210122
Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software
Foundation, Inc. Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it. This is free software: you are free to change and redistribute it.
GNU parallel comes with no warranty. GNU parallel comes with no warranty.
Web site: http://www.gnu.org/software/parallel Web site: https://www.gnu.org/software/parallel
When using programs that use GNU Parallel to process data for publication When using programs that use GNU Parallel to process data for publication
please cite as described in 'parallel --citation'. please cite as described in 'parallel --citation'.
@ -3045,7 +3045,7 @@ Output:
month = {Feb}, month = {Feb},
number = {1}, number = {1},
volume = {36}, volume = {36},
url = {http://www.gnu.org/s/parallel}, url = {https://www.gnu.org/s/parallel},
year = {2011}, year = {2011},
pages = {42-47}, pages = {42-47},
doi = {10.5281/zenodo.16303} doi = {10.5281/zenodo.16303}
@ -3121,7 +3121,7 @@ If you like GNU B<parallel>:
=item * =item *
(Re-)walk through the tutorial if you have not done so in the past year (Re-)walk through the tutorial if you have not done so in the past year
(http://www.gnu.org/software/parallel/parallel_tutorial.html) (https://www.gnu.org/software/parallel/parallel_tutorial.html)
=item * =item *

View file

@ -14,7 +14,7 @@
# General Public License for more details. # General Public License for more details.
# #
# You should have received a copy of the GNU General Public License # You should have received a copy of the GNU General Public License
# along with this program; if not, see <http://www.gnu.org/licenses/> # along with this program; if not, see <https://www.gnu.org/licenses/>
# or write to the Free Software Foundation, Inc., 51 Franklin St, # or write to the Free Software Foundation, Inc., 51 Franklin St,
# Fifth Floor, Boston, MA 02110-1301 USA # Fifth Floor, Boston, MA 02110-1301 USA
# #

View file

@ -121,7 +121,7 @@ GetOptions(
"help" => \$opt::dummy, "help" => \$opt::dummy,
) || exit(255); ) || exit(255);
$Global::progname = ($0 =~ m:(^|/)([^/]+)$:)[1]; $Global::progname = ($0 =~ m:(^|/)([^/]+)$:)[1];
$Global::version = 20210323; $Global::version = 20210422;
if($opt::version) { version(); exit 0; } if($opt::version) { version(); exit 0; }
@Global::sortoptions = @Global::sortoptions =
shell_quote(@ARGV_before[0..($#ARGV_before-$#ARGV-1)]); shell_quote(@ARGV_before[0..($#ARGV_before-$#ARGV-1)]);

View file

@ -265,7 +265,7 @@ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details. GNU General Public License for more details.
You should have received a copy of the GNU General Public License You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>. along with this program. If not, see <https://www.gnu.org/licenses/>.
=head2 Documentation license I =head2 Documentation license I

View file

@ -600,7 +600,7 @@ $Global::Initfile && unlink $Global::Initfile;
exit ($err); exit ($err);
sub parse_options { sub parse_options {
$Global::version = 20210323; $Global::version = 20210422;
$Global::progname = 'sql'; $Global::progname = 'sql';
# This must be done first as this may exec myself # This must be done first as this may exec myself

View file

@ -1,3 +1,7 @@
#!/bin/bash #!/bin/bash
# SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
#
# SPDX-License-Identifier: GPL-3.0-or-later
ls tests-to-run/test*.sh | perl -pe 's:(.*/(.*)).sh:cp actual-results/$2 wanted-results/$2:' | sh -x ls tests-to-run/test*.sh | perl -pe 's:(.*/(.*)).sh:cp actual-results/$2 wanted-results/$2:' | sh -x

View file

@ -1,3 +1,7 @@
# SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
#
# SPDX-License-Identifier: GPL-3.0-or-later
### Full test set (except mem) ### Full test set (except mem)
testsuite: 3 testsuite: 3
true true
@ -40,17 +44,15 @@ polar: ../src/parallel tests-to-run/*polar* wanted-results/*polar* prereqlocal
touch ~/.parallel/will-cite touch ~/.parallel/will-cite
# *local* only # *local* only
local: testlocal local: ../src/parallel tests-to-run/*local* wanted-results/*local* prereqlocal installparallel
true
testlocal: ../src/parallel tests-to-run/*local* wanted-results/*local* prereqlocal installparallel
time bash Start.sh local mem time bash Start.sh local mem
# short and local # short
short: testlocalshort short: ../src/parallel tests-to-run/* wanted-results/* prereqlocal installparallel
true TRIES=1 time bash Start.sh '' 'polar|mem|100s|300s'
testlocalshort: ../src/parallel tests-to-run/*local* wanted-results/*local* prereqlocal installparallel # short and local
localshort: ../src/parallel tests-to-run/*local* wanted-results/*local* prereqlocal installparallel
TRIES=1 time bash Start.sh local 'mem|100s|300s' TRIES=1 time bash Start.sh local 'mem|100s|300s'
# sql # sql
@ -71,7 +73,6 @@ REQUIRE=bash -c '\
export -f is_installed; \ export -f is_installed; \
parallel -n2 --tag is_installed ::: \ parallel -n2 --tag is_installed ::: \
script "echo | script -c echo -q /dev/null" \ script "echo | script -c echo -q /dev/null" \
lsh "lsh lo echo lsh is installed" \
"ssh sh@lo" "timeout 20 ssh sh@lo true" \ "ssh sh@lo" "timeout 20 ssh sh@lo true" \
"ssh csh@lo" "timeout 5 ssh csh@lo true" \ "ssh csh@lo" "timeout 5 ssh csh@lo true" \
niceload "niceload echo niceload is installed" \ niceload "niceload echo niceload is installed" \
@ -86,6 +87,7 @@ REQUIRE=bash -c '\
pv "echo | pv -qL 10" \ pv "echo | pv -qL 10" \
timeout "timeout 1 echo timeout is installed" \ timeout "timeout 1 echo timeout is installed" \
' '
# lsh "lsh lo echo lsh is installed" \
prereqlocal: installparallel startvm prereqlocal: installparallel startvm
${REQUIRE} ${REQUIRE}

View file

@ -1,5 +1,9 @@
#!/bin/bash #!/bin/bash
# SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
#
# SPDX-License-Identifier: GPL-3.0-or-later
export LANG=C export LANG=C
SHFILE=/tmp/unittest-parallel.sh SHFILE=/tmp/unittest-parallel.sh

View file

@ -1,5 +1,9 @@
#!/bin/bash #!/bin/bash
# SPDX-FileCopyrightText: 2002-2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
#
# SPDX-License-Identifier: GPL-3.0-or-later
install_packages() { install_packages() {
# After first run, set this: # After first run, set this:
# export INSTALL=echo # export INSTALL=echo
@ -10,7 +14,7 @@ install_packages() {
test_pkgs="$test_pkgs pod2pdf gawk lua5.3 clisp php-cli nodejs" test_pkgs="$test_pkgs pod2pdf gawk lua5.3 clisp php-cli nodejs"
test_pkgs="$test_pkgs mono-csharp-shell libevent-dev tcl libtext-csv-perl" test_pkgs="$test_pkgs mono-csharp-shell libevent-dev tcl libtext-csv-perl"
test_pkgs="$test_pkgs xterm libc6-i386 libcrypt1:i386" test_pkgs="$test_pkgs xterm libc6-i386 libcrypt1:i386"
test_pkgs="$test_pkgs libtest-nowarnings-perl" test_pkgs="$test_pkgs libtest-nowarnings-perl xemacs21-bin"
# Debian package # Debian package
packaging_pkgs="dpkg-dev build-essential debhelper osc cvs automake-1.15" packaging_pkgs="dpkg-dev build-essential debhelper osc cvs automake-1.15"
@ -377,16 +381,23 @@ rsync_versions() {
) )
} }
install_tangetools() {
git clone https://gitlab.com/ole.tange/tangetools.git
cd tangetools &&
make &&
sudo make install
}
misc() { misc() {
parallel --record-env parallel --record-env
#sudo cp /usr/bin/tmux /usr/local/bin/tmux1.8
# Build locale for LC_ALL=zh_HK.big5hkscs perl -e 1 # Build locale for LC_ALL=zh_HK.big5hkscs perl -e 1
sudo locale-gen zh_HK sudo locale-gen zh_HK
} }
run() { run() {
install_packages && install_packages &&
install_tangetools &&
install_oracle_client && install_oracle_client &&
setup_databases && setup_databases &&
add_server_to_hosts && add_server_to_hosts &&

View file

@ -1,5 +1,9 @@
#!/bin/bash -x #!/bin/bash -x
# SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
#
# SPDX-License-Identifier: GPL-3.0-or-later
# Argument can be substring of tests (such as 'local') # Argument can be substring of tests (such as 'local')
export LANG=C export LANG=C

View file

@ -1,16 +0,0 @@
stdout parallel -i -s28 -0 true from \{\} to x{}y < items-0.xi | egrep -v 'exit|Command|\.\.\.' | grep .
# Fejler
#stdout parallel -Di -s28 -0 true from \{\} to x{}y < items-0.xi > /dev/null
# stdout parallel -Di -s28 -0 echo from \{\} to x{}y < items-0.xi > /dev/null
#grep Segmentation /tmp/out && cat >/tmp/:out < /tmp/out
# Denne fejler: seq 1 100 | parallel --eta trysegfault
#stdout stdout /usr/local/bin/parallel -Di -s26 -0 echo from \{\} to x{}y < items-0.xi > /tmp/out;
#grep Segmentation /tmp/out && cat >/tmp/:out < /tmp/out
#/usr/local/bin/parallel -s26 -0 echo < items-0.xi > /tmp/out

View file

@ -1,4 +0,0 @@
#!/usr/local/bin/parallel --shebang-wrap -k A={} /usr/bin/gnuplot
name=system("echo $A")
print name

View file

@ -1,6 +0,0 @@
#!/usr/local/bin/parallel --shebang-wrap -k /usr/bin/octave -qf
arg_list = argv ();
filename = arg_list{1};
printf(filename);
printf("\n");

View file

@ -1,3 +0,0 @@
#!/usr/local/bin/parallel --shebang-wrap -k /usr/bin/perl
print @ARGV,"\n";

View file

@ -1,5 +0,0 @@
#!/usr/local/bin/parallel --shebang-wrap -k /usr/bin/python3
import sys
print(str(sys.argv[1]))

View file

@ -1,4 +0,0 @@
#!/usr/local/bin/parallel --shebang-wrap -k /usr/bin/Rscript --vanilla --slave
options <- commandArgs(trailingOnly = TRUE)
options

View file

@ -1,3 +0,0 @@
#!/usr/local/bin/parallel --shebang-wrap -k /usr/bin/ruby
p ARGV

View file

@ -1,3 +0,0 @@
#!/usr/local/bin/parallel --shebang-wrap -k /bin/sh
echo "$@"

View file

@ -1,3 +0,0 @@
flyp
_PRE 8
hatchname> 8

View file

@ -1,3 +0,0 @@
flyp
_PRE 9
hatchname> 8

View file

@ -1,3 +0,0 @@
flyp
_PRE 19
hatchname> 19

View file

@ -1,4 +0,0 @@
flyp
_PRE 19
hatchname> 9

View file

@ -59,8 +59,13 @@ par_open_files_blocks() {
yes "`seq 3000`" | yes "`seq 3000`" |
head -c 20M | head -c 20M |
stdout parallel -j10 --pipe -k echo {#} of 21 | stdout parallel -j10 --pipe -k echo {#} of 21 |
grep -v 'parallel: Warning: No more file handles.' | grep -v 'No more file handles.' |
grep -v 'Warning: Raising ulimit -n or /etc/security/limits.conf' grep -v 'Only enough file handles to run .* jobs in parallel.' |
grep -v 'Raising ulimit -n or /etc/security/limits.conf' |
grep -v 'Try running .parallel -j0 -N .* --pipe parallel -j0.' |
grep -v 'or increasing .ulimit -n. .try: ulimit -n .ulimit -Hn..' |
grep -v 'or increasing .nofile. in /etc/security/limits.conf' |
grep -v 'or increasing /proc/sys/fs/file-max'
} }
par_pipe_unneeded_procs() { par_pipe_unneeded_procs() {
@ -690,7 +695,7 @@ par_test_cpu_detection_cpuinfo() {
} }
export -f test_one export -f test_one
compgen -A function | grep ^cpu | sort | parallel -j0 -k test_one compgen -A function | grep ^cpu | sort | parallel -j0 -k test_one
rm ~/.parallel/tmp/sshlogin/*/cpuspec rm ~/.parallel/tmp/sshlogin/*/cpuspec 2>/dev/null
} }
par_test_cpu_detection_lscpu() { par_test_cpu_detection_lscpu() {

View file

@ -469,6 +469,8 @@ par_keeporder_roundrobin() {
fi fi
} }
# was -j6 before segfault circus
export -f $(compgen -A function | grep par_) export -f $(compgen -A function | grep par_)
compgen -A function | grep par_ | sort | compgen -A function | grep par_ | sort |
parallel --delay 0.3 --timeout 1000% -j6 --tag -k --joblog /tmp/jl-`basename $0` '{} 2>&1' # parallel --delay 0.3 --timeout 1000% -j6 --tag -k --joblog /tmp/jl-`basename $0` '{} 2>&1'
parallel --delay 0.3 --timeout 1000% -j1 --lb --tag -k --joblog /tmp/jl-`basename $0` '{} 2>&1'

View file

@ -8,6 +8,75 @@
# Each should be taking 3-10s and be possible to run in parallel # Each should be taking 3-10s and be possible to run in parallel
# I.e.: No race conditions, no logins # I.e.: No race conditions, no logins
par_shebang() {
echo '### Test different shebangs'
gp() {
cat <<'EOF'
#!/usr/local/bin/parallel --shebang-wrap -k A={} /usr/bin/gnuplot
name=system("echo $A")
print name
EOF
true
}
oct() {
cat <<'EOF'
#!/usr/local/bin/parallel --shebang-wrap -k /usr/bin/octave -qf
arg_list = argv ();
filename = arg_list{1};
printf(filename);
printf("\n");
EOF
true
}
pl() {
cat <<'EOF'
#!/usr/local/bin/parallel --shebang-wrap -k /usr/bin/perl
print @ARGV,"\n";
EOF
true
}
py() {
cat <<'EOF'
#!/usr/local/bin/parallel --shebang-wrap -k /usr/bin/python3
import sys
print(str(sys.argv[1]))
EOF
true
}
r() {
cat <<'EOF'
#!/usr/local/bin/parallel --shebang-wrap -k /usr/bin/Rscript --vanilla --slave
options <- commandArgs(trailingOnly = TRUE)
options
EOF
true
}
rb() {
cat <<'EOF'
#!/usr/local/bin/parallel --shebang-wrap -k /usr/bin/ruby
p ARGV
EOF
true
}
sh() {
cat <<'EOF'
#!/usr/local/bin/parallel --shebang-wrap -k /bin/sh
echo "$@"
EOF
true
}
run() {
tmp=`tempfile`
"$@" > "$tmp"
chmod +x "$tmp"
"$tmp" A B C
rm "$tmp"
}
export -f run gp oct pl py r rb sh
parallel --tag -k run ::: gp oct pl py r rb sh
}
par_pipe_regexp() { par_pipe_regexp() {
echo '### --pipe --regexp' echo '### --pipe --regexp'
gen() { gen() {
@ -36,6 +105,33 @@ EOF
$p --recstart '.*, Start' -N1 'echo Record;cat' $p --recstart '.*, Start' -N1 'echo Record;cat'
} }
par_pipe_regexp_non_quoted() {
echo '### --pipe --regexp non_quoted \n'
gen() {
cat <<EOF
Start
foo
End
Start
Start this line is a false Start line
End this line is a false End line
End
EOF
true
}
p="ppar --pipe --regexp -k"
p="parallel --pipe --regexp -k"
gen | $p --recend '' --recstart '^Start$' -N1 'echo :::Single record;cat'
gen | $p --recend '' --recstart 'Start\n' -N1 'echo :::Single record;cat'
gen | $p --recend '' --recstart 'Start
' -N1 'echo :::Single record;cat'
gen | $p --recend 'End$' --recstart '' -N1 'echo :::Single record;cat'
gen | $p --recend 'End\n' --recstart '' -N1 'echo :::Single record;cat'
gen | $p --recend 'End
' --recstart '' -N1 'echo :::Single record;cat'
}
par_delay_halt_soon() { par_delay_halt_soon() {
echo "bug #59893: --halt soon doesn't work with --delay" echo "bug #59893: --halt soon doesn't work with --delay"
seq 0 10 | seq 0 10 |
@ -169,7 +265,10 @@ par_tee_with_premature_close() {
par_tee_too_many_args() { par_tee_too_many_args() {
echo '### Fail if there are more arguments than --jobs' echo '### Fail if there are more arguments than --jobs'
seq 11 | stdout parallel -k --tag --pipe -j4 --tee grep {} ::: {1..4} seq 11 | stdout parallel -k --tag --pipe -j4 --tee grep {} ::: {1..4}
seq 11 | stdout parallel -k --tag --pipe -j0 --tee grep {} ::: {1..10000} tmp=`tempfile`
seq 11 | parallel -k --tag --pipe -j0 --tee grep {} ::: {1..10000} 2> "$tmp"
cat "$tmp" | perl -pe 's/\d+/999/g'
rm "$tmp"
} }
par_maxargs() { par_maxargs() {
@ -197,7 +296,7 @@ par_totaljob_repl() {
parallel -k -N7 --plus echo {#} {##} ::: {1..14} parallel -k -N7 --plus echo {#} {##} ::: {1..14}
parallel -k -N7 --plus echo {#} {##} ::: {1..15} parallel -k -N7 --plus echo {#} {##} ::: {1..15}
parallel -k -S 8/: -X --plus echo {#} {##} ::: {1..15} parallel -k -S 8/: -X --plus echo {#} {##} ::: {1..15}
parallel -k --plus -j 10 echo '{0#}/{##}:{0%}' ::: {1..5} ::: {1..4} parallel -k --plus --delay 0.1 -j 10 'echo {0#}/{##}:{0%}' ::: {1..5} ::: {1..4}
} }
par_jobslot_repl() { par_jobslot_repl() {

View file

@ -192,12 +192,15 @@ par_timeout_onall() {
par_rsync_3.2.3() { par_rsync_3.2.3() {
echo "bug #59006: rsync version 3.2.3 is not detected correctly" echo "bug #59006: rsync version 3.2.3 is not detected correctly"
PATH=$HOME/bin/rsync:$PATH tmp=$(mktemp -d)
cp $(which rsync-v3.2.3) "$tmp"/rsync
PATH="$tmp":$PATH
rsync --version | grep version rsync --version | grep version
rm -f bug59006 rm -f bug59006
parallel --return {} -Sparallel@lo touch ::: bug59006 parallel --return {} -Sparallel@lo touch ::: bug59006
ls bug59006 ls bug59006
rm -f bug59006 rm -f bug59006
rm -r "$tmp"
} }
export -f $(compgen -A function | grep par_) export -f $(compgen -A function | grep par_)

View file

@ -20,6 +20,33 @@ resize=`resize`
# sleep 2 # sleep 2
# rm /tmp/parallel-script-for-script # rm /tmp/parallel-script-for-script
echo '### Test xargs compatibility'
echo /tmp/1 > /tmp/files
echo 1 > /tmp/1
echo 'xargs Expect: 3 1'
echo 3 | xargs -P 1 -n 1 -a /tmp/files cat -
echo 'parallel Expect: 3 1 via psedotty 2'
cat >/tmp/parallel-script-for-script <<EOF
#!/bin/bash
echo 3 | parallel --tty -k -P 1 -n 1 -a /tmp/files cat -
EOF
chmod 755 /tmp/parallel-script-for-script
echo via pseudotty | script -q -f -c /tmp/parallel-script-for-script /dev/null
sleep 1
echo 'xargs Expect: 1 3'
echo 3 | xargs -I {} -P 1 -n 1 -a /tmp/files cat {} -
echo 'parallel Expect: 1 3 2 via pseudotty'
cat >/tmp/parallel-script-for-script2 <<EOF
#!/bin/bash
echo 3 | parallel --tty -k -I {} -P 1 -n 1 -a /tmp/files cat {} -
EOF
chmod 755 /tmp/parallel-script-for-script2
echo via pseudotty | script -q -f -c /tmp/parallel-script-for-script2 /dev/null
sleep 1
echo '### Test stdin goes to first command only ("cat" as argument)' echo '### Test stdin goes to first command only ("cat" as argument)'
cat >/tmp/parallel-script-for-script2 <<EOF cat >/tmp/parallel-script-for-script2 <<EOF
#!/bin/bash #!/bin/bash

View file

@ -4,7 +4,6 @@
# #
# SPDX-License-Identifier: GPL-3.0-or-later # SPDX-License-Identifier: GPL-3.0-or-later
rsync -Ha --delete input-files/segfault/ tmp/
cd tmp cd tmp
median() { perl -e '@a=sort {$a<=>$b} <>;print $a[$#a/2]';} median() { perl -e '@a=sort {$a<=>$b} <>;print $a[$#a/2]';}
@ -71,11 +70,6 @@ echo '### Test make .deb package';
cd ~/privat/parallel/packager/debian; cd ~/privat/parallel/packager/debian;
stdout make | grep 'To install the GNU Parallel Debian package, run:' stdout make | grep 'To install the GNU Parallel Debian package, run:'
echo '### Test of segfaulting issue'
echo 'This gave ~/bin/stdout: line 3: 20374 Segmentation fault "$@" 2>&1';
echo 'before adding wait() before exit';
seq 1 300 | stdout parallel ./trysegfault
echo '### Test basic --arg-sep' echo '### Test basic --arg-sep'
parallel -k echo ::: a b parallel -k echo ::: a b

View file

@ -1,13 +0,0 @@
#!/bin/bash
# SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
#
# SPDX-License-Identifier: GPL-3.0-or-later
# -L1 will join lines ending in ' '
cat <<'EOF' | sed -e 's/;$/; /;s/$SERVER1/'$SERVER1'/;s/$SERVER2/'$SERVER2'/' | stdout parallel -vj0 -k --joblog /tmp/jl-`basename $0` -L1 -r
echo "### test08"
cd input-files/test08;
ls | parallel -q perl -ne '/_PRE (\d+)/ and $p=$1; /hatchname> (\d+)/ and $1!=$p and print $ARGV,"\n"' | sort
EOF

View file

@ -66,9 +66,6 @@ echo '### --header regexp --round-robin'
echo '### --header num --round-robin' echo '### --header num --round-robin'
(echo %head1; echo %head2; seq 5) | $NICEPAR -kj2 --pipe -N2 --round --header 2 echo JOB{#}\;wc | sort (echo %head1; echo %head2; seq 5) | $NICEPAR -kj2 --pipe -N2 --round --header 2 echo JOB{#}\;wc | sort
echo '### shebang-wrap'
$NICEPAR -k {} {} A B C ::: ./input-files/shebang/shebangwrap.*[^~]
echo 'bug #43967: Error if there exists a bin/zsh or bin/bash dir (with zsh or bash).' echo 'bug #43967: Error if there exists a bin/zsh or bin/bash dir (with zsh or bash).'
mkdir -p /tmp/bash$$/bash; PATH=/tmp/bash$$:$PATH parallel echo ::: OK; rm -rf /tmp/bash$$ mkdir -p /tmp/bash$$/bash; PATH=/tmp/bash$$:$PATH parallel echo ::: OK; rm -rf /tmp/bash$$

View file

@ -1,35 +0,0 @@
#!/bin/bash
# SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
#
# SPDX-License-Identifier: GPL-3.0-or-later
# Test xargs compatibility
#rm -f ~/.parallel/will-cite
cd input-files/test15 || cd ../input-files/test15
echo 'xargs Expect: 3 1'
echo 3 | xargs -P 1 -n 1 -a files cat -
echo 'parallel Expect: 3 1 via psedotty 2'
cat >/tmp/parallel-script-for-script <<EOF
#!/bin/bash
echo 3 | parallel --tty -k -P 1 -n 1 -a files cat -
EOF
chmod 755 /tmp/parallel-script-for-script
echo via pseudotty | script -q -f -c /tmp/parallel-script-for-script /dev/null
sleep 1
echo 'xargs Expect: 1 3'
echo 3 | xargs -I {} -P 1 -n 1 -a files cat {} -
echo 'parallel Expect: 1 3 2 via pseudotty'
cat >/tmp/parallel-script-for-script2 <<EOF
#!/bin/bash
echo 3 | parallel --tty -k -I {} -P 1 -n 1 -a files cat {} -
EOF
chmod 755 /tmp/parallel-script-for-script2
echo via pseudotty | script -q -f -c /tmp/parallel-script-for-script2 /dev/null
sleep 1
touch ~/.parallel/will-cite

View file

@ -1,5 +1,7 @@
# -*- mode: ruby -*- # -*- mode: ruby -*-
# vi: set ft=ruby : # vi: set ft=ruby :
# SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
# SPDX-License-Identifier: GPL-3.0-or-later
# All Vagrant configuration is done below. The "2" in Vagrant.configure # All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for # configures the configuration version (we support older styles for

View file

@ -265,11 +265,6 @@ par_null_resume A
par_null_resume B par_null_resume B
par_null_resume C par_null_resume C
par_open_files_blocks bug #38439: "open files" with --files --pipe blocks after a while par_open_files_blocks bug #38439: "open files" with --files --pipe blocks after a while
par_open_files_blocks parallel: Warning: Only enough file handles to run 2 jobs in parallel.
par_open_files_blocks parallel: Warning: Try running 'parallel -j0 -N 2 --pipe parallel -j0'
par_open_files_blocks parallel: Warning: or increasing 'ulimit -n' (try: ulimit -n `ulimit -Hn`)
par_open_files_blocks parallel: Warning: or increasing 'nofile' in /etc/security/limits.conf
par_open_files_blocks parallel: Warning: or increasing /proc/sys/fs/file-max
par_open_files_blocks 1 of 21 par_open_files_blocks 1 of 21
par_open_files_blocks 2 of 21 par_open_files_blocks 2 of 21
par_open_files_blocks 3 of 21 par_open_files_blocks 3 of 21

View file

@ -282,6 +282,62 @@ par_pipe_regexp A2, 00100, 7
par_pipe_regexp Record par_pipe_regexp Record
par_pipe_regexp A2, Start, 7 par_pipe_regexp A2, Start, 7
par_pipe_regexp A2, 00200, 8 par_pipe_regexp A2, 00200, 8
par_pipe_regexp_non_quoted ### --pipe --regexp non_quoted \n
par_pipe_regexp_non_quoted :::Single record
par_pipe_regexp_non_quoted Start
par_pipe_regexp_non_quoted foo
par_pipe_regexp_non_quoted End
par_pipe_regexp_non_quoted :::Single record
par_pipe_regexp_non_quoted Start
par_pipe_regexp_non_quoted Start this line is a false Start line
par_pipe_regexp_non_quoted End this line is a false End line
par_pipe_regexp_non_quoted End
par_pipe_regexp_non_quoted :::Single record
par_pipe_regexp_non_quoted Start
par_pipe_regexp_non_quoted foo
par_pipe_regexp_non_quoted End
par_pipe_regexp_non_quoted :::Single record
par_pipe_regexp_non_quoted Start
par_pipe_regexp_non_quoted Start this line is a false Start line
par_pipe_regexp_non_quoted End this line is a false End line
par_pipe_regexp_non_quoted End
par_pipe_regexp_non_quoted :::Single record
par_pipe_regexp_non_quoted Start
par_pipe_regexp_non_quoted foo
par_pipe_regexp_non_quoted End
par_pipe_regexp_non_quoted :::Single record
par_pipe_regexp_non_quoted Start
par_pipe_regexp_non_quoted Start this line is a false Start line
par_pipe_regexp_non_quoted End this line is a false End line
par_pipe_regexp_non_quoted End
par_pipe_regexp_non_quoted :::Single record
par_pipe_regexp_non_quoted Start
par_pipe_regexp_non_quoted foo
par_pipe_regexp_non_quoted End:::Single record
par_pipe_regexp_non_quoted
par_pipe_regexp_non_quoted Start
par_pipe_regexp_non_quoted Start this line is a false Start line
par_pipe_regexp_non_quoted End this line is a false End line
par_pipe_regexp_non_quoted End:::Single record
par_pipe_regexp_non_quoted
par_pipe_regexp_non_quoted :::Single record
par_pipe_regexp_non_quoted Start
par_pipe_regexp_non_quoted foo
par_pipe_regexp_non_quoted End
par_pipe_regexp_non_quoted :::Single record
par_pipe_regexp_non_quoted Start
par_pipe_regexp_non_quoted Start this line is a false Start line
par_pipe_regexp_non_quoted End this line is a false End line
par_pipe_regexp_non_quoted End
par_pipe_regexp_non_quoted :::Single record
par_pipe_regexp_non_quoted Start
par_pipe_regexp_non_quoted foo
par_pipe_regexp_non_quoted End
par_pipe_regexp_non_quoted :::Single record
par_pipe_regexp_non_quoted Start
par_pipe_regexp_non_quoted Start this line is a false Start line
par_pipe_regexp_non_quoted End this line is a false End line
par_pipe_regexp_non_quoted End
par_progress ### Test of --progress par_progress ### Test of --progress
par_progress 16 par_progress 16
par_progress ### Test of --progress with no jobs par_progress ### Test of --progress with no jobs
@ -327,6 +383,28 @@ par_replacement_slashslash /a/b /a/b/c.jpg
par_replacement_slashslash . ./a.jpg par_replacement_slashslash . ./a.jpg
par_replacement_slashslash ./a ./a/b.jpg par_replacement_slashslash ./a ./a/b.jpg
par_replacement_slashslash ./a/b ./a/b/c.jpg par_replacement_slashslash ./a/b ./a/b/c.jpg
par_shebang ### Test different shebangs
par_shebang gp A
par_shebang gp B
par_shebang gp C
par_shebang oct A
par_shebang oct B
par_shebang oct C
par_shebang pl A
par_shebang pl B
par_shebang pl C
par_shebang py A
par_shebang py B
par_shebang py C
par_shebang r [1] "A"
par_shebang r [1] "B"
par_shebang r [1] "C"
par_shebang rb ["A"]
par_shebang rb ["B"]
par_shebang rb ["C"]
par_shebang sh A
par_shebang sh B
par_shebang sh C
par_show_limits ### Test --show-limits par_show_limits ### Test --show-limits
par_show_limits Maximal size of command: 131xxx par_show_limits Maximal size of command: 131xxx
par_show_limits Maximal used size of command: 131xxx par_show_limits Maximal used size of command: 131xxx
@ -372,12 +450,12 @@ par_tee_too_many_args 1 11
par_tee_too_many_args 2 2 par_tee_too_many_args 2 2
par_tee_too_many_args 3 3 par_tee_too_many_args 3 3
par_tee_too_many_args 4 4 par_tee_too_many_args 4 4
par_tee_too_many_args parallel: Warning: Only enough file handles to run 251 jobs in parallel. par_tee_too_many_args parallel: Warning: Only enough file handles to run 999 jobs in parallel.
par_tee_too_many_args parallel: Warning: Try running 'parallel -j0 -N 251 --pipe parallel -j0' par_tee_too_many_args parallel: Warning: Try running 'parallel -j999 -N 999 --pipe parallel -j999'
par_tee_too_many_args parallel: Warning: or increasing 'ulimit -n' (try: ulimit -n `ulimit -Hn`) par_tee_too_many_args parallel: Warning: or increasing 'ulimit -n' (try: ulimit -n `ulimit -Hn`)
par_tee_too_many_args parallel: Warning: or increasing 'nofile' in /etc/security/limits.conf par_tee_too_many_args parallel: Warning: or increasing 'nofile' in /etc/security/limits.conf
par_tee_too_many_args parallel: Warning: or increasing /proc/sys/fs/file-max par_tee_too_many_args parallel: Warning: or increasing /proc/sys/fs/file-max
par_tee_too_many_args parallel: Error: --tee requires --jobs to be higher. Try --jobs 0. par_tee_too_many_args parallel: Error: --tee requires --jobs to be higher. Try --jobs 999.
par_tee_with_premature_close --tee --pipe should send all data to all commands par_tee_with_premature_close --tee --pipe should send all data to all commands
par_tee_with_premature_close even if a command closes stdin before reading everything par_tee_with_premature_close even if a command closes stdin before reading everything
par_tee_with_premature_close tee with --output-error=warn-nopipe support par_tee_with_premature_close tee with --output-error=warn-nopipe support

View file

@ -76,7 +76,7 @@ par_remote_load parallel: Warning: This job was killed because it timed out:
par_remote_load parallel: Warning: perl -e 'while(1){ }' par_remote_load parallel: Warning: perl -e 'while(1){ }'
par_remote_load OK par_remote_load OK
par_remote_nice ### Test --nice remote par_remote_nice ### Test --nice remote
par_remote_nice ssh lo -- exec 'perl -e '"'"'$ENV{"PARALLEL_PID"}="XXXXX";$ENV{"PARALLEL_SEQ"}="1";$ENV{"PARALLEL_SSHLOGIN"}="lo";$ENV{"PARALLEL_SSHHOST"}="lo";$ENV{"PARALLEL_JOBSLOT"}="1";$bashfunc = "";@ARGV="PAR=a bash -c \"echo \\\$PAR b\"";$shell="$ENV{SHELL}";$tmpdir="/tmp/parallel-local-ssh1-tmpdir";$nice=1;$termseq="";do{$ENV{PARALLEL_TMP}=$tmpdir."/par".join"",map{(0..9,"a".."z","A".."Z")[rand(62)]}(1..5);}while(-e$ENV{PARALLEL_TMP});$SIG{CHLD}=sub{$done=1;};$pid=fork;unless($pid){eval{setpgrp};eval{setpriority(0,0,$nice)};exec$shell,"-c",($bashfunc."@ARGV");die"exec:$!\n";}do{$s=$s<1?0.001+$s*1.03:$s;select(undef,undef,undef,$s);}until($done||getppid==1);if(not$done){my@term_seq=split/,/,$termseq;if(not@term_seq){@term_seq=("TERM",200,"TERM",100,"TERM",50,"KILL",25);}while(@term_seq&&kill(0,-$pid)){kill(shift@term_seq,-$pid);select(undef,undef,undef,(shift@term_seq)/1000);}}wait;exit($?&127?128+($?&127):1+$?>>8)'"'"; par_remote_nice ssh lo -- exec 'perl -e '"'"'$ENV{"PARALLEL_PID"}="XXXXX";$ENV{"PARALLEL_SEQ"}="1";$ENV{"PARALLEL_SSHLOGIN"}="lo";$ENV{"PARALLEL_SSHHOST"}="lo";$ENV{"PARALLEL_JOBSLOT"}="1";$bashfunc = "";@ARGV="PAR=a bash -c \"echo \\\$PAR b\"";$shell="$ENV{SHELL}";$tmpdir="/tmp/parallel-local-ssh1-tmpdir";$nice=1;$termseq="";do{$ENV{PARALLEL_TMP}=$tmpdir."/par".join"",map{(0..9,"a".."z","A".."Z")[rand(62)]}(1..5);}while(-e$ENV{PARALLEL_TMP});$SIG{CHLD}=sub{$done=1;};$pid=fork;unless($pid){eval{setpgrp};eval{setpriority(0,0,$nice)};exec$shell,"-c",($bashfunc."@ARGV");die"exec:$!\n";}my$parent=getppid;do{$s=$s<1?0.001+$s*1.03:$s;select(undef,undef,undef,$s);}until($done||getppid!=$parent);if(not$done){my@term_seq=split/,/,$termseq;if(not@term_seq){@term_seq=("TERM",200,"TERM",100,"TERM",50,"KILL",25);}while(@term_seq&&kill(0,-$pid)){kill(shift@term_seq,-$pid);select(undef,undef,undef,(shift@term_seq)/1000);}}wait;exit($?&127?128+($?&127):1+$?>>8)'"'";
par_remote_nice a b par_remote_nice a b
par_ssh ### use --ssh par_ssh ### use --ssh
par_ssh Run through BARSSH? par_ssh Run through BARSSH?

View file

@ -1,3 +1,48 @@
### Test xargs compatibility
xargs Expect: 3 1
3
1
parallel Expect: 3 1 via psedotty 2
via pseudotty
Academic tradition requires you to cite works you base your article on.
If you use programs that use GNU Parallel to process data for an article in a
scientific publication, please cite:
Tange, O. (2021, March 22). GNU Parallel 20210322 ('2002-01-06').
Zenodo. https://doi.org/10.5281/zenodo.4628277
This helps funding further development; AND IT WON'T COST YOU A CENT.
If you pay 10000 EUR you should feel free to use GNU Parallel without citing.
More about funding GNU Parallel and the citation notice:
https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice
To silence this citation notice: run 'parallel --citation' once.
via pseudotty
1
xargs Expect: 1 3
1
3
parallel Expect: 1 3 2 via pseudotty
via pseudotty
Academic tradition requires you to cite works you base your article on.
If you use programs that use GNU Parallel to process data for an article in a
scientific publication, please cite:
Tange, O. (2021, March 22). GNU Parallel 20210322 ('2002-01-06').
Zenodo. https://doi.org/10.5281/zenodo.4628277
This helps funding further development; AND IT WON'T COST YOU A CENT.
If you pay 10000 EUR you should feel free to use GNU Parallel without citing.
More about funding GNU Parallel and the citation notice:
https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice
To silence this citation notice: run 'parallel --citation' once.
1
via pseudotty
### Test stdin goes to first command only ("cat" as argument) ### Test stdin goes to first command only ("cat" as argument)
via pseudotty via pseudotty
Academic tradition requires you to cite works you base your article on. Academic tradition requires you to cite works you base your article on.

View file

@ -144,11 +144,6 @@ echo "### bug #35268: shell_quote doesn't treats [] brackets correctly"
echo '### Test make .deb package'; cd ~/privat/parallel/packager/debian; stdout make | grep 'To install the GNU Parallel Debian package, run:' echo '### Test make .deb package'; cd ~/privat/parallel/packager/debian; stdout make | grep 'To install the GNU Parallel Debian package, run:'
### Test make .deb package ### Test make .deb package
To install the GNU Parallel Debian package, run: To install the GNU Parallel Debian package, run:
echo '### Test of segfaulting issue'
### Test of segfaulting issue
echo 'This gave ~/bin/stdout: line 3: 20374 Segmentation fault "$@" 2>&1'; echo 'before adding wait() before exit'; seq 1 300 | stdout parallel ./trysegfault
This gave ~/bin/stdout: line 3: 20374 Segmentation fault "$@" 2>&1
before adding wait() before exit
echo '### Test basic --arg-sep' echo '### Test basic --arg-sep'
### Test basic --arg-sep ### Test basic --arg-sep
parallel -k echo ::: a b parallel -k echo ::: a b

View file

@ -1,5 +0,0 @@
echo "### test08"
### test08
cd input-files/test08; ls | parallel -q perl -ne '/_PRE (\d+)/ and $p=$1; /hatchname> (\d+)/ and $1!=$p and print $ARGV,"\n"' | sort
b
d

View file

@ -41,4 +41,4 @@ par_special_char a' * ? >o <i*? ][\!#¤%=( ) | }b 5
par_special_char a' * ? >o <i*? ][\!#¤%=( ) | }b 5 par_special_char a' * ? >o <i*? ][\!#¤%=( ) | }b 5
par_special_char a' * ? >o <i*? ][\!#¤%=( ) | }b 5 par_special_char a' * ? >o <i*? ][\!#¤%=( ) | }b 5
par_special_char a' * ? >o <i*? ][!#¤%=( ) | }b 5 par_special_char a' * ? >o <i*? ][!#¤%=( ) | }b 5
par_special_char a' * ? >o <i*? ][!#¤%=( ) | }b 5 par_special_char a' * ? >o <i*? ][\!#¤%=( ) | }b 5

View file

@ -137,37 +137,6 @@ echo '### --header num --round-robin'
5 5 20 5 5 20
JOB1 JOB1
JOB2 JOB2
echo '### shebang-wrap'
### shebang-wrap
$NICEPAR -k {} {} A B C ::: ./input-files/shebang/shebangwrap.*[^~]
./input-files/shebang/shebangwrap.oct
A
B
C
./input-files/shebang/shebangwrap.pl
A
B
C
./input-files/shebang/shebangwrap.py
A
B
C
[1] "./input-files/shebang/shebangwrap.r"
[1] "A"
[1] "B"
[1] "C"
["./input-files/shebang/shebangwrap.rb"]
["A"]
["B"]
["C"]
./input-files/shebang/shebangwrap.sh
A
B
C
./input-files/shebang/shebangwrap.gp
A
B
C
echo 'bug #43967: Error if there exists a bin/zsh or bin/bash dir (with zsh or bash).' echo 'bug #43967: Error if there exists a bin/zsh or bin/bash dir (with zsh or bash).'
bug #43967: Error if there exists a bin/zsh or bin/bash dir (with zsh or bash). bug #43967: Error if there exists a bin/zsh or bin/bash dir (with zsh or bash).
mkdir -p /tmp/bash$$/bash; PATH=/tmp/bash$$:$PATH parallel echo ::: OK; rm -rf /tmp/bash$$ mkdir -p /tmp/bash$$/bash; PATH=/tmp/bash$$:$PATH parallel echo ::: OK; rm -rf /tmp/bash$$

View file

@ -845,7 +845,7 @@ For details: see man env_parallel
export -f my_func3 export -f my_func3
parallel -vv --workdir ... --nice 17 --env _ --trc {}.out \ parallel -vv --workdir ... --nice 17 --env _ --trc {}.out \
-S $SERVER1 my_func3 {} ::: abc-file -S $SERVER1 my_func3 {} ::: abc-file
ssh -l parallel lo -- mkdir -p ./.TMPWORKDIR && rsync --protocol 30 -rlDzR -e'ssh -l parallel' ./abc-file lo:./.TMPWORKDIR;ssh -l parallel lo -- exec perl -e "'"'@GNU_Parallel=split/_/,"use_IPC::Open3;_use_MIME::Base64";eval"@GNU_Parallel";$chld=$SIG{CHLD};$SIG{CHLD}="IGNORE";my$zip=(grep{-x$_}"/usr/local/bin/bzip2")[0]||"bzip2";open3($in,$out,">&STDERR",$zip,"-dc");if(my$perlpid=fork){close$in;$eval=join"",<$out>;close$out;}else{close$out;print$in(decode_base64(join"",@ARGV));close$in;exit;}wait;$SIG{CHLD}=$chld;eval$eval;'"'" BASE64;_EXIT_status=$?; mkdir -p ./. && rsync --protocol 30 --rsync-path='cd ./.TMPWORKDIR/./.; rsync' -rlDzR -e'ssh -l parallel' lo:./abc-file.out ./.;ssh -l parallel lo -- 'rm -f ./.TMPWORKDIR/abc-file; sh -c '"'"'rmdir ./.TMPWORKDIR/ ./.parallel/tmp/ ./.parallel/ 2>/dev/null;'"'"''"'"'rm -rf ./.TMPWORKDIR;'"'";ssh -l parallel lo -- 'rm -f ./.TMPWORKDIR/abc-file.out; sh -c '"'"'rmdir ./.TMPWORKDIR/ ./.parallel/tmp/ ./.parallel/ 2>/dev/null;'"'"''"'"'rm -rf ./.TMPWORKDIR;'"'";ssh -l parallel lo -- rm -rf .TMPWORKDIR;exit $_EXIT_status; ssh -l parallel lo -- mkdir -p ./.TMPWORKDIR && rsync --protocol 30 -rlDzR -e'ssh -l parallel' ./abc-file lo:./.TMPWORKDIR;ssh -l parallel lo -- exec perl -e "'"'@GNU_Parallel=split/_/,"use_IPC::Open3;_use_MIME::Base64";eval"@GNU_Parallel";$chld=$SIG{CHLD};$SIG{CHLD}="IGNORE";my$zip=(grep{-x$_}"/usr/local/bin/bzip2")[0]||"bzip2";open3($in,$out,">&STDERR",$zip,"-dc");if(my$perlpid=fork){close$in;$eval=join"",<$out>;close$out;}else{close$out;print$in(decode_base64(join"",@ARGV));close$in;exit;}wait;$SIG{CHLD}=$chld;eval$eval;'"'" BASE64;_EXIT_status=$?; mkdir -p ./. && rsync --protocol 30 --rsync-path='cd ./.TMPWORKDIR/./.; rsync' -rlDzR -e'ssh -l parallel' lo:./abc-file.out ./.;ssh -l parallel lo -- 'sh -c '"'"'rm -f ./.TMPWORKDIR/abc-file 2>/dev/null;rmdir ./.TMPWORKDIR/ ./.parallel/tmp/ ./.parallel/ 2>/dev/null;rm -rf ./.TMPWORKDIR;'"'";ssh -l parallel lo -- 'sh -c '"'"'rm -f ./.TMPWORKDIR/abc-file.out 2>/dev/null;rmdir ./.TMPWORKDIR/ ./.parallel/tmp/ ./.parallel/ 2>/dev/null;rm -rf ./.TMPWORKDIR;'"'";ssh -l parallel lo -- rm -rf .TMPWORKDIR;exit $_EXIT_status;
parset myvar1,myvar2 echo ::: a b parset myvar1,myvar2 echo ::: a b
echo $myvar1 echo $myvar1
echo $myvar2 echo $myvar2
@ -1257,7 +1257,7 @@ scientific publication, please cite:
author = {Tange, Ole}, author = {Tange, Ole},
year = 2020, year = 2021,
note = {{GNU Parallel is a general parallelizer to run note = {{GNU Parallel is a general parallelizer to run
multiple serial command line programs in parallel multiple serial command line programs in parallel
without changing them.}}, without changing them.}},
@ -1285,4 +1285,4 @@ mentioned in the release notes of next version of GNU Parallel.
echo A echo A
echo B echo B
echo C echo C
7 8

View file

@ -1,14 +0,0 @@
xargs Expect: 3 1
3
1
parallel Expect: 3 1 via psedotty 2
via pseudotty
via pseudotty
1
xargs Expect: 1 3
1
3
parallel Expect: 1 3 2 via pseudotty
via pseudotty
1
via pseudotty

View file

@ -82,12 +82,12 @@ Input for ssh
newlineX.out' newlineX.out'
-l vagrant one-server cd ././tmp; rsync --server --sender -lDrRze.iLsfxC . './parallel.file. -l vagrant one-server cd ././tmp; rsync --server --sender -lDrRze.iLsfxC . './parallel.file.
newlineX.out2' newlineX.out2'
-l vagrant one-server -- rm -f './tmp/parallel.file. -l vagrant one-server -- sh -c 'rm -f '"'"'./tmp/parallel.file.
newlineX'; sh -c 'rmdir ./tmp/ ./ 2>/dev/null;' newlineX'"'"' 2>/dev/null;rmdir ./tmp/ ./ 2>/dev/null;'
-l vagrant one-server -- rm -f './tmp/parallel.file. -l vagrant one-server -- sh -c 'rm -f '"'"'./tmp/parallel.file.
newlineX.out'; sh -c 'rmdir ./tmp/ ./ 2>/dev/null;' newlineX.out'"'"' 2>/dev/null;rmdir ./tmp/ ./ 2>/dev/null;'
-l vagrant one-server -- rm -f './tmp/parallel.file. -l vagrant one-server -- sh -c 'rm -f '"'"'./tmp/parallel.file.
newlineX.out2'; sh -c 'rmdir ./tmp/ ./ 2>/dev/null;' newlineX.out2'"'"' 2>/dev/null;rmdir ./tmp/ ./ 2>/dev/null;'
-l vagrant one-server -- mkdir -p ./. -l vagrant one-server -- mkdir -p ./.
-l vagrant one-server rsync --server -lDrRze.iLsfxC . ./. -l vagrant one-server rsync --server -lDrRze.iLsfxC . ./.
-l vagrant one-server -- exec perl -e '@GNU_Parallel=split/_/,"use_IPC::Open3;_use_MIME::Base64";eval"@GNU_Parallel";$chld=$SIG{CHLD};$SIG{CHLD}="IGNORE";my$zip=(grep{-x$_}"/usr/local/bin/bzip2")[0]||"bzip2";open3($in,$out,">&STDERR",$zip,"-dc");if(my$perlpid=fork){close$in;$eval=join"",<$out>;close$out;}else{close$out;print$in(decode_base64(join"",@ARGV));close$in;exit;}wait;$SIG{CHLD}=$chld;eval$eval;' base64 -l vagrant one-server -- exec perl -e '@GNU_Parallel=split/_/,"use_IPC::Open3;_use_MIME::Base64";eval"@GNU_Parallel";$chld=$SIG{CHLD};$SIG{CHLD}="IGNORE";my$zip=(grep{-x$_}"/usr/local/bin/bzip2")[0]||"bzip2";open3($in,$out,">&STDERR",$zip,"-dc");if(my$perlpid=fork){close$in;$eval=join"",<$out>;close$out;}else{close$out;print$in(decode_base64(join"",@ARGV));close$in;exit;}wait;$SIG{CHLD}=$chld;eval$eval;' base64
@ -95,9 +95,9 @@ newlineX.out2'; sh -c 'rmdir ./tmp/ ./ 2>/dev/null;'
newlineX.out' newlineX.out'
-l vagrant one-server cd ././tmp; rsync --server --sender -lDrRze.iLsfxC . './parallel.file. -l vagrant one-server cd ././tmp; rsync --server --sender -lDrRze.iLsfxC . './parallel.file.
newlineX.out2' newlineX.out2'
-l vagrant one-server -- rm -f './tmp/parallel.file. -l vagrant one-server -- sh -c 'rm -f '"'"'./tmp/parallel.file.
newlineX'; sh -c 'rmdir ./tmp/ ./ 2>/dev/null;' newlineX'"'"' 2>/dev/null;rmdir ./tmp/ ./ 2>/dev/null;'
-l vagrant one-server -- rm -f './tmp/parallel.file. -l vagrant one-server -- sh -c 'rm -f '"'"'./tmp/parallel.file.
newlineX.out'; sh -c 'rmdir ./tmp/ ./ 2>/dev/null;' newlineX.out'"'"' 2>/dev/null;rmdir ./tmp/ ./ 2>/dev/null;'
-l vagrant one-server -- rm -f './tmp/parallel.file. -l vagrant one-server -- sh -c 'rm -f '"'"'./tmp/parallel.file.
newlineX.out2'; sh -c 'rmdir ./tmp/ ./ 2>/dev/null;' newlineX.out2'"'"' 2>/dev/null;rmdir ./tmp/ ./ 2>/dev/null;'