tange/parallel
1
0
Fork 0
mirror of https://git.savannah.gnu.org/git/parallel.git synced 2024-11-21 21:47:54 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Fixed bug #59006: rsync version 3.2.3 is not detected correctly.

Browse source
This commit is contained in:
Ole Tange 2020-08-29 22:12:46 +02:00
parent 5ea74055dd
commit e6c9dfe010
12 changed files with 94 additions and 47 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

10
doc/haikus
View file

@ -1,14 +1,15 @@
Quote of the month: Quote of the month:
Gnu parallel is also awesome, fwiw. Great tool, gets jobs done fast.
-- Rogan Dawes @RoganDawes@twitter Great tool, gets jobs done fast.
Great tool, gets jobs done fast.
-- Paul F. De La Cruz @pdelacruzcc
I get a weird sense of satisfaction every single time I see the I get a weird sense of satisfaction every single time I see the
lovely logo of #GNU Parallel (plus, what an underrated piece of lovely logo of #GNU Parallel (plus, what an underrated piece of
great software!) great software!)
-- Emre Sevinç @EmreSevinc@twitter -- Emre Sevinç @EmreSevinc@twitter
I have gotten a *ton* of mileage out of jq, awk, and GNU parallel, I have gotten a *ton* of mileage out of jq, awk, and GNU parallel,
even at multi-GB sizes. even at multi-GB sizes.
-- Eric Wolak @ericthewolak@twitter -- Eric Wolak @ericthewolak@twitter
@ -68,6 +69,9 @@ https://negfeedback.blogspot.com/2020/05/indispensable-command-line-tools.html
=== Used === === Used ===
Gnu parallel is also awesome, fwiw.
-- Rogan Dawes @RoganDawes@twitter
With multicore systems everywhere GNU Parallel is a must have tool. With multicore systems everywhere GNU Parallel is a must have tool.
-- Neil H. Watson @neil_h_watson@twitter -- Neil H. Watson @neil_h_watson@twitter

11
doc/release_new_version
View file

@ -123,7 +123,7 @@ torsocks git push origin $YYYYMMDD
== Zenodo == == Zenodo ==
Add tar.bz2 and publish. Add tar.bz2 [Start upload] and [Publish].
== Update documentation == == Update documentation ==
@ -134,6 +134,7 @@ export YYYYMMDD=${YYYYMMDD:0:6}23
echo $YYYYMMDD echo $YYYYMMDD
perl -i -pe "/version/ and s/20\d\d\d\d\d\d/$YYYYMMDD/" src/parallel perl -i -pe "/version/ and s/20\d\d\d\d\d\d/$YYYYMMDD/" src/parallel
perl -i -pe "/version/ and s/20\d\d\d\d\d\d/$YYYYMMDD/" src/sql perl -i -pe "/version/ and s/20\d\d\d\d\d\d/$YYYYMMDD/" src/sql
perl -i -pe "/version/ and s/20\d\d\d\d\d\d/$YYYYMMDD/" src/parsort
perl -i -pe "/version/ and s/20\d\d\d\d\d\d/$YYYYMMDD/" src/niceload perl -i -pe "/version/ and s/20\d\d\d\d\d\d/$YYYYMMDD/" src/niceload
Unmodified beta since last version => production Unmodified beta since last version => production
@ -189,9 +190,9 @@ from:tange@gnu.org
to:parallel@gnu.org, bug-parallel@gnu.org to:parallel@gnu.org, bug-parallel@gnu.org
stable-bcc: Jesse Alama <jessealama@fastmail.fm> stable-bcc: Jesse Alama <jessealama@fastmail.fm>
Subject: GNU Parallel 20200822 ('Beirut') released <<[stable]>> Subject: GNU Parallel 20200922 ('Belarus/Lukashenko') released <<[stable]>>
GNU Parallel 20200822 ('') <<[stable]>> has been released. It is available for download at: http://ftpmirror.gnu.org/parallel/ GNU Parallel 20200922 ('') <<[stable]>> has been released. It is available for download at: http://ftpmirror.gnu.org/parallel/
<<No new functionality was introduced so this is a good candidate for a stable release.>> <<No new functionality was introduced so this is a good candidate for a stable release.>>
@ -207,6 +208,10 @@ New in this release:
News about GNU Parallel: News about GNU Parallel:
https://floki.blog/2020/08/more_unix_tools
https://computeontario.ca/event/webinar-options-for-solving-jobs-with-many-tasks/
Get the book: GNU Parallel 2018 http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html Get the book: GNU Parallel 2018 http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html

11
packager/obs/Makefile
View file

@ -13,12 +13,13 @@ all:
cd home\:tange/parallel/ && osc up cd home\:tange/parallel/ && osc up
cd home\:tange/parallel/ && parallel osc add ::: *.spec *.dsc *.tar.gz *.tar.bz2 && echo Src added OK || true cd home\:tange/parallel/ && parallel osc add ::: *.spec *.dsc *.tar.gz *.tar.bz2 && echo Src added OK || true
cd home\:tange/parallel/ && osc ci -m "New release" cd home\:tange/parallel/ && osc ci -m "New release"
# wait for "building" state # wait for "building" state
cd home\:tange/parallel/ && yes building | parallel -j1 --delay 10 --halt now,success=1 'osc results|G -E {}' cd home\:tange/parallel/ && yes building | parallel -j1 --delay 10 --halt now,success=1 'osc results|G -E {}'
# wait for "building" state to end # wait for "building" state to end
cd home\:tange/parallel/ && yes building | parallel -j1 --delay 10 --halt now,fail=1 'osc results|G -v Mageia -E {}' || true # Ignore Mageia and Raspbian_9.0 that are broken
# wait for "finished" state of .deb cd home\:tange/parallel/ && yes building | parallel -j1 --delay 10 --halt now,fail=1 'osc results|G -v Mageia -v Raspbian_9.0 -E {}' || true
# wait for "finished" state of .deb
cd home\:tange/parallel/ && echo succeeded | parallel -j1 --retries 30 --delay 10 --halt now,success=1 'osc results|G -E "(Debian|Ubuntu).*{}"' cd home\:tange/parallel/ && echo succeeded | parallel -j1 --retries 30 --delay 10 --halt now,success=1 'osc results|G -E "(Debian|Ubuntu).*{}"'
# wait for "finished" state of .rpm # wait for "finished" state of .rpm
cd home\:tange/parallel/ && echo succeeded | parallel -j1 --retries 30 --delay 10 --halt now,success=1 'osc results|G -E "(SUSE|SLE|Scientific|RHEL|Fedora|CentOS).*{}"' cd home\:tange/parallel/ && echo succeeded | parallel -j1 --retries 30 --delay 10 --halt now,success=1 'osc results|G -E "(SUSE|SLE|Scientific|RHEL|Fedora|CentOS).*{}"'

6
src/niceload
View file

@ -23,7 +23,7 @@
use strict; use strict;
use Getopt::Long; use Getopt::Long;
$Global::progname="niceload"; $Global::progname="niceload";
$Global::version = 20200822; $Global::version = 20200823;
Getopt::Long::Configure("bundling","require_order"); Getopt::Long::Configure("bundling","require_order");
get_options_from_array(\@ARGV) || die_usage(); get_options_from_array(\@ARGV) || die_usage();
if($opt::version) { if($opt::version) {
@ -1147,7 +1147,9 @@ sub io_status_linux {
my @iostat; my @iostat;
for(reverse @iostat_out) { for(reverse @iostat_out) {
/Device/ and last; /Device/ and last;
push @iostat, (split(/\s+/,$_))[13]; my @col = (split(/\s+/,$_));
# Util% is last column
push @iostat, pop @col;
} }
my $io = ::max(@iostat); my $io = ::max(@iostat);
return undef_as_zero($io)/10; return undef_as_zero($io)/10;

6
src/parallel
View file

@ -2160,7 +2160,7 @@ sub check_invalid_option_combinations() {
sub init_globals() { sub init_globals() {
# Defaults: # Defaults:
$Global::version = 20200822; $Global::version = 20200823;
$Global::progname = 'parallel'; $Global::progname = 'parallel';
$::name = "GNU Parallel"; $::name = "GNU Parallel";
$Global::infinity = 2**31; $Global::infinity = 2**31;
@ -7836,7 +7836,9 @@ sub cleanup_cmd($$$) {
if(not $rsync) { if(not $rsync) {
my @out = `rsync --version`; my @out = `rsync --version`;
for (@out) { for (@out) {
if(/version (\d+.\d+)(.\d+)?/) { # rsync version 3.1.3 protocol version 31
# rsync version v3.2.3 protocol version 31
if(/version v?(\d+.\d+)(.\d+)?/) {
if($1 >= 3.1) { if($1 >= 3.1) {
# Version 3.1.0 or later: Downgrade to protocol 30 # Version 3.1.0 or later: Downgrade to protocol 30
$rsync = "rsync --protocol 30"; $rsync = "rsync --protocol 30";

19
src/parcat.pod
View file

@ -16,6 +16,10 @@ you use:
(cat file1 & cat file2 &) | ... (cat file1 & cat file2 &) | ...
It is faster than doing:
parallel -j0 --lb cat ::: file*
Arguments can be given on the command line or passed in on stdin Arguments can be given on the command line or passed in on stdin
(standard input). (standard input).
@ -44,13 +48,16 @@ the files.
=head2 Simple line buffered output =head2 Simple line buffered output
GNU B<parallel> saves output to tempfiles. If the amount of data is B<traceroute> will often print half a line. If run in parallel, two
bigger than the free disk space, then you can use this technique to do instances may half-lines of their output. This can be avoided by
line buffering without saving to disk: saving the output to a fifo and then using B<parcat> to read the two
fifos in parallel:
mkfifo freenetproject.org.fifo tange.dk.fifo
traceroute freenetproject.org > freenetproject.org.fifo &
traceroute tange.dk > tange.dk.fifo &
parcat --rm *fifo
mkfifo slot-{1..5}
seq 10000000 | parallel -j5 --round --pipe 'cat > slot-{%}' &
parcat slot-{1..5} | wc
=head1 REPORTING BUGS =head1 REPORTING BUGS

16
src/parsort
View file

@ -117,7 +117,7 @@ GetOptions(
"help" => \$opt::dummy, "help" => \$opt::dummy,
) || exit(255); ) || exit(255);
$Global::progname = ($0 =~ m:(^|/)([^/]+)$:)[1]; $Global::progname = ($0 =~ m:(^|/)([^/]+)$:)[1];
$Global::version = 20200412; $Global::version = 20200823;
if($opt::version) { version(); exit 0; } if($opt::version) { version(); exit 0; }
@Global::sortoptions = @ARGV_before[0..($#ARGV_before-$#ARGV-1)]; @Global::sortoptions = @ARGV_before[0..($#ARGV_before-$#ARGV-1)];
#if($opt::zero_terminated) { $/ = "\0"; } #if($opt::zero_terminated) { $/ = "\0"; }
@ -256,6 +256,20 @@ sub which {
return wantarray ? @which : $which[0]; return wantarray ? @which : $which[0];
} }
sub version() {
# Returns: N/A
print join
("\n",
"GNU $Global::progname $Global::version",
"Copyright (C) 2020 Ole Tange, http://ole.tange.dk and Free Software",
"Foundation, Inc.",
"License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>",
"This is free software: you are free to change and redistribute it.",
"GNU $Global::progname comes with no warranty.",
"",
"Web site: https://www.gnu.org/software/parallel\n",
);
}
if(@ARGV) { if(@ARGV) {
sort_files(@ARGV); sort_files(@ARGV);

2
src/sql
View file

@ -574,7 +574,7 @@ $Global::Initfile && unlink $Global::Initfile;
exit ($err); exit ($err);
sub parse_options { sub parse_options {
$Global::version = 20200822; $Global::version = 20200823;
$Global::progname = 'sql'; $Global::progname = 'sql';
# This must be done first as this may exec myself # This must be done first as this may exec myself

10
testsuite/tests-to-run/parallel-local-ssh6.sh
View file

@ -147,6 +147,16 @@ par_timeout_onall() {
echo jobs failed: $? echo jobs failed: $?
} }
par_rsync_3.2.3() {
echo "bug #59006: rsync version 3.2.3 is not detected correctly"
PATH=$HOME/bin/rsync:$PATH
rsync --version | grep version
rm -f bug59006
parallel --return {} -Sparallel@lo touch ::: bug59006
ls bug59006
rm -f bug59006
}
export -f $(compgen -A function | grep par_) export -f $(compgen -A function | grep par_)
#compgen -A function | grep par_ | sort | parallel --delay $D -j$P --tag -k '{} 2>&1' #compgen -A function | grep par_ | sort | parallel --delay $D -j$P --tag -k '{} 2>&1'
compgen -A function | grep par_ | sort | compgen -A function | grep par_ | sort |

21
testsuite/tests-to-run/sql01.sh
View file

@ -1,18 +1,17 @@
#!/bin/bash #!/bin/bash
mysqlrootpass=${mysqlrootpass:-M-b+Ydjq4ejT4E}
MYSQL_ADMIN_DBURL=mysql://root:$mysqlrootpass@/mysql
exec 2>&1 exec 2>&1
# Setup ## Setup
sql $MYSQL_ADMIN_DBURL "drop user 'sqlunittest'@'localhost'" # mysqlrootpass=${mysqlrootpass:-M-b+Ydjq4ejT4E}
sql $MYSQL_ADMIN_DBURL DROP DATABASE sqlunittest; # MYSQL_ADMIN_DBURL=mysql://root:$mysqlrootpass@/mysql
sql $MYSQL_ADMIN_DBURL CREATE DATABASE sqlunittest; # sql $MYSQL_ADMIN_DBURL "drop user 'sqlunittest'@'localhost'"
sql $MYSQL_ADMIN_DBURL "CREATE USER 'sqlunittest'@'localhost' IDENTIFIED BY 'CB5A1FFFA5A';" # sql $MYSQL_ADMIN_DBURL DROP DATABASE sqlunittest;
sql $MYSQL_ADMIN_DBURL "GRANT ALL PRIVILEGES ON sqlunittest.* TO 'sqlunittest'@'localhost';" # sql $MYSQL_ADMIN_DBURL CREATE DATABASE sqlunittest;
# sql $MYSQL_ADMIN_DBURL "CREATE USER 'sqlunittest'@'localhost' IDENTIFIED BY 'CB5A1FFFA5A';"
# sql $MYSQL_ADMIN_DBURL "GRANT ALL PRIVILEGES ON sqlunittest.* TO 'sqlunittest'@'localhost';"
MYSQL_TEST_DBURL=mysql://sqlunittest:CB5A1FFFA5A@/ MYSQL_TEST_DBURL=mysql://tange:tange@/
echo '### Test of #! -Y with file as input' echo '### Test of #! -Y with file as input'
cat >/tmp/shebang <<EOF cat >/tmp/shebang <<EOF
@ -49,7 +48,7 @@ EOF
sql $MYSQL_TEST_DBURL/sqlunittest </tmp/unittest.sql sql $MYSQL_TEST_DBURL/sqlunittest </tmp/unittest.sql
echo '### Test dburl with username password host port' echo '### Test dburl with username password host port'
sql mysql://sqlunittest:CB5A1FFFA5A@localhost:3306/sqlunittest </tmp/unittest.sql sql mysql://tange:tange@localhost:3306/tange </tmp/unittest.sql
echo "### Test .sql/aliases" echo "### Test .sql/aliases"
mkdir -p ~/.sql mkdir -p ~/.sql

3
testsuite/wanted-results/parallel-local-ssh6
View file

@ -26,6 +26,9 @@ par_read_sshloginfile_from_stdin foo
par_remote_symlink_dir bug #51293: parallel does not preserve symlinked directory structure on remote par_remote_symlink_dir bug #51293: parallel does not preserve symlinked directory structure on remote
par_remote_symlink_dir OK: wd is still a symlink with --rsync-opts par_remote_symlink_dir OK: wd is still a symlink with --rsync-opts
par_remote_symlink_dir OK: wd is still a symlink with PARALLEL_RSYNC_OPTS par_remote_symlink_dir OK: wd is still a symlink with PARALLEL_RSYNC_OPTS
par_rsync_3.2.3 bug #59006: rsync version 3.2.3 is not detected correctly
par_rsync_3.2.3 rsync version v3.2.3 protocol version 31
par_rsync_3.2.3 bug59006
par_sshlogin_replacement ### show {sshlogin} and {host} par_sshlogin_replacement ### show {sshlogin} and {host}
par_sshlogin_replacement parallel@lo and lo par_sshlogin_replacement parallel@lo and lo
par_sshlogin_replacement /usr/bin/ssh parallel@lo and lo par_sshlogin_replacement /usr/bin/ssh parallel@lo and lo

26
testsuite/wanted-results/sql01
View file

@ -39,14 +39,14 @@ def 3
2 17 2 17
2 17 2 17
### Test --db-size --dbsize ### Test --db-size --dbsize
2 6 54 2 6 48
2 6 54 2 6 48
### Test --table-size --tablesize ### Test --table-size --tablesize
63 83
63 83
### Test --debug ### Test --debug
dburl mysql://sqlunittest:CB5A1FFFA5A@localhost:3306/sqlunittest dburl mysql://tange:tange@localhost:3306/tange
databasedriver mysql user sqlunittest password CB5A1FFFA5A host localhost port 3306 database sqlunittest query databasedriver mysql user tange password tange host localhost port 3306 database tange query
Test if --debug works Test if --debug works
Yes it does Yes it does
### Test --version -V ### Test --version -V
@ -55,28 +55,28 @@ Yes it does
### Test -r ### Test -r
dburl pg://nongood@127.0.0.3:2227/ dburl pg://nongood@127.0.0.3:2227/
databasedriver pg user nongood password host 127.0.0.3 port 2227 database nongood query databasedriver pg user nongood password host 127.0.0.3 port 2227 database nongood query
psql: could not connect to server: Connection refused psql: error: could not connect to server: could not connect to server: Connection refused
Is the server running on host "127.0.0.3" and accepting Is the server running on host "127.0.0.3" and accepting
TCP/IP connections on port 2227? TCP/IP connections on port 2227?
psql: could not connect to server: Connection refused psql: error: could not connect to server: could not connect to server: Connection refused
Is the server running on host "127.0.0.3" and accepting Is the server running on host "127.0.0.3" and accepting
TCP/IP connections on port 2227? TCP/IP connections on port 2227?
psql: could not connect to server: Connection refused psql: error: could not connect to server: could not connect to server: Connection refused
Is the server running on host "127.0.0.3" and accepting Is the server running on host "127.0.0.3" and accepting
TCP/IP connections on port 2227? TCP/IP connections on port 2227?
### Test --retries=s ### Test --retries=s
dburl pg://nongood@127.0.0.3:2227/ dburl pg://nongood@127.0.0.3:2227/
databasedriver pg user nongood password host 127.0.0.3 port 2227 database nongood query databasedriver pg user nongood password host 127.0.0.3 port 2227 database nongood query
psql: could not connect to server: Connection refused psql: error: could not connect to server: could not connect to server: Connection refused
Is the server running on host "127.0.0.3" and accepting Is the server running on host "127.0.0.3" and accepting
TCP/IP connections on port 2227? TCP/IP connections on port 2227?
psql: could not connect to server: Connection refused psql: error: could not connect to server: could not connect to server: Connection refused
Is the server running on host "127.0.0.3" and accepting Is the server running on host "127.0.0.3" and accepting
TCP/IP connections on port 2227? TCP/IP connections on port 2227?
psql: could not connect to server: Connection refused psql: error: could not connect to server: could not connect to server: Connection refused
Is the server running on host "127.0.0.3" and accepting Is the server running on host "127.0.0.3" and accepting
TCP/IP connections on port 2227? TCP/IP connections on port 2227?
psql: could not connect to server: Connection refused psql: error: could not connect to server: could not connect to server: Connection refused
Is the server running on host "127.0.0.3" and accepting Is the server running on host "127.0.0.3" and accepting
TCP/IP connections on port 2227? TCP/IP connections on port 2227?
### Test --help -h ### Test --help -h