sql: Added unittest, --shebang, --version, --help.

parallel: bugfix in unittest.
Passes unittest
This commit is contained in:
Ole Tange 2010-09-01 15:26:45 +02:00
parent 16939a0fa6
commit 32a3f8340b
7 changed files with 356 additions and 58 deletions

View file

@ -1,17 +1,19 @@
sql: Added unittest, --shebang, --version, --help.
parallel: bugfix in unittest.
== FEX ==
fex syntax for splitting fields
http://www.semicomplete.com/projects/fex/
sql :foo 'select * from bar' | parallel --fex '|{1,2}' do_stuff {2} {1}
== SQL ==
Example:
sql :foo 'select * from bar' | parallel --colsep '\|' do_stuff {4} {1}
Virker shebang?
#!/usr/bin/sql -t
== Build ==
GNU kræver:
--version
--help
-h
build.opensuse.org
--autocolsep: Læs alle linjer.
@ -51,10 +53,6 @@ colsep = [sepchars]{no_of_sepchars}
# TODO Debian package
# TODO to kill from a run script parallel should set PARALLEL_PID that can be sig termed
# TAGS: parallel | parallel processing | multicore | multiprocessor | Clustering/Distributed Networks
# job control | multiple jobs | parallelization | text processing | cluster | filters
# Clustering Tools | Command Line Tools | Utilities | System Administration
# Bash parallel
=head1 YouTube video2
@ -235,25 +233,9 @@ If there are nomore jobs (STDIN is eof) then make sure to
distribute the arguments evenly if running -X.
=head1 search terms
GNU parallel execution shell bash script simultaneous concurrent linux
scripting run xargs ppss code.google.com/p/ppss/
=head1 options
One char options not used: F G J K P Q Y
Skilletegn i sshlogin:
#=item B<--sshlogin> I<[ncpu/]sshlogin[,[ncpu/]sshlogin[,...]]> (beta testing)
# Skilletegn:
# No: "#!&()?\<>|;*'~ shellspecial
# No: @.- part of user@i.p.n.r i.p.n.r host-name
# No: , separates different sshlogins
# No: space Will make it hard to do: 8/server1,server2
# Maybe: / 8//usr/bin/myssh,//usr/bin/ssh
# %/=:_^
One char options not used: A D F G J K O Q R T Z
=head2 mutex

View file

@ -58,6 +58,18 @@ gpg -b parallel-$YYYYMMDD.tar.bz2
YYYYMMDD=`yyyymmdd`
echo put parallel-$YYYYMMDD.tar.bz2{,.sig,*asc} | ncftp ftp://ftp-upload.gnu.org/incoming/ftp/
== Download and test ==
cd /tmp
wget http://ftp.gnu.org/gnu/parallel/parallel-$YYYYMMDD.tar.bz2
tar xjvf parallel-$YYYYMMDD.tar.bz2
cd parallel-$YYYYMMDD
make -j && sudo make -j install
== Update OpenSUSE build system ==
https://build.opensuse.org/package/show?package=parallel&project=home%3Atange
== Update Savannah ==
https://savannah.gnu.org/news/submit.php?group=parallel
@ -85,24 +97,43 @@ http://groups.google.com/group/comp.unix.shell/post
Newsgroups: comp.unix.shell,comp.unix.admin
<<<<<
from:tange@gnu.org
to:parallel@gnu.org, bug-parallel@gnu.org, info-gnu@gnu.org, bug-directory@gnu.org
cc:Peter Simons <simons@cryp.to>, Sandro Cazzaniga <kharec@mandriva.org>,
Tim Cuthbertson <tim3d.junk@gmail.com>, Ludovic Courtès <ludo@gnu.org>,
Markus Ammer <mkmm@gmx-topmail.de>, Pavel Nuzhdin <pnzhdin@gmail.com>,
Phil Sung <psung@alum.mit.edu>, Michael Shigorin <mike@altlinux.org>
Phil Sung <psung@alum.mit.edu>, Michael Shigorin <mike@altlinux.org>,
Andrew McFague <amcfague@wgen.net>
Subject: GNU Parallel 20100822 released
Subject: GNU Parallel 2010XXXX released
GNU Parallel 20100822 has been released. It is available for
GNU Parallel 2010XXXX has been released. It is available for
download at: http://ftp.gnu.org/gnu/parallel/
New in this release:
* sql - a small script to access sql bases from the command line which
is a handy companion to parallel --colsep
* Using --shebang GNU Parallel can be used as the parser for a script.
E.g: #!/usr/bin/parallel --shebang traceroute (followed by lines of
hosts)
* First community generated bugfixes
* Alt Linux package of GNU Parallel. Thanks to Michael Shigorin <mike
at altlinux dot org>
* Untested CentOS, Fedora, Mandriva, RedHat, and SUSE packages
available through OpenSUSE build service:
https://build.opensuse.org/package/show?package=parallel&project=home%3Atange
* Review of GNU Parallel. Thanks to Andrew McFague amcfague at wgen dot net
http://www.andrew-mcfague.com/linux/utilities-linux/
commands-every-serious-nix-user-should-know/#parallel
* First 1000 views of the intro video
= About GNU Parallel =
GNU Parallel is a shell tool for executing jobs in parallel using one

View file

@ -14,6 +14,8 @@ B<parallel> [options] [I<command> [arguments]] B<::::> argfile(s)
B<parallel> --semaphore [options] I<command>
B<#!/usr/bin/parallel> --shebang [options] [I<command> [arguments]]
=head1 DESCRIPTION
GNU B<parallel> is a shell tool for executing jobs in parallel locally
@ -1535,7 +1537,8 @@ Inputs
I3. Arguments can be read from multiple files
I4. Arguments can be read from command line
I5. Arguments can be read from a table
I6. Line oriented input as default (Quoting of special chars not needed)
I6. Arguments can be read from the same file using #! (shebang)
I7. Line oriented input as default (Quoting of special chars not needed)
Manipulation of input
M1. Composed command
@ -1567,22 +1570,22 @@ Legend
As every new version of the programs are not tested the table may be
outdated. Please file a bug-report if you find errors.
parallel: E1 E2 E3 E4 E5 O1 O2 O3 O4 O5 O6 I1 I2 I3 I4 I5 I6 M1 M2 M3
M4 M5 M6 R1 R2 R3 R4 R5 R6 R7 R8 S1 S2
parallel: E1 E2 E3 E4 E5 O1 O2 O3 O4 O5 O6 I1 I2 I3 I4 I5 I6 I7 M1 M2
M3 M4 M5 M6 R1 R2 R3 R4 R5 R6 R7 R8 S1 S2
xargs: E1 - - - - - O2 O3 - O5 O6 I1 I2 - - - - - M2 M3 -
xargs: E1 - - - - - O2 O3 - O5 O6 I1 I2 - - - - - - M2 M3 -
- - - - - - - x - - -
find -exec: - - - x - x O2 O3 O4 O5 O6 - - - - - - - M2 M3 -
find -exec: - - - x - x O2 O3 O4 O5 O6 - - - - - - - - M2 M3 -
- - - - - - - - - x x
make -j: E1 - - - E5 O1 O2 O3 - x O6 - - - - - - - - - - - - - - - - -
make -j: E1 - - - E5 O1 O2 O3 - x O6 - - - - - - - - - - - - - - - - - -
- - -
ppss: E1 E2 ?E3 E4 - O1 - - x - - I1 I2 - - - I6 M1 - M3 - - M6 R1 R2
ppss: E1 E2 ?E3 E4 - O1 - - x - - I1 I2 - - - - I7 M1 - M3 - - M6 R1 R2
R3 R4 - - ?R7 - -
pexec: E1 - - E4 - O1 O2 O3 - O5 O6 I1 I2 - I4 I5 - M1 - M3 - - M6 R1
pexec: E1 - - E4 - O1 O2 O3 - O5 O6 I1 I2 - I4 I5 - - M1 - M3 - - M6 R1
- - - - R6 - S1 -
xjobs: TODO
@ -2223,7 +2226,6 @@ sub parse_options {
exec "$0 --skip-first-line -a $argfile @ARGV";
}
Getopt::Long::Configure ("bundling","require_order");
# Add options from .parallel/config
my $parallel_config = $ENV{'HOME'}."/.parallel/config";
if(-r $parallel_config) {
@ -2237,6 +2239,7 @@ sub parse_options {
}
# Add options from shell variable $PARALLEL
$ENV{'PARALLEL'} and unshift @ARGV, split/\n/, $ENV{'PARALLEL'};
Getopt::Long::Configure ("bundling","require_order");
GetOptions("debug|D" => \$::opt_D,
"xargs|m" => \$::opt_m,
"X" => \$::opt_X,
@ -2388,7 +2391,8 @@ sub parse_options {
}
# Semaphore defaults
# Must be done before computing number of processes and max_line_length as no args
# Must be done before computing number of processes and max_line_length
# because when running as a semaphore GNU Parallel does not read args
$Global::semaphore ||= ($0 =~ m:(^|/)sem$:); # called as 'sem'
if($Global::semaphore) {
# A semaphore does not take input from neither stdin nor file
@ -2407,7 +2411,7 @@ sub parse_options {
}
if(defined $::opt_eta) {
# must be done after opt_a
# must be done after opt_a because we need to read all args
$::opt_progress = $::opt_eta;
my @args = ();
while(more_arguments()) {
@ -2493,7 +2497,7 @@ sub read_args_from_command_line {
}
sub convert_argfiles_from_command_line_to_multiple_opt_a {
# convert :::: to multiple -a
# Convert :::: to multiple -a
# Remove :::: from @ARGV and move the following arguments to @::opt_a
# Returns:
# @ARGV without :::: and following args
@ -2518,6 +2522,7 @@ sub argfiles_xapply_style {
# Convert the n files into one queue
# Every n'th entry is from the same file
# Set opt_N to read n entries per invocation
# Returns: N/A
$Global::argfile = open_or_exit("/dev/null");
$::opt_N = $#::opt_a+1;
$Global::max_number_of_args = $#::opt_a+1;

119
src/sql
View file

@ -6,8 +6,9 @@ sql - execute a command on a database determined by a dburl
=head1 SYNOPSIS
B<sql> [B<-hnr>] [B<--table-size>] [B<--db-size>] [B<-p> I<pass-through>] [B<-s> I<string>] I<dburl> [I<command>]
B<sql> [B<-hnr>] [B<--table-size>] [B<--db-size>] [B<-p> I<pass-through>] [B<-s> I<string>] I<dburl> [I<sqlcommand>]
B<#!/usr/bin/sql> --shebang [options] I<dburl>
=head1 DESCRIPTION
@ -110,6 +111,12 @@ retry the command. Default is I<--retries 1>.
Field separator. Use I<string> as seperator between columns.
=item B<--skip-first-line>
Do not use the first line of input (used by GNU B<sql> itself
when called with B<--shebang>).
=item B<--table-size>
=item B<--tablesize>
@ -124,6 +131,18 @@ Size of tables. Show the size of the tables in the database.
Print the version GNU B<sql> and exit.
=item B<--shebang>
=item B<-Y>
GNU B<sql> can be called as a shebang (#!) command as the first line of a script. Like this:
#!/usr/bin/sql -Y mysql:///
select * from users;
For this to work B<--shebang> or B<-Y> must be set as the first option.
=back
=head1 DBURL
@ -172,6 +191,19 @@ sql -n mysql:/// 'show tables' | parallel sql mysql:/// delete from {};
sql -n pg:/// '\dt' | awk '{print $3}' | parallel -r sql pg:/// drop table {};
=head2 Run as a script
Create a script called I<demosql>:
#!/usr/bin/sql -Y mysql:///
select * from users;
Then do:
chmod 755 demosql; ./demosql
=head1 REPORTING BUGS
GNU B<sql> is part of GNU B<parallel>. Report bugs to <bug-parallel@gnu.org>.
@ -179,7 +211,7 @@ GNU B<sql> is part of GNU B<parallel>. Report bugs to <bug-parallel@gnu.org>.
=head1 AUTHOR
Copyright (C) 2008,2009,2010 Ole Tange, Ange Optimization http://ange.dk
Copyright (C) 2008,2009,2010 Ole Tange http://ole.tange.dk
=head1 LICENSE
@ -310,17 +342,8 @@ B<mysql>(1), B<psql>(1), B<sqlplus>(1), B<rlwrap>(1)
use Getopt::Long;
use strict;
Getopt::Long::Configure ("bundling","require_order");
GetOptions("passthrough|p=s@" => \$::opt_p,
"sep|s=s" => \$::opt_s,
"html" => \$::opt_html,
"show-processlist|proclist|listproc" => \$::opt_processlist,
"db-size|dbsize" => \$::opt_dbsize,
"table-size|tablesize" => \$::opt_tablesize,
"noheaders|no-headers|n" => \$::opt_n,
"r" => \$::opt_retry,
"retries=s" => \$::opt_retries,
"debug" => \$::opt_debug);
parse_options();
my $pass_through_options = (defined $::opt_p) ? join(" ",@{$::opt_p}) : "";
@ -377,6 +400,45 @@ do {
$Global::Initfile && unlink $Global::Initfile;
exit ($err);
sub parse_options {
$Global::version = 20100822;
$Global::progname = 'sql';
# This must be done first as this may exec myself
if(defined $ARGV[0] and ($ARGV[0]=~/^-Y/ or $ARGV[0]=~/^--shebang /)) {
# Program is called from #! line in script
$ARGV[0]=~s/^-Y //; # remove -Y if on its own
$ARGV[0]=~s/^-Y/-/; # remove -Y if bundled with other options
$ARGV[0]=~s/^--shebang //; # remove --shebang if it is set
my $argfile = pop @ARGV;
# exec myself to split @ARGV into separate fields
exec "$0 --skip-first-line < $argfile @ARGV";
}
Getopt::Long::Configure ("bundling","require_order");
GetOptions("passthrough|p=s@" => \$::opt_p,
"sep|s=s" => \$::opt_s,
"html" => \$::opt_html,
"show-processlist|proclist|listproc" => \$::opt_processlist,
"db-size|dbsize" => \$::opt_dbsize,
"table-size|tablesize" => \$::opt_tablesize,
"noheaders|no-headers|n" => \$::opt_n,
"r" => \$::opt_retry,
"retries=s" => \$::opt_retries,
"debug" => \$::opt_debug,
# Shebang #!/usr/bin/parallel -Yotheroptions
"Y|shebang" => \$::opt_shebang,
"skip-first-line" => \$::opt_skip_first_line,
# GNU requirements
"help|h" => \$::opt_help,
"version|V" => \$::opt_version,
) || die_usage();
if(defined $::opt_help) { die_usage(); }
if(defined $::opt_version) { version(); exit(0); }
}
sub database_driver_alias {
my $driver = shift;
my %database_driver_alias = ("mysql" => "mysql",
@ -498,7 +560,6 @@ sub oracle_commands {
return($batch_command,$interactive_command);
}
# Return the code for 'show processlist' in the chosen database dialect
sub processlist {
my $dbdriver = shift;
@ -694,4 +755,34 @@ sub undef_if_empty {
return $_[0];
}
sub version {
# Returns: N/A
print join("\n",
"GNU $Global::progname $Global::version",
"Copyright (C) 2009,2010 Ole Tange and Free Software Foundation, Inc.",
"License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>",
"This is free software: you are free to change and redistribute it.",
"GNU $Global::progname comes with no warranty.",
"",
"Web site: http://www.gnu.org/software/${Global::progname}\n"
);
}
sub die_usage {
# Returns: N/A
usage();
exit(255);
}
sub usage {
# Returns: N/A
print "Usage:\n";
print "$Global::progname [options] dburl [sqlcommand]\n";
print "$Global::progname [options] dburl < sql_command_file\n";
print "\n";
print "See 'man $Global::progname' for the options\n";
}
# TODO --list-databases: psql -l eller '\l'. mysql show databases. Oracle ?
$::opt_skip_first_line = $::opt_shebang = 0;

98
unittest/tests-to-run/sql01.sh Executable file
View file

@ -0,0 +1,98 @@
#!/bin/bash
# Setup
sql mysql://root@ "drop user 'sqlunittest'@'localhost'"
sql mysql://root@ DROP DATABASE sqlunittest;
sql mysql://root@ CREATE DATABASE sqlunittest;
sql mysql://root@ "CREATE USER 'sqlunittest'@'localhost' IDENTIFIED BY 'CB5A1FFFA5A';"
sql mysql://root@ "GRANT ALL PRIVILEGES ON sqlunittest.* TO 'sqlunittest'@'localhost';"
echo '### Test of #! -Y with file as input'
cat >/tmp/shebang <<EOF
#!/usr/local/bin/sql -Y mysql:///tange
SELECT 'Yes it does' AS 'Testing if -Y works';
EOF
chmod 755 /tmp/shebang
/tmp/shebang
echo '### Test of #! --shebang with file as input'
cat >/tmp/shebang <<EOF
#!/usr/local/bin/sql --shebang mysql:///tange
SELECT 'Yes it does' AS 'Testing if --shebang works';
EOF
chmod 755 /tmp/shebang
/tmp/shebang
echo '### Test reading sql on command line'
sql mysql:///tange "SELECT 'Yes it does' as 'Test reading SQL from command line';"
echo '### Test reading sql from file'
cat >/tmp/unittest.sql <<EOF
DROP TABLE IF EXISTS unittest;
CREATE TABLE unittest (
id INT,
data VARCHAR(100)
);
INSERT INTO unittest VALUES (1,'abc');
INSERT INTO unittest VALUES (3,'def');
SELECT 'Yes it does' as 'Test reading SQL from file works';
EOF
sql mysql:///tange </tmp/unittest.sql
echo '### Test dburl with username password host port'
sql mysql://sqlunittest:CB5A1FFFA5A@localhost:3306/sqlunittest </tmp/unittest.sql
echo "### Test .dburl.aliases"
echo :sqlunittest mysql://sqlunittest:CB5A1FFFA5A@localhost:3306/sqlunittest >> ~/.dburl.aliases
sql :sqlunittest "SELECT 'Yes it does' as 'Test if .dburl.aliases works';"
echo "### Test --noheaders --no-headers -n"
sql -n :sqlunittest 'select * from unittest' | parallel --colsep '\t' echo {2} {1}
sql --noheaders :sqlunittest 'select * from unittest' | parallel --colsep '\t' echo {2} {1}
sql --no-headers :sqlunittest 'select * from unittest' | parallel --colsep '\t' echo {2} {1}
echo "### Test --sep -s";
sql --no-headers -s : pg:/// 'select 1,2' | parallel --colsep ':' echo {2} {1}
sql --no-headers --sep : pg:/// 'select 1,2' | parallel --colsep ':' echo {2} {1}
echo "### Test --passthrough -p";
sql -p -H :sqlunittest 'select * from unittest'
echo
sql --passthrough -H :sqlunittest 'select * from unittest'
echo
echo "### Test --html";
sql --html mysql:///tange 'select * from unittest'
echo
echo "### Test --show-processlist|proclist|listproc";
sql --show-processlist :sqlunittest | wc
sql --proclist :sqlunittest | wc
sql --listproc :sqlunittest | wc
echo "### Test --db-size --dbsize";
sql --dbsize :sqlunittest | wc
sql --db-size :sqlunittest | wc
echo "### Test --table-size --tablesize"
sql --tablesize :sqlunittest | wc -l
sql --table-size :sqlunittest | wc -l
echo "### Test --debug"
sql --debug :sqlunittest "SELECT 'Yes it does' as 'Test if --debug works';"
echo "### Test --version -V"
sql --version | wc
sql -V | wc
echo "### Test -r"
stdout sql -r --debug pg://nongood@127.0.0.3:2227/ "SELECT 'This should fail 3 times';"
echo "### Test --retries=s"
stdout sql --retries=4 --debug pg://nongood@127.0.0.3:2227/ "SELECT 'This should fail 4 times';"
echo "### Test --help -h"
sql --help
sql -h

View file

@ -2,7 +2,7 @@
echo '### Test of #! with file as input'
cat >/tmp/shebang <<EOF
#!/usr/local/bin/parallel -Yr echo
#!/usr/local/bin/parallel -Yrk echo
A
B
C
@ -12,7 +12,7 @@ chmod 755 /tmp/shebang
echo '### Test of #! with 2 files as input'
cat >/tmp/shebang <<EOF
#!/usr/local/bin/parallel -Yr -a /tmp/123 echo
#!/usr/local/bin/parallel -Yrk -a /tmp/123 echo
A
B
C

View file

@ -0,0 +1,91 @@
### Test of #! -Y with file as input
Testing if -Y works
Yes it does
### Test of #! --shebang with file as input
Testing if --shebang works
Yes it does
### Test reading sql on command line
Test reading SQL from command line
Yes it does
### Test reading sql from file
Test reading SQL from file works
Yes it does
### Test dburl with username password host port
Test reading SQL from file works
Yes it does
### Test .dburl.aliases
Test if .dburl.aliases works
Yes it does
### Test --noheaders --no-headers -n
abc 1
def 3
abc 1
def 3
abc 1
def 3
### Test --sep -s
2 1
2 1
### Test --passthrough -p
<TABLE BORDER=1><TR><TH>id</TH><TH>data</TH></TR><TR><TD>1</TD><TD>abc</TD></TR><TR><TD>3</TD><TD>def</TD></TR></TABLE>
<TABLE BORDER=1><TR><TH>id</TH><TH>data</TH></TR><TR><TD>1</TD><TD>abc</TD></TR><TR><TD>3</TD><TD>def</TD></TR></TABLE>
### Test --html
<TABLE BORDER=1><TR><TH>id</TH><TH>data</TH></TR><TR><TD>1</TD><TD>abc</TD></TR><TR><TD>3</TD><TD>def</TD></TR></TABLE>
### Test --show-processlist|proclist|listproc
2 17 108
2 17 108
2 17 108
### Test --db-size --dbsize
3 12 89
3 12 89
### Test --table-size --tablesize
30
30
### Test --debug
mysql -C --host=localhost --user=sqlunittest --port=3306 --password=CB5A1FFFA5A sqlunittest
Test if --debug works
Yes it does
### Test --version -V
7 43 306
7 43 306
### Test -r
psql -h 127.0.0.3 -U nongood -p 2227
psql: could not connect to server: Connection refused
Is the server running on host "127.0.0.3" and accepting
TCP/IP connections on port 2227?
psql -h 127.0.0.3 -U nongood -p 2227
psql: could not connect to server: Connection refused
Is the server running on host "127.0.0.3" and accepting
TCP/IP connections on port 2227?
psql -h 127.0.0.3 -U nongood -p 2227
psql: could not connect to server: Connection refused
Is the server running on host "127.0.0.3" and accepting
TCP/IP connections on port 2227?
### Test --retries=s
psql -h 127.0.0.3 -U nongood -p 2227
psql: could not connect to server: Connection refused
Is the server running on host "127.0.0.3" and accepting
TCP/IP connections on port 2227?
psql -h 127.0.0.3 -U nongood -p 2227
psql: could not connect to server: Connection refused
Is the server running on host "127.0.0.3" and accepting
TCP/IP connections on port 2227?
psql -h 127.0.0.3 -U nongood -p 2227
psql: could not connect to server: Connection refused
Is the server running on host "127.0.0.3" and accepting
TCP/IP connections on port 2227?
psql -h 127.0.0.3 -U nongood -p 2227
psql: could not connect to server: Connection refused
Is the server running on host "127.0.0.3" and accepting
TCP/IP connections on port 2227?
### Test --help -h
Usage:
sql [options] dburl [sqlcommand]
sql [options] dburl < sql_command_file
See 'man sql' for the options
Usage:
sql [options] dburl [sqlcommand]
sql [options] dburl < sql_command_file
See 'man sql' for the options