diff --git a/NEWS b/NEWS index 5fa07548..1aa3c5f5 100644 --- a/NEWS +++ b/NEWS @@ -1,3 +1,17 @@ +20120722 + +* GNU Parallel was presented at Chiang Mai Bar Camp + http://barcampchiangmai.org/ + See write up here: + http://www.cnx-software.com/2012/07/02/use-gnu-parallel-to-speed-up-script-execution-on-multiple-cores-andor-machines/ + +* I don't always max out all my CPUs, but when I do, I do it with GNU + Parallel. + http://memegenerator.net/instance/22638454 + +* Bug fixes and man page updates. + + 20120622 * '-L n --pipe' will use records of n lines. This is useful when @@ -15,11 +29,11 @@ be better on computers with >10 cores while remaining the same on computers with few cores. -* GNU Parallel will be represented at Chiang Mai Bar Camp +* GNU Parallel will be represented at Chiang Mai Bar Camp. http://barcampchiangmai.org/ * Indexing Big Data on Amazon AWS: The Screencast - (Check out his T-shirt at 18:40-21:00) + (Check out his T-shirt at 18:40-21:00). http://www.opensourceconnections.com/2012/06/06/indexing-big-data-on-amazon-aws-screencast/ * biotoolbox uses GNU Parallel. @@ -28,10 +42,10 @@ * Spiceweasel uses GNU Parallel. https://github.com/mattray/spiceweasel -* GNU Parallel part of The Administrators Challenge +* GNU Parallel part of The Administrators Challenge. http://challenge.twistedrack.com/questions/qs3.php -* Finding Oldest Firefox Code using GNU Parallel +* Finding Oldest Firefox Code using GNU Parallel. http://gregoryszorc.com/blog/2012/06/18/finding-oldest-firefox-code/ * Bug fixes (quite a few for remote job running) and man page updates. diff --git a/configure b/configure index 93f73630..fae4ebbb 100755 --- a/configure +++ b/configure @@ -1,6 +1,6 @@ #! /bin/sh # Guess values for system-dependent variables and create Makefiles. -# Generated by GNU Autoconf 2.68 for parallel 20120622. +# Generated by GNU Autoconf 2.68 for parallel 20120722. # # Report bugs to . # @@ -559,8 +559,8 @@ MAKEFLAGS= # Identity of this package. PACKAGE_NAME='parallel' PACKAGE_TARNAME='parallel' -PACKAGE_VERSION='20120622' -PACKAGE_STRING='parallel 20120622' +PACKAGE_VERSION='20120722' +PACKAGE_STRING='parallel 20120722' PACKAGE_BUGREPORT='bug-parallel@gnu.org' PACKAGE_URL='' @@ -1176,7 +1176,7 @@ if test "$ac_init_help" = "long"; then # Omit some internal or obsolete options to make the list less imposing. # This message is too long to be a string in the A/UX 3.1 sh. cat <<_ACEOF -\`configure' configures parallel 20120622 to adapt to many kinds of systems. +\`configure' configures parallel 20120722 to adapt to many kinds of systems. Usage: $0 [OPTION]... [VAR=VALUE]... @@ -1242,7 +1242,7 @@ fi if test -n "$ac_init_help"; then case $ac_init_help in - short | recursive ) echo "Configuration of parallel 20120622:";; + short | recursive ) echo "Configuration of parallel 20120722:";; esac cat <<\_ACEOF @@ -1309,7 +1309,7 @@ fi test -n "$ac_init_help" && exit $ac_status if $ac_init_version; then cat <<\_ACEOF -parallel configure 20120622 +parallel configure 20120722 generated by GNU Autoconf 2.68 Copyright (C) 2010 Free Software Foundation, Inc. @@ -1326,7 +1326,7 @@ cat >config.log <<_ACEOF This file contains any messages produced by compilers while running configure, to aid debugging if configure makes a mistake. -It was created by parallel $as_me 20120622, which was +It was created by parallel $as_me 20120722, which was generated by GNU Autoconf 2.68. Invocation command line was $ $0 $@ @@ -2141,7 +2141,7 @@ fi # Define the identity of the package. PACKAGE='parallel' - VERSION='20120622' + VERSION='20120722' cat >>confdefs.h <<_ACEOF @@ -2704,7 +2704,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 # report actual input values of CONFIG_FILES etc. instead of their # values after options handling. ac_log=" -This file was extended by parallel $as_me 20120622, which was +This file was extended by parallel $as_me 20120722, which was generated by GNU Autoconf 2.68. Invocation command line was CONFIG_FILES = $CONFIG_FILES @@ -2766,7 +2766,7 @@ _ACEOF cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`" ac_cs_version="\\ -parallel config.status 20120622 +parallel config.status 20120722 configured by $0, generated by GNU Autoconf 2.68, with options \\"\$ac_cs_config\\" diff --git a/configure.ac b/configure.ac index 15d9ad30..e4eefb4b 100644 --- a/configure.ac +++ b/configure.ac @@ -1,4 +1,4 @@ -AC_INIT([parallel], [20120622], [bug-parallel@gnu.org]) +AC_INIT([parallel], [20120722], [bug-parallel@gnu.org]) AM_INIT_AUTOMAKE([-Wall -Werror foreign]) AC_CONFIG_HEADERS([config.h]) AC_CONFIG_FILES([ diff --git a/doc/release_new_version b/doc/release_new_version index 3fbe3f2a..16c2de8e 100644 --- a/doc/release_new_version +++ b/doc/release_new_version @@ -171,48 +171,26 @@ cc:Sandro Cazzaniga , Ryoichiro Suzuki , Jesse Alama -Subject: GNU Parallel 20120622 ('Chiang Mai') released +Subject: GNU Parallel 20120722 ('Nai Soi') released -GNU Parallel 20120622 ('Chiang Mai') has been released. It is +GNU Parallel 20127022 ('Nai Soi') has been released. It is available for download at: http://ftp.gnu.org/gnu/parallel/ +This is a bugfix release with no new features. Probably a good release +for stable long-term use. + New in this release: -* '-L n --pipe' will use records of n lines. This is useful when - processing data that have fixed records with a fixed number of - lines (e.g. fastq). - -* --filter-hosts will remove down hosts. For each remote host: check - that login through ssh works. If not: do not use this host. - Currently you can not put --filter-hosts in a profile, $PARALLEL, - /etc/parallel/config or similar. This is because GNU Parallel uses - GNU Parallel to compute this, so you will get an infinite loop. This - will likely be fixed in a later release. - -* --pipe now uses fork() instead of busy wait. The performance should - be better on computers with >10 cores while remaining the same on - computers with few cores. - -* GNU Parallel will be represented at Chiang Mai Bar Camp +* GNU Parallel was presented at Chiang Mai Bar Camp http://barcampchiangmai.org/ + See write up here: + http://www.cnx-software.com/2012/07/02/use-gnu-parallel-to-speed-up-script-execution-on-multiple-cores-andor-machines/ -* Indexing Big Data on Amazon AWS: The Screencast - (Check out his T-shirt at 18:40-21:00) - http://www.opensourceconnections.com/2012/06/06/indexing-big-data-on-amazon-aws-screencast/ +* I don't always max out all my CPUs, but when I do, I do it with GNU + Parallel. + http://memegenerator.net/instance/22638454 -* biotoolbox uses GNU Parallel. - https://code.google.com/p/biotoolbox/wiki/Pod_novo_wrapper - -* Spiceweasel uses GNU Parallel. - https://github.com/mattray/spiceweasel - -* GNU Parallel part of The Administrators Challenge - http://challenge.twistedrack.com/questions/qs3.php - -* Finding Oldest Firefox Code using GNU Parallel - http://gregoryszorc.com/blog/2012/06/18/finding-oldest-firefox-code/ - -* Bug fixes (quite a few for remote job running) and man page updates. +* Bug fixes and man page updates. = About GNU Parallel = diff --git a/packager/obs/home:tange/parallel/parallel.spec b/packager/obs/home:tange/parallel/parallel.spec index 81866d18..4eac07ba 100644 --- a/packager/obs/home:tange/parallel/parallel.spec +++ b/packager/obs/home:tange/parallel/parallel.spec @@ -1,6 +1,6 @@ Summary: Shell tool for executing jobs in parallel Name: parallel -Version: 20120622 +Version: 20120722 Release: 1 License: GPL Group: Productivity/File utilities diff --git a/src/niceload b/src/niceload index 864a8b12..1cf2f835 100755 --- a/src/niceload +++ b/src/niceload @@ -24,7 +24,7 @@ use strict; use Getopt::Long; $Global::progname="niceload"; -$Global::version = 20120622; +$Global::version = 20120722; Getopt::Long::Configure("bundling","require_order"); get_options_from_array(\@ARGV) || die_usage(); if($::opt_version) { diff --git a/src/parallel b/src/parallel index 13e952e9..255b14d2 100755 --- a/src/parallel +++ b/src/parallel @@ -584,7 +584,7 @@ sub get_options_from_array { sub parse_options { # Returns: N/A # Defaults: - $Global::version = 20120622; + $Global::version = 20120722; $Global::progname = 'parallel'; $Global::infinity = 2**31; $Global::debug = 0; @@ -2239,7 +2239,7 @@ sub swap_activity { # If the (remote) machine is Mac we should use vm_stat # swap_in and swap_out on GNU/Linux is $7 and $8 # swap_in and swap_out on Mac is $10 and $11 - $swap_activity = q[ { vmstat 1 2> /dev/null || vm_stat 1; } | ]. + $swap_activity = q[ { vmstat 1 2 2> /dev/null || vm_stat 1; } | ]. q[ awk 'NR!=4{next} NF==16{print $7*$8} NF==11{print $10*$11} {exit}' ]; if($self->{'string'} ne ":") { $swap_activity = $self->sshcommand() . " " . $self->serverlogin() . " " . diff --git a/src/parallel.pod b/src/parallel.pod index cfdde60f..0580b32e 100644 --- a/src/parallel.pod +++ b/src/parallel.pod @@ -462,7 +462,7 @@ See also: B<--bg>, B Implies B<--semaphore>. -=item B<--filter-hosts> (alpha testing) +=item B<--filter-hosts> (beta testing) Remove down hosts. For each remote host: check that login through ssh works. If not: do not use this host. @@ -549,7 +549,7 @@ specified, and for B<-I>{} otherwise. This option is deprecated; use B<-I> instead. -=item B<--joblog> I (alpha testing) +=item B<--joblog> I (beta testing) Logfile for executed jobs. Save a list of the executed jobs to I in the following TAB separated format: sequence number, @@ -645,7 +645,7 @@ to see the difference: parallel -j4 sleep {}\; echo {} ::: 2 1 4 3 parallel -j4 -k sleep {}\; echo {} ::: 2 1 4 3 -=item B<-L> I (alpha testing) +=item B<-L> I (beta testing) When used with B<--pipe>: Read records of I. @@ -675,7 +675,7 @@ B<-l 0> is an alias for B<-l 1>. Implies B<-X> unless B<-m>, B<--xargs>, or B<--pipe> is set. -=item B<--load> I (alpha testing) +=item B<--load> I (beta testing) Do not start new jobs on a given computer unless the load is less than I. I uses the same syntax as B<--jobs>, so I<100%> @@ -770,9 +770,9 @@ Instead of printing the output to stdout (standard output) the output of each job is saved in a file and the filename is then printed. -=item B<--pipe> (alpha testing) +=item B<--pipe> (beta testing) -=item B<--spreadstdin> (alpha testing) +=item B<--spreadstdin> (beta testing) Spread input to jobs on stdin (standard input). Read a block of data from stdin (standard input) and give one block of data as input to one @@ -1232,14 +1232,14 @@ different dir for the files. Setting B<--tmpdir> is equivalent to setting $TMPDIR. -=item B<--timeout> I (alpha testing) +=item B<--timeout> I (beta testing) Time out for command. If the command runs for longer than I seconds it will get killed with SIGTERM, followed by SIGTERM 200 ms later, followed by SIGKILL 200 ms later. -=item B<--tollef> (alpha testing) +=item B<--tollef> (beta testing) Make GNU B behave more like Tollef's parallel command. It activates B<-u>, B<-q>, and B<--arg-sep -->. It also causes B<-l> to @@ -1445,9 +1445,9 @@ Compare these two: See also B<--header>. -=item B<--shebang> (alpha testing) +=item B<--shebang> (beta testing) -=item B<--hashbang> (alpha testing) +=item B<--hashbang> (beta testing) GNU B can be called as a shebang (#!) command as the first line of a script. Like this: diff --git a/src/parallel.texi b/src/parallel.texi index 2680ad7c..2fd7b223 100644 --- a/src/parallel.texi +++ b/src/parallel.texi @@ -489,8 +489,8 @@ See also: @strong{--bg}, @strong{man sem} Implies @strong{--semaphore}. -@item @strong{--filter-hosts} (alpha testing) -@anchor{@strong{--filter-hosts} (alpha testing)} +@item @strong{--filter-hosts} (beta testing) +@anchor{@strong{--filter-hosts} (beta testing)} Remove down hosts. For each remote host: check that login through ssh works. If not: do not use this host. @@ -581,8 +581,8 @@ This option is a synonym for @strong{-I}@emph{replace-str} if @emph{replace-str} specified, and for @strong{-I}@{@} otherwise. This option is deprecated; use @strong{-I} instead. -@item @strong{--joblog} @emph{logfile} (alpha testing) -@anchor{@strong{--joblog} @emph{logfile} (alpha testing)} +@item @strong{--joblog} @emph{logfile} (beta testing) +@anchor{@strong{--joblog} @emph{logfile} (beta testing)} Logfile for executed jobs. Save a list of the executed jobs to @emph{logfile} in the following TAB separated format: sequence number, @@ -696,8 +696,8 @@ to see the difference: parallel -j4 -k sleep {}\; echo {} ::: 2 1 4 3 @end verbatim -@item @strong{-L} @emph{max-lines} (alpha testing) -@anchor{@strong{-L} @emph{max-lines} (alpha testing)} +@item @strong{-L} @emph{max-lines} (beta testing) +@anchor{@strong{-L} @emph{max-lines} (beta testing)} When used with @strong{--pipe}: Read records of @emph{max-lines}. @@ -727,8 +727,8 @@ standard specifies @strong{-L} instead. Implies @strong{-X} unless @strong{-m}, @strong{--xargs}, or @strong{--pipe} is set. -@item @strong{--load} @emph{max-load} (alpha testing) -@anchor{@strong{--load} @emph{max-load} (alpha testing)} +@item @strong{--load} @emph{max-load} (beta testing) +@anchor{@strong{--load} @emph{max-load} (beta testing)} Do not start new jobs on a given computer unless the load is less than @emph{max-load}. @emph{max-load} uses the same syntax as @strong{--jobs}, so @emph{100%} @@ -825,11 +825,11 @@ all the output from one server will be grouped together. Instead of printing the output to stdout (standard output) the output of each job is saved in a file and the filename is then printed. -@item @strong{--pipe} (alpha testing) -@anchor{@strong{--pipe} (alpha testing)} +@item @strong{--pipe} (beta testing) +@anchor{@strong{--pipe} (beta testing)} -@item @strong{--spreadstdin} (alpha testing) -@anchor{@strong{--spreadstdin} (alpha testing)} +@item @strong{--spreadstdin} (beta testing) +@anchor{@strong{--spreadstdin} (beta testing)} Spread input to jobs on stdin (standard input). Read a block of data from stdin (standard input) and give one block of data as input to one @@ -1311,15 +1311,15 @@ into temporary files in /tmp. By setting @strong{--tmpdir} you can use a different dir for the files. Setting @strong{--tmpdir} is equivalent to setting $TMPDIR. -@item @strong{--timeout} @emph{sec} (alpha testing) -@anchor{@strong{--timeout} @emph{sec} (alpha testing)} +@item @strong{--timeout} @emph{sec} (beta testing) +@anchor{@strong{--timeout} @emph{sec} (beta testing)} Time out for command. If the command runs for longer than @emph{sec} seconds it will get killed with SIGTERM, followed by SIGTERM 200 ms later, followed by SIGKILL 200 ms later. -@item @strong{--tollef} (alpha testing) -@anchor{@strong{--tollef} (alpha testing)} +@item @strong{--tollef} (beta testing) +@anchor{@strong{--tollef} (beta testing)} Make GNU @strong{parallel} behave more like Tollef's parallel command. It activates @strong{-u}, @strong{-q}, and @strong{--arg-sep --}. It also causes @strong{-l} to @@ -1540,11 +1540,11 @@ Compare these two: See also @strong{--header}. -@item @strong{--shebang} (alpha testing) -@anchor{@strong{--shebang} (alpha testing)} +@item @strong{--shebang} (beta testing) +@anchor{@strong{--shebang} (beta testing)} -@item @strong{--hashbang} (alpha testing) -@anchor{@strong{--hashbang} (alpha testing)} +@item @strong{--hashbang} (beta testing) +@anchor{@strong{--hashbang} (beta testing)} GNU @strong{Parallel} can be called as a shebang (#!) command as the first line of a script. Like this: @@ -3493,6 +3493,9 @@ combination of @strong{seq}, @strong{cat}, @strong{echo}, and @strong{sleep} can most errors. If your example requires large files, see if you can make them by something like @strong{seq 1000000} > @strong{file}. +@item The output of your example. If your problem is not easily reproduced +by others, the output might help them figure out the problem. + @end itemize If you suspect the error is dependent on your distribution, please see @@ -3502,6 +3505,11 @@ http://sourceforge.net/projects/virtualboximage/files/ Specifying the name of your distribution is not enough as you may have installed software that is not in the VirtualBox images. +If you cannot reproduce the error on any of the VirtualBox images +above, you should assume the debugging will be done through you. That +will put more burden on you and it is extra important you give any +information that help. + @chapter AUTHOR @anchor{AUTHOR} diff --git a/src/sql b/src/sql index 5e5e65f0..9c737f68 100755 --- a/src/sql +++ b/src/sql @@ -556,7 +556,7 @@ $Global::Initfile && unlink $Global::Initfile; exit ($err); sub parse_options { - $Global::version = 20120622; + $Global::version = 20120722; $Global::progname = 'sql'; # This must be done first as this may exec myself diff --git a/testsuite/wanted-results/parallel-local20 b/testsuite/wanted-results/parallel-local20 index ab50dcac..c3a48fd9 100644 --- a/testsuite/wanted-results/parallel-local20 +++ b/testsuite/wanted-results/parallel-local20 @@ -12,3 +12,6 @@ B 2 C 3 ### bug #36595: silent loss of input with --pipe and --sshlogin 1 10000 48894 +bug #36707: --controlmaster eats jobs +OK1 +OK2 diff --git a/testsuite/wanted-results/test30 b/testsuite/wanted-results/test30 index fb7b1a2c..59fc6f25 100644 --- a/testsuite/wanted-results/test30 +++ b/testsuite/wanted-results/test30 @@ -22,7 +22,6 @@ Computer:jobs running/jobs completed/%of started jobs/Average seconds to complet local:1/0/100%/0.0s ### --timeout on remote machines slept 1 -slept 1 jobs failed: 2 ### --pipe without command parallel: --pipe must have a command to pipe into (e.g. 'cat')