diff --git a/NEWS b/NEWS index 17f40fd7..cfd71102 100644 --- a/NEWS +++ b/NEWS @@ -1,3 +1,68 @@ +20161222 + +* --results foo.csv will save output as a CSV-file. Can be directly + imported into R or LibreCalc. + +* When using --pipepart a negative --block-size is not interpreted as + a block size but as the number of blocks each jobslot should have. + +* --sqlmaster/--sqlandworker will append jobs to the DBURL if the + DBURL is prepended with +. + +* GNU Parallel was cited in: A cloud-based workflow to quantify + transcript-expression levels in public cancer compendia + http://www.nature.com/articles/srep39259 + +* GNU Parallel was cited in: Learning string distance with smoothing + for OCR spelling correction + http://paperity.org/p/78557440/learning-string-distance-with-smoothing-for-ocr-spelling-correction + +* GNU Parallel was cited in: Transient compute clustering with GNU + Parallel and sshfs + https://gist.github.com/Brainiarc7/24c966c8a001061ee86cc4bc05826bf4 + +* GNU Parallel was cited in: Determination of crystal structures of + proteins of unknown identity using a marathon molecular replacement + procedure: Structure of Stenotrophomonas maltophilia + phosphate-binding protein + https://www.researchgate.net/publication/308186413_Determination_of_crystal_structures_of_proteins_of_unknown_identity_using_a_marathon_molecular_replacement_procedure_Structure_of_Stenotrophomonas_maltophilia_phosphate-binding_protein + +* GNU Parallel was cited in: The Outer Solar System Origins Survey: + I. Design and First-Quarter Discoveries + https://arxiv.org/pdf/1511.02895v2.pdf + +* GNU Parallel was cited in: Large-scale benchmarking reveals false + discoveries and count transformation sensitivity in 16S rRNA gene + amplicon data analysis methods used in microbiome studies + http://microbiomejournal.biomedcentral.com/articles/10.1186/s40168-016-0208-8 + +* GNU Parallel was cited in: Decomposing Images into Layers via + RGB-space Geometry + https://cs.gmu.edu/~ygingold/singleimage/Decomposing%20Images%20into%20Layers%20via%20RGB-space%20Geometry%20(Tan%20et%20al%202016%20TOG)%20small.pdf + +* 4 Ways to Batch Convert Your PNG to JPG and Vice-Versa + http://www.tecmint.com/linux-image-conversion-tools/ + +* All's Fair in Love and Distributed Storage + http://cohesity.com/blog/alls-fair-love-distributed-storage/ + +* How can I use GNU Parallel to run a lot of commands in parallel + https://www.msi.umn.edu/support/faq/how-can-i-use-gnu-parallel-run-lot-commands-parallel + +* 정해영의 블로그 - JEONG Haeyoung's blog + http://blog.genoglobe.com/2016/11/gnu-parallel.html + +* 在Linux下将PNG和JPG批量互转的四种方法 + http://os.51cto.com/art/201612/524182.htm + +* Running in parallel http://tomkimpson.com/howto/gnuparallel/ + +* Taco Bell Parallel Programming + https://giorgos.sealabs.net/taco-bell-parallel-programming.html + +* Bug fixes and man page updates. + + 20161122 * --record-env can now be used with env_parallel for bash, ksh, pdksh, @@ -16,6 +81,7 @@ * Tutorial: Parallel web scraping with CasperJS and GNU Parallel http://g-liu.com/blog/2016/10/tutorial-parallel-web-scraping-with-casperjs-and-gnu-parallel/ + * Bug fixes and man page updates. diff --git a/README b/README index 8132c8e6..8ec329a2 100644 --- a/README +++ b/README @@ -44,9 +44,9 @@ document. Full installation of GNU Parallel is as simple as: - wget http://ftpmirror.gnu.org/parallel/parallel-20161122.tar.bz2 - bzip2 -dc parallel-20161122.tar.bz2 | tar xvf - - cd parallel-20161122 + wget http://ftpmirror.gnu.org/parallel/parallel-20161222.tar.bz2 + bzip2 -dc parallel-20161222.tar.bz2 | tar xvf - + cd parallel-20161222 ./configure && make && sudo make install @@ -55,9 +55,9 @@ Full installation of GNU Parallel is as simple as: If you are not root you can add ~/bin to your path and install in ~/bin and ~/share: - wget http://ftpmirror.gnu.org/parallel/parallel-20161122.tar.bz2 - bzip2 -dc parallel-20161122.tar.bz2 | tar xvf - - cd parallel-20161122 + wget http://ftpmirror.gnu.org/parallel/parallel-20161222.tar.bz2 + bzip2 -dc parallel-20161222.tar.bz2 | tar xvf - + cd parallel-20161222 ./configure --prefix=$HOME && make && make install Or if your system lacks 'make' you can simply copy src/parallel diff --git a/configure b/configure index 476e8e8e..e505ef59 100755 --- a/configure +++ b/configure @@ -1,6 +1,6 @@ #! /bin/sh # Guess values for system-dependent variables and create Makefiles. -# Generated by GNU Autoconf 2.69 for parallel 20161122. +# Generated by GNU Autoconf 2.69 for parallel 20161222. # # Report bugs to . # @@ -579,8 +579,8 @@ MAKEFLAGS= # Identity of this package. PACKAGE_NAME='parallel' PACKAGE_TARNAME='parallel' -PACKAGE_VERSION='20161122' -PACKAGE_STRING='parallel 20161122' +PACKAGE_VERSION='20161222' +PACKAGE_STRING='parallel 20161222' PACKAGE_BUGREPORT='bug-parallel@gnu.org' PACKAGE_URL='' @@ -1203,7 +1203,7 @@ if test "$ac_init_help" = "long"; then # Omit some internal or obsolete options to make the list less imposing. # This message is too long to be a string in the A/UX 3.1 sh. cat <<_ACEOF -\`configure' configures parallel 20161122 to adapt to many kinds of systems. +\`configure' configures parallel 20161222 to adapt to many kinds of systems. Usage: $0 [OPTION]... [VAR=VALUE]... @@ -1269,7 +1269,7 @@ fi if test -n "$ac_init_help"; then case $ac_init_help in - short | recursive ) echo "Configuration of parallel 20161122:";; + short | recursive ) echo "Configuration of parallel 20161222:";; esac cat <<\_ACEOF @@ -1345,7 +1345,7 @@ fi test -n "$ac_init_help" && exit $ac_status if $ac_init_version; then cat <<\_ACEOF -parallel configure 20161122 +parallel configure 20161222 generated by GNU Autoconf 2.69 Copyright (C) 2012 Free Software Foundation, Inc. @@ -1362,7 +1362,7 @@ cat >config.log <<_ACEOF This file contains any messages produced by compilers while running configure, to aid debugging if configure makes a mistake. -It was created by parallel $as_me 20161122, which was +It was created by parallel $as_me 20161222, which was generated by GNU Autoconf 2.69. Invocation command line was $ $0 $@ @@ -2225,7 +2225,7 @@ fi # Define the identity of the package. PACKAGE='parallel' - VERSION='20161122' + VERSION='20161222' cat >>confdefs.h <<_ACEOF @@ -2867,7 +2867,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 # report actual input values of CONFIG_FILES etc. instead of their # values after options handling. ac_log=" -This file was extended by parallel $as_me 20161122, which was +This file was extended by parallel $as_me 20161222, which was generated by GNU Autoconf 2.69. Invocation command line was CONFIG_FILES = $CONFIG_FILES @@ -2929,7 +2929,7 @@ _ACEOF cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`" ac_cs_version="\\ -parallel config.status 20161122 +parallel config.status 20161222 configured by $0, generated by GNU Autoconf 2.69, with options \\"\$ac_cs_config\\" diff --git a/configure.ac b/configure.ac index 42994e4f..02d4937b 100644 --- a/configure.ac +++ b/configure.ac @@ -1,4 +1,4 @@ -AC_INIT([parallel], [20161122], [bug-parallel@gnu.org]) +AC_INIT([parallel], [20161222], [bug-parallel@gnu.org]) AM_INIT_AUTOMAKE([-Wall -Werror foreign]) AC_CONFIG_HEADERS([config.h]) AC_CONFIG_FILES([ diff --git a/install-sh b/install-sh index 59990a10..377bb868 100755 --- a/install-sh +++ b/install-sh @@ -1,7 +1,7 @@ #!/bin/sh # install - install a program, script, or datafile -scriptversion=2014-09-12.12; # UTC +scriptversion=2011-11-20.07; # UTC # This originates from X11R5 (mit/util/scripts/install.sh), which was # later released in X11R6 (xc/config/util/install.sh) with the @@ -41,15 +41,19 @@ scriptversion=2014-09-12.12; # UTC # This script is compatible with the BSD install script, but was written # from scratch. -tab=' ' nl=' ' -IFS=" $tab$nl" +IFS=" "" $nl" -# Set DOITPROG to "echo" to test this script. +# set DOITPROG to echo to test this script +# Don't use :- since 4.3BSD and earlier shells don't like it. doit=${DOITPROG-} -doit_exec=${doit:-exec} +if test -z "$doit"; then + doit_exec=exec +else + doit_exec=$doit +fi # Put in absolute file names if you don't have them in your path; # or use environment vars. @@ -64,6 +68,17 @@ mvprog=${MVPROG-mv} rmprog=${RMPROG-rm} stripprog=${STRIPPROG-strip} +posix_glob='?' +initialize_posix_glob=' + test "$posix_glob" != "?" || { + if (set -f) 2>/dev/null; then + posix_glob= + else + posix_glob=: + fi + } +' + posix_mkdir= # Desired mode of installed file. @@ -82,7 +97,7 @@ dir_arg= dst_arg= copy_on_change=false -is_target_a_directory=possibly +no_target_directory= usage="\ Usage: $0 [OPTION]... [-T] SRCFILE DSTFILE @@ -122,57 +137,46 @@ while test $# -ne 0; do -d) dir_arg=true;; -g) chgrpcmd="$chgrpprog $2" - shift;; + shift;; --help) echo "$usage"; exit $?;; -m) mode=$2 - case $mode in - *' '* | *"$tab"* | *"$nl"* | *'*'* | *'?'* | *'['*) - echo "$0: invalid mode: $mode" >&2 - exit 1;; - esac - shift;; + case $mode in + *' '* | *' '* | *' +'* | *'*'* | *'?'* | *'['*) + echo "$0: invalid mode: $mode" >&2 + exit 1;; + esac + shift;; -o) chowncmd="$chownprog $2" - shift;; + shift;; -s) stripcmd=$stripprog;; - -t) - is_target_a_directory=always - dst_arg=$2 - # Protect names problematic for 'test' and other utilities. - case $dst_arg in - -* | [=\(\)!]) dst_arg=./$dst_arg;; - esac - shift;; + -t) dst_arg=$2 + # Protect names problematic for 'test' and other utilities. + case $dst_arg in + -* | [=\(\)!]) dst_arg=./$dst_arg;; + esac + shift;; - -T) is_target_a_directory=never;; + -T) no_target_directory=true;; --version) echo "$0 $scriptversion"; exit $?;; - --) shift - break;; + --) shift + break;; - -*) echo "$0: invalid option: $1" >&2 - exit 1;; + -*) echo "$0: invalid option: $1" >&2 + exit 1;; *) break;; esac shift done -# We allow the use of options -d and -T together, by making -d -# take the precedence; this is for compatibility with GNU install. - -if test -n "$dir_arg"; then - if test -n "$dst_arg"; then - echo "$0: target directory not allowed when installing a directory." >&2 - exit 1 - fi -fi - if test $# -ne 0 && test -z "$dir_arg$dst_arg"; then # When -d is used, all remaining arguments are directories to create. # When -t is used, the destination is already specified. @@ -203,15 +207,6 @@ if test $# -eq 0; then exit 0 fi -if test -z "$dir_arg"; then - if test $# -gt 1 || test "$is_target_a_directory" = always; then - if test ! -d "$dst_arg"; then - echo "$0: $dst_arg: Is not a directory." >&2 - exit 1 - fi - fi -fi - if test -z "$dir_arg"; then do_exit='(exit $ret); exit $ret' trap "ret=129; $do_exit" 1 @@ -228,16 +223,16 @@ if test -z "$dir_arg"; then *[0-7]) if test -z "$stripcmd"; then - u_plus_rw= + u_plus_rw= else - u_plus_rw='% 200' + u_plus_rw='% 200' fi cp_umask=`expr '(' 777 - $mode % 1000 ')' $u_plus_rw`;; *) if test -z "$stripcmd"; then - u_plus_rw= + u_plus_rw= else - u_plus_rw=,u+rw + u_plus_rw=,u+rw fi cp_umask=$mode$u_plus_rw;; esac @@ -274,15 +269,41 @@ do # If destination is a directory, append the input filename; won't work # if double slashes aren't ignored. if test -d "$dst"; then - if test "$is_target_a_directory" = never; then - echo "$0: $dst_arg: Is a directory" >&2 - exit 1 + if test -n "$no_target_directory"; then + echo "$0: $dst_arg: Is a directory" >&2 + exit 1 fi dstdir=$dst dst=$dstdir/`basename "$src"` dstdir_status=0 else - dstdir=`dirname "$dst"` + # Prefer dirname, but fall back on a substitute if dirname fails. + dstdir=` + (dirname "$dst") 2>/dev/null || + expr X"$dst" : 'X\(.*[^/]\)//*[^/][^/]*/*$' \| \ + X"$dst" : 'X\(//\)[^/]' \| \ + X"$dst" : 'X\(//\)$' \| \ + X"$dst" : 'X\(/\)' \| . 2>/dev/null || + echo X"$dst" | + sed '/^X\(.*[^/]\)\/\/*[^/][^/]*\/*$/{ + s//\1/ + q + } + /^X\(\/\/\)[^/].*/{ + s//\1/ + q + } + /^X\(\/\/\)$/{ + s//\1/ + q + } + /^X\(\/\).*/{ + s//\1/ + q + } + s/.*/./; q' + ` + test -d "$dstdir" dstdir_status=$? fi @@ -293,81 +314,74 @@ do if test $dstdir_status != 0; then case $posix_mkdir in '') - # Create intermediate dirs using mode 755 as modified by the umask. - # This is like FreeBSD 'install' as of 1997-10-28. - umask=`umask` - case $stripcmd.$umask in - # Optimize common cases. - *[2367][2367]) mkdir_umask=$umask;; - .*0[02][02] | .[02][02] | .[02]) mkdir_umask=22;; + # Create intermediate dirs using mode 755 as modified by the umask. + # This is like FreeBSD 'install' as of 1997-10-28. + umask=`umask` + case $stripcmd.$umask in + # Optimize common cases. + *[2367][2367]) mkdir_umask=$umask;; + .*0[02][02] | .[02][02] | .[02]) mkdir_umask=22;; - *[0-7]) - mkdir_umask=`expr $umask + 22 \ - - $umask % 100 % 40 + $umask % 20 \ - - $umask % 10 % 4 + $umask % 2 - `;; - *) mkdir_umask=$umask,go-w;; - esac + *[0-7]) + mkdir_umask=`expr $umask + 22 \ + - $umask % 100 % 40 + $umask % 20 \ + - $umask % 10 % 4 + $umask % 2 + `;; + *) mkdir_umask=$umask,go-w;; + esac - # With -d, create the new directory with the user-specified mode. - # Otherwise, rely on $mkdir_umask. - if test -n "$dir_arg"; then - mkdir_mode=-m$mode - else - mkdir_mode= - fi + # With -d, create the new directory with the user-specified mode. + # Otherwise, rely on $mkdir_umask. + if test -n "$dir_arg"; then + mkdir_mode=-m$mode + else + mkdir_mode= + fi - posix_mkdir=false - case $umask in - *[123567][0-7][0-7]) - # POSIX mkdir -p sets u+wx bits regardless of umask, which - # is incompatible with FreeBSD 'install' when (umask & 300) != 0. - ;; - *) - # $RANDOM is not portable (e.g. dash); use it when possible to - # lower collision chance - tmpdir=${TMPDIR-/tmp}/ins$RANDOM-$$ - trap 'ret=$?; rmdir "$tmpdir/a/b" "$tmpdir/a" "$tmpdir" 2>/dev/null; exit $ret' 0 + posix_mkdir=false + case $umask in + *[123567][0-7][0-7]) + # POSIX mkdir -p sets u+wx bits regardless of umask, which + # is incompatible with FreeBSD 'install' when (umask & 300) != 0. + ;; + *) + tmpdir=${TMPDIR-/tmp}/ins$RANDOM-$$ + trap 'ret=$?; rmdir "$tmpdir/d" "$tmpdir" 2>/dev/null; exit $ret' 0 - # As "mkdir -p" follows symlinks and we work in /tmp possibly; so - # create the $tmpdir first (and fail if unsuccessful) to make sure - # that nobody tries to guess the $tmpdir name. - if (umask $mkdir_umask && - $mkdirprog $mkdir_mode "$tmpdir" && - exec $mkdirprog $mkdir_mode -p -- "$tmpdir/a/b") >/dev/null 2>&1 - then - if test -z "$dir_arg" || { - # Check for POSIX incompatibilities with -m. - # HP-UX 11.23 and IRIX 6.5 mkdir -m -p sets group- or - # other-writable bit of parent directory when it shouldn't. - # FreeBSD 6.1 mkdir -m -p sets mode of existing directory. - test_tmpdir="$tmpdir/a" - ls_ld_tmpdir=`ls -ld "$test_tmpdir"` - case $ls_ld_tmpdir in - d????-?r-*) different_mode=700;; - d????-?--*) different_mode=755;; - *) false;; - esac && - $mkdirprog -m$different_mode -p -- "$test_tmpdir" && { - ls_ld_tmpdir_1=`ls -ld "$test_tmpdir"` - test "$ls_ld_tmpdir" = "$ls_ld_tmpdir_1" - } - } - then posix_mkdir=: - fi - rmdir "$tmpdir/a/b" "$tmpdir/a" "$tmpdir" - else - # Remove any dirs left behind by ancient mkdir implementations. - rmdir ./$mkdir_mode ./-p ./-- "$tmpdir" 2>/dev/null - fi - trap '' 0;; - esac;; + if (umask $mkdir_umask && + exec $mkdirprog $mkdir_mode -p -- "$tmpdir/d") >/dev/null 2>&1 + then + if test -z "$dir_arg" || { + # Check for POSIX incompatibilities with -m. + # HP-UX 11.23 and IRIX 6.5 mkdir -m -p sets group- or + # other-writable bit of parent directory when it shouldn't. + # FreeBSD 6.1 mkdir -m -p sets mode of existing directory. + ls_ld_tmpdir=`ls -ld "$tmpdir"` + case $ls_ld_tmpdir in + d????-?r-*) different_mode=700;; + d????-?--*) different_mode=755;; + *) false;; + esac && + $mkdirprog -m$different_mode -p -- "$tmpdir" && { + ls_ld_tmpdir_1=`ls -ld "$tmpdir"` + test "$ls_ld_tmpdir" = "$ls_ld_tmpdir_1" + } + } + then posix_mkdir=: + fi + rmdir "$tmpdir/d" "$tmpdir" + else + # Remove any dirs left behind by ancient mkdir implementations. + rmdir ./$mkdir_mode ./-p ./-- 2>/dev/null + fi + trap '' 0;; + esac;; esac if $posix_mkdir && ( - umask $mkdir_umask && - $doit_exec $mkdirprog $mkdir_mode -p -- "$dstdir" + umask $mkdir_umask && + $doit_exec $mkdirprog $mkdir_mode -p -- "$dstdir" ) then : else @@ -377,51 +391,53 @@ do # directory the slow way, step by step, checking for races as we go. case $dstdir in - /*) prefix='/';; - [-=\(\)!]*) prefix='./';; - *) prefix='';; + /*) prefix='/';; + [-=\(\)!]*) prefix='./';; + *) prefix='';; esac + eval "$initialize_posix_glob" + oIFS=$IFS IFS=/ - set -f + $posix_glob set -f set fnord $dstdir shift - set +f + $posix_glob set +f IFS=$oIFS prefixes= for d do - test X"$d" = X && continue + test X"$d" = X && continue - prefix=$prefix$d - if test -d "$prefix"; then - prefixes= - else - if $posix_mkdir; then - (umask=$mkdir_umask && - $doit_exec $mkdirprog $mkdir_mode -p -- "$dstdir") && break - # Don't fail if two instances are running concurrently. - test -d "$prefix" || exit 1 - else - case $prefix in - *\'*) qprefix=`echo "$prefix" | sed "s/'/'\\\\\\\\''/g"`;; - *) qprefix=$prefix;; - esac - prefixes="$prefixes '$qprefix'" - fi - fi - prefix=$prefix/ + prefix=$prefix$d + if test -d "$prefix"; then + prefixes= + else + if $posix_mkdir; then + (umask=$mkdir_umask && + $doit_exec $mkdirprog $mkdir_mode -p -- "$dstdir") && break + # Don't fail if two instances are running concurrently. + test -d "$prefix" || exit 1 + else + case $prefix in + *\'*) qprefix=`echo "$prefix" | sed "s/'/'\\\\\\\\''/g"`;; + *) qprefix=$prefix;; + esac + prefixes="$prefixes '$qprefix'" + fi + fi + prefix=$prefix/ done if test -n "$prefixes"; then - # Don't fail if two instances are running concurrently. - (umask $mkdir_umask && - eval "\$doit_exec \$mkdirprog $prefixes") || - test -d "$dstdir" || exit 1 - obsolete_mkdir_used=true + # Don't fail if two instances are running concurrently. + (umask $mkdir_umask && + eval "\$doit_exec \$mkdirprog $prefixes") || + test -d "$dstdir" || exit 1 + obsolete_mkdir_used=true fi fi fi @@ -456,12 +472,15 @@ do # If -C, don't bother to copy if it wouldn't change the file. if $copy_on_change && - old=`LC_ALL=C ls -dlL "$dst" 2>/dev/null` && - new=`LC_ALL=C ls -dlL "$dsttmp" 2>/dev/null` && - set -f && + old=`LC_ALL=C ls -dlL "$dst" 2>/dev/null` && + new=`LC_ALL=C ls -dlL "$dsttmp" 2>/dev/null` && + + eval "$initialize_posix_glob" && + $posix_glob set -f && set X $old && old=:$2:$4:$5:$6 && set X $new && new=:$2:$4:$5:$6 && - set +f && + $posix_glob set +f && + test "$old" = "$new" && $cmpprog "$dst" "$dsttmp" >/dev/null 2>&1 then @@ -474,24 +493,24 @@ do # to itself, or perhaps because mv is so ancient that it does not # support -f. { - # Now remove or move aside any old file at destination location. - # We try this two ways since rm can't unlink itself on some - # systems and the destination file might be busy for other - # reasons. In this case, the final cleanup might fail but the new - # file should still install successfully. - { - test ! -f "$dst" || - $doit $rmcmd -f "$dst" 2>/dev/null || - { $doit $mvcmd -f "$dst" "$rmtmp" 2>/dev/null && - { $doit $rmcmd -f "$rmtmp" 2>/dev/null; :; } - } || - { echo "$0: cannot unlink or rename $dst" >&2 - (exit 1); exit 1 - } - } && + # Now remove or move aside any old file at destination location. + # We try this two ways since rm can't unlink itself on some + # systems and the destination file might be busy for other + # reasons. In this case, the final cleanup might fail but the new + # file should still install successfully. + { + test ! -f "$dst" || + $doit $rmcmd -f "$dst" 2>/dev/null || + { $doit $mvcmd -f "$dst" "$rmtmp" 2>/dev/null && + { $doit $rmcmd -f "$rmtmp" 2>/dev/null; :; } + } || + { echo "$0: cannot unlink or rename $dst" >&2 + (exit 1); exit 1 + } + } && - # Now rename the file to the real destination. - $doit $mvcmd "$dsttmp" "$dst" + # Now rename the file to the real destination. + $doit $mvcmd "$dsttmp" "$dst" } fi || exit 1 diff --git a/missing b/missing index f62bbae3..db98974f 100755 --- a/missing +++ b/missing @@ -3,7 +3,7 @@ scriptversion=2013-10-28.13; # UTC -# Copyright (C) 1996-2014 Free Software Foundation, Inc. +# Copyright (C) 1996-2013 Free Software Foundation, Inc. # Originally written by Fran,cois Pinard , 1996. # This program is free software; you can redistribute it and/or modify diff --git a/packager/obs/home:tange/parallel/parallel.spec b/packager/obs/home:tange/parallel/parallel.spec index fff4ecc3..86b680ac 100644 --- a/packager/obs/home:tange/parallel/parallel.spec +++ b/packager/obs/home:tange/parallel/parallel.spec @@ -1,6 +1,6 @@ Summary: Shell tool for executing jobs in parallel Name: parallel -Version: 20161122 +Version: 20161222 Release: 1.1 License: GPL Group: Productivity/File utilities diff --git a/src/niceload b/src/niceload index 1afce95a..8726141e 100755 --- a/src/niceload +++ b/src/niceload @@ -24,7 +24,7 @@ use strict; use Getopt::Long; $Global::progname="niceload"; -$Global::version = 20161123; +$Global::version = 20161222; Getopt::Long::Configure("bundling","require_order"); get_options_from_array(\@ARGV) || die_usage(); if($opt::version) { diff --git a/src/parallel b/src/parallel index 963bd56a..ac8e39bf 100755 --- a/src/parallel +++ b/src/parallel @@ -227,8 +227,10 @@ if($opt::halt and $Global::halt_when ne "never") { wait_and_exit(min(undef_as_zero($Global::exitstatus),101)); } + sub __PIPE_MODE__ {} + sub pipe_part_files { # Input: # $file = the file to read @@ -769,8 +771,10 @@ sub write_record_to_pipe { return 1; } + sub __SEM_MODE__ {} + sub acquire_semaphore { # Acquires semaphore. If needed: spawns to the background # Uses: @@ -793,8 +797,10 @@ sub acquire_semaphore { return $sem; } + sub __PARSE_OPTIONS__ {} + sub options_hash { # Returns: # %hash = the GetOptions config @@ -1078,10 +1084,13 @@ sub parse_options { } if(defined $opt::results) { + # Is the output a dir or CSV-file? if($opt::results =~ /\.csv$/i) { + # CSV with , as separator $Global::csv = ","; $Global::membuffer ||= 1; } elsif($opt::results =~ /\.tsv$/i) { + # CSV with TAB as separator $Global::csv = "\t"; $Global::membuffer ||= 1; } @@ -1250,7 +1259,7 @@ sub check_invalid_option_combinations { sub init_globals { # Defaults: - $Global::version = 20161208; + $Global::version = 20161222; $Global::progname = 'parallel'; $Global::infinity = 2**31; $Global::debug = 0; @@ -1636,7 +1645,7 @@ sub find_compression_program { # parallel --shuf -j1 --joblog jl-m --arg-sep , parallel --compress-program \'{3}" "-{2}\' cat ::: 1gb '>'/dev/null , 1 2 3 , {1..3} , $multithread # sort -nk4 jl-? # 1-core: - # 2-cores: + # 2-cores: lz4 lzop zstd gzip bzip2 lzma xz clzip pigz lbzip2 pbzip2 lzip lrz plzip pxz pzstd # 4-cores: # 8-cores: pzstd lz4 zstd pigz lzop lbzip2 pbzip2 gzip lzip lrz plzip pxz bzip2 lzma xz clzip # 16-cores: pzstd lz4 pigz lzop lbzip2 pbzip2 plzip lzip lrz pxz gzip lzma xz bzip2 @@ -1653,7 +1662,6 @@ sub find_compression_program { return ("cat","cat"); } - sub read_options { # Read options from command line, profile and $PARALLEL # Uses: @@ -1889,8 +1897,10 @@ sub cleanup { } } + sub __QUOTING_ARGUMENTS_FOR_SHELL__ {} + sub shell_quote { # Input: # @strings = strings to be quoted @@ -2038,6 +2048,7 @@ sub unquote_printf { return $_; } + sub __FILEHANDLES__ {} @@ -2154,6 +2165,7 @@ sub set_fh_non_blocking { sub __RUNNING_THE_JOBS_AND_PRINTING_PROGRESS__ {} + # Variable structure: # # $Global::running{$pid} = Pointer to Job-object @@ -2870,8 +2882,10 @@ sub get_job_with_sshlogin { return $job; } + sub __REMOTE_SSH__ {} + sub read_sshloginfiles { # Read a list of --slf's # Input: @@ -3106,7 +3120,7 @@ sub cleanup_basefile { sub run_parallel { my ($stdin,@args) = @_; my $cmd = join "",map { " $_ & " } split /\n/, $stdin; -print $Global::original_stderr ` $cmd wait` ; + print $Global::original_stderr ` $cmd wait` ; return 0 } @@ -3507,8 +3521,10 @@ sub onall { } } + sub __SIGNAL_HANDLING__ {} + sub tstp { # Send TSTP signal (Ctrl-Z) to all children process groups # Uses: @@ -3520,7 +3536,6 @@ sub tstp { kill "TSTP", $$; } - sub save_original_signal_handler { # Remember the original signal handler # Uses: @@ -3662,8 +3677,10 @@ sub reaper { return wantarray ? @pids_reaped : $children_reaped; } + sub __USAGE__ {} + sub killall { # Kill all jobs by killing their process groups @@ -3790,7 +3807,6 @@ sub usage { "",); } - sub citation_notice { # if --will-cite or --plain: do nothing # if stderr redirected: do nothing @@ -3953,8 +3969,10 @@ sub show_limits { "press CTRL-D or CTRL-C\n"); } + sub __GENERIC_COMMON_FUNCTION__ {} + sub mkdir_or_die { # If dir is not executable: die my $dir = shift; @@ -4003,13 +4021,13 @@ sub tmpname { return $tmpname; } -sub tmpfifo { - # Find an unused name and mkfifo on it - use POSIX qw(mkfifo); - my $tmpfifo = tmpname("fif",@_); - mkfifo($tmpfifo,0600); - return $tmpfifo; -} +#sub tmpfifo { +# # Find an unused name and mkfifo on it +# use POSIX qw(mkfifo); +# my $tmpfifo = tmpname("fif",@_); +# mkfifo($tmpfifo,0600); +# return $tmpfifo; +#} sub rm { # Remove file and remove it from %Global::unlink @@ -4484,8 +4502,10 @@ sub kill_youngster_if_not_enough_mem { } } + sub __DEBUGGING__ {} + sub debug { # Uses: # $Global::debug @@ -4578,8 +4598,10 @@ sub my_carp { carp(@_); } + sub __OBJECT_ORIENTED_PARTS__ {} + package SSHLogin; sub new { @@ -7382,7 +7404,9 @@ sub sshlogin_wrap { my @vars = vars_to_export(); my $csh_friendly = not grep { /\n/ } @ENV{@vars}; my @bash_functions = grep { substr($ENV{$_},0,4) eq "() {" } @vars; - my @non_functions = grep { substr($ENV{$_},0,4) ne "() {" } @vars; + my @non_functions = (grep { !/PARALLEL_ENV/ } + grep { substr($ENV{$_},0,4) ne "() {" } @vars); + # eval of @envset will set %ENV my $envset = join"", map { '$ENV{"'.::perl_quote_scalar($_).'"}="'. @@ -8245,7 +8269,7 @@ sub print { # Memory optimization: Overwrite with the joined output $self->{'output'}{1} = join("", @{$self->{'output'}{1}}); $self->{'output'}{2} = join("", @{$self->{'output'}{2}}); - print $Global::csv_fh + print $Global::csv_fh (map { $$_ } combine_ref ($self->seq(), @@ -8296,7 +8320,7 @@ sub combine_ref { $must_be_quoted ||=1; } if($must_be_quoted){ - push \@out, \$sep, \$quot, $column, \$quot; + push @out, \$sep, \$quot, $column, \$quot; } else { push @out, \$sep, $column; } @@ -8306,46 +8330,6 @@ sub combine_ref { return @out; } -sub combine { - # Inspired by Text::CSV_PP::_combine (by Makamaka Hannyaharamitu) - my @part = @_; - my $sep = $Global::csv; - my $quot = '"'; - - my $must_be_quoted; - for my $column (@part) { - # Memory optimization: Content transferred as reference - if(ref $column ne "SCALAR") { - ($column,$$column) = ($$column,""); - } - if(not defined $column) { - $column = ''; - next; - } - - $must_be_quoted = 0; - - if($column =~ s/$quot/$quot$quot/go){ - # Replace " => "" - $must_be_quoted ||=1; - } - if($column =~ /[\s\Q$sep\E]/o){ - # Put quotes around if the column contains , - $must_be_quoted ||=1; - } - - $Global::use{"bytes"} ||= eval "use bytes; 1;"; - if ($column =~ /\0/) { - # Contains \0 => put quotes around - $must_be_quoted ||=1; - } - if($must_be_quoted){ - $column = $quot . $column . $quot; - } - } - return join($sep, @part) . "\n"; -} - sub print_files { # Print the name of the file containing stdout on stdout # Uses: @@ -8702,6 +8686,7 @@ sub set_exitsignal { } } + package CommandLine; sub new { diff --git a/src/parallel.pod b/src/parallel.pod index 6b334cb7..365314c3 100644 --- a/src/parallel.pod +++ b/src/parallel.pod @@ -499,7 +499,7 @@ If you use B<-N>, B<--block-size> should be bigger than N+1 records. I defaults to 1M. -When using B<--pipepart> negative a block size is not interpreted as a +When using B<--pipepart> a negative block size is not interpreted as a blocksize but as the number of blocks each jobslot should have. So this will run 10*5 = 50 jobs in total: @@ -1670,9 +1670,9 @@ B<--return> is ignored when used with B<--sshlogin :> or when not used with B<--sshlogin>. -=item B<--round-robin> +=item B<--round-robin> (alpha testing) -=item B<--round> +=item B<--round> (alpha testing) Normally B<--pipe> will give a single block to each instance of the command. With B<--round-robin> all blocks will at random be written to @@ -3427,7 +3427,7 @@ Note: The default for GNU B is to remove the spaces around the columns parallel -a table_file.tsv --trim n --colsep '\t' cmd -o {2} -i {1} -=head1 EXAMPLE: Output as CSV-file +=head1 EXAMPLE: Output to database GNU B can output to a database table and a CSV-file: @@ -3464,6 +3464,18 @@ Or MySQL: perl -pe 's/"/""/g;s/\t/","/g;s/^/"/;s/$/"/;s/\\\\/\\/g;s/\\t/\t/g;s/\\n/\n/g;' mytable.tsv +=head1 EXAMPLE: Output to CSV-file for R + +If you have no need for the advanced job distribution control that a +database provides, but you simply want output into a CSV file that you +can read into R or LibreCalc, then you can use B<--results>: + + parallel --results my.csv seq ::: 10 20 30 + R + > mydf <- read.csv("my.csv"); + > print(mydf[2,]) + > write(as.character(mydf[2,c("Stdout")]),'') + =head1 EXAMPLE: Run the same command 10 times If you want to run the same command with the same arguments 10 times diff --git a/src/parallel_design.pod b/src/parallel_design.pod index 536b1cef..11798e60 100644 --- a/src/parallel_design.pod +++ b/src/parallel_design.pod @@ -69,12 +69,6 @@ The only way to have access to the environment is directly from the shell, so the program must be written in a shell script that will be sourced and there has to deal with the dialect of the relevant shell. -Dumping the environment into a format that can be activated again is -easy in Bash: B and B basically give you the commands -you must run to re-create the environment. It is much harder in B -where there is no easy way to convert an environment into commands -that generate the environment. - =head3 env_parallel.* @@ -183,10 +177,10 @@ compresses the buffered data. This is a bit tricky because there should be no files to clean up if GNU B is killed by a power outage. -GNU B first selects a compression program. If the user has not -selected one, the first of these that is in $PATH is used: B. They are sorted by -speed on a 16 core machine. +GNU B first selects a compression program. If the user has +not selected one, the first of these that is in $PATH is used: B. They are sorted by speed on a 32 core machine. Schematically the setup is as follows: @@ -449,23 +443,6 @@ that correctly for all corner cases is next to impossible to do by hand. -=head2 --block-size adjustment - -Every time GNU B detects a record bigger than -B<--block-size> it increases the block size by 30%. A small -B<--block-size> gives very poor performance; by exponentially -increasing the block size performance will not suffer. - -GNU B will waste CPU power if B<--block-size> does not -contain a full record, because it tries to find a full record and will -fail to do so. The recommendation is therefore to use a -B<--block-size> > 2 records, so you always get at least one full -record when you read one block. - -If you use B<-N> then B<--block-size> should be big enough to contain -N+1 records. - - =head2 Shell shock The shell shock bug in B did not affect GNU B, but the @@ -736,6 +713,36 @@ data. See this for a surprising example: yes | dd bs=1024k count=10 | wc + +=head2 --block-size adjustment + +Every time GNU B detects a record bigger than +B<--block-size> it increases the block size by 30%. A small +B<--block-size> gives very poor performance; by exponentially +increasing the block size performance will not suffer. + +GNU B will waste CPU power if B<--block-size> does not +contain a full record, because it tries to find a full record and will +fail to do so. The recommendation is therefore to use a +B<--block-size> > 2 records, so you always get at least one full +record when you read one block. + +If you use B<-N> then B<--block-size> should be big enough to contain +N+1 records. + + +=head2 Automatic --block-size computation + +With B<--pipepart> GNU B can compute the B<--block-size> +automatically. A B<--block-size> of B<-1> will use a block size so +that each jobslot will receive approximately 1 block. B<--block -2> +will pass 2 blocks to each jobslot and B<-I> will pass I blocks +to each jobslot. + +This can be done because B<--pipepart> reads from files, and we can +compute the total size of the input. + + =head2 --jobs and --onall When running the same commands on many servers what should B<--jobs> @@ -968,11 +975,12 @@ single string. The DBURL must point to a table name. The table will be dropped and created. The reason for not reusing an exising table is that the user may have added more input sources which would require more columns in -the table. +the table. By prepending '+' to the DBURL the table will not be +dropped. The table columns are similar to joblog with the addition of B -.. B which are values from the input sources, and stdout and -stderr which are the output from standard output and standard error, +.. B which are values from the input sources, and Stdout and +Stderr which are the output from standard output and standard error, respectively. The Signal column has been renamed to _Signal due to Signal being a diff --git a/src/sql b/src/sql index 337b8bed..5b6f8552 100755 --- a/src/sql +++ b/src/sql @@ -566,7 +566,7 @@ $Global::Initfile && unlink $Global::Initfile; exit ($err); sub parse_options { - $Global::version = 20161123; + $Global::version = 20161222; $Global::progname = 'sql'; # This must be done first as this may exec myself