diff --git a/CITATION b/CITATION index 60eb9e87..0d8cc1d4 100644 --- a/CITATION +++ b/CITATION @@ -1,16 +1,17 @@ When using programs that use GNU Parallel to process data for publication please cite: @article{Tange2011a, - title = {GNU Parallel - The Command-Line Power Tool}, - author = {O. Tange}, - address = {Frederiksberg, Denmark}, - journal = {;login: The USENIX Magazine}, - month = {Feb}, - number = {1}, - volume = {36}, - url = {http://www.gnu.org/s/parallel}, - year = {2011}, - pages = {42-47} + title = {GNU Parallel - The Command-Line Power Tool}, + author = {O. Tange}, + address = {Frederiksberg, Denmark}, + journal = {;login: The USENIX Magazine}, + month = {Feb}, + number = {1}, + volume = {36}, + url = {http://www.gnu.org/s/parallel}, + year = {2011}, + pages = {42-47} + doi = {10.5281/zenodo.16303} } (Feel free to use \nocite{Tange2011a}) diff --git a/NEWS b/NEWS index 30356feb..b2dcf0a3 100644 --- a/NEWS +++ b/NEWS @@ -2,45 +2,73 @@ New in this release: -* Security fix. An attacker on the local system could make you overwrite one of your own files with a single byte. The problem exist when you use --compress or --tmux or --pipe or --cat or --fifo. The attacker must figure out the randomly chosen file name and create a symlink within a time window of 15 ms. +* Security fix. An attacker on the local system could make you + overwrite one of your own files with a single byte. The problem + exist when you use --compress or --tmux or --pipe or --cat or + --fifo. The attacker must figure out the randomly chosen file name + and create a symlink within a time window of 15 ms. -* --shuf will generate all jobs, and shuffle them before running them. This is useful to get a quick preview of the results before running the full batch. +* --shuf will generate all jobs, and shuffle them before running + them. This is useful to get a quick preview of the results before + running the full batch. * GNU Parallel now has a DOI: https://dx.doi.org/10.5281/zenodo.16303 -* GNU Parallel was cited in: Scaling Machine Learning for Target Prediction in Drug Discovery using Apache Spark https://cris.cumulus.vub.ac.be/portal/files/5147244/spark.pdf +* GNU Parallel was cited in: Scaling Machine Learning for Target + Prediction in Drug Discovery using Apache Spark + https://cris.cumulus.vub.ac.be/portal/files/5147244/spark.pdf -* GNU Parallel was cited in: Bayesian inference of protein structure from chemical shift data https://peerj.com/articles/861/ +* GNU Parallel was cited in: Bayesian inference of protein structure + from chemical shift data https://peerj.com/articles/861/ -* GNU Parallel was cited in: Toward Enhanced Metadata Quality of Large-Scale Digital Libraries: Estimating Volume Time Range https://www.ideals.illinois.edu/bitstream/handle/2142/73656/186_ready.pdf +* GNU Parallel was cited in: Toward Enhanced Metadata Quality of + Large-Scale Digital Libraries: Estimating Volume Time Range + https://www.ideals.illinois.edu/bitstream/handle/2142/73656/186_ready.pdf -* GNU Parallel was cited in: Contrasting regional architectures of schizophrenia and other complex diseases using fast variance components analysis http://biorxiv.org/content/biorxiv/early/2015/03/13/016527.full.pdf +* GNU Parallel was cited in: Contrasting regional architectures of + schizophrenia and other complex diseases using fast variance + components analysis + http://biorxiv.org/content/biorxiv/early/2015/03/13/016527.full.pdf -* GNU Parallel was cited in: Experimental study on the Wind Farm Substation Cable Installation Problem http://i11www.iti.uni-karlsruhe.de/_media/teaching/theses/ma-schmitz-14.pdf +* GNU Parallel was cited in: Experimental study on the Wind Farm + Substation Cable Installation Problem + http://i11www.iti.uni-karlsruhe.de/_media/teaching/theses/ma-schmitz-14.pdf -* GNU Parallel was used in: landsat-gifworks https://github.com/KAPPS-/landsat-gifworks +* GNU Parallel was used in: landsat-gifworks + https://github.com/KAPPS-/landsat-gifworks * GNU Parallel was used in: https://github.com/exascience/elprep -* GNU Parallel was used in: http://search.cpan.org/~jillrowe/HPC-Runner-GnuParallel-0.01/ +* GNU Parallel was used in: + http://search.cpan.org/~jillrowe/HPC-Runner-GnuParallel-0.01/ -* GNU Parallel was used in: http://search.cpan.org/~ajpage/Bio-Roary-2.0.7/lib/Bio/Roary/JobRunner/Parallel.pm +* GNU Parallel was used in: + http://search.cpan.org/~ajpage/Bio-Roary-2.0.7/lib/Bio/Roary/JobRunner/Parallel.pm -* GNU Parallel was used in: Rapid haploid SNP calling https://github.com/tseemann/snippy +* GNU Parallel was used in: Rapid haploid SNP calling + https://github.com/tseemann/snippy -* Pictures and Metadata http://www.ozzy.no/2015/02/05/pictures-and-metadata/ +* Pictures and Metadata + http://www.ozzy.no/2015/02/05/pictures-and-metadata/ -* Task automation with bash and parallel https://biowize.wordpress.com/2015/03/23/task-automation-with-bash-and-parallel/ +* Task automation with bash and parallel + https://biowize.wordpress.com/2015/03/23/task-automation-with-bash-and-parallel/ -* How To: Speed Up File Transfers in Linux using RSync with GNU Parallel http://www.yourownlinux.com/2015/04/speed-up-file-transfers-using-rsync-with-gnu-parallel.html +* How To: Speed Up File Transfers in Linux using RSync with GNU + Parallel + http://www.yourownlinux.com/2015/04/speed-up-file-transfers-using-rsync-with-gnu-parallel.html -* Mirroring a Git Repository https://avacariu.me/articles/mirroring-a-git-repository.html +* Mirroring a Git Repository + https://avacariu.me/articles/mirroring-a-git-repository.html -* Add an ENVI header to JAXA Global Mangrove Watch PALSAR tiles https://spectraldifferences.wordpress.com/2015/03/24/add-an-envi-header-to-jaxa-global-mangrove-watch-palsar-tiles/ +* Add an ENVI header to JAXA Global Mangrove Watch PALSAR tiles + https://spectraldifferences.wordpress.com/2015/03/24/add-an-envi-header-to-jaxa-global-mangrove-watch-palsar-tiles/ -* GNU Parallel (Sebuah Uji Coba) http://kaka.prakasa.my.id/2014/09/04/gnu-parallel-sebuah-uji-coba/ +* GNU Parallel (Sebuah Uji Coba) + http://kaka.prakasa.my.id/2014/09/04/gnu-parallel-sebuah-uji-coba/ -* 你见过的最牛逼的命令行程序是什么？ http://www.zhihu.com/question/29257300 +* 你见过的最牛逼的命令行程序是什么？ + http://www.zhihu.com/question/29257300 * Bug fixes and man page updates. diff --git a/doc/release_new_version b/doc/release_new_version index 97c0779b..9e38d778 100644 --- a/doc/release_new_version +++ b/doc/release_new_version @@ -58,6 +58,7 @@ pushd == Upload == export YYYYMMDD=`yyyymmdd` +eval `gpg-agent --daemon` make upload # Only needed for alpha: @@ -141,8 +142,7 @@ YYYYMMDD=`yyyymmdd` TAG=MyTag echo "Released as $YYYYMMDD ('$TAG')." -git commit -a - +git commit -a -m "Released as $YYYYMMDD ('$TAG')" git tag -s -m "Released as $YYYYMMDD ('$TAG')" $TAG git tag -s -m "Released as $YYYYMMDD ('$TAG')" $YYYYMMDD @@ -209,66 +209,43 @@ cc:Tim Cuthbertson , Ryoichiro Suzuki , Jesse Alama -Subject: GNU Parallel 20150422 ('Germanwings') released +Subject: GNU Parallel 20150522 ('Nepal') released -GNU Parallel 20150422 ('Germanwings') has been released. It is available for download at: http://ftp.gnu.org/gnu/parallel/ +GNU Parallel 20150522 ('Nepal') has been released. It is available for download at: http://ftp.gnu.org/gnu/parallel/ Haiku of the month: - SSH set up? - Instant cluster needed now? + Many jobs to run. + Idle cores is not your thing. Use GNU Parallel. - -- Ole Tange + -- Ole Tange New in this release: -* Security fix. An attacker on the local system could make you overwrite one of your own files with a single byte. The problem exist when you use --compress or --tmux or --pipe or --cat or --fifo. The attacker must figure out the randomly chosen file name and create a symlink within a time window of 15 ms. - -* GNU Parallel now has a DOI: https://dx.doi.org/10.5281/zenodo.16303 - -* GNU Parallel was cited in: Scaling Machine Learning for Target Prediction in Drug Discovery using Apache Spark https://cris.cumulus.vub.ac.be/portal/files/5147244/spark.pdf - -* GNU Parallel was cited in: Bayesian inference of protein structure from chemical shift data https://peerj.com/articles/861/ - -* GNU Parallel was cited in: Toward Enhanced Metadata Quality of Large-Scale Digital Libraries: Estimating Volume Time Range https://www.ideals.illinois.edu/bitstream/handle/2142/73656/186_ready.pdf - -* GNU Parallel was cited in: Contrasting regional architectures of schizophrenia and other complex diseases using fast variance components analysis http://biorxiv.org/content/biorxiv/early/2015/03/13/016527.full.pdf - -* GNU Parallel was cited in: Experimental study on the Wind Farm Substation Cable Installation Problem http://i11www.iti.uni-karlsruhe.de/_media/teaching/theses/ma-schmitz-14.pdf - * <> Comparing the CarbonTracker and TM5-4DVar data assimilation systems for CO2 surface flux inversions http://www.atmos-chem-phys-discuss.net/15/8883/2015/acpd-15-8883-2015-discussion.html * <> CIDER: a pipeline for detecting waves of coordinated transcriptional regulation in gene expression time-course data http://biorxiv.org/content/biorxiv/early/2015/03/17/012518.full.pdf + * <> GNU Parallel was used (unfortunately without citation) in: MUGBAS: a species free gene-based programme suite for post-GWAS analysis http://www.ncbi.nlm.nih.gov/pubmed/25765345 taxator-tk http://algbio.cs.uni-duesseldorf.de/webapps/wa-download/ (check it) * << afventer svar fra Rachel >> GNU Parallel was used in: SISRS: Site Identification from Short Read Sequences https://github.com/rachelss/SISRS/ -* GNU Parallel was used in: landsat-gifworks https://github.com/KAPPS-/landsat-gifworks +* GNU Parallel was cited in: Sequencing the cap-snatching repertoire of H1N1 influenza provides insight into the mechanism of viral transcription initiation http://nar.oxfordjournals.org/content/early/2015/04/20/nar.gkv333.full.pdf -* GNU Parallel was used in: https://github.com/exascience/elprep +* GNU Parallel was cited in: Genome assemblyusing Nanopore-guided long and error-free DNA reads http://www.biomedcentral.com/content/pdf/s12864-015-1519-z.pdf -* GNU Parallel was used in: http://search.cpan.org/~jillrowe/HPC-Runner-GnuParallel-0.01/ +* GNU Parallel was cited in: Contrasting regional architectures of schizophrenia and other complex diseases using fast variance components analysis http://biorxiv.org/content/biorxiv/early/2015/03/13/016527.full.pdf -* GNU Parallel was used in: http://search.cpan.org/~ajpage/Bio-Roary-2.0.7/lib/Bio/Roary/JobRunner/Parallel.pm +* <> GNU Parallel was used (unfortunately with wrong citation) in: TADSim: Discrete Event-Based Performance Prediction for Temperature-Accelerated Dynamics http://vruehle.de/publications/2015c.pdf -* GNU Parallel was used in: Rapid haploid SNP calling https://github.com/tseemann/snippy +* GNU Parallel was used in: Gene Set Omic Analysis (GSOA) method https://bitbucket.org/srp33/gsoa -* Pictures and Metadata http://www.ozzy.no/2015/02/05/pictures-and-metadata/ +* Execute commands on multiple computers using GNU Parallel (setting up a cluster on the cheap) https://spectraldifferences.wordpress.com/2015/04/26/execute-commands-on-multiple-computers-using-gnu-parallel-setting-up-a-cluster-on-the-cheap/ -* Task automation with bash and parallel https://biowize.wordpress.com/2015/03/23/task-automation-with-bash-and-parallel/ - -* How To: Speed Up File Transfers in Linux using RSync with GNU Parallel http://www.yourownlinux.com/2015/04/speed-up-file-transfers-using-rsync-with-gnu-parallel.html - -* Mirroring a Git Repository https://avacariu.me/articles/mirroring-a-git-repository.html - -* Add an ENVI header to JAXA Global Mangrove Watch PALSAR tiles https://spectraldifferences.wordpress.com/2015/03/24/add-an-envi-header-to-jaxa-global-mangrove-watch-palsar-tiles/ - -* GNU Parallel (Sebuah Uji Coba) http://kaka.prakasa.my.id/2014/09/04/gnu-parallel-sebuah-uji-coba/ - -* 你见过的最牛逼的命令行程序是什么？ http://www.zhihu.com/question/29257300 +* Functions and GNU parallel for effective cluster load management http://genomespot.blogspot.dk/2015/04/functions-and-gnu-parallel-for.html * Bug fixes and man page updates. diff --git a/src/parallel b/src/parallel index 741ec1dd..4903133a 100755 --- a/src/parallel +++ b/src/parallel @@ -725,7 +725,6 @@ sub options_hash { "use-cpus-instead-of-cores" => \$opt::use_cpus_instead_of_cores, "shellquote|shell_quote|shell-quote" => \$opt::shellquote, "nice=i" => \$opt::nice, - "timeout=s" => \$opt::timeout, "tag" => \$opt::tag, "tagstring|tag-string=s" => \$opt::tagstring, "onall" => \$opt::onall, @@ -751,9 +750,7 @@ sub options_hash { "compress" => \$opt::compress, "tty" => \$opt::tty, "T" => \$opt::retired, - "halt-on-error|halt=s" => \$opt::halt, "H=i" => \$opt::retired, - "retries=i" => \$opt::retries, "dry-run|dryrun" => \$opt::dryrun, "progress" => \$opt::progress, "eta" => \$opt::eta, @@ -779,7 +776,11 @@ sub options_hash { "xapply" => \$opt::xapply, "bibtex" => \$opt::bibtex, "wc|willcite|will-cite|nn|nonotice|no-notice" => \$opt::willcite, + # Termination and retries + "halt-on-error|halt=s" => \$opt::halt, "memfree=s" => \$opt::memfree, + "retries=i" => \$opt::retries, + "timeout=s" => \$opt::timeout, # xargs-compatibility - implemented, man, testsuite "max-procs|P=s" => \$opt::jobs, "delimiter|d=s" => \$opt::d, @@ -1053,7 +1054,7 @@ sub parse_options { sub init_globals { # Defaults: - $Global::version = 20150422; + $Global::version = 20150424; $Global::progname = 'parallel'; $Global::infinity = 2**31; $Global::debug = 0; @@ -3276,17 +3277,17 @@ sub bibtex { "please cite:", "", "\@article{Tange2011a,", - " title = {GNU Parallel - The Command-Line Power Tool},", - " author = {O. Tange},", - " address = {Frederiksberg, Denmark},", - " journal = {;login: The USENIX Magazine},", - " month = {Feb},", - " number = {1},", - " volume = {36},", - " url = {http://www.gnu.org/s/parallel},", - " year = {2011},", - " pages = {42-47}", - " doi = {10.5281/zenodo.16303}", + " title = {GNU Parallel - The Command-Line Power Tool},", + " author = {O. Tange},", + " address = {Frederiksberg, Denmark},", + " journal = {;login: The USENIX Magazine},", + " month = {Feb},", + " number = {1},", + " volume = {36},", + " url = {http://www.gnu.org/s/parallel},", + " year = {2011},", + " pages = {42-47}", + " doi = {10.5281/zenodo.16303}", "}", "", "(Feel free to use \\nocite{Tange2011a})", diff --git a/src/parallel.pod b/src/parallel.pod index 43802e1c..09160700 100644 --- a/src/parallel.pod +++ b/src/parallel.pod @@ -488,7 +488,7 @@ I is a Perl Regular Expression: http://perldoc.perl.org/perlre.html -=item B<--compress> (beta testing) +=item B<--compress> Compress temporary files. If the output is big and very compressible this will take up less disk space in $TMPDIR and possibly be faster @@ -499,9 +499,9 @@ B, B, B, B, B in that order, and use the first available. -=item B<--compress-program> I (beta testing) +=item B<--compress-program> I -=item B<--decompress-program> I (beta testing) +=item B<--decompress-program> I Use I for (de)compressing temporary files. It is assumed that I will decompress stdin (standard input) to stdout (standard @@ -514,9 +514,9 @@ If receiving SIGNING, GNU B will send SIGINT to tasks running on remote computers thus killing them. -=item B<--delimiter> I (beta testing) +=item B<--delimiter> I -=item B<-d> I (beta testing) +=item B<-d> I Input items are terminated by I. Quotes and backslash are not special; every character in the input is taken literally. Disables @@ -568,7 +568,7 @@ If I is omitted, there is no end of file string. If neither B<-E> nor B<-e> is used, no end of file string is used. -=item B<--env> I (alpha testing) +=item B<--env> I (beta testing) Copy environment variable I. This will copy I to the environment that the command is run in. This is especially useful for @@ -782,7 +782,7 @@ specified, and for B<-I>{} otherwise. This option is deprecated; use B<-I> instead. -=item B<--joblog> I (beta testing) +=item B<--joblog> I Logfile for executed jobs. Save a list of the executed jobs to I in the following TAB separated format: sequence number, @@ -1039,9 +1039,9 @@ of each job is saved in a file and the filename is then printed. See also: B<--results> -=item B<--pipe> (beta testing) +=item B<--pipe> -=item B<--spreadstdin> (beta testing) +=item B<--spreadstdin> Spread input to jobs on stdin (standard input). Read a block of data from stdin (standard input) and give one block of data as input to one @@ -1068,7 +1068,7 @@ B<--files> is often used with B<--pipe>. See also: B<--recstart>, B<--recend>, B<--fifo>, B<--cat>, B<--pipepart>. -=item B<--pipepart> (alpha testing) +=item B<--pipepart> (beta testing) Pipe parts of a physical file. B<--pipepart> works similar to B<--pipe>, but is much faster. It has a few limitations: @@ -1162,13 +1162,13 @@ exit (used by GNU B itself to determine the line length on remote computers). -=item B<--number-of-cpus> (beta testing) +=item B<--number-of-cpus> Print the number of physical CPUs and exit (used by GNU B itself to determine the number of physical CPUs on remote computers). -=item B<--number-of-cores> (beta testing) +=item B<--number-of-cores> Print the number of CPU cores and exit (used by GNU B itself to determine the number of CPU cores on remote computers). @@ -1180,7 +1180,7 @@ Overrides an earlier B<--keep-order> (e.g. if set in B<~/.parallel/config>). -=item B<--nice> I (beta testing) +=item B<--nice> I Run the command at this niceness. For simple commands you can just add B in front of the command. But if the command consists of more @@ -1441,7 +1441,7 @@ B<--keep-order> will not work with B<--round-robin> as it is impossible to track which input block corresponds to which output. -=item B<--rpl> 'I I' (beta testing) +=item B<--rpl> 'I I' Use I as a replacement string for I. This makes it possible to define your own replacement strings. GNU B's @@ -1616,7 +1616,7 @@ Does not run the command but quotes it. Useful for making quoted composed commands for GNU B. -=item B<--shuf> (alpha testing) +=item B<--shuf> (beta testing) Shuffle jobs. When having multiple input sources it is hard to randomize jobs. --shuf will generate all jobs, and shuffle them before @@ -1781,7 +1781,7 @@ the lines will be prepended with the sshlogin instead. B<--tag> is ignored when using B<-u>. -=item B<--tagstring> I (beta testing) +=item B<--tagstring> I Tag lines with a string. Each output line will be prepended with I and TAB (\t). I can contain replacement strings such as @@ -1798,7 +1798,7 @@ different dir for the files. Setting B<--tmpdir> is equivalent to setting $TMPDIR. -=item B<--tmux> (beta testing) +=item B<--tmux> Use B for output. Start a B session and run each job in a window in that session. No other output will be produced.