From 585a29c0629778ca470bf118ae3d185ce3f9dda1 Mon Sep 17 00:00:00 2001 From: Ole Tange Date: Sun, 22 Mar 2015 14:14:50 +0100 Subject: [PATCH] Cleanup for next release. --- doc/release_new_version | 68 +++++------------------------------------ src/parallel | 14 +++------ src/parallel.pod | 30 +++++++++--------- 3 files changed, 26 insertions(+), 86 deletions(-) diff --git a/doc/release_new_version b/doc/release_new_version index 5942b54f..a6d99ac3 100644 --- a/doc/release_new_version +++ b/doc/release_new_version @@ -4,7 +4,7 @@ Check that documentation is updated (compare to web): -Fixet for 20150122 +Fixet for 20150422 git diff last-release-commit Unmodified beta since last version => production Unmodified alpha since last version => beta @@ -208,75 +208,21 @@ cc:Tim Cuthbertson , Ryoichiro Suzuki , Jesse Alama -Subject: GNU Parallel 20150322 ('Hellwig') released +Subject: GNU Parallel 20150422 ('') released -GNU Parallel 20150322 ('Hellwig') has been released. It is available for download at: http://ftp.gnu.org/gnu/parallel/ +GNU Parallel 20150422 ('') has been released. It is available for download at: http://ftp.gnu.org/gnu/parallel/ Haiku of the month: - Stand up for your rights. - VMware does not comply. - Hellwig goes to court. - - (https://sfconservancy.org/linux-compliance/vmware-lawsuit-faq.html) + <<>> New in this release: -* --number-of-cores respects 'taskset' on GNU/Linux. +* <> CIDER: a pipeline for detecting waves of coordinated transcriptional regulation in gene expression time-course data http://biorxiv.org/content/biorxiv/early/2015/03/17/012518.full.pdf -* --joblog --pipe gives the data send and received in the log. +* <> GNU Parallel was used (unfortunately without citation) in: MUGBAS: a species free gene-based programme suite for post-GWAS analysis http://www.ncbi.nlm.nih.gov/pubmed/25765345 -* GNU Parallel was tested to support 100 GB sized records in --pipe. A few bugs was fixed to support >2 GB records. It works, but is rather slow. - -* GNU Parallel was cited in: RIG: Recalibration and Interrelation of genomic sequence data with the GATK http://www.g3journal.org/content/early/2015/02/13/g3.115.017012.full.pdf+html - -* GNU Parallel was cited in: MPI-blastn and NCBI-TaxCollector: Improving metagenomic analysis with high performance classification and wide taxonomic attachment http://www.worldscientific.com/doi/abs/10.1142/S0219720014500139?af=R& - -* GNU Parallel was cited in: Recent evolution in Rattus norvegicus is shaped by declining effective population size http://biorxiv.org/content/biorxiv/early/2015/03/01/015818.full.pdf - -* GNU Parallel was cited in: Evidence for DCO+ as a probe of ionization in the warm disk surface http://arxiv.org/pdf/1503.02659.pdf - -* GNU Parallel was cited in: De novo assembly and annotation of the Asian tiger mosquito (Aedes albopictus) repeatome with dnaPipeTE from raw genomic reads and comparative analysis with the yellow fever mosquito (Aedes aegypti) http://gbe.oxfordjournals.org/content/early/2015/03/11/gbe.evv050.full.pdf - -* GNU Parallel was cited in: A General Approach to Network Configuration Analysis http://research.microsoft.com/en-us/um/people/ratul/papers/nsdi2015-batfish.pdf - -* GNU Parallel was cited in: Scrimer: designing primers from transcriptome data http://onlinelibrary.wiley.com/doi/10.1111/1755-0998.12403/pdf - -* GNU Parallel was cited in: Efficient Retrieval of Key Material for Inspecting Potentially Malicious Traffic in the Cloud http://sacko.uk/pdf/2015.1.pdf - -* GNU Parallel will be presented at Strataconf: Poor Man's Parallel Pipelines http://strataconf.com/big-data-conference-uk-2015/public/schedule/detail/40031 - -* GNU Parallel was used in: https://github.com/alexbyrnes/FCC-Political-Ads_The-Code - -* GNU Parallel was used in: https://github.com/martymac/fpart - -* GNU Parallel was used in: https://github.com/hoytak/diabetic-retinopathy-code - -* GNU Parallel was used in: https://github.com/mehmattski/HybSeqPipeline - -* GNU Parallel was used in: http://search.cpan.org/~ajpage/Bio-Roary-2.0.0/lib/Bio/Roary/JobRunner/Parallel.pm - -* Using GNU Parallel on a Raspberry Pi cluster: http://www.dcglug.org.uk/cluster-progress/ - -* An introduction to vector tiling and map reduce in postgis: http://dimensionaledge.com/intro-vector-tiling-map-reduce-postgis/ - -* Running scripts in parallel with GNU Parallel: http://code.jasonbhill.com/2015/03/ - -* Mahout: Parallelising the creation of DecisionTrees: http://www.markhneedham.com/blog/2012/12/27/mahout-parallelising-the-creation-of-decisiontrees/ - -* High-Performance Scientific Computing: Running serial jobs in parallel https://support.scinet.utoronto.ca/education/staticpublic/course178/serial.pdf - -* Parallel cardinality on your laptop https://highonscience.wordpress.com/2015/03/15/parallel-cardinality-on-your-laptop/ - -* Experiments about a better locate using grep http://a3nm.net/blog/better_locate.html - -* Homework with GNU Parallel: https://support.scinet.utoronto.ca/education/staticpublic/course178content349.html - -* Parallel - Jobs in Skripten parallelisieren: https://slzm.de/blog/linux-tool-des-tages-parallel-jobs-in-skripten-parallelisieren/#more-612 - -* GNU Parallel, czyli 100% użycia procesora http://matmatyk.blogspot.dk/2015/03/gnu-parallel-czyli-100-uzycia-procesora.html - -* Bug fixes and man page updates. +taxator-tk http://algbio.cs.uni-duesseldorf.de/webapps/wa-download/ (check it) GNU Parallel - For people who live life in the parallel lane. diff --git a/src/parallel b/src/parallel index 369a59e9..90dc34f3 100755 --- a/src/parallel +++ b/src/parallel @@ -1052,7 +1052,7 @@ sub parse_options { sub init_globals { # Defaults: - $Global::version = 20150322; + $Global::version = 20150323; $Global::progname = 'parallel'; $Global::infinity = 2**31; $Global::debug = 0; @@ -3524,6 +3524,7 @@ sub tmpfile { # N/A if(not $disk_full_fh) { ($disk_full_fh, $name) = ::tmpfile(SUFFIX => ".df"); + # Separate unlink due to NFS dealing badly with File::Temp unlink $name; $b8193 = "x"x8193; } @@ -3787,14 +3788,7 @@ sub memfree { # Returns: # $memfree in bytes my $self = shift; - if(not $self->{'memfree'}) { - $self->memfree_recompute(); - } -# if(time - $self->{'last_memfree'} >= 1) { - # More than 10 seconds old: Recompute - $self->{'last_memfree'} = time; - $self->memfree_recompute(); -# } + $self->memfree_recompute(); return (not defined $self->{'memfree'} or $self->{'memfree'}) } @@ -5640,7 +5634,7 @@ sub max_file_name_length { rmdir($testdir."/".$dir); $len *= 16; $dir = "x"x$len; - } while (mkdir $testdir."/".$dir); + } while ($len < $upper and mkdir $testdir."/".$dir); # Then search for the actual max length between $len/16 and $len my $min = $len/16; my $max = $len; diff --git a/src/parallel.pod b/src/parallel.pod index 1715e054..2114224c 100644 --- a/src/parallel.pod +++ b/src/parallel.pod @@ -488,7 +488,7 @@ I is a Perl Regular Expression: http://perldoc.perl.org/perlre.html -=item B<--compress> (alpha testing) +=item B<--compress> (beta testing) Compress temporary files. If the output is big and very compressible this will take up less disk space in $TMPDIR and possibly be faster @@ -499,9 +499,9 @@ B, B, B, B, B in that order, and use the first available. -=item B<--compress-program> I (alpha testing) +=item B<--compress-program> I (beta testing) -=item B<--decompress-program> I (alpha testing) +=item B<--decompress-program> I (beta testing) Use I for (de)compressing temporary files. It is assumed that I will decompress stdin (standard input) to stdout (standard @@ -514,9 +514,9 @@ If receiving SIGNING, GNU B will send SIGINT to tasks running on remote computers thus killing them. -=item B<--delimiter> I (alpha testing) +=item B<--delimiter> I (beta testing) -=item B<-d> I (alpha testing) +=item B<-d> I (beta testing) Input items are terminated by I. Quotes and backslash are not special; every character in the input is taken literally. Disables @@ -780,7 +780,7 @@ specified, and for B<-I>{} otherwise. This option is deprecated; use B<-I> instead. -=item B<--joblog> I (alpha testing) +=item B<--joblog> I (beta testing) Logfile for executed jobs. Save a list of the executed jobs to I in the following TAB separated format: sequence number, @@ -1037,9 +1037,9 @@ of each job is saved in a file and the filename is then printed. See also: B<--results> -=item B<--pipe> (alpha testing) +=item B<--pipe> (beta testing) -=item B<--spreadstdin> (alpha testing) +=item B<--spreadstdin> (beta testing) Spread input to jobs on stdin (standard input). Read a block of data from stdin (standard input) and give one block of data as input to one @@ -1160,13 +1160,13 @@ exit (used by GNU B itself to determine the line length on remote computers). -=item B<--number-of-cpus> (alpha testing) +=item B<--number-of-cpus> (beta testing) Print the number of physical CPUs and exit (used by GNU B itself to determine the number of physical CPUs on remote computers). -=item B<--number-of-cores> (alpha testing) +=item B<--number-of-cores> (beta testing) Print the number of CPU cores and exit (used by GNU B itself to determine the number of CPU cores on remote computers). @@ -1178,7 +1178,7 @@ Overrides an earlier B<--keep-order> (e.g. if set in B<~/.parallel/config>). -=item B<--nice> I (alpha testing) +=item B<--nice> I (beta testing) Run the command at this niceness. For simple commands you can just add B in front of the command. But if the command consists of more @@ -1387,7 +1387,7 @@ useful if some jobs fail for no apparent reason (such as network failure). -=item B<--return> I (beta testing) +=item B<--return> I Transfer files from remote computers. B<--return> is used with B<--sshlogin> when the arguments are files on the remote computers. When @@ -1435,7 +1435,7 @@ B<--keep-order> will not work with B<--round-robin> as it is impossible to track which input block corresponds to which output. -=item B<--rpl> 'I I' (alpha testing) +=item B<--rpl> 'I I' (beta testing) Use I as a replacement string for I. This makes it possible to define your own replacement strings. GNU B's @@ -1763,7 +1763,7 @@ the lines will be prepended with the sshlogin instead. B<--tag> is ignored when using B<-u>. -=item B<--tagstring> I (alpha testing) +=item B<--tagstring> I (beta testing) Tag lines with a string. Each output line will be prepended with I and TAB (\t). I can contain replacement strings such as @@ -1780,7 +1780,7 @@ different dir for the files. Setting B<--tmpdir> is equivalent to setting $TMPDIR. -=item B<--tmux> (alpha testing) +=item B<--tmux> (beta testing) Use B for output. Start a B session and run each job in a window in that session. No other output will be produced.