tange/parallel
1
0
Fork 0
mirror of https://git.savannah.gnu.org/git/parallel.git synced 2024-11-26 16:07:54 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

parallel: Transfer functions through csh if PARALLEL_SHELL is set to bash.

Browse source
This commit is contained in:
Ole Tange 2016-05-04 19:28:40 +02:00
parent e4738dfb97
commit f8d1474943
9 changed files with 60 additions and 38 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

50
doc/release_new_version
View file

@ -219,9 +219,9 @@ cc:Tim Cuthbertson <tim3d.junk@gmail.com>,
Ryoichiro Suzuki <ryoichiro.suzuki@gmail.com>, Ryoichiro Suzuki <ryoichiro.suzuki@gmail.com>,
Jesse Alama <jesse.alama@gmail.com> Jesse Alama <jesse.alama@gmail.com>
Subject: GNU Parallel 20160422 ('PanamaPapers') released <<[stable]>> Subject: GNU Parallel 20160522 ('ttipleaks') released <<[stable]>>
GNU Parallel 20160422 ('PanamaPapers') <<[stable]>> has been released. It is available for download at: http://ftp.gnu.org/gnu/parallel/ GNU Parallel 20160522 ('ttipleaks') <<[stable]>> has been released. It is available for download at: http://ftp.gnu.org/gnu/parallel/
<<No new functionality was introduced so this is a good candidate for a stable release.>> <<No new functionality was introduced so this is a good candidate for a stable release.>>
@ -232,6 +232,8 @@ Haiku of the month:
New in this release: New in this release:
* Vote for GNU Parallel's community ad on https://meta.askubuntu.com/questions/14925/community-promotion-ads-2016/15046#15046
* << kontakt GNU Parallel was used (unfortunately without citation) in: Instrumentation and Trace Analysis for Ad-hoc Python Workflows in Cloud Environments http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7214035>> * << kontakt GNU Parallel was used (unfortunately without citation) in: Instrumentation and Trace Analysis for Ad-hoc Python Workflows in Cloud Environments http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7214035>>
* <<Har angiveligt submittet ny version - afventer opdatering>> GNU Parallel was used (unfortunately without citation) in: MUGBAS: a species free gene-based programme suite for post-GWAS analysis http://www.ncbi.nlm.nih.gov/pubmed/25765345 * <<Har angiveligt submittet ny version - afventer opdatering>> GNU Parallel was used (unfortunately without citation) in: MUGBAS: a species free gene-based programme suite for post-GWAS analysis http://www.ncbi.nlm.nih.gov/pubmed/25765345
@ -248,37 +250,49 @@ for Big Data Applications https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumb
* <<Citation needed: Introspecting for RSA Key Material to Assist Intrusion Detection http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=7331177&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D7331177>> * <<Citation needed: Introspecting for RSA Key Material to Assist Intrusion Detection http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=7331177&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D7331177>>
* :::+ and ::::+ work like ::: and :::: but links this input source to the previous input source in a --xapply fashion. Contrary to --xapply values do not wrap: The shortest input source determines the length. * GNU Parallel was cited in: Structure and evolutionary history of a large family of NLR proteins in the zebrafish http://rsob.royalsocietypublishing.org/content/royopenbio/6/4/160009.full.pdf
* --line-buffer --keep-order now outputs continously from the oldest job still running. This is more what you would expect than the earlier behaviour where --keep-order had no effect with --line-buffer. * GNU Parallel was cited in: Data processing pipeline for serial femtosecond crystallography at SACLA http://journals.iucr.org/j/issues/2016/03/00/zw5001/index.html
* env_parallel supports tcsh, csh, pdksh. In fish it now supports arrays. In csh/tcsh it now supports variables, aliases, and arrays with no special chars. In pdksh it supports aliases, functions, variables, and arrays. * GNU Parallel was cited in: Reconstruction of Fine-Scale Auroral Dynamics http://arxiv.org/pdf/1512.01460.pdf
* Function exporting on Mac OS X works around old Bash version. * GNU Parallel was cited in: The Outer Solar System Origins Survey: I. Design and First-Quarter Discoveries http://arxiv.org/pdf/1511.02895.pdf
* Better CPU detection on OpenIndiana. * GNU Parallel was cited in: Multiscale Estimation of Binding Kinetics Using Brownian Dynamics, Molecular Dynamics and Milestoning http://journals.plos.org/ploscompbiol/article/asset?id=10.1371%2Fjournal.pcbi.1004381.PDF
* GNU Parallel was cited in: How Can We Measure the Similarity Between Resumes of Selected Candidates for a Job? https://www.researchgate.net/publication/275954089_How_can_we_measure_the_similarity_between_resumes_of_selected_candidates_for_a_job * GNU Parallel was cited in: Genomic legacy of the African cheetah, Acinonyx jubatus https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4676127/pdf/13059_2015_Article_837.pdf
* GNU Parallel was cited in: Automatic Methods for Assisted Recruitment https://www.researchgate.net/publication/297738658_Automatic_Methods_for_Assisted_Recruitment * GNU Parallel was cited in: Don’t Forget to Lock the Back Door! A Characterization of IPv6 Network Security Policy http://benign-research-probe2.eecs.umich.edu/ndss16_ipv6_final.pdf
* GNU Parallel was cited in: Tools and techniques for computational reproducibility http://biorxiv.org/content/biorxiv/early/2016/03/17/022707.full.pdf * GNU Parallel was cited in: Comprehensive Annotation of the Parastagonospora nodorum Reference Genome Using Next-Generation Genomics, Transcriptomics and Proteogenomics http://journals.plos.org/plosone/article/asset?id=10.1371%2Fjournal.pone.0147221.PDF
* GNU Parallel was cited in: Reinterpretation of ATLAS 8 TeV searches for Natural SUSY with a R-Sneutrino LSP http://arxiv.org/pdf/1603.06130.pdf * GNU Parallel was cited in: Stride Search: a general algorithm for storm detection in high-resolution climate data http://www.geosci-model-dev-discuss.net/8/7727/2015/gmdd-8-7727-2015.pdf
* GNU Parallel was cited in: An Operational Radiometric Landsat Preprocessing Framework for Large-Area Time Series Applications https://www.uni-trier.de/fileadmin/fb6/prof/FER/Publikationen/frantz_et_al_ieee-tgrs-2016-post-print.pdf * GNU Parallel was cited in: How attention influences perceptual decision making: Single-trial EEG correlates of drift-diffusion model parameters http://www.cidlab.com/prints/nunez2016attention.pdf
* A basic demo of how GNU Parallel can speed up execution of commands https://www.youtube.com/watch?v=kl8LO2jcvMc * GNU Parallel was cited in: AT-GIS: Highly Parallel Spatial Query Processing with Associative Transducers http://lsds.doc.ic.ac.uk/sites/default/files/ATGIS-SIGMOD16.pdf
* Downloading a list of URLs http://blog.gypsydave5.com/2016/02/04/xargs-and-curl/ * GNU Parallel was cited in: PVAIR: Partial Variable Assignment InterpolatoR http://verify.inf.usi.ch/sites/default/files/main-2.pdf
* qbatch uses GNU Parallel: https://pypi.python.org/pypi/qbatch/1.0rc2 * GNU Parallel was cited in: Shannon: An Information-Optimal de NovoRNA-Seq Assembler http://biorxiv.org/content/biorxiv/early/2016/02/09/039230.full.pdf
* FaceCrop uses GNU Parallel: https://github.com/EderSantana/FaceCrop * GNU Parallel was cited in: Computational Design of DNA-Binding Proteins http://link.springer.com/protocol/10.1007/978-1-4939-3569-7_16
* Parallel Processing with Catmandu https://librecatproject.wordpress.com/2016/04/20/parallel-processing-with-catmandu/ * GNU Parallel was cited in: Reference genotype and exome data from an Australian Aboriginal population for health-based research http://www.nature.com/articles/sdata201623
* GNU parallel 應用範例 http://staypython.blogspot.dk/2016/04/gnu-parallel.html * GNU Parallel was cited in: Do aye-ayes echolocate? http://biorxiv.org/content/biorxiv/early/2016/04/11/048165.full.pdf
* GNU Parallel was cited in: Functional enrichments of disease variants across thousands of independent loci in eight diseases http://biorxiv.org/content/biorxiv/early/2016/04/11/048066.full.pdf
* GNU Parallel was cited in: From genomes to phenotypes: Traitar, the microbial trait analyzer http://biorxiv.org/content/biorxiv/early/2016/03/12/043315.full.pdf
* GNU Parallel was mentioned in: Fast Playback Framework for Analysis of Ground-Based Doppler Radar Observations Using MapReduce Technology http://journals.ametsoc.org/doi/pdf/10.1175/JTECH-D-15-0118.1
* NCBI blast tutorial https://github.com/enormandeau/ncbi_blast_tutorial
* Distributed Preservation Made Simple https://blog.archive.org/2016/02/26/distributed-preservation-made-simple/
* Parallel Jobs in Luigi http://rjbaxley.com/posts/2016/03/13/parallel_jobs_in_luigi.html
* Bug fixes and man page updates. * Bug fixes and man page updates.

6
src/env_parallel.tcsh
View file

@ -1,8 +1,8 @@
#!/bin/csh #!/bin/csh
# This file must be sourced in csh: # This file must be sourced in tcsh:
# #
# source `which env_parallel.csh` # source `which env_parallel.tcsh`
# #
# after which 'env_parallel' works # after which 'env_parallel' works
# #
@ -60,8 +60,6 @@ else
# Quoted: s/\\047/\\047\\042\\047\\042\\047/g\; # Quoted: s/\\047/\\047\\042\\047\\042\\047/g\;
# Remove () from second column # Remove () from second column
# s/^(\S+)(\s+)\((.*)\)/\1\2\3/
# \047 => '
# s/^(\S+)(\s+)\((.*)\)/\1\2\3/; # s/^(\S+)(\s+)\((.*)\)/\1\2\3/;
# Quoted: s/\^\(\\S+\)\(\\s+\)\\\(\(.\*\)\\\)/\\1\\2\\3/\; # Quoted: s/\^\(\\S+\)\(\\s+\)\\\(\(.\*\)\\\)/\\1\\2\\3/\;

8
src/niceload
View file

@ -24,7 +24,7 @@
use strict; use strict;
use Getopt::Long; use Getopt::Long;
$Global::progname="niceload"; $Global::progname="niceload";
$Global::version = 20160422; $Global::version = 20160423;
Getopt::Long::Configure("bundling","require_order"); Getopt::Long::Configure("bundling","require_order");
get_options_from_array(\@ARGV) || die_usage(); get_options_from_array(\@ARGV) || die_usage();
if($opt::version) { if($opt::version) {
@ -346,9 +346,9 @@ sub die_bug {
sub usleep { sub usleep {
# Sleep this many milliseconds. # Sleep this many milliseconds.
my $secs = shift; my $ms = shift;
::debug("Sleeping ",$secs," millisecs\n"); ::debug("Sleeping ",$ms," millisecs\n");
select(undef, undef, undef, $secs/1000); select(undef, undef, undef, $ms/1000);
} }

10
src/parallel
View file

@ -764,7 +764,7 @@ sub options_hash {
"sqlmaster=s" => \$opt::sqlmaster, "sqlmaster=s" => \$opt::sqlmaster,
"sqlworker=s" => \$opt::sqlworker, "sqlworker=s" => \$opt::sqlworker,
"sqlandworker=s" => \$opt::sqlandworker, "sqlandworker=s" => \$opt::sqlandworker,
"joblog=s" => \$opt::joblog, "joblog|jl=s" => \$opt::joblog,
"results|result|res=s" => \$opt::results, "results|result|res=s" => \$opt::results,
"resume" => \$opt::resume, "resume" => \$opt::resume,
"resume-failed|resumefailed" => \$opt::resume_failed, "resume-failed|resumefailed" => \$opt::resume_failed,
@ -1183,7 +1183,7 @@ sub check_invalid_option_combinations {
sub init_globals { sub init_globals {
# Defaults: # Defaults:
$Global::version = 20160422; $Global::version = 20160423;
$Global::progname = 'parallel'; $Global::progname = 'parallel';
$Global::infinity = 2**31; $Global::infinity = 2**31;
$Global::debug = 0; $Global::debug = 0;
@ -7112,13 +7112,13 @@ sub sshlogin_wrap {
my $bashfuncset; my $bashfuncset;
if(@bashfunc) { if(@bashfunc) {
# Functions are not supported for all shells # Functions are not supported for all shells
if($Global::shell !~ m:/(bash|rbash|zsh|rzsh|dash|ksh):) { if($Global::shell !~ m:(bash|rbash|zsh|rzsh|dash|ksh):) {
::warning("Shell functions may not be supported in $Global::shell."); ::warning("Shell functions may not be supported in $Global::shell.");
} }
$bashfuncset = $bashfuncset =
'@bash_functions=qw('."@bash_functions".");". '@bash_functions=qw('."@bash_functions".");".
::spacefree(1,q{ ::spacefree(1,'$shell="'.($ENV{'PARALLEL_SHELL'} || '$ENV{SHELL}').'";'.q{
if($ENV{"SHELL"}=~/csh/) { if($shell=~/csh/) {
print STDERR "CSH/TCSH DO NOT SUPPORT newlines IN VARIABLES/FUNCTIONS. Unset @bash_functions\n"; print STDERR "CSH/TCSH DO NOT SUPPORT newlines IN VARIABLES/FUNCTIONS. Unset @bash_functions\n";
exec "false"; exec "false";
} }

10
src/parallel.pod
View file

@ -285,7 +285,7 @@ positional replacement strings see B<{>I<n>B<}>.
See also: B<{=perl expression=}> B<{>I<n>B<}>. See also: B<{=perl expression=}> B<{>I<n>B<}>.
=item B<:::> I<arguments> (alpha testing) =item B<:::> I<arguments> (beta testing)
Use arguments from the command line as input source instead of stdin Use arguments from the command line as input source instead of stdin
(standard input). Unlike other options for GNU B<parallel> B<:::> is (standard input). Unlike other options for GNU B<parallel> B<:::> is
@ -326,7 +326,7 @@ B<:::> and B<::::> can be mixed. So these are equivalent:
::: 1 2 3 ::: 1 2 3
=item B<:::+> I<arguments> (alpha testing) =item B<:::+> I<arguments> (beta testing)
Like B<:::> but linked like B<--xapply> to the previous input source. Like B<:::> but linked like B<--xapply> to the previous input source.
@ -338,7 +338,7 @@ Example:
parallel echo ::: a b c :::+ 1 2 3 ::: X Y :::+ 11 22 parallel echo ::: a b c :::+ 1 2 3 ::: X Y :::+ 11 22
=item B<::::> I<argfiles> (alpha testing) =item B<::::> I<argfiles> (beta testing)
Another way to write B<-a> I<argfile1> B<-a> I<argfile2> ... Another way to write B<-a> I<argfile1> B<-a> I<argfile2> ...
@ -347,7 +347,7 @@ B<:::> and B<::::> can be mixed.
See B<-a>, B<:::> and B<--xapply>. See B<-a>, B<:::> and B<--xapply>.
=item B<::::+> I<argfiles> (alpha testing) =item B<::::+> I<argfiles> (beta testing)
Like B<::::+> but linked like B<--xapply> to the previous input source. Like B<::::+> but linked like B<--xapply> to the previous input source.
@ -611,7 +611,7 @@ If I<eof-str> is omitted, there is no end of file string. If neither
B<-E> nor B<-e> is used, no end of file string is used. B<-E> nor B<-e> is used, no end of file string is used.
=item B<--env> I<var> (alpha testing) =item B<--env> I<var> (beta testing)
Copy environment variable I<var>. This will copy I<var> to the Copy environment variable I<var>. This will copy I<var> to the
environment that the command is run in. This is especially useful for environment that the command is run in. This is especially useful for

2
src/sql
View file

@ -566,7 +566,7 @@ $Global::Initfile && unlink $Global::Initfile;
exit ($err); exit ($err);
sub parse_options { sub parse_options {
$Global::version = 20160422; $Global::version = 20160423;
$Global::progname = 'sql'; $Global::progname = 'sql';
# This must be done first as this may exec myself # This must be done first as this may exec myself

6
testsuite/tests-to-run/parallel-local-ssh3.sh
View file

@ -77,4 +77,10 @@ echo '### bug #45907: --header : + --return {header}'
echo "### bug #47608: parallel --nonall -S lo 'echo ::: ' blocks" echo "### bug #47608: parallel --nonall -S lo 'echo ::: ' blocks"
parallel --nonall -S lo 'echo ::: ' parallel --nonall -S lo 'echo ::: '
echo '### exported function to csh but with PARALLEL_SHELL=bash'
doit() { echo "$1"; };
export -f doit;
stdout parallel --env doit -S csh@lo doit ::: not_OK;
PARALLEL_SHELL=bash parallel --env doit -S csh@lo doit ::: OK
EOF EOF

1
testsuite/wanted-results/parallel-freebsd
View file

@ -62,7 +62,6 @@ bash -c 'echo bug \#43358: shellshock breaks exporting functions using --env _;
bug #43358: shellshock breaks exporting functions using --env _ bug #43358: shellshock breaks exporting functions using --env _
Non-shellshock-hardened to non-shellshock-hardened Non-shellshock-hardened to non-shellshock-hardened
Function non-shellshock-hardened Function non-shellshock-hardened
parallel: Warning: Shell functions may not be supported in bash.
bash -c 'echo bug \#43358: shellshock breaks exporting functions using --env _; echo Non-shellshock-hardened to shellshock-hardened; funky() { echo Function $1; }; export -f funky; parallel --env funky -S parallel@192.168.1.72 funky ::: shellshock-hardened' bash -c 'echo bug \#43358: shellshock breaks exporting functions using --env _; echo Non-shellshock-hardened to shellshock-hardened; funky() { echo Function $1; }; export -f funky; parallel --env funky -S parallel@192.168.1.72 funky ::: shellshock-hardened'
bug #43358: shellshock breaks exporting functions using --env _ bug #43358: shellshock breaks exporting functions using --env _
Non-shellshock-hardened to shellshock-hardened Non-shellshock-hardened to shellshock-hardened

5
testsuite/wanted-results/parallel-local-ssh3
View file

@ -117,3 +117,8 @@ echo "### bug #47608: parallel --nonall -S lo 'echo ::: ' blocks"
### bug #47608: parallel --nonall -S lo 'echo ::: ' blocks ### bug #47608: parallel --nonall -S lo 'echo ::: ' blocks
parallel --nonall -S lo 'echo ::: ' parallel --nonall -S lo 'echo ::: '
::: :::
echo '### exported function to csh but with PARALLEL_SHELL=bash'
### exported function to csh but with PARALLEL_SHELL=bash
doit() { echo "$1"; }; export -f doit; stdout parallel --env doit -S csh@lo doit ::: not_OK; PARALLEL_SHELL=bash parallel --env doit -S csh@lo doit ::: OK
CSH/TCSH DO NOT SUPPORT newlines IN VARIABLES/FUNCTIONS. Unset doit
OK