From 9892e7deac54a88ae634cc9f2d4865a3c98622ae Mon Sep 17 00:00:00 2001 From: Ole Tange Date: Mon, 4 Apr 2016 23:33:30 +0200 Subject: [PATCH] env_parallel.bash tested: Works better as variable than as tempfile. --- README | 2 +- doc/release_new_version | 14 ++++++++++++-- src/env_parallel.bash | 15 +++++++++++++++ src/env_parallel.pod | 13 ++++++++++--- 4 files changed, 38 insertions(+), 6 deletions(-) diff --git a/README b/README index 85590d35..2ed882e7 100644 --- a/README +++ b/README @@ -6,7 +6,7 @@ Please send problems and feedback to bug-parallel@gnu.org. = Presentation of GNU Parallel = GNU Parallel is a shell tool for executing jobs in parallel using one -or more computers. A job is can be a single command or a small script +or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from diff --git a/doc/release_new_version b/doc/release_new_version index 28f34012..cce04d97 100644 --- a/doc/release_new_version +++ b/doc/release_new_version @@ -219,9 +219,9 @@ cc:Tim Cuthbertson , Ryoichiro Suzuki , Jesse Alama -Subject: GNU Parallel 20160422 ('') released <<[stable]>> +Subject: GNU Parallel 20160422 ('Gulshan-i-Iqbal') released <<[stable]>> -GNU Parallel 20160422 ('') <<[stable]>> has been released. It is available for download at: http://ftp.gnu.org/gnu/parallel/ +GNU Parallel 20160422 ('Gulshan-i-Iqbal') <<[stable]>> has been released. It is available for download at: http://ftp.gnu.org/gnu/parallel/ <> @@ -248,8 +248,18 @@ for Big Data Applications https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumb * <> +* GNU Parallel was cited in: How Can We Measure the Similarity Between Resumes of Selected Candidates for a Job? https://www.researchgate.net/publication/275954089_How_can_we_measure_the_similarity_between_resumes_of_selected_candidates_for_a_job + +* GNU Parallel was cited in: Automatic Methods for Assisted Recruitment https://www.researchgate.net/publication/297738658_Automatic_Methods_for_Assisted_Recruitment + * GNU Parallel was cited in: Tools and techniques for computational reproducibility http://biorxiv.org/content/biorxiv/early/2016/03/17/022707.full.pdf +* GNU Parallel was cited in: Reinterpretation of ATLAS 8 TeV searches for Natural SUSY with a R-Sneutrino LSP http://arxiv.org/pdf/1603.06130.pdf + +* GNU Parallel was cited in: An Operational Radiometric Landsat Preprocessing Framework for Large-Area Time Series Applications https://www.uni-trier.de/fileadmin/fb6/prof/FER/Publikationen/frantz_et_al_ieee-tgrs-2016-post-print.pdf + +* Downloading a list of URLs http://blog.gypsydave5.com/2016/02/04/xargs-and-curl/ + * qbatch uses GNU Parallel: https://pypi.python.org/pypi/qbatch/1.0rc2 * FaceCrop uses GNU Parallel: https://github.com/EderSantana/FaceCrop diff --git a/src/env_parallel.bash b/src/env_parallel.bash index 4f36bdce..54f64171 100755 --- a/src/env_parallel.bash +++ b/src/env_parallel.bash @@ -25,6 +25,7 @@ # or write to the Free Software Foundation, Inc., 51 Franklin St, # Fifth Floor, Boston, MA 02110-1301 USA +# Supports env of 127426 bytes env_parallel() { export PARALLEL_ENV="$(echo "shopt -s expand_aliases 2>/dev/null"; alias;typeset -p | grep -vFf <(readonly) | @@ -33,3 +34,17 @@ env_parallel() { `which parallel` "$@"; unset PARALLEL_ENV; } + +# Supports env of 127375 bytes +# +# env_parallel() { +# # Saving to a tempfile +# export PARALLEL_ENV=`tempfile`; +# (echo "shopt -s expand_aliases 2>/dev/null"; alias;typeset -p | +# grep -vFf <(readonly) | +# grep -v 'declare .. (GROUPS|FUNCNAME|DIRSTACK|_|PIPESTATUS|USERNAME|BASH_[A-Z_]+) '; +# typeset -f) > $PARALLEL_ENV +# `which parallel` "$@"; +# rm "$PARALLEL_ENV" +# unset PARALLEL_ENV; +# } diff --git a/src/env_parallel.pod b/src/env_parallel.pod index acca97d9..a8b6a0b1 100644 --- a/src/env_parallel.pod +++ b/src/env_parallel.pod @@ -20,10 +20,17 @@ environment to GNU Parallel. If the shell function is not loaded, a dummy script will be run instead that explains how to install the function. +=head2 Environment space + B only works if the size of the current environment is -smaller than half of the max size and smaller than 25% of the max if -running remotely. E.g. The max size of Bash's environment is 256 KB, -so B will fail if B is bigger than 128 KB. +smaller than the maximal length of a command and smaller than half of +the max if running remotely. E.g. The max size of Bash's command is +128 KB, so B will fail if B is bigger than +128 KB. Technically the limit is in execve(1) which IPC::open3 uses. + +Bash completion functions are well-known for taking up well over 128 +KB of environment space and the primary reason for causing +B to fail. =head1 OPTIONS