env_parallel.bash tested: Works better as variable than as tempfile.

This commit is contained in:
Ole Tange 2016-04-04 23:33:30 +02:00
parent e663fe2636
commit 9892e7deac
4 changed files with 38 additions and 6 deletions

2
README
View file

@ -6,7 +6,7 @@ Please send problems and feedback to bug-parallel@gnu.org.
= Presentation of GNU Parallel =
GNU Parallel is a shell tool for executing jobs in parallel using one
or more computers. A job is can be a single command or a small script
or more computers. A job can be a single command or a small script
that has to be run for each of the lines in the input. The typical
input is a list of files, a list of hosts, a list of users, a list of
URLs, or a list of tables. A job can also be a command that reads from

View file

@ -219,9 +219,9 @@ cc:Tim Cuthbertson <tim3d.junk@gmail.com>,
Ryoichiro Suzuki <ryoichiro.suzuki@gmail.com>,
Jesse Alama <jesse.alama@gmail.com>
Subject: GNU Parallel 20160422 ('') released <<[stable]>>
Subject: GNU Parallel 20160422 ('Gulshan-i-Iqbal') released <<[stable]>>
GNU Parallel 20160422 ('') <<[stable]>> has been released. It is available for download at: http://ftp.gnu.org/gnu/parallel/
GNU Parallel 20160422 ('Gulshan-i-Iqbal') <<[stable]>> has been released. It is available for download at: http://ftp.gnu.org/gnu/parallel/
<<No new functionality was introduced so this is a good candidate for a stable release.>>
@ -248,8 +248,18 @@ for Big Data Applications https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumb
* <<Citation needed: Introspecting for RSA Key Material to Assist Intrusion Detection http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=7331177&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D7331177>>
* GNU Parallel was cited in: How Can We Measure the Similarity Between Resumes of Selected Candidates for a Job? https://www.researchgate.net/publication/275954089_How_can_we_measure_the_similarity_between_resumes_of_selected_candidates_for_a_job
* GNU Parallel was cited in: Automatic Methods for Assisted Recruitment https://www.researchgate.net/publication/297738658_Automatic_Methods_for_Assisted_Recruitment
* GNU Parallel was cited in: Tools and techniques for computational reproducibility http://biorxiv.org/content/biorxiv/early/2016/03/17/022707.full.pdf
* GNU Parallel was cited in: Reinterpretation of ATLAS 8 TeV searches for Natural SUSY with a R-Sneutrino LSP http://arxiv.org/pdf/1603.06130.pdf
* GNU Parallel was cited in: An Operational Radiometric Landsat Preprocessing Framework for Large-Area Time Series Applications https://www.uni-trier.de/fileadmin/fb6/prof/FER/Publikationen/frantz_et_al_ieee-tgrs-2016-post-print.pdf
* Downloading a list of URLs http://blog.gypsydave5.com/2016/02/04/xargs-and-curl/
* qbatch uses GNU Parallel: https://pypi.python.org/pypi/qbatch/1.0rc2
* FaceCrop uses GNU Parallel: https://github.com/EderSantana/FaceCrop

View file

@ -25,6 +25,7 @@
# or write to the Free Software Foundation, Inc., 51 Franklin St,
# Fifth Floor, Boston, MA 02110-1301 USA
# Supports env of 127426 bytes
env_parallel() {
export PARALLEL_ENV="$(echo "shopt -s expand_aliases 2>/dev/null"; alias;typeset -p |
grep -vFf <(readonly) |
@ -33,3 +34,17 @@ env_parallel() {
`which parallel` "$@";
unset PARALLEL_ENV;
}
# Supports env of 127375 bytes
#
# env_parallel() {
# # Saving to a tempfile
# export PARALLEL_ENV=`tempfile`;
# (echo "shopt -s expand_aliases 2>/dev/null"; alias;typeset -p |
# grep -vFf <(readonly) |
# grep -v 'declare .. (GROUPS|FUNCNAME|DIRSTACK|_|PIPESTATUS|USERNAME|BASH_[A-Z_]+) ';
# typeset -f) > $PARALLEL_ENV
# `which parallel` "$@";
# rm "$PARALLEL_ENV"
# unset PARALLEL_ENV;
# }

View file

@ -20,10 +20,17 @@ environment to GNU Parallel.
If the shell function is not loaded, a dummy script will be run
instead that explains how to install the function.
=head2 Environment space
B<env_parallel> only works if the size of the current environment is
smaller than half of the max size and smaller than 25% of the max if
running remotely. E.g. The max size of Bash's environment is 256 KB,
so B<env_parallel> will fail if B<set | wc -c> is bigger than 128 KB.
smaller than the maximal length of a command and smaller than half of
the max if running remotely. E.g. The max size of Bash's command is
128 KB, so B<env_parallel> will fail if B<set | wc -c> is bigger than
128 KB. Technically the limit is in execve(1) which IPC::open3 uses.
Bash completion functions are well-known for taking up well over 128
KB of environment space and the primary reason for causing
B<env_parallel> to fail.
=head1 OPTIONS