env_parallel.bash tested: Works better as variable than as tempfile.

This commit is contained in:
Ole Tange 2016-04-04 23:33:30 +02:00
parent e663fe2636
commit 9892e7deac
4 changed files with 38 additions and 6 deletions

2
README
View file

@ -6,7 +6,7 @@ Please send problems and feedback to bug-parallel@gnu.org.
= Presentation of GNU Parallel = = Presentation of GNU Parallel =
GNU Parallel is a shell tool for executing jobs in parallel using one GNU Parallel is a shell tool for executing jobs in parallel using one
or more computers. A job is can be a single command or a small script or more computers. A job can be a single command or a small script
that has to be run for each of the lines in the input. The typical that has to be run for each of the lines in the input. The typical
input is a list of files, a list of hosts, a list of users, a list of input is a list of files, a list of hosts, a list of users, a list of
URLs, or a list of tables. A job can also be a command that reads from URLs, or a list of tables. A job can also be a command that reads from

View file

@ -219,9 +219,9 @@ cc:Tim Cuthbertson <tim3d.junk@gmail.com>,
Ryoichiro Suzuki <ryoichiro.suzuki@gmail.com>, Ryoichiro Suzuki <ryoichiro.suzuki@gmail.com>,
Jesse Alama <jesse.alama@gmail.com> Jesse Alama <jesse.alama@gmail.com>
Subject: GNU Parallel 20160422 ('') released <<[stable]>> Subject: GNU Parallel 20160422 ('Gulshan-i-Iqbal') released <<[stable]>>
GNU Parallel 20160422 ('') <<[stable]>> has been released. It is available for download at: http://ftp.gnu.org/gnu/parallel/ GNU Parallel 20160422 ('Gulshan-i-Iqbal') <<[stable]>> has been released. It is available for download at: http://ftp.gnu.org/gnu/parallel/
<<No new functionality was introduced so this is a good candidate for a stable release.>> <<No new functionality was introduced so this is a good candidate for a stable release.>>
@ -248,8 +248,18 @@ for Big Data Applications https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumb
* <<Citation needed: Introspecting for RSA Key Material to Assist Intrusion Detection http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=7331177&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D7331177>> * <<Citation needed: Introspecting for RSA Key Material to Assist Intrusion Detection http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=7331177&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D7331177>>
* GNU Parallel was cited in: How Can We Measure the Similarity Between Resumes of Selected Candidates for a Job? https://www.researchgate.net/publication/275954089_How_can_we_measure_the_similarity_between_resumes_of_selected_candidates_for_a_job
* GNU Parallel was cited in: Automatic Methods for Assisted Recruitment https://www.researchgate.net/publication/297738658_Automatic_Methods_for_Assisted_Recruitment
* GNU Parallel was cited in: Tools and techniques for computational reproducibility http://biorxiv.org/content/biorxiv/early/2016/03/17/022707.full.pdf * GNU Parallel was cited in: Tools and techniques for computational reproducibility http://biorxiv.org/content/biorxiv/early/2016/03/17/022707.full.pdf
* GNU Parallel was cited in: Reinterpretation of ATLAS 8 TeV searches for Natural SUSY with a R-Sneutrino LSP http://arxiv.org/pdf/1603.06130.pdf
* GNU Parallel was cited in: An Operational Radiometric Landsat Preprocessing Framework for Large-Area Time Series Applications https://www.uni-trier.de/fileadmin/fb6/prof/FER/Publikationen/frantz_et_al_ieee-tgrs-2016-post-print.pdf
* Downloading a list of URLs http://blog.gypsydave5.com/2016/02/04/xargs-and-curl/
* qbatch uses GNU Parallel: https://pypi.python.org/pypi/qbatch/1.0rc2 * qbatch uses GNU Parallel: https://pypi.python.org/pypi/qbatch/1.0rc2
* FaceCrop uses GNU Parallel: https://github.com/EderSantana/FaceCrop * FaceCrop uses GNU Parallel: https://github.com/EderSantana/FaceCrop

View file

@ -25,6 +25,7 @@
# or write to the Free Software Foundation, Inc., 51 Franklin St, # or write to the Free Software Foundation, Inc., 51 Franklin St,
# Fifth Floor, Boston, MA 02110-1301 USA # Fifth Floor, Boston, MA 02110-1301 USA
# Supports env of 127426 bytes
env_parallel() { env_parallel() {
export PARALLEL_ENV="$(echo "shopt -s expand_aliases 2>/dev/null"; alias;typeset -p | export PARALLEL_ENV="$(echo "shopt -s expand_aliases 2>/dev/null"; alias;typeset -p |
grep -vFf <(readonly) | grep -vFf <(readonly) |
@ -33,3 +34,17 @@ env_parallel() {
`which parallel` "$@"; `which parallel` "$@";
unset PARALLEL_ENV; unset PARALLEL_ENV;
} }
# Supports env of 127375 bytes
#
# env_parallel() {
# # Saving to a tempfile
# export PARALLEL_ENV=`tempfile`;
# (echo "shopt -s expand_aliases 2>/dev/null"; alias;typeset -p |
# grep -vFf <(readonly) |
# grep -v 'declare .. (GROUPS|FUNCNAME|DIRSTACK|_|PIPESTATUS|USERNAME|BASH_[A-Z_]+) ';
# typeset -f) > $PARALLEL_ENV
# `which parallel` "$@";
# rm "$PARALLEL_ENV"
# unset PARALLEL_ENV;
# }

View file

@ -20,10 +20,17 @@ environment to GNU Parallel.
If the shell function is not loaded, a dummy script will be run If the shell function is not loaded, a dummy script will be run
instead that explains how to install the function. instead that explains how to install the function.
=head2 Environment space
B<env_parallel> only works if the size of the current environment is B<env_parallel> only works if the size of the current environment is
smaller than half of the max size and smaller than 25% of the max if smaller than the maximal length of a command and smaller than half of
running remotely. E.g. The max size of Bash's environment is 256 KB, the max if running remotely. E.g. The max size of Bash's command is
so B<env_parallel> will fail if B<set | wc -c> is bigger than 128 KB. 128 KB, so B<env_parallel> will fail if B<set | wc -c> is bigger than
128 KB. Technically the limit is in execve(1) which IPC::open3 uses.
Bash completion functions are well-known for taking up well over 128
KB of environment space and the primary reason for causing
B<env_parallel> to fail.
=head1 OPTIONS =head1 OPTIONS