tange/parallel
1
0
Fork 0
mirror of https://git.savannah.gnu.org/git/parallel.git synced 2024-11-22 14:07:55 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Fixed bug #47086: [PATCH] Initialize total_completed from joblog.

Browse source
This commit is contained in:
Ole Tange 2016-02-19 22:15:08 +01:00
parent 794315c676
commit b1447b6bad
8 changed files with 77 additions and 36 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
doc/release_new_version
View file

@ -213,9 +213,9 @@ cc:Tim Cuthbertson <tim3d.junk@gmail.com>,
Ryoichiro Suzuki <ryoichiro.suzuki@gmail.com>, Ryoichiro Suzuki <ryoichiro.suzuki@gmail.com>,
Jesse Alama <jesse.alama@gmail.com> Jesse Alama <jesse.alama@gmail.com>
Subject: GNU Parallel 20160122 ('') released <<[stable]>> Subject: GNU Parallel 20160222 ('N977GA') released <<[stable]>>
GNU Parallel 20160122 ('') <<[stable]>> has been released. It is available for download at: http://ftp.gnu.org/gnu/parallel/ GNU Parallel 20160222 ('N977GA') <<[stable]>> has been released. It is available for download at: http://ftp.gnu.org/gnu/parallel/
<<No new functionality was introduced so this is a good candidate for a stable release.>> <<No new functionality was introduced so this is a good candidate for a stable release.>>
@ -243,6 +243,8 @@ for Big Data Applications https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumb
* <<Citation needed: Introspecting for RSA Key Material to Assist Intrusion Detection http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=7331177&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D7331177>> * <<Citation needed: Introspecting for RSA Key Material to Assist Intrusion Detection http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=7331177&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D7331177>>
* Daily Quick Tip: parallel https://medium.com/@jacaetevha/daily-quick-tip-parallel-8772435f6582
* Downloading a list of URLs http://blog.gypsydave5.com/2016/02/04/xargs-and-curl/ * Downloading a list of URLs http://blog.gypsydave5.com/2016/02/04/xargs-and-curl/
* Using GNU Parallel and ImageMagick for Conversions http://blog.fractalcomic.com/tutorials/using-gnu-parallel-and-imagemagick-for-conversions/ * Using GNU Parallel and ImageMagick for Conversions http://blog.fractalcomic.com/tutorials/using-gnu-parallel-and-imagemagick-for-conversions/

6
src/parallel
View file

@ -1182,6 +1182,7 @@ sub init_globals {
$Global::debug = 0; $Global::debug = 0;
$Global::verbose = 0; $Global::verbose = 0;
$Global::quoting = 0; $Global::quoting = 0;
$Global::total_completed = 0;
# Read only table with default --rpl values # Read only table with default --rpl values
%Global::replace = %Global::replace =
( (
@ -1460,6 +1461,7 @@ sub open_joblog {
if(/$joblog_regexp/o) { if(/$joblog_regexp/o) {
# This is 30% faster than set_job_already_run($1); # This is 30% faster than set_job_already_run($1);
vec($Global::job_already_run,($1||0),1) = 1; vec($Global::job_already_run,($1||0),1) = 1;
$Global::total_completed++;
$group[$1-1] = "true"; $group[$1-1] = "true";
} elsif(/(\d+)\s+\S+(\s+[-0-9.]+){6}\s+(.*)$/) { } elsif(/(\d+)\s+\S+(\s+[-0-9.]+){6}\s+(.*)$/) {
$group[$1-1] = $3 $group[$1-1] = $3
@ -1495,6 +1497,7 @@ sub open_joblog {
if(/$joblog_regexp/o) { if(/$joblog_regexp/o) {
# This is 30% faster than set_job_already_run($1); # This is 30% faster than set_job_already_run($1);
vec($Global::job_already_run,($1||0),1) = 1; vec($Global::job_already_run,($1||0),1) = 1;
$Global::total_completed++;
} elsif(not /\d+\s+[^\s]+\s+([-0-9.]+\s+){6}/) { } elsif(not /\d+\s+[^\s]+\s+([-0-9.]+\s+){6}/) {
::error("Format of '$opt::joblog' is wrong: $_"); ::error("Format of '$opt::joblog' is wrong: $_");
::wait_and_exit(255); ::wait_and_exit(255);
@ -2026,7 +2029,6 @@ sub init_run_jobs {
# Returns: N/A # Returns: N/A
$Global::total_running = 0; $Global::total_running = 0;
$Global::total_started = 0; $Global::total_started = 0;
$Global::total_completed = 0;
$Global::tty_taken = 0; $Global::tty_taken = 0;
$SIG{USR1} = \&list_running_jobs; $SIG{USR1} = \&list_running_jobs;
$SIG{USR2} = \&toggle_progress; $SIG{USR2} = \&toggle_progress;
@ -4747,7 +4749,7 @@ sub loadavg_too_high {
'ultrix' => "ps -ax | awk '{print \$3,\$5}'", 'ultrix' => "ps -ax | awk '{print \$3,\$5}'",
); );
print `$ps{$^O}`; print `$ps{$^O}`;
}; });
$cmd = "perl -e ".::shell_quote_scalar($ps); $cmd = "perl -e ".::shell_quote_scalar($ps);
} }
return $cmd; return $cmd;

8
src/parallel.pod
View file

@ -2420,6 +2420,14 @@ Find the files in a list that do not exist
cat file_list | parallel 'if [ ! -e {} ] ; then echo {}; fi' cat file_list | parallel 'if [ ! -e {} ] ; then echo {}; fi'
=head1 EXAMPLE: Composed command with multiple input sources
You have a dir with files named as 24 hours in 5 minute intervals:
00:00, 00:05, 00:10 .. 23:55. You want to find the files missing:
parallel [ -f {1}:{2} ] "||" echo {1}:{2} does not exist ::: {00..23} ::: {00..55..5}
=head1 EXAMPLE: Calling Bash functions =head1 EXAMPLE: Calling Bash functions
If the composed command is longer than a line, it becomes hard to If the composed command is longer than a line, it becomes hard to

4
src/parallel_design.pod
View file

@ -754,8 +754,8 @@ Unfortunately it is not always possible to predict the root cause of the error.
Contrary to the obvious B<--load> does not use load average. This is Contrary to the obvious B<--load> does not use load average. This is
due to load average rising too slowly. Instead it uses B<ps> to list due to load average rising too slowly. Instead it uses B<ps> to list
the number of jobs in running or blocked state (state D, O or R). This the number of threads in running or blocked state (state D, O or
gives an instant load. R). This gives an instant load.
As remote calculation of load can be slow, a process is spawned to run As remote calculation of load can be slow, a process is spawned to run
B<ps> and put the result in a file, which is then used next time. B<ps> and put the result in a file, which is then used next time.

4
src/parallel_tutorial.html
View file

@ -590,6 +590,10 @@
<pre><code> parallel echo {} shell quoted is {= &#39;$_=Q($_)&#39; =} ::: &#39;*/!#$&#39;</code></pre> <pre><code> parallel echo {} shell quoted is {= &#39;$_=Q($_)&#39; =} ::: &#39;*/!#$&#39;</code></pre>
<p>Output:</p>
<pre><code> */!#$ shell quoted is \*/\!\#\$</code></pre>
<p><b>$job-</b>&gt;<b>skip()</b> skips the job:</p> <p><b>$job-</b>&gt;<b>skip()</b> skips the job:</p>
<pre><code> parallel echo {= &#39;if($_==3) { $job-&gt;skip() }&#39; =} ::: {1..5}</code></pre> <pre><code> parallel echo {= &#39;if($_==3) { $job-&gt;skip() }&#39; =} ::: {1..5}</code></pre>

4
src/parallel_tutorial.pod
View file

@ -477,6 +477,10 @@ B<Q(...)> shell quotes the string:
parallel echo {} shell quoted is {= '$_=Q($_)' =} ::: '*/!#$' parallel echo {} shell quoted is {= '$_=Q($_)' =} ::: '*/!#$'
Output:
*/!#$ shell quoted is \*/\!\#\$
B<$job->>B<skip()> skips the job: B<$job->>B<skip()> skips the job:
parallel echo {= 'if($_==3) { $job->skip() }' =} ::: {1..5} parallel echo {= 'if($_==3) { $job->skip() }' =} ::: {1..5}

38
testsuite/tests-to-run/parallel-local-0.3s.sh
View file

@ -125,26 +125,26 @@ echo 'bug #41613: --compress --line-buffer - no newline';
parallel --line-buffer echo {} O ::: -n; echo "K" parallel --line-buffer echo {} O ::: -n; echo "K"
echo 'Compress with failing (de)compressor' echo 'Compress with failing (de)compressor'
parallel -k --tag --compress --compress-program 'cat;true' --decompress-program 'cat;true' echo ::: tag true true parallel -k --tag --compress --compress-program 'cat;true' --decompress-program 'cat;true' echo ::: tag true true
parallel -k --tag --compress --compress-program 'cat;false' --decompress-program 'cat;true' echo ::: tag false true parallel -k --tag --compress --compress-program 'cat;false' --decompress-program 'cat;true' echo ::: tag false true
parallel -k --tag --compress --compress-program 'cat;false' --decompress-program 'cat;false' echo ::: tag false false parallel -k --tag --compress --compress-program 'cat;false' --decompress-program 'cat;false' echo ::: tag false false
parallel -k --tag --compress --compress-program 'cat;true' --decompress-program 'cat;false' echo ::: tag true false parallel -k --tag --compress --compress-program 'cat;true' --decompress-program 'cat;false' echo ::: tag true false
parallel -k --compress --compress-program 'cat;true' --decompress-program 'cat;true' echo ::: true true parallel -k --compress --compress-program 'cat;true' --decompress-program 'cat;true' echo ::: true true
parallel -k --compress --compress-program 'cat;false' --decompress-program 'cat;true' echo ::: false true parallel -k --compress --compress-program 'cat;false' --decompress-program 'cat;true' echo ::: false true
parallel -k --compress --compress-program 'cat;false' --decompress-program 'cat;false' echo ::: false false parallel -k --compress --compress-program 'cat;false' --decompress-program 'cat;false' echo ::: false false
parallel -k --compress --compress-program 'cat;true' --decompress-program 'cat;false' echo ::: true false parallel -k --compress --compress-program 'cat;true' --decompress-program 'cat;false' echo ::: true false
parallel -k --line-buffer --compress --compress-program 'cat;true' --decompress-program 'cat;true' echo ::: line-buffer true true parallel -k --line-buffer --compress --compress-program 'cat;true' --decompress-program 'cat;true' echo ::: line-buffer true true
parallel -k --line-buffer --compress --compress-program 'cat;false' --decompress-program 'cat;true' echo ::: line-buffer false true parallel -k --line-buffer --compress --compress-program 'cat;false' --decompress-program 'cat;true' echo ::: line-buffer false true
parallel -k --line-buffer --compress --compress-program 'cat;false' --decompress-program 'cat;false' echo ::: line-buffer false false parallel -k --line-buffer --compress --compress-program 'cat;false' --decompress-program 'cat;false' echo ::: line-buffer false false
parallel -k --tag --line-buffer --compress --compress-program 'cat;true' --decompress-program 'cat;false' echo ::: tag line-buffer true false parallel -k --tag --line-buffer --compress --compress-program 'cat;true' --decompress-program 'cat;false' echo ::: tag line-buffer true false
parallel -k --tag --line-buffer --compress --compress-program 'cat;true' --decompress-program 'cat;true' echo ::: tag line-buffer true true parallel -k --tag --line-buffer --compress --compress-program 'cat;true' --decompress-program 'cat;true' echo ::: tag line-buffer true true
parallel -k --tag --line-buffer --compress --compress-program 'cat;false' --decompress-program 'cat;true' echo ::: tag line-buffer false true parallel -k --tag --line-buffer --compress --compress-program 'cat;false' --decompress-program 'cat;true' echo ::: tag line-buffer false true
parallel -k --tag --line-buffer --compress --compress-program 'cat;false' --decompress-program 'cat;false' echo ::: tag line-buffer false false parallel -k --tag --line-buffer --compress --compress-program 'cat;false' --decompress-program 'cat;false' echo ::: tag line-buffer false false
parallel -k --tag --line-buffer --compress --compress-program 'cat;true' --decompress-program 'cat;false' echo ::: tag line-buffer true false parallel -k --tag --line-buffer --compress --compress-program 'cat;true' --decompress-program 'cat;false' echo ::: tag line-buffer true false
parallel -k --files --compress --compress-program 'cat;true' --decompress-program 'cat;true' echo ::: files true true | parallel rm parallel -k --files --compress --compress-program 'cat;true' --decompress-program 'cat;true' echo ::: files true true | parallel rm
parallel -k --files --compress --compress-program 'cat;false' --decompress-program 'cat;true' echo ::: files false true | parallel rm parallel -k --files --compress --compress-program 'cat;false' --decompress-program 'cat;true' echo ::: files false true | parallel rm
parallel -k --files --compress --compress-program 'cat;false' --decompress-program 'cat;false' echo ::: files false false | parallel rm parallel -k --files --compress --compress-program 'cat;false' --decompress-program 'cat;false' echo ::: files false false | parallel rm
parallel -k --files --compress --compress-program 'cat;true' --decompress-program 'cat;false' echo ::: files true false | parallel rm parallel -k --files --compress --compress-program 'cat;true' --decompress-program 'cat;false' echo ::: files true false | parallel rm
echo 'bug #44250: pxz complains File format not recognized but decompresses anyway' echo 'bug #44250: pxz complains File format not recognized but decompresses anyway'
# The first line dumps core if run from make file. Why?! # The first line dumps core if run from make file. Why?!
@ -263,6 +263,14 @@ echo 'bug #47002: --tagstring with -d \n\n'
echo '**' echo '**'
echo 'bug #47086: [PATCH] Initialize total_completed from joblog'
rm -f /tmp/parallel-47086;
parallel -j1 --joblog /tmp/parallel-47086 --halt now,fail=1 echo '{= $_=$Global::total_completed =};exit {}' ::: 0 0 0 1 0 0;
parallel -j1 --joblog /tmp/parallel-47086 --halt now,fail=1 --resume echo '{= $_=$Global::total_completed =};exit {}' ::: 0 0 0 1 0 0
echo '**'
EOF EOF
echo '### 1 .par file from --files expected' echo '### 1 .par file from --files expected'

43
testsuite/wanted-results/parallel-local-0.3s
View file

@ -168,56 +168,56 @@ nopipe nocompress notagstring
OK OK
echo 'Compress with failing (de)compressor' echo 'Compress with failing (de)compressor'
Compress with failing (de)compressor Compress with failing (de)compressor
parallel -k --tag --compress --compress-program 'cat;true' --decompress-program 'cat;true' echo ::: tag true true parallel -k --tag --compress --compress-program 'cat;true' --decompress-program 'cat;true' echo ::: tag true true
tag tag tag tag
true true true true
true true true true
parallel -k --tag --compress --compress-program 'cat;false' --decompress-program 'cat;true' echo ::: tag false true parallel -k --tag --compress --compress-program 'cat;false' --decompress-program 'cat;true' echo ::: tag false true
tag tag tag tag
false false false false
true true true true
parallel: Error: cat;false failed. parallel: Error: cat;false failed.
parallel: Error: cat;false failed. parallel: Error: cat;false failed.
parallel: Error: cat;false failed. parallel: Error: cat;false failed.
parallel -k --tag --compress --compress-program 'cat;false' --decompress-program 'cat;false' echo ::: tag false false parallel -k --tag --compress --compress-program 'cat;false' --decompress-program 'cat;false' echo ::: tag false false
tag tag tag tag
false false false false
false false false false
parallel: Error: cat;false failed. parallel: Error: cat;false failed.
parallel: Error: cat;false failed. parallel: Error: cat;false failed.
parallel: Error: cat;false failed. parallel: Error: cat;false failed.
parallel -k --tag --compress --compress-program 'cat;true' --decompress-program 'cat;false' echo ::: tag true false parallel -k --tag --compress --compress-program 'cat;true' --decompress-program 'cat;false' echo ::: tag true false
tag tag tag tag
true true true true
false false false false
parallel -k --compress --compress-program 'cat;true' --decompress-program 'cat;true' echo ::: true true parallel -k --compress --compress-program 'cat;true' --decompress-program 'cat;true' echo ::: true true
true true
true true
parallel -k --compress --compress-program 'cat;false' --decompress-program 'cat;true' echo ::: false true parallel -k --compress --compress-program 'cat;false' --decompress-program 'cat;true' echo ::: false true
false false
true true
parallel: Error: cat;false failed. parallel: Error: cat;false failed.
parallel: Error: cat;false failed. parallel: Error: cat;false failed.
parallel -k --compress --compress-program 'cat;false' --decompress-program 'cat;false' echo ::: false false parallel -k --compress --compress-program 'cat;false' --decompress-program 'cat;false' echo ::: false false
false false
false false
parallel: Error: cat;false failed. parallel: Error: cat;false failed.
parallel: Error: cat;false failed. parallel: Error: cat;false failed.
parallel -k --compress --compress-program 'cat;true' --decompress-program 'cat;false' echo ::: true false parallel -k --compress --compress-program 'cat;true' --decompress-program 'cat;false' echo ::: true false
true true
false false
parallel -k --line-buffer --compress --compress-program 'cat;true' --decompress-program 'cat;true' echo ::: line-buffer true true parallel -k --line-buffer --compress --compress-program 'cat;true' --decompress-program 'cat;true' echo ::: line-buffer true true
line-buffer line-buffer
true true
true true
parallel -k --line-buffer --compress --compress-program 'cat;false' --decompress-program 'cat;true' echo ::: line-buffer false true parallel -k --line-buffer --compress --compress-program 'cat;false' --decompress-program 'cat;true' echo ::: line-buffer false true
line-buffer line-buffer
false false
true true
parallel: Error: cat;false failed. parallel: Error: cat;false failed.
parallel: Error: cat;false failed. parallel: Error: cat;false failed.
parallel: Error: cat;false failed. parallel: Error: cat;false failed.
parallel -k --line-buffer --compress --compress-program 'cat;false' --decompress-program 'cat;false' echo ::: line-buffer false false parallel -k --line-buffer --compress --compress-program 'cat;false' --decompress-program 'cat;false' echo ::: line-buffer false false
line-buffer line-buffer
false false
false false
@ -257,16 +257,16 @@ tag tag
line-buffer line-buffer line-buffer line-buffer
true true true true
false false false false
parallel -k --files --compress --compress-program 'cat;true' --decompress-program 'cat;true' echo ::: files true true | parallel rm parallel -k --files --compress --compress-program 'cat;true' --decompress-program 'cat;true' echo ::: files true true | parallel rm
parallel -k --files --compress --compress-program 'cat;false' --decompress-program 'cat;true' echo ::: files false true | parallel rm parallel -k --files --compress --compress-program 'cat;false' --decompress-program 'cat;true' echo ::: files false true | parallel rm
parallel: Error: cat;false failed. parallel: Error: cat;false failed.
parallel: Error: cat;false failed. parallel: Error: cat;false failed.
parallel: Error: cat;false failed. parallel: Error: cat;false failed.
parallel -k --files --compress --compress-program 'cat;false' --decompress-program 'cat;false' echo ::: files false false | parallel rm parallel -k --files --compress --compress-program 'cat;false' --decompress-program 'cat;false' echo ::: files false false | parallel rm
parallel: Error: cat;false failed. parallel: Error: cat;false failed.
parallel: Error: cat;false failed. parallel: Error: cat;false failed.
parallel: Error: cat;false failed. parallel: Error: cat;false failed.
parallel -k --files --compress --compress-program 'cat;true' --decompress-program 'cat;false' echo ::: files true false | parallel rm parallel -k --files --compress --compress-program 'cat;true' --decompress-program 'cat;false' echo ::: files true false | parallel rm
echo 'bug #44250: pxz complains File format not recognized but decompresses anyway' echo 'bug #44250: pxz complains File format not recognized but decompresses anyway'
bug #44250: pxz complains File format not recognized but decompresses anyway bug #44250: pxz complains File format not recognized but decompresses anyway
# The first line dumps core if run from make file. Why?! # The first line dumps core if run from make file. Why?!
@ -575,5 +575,18 @@ bug #47002: --tagstring with -d \n\n
2 2
echo '**' echo '**'
** **
echo 'bug #47086: [PATCH] Initialize total_completed from joblog'
bug #47086: [PATCH] Initialize total_completed from joblog
rm -f /tmp/parallel-47086; parallel -j1 --joblog /tmp/parallel-47086 --halt now,fail=1 echo '{= $_=$Global::total_completed =};exit {}' ::: 0 0 0 1 0 0; parallel -j1 --joblog /tmp/parallel-47086 --halt now,fail=1 --resume echo '{= $_=$Global::total_completed =};exit {}' ::: 0 0 0 1 0 0
0
1
2
3
4
5
parallel: This job failed:
echo 3;exit 1
echo '**'
**
### 1 .par file from --files expected ### 1 .par file from --files expected
1 1