mirror of
https://git.savannah.gnu.org/git/parallel.git
synced 2024-10-18 12:59:39 +00:00
1629 lines
54 KiB
Plaintext
1629 lines
54 KiB
Plaintext
20140322
|
||
|
||
* Offical package for Alpine Linux now exists:
|
||
http://alpinelinux.org/apk/main/x86/parallel
|
||
|
||
* GNU Parallel was cited in: Estimating uncertainties in statistics
|
||
computed from direct numerical simulation
|
||
http://dx.doi.org/10.1063/1.4866813
|
||
|
||
* GNU Parallel was cited in: Vehicle routing problems with time
|
||
windows and multiple service workers: a systematic comparison
|
||
between ACO and GRASP
|
||
http://link.springer.com/article/10.1007/s10100-014-0341-z
|
||
|
||
* GNU Parallel, quick and easy
|
||
http://www.reganmian.net/blog/2014/03/09/gnu-parallel-quick-and-easy/
|
||
|
||
* Using Multiple CPU Cores With Command Line Tools
|
||
http://prateekvjoshi.com/2014/03/07/using-multiple-cpu-cores-with-command-line-tools/
|
||
|
||
* Batch convert mp3 CBR to VBR
|
||
http://blogs.zercle.com/bouroo/2013/03/batch-convert-mp3-cbr-to-vbr/
|
||
|
||
* Running Behat in parallel with Selenium grid
|
||
http://zalas.eu/running-behat-in-parallel-with-selenium-grid/
|
||
|
||
* Git-Mirror per cronjob aktuell halten
|
||
https://www.schtobia.de/2013/git-mirror-per-cronjob-aktuell-halten/
|
||
|
||
* Пишем многопоточный скрипт на bash
|
||
http://dafter.ru/duf/howto/130.html
|
||
|
||
* 如何利用多核CPU来加速你的Linux命令
|
||
http://www.chinadmd.com/file/xpcr6wpctzwtuu3ivwiaatcs_1.html
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20140222
|
||
|
||
New in this release:
|
||
|
||
* --tollef has been retired.
|
||
|
||
* --compress has be redesigned due to bugs.
|
||
|
||
* Format of --eta and --joblog has changed slightly.
|
||
|
||
* GNU Parallel was cited in: Speaker Verification and Keyword Spotting
|
||
Systems for Forensic Applications
|
||
http://lantana.tenet.res.in/website_files/thesis/Phd/srikanth.pdf
|
||
|
||
* GNU Parallel was cited in: Scaleable Code Clone Detection
|
||
http://scg.unibe.ch/archive/phd/schwarz-phd.pdf
|
||
|
||
* GNU Parallel was cited in: Clone Detection that scales
|
||
http://scg.unibe.ch/archive/masters/Vogt14a.pdf
|
||
|
||
* GNU Parallel was cited in: Scalable Positional Analysis for Studying
|
||
Evolution of Nodes in Networks
|
||
http://arxiv-web3.library.cornell.edu/pdf/1402.3797
|
||
|
||
* GNU Parallel was cited in: Aggregate Characterization of User
|
||
Behavior in Twitter and Analysis of the Retweet Graph
|
||
http://arxiv.org/pdf/1402.2671v1
|
||
|
||
* GNU Parallel was used (unfortunately without citation) in: Causal
|
||
Network Models for Predicting Compound Targets and Driving Pathways
|
||
in Cancer
|
||
http://jbx.sagepub.com/content/early/2014/02/10/1087057114522690.full
|
||
|
||
* GNU Parallel was used (unfortunately without citation) in:
|
||
Performance and Energy Efficiency of Common Compression /
|
||
Decompression Utilities
|
||
http://www.researchgate.net/publication/243962643_Performance_and_Energy_Efficiency_of_Common_CompressionDecompression_Utilities_An_Experimental_Study_in_Mobile_and_Workstation_Computer_Platforms/file/3deec51d1dbc0474f9.pdf
|
||
|
||
* GNU Parallel was used (unfortunately without citation) in: SpringFS:
|
||
Bridging Agility and Performance in Elastic Distributed Storage
|
||
http://www.istc-cc.cmu.edu/publications/papers/2014/SpringFS-FAST14.pdf
|
||
|
||
* GNU Parallel was recommended (without citation) in: Redesigning the
|
||
Specificity of Protein-DNA Interactions with Rosetta
|
||
http://link.springer.com/protocol/10.1007/978-1-62703-968-0_17
|
||
|
||
* GNU Parallel is co-distributed with RepeatExplorer
|
||
http://www.vcru.wisc.edu/simonlab/bioinformatics/programs/repeatexplorer/README.txt
|
||
|
||
* Convert MP3 to a scrolling spectrum waterfall plot video
|
||
http://jdesbonnet.blogspot.dk/2014/02/convert-mp3-to-scrolling-spectrum.html
|
||
|
||
* PHP wrapper class for the GNU Parallel tool
|
||
https://github.com/geopal-solutions/gnu-parallel-wrapper/tree/master
|
||
|
||
* Exploratory Data Analysis http://www.slideshare.net/thinrhino/gnunify
|
||
|
||
* Copias de archivos con rsync y parallel
|
||
http://linuxmanr4.com/2014/02/20/copias-de-archivos-con-rsync-y-parallel
|
||
|
||
* Single-Thread-Programme auf Multicore-Rechnern parallelisieren
|
||
http://www.adlerweb.info/blog/2014/02/08/linux-single-thread-programme-auf-multicore-rechnern-parallelisieren
|
||
|
||
* Распараллеливаем процессы для ускорения вычислений и выполнения
|
||
заданий в Linux http://savepearlharbor.com/?p=210480
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20140122
|
||
|
||
* GNU Parallel was cited in: On the likelihood of multiple bit upsets
|
||
in logic circuits
|
||
http://arxiv-web3.library.cornell.edu/pdf/1401.1003
|
||
|
||
* HaploClique uses GNU Parallel
|
||
https://github.com/armintoepfer/haploclique
|
||
|
||
* Scraping NSScreencast
|
||
https://blog.nicolai86.eu/posts/2014-01-12/scraping-nsscreencast/
|
||
|
||
* 30 Cool Open Source Software I Discovered in 2013
|
||
http://www.cyberciti.biz/open-source/30-cool-best-open-source-softwares-of-2013/
|
||
|
||
* [Unix] 13. The power of doing things in Parallel
|
||
http://leetaey.tistory.com/384
|
||
|
||
* Parallel the execution of a job that read from stdin
|
||
http://www.linuxask.com/questions/parallel-the-execution-of-a-job-that-read-from-stdin
|
||
|
||
* Mon Make à moi (6:38-11:50)
|
||
http://videos.rennes.inria.fr/ReNaBI-GO2013/indexPierreLindenbaum.html
|
||
|
||
* Shell-Abarbeitung beschleunigen: Wie Sie mit parallelen Prozesse
|
||
effizienter in der Shell arbeiten
|
||
https://www.hosteurope.ch/blog/shell-abarbeitung-beschleunigen-wie-sie-mit-parallelen-prozesse-effizienter-in-der-shell-arbeiten/
|
||
|
||
* Summary of GNU Parallel tutorial
|
||
http://hacktracking.blogspot.dk/2014/01/gnu-parallel-tutorial.html
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20131222
|
||
|
||
* GNU Parallel now has a motto: For people who live life in the
|
||
parallel lane.
|
||
|
||
* Detect if the buffer dir $TMPDIR runs out of space. This changes a
|
||
central piece of code, making this release alpha quality.
|
||
|
||
* --keep-order --(n)onall will sort according to the sshlogin.
|
||
|
||
* Detect the number of CPUs on DEC Tru64.
|
||
|
||
* GNU sql: --list-tables for sqlite3.
|
||
|
||
* GNU Parallel was cited in: Variational Message-Passing: Extension to
|
||
Continuous Variables and Applications in Multi-Target Tracking
|
||
http://www.ics.uci.edu/~ajfrank/pubs/thesis.pdf
|
||
|
||
* GNU Parallel was cited in: Genome Assembly: Scaffolding Guided by
|
||
Related Genomes
|
||
https://www.duo.uio.no/handle/10852/37431
|
||
|
||
* GNU Parallel was cited in: Metagenomic analysis of Mariana Trench
|
||
sediment samples
|
||
http://repositorio.ul.pt/handle/10451/9436
|
||
|
||
* GNU Parallel was cited in: Diacritization, automatic segmentation
|
||
and labeling for Levantine Arabic speech
|
||
http://ieeexplore.ieee.org/xpl/articleDetails.jsp?tp=&arnumber=6642556
|
||
|
||
* GNU Parallel was cited in: Investigation of emotion classification
|
||
using speech rhythm metrics
|
||
http://ieeexplore.ieee.org/xpl/articleDetails.jsp?tp=&arnumber=6642591
|
||
|
||
* GNU Parallel was cited in: A Rhythm-Based Analysis of Arabic Native
|
||
and Non-Native Speaking Styles
|
||
http://www.ijsps.com/uploadfile/2013/1128/20131128014413279.pdf
|
||
|
||
* GNU Parallel was cited in: Evolutionary optimization of wetlands
|
||
design
|
||
http://dl.acm.org/citation.cfm?id=2480400
|
||
|
||
* GNU Parallel was cited (unfortunately with wrong citation) in:
|
||
RDBMS vs NoSQL: Performance and Scaling Comparison
|
||
http://www.epcc.ed.ac.uk/sites/default/files/Dissertations/2012-2013/RDBMS%20vs%20NoSQL%20-%20Performance%20and%20Scaling%20Comparison.pdf
|
||
|
||
* GNU Parallel was used (without proper citation) in: Comprehensive
|
||
Analysis Pipeline for Discovery of Human Genetic Variation
|
||
http://www.google.com/patents/US20130311106
|
||
|
||
* Encrypt and decrypt in parallel with GPG
|
||
http://blog.tkassembled.com/412/encrypt-and-decrypt-in-parallel-with-gpg/
|
||
|
||
* Using GNU Parallel to create a SVM classifier
|
||
http://ivanyu.me/blog/2013/12/01/parallel/
|
||
|
||
* Parallel xz compression
|
||
http://anthon.home.xs4all.nl/rants/2013/parallel_xz/
|
||
|
||
* Got CPUs to burn? Put 'em to work with GNU parallel
|
||
http://baoilleach.blogspot.dk/2013/11/got-cpus-to-burn-put-em-to-work-with.html
|
||
|
||
* Parallel rsync
|
||
http://pastebin.com/JmnB9ffq
|
||
|
||
* Gnu Parallel for fun and profit
|
||
https://gist.github.com/celoyd/f7eb55ad69c9b33fd8c3
|
||
|
||
* Parallelization of Exploitation
|
||
http://rantsideasstuff.com/parallelization-of-exploitation/
|
||
|
||
* Running assemblies in parallel
|
||
https://khmer-protocols.readthedocs.org/en/v0.8.3/metagenomics/4-assemble.html
|
||
|
||
* Procesando la contabilidad del PP
|
||
http://www.neorazorx.com/2013/07/procesando-la-contabilidad-del-pp.html
|
||
|
||
* Mon make à moi (French)
|
||
http://www.slideshare.net/lindenb/mon-make-moi-tout-sauf-galaxy
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20131122
|
||
|
||
* A citation notice is printed on stderr only if stderr is a terminal,
|
||
the user has not specified --no-notice and the user has not run
|
||
--bibtex once. This makes the release alpha quality.
|
||
|
||
* --compress will compress temporary files. If the output is big and
|
||
very compressible this will take up less disk space in $TMPDIR and
|
||
possibly be faster due to less disk I/O.
|
||
|
||
* --compress-program comtrols which program to use for compressing
|
||
temporary files.
|
||
|
||
* --bar show progress as a progress bar compatible with zenity.
|
||
|
||
* --resume can now be used with --result: Jobs already run will be
|
||
skipped.
|
||
|
||
* --transfer and --basefile support paths relative to the --workdir by
|
||
inserting /./ into the path.
|
||
|
||
* GNU Parallel was cited in: 'fastphylo: Fast tools for phylogenetics'
|
||
http://www.biomedcentral.com/1471-2105/14/334/abstract
|
||
|
||
* Using GNU parallel
|
||
http://davetang.org/muse/2013/11/18/using-gnu-parallel/
|
||
|
||
* Techlux - GNU - Parallel (German)
|
||
https://techlux.de/blog/2013/11/07/gnu-parallel/
|
||
|
||
* awk, sed, bzip2, grep, wc на всех ядрах
|
||
http://vk.com/page-30666517_45528467
|
||
|
||
* 如何利用多核CPU來加速你的Linux命令 — awk, sed, bzip2, grep, wc等
|
||
http://www.hksilicon.com/kb/articles/290543/CPULinuxawk-sed-bzip2-grep-wc
|
||
|
||
* GNU Parallel (Japanese)
|
||
http://jarp.does.notwork.org/diary/201311b.html#20131117
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20131022
|
||
|
||
* --transfer files with /./ in the path will copy the files relative
|
||
to the --workdir.
|
||
|
||
* The maximal command length is now cached in a file halfing the
|
||
startup time.
|
||
|
||
* GNU Parallel was cited in: Investigating speaker gender using rhythm
|
||
metrics in Arabic dialects
|
||
http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=6602389
|
||
|
||
* GNU Parallel was cited in: Tiedon erist ̈minen ja visualisointi a
|
||
lastensuojelun asiakaskertomuksista
|
||
https://tampub.uta.fi/handle/10024/84978
|
||
|
||
* Some Additional Useful Bioinformatics Software
|
||
http://nix-bio.blogspot.dk/2013/10/additional-software-for-bioinformatics.html
|
||
|
||
* GNU Parallel Example Screen Cast (bil to Geotiff conversion)
|
||
http://www.youtube.com/watch?v=PpF27eC7Mys
|
||
|
||
* Using GNU Parallel to roll-your-own Map Reduce!
|
||
http://www.rankfocus.com/hello-world/
|
||
|
||
* Use multiple CPU Cores with your Linux commands
|
||
http://www.rankfocus.com/use-cpu-cores-linux-commands/
|
||
|
||
* Using GNU Parallel at HPC @ Uni.lu
|
||
https://hpc.uni.lu/users/use_cases/
|
||
|
||
* Scaling up with parallelization
|
||
https://www.msi.umn.edu/sites/default/files/AdvPython_1.pdf
|
||
|
||
* How to teach yourself to be a technical tester
|
||
http://infiniteundo.com/post/62421363463/qa-toc
|
||
|
||
* Faster Rasters For All
|
||
http://2013.foss4g.org/conf/programme/presentations/52/
|
||
|
||
* Optimizing translated file downloads
|
||
http://www.smartling.com/blog/2013/05/20/optimizing-translated-file-downloads/
|
||
|
||
* EMC NetWorker parallel saveset cloning with nsrclone and GNU parallel
|
||
http://www.beebotech.com.au/2013/10/emc-networker-parallel-saveset-cloning-with-nsrclone-and-gnu-parallel/
|
||
|
||
* Best Practices for Amazon EMR
|
||
http://media.amazonwebservices.com/AWS_Amazon_EMR_Best_Practices.pdf
|
||
|
||
* Computing checksums for backup
|
||
https://identi.ca/evan/note/6yf1GzAARtyBhj__xzMvAg
|
||
|
||
* Un exemplu de utilizare a programului GNU parallel
|
||
http://vundicind.blogspot.dk/2013/10/exemplu-utilizare-gnu-parallel.html
|
||
|
||
* 平行化你的工作
|
||
http://www.slideshare.net/drakeguan/part1-23705978
|
||
|
||
* なんか気持ちよかです Gnu Parallel
|
||
http://a3-lab.jpn.org/note/%E3%81%AA%E3%82%93%E3%81%8B%E6%B0%97%E6%8C%81%E3%81%A1%E3%82%88%E3%81%8B%E3%81%A7%E3%81%99-gnu-parallel
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20130922
|
||
|
||
* PDF-files of documentation is now included.
|
||
|
||
* Cloning at warp speed 100 VMs in 1 minute using one command line
|
||
http://www.beebotech.com.au/tag/command-line/
|
||
|
||
* Calcul Quebec's introduction to GNU Parallel
|
||
https://wiki.calculquebec.ca/w/GNU_parallel
|
||
|
||
* Brincando com o GNU parallel
|
||
http://blog.tadeucruz.com/2013/09/brincando-com-o-gnu-parallel/
|
||
|
||
* GNU Parallelで並列処理を行う
|
||
http://qiita.com/PENGUINANA_/items/af27a833e835fe17f09b
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20130822
|
||
|
||
* A tutorial showing much of GNU Parallel's functionality. Spend an
|
||
hour walking through the tutorial. Your commandline will love you
|
||
for it. man parallel_tutorial
|
||
|
||
* --line-buffer will buffer output on line basis. --group keeps the
|
||
output together for a whole job. --ungroup allows output to mixup
|
||
with half a line coming from one job and half a line coming from
|
||
another job. --line-buffer fits between these two; it prints a full
|
||
line, but will allow for mixing lines of different jobs.
|
||
|
||
* --record-env records all variables and functions set. Useful to
|
||
record the normal environment before using --env _.
|
||
|
||
* --env _ exports all functions and variables to the remote server
|
||
except for the ones recorded.
|
||
|
||
* New signing key. Due to recommendations from NIST
|
||
http://www.keylength.com/en/4/ the signing key was changed from
|
||
1024D/ID:FFFFFFF1 to 9888R/ID:88888888.
|
||
|
||
* Agalma: an automated phylogenomics workflow
|
||
http://arxiv.org/pdf/1307.6432
|
||
|
||
* Job ad asking for GNU Parallel expertise
|
||
http://seattle.craigslist.org/est/sof/4006079352.html
|
||
|
||
* Transient Beowulf Clustering with GNU Parallel and SSHfs
|
||
http://www.reddit.com/r/linux/comments/1ka8mn/transient_beowulf_clustering_with_gnu_parallel/
|
||
|
||
* Aligning to unique regions
|
||
http://davetang.org/muse/2013/07/22/aligning-to-unique-regions/
|
||
|
||
* Top 10 scripting tricks for basic bioinformatics
|
||
http://students.washington.edu/bowmanjs/wordpress/?p=873
|
||
|
||
* Using GNU Parallel to speed up and simplify data analyzes
|
||
http://www.sergeymarkov.com/blog/2013/07/using-gnu-parallel-to-speed-up-and-simplify-data-analyzes/
|
||
|
||
* Compression of files in parallel using GNU parallel
|
||
http://codextechnicanum.blogspot.dk/2013/07/compression-of-files-in-parallel-using.html
|
||
|
||
* Using GNU Parallel to roll-your-own Map Reduce!
|
||
http://www.rankfocus.com/?p=1
|
||
|
||
* Using GNU Parallel with Amazon
|
||
media.amazonwebservices.com/AWS_Amazon_EMR_Best_Practices.pdf
|
||
|
||
* Some useful comments on GNU Parallel
|
||
https://news.ycombinator.com/item?id=6209767
|
||
|
||
* Using GNU Parallel to count reads mapping to genes for multiple BAMs
|
||
http://drduanehassane.com/blog/sequencing-resources
|
||
|
||
* TimeMachineっぽいバックアップスクリプト
|
||
http://rio.tc/2013/07/timemachine-1.html
|
||
|
||
* GNU ParallelでAndroid NDKの全バージョンを一括ダウンロード
|
||
http://qiita.com/mazgi/items/b10bf0ff3da2045d19ab
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20130722
|
||
|
||
* --round-robin with --pipe will write all blocks to already running
|
||
jobs.
|
||
|
||
* --env can now transfer Bash function for remote execution. That is
|
||
pretty cool!
|
||
|
||
* GNU Parallel was used (unfortunately with improper citation) in:
|
||
Understanding the Impact of E-Commerce Software on the Adoption of Structured Data on the Web
|
||
http://link.springer.com/chapter/10.1007/978-3-642-38366-3_9#page-1
|
||
|
||
* GNU Parallel was used (unfortunately with improper citation) in:
|
||
CWI at TREC 2012, KBA track and Session Track
|
||
http://trec.nist.gov/pubs/trec21/papers/CWI.kba.session.final.pdf
|
||
|
||
* Mitigation of Adverse Effects Caused by Shock Wave Boundary Layer
|
||
Interactions through Optimal Wall Shaping.
|
||
http://arc.aiaa.org/doi/abs/10.2514/6.2013-2653
|
||
|
||
* Using GNU parallel to convert images.
|
||
http://www.brunokim.com.br/blog/?p=18
|
||
|
||
* A quick way to parallelize.
|
||
http://timotheepoisot.fr/2013/07/08/parallel/
|
||
|
||
* GNU Parallel 20130522 ('Rana Plaza') 发布，并行作业执行.
|
||
http://www.open-open.com/news/view/371301
|
||
|
||
* Quite a few bug fixes and man page updates.
|
||
|
||
|
||
20130622
|
||
|
||
* --xapply now recycles arguments if an input source has more
|
||
arguments than others.
|
||
|
||
* The sleep time between jobs is now both increased and decreased
|
||
exponentially.
|
||
|
||
* 10 seconds installation check the signature using GnuPG if GnuPG is
|
||
installed.
|
||
|
||
* Developer job asking for GNU Parallel expertise.
|
||
http://careers.stackoverflow.com/jobs/35562/developer-big-data-geo-and-web-climate-central
|
||
|
||
* A small utility program to run youtube-dl in parallel.
|
||
https://github.com/dlh/youtube-dl-parallel
|
||
|
||
* Parallelizing Freesurfer:
|
||
http://blog.cogneurostats.com/?p=148
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20130522
|
||
|
||
* --ctrlc is now default if --pipe is not used; thus making it
|
||
possible to kill remotely started jobs simply by pressing Ctrl-C.
|
||
|
||
* --timeout 200% now means kill jobs that take more than twice the
|
||
time of the median run time of a job.
|
||
|
||
* Jobs are now distributed round robin when having mulitiple
|
||
--sshlogin instead of filling up one --sshlogin at a time.
|
||
|
||
* niceload: darwin version of --io --mem --swap. Thanks to Anders F
|
||
Björklund.
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20130422
|
||
|
||
* 10 seconds installation: wget -O - pi.dk/3 | sh
|
||
|
||
* HPUX CPU/core detection. Thanks to Javier Tarradas.
|
||
|
||
* CPU detection for HURD, IRIX, SCO OpenServer and (old) HPUX.
|
||
|
||
* --ctrlc will send SIGINT to tasks running on remote computers and
|
||
thus killing them.
|
||
|
||
* --load now uses 'ps' to see immediately see the number of running
|
||
processes instead of 'uptime' thus making --load react much faster.
|
||
|
||
* Testing on Centos 3.9 and FreeBSD 7 revealed a lot of compability
|
||
bugs. Some of these required quite extensive changes making this
|
||
release beta quality.
|
||
|
||
* --retries works with --onall.
|
||
|
||
* The new --load computation now works on FreeBSD 7.
|
||
|
||
* --nice works under tcsh.
|
||
|
||
* GNU Parallel is officially supported in NetBSD.
|
||
http://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc/parallel/parallel/README.html
|
||
|
||
* GNU Parallel is accepted for openSUSE.
|
||
http://software.opensuse.org/package/gnu_parallel
|
||
|
||
* GNU Parallel can be installed under Microsoft Windows (CygWin).
|
||
http://blogs.msdn.com/b/hpctrekker/archive/2013/03/30/preparing-and-uploading-datasets-for-hdinsight.aspx
|
||
|
||
* Excuses for not installing GNU Parallel.
|
||
http://oletange.blogspot.dk/2013/04/why-not-install-gnu-parallel.html
|
||
|
||
* Job advert that requires GNU Parallel competence.
|
||
http://versium.com/about/careers/
|
||
|
||
* Parallelizing Batch Jobs for Fun and Profit.
|
||
http://mikeseidle.com/tech/programming/2013/03/parallelizing-batch-jobs
|
||
|
||
* Processing Transcription Start Sites(TSS) for the entire Mouse genome.
|
||
http://qbrc.swmed.edu/2013/03/gnu-parallel-speeding-up-unix-commands-and-scripts/
|
||
|
||
* GNU parallel is used throughout Scrimer.
|
||
http://scrimer.readthedocs.org/en/latest/
|
||
|
||
* GNU Parallel helped making public documents searchable.
|
||
http://danpalmer.me/blog/articles/2013-04-06-unlocking-hillsborough.html
|
||
|
||
* Identifying big movie files (German).
|
||
http://blackappsolutions.wordpress.com/2013/03/23/wenn-der-plattenplatz-knapp-wird-filme-identifizieren-die-viel-platz-belegen/
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20130222
|
||
|
||
* --resume works with --pipe.
|
||
|
||
* --resume-failed will go through --joblog, redo the failed jobs and
|
||
then continue like --resume.
|
||
|
||
* Negative positional arguments count from the end: {-1} means the
|
||
last argument, {-2} the second to last.
|
||
|
||
* NetBSD CPU detection.
|
||
|
||
* --blocksize increases exponentially if it smaller than a full
|
||
record.
|
||
|
||
* Processing n-line records (--pipe -L n) is now much faster.
|
||
|
||
* --tollef is obsoleted after discussion on the mailing list.
|
||
|
||
* GNU Parallel is the highest rated tool on:
|
||
http://www.biostars.org/show/tools/?sort=votes&since=all%20time
|
||
|
||
* GNU Parallel was loved during FSFE's #ilovefs campaign.
|
||
http://fsfe.org/news/2013/news-20130212-01.en.html
|
||
|
||
* Using GNU Parallel with s3cmd (Japanese).
|
||
http://blog.suz-lab.com/2013/02/s3cmd-gnu-paralells3.html
|
||
|
||
* Intro to GNU Parallel (Chinese).
|
||
http://guiquanz.github.com/2013/02/12/gnu-parallel-intro/
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20130122
|
||
|
||
* --sshdelay Delay starting next ssh by secs seconds. GNU parallel
|
||
will pause secs seconds after starting each ssh. secs can be less
|
||
than 1 seconds.
|
||
|
||
* Official OpenBSD port:
|
||
http://ftp.openbsd.org/ports/sysutils/parallel/
|
||
|
||
* Official DragonFlyBSD package:
|
||
http://www.mirrorservice.org/sites/ftp.dragonflybsd.org/packages/amd64/DragonFly-3.0.0/stable/parallel/
|
||
|
||
* Post about niceload in Hungarian:
|
||
http://commandline.blog.hu/2013/01/02/niceload
|
||
|
||
* 自炊スキャンデータをKobo Gloに最適化
|
||
http://interstadial.wordpress.com/2013/01/20/
|
||
|
||
* GNU parallel+ssh で複数のリモートホストに複数のコマンドを実行させる
|
||
http://oshiire.to/archives/1686
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20121222
|
||
|
||
* --shebang-wrap makes is possible to parallelize scripts by putting
|
||
parallel in the shebang line (the first #! line in scripts).
|
||
|
||
* --delay puts in a delay after starting each job. Useful to avoid
|
||
raceconditions and thundering herd problems.
|
||
|
||
* --results now uses / as separator instead of TAB and thus creates
|
||
subdirs. Thanks to Drew Frank.
|
||
|
||
* parallel: OpenBSD CPU detection. Thanks to Michał Markowski.
|
||
|
||
* niceload --sensor allows for negative numbers to signify a lower
|
||
number means the limit is lower.
|
||
|
||
* niceload --prg suspend a named program that is already running.
|
||
|
||
* niceload multiple --pid suspend process ids that are already running.
|
||
|
||
* Convertir des images avec find et imagemagick
|
||
http://roger.b.over-blog.com/article-convertir-des-images-avec-find-et-imagemagick-113416637.html
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20121122
|
||
|
||
* niceload --sensor makes it possible to use a program to control if a
|
||
program should be suspended.
|
||
|
||
* GNU Parallel was part of the justification for Nordic Free Software
|
||
Award 2012
|
||
https://fsfe.org/news/2012/news-20121112-01.html
|
||
|
||
* Dateien schneller packen und konvertieren mit GNU parallel
|
||
http://www.linux-community.de/Archiv/Tipp-der-Woche/Dateien-schneller-packen-und-konvertieren-mit-GNU-parallel
|
||
|
||
* Processing images from Lunar Reconnaissance Orbiter Narrow Angle
|
||
Camera http://lunokhod.org/?p=774
|
||
|
||
* GNU Parallel is used in types2: Type and Hapax Accumulation Curves
|
||
http://www.cs.helsinki.fi/u/josuomel/types2/
|
||
|
||
* GNU Parallel used in: A Hybrid Factor Analysis and Probabilistic
|
||
PCA-based system for Dictionary Learning and Encoding for Robust
|
||
Speaker Recognition
|
||
http://www.superlectures.com/odyssey2012/lecture.php?id=8&lang=en
|
||
http://www.cse.iitm.ac.in/~mrsri/ppca.pdf
|
||
|
||
* Mapreduce avec parallel, cat et une redirection
|
||
http://datacratic.com/site/blog/mapreduce-avec-parallel-cat-et-une-redirection
|
||
|
||
* GNU Parallel for Large Batches of Small Jobs
|
||
https://support.scinet.utoronto.ca/wiki/images/7/7b/Tech-talk-gnu-parallel.pdf
|
||
|
||
* GNU Parallel used in black belt tip (8:50-10:12)
|
||
https://aws.amazon.com/apac/awspodcast/episode20/
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20121022
|
||
|
||
* --results structures output files so they can be imported using
|
||
Pandas (http://pandas.pydata.org/). Useful when doing simulations to
|
||
select output from some arguments.
|
||
|
||
* Use --env to transfer environment variables through ssh to remote
|
||
machines.
|
||
|
||
* GNU Parallel is accepted for Ubuntu Quantal.
|
||
http://packages.ubuntu.com/quantal/parallel
|
||
|
||
* GNU Parallel was requested for OpenSUSE (go vote!)
|
||
https://features.opensuse.org/314461
|
||
|
||
* Kneth's Korner: Map/Reduce and GNU Parallel
|
||
http://kenneth.geisshirt.dk/2012/10/mapreduce-and-gnu-parallel.html
|
||
|
||
* Blog post in Japanese by Siguniang.
|
||
https://siguniang.wordpress.com/2012/09/09/notes-on-gnu-parallel-the-command-line-power-tool/
|
||
(Siguniang mentions the logo is called 'The Cafe Wall Illusion')
|
||
http://www.michaelbach.de/ot/ang_cafewall/index.html
|
||
|
||
* GNU Parallel, where have you been all my life?
|
||
http://ssdigit.nothingisreal.com/2012/09/gnu-parallel-where-have-you-been-all-my.html
|
||
|
||
* Reference-free ddRADseq analysis tools uses GNU Parallel.
|
||
https://github.com/brantp/rtd#readme
|
||
|
||
* When bash just isn’t quick enough.
|
||
https://soimasysadmin.wordpress.com/2012/08/15/gnu-parallel-when-bash-just-isnt-quick-enough/
|
||
|
||
* Using GNU Parallel to Package Multiple Jobs in a Single PBS Job.
|
||
http://www.nas.nasa.gov/hecc/support/kb/Using-GNU-Parallel-to-Package-Multiple-Jobs-in-a-Single-PBS-Job_303.html
|
||
|
||
* GNU Parallel Notes.
|
||
http://xed.ch/help/parallel.html
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20120822
|
||
|
||
* Due to a bugfix the main flow has changed. Making this release beta
|
||
quality.
|
||
|
||
* Parallel Proxy Scraper & Checker Tool
|
||
https://www.youtube.com/watch?v=iXCeR_XsP6o
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20120722
|
||
|
||
* GNU Parallel was presented at Chiang Mai Bar Camp
|
||
http://barcampchiangmai.org/
|
||
See write up here:
|
||
http://www.cnx-software.com/2012/07/02/use-gnu-parallel-to-speed-up-script-execution-on-multiple-cores-andor-machines/
|
||
|
||
* I don't always max out all my CPUs, but when I do, I do it with GNU
|
||
Parallel.
|
||
http://memegenerator.net/instance/22638454
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20120622
|
||
|
||
* '-L n --pipe' will use records of n lines. This is useful when
|
||
processing data that have fixed records with a fixed number of
|
||
lines (e.g. fastq).
|
||
|
||
* --filter-hosts will remove down hosts. For each remote host: check
|
||
that login through ssh works. If not: do not use this host.
|
||
Currently you can not put --filter-hosts in a profile, $PARALLEL,
|
||
/etc/parallel/config or similar. This is because GNU Parallel uses
|
||
GNU Parallel to compute this, so you will get an infinite loop. This
|
||
will likely be fixed in a later release.
|
||
|
||
* --pipe now uses fork() instead of busy wait. The performance should
|
||
be better on computers with >10 cores while remaining the same on
|
||
computers with few cores.
|
||
|
||
* GNU Parallel will be represented at Chiang Mai Bar Camp.
|
||
http://barcampchiangmai.org/
|
||
|
||
* Indexing Big Data on Amazon AWS: The Screencast
|
||
(Check out his T-shirt at 18:40-21:00).
|
||
http://www.opensourceconnections.com/2012/06/06/indexing-big-data-on-amazon-aws-screencast/
|
||
|
||
* biotoolbox uses GNU Parallel.
|
||
https://code.google.com/p/biotoolbox/wiki/Pod_novo_wrapper
|
||
|
||
* Spiceweasel uses GNU Parallel.
|
||
https://github.com/mattray/spiceweasel
|
||
|
||
* GNU Parallel part of The Administrators Challenge.
|
||
http://challenge.twistedrack.com/questions/qs3.php
|
||
|
||
* Finding Oldest Firefox Code using GNU Parallel.
|
||
http://gregoryszorc.com/blog/2012/06/18/finding-oldest-firefox-code/
|
||
|
||
* Bug fixes (quite a few for remote job running) and man page updates.
|
||
|
||
|
||
20120522
|
||
|
||
* Timings in --joblog now with milliseconds.
|
||
|
||
* Thesis using GNU Parallel (correctly cited; hooray!).
|
||
http://scholarworks.sjsu.edu/etd_projects/222/
|
||
|
||
* Package for AIX.
|
||
http://www.perzl.org/aix/index.php?n=Main.Parallel
|
||
|
||
* Prokka uses GNU Parallel.
|
||
http://bioinformatics.net.au/prokka-manual.html
|
||
|
||
* GNU Parallel was presented at MUUG.
|
||
http://muug.mb.ca/blog/meetings/2012/05/09/may-8th-2012-rtfm-gnu-parallel1/
|
||
|
||
* Blog post: Manipulating many or large files with GNU parallel.
|
||
http://compbiously.blogspot.com/2012/05/manipulating-many-or-large-files-with.html
|
||
|
||
* Blog post: Running commands in parallel using bash.
|
||
http://blog.michaelboman.org/2012/04/running-commands-in-parallel-using-bash.html
|
||
|
||
* Blog post: Efficient log processing.
|
||
http://www.francoismaillet.com/blog/?p=399
|
||
|
||
* Blog post about GNU Parallel and Amazon EC2.
|
||
http://blog.kevinformatics.com/post/4970574713/interested-in-your-experience-using-gnu-parallel-in
|
||
|
||
* Blog post in Hungarian.
|
||
http://commandline.blog.hu/2012/05/03/gnu_parallel
|
||
|
||
* Popularity of GNU Parallel in Debian explodes.
|
||
http://qa.debian.org/popcon-graph.php?packages=parallel
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20120422
|
||
|
||
* A race condition bug caused restructuring of a central piece of the
|
||
code. Thus this release of GNU Parallel should be considered beta
|
||
quality.
|
||
|
||
* sql --dbsize will now list sizes of all databases if given no
|
||
database.
|
||
|
||
* GNU Parallel is now in Debian Unstable.
|
||
apt-get install parallel
|
||
|
||
* Article: Computational and experimental analyses of
|
||
retrotransposon-associated minisatellite DNAs in the soybean genome.
|
||
http://www.biomedcentral.com/content/supplementary/1471-2105-13-s2-s13-s1.pdf
|
||
|
||
* Blog post: GNU parallel - the best thing since sliced bread.
|
||
https://arrayal.wordpress.com/2012/03/27/gnu-parallel-the-best-thing-since-sliced-bread/
|
||
|
||
* Blog post: GNU Parallel makes everything faster.
|
||
http://compbiously.blogspot.com/2012/03/gnu-parallel-makes-everything-faster.html
|
||
|
||
* Blog post (German): Howto: GNU parallel.
|
||
http://d24m.de/2012/04/05/howto-gnu-parallel/
|
||
|
||
* Blog post: Running in Parallel
|
||
http://interactivity.ifactory.com/2012/04/running-in-parallel/
|
||
|
||
* 20000 watched the first intro video.
|
||
http://pi.dk/1
|
||
|
||
* The first 5 have received their GNU Parallel merchandise
|
||
https://www.gnu.org/software/parallel/merchandise.html
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20120322
|
||
|
||
* Parallel Process Database Dumps.
|
||
http://blog.mattoates.co.uk/2012/02/parallel-process-database-dumps.html
|
||
|
||
* Using GNU Parallel to process images from Mars.
|
||
http://lunokhod.org/?p=468
|
||
|
||
* Using GNU Parallel with bzgrep.
|
||
http://filip.rembialkowski.net/did-you-know-gnu-parallel/
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20120222
|
||
|
||
* --workdir . will use the current working dir. If the current working
|
||
dir is beneath your home dir, the value . is treated as the relative
|
||
path to your home dir. This means that if your home dir is different
|
||
on remote computers (e.g. if your login is different) the relative
|
||
path will still be relative to your home dir.
|
||
|
||
* A texinfo file is now generated from the POD file using pod2texi.
|
||
|
||
* The bioinformatics article "Fast large-scale clustering of protein
|
||
structures using Gauss integrals" mentions GNU Parallel. They forgot
|
||
to add the reference. Please remember --bibtex if you use GNU
|
||
Parallel in an article.
|
||
http://bioinformatics.oxfordjournals.org/content/28/4/510.short
|
||
|
||
* The first supporter received his merchandise. Did you order yours?
|
||
https://www.gnu.org/software/parallel/merchandise.html
|
||
|
||
* How to use GNU Parallel with samtools.
|
||
http://zvfak.blogspot.com/2012/02/samtools-in-parallel.html
|
||
|
||
* How to use GNU Parallel for vacuuming PostgreSQL tables.
|
||
http://blog.timmattison.com/archives/2012/01/24/mini-hack-parallel-vacuuming-in-postgresql/
|
||
|
||
* Converting e-books with Calibre and GNU Parallel.
|
||
http://www.linkhal.de/blog/2012/01/converting-massive-e-book-collections-with-calibre-and-gnu-parrallel/
|
||
|
||
* Using GNU Parallel for tailing logfiles from multiple servers.
|
||
http://silviud.blogspot.com/2012/02/shell-parallel-processing.html
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20120122
|
||
|
||
* --header : uses the first input line as column names and you can
|
||
then use {colname} as a replacement string in the command. This also
|
||
works with multiple :::'s.
|
||
|
||
* --header <regexp> matches a header as a regular expression and
|
||
repeats the header for each block with --pipe.
|
||
|
||
* --resume resumes from the last unfinished job. Useful if you stop
|
||
GNU Parallel and restart it later with the same arguments.
|
||
|
||
* niceload now propagates exit status correctly.
|
||
|
||
* Show your support for GNU Parallel. For a limited time you can get a
|
||
GNU Parallel T-shirt+mug+pen+100 postcards at 20 EUR. See
|
||
https://www.gnu.org/software/parallel/merchandise.html
|
||
|
||
* Options -g -B -T -U -W -Y are retired as warned 6 months ago.
|
||
|
||
* GNU Parallel referenced in article on Holographic Grid Cloud. Thanks
|
||
to Stefano Gallozzi. http://arxiv.org/pdf/1112.6128
|
||
|
||
* Article in IEEE Software on GNU Parallel. Thanks to Diomidis
|
||
Spinellis. http://www.spinellis.gr/blog/20110911/
|
||
|
||
* An article on Narwhal which uses GNU Parallel. They forgot to add
|
||
the reference. Please remember --bibtex if you use GNU Parallel in
|
||
an article.
|
||
http://bioinformatics.oxfordjournals.org/content/early/2011/11/08/bioinformatics.btr613.full.pdf
|
||
|
||
* Blog post on using GNU Parallel to speed up BLAST queries:
|
||
http://blog.mckuhn.de/2012/01/embarrassingly-parallel-blast-search.html
|
||
|
||
* Video showing Wake-on-LAN with GNU Parallel.
|
||
https://www.youtube.com/watch?v=0mB-yIyKFLQ
|
||
|
||
* Using GNU Parallel with Solr.
|
||
http://sujitpal.blogspot.com/2011/12/solr-report-generation-with-python.html
|
||
|
||
* First job ad including GNU Parallel:
|
||
http://seeker.dice.com/jobsearch/servlet/JobSearch?op=101&dockey=xml/7/6/76f858de083a094f74b1a5d3ba53ffc5@endecaindex
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20111222
|
||
|
||
* --timeout will now kill grandchildren.
|
||
|
||
* Interview in Hacker Public Radio.
|
||
http://hackerpublicradio.org/eps.php?id=0860
|
||
|
||
* Blog post in Chinese.
|
||
http://blog.sina.com.cn/s/blog_3f7652740100y0ju.html
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20111122
|
||
|
||
* GNU Parallel package for Fedora.
|
||
https://admin.fedoraproject.org/pkgdb/acls/name/parallel
|
||
|
||
* Using GNU Parallel with Mozilla cross referencing.
|
||
https://blog.mozilla.com/it/2011/11/15/mxr-improvements/
|
||
|
||
* Neat flash presentation of GNU Parallel.
|
||
http://prezi.com/0nq74b2elgtx/about-gnu-parallel/
|
||
|
||
* How to create tar files of dirs in parallel.
|
||
https://joneslee85.wordpress.com/2011/11/22/shell-tips-how-to-recursively-archive-many-folders/
|
||
|
||
* Article in Admin Network&Security by Ben Martin.
|
||
http://www.admin-magazine.com/HPC/Articles/GNU-Parallel
|
||
|
||
* Blog post in Polish.
|
||
http://blog.tiger.com.pl/2011/01/05/gnu-parallel/
|
||
|
||
* Blog post on using GNU Parallel of Mac OS X.
|
||
http://www.0xcb0.com/2011/10/19/running-parallel-bash-tasks-on-os-x/
|
||
|
||
* Blog post in English about using zip with GNU Parallel.
|
||
http://nuclear-imaging.info/site_content/2011/05/11/gnu-parallel/
|
||
|
||
* Blog post in English by researcher.
|
||
http://zandyware.wordpress.com/2011/10/15/gnu-parallell-make-best-use-of-your-multicore-computer/
|
||
|
||
* Blog entry in Chinese.
|
||
http://blog.csdn.net/tossense/article/details/6889026
|
||
|
||
* Blog entry in Japanese on managing many servers.
|
||
http://hiroakis.com/blog/2011/11/13/gnu-parallel/
|
||
|
||
* GNU Parallel is now part of ppbs.
|
||
http://physiology.med.cornell.edu/faculty/mason/lab/ppbs/introduction/introduction.html
|
||
|
||
* How to use GNU Parallel with MCX.
|
||
http://mcx.sourceforge.net/cgi-bin/index.cgi?MMC/Doc/MMCCluster
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20111022
|
||
|
||
* --tag prepends lines with the argument(s). If used with --(n)onall
|
||
the line will be prepended with the sshlogin instead.
|
||
|
||
* --shellquote does not run the command but quotes it using \'s.
|
||
Useful for making quoted composed commands.
|
||
|
||
* --profile can now be repeated merging multiple profiles.
|
||
|
||
* --bibtex now gives the BibTeX entry.
|
||
Thanks to Aleksandr Levchuk for testing.
|
||
|
||
* Makefile for simple .deb package.
|
||
cd packager/debian; make
|
||
|
||
* sql: --list-databases lists the databases.
|
||
|
||
* GNU Parallel officially in Fedora 16
|
||
http://www.spinics.net/lists/fedora-package-announce/msg67130.html
|
||
|
||
* Blog post on recompression FLAC audio.
|
||
http://blog.oxplot.com/2011/08/reflac-flac-cleanser.html
|
||
|
||
* The most complex use of ::: I have seen so far.
|
||
http://agentzlerich.blogspot.com/2011/09/following-up-on-previous-failures-ive.html
|
||
|
||
* Example of using GNU Parallel with sudo.
|
||
http://loopkid.net/articles/2011/10/07/recursive-chmod-and-chown-on-files
|
||
|
||
* Use wget and gnu parallel to download mp3s at rocket speed.
|
||
http://sdbrain.posterous.com/use-wget-and-gnu-parallel-to-download-mp3s-at
|
||
|
||
* First CV referring to GNU Parallel.
|
||
http://www.peopleperhour.com/freelancers/stephen_/internet_researcher_and_protocol_develop/140602
|
||
|
||
* Cartwright labs usage of GNU Parallel.
|
||
http://cartwrightlab.wikispaces.com/The+Most+Bang+for+Your+Buck
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20110822
|
||
|
||
* --timeout implemented so that slow commands can be killed
|
||
|
||
* CPU detection improved for Mac OSX. Thanks to Wayne E. Seguin.
|
||
|
||
* Example of a parallel webcrawler in the man page.
|
||
|
||
* Backup up PostgreSQL with GNU Parallel. Thanks to Stephane Wirtel.
|
||
http://wirtel.be/2011/07/15/rsync_parallel/
|
||
|
||
* Blog post in Japanese.
|
||
http://dminor11th.blogspot.com/2011/08/gnu-parallel.html
|
||
|
||
* Blog post about optimizing JPEGs. Thanks to Thomas Jost.
|
||
http://schnouki.net/2011/07/22/optimizing-jpeg-pictures/
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20110722
|
||
|
||
* niceload: --hard will suspend a program if a limit is reached - as
|
||
opposed to just slowing the program down.
|
||
|
||
* niceload: --soft will slow the program down - as opposed to
|
||
suspending the program completely.
|
||
|
||
* niceload: --run-io will slow down a program if disk io goes above a
|
||
certain limit.
|
||
|
||
* niceload: --run-load will slow down a program if loadaverage goes
|
||
above a certain limit.
|
||
|
||
* niceload: --run-mem will slow down a program if free memory goes
|
||
below a certain limit.
|
||
|
||
* niceload: --run-noswap will slow down a program if the computer is
|
||
swapping.
|
||
|
||
* niceload: --start-io, --start-load, --start-mem, --start-noswap will
|
||
defer starting a program until the system is below the limit.
|
||
|
||
* --io, --load, --mem, and --noswap sets both --run-* and --start-*.
|
||
|
||
* niceload got a major rewrite and is now object oriented.
|
||
|
||
* GNU Parallel was presented at Nordic Perl Workshop 2011.
|
||
http://conferences.yapceurope.org/npw2011/talk/3416
|
||
|
||
* Blog post about zcat and GNU Parallel. Thanks to Dr. John.
|
||
http://drjohnstechtalk.com/blog/2011/06/gnu-parallel-really-helps-with-zcat/
|
||
|
||
* 2 blog posts in Japanese. Thanks to Negima.
|
||
http://d.hatena.ne.jp/negima1976/20110607/1307412660
|
||
http://d.hatena.ne.jp/negima1976/20110628/1309252494
|
||
|
||
* Blog post for bioinformatics. Thanks to Chris Miller.
|
||
http://chrisamiller.com/science/2010/05/26/use-parallel-for-easy-multi-processor-execution/
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20110622
|
||
|
||
* --onall will run all the jobs on all the computers. This is useful
|
||
for systemadinistrators having multiple servers.
|
||
|
||
* --nonall runs like --onall but reads no arguments from stdin
|
||
(standard input), so is possible to do:
|
||
parallel --nonall -S computer1,computer2 uptime
|
||
|
||
* Context replace now works with multiple input sources:
|
||
parallel -X echo {1}-{2} ::: 1 2 3 ::: a b c
|
||
|
||
* --noswap do not start a job on a server that is swapping.
|
||
|
||
* Article: Parallel processing without Hadoop!
|
||
http://www.solutionhacker.com/parallel-processing-without-hadoop/
|
||
|
||
* Article in Linux Magazine (Spanish). Thanks to Ben Martin.
|
||
http://www.linux-magazine.es/issue/67/
|
||
|
||
* First blog post in Korean. Thanks to 민병국.
|
||
http://blog.daum.net/_blog/BlogTypeView.do?blogid=02RQ3&articleno=13380895&admin=#
|
||
|
||
* Blog post in Japanese. Thanks to Naoya Nakazawa.
|
||
http://www.sssg.org/blogs/naoya/archives/2003
|
||
|
||
* Blog post in English. Thanks to Dagon.
|
||
http://hekate.homeip.net/2011/05/29/
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20110522
|
||
|
||
* Multiple ::: can be put on the command line and will be treated
|
||
similar to contents from multiple -a's.
|
||
|
||
* ::: and :::: can now be mixed on the command line. Use {1} .. {n} to
|
||
refer to inputs from the different input sources.
|
||
|
||
* --xapply is now needed to get xapply's behaviour of reading one line
|
||
from each of the input sources.
|
||
|
||
* Multiple input sources will cause all combinations of arguments from
|
||
the sources to be generated. E.g. 'parallel echo {1}+{2} ::: 1 2 :::
|
||
a b c' will print 1+a 1+b 1+c 2+a 2+b 2+c. This can often replace
|
||
nested loops.
|
||
|
||
* Implemented {//} for the input line with the basename removed (dirname).
|
||
|
||
* New video showing the new ::: and :::: input sources.
|
||
http://tinyogg.com/watch/iOdhU/ http://www.youtube.com/watch?v=fOX1EyHkQwc
|
||
|
||
* GNU Parallel now has a logo.
|
||
http://www.gnu.org/software/parallel/logo.png
|
||
|
||
* Article about GNU SQL in USENIX Magazine ;login: (print)
|
||
http://www.usenix.org/publications/login/2011-04/
|
||
|
||
* Using GNU Parallel with EC2. Thanks to Kevin Wu.
|
||
http://blog.kevinformatics.com/post/4970574713/interested-in-your-experience-using-gnu-parallel-in
|
||
|
||
* Review with idea for {..} and {...} in Japanese. Thanks to ichii386.
|
||
http://d.hatena.ne.jp/ichii386/20110426
|
||
|
||
* Upgrade GNU Parallel using Macports. Thanks to Phil Hollenback.
|
||
http://www.hollenback.net/index.php/MacportsParallel
|
||
|
||
* Robert from Echo One discusses using processes instead of threads:
|
||
http://rrees.wordpress.com/2011/04/25/many-cores-many-threads/
|
||
|
||
* How to run GNU Parallel on a SLURM cluster. Thanks to Kevin Stock.
|
||
http://www.kevinstock.org/2011/04/using-the-ri-cluster/
|
||
|
||
* Short article about using GNU Parallel with lame:
|
||
http://loopkid.net/articles/2011/04/30/accelerate-lame-mp3-conversion
|
||
|
||
* Using GNU Parallel to run tail -f in Japanese. Thanks to Clouder.
|
||
http://blog.clouder.jp/archives/001140.html
|
||
|
||
* BBC Research & Development uses GNU Parallel:
|
||
http://www.bbc.co.uk/blogs/researchanddevelopment/2010/11/prototyping-weeknotes-41-26112.shtml
|
||
|
||
* Short article about using GNU Parallel on RHEL. Thanks to Rishi Deshpande.
|
||
http://nuclear-imaging.info/site_content/2011/05/11/gnu-parallel/
|
||
|
||
* Using GNU Parallel for FLAC->MP3 conversion. Thanks to Derek Marcotte.
|
||
http://derek.chezmarcotte.ca/?p=286
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20110422
|
||
|
||
* {#} now works as an alias for $PARALLEL_PID.
|
||
|
||
* --eta now estimates way more accurately.
|
||
|
||
* CPU detection code for AIX thanks to Christian Netrwal.
|
||
|
||
* --joblog contains exitcode.
|
||
|
||
* Thanks to Ævar Arnfjörð Bjarmason for reading my code.
|
||
|
||
* GNU Parallel was presented at:
|
||
- LUGA, Augsburg, 2011-03-26, http://www.luga.de/Aktionen/LIT-2011/Programm
|
||
- OSAA.dk, Aarhus, 2011-04-12
|
||
|
||
* Blog entry in Japanese. Thanks to Riywo.
|
||
http://blog.riywo.com/2011/04/19/022802
|
||
|
||
* Example of how to use GNU Parallel on PBS clusters. Thanks to Andrew
|
||
J Dolgert: http://web0.tc.cornell.edu/wiki/index.php?title=Gnu_Parallel
|
||
|
||
* First example of using GNU Parallel in Hebrew. Thanks to omry.
|
||
http://nd.gd/jk
|
||
|
||
* Intro video got past 10000 hits.
|
||
http://www.youtube.com/watch?v=OpaiGYxkSuQ
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20110322
|
||
|
||
* --tollef to be switch compatible with Tollef's parallel. This will
|
||
cause -l to mean --load, and the argument separator will be --
|
||
instead of :::
|
||
|
||
* --gnu will force GNU Parallel to behave like GNU Parallel even if
|
||
--tollef is set.
|
||
|
||
* Site wide config file: /etc/parallel/config
|
||
|
||
This should solve the issue with some packagers renaming GNU
|
||
Parallel to gparallel to avoid the naming conflict.
|
||
|
||
By putting --tollef in the site wide config file you can deinstall
|
||
Tollef's parallel and install GNU Parallel instead without any
|
||
change for users or scripts. This is useful for packagers that
|
||
currently rename GNU Parallel or simply do not distribute GNU
|
||
Parallel because the command name conflicts with Tollef's parallel.
|
||
|
||
* -L 0 -n 0, and -N 0 implemented. They will read one argument,
|
||
but insert 0 arguments on the command line. Useful if you just want
|
||
to run the same command with the same arguments a number of times.
|
||
|
||
* GNU Parallel is now in Fink. Thanks to Jesse Alama.
|
||
http://pdb.finkproject.org/pdb/package.php/parallel
|
||
|
||
* Opscode Cookbook for Chef. Thanks to Joshua Timberman.
|
||
https://github.com/opscode/cookbooks/tree/master/gnu_parallel
|
||
|
||
* Man page examples translated into Japanese. Thanks to Koshigoe.
|
||
http://w.koshigoe.jp/study/?%5Bsystem%5D+GNU+parallel+%BB%C8%CD%D1%CE%E3#l13
|
||
|
||
* GNU Parallel will be presented at:
|
||
- LUGA, Augsburg, 2011-03-26, http://www.luga.de/Aktionen/LIT-2011/Programm
|
||
- OSAA.dk, Aarhus, 2011-04-12
|
||
|
||
* Video of presentation from FSCONS 2010-11-07. The presenter was
|
||
_really_ hoarse that day (Something to do with loads of alcohol the
|
||
night before). http://vimeo.com/20838834
|
||
|
||
* Review with examples in German. Thanks to M. Nieberg.
|
||
http://kenntwas.de/2011/linux/gnu-parallel/
|
||
|
||
* A review of the --pipe option in German. Thanks to Mathias Huber.
|
||
http://www.linux-magazin.de/NEWS/GNU-Parallel-20110205-stueckelt-Daten
|
||
|
||
* A small example of grepping maillogs. Thanks to François Maillet.
|
||
http://www.francoismaillet.com/blog/?p=399
|
||
|
||
* Using GNU Parallel instead of xargs. Thanks to James Cuff.
|
||
http://blog.jcuff.net/2011/02/on-train-ride-in.html
|
||
|
||
* Using GNU Parallel from 0install (German). Thanks to AdaMin.
|
||
http://forum.ubuntuusers.de/topic/gnu-parallel-mit-zero-install-kurzinfo-kein-wi/
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20110205
|
||
|
||
* --pipe splits piped data into blocks. Each block is piped to a
|
||
program for processing. The piping and the programs will be run in
|
||
parallel. Useful if the data you want to process is data for a
|
||
program and not arguments.
|
||
|
||
* --blocksize sets the blocksize in bytes for --pipe. The blocksize is
|
||
approximate. It can deviate as much as the size of one record.
|
||
Default is 1M.
|
||
|
||
* --recstart sets the string matching the start of a
|
||
record. Default is "".
|
||
|
||
* --recend sets the string matching the end of a
|
||
record. Default is '\n'. To specify none use --recend "".
|
||
|
||
If both --recstart and --recend are set, the end of a record must be
|
||
followed immediately by a start of a record. This is useful if
|
||
either recend or recstart can occur in the middle of a record.
|
||
|
||
* --remove-rec-sep removes the string matched by --recstart and
|
||
--recend.
|
||
|
||
* --regexp will make GNU Parallel treat --recstart and --recend as
|
||
regular expressions.
|
||
|
||
* --output-as-files will put the output of the programs into files and
|
||
instead of giving the output GNU Parallel will output the name of
|
||
these files.
|
||
|
||
* -N if used with --pipe sets the number of records to read.
|
||
|
||
* New video showing --pipe at
|
||
http://www.youtube.com/watch?v=1ntxT-47VPA
|
||
|
||
* GNU Parallel was presented at FOSDEM.
|
||
|
||
* Article in USENIX Magazine ;login: (print)
|
||
http://www.usenix.org/publications/login/2011-02/
|
||
|
||
* GNU Parallel is now on ohloh.net. Thanks to Wim Muskee.
|
||
https://www.ohloh.net/p/gnu-parallel
|
||
|
||
* Advanced recursive example. Thanks to Ruarí Ødegaard.
|
||
http://my.opera.com/ruario/blog/2011/01/24/editing-debian-packages-more-fun-with-gnu
|
||
|
||
* Small example on using GNU Parallel through 0install (German).
|
||
http://forum.ubuntuusers.de/topic/gnu-parallel-mit-zero-install-kurzinfo-kein-wi/
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20110122
|
||
|
||
* --joblog makes a simple log of completed jobs.
|
||
|
||
* -X now spreads arguments between job slots when reaching last
|
||
argument. Use -j1 to avoid this.
|
||
|
||
* People on the email list have voted -j+0 to be the new default
|
||
instead of -j9.
|
||
|
||
* First review in Polish. Thanks to Patryk Krawaczyński.
|
||
http://nfsec.pl/root/2458
|
||
|
||
* Review in Spanish (in print).
|
||
https://www.linux-magazine.es/issue/67/moreinfo.html
|
||
|
||
* Review in English. Thanks to Brian Gough.
|
||
http://blogs.fsfe.org/bjg/2011/01/gnu-parallel-a-map-operator-for-the-command-line/
|
||
|
||
* Review in French. Thanks to Denis Dordoigne.
|
||
http://linuxfr.org/2010/12/29/27715.html
|
||
|
||
* Review in Spanish.
|
||
http://gufete.net/index.php?entry=entry110116-200022
|
||
|
||
* Article with advanced recursive example. Thanks to Ruarí Ødegaard
|
||
http://my.opera.com/ruario/blog/2011/01/18/fun-with-gnu-parallel
|
||
|
||
* Use case for memcache.
|
||
http://www.dctrwatson.com/2010/12/how-to-dump-memcache-keyvalue-pairs-fast/
|
||
|
||
* Bug fixes and man page updates.
|
||
|
||
|
||
20101222
|
||
|
||
* GNU niceload is now part of GNU Parallel. GNU niceload slows down a
|
||
program if the load average is above a certain limit.
|
||
|
||
* Implemented --tmpdir to buffer standard output and standard error in
|
||
a different place.
|
||
|
||
* Implemented --load to wait until the load is below a limit before
|
||
starting another job on that computer.
|
||
|
||
* Implemented --nice set the niceness of jobs running both locally and
|
||
remotely.
|
||
|
||
* Implemented --dry-run to print the job without running it.
|
||
|
||
* Implemented --tty as the old default of assigning a tty to the first
|
||
job causes problems.
|
||
|
||
* Review with focus on clusters. Thanks to Taylor Gillespie
|
||
http://www.unixpronews.com/unixpronews-49-20101019GNUParallelSpeedUpProcessingWithMulticoresClusters.html
|
||
|
||
* Review with focus on protein similarity.
|
||
http://kevinformatics.tumblr.com/post/2142473893/cluster-like-computing-using-gnu-parallel
|
||
|
||
* Review in Spanish.
|
||
http://gr3p.com/2010/12/gnu-parallel-acelera-tus-scripts-en-linux
|
||
|
||
* Quite a few bug fixes and man page updates.
|
||
|
||
|
||
20101202
|
||
|
||
* Implemented {/} for the input line with the path removed (basename).
|
||
|
||
* Implemented {/.} for the input line with extension and path removed
|
||
(basename).
|
||
|
||
* Output from --progress is now sent to standard error instead of
|
||
standard output.
|
||
|
||
* --eta was broken and counted down from 0. Now fixed.
|
||
|
||
* Standard output and standard error are flushed after every job so if
|
||
standard output does not end with a newline it will still be grouped
|
||
with the rest of standard output.
|
||
|
||
* --command, -c, --file, and -f are now removed as options. They were
|
||
never used in practice.
|
||
|
||
* GetOptionsFromArray rewritten to work with old Perl libraries.
|
||
|
||
* The file COPYING now contains the GNU General Public License 3
|
||
|
||
* Major rewrite to make the code more object oriented and easier to
|
||
maintain in the future.
|
||
|
||
|
||
20101113
|
||
|
||
* Using -j myfile the number of jobs can be changed while GNU Parallel
|
||
is running simply by changing the content of myfile.
|
||
|
||
* Implemented --profile to use different .parallel/config for
|
||
different situations.
|
||
|
||
* Ugly newlines in $PARALLEL and .parallel/config are no longer
|
||
needed. Instead you have to use \ in front of special shell
|
||
characters.
|
||
|
||
* --workdir puts the files transferred to remote machines in a
|
||
specified directory.
|
||
|
||
* $PARALLEL_PID is set to the process id of GNU Parallel.
|
||
|
||
* $PARALLEL_SEQ is set to the sequence number of the job.
|
||
|
||
* -v now only shows the command to run. Use -v -v to show the
|
||
ssh/rsync wrapping.
|
||
|
||
* Slow spawning error is now only a warning.
|
||
|
||
* If stdin is a tty and input is read from stdin you now get a
|
||
warning.
|
||
|
||
* GNU sql: \n and \x0a in arguments is replaced with newline.
|
||
|
||
* Patch for Debian package and spelling mistakes. Thanks to Rogério
|
||
Brito <rbrito at ime dot usp dot br>
|
||
|
||
* Mac OS X Homebrew package. Thanks to Jonathan Palardy <jonathan dot
|
||
palardy at gmail dot com>
|
||
|
||
* FreeBSD port. Thanks to Chris Howey <howeyc at gmail dot com>
|
||
|
||
* Pardus package. Thanks to Fethican Coşkuner
|
||
<fethicanc at gmail dot com>
|
||
|
||
* First review in Chinese. Thanks to 曾義峰:
|
||
http://antbsd.twbbs.org/~ant/wordpress/?p=2876
|
||
|
||
* First review in print:
|
||
http://www.linux-magazine.com/Issues/2010 Nov 2010
|
||
|
||
* First review in Spanish:
|
||
http://www.muylinux.com/2010/10/18/gnu-parallel-computacion-paralela-a-golpe-de-comando
|
||
|
||
* First review in Dutch thanks to Koen Vervloesem <koen at vervloesem
|
||
dot eu>:
|
||
http://techworld.nl/technologie/33493/gebruik-al-je-processorkernen-met-gnu-parallel.html
|
||
|
||
* Blog review thanks to R. Tyler Croy <tyler at monkeypox dot org>:
|
||
http://unethicalblogger.com/posts/2010/11/gnuparallel_changed_my_life
|
||
|
||
* 5000 views of the intro video:
|
||
http://www.youtube.com/watch?v=OpaiGYxkSuQ
|
||
|
||
* As usual a bunch of bugfixes and more usage examples in the man
|
||
page.
|
||
|
||
* GNU Parallel was presented at FSCONS 2010-11-07:
|
||
http://www.fscons.org/fs/gnu-parallel Hopefully the
|
||
video will be online soon.
|
||
|
||
|
||
20100922
|
||
|
||
* See GNU Parallel live at FSCONS 2010-11-07:
|
||
http://www.fscons.org/fs/gnu-parallel
|
||
|
||
* Untested Debian and xUbuntu packages available through OpenSUSE
|
||
build service:
|
||
https://build.opensuse.org/package/show?package=parallel&project=home%3Atange
|
||
|
||
* Using --retries a job will be retried on another computer if it
|
||
fails. This is useful if some jobs fail for no apparent reason (such
|
||
as network failure).
|
||
|
||
* BSD xargs -o (open /dev/tty) is now default for the job running in
|
||
foreground. Useful for interactive commands like:
|
||
ls | parallel -Xuj1 vi
|
||
|
||
* GNU sql now supports SQLite.
|
||
|
||
* Renamed .dburl.aliases to .sql/aliases and /etc/sql/aliases.
|
||
|
||
* GNU sql now support --list-tables
|
||
|
||
* Alias for DBURL can contain '?query' part with %-quoting.
|
||
|
||
|
||
20100906
|
||
|
||
* Using --shebang GNU Parallel can be used as the parser for a script.
|
||
E.g: #!/usr/bin/parallel --shebang traceroute (followed by lines of
|
||
hosts)
|
||
|
||
* First community generated bugfixes
|
||
|
||
* Alt Linux package of GNU Parallel. Thanks to Michael Shigorin <mike
|
||
at altlinux dot org>
|
||
|
||
* Sunfreeware package of GNU Parallel. Thanks to Steven M. Christensen
|
||
<sunfreeware at gmail.com>
|
||
|
||
* Untested CentOS, Fedora, Mandriva, RedHat, and SUSE packages
|
||
available through OpenSUSE build service:
|
||
https://build.opensuse.org/package/show?package=parallel&project=home%3Atange
|
||
|
||
* Review of GNU Parallel. Thanks to Andrew McFague amcfague at wgen dot net
|
||
http://www.andrew-mcfague.com/linux/utilities-linux/
|
||
commands-every-serious-nix-user-should-know/#parallel
|
||
|
||
* First 1000 views of the intro video
|
||
|
||
* sql - a small script to access sql bases from the command line which
|
||
is a handy companion to parallel --colsep
|
||
|
||
|
||
20100822
|
||
|
||
* Counting semaphore functionality: start a job in the background. If
|
||
N jobs are already running, wait for one to complete. Examples:
|
||
sem 'sleep 2; echo foo'; sem 'sleep 1; echo bar'; sem --wait
|
||
sem -j2 'sleep 2; echo foo'; sem -j2 'sleep 1; echo bar'; sem --wait
|
||
|
||
* With --colsep a table can be used as input. Example:
|
||
cat tab_sep_table | parallel --colsep '\t' echo col1 {1} col2 {2}
|
||
|
||
* --trim can remove white space around arguments.
|
||
|
||
* --sshloginfile '..' means use ~/.parallel/sshloginfile
|
||
|
||
* Zero install package. Thanks to Tim Cuthbertson <tim3d dot junk at
|
||
gmail dot com>
|
||
|
||
* OpenSUSE package. Thanks to Markus Ammer <mkmm at gmx-topmail dot
|
||
de>
|
||
|
||
* NixOS package. Thanks to Ludovic Courtès <ludo at gnu dot org>
|
||
|
||
* Web review http://oentend.blogspot.com/2010/08/gnu-parallel.html
|
||
Thanks to Pavel Nuzhdin <pnzhdin at gmail dot com>
|
||
|
||
* Web review http://psung.blogspot.com/2010/08/gnu-parallel.html
|
||
Thanks to Phil Sung <psung at alum dot mit dot edu>
|
||
|
||
|
||
20100722
|
||
|
||
* Arguments can now be given on the command line. Example:
|
||
parallel bzip2 ::: *.txt
|
||
|
||
* xapply like functionality. Example:
|
||
parallel diff {1} {2} :::: filelist1 filelist2
|
||
parallel diff {1} {2} ">"{1.}-{2.}.diff :::: filelist1 filelist2
|
||
|
||
* Arch Linux package. Thanks to Peter Simons <simons at cryp dot to>
|
||
|
||
* Mandriva package. Thanks to Sandro Cazzaniga <kharec at mandriva dot
|
||
org>
|
||
|
||
* -L -l -n -s now implies -X unless -m is set
|
||
|
||
|
||
20100620
|
||
|
||
* New video showing the new options.
|
||
http://www.youtube.com/watch?v=OpaiGYxkSuQ or at
|
||
http://tinyogg.com/watch/TORaR/ and http://tinyogg.com/watch/hfxKj/
|
||
|
||
* 100% options complete with xargs. All options for xargs can now be
|
||
used in GNU Parallel - even the more exotic.
|
||
|
||
* --basefile for transferring basedata. When running jobs on remote
|
||
computers --basefile will transfer files before the first jobs is
|
||
run. It can be used to transfer data that remains the same for each
|
||
job such as scripts or lookup tables.
|
||
|
||
* --progress shows progress. To see how many jobs is running on each
|
||
server use --progress. It can be turned on even after GNU Parallel
|
||
is started.
|
||
|
||
* --eta shows estimated time left in seconds.
|
||
|
||
* --halt-on-error stops if an error occurs. GNU Parallel will default
|
||
to run all jobs - even if some of them fail. With --halt-on-error
|
||
GNU Parallel can ignore errors, wait for the currently running jobs
|
||
to finish, or stop immediately when an error occurs.
|
||
|
||
|
||
20100601
|
||
|
||
* GNU Parallel now has support for running jobs on remote machines
|
||
|
||
* Intro video http://www.youtube.com/watch?v=LlXDtd_pRaY
|
||
http://openvideo.dailymotion.com/video/xdmqwz_gnu-parallel-script-processing-and_tech
|
||
|
||
|
||
20100424
|
||
|
||
* GNU Parallel is now official GNU software
|