diff --git a/COPYING b/COPYING
new file mode 100644
index 00000000..94a9ed02
--- /dev/null
+++ b/COPYING
@@ -0,0 +1,674 @@
+ GNU GENERAL PUBLIC LICENSE
+ Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc.
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+ Preamble
+
+ The GNU General Public License is a free, copyleft license for
+software and other kinds of works.
+
+ The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works. By contrast,
+the GNU General Public License is intended to guarantee your freedom to
+share and change all versions of a program--to make sure it remains free
+software for all its users. We, the Free Software Foundation, use the
+GNU General Public License for most of our software; it applies also to
+any other work released this way by its authors. You can apply it to
+your programs, too.
+
+ When we speak of free software, we are referring to freedom, not
+price. Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+them if you wish), that you receive source code or can get it if you
+want it, that you can change the software or use pieces of it in new
+free programs, and that you know you can do these things.
+
+ To protect your rights, we need to prevent others from denying you
+these rights or asking you to surrender the rights. Therefore, you have
+certain responsibilities if you distribute copies of the software, or if
+you modify it: responsibilities to respect the freedom of others.
+
+ For example, if you distribute copies of such a program, whether
+gratis or for a fee, you must pass on to the recipients the same
+freedoms that you received. You must make sure that they, too, receive
+or can get the source code. And you must show them these terms so they
+know their rights.
+
+ Developers that use the GNU GPL protect your rights with two steps:
+(1) assert copyright on the software, and (2) offer you this License
+giving you legal permission to copy, distribute and/or modify it.
+
+ For the developers' and authors' protection, the GPL clearly explains
+that there is no warranty for this free software. For both users' and
+authors' sake, the GPL requires that modified versions be marked as
+changed, so that their problems will not be attributed erroneously to
+authors of previous versions.
+
+ Some devices are designed to deny users access to install or run
+modified versions of the software inside them, although the manufacturer
+can do so. This is fundamentally incompatible with the aim of
+protecting users' freedom to change the software. The systematic
+pattern of such abuse occurs in the area of products for individuals to
+use, which is precisely where it is most unacceptable. Therefore, we
+have designed this version of the GPL to prohibit the practice for those
+products. If such problems arise substantially in other domains, we
+stand ready to extend this provision to those domains in future versions
+of the GPL, as needed to protect the freedom of users.
+
+ Finally, every program is threatened constantly by software patents.
+States should not allow patents to restrict development and use of
+software on general-purpose computers, but in those that do, we wish to
+avoid the special danger that patents applied to a free program could
+make it effectively proprietary. To prevent this, the GPL assures that
+patents cannot be used to render the program non-free.
+
+ The precise terms and conditions for copying, distribution and
+modification follow.
+
+ TERMS AND CONDITIONS
+
+ 0. Definitions.
+
+ "This License" refers to version 3 of the GNU General Public License.
+
+ "Copyright" also means copyright-like laws that apply to other kinds of
+works, such as semiconductor masks.
+
+ "The Program" refers to any copyrightable work licensed under this
+License. Each licensee is addressed as "you". "Licensees" and
+"recipients" may be individuals or organizations.
+
+ To "modify" a work means to copy from or adapt all or part of the work
+in a fashion requiring copyright permission, other than the making of an
+exact copy. The resulting work is called a "modified version" of the
+earlier work or a work "based on" the earlier work.
+
+ A "covered work" means either the unmodified Program or a work based
+on the Program.
+
+ To "propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy. Propagation includes copying,
+distribution (with or without modification), making available to the
+public, and in some countries other activities as well.
+
+ To "convey" a work means any kind of propagation that enables other
+parties to make or receive copies. Mere interaction with a user through
+a computer network, with no transfer of a copy, is not conveying.
+
+ An interactive user interface displays "Appropriate Legal Notices"
+to the extent that it includes a convenient and prominently visible
+feature that (1) displays an appropriate copyright notice, and (2)
+tells the user that there is no warranty for the work (except to the
+extent that warranties are provided), that licensees may convey the
+work under this License, and how to view a copy of this License. If
+the interface presents a list of user commands or options, such as a
+menu, a prominent item in the list meets this criterion.
+
+ 1. Source Code.
+
+ The "source code" for a work means the preferred form of the work
+for making modifications to it. "Object code" means any non-source
+form of a work.
+
+ A "Standard Interface" means an interface that either is an official
+standard defined by a recognized standards body, or, in the case of
+interfaces specified for a particular programming language, one that
+is widely used among developers working in that language.
+
+ The "System Libraries" of an executable work include anything, other
+than the work as a whole, that (a) is included in the normal form of
+packaging a Major Component, but which is not part of that Major
+Component, and (b) serves only to enable use of the work with that
+Major Component, or to implement a Standard Interface for which an
+implementation is available to the public in source code form. A
+"Major Component", in this context, means a major essential component
+(kernel, window system, and so on) of the specific operating system
+(if any) on which the executable work runs, or a compiler used to
+produce the work, or an object code interpreter used to run it.
+
+ The "Corresponding Source" for a work in object code form means all
+the source code needed to generate, install, and (for an executable
+work) run the object code and to modify the work, including scripts to
+control those activities. However, it does not include the work's
+System Libraries, or general-purpose tools or generally available free
+programs which are used unmodified in performing those activities but
+which are not part of the work. For example, Corresponding Source
+includes interface definition files associated with source files for
+the work, and the source code for shared libraries and dynamically
+linked subprograms that the work is specifically designed to require,
+such as by intimate data communication or control flow between those
+subprograms and other parts of the work.
+
+ The Corresponding Source need not include anything that users
+can regenerate automatically from other parts of the Corresponding
+Source.
+
+ The Corresponding Source for a work in source code form is that
+same work.
+
+ 2. Basic Permissions.
+
+ All rights granted under this License are granted for the term of
+copyright on the Program, and are irrevocable provided the stated
+conditions are met. This License explicitly affirms your unlimited
+permission to run the unmodified Program. The output from running a
+covered work is covered by this License only if the output, given its
+content, constitutes a covered work. This License acknowledges your
+rights of fair use or other equivalent, as provided by copyright law.
+
+ You may make, run and propagate covered works that you do not
+convey, without conditions so long as your license otherwise remains
+in force. You may convey covered works to others for the sole purpose
+of having them make modifications exclusively for you, or provide you
+with facilities for running those works, provided that you comply with
+the terms of this License in conveying all material for which you do
+not control copyright. Those thus making or running the covered works
+for you must do so exclusively on your behalf, under your direction
+and control, on terms that prohibit them from making any copies of
+your copyrighted material outside their relationship with you.
+
+ Conveying under any other circumstances is permitted solely under
+the conditions stated below. Sublicensing is not allowed; section 10
+makes it unnecessary.
+
+ 3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+ No covered work shall be deemed part of an effective technological
+measure under any applicable law fulfilling obligations under article
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
+similar laws prohibiting or restricting circumvention of such
+measures.
+
+ When you convey a covered work, you waive any legal power to forbid
+circumvention of technological measures to the extent such circumvention
+is effected by exercising rights under this License with respect to
+the covered work, and you disclaim any intention to limit operation or
+modification of the work as a means of enforcing, against the work's
+users, your or third parties' legal rights to forbid circumvention of
+technological measures.
+
+ 4. Conveying Verbatim Copies.
+
+ You may convey verbatim copies of the Program's source code as you
+receive it, in any medium, provided that you conspicuously and
+appropriately publish on each copy an appropriate copyright notice;
+keep intact all notices stating that this License and any
+non-permissive terms added in accord with section 7 apply to the code;
+keep intact all notices of the absence of any warranty; and give all
+recipients a copy of this License along with the Program.
+
+ You may charge any price or no price for each copy that you convey,
+and you may offer support or warranty protection for a fee.
+
+ 5. Conveying Modified Source Versions.
+
+ You may convey a work based on the Program, or the modifications to
+produce it from the Program, in the form of source code under the
+terms of section 4, provided that you also meet all of these conditions:
+
+ a) The work must carry prominent notices stating that you modified
+ it, and giving a relevant date.
+
+ b) The work must carry prominent notices stating that it is
+ released under this License and any conditions added under section
+ 7. This requirement modifies the requirement in section 4 to
+ "keep intact all notices".
+
+ c) You must license the entire work, as a whole, under this
+ License to anyone who comes into possession of a copy. This
+ License will therefore apply, along with any applicable section 7
+ additional terms, to the whole of the work, and all its parts,
+ regardless of how they are packaged. This License gives no
+ permission to license the work in any other way, but it does not
+ invalidate such permission if you have separately received it.
+
+ d) If the work has interactive user interfaces, each must display
+ Appropriate Legal Notices; however, if the Program has interactive
+ interfaces that do not display Appropriate Legal Notices, your
+ work need not make them do so.
+
+ A compilation of a covered work with other separate and independent
+works, which are not by their nature extensions of the covered work,
+and which are not combined with it such as to form a larger program,
+in or on a volume of a storage or distribution medium, is called an
+"aggregate" if the compilation and its resulting copyright are not
+used to limit the access or legal rights of the compilation's users
+beyond what the individual works permit. Inclusion of a covered work
+in an aggregate does not cause this License to apply to the other
+parts of the aggregate.
+
+ 6. Conveying Non-Source Forms.
+
+ You may convey a covered work in object code form under the terms
+of sections 4 and 5, provided that you also convey the
+machine-readable Corresponding Source under the terms of this License,
+in one of these ways:
+
+ a) Convey the object code in, or embodied in, a physical product
+ (including a physical distribution medium), accompanied by the
+ Corresponding Source fixed on a durable physical medium
+ customarily used for software interchange.
+
+ b) Convey the object code in, or embodied in, a physical product
+ (including a physical distribution medium), accompanied by a
+ written offer, valid for at least three years and valid for as
+ long as you offer spare parts or customer support for that product
+ model, to give anyone who possesses the object code either (1) a
+ copy of the Corresponding Source for all the software in the
+ product that is covered by this License, on a durable physical
+ medium customarily used for software interchange, for a price no
+ more than your reasonable cost of physically performing this
+ conveying of source, or (2) access to copy the
+ Corresponding Source from a network server at no charge.
+
+ c) Convey individual copies of the object code with a copy of the
+ written offer to provide the Corresponding Source. This
+ alternative is allowed only occasionally and noncommercially, and
+ only if you received the object code with such an offer, in accord
+ with subsection 6b.
+
+ d) Convey the object code by offering access from a designated
+ place (gratis or for a charge), and offer equivalent access to the
+ Corresponding Source in the same way through the same place at no
+ further charge. You need not require recipients to copy the
+ Corresponding Source along with the object code. If the place to
+ copy the object code is a network server, the Corresponding Source
+ may be on a different server (operated by you or a third party)
+ that supports equivalent copying facilities, provided you maintain
+ clear directions next to the object code saying where to find the
+ Corresponding Source. Regardless of what server hosts the
+ Corresponding Source, you remain obligated to ensure that it is
+ available for as long as needed to satisfy these requirements.
+
+ e) Convey the object code using peer-to-peer transmission, provided
+ you inform other peers where the object code and Corresponding
+ Source of the work are being offered to the general public at no
+ charge under subsection 6d.
+
+ A separable portion of the object code, whose source code is excluded
+from the Corresponding Source as a System Library, need not be
+included in conveying the object code work.
+
+ A "User Product" is either (1) a "consumer product", which means any
+tangible personal property which is normally used for personal, family,
+or household purposes, or (2) anything designed or sold for incorporation
+into a dwelling. In determining whether a product is a consumer product,
+doubtful cases shall be resolved in favor of coverage. For a particular
+product received by a particular user, "normally used" refers to a
+typical or common use of that class of product, regardless of the status
+of the particular user or of the way in which the particular user
+actually uses, or expects or is expected to use, the product. A product
+is a consumer product regardless of whether the product has substantial
+commercial, industrial or non-consumer uses, unless such uses represent
+the only significant mode of use of the product.
+
+ "Installation Information" for a User Product means any methods,
+procedures, authorization keys, or other information required to install
+and execute modified versions of a covered work in that User Product from
+a modified version of its Corresponding Source. The information must
+suffice to ensure that the continued functioning of the modified object
+code is in no case prevented or interfered with solely because
+modification has been made.
+
+ If you convey an object code work under this section in, or with, or
+specifically for use in, a User Product, and the conveying occurs as
+part of a transaction in which the right of possession and use of the
+User Product is transferred to the recipient in perpetuity or for a
+fixed term (regardless of how the transaction is characterized), the
+Corresponding Source conveyed under this section must be accompanied
+by the Installation Information. But this requirement does not apply
+if neither you nor any third party retains the ability to install
+modified object code on the User Product (for example, the work has
+been installed in ROM).
+
+ The requirement to provide Installation Information does not include a
+requirement to continue to provide support service, warranty, or updates
+for a work that has been modified or installed by the recipient, or for
+the User Product in which it has been modified or installed. Access to a
+network may be denied when the modification itself materially and
+adversely affects the operation of the network or violates the rules and
+protocols for communication across the network.
+
+ Corresponding Source conveyed, and Installation Information provided,
+in accord with this section must be in a format that is publicly
+documented (and with an implementation available to the public in
+source code form), and must require no special password or key for
+unpacking, reading or copying.
+
+ 7. Additional Terms.
+
+ "Additional permissions" are terms that supplement the terms of this
+License by making exceptions from one or more of its conditions.
+Additional permissions that are applicable to the entire Program shall
+be treated as though they were included in this License, to the extent
+that they are valid under applicable law. If additional permissions
+apply only to part of the Program, that part may be used separately
+under those permissions, but the entire Program remains governed by
+this License without regard to the additional permissions.
+
+ When you convey a copy of a covered work, you may at your option
+remove any additional permissions from that copy, or from any part of
+it. (Additional permissions may be written to require their own
+removal in certain cases when you modify the work.) You may place
+additional permissions on material, added by you to a covered work,
+for which you have or can give appropriate copyright permission.
+
+ Notwithstanding any other provision of this License, for material you
+add to a covered work, you may (if authorized by the copyright holders of
+that material) supplement the terms of this License with terms:
+
+ a) Disclaiming warranty or limiting liability differently from the
+ terms of sections 15 and 16 of this License; or
+
+ b) Requiring preservation of specified reasonable legal notices or
+ author attributions in that material or in the Appropriate Legal
+ Notices displayed by works containing it; or
+
+ c) Prohibiting misrepresentation of the origin of that material, or
+ requiring that modified versions of such material be marked in
+ reasonable ways as different from the original version; or
+
+ d) Limiting the use for publicity purposes of names of licensors or
+ authors of the material; or
+
+ e) Declining to grant rights under trademark law for use of some
+ trade names, trademarks, or service marks; or
+
+ f) Requiring indemnification of licensors and authors of that
+ material by anyone who conveys the material (or modified versions of
+ it) with contractual assumptions of liability to the recipient, for
+ any liability that these contractual assumptions directly impose on
+ those licensors and authors.
+
+ All other non-permissive additional terms are considered "further
+restrictions" within the meaning of section 10. If the Program as you
+received it, or any part of it, contains a notice stating that it is
+governed by this License along with a term that is a further
+restriction, you may remove that term. If a license document contains
+a further restriction but permits relicensing or conveying under this
+License, you may add to a covered work material governed by the terms
+of that license document, provided that the further restriction does
+not survive such relicensing or conveying.
+
+ If you add terms to a covered work in accord with this section, you
+must place, in the relevant source files, a statement of the
+additional terms that apply to those files, or a notice indicating
+where to find the applicable terms.
+
+ Additional terms, permissive or non-permissive, may be stated in the
+form of a separately written license, or stated as exceptions;
+the above requirements apply either way.
+
+ 8. Termination.
+
+ You may not propagate or modify a covered work except as expressly
+provided under this License. Any attempt otherwise to propagate or
+modify it is void, and will automatically terminate your rights under
+this License (including any patent licenses granted under the third
+paragraph of section 11).
+
+ However, if you cease all violation of this License, then your
+license from a particular copyright holder is reinstated (a)
+provisionally, unless and until the copyright holder explicitly and
+finally terminates your license, and (b) permanently, if the copyright
+holder fails to notify you of the violation by some reasonable means
+prior to 60 days after the cessation.
+
+ Moreover, your license from a particular copyright holder is
+reinstated permanently if the copyright holder notifies you of the
+violation by some reasonable means, this is the first time you have
+received notice of violation of this License (for any work) from that
+copyright holder, and you cure the violation prior to 30 days after
+your receipt of the notice.
+
+ Termination of your rights under this section does not terminate the
+licenses of parties who have received copies or rights from you under
+this License. If your rights have been terminated and not permanently
+reinstated, you do not qualify to receive new licenses for the same
+material under section 10.
+
+ 9. Acceptance Not Required for Having Copies.
+
+ You are not required to accept this License in order to receive or
+run a copy of the Program. Ancillary propagation of a covered work
+occurring solely as a consequence of using peer-to-peer transmission
+to receive a copy likewise does not require acceptance. However,
+nothing other than this License grants you permission to propagate or
+modify any covered work. These actions infringe copyright if you do
+not accept this License. Therefore, by modifying or propagating a
+covered work, you indicate your acceptance of this License to do so.
+
+ 10. Automatic Licensing of Downstream Recipients.
+
+ Each time you convey a covered work, the recipient automatically
+receives a license from the original licensors, to run, modify and
+propagate that work, subject to this License. You are not responsible
+for enforcing compliance by third parties with this License.
+
+ An "entity transaction" is a transaction transferring control of an
+organization, or substantially all assets of one, or subdividing an
+organization, or merging organizations. If propagation of a covered
+work results from an entity transaction, each party to that
+transaction who receives a copy of the work also receives whatever
+licenses to the work the party's predecessor in interest had or could
+give under the previous paragraph, plus a right to possession of the
+Corresponding Source of the work from the predecessor in interest, if
+the predecessor has it or can get it with reasonable efforts.
+
+ You may not impose any further restrictions on the exercise of the
+rights granted or affirmed under this License. For example, you may
+not impose a license fee, royalty, or other charge for exercise of
+rights granted under this License, and you may not initiate litigation
+(including a cross-claim or counterclaim in a lawsuit) alleging that
+any patent claim is infringed by making, using, selling, offering for
+sale, or importing the Program or any portion of it.
+
+ 11. Patents.
+
+ A "contributor" is a copyright holder who authorizes use under this
+License of the Program or a work on which the Program is based. The
+work thus licensed is called the contributor's "contributor version".
+
+ A contributor's "essential patent claims" are all patent claims
+owned or controlled by the contributor, whether already acquired or
+hereafter acquired, that would be infringed by some manner, permitted
+by this License, of making, using, or selling its contributor version,
+but do not include claims that would be infringed only as a
+consequence of further modification of the contributor version. For
+purposes of this definition, "control" includes the right to grant
+patent sublicenses in a manner consistent with the requirements of
+this License.
+
+ Each contributor grants you a non-exclusive, worldwide, royalty-free
+patent license under the contributor's essential patent claims, to
+make, use, sell, offer for sale, import and otherwise run, modify and
+propagate the contents of its contributor version.
+
+ In the following three paragraphs, a "patent license" is any express
+agreement or commitment, however denominated, not to enforce a patent
+(such as an express permission to practice a patent or covenant not to
+sue for patent infringement). To "grant" such a patent license to a
+party means to make such an agreement or commitment not to enforce a
+patent against the party.
+
+ If you convey a covered work, knowingly relying on a patent license,
+and the Corresponding Source of the work is not available for anyone
+to copy, free of charge and under the terms of this License, through a
+publicly available network server or other readily accessible means,
+then you must either (1) cause the Corresponding Source to be so
+available, or (2) arrange to deprive yourself of the benefit of the
+patent license for this particular work, or (3) arrange, in a manner
+consistent with the requirements of this License, to extend the patent
+license to downstream recipients. "Knowingly relying" means you have
+actual knowledge that, but for the patent license, your conveying the
+covered work in a country, or your recipient's use of the covered work
+in a country, would infringe one or more identifiable patents in that
+country that you have reason to believe are valid.
+
+ If, pursuant to or in connection with a single transaction or
+arrangement, you convey, or propagate by procuring conveyance of, a
+covered work, and grant a patent license to some of the parties
+receiving the covered work authorizing them to use, propagate, modify
+or convey a specific copy of the covered work, then the patent license
+you grant is automatically extended to all recipients of the covered
+work and works based on it.
+
+ A patent license is "discriminatory" if it does not include within
+the scope of its coverage, prohibits the exercise of, or is
+conditioned on the non-exercise of one or more of the rights that are
+specifically granted under this License. You may not convey a covered
+work if you are a party to an arrangement with a third party that is
+in the business of distributing software, under which you make payment
+to the third party based on the extent of your activity of conveying
+the work, and under which the third party grants, to any of the
+parties who would receive the covered work from you, a discriminatory
+patent license (a) in connection with copies of the covered work
+conveyed by you (or copies made from those copies), or (b) primarily
+for and in connection with specific products or compilations that
+contain the covered work, unless you entered into that arrangement,
+or that patent license was granted, prior to 28 March 2007.
+
+ Nothing in this License shall be construed as excluding or limiting
+any implied license or other defenses to infringement that may
+otherwise be available to you under applicable patent law.
+
+ 12. No Surrender of Others' Freedom.
+
+ If conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License. If you cannot convey a
+covered work so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you may
+not convey it at all. For example, if you agree to terms that obligate you
+to collect a royalty for further conveying from those to whom you convey
+the Program, the only way you could satisfy both those terms and this
+License would be to refrain entirely from conveying the Program.
+
+ 13. Use with the GNU Affero General Public License.
+
+ Notwithstanding any other provision of this License, you have
+permission to link or combine any covered work with a work licensed
+under version 3 of the GNU Affero General Public License into a single
+combined work, and to convey the resulting work. The terms of this
+License will continue to apply to the part which is the covered work,
+but the special requirements of the GNU Affero General Public License,
+section 13, concerning interaction through a network will apply to the
+combination as such.
+
+ 14. Revised Versions of this License.
+
+ The Free Software Foundation may publish revised and/or new versions of
+the GNU General Public License from time to time. Such new versions will
+be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+ Each version is given a distinguishing version number. If the
+Program specifies that a certain numbered version of the GNU General
+Public License "or any later version" applies to it, you have the
+option of following the terms and conditions either of that numbered
+version or of any later version published by the Free Software
+Foundation. If the Program does not specify a version number of the
+GNU General Public License, you may choose any version ever published
+by the Free Software Foundation.
+
+ If the Program specifies that a proxy can decide which future
+versions of the GNU General Public License can be used, that proxy's
+public statement of acceptance of a version permanently authorizes you
+to choose that version for the Program.
+
+ Later license versions may give you additional or different
+permissions. However, no additional obligations are imposed on any
+author or copyright holder as a result of your choosing to follow a
+later version.
+
+ 15. Disclaimer of Warranty.
+
+ THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
+APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
+HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
+OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
+IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
+ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+ 16. Limitation of Liability.
+
+ IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
+THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
+GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
+USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
+DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
+PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
+EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGES.
+
+ 17. Interpretation of Sections 15 and 16.
+
+ If the disclaimer of warranty and limitation of liability provided
+above cannot be given local legal effect according to their terms,
+reviewing courts shall apply local law that most closely approximates
+an absolute waiver of all civil liability in connection with the
+Program, unless a warranty or assumption of liability accompanies a
+copy of the Program in return for a fee.
+
+ END OF TERMS AND CONDITIONS
+
+ How to Apply These Terms to Your New Programs
+
+ If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+ To do so, attach the following notices to the program. It is safest
+to attach them to the start of each source file to most effectively
+state the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+
+ Copyright (C)
+
+ This program is free software: you can redistribute it and/or modify
+ it under the terms of the GNU General Public License as published by
+ the Free Software Foundation, either version 3 of the License, or
+ (at your option) any later version.
+
+ This program is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ GNU General Public License for more details.
+
+ You should have received a copy of the GNU General Public License
+ along with this program. If not, see .
+
+Also add information on how to contact you by electronic and paper mail.
+
+ If the program does terminal interaction, make it output a short
+notice like this when it starts in an interactive mode:
+
+ Copyright (C)
+ This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+ This is free software, and you are welcome to redistribute it
+ under certain conditions; type `show c' for details.
+
+The hypothetical commands `show w' and `show c' should show the appropriate
+parts of the General Public License. Of course, your program's commands
+might be different; for a GUI interface, you would use an "about box".
+
+ You should also get your employer (if you work as a programmer) or school,
+if any, to sign a "copyright disclaimer" for the program, if necessary.
+For more information on this, and how to apply and follow the GNU GPL, see
+.
+
+ The GNU General Public License does not permit incorporating your program
+into proprietary programs. If your program is a subroutine library, you
+may consider it more useful to permit linking proprietary applications with
+the library. If this is what you want to do, use the GNU Lesser General
+Public License instead of this License. But first, please read
+.
diff --git a/Makefile.am b/Makefile.am
index f2689244..af437a64 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -1,2 +1 @@
SUBDIRS = src
-
diff --git a/Makefile.in b/Makefile.in
index 4f0bc031..65fae61e 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -34,7 +34,7 @@ POST_UNINSTALL = :
subdir = .
DIST_COMMON = README $(am__configure_deps) $(srcdir)/Makefile.am \
$(srcdir)/Makefile.in $(srcdir)/config.h.in \
- $(top_srcdir)/configure NEWS install-sh missing
+ $(top_srcdir)/configure COPYING NEWS install-sh missing
ACLOCAL_M4 = $(top_srcdir)/aclocal.m4
am__aclocal_m4_deps = $(top_srcdir)/configure.ac
am__configure_deps = $(am__aclocal_m4_deps) $(CONFIGURE_DEPENDENCIES) \
diff --git a/configure b/configure
index c5979160..0631ff9d 100755
--- a/configure
+++ b/configure
@@ -1,6 +1,6 @@
#! /bin/sh
# Guess values for system-dependent variables and create Makefiles.
-# Generated by GNU Autoconf 2.67 for parallel 20101115.
+# Generated by GNU Autoconf 2.67 for parallel 20101122.
#
# Report bugs to .
#
@@ -551,8 +551,8 @@ MAKEFLAGS=
# Identity of this package.
PACKAGE_NAME='parallel'
PACKAGE_TARNAME='parallel'
-PACKAGE_VERSION='20101115'
-PACKAGE_STRING='parallel 20101115'
+PACKAGE_VERSION='20101122'
+PACKAGE_STRING='parallel 20101122'
PACKAGE_BUGREPORT='bug-parallel@gnu.org'
PACKAGE_URL=''
@@ -1168,7 +1168,7 @@ if test "$ac_init_help" = "long"; then
# Omit some internal or obsolete options to make the list less imposing.
# This message is too long to be a string in the A/UX 3.1 sh.
cat <<_ACEOF
-\`configure' configures parallel 20101115 to adapt to many kinds of systems.
+\`configure' configures parallel 20101122 to adapt to many kinds of systems.
Usage: $0 [OPTION]... [VAR=VALUE]...
@@ -1234,7 +1234,7 @@ fi
if test -n "$ac_init_help"; then
case $ac_init_help in
- short | recursive ) echo "Configuration of parallel 20101115:";;
+ short | recursive ) echo "Configuration of parallel 20101122:";;
esac
cat <<\_ACEOF
@@ -1301,7 +1301,7 @@ fi
test -n "$ac_init_help" && exit $ac_status
if $ac_init_version; then
cat <<\_ACEOF
-parallel configure 20101115
+parallel configure 20101122
generated by GNU Autoconf 2.67
Copyright (C) 2010 Free Software Foundation, Inc.
@@ -1318,7 +1318,7 @@ cat >config.log <<_ACEOF
This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.
-It was created by parallel $as_me 20101115, which was
+It was created by parallel $as_me 20101122, which was
generated by GNU Autoconf 2.67. Invocation command line was
$ $0 $@
@@ -2133,7 +2133,7 @@ fi
# Define the identity of the package.
PACKAGE='parallel'
- VERSION='20101115'
+ VERSION='20101122'
cat >>confdefs.h <<_ACEOF
@@ -2684,7 +2684,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
# report actual input values of CONFIG_FILES etc. instead of their
# values after options handling.
ac_log="
-This file was extended by parallel $as_me 20101115, which was
+This file was extended by parallel $as_me 20101122, which was
generated by GNU Autoconf 2.67. Invocation command line was
CONFIG_FILES = $CONFIG_FILES
@@ -2746,7 +2746,7 @@ _ACEOF
cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`"
ac_cs_version="\\
-parallel config.status 20101115
+parallel config.status 20101122
configured by $0, generated by GNU Autoconf 2.67,
with options \\"\$ac_cs_config\\"
diff --git a/configure.ac b/configure.ac
index a5cc4c1b..d587b1ee 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1,4 +1,4 @@
-AC_INIT([parallel], [20101115], [bug-parallel@gnu.org])
+AC_INIT([parallel], [20101122], [bug-parallel@gnu.org])
AM_INIT_AUTOMAKE([-Wall -Werror foreign])
AC_CONFIG_HEADERS([config.h])
AC_CONFIG_FILES([
diff --git a/doc/FUTURE_IDEAS b/doc/FUTURE_IDEAS
index dc9c1aa5..db27f1e8 100644
--- a/doc/FUTURE_IDEAS
+++ b/doc/FUTURE_IDEAS
@@ -1,3 +1,10 @@
+parallel: Argument handling re-written to OO.
+The code is quite messy, the implementation is fairly slow, but the
+structure seems sound and it passes the testsuite.
+basename {/} and {/.} implemented.
+Flushing of STDERR and STDOUT after each job completes.
+
+
== Compare ==
http://code.google.com/p/spawntool/
@@ -22,6 +29,10 @@ Only negative.
Bug: {2} in --return
parallel -N2 --workdir ... --cleanup --transfer --return {2}.2 -S .. -v echo {} ">{2}.2" ::: ./tmp/foo ./tmp/bar
+== load ==
+
+Include niceload
+Have an option to not spawn new tasks until load is below a limit computed like number of jobs.
== SQL ==
diff --git a/src/parallel b/src/parallel
index 7a770dd3..b52aa015 100755
--- a/src/parallel
+++ b/src/parallel
@@ -2468,14 +2468,65 @@ use POSIX qw(:sys_wait_h setsid);
use File::Temp qw(tempfile tempdir);
use Getopt::Long;
use strict;
+use Carp;
+
+$::oodebug=0;
+$Global::original_sigterm = $SIG{TERM};
+$SIG{TERM} = sub {}; # Dummy until jobs really start
do_not_reap();
parse_options();
+my $number_of_args;
+if($Global::max_number_of_args) {
+ $number_of_args=$Global::max_number_of_args;
+} elsif ($Global::Xargs or $Global::xargs) {
+ $number_of_args = undef;
+} else {
+ $number_of_args = 1;
+}
+
+my @fhlist;
+@fhlist = map { open_or_exit($_) } @::opt_a;
+if(not @fhlist) {
+ @fhlist = (*STDIN);
+}
+if($::opt_skip_first_line) {
+ # Skip the first line for the first file handle
+ my $fh = $fhlist[0];
+ <$fh>;
+# for my $fh (@fhlist) {
+# <$fh>; # Read first line and forget it
+# }
+}
+
+$Global::CommandLineQueue = CommandLineQueue->new(join(" ",@ARGV),\@fhlist,$Global::Xargs,$number_of_args,\@Global::ret_files);
+#my $cmdline = $cmdqueue->get();
+#
+#my_dump($cmdline);
+#print "Replaced(",length $cmdline->replaced(),"):",$cmdline->replaced(),"\n";
+#print $cmdline->len(),"\n";
+#
+#while(not $cmdqueue->empty()) {
+# my $cmdline = $cmdqueue->get();
+# print "Replaced(",length $cmdline->replaced(),"):",$cmdline->replaced(),"\n";
+# my_dump($cmdline);
+# print $cmdline->len(),"\n";
+#}
+#exit;
+
init_run_jobs();
+if(defined $::opt_P) {
+ compute_number_of_processes_for_sshlogins();
+}
my $sem;
if($Global::semaphore) {
+ # $Global::host{':'}{'max_no_of_running'} must be set
+ if(not defined $Global::host{':'}{'max_no_of_running'}) {
+ compute_number_of_processes_for_sshlogins();
+ }
$sem = acquire_semaphore();
}
+$SIG{TERM} = \&start_no_new_jobs;
start_more_jobs();
reap_if_needed();
drain_job_queue();
@@ -2539,8 +2590,8 @@ sub get_options_from_array {
"keep-order|keeporder|k" => \$::opt_k,
"group|g" => \$::opt_g,
"ungroup|u" => \$::opt_u,
- "command|c" => \$::opt_c,
- "file|f" => \$::opt_f,
+# "command|c" => \$::opt_c,
+# "file|f" => \$::opt_f,
"null|0" => \$::opt_0,
"quote|q" => \$::opt_q,
"I=s" => \$::opt_I,
@@ -2608,18 +2659,20 @@ sub get_options_from_array {
sub parse_options {
# Returns: N/A
# Defaults:
- $Global::version = 20101115;
+ $Global::version = 20101122;
$Global::progname = 'parallel';
$Global::debug = 0;
$Global::verbose = 0;
$Global::grouped = 1;
$Global::keeporder = 0;
$Global::quoting = 0;
- $Global::replacestring = '{}';
- $Global::replace_no_ext = '{.}';
+ $Global::replace{'{}'} = '{}';
+ $Global::replace{'{.}'} = '{.}';
+ $Global::replace{'{/}'} = '{/}';
+ $Global::replace{'{/.}'} = '{/.}';
$/="\n";
$Global::ignore_empty = 0;
- $Global::argfile = *STDIN;
+ #$Global::argfile = *STDIN;
$Global::interactive = 0;
$Global::stderr_verbose = 0;
$Global::default_simultaneous_sshlogins = 9;
@@ -2640,17 +2693,17 @@ sub parse_options {
if(defined $::opt_k) { $Global::keeporder = 1; }
if(defined $::opt_g) { $Global::grouped = 1; }
if(defined $::opt_u) { $Global::grouped = 0; }
- if(defined $::opt_c) { $Global::input_is_filename = 0; }
- if(defined $::opt_f) { $Global::input_is_filename = 1; }
if(defined $::opt_0) { $/ = "\0"; }
if(defined $::opt_d) { my $e="sprintf \"$::opt_d\""; $/ = eval $e; }
if(defined $::opt_p) { $Global::interactive = $::opt_p; }
if(defined $::opt_q) { $Global::quoting = 1; }
if(defined $::opt_r) { $Global::ignore_empty = 1; }
if(defined $::opt_verbose) { $Global::stderr_verbose = 1; }
- if(defined $::opt_I) { $Global::replacestring = $::opt_I; }
- if(defined $::opt_U) { $Global::replace_no_ext = $::opt_U; }
- if(defined $::opt_i and $::opt_i) { $Global::replacestring = $::opt_i; }
+ if(defined $::opt_I) { $Global::replace{'{}'} = $::opt_I; }
+ if(defined $::opt_U) { $Global::replace{'{.}'} = $::opt_U; }
+ if(defined $::opt_i and $::opt_i) { $Global::replace{'{}'} = $::opt_i; }
+ if(defined $::opt_basenamereplace) { $Global::replace{'{/}'} = $::opt_basenamereplace; }
+ if(defined $::opt_basenameextensionreplace) { $Global::replace{'{/.}'} = $::opt_basenameextensionreplace; }
if(defined $::opt_E and $::opt_E) { $Global::end_of_file_string = $::opt_E; }
if(defined $::opt_n and $::opt_n) { $Global::max_number_of_args = $::opt_n; }
if(defined $::opt_N and $::opt_N) { $Global::max_number_of_args = $::opt_N; }
@@ -2661,7 +2714,7 @@ sub parse_options {
if(defined $::opt_arg_file_sep) { $Global::arg_file_sep = $::opt_arg_file_sep; }
if(defined $::opt_number_of_cpus) { print no_of_cpus(),"\n"; wait_and_exit(0); }
if(defined $::opt_number_of_cores) { print no_of_cores(),"\n"; wait_and_exit(0); }
- if(defined $::opt_max_line_length_allowed) { print real_max_length(),"\n"; wait_and_exit(0); }
+ if(defined $::opt_max_line_length_allowed) { print CommandMaxLength::real_max_length(),"\n"; wait_and_exit(0); }
if(defined $::opt_version) { version(); wait_and_exit(0); }
if(defined $::opt_show_limits) { show_limits(); }
if(defined @::opt_sshlogin) { @Global::sshlogin = @::opt_sshlogin; }
@@ -2681,7 +2734,14 @@ sub parse_options {
if(defined $::opt_L and $::opt_L or defined $::opt_l) {
$Global::max_lines = $::opt_l || $::opt_L || 1;
$Global::max_number_of_args ||= $Global::max_lines;
+# warn $Global::max_lines;
+# warn $Global::max_number_of_args;
}
+ if($::opt_s) {
+ #$Global::max_number_of_args ||= 10000;
+ }
+ if($::opt_n || $::opt_N) { }
+ %Global::replace_rev = reverse %Global::replace;
if(grep /^$Global::arg_sep$/o, @ARGV) {
# Deal with :::
@@ -2693,31 +2753,14 @@ sub parse_options {
@ARGV=convert_argfiles_from_command_line_to_multiple_opt_a();
}
- # must be done after ::: and :::: because they mess with @ARGV
- $Global::input_is_filename ||= (@ARGV);
-
- if(@::opt_a) {
- # must be done after
- # convert_argfiles_from_command_line_to_multiple_opt_a
- if($#::opt_a == 0) {
- # One -a => xargs compatibility
- $Global::argfile = open_or_exit($::opt_a[0]);
- if($::opt_skip_first_line) {
- <$Global::argfile>; # Read first line and forget it
- }
- } else {
- # Multiple -a => xapply style
- argfiles_xapply_style();
- }
- }
-
if(($::opt_l || $::opt_L || $::opt_n || $::opt_N || $::opt_s ||
$::opt_colsep) and not ($::opt_m or $::opt_X)) {
# The options --max-line, -l, -L, --max-args, -n, --max-chars, -s
# do not make sense without -X or -m
# so default to -X
# Needs to be done after :::: and @opt_a, as they can set $::opt_N
- $Global::Xargs = 1;
+ #$Global::Xargs = 1;
+ # TODO Somehow set context_replace or not
}
# Semaphore defaults
@@ -2726,8 +2769,11 @@ sub parse_options {
$Global::semaphore ||= ($0 =~ m:(^|/)sem$:); # called as 'sem'
if($Global::semaphore) {
# A semaphore does not take input from neither stdin nor file
- $Global::argfile = open_or_exit("/dev/null");
- unget_arg("");
+ @::opt_a = ("/dev/null");
+ push(@Global::unget_argv, [Arg->new("")]);
+ #$Global::argfile = open_or_exit("/dev/null");
+ #$Global::arg_queue ||= ArgQueue->new([$Global::argfile]);
+ #$Global::arg_queue->unget(Arg->new(""));
$Semaphore::timeout = $::opt_semaphoretimeout || 0;
if(defined $::opt_semaphorename) {
$Semaphore::name = $::opt_semaphorename;
@@ -2739,16 +2785,9 @@ sub parse_options {
$Semaphore::wait = $::opt_wait;
$Global::default_simultaneous_sshlogins = 1;
}
-
if(defined $::opt_eta) {
- # must be done after opt_a because we need to read all args
$::opt_progress = $::opt_eta;
- my @args = ();
- while(more_arguments()) {
- # This will read all arguments and compute $Global::total_jobs
- push @args, get_arg();
- }
- unget_arg(@args);
+ # use $CommandLineQueue->size(); when needed
}
if(@ARGV) {
@@ -2772,17 +2811,13 @@ sub parse_options {
# Needs to be done after setting $Global::command and $Global::command_line_max_len
# as '-m' influences the number of commands that needs to be run
if(defined $::opt_P) {
- compute_number_of_processes_for_sshlogins();
+# compute_number_of_processes_for_sshlogins();
} else {
for my $sshlogin (keys %Global::host) {
$Global::host{$sshlogin}{'max_no_of_running'} =
$Global::default_simultaneous_sshlogins;
}
}
-
- if(-t $Global::argfile) {
- print STDERR "$Global::progname: Input is tty. Press CTRL-D to exit.\n";
- }
}
sub read_options {
@@ -2858,24 +2893,34 @@ sub read_args_from_command_line {
while(@ARGV) {
my $arg = shift @ARGV;
if($arg eq $Global::arg_sep) {
- $Global::input_is_filename = (@new_argv);
+ my $prepend="";
while(@ARGV) {
my $arg = shift @ARGV;
+ if($Global::ignore_empty) {
+ if($arg =~ /^\s*$/) { next; }
+ }
if($Global::end_of_file_string and
$arg eq $Global::end_of_file_string) {
# Ignore the rest of ARGV
@ARGV=();
+ if($prepend) {
+ push(@Global::unget_argv, [Arg->new($prepend)]);
+ $Global::total_jobs++;
+ }
+ last;
}
- if($Global::ignore_empty) {
- if($arg =~ /^\s*$/) { next; }
+ if($prepend) {
+ $arg = $prepend.$arg; # For line continuation
+ $prepend = ""; #undef;
}
- if($Global::max_lines and $#ARGV >=0) {
+ if($Global::max_lines) {
if($arg =~ /\s$/) {
# Trailing space => continued on next line
- $arg .= shift @ARGV;
+ $prepend = $arg;
+ redo;
}
}
- unget_argv($arg);
+ push(@Global::unget_argv, [Arg->new($arg)]);
$Global::total_jobs++;
}
last;
@@ -2914,39 +2959,46 @@ sub argfiles_xapply_style {
# Every n'th entry is from the same file
# Set opt_N to read n entries per invocation
# Returns: N/A
- $Global::argfile = open_or_exit("/dev/null");
+ #$Global::argfile = open_or_exit("/dev/null");
$::opt_N = $#::opt_a+1;
$Global::max_number_of_args = $#::opt_a+1;
- # read the files
- my @content;
- my $max_lineno = 0;
- my $in_fh = gensym;
- for (my $fileno = 0; $fileno <= $#::opt_a; $fileno++) {
- $in_fh = open_or_exit($::opt_a[$fileno]);
- if($::opt_skip_first_line and $fileno == 0) {
- <$in_fh>; # Read first line and forget it
- }
- for (my $lineno=0;
- $content[$fileno][$lineno] = get_arg($in_fh);
- $lineno++) {
- $max_lineno = max($max_lineno,$lineno);
- }
- close $in_fh;
- }
- for (my $lineno=0; $lineno <= $max_lineno; $lineno++) {
- for (my $fileno = 0; $fileno <= $#::opt_a; $fileno++) {
- my $arg = $content[$fileno][$lineno];
- if($Global::trim ne 'n') {
- $arg = trim($arg);
- }
- if(defined $arg) {
- unget_arg($arg);
- } else {
- unget_arg("");
- }
- }
- }
- $Global::total_jobs += $max_lineno;
+# my @fhlist = map { open_or_exit($_) } @::opt_a;
+# $Global::arg_queue = RecordQueue->new(\@fhlist);
+# ::my_dump($Global::arg_queue);
+
+# # read the files
+# my @content;
+# my $max_lineno = 0;
+# my $in_fh = gensym;
+# for (my $fileno = 0; $fileno <= $#::opt_a; $fileno++) {
+# $in_fh = open_or_exit($::opt_a[$fileno]);
+# $Global::arg_queue{$in_fh} ||= ArgQueue->new([$in_fh]);
+# if($::opt_skip_first_line and $fileno == 0) {
+# <$in_fh>; # Read first line and forget it
+# }
+# for (my $lineno=0;
+# $content[$fileno][$lineno] = $Global::arg_queue{$in_fh}->get();
+# $lineno++) {
+# $max_lineno = max($max_lineno,$lineno);
+# }
+# close $in_fh;
+# }
+# for (my $lineno=0; $lineno <= $max_lineno; $lineno++) {
+# for (my $fileno = 0; $fileno <= $#::opt_a; $fileno++) {
+# my $arg = $content[$fileno][$lineno];
+# if($Global::trim ne 'n') {
+# $arg = trim($arg);
+# }
+# $Global::arg_queue ||= ArgQueue->new([$Global::argfile]);
+# if(defined $arg) {
+# $Global::arg_queue->unget($arg);
+# } else {
+# die;
+# $Global::arg_queue->unget(Arg->new(""));
+# }
+# }
+# }
+# $Global::total_jobs += $max_lineno;
}
sub open_or_exit {
@@ -2975,225 +3027,71 @@ sub cleanup {
# Generating the command line
#
-sub no_extension {
- # Returns:
- # argument with .extension removed if any
- my $no_ext = shift;
- $no_ext =~ s:\.[^/\.]*$::; # Remove .ext from argument
- return $no_ext;
-}
-sub trim {
- # Removes white space as specifed by --trim:
- # n = nothing
- # l = start
- # r = end
- # lr|rl = both
- # Returns:
- # string with white space removed as needed
- my (@strings) = map { defined $_ ? $_ : "" } (@_);
- my $arg;
- if($Global::trim eq "n") {
- # skip
- } elsif($Global::trim eq "l") {
- for $arg (@strings) { $arg =~ s/^\s+//; }
- } elsif($Global::trim eq "r") {
- for $arg (@strings) { $arg =~ s/\s+$//; }
- } elsif($Global::trim eq "rl" or $Global::trim eq "lr") {
- for $arg (@strings) { $arg =~ s/^\s+//; $arg =~ s/\s+$//; }
- } else {
- print STDERR "$Global::progname: --trim must be one of: r l rl lr\n";
- wait_and_exit(255);
- }
- return wantarray ? @strings : "@strings";
-}
+#sub generate_command_line {
+# # Returns:
+# # the full job line to run
+# # list of quoted arguments on that line
+# my $command = shift;
+# my ($job_line,$last_good);
+# my ($quoted_args) =
+# get_multiple_args($command,Limits::Command::max_length());
+#
+# if(@$quoted_args) {
+# $job_line = $command;
+# die;
+# $job_line = context_replace($command, @$quoted_args);
+# debug("Return jobline(",length($job_line),"): !",$job_line,"!\n");
+# }
+# return ($job_line,$quoted_args);
+#}
-
-sub generate_command_line {
- # Returns:
- # the full job line to run
- # list of quoted arguments on that line
- my $command = shift;
- my ($job_line,$last_good);
- my ($quoted_args,$quoted_args_no_ext) =
- get_multiple_args($command,max_length_of_command_line(),0);
- my $is_substituted = 0;
-
- if(@$quoted_args) {
- $job_line = $command;
- if(defined $job_line and
- ($job_line =~/\Q$Global::replacestring\E/o or
- $job_line =~/\Q$Global::replace_no_ext\E/o)) {
- # substitute {} and {.} with args
- if($Global::Xargs) {
- # Context sensitive replace (foo{}bar with fooargsbar)
- $job_line =
- context_replace($job_line, $quoted_args, $quoted_args_no_ext);
- } else {
- # Normal replace {} with args and {.} with args without extension
- my $arg=join(" ",@$quoted_args);
- my $arg_no_ext=join(" ",@$quoted_args_no_ext);
- $job_line =~ s/\Q$Global::replacestring\E/$arg/go;
- $job_line =~ s/\Q$Global::replace_no_ext\E/$arg_no_ext/go;
- }
- $is_substituted = 1;
- }
- if(defined $job_line and $::opt_N) {
- if($job_line =~/\{\d+\}/o) {
- # substitute {#} with args
- for my $argno (1..$::opt_N) {
- my $arg = $quoted_args->[$argno-1];
- if(defined $arg) {
- $job_line =~ s/\{$argno\}/$arg/g;
- } else {
- $job_line =~ s/\{$argno\}//g;
- }
- }
- $is_substituted = 1;
- }
- if($job_line =~/\{\d+\.\}/o) {
- # substitute {#.} with args
- for my $argno (1..$::opt_N) {
- my $arg = no_extension($quoted_args->[$argno-1]);
- if(defined $arg) {
- $job_line =~ s/\{$argno\.\}/$arg/g;
- } else {
- $job_line =~ s/\{$argno\.\}//g;
- }
- }
- $is_substituted = 1;
- }
- }
- if (not $is_substituted) {
- # append args
- my $arg=join(" ",@$quoted_args);
- if($job_line) {
- $job_line .= " ".$arg;
- } else {
- # Parallel behaving like '|sh'
- $job_line = $arg;
- }
- }
- debug("Return jobline(",length($job_line),"): !",$job_line,"!\n");
- }
- return ($job_line,$quoted_args);
-}
-
-sub get_multiple_args {
- # Returns:
- # \@quoted_args - empty if no more args
- # \@quoted_args_no_ext
- my ($command,$max_length_of_command_line,$test_only_mode) = (@_);
- my ($next_arg,@quoted_args,@quoted_args_no_ext,$arg_length);
- my ($number_of_substitution,
- $number_of_substitution_no_ext,$spaces,
- $length_of_command_no_args,$length_of_context) =
- xargs_computations($command);
- my $number_of_args = 0;
- while (defined($next_arg = get_arg())) {
- my $next_arg_no_ext = no_extension($next_arg);
- push (@quoted_args, $next_arg);
- push (@quoted_args_no_ext, $next_arg_no_ext);
- $number_of_args++;
-
- # Emulate xargs if there is a command and -x or -X is set
- my $next_arg_len =
- $number_of_substitution * (length ($next_arg) + $spaces)
- + $number_of_substitution_no_ext * (length ($next_arg_no_ext) + $spaces)
- + $length_of_context;
- $arg_length += $next_arg_len;
- my $job_line_length = $length_of_command_no_args + $arg_length;
- if($job_line_length >= $max_length_of_command_line) {
- unget_arg(pop @quoted_args);
- pop @quoted_args_no_ext;
- if($test_only_mode) {
- last;
- }
- if($::opt_x and $length_of_command_no_args + $next_arg_len
- >= $max_length_of_command_line) {
- # To be compatible with xargs -x
- print STDERR ("Command line too long ($job_line_length >= "
- . $max_length_of_command_line .
- ") at number $number_of_args: ".
- (substr($next_arg,0,50))."...\n");
- wait_and_exit(255);
- }
- if(defined $quoted_args[0]) {
- last;
- } else {
- print STDERR ("Command line too long ($job_line_length >= "
- . $max_length_of_command_line .
- ") at number $number_of_args: ".
- (substr($next_arg,0,50))."...\n");
- wait_and_exit(255);
- }
- }
- if($Global::max_number_of_args and
- $number_of_args >= $Global::max_number_of_args) {
- last;
- }
- if(not $Global::xargs and not $Global::Xargs) {
- # No xargs-mode: Just one argument per line
- last;
- }
- }
- return (\@quoted_args,\@quoted_args_no_ext);
-}
-
-
-sub xargs_computations {
- # Returns:
- # $number_of_substitution = number of {}'s
- # $number_of_substitution_no_ext = number of {.}'s
- # $spaces = is a single space needed at the start?
- # $length_of_command_no_args = length of command line with args removed
- # $length_of_context = context needed for each additional arg
-
- my $command = shift;
- if(not @Calculated::xargs_computations) {
- my ($number_of_substitution, $number_of_substitution_no_ext,
- $spaces,$length_of_command_no_args,$length_of_context)
- = (1,0,0,0,0);
- if($command) {
- if($command !~ /\s\S*\Q$Global::replacestring\E\S*|\s\S*\Q$Global::replace_no_ext\E\S*/o) {
- # No replacement strings: add {}
- $command .= " ".$Global::replacestring;
- }
- # Count number of {}'s on the command line
- my $no_of_replace =
- ($command =~ s/\Q$Global::replacestring\E/$Global::replacestring/go);
- $number_of_substitution = $no_of_replace || 1;
- # Count number of {.}'s on the command line
- my $no_of_no_ext =
- ($command =~ s/\Q$Global::replace_no_ext\E/$Global::replace_no_ext/go);
- $number_of_substitution_no_ext = $no_of_no_ext || 0;
- # Count
- my $c = $command;
- if($Global::Xargs) {
- $c =~ s/\s\S*\Q$Global::replacestring\E\S*|\s\S*\Q$Global::replace_no_ext\E\S*//go;
- $length_of_command_no_args = length($c);
- $length_of_context = length($command) - $length_of_command_no_args
- - $no_of_replace * length($Global::replacestring)
- - $no_of_no_ext * length($Global::replace_no_ext);
- $spaces = 0;
- debug("length_of_command_no_args ",$length_of_command_no_args,"\n");
- debug("length_of_context ",$length_of_context,"\n");
- debug("no_of_replace ",$no_of_replace," no_of_no_ext ",$no_of_no_ext,"\n");
- } else {
- # remove all {}s
- $c =~ s/\Q$Global::replacestring\E|\Q$Global::replace_no_ext\E//og;
- $length_of_command_no_args = length($c) -
- $no_of_replace - $no_of_no_ext;
- $length_of_context = 0;
- $spaces = 1;
- }
- }
- @Calculated::xargs_computations =
- ($number_of_substitution, $number_of_substitution_no_ext,
- $spaces,$length_of_command_no_args,$length_of_context);
- }
- return (@Calculated::xargs_computations);
-}
+# sub get_multiple_args {
+# # Returns:
+# # \@quoted_args - empty if no more args
+# my ($command,$max_length_of_command_line) = (@_);
+# my ($next_arg,@args,$arg_length);
+# my $number_of_args = 0;
+# $Global::arg_queue ||= ArgQueue->new($Global::argfile);
+# while (defined($next_arg = $Global::arg_queue->get())) {
+# push (@args, $next_arg);
+# ::debug("Next '$next_arg'\n");
+# $number_of_args++;
+#
+# # Emulate xargs if there is a command and -x or -X is set
+# my $next_arg_len = 1;
+#
+# my $job_line = context_replace($command, @args);
+# my $job_line_length = length $job_line;
+#
+# if($job_line_length >= $max_length_of_command_line) {
+# $Global::arg_queue ||= ArgQueue->new($Global::argfile);
+# $Global::arg_queue->unget(pop @args);
+# if($::opt_x and $number_of_args == 1 and
+# $job_line_length >= $max_length_of_command_line) {
+# # To be compatible with xargs -x
+# }
+# if(defined $args[0]) {
+# last;
+# } else {
+# print STDERR ("Command line too long ($job_line_length >= "
+# . $max_length_of_command_line .
+# ") at number $number_of_args: ".
+# (substr($next_arg,0,50))."...\n");
+# wait_and_exit(255);
+# }
+# }
+# if($Global::max_number_of_args and
+# $number_of_args >= $Global::max_number_of_args) {
+# last;
+# }
+# if(not $Global::xargs and not $Global::Xargs) {
+# # No xargs-mode: Just one argument per line
+# last;
+# }
+# }
+# return (\@args);
+# }
sub shell_quote {
@@ -3229,115 +3127,61 @@ sub shell_unquote {
}
-sub context_replace {
- # Replace foo{}bar or foo{.}bar
- # Returns:
- # jobline with {} and {.} expanded to args
- my ($job_line,$quoted,$no_ext) = (@_);
- while($job_line =~/\Q$Global::replacestring\E|\Q$Global::replace_no_ext\E/o) {
- $job_line =~ /(\S*(\Q$Global::replacestring\E|\Q$Global::replace_no_ext\E)\S*)/o
- or die ("This should never happen");
- my $wordarg = $1; # This is the context that needs to be substituted
- my @all_word_arg;
- for my $n (0 .. $#$quoted) {
- my $arg = $quoted->[$n];
- my $arg_no_ext = $no_ext->[$n];
- my $substituted = $wordarg;
- $substituted=~s/\Q$Global::replacestring\E/$arg/go;
- $substituted=~s/\Q$Global::replace_no_ext\E/$arg_no_ext/go;
- push @all_word_arg, $substituted;
- }
- my $all_word_arg = join(" ",@all_word_arg);
- $job_line =~ s/\Q$wordarg\E/$all_word_arg/;
- }
- return $job_line;
-}
+# sub context_replace {
+# return replace(1,@_);
+# }
+#
+# sub replace {
+# # Replace foo{...}bar
+# # Returns:
+# # jobline with {...} expanded to the relevant args
+# # $job_line, @args
+# # Create replacement matrix:
+# # $replace{n}{"{}"} = replacement of {} for arg n
+# # $replace{-1}{"{n}"} = replacement of {n}
+# my($context_replace, $job_line, @args) = (@_);
+# my $context_regexp = $context_replace ? '\S*' : ''; # Regexp to match surrounding context
+# my %replace;
+# my %replace_single;
+# my %replace_multi;
+# die;
+# for my $n (0 .. $#args) {
+# my $m = $n+1;
+# $replace{'{}'} = $args[$n]->orig(); # {}
+# $replace{'{.}'} = $args[$n]->no_extension(); # {.}
+# $replace{'{/}'} = $args[$n]->basename(); # {/}
+# $replace{'{/.}'} = $args[$n]->basename_no_extension(); # {/.}
+# $replace_single{"{$m}"} = $replace{'{}'}; # {2}
+# $replace_single{"{$m.}"} = $replace{'{.}'}; # {2.}
+# $replace_single{"{$m/}"} = $replace{'{/}'}; # {2/}
+# $replace_single{"{$m/.}"} = $replace{'{/.}'}; # {2/.}
+# push(@{$replace_multi{$Global::replace{"{}"}}}, $replace{'{}'}); # {}
+# push(@{$replace_multi{$Global::replace{"{.}"}}}, $replace{'{.}'}); # {.}
+# push(@{$replace_multi{$Global::replace{"{/}"}}}, $replace{'{/}'}); # {/}
+# push(@{$replace_multi{$Global::replace{"{/.}"}}}, $replace{'{/.}'}); # {/.}
+# }
+# my $single_regexp = join('|', map {$_=~s/(\W)/\\$1/g; $_} sort keys %replace_single);
+# my $replacements = 0;
+# die;
+# $replacements += ($job_line =~ s/($single_regexp)/$replace_single{$1}/ge);
+# my $multi_regexp = join('|', map {$_=~s/(\W)/\\$1/g; $_} sort keys %replace_multi);
+# $replacements += ($job_line =~ s/($context_regexp)($multi_regexp)($context_regexp)/
+# join(" ",map {$1.$_.$3} @{$replace_multi{$2}})/gex);
+# die;
+# if(not $replacements) {
+# # no {...} in line. Add one and replace that
+# $job_line .=" {}";
+# $replacements += ($job_line =~ s/($context_regexp)($multi_regexp)($context_regexp)/
+# join(" ",map {$1.$_.$3} @{$replace_multi{$2}})/gex);
+# if(not $replacements) {
+# warn("No replacements. This should not happen");
+# }
+# }
+# return $job_line;
+# }
sub __NUMBER_OF_PROCESSES_FILEHANDLES_MAX_LENGTH_OF_COMMAND_LINE__ {}
-# Maximal command line length (for -m and -X)
-sub max_length_of_command_line {
- # Find the max_length of a command line
- # Returns:
- # number of chars on the longest command line allowed
- if(not $Private::command_line_max_len) {
- $Private::command_line_max_len = limited_max_length();
- if($::opt_s) {
- if($::opt_s <= $Private::command_line_max_len) {
- $Private::command_line_max_len = $::opt_s;
- } else {
- print STDERR "$Global::progname: ",
- "value for -s option should be < $Private::command_line_max_len\n";
- }
- }
- }
- return $Private::command_line_max_len;
-}
-
-sub max_length_limited_by_opt_s {
- # Returns:
- # min(opt_s, number of chars on the longest command line allowed)
- if(is_acceptable_command_line_length($::opt_s)) {
- debug("-s is OK: ",$::opt_s,"\n");
- return $::opt_s;
- }
- # -s is too long: Find the correct
- return binary_find_max_length(0,$::opt_s);
-}
-
-sub limited_max_length {
- # Returns:
- # min(opt_s, number of chars on the longest command line allowed)
- if($::opt_s) { return max_length_limited_by_opt_s() }
-
- return real_max_length();
-}
-
-sub real_max_length {
- # Returns:
- # The maximal command line length
- # Use an upper bound of 8 MB if the shell allows for for infinite long lengths
- my $upper = 8_000_000;
- my $len = 8;
- do {
- if($len > $upper) { return $len };
- $len *= 16;
- } while (is_acceptable_command_line_length($len));
- # Then search for the actual max length between 0 and upper bound
- return binary_find_max_length(int($len/16),$len);
-}
-
-sub binary_find_max_length {
- # Given a lower and upper bound find the max_length of a command line
- # Returns:
- # number of chars on the longest command line allowed
- my ($lower, $upper) = (@_);
- if($lower == $upper or $lower == $upper-1) { return $lower; }
- my $middle = int (($upper-$lower)/2 + $lower);
- debug("Maxlen: $lower,$upper,$middle\n");
- if (is_acceptable_command_line_length($middle)) {
- return binary_find_max_length($middle,$upper);
- } else {
- return binary_find_max_length($lower,$middle);
- }
-}
-
-sub is_acceptable_command_line_length {
- # Test if a command line of this length can run
- # Returns:
- # 0 if the command line length is too long
- # 1 otherwise
- my $len = shift;
-
- $Private::is_acceptable_command_line_length++;
- debug("$Private::is_acceptable_command_line_length $len\n");
- local *STDERR;
- open (STDERR,">/dev/null");
- system "true "."x"x$len;
- close STDERR;
- debug("$len $?\n");
- return not $?;
-}
# Number of parallel processes to run
@@ -3375,7 +3219,7 @@ sub processes_available_by_system_limit {
my $system_limit=0;
my @command_lines=();
my ($next_command_line, $args_ref);
- my $more_filehandles;
+ my $more_filehandles=1;
my $max_system_proc_reached=0;
my $slow_spawining_warning_printed=0;
my $time = time;
@@ -3389,18 +3233,20 @@ sub processes_available_by_system_limit {
for my $i (1..8) {
open($fh{"init-$i"},"empty() or $Global::semaphore)
+ and $more_filehandles
+ and not $max_system_proc_reached) {
$system_limit++;
if(not $Global::semaphore) {
# If there are no more command lines, then we have a process
# per command line, so no need to go further
- ($next_command_line, $args_ref) = get_command_line();
- if(defined $next_command_line) {
- push(@command_lines, $next_command_line, $args_ref);
- }
- }
+ ($next_command_line) = $Global::CommandLineQueue->get();
+ push(@command_lines, $next_command_line);
+ }
+
# Every simultaneous process uses 2 filehandles when grouping
$more_filehandles = open($fh{$system_limit*2},"unget(@command_lines);
if($sshlogin ne ":" and
$system_limit > $Global::default_simultaneous_sshlogins) {
$system_limit =
@@ -3792,6 +3635,12 @@ sub undef_as_zero {
return $a ? $a : 0;
}
+
+sub undef_as_empty {
+ my $a = shift;
+ return $a ? $a : "";
+}
+
sub hostname {
if(not $Private::hostname) {
my $hostname = `hostname`;
@@ -3825,7 +3674,6 @@ sub __RUNNING_AND_PRINTING_THE_JOBS__ {}
# $Global::total_started = total jobs started
# $Global::total_jobs = total jobs to be started at all
# $Global::total_completed = total jobs completed
-# @Global::unget_arg = arguments quoted as needed ready to use
# @Global::unget_lines = raw argument lines - needs quoting and splitting
#
# Flow:
@@ -3848,8 +3696,6 @@ sub init_run_jobs {
$Global::tty_taken = 0;
$SIG{USR1} = \&list_running_jobs;
$SIG{USR2} = \&toggle_progress;
- $Global::original_sigterm = $SIG{TERM};
- $SIG{TERM} = \&start_no_new_jobs;
if(@::opt_basefile) {
setup_basefile();
}
@@ -4082,6 +3928,9 @@ sub start_more_jobs {
debug("Running jobs on $sshlogin: $Global::host{$sshlogin}{'no_of_running'}\n");
while ($Global::host{$sshlogin}{'no_of_running'} <
$Global::host{$sshlogin}{'max_no_of_running'}) {
+ if($Global::CommandLineQueue->empty()) {
+ last;
+ }
if(start_another_job($sshlogin) == 0) {
# No more jobs to start
last;
@@ -4104,9 +3953,16 @@ sub start_another_job {
my $sshlogin = shift;
# Do we have enough file handles to start another job?
if(enough_file_handles()) {
- my ($command,$clean_command) = get_command_line_with_sshlogin($sshlogin);
- if(defined $command) {
- debug("Command to run on '$sshlogin': $command\n");
+ if($Global::CommandLineQueue->empty()) {
+ # No more commands to run
+ return 0;
+ } else {
+ my ($command,$clean_command) = get_command_line_with_sshlogin($sshlogin);
+ if(not defined $command) {
+ # No command available for that sshlogin
+ return 0;
+ }
+ debug("Command to run on '$sshlogin': '$command'\n");
my %jobinfo = start_job($command,$sshlogin,$clean_command);
if(%jobinfo) {
$Global::running{$jobinfo{"pid"}} = \%jobinfo;
@@ -4116,9 +3972,6 @@ sub start_another_job {
# If interactive says: Dont run the job, then skip it and run the next
return start_another_job($sshlogin);
}
- } else {
- # No more commands to run
- return 0;
}
} else {
# No more file handles
@@ -4270,12 +4123,14 @@ sub print_job {
while(sysread($err,$buf,1000_000)) {
print STDERR $buf;
}
+ flush STDERR;
if($Global::debug) {
print STDOUT "OUT:\n";
}
while(sysread($out,$buf,1000_000)) {
print STDOUT $buf;
}
+ flush STDOUT;
debug("<empty()) {
+ Carp::confess("get_command_line_with_sshlogin should never be called if empty");
+ }
+ my ($next_command_ref) = $Global::CommandLineQueue->get();
+ if(not defined $next_command_ref) {
+ # No more commands
+ return undef;
+ }
+ my ($next_command_line) = $next_command_ref->replaced();
my ($clean_command) = $next_command_line;
+ if($clean_command =~ /^\s*$/) {
+ # Do not run empty lines
+ if(not $Global::CommandLineQueue->empty()) {
+ return get_command_line_with_sshlogin($sshlogin);
+ } else {
+ return undef;
+ }
+ }
if($::opt_retries and $clean_command and
$Global::failed{$clean_command}{'count'}{$sshlogin}) {
# This command with these args failed for this sshlogin
@@ -4302,47 +4174,16 @@ sub get_command_line_with_sshlogin {
} else {
# If it failed fewer times on another host:
# Find another job to run
- my @next_job_to_run = get_command_line_with_sshlogin($sshlogin);
+ my @next_job_to_run;
+ if(not $Global::CommandLineQueue->empty()) {
+ @next_job_to_run = get_command_line_with_sshlogin($sshlogin);
+ }
# Push the command back on the queue
- unget_command_line($next_command_line,$args_ref);
+ $Global::CommandLineQueue->unget($next_command_ref);
return @next_job_to_run;
}
}
-
- my ($sshcmd,$serverlogin) = sshcommand_of_sshlogin($sshlogin);
- my ($pre,$post)=("","");
- if($next_command_line and $serverlogin ne ":") {
- $Global::transfer_seq++;
- for my $file (@$args_ref) {
- if($::opt_transfer) {
- # --transfer
- $pre .= sshtransfer($sshlogin,$file).";";
- }
- if(@Global::ret_files) {
- # --return or --trc
- $post .= sshreturn($sshlogin,$file).";";
- }
- if($::opt_cleanup) {
- # --cleanup
- $post .= sshcleanup($sshlogin,$file).";";
- }
- }
- if($post) {
- # We need to save the exit status of the job
- $post = '_EXIT_status=$?; '.$post.' exit $_EXIT_status;';
- }
- my $parallel_env = 'PARALLEL_SEQ=$PARALLEL_SEQ\;export PARALLEL_SEQ\;'.
- 'PARALLEL_PID=$PARALLEL_PID\;export PARALLEL_PID\;';
- if($::opt_workdir) {
- return ($pre . "$sshcmd $serverlogin $parallel_env ".shell_quote("cd ".workdir()." && ")
- .shell_quote($next_command_line).";".$post,$clean_command);
- } else {
- return ($pre . "$sshcmd $serverlogin $parallel_env "
- .shell_quote($next_command_line).";".$post,$clean_command);
- }
- } else {
- return ($next_command_line,$clean_command);
- }
+ return $next_command_ref->sshlogin_wrap($sshlogin);
}
sub workdir {
@@ -4364,172 +4205,6 @@ sub workdir {
return $workdir;
}
-sub get_command_line {
- # Returns:
- # next command line
- # list of arguments for the line
- my ($cmd_line,$args_ref);
- if(@Global::unget_next_command_line) {
- $cmd_line = shift @Global::unget_next_command_line;
- $args_ref = shift @Global::unget_next_command_line;
- } else {
- do {
- ($cmd_line,$args_ref) = generate_command_line($Global::command);
- } while (defined $cmd_line and $cmd_line =~ /^\s*$/); # Skip empty lines
- }
- return ($cmd_line,$args_ref);
-}
-
-sub unget_command_line {
- # Returns: N/A
- unshift @Global::unget_next_command_line, @_;
-}
-
-sub more_arguments {
- # Returns:
- # whether there are more arguments to be processed or not
- my $fh = shift || $Global::argfile;
- return (
- @Global::unget_argv or
- (defined $Global::unget_line{$fh} and @{$Global::unget_line{$fh}}) or
- (defined $Global::unget_col{$fh} and @{$Global::unget_col{$fh}})
- or @Global::unget_arg or not eof $fh);
-}
-
-sub get_line_from_fh {
- # Returns:
- # next line from file handle from file or stdin. Delimiter removed
- # undef if end of file
- my $fh = shift;
- my $arg;
- if(@Global::unget_argv) {
- # Ungotten args from command line exists
- debug("get_line_from_fh ",$Global::unget_argv[0],"\n");
- return shift @Global::unget_argv;
- }
- if(not $Global::unget_line{$fh}) {
- @{$Global::unget_line{$fh}} = ();
- }
- my $unget_ref = $Global::unget_line{$fh};
- if(@$unget_ref) {
- # Ungotten arg exists
- debug("get_line_from_fh ",$$unget_ref[0],"\n");
- return shift @$unget_ref;
- }
- if(eof($fh)) {
- return undef;
- }
- $arg = <$fh>;
- # Remove delimiter
- $arg =~ s:$/$::;
- if($Global::end_of_file_string and
- $arg eq $Global::end_of_file_string) {
- # Ignore the rest of input file
- while (<$fh>) {}
- return undef;
- }
- if($Global::ignore_empty) {
- if($arg =~ /^\s*$/) {
- return get_line_from_fh($fh);
- }
- }
- if($Global::max_lines) {
- if($arg =~ /\s$/) {
- # Trailing space => continued on next line
- my $cont = get_line_from_fh($fh);
- if(defined $cont) {
- $arg .= $cont;
- }
- }
- }
- debug("get_line_from_fh ",$arg,"\n");
- return $arg;
-}
-
-sub unget_line_from_fh {
- # Returns: N/A
- my $fh = shift;
- if(not $Global::unget_line{$fh}) {
- @{$Global::unget_line{$fh}} = ();
- }
- my $unget_ref = $Global::unget_line{$fh};
- push @$unget_ref, @_;
-}
-
-sub unget_argv {
- # Returns: N/A
- push @Global::unget_argv, @_;
-}
-
-sub get_column {
- # Return:
- # Column unquoted untrimmed
- # undef if no more
- my $fh = shift;
- if(not $Global::unget_col{$fh}) {
- @{$Global::unget_col{$fh}} = ();
- }
- my $unget_ref = $Global::unget_col{$fh};
- if(@$unget_ref) {
- # Ungotten col exists
- return shift @$unget_ref;
- }
- my $line = get_line_from_fh($fh);
- if(defined $line) {
- if($line ne "") {
- push @$unget_ref, split /$::opt_colsep/o, $line;
- } else {
- push @$unget_ref, "";
- }
- $::opt_N = $#$unget_ref+1;
- $Global::max_number_of_args = $::opt_N;
- debug("col_unget_ref: @$unget_ref\n");
- return shift @$unget_ref;
- } else {
- return undef;
- }
-}
-
-sub unget_column {
- # Returns: N/A
- my $fh = shift;
- if(not $Global::unget_col{$fh}) {
- @{$Global::unget_col{$fh}} = ();
- }
- my $unget_ref = $Global::unget_col{$fh};
- push @$unget_ref, @_;
-}
-
-sub get_arg {
- # Returns:
- # next argument quoted and trimmed as needed
- # (from $Global::argfile or $fh if given)
- # undef if end of file
- my $arg;
- my $fh = shift || $Global::argfile;
- if(@Global::unget_arg) {
- return shift @Global::unget_arg;
- }
- if($::opt_colsep) {
- $arg = get_column($fh);
- } else {
- $arg = get_line_from_fh($fh);
- }
- if(defined $arg) {
- if($Global::trim ne 'n') {
- $arg = trim($arg);
- }
- if($Global::input_is_filename) {
- $arg = shell_quote($arg);
- }
- }
- return $arg;
-}
-
-sub unget_arg {
- # Returns: N/A
- push @Global::unget_arg, @_;
-}
sub __REMOTE_SSH__ {}
@@ -4571,7 +4246,7 @@ sub parse_sshlogin {
}
}
$Global::host{$sshlogin}{'no_of_running'} = 0;
- $Global::host{$sshlogin}{'maxlength'} = max_length_of_command_line();
+ $Global::host{$sshlogin}{'maxlength'} = Limits::Command::max_length();
}
debug("sshlogin: ", my_dump(%Global::host),"\n");
if($::opt_transfer or @::opt_return or $::opt_cleanup or @::opt_basefile) {
@@ -4652,12 +4327,12 @@ sub control_path_dir {
return $Private::control_path_dir;
}
-sub sshtransfer {
- # Return the sshcommand needed to transfer the file
- # Returns:
- # ssh command needed to transfer file to sshlogin
- return sshtransferreturn(@_,1,0);
-}
+#sub sshtransfer {
+# # Return the sshcommand needed to transfer the file
+# # Returns:
+# # ssh command needed to transfer file to sshlogin
+# return sshtransferreturn(@_,1,0);
+#}
sub sshreturn {
# Return the sshcommand needed to returning the file
@@ -4727,21 +4402,23 @@ sub sshtransferreturn {
}
} else {
# Return or cleanup
- my $noext = no_extension($file); # Remove .ext before prepending ./
+ #my $noext = no_extension($file); # Remove .ext before prepending ./
my @cmd = ();
my $rsync_destdir = ($relpath ? "./" : "/");
- for my $ret_file (@Global::ret_files) {
+ #for my $ret_file (@Global::ret_files) {
+ my $ret_file = $file;
my $remove = $removesource ? "--remove-source-files" : "";
# If relative path: prepend workdir/./ to avoid problems if the dir contains ':'
# and to get the right relative return path
my $replaced = ($relpath ? workdir()."/./" : "") .
- context_replace($ret_file,[$file],[$noext]);
+ $file;
+# context_replace($ret_file,[$file]);
# --return
# Abs path: rsync -rlDzRE server:/home/tange/dir/subdir/file.gz /
# Rel path: rsync -rlDzRE server:./subsir/file.gz ./
push(@cmd, "rsync $rsync_opt $remove $serverlogin:".
shell_quote($replaced) . " ".$rsync_destdir);
- }
+ #}
return join(";",@cmd);
}
}
@@ -4797,13 +4474,13 @@ sub list_running_jobs {
sub start_no_new_jobs {
# Returns: N/A
+ $SIG{TERM} = $Global::original_sigterm;
print STDERR
("$Global::progname: SIGTERM received. No new jobs will be started.\n",
"$Global::progname: Waiting for these ", scalar(keys %Global::running),
" jobs to finish. Send SIGTERM again to stop now.\n");
list_running_jobs();
$Global::start_no_new_jobs++;
- $SIG{TERM} = $Global::original_sigterm;
}
sub count_sig_child {
@@ -4871,7 +4548,10 @@ sub reaper {
delete $Global::failed{$clean_command};
} else {
# This command should be retried
- unget_command_line($clean_command,[]);
+ # $Global::CommandLineQueue ||= CommandLineQueue->new($Global::argfile);
+ # Convert $clean_command to the relevant command object
+ my $cmdobject = $Global::clean_commands{$clean_command};
+ $Global::CommandLineQueue->unget($cmdobject);
$retry_job = 1;
}
}
@@ -4948,7 +4628,7 @@ sub die_usage {
sub usage {
# Returns: N/A
print "Usage:\n";
- print "$Global::progname [options] [command [arguments]] < list_of_arguments)\n";
+ print "$Global::progname [options] [command [arguments]] < list_of_arguments\n";
print "$Global::progname [options] [command [arguments]] ::: arguments\n";
print "$Global::progname [options] [command [arguments]] :::: argfile(s)\n";
print "\n";
@@ -4970,8 +4650,8 @@ sub version {
sub show_limits {
# Returns: N/A
- print("Maximal size of command: ",real_max_length(),"\n",
- "Maximal used size of command: ",max_length_of_command_line(),"\n",
+ print("Maximal size of command: ",Limits::Command::real_max_length(),"\n",
+ "Maximal used size of command: ",Limits::Command::max_length(),"\n",
"\n",
"Execution of will continue now, and it will try to read its input\n",
"and run commands; if this is not what you wanted to happen, please\n",
@@ -5052,6 +4732,1272 @@ sub my_dump {
}
}
+###
+##### OO Parts below
+###
+
+package CommandLine;
+
+sub new {
+ my $class = shift;
+ my $command = ::undef_as_empty(shift);
+ my $arg_queue = shift;
+ my $context_replace = shift;
+ my $max_number_of_args = shift; # for -N and normal (-N1)
+ my $return_files = shift;
+ my $len = {
+ '{}' => 0, # Total length of all {} replaced with all args
+ '{/}' => 0, # Total length of all {/} replaced with all args
+ '{.}' => 0, # Total length of all {.} replaced with all args
+ '{/.}' => 0, # Total length of all {/.} replaced with all args
+ 'no_args' => undef, # Length of command with all replacement args removed
+ 'context' => undef, # Length of context of an additional arg
+ };
+ my($sum,%count);
+ ($sum,$len->{'no_args'},$len->{'context'},$len->{'contextgroups'},%count) =
+ number_of_replacements($command,$context_replace);
+ if($sum == 0) {
+ if($command eq "") {
+ $command = $Global::replace{'{}'};
+ } else {
+ $command .=" ".$Global::replace{'{}'}; # Add {} to the command if there are no {...}'s
+ }
+ }
+ ($sum,$len->{'no_args'},$len->{'context'},$len->{'contextgroups'},%count) =
+ number_of_replacements($command,$context_replace);
+# if(not $max_number_of_args) {
+# $max_number_of_args = 2**31; # As many as possible (-X or -m)
+# }
+ return bless {
+ 'command' => $command,
+ 'len' => $len,
+ 'arg_list' => [],
+ 'arg_queue' => $arg_queue,
+ 'max_number_of_args' => $max_number_of_args,
+ 'replacecount' => \%count,
+ 'context_replace' => $context_replace,
+ 'return_files' => $return_files,
+ }, ref($class) || $class;
+}
+
+sub populate {
+ # Add arguments from arg_queue until the number of arguments or
+ # max line length is reached
+ my $self = shift;
+ my $next_arg;
+ while (not $self->{'arg_queue'}->empty()) {
+ $next_arg = $self->{'arg_queue'}->get();
+ if(not defined $next_arg) {
+ next;
+ }
+ $self->push($next_arg);
+ #::debug("if(".$self->len()." >= ".Limits::Command::max_length().") ".length $self->replaced()."\n");
+ if($self->len() >= Limits::Command::max_length()) {
+ # TODO stuff about -x opt_x
+ if($self->number_of_args() > 1) {
+ # There is something to work on
+ $self->{'arg_queue'}->unget($self->pop());
+ last;
+ } else {
+ my $args = join(" ", map { $_->orig() } @$next_arg);
+ print STDERR ("Command line too long (",
+ $self->len(), " >= ",
+ Limits::Command::max_length(),
+ ") at number ",
+ $self->{'arg_queue'}->arg_number(),
+ ": ".
+ (substr($args,0,50))."...\n");
+ $self->{'arg_queue'}->unget($self->pop());
+ ::wait_and_exit(255);
+ }
+ }
+
+ if(defined $self->{'max_number_of_args'}) {
+ if($self->number_of_args() >= $self->{'max_number_of_args'}) {
+ last;
+ }
+ }
+ }
+ if($self->number_of_args() > 0) {
+ # Fill up if we have a half completed line
+ if(defined $self->{'max_number_of_args'}) {
+ # If you want a number of args and do not have it then fill out the rest with empties
+ while($self->number_of_args() < $self->{'max_number_of_args'}) {
+ $self->push([Arg->new("")]);
+ }
+ }
+ }
+}
+
+sub push {
+ # Add one or more records as arguments
+ my $self = shift;
+ for my $record (@_) {
+ push @{$self->{'arg_list'}}, $record;
+ #::my_dump($record);
+ my $arg_no = ($self->number_of_args()-1) * ($#$record+1);
+
+ for my $arg (@$record) {
+ $arg_no++;
+ if(defined $arg) {
+ for my $used (keys %{$self->{'replacecount'}}) {
+ if($used =~ /^{(\d+)(\D*)}$/) {
+ my $positional = $1; # number if any
+ my $replacementfunction = "{".::undef_as_empty($2)."}"; # {} {/} {.} or {/.}
+ # Find the single replacements
+ if($arg_no == $positional) {
+ $self->{'len'}{$used} += length $arg->replace($replacementfunction);
+ }
+ } elsif($used =~ /^{(\D*)}$/) {
+ # Add to the multireplacement
+ $self->{'len'}{$used} += length $arg->replace($used);
+ } else {
+ die;
+ }
+ }
+ }
+ }
+ }
+}
+
+sub pop {
+ # Remove last argument
+ my $self = shift;
+ my $record = pop @{$self->{'arg_list'}};
+ for my $arg (@$record) {
+ if(defined $arg) {
+ $self->{'len'}{'{}'} -= length $arg->orig();
+ $self->{'len'}{'{/}'} -= length $arg->basename();
+ $self->{'len'}{'{.}'} -= length $arg->no_extension();
+ $self->{'len'}{'{/.}'} -= length $arg->basename_no_extension();
+ }
+ }
+ return $record;
+}
+
+sub number_of_args {
+ my $self = shift;
+ # This is really number of records
+ return $#{$self->{'arg_list'}}+1;
+}
+
+sub len {
+ # The length of the command line with args substituted
+ my $self = shift;
+ my $len = 0;
+ # Add length of the original command with no args
+ $len += $self->{'len'}{'no_args'};
+ if($self->{'context_replace'}) {
+ $len += $self->number_of_args()*$self->{'len'}{'context'};
+ for my $replstring (keys %{$self->{'replacecount'}}) {
+ if(defined $self->{'len'}{$replstring}) {
+ $len += $self->{'len'}{$replstring} * $self->{'replacecount'}{$replstring};
+ }
+ }
+ $len += ($self->number_of_args()-1) * $self->{'len'}{'contextgroups'};
+
+# # One space between two args
+# $len += $self->number_of_args()-1;
+# # The context is repeated for each arg
+# $len += $self->number_of_args() * $self->{'len'}{'context'};
+# # A context group inserts a space
+# $len += $self->number_of_args() * ($self->{'len'}{'contextgroups'} -1);
+# $len += -$self->{'len'}{'contextgroups'}+1;
+# # Each replacement string may occur several times
+# # Add the length for each time
+# for my $replstring (keys %{$self->{'replacecount'}}) {
+# if(defined $self->{'len'}{$replstring}) {
+# $len += $self->{'len'}{$replstring} * $self->{'replacecount'}{$replstring};
+# $len -= length $Global::replace{$replstring};
+# }
+#
+# }
+# $len+= $self->number_of_args() +1;
+# ::my_dump($self);
+ } else {
+ # Each replacement string may occur several times
+ # Add the length for each time
+ for my $replstring (keys %{$self->{'replacecount'}}) {
+ if(defined $self->{'len'}{$replstring}) {
+ $len += $self->{'len'}{$replstring} *
+ $self->{'replacecount'}{$replstring};
+ }
+ if($Global::replace{$replstring}) {
+ # This is a multi replacestring ({} {/} {.} {/.})
+ # Add each space between two arguments
+ my $number_of_args = ($#{$self->{'arg_list'}[0]}+1)*$self->number_of_args();
+ $len += ($number_of_args-1) * $self->{'replacecount'}{$replstring};
+ }
+ }
+ }
+ return $len;
+}
+
+sub multi_regexp {
+ if(not $CommandLine::multi_regexp) {
+ $CommandLine::multi_regexp =
+ "(?:".
+ join("|",map {my $a=$_; $a =~ s/(\W)/\\$1/g; $a}
+ ($Global::replace{"{}"},
+ $Global::replace{"{.}"},
+ $Global::replace{"{/}"},
+ $Global::replace{"{/.}"})
+ ).")";
+ }
+ return $CommandLine::multi_regexp;
+}
+
+sub number_of_replacements {
+ # Returns:
+ # sum_of_count, length_of_command_with_no_args, length_of_context { 'replacementstring' => count }
+ my $command = shift;
+ my $context_replace = shift;
+ my %count = ();
+ my $sum = 0;
+ my $cmd = $command;
+ my $multi_regexp = multi_regexp();
+ my $replacement_regexp =
+ "(?:".
+ '\{\d+/?\.?\}'. # {n}, {n.} {n/.} {n/}
+ '|'.
+ join("|",map {$a=$_;$a=~s/(\W)/\\$1/g; $a} values %Global::replace).
+ ")";
+ while($cmd =~ s/($replacement_regexp)/\0/o) {
+ # substitute with \0 to avoid {{}} being interpreted as two {}'s
+ if(defined $Global::replace_rev{$1}) {
+ $count{$Global::replace_rev{$1}}++;
+ } else {
+ $count{$1}++;
+ }
+ $sum++;
+ }
+
+ my $number_of_context_groups = 0;
+ my $no_args;
+ my $context;
+ if($context_replace) {
+ $cmd = $command;
+ while($cmd =~ s/\S*$multi_regexp\S*//o) {
+ $number_of_context_groups++;
+ }
+ $no_args = length $cmd;
+ $context = length($command) - $no_args;
+ } else {
+ $cmd = $command;
+ $cmd =~ s/$multi_regexp//go;
+ $cmd =~ s/$replacement_regexp//go;
+ $no_args = length($cmd);
+ $context = length($command) - $no_args;
+ }
+ #warn("Command:$command no_args:$no_args context:$context");
+ #warn(%count);
+ for my $k (keys %count) {
+ if(defined $Global::replace{$k}) {
+ # {} {/} {.} {/.}
+ $context -= (length $Global::replace{$k}) * $count{$k};
+ } else {
+ # {#}
+ $context -= (length $k) * $count{$k};
+ }
+ }
+ #die("Command:$command no_args:$no_args context:$context");
+ return ($sum,$no_args,$context,$number_of_context_groups,%count);
+}
+
+sub sshlogin_wrap {
+ # Wrap the command with the commands needed to run remotely
+ my $self = shift;
+ my $sshlogin = shift;
+ my ($sshcmd,$serverlogin) = ::sshcommand_of_sshlogin($sshlogin);
+ my ($next_command_line) = $self->replaced();
+ my ($clean_command) = $next_command_line;
+ my ($pre,$post,$cleanup)=("","","");
+ if($serverlogin ne ":") {
+ $Global::transfer_seq++;
+ # --transfer
+ $pre .= $self->sshtransfer($sshlogin);
+
+ for my $file ($self->cleanup()) {
+ $cleanup .= ::sshcleanup($sshlogin,$file).";";
+ }
+ for my $file ($self->return()) {
+ $post .= ::sshreturn($sshlogin,$file).";";
+ }
+ $post.=$cleanup;
+ if($post) {
+ # We need to save the exit status of the job
+ $post = '_EXIT_status=$?; '.$post.' exit $_EXIT_status;';
+ }
+ my $parallel_env = 'PARALLEL_SEQ=$PARALLEL_SEQ\;export PARALLEL_SEQ\;'.
+ 'PARALLEL_PID=$PARALLEL_PID\;export PARALLEL_PID\;';
+ if($::opt_workdir) {
+ return ($pre . "$sshcmd $serverlogin $parallel_env ".::shell_quote("cd ".::workdir()." && ")
+ .::shell_quote($next_command_line).";".$post,$clean_command);
+ } else {
+ return ($pre . "$sshcmd $serverlogin $parallel_env "
+ .::shell_quote($next_command_line).";".$post,$clean_command);
+ }
+ } else {
+ return ($next_command_line,$clean_command);
+ }
+}
+
+sub transfer {
+ # Files to transfer
+ my $self = shift;
+ my @transfer = ();
+ if($::opt_transfer) {
+ for my $record (@{$self->{'arg_list'}}) {
+ # Merge arguments from records into args
+ for my $arg (@$record) {
+ CORE::push @transfer, $arg->orig();
+ }
+ }
+ }
+ return @transfer;
+}
+
+sub sshtransfer {
+ my $self = shift;
+ my $sshlogin = shift;
+ my $pre = "";
+ for my $file ($self->transfer()) {
+ $pre .= ::sshtransferreturn($sshlogin,$file,1,0).";";
+ }
+ return $pre;
+}
+
+sub return {
+ # Files to return
+ # Quoted and with {...} substituted
+ my $self = shift;
+ my @return = ();
+ for my $return (@{$self->{'return_files'}}) {
+ CORE::push @return, $self->replace_placeholders($return);
+ }
+ return @return;
+}
+
+sub sshreturn {
+ my $self = shift;
+ my $sshlogin = shift;
+ my $pre = "";
+ for my $file ($self->transfer()) {
+ $pre .= ::sshtransferreturn($sshlogin,$file,0,$::opt_cleanup).";";
+ }
+ return $pre;
+}
+
+sub cleanup {
+ # Files to remove at cleanup
+ my $self = shift;
+ if($::opt_cleanup) {
+ my @transfer = $self->transfer();
+ return @transfer;
+ } else {
+ return ();
+ }
+}
+
+sub replaced {
+ my $self = shift;
+ if(not $self->{'replaced'}) {
+ $self->{'replaced'} = $self->replace_placeholders($self->{'command'});
+ }
+ if($::oodebug and length($self->{'replaced'}) != ($self->len())) {
+ ::my_dump($self);
+ Carp::cluck("replaced len=".length($self->{'replaced'})." computed=".($self->len())."\n");
+ }
+ return $self->{'replaced'};
+}
+
+sub replace_placeholders {
+ my $self = shift;
+ my($target) = shift;
+ my($context_replace) = $self->{'context_replace'};
+
+ my $context_regexp = $context_replace ? '\S*' : ''; # Regexp to match surrounding context
+ if($self->number_of_args() == 0) {
+ Carp::confess("0 args should never call replaced");
+ }
+
+ my %replace;
+ my %replace_single;
+ my %replace_multi;
+ my @replace_context;
+ my @args=();
+ my @used_multi;
+
+ for my $record (@{$self->{'arg_list'}}) {
+ # Merge arguments from records into args
+ CORE::push @args, @$record;
+ }
+ #::my_dump(@args);
+ for my $used (keys %{$self->{'replacecount'}}) {
+ if($used =~ /^{(\d+)(\D*)}$/) {
+ my $positional = $1; # number if any
+ my $replacementfunction = "{".::undef_as_empty($2)."}"; # {} {/} {.} or {/.}
+ # Find the single replacements
+ if(defined $args[$positional-1]) {
+ # we have a matching argument for {n}
+ $replace_single{$used} = $args[$positional-1]->replace($replacementfunction);
+ }
+ } elsif($used =~ /^{\D*}$/) {
+ # Add to the multireplacement
+ my $replacementfunction = $used; # {} {/} {.} or {/.}
+ CORE::push @used_multi, $replacementfunction;
+ if($self->{'context_replace'}) {
+ for my $n (0 .. $#args) {
+ $replace_context[$n]{$replacementfunction} =
+ $args[$n]->replace($replacementfunction);
+ }
+ } else {
+ CORE::push(@{$replace_multi{$replacementfunction}},
+ map { $args[$_]->replace($replacementfunction) }
+ 0 .. $#args);
+ }
+ } else {
+ die;
+ }
+ }
+
+# for my $n (0 .. $#args) {
+# my $m = $n+1;
+# $replace{'{}'} = $args[$n]->replace('{}');
+# $replace{'{.}'} = $args[$n]->replace('{.}');
+# $replace{'{/}'} = $args[$n]->replace('{/}');
+# $replace{'{/.}'} = $args[$n]->replace('{/.}');
+# $replace_single{"{$m}"} = $replace{'{}'}; # {2}
+# $replace_single{"{$m.}"} = $replace{'{.}'}; # {2.}
+# $replace_single{"{$m/}"} = $replace{'{/}'}; # {2/}
+# $replace_single{"{$m/.}"} = $replace{'{/.}'}; # {2/.}
+#
+# CORE::push(@{$replace_multi{"{.}"}}, $replace{'{.}'}); # {.}
+# CORE::push(@{$replace_multi{"{/}"}}, $replace{'{/}'}); # {/}
+# CORE::push(@{$replace_multi{"{/.}"}}, $replace{'{/.}'}); # {/.}
+# $replace_context[$n]{"{}"} = $replace{'{}'}; # {}
+# $replace_context[$n]{"{.}"} = $replace{'{.}'}; # {.}
+# $replace_context[$n]{"{/}"} = $replace{'{/}'}; # {/}
+# $replace_context[$n]{"{/.}"} = $replace{'{/.}'}; # {/.}
+# }
+ #::my_dump(%replace_single);
+ # Time: 0.146 0.392
+ # Time: 0.066 0.392
+ # Time: 0.046 0.372
+ # Time: 0.096 0.372
+ # Time: 0.066 0.402
+# my $single_regexp = join('|', map { $_=~s/(\W)/\\$1/g; $_} sort keys %replace_single);
+ # Time: 0.214 0.598
+ # Time: 0.204 0.538
+ # Time: 0.234 0.568
+ my $replacements = 0;
+ if(%replace_single) {
+ my $single_regexp = join('|', map { $_=~s/(\W)/\\$1/g; $_} sort keys %replace_single);
+ $replacements += ($target =~ s/($single_regexp)/$replace_single{$1}/ge);
+ }
+ my $orig_target = $target;
+ if(@used_multi) {
+ my $multi_regexp = join('|', map {
+ $a=$Global::replace{"$_"};
+ $a=~s/(\W)/\\$1/g; $a
+ } @used_multi);
+ my %wordargs;
+ while($target =~ s/(\S*($multi_regexp)\S*)/\0/o) {
+ my $wordarg = $1;
+ my $pattern = $2;
+ if($self->{'context_replace'}) {
+ my $substituted = $wordarg;
+ my @all=();
+ for my $argref (@replace_context) {
+ # for each argument convert a{}b to a1b a2b
+ my $substituted = $wordarg;
+ $substituted =~ s/($multi_regexp)/$argref->{$Global::replace_rev{$1}}/g;
+ CORE::push @all,$substituted;
+ }
+ $wordargs{$wordarg} = join" ",@all;
+ } else {
+ my $substituted = $wordarg;
+ $substituted =~ s/($multi_regexp)/join(" ",map {$_} @{$replace_multi{$Global::replace_rev{$1}}})/eg;
+ $wordargs{$wordarg} = $substituted;
+ }
+ }
+ my @k=keys %wordargs;
+ for(@k) {s/(\W)/\\$1/g};
+ my $regexp=join("|",@k);
+ $orig_target =~s/($regexp)/$wordargs{$1}/g;
+ }
+
+
+# ::my_dump(@args);
+# ::my_dump(@replace_context);
+# ::my_dump($multi_regexp);
+# ::my_dump($target);
+
+
+ return $orig_target;
+}
+
+
+package CommandLineQueue;
+
+sub new {
+ my $class = shift;
+ my $command = shift;
+ my $read_from = shift;
+ my $context_replace = shift;
+ my $max_number_of_args = shift;
+ my $return_files = shift;
+ my @unget = ();
+ return bless {
+ 'unget' => \@unget,
+ 'command' => $Global::command,
+ 'arg_queue' => RecordQueue->new($read_from,$::opt_colsep),
+ 'context_replace' => $context_replace,
+ 'max_number_of_args' => $max_number_of_args,
+ 'size' => undef,
+ 'return_files' => $return_files,
+ }, ref($class) || $class;
+}
+
+sub quote_args {
+ my $self = shift;
+ # If there is not command emulate |bash
+ return $self->{'command'};
+}
+
+sub get {
+ my $self = shift;
+ if(@{$self->{'unget'}}) {
+ my $cmd_line = shift @{$self->{'unget'}};
+ return ($cmd_line);
+ } else {
+ my ($cmd_line);
+# do {
+ $cmd_line = CommandLine->new($self->{'command'},
+ $self->{'arg_queue'},
+ $self->{'context_replace'},
+ $self->{'max_number_of_args'},
+ $self->{'return_files'},
+ );
+ $cmd_line->populate();
+# ::my_dump($cmd_line);
+# } while (defined $cmd_line and $cmd_line =~ /^\s*$/); # Skip empty lines
+# } while (defined $cmd_line); # Skip empty lines
+ ::debug("cmd_line->number_of_args ".$cmd_line->number_of_args()."\n");
+ if($cmd_line->number_of_args() == 0) {
+ # We did not get more args - maybe at EOF string?
+ return undef;
+ } else {
+ # TODO get rid of this hash (needed in sub reaper)
+ $Global::clean_commands{$cmd_line->replaced()} = $cmd_line;
+ return ($cmd_line);
+ }
+ }
+}
+
+sub unget {
+ my $self = shift;
+ unshift @{$self->{'unget'}}, @_;
+}
+
+sub empty {
+ my $self = shift;
+ my $empty = (not @{$self->{'unget'}}) && $self->{'arg_queue'}->empty();
+ ::debug("CommandLineQueue->empty $empty\n");
+ return $empty;
+}
+
+sub size {
+ my $self = shift;
+ if(not $self->{'size'}) {
+ my @all_lines = ();
+ while(not $self->{'arg_queue'}->empty()) {
+ push @all_lines, CommandLine->new($self->{'command'},
+ $self->{'arg_queue'},
+ $self->{'context_replace'},
+ $self->{'max_number_of_args'});
+ }
+ $self->{'size'} = @all_lines;
+ $self->unget(@all_lines);
+ }
+ return $self->{'size'};
+}
+
+#sub generate_command_line {
+# # Returns:
+# # the full job line to run
+# # list of quoted arguments on that line
+# my $self = shift;
+# my $command = $self->{'command'};
+# my ($job_line,$last_good);
+# my ($quoted_args) =
+# ::get_multiple_args($command,Limits::Command::max_length());
+#
+# if(@$quoted_args) {
+# $job_line = $command;
+#
+# $job_line = ::context_replace($command, @$quoted_args);
+# ::debug("Return jobline(",length($job_line),"): !",$job_line,"!\n");
+# }
+# return ($job_line,$quoted_args);
+#}
+
+package Limits::Command;
+
+# Maximal command line length (for -m and -X)
+sub max_length {
+ # Find the max_length of a command line
+ # Returns:
+ # number of chars on the longest command line allowed
+ if(not $Limits::Command::line_max_len) {
+ if($::opt_s) {
+ if(is_acceptable_command_line_length($::opt_s)) {
+ $Limits::Command::line_max_len = $::opt_s;
+ } else {
+ # -s is too long: Find the correct
+ $Limits::Command::line_max_len = binary_find_max_length(0,$::opt_s);
+ }
+ if($::opt_s <= $Limits::Command::line_max_len) {
+ $Limits::Command::line_max_len = $::opt_s;
+ } else {
+ print STDERR "$Global::progname: ",
+ "value for -s option should be < $Limits::Command::line_max_len\n";
+ }
+ } else {
+ $Limits::Command::line_max_len = real_max_length();
+ }
+ }
+ return $Limits::Command::line_max_len;
+}
+
+sub real_max_length {
+ # Returns:
+ # The maximal command line length
+ # Use an upper bound of 8 MB if the shell allows for for infinite long lengths
+ my $upper = 8_000_000;
+ my $len = 8;
+ do {
+ if($len > $upper) { return $len };
+ $len *= 16;
+ } while (is_acceptable_command_line_length($len));
+ # Then search for the actual max length between 0 and upper bound
+ return binary_find_max_length(int($len/16),$len);
+}
+
+sub binary_find_max_length {
+ # Given a lower and upper bound find the max_length of a command line
+ # Returns:
+ # number of chars on the longest command line allowed
+ my ($lower, $upper) = (@_);
+ if($lower == $upper or $lower == $upper-1) { return $lower; }
+ my $middle = int (($upper-$lower)/2 + $lower);
+ ::debug("Maxlen: $lower,$upper,$middle\n");
+ if (is_acceptable_command_line_length($middle)) {
+ return binary_find_max_length($middle,$upper);
+ } else {
+ return binary_find_max_length($lower,$middle);
+ }
+}
+
+sub is_acceptable_command_line_length {
+ # Test if a command line of this length can run
+ # Returns:
+ # 0 if the command line length is too long
+ # 1 otherwise
+ my $len = shift;
+
+ $CommandMaxLength::is_acceptable_command_line_length++;
+ ::debug("$CommandMaxLength::is_acceptable_command_line_length $len\n");
+ local *STDERR;
+ open (STDERR,">/dev/null");
+ system "true "."x"x$len;
+ close STDERR;
+ ::debug("$len $?\n");
+ return not $?;
+}
+
+package RecordQueue;
+
+sub new {
+ my $class = shift;
+ my $fhs = shift;
+ my $colsep = shift;
+ my @unget = ();
+ my $arg_sub_queue;
+ if($colsep) {
+ # Open one file with colsep
+ $arg_sub_queue = RecordColQueue->new($fhs);
+ } else {
+ # Open one or more files if multiple -a
+ $arg_sub_queue = MultifileQueue->new($fhs);
+ }
+ return bless {
+ 'unget' => \@unget,
+ 'arg_number' => 0,
+ 'arg_sub_queue' => $arg_sub_queue,
+ }, ref($class) || $class;
+}
+
+sub get {
+ my $self = shift;
+ my $arg;
+ if(@{$self->{'unget'}}) {
+ $arg = shift @{$self->{'unget'}};
+ ::debug("RecordQueue-get from unget $arg\n");
+ return $arg;
+ }
+ if($::oodebug and $self->empty()) {
+ Carp::croak("RecordQueue->get should never be called if empty");
+ }
+ $arg = $self->{'arg_sub_queue'}->get();
+ ::debug("RecordQueue-get ".::undef_as_empty($arg)."\n");
+ $self->{'arg_number'}++;
+ return $arg;
+}
+
+sub unget {
+ my $self = shift;
+ ::debug("RecordQueue-unget '@_'\n");
+ $self->{'arg_number'}--;
+ unshift @{$self->{'unget'}}, @_;
+}
+
+sub empty {
+ my $self = shift;
+ my $empty = not @{$self->{'unget'}};
+ $empty &&= $self->{'arg_sub_queue'}->empty();
+ ::debug("RecordQueue->empty $empty\n");
+ return $empty;
+}
+
+sub arg_number {
+ my $self = shift;
+ return $self->{'arg_number'};
+}
+
+package RecordColQueue;
+
+sub new {
+ my $class = shift;
+ my $fhs = shift;
+ my @unget = ();
+ my $arg_sub_queue = MultifileQueue->new($fhs);
+ return bless {
+ 'unget' => \@unget,
+ 'arg_sub_queue' => $arg_sub_queue,
+ }, ref($class) || $class;
+}
+
+sub get {
+ my $self = shift;
+ if(@{$self->{'unget'}}) {
+ return shift @{$self->{'unget'}};
+ }
+ my $unget_ref=$self->{'unget'};
+ if($self->{'arg_sub_queue'}->empty()) {
+ return undef;
+ }
+ my $in_record = $self->{'arg_sub_queue'}->get();
+ if(defined $in_record) {
+ my @out_record = ();
+ for my $arg (@$in_record) {
+ ::debug("RecordColQueue::arg $arg\n");
+ my $line = ::shell_unquote($arg->orig());
+ ::debug("line='$line'\n");
+ if($line ne "") {
+ for my $s (split /$::opt_colsep/o, $line) {
+ push @out_record, Arg->new($s);
+ }
+ } else {
+ push @out_record, Arg->new("");
+ }
+ }
+ return \@out_record;
+ } else {
+ return undef;
+ }
+}
+
+sub unget {
+ my $self = shift;
+ ::debug("RecordColQueue-unget '@_'\n");
+ unshift @{$self->{'unget'}}, @_;
+}
+
+sub empty {
+ my $self = shift;
+ my $empty = (not @{$self->{'unget'}} and $self->{'arg_sub_queue'}->empty());
+ ::debug("RecordColQueue->empty $empty");
+ return $empty;
+}
+
+
+
+package MultifileQueue;
+
+@Global::unget_argv=();
+
+sub new {
+ my $class = shift;
+ my $fhs = shift;
+ my @unget = ();
+ for my $fh (@$fhs) {
+ if(-t $fh) {
+ print STDERR "$Global::progname: Input is tty. Press CTRL-D to exit.\n";
+ }
+ }
+ return bless {
+ 'unget' => \@unget,
+ 'fhs' => $fhs,
+ }, ref($class) || $class;
+}
+
+sub get {
+ my $self = shift;
+ if(@Global::unget_argv) {
+ return shift @Global::unget_argv;
+ } else {
+ if(@{$self->{'unget'}}) {
+ return shift @{$self->{'unget'}};
+ }
+ my @record = ();
+ my $prepend = undef;
+ my $empty = 1;
+ for my $fh (@{$self->{'fhs'}}) {
+ ::debug("Reading $fh\n");
+ if(eof($fh)) {
+ if(defined $prepend) {
+ push @record, Arg->new($prepend);
+ $empty = 0;
+ } else {
+# push @record, undef;
+ push @record, Arg->new($prepend);
+ }
+ next;
+ }
+ my $arg = <$fh>;
+ # Remove delimiter
+ $arg =~ s:$/$::;
+ if($Global::end_of_file_string and
+ $arg eq $Global::end_of_file_string) {
+ # Ignore the rest of input file
+ while (<$fh>) {}
+ ::debug("EOF-string $arg\n");
+ if(defined $prepend) {
+ push @record, Arg->new($prepend);
+ $empty = 0;
+ } else {
+# push @record, undef;
+ push @record, Arg->new($prepend);
+ }
+ ::debug("Is empty? $empty");
+ next;
+ }
+ if(defined $prepend) {
+ $arg = $prepend.$arg; # For line continuation
+ $prepend = undef; #undef;
+ }
+ if($Global::ignore_empty) {
+ if($arg =~ /^\s*$/) {
+ redo; # Try the next line
+ }
+ }
+ if($Global::max_lines) {
+ if($arg =~ /\s$/) {
+ # Trailing space => continued on next line
+ $prepend = $arg;
+ redo;
+ }
+ }
+ ::debug("ArgLineQueue->get '",$arg,"'\n");
+ push @record, Arg->new($arg);
+ $empty = 0;
+ }
+ if($empty) {
+ ::debug("Return undef");
+ return undef;
+ } else {
+ ::debug("return [@record]");
+ return [@record];
+ }
+ }
+}
+
+sub unget {
+ my $self = shift;
+ ::debug("MultifileQueue-unget '@_'\n");
+ unshift @{$self->{'unget'}}, @_;
+}
+
+sub empty {
+ my $self = shift;
+ my $empty = (not @Global::unget_argv
+ and not @{$self->{'unget'}});
+ for my $fh (@{$self->{'fhs'}}) {
+ $empty &&= eof($fh);
+ }
+ ::debug("MultifileQueue->empty $empty\n");
+ return $empty;
+}
+
+
+#package _ArgQueue;
+#
+#sub new {
+# my $class = shift;
+# my $fhs = shift;
+# my @unget = ();
+# my $arg_sub_queue;
+# if($::opt_colsep) {
+# # Open one file with colsep
+# $arg_sub_queue->[0] = ArgColQueue->new($fhs->[0]);
+# } else {
+# # Open one or more files if multiple -a
+# my $sub_queue_no = 0;
+# for my $fh (@{$fhs}) {
+# $arg_sub_queue->[$sub_queue_no] = ArgNoColQueue->new($fh);
+# }
+# }
+# return bless {
+# 'unget' => \@unget,
+# 'arg_number' => 0,
+# 'arg_sub_queue' => $arg_sub_queue,
+# }, ref($class) || $class;
+#}
+#
+#sub get {
+# my $self = shift;
+# my $arg;
+# if(@{$self->{'unget'}}) {
+# $arg = shift @{$self->{'unget'}};
+# ::debug("ArgQueue-get from unget $arg\n");
+# $self->{'arg_number'}++;
+# return $arg;
+# }
+# # Read from the filehandles in round robin in case there is more than one
+# my $sub_queue_no = $self->{'arg_number'} % ($#{$self->{'arg_sub_queue'}}+1);
+# $arg = $self->{'arg_sub_queue'}->[$sub_queue_no]->get();
+# ::debug("ArgQueue-get ".::undef_as_empty($arg)."\n");
+# $self->{'arg_number'}++;
+# return $arg;
+#}
+#
+#sub unget {
+# my $self = shift;
+# ::debug("ArgQueue-unget '@_'\n");
+# unshift @{$self->{'unget'}}, @_;
+# $self->{'arg_number'}--;
+#}
+#
+#sub arg_number {
+# my $self = shift;
+# return $self->{'arg_number'};
+#}
+#
+#sub empty {
+# my $self = shift;
+# my $empty = not @{$self->{'unget'}};
+# for my $sub_queues (@{$self->{'arg_sub_queue'}}) {
+# $empty &&= $sub_queues->empty();
+# }
+# ::debug("ArgQueue->empty $empty\n");
+# return $empty;
+#}
+
+package Arg;
+
+sub new {
+ my $class = shift;
+ my $orig = shift;
+ ::debug("Arg->new '",$orig,"'\n");
+ if($Global::trim) {
+ $orig = trim_of($orig);
+ }
+ $orig =~ m:^(.*/)?([^/]*?)(\.[^/.]*)?$: or
+ Carp::croak($orig." does not match an argument. File a bug report");
+ my $p1 = ::undef_as_empty($1);
+ my $p2 = ::undef_as_empty($2);
+ my $p3 = ::undef_as_empty($3);
+ return bless {
+ 'orig' => $orig,
+ 'basename' => undef, #$p2.$p3, Problem if quoted or not
+ 'no_extension' => undef, #$p1.$p2,
+ 'basename_no_extension' => undef, #$p2,
+ 'trim' => $::opt_trim,
+ }, ref($class) || $class;
+}
+
+sub replace {
+ my $self = shift;
+ my $replacement_string = shift;
+ my %jumptable = ("{}" => sub { $self->orig() },
+ "{.}" => sub { $self->no_extension() },
+ "{/}" => sub { $self->basename() },
+ "{/.}" => sub { $self->basename_no_extension() },
+ );
+# my %jumptable = ("{}" => \&orig(),
+# "{.}" => sub { $self->no_extension() },
+# "{/}" => sub { $self->basename() },
+# "{/.}" => sub { $self->basename_no_extension() },
+# );
+ $self->{'jumptable'} = \%jumptable;
+# Carp::cluck($replacement_string);
+# warn($replacement_string);
+ return $self->{'jumptable'}{$replacement_string}();
+}
+
+sub orig {
+ my $self = shift;
+
+ if(not defined $self->{'myorig'}) {
+ if($Global::CommandLineQueue->quote_args()) {
+ $self->{'myorig'} = ::shell_quote($self->{'orig'});
+ } else {
+ $self->{'myorig'} = $self->{'orig'};
+ }
+ }
+ return $self->{'myorig'};
+}
+
+sub basename {
+ my $self = shift;
+ if(not defined $self->{'basename'}) {
+ if($Global::CommandLineQueue->quote_args()) {
+ $self->{'basename'} = ::shell_quote(basename_of($self->{'orig'}));
+ } else {
+ $self->{'basename'} = basename_of($self->{'orig'});
+ }
+ }
+ return $self->{'basename'};
+}
+
+sub no_extension {
+ my $self = shift;
+ if(not defined $self->{'no_extension'}) {
+ if($Global::CommandLineQueue->quote_args()) {
+ $self->{'no_extension'} = ::shell_quote(no_extension_of($self->{'orig'}));
+ } else {
+ $self->{'no_extension'} = no_extension_of($self->{'orig'});
+ }
+ }
+ return $self->{'no_extension'};
+}
+
+sub basename_no_extension {
+ my $self = shift;
+ if(not defined $self->{'basename_no_extension'}) {
+ if($Global::CommandLineQueue->quote_args()) {
+ $self->{'basename_no_extension'} = ::shell_quote(no_extension_of(basename_of($self->{'orig'})));
+ } else {
+ $self->{'basename_no_extension'} = no_extension_of(basename_of($self->{'orig'}));
+ }
+ }
+ return $self->{'basename_no_extension'};
+}
+
+sub trim {
+ my $self = shift;
+ if(not defined $self->{'trim'}) {
+ if($Global::CommandLineQueue->quote_args()) {
+ $self->{'trim'} = ::shell_quote(trim_of($self->{'orig'}));
+ } else {
+ $self->{'trim'} = trim_of($self->{'orig'});
+ }
+ }
+ return $self->{'trim'};
+}
+
+sub basename_of {
+ # Returns:
+ # argument with dir removed if any
+ my $basename = shift;
+ $basename =~ s:^.*/([^/]+)/?$:$1:; # Remove dir from argument. If ending in /, remove final /
+ return $basename;
+}
+
+sub no_extension_of {
+ # Returns:
+ # argument with .extension removed if any
+ my $no_ext = shift;
+ $no_ext =~ s:\.[^/\.]*$::; # Remove .ext from argument
+ return $no_ext;
+}
+
+sub trim_of {
+ # Removes white space as specifed by --trim:
+ # n = nothing
+ # l = start
+ # r = end
+ # lr|rl = both
+ # Returns:
+ # string with white space removed as needed
+ my (@strings) = map { defined $_ ? $_ : "" } (@_);
+ my $arg;
+ if($Global::trim eq "n") {
+ # skip
+ } elsif($Global::trim eq "l") {
+ for $arg (@strings) { $arg =~ s/^\s+//; }
+ } elsif($Global::trim eq "r") {
+ for $arg (@strings) { $arg =~ s/\s+$//; }
+ } elsif($Global::trim eq "rl" or $Global::trim eq "lr") {
+ for $arg (@strings) { $arg =~ s/^\s+//; $arg =~ s/\s+$//; }
+ } else {
+ print STDERR "$Global::progname: --trim must be one of: r l rl lr\n";
+ ::wait_and_exit(255);
+ }
+ return wantarray ? @strings : "@strings";
+}
+
+
+#package _ArgColQueue;
+#
+#sub new {
+# my $class = shift;
+# my $fh = shift;
+# my @unget = ();
+# my $arg_line_queue = ArgLineQueue->new($fh);
+# return bless {
+# 'unget' => \@unget,
+# 'fh' => $fh,
+# 'arg_line_queue' => $arg_line_queue,
+# }, ref($class) || $class;
+#}
+#
+#sub get {
+# my $self = shift;
+# if(@{$self->{'unget'}}) {
+# return shift @{$self->{'unget'}};
+# }
+# my $unget_ref=$self->{'unget'};
+# my $line = $self->{'arg_line_queue'}->get();
+# if(defined $line) {
+# ::debug("ArgColQueue::line $line\n");
+# if($line ne "") {
+# for my $s (split /$::opt_colsep/o, $line) {
+# push @$unget_ref, Arg->new($s);
+# }
+# } else {
+# push @$unget_ref, Arg->new("");
+# }
+# $::opt_N = $#$unget_ref+1;
+# $Global::max_number_of_args = $::opt_N;
+# ::debug("ArgColQueue_unget_ref: @$unget_ref\n");
+# return shift @$unget_ref;
+# } else {
+# return undef;
+# }
+#}
+#
+#sub unget {
+# my $self = shift;
+# unshift @{$self->{'unget'}}, @_;
+#}
+#
+#sub empty {
+# my $self = shift;
+# my $empty = (not @{$self->{'unget'}} and $self->{'arg_line_queue'}->empty());
+# ::debug("ArgColQueue->empty $empty");
+# return $empty;
+#}
+#
+#
+#
+##package _ArgNoColQueue;
+#
+#sub new {
+# my $class = shift;
+# my $fh = shift;
+# my $linequeue = ArgLineQueue->new($fh);
+# return bless {
+# 'linequeue' => $linequeue,
+# }, ref($class) || $class;
+#}
+#
+#sub get {
+# my $self = shift;
+# my $arg = $self->{'linequeue'}->get();
+# if(defined $arg) {
+# return Arg->new($arg);
+# } else {
+# return undef;
+# }
+#}
+#
+#sub empty {
+# my $self = shift;
+# my $empty = ($self->{'linequeue'}->empty());
+# ::debug("ArgNoColQueue->empty $empty");
+# return $empty;
+#}
+#
+##package _ArgLineQueue;
+#
+#@Global::unget_argv=();
+#
+#sub new {
+# my $class = shift;
+# my $fh = shift;
+# my @unget = ();
+# return bless {
+# 'unget' => \@unget,
+# 'fh' => $fh,
+# }, ref($class) || $class;
+#}
+#
+#sub get {
+# my $self = shift;
+# if(@Global::unget_argv) {
+# return shift @Global::unget_argv;
+# } else {
+# if(@{$self->{'unget'}}) {
+# return shift @{$self->{'unget'}};
+# }
+# my $fh =$self->{'fh'};
+# if(eof($fh)) {
+# return undef;
+# }
+# my $arg = <$fh>;
+# # Remove delimiter
+# $arg =~ s:$/$::;
+# if($Global::end_of_file_string and
+# $arg eq $Global::end_of_file_string) {
+# # Ignore the rest of input file
+# while (<$fh>) {}
+# return undef;
+# }
+# if($Global::ignore_empty) {
+# if($arg =~ /^\s*$/) {
+# return $self->get();
+# }
+# }
+# if($Global::max_lines) {
+# if($arg =~ /\s$/) {
+# # Trailing space => continued on next line
+# my $cont = $self->get();
+# if(defined $cont) {
+# $arg .= $cont;
+# }
+# }
+# }
+# ::debug("ArgLineQueue->get ",$arg,"\n");
+# return $arg;
+# }
+#}
+#
+#sub unget {
+# my $self = shift;
+# unshift @Global::unget_argv, @_;
+# #push @{$self->{'unget'}}, @_;
+#}
+#
+#sub empty {
+# my $self = shift;
+# my $empty = (not @Global::unget_argv and not @{$self->{'unget'}} and eof($self->{'fh'}));
+# ::debug("ArgLineQueue->empty $empty");
+# return $empty;
+#}
+#
+
package Semaphore;
# This package provides a counting semaphore
@@ -5076,6 +6022,7 @@ sub new {
-d $parallel_locks or mkdir $parallel_locks;
my $lockdir = "$parallel_locks/$id";
my $lockfile = $lockdir.".lock";
+# Carp::cluck($count);
if($count < 1) { die "Semaphore count = $count"; }
return bless {
'lockfile' => $lockfile,
@@ -5179,6 +6126,6 @@ sub unlock {
# Keep perl -w happy
-$::opt_workdir = $Private::control_path = $Semaphore::timeout = $Semaphore::wait =
+$::opt_x = $::opt_workdir = $Private::control_path = $Semaphore::timeout = $Semaphore::wait =
$::opt_skip_first_line = $::opt_shebang = 0 ;
diff --git a/src/sql b/src/sql
index 1991550c..9e5c0c90 100755
--- a/src/sql
+++ b/src/sql
@@ -528,7 +528,7 @@ $Global::Initfile && unlink $Global::Initfile;
exit ($err);
sub parse_options {
- $Global::version = 20101115;
+ $Global::version = 20101122;
$Global::progname = 'sql';
# This must be done first as this may exec myself
diff --git a/testsuite/tests-to-run/test03.sh b/testsuite/tests-to-run/test03.sh
index 46f15f3e..5c1d1402 100755
--- a/testsuite/tests-to-run/test03.sh
+++ b/testsuite/tests-to-run/test03.sh
@@ -7,7 +7,7 @@ cp -a input-files/testdir2 tmp
cd tmp
# tests if -c (cat | sh) works
-perl -e 'for(1..25) {print "echo a $_; echo b $_\n"}' | $PAR -c 2>&1 | sort
+perl -e 'for(1..25) {print "echo a $_; echo b $_\n"}' | $PAR 2>&1 | sort
cd ..
rm -rf tmp
diff --git a/testsuite/tests-to-run/test06.sh b/testsuite/tests-to-run/test06.sh
index b3878209..bdd38e5e 100755
--- a/testsuite/tests-to-run/test06.sh
+++ b/testsuite/tests-to-run/test06.sh
@@ -6,11 +6,11 @@ rm -rf tmp 2>/dev/null
cp -a input-files/testdir2 tmp
cd tmp
-# tests if -c (cat | sh) works
-perl -e 'for(1..25) {print "echo a $_; echo b $_\n"}' | $PAR -c 2>&1 | sort
+# tests if cat | sh-mode works
+perl -e 'for(1..25) {print "echo a $_; echo b $_\n"}' | $PAR 2>&1 | sort
-# tests if -f (xargs) works
-perl -e 'for(1..25) {print "a $_\nb $_\n"}' | $PAR -f echo 2>&1 | sort
+# tests if xargs-mode works
+perl -e 'for(1..25) {print "a $_\nb $_\n"}' | $PAR echo 2>&1 | sort
cd ..
rm -rf tmp
diff --git a/testsuite/tests-to-run/test15.sh b/testsuite/tests-to-run/test15.sh
index 35cc10e3..acd5b174 100755
--- a/testsuite/tests-to-run/test15.sh
+++ b/testsuite/tests-to-run/test15.sh
@@ -68,18 +68,16 @@ perl -e 'print "z"x1000000' | parallel echo 2>&1
perl -e 'print "z"x1000000' | xargs echo 2>&1
(seq 1 10; perl -e 'print "z"x1000000'; seq 12 15) | stdout parallel -j1 -km -s 10 echo
(seq 1 10; perl -e 'print "z"x1000000'; seq 12 15) | stdout xargs -s 10 echo
-(seq 1 10; perl -e 'print "z"x1000000'; seq 12 15) | stdout parallel -j1 -k -s 10 echo
+(seq 1 10; perl -e 'print "z"x1000000'; seq 12 15) | stdout parallel -j1 -kX -s 10 echo
echo '### Test -x'
(seq 1 10; echo 12345; seq 12 15) | stdout parallel -j1 -km -s 10 -x echo
-(seq 1 10; echo 12345; seq 12 15) | stdout parallel -j1 -k -s 10 -x echo
+(seq 1 10; echo 12345; seq 12 15) | stdout parallel -j1 -kX -s 10 -x echo
(seq 1 10; echo 12345; seq 12 15) | stdout xargs -s 10 -x echo
(seq 1 10; echo 1234; seq 12 15) | stdout parallel -j1 -km -s 10 -x echo
-(seq 1 10; echo 1234; seq 12 15) | stdout parallel -j1 -k -s 10 -x echo
+(seq 1 10; echo 1234; seq 12 15) | stdout parallel -j1 -kX -s 10 -x echo
(seq 1 10; echo 1234; seq 12 15) | stdout xargs -s 10 -x echo
-echo '### Test bugfix if no command given'
-(echo echo; seq 1 5; perl -e 'print "z"x1000000'; seq 12 15) | stdout parallel -j1 -km -s 10
-
-
+#echo '### Test bugfix if no command given'
+#(echo echo; seq 1 5; perl -e 'print "z"x1000000'; seq 12 15) | stdout parallel -j1 -km -s 10
echo '### Test -a and --arg-file: Read input from file instead of stdin'
seq 1 10 >/tmp/$$
@@ -185,4 +183,4 @@ echo '### Test --verbose and -t'
echo '### Test --show-limits'
(echo b; echo c; echo f) | parallel -k --show-limits echo {}ar
-(echo b; echo c; echo f) | parallel -k --show-limits -s 100 echo {}ar
+(echo b; echo c; echo f) | parallel -kX --show-limits -s 100 echo {}ar
diff --git a/testsuite/tests-to-run/test27.sh b/testsuite/tests-to-run/test27.sh
index 0b4f092d..4e6a7a54 100644
--- a/testsuite/tests-to-run/test27.sh
+++ b/testsuite/tests-to-run/test27.sh
@@ -14,7 +14,7 @@ stdout xargs -d o -n1 echo < helloworld.xi
stdout parallel -k -d o -n1 echo < helloworld.xi
echo '### -E_ -0 echo < eof_-0.xi'
stdout xargs -E_ -0 echo < eof_-0.xi
-stdout parallel -k -E_ -0 echo < eof_-0.xi
+stdout parallel -X -k -E_ -0 echo < eof_-0.xi
echo '### -i -0 echo from \{\} to x{}y < items-0.xi'
stdout xargs -i -0 echo from \{\} to x{}y < items-0.xi
stdout parallel -k -i -0 echo from \{\} to x{}y < items-0.xi
@@ -23,10 +23,10 @@ stdout xargs -i -s26 -0 echo from \{\} to x{}y < items-0.xi
stdout parallel -k -i -s26 -0 echo from \{\} to x{}y < items-0.xi
echo '### -l -0 echo < ldata-0.xi'
stdout xargs -l -0 echo < ldata-0.xi
-stdout parallel -k -l -0 echo < ldata-0.xi
+stdout parallel -l -k -0 echo < ldata-0.xi
echo '### -l -0 echo < ldatab-0.xi'
stdout xargs -l -0 echo < ldatab-0.xi
-stdout parallel -k -l -0 echo < ldatab-0.xi
+stdout parallel -l -k -0 echo < ldatab-0.xi
echo '### -L2 -0 echo < ldata-0.xi'
stdout xargs -L2 -0 echo < ldata-0.xi
stdout parallel -k -L2 -0 echo < ldata-0.xi
@@ -64,19 +64,19 @@ stdout xargs -r echo this plus that < blank.xi
stdout parallel -k -r echo this plus that < blank.xi
echo '### -0 -s118 echo < stairs-0.xi'
stdout xargs -0 -s118 echo < stairs-0.xi
-stdout parallel -k -0 -s118 echo < stairs-0.xi
+stdout parallel -k -X -0 -s118 echo < stairs-0.xi
echo '### -0 -s19 echo < stairs-0.xi'
stdout xargs -0 -s19 echo < stairs-0.xi
-stdout parallel -k -0 -s19 echo < stairs-0.xi
+stdout parallel -k -X -0 -s19 echo < stairs-0.xi
echo '### -0 -s19 echo < stairs2-0.xi'
stdout xargs -0 -s19 echo < stairs2-0.xi
-stdout parallel -k -0 -s19 echo < stairs2-0.xi
+stdout parallel -k -X -0 -s19 echo < stairs2-0.xi
echo '### -0 -s20 echo < stairs-0.xi'
stdout xargs -0 -s20 echo < stairs-0.xi
-stdout parallel -k -0 -s20 echo < stairs-0.xi
+stdout parallel -k -X -0 -s20 echo < stairs-0.xi
echo '### -0 -s30 echo < stairs-0.xi'
stdout xargs -0 -s30 echo < stairs-0.xi
-stdout parallel -k -0 -s30 echo < stairs-0.xi
+stdout parallel -k -X -0 -s30 echo < stairs-0.xi
echo '### -0 echo this plus that < space.xi'
stdout xargs -0 echo this plus that < space.xi
stdout parallel -k -0 echo this plus that < space.xi
@@ -109,7 +109,7 @@ stdout xargs -E_ -IARG echo from ARG to xARGy < eof_.xi
stdout parallel -k -E_ -IARG echo from ARG to xARGy < eof_.xi
echo '### -s470 echo hi there < files.xi'
stdout xargs -s470 echo hi there < files.xi
-stdout parallel -k -s470 echo hi there < files.xi
+stdout parallel -k -s470 -X echo hi there < files.xi
echo '### -IARG echo from ARG to xARGy -E_ < eof_.xi'
stdout xargs -IARG echo from ARG to xARGy -E_ < eof_.xi
stdout parallel -k -IARG echo from ARG to xARGy -E_ < eof_.xi
@@ -118,7 +118,7 @@ stdout xargs -IARG echo from ARG to xARGy < items.xi
stdout parallel -k -IARG echo from ARG to xARGy < items.xi
echo '### -IARG -s15 echo ARG < stairs.xi'
stdout xargs -IARG -s15 echo ARG < stairs.xi
-stdout parallel -k -IARG -s15 echo ARG < stairs.xi
+stdout parallel -k -IARG -X -s15 echo ARG < stairs.xi
echo '### -L2 echo < ldatab.xi'
stdout xargs -L2 echo < ldatab.xi
stdout parallel -k -L2 echo < ldatab.xi
@@ -154,31 +154,31 @@ stdout xargs echo < quotes.xi
stdout parallel -k echo < quotes.xi
echo '### -s118 echo < stairs.xi'
stdout xargs -s118 echo < stairs.xi
-stdout parallel -k -s118 echo < stairs.xi
+stdout parallel -k -X -s118 echo < stairs.xi
echo '### -s19 echo < stairs2.xi'
stdout xargs -s19 echo < stairs2.xi
-stdout parallel -k -s19 echo < stairs2.xi
+stdout parallel -k -X -s19 echo < stairs2.xi
echo '### -s19 echo < stairs.xi'
stdout xargs -s19 echo < stairs.xi
-stdout parallel -k -s19 echo < stairs.xi
+stdout parallel -k -X -s19 echo < stairs.xi
echo '### -s20 echo < stairs.xi'
stdout xargs -s20 echo < stairs.xi
-stdout parallel -k -s20 echo < stairs.xi
+stdout parallel -k -X -s20 echo < stairs.xi
echo '### -s30 echo < stairs.xi'
stdout xargs -s30 echo < stairs.xi
-stdout parallel -k -s30 echo < stairs.xi
+stdout parallel -k -X -s30 echo < stairs.xi
echo '### -s470 echo < files.xi'
stdout xargs -s470 echo < files.xi
-stdout parallel -k -s470 echo < files.xi
+stdout parallel -k -X -s470 echo < files.xi
echo '### -s47 echo < files.xi'
stdout xargs -s47 echo < files.xi
-stdout parallel -k -s47 echo < files.xi
+stdout parallel -k -X -s47 echo < files.xi
echo '### -s48 echo < files.xi'
stdout xargs -s48 echo < files.xi
-stdout parallel -k -s48 echo < files.xi
+stdout parallel -k -X -s48 echo < files.xi
echo '### -s6 echo < files.xi'
stdout xargs -s6 echo < files.xi
-stdout parallel -k -s6 echo < files.xi
+stdout parallel -k -X -s6 echo < files.xi
echo '### -iARG -s86 echo ARG is xARGx < files.xi'
stdout xargs -iARG -s86 echo ARG is xARGx < files.xi
stdout parallel -k -iARG -s86 echo ARG is xARGx < files.xi
@@ -241,7 +241,7 @@ stdout xargs -l2 echo < files.xi
stdout parallel -k -l2 echo < files.xi
echo '### -s30 -t echo < stairs.xi'
stdout xargs -s30 -t echo < stairs.xi
-stdout parallel -k -s30 -t echo < stairs.xi
+stdout parallel -k -X -s30 -t echo < stairs.xi
echo '### -t echo this plus that < space.xi'
stdout xargs -t echo this plus that < space.xi
stdout parallel -k -t echo this plus that < space.xi
diff --git a/testsuite/tests-to-run/test29.sh b/testsuite/tests-to-run/test29.sh
index 4bf590c2..8e233420 100644
--- a/testsuite/tests-to-run/test29.sh
+++ b/testsuite/tests-to-run/test29.sh
@@ -19,6 +19,7 @@ echo '### Test of trailing space continuation'
(echo foo; echo '';echo 'ole ';echo bar;echo quux) | xargs -r -L2 echo
(echo foo; echo '';echo 'ole ';echo bar;echo quux) | parallel -kr -L2 echo
parallel -kr -L2 echo ::: foo '' 'ole ' bar quux
+echo '### Test of trailing space continuation with -E eof'
(echo foo; echo '';echo 'ole ';echo bar;echo quux) | xargs -r -L2 -E bar echo
(echo foo; echo '';echo 'ole ';echo bar;echo quux) | parallel -kr -L2 -E bar echo
parallel -kr -L2 -E bar echo ::: foo '' 'ole ' bar quux
diff --git a/testsuite/tests-to-run/test34.sh b/testsuite/tests-to-run/test34.sh
index c9b9308d..021994ea 100644
--- a/testsuite/tests-to-run/test34.sh
+++ b/testsuite/tests-to-run/test34.sh
@@ -2,6 +2,6 @@
echo '### Test too slow spawning'
(sleep 0.3; seq 1 10 | parallel -j50 burnP6) &
-seq 1 500 | nice nice stdout timeout 10 \
+seq 1 500 | nice nice stdout timeout -k 1 10 \
parallel -j500 "killall -9 burnP6 2>/dev/null ; echo {} >/dev/null"
killall -9 burnP6
diff --git a/testsuite/tests-to-run/test39.sh b/testsuite/tests-to-run/test39.sh
new file mode 100644
index 00000000..26f1a0a2
--- /dev/null
+++ b/testsuite/tests-to-run/test39.sh
@@ -0,0 +1,62 @@
+#!/bin/bash
+
+
+# Tests that failed for OO-rewrite
+
+parallel -u --semaphore seq 1 10 '|' pv -qL 20
+sem --wait; echo done
+
+
+echo a | parallel echo {1}
+
+echo "echo a" | parallel
+
+parallel -j1 -I :: -X echo 'a::b::^c::[.}c' ::: 1
+
+echo "### BUG: The length for -X is not close to max (131072)"
+seq 1 4000 | parallel -X echo {.} aa {}{.} {}{}d{} {}dd{}d{.} |head -n 1 |wc
+
+echo "### BUG: empty lines with --show-limit"
+echo | parallel --show-limits
+
+echo '### Test -N'
+seq 1 5 | parallel -kN3 echo {1} {2} {3}
+
+echo '### Test --arg-file-sep with files of different lengths'
+parallel --arg-file-sep :::: -k echo {1} {2} :::: <(seq 1 1) <(seq 3 4)
+
+echo '### Test respect -s'
+parallel -kvm -IARG -s15 echo ARG ::: 1 22 333 4444 55555 666666 7777777 88888888 999999999
+
+echo '### Test eof string after :::'
+parallel -k -E ole echo ::: foo ole bar
+
+echo '### Test -C and --trim rl'
+parallel -k -C %+ echo '"{1}_{3}_{2}_{4}"' ::: 'a% c %%b' 'a%c% b %d'
+
+echo '### Test empty input'
+= 10) at number 1: zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz...
9 10
+Command line too long (1000007 >= 10) at number 6: zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz...
1 2
3 4
5 6
@@ -90,19 +90,21 @@ xargs: argument line too long
3 4
5 6
7 8
-Command line too long (1000007 >= 10) at number 1: zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz...
9 10
+Command line too long (1000007 >= 10) at number 6: zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz...
### Test -x
1 2
3 4
5 6
-Command line too long (15 >= 10) at number 3: 12345...
7 8
+9 10
+Command line too long (10 >= 10) at number 6: 12345...
1 2
3 4
5 6
-Command line too long (15 >= 10) at number 3: 12345...
7 8
+9 10
+Command line too long (10 >= 10) at number 6: 12345...
1 2
3 4
5 6
@@ -138,9 +140,6 @@ xargs: argument line too long
13
14
15
-### Test bugfix if no command given
-Command line too long (1000002 >= 10) at number 1: zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz...
-1 2 3 4 5
### Test -a and --arg-file: Read input from file instead of stdin
1
2
diff --git a/testsuite/wanted-results/test27 b/testsuite/wanted-results/test27
index c8c3a383..8f5a7193 100644
--- a/testsuite/wanted-results/test27
+++ b/testsuite/wanted-results/test27
@@ -25,8 +25,7 @@ hell
rld
### -E_ -0 echo < eof_-0.xi
one two _ three four
-one
-two
+one two
### -i -0 echo from \{\} to x{}y < items-0.xi
from one to xoney
from
@@ -71,7 +70,7 @@ y
### -i -s26 -0 echo from \{\} to x{}y < items-0.xi
xargs: argument list too long
from one to xoney
-Command line too long (42 >= 26) at number 1: \ \\
\ \'
+Command line too long (42 >= 26) at number 2: \ \\
\ \'
'...
### -l -0 echo < ldata-0.xi
1 22 333 4444
@@ -84,6 +83,14 @@ Command line too long (42 >= 26) at number 1: \ \\
\ \'
88888888
999999999
1 22 333 4444
+55555 666666
+7777777
+88888888
+999999999 1 22
+333 4444 55555
+666666 7777777
+88888888
+999999999
### -l -0 echo < ldatab-0.xi
1 22 333 4444
@@ -96,6 +103,12 @@ Command line too long (42 >= 26) at number 1: \ \\
\ \'
88888888
999999999
+1 22 333 4444 55555 666666
+7777777 88888888 999999999 1 22
+333 4444 55555
+666666 7777777
+88888888
+999999999
### -L2 -0 echo < ldata-0.xi
1 22 333 4444 55555 666666
7777777 88888888
@@ -1075,6 +1088,6 @@ foo bar
echo baz ugh
baz ugh
echo foo\ bar baz\
-echo ugh
+echo ugh
foo bar baz
ugh
diff --git a/testsuite/wanted-results/test29 b/testsuite/wanted-results/test29
index 7b9e9323..f4e2c4be 100644
--- a/testsuite/wanted-results/test29
+++ b/testsuite/wanted-results/test29
@@ -6,7 +6,6 @@
parallel: --trim must be one of: r l rl lr
### Test of eof string on :::
foo
-ole
### Test of ignore-empty string on :::
foo
ole
@@ -18,10 +17,10 @@ foo ole bar
quux
foo ole bar
quux
+### Test of trailing space continuation with -E eof
foo ole
foo ole
-foo ole bar
-quux
+foo ole
### Test of --colsep
a b c
a b c {4}
@@ -40,9 +39,7 @@ abc def
ghi jkl
### Test of multiple -a plus colsep
abc def
-ghi jkl
-mno
-pqr
+mno jkl
### Test of multiple -a no colsep
abc def ghi
jkl mno pqr
diff --git a/testsuite/wanted-results/test33 b/testsuite/wanted-results/test33
index 3e1ad0e4..85d4c751 100644
--- a/testsuite/wanted-results/test33
+++ b/testsuite/wanted-results/test33
@@ -1,7 +1,7 @@
### Test of -j filename
Parsing of --jobs/-j/--max-procs/-P failed
Usage:
-parallel [options] [command [arguments]] < list_of_arguments)
+parallel [options] [command [arguments]] < list_of_arguments
parallel [options] [command [arguments]] ::: arguments
parallel [options] [command [arguments]] :::: argfile(s)
diff --git a/testsuite/wanted-results/test39 b/testsuite/wanted-results/test39
new file mode 100644
index 00000000..24184ff9
--- /dev/null
+++ b/testsuite/wanted-results/test39
@@ -0,0 +1,70 @@
+1
+2
+3
+4
+5
+6
+7
+8
+9
+10
+done
+a
+a
+a1b1^c1[.}c
+### BUG: The length for -X is not close to max (131072)
+ 1 12821 131060
+### BUG: empty lines with --show-limit
+Maximal size of command: 131071
+Maximal used size of command: 131071
+
+Execution of will continue now, and it will try to read its input
+and run commands; if this is not what you wanted to happen, please
+press CTRL-D or CTRL-C
+### Test -N
+1 2 3
+4 5
+### Test --arg-file-sep with files of different lengths
+1 3
+4
+### Test respect -s
+echo 1 22 333
+1 22 333
+echo 4444
+4444
+echo 55555
+55555
+echo 666666
+666666
+echo 7777777
+7777777
+echo 88888888
+88888888
+echo 999999999
+999999999
+### Test eof string after :::
+foo
+### Test -C and --trim rl
+a_b_c_{4}
+a_b_c_d
+### Test empty input
+### Test -m
+1 2
+### Test :::
+1
+### Test context_replace
+'a'
+### Test -N2 {2}
+arg1:1 seq:1 arg2:2
+arg1:3 seq:2 arg2:4
+### Test -E (should only output foo ole)
+foo ole
+foo ole
+### Test -r (should only output foo ole bar\nquux)
+foo ole bar
+quux
+### Test of tab as colsep
+abc def
+ghi jkl
+abc def
+ghi jkl
diff --git a/testsuite/wanted-results/test40 b/testsuite/wanted-results/test40
new file mode 100644
index 00000000..ba840ed0
--- /dev/null
+++ b/testsuite/wanted-results/test40
@@ -0,0 +1,8 @@
+### Computing length of command line
+1 2
+11 1
+12 2
+3
+a_b_c_{4}
+a_b_c_d
+{4}