GNU shtool - The GNU Portable Shell Tool
GNU shtool SHTOOL_VERSION_STR
shtool [ global_options ] command [ command_options ] [ command_args ]
GNU shtool is a compilation of small but very stable and portable shell scripts into a single shell tool. All ingredients were in successful use over many years in various free software projects. The compiled shtool script is intended to be used inside the source tree of those free software packages. There it can take over various (usually non-portable) tasks related to the building and installation of such packages.
For the configuration, build and installation environment of modern free
software packages one nowadays uses GNU autoconf and friends (i.e.
usually autoconf, automake and libtool). Autoconf covers the
configuration, automake covers the generation of the build environment and
libtool covers most of a libraries build process. But at least when it
comes to the installation step one usually have to use a few auxiliary scripts
like mkdir.sh, install.sh, etc. These scripts are usually replacements
for system tools and are required mainly for portability reasons. The result
is usually an etc/ subdirectory in the source tree where over time a
lot shell scripts accumulate.
The problem with those etc/ shell scripts starts if one has to maintain
lots of free software packages as it's the case for the author of shtool.
Then over time all etc/ directories diverge and with every day it gets more
and more nasty to always keep them in sync. Especially if some scripts
were locally adjusted because no centralized maintainance location exists, of
course. For autoconf no such problem exists, because the resulting
configure script is generated on-the-fly. The same applies to automake
and the various Makefile.in files.
Only for libtool one always has to grab the latest copy. But because it's
just two files (ltmain.sh and ltconfig), keeping a source trees in sync
is not too complicated (especially not if using the handy libtoolize
program). But the etc/ shell script mess is nasty, especially because there
is no master version on the net. Additionally everytime one starts a new
project, one has to establish a new source tree. For a GNU hacker it's
immediately clear that autoconf and friends are part of the game. But which
etc/ shell scripts are needed this time? And from which existing source
tree to copy them from? Hmmm... the same procedure as last year?!
When a free software package has a large source tree (say, more than 50 files
and especially with one or more subdirectories) it's usually no problem to
have an additional etc/ subdirectory with some scripts. They then totally
go down. But for smaller packages, especially those living in a single source
directory (a degenerated tree), some people (like the author of shtool)
have aesthetic problems. Because it looks strange to them that 20% of the
files in the source tree are just auxiliary scripts. Sure, the actual amount
of script code even shtool cannot reduce, but shtool merges them
together into a single file and this way they optically totally disappear from
the source tree.
This is a pure aesthetical issue, of course. But keep in mind that hacking is a piece of art. And a well layouted source tree is a piece of art for real hackers, too. Oh, and for those who really insist on a technical reason: it's also easier to upgrade a single file than multiple files ;)
So, wouldn't it be nice to have a fourth package (beside autoconf,
automake and libtool) which fills the gap, i.e. which provides the
functionality of the old files in etc/, is maintained at a centralized
location and even consists of just a single (perhaps large) script one can
threat as a black box the same way one already does this for libtool? The
author thought this would be actually very useful and the result is the
current GNU shtool package which at least successfully solved the above
problems of the author.
To better understand the intentions behind shtool and to avoid confusion, here are the original goals of the shtool script:
fmt, tr, etc) or
unportable features of well-known Unix programs (like shell functions, special
sed features, etc. pp). That's why shtool's code sometimes looks crazy
and like overkill to you. Don't think this is because of the authors
crazyness. Instead it's most of the time mainly for portability reasons.
shtoolize(1) which can be used to
build individual shtool scripts out of the ingredient shell scripts. This
way if one don't need all the available functionality one can assemble
together an individual shtool script.
The following global options are available for shtool. Any commands are ignored if one of them is present on the shtool command line.
shtoolize(1) call.
The following commands are provided by shtool. They are all called via
``shtool command''. Any trailing command_options are specific to the
particular command. They are listed (here and also below) sorted by topic,
i.e. related commands are listed side-by-side.
echo(1) style print command providing special expansion constructs (terminal
bold mode, environment details, date) and newline control.
mv(1) style command, but can rename/move multiple files at once and allows
source files just to be deleted if contents didn't change.
mkdir(1) style command providing support for auto-parent-dir creation,
directory permission control and smart skipping if directory already
exists.
ln(1) style command providing automatic calculation and usage of relative
links if possible.
sed(1) substitution operations.
cpp(1) code, internal
variables and internal functions.
In the following the available commands and their corresponding command_options are described in detail.
echo(1) style print command which provides special expansion
constructs (terminal bold mode, environment details, date) and newline
control. Per default string is written to stdout followed by a newline
character (``\n''). When option ``-n'' is used this newline character is
left out.
The str can contain special ``%x'' constructs which which are expanded before the output is written if option ``-e'' is used. Currently the following constructs are recognized: ``%B'' for switching to terminal bold mode, ``%b'' for switching terminal mode back to normal display mode, ``%u'' for the current user name, ``%U'' for the current user id (numerical), ``%g'' for the current group name, ``%G'' for the current group id (numerical), ``%h'' for the current hostname, ``%d'' for the current domain name, ``%D'' for the current day of the month, ``%M'' for the current month (numerical), ``%m'' for the current month name and ``%Y'' for the current year.
The trick of this command is that it provides a portable ``-n'' option and hides the gory details needed to find out the environment details.
Examples:
# shell script shtool echo -n -e "Enter your name [%B%u%b]: "; read name shtool echo -e "Your Email address might be %u@%h%d" shtool echo -e "The current date is %D-%m-%Y"
The default for spec is ``dmy'' which means an output of ``<day> <month>
<year>''. Any combination of the chars ``d'', ``m'' and ``y'' or
allowed for spec.
The trick of this command is that it provides a portable way to find out the date of a file or directory while still allowing one to specify the format of the date display.
Examples:
# shell script shtool mdate -n / shtool mdate -f '/' -z -d -o ymd foo.txt shtool mdate -f '-' -s foo.txt
The trick of this command is that it avoids to use the unportable tr(1) and
fmt(1) commands and instead is based entirely on sh(1), awk(1) and sed(1)
functionality.
Example:
# shell script shtool table -F , -w 5 -c 4 "1,2,3,4,5,6,7,8,9,10,11,12"
|'') sequence where on stdin
logging/processing informations found. For every line on stdin the
propeller cycles one step clock-wise.
The trick of this command is that it provides a portable and easy to use way to display such nice and psychologically important process indicators.
Example:
# shell script
configure 2>&1 |\
tee logfile |\
shtool prop -p "Configuring sources"
mv(1) style command, but with two special features: First if
option ``-e'' (`expand') is used and an asterisk occurs somewhere in src
one can use ``%n'' (where n is 1,2,...) in dst-file. This is
useful for renaming multiple files at once. Second, if option ``-p''
(for `preserve') is used and src-file and dst-file are byte-wise the
same it just deletes src-file. The intention is that the permissions and
time stamps on dst aren't changed which is important if dst-file is
used in conjunction with Makefiles. Option ``-v'' (verbose) can be used to
enable the output of extra processing information. Option ``-t'' (trace)
can be used to enable the output of the essential shell commands which are
executed.
The trick of this command is that it can rename multiple files at once and preserves the timestamps if the contents isn't changed.
Examples:
# shell script shtool move -v -e '*.txt' %1.asc
# Makefile
scanner.c: scanner.l
lex scanner.l
shtool move -t -p lex.yy.c scanner.c
install(1)
command. Per default file is moved to the target path, but with option
``-c'' file is copied. The target file is created with owner/group set
to the current active uid/gid, but if this script is called as root (uid 0)
the options ``-o'' and ``-g'' can be used to override this.
Additionally program executables is stripped with strip(1) after
installation if option ``-s'' is used. Option ``-C'' is like
``-c'', except if the destination file already exists and the files
are the same, the source is just removed. Option ``-e'' can be used
one or multiple times to apply one or more sed(1) commands on-the-fly
to the contents of the input file before the output file is created.
Option ``-v'' (verbose) can be used to enable the output of extra
processing information. Option ``-t'' (trace) can be used to enable
the output of the essential shell commands which are executed.
The trick of this command is that it provides the functionality of BSD
install(1) in a portable emulated way.
Example:
# Makefile
install:
:
shtool install -c -s -m 4755 foo $(bindir)/
shtool install -c -m 644 foo.man $(mandir)/man1/foo.1
shtool install -c -m 644 -e "s/@p@/$prefix/g" foo.conf $(etcdir)/
mkdir(1) style command providing support for automatic parent
directory creation (if option ``-p'' is used), directory permission
control (with option ``-m mode'' where mode can be in any of
the formats specified to the chmod(1) command) and smart skipping if
dir already exists (triggered by the force option ``-f''). Option
``-t'' (trace) can be used to enable the output of the essential
shell commands which are executed.
The trick of this command is that it provides both a portable ``-p'' functionality and the ability to be smart if the directory already exists which is important for installation procedures.
Example:
# Makefile
install:
shtool mkdir -f -p -m 755 $(bindir)
shtool mkdir -f -p -m 755 $(mandir)/man1
:
ln(1) style command which provides automatic calculation and usage
of relative links if possible, i.e. usually if src-path and dst-path
are not absolute paths or at least they share a common prefix except the root
directory (``/''). When more than one src-path is specified, all of them
are linked into dst-path. Options ``-f'' and ``-s'' are similar to
ln(1), i.e. they force the creation of the link (even if it exists) and
create a symbolic link instead of a hard-link. Option ``-t'' (trace) can
be used to enable the output of the essential ``ln'' command which is
executed.
The trick of this command is that it tried hard to calculate the paths to get the maximum possible relative paths.
Example:
# shell script shtool mkln -s foo/bar baz/quux
Option ``-v'' (verbose) can be used to enable some displaying of processing information. Option ``-t'' (trace) can be used to display all commands which are executed in order to construct dst-dir. Option ``-a'' (all) can be used to really shadow all files and directories in src-dir. Per default CVS related files and directories, backup files, object files, etc. are not shadowed.
The trick of this is that is provides such a high-level functionality with a single command and hides all gory details.
Example:
# shell script shtool mkshadow -v -a . /tmp/shadow
The trick is that this is more convenient that having to set the permissions manually or with a large file list.
Example:
# Makefile.in
dist:
shtool fixperm -v *
...
A rotation step consists of the following steps: 1. remove archive file number count-1; 2. move archive file number N-1 to N for N counting from 1 to count-1; 3. move file to archive file number 0; 4. creating a new and empty instance of file.
Option ``-s'' can be used to only start a rotation step if file is
at least size bytes long. The argument size can be specified also
with the trailing units K (kilo), M (mega) or G (giga).
Option ``-c'' changes the approach of moving file to archive file number 0: instead of a move followed by the creation of a new file, a copy is performed followed by a truncation of file. The difference is that in the first case (the default), if an application has file still opened, after the rotation step it will have archive file number 0 opened and usually has to reopen the new file, while in the second case the application can keep its open file handles to file. The drawback of the second approach is that logfile entries are lost when they are written to file between the execution of the copy and the subsequent truncation operation.
Option ``-r'' removes file after rotation instead of providing a new empty file. Option ``-a'' forces archive files to be created in the separate directory dir.
Option ``-z'' enables compression of archive files with compression
level level (if option ``-b'' is present, compression takes place in
background). By default, the tools bzip2(1), gzip(1) and compress(1) are
searched for in $PATH (in this order), but one also can override this
by prefixing the compression level with one of the three particular tool
names. Option ``-d'' delays the compression of archive file number 0.
This is useful if option ``-c'' is not used, because an application
might still write to archive file 0 (through an open file handle).
Option ``-p'' enables padding with leading zeros in the number part of the filename ``file.numbercompress-suffix''. The default padding len is 1. This is interesting if more than 10 archive files are used, because it leads to still sorted directory listings.
Options ``-o'', ``-g'' and ``-m'' can be used to make sure that
the created files have particular file attributes. The valid arguments
are the same as for chown(1), chgrp(1) and chmod(1). Be aware that using
options ``-o'' and ``-g'' require root privileges.
Option ``-M'' allows one to execute a ``migration'' command just before the archive file number count-1 is removed from the filesystem. The specified cmd gets the archive filename as an argument appended. Options ``-P'' (prolog) and ``-E'' (epilog) can be used to execute commands before and after the rotation step. They are interesting in conjunction with option ``-s'', because they are not executed at all if it is decided that no rotation step is performed.
Option ``-f'' (force) can be used to allow the archive directory (option ``-a'') to be silently created if it still does not exist and that still not existing intermediate logfiles are silently skipped in the rotation step.
Option ``-v'' (verbose) can be used to display the files which are rotated. Option ``-t'' (trace) can be used to enable the output of the essential shell commands which are executed for the rotation step.
Example:
# shell script shtool rotate -n10 -s1M -zbzip2:9 -d -r /var/log/ap.access.log shtool rotate -n5 -s128K -zbzip2:9 -d -r /var/log/ap.error.log apachectl graceful
tar'. The important aspects of
standardized free software tarballs are: first they have to unpack into a
single top-level directory; second this top-level directory should correspond
to the tarball filename (i.e. a tarball `foobar-0.8.15.tar' per convention
unpacks into a top-level directory `foobar-0.8.15/'); third the files in
the tarball should be sorted to allow users to use the `tar tvf -' command
in a reasonable way; fourth the owner and group of the files in the tarball
for security reasons can be set to arbitrary names.
The input files are given by the file or directory arguments path.
Directories are expanded before the comma-separated exclude (option -e)
patterns (grep regular expressions) are used to filter the list. The
default filter is ``CVS,\\.cvsignore,\\.[oa]\$''. Then the tarball is
created with its files owned by user (option -u) and group (option
-g). Finally the resulting tarball is piped through an optional compression
(option -c) program and written to the output file tarball (option
-o). Option ``-v'' can be used to display the files which are stored in
the tarball. Option ``-t'' (trace) can be used to enable the output of the
essential shell commands which are executed.
The trick of this command is that it combines the complex process of rolling a good tarball into a single command.
Example:
# Makefile.in
dist:
...
V=`shtool version -d short ...'; \
shtool tarball -o foobar-$$V.tar.gz -c 'gzip -9' \
-u bar -g gnu -e 'CVS,\.cvsignore' .
sed(1) substitution operations to
stdin or any number of files. The sed(1) operations are either
specified on the command line with option ``-e'' or are contained
in a file cmd-file and are specified with option ``-f''. The
original untouched file is usually overridden. If option ``-b''
is given and specifies a file extension, the original file is preserved
as ``file.ext''. If option ``-s'' (stealth) is specified,
the timestamp is preserved on file, too. Option ``-i'' enables
interactive mode where the user has to approve each operation. Option
``-n'' (no operation) can be used to disable the actual execution of
the essential shell commands which would be executed. Option ``-t''
(trace) can be used to enable the output of the essential shell commands
which are executed. Option ``-v'' (verbose) can be used to display
the files which are patched.
Example:
# interactive shell shtool subst -i -e 's;(c) \([0-9]*\)-2000;(c) \1-2001;' *.[ch]
# RPM spec-file
%install
shtool subst -v -n \
-e 's;^\(prefix=\).*;\1 $RPM_BUILD_ROOT%{_prefix};g' \
-e 's;^\(sysconfdir=\).*;\1 $RPM_BUILD_ROOT%{_prefix}/etc;g' \
`find . -name Makefile -print`
make install
i686-pc-freebsd3.1''. When you
need a more sophisticated platform guesser, use the GNU
config.guess/config.sub scripts, please.
# configure.in OS=`shtool guessos`
ar'') tool. It provides the ability
to create archives out of existing archives, i.e. if one of file matches
``*.a'' the archive member files of file are used instead of file
itself. When option ``-t'' (trace) is given arx shows the actually
involved shell commands. Option ``-C'' can be used to set the ``ar''
command to cmd.
The trick of this command is the automatic handling of archive members which is especially interesting if one wants to construct a (usually top-level) library archive out of pre-build sub-library archives (usually staying inside subdirs) in a large source tree.
Example:
# Makefile
AR=ar
RANLIB=ranlib
:
OBJS=foo.o bar.o
LIBS=baz/libbaz.a quux/libquux.a
:
libfoo.a: $(OBJS) $(LIBS)
shtool arx -C $(AR) rc libfoo.a $(OBJS) $(LIBS)
$(RANLIB) libfoo.a
stdout which
defines six variables containing the ``-L'' and ``-l'' options sorted by
class:
``SLO_DIRS_OBJ'' and ``SLO_LIBS_OBJ'' contains the ``-L'' and
``-l'' options of static libraries, ``SLO_DIRS_PIC'' and
``SLO_LIBS_PIC'' contains the ``-L'' and ``-l'' options of static
libraries containing PIC (``Position Independent Code'') and
``SLO_DIRS_DSO'' and ``SLO_LIBS_DSO'' contains the ``-L'' and
``-l'' options of shared libraries. The -p option can be used to
change the default variable prefix from ``SLO_'' to str.
The intent of this separation is to provide a way between static and shared libraries which is important if one wants to link custom DSOs against libraries, because not all platforms all one to link these DSOs against shared libraries. So one first has to separate out the shared libraries and link the DSO only against the static libraries. One can use this command also to just sort the options.
Example:
# configure.in
LINK_STD="$LDFLAGS $LIBS"
eval `shtool slo $LINK_STD`
LINK_DSO="$SLO_DIRS_OBJ $SLO_LIBS_OBJ $SLO_DIRS_PIC $SLO_LIBS_PIC"
:
cpp(1) code segments, internal variables and internal functions. The intention
for this comes from writing libraries in ANSI C. Here a common shared internal
header file is usually used for sharing information between the library
source files.
The operation is to parse special constructs in files, generate a few
things out of these constructs and insert them at position mark in tfile
by writing the output to ofile. Additionally the files are never touched
or modified. Instead the constructs are removed later by the cpp(1) phase of
the build process. The only prerequisite is that every file has a
``#include "ofile"'' at the top.
This command provides the following features: First it avoids namespace
pollution and reduces prototyping efforts for internal symbols by recognizing
functions and variables which are defined with the storage class identifier
``cname''. For instance if cname is ``intern'', a function ``intern
void *foobar(int quux)'' in one of the files is translated into both a
``#define foobar __foobar'' and a ``extern void *foobar(int quux);'' in
ofile. Additionally a global ``#define cname /**/'' is also
created in ofile to let the compiler silently ignore this additional
storage class identifier.
Second, the library source files usually want to share typedefs,
#defines, etc. over the source file boundaries. To achieve this one can
either place this stuff manually into tfile or use the second feature of
scpp: All code in files encapsulated with ``#if dname ...
#endif'' is automatically copied to ofile. Additionally a global
``#define dname 0'' is also created in ofile to let the compiler
silently skip this parts (because it was already found in the header).
Option ``-v'' can be used to enable some processing output. Option
``-p'' can be used to make the decision whether to overwrite ofile
independent of the generated ``#line'' lines. This is useful for
Makefiles if the real contents of ofile will not change, just
line numbers. Option ``-f'' (which can occur multiple times) can
be used to apply one or more pre-processing sed(1) filter commands
(usually of type ``s/.../.../'') to each input file before their
input is parsed.
Example:
# Makefile
SRCS=foo_bar.c foo_quux.c
foo_p.h: foo_p.h.in
shtool scpp -o foo_p.h -t foo_p.h.in \
-M %%MARK%% -D cpp -C intern $(SRCS)
/* foo_p.h.in */ #ifndef FOO_P_H #define FOO_P_H %%MARK%% #endif /* FOO_P_H */
/* foo_bar.c */
#include "foo_p.h"
#if cpp
#define OURS_INIT 4711
#endif
intern int ours;
static int myone = 0815;
intern int bar(void)
{
ours += myone;
}
/* foo_quux.c */
#include "foo_p.h"
int main(int argc, char *argv[])
{
int i;
ours = OURS_INIT
for (i = 0; i < 10; i++) {
bar();
printf("ours now %d\n", ours);
}
return 0;
}
txt''), ANSI C (lang=``c''), Perl (lang=``perl'') or
Python (lang=``python'') language. The version is always described
with a triple <version,revision,level> and is
represented by a string which always matches the regular expression
``[0-9]+\.[0-9]+[sabp.][0-9]+''. When the option ``-s'' is given,
the contents of file is overridden with the specified version.
When option ``-i'' is used, the current version in file is updated
by increasing one element of the version where knob can be one of
the following: ``v'' for increasing the version by 1 (and resetting
revision and level to 0), ``r'' for increasing the revision by 1 (and
resetting level to 0) or ``l'' for increasing the level by 1. Option
``-e'' can be used to interactively enter a new version.
Unless option ``-e'', ``-i'' or ``-s'' is specified, the performed
action is to display the current version. Option ``-d'' then can be used
to control the display type: ``short'' for a short version display, ``long''
for a longer version display, ``hex'' for a hexadecimal display of the version
and ``libtool'' for a format suitable for use with GNU libtool.
The hexadecimal format for a version v.rtl is VVRRTLL where VV
and RR directly correspond to v and r, T encodes the level
type as 9, 2, 1, 0 (representing s, p/., b, a
in this order) and LL is either directly corresponding to l or set
to 99 if level type is s.
Example:
# shell script shtool version -l c -n FooBar -p foobar -s 1.2b3 version.c
# configure.in V=`shtool version -l c -d long version.c` echo "Configuring FooBar, Version $V"
$PATH variables. It can find a program
executable in $PATH or path through one or more filenames (given by one or
more str arguments). The result is the absolute filesystem path to the
program displayed on stdout plus an exit code of 0 if it was really
found.
The option ``-s'' can be used to suppress the output which is useful to
just test whether a program exists with the help of the return code. The
option ``-m'' enables some magic where currently for the programs
``perl'' and ``cpp'' an advanced magic search is done. The option
``-r'' can be used to transform a forward path to a subdirectory into a
reverse path. Option ``-d'' and ``-b'' just output the directory or base
name of str.
Examples:
# shell script
awk=`shtool path -p "${PATH}:." gawk nawk awk`
perl=`shtool path -m perl`
cpp=`shtool path -m cpp`
revpath=`shtool path -r path/to/subdir`
sh(1), cp(1), rm(1), mkdir(1), awk(1), sed(1).
Some scripts contained in GNU shtool were already written in 1994 by Ralf S. Engelschall for use inside some private source trees. Then they evolved into more elaborated versions over the years and were used in various free software projects like ePerl, WML, iSelect, gFONT, etc. They were complemented with other scripts from the author which he wrote in March 1998 for the ``Apache Autoconf-style Interface'' (APACI) for Apache 1.3. In April 1999 the shtool package was created out of the accumulated master versions of the scripts and in June 1999 it entered the status of an official GNU program and this way finally joined the group of GNU autoconf, GNU automake and GNU libtool.
Ralf S. Engelschall rse@engelschall.com www.engelschall.com