The cmdargs Tcl extension

Introduction

Cmdargs is a Tcl extension that provides a command-line interface to any user-created proc.

Tcl makes heavy use of the "-switch" convention for specifying command-line options. For example,

set res [glob -nocomplain -directory "../dir" -types {f l} *]

Tcl users are familiar with this convention. This style of input is useful when a proc has several optional inputs or multiple parameters. Unfortunately, out of the box, user-created commands have only positional parmeters with (some) support for default values and no support for arbitrary-order, named parameters.

Say we've written a proc:

proc setdata {name {addr {}} {city {}} {age {}} {email {}}} {
    # do something with inputs...
}

Instead of entering

setdata Sam "" "" "" sam@email.net

it would be better if this were possible:

setdata -name Sam -email sam@email.net

This extension cmdargs makes this kind of command-line easy to accomplish. For any practical number of command-line options cmdargs is also quite performant. It has a small amount of overhead, so it may not be optimum for procedures called in tight loops. However, for many or most purposes it's slight performance cost is insignificant.

Installation

Binary installation is straightforward. Download the archive in zip or tgz format. Use zip or tar to decompress into any convenient location.

The archive structure is as follows:

  CmdArgs-ext
    bin
      config
      config.exe
      pkgNdx
      pkgNdx.exe
    doc
    generic
    tcl8
      unix
        cmdargs1.0
          cmdargs10.so
          pkgIndex.tcl
        config.def
      win-msys
        (as in unix)
    tcl9
      unix
        (as above)
      win-msys
        (as above)
    tools
    win

The extension is a shared library found in the cmdargs1.0 directory in each of the 4 directories: Tcl8/unix, Tcl8/win-msys, Tcl9/unix and Tcl9/win-msys. It's only necessary to copy the suitable cmdargs1.0 directory to the package directory used by your installed Tcl. (Usually that's /usr/lib, /usr/local/lib, or TclNNN/lib directory.)

Then put package require cmdargs at the top of the Tcl file.

The use of the other directories is described below, see "Compiling".

Usage

Syntax

cmdargs::chkArgs* <input (list)> <defaults (list[s])> ?<filter> {list}?

<input> : {-optionA ?valueA? ... ?--? ...}  
      # typically collected in <$args> variable
      # <value> isn't required for boolean options 
      # special input "--" signals end of options 
      # everything after "--" is collected in variable <_CA_rest>
<defaults> : <type> <defaults-list> ... 
      # <one (or more) type/defaults pairs>
<type designator> : ?-?(dbool|dnum|dstr|denum) 
      # leading dash in type designator is optional
<defaults-list> : {optionNameA defaultA ...}
<filter> : ?-?dfilter {<nonulls | nozero | noneg> {option-names} ...}

Syntax details

The "cmdargs::" namespace provides 4 commands, chkArgs, chkArgsCi, chkArgsv, and chkArgsvCi. chkArgsCi is the case-insentive variant of chkArgs. The 'v' variants provide a variable, _CA_allvar in the caller. This variable has a list of all varnames created by chkArgs*. The syntax:

cmdargs::chkArgs?vCi? <cmdline (list)> ?-dbool {...}? ?-dnum {...}? ?-dstr {...}? ?-denum {...}? ?-dfilter {...}?

cmdline is the input to the proc, typically using args. proc input is in -option value format, except for boolean options which don't require input values. Typically chkArgs* is the first command in a procedure, like this:

proc test {args} {
    cmdargs::chkArgs $args -dbool ....
    ...
}

Remainder of chkArgs invocation consists of paired parameters, one of <?-?dbool dnum dstr denum> followed by a list of <option value> pairs. For example, CmdArgs::chkArgs $args dbool {flagc 0 flagd 1} dnum {nlines 80 npages 23} ...

The d... identifiers denote the type of the values in the associated list.

IOW using cmdargs provides a basic level of type-checking and validation of proc inputs. This not only reduces errors, it also can save programmers some work validating proc input values.

Note that in the lists the option names (even-numbered elements) will be names of variables created by cmdargs. Values (odd-numbered elements) are the defaults assigned to the named variables. Command-line options/values override these defaults.

Cmdline input

<cmdname> -option value -option1 value1 -booloption -option2 -- <rest>...

Each -option sets a variable option with value. Some conditions apply:

• Option must be among those in the typed lists, it's an error if it's not. (With chkArgsCi command-line options match options in defaults lists without regard to case. So "-OPTION" and "-option" work equally well. Be aware that variables are always created with names matching what's specified in the defaults lists.)
• Value must match the designated type (boolean, numeric, string, or enumerated item). A bool -option doesn't need to be followed by a value.
• Except for numeric values, a value with a leading dash will cause errors. (That's because the value can't be distinguished from an "-option".) If the value must have a leading dash use this format: " -value". Negative numbers are accepted in value positions.
• On the command-line "--" signals the end of option/value inputs. Everything following the "--" is collected in a variable "_CA_rest". This variable is not set if "--" is not present.

Of course, as is standard practice the command-line is easily "shared" with preceding positional arguments:

proc myproc {a b args} {
    chkArgs $args -dstr {tabby "HUGE cat"} -dnum {wt 23.5} 
    if {$a eq "BIG dog"} { ... }
    ...
}

Defaults lists have a type

# boolean
?-?dbool {flagx 0 flagy 1} ## sets flagx 0, flagy 1
cmdline: -flagx -option ... ## flagx is now 1

# numeric
?-?dnum {qty 12 area 234.5 ...} 
cmdline: -qty 23 
         -qty abc -> error (non-numeric value)
         -qtty 23 -> error (no such option)

# string
?-?dstr {item1 "green" ...} ## item1 -> green
cmdline: -item1 reddish     ## item1 -> reddish
         -item1 -itemX ...  ## error (missing value)

# enumerated
?-?denum {food {veggie fruit legume}} ## food -> veggie
# first elem in list "veggie" is default value of var food
# enum list elems can be of any type
cmdline: -food legume -> sets food to "legume"
         -food lamb   -> error ("lamb" not member of enum)

The leading dash is optional for type designators.

At runtime, using "-opts?" or "-options?" prompts cmdargs to print a list of cmdargs-created variables and default values:

    source testfile.tcl
    myproc -opts?
        ----dstr----
            kitchen: modern stuff
             floors: linoleum
              walls: off-white
               auto: Honda
        ----dnum----
               frac: 0.5
               area: 2000
              rooms: 12
              doors: 4
        ---denum----
            critter: cat dog racoon deer ants birds
               food: vegetable fruit meat legume grain
        ---dbool----
              flagd: 0
              flagc: 1

Filters

Filters prevent certain input from being accepted. cmdargs has 3 filters:

• nonulls disallows empty (aka null) string input.
• nozero blocks numeric input when the number is 0 or 0.0.
• noneg screens out input of numbers less than 0.

Filters work on individual options as given in the defaults lists. Filter syntax is straightforward:

# define "myproc"
proc myproc {args} {
  chkArgs $args -dstr {str0 cat str1 dog} \
      -dnum {numA 202 numB 5.5} \
      -dfilter {nonulls {str0} nozero {numA} noneg {numA numB}}
  ...
}

# call myproc with options...
myproc -str0 "" -> Error: empty string not allowed
myproc -str1 "" -> OK.
myproc -numA 0  -> Error: numeric value == 0 not allowed.
myproc -numB 0  -> OK.
myproc -numA -2 -> Error: numeric value < 0 not allowed

The "-dfilter {...}" pair can appear before or after other pairs following "$args". The set of type designator and associated list must be kept together but the order of sets doesn't matter.

Benchmarks

Some benchmark results are listed in the table. Overall, the extension is ~ 7-9 times faster than the pure Tcl reference implementation (test/CmdArgs.tcl). Also, the chkArgs variant is fastest, chkArgsvCi slowest. The difference is ~25%.

Table. Benchmark times (μsec) for each implementation. tC/tc = time(CmdArgs)/time(cmdargs) Tests running on a laptop with I7, 16GB, Windows 11, WSL2/Ubuntu 22.04.

Tests performed using "tmtest" (tmtest -batch 7), in "test/cmdargs-test.tcl" (for cmdargs::chkArgs*) and "test/cmdargs-test-2.tcl" (for CmdArgs::chkArgs*)

Compiling

While the supplied precompiled shared libraries should work on Linux and Windows systems, there's always a chance of problems.

The archive contains the full source code and a build system. The supplied Makefiles will probably need to be regenerated to suit individual operating systems and environments.

The included build system is simple but can be elaborated to adapt to situations as necessary. The bin directory contains two executables, config and pkgNdx (or config.exe, pkgNdx.exe). These two files can be copied to a directory on the PATH. These may also be run from the bin directory as well. On unix-like systems check that bin/config,pkgNdx have exec bits "on". If not, use chmod a+x config pkgNdx to fix it.

The primary configuration file is config.def found in each Tcl8/9/unix/win directory. (The tools directory also has config.def in its unix and win subdirectories. The provided bin/ files will probably not need to be regenerated.)

In config.def change the "tcldir" default path near the top of the file. Change the installto list to reflect your system's Tcl package directories.

set Defaults {
    tcldir /usr/local/opt/tcl87s <- (replace with location on YOUR system...)
    installto {directories to cp <cmdargs1.0> to}
    <other settings...>
}

Now run

../../bin/config

the Makefile will be regenerated. Then run make install and it should be good to go.

Regenerating config and pkgNdx

As already noted downloaded archives contain pre-compiled binaries for Windows and Linux. Binaries include working config and pkgNdx (or config.exe and pkgNdx.exe).

Regenerating these programs isn't too difficult. Necessary tools are located in the tools directory. Therein should be the following directories/files:

> config8.vfs
    > tcl_library
    config.tcl
    main.tcl
> config9.vfs
    > tcl_library
    config.tcl
    main.tcl
> unix
    config.def
    Makefile
> win
    config.def
    Makefile
config.def
Makefile
mkconfig8.tcl
mkconfig9.tcl
mkpkgndx.c

Configuration

In the tools directory is found a unique configuration system designed around creating Tcl extensions and projects. (But not limited to Tcl software.)

It has two components, config.tcl and config.def. Config.tcl takes configuration info in config.def and "translates" into a standard Gnu makefile. Note that these makefiles are intended for processing by Gnu make. That should never be a problem on Linux systems. Under Windows it's highly recommended to use msys2 which provides a Unix-like environment for producing Windows binaries. For *BSD and other systems, use the appropriate package utility or ports collection to install gmake or equivalent.

Config.tcl can be used in two ways. One is using a suitable tclsh* to run config.tcl:

cd /path/to/target-directory-with-config.def
/path/to/tclsh(.exe) config.tcl</code>. 

This uses the target directory's config.def file to produce a Makefile. config.tcl is provided in the tools directory and can be copied as needed. However, it's not very convenient if used this way more than occasionally.

The other (arguably better) way is creating a config executable from one of the "mkconfig*.tcl" supplied. There are a few steps involved:

1. In the win/ or unix/ directory, run tclsh ../mkconfig(8/9).tcl. to create config8(.exe) or config9(.exe). But which tclsh versions should be used?
   • Creating a config executable relies on the excellent zipfs facility to produce it. Since zipfs is new to tcl8.7 and 9.0, it's necessary to have tclsh8.7 or 9.0 (tclsh87/90 on Windows). Furthermore, the idea is to create a standalone binary that works without having an installed Tcl. That requires a statically compiled tclsh8.7/9.0. (Compiling Tcl from source is pretty straightforward even on Windows.)
   • The *.vfs directories include tcl_library/ whose contents are necessary for initializing Tcl. Bear in mind that the tcl_library/ version must match the tclsh version. If needed, copy the library/ directory from the Tcl source tree to config8.vfs or config9.vfs. Rename library/ to tcl_library/.
   • Use mkconfig8.tcl with Tcl8, and mkconfig9.tcl with Tcl9.
2. Modify tools/unix-or-win/config.def to suit your system. Most likely only one or two changes are needed. Set the defaults: for "tcldir" to the location where Tcl is installed, and "installdest" to a location on the exec path where config and pkgNdx will reside. (See Using config.def.)
3. Run config(.exe) in tools/unix-or-win directory to generate a new Makefile.
4. Run (g)make to generate config8/9 and pkgNdx. Use ?sudo? make install to install. make install-bin to copy these binaries to the CmdArgs-ext/bin directory. Use make clean to remove compiled files from unix or win.
5. Note that either tclsh8.7 or tclsh9.0 can be used to create config/pkgNdx. The resulting files work exactly the same way, so it doesn't matter whether 8.7 or 9 was used to generate these binaries.

Using config.def

Config.def is designed as a per directory configuration resource. It consists of several sections where users enter the directives that config (config.tcl) needs to have in order to generate a proper Makefile for the application.

— Config.def sections:

Sections are Tcl lists with varying structures.

• Vars:
  • List of variable names that are referred to in later sections. This is list of names only. They're instantiated as Tcl variables and assigned a value specified in Defaults or via the config command-line.
• Defaults:
  • The Defaults list consists of pairs where the first element is a name included in the Vars list. The second element is the default value for the named variable. A different value can be assigned on the command-line.
  • If a variable declared in Vars is not listed in Defaults, then its value must be provided on the command-line, and if not, an error is generated.
  • In any case, the Vars name becomes the name of a global Tcl variable with a value determined by Defaults or command-line specification.
• helptext:
  • Like Defaults, the "helptext" list contains paired elements. The first is a variable declared in Vars, the second is a brief description of the variable or its purpose. This shows up when config -help is run or an error occurs.
  • Declared variables (per Vars list) without an entry in Defaults are considered mandatory command-line settings. These settings should be described in the helptext list.
• helptext_optional:
  • Same structure as "helptext", except used for variables with Defaults. Providing a value on the command-line is considered optional.
• DEFINES:
  • A list composed of element pairs. Each pair is a Makefile variable definition.
  • First element is the Makefile variable name which can be a built-in, declared in Vars or an arbitrary string.
  • The second element is a valid Tcl expression. This expression is evaluated and the result becomes the value of the Makefile variable. The expression can be a constant (string or number), a simple variable deference, an elaborate conditional or other Tcl expression.
    • Makefile variable names are transformed to upper case followed by an " = " sequence.
    • The second element is output following " = " to the end of the line. Dereferenced Makefile variables appearing in the expression must have their "$" escaped, e.g., $(CFLAGS) or else will cause errors ("can't read "varname": no such variable").
    • Therefore, the definition "tclver $tclver", becomes "TCLVER = 9.0" in the Makefile. Variable definition statements are written to the Makefile as a single line.
    • Note that declared variables (per Vars) can be used in DEFINES right-hand side expressions without global namespace qualifiers. So even though tclver is a global variable it can be used as $tclver. (Though $::tclver isn't wrong, just a bit noisier in this context.)
• RULES:
  • The last section of config.def
  • RULES is a list with each rule consisting of 3 elements.
    • Element 1: the TARGET. ("config$(EXT)")
    • Element 2: the DEPENDENCIES. ("$(OBJS)")
    • Element 3: the RECIPE. ("$(CC) -o $@ ...")
      • Multiple steps can be provided. Enclose the list of steps in braces as well as the steps,
        {
           { step 1... }
           { step 2... }
        }.
      • Recipe steps can contain Tcl commands.
      • Because Makefile variables of the form "$(...)" or "$@" are very commonly used in recipes, Tcl variables are NOT expanded unless within a command like [expr {...}].

— Built-in or predefined variables:

Certain variables are predefined by config.tcl.

• cc: defined as "gcc". Becomes "CC = gcc" or as defined by the user.
• exe: set to .exe on Windows systems, "" otherwise. "EXE = .exe"
• libext: "dll" on Windows, "so" on unix.

Config.tcl also provides a command stdrecp that outputs a standard recipe for compiling a C object file: $(CC) $(CFLAGS) -c $< -o $@

As a special convience for Tcl programmers, config.tcl accepts input of a variable "tcldir". This built-in need only be listed in Defaults with the location of the desired Tcl installation directory. With no further work required of the programmer, config.tcl will process this to provide a number of useful variables. These include:

• Tcl include and lib directories (tclinc, tcllib),
• Tcl stub name (tclstub), and
• Path to the tclsh executable file (tclsh).

These variables are used in DEFINES like other built-ins. IOW the pair "tclinc $tclinc" expands to TCLINC = path-to-installed-tcl/include.

— The config.tcl command-line:

The config command-line is the simple and preferred way to configure and generate a Makefile. Currently, required config parameters include tcldir, tclsrc, and installto.

The syntax is simple: -variableName=value. This is a valid invocation:

config -tcldir=/usr/opt/tcl9

The variable name is preceded by one or two dashes and must a built-in like "cc", "exe", "libext" or declared in the config.def "Vars" list. The variable gets its value from the Defaults list, or overridden by a command-line value.

If a variable declared in "Vars" does not have a Default value, then the command-line must be used to assign a value. Empty value is signified by absent right-hand side:

config -myvar=

Extending config.tcl or config.def

Since config.def is an ordinary Tcl file additional functionality can be included via sourcing new Tcl files into a local config.def. Another approach is modifying config.tcl to enhance capabilities. Programmers are invited to contribute their enhancements to the project (via the project website, fossil, etc.). If the config/config.def configuration is successfully used for non-Tcl projects that will be of particular interest in shaping its future development.

J Altfas
Sun 9 Apr 2023 00:23:01 -07:00
cmdargs-doc.md:v1.0.9B