Thinairarts icon (fan-shaped) Go to Main Page
Thinairarts logo (like a Scheme program...)

Open Source Software: Tcl Web Gen

Tcl Web Gen is a small set of tools for developing websites as simple or complex as requirements demand. Like Tcl itself, TWG tools are a foundation meant to be built on, so all kinds of websites are possible. To date TWG has achieved the goal of producing fully static sites, but can also generate pages dynamically from the same code. Dynamic content is easily added, for example via AJAX calls, it's simply another bit of code added to the toolset.

Illustrating what TWG can do, this entire website is produced with Tcl utilizing TWG tools. Of course we all know building a site isn't a one-time operation, that is, generating all or part of site pages has to be done frequently in order to grow the site and maintain the content.

TWG is easily scripted to render all pages from a single proc call. Once the up front work in design and initial implementation is completed and structure is established it becomes trivial to regenerate all or part of a site whenever necessary.

Tcl is a wonderful language to use for this purpose, given its malleability and stability. While time-critical performance isn't a huge factor in site generation, nonetheless operations are surprisingly brisk, for example, it takes less than one second to generate all 57 pages of this site. That's more than satisfactory for this kind of application.

New! Upcoming MAJOR changes!

FYI. In the coming weeks we will be rolling out a new completely revised version of this software. The new version is written using robust TclOO classes and objects. As a result the code is simpler and easier to apply to problem domains.

In fact this site (approximately 75 pages) is being rewritten using the upcoming version. We anticipate it will be more reliable and maintainable, website qualities that are very desirable to build in whenever possible.

A result of this exciting development is that all that's written below will be changing. Syntax will be the most obviously different aspect of the changeover. When you come back in January 2022 this page will be populated with detailed information about our new approach (and updated repos).

Download TWG

You can access the latest version of TWG using our TWG fossil repository

Or if you prefer, go directly to the repo download page: TWG Downloads

It's easy to install fossil and clone the repo, or just get everything in tar.gz or zip format—it really couldn't be simpler!

HTML Templates

NOTE: This section is going to be completely revised in the next weeks.

TWG is based on parsing an HTML template expressed as a Tcl list. This approach was very much inspired by similar applications using Lisp/Scheme list templates. However putting this idea into practice proved more difficult to accomplish in Tcl than Scheme, largely because Tcl strings and lists are not distinct types. Nonethless despite the differences our Tcl implementation was constructed to work effectively and retains much of the elegance of its inherent Lisp-like nature.

The basic outline looks like this:

{html
  {head ...}
  {body ...}}

HTML tags are the first element of the Tcl list. So adding a headline to the body:

class template
{body
  {h3 new {My Wonderful Site}}}

Note that there are no closing tags. When the list is parsed the closing tag, if it's required, is provided automatically. This makes the markup more clear.

The h3 tag is followed by ! {! ... }. The "!" is obviously not an HTML tag. In fact the "tag position" in templates is not restricted to official HTML tags but can be nearly anything. Here "!" has a special meaning, that the rest of the list is taken literally and inserted into the resulting HTML as is. (There are cases where additional quoting would be needed as Tcl rules require.)

Another special "tag" is @. Placed right after an HTML tag, {@ ...} signifies an attribute list. Attributes are given as pairs, where the attribute name is followed by its value. For example,

class template
{div new -x {class myclass id topdiv} ...}

The template structure is made useful by inserting replaceable content in the form of variables or procedure calls. This is easily accomplished by using the Tcl command.

set ttl "The Great Site"
proc mk-top-headline {ttl} {
  return "Introducing: <em\>$ttl</em>"
}
[subst {{html
  {head
    {title {! $ttl}}}
  {body {@ onload pageinit()}
    {h3 new {[mk-top-headline $ttl]}} ... }}]

When rendered (described below) this example produces:

<!DOCTYPE html>
<html>
  <head>
    <title>
      The Great Site
    </title>
  </head>
  <body onload="pageinit()">
    <h3>
      Introducing: <em\>The Great Site</em>
    </h3>
    ...
  </body>
</html>

The Toolkit

  • httm.tcl

    The procedure httm::parse_stream is the heart and soul of TWG. It takes one argument, a string in the form of the html template.

    httm::parse_stream recognizes the following "tag" forms:

    • Any html tag. meta, input, hr, link, img, area don't get end tags, the others will.
    • An HTML tag with "-" appended will have no spaces or newline added after the tag. It's also necessary to use !- or !!!- to make sure no spaces are inserted after the content.
    • @. The @ symbol specifies an attribute list, containing key value pairs as in {@ key0 value0 key1 value1 ...}. There must be a value for every key. If a key doesn't take a value, use \"\".
    • ! or !!!. The remainder of the list is taken as literal text and will be written to output as is. The difference is described below. Append - to eliminate spaces around content.

      Input can also use the form {"..."}. Avoid spaces after the opening brace or before the closing, because Tcl will consider it to mean the string and braces are separate lists.
    • !-. No spaces or newline will be appended before the end tag is applied. This eliminates having an extra space between adjacent content in the final page.
    • !!!. Used to prevent replacing double quote characters and curly braces with HTML entities. Note that ! is automatically treated as !!! in script and style tags so writing ! is sufficient. Once in a while the !!! "tag" may be useful elsewhere.

    Text punctuation rules:

    • Double quote " and curly brace { } are problematic because these characters signify list structure to Tcl. To deal with this, stream->html will convert these symbols to HTML entities when encountered in text within {! ...} or {"..."} blocks.
    • Sometimes the conversion is troublesome. In script or style sections, the conversion is supressed. Elsewhere using {!!! ...} suppresses conversion leaving the text as is.
    • The characters \[ and \] are special to Tcl and must be backslash-quoted to avoid errors re: enclosed text not being a valid procedure. But don't quote brackets of inserted Tcl code in {script {! ...}} or {style {! ...}} sections (brackets are syntax in those locations and need to remain "plain").
    • Other characters that require backslash quoting are $ and the backslash itself, \\.
    • The characters < and > have special meaning in HTML. Unless it's indended to insert a tag, they must be written as &lt; and &gt; respectively. It's a bit of extra typing, but very usefully allows HTML markup to be included in template text. It's expected authors are familiar with markup like <b>, <em>, <code>, <mark> and other phrase tags. It's far easier to use markup vs. {b ...} or {em ...} for just a couple of words.

    Addition procedures in httm.tcl include the following:

    • car, caar, cdr, cadr, cddr. These have the same meaning as in Lisp. Used internally in stream->html but intended to be useful elsewhere as well.
    • stream->html, stream->htmlpart. stream->html takes a stream as input and outputs a complete page to standard out. stream->htmlpart renders a partial stream, a section, div, etc., as a proper list.
    • stream->file, stream->filepart. Same as "html" counterpart except output is written to the file specified on input.
  • bool.tcl

    bool.tcl contains several procedures also inspired by Lisp. and, or can be handy in expressions as they return values, unlike Tcl if, etc. nullstr* and nullstr? are well-described in bool.tcl.

  • cmd-args.tcl

    Provides a command-line-like interface for Tcl procedures. The procArgs::chkargs proc takes as input the args parameter to a procedure and 4 dicts that define the CL arguments permitted and default values. Details are described in the cmd-args.tcl file.

    While not equivalent to command-line args in native Tcl commands, nonetheless the cmd-args utility is very useful and widely used in TWG. It's performance is also acceptable, clocking in at 30-40 microsec/iteration depending on number of CL arguments to process. In the context of TWG it's plenty fast enough.

  • component.tcl

    Components are bundled sets of styles, HTML markup, and Javascript which are bound by unique alphanumeric identifiers so the settings for one instance of the component don't affect others. This is accomplished by storing randomly created selector identifiers in a Tcl dictionary so that they can be applied consistently.

    There are 4 procs in the Component namespace. classdict sets up a dict containing selectors as keys and assigned random identifiers in the form of "s012345".

    selector takes one argument, the CSS class name to amend. Thus, \[Component::selector myclass\] returns myclass\[s234567\] to use in the component <style> section.

    attr is used in \{@ ...\} lists to insert the identifier corresponding to the style selector. \[Component::attr myclass\] outputs s234567 "" class myclass. Note that the Component procedures take care to distinguish "#" from "." selectors and to use the correct notation, inserting a "." in selectors and replacing the "." with a space in attribute lists.

    For javascript, there's jsfunc and ident (which work exactly the same way) to add unique identifiers to javascript functions and variables when these need to be part of a component.

  • menu.tcl, panelNav.tcl and slider1.tcl

    These Tcl files are useful components (in an extended sense) and can be customized to suit a variety of purposes and sites. They are not themselves "fundamental" in the same way as those described above. These "derived" utilities are somewhat more complex, especially slider1.tcl, but they show how the primary tools can be used to build desired functionality.

    menu.tcl illustrates individualizing styles through Tcl parameterization, though doesn't make full use of Component capabilites. Obviously it's better to use top level CSS whenver possible since styles then are much easier to change vs. having to reconfigure the Tcl code and regenerate all the site pages to propagate the new settings.

    While menu.tcl is still part of the bundle it's been supplanted by panelNav.tcl for most purposes in projects using TWG.

    panelNav.tcl takes the latter approach, keeping styles in a global level CSS file. The logic is that a page is unlikely to have more than one panelNav, but of course that's fully possible. In that case the styles can be bundled with the nav, in fact that's about the only way it could happen.

    slider1.tcl makes full use of componentization. FWIW slider1 is an elaborate example that could be somewhat hard to follow. The basic idea is moving images in and out of view in a defined order. A tricky part is having the image move the correct distance and direction. Almost all of it is accomplished in CSS, javascript is invoked only to automatically rotate the "slides" sequentially at a prescribed pace.

    It's mainly a matter of selecting the right geometry and applying it consistently. The CSS "trick" is using checkboxes and labels to accomplish the alternation of putting slides in view. When the labels (the tabs at the top) are "checked" it triggers a set of conditions that put the slide on display, but when unchecked the slide resumes its default, out of view placement.

  • mod-page-common.tcl

    This file contains several procedures for assembling a complete HTML page. The idea is constructing parts of the structure then combining the pieces into the final output. It has very recently been completely revised. Now the generating procedures are entirely generic, contain no code specific to any site!

    The "high level" procedure is common::HTML which takes CL args for lists of CSS and JS files, and template for the <body> of the page. It's output is a complete HTML document as shown above. Setting common:: variables permits deep customization of pages. In particular setting main_body_proc to a procedure that accepts bodyspec arguments from common::HTML allows maximum flexibility for website developers.

    common::HTML calls common::htmlFromParts which in turn calls bodyParts and headParts which insert boiler-plate code into the respective HTML sections. This delegation of the processing gives greater flexibility in organizing a site allowing developers alternate ways to assemble the pieces.

    Normally mod-page-common.tcl is used in tandem with an adapter file to set and override common:: namespace variables. (This can be in the common:: namespace or another as the user prefers.) The adapter file can also define procedures including a main-body-proc that generates the template for a particular website. Create a file that sources mod-page-common.tcl and the adapter file in that order and save as "page-common.tcl".

    Utility procedures include filesSortByMtime and filesSortByName—collect a file list from directories sorted in ascending or descending order. Tcl programmers know the common namespace can be modified or extended to suit the needs of projects.

  • Web server (TWG-serv.tcl, et. al.)

    The bundle also includes a very useful web server which consists of several files, the main one is TWG-serv.tcl. Running the server is simple, open a shell in the archive directory and run "tclsh TWG-serv.tcl -port 8080" or whatever port you wish to use. In your browser type "localhost:8080/software.html" and you should get this page. The server can produce pages statically or dynamically. See "serv-pages.tcl" for examples of how to set up dynamic page generation.

  • Generating website files.

    There are many programs available to expedite batch processing files, make is the classic example. However for many static sites such management tools may be overkill. pages.tcl illustrates a simple approach to setting up a proc that (re)generates a whole site. Obviously with elaborate sites a more sophisticated method of update management is probably called for, and in those cases, make or other tools would be superior.

Comments