As you can see from the formatting changes in this website, I've been a but busy the last few days renovating the guts of the Httpd Module for Tcllib as well as the Toadhttpd Webserver that is based on it.

The changes stem from the ideas I had put forward in Adding a DOM to tcllib's HTTPD module. But, of course, all great ideas run up against the brick wall of reality. In the process of introducing a DOM based content generator, I still had to account for the vast amount of content on my existing websites that are based on the old stream model, or which are encoded in Markdown. After trial, error, denial, anger, bargaining, depression, and acceptance I finally hammered out a model by which DOM based documents can be the norm while still grandfathering in all of the other bizzare templating schemes my website (or any other website that is using me code) depend on.

There are still calls for streaming buffers of content, and for now that is still the default state of affairs for an httpd::reply. However this streaming concept is relegated to a mixin that is applied if nobody tells us any better.

The Dispatch method is now really only the standard. Everything that Dispatch calls is assumed to be internal to the content generating mixin. In fact, the implementation of Dispatch is assumed to be internal to the content generating mixin.

Another change was that dispatch helper objects were added as a concept. Instead of plugins monkey patching the server's Dispatch method, each helper now registers itself, complete with where it would like to be on the list of objects to be consulted. If any dispatch helper along the way decides "Hey I want to answer that", they simply need to return a non-empty dict. The server consults each dispatch helper via the public method claim

To not completely make a mess of things, the server itself inherits a reasonably complex set of dispatcher behaviors. If no dispatch object claims a query, the server falls back to it's own implementation of claim which handles consulting registered URLs, and handling static content under doc_root.

The server has also inherited most of the better ideas from toadhttpd as far as dispatching from an sqlite table, as well as managing a realtime black hole and setting up a honeypot to detect evildoers and put them into the black hole. (Things like asking for php, or cgi-bin, or known management URLs for web servers with vulnerabilities.) The system also now flags empty agent strings, or those from notoriously trolly web spiders.

Another feature of the sqlite powered server is that it can now support the same index delivered documents as toadhttpd. Naturally, how data gets into that index is still a matter that is left to the user. (I personally use the indexer in toadhttpd's facade plugin.)

Some random and I have to explain them changes occurred in httpd::reply.file. The old implementation was part dispatcher, part content deliverer, and full time mess. In the new system, httpd::reply.file now does one and only one job: ship a static file. The complexity of sorting out the headers and detecting paths and all of that is now a fancy new method in the dispatcher.

The dispatcher can also now decide that a file is not to be delivered verbatim, and instead should be processed as a template. To help website developers maintain their sanity, httpd::reply.cunieform can now deliver clay based content, subst based templates, or markdown. The dispatcher determines which, and informs the responder with the new CONTENT block of the reply. This CONTENT block can also contain a field called body which is often the contents of the file sucked off the disk. The reply.cunieform implementation still wraps the rest of the site's template, the final "content" is just inserted into a pre-agreed location on that template.

All of this new complexity comes in at about 900 more lines of code. Though, there is now a dependency on two new modules: clay-db and cuneiform. Toadhttpd itself, however, is getting much shorter. Those 900 lines httpd gained were largely copied and pasted from toadhttpd.