]

The Httpd Module and Toadhttpd

Sean Woods

Test & Evaluation Solutions, LLC

(Press the right arrow on your keyboard to advance the slides or you can just watch Sean present it himself on YouTube)

In this presentation we will cover:
  • Why we need a reference web server
  • The Httpd Module in Tcllib
  • The Toadhttpd web server project
On an interesting note, this presentation is running from an instance of Httpd

Folks at home can tune into:

http://www.etoyoc.com/yoda/papers/tcl2018/Httpd_Slides.html

and replay this presentation on their own.
It's actually a cute bit of Javascript I stole picked up from Cyan Ogilvie.
Error serving /yoda/papers/tcl2018.Httpd_slides.tml:

The server encountered an internal server error:

Must crash here

 while executing
"error "Must Crash here""
    invoked from within
"subst $tmltxt"
    (class "::httpd::content.file" method "content" line 42)
    invoked from within
"my content"

This is just joke, go onto the next slide

Why we need a reference web server

The user interface of choice in this day an age is no longer Tk.

Why we need a reference web server

The user interface of choice in this day an age is no longer Tk:
  • Tcl needs to explore new ways of generating human machine interfaces.
  • Newer platforms are stressing browser based interactions.
  • Especially on mobile platforms.
But hypnotoad, don't we already have a host of web servers for Tcl already?

Why we need a REFERENCE web server

The Tcl community has already developed a number of web servers.

The problem is:

  • The serious implementations are too massive, complex, and/or needy to be embedded in a desktop application.
  • The embeddable implementations are basically toys.
  • The implementations which support server side scripting, each go off an implement server side scripting in a completely different way.
  • Testing. Testing. Testing.

The Httpd Module

  • Started off life as version 4 of Tclhttpd.
  • Introduced Coroutines and TclOO to the core of the httpd server
  • Added bootstrap and jquery integration
  • Added a highly adaptable dispatch engine
  • Server was implemented as an OO object
  • Could be loaded in a package inside of normal Tcl software
Ended up being nothing like Tclhttpd.

Simple Dustmote Example:


package require httpd

::httpd::server create HTTPD port 8015 doc_root ~/htdocs
HTTPD start
QED
Layer 1 But Wait: There's MORE
Layer 1 Server Reply Coroutine

There are three players: The Server, the Coroutine, and the Reply Object

The players

Layer 1 Server Socket method connect

The process starts when a socket event invokes a call to the server's connect method.

The players

Layer 1 Server Socket method connect Coroutine

The connect method creates a coroutine which takes control of the socket, and uses the private Connect method of the server to bootstrap the rest of the reply process.

The players

Layer 1 Connect Server Socket Coroutine Connect dispatch

The coroutine reads the request line and the MIME headers, and passes that information to the Server's dispatch method

The players

Layer 1 Connect Server Socket Coroutine Connect dispatch Dict

The dispatch method returns a dict

The players

Layer 1 Connect Server Socket Coroutine Dict Reply

The coroutine then creates an instance of httpd::reply

The players

Layer 1 Connect Server Socket Coroutine Dict Reply dispatch content The coroutine passes the dict and the socket over to the Reply's dispatch method.
Layer 1 Connect Server Coroutine Socket Reply Mixins Mixin content final output
The reply object mixes in behaviors according to the instructions in the dict, and uses those mixed in behaviors to generate the reply.

The players

Layer 1 Server After the reply is returned, the socket is closed, the object is destroyed, and the coroutine terminates. Leaving only the server, to wait for the next socket open.

Simple Serverside Content Example

The Httpd module only provides a fallback hander for static server content from doc_root.

To generate server side content directly from a url requires modifying the Server object. The Dispatch_Local method is empty by default, and intended to be replaced by the application. If that method returns a non-empty dict, the server ends the dispatch process, and returns that reply to the coroutine, who will in turn pass that to the httpd::reply instance.

Simple Serverside Content Example


# Hijack the dispatcher method on the server
oo::objdefine HTTPD {
  method Dispatch_Local data {
    # Only match up requests for demo
    if {[dict get $data REQUEST_URI] eq "demo"} {
      set reply $data
      dict set reply mixin content ::mydemo
      return $reply
    }
  }
}

Simple Serverside Content Example

The default behavior for an httpd::reply is to invoke a method named content. It is intended that the content method be mixed in from some other class. The content method populates an internal value named reply_body. For easy of use, the httpd::reply class includes a puts method which will append to that buffer, and mimic the heady days of CGI where we used to spew data to stdout.

Simple Serverside Content Example



clay::define ::mydemo {
  method content {} {
    my puts [my html_header {Hello World!}]
    my puts {<h1>Demo page</h1>}
    my puts "Generated at [clock format [clock seconds]]"
    my puts [my html_footer]
    # NOTE METHOD DOES NOT RETURN A VALUE
  }
}

Simpler Serverside Content Example

Httpd has a plugin to automate the process of adding simple direct Urls:


HTTPD plugin dispatch ::httpd::plugin.dict_dispatch

HTTPD uri direct * demo {} {
  my puts [my html_header {Hello World!}]
  my puts {<h1>Demo page</h1>}
  my puts "Generated at [clock format [clock seconds]]"
  my puts [my html_footer]
  # NOTE METHOD DOES NOT RETURN A VALUE
}

Simple Serverside Content Example

GET/POST form information is available as a dict from the FormData method.


HTTPD uri direct * time {} {
  my puts [my html_header {Hello World!}]
  set formdat [my FormData]
  my puts {<h1>Demo page</h1>}
  if {[dict exists $formdat format]} {
    set format [dict get $formdat]
  } else {
    set format "%Y-%m-%d %H:%M:%S"
  }
  my puts "Generated at [clock format [clock seconds] -format $format]"
  my puts {<form action=/[my request get REQUEST_URI]>}
  my puts "<input name=format value=\"$format\">"
  my puts {<input type=submit></form>}
  my puts [my html_footer]
}

Simple Serverside Content Example

The raw mime data is also available for complex request handling. In fact, the entire dict that was built by the dispatch process is stored in the objects clay data structure.


HTTPD uri direct * rawmime {} {
  my puts [my html_header {Hello World!}]
  set mimetxt [my clay get mimetxt]
  my puts {<h1>Raw Mime</h1>}
  my puts "<pre>$mimetxt</pre>"
  my puts [my html_footer]
}

Simple Serverside Content Example

The headers for the reply can be controlled via the reply ensemble. CGI encoded data from the request is available through the request method.


HTTPD uri direct * texttime {} {
  my reply set Content-Type text/plain
  my puts "This URI was [my request get REQUEST_URI]"
  my puts "Generated at [clock format [clock seconds]]"
}

.tml files

The httpd module's file server mechanism also understands Tclhttpd's old .tml file templates.

> cat ~/htdocs/demotml.tml

[my html_header {Hello World}]

<h1>Demo page</h1>
Generated at [clock format [clock seconds]]

[my html_footer]

Note: the tml files are run inside of httpd::reply objects, so your templates can (but do not have to) invoke the methods of that object.

Rest API Example



HTTPD uri direct * restapi {} {
  # Have the reply do its own dispatching
  set URI [split [my request get REQUEST_URI] /]
  lassign [split $URI] base method uuid
  set class [find_class $method]
  if {$class eq {}} {
    # Errors will turned into the appropriate HTTP reply
    tailcall my error 404 {Not Found} {}
  }
  # Mix in a new behavior
  my clay mixinmap rest $class
  my reply set Content-Type application/json
  # Pass off control to a method brought in by the mixin
  tailcall my RestContent
}

Toadhttpd

The httpd module is very basic. It is intended to be a foundation to build on, not the entire building.

To run a real web facing server requires a lot of facilities that are beyond the scope of what an embedded web server would provide. At least a web server that is small enough for inclusion in tcllib.

A parallel project, Toadhttpd extends the httpd module in tcllib into fully fledged web server.

Toadhttpd

Fossil Repo: http://fossil.etoyoc.com/fossil/toadhttpd
  • Hardened against botnet attacks
  • Plug-in architecture
  • Bootstrap and jQuery Integration
  • Sqlite Based:
    • Dispatch Engine
    • Cacheing Mechanism
    • Logging
    • IP Blocking
  • SCGI Proxy Support
  • Built-in Fossil Mirroring Support

Q&A

And if you want to play with httpd, (and you have a recent Tcllib installed)

just:



package require httpd