TIP 308: Tcl Database Connectivity (TDBC)

Login
Home Timeline Branches Tags Tickets Help
Bounty program for improvements to Tcl and certain Tcl packages. 
Author:         Kevin B. Kenny <kennykb@acm.org>
Author:         Artur Trzewik <mail@xdobry.de>
Author:         Andreas Leitgeb <avl@logic.at>
Author:		Donal K. Fellows <donal.k.fellows@manchester.ac.uk>
State:          Final
Type:           Informative
Vote:           Done
Created:        15-Nov-2007
Post-History:   
Obsoleted-By:	350

Abstract

This TIP defines a common database access interface for Tcl scripts.

Corrections

There are corrections to this TIP in [350] as well. Readers of this document should see that one too.

Introduction

There has been a fair amount of discussion, that flares and dies back, regarding the need for a "Tcl database connectivity layer" in the Tcl core. This document specifies what this discussion means. At its present stage of development, it is to be considered very much a draft; discussion is actively solicited.

Parties who are interested in a detailed background of this TIP may a more extensive discussion of motivations and objectives in the author's posting to comp.lang.tcl and the tcl-core newsgroup, obtainable from http://groups.google.com/group/comp.lang.tcl/msg/9351d1b2a59ee2ca or http://aspn.activestate.com/ASPN/Mail/Message/tcl-core/3581757 .

What is Tcl's Database Connectivity Layer?

If we look at other database connectivity layers such as ODBC/DAO, JDBC, Perl's DBD/DBI, we find that there really isn't very much, if anything, inside them. Rather than being a body of code, they consist primarily of specifications of the interfaces to which the author of a database connectivity module must conform. The real work of connecting to the databases happens inside the connectivity modules, which are generally speaking under the control of the database teams. In terms of practical politics, there isn't really any other way to do it; the Tcl maintainers are highly unlikely to want to take on the job of connecting to arbitrary database API's.

In other languages, such as C++ and Java, it is often necessary to have interface definitions that are understood by a compiler in order to get the "pluggability" of arbitrary database connectivity. In Tcl, however, an "interface" is best understood as an ensemble implementing a predetermined set of commands. There is no counterpart to a Java or C++ interface definition, nor does there need to be. For this reason, the work product of a "Tcl database connectivity" development effort is likely (at least at the first stage) to consist primarily of a specification document, perhaps with reference implementations for one or a few popular databases. To be considered "in the core", the specification should be included with the Tcl documentation, and be under control of the TIP process. The database implementations should be considered "extensions," and have their own configuration management. This statement doesn't say that we can't choose from among them a set that we will package with releases of the Tcl core. In fact, I hope that this effort will be one driver for the TCT to sort out the management of "bundled extensions."

Mechanics of This Document

I write this document in "standards committee prose". (While turgid, it at least is often well-understood; I offer no further defence.) In particular:

  • the word "MAY" is construed as allowing a given behaviour but imposing no requirement other than that clients be prepared for it;

  • the word "MUST" (and conversely "MUST NOT") is construed as requiring a given behaviour; implementations that fail one or more requirements given by "must" are non-compliant;

  • the word "SHOULD" (and conversely "SHOULD NOT") indicates that a given behaviour is expected of an implementation unless there is a compelling reason not to include it; while not formally non-compliant, implementations that fail one or more requirements given by "SHOULD" can be understood to have issues with respect to "quality of implementation."

  • the future of determination ("SHALL" or "WILL" according to the usual customs of formal written English) is construed as a promise to which the Tcl Core or the Tcl Core Team, as appropriate, shall adhere. It describes requirements of the Tcl Core, rather than of database connection modules.

  • the term, "integer value" refers to any string acceptable to Tcl_GetBignumFromObj; the term "native integer value" refers to a value acceptable to Tcl_GetIntFromObj, and hence to a value that can be represented by a C int on the target machine.

  • the term, "boolean value" refers to any string acceptable to Tcl_GetBooleanFromObj and hence includes at least '1', '0', 'on', 'off', 'yes', 'no', 'true', and 'false'.

  • the term "ensemble" refers to a Tcl command that accepts subcommands. It does not imply that any given command is implemented using the namespace ensemble mechanism.

Specification

Connecting to a Database

Obviously the first thing that any connectivity layer has to offer is the ability to select a database. The way databases are named is quite specific to the database manager, as is the way access is negotiated (credentials such as user name and password may be required, session keys may be negotiated for privacy and authentication, and so on). All of this machinery is formally out of scope for this specification. Similarly, the machinery of database administration (at least at the level of creating/deleting entire databases, managing the physical layer, and authorizing clients) is presumed to be already taken care of. We need merely specify that a connectivity layer must provide at least one command that accepts arguments describing the desired connection and returns a database handle - defined to be an ensemble through which interactions with the given database instance will take place. Here, database instance means the database, or databases, that the given handle can access; rather a circular definition. In many SQL systems, it is possible for a single connection to access several "databases" managed by SQL CREATE DATABASE statments, or several "tablespaces" or similar constructs. We presume that database module implementors will know what is appropriate for their systems, and intentionally leave this particular matter somewhat vague.

Basic Mechanics of Database Interfaces

Database handles are Tcl ensembles, meaning that they are commands that support subcommands. Other ensembles, such as statement handles, are also defined in this specification. Any of the ensembles MAY support abbreviation of its subcommands according to the rules defined by Tcl_GetIndexFromObj; nevertherless, code that uses the database interface SHOULD spell out subcommands in full.

Many of the subcommands are expected to take options in Tcl's usual syntax of:

?-option ?value?? ?-option value?...

In all of the places where this syntax is expected, a database module MAY support abbreviation of options according to the rules of Tcl_GetIndexFromObj(); once again, code that uses the interface SHOULD spell out options in full.

All the database objects (connections, statements and result sets) are "duck typed" - that is, "If it walks like a duck and quacks like a duck, I would call it a duck. (James Whitcomb Riley)." In other words, the ensembles may be implemented using any available functionality as long as the result is that they use the interfaces described. Nevertheless, as a convenience to implementors, a set of base classes, called tdbc::connection, tdbc::statement, and tdbc::resultset, SHALL be provided using Tcl's native object orientation as described in [257]. Certain advantages will accrue to database implementors by using these base classes. In particular, the tdbc::* classes SHALL do all the bookkeeping needed to determine what statements and result sets are open, SHALL provide the internal iterators allrows and foreach, SHALL implement the transaction method on connections, and SHALL ensure that the close method on the objects functions the same as renaming the object to the null string.

Configuring a Database Handle

Once a handle is returned, there are a number of session-level attributes that may be controllable. Every database handle MUST provide a configure subcommand that takes the form:

dbHandle configure ?-option ?value?? ?-option value?...

This configuration process is analogous to configuring a Tk widget. If there are no arguments presented to configure, the return value MUST be a list of alternating options and values describing the configuration parameters currently in effect. If a single argument is presented, it MUST be the name of a configuration parameter, and the return value MUST be current value for that parameter. Finally, if more than one argument is presented, they MUST be a list of alternating parameter names and values. This last form is an order to set the given parameters to the given values.

The connectivity layer SHOULD implement the following parameters, and MAY implement others:

(Note: an earlier draft of this TIP specified a -autocommit option; this option has been removed because it is redundant with the transaction management primitives below.)

  • -encoding name

    Requests that the encoding to be used in database communication protocol be changed to the one given by name, which MAY be any name acceptable to the [encoding] command. A well-designed database interface SHOULD NOT require this command; however, some backends make it virtually inevitable that mid-stream changes of encodings will be required.

  • -isolation level

    Requests that transactions performed on the database be executed at the given isolation level. The acceptable values for level are:

    * readuncommitted

    Allows the transaction to read "dirty", that is, uncommitted data. This isolation level may compromise data integrity, does not guarantee that foreign keys or uniqueness constraints are satisfied, and in generall does not guarantee data consistency.

    * readcommitted

    Forbids the transaction from reading "dirty" data, but does not guarantee repeatable reads; if a transaction reads a row of a database at a given time, there is no guarantee that the same row will be available at a later time in the same transaction.

    * repeatableread

    Guarantees that any row of the database, once read, will have the same values for the life of a transaction. Still permits "phantom reads" (that is, newly-added rows appearing if a table is queried a second time).

    * serializable

    The most restrictive (and most expensive) level of transaction isolation. Any query to the database, if repeated, will return precisely the same results for the life of the transaction, exactly as if the transaction is the only user of the database.

    * readonly

    Behaves like serializable in that the only results visible to the transaction are those that were committed prior to the start of the transaction, but forbids the transaction from modifying the database.

    A database that does not implement one of these isolation levels SHOULD instead use the next more restrictive isolation level. If the given level of isolation cannot be obtained, the database interface MUST throw an error reporting the fact. The default isolation level SHOULD be at least readcommitted.

    A database interface MAY forbid changing the isolation level when a transaction is in progress.

  • -timeout ms

    Requests that operations requested on the database SHOULD time out after the given number of milliseconds, if such an option is supported by the underlying connectivity layer.

  • -readonly boolean

    Notifies that the application will, or will not, limit its activity to operations that do not modify the content of the database. This option MAY have the effect of adjusting the transaction isolation level.

The command that returns a database handle SHOULD also accept these options.

Transaction Isolation

A database handle MUST implement the three subcommands starttransaction, commit and rollback:

dbHandle starttransaction

Begins an atomic transaction on the database. If the underlying database does not implement atomic transactions or rollback, the starttransaction subcommand MUST throw an error reporting the fact.

If the underlying database does not implement nested transactions, a starttransaction command that is executed when there is a transaction already in progress (started, but neither committed nor rolled back) MUST result in an error.

dbHandle commit

Commits a transaction to the database, making the changes durable.

dbHandle rollback

Rolls back a transaction against the database, cancelling any changes made during the transaction.

Statements executed against the database when no transaction is in progress (before the first starttransaction or after all started transactions have been either committed or rolled back) SHOULD be auto-committed; that is, each such statement SHOULD be executed as if a starttransaction command preceded the statement and a commit command followed it (assuming that the statement succeeded; errors should result in rollback of course).

These commands are provided primarily to support the construction of higher-level operations. In particular, most simple transactions against a database can be handled using the transaction command:

dbHandle transaction script

Executes the given script with transaction isolation. In this command, the dbHandle argument is a handle to a database connection, and the script argument is a Tcl script to be evaluated in the calling scope. The script is treated as a single atomic database transaction. The starttransaction command is executed against the given database connection, and then the script is evaluated. If it completes successfully (TCL_OK), the transaction SHALL be committed to the database. If it fails, (TCL_ERROR), the transaction SHALL be rolled back and not visible to other users of the database. TCL_BREAK, TCL_CONTINUE and TCL_RETURN SHALL result in a commit and subsequently rethrow the same exception status outside the transaction. Exception status codes other than these five SHALL rollback the transaction and be rethrown.

(Note: Scripts inside a transaction command SHOULD avoid use of the return -code or return -level operations. If a script returns from a transaction, with any combination of return options, the transaction SHALL be committed.)

Just as with starttransaction, if a [dbHandle transaction] command is executed while another transaction is already in progress, it is requesting nested transaction semantics. A database handle to an engine that supports nested transactions MUST treat this case correctly; a database handle to an engine that does not support nested transactions (including one that does not support transactions at all) MUST throw an error.

The transaction subcommand SHALL be provided by the tdbc::connection base class; database interfaces that use the TclOO features and the TDBC base classes do not need to implement it.

Closing a Database Connection

A database handle MUST implement the command:

dbHandle close

This command MUST dismiss the connection to the database and is expected to clean up the system resources associated with it. If there is an uncommitted transaction, it SHOULD be rolled back. Any handles to other objects associated with the database SHOULD become invalid.

A database interface also SHOULD perform the same actions if a handle is deleted by means of the rename command. (Interfaces that are implemented in Tcl may be notified of this action by creating a deletion trace with trace add command.) It is recognized that command deletion traces present difficulties in situations like namespace and interpreter deletion; the close subcommand shall therefore be considered the preferred way to terminate connections.

A database interface SHOULD attempt to arrange, if possible, to rollback unfinished transactions and clean up on process exit. In particular, if the underlying database engine supports transactions, it SHOULD be considered an error to commit any work that remains uncommitted on process exit.

The close command SHALL be provided by the tdbc::connection base class; database interfaces that use the TDBC base classes do not need to implement it. The base class implementation destroys the object using my destroy. As a result, any statements obtained from the connection are also destroyed, since they are stored in a namespace that is subordinate to the connection's namespace. The destructor of the connection object is expected to close the underlying database connection and release any system resources associated with it.

Preparing Statements

A database handle must support the prepare command, which has the syntax:

dbHandle prepare SQL-code

The SQL-code argument is a SQL statement that is to be executed against the given database connection. This command does not execute the statement directly; rather, it prepares to execute the statement, possibly performing tasks such as code compilation and query optimisation.

The database interface MUST support substitutions in SQL-code. Each substitution request has the form :variableName. That is, each substitution request begins with a literal colon (:), followed by a letter or underscore, followed by zero or more letters, digits, or underscores. The database interface is responsible for translating from this syntax to whatever the underlying engine requires. Typical strings required in database interfaces are :name, :number, @name, @number, and ?.

The return value from the prepare command is a statement handle, discussed under "The statement interface" below.

Rationale. The choice of the colon deserves some discussion. It would surely be more natural for Tcl to use a literal dollar sign to introduce a variable name. This choice, however, seems unwise, since several databases (most notably Oracle) allow the use of table and view names that contain dollar signs. While it might be possible to continue to use these while allowing for variable substitution (for instance, by mandating that table or view names with dollar signs be enclosed in double quotes), it seems unnatural. The colon is syntax that is recognized by JDBC, ODBC, and Oracle's native API, and as such will be familiar to most SQL programmers and unlikely to collide with native syntax.

The requirement to support prepared statements is intended to guard against SQL insertion attacks. An interface to a database whose native API does not support prepared statements MUST simulate them. In particular, when the execute command is executed on a statement, substitution must be performed in a safe fashion with whatever magic quoting is required. In any case, magic quoting should be regarded as an infelicitous expedient and avoided if at all possible.

If a database interface uses the tdbc::connection base class, then a prepare method will be provided for it. If this method is not overridden, then the database interface MUST arrange that the constructor of the connection sets the instance variable, statementClass, to the fully qualified name of the command that constructs statements. The prepare method SHALL invoke that command with a call of the form:

statementClass create handle connectionHandle sql

where handle is the name of the new statement being created, connectionHandle is the handle to the connection creating it, and sql is the SQL statement being prepared.

Stored Procedure Calls

A second way to prepare statements is to prepare a stored procedure call. If a database interface supports stored procedures, it MUST support the preparecall command:

dbHandle preparecall call

call is a string that describes a call to a stored procedure. It takes the form:

?:varName =? procName ( ?:varName? ?, varName?... )

The result of the preparecall command is a statement handle. The statement handle may be used just as any other statement handle.

The preparecall method SHALL not be provided in the tdbc::connection base class; individual database interfaces are expected to do so. They MAY do so by rewriting the call to whatever syntax the native database requires, and delegating to the prepare method to prepare that, or they MAY instead prepare another ensemble. (See "The TDBC base classes" below for details of integrating this mechanism with the base classes.)

Quasi-Direct Execution

A database handle MUST support the following two calls:

dbHandle allrows ?-as lists|dicts? ?-columnsvariable varName? ?--? sql ?dictionary?

This command prepares the SQL statement given by the sql parameter, and immediately executes it. Variable substitutions inside the sql parameter are satisfied from the given dictionary, if one is supplied, and from variables in the caller's scope otherwise. The -as option determines the form of the result, and the -columnsvariable option provides an optional variable in which the names of the result columns will be stored. Upon termination of the command, whether successful or unsuccessful, the prepared statement is closed. The command returns a list of the rows returned by the affected statement. (If the affected statement does not yield a set of rows, the return value from the allrows command is an empty list.)

This command MUST function the same way as preparing the statement explicitly, executing the statement allrows call (see below) on the resulting statement handle, and then (irrespective of whether the operation succeeeded) destroying the statement handle.

dbHandle foreach?-as lists|dicts? ?-columnsvariable varName? ?--? sql ?dictionary? varName script

This command prepares the SQL statement given by the sql parameter, and immediately executes it. Variable substitutions inside the sql parameter are satisfied from the given dictionary, if one is supplied, and from variables in the caller's scope otherwise. The -as option determines the form of the result, and the -columnsvariable option provides an optional variable in which the names of the result columns will be stored. For each row returned by the given statement, the given varName is set to a list or dictionary containing the returned row, and the given script is executed in the caller's scope. Upon termination of the command, whether successful or unsuccessful, the prepared statement is closed.

This command MUST function the same way as preparing the statement explicitly and then executing the statement foreach call on the resulting statement handle.

Both of these commands SHALL be provided in the tdbc::connection base class.

Introspecting the Sets of Handles

A database handle MUST support the statements command:

dbHandle statements

This command MUST return a list of the statements that have been prepared by means of [dbHandle prepare] but not yet dismissed using [statementHandle close].

Likewise, a database handle MUST support the resultsets command:

dbHandle resultsets

This command MUST return a list of the result sets that have been returned (by executing statements, or by querying metadata) and have not yet been dismissed using [resultSetHandle close].

Both of these commands SHALL be provided in the tdbc::connection base class. Using the base class implementations imposes certain restrictions on derived classes. (See "The TDBC base classes" below for details of integrating this mechanism with the base classes.)

Querying Metadata

A database interface MUST provide a way of enumerating the tables in the database. The syntax for querying tables MUST be:

dbHandle tables ?matchPattern?

The optional argument matchPattern, if supplied, is a pattern against which the table names are to be matched. The database interface MUST recognize the SQL wildcards % and _ in the pattern.

A database interface MUST provide a way of enumerating the columns in a database table. The syntax for querying columns MUST be:

dbHandle columns tableName ?matchPattern?

The return value from the tables and columns commands MUST be a dictionary. The keys of the dictionary MUST be the names of the tables in the database, or respectively the columns in the given table.

The values stored in the dictionary returned from the tables command MUST be dictionaries. The keys and values of these dictionaries, nevertheless, are implementation-defined; only the keys are mandated in this specification.

The values stored in the dictionary returned from the columns command MUST themselves be dictionaries. These subdictionaries MUST include the keys, type, precision, scale, and nullable. The type value MUST be the data type of the column, and SHOULD be chosen from among the standard types bigint, binary, bit, char, date, decimal, double, float, integer, longvarbinary, longvarchar, numeric, real, time, timestamp, smallint, tinyint, varbinary, and _varchar'. The precision and scale values SHOULD give the precision and scale of the column, and the nullable value SHOULD give a boolean value that represents whether the given column can contain NULL values.

Other keys MAY be included in the subdictionaries returned from tables and columns, and SHALL be added to this document (as optional columns) on request from the implementors of database interfaces.

The Statement Interface

The statement handle returned from the prepare command on a database interface must itself be an ensemble. The following subcommands MUST be accepted:

statementHandle params

Requests a description of the names and expected data types of the parameters to the given statement. The return value from the params command MUST be a dictionary whose keys are the names of the parameters and whose values are themselves dictionaries. The keys of the subdictionaries MUST include name, type, precision, scale, and nullable. They are interpreted in the same way as those of the columns subcommand to a database interface (shown above). The subdictionaries also MUST include the key, direction, whose value identifies the direction of parameter transmission, and MUST be chosen from among in, out and inout.

statementHandle execute ?dictionary?

Executes a statement against a database. Any variable substitution present in the SQL that was provided when the statement was created MUST be performed at this time. The variable values MUST be obtained from the given dictionary, if one is supplied. If the dictionary does not contain a key equal to a variable name in the statement, a NULL value MUST be provided.

If the dictionary argument is omitted, the variable values MUST be obtained from the scope in which the execute command was evaluated. Any variable that is undefined in that scope must be replaced with a NULL value. An array variable provided to a substituent MUST result in an error. Read traces against the substituted variables SHOULD fire, in left-to-right order as they appeared in the SQL statement. The result of the execute command SHOULD be a result set, as defined under "The result set interface" below.

This method is provided by the tdbc::connection base class. In the base class, it works by creating an instance of the class whose name appears in the statementClass instance variable. See "The TDBC base classes" below for the details of how the derived classes should be implemented to avail themselves of this method.

statementHandle close

Announces that a statement is no longer required, and frees all system resources associated with it. The close command MAY invalidate any result sets that were obtained by the params and execute commands.

As with database connections, the database interface SHOULD also clean up if a statement handle is removed with [rename $statement {}]. Once again, it is recognized that the strange side effects of namespace and interpreter deletion may make this cleanup impossible in some interfaces, so close SHALL be considered the standard means of discarding statements.

The close command SHALL be provided in the tdbc::statement base class. Database interfaces that use the TDBC base classes do not need to implement it. The base class implementation destroys the object using my destroy. As a result, any result sets obtained from the statement are also destroyed, since they are stored in a namespace that is subordinate to the statement's namespace. The destructor of the statement object is expected to release any system resources associated with it.

Data Types of Parameters to Prepared Statements

The syntax described so far presumes that the database interface can determine the expected types of the variables that appear in a prepared statement, or at the very least can accept some sort of variant type and perform automatic type coercion. This requirement does not seem horribly onerous at first inspection, since SQLite allows for "everything is a string" parameters; ODBC offers parameter introspection via the SQLDescribeParam call; and JDBC offers it via the getParameterMetaData method of the PreparedStatement interface.

Nevertheless, a deeper examination discovers that in at least ODBC, a driver is allowed to fail to offer SQLDescribeParam. Inspection of the JDBC-ODBC bridge reveals that in this case, JDBC will return a ParameterMetaData object that throws a SQLException on any attempt to query specific data. The result is that, while the APIs to introspect parameter types are available, they may be unusable against a particular database engine. In these cases, a backup is needed.

For this reason, a database interface MUST support allowing the user to specify types of the parameters of a prepared statement. The syntax for doing so MUST be:

statementHandle paramtype paramName ?direction? type ?precision? ?scale?

Defines that the parameter identified by paramName in the given statement is to be of type type. The type MUST be chosen from among the names bigint, binary, bit, char, date, decimal, double, float, integer, longvarbinary, longvarchar, numeric, real, time, timestamp, smallint, tinyint, varbinary, and varchar.

(Rationale: These types appear to suffice for ODBC, and we can always come back and extend them later if needed.)

The precision of a parameter defines the number of characters or digits that it requires, and its scale defines the number of digits after the decimal point, if neeeded. A database interface MAY allow negative numbers for scale in contexts where they make sense. For example, a scale of -3, if allowed, SHOULD indicate that quantities in the given column are all multiples of 1000. The precision and scale are not required by all types.

A direction must be one of the words, in, out or inout. It specifies that the given parameter is an input to the statement, an output from the statement, or both. It is usually meaningful only in stored procedure calls. Default is in, unless the parameter appears on the left-hand side of an equal side in a stored procedure call, in which case the default is out.

~Examples

 $statement paramtype name varchar 40
 $statement paramtype balance in decimal 10 2
 $statement paramtype transactionDate timestamp

Implementors of database APIs SHOULD make every effort to do appropriate type introspection so that programmers can avoid needing to include explicit type information in their SQL statements.

Internal Iterators

A statement handle MUST support the following two calls:

statement allrows ?-as lists|dicts? ?-columnsvariable varName? ?--? ?dictionary?

This command executes the given statement. Variable substitutions inside the statement are satisfied from the given dictionary, if one is supplied, and from variables in the caller's scope otherwise. The -as option determines the form of the result, and the -columnsvariable option provides an optional variable in which the names of the result columns will be stored. Upon termination of the command, whether successful or unsuccessful, the prepared statement is closed. The command returns a list of the rows returned by the affected statement. (If the affected statement does not yield a set of rows, the return value from the allrows command is an empty list.)

This command MUST function the same way as executing the statement explicitly (with the given dictionary argument if one is supplied), executing the resultset allrows call (see below) on the resulting result set, and then (irrespective of whether the operation succeeeded) destroying the result set.

statement foreach?-as lists|dicts? ?-columnsvariable varName? ?--? ?dictionary? varName script

This command executes the given statement. Variable substitutions inside the statement are satisfied from the given dictionary, if one is supplied, and from variables in the caller's scope otherwise. The -as option determines the form of the result, and the -columnsvariable option provides an optional variable in which the names of the result columns will be stored. For each row in the result set, the given varName is set to a list or dictionary containing the returned row, and the given script is executed in the caller's scope. Upon termination of the command, whether successful or unsuccessful, the result set is closed.

This command MUST function the same way as executing the statement explicitly, executing the resultset foreach call on the resulting statement handle, and then (irrespective of whether the operation succeeded) closing the result set.

Both of these commands SHALL be provided in the tdbc::statement base class.

The Result Set Interface

Result sets represent the results of operations performed on the database. A preferred implementation for large result sets is that they be implemented as database cursors, so that it is possible to iterate over result sets that will not fit in memory. A result set MUST be an ensemble. The following subcommands MUST be accepted:

resultSetHandle rowcount

Determines the number of rows affected by a SQL statement such as INSERT, DELETE or UPDATE. This count MUST be returned as an integer. It should not be confused with the number of rows in the result set. A database interface need not provide any interface to determine the latter number (often, the only way to determine it is to read all the rows). For this reason, the rowcount command MAY return an empty string, or a non-positive number, for SELECT operations (and any other operations that do not modify rows of the database).

resultSetHandle columns

Determines the set of columns contained in the result set. The set of columns is returned simply as a list of column names, in the order in which they appear in the results.

resultSetHandle nextrow ?-as lists|dicts? ?--? variableName

(This interface SHALL be provided by the tdbc::resultset base class. The default implementation SHALL delegate to either the nextlist or nextdict methods, below.

Fetches a row of data from the result set and stores it in the given variable in the caller's context.

If -as dicts is specified (the default), the row MUST be represented as a dictionary suitable for use with the dict command. The keys in the dictionary SHALL be the column names, and the values SHALL be the values of the cells. If no rows remain, the **nextrow_ command MUST store an empty dictionary. If a cell in the row is NULL, the key MUST be omitted from the dictionary. A database interface MUST NOT use a special value of any kind to represent a NULL in a dictionary.

If -as lists is specified, the row MUST be represented as a list of values, in the order in which they appear in the query. (If the statement is a stored procedure call, the values comprise all the out or inout parameters.) If no rows remain, the nextrow command MUST store an empty list. If a cell in the row is NULL, an empty string MUST be stored as its value.

The return value of nextrow MUST be 1 if a row has been returned, and 0 if no rows remain in the result set.

In the result set, values of type bigint, bit, decimal, double, float, integer, numeric, real, smallint, and tinyint MUST receive their natural representation as decimal numbers. Ideally, they should be returned as "pure" numbers with their string representations generated only on demand. Values of type char, longvarchar and varchar MUST be returned as Tcl strings. A database interface implemented in C _MUST take care that all strings are well-formed UTF-8._ Values of type date and timestamp MUST be returned as a numeric count of seconds from the Tcl epoch; if necessary, this count may have a decimal point and an appropriate number of additional decimal places appended to it. Values of type time MUST be returned as a integer count of seconds since midnight, to which MAY be appended a decimal point and a fraction of a second. Values of type binary, longvarbinary and varbinary MUST be returned as Tcl byte arrays.

Rationale: Dictionaries and lists are both useful in representing the result set rows. Dictionaries allow for a ready distinction between NULL values in a database and any other string. With any scheme where values that can include NULLs can appear in Tcl objects, the problem arises that NULL must be distinguished from any other string, particularly including the empty string and the word "NULL". The lack of such a distinction has led to several ill-advised proposals, such as [185], for representing NULLs in Tcl. These alternatives founder on the principle of "everything is a string". The NULL value is not any string. Dictionaries also have the advantage that results can be addressed by name rather than by position. On the other hand, lists are convenient when formatting tabular results from ad hoc queries. The brevity of code that can be achieved with them is also attractive. For this reason, this TIP requires both formats to be made available.

resultSetHandle nextdict variableName

resultSetHandle nextlist variableName

These two calls are precisely equivalent to calls to the nextrow command with the -as dicts and -as lists option respectively. A database interface MUST provide both of these, and they are the fundamental means for retrieving rows from the result set.

resultSetHandle close

Dismisses a result set and releases any system resources associated with it.

As with statements and database connections, the database interface SHOULD also clean up if a resut set handle is removed with [rename $statement {}]. Once again, it is recognized that the strange side effects of namespace and interpreter deletion may make this cleanup impossible in some interfaces, so close SHALL be considered the standard means of discarding result sets.

The close command SHALL be provided by the tdbc::resultset base class. The base class implementation destroys the object using my destroy. The destructor of the result object is expected to release any system resources associated with it.

Internal Iterators

A result set handle MUST support the following two calls:

resultset allrows ?-as lists|dicts? ?-columnsvariable varName ?--?

This command executes the nextrow command repeatedly, producing a list of dictonaries or of lists (according to the value of the -as option). The allrowscommand returns the resulting list. Optionally, the names of the columns of the result set are also stored in the named variable given by the -columnsvariable option.

statement foreach?-as lists|dicts? ?-columnsvariable varName? ?--? varName script

This command optionally stores the names of the columns of the result set in the variable designated by the -columnsvariable option. It then executes the nextrow command repeatedly until all rows of the result set have been processed. The nextrow command receives the given varName and -as option, and stores the row in the named variable. For each row processed, the given script is executed in the caller's scope.

Both of these commands SHALL be provided in the tdbc::resultset base class.

The TDBC Base Classes

Most implementations of database drivers SHOULD, as mentioned before, use Tcl objects (as in [257]) that inherit from the tdbc::connection, tdbc::statement and tdbc::resultset classes. The foregoing discussion has described the user-visible methods that are provided by doing so (and must otherwise be implemented). This section is directed to the driver implementor, and discusses certain necessary housekeeping issues.

Database Connections

However a database connection object is constructed, its constructor will need to seize resources (such as opening a database connection to the underlying database system). If the bookkeeping done by the base classes is to work correctly, initialization of the tdbc::connection base class needs to happen before external resources are seized. In addition, if the prepare method is not overloaded (and the driver SHOULD NOT have to overload it), the name of the class that implemements the statement interface needs to be provided at this time. The recommended sequence for connection construction is:

  constructor args {
      next;                         # Initialize tdbc::connection
      my variable statementClass
      set statementClass ::whatever;# Tell tdbc::connection what
                                    # class must be instantiated by
                                    # the 'prepare' method
      my init {*}$args              # Perform implementation-specific
                                    # initialization
  }

Some database interfaces have a different API to stored procedures than to ordinary SQL statements. These databases may need a separate type of statement object from the one that implements ordinary statements. This object can be managed as a statement owned by the connection by using a prepareCall method that looks like:

  method prepareCall {call} {
      my variable statementSeq;   # Provided in the
                                  # tdbc::connection base class
      return [preparedStatementClass create \
                  Stmt::[incr statementSeq] [self] $call]
  }

In this call, preparedStatementClass is the name of the class that implements prepared statements. Its constructor is expected to accept two arguments: the handle to the database connection, and the prepared statement that was passed to prepareCall. Placing the resulting object inside the Stmt namespace under the current object (this namespace is created by the constructor of tdbc::connection) allows for its destruction to be sequenced correctly when the connection is destroyed.

The methods that a derived class from tdbc::connection MUST implement are prepareCall, begintransaction, commit, and rollback. In addition, system resources belonging to the connection itself MUST be cleaned up by a destructor or by a deletion callback at C level. (Statements and result sets MUST not be deleted then; the base classes take care of that.) See "Best practices for memory management" below for further discussion.

Statements

The class that implements a statement SHOULD normally inherit from the tdbc::statement base class. Its constructor accepts the connection handle and the SQL statement to prepare. The constructor is responsible for invoking the base class constructor with next, setting an instance variable resultSetClass to the name of the class that implements its result set, and then preparing the statement. (The constructor is invoked by the prepare method of **tdbc::connection.) A sample constructor looks like:

  constructor {connection sql} {
      next;                        # initialize the base class
      my variable resultSetClass 
      set resultSetClass whatever; # Tell the base class what class
                                   # to use for result sets
      my init $connection $sql;    # The [[init]] method should do
                                   # whatever is necessary to prepare
                                   # the statement
  }

Derived classes from tdbc::statement MUST also implement the params and paramtype methods. In addition, system resources belonging to the statement itself MUST be cleaned up by a destructor or by a deletion callback at C level. (Result sets MUST not be deleted then; the base classes take care of that.) See "Best practices for memory management" below for further discussion.

Result Sets

The class that implements a result set SHOULD normally inherit from the tdbc::resultset base class. Its constructor accepts the statement handle and the arguments to the execute method. The constructor is responsible for invoking the base class constructor with next, and executing the statement. A sample constructor looks like:

  constructor {statement args} {
      next
      uplevel 1 [list {*}[namespace code {my init}] $statement {*}$args]
  }

Note the peculiar form of invocation for the init method in the example above. Since the init method needs access to local variables in the caller's context to do variable substitution, it needs to be executed at the same stack level as the constructor itself. The [namespace code {my init}] call gives a command prefix that can be used to invoke the method in a foreign context, and this command is then executed with [uplevel 1] to do the initialization.

Besides the constructor and init, the other methods that a result set class MUST implement are columns, nextrow, and rowcount. In addition, a destructor (or a C deletion callback) MUST clean up any system resources belonging to the result set.

Best Practices for Memory Management in Database Interfaces

Since the TclOO interfaces are so new, it seems wise to give developers of database interfaces written in C some guidance about effective ways to manage memory. A C-level extension, if written correctly, gets considerable assistance in releasing memory at the appropriate times from TclOO and the tdbc base classes.

When a database interface is first loaded as an extension, it is entered through its PackageName_Init function. It will call, in order, Tcl_InitStubs, Tcloo_InitStubs, and Tdbc_InitStubs so that Tcl, the TclOO system, and the TDBC base classes are all available. Its next task is to allocate any per-interpreter data that may be required. (In the case of the tdbc::odbc bridge, the per-interpreter data include an ODBC environment handle and a string literal pool.) The per-interpreter data structure SHOULD be reference counted, since the order of destruction of the objects that refer to it is unpredictable. Next, the initialization function creates the classes, usually by evaluating an initialization script containing a call to tcl_findLibrary, where the Tcl code contains the skeletons of the class definitions. With the class definitions in hand, methods that are implemented in C can be attached to them. Any methods that need the per-interpreter data can receive it as ClientData. The reference count of the per-interpreter data SHOULD be incremented for these, and the method delete procedures should be responsible for decrementing the reference count.

Each of the three classes that make up a database interface SHOULD have a reference-counted data structure to hold any instance data. This structure SHOULD be created within the init method, and attached to the object with Tcl_ObjectSetMetadata. The metadata type structure SHOULD designate a delete procedure that decrements the reference count. The type structure MAY designate a clone procedure that returns TCL_ERROR; it is entirely permissible for TDBC objects not to be clonable.

Generally speaking, each object's instance data structure will contain a pointer to (and hold a counted reference to) the next higher object in the ownership hierarchy. A result set will refer to the statement that produced it; a statement will refer to the connection in which it executes, and a connection will refer to the per-interp data.

With this infrastructure in place, object destruction becomes strictly a local matter. Any object, when its reference count becomes zero, MUST release any system resources that belong to it, and decrement the reference count of the next object up. There is no need for a connection to track its statements, or a statement to track its result sets. This happens automatically because the prepare and execute methods create statements in a namespace subordinate to the namespace of the owning connection, and create result sets in a namespace subordinate to that of the owning statement. When the owning objects are destroyed, the subordinate namespaces are also destroyed, invoking the destructors of the objects within them.

This whole scheme is simpler than it sounds, and is observed to work well for the tdbc::odbc bridge (see the source code of the bridge for further details). Closing a connection gracefully deletes the statement and result class objects (in Tcl) from top to bottom, and then deletes the corresponding C data structures from bottom to top, finally cleaning up the connection data itself.

Note that, since TclOO does not guarantee to run destructors on exit, if a database interface needs to always close the underlying connection on termination, the implementation code should install an exit handler with Tcl_CreateExitHandler if it needs to.

Support Procedures for Implementors of Database Interfaces

In addition to the convenience commands discussed above, the Tcl system SHALL provide certain commands to aid the job of database implementors.

SQL Tokenisation

The task of mapping variable substituions in the form, :varName into whatever form that a native database API can handle is a somewhat tricky one. For instance, substitutions that appear inside quoted strings MUST NOT be mapped. In order to aid in this task, the Tcl system SHALL provide a command, ::tdbc::tokenize. This command SHALL accept a SQL statement as its sole parameter, and return a list of tokens. The lexical value of the tokens can be distinguished by their first characters:

  • '$', ':' and '@' are all variable substitutions; the remainder of the token string is a variable name.

  • ';' is a statement separator, for databases that allow multiple statements to be prepared together.

  • '-' is a comment

  • Anything else is literal text to be copied into a SQL statement.

Assuming that a native database's lexical structure conforms with standard SQL, the variable names can be substituted with parameter numbers, question marks, or whatever the database needs, to yield the native SQL that must be prepared.

Tokenisation is also available at the C level; to access it, a C extension MUST first call Tdbc_InitStubs; it is a macro that behaves as if it is a function with the type signature

int Tdbc_InitStubs(Tcl_Interp *interp);

where interp is a Tcl interpreter. The function returns TCL_OK if successful, and TCL_ERROR (with an error message left in the interpreter) in the case of failure.

The tokenisation is then available by calling

Tcl_Obj *Tdbc_TokenizeSql(Tcl_Interp *interp, const char *sqlCode);

In this call, interp is a Tcl interpreter, and sqlCode is a SQL statement to parse. If the parse is successful, the return value is a Tcl object with a reference count of zero that contains a list of token strings as with the **tdbc::tokenize_ call.

References

This specification is largely built from studying existing cross-platform database APIs and deriving a comon set of requirements from them. These include both popular offerings in lower-level languages (ODBC and JDBC) and Tcl-level ones (notably the 'nstcl-database' package, the SQLite API and tclodbc).

"ODBC Programmer's Reference." Redmond, Wash.: Microsoft Corporation, 2007. http://msdn2.microsoft.com/library/ms714177.aspx .

"Java Platform Standard Edition 6 API Specification." Santa Clara, Calif.: Sun Microsystems, 2007 http://java.sun.com/javase/6/docs/api/ ; in particular the package named, java.sql.

Cleverly, Michael. "nstcl-database Package." http://nstcl.sourceforge.net/docs/nstcl-database/ .

Hipp, D. Richard. "The Tcl interface to the Sqlite library." http://www.sqlite.org/tclsqlite.html .

Nurmi, Roy. "Tclodbc v 2.3 Reference." Available as part of the Tclodbc distribution at http://sourceforge.net/projects/tclodbc/ , in the file, DOC/REFERENC.HTM.

License

This file is explicitly released to the public domain and the author explicitly disclaims all rights under copyright law.

Appendix. Additional Possibilities.

An earlier version of this TIP specified several more requirements for the TDBC statement objects. In the current version, these requirements have been lifted. The three areas that have been removed are batch processing, asynchronous query handling, and references to cursors.

Rationale: Specifying an interface like this one is always a tradeoff between capability of the interface and burden upon the implementors. The earlier requirement for handling these three areas seems improvident.

The handling of bulk data ("batch processing") is to a large extent a performance issue. In most cases, if the performance of bulk data handling is critical, an implementor will resort to a compiled language rather than to Tcl to do so. The reporting of errors on bulk operations is complicated, as is the specification of what will happen if certain parameter sets succeed while others fail. The benefit of bulk data handling at the Tcl level was not deemed adequate to justify the implementation complexity.

The handling of asynchronous queries is also chiefly a performance issue in that it is intended to enable keeping a GUI live while long-running database operations are in progress. This "keep the GUI alive during long operations" requirement is equally well satisfied by performing database operations in a separate thread (for a thread-enabled Tcl) or a separate subprocess, and these techniques are familiar to Tcl programmers. For similar reasons, the ODBC manual now formally deprecates using ODBC's asynchronous operations on operating systems that support multithreading. Again, the benefits of integrating TDBC into the event loop do not appear to justify the cost in complexity to be gained.

References to cursors are a feature that is highly dependent on the underlying database. It is not clear that the specification described below is even readily implementable on all the platforms that have refcursors. Most of these, in any case, provide some other way of achieving the same end. For instance, Oracle allows returning a cursor by name, and then executing a statment, "FETCH ALL FROM :cursorName", to retrieve the data from the cursor. Again, here is a feature that adds complexity out of proportion to the benefits achieved.

Batch Processing

Some databases provide an interface to pass bulk data into a statement, in order to provide an efficient means for doing tasks such as inserting a large number of rows into a table at once. A statement handle MUST provide the subcommands:

statement startbatch

Prepares to perform batch processing on the specified statement.

statement addtobatch dictionary

Adds the values given by dictionary into the specified statement. The dictionary argument is exactly the same as the dictionary argument to [statement execute].

If no batch operation is in progress, the database interface MUST throw an error.

statement executebatch

Executes the batch of operations accumulated by [statement addToBatch].

The result of executebatch MUST be a result set. The rows of the result set are the result of concatenating the rows returned from the individual operations.

If no batch operation is in progress, the database interface MUST return an error.

If an underlying database does not support batch operations, the database interface SHOULD simulate them by accumulating the data in memory and executing the statement repeatedly when the executeBatch operation is requested.

The database interface MUST return an error if an attempt is made to execute a statement in the ordinary manner or to request a commit while there is an unfinished batch in progress. A rollback, or closing the statement, or closing the database connection, while a batch is in progress MUST result in abandoning the batch without applying any changes to the database.

Asynchronous Queries

Some database operations take a long time to complete. In order to avoid freezing the event loop, a database interface MAY provide an asynchronous query mechanism. If it does so, it MUST take the form:

resultSet whenready script

In this interface, resultSet is the handle of a result set. The whenready command requests that script be evaluated at the global level once for each row of the result set, plus once after all rows have been returned. The script SHOULD execute nextrow to retrieve the next row or get the indication that no rows remain.

References to Cursors

Some databases allow stored procedures to return references to cursors. If a column of a result set contains a reference to a cursor, it MUST be represented in Tcl as another result set handle. A Tcl script can then iterate over this included result set to use the reference to a cursor.

The given result set MUST be destroyed upon the next call to nextrow. For this reason, Tcl code MUST not use the allrows command with a statement that can return references to cursors.

Appendix. Change Summary

2008-04-27: Removed asynchronous queries, refcursors, and batch updates from the main body of the spec. Performed a good bit of general cleanup to bring the spec back in line with the reference implementation being developed.

2007-11-23: Expanded transaction management to have both the transaction command and explicit transaction boundaries. Added transaction isolation levels.

Added lists as an alternative to dicts as a representation of rows in result sets. Added a side interface for retrieving the set of column names in the convenience procedures.

Simplified introspection to return lists instead of result sets

Added batch processing.

Added asynchronous query processing.

Added an interface for stored procedures.

Added a discussion of returning refcursors.

2007-11-16: Changed the transaction management API from explicit commit and rollback to a model where a script is executed as an atomic operation.

Changed the "execute" API and the convenience procedures that use it to accept an optional dictionary containing substituents, so the substituents need not pollute the local namespace. The version accepting variables is still provided, because it is useful in the case of static queries where the substitutions follow a predetermined pattern.

Added reference to the author's cover letter on tcl-core.

Added missing citation of the nstcl-database API.

Appendix. Comments

Artur Trzewik (2007-11-19):

I miss defined error handling. Current DB-Api handles them in different way. How to obtain SQL-error message from server. If "execute" fails should it return TCL_ERROR or it should be special api for error code.

I miss C-framework or template to implement such API. Writing everything from scratch for all DB will be quite painfully. There are many things which can be reused: New Tcl objects, handles managing, thread-safe managing, encoding. Also prepared statements are not so easy. For example mysql requires that one allocate fixed size memory for all variables. It does not fit well with Tcl.

Kevin Kenny (2007-11-23):

Rest assured that at least one reference implementation will be published before this TIP is considered FINAL; database implementors are not going to be abandoned.

Andreas Leitgeb (2008-06-17):

For allrows and foreach calls, there has been some discussion about replacing the idiom -as list|dict by separate methods. Was this discussion dropped, or has it just not yet been reflected here?

For allrows and foreach calls there is ?-columnsvariable varName? ... I think, another option: ?-indicatorvariable varName? would be useful, as it allows both NULLs and equally named columns at the same time. Without indicator, dicts can handle only the former, and lists only the latter.

History
This page was generated in about 0.006s by Fossil 2.7 [9aa9ba8bd2] 2018-09-22 17:40:11