TIP 579: Improved auto_path for Safe Base interpreters

Login
Home Timeline Branches Tags Tickets Help
Bounty program for improvements to Tcl and certain Tcl packages. 
Author:         Keith Nash <k.j.nash@usa.net>
State:          Draft
Type:           Project
Vote:           Pending
Created:        07-Jul-2020
Version:        25-Jul-2020
Post-History:
Tcl-Version:    8.7
Keywords:       safe interp auto_path access_path
Tcl-Branch:     tip-579-8-7

Abstract

This TIP proposes modifying the Safe Base to give users the option of defining the ::auto_path in a Safe Base interpreter to conform to the specification given in pkg_mkIndex(n) and library(n). The default is to preserve the existing behavior of the Safe Base, in which the ::auto_path includes a large number of additional directories.

Rationale

A Safe Base interpreter is a safe slave interpreter that can apply the commands load and source to files in certain directories specified by its master interpreter. The list of those directories is called the access path and is recorded in the master interpreter. In the Safe Base slave, each directory of the access path is represented by a token of the form $p(:34:) which includes a literal dollar character and an integer. A filename can be composed as, for example, [file join {$p(:34:)} pkgIndex.tcl]. The Safe Base interpreter uses modified source and load commands that convert a filename rooted at an access path token to a real filename. The tclIndex autoloader, package command, and tcl::tm command in Safe Base interpreters are compatible with such tokenized filenames.

The access path comprises the directories of the requested ::auto_path and all their children, and the directories of the module path [tcl::tm::list] and all their descendants.

In a Safe Base interpreter, the value of ::auto_path is currently set to the tokenized access path, rather than to a tokenized form of the usual definition given by pkg_mkIndex(n) and library(n). This feature, not documented in safe(n), allowed package to work correctly despite the absence of the glob command in a Safe Base interpreter. The value of ::auto_path is synchronized with the tokenized access path whenever the latter is changed by a safe::* command in the master interpreter.

When tcl::tm facilities were added to Tcl, a restricted form of glob was added to Safe Base interpreters. This new glob command allows package to operate correctly when ::auto_path conforms to the specification in pkg_mkIndex(n) and library(n). However, the Safe Base still defines the ::auto_path to be equal to the tokenized access path, and as a result it lists a large number of directories that are used by tcl::tm, and have nothing to do with the purpose of ::auto_path - seeking pkgIndex.tcl or tclIndex files. This feature is unhelpful when debugging, because the list of directories in the access path can be very long.

Proposed Change

(a) A new exported command safe::setSyncMode

(b) A new option for Safe Base interpreters, -autoPath, defined under certain conditions explained below.

(c) Revised behavior for the commands that can alter the access path, i.e. safe::interpCreate, safe::interpInit, safe::interpConfigure, and safe::interpAddToAccessPath.

The new command is

safe::setSyncMode ?newValue?

When an argument is supplied, the command returns an error in two cases: (a) if the argument is not a boolean value; (b) if any Safe Base interpreters exist.

Otherwise, the command is an accessor for a new namespace variable safe::AutoPathSync. Normal return supplies the value of this variable. The value is initialized when the file tcl8.6/safe.tcl is sourced by the autoloader.

The value [safe::setSyncMode] controls the behavior of all Safe Base interpreters, and can be changed only when no Safe Base interpreters are defined. Typically the value will be set as part of initialization.

When [safe::setSyncMode] is true, the behavior is the same as in current Tcl. The Safe Base interpreter's ::auto_path is defined as a tokenized form of the access path. Operations that change the Safe Base interpreter's access path also change its value of ::auto_path.

When [safe::setSyncMode] is false, ::auto_path is defined to fulfil the requirements of pkg_mkIndex(n) and library(n) only. It is not modified when the access path changes, except by operations that give the slave default paths derived from those of its master - in these cases the slave's ::auto_path is set to a tokenized form of the master's ::auto_path.

When [safe::setSyncMode] is false, each Safe Base interpreter is given a new option "-autoPath" which can be used with the commands safe::interpCreate, safe::interpInit, and safe::interpConfigure in the same way as other options. The -autoPath option is undefined when [safe::setSyncMode] is true, permitting 100% compatibility with the existing Tcl Safe Base.

In order to implement this new definition of ::auto_path in a Safe Base interpreter, some changes must be made to the behavior of existing Safe Base commands when [safe::setSyncMode] is false. These changes are specified below.

Specification and Examples

The commands that can alter the access path are safe::interpCreate, safe::interpInit, safe::interpConfigure, and safe::interpAddToAccessPath. The effects of these commands on the access path and ::auto_path are described below, including the changes that are proposed by this TIP.

In this section:

  • The Existing Case is defined to mean Tcl either in its current form, or with this TIP implemented and with [safe::setSyncMode] true.
  • The Extra Case is defined to mean Tcl with this TIP implemented and with [safe::setSyncMode] false. The differences from the Existing Case are the changes proposed by this TIP.
  • Code Examples are given for the Extra Case.

The effects of options -accessPath and -autoPath on the value of the slave's ::auto_path can be expressed in four rules, with a fifth rule for the command safe::interpAddToAccessPath.

(1) Calling safe::interpCreate, safe::interpInit, or safe::interpConfigure with an option/value pair for -autoPath.

  • Existing Case
    • The option -autoPath is undefined. The command will raise an error.
  • Extra Case
    • The presence of the "-autoPath value" arguments has no effect on the access path.
    • The command applies option -autoPath after it determines the value of access path from the other arguments.
    • The command will set the slave's ::auto_path to a tokenized form of the specified value.

(2) Calling safe::interpCreate or safe::interpInit without the option -accessPath, or with -accessPath {}; or calling safe::interpConfigure with -accessPath {}.

  • This remains the most convenient way to initialize a Safe Base interpreter.
  • Each command sets the access path to the master interpreter's ::auto_path and its children, and then appends the directories of [tcl::tm::list] and their descendants.
  • Each command sets the slave's [tcl::tm::list] to a tokenized form of the master's [tcl::tm::list].
  • Existing Case
    • Each command sets ::auto_path to the tokenized access path.
  • Extra Case
    • If the option -autoPath is specified, the slave's ::auto_path is determined by rule (1).
    • If the option -autoPath is not specified, each command sets the slave's ::auto_path to a tokenized form of the master's ::auto_path.

Examples of calls to commands safe::* that belong to this category are:

        safe::interpCreate foo
        safe::interpCreate foo -accessPath {}
        safe::interpInit bar
        safe::interpInit bar -accessPath {}
        safe::interpConfigure foo -accessPath {}

(3) Calling safe::interpCreate, safe::interpInit, or safe::interpConfigure with a non-empty value of -accessPath.

Each such command:

  • Sets the access path to the requested value, then appends [tcl::tm::list] and its descendants.
  • Sets the slave's [tcl::tm::list] to a tokenized form of the master's [tcl::tm::list].
  • Existing Case
    • Sets the slave's ::auto_path to the tokenized access path.
  • Extra Case
    • If option -autoPath is also present, the slave's ::auto_path is determined by rule (1).
    • If the option -autoPath is not specified, commands safe::interpCreate and safe::interpInit each set the slave's ::auto_path to {}.
    • If the option -autoPath is not specified, command safe::interpConfigure leaves the slave's ::auto_path unchanged.

e.g.

        safe::interpCreate foo -accessPath {
            /usr/local/TclHome/lib/tcl8.6
            /usr/local/TclHome/lib/tcl8.6/http1.0
            /usr/local/TclHome/lib/tcl8.6/opt0.4
            /usr/local/TclHome/lib/tcl8.6/msgs
            /usr/local/TclHome/lib/tcl8.6/encoding
            /usr/local/TclHome/lib
        }

        # The slave's ::auto_path must be given a suitable value:

        safe::interpConfigure foo -autoPath {
            /usr/local/TclHome/lib/tcl8.6
            /usr/local/TclHome/lib
        }

        # The two commands can be combined:

        safe::interpCreate foo -accessPath {
            /usr/local/TclHome/lib/tcl8.6
            /usr/local/TclHome/lib/tcl8.6/http1.0
            /usr/local/TclHome/lib/tcl8.6/opt0.4
            /usr/local/TclHome/lib/tcl8.6/msgs
            /usr/local/TclHome/lib/tcl8.6/encoding
            /usr/local/TclHome/lib
        } -autoPath {
            /usr/local/TclHome/lib/tcl8.6
            /usr/local/TclHome/lib
        }

(4) Other calls of command safe::interpConfigure with option/value pairs.

The only case not covered by rules (1) to (3) is call to safe::interpConfigure with neither option -accessPath nor -autoPath. Such a call does not alter either the access path or the ::auto_path.

(5) Calling safe::interpAddToAccessPath.

  • The command appends a directory to the slave's existing access path.
  • It does not change the slave's [tcl::tm::list].
  • Existing Case
    • It sets ::auto_path to the tokenized access path
  • Extra Case
    • It does not change the slave's ::auto_path.

e.g.

  • The example code below can be appended to the example code that defines the interpreter foo in rule (3) above.
  • In the Extra Case only, the slave's auto_path must be modified to allow the interpreter to use the Img package.
        safe::interpAddToAccessPath foo /usr/local/TclHome/lib/extras/Img1.4.11

        lassign [safe::interpConfigure foo -autoPath] DUM slaveAutoPath
        lappend slaveAutoPath /usr/local/TclHome/lib/extras/Img1.4.11
        safe::interpConfigure foo -autoPath $slaveAutoPath
  • In the examples below, no change is needed to ::auto_path, either because the existing value allows access to the new package directories, or because the added directories (e.g. timezone data) do not concern packages.
        safe::interpAddToAccessPath foo /usr/local/TclHome/lib/sqlite3.30.1.2

        foreach dir {
            tzdata
            tzdata/Africa
            ...
            tzdata/US
        } {
            safe::interpAddToAccessPath foo /usr/local/TclHome/lib/tcl8.6/$path
        }

Summary of Specification

  • Existing Case
    • Safe Base commands set the slave's ::auto_path to the tokenized access path.
    • Advantages: the caller need not be concerned with the slave's ::auto_path.
    • Disadvantages: does not conform to pkg_mkIndex(n) and library(n); the slave's ::auto_path includes many directories that do not belong in the search path for pkgIndex.tcl or tclIndex files; the list of directories is very long and this is unhelpful when debugging.
  • Extra Case
    • The commands in rule (2) set the slave's ::auto_path to a tokenized form of the master interpreter's.
    • When other commands change the access path, the caller is responsible for making any necessary changes to the slave's ::auto_path. The new -autoPath option simplifies this task.

Discussion

Because this TIP adds a new option -autoPath, it must go into detail about the interplay between -autoPath and -accessPath, especially when values are changed by safe::interpConfigure. This leads to a fairly lengthy consideration of multiple cases (above).

When writing tests for the reference implementation, many bugs were discovered in the existing Safe Base code, most especially in handling changes of options by safe::interpConfigure. It is hard to avoid the conclusion that (like the present author) most users create a Safe Base interpreter with the properties that they want, and seldom if ever need to call safe::interpConfigure.

In this usage case, the entirety of the TIP's specification is greatly simplified:

Typical Use

In many cases, the properties of a Safe Base interpreter can be specified when the interpreter is created, and then left unchanged for the lifetime of the interpreter.

If you wish to use Safe Base interpreters with "Sync Mode" off, evaluate the command

       safe::setSyncMode 0

Use safe::interpCreate or safe::interpInit to create an interpreter with the properties that you require. The simplest way is not to specify -accessPath or -autoPath, which means the safe interpreter will use the same paths as the master interpreter. However, if -accessPath is specified, then -autoPath must also be specified, or else it will be set to {}.

The value of -autoPath is that required to access tclIndex and pkgIndex.txt files according to the same rules as an unsafe interpreter (see pkg_mkIndex(n) and library(n)).

With "Sync Mode" on (the default), the option -autoPath is undefined, and the Safe Base sets the safe interpreter's ::auto_path to a tokenized form of the access path. In addition to the directories present if "Safe Mode" is off, the ::auto_path includes the numerous subdirectories and module paths that belong to the access path.

Reference Implementation

Branch TIP-579-8-7 in the public fossil repository for Tcl.

Compatibility

The default value of [safe::setSyncMode] is 1. The new interpreter option -autoPath is undefined in this case. All existing code will behave in the same way as before.

If desired, the default value of [safe::setSyncMode] could be changed to 0 at a future major revision of Tcl.

Copyright

This document has been placed in the public domain.

History
This page was generated in about 0.009s by Fossil 2.7 [9aa9ba8bd2] 2018-09-22 17:40:11