TIP 738: Add -relative Option to source Command

Login
Home Timeline Branches Tags Tickets Help
Bounty program for improvements to Tcl and certain Tcl packages. 
    Author: Eric Taylor <[et99@rocketship1.me]>
    State: Withdrawn
    Type: Project
    Vote: Pending
    Tcl-Version: 9.1
    Created: 20-Nov-2025
    Post-History:

Abstract

Add a -relative option to the source command to load files from the directory containing the currently executing script, rather than from the current working directory.

Rationale

Motivation: Code Files vs. Data Files

Tcl's current pwd-relative path resolution is correct for data files - files the user is working on

# User runs: cd ~/documents && tclsh ~/tools/process.tcl
set f [open "data.txt" r]    # Correctly opens ~/documents/data.txt
set f [open "output.csv" w]  # Correctly creates in ~/documents/

However, when scripts source code files (other script components), they should be found relative to the script's location, not the user's working directory:

# Inside ~/tools/process.tcl
source helpers.tcl    # Should find ~/tools/helpers.tcl
                     # Currently looks in ~/documents/helpers.tcl - WRONG!

This approach is consistent with other scripting languages: Ruby provides require_relative 'filename' for exactly this purpose, Python's import system resolves modules relative to the package location. The -relative option brings Tcl in line with this established pattern of explicit, script-relative code loading.

Distinction from Package Mechanisms

The existing Tcl package system is designed for distributing reusable library code installed in system-wide directories, typically requiring administrator privileges. In contrast, source -relative addresses a different use case: a main Tcl script loading its own supporting files for modular organization.

These supporting files are not libraries for other programs - they are internal components that travel with the application. Requiring package infrastructure for such files forces either system-wide installation or complex path configuration. The source -relative option enables the simpler model where an application and its supporting files can be unpacked together anywhere and work immediately, without installation or special privileges.

Self-Contained Applications

Consider distributing a Tcl application as a zip file:

myapp.zip contains:
myapp/
  +-- myapp.tcl          # Main entry point
  +-- gui.tcl
  +-- logic.tcl
  +-- lib/
      +-- database.tcl
      +-- utils.tcl

Users expect this to work no matter where unzipped:

unzip myapp.zip
cd ~/documents              # User's working directory
tclsh ~/myapp/myapp.tcl    # Run from anywhere

Inside myapp.tcl, the application needs to source its code components (gui.tcl, logic.tcl) which are in ~/myapp/, not in ~/documents/ where the user is working.

The standard solution requires this verbose idiom:

source [file join [file dirname [file normalize [info script]]] gui.tcl]
source [file join [file dirname [file normalize [info script]]] lib/utils.tcl]

This idiom is not self-documenting and creates a barrier for newcomers. It makes "unzip and run" applications unnecessarily complex.

With -relative:

# Clean and obvious - find code relative to script location
source -relative gui.tcl
source -relative lib/utils.tcl

# Data files still use pwd naturally
set f [open "userdata.txt" r]  # Opens ~/documents/userdata.txt

Specification

Syntax

source ?-encoding encodingName? ?-relative? fileName

Semantics

When a script is being sourced (i.e., [info script] returns a non-empty value), source -relative filename resolves the path relative to the directory containing the currently executing script.

Internally, this is accomplished by:

  1. Getting the normalized path of the current script: [file normalize [info script]]
  2. Extracting its directory: [file dirname ...]
  3. Joining with the provided filename: [file join ... filename]

Using file normalize also ensures symbolic links are resolved to their actual file location, and handles other path syntax such as ~ expansion and .. resolution.

This is similar (but not identical) to the idiom:

source [file join [file dirname [file normalize [info script]]] filename]

Key difference: When [info script] returns empty (no script context), the behaviors differ:

  • The idiom: [file dirname ""] returns ".", silently sourcing from pwd
  • source -relative: Errors with "requires a script context when sourcing <filename>"

Use case: source -relative is intended for initialization-time sourcing - when scripts load their component files during startup, including nested sourcing where one script sources another. It is not designed for runtime dynamic loading inside procedures.

Examples

source -relative code.tcl              # same directory
source -relative lib/utilities.tcl     # subdirectory
source -relative ../common.tcl         # parent directory
source -encoding utf-8 -relative file.tcl  # with encoding
source -relative -encoding utf-8 file.tcl  # order doesn't matter

Limitation: Relative Paths and Directory Changes

The -relative option has a limitation when the main script is sourced using a relative path and then changes directories before sourcing component files.

Example of the problem:

# User runs: tclsh myapp.tcl (relative path)
# Inside myapp.tcl:
cd subdirectory
source -relative component.tcl  # Fails - looks in wrong directory

This occurs because [info script] stores the path exactly as provided to source. When the path is relative ("myapp.tcl"), file normalize uses the current working directory, which after the cd is now the wrong directory.

Note: This is not a new limitation introduced by -relative - the idiom source [file join [file dirname [file normalize [info script]]] file] has the identical problem.

Workarounds:

  1. Don't change directories before sourcing component files
  2. Source the main script with an absolute path: tclsh /full/path/myapp.tcl

This limitation is inherent to how [info script] and file normalize interact with relative paths.

Implementation

The implementation modifies TclNRSourceObjCmd in generic/tclCmdMZ.c. The complete modified function is:

int
TclNRSourceObjCmd(
    TCL_UNUSED(void *),
    Tcl_Interp *interp,         /* Current interpreter. */
    int objc,                   /* Number of arguments. */
    Tcl_Obj *const objv[])      /* Argument objects. */
{
    const char *encodingName = NULL;
    Tcl_Obj *fileName;
    int result;
    void **pkgFiles = NULL;
    void *names = NULL;
    int useRelative = 0;
    int nextArg = 1;

    if (objc < 2 || objc > 5) {
        Tcl_WrongNumArgs(interp, 1, objv, 
            "?-encoding encoding? ?-relative? fileName");
        return TCL_ERROR;
    }

    /* Parse options */
    while (nextArg < objc - 1) {
        const char *arg = TclGetString(objv[nextArg]);
        
        if (strcmp(arg, "-encoding") == 0) {
            if (nextArg + 1 >= objc - 1) {
                Tcl_WrongNumArgs(interp, 1, objv,
                    "?-encoding encoding? ?-relative? fileName");
                return TCL_ERROR;
            }
            encodingName = TclGetString(objv[nextArg + 1]);
            nextArg += 2;
        } else if (strcmp(arg, "-relative") == 0) {
            useRelative = 1;
            nextArg++;
        } else if (strcmp(arg, "-nopkg") == 0) {
            /* Handle undocumented -nopkg option */
            pkgFiles = (void **)Tcl_GetAssocData(interp, "tclPkgFiles", NULL);
            names = *pkgFiles;
            *pkgFiles = NULL;
            nextArg++;
        } else {
            Tcl_SetObjResult(interp, Tcl_ObjPrintf(
                "bad option \"%s\": must be -encoding or -relative", arg));
            Tcl_SetErrorCode(interp, "TCL", "LOOKUP", "INDEX", "option",
                arg, (char *)NULL);
            return TCL_ERROR;
        }
    }

    fileName = objv[objc - 1];

    /* Handle -relative option */
    if (useRelative) {
        Tcl_Obj *scriptPathObj;
        
        /* Evaluate [file normalize [info script]] to get the current script path */
        result = Tcl_EvalEx(interp, "file normalize [info script]", -1, TCL_EVAL_GLOBAL);        
        if (result != TCL_OK) {
            return TCL_ERROR;
        }
        
        scriptPathObj = Tcl_GetObjResult(interp);
        
        /* Check if we got an empty string (no script context) */
        if (Tcl_GetString(scriptPathObj)[0] == '\0') {
            Tcl_SetObjResult(interp, Tcl_ObjPrintf(
                "-relative option requires a script context when sourcing \"%s\"",
                Tcl_GetString(fileName)));
            Tcl_SetErrorCode(interp, "TCL", "OPERATION", "SOURCE",
                "NOCONTEXT", (char *)NULL);
            return TCL_ERROR;
        }        
        Tcl_IncrRefCount(scriptPathObj);
        
        /* Get the directory containing the script */
        Tcl_Obj *scriptDir = TclPathPart(interp, scriptPathObj,
            TCL_PATH_DIRNAME);
        
        if (scriptDir == NULL) {
            Tcl_DecrRefCount(scriptPathObj);
            return TCL_ERROR;
        }
        Tcl_IncrRefCount(scriptDir);
        
        /* Join script directory with the provided filename */
        Tcl_Obj *pathParts[2];
        pathParts[0] = scriptDir;
        pathParts[1] = fileName;
        fileName = Tcl_FSJoinPath(Tcl_NewListObj(2, pathParts), 2);
        Tcl_IncrRefCount(fileName);
        
        /* Evaluate the file */
        result = TclNREvalFile(interp, fileName, encodingName);
        
        Tcl_DecrRefCount(fileName);
        Tcl_DecrRefCount(scriptDir);
        Tcl_DecrRefCount(scriptPathObj);
    } else {
        result = TclNREvalFile(interp, fileName, encodingName);
    }
    
    if (pkgFiles) {
        /* Restore "tclPkgFiles" assocdata to how it was. */
        *pkgFiles = names;
    }
    
    return result;
}

Key Changes

  • Added useRelative flag and nextArg index for option parsing
  • Changed objc check to allow up to 5 arguments (for both -encoding and -relative)
  • Added option parsing loop that handles -encoding, -relative, and existing -nopkg
  • When -relative is set:
    • Evaluates [info script] to get current script path
    • Extracts directory with TclPathPart
    • Joins with filename using Tcl_FSJoinPath
    • Passes resolved path to TclNREvalFile

The implementation reuses existing Tcl path manipulation functions and follows the established pattern for option parsing.

Backwards Compatibility

Fully compatible. New optional flag, no changes to existing behavior. Scripts using -relative fail gracefully on older Tcl with "bad option" error.

Testing

The implementation has been tested with various scenarios:

  • Basic usage: source -relative file.tcl
  • Subdirectories: source -relative lib/utils.tcl
  • With encoding: source -encoding utf-8 -relative file.tcl
  • Both option orders: source -relative -encoding utf-8 file.tcl
  • Error case: Using -relative at interactive prompt correctly errors

A complete test suite for tests/source.test will be provided when approved.

Future Considerations

The load command exhibits similar pwd-relative behavior for binary extensions. However, since binary extensions are typically distributed via the package system rather than direct load calls, this is less commonly a problem. If -relative proves valuable for source, a future TIP could extend it to load for consistency.

Withdrawal Notice

This TIP has been withdrawn. During review, several fundamental limitations were identified:

  1. Relative path handling: When scripts are sourced with relative paths and change directories before loading components, path resolution fails
  2. Symbolic links: file normalize does not fully resolve symlinks to script files
  3. Thread safety: In multi-threaded applications, cd is process-wide, making any relative path resolution inherently unsafe

These limitations are fundamental to how [info script] and file system operations work in Tcl, and cannot be resolved within the scope of this TIP.

Copyright

This document has been placed in the public domain.

History
This page was generated in about 0.007s by Fossil 2.26 [935cd45f03] 2025-02-06 15:57:56