Author:         George Petasis <petasis@iit.demokritos.gr>
State:          Final
Type:           Project
Vote:           Done
Created:        11-Jun-2002
Post-History:   
Discussions-To: news:comp.lang.tcl
Keywords:       load,unload,dynamic library
Tcl-Version:    8.5

Abstract

Tcl already provides facilities for loading dynamic libraries, through the load command. However no facilities are currently offered in order to unload dynamic libraries already loaded with the load command. This TIP tries to add support for unloading libraries, by introducing a new Tcl command (unload) and the guidelines that dynamic libraries must follow, in order to be unloadable. Note that the unload command will operate only on libraries that are designed to be unloadable by their developers. This way backward compatibility with older extensions is maintained, as unload will never try to unload libraries unaware of this new functionality.

Rationale

Tcl is an ideal language for component-based applications. Usually these applications offer a framework in which components developed by the users of the application can be embedded, in order to extend the functionality of the framework. Usually, these extensions are implemented as C/C++ dynamic libraries that are loaded through the load Tcl command.

However the development of such components can be a time-consuming process, as developers have to restart the framework application in order to be able to reload the library into it and test its altered functionality. And this can be quite annoying (depending on the application of course), as usually processing within the application is required in order to bring it into a proper state before testing the library.

The development cycle can be significantly shortened if Tcl provides a mechanism for unloading a dynamic library. A new version of the library can be created, as its object file can now be written, and the updated library can be re-loaded.

However, this is not the only application of unload. Services running for long periods of time and want to unload no longer needed functionality, replacing parts of an applications (i.e. from an automatic update procedure) or functionality temporarily needed (i.e. a web browser that loads a plugin to display a file of a particular file type) are some additional fields of application.

Introduction

Unload functionality has been left out of the Tcl core, mainly because library unloading was poorly implemented in many operating systems. But almost all operating systems have been improved in the meantime, and as a result most modern operating systems now support library unloading.

The main idea of this TIP is to enable dynamic library unloading at the Tcl level, in the same sense load provides dynamic library loading. However, library unloading will be provided only when the underlying operating system support this feature (as is also the case for load) and only when the library to be unloaded provides a set of functions that can "undo" the changes the library has made to the interpreter. In all other cases, unloading a library will result in an error.

This TIP proposes the insertion of a new Tcl command named unload and two functions pkg_Unload and pkg_SafeUnload, modelled after pkg_Init and pkg_SafeInit, that libraries which can be unloaded should implement.

Unloadable Libraries

A main concern is related to when a shared library can be "unloadable". An unloadable library is a library characterised as such by its developer. The developer of the library must export a function from the library, similar to the library's initialisation function. The unload command will never try to unload a library that does not provide such a function. This makes old libraries (before the introduction of the unload functionality) safe.

There is a category of libraries that can never be unloaded (i.e. libraries that register new tcl object types). However, the choice is upon the developer: the developer knows if the library can be unloadable. The simpler case, libraries that only register new commands are the most probable libraries to be unloadable. Libraries that export functions through a stub mechanism cannot be unloaded, as the were meant for having dependencies with other libraries that use the exported API. There is no way for the provider library to know wether it is used or not.

Specification

Actually, all the facilities for unloading dynamic libraries already exist in the Tcl core and simply they are not yet exposed at the Tcl level. As a result, the implementation of the unload command should be fairly easy.

load as it is currently implemented loads a dynamic library only the first time this library is loaded. It keeps an internal cache of all loaded libraries and if an already loaded library is requested again, only its initialisation function is called. This cache should be extended to keep some additional information:

Two reference counts, counting how many times a specific library has been loaded. This reference count should be increased by each load and decreased for each unload. When it reaches 0, the library can be unloaded. Safe interpreters and normal interpreters should have different reference counts. Both should be 0 in order for a library to be unloaded.
The addresses of the pkg_Unload and pkg_SafeUnload functions, if these are implemented by the library. If both of these functions are missing, the library will never be unloaded. If only pkg_Unload is implemented, the library can be unloaded if it never has been loaded in a safe interpreter. Finally, if pkg_SafeUnload is implemented, the library can be unloaded if it has never been loaded in a normal interpreter.

The unload command will always return an error, if the operating system does not support library unloading. In case the operating system supports library unloading:

unload will examine the cache of load to locate the entry for the library to be unloaded. It is an error to unload a library that has not been loaded with load.
If the entry in the cache is found, unload checks whether the corresponding for the interpreter type unload function pointer is NULL or not. If it is NULL, the library cannot be unloaded under this interpreter and again an error is returned.
If the unload function pointer is not NULL, it is executed. If an error is returned by this function, unload also returns an error.
If the unload function finishes without errors, the reference count corresponding to the interpreter type is decreased. If both reference counts reach 0, the library is unloaded.

Responsibilities of the Unload Functions

Its up to the developer of the library to decide if its library can be unloaded or not. A library can be unloaded if the function pkg_Unload is implemented and exported as a symbol from the library, and the library will never be loaded in a safe interpreter. A library that can be also loaded in safe interpreters is unloadable if the function pkg_SafeUnload is also available. These two functions will accept two arguments, the interpreter under which the library is unloaded and an integer, holding various flags. The flags argument can be either TCL_UNLOAD_DETACH_FROM_INTERPRETER or TCL_UNLOAD_DETACH_FROM_PROCESS. In case the library will remain attached to the process after the unload procedure returns (i.e. because the library is used by other interpreters), TCL_UNLOAD_DETACH_FROM_INTERPRETER will be defined. However, if the library is used only by the target interpreter and the library will be detached from the application as soon as the unload procedure returns, the flags argument will be set to TCL_UNLOAD_DETACH_FROM_PROCESS.

The main responsibility of these functions is to remove from the interpreter they are unloaded under any reference to a function residing inside the library. For example, such a function must:

Unregister any commands that have been registered by the Init() function to the interpreter given by the first argument. In order to do this, the library should keep internally the tokens returned by each Tcl_Create*Command, as command may have been renamed.
Unregister any other commands that may have been registered to the interpreter during the use of the library (usually used to represent special special data structures).

If the flag TCL_UNLOAD_DETACH_FROM_PROCESS is defined, the developer must do additional task, that are not normally required when the library gets unloaded from an interpreter:

Free any memory occupied by the internal structures of the library.
In general, try to remove any references Tcl may have to functions provided by the library.

If the developer cannot remove all reference to functions to the library, its better to not provide at all these two functions, so as unload to never attempt to unload the library.

Dependencies Among Libraries

It is possible that a library A has been loaded that exports some symbols. Then a library B is loaded, that has dependencies (i.e. uses some exported symbols) on A. What if A gets unloaded?

Actually, most modern operating systems seem to provide a solution to this problem, as reference counts are hold internally by the operating system for each library. Newer Windows, Solaris and Linux seem to provide similar solutions and in reality they don't unload the library if such symbols remain, even if the unload system call has been made for the library. Both libraries A and B have to be unloaded in order for A to be really removed from the address space.

Reference Implementation

A reference implementation can be found at: http://sf.net/tracker/?func=detail&aid=823486&group\_id=10894&atid=310894

Copyright

This document has been placed in the public domain.

Appendix: The unload man page.

 NAME
 unload - Unload machine code.
 SYNOPSIS
 unload ?switches? fileName
 unload ?switches? fileName packageName
 unload ?switches? fileName packageName interp
 DESCRIPTION
 -nocomplain
 -keeplibrary
 - -
 PORTABILITY ISSUES
 Unix
 Macintosh
 BUGS
 SEE ALSO
 KEYWORDS

NAME

unload - Unload machine code.

SYNOPSIS

unload ?switches? fileName

unload ?switches? fileName packageName

unload ?switches? fileName packageName interp

DESCRIPTION

This command tries to unload shared libraries previously loaded with load from the application's address space. fileName is the name of the file containing the library file to be unload; it must be the same as the filename provided to load for loading the library. packageName is the name of the package, and is used to compute the name of the unload procedure. interp is the path name of the interpreter from which to unload the package (see the interp manual entry for details); if interp is omitted, it defaults to the interpreter in which the unload command was invoked.

If the initial arguments to unload start with - then they are treated as switches. The following switches are currently supported:

-nocomplain Supresses all error messages. If this switch is given unload will never report an error.

-keeplibrary This switch will prevent unload from issuing the operating system call that will unload the library from the process.

 Marks the end of switches. The argument following
 this one will be treated as a fileName even if it
 starts with a -.

When a file containing a shared library is loaded through the load command, Tcl associates two reference counts to the library file. The first counter shows how many times the library has been loaded into normal (trusted) interpreters while the second describes how many times the library has been loaded into safe interpreters. As a file containing a shared library can be loaded only once by Tcl (with the first load call on the file), these counters track how many interpreters use the library. Each subsequent call to load after the first, simply increaments the proper reference count.

unload works in the opposite direction. As a first step, unload will check whether the library is unloadable: an unloadable library exports a special unload procedure. The name of the unload procedure is determined by packageName and whether or not the target interpreter is a safe one. For normal interpreters the name of the initialization procedure will have the form pkg_Unload, where pkg is the same as packageName except that the first letter is converted to upper case and all other letters are converted to lower case. For example, if packageName is foo or FOo, the initialization procedure's name will be Foo_Unload. If the target interpreter is a safe interpreter, then the name of the initialization procedure will be pkg_SafeUnload instead of pkg_Unload.

If unload determines that a library is not unloadable (or unload functionality has been disabled during compilation), an error will be returned. If the library is unloadable, then unload will call the unload procedure. If the unload procedure returns TCL_OK, unload will proceed and decrease the proper reference count (depending on the target interpreter type). When both reference counts have reached 0, the library will be detached from the process.

The unload procedure must match the following prototype: typedef int Tcl_PackageUnloadProc(Tcl_Interp *interp, int flags);

The interp argument identifies the interpreter from which the library is to be unloaded. The unload procedure must return TCL_OK or TCL_ERROR to indicate whether or not it completed successfully; in the event of an error it should set the interpreter's result to point to an error message. In this case, the result of the unload command will be the result returned by the unload procedure. The flags argument can be either TCL_UNLOAD_DETACH_FROM_INTERPRETER or TCL_UNLOAD_DETACH_FROM_PROCESS. In case the library will remain attached to the process after the unload procedure returns (i.e. because the library is used by other interpreters), TCL_UNLOAD_DETACH_FROM_INTERPRETER will be defined. However, if the library is used only by the target interpreter and the library will be detached from the application as soon as the unload procedure returns, the flags argument will be set to TCL_UNLOAD_DETACH_FROM_PROCESS.

The unload command cannot unload libraries that are statically linked with the application. If fileName is an empty string, then packageName must be specified. If packageName is omitted or specified as an empty string, Tcl tries to guess the name of the package. This may be done differently on different platforms. The default guess, which is used on most UNIX platforms, is to take the last element of fileName, strip off the first three characters if they are lib, and use any following alphabetic and underline characters as the module name. For example, the command unload libxyz4.2.so uses the module name xyz and the command unload bin/last.so {} uses the module name last.

PORTABILITY ISSUES

Unix Not all unix operating systems support library unloading. Under such an operating system unload returns an error (unless -nocomplain has been specified).

Macintosh

BUGS

If the same file is loaded by different fileNames, it will be loaded into the process's address space multiple times. The behavior of this varies from system to system (some systems may detect the redundant loads, others may not). In case a library has been silently detached by the operating system (and as a result Tcl thinks the library is still loaded), it may be dangerous to use unload on such a library (as the library will be completely detached from the application while some interpreters will continue to use it).

TIP 100: Add Support for Unloading Dynamic Libraries Loaded with [load]