Author:         Frédéric Bonnet <fredericbonnet@free.fr>
State:          Final
Type:           Project
Vote:           Done
Created:        23-Jan-2017
Post-History:   
Tcl-Version:    8.7
Tcl-Branch:     tip-462

Abstract

This TIP proposes to improve Tcl's handling of subprocesses created by the exec and open commands by adding a new ::tcl::process ensemble.

Rationale

This TIP is inspired by a request from FlightAware to fix the way Tcl currently handles child process exit status.

Subprocess creation can be either synchronous or asynchronous. In either case, a children with a non-zero return value indicates an error condition that is bubbled up to the Tcl error handling mechanism.

Synchronous subprocesses

Synchronous subprocesses are created using the exec command with no & terminal argument. Errors are raised synchronously as well.

Asynchronous subprocesses

Asynchronous subprocesses can be created using two distinct methods:

exec command with a & terminal argument. In this case the command returns immediately with a list of the PIDs for all the subprocesses in the pipeline.
open "| command". In this case the command returns immediately with the channel id of the pipe (hereafter $ch). The subprocess IDs are given by the [pid $ch] command. The subprocess status code and error conditions are processed upon channel closure with the [close $ch].

Error handling and status code

Errors are caught with the catch and try commands, with status codes given in the -errorcode options dictionary entry and the errorCode global variable in the form {CHILDKILLED pid sigName msg} / {CHILDSTATUS pid code} / {CHILDSUSP pid sigName msg}.

C-level access

The Tcl library provides the following procedures for managing subprocesses (excerpts from the Tcl documentation):

Tcl_DetachPids may be called to ask Tcl to take responsibility for one or more processes whose process ids are contained in the pidPtr array passed as argument. The caller presumably has started these processes running in background and does not want to have to deal with them again.
Tcl_ReapDetachedProcs invokes the waitpid kernel call on each of the background processes so that its state can be cleaned up if it has exited. If the process has not exited yet, Tcl_ReapDetachedProcs does not wait for it to exit; it will check again the next time it is invoked. Tcl automatically calls Tcl_ReapDetachedProcs each time the exec command is executed, so in most cases it is not necessary for any code outside of Tcl to invoke Tcl_ReapDetachedProcs. However, if you call Tcl_DetachPids in situations where the exec command may never get executed, you may wish to call Tcl_ReapDetachedProcs from time to time so that background processes can be cleaned up.
Tcl_WaitPid is a thin wrapper around the facilities provided by the operating system to wait on the end of a spawned process and to check a whether spawned process is still running. It is used by Tcl_ReapDetachedProcs and the channel system to portably access the operating system.

Moreover, Tcl_WaitPid is blocking unless called with the WNOHANG option.

Limitations

The current implementation is lacking several key features:

There is no way to get subprocess status other than through the error handling mechanism.
Consequently, there is no way to collect the status code of a asychronous subprocess created with the exec & method because such commands don't raise errors once the subprocesses are launched.
There is no non-blocking way to query asynchronous subprocess status codes; catch/try upon open "| command" pipe closure is blocking.
Moreover, exec and open call Tcl_ReapDetachedProcs, thereby cleaning up all pending information on terminated subprocesses. This prevents any advanced subprocess monitoring at the script level.
While reasonable in the general case, a non-zero return value does not always indicates an error condition for all kinds of programs, so it is desirable to provide a subprocess-specific mechanism that does not rely on Tcl's standard error handling facility.

Specifications

A new ::tcl::process will be created:

::tcl::process subcommand ?arg ...: Subprocess management.

The following subcommand values are supported by ::tcl::process:

::tcl::process list: Returns the list of subprocess PIDs.
::tcl::process status ?switches? ?pids?: Returns a dictionary mapping subprocess PIDs to their respective statuses. If pids is specified as a list of PIDs then the command only returns the status of the matching subprocesses if they exist, and raises an error otherwise. For active processes, the status is an empty value. For terminated processes, the status is a list with the following format: {code ?msg errorCode?}, where:
- code is a standard Tcl return code,
- msg is the human-readable error message,
- errorCode uses the same format as the ::errorCode global variable.

Note that msg and errorCode are only present for abnormally terminated processes (i.e. those where code is nonzero). Under the hood this command calls Tcl_WaitPid with the WNOHANG flag set for non-blocking behavior, unless the -wait switch is set (see below).

::tcl::process purge ?pids?: Cleans up all data associated with terminated subprocesses. If pids is specified as a list of PIDs then the command only cleanup data for the matching subprocesses if they exist, and raises an error otherwise. If the process is still active then it does nothing.
::tcl::process autopurge ?flag?: Automatic purge facility. If flag is specified as a boolean value then it activates or deactivate autopurge. In all cases it returns the current status as a boolean value. When autopurge is active, Tcl_ReapDetachedProcs is called each time the exec command is executed or a pipe channel created by open is closed. When autopurge is inactive, ::tcl::process purge must be called explicitly. By default autopurge is active and replicates the current Tcl behavior.

Additionally, ::tcl::process status accepts the following switches:

-wait: By default the command returns immediately (the underlying Tcl_WaitPid is called with the WNOHANG flag set) unless this switch is set. If pids is specified as a list of PIDs then the command waits until the matching subprocess statuses are available. If pids is not specified then it waits for all known subprocesses.
--: Marks the end of switches. The argument following this one will be treated as the first arg even if it starts with a -.

Examples

% ::tcl::process autopurge
true
% ::tcl::process autopurge false
false

% set pid1 [exec command1 a b c | command2 d e f &]
123 456
% set chan [open "|command1 a b c | command2 d e f"]
file123
% set pid2 [pid $chan]
789 1011

% ::tcl::process list
123 456 789 1011

% ::tcl::process status
123 0 456 {1 "child killed: write on pipe with no readers" {CHILDKILLED 456 SIGPIPE "write on pipe with no readers"}} 789 {1 "child suspended: background tty read" {CHILDSUSP 789 SIGTTIN "background tty read"}} 1011 {}

% ::tcl::process status 123
123 0

% ::tcl::process status 1011
1011 {}

% ::tcl::process status -wait
123 0 456 {1 "child killed: write on pipe with no readers" {CHILDKILLED 456 SIGPIPE "write on pipe with no readers"}} 789 {1 "child suspended: background tty read" {CHILDSUSP 789 SIGTTIN "background tty read"}} 1011 {1 "child process exited abnormally" {CHILDSTATUS 1011 -1}}

% ::tcl::process status 1011
1011 {1 "child process exited abnormally" {CHILDSTATUS 1011 -1}}

% ::tcl::process purge
% exec command1 1 2 3 &
1213
% ::tcl::process list
1213

Rejected Alternatives

The first version proposed to implement the feature as a new ps option to the existing info command. However, almost all operations in [info] are things that just examine state, not change it, and that's a principle-of-least-astonishment that should be upheld for the sake of less experienced users.

Reference implementation

The reference implementation is available on branch tip-462 in the Tcl Fossil repository.

Copyright

This document has been placed in the public domain.

TIP 462: Add New [::tcl::process] Ensemble for Subprocess Management