Author:         Kevin Kenny <kennykb@acm.org>
State:          Final
Type:           Project
Vote:           Done
Created:        11-Feb-2012
Post-History:   
Keywords:       coroutine,yield,yieldto
Obsoletes:      372
Tcl-Version:    8.6

---

Abstract

A new command, yieldto, is proposed that allows a corouting to suspend its execution and tailcall into an arbitrary command. If the new command is another coroutine's resume command, we obtain symmetric coroutines. If the new command is the return command, we obtain the ability for a coroutine to present an unusual status (e.g., error, break, continue or return) to its caller. The yieldto command also marshals the arguments as a list when the yielding coroutine is next invoked, allowing for the transmission of multiple arguments.

Rationale

Several TIPS (at least [372], [373], [375], and [383]) have been advanced to propose various improvements to the coroutine control transfer provided by the yield command. This TIP attempts to distill the requirements of these TIPs into an irreducible minimum for implementation in 8.6 (resolving a blocking issue for an 8.6 release).

This TIP intentionally leaves out of scope some of the more complex or controversial issues, such as enhancements to info args and info default, unusual return from a `yield` operation. and code injection into coroutines. It is believed that all of these can be added later, without introducing needless incompatibilities into the basic mechanisms of coroutine construction, invocation, and yielding.

Requirements that are thought to be essential for this TIP include:

The ability for a coroutine invocation to implement any argument signature that an Tcl command can implement. A coroutine invocation must be able to accept multiple arguments, and to allow for call-by-name (or rather, call-by-quasi-value-result, see below) parameter transmission.

The ability to return an unusual status. A coroutine invocation must be able to return an error status or another unusual status (e.g., break, continue or return) to its caller, and to perform a tailcall.

Support for symmetric coroutines. Although it is well known that asymmetric coroutines (such as Tcl 8.6 implements today) and symmetric coroutines have equivalent power, the implementation of symmetric coroutines in a system that supports only asymmetric ones is possible only by coding a separate scheduler that allows an active coroutine to detach with a request to resume another. If symmetric coroutines are not implemented directly, it is likely that multiple incompatible schedulers will spring up in user code, greatly impeding a later unification.

Proposal

The new command:

yieldto cmd ?arg1...?

shall accept one or more arguments:

cmd - The name of a command to invoke in place of the current coroutine invocation.

arg1... - The arguments to pass to the given command.

It shall have the following effects:

1. The cmd argument shall be resolved in the current coroutine's context, resulting in a command to invoke. If resolution fails, the error is presented in the coroutine's context.

2. The current coroutine shall suspend its execution in the same way as with the yield command.

3. The command that invokes the coroutine shall be placed into a state such that it will accept multiple arguments when it is next invoked, rather than the single argument demanded by yield.

4. The command and arguments shall be invoked in just the same way as if they had been called directly from the coroutine's caller. The given command replaces the coroutine invocation on the runtime stack. Data and status returned from the given command are returned to the context that invoked the coroutine.

5. When the coroutine is resumed, any arguments passed into the coroutine command are assembled into a list and returned as the value of the yieldto command.

In other words, yieldto means "suspend the current coroutine and tailcall the given command: yieldto is to yield as tailcall is to return." In addition, yieldto causes the current coroutine to accept multiple arguments on its next invocation.

Relationship with the Earlier Proposals

[372] proposes a yieldm command that allows a coroutine to yield and subsequently accept multiple arguments when resumed. The requested functionality of [372] can be layered trivially atop this proposal: a one-line implementation of the yieldm command would be:

interp alias {} yieldm {} yieldto return -level 0

[373] proposed yieldto together with a separate yieldset command. The latter allowed a coroutine to designate a set of arguments and defaults. The advantage over simply passing the arguments as a list was that error messages for incorrect numbers of arguments could be generated automatically, and that info args and info body could introspect into the desired argument list. Since the error message generation can be done readily by auxiliary procedures, and the introspection is something of a nicety, this proposal defers the implementation of yieldset.

[375], which replaced [373], proposed a yieldto command that is the same as the current proposal's, except that it could transmit only a single argument when the coroutine was resumed. As such, it was incomplete as it stands.

[387] proposed a unified syntax for all of the above TIPs. The proposed syntax was quite complex, and its only advantage over the current proposal is that is allowed for introspection using info args and info body and for automatic generation of error messages for incorrect arguments. Since these are regarded as something of a nicety, the author of the current proposal believes that their consideration can be deferred in favour of the current proposal.

The related [383] addresses a different set of issues: injecting code into a suspended coroutine for the purpose of debugging. Its implementation can be decided on independently of this TIP.

Examples:

1. Multiple arguments and error returns.

Let us assume that there is a coroutine foo that wishes to accept at each invocation two arguments bar and grill. It therefore needs to accept multiple arguments, and to check the number of arguments, returning an error to its caller if they are incorrect. Code structured like the following can serve both purposes:

 # presume that $value is the value to return to the last invocation
 for {set args [yieldto return -level 0 $value]} \
     {[llength $args] != 2} \
     {set args [yieldto return -level 0 -code error \
                    -errorCode {MYCORO WRONGNUMARGS} \
                    "wrong # args, should be \"foo bar grill\""]} {
         # do nothing
 }
 lassign $args bar grill

2. Symmetric coroutines.

It may not be obvious from the foregoing discussion that the original purpose of yieldto was imagined to be passing of control between peer coroutines. For instance, if we assume that there are two coroutines, producer and consumer, and that calling producer returns a string while calling consumer accepts a string and returns nothing, then each coroutine may yield to the other:

Code in producer:

 yieldto consumer $string

Code in consumer:

 lassign [yieldto producer] string

3. Call by name.

It turns out that yieldto allows for a rough simulation of Tcl's call by name, although it requires explicit transmission of values between coroutines, rather than a simple uplevel operation. Let us assume that a caller invokes a coroutine with the name of an array, and that the coroutine wishes to set elements of that array. A procedure like the following will give approximately the desired result.

 proc remote-set {varName value} {
     lassign [yieldto remote-set-worker $varName $value [info coroutine]] \
         status result
     return -code $status $result
 }
 proc remote-set-worker {varName value coro} {
     tailcall $coro \
         [catch {uplevel 1 [list set $varName $value]} result] \
         $result
 }

(Proper transmission of the options dictionary from catch is omitted for the sake of simplicity.)

Let's look at what happens when code in the coroutine mycoro does

 remote-set array(key) value

1. The remote-set procedure yields to the command

            remote-set-worker array(key) value mycoro
   

2. The remote-set-worker, executing in the calling coroutine, invokes

            set array(key) value
   

   in its caller's scope, capturing the status and return value. Presuming that the variable was set successfully, It then invokes

            mycoro 0 value
   

   as a tailcall, re-entering the mycoro coroutine. The two-element list, {0 value}, becomes the return value of yieldto

3. The remote-set procedure returns the same status and result that the set command returned.

This technique actually shows how arbitrary commands may be executed in a calling coroutine. The way this is done opens some interesting possibilities for tools such as debugging interactors. For instance, a trace callback might invoke a debugging coroutine to interact with the user, allow the user to execute arbitrary commands to probe the state of the code being debugged, and then transparently resume at the place that the code left off, without otherwise disturbing its stack.

Implementation notes

The yieldto return idiom is expected to be used widely, since it is the way to present unusual status codes, and internally is used in the implementation of yieldm. For this reason, the implementation may contain code to optimize it specially.

Acknowledgments

The implementation of this TIP, and most of the details of its specification, are actually the brainchild of Miguel Sofer, who expresses a continued interest in seeing it implemented while being unwilling at the present time to shepherd it through the sometimes acrimonious discussions on tcl-core. To him belongs all the credit for the ideas: any errors in the specification as presented here are the author's.

Thanks are due to Donal Fellows, Joe English, Andreas Leitgeb, and Lars Hellström for making needed corrections to the initial version of this TIP.

Copyright

This document has been placed in the public domain.