Author:         Jan Nijtmans <jan.nijtmans@gmail.com>
State:          Final
Type:           Project
Vote:           Done
Created:        29-Dec-2017
Post-History:
Keywords:       tcl
Tcl-Version:    9.0
Tcl-Branch:     memory-API

Abstract

This TIP describes the non-controversial part of the Tcl 9 changes: Make Tcl 9 ready for the 64-bit era.

Rationale

Many Tcl API functions and struct fields have int parameters, which don't provide sufficient room on 64-bit platforms.

Proposal

Enhance the hash functions, such that the hash value is stored in a size_t in stead of unsigned int. This allows hash tables to grow beyond 4Gb on 64-bit platforms
Enhance all struct fields representing refCounts or epochs, make them of type size_t
All memory-related functions, such as Tcl_Alloc(), will change its size argument from int to size_t, and its "char *" arguments to "void *"
Many functions, which have size parameters of type int (but NOT int *) will change to type size_t
All ClientData type arguments will be changed to void * arguments. This is actually the same type, but it prevents the need for type-casts in some situations.
Provide a compilation option -DTCL_8_COMPAT, which provides fully source compatibility. Using this option, extensions compile and run fine using either Tcl 8 or Tcl 9 headers. More explanation below.
Provide 2 new macro's TCL_IO_FAILURE and TCL_AUTO_LENGTH, both equal to ((size_t)-1). They can help making extensions use the full 64-bit range with Tcl 9, which still compile with Tcl 8 as well (see below). Tcl 8.7 will get those macro's too, but the value there is (-1)
The functions Tcl_Alloc(), Tcl_Free() (and friends) change to redirect to their debugging variants if the TCL_MEM_DEBUG is defined. So, Tcl_Alloc() becomes the same as ckalloc(), ending the general confusion regarding the difference between those two groups of functions: Starting with Tcl 9.0 there is no difference anymore. ckalloc() and friends are deprecated starting with 9.0, but there are no plans to actually remove those and no deprecation warning will be given if extensions use it.

On 32-bit platforms, this is all 100% upwards binary compatible, provided no internal API is used (since some internal structs might have incompatible but externally invisible changes)

On 64-bit platforms, those changes cause binary incompatibility. Therefore the TCL_STUB_MAGIC value needs to change, so extensions compiled using Tcl 9 headers will not load in Tcl 8 and reverse.

Implications

Although those changes are binary compatible on 32-bit platforms, there is a small potential source incompatibility. There are 10 functions which previously had an int return argument, which is changed to size_t (for now, see below). This signed/unsigned change might lead to subtle difference in behavior.

The 10 functions are:

Tcl_Gets
Tcl_GetsObj
Tcl_Read
Tcl_ReadChars
Tcl_ReadRaw
Tcl_Write
Tcl_WriteChars
Tcl_WriteObj
Tcl_WriteRaw
Tcl_Ungets

If the return value of such function is directly used in a compare, this could lead to the use of an unsigned compare in stead of a signed compare. If you compile your extension with -DTCL_8_COMPAT, those 10 functions will be changed to wrapper macro's which makes everything behave as if those functions return Tcl_WideInt. That's the easiest way to resolve this potential problem.

There are 2 other ways to make this change, which can do it without the use of the TCL_8_COMPAT macro:

Add 1 to the left and right hand side of the comparison. E.g. as here
Don't compare to -1 using < or >, but always to TCL_IO_FAILURE using == or != . E.g. as here

Tk and sqlite are already converted to make full use of TIP #494. Other extensions included with Tcl (e.g. tdbc) are unaffected.

Open issues

There has been discussion, whether the return-type of these 10 functions (and possibly other functions) should actually be size_t or maybe some other type. A Tcl-specific type Tcl_Size could also be defined for this. Actually, this TIP provides 2 possibilities: Without -DTCL_8_COMPAT, the type is size_t, with -DTCL_8_COMPAT it is Tcl_WideInt. This gives enough room for experimenting which one is best, maybe it would be better to make it ssize_t or ptrdiff_t.

There - for sure - will be more future TIP's, which propose more API changes for 9.0. This one - definitely - is not the last one. So, this TIP doesn't make a final decision yet what will be the final type returned by those 10 functions.

Implementation

The implementation of this TIP can be found in the memory-API branch.

Copyright

This document has been placed in the public domain.