Author:         Jan Nijtmans <jan.nijtmans@gmail.com>
Author:	        Christian Werner <Christian.Werner@t-online.de>
State:          Draft
Type:           Project
Tcl-Version:    8.6, 9.0
Tcl-Branch:     tip-709

Abstract

This TIP proposes to accept MemoryModule.[ch] in the Tcl Repository for the Windows build. These files are under the Mozilla Public License as prominently stated at the very top of these files.

In the default Windows build, MemoryModule.[ch] is not used. In Tcl 8.6 and 9.0 it is only enabled when the build is configured with --enable-memorymodule. And - even if it is enabled - dll's containing a TLS (= Thread Local Storage, not Transport Layer Security) section won't be loaded by it.

Rationale

This TIP is meant to solve a longstanding bug in the implementation of the VFS loader for Windows. On Windows a dynamically-linked library (DLL) residing in a VFS (e.g. a mounted zip-file) is copied to a temporary directory before being loaded. This directory is never cleaned up, resulting in left-over temporary directories with every tclsh/wish run. The bug is present in all Tcl releases (8.6 too), but it became more visible in 9.0 because of the added support for a zipfs, which is the reason this TIP targets 8.6 and 9.0.

A TIP is needed since MemoryModule.[ch] is distributed under a different license, compatible with the Tcl license.

The Mozilla Public License doesn't impose any restrictions on software linked to it: the whole combined work can still be distributed under any license. But there are some restrictions on the two mentioned files:

Any modification of those 2 files must be done under the MPL license too.
The source code of such a modification must be published.

Tcl is compliant with those restrictions.

The current implementation of MemoryModule.[ch] is derived from MemoryModule master branch, with 4 additional pull-requests added. And it contains a trivial fix, making it work on ARM64 too. Since the TLS implementation in the Windows loader is a lot different in Vista than in XP, and MemoryModule still only implements the XP part, MemoryModule.[ch] contains a small additional patch which lets it refuse to load any dll which contains a TLS section, so any dll which uses at least one __declspec(thread) variable. Most Tcl extensions don't use such variables, so they can safely be loaded in memory.

Specification

This is an opt-in feature which must explicitly enabled by the option --enable-memorymodule when using the Windows Makefile build or OPTS=memorymodule when using the makefile.vc build."

When enabling this, a DLL inside a VFS is no longer copied to a temporary directory in order to load it. The DLL is loaded directly in memory using the functionality of MemoryModule.

Known limitations

There are three known limitations at the moment. If any of the first two affect you, don't configure Tcl with --enable-memorymodule. The third limitation is handled at runtime, causing the fallback to be triggered. The limitations might be lifted in the future.

Loading a DLL which depends on another DLL that is also in a VFS doesn't work, but that doesn't work without MemoryModule either, so it cannot be seen as a limitation of MemoryModule itself.
A DLL that is protected with WinLicense does not work.
A DLL using TLS (Thread Local Storage)

If the loading by MemoryModule fails, a fallback to the copy strategy takes place. (This is not tested for situation 2), so be warned!)

Compatibility

This change is 100% compatible: In the default build, the memorymodule is not used at all. Only when people do a special build with --enable-memorymodule, the step copying the dll to the /tmp directory is skipped.

What to do when my dll is not working well with memorymodule?

Don't use --enable-memorymodule, just use the default build.
Don't use a dll, but compile the code as a .lib file, and link it in statically.
Make sure that your dll has a TLS section. You can do that by creating a __declspec(thread) variable somewhere in the code.

I bet that your dll is not working well with memorymodule because it already has a TLS section. In that case, there's nothing special to be done.

In my view it's overkill to provide a "-memory" option to the load command, because all known dll's work fine with memorymodule except the ones containing a TLS section. Extensive tests have been done to assure that (all tests supplied by Ashok and more will be integrated in the test-suite to assure that).

Any dll which doesn't work well with memorymodule, and doesn't have a TLS section (the first one still isn't found) can be worked-around by introducing a __declspec(thread) variable. Since this situation is so rare, a --nomemory option would be overkill.

Implementation

See the tip-709 branch.

A new win/dltest directory with testcases for the MemoryModule functionality is added.

Extensive tests are currently done using the LUCK (Lean Undroidwish Construction Kit) environment.

Copyright

This document has been placed in the public domain.