Error when calling (rebuild-ccl ...) in a newly compiled wx86cl64 kernel

[symingz]

I successfully built a wx86cl64 kernel with mingw-w64 + cygwin + win7-64bit (mingw-w64-bin_i686-cygwin-1.7.6-1_20100902, cygwin 1.7.7), using the method mentioned in ticket #733. But when I run (rebuild-ccl :clean t) using the new kernel and the svn check out image, an error occured, saying something like "an error occured during read of memory address 0x2e when calling (ccl:get-native-utf-16-cstring)". Then I changed back to the svn check out kernel, and everything was fine, a new image was successfully built. And I tried again with the new kernel + the new image, but the same error occured.

[gb]

From a quick glance, it appears that the fault at 0x2e is caused by the fact that data structures used by mingw's C runtime library have changed (and no longer match the headers used to generate the interfaces that CCL's using.)

If there's some reason to believe that the mingw64 interfaces are now stable, then clearly the right thing to do is to change CCL to use the newer interfaces. (That's going to introduce binary incompatibility, but it's the right thing to do moving forward. It's also a non-trivial change, and the benefits to users aren't clear. People who want to build the kernel with a recent mingw64 distribution will be able to do so, and the newer C library might be better/more complete or something.)

(Sometimes, when people are able to anticipate that data structures might change, the data structures in question contain version info, and a function's caller can examine that info and try to adapt. I don't know if that's a factor here, but it seems clear that the function _wreaddir - which is used to enumerate directory entries - isn't returning a _wdirent structure that looks like what the lisp code that calls that function expects. That's a (static) mingw C library issue, not a toolchain issue: we're clearly calling _wreaddir and it's returning a structure, but that structure's apparently laid out differently from the way that it used to be. The #x2e seems to be a "." in the directory entry's name, and we're expecting to see a pointer to a UTF-16 string containing that "."

The Mingw64 Sourceforge project doesn't seem to maintain archives of older distributions. I've been using something that's around a year old; I uploaded the archive to < ftp://clozure.com/pub/mingw-w64-bin_i686-cygwin-1.5.25-15_4.4.1a.tar.bz2>; the tools in that archive seem to run fine in Cygwin 1.7.x.

If my understanding of the issues is correct, we sort of have to use the older static library in that archive until lisp's interfaces have been updated to be compatible with the newer library.

(Another approach - since we barely use the Mingw C library at all - would be to stop using it at all. This particular function - _wreaddir - is one of the few things that we do use, and we could just incorporate the old version of the function into the lisp kernel sources, assuming that we can find the source to that old version ...)

None of this has too much to do with CCL, aside from its dependence on the Mingw64 libraries and toolchain. If better (more stable) alternatives existed, I'd certainly be interested in using them. (I don't know that it makes sense to try to use MS tools and libraries; that may just create its own set of problems.) We do have to do something (other than say "try to find and use an older mingw64 release", especially since the current mingw64 release isn't completely broken (and some recent versions may have been.)

[johnfredcee]

Having tracked down the commit in question, my guess is that the reason for the commit is to increase general POSIX/GNU compatibility as both require a plain char array and not a pointer in that field.

 The Checkin

[rme]

We now (as of r14581) use the native Windows calls #_FindFirstFileW/#_FindNextFileW/#_FindClose.