wiki:PortToIA-32

Version 27 (modified by rme, 6 years ago) (diff)

--

Notes on an IA-32 port

A port of OpenMCL to IA-32 (Intel's name for the 32 bit x86 architecture) would be a win for several reasons.

The most obvious reason would be to support Apple hardware that isn't 64-bit capable. This includes the first-generation Intel-based iMac, MacBook, MacBook Pro, and all Intel-based Mac Minis. Of course, support for non-Apple IA-32 hardware would also be nice.

Another reason would be to support access to Cocoa and Carbon (and other frameworks) on Intel-based Macintoshes running Mac OS X Tiger.

Although Apple has announced that Cocoa will be 64 bit in Leopard, they have publically confirmed that Carbon won't be. Therefore, a 32 bit lisp would still be needed to use Carbon, even on Leopard running on 64 bit hardware.

It would be interesting to support the AMD Geode LX (as used in the  OLPC laptop) as the minimum processor. This processor supports the P6 family instructions, including MMX instructions. We can therefore use the conditional move instructions, and maybe some MMX instructions to help out with bignums.

On the other hand, the AMD Geode LX processor doesn't support any of the SSE/SSE2/SSE3 instructions; this means that we'd have to use the x87 FPU (which is sort of funky). This would require modifications to the compiler, which believes that every floating point register can be accessed independently.

It might be reasonable to target the Core Solo/Duo? processor, at least to begin with. This would cover all Intel-based Macintosh systems ever shipped, and would allow us to avoid adding x87 FPU support for now.

Register usage and tagging

(See also  http://openmcl.clozure.com/Doc/index.html#Register-usage-and-tagging)

We want to keep the precise GC, but the limited number of registers that we have will probably make it impossible to statically partition the register file into immediate and tagged sets.

Anyway, we're looking at something like this:

eax 	 imm0
ecx 	 temp0
edx 	 temp1, nargs
ebx 	 arg_z
esp 	 stack pointer
ebp 	 frame pointer
esi 	 arg_y
edi 	 fn

We will augment this with a dynamic scheme: we will set or clear a bit in thread-private memory whenever a register transitions from one class to another. The GC will then look at these flag bits to decide how to treat the registers. (This may make the lispy register names confusing, since at times imm0 might actually contain a node, or arg_y an immediate.)

Callee-saved "non-volatile" registers are probably a non-starter.

The tagging scheme can basically follow the PPC32 port. An important difference is that the three-bit tag #b101, which is for NIL on PPC32, would be used for a thing called a tagged return addresses on IA-32. (More on this later.)

Comment by gb on Wed Aug 1 22:16:05 2007

It's probably sanest to think of the dynamic register partitioning as being a set of (local, temporary) changes relative to a baseline scheme, where the baseline scheme is in effect any time a function is entered (and therefore at the time of a function call). At that time, we probably need more node regs and fewer imm regs than the scheme suggested above provides, and we can probably overload nargs and imm0.

If we pass two arguments in registers, then we probably need a node register to address the callee on a function call (something like:

(movl (@ 'foo (% fn)) (% temp0))
(:talign ia32::fulltag-tra)
(call (@ ia32::symbol.fcell (% temp0))
(movl ($ :self) (% fn))

The CLOS implementation will sometimes funcall a method-function with an invisible argument (not counted against nargs) in a node register. (That's context information for CALL-NEXT-METHOD and it's done in a way that's not MOP-compliant.)

I think that in general if we err on the side of "too many node regs" in the baseline partitioning, we always (cheaply) save the values in those node regs if we need to temporarily make the register immediate for consing, shifts, multiply/divide, memory assignment, whatever else needs more than one imm reg.)

I think that having imm0=nargs would work fairly well, since we're usually either validating/defaulting based on nargs or doing tag/bounds checking, but rarely if ever need to do both at the same time.

Comment by rme

It turns out that there's a bit of a wrinkle in using imm0 as nargs, and that wrinkle is funcall. Funcall needs to use an imm reg for tag checking, and if our only imm reg contains nargs, well, that's going to be a problem. We'll use temp1 for nargs instead.

Another possibility might be a minor design change: make CALL-ARGUMENTS-LIMIT smaller, and use %ah for nargs, which would leave %al free for tag checking.

Threads

We don't have the __thread storage class on Darwin, so we will need to use i386_set_ldt to install a segment descriptor for each thread into the LDT. When a thread's segment descriptor is loaded into the %fs segment register, %fs can be used to refer to thread-local storage.

(This implies an 8K limit to the number of threads, by the way. Probably not a big deal for a 32 bit lisp.)

LAP notes

quoted numbers are fixnums, e.g., for x8664, (add ($ '2) (% rsp)) adds 16 to %rsp.

memory operand form: ([seg] [disp] [base] [index] [scale])

Bootstrapping notes

Follow these instructions to use a current Darwin/x86-64 lisp to build the ia32 branch. (I think that the 070722 snapshot should also work.)

Check out the ia32 branch from svn.

$ svn co svn+ssh://svn.clozure.com/usr/local/publicsvn/openmcl/branches/ia32

Use

$ svn co http://svn.clozure.com/publicsvn/openmcl/branches/ia32

if you don't have write access to the repository.

Next, copy a kernel, image, and interfaces into the ia32 tree.

$ p=/path/to/trunk/ccl/
$ cp $p/{dx86cl64,dx86cl64.image} .
$ cd darwin-x86-headers64/libc
$ cp $p/darwin-x86-headers64/libc/*.cdb .

Build the ia32 branch sources. Follow the directions below.

;;; pick up operators i386-ff-call and i386-syscall
(load "compiler/nxenv.lisp")
(load "compiler/nx1.lisp")

;;; define some stuff from x86-asm.lisp
(in-package "X86")
(defparameter *opcode-flags*
  `((:jump . ,(ash 1 0))		;special case for jump insns
    (:CpuNo64 . ,(ash 1 16))		;not supported in 64 bit mode
    (:Cpu64 . ,(ash 1 17))		;64 bit mode required
    (:CpuSSE . ,(ash 1 18))		;SSE extensions required
    (:CpuSSE2 . ,(ash 1 19))		;SSE2 extensions required
    (:CpuSSE3 . ,(ash 1 20))		;SSE3 extensions required
))

(defun %encode-opcode-flags (flags &optional errorp)
  (flet ((encode-atomic-flag (f)
	   (if f
	     (cdr (assoc f *opcode-flags*))
	     0)))
    (or
     (if (atom flags)
       (encode-atomic-flag flags)
       (let* ((k 0))
	 (dolist (f flags k)
	   (let* ((k0 (encode-atomic-flag f)))
	     (if k0
	       (setq k (logior k0 k))
	       (return))))))
     (if errorp (error "Unknown x86 opcode flags: ~s" flags)))))

(defmacro encode-opcode-flags (&rest flags)
  (%encode-opcode-flags flags t))

(in-package "CCL")

(load "compiler/X86/X8632/x8632-arch.lisp")
(load "lib/x8632env.lisp")
(compile-file "compiler/X86/X8664/x8664-backend.lisp")
(compile-file "compiler/X86/x86-asm.lisp")
(compile-file "compiler/X86/x86-lap.lisp")
(compile-file "compiler/X86/x862.lisp")

;;; Ignore the warnings about the undeclared x86::*x86...* free variables.

(load "compiler/X86/X8664/x8664-backend")
(load "compiler/X86/x86-asm")
(load "compiler/X86/x86-lap")
(load "compiler/X86/x862")

(compile-ccl t)
(xload-level-0)

Exit, start up with the bootstrap image, and then save a new full image.

Darwin/IA-32 interface databases

To build interface files, get an ffigen binary from clozure.com:/pub/testing and install it.

$ cd darwin-x86-headers/libc/C
$ ./populate.sh
+++ /Developer/SDKs/MacOSX10.4u.sdk/usr/include/ar.h
+++ /Developer/SDKs/MacOSX10.4u.sdk/usr/include/arpa/ftp.h
+++ /Developer/SDKs/MacOSX10.4u.sdk/usr/include/arpa/inet.h
[...]
$

There's one more step to making these interfaces available to the lisp, but we'll get to that in a minute.

Cross compiling

Given an image made as above, to set up for cross-compiling:

(in-package "CCL")
(set-development-environment)
;;; may not actually need all this stuff
(load "compiler/X86/X8632/x8632-arch.lisp")
(require "X8632ENV")
(require "X8632-ARCH")
(defpackage "X86-DARWIN32")
(compile-file "compiler/X86/x86-lap.lisp")
(load "compiler/X86/x86-lap")
(load "compiler/X86/x86-backend.lisp")
(load "compiler/X86/x86-disassemble.lisp")
(load "compiler/X86/X8632/x8632-backend.lisp")

(let ((*target-backend* *x8632-backend*))
  (load "ccl:compiler;X86;X8632;x8632-vinsns.lisp")
  (load "ccl:compiler;X86;x86-lapmacros.lisp"))

(load "lib/ffi-darwinx8632.lisp")

(require-update-modules *x8632-xload-modules* t)

The first time you do this, you need to finish processing the interface databases.

(require "PARSE-FFI")
(parse-standard-ffi-files :libc :darwinx8632)
;;; lots of output...
;;; Do it once more...
(parse-standard-ffi-files :libc :darwinx8632)

Evaluating

(cross-xload-level-0 :darwinx8632 :force)

will create a boot image in x86-boot32.image.

To cross-compile the rest of the lisp, evaluate:

(cross-compile-ccl :darwinx8632 t)

Note that several files will not compile yet; just pick the "skip this file" restart and carry on for now.

To build an IA-32 lisp kernel:

$ cd lisp-kernel/darwinx8632
$ make

The current state of the port is that it will map in the boot image and start fasloading files. It'll die at l1-utils.dx32fsl.

To run the IA-32 lisp (from the lisp-kernel/darwinx8632 directory):

$ gdb ../../dx86cl
GNU gdb 6.3.50-20050815 (Apple version gdb-768) (Tue Oct  2 04:07:49 UTC 2007)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-apple-darwin"...Reading symbols for shared libraries .. done

(gdb) run ../../x86-boot32.image
Starting program: /Users/rme/ccl/dx86cl x86-boot32.image
Reading symbols for shared libraries +. done
obtaining Mach exception lock in exception thread
Setting up exception handling for 0x100450
Set up exception context for 0x100450 at 0x342f80
obtaining Mach exception lock in exception thread
doing pseudo_sigreturn for 0x100450
did pseudo_sigreturn for 0x100450
;Loading level-1.dx32fsl
;Loading ./l1-fasls/l1-cl-package.dx32fsl
obtaining Mach exception lock in exception thread
Setting up exception handling for 0x100450
Set up exception context for 0x100450 at 0x342e60
obtaining Mach exception lock in exception thread
doing pseudo_sigreturn for 0x100450
did pseudo_sigreturn for 0x100450
;Loading ./l1-fasls/l1-utils.dx32fsl

[and it chokes...]

If you take a look and have questions or comments, please send mail to openmcl-devel@…, or to me directly (rme@…).