wiki:ForeignFunctionInterface

Using Clozure CL's FFI

BLAS and LAPACK libraries

The BLAS and LAPACK libraries were originally written in Fortran. It's certainly possible to call the Fortran-style routines directly from CCL, but some care is needed, because Fortran conventions differ from C conventions. (CCL's FFI is geared towards calling C functions.)

In Fortran, all arguments to a function, even scalar values, are passed by reference (that is, as pointers). Arrays are also different. In Common Lisp, arrays are indexed from 0 and are stored in row-major order. In Fortran, arrays are indexed from 1, and are stored in column-major order.

To show how this works in practice, we'll call the BLAS routine idamax, which finds the index of the element of a double-float vector with the maximum absolute value.

Welcome to Clozure Common Lisp Version 1.7-dev-r14780  (LinuxARM32)!
? (open-shared-library "libblas.so")
#<SHLIB libblas.so.3gf #x5454AF36>
? (rletz ((n :int 10)
	  (dx (:array :double 10))
	  (incx :int 1))
     (setf (paref dx (:array :double) 5) 100d0
	   (paref dx (:array :double) 8) -1000d0)
     (external-call "idamax_" (:* int) n (:* double) dx (:* int) incx :int))
9
?

Notice that the function parameters are passed by reference (that is, as pointers). Second, note that the returned answer is 9 and not 8 as we might expect. This is because Fortran array indexes start at 1, not at 0 as they do in Common Lisp. Finally, note the "_" suffix on the name of the Fortran function---that's simply a common Unix convention.

There is also a C interface to the BLAS. This is somewhat more convenient to use, as the following example shows:

? (rletz ((x (:array :double 10)))
   (setf (paref x (:array :double) 5) 100d0
	 (paref x (:array :double) 8) -1000d0)
   (external-call "cblas_idamax" :int 10 (:* double) x :int 1 :int))
8

With the C interface, we don't have to pass scalar parameters by reference, and the indexes are 0-based, as a Lisp programmer would expect. (The libblas.so that we loaded earlier also contains the C interface functions on this system.)

We can make this more convenient still by using the interface translator to parse the cblas.h header file and generate an interface database for it.

The directions at http://trac.clozure.com/ccl/wiki/CustomFramework explain how to do this. With the interface database present, calling the BLAS function becomes even easier:

? (use-interface-dir :cblas)
#<INTERFACE-DIR :CBLAS #P"cblas/" #x54547E8E>
? (rletz ((x (:array :double 10)))
    (setf (paref x (:array :double) 5) 100d0
	  (paref x (:array :double) 8) -1000d0)
    (#_cblas_idamax 10 x 1))
8

Now we can use the handy #_ reader macro, and we no longer have to provide the return type of the function, or the types of its arguments. The #_ reader macro looks them up in the interface database and generates the correct external-call form automatically.

Lisp Arrays and Foreign Arrays

You may be wondering if there is some way to pass lisp arrays to foreign functions. In CCL, this is generally not possible. Even if it were possible to obtain a pointer to the array data, and assuming that the lisp array data is stored in the format that the foreign function expects, the GC might run at any time and move the underlying object. (Keep in mind that CCL uses multiple threads, and any thread might initiate a GC at arbitrary times.)

Data in a lisp array must therefore be copied to memory either on the stack or on the foreign heap before being passed to a foreign function.

There are, however, a few features in CCL that may be useful for avoiding the need to copy.

The first is an object called a heap ivector.  http://ccl.clozure.com/ccl-documentation.html#f_make-heap-ivector

Here's an example. (Presumably you would keep the heap ivector object around somewhere, and free it when you were completely done with it.)

(multiple-value-bind (vector ptr)
    (ccl:make-heap-ivector 10 'double-float)
  (setf (aref vector 5) 100d0
	(aref vector 8) -1000d0)
  (prog1
      (#_cblas_idamax 10 ptr 1)
    (ccl:dispose-heap-ivector vector)))

The second is a macro called ccl:with-pointer-to-ivector. See the docstring. This must be used with some care, because it inhibits the GC within its body.

(let ((vector (make-array 10 :element-type 'double-float)))
  (setf (aref vector 5) 100d0
	(aref vector 8) -1000d0)
  (ccl:with-pointer-to-ivector (ptr vector)
    (#_cblas_idamax 10 ptr 1)))

The nice thing about the above two examples is that you can still access the vector in the normal way.

In some earlier examples, we used ccl:rlet to allocate a vector on the stack. We can also allocate memory on the foreign heap either with #_malloc or with the slightly higher-level ccl:make-record. In either case, you must explicitly call ccl:free.

(let ((x (ccl:make-record (:array :double 10))))
  (setf (ccl:paref x (:array :double) 5) 100d0
        (ccl:paref x (:array :double) 8) -1000d0)
  (prog1
      (#_cblas_idamax 10 x 1)
    (ccl:free x)))

It's also possible to use ccl:make-gcable-record to allocate foreign memory that will be freed when it is unreachable from lisp.

(let ((x (ccl:make-gcable-record (:array :double 10))))
  (setf (paref x (:array :double) 5) 100d0
        (paref x (:array :double) 8) -1000d0)
  (#_cblas_idamax 10 x 1))

This works well as long as the gcable pointer is the only thing (either in the lisp world or foreign world) that refers to the memory. See the manual for a more detailed explanation:  http://ccl.clozure.com/ccl-documentation.html#f_make-gcable-record