Ticket #731 (closed defect: fixed)

Opened 4 years ago

Last modified 3 years ago

Lisp kernel terminated with segmentation fault

Reported by: llibra Owned by: gb
Priority: blocker Milestone:
Component: Runtime (threads, GC) Version: trunk
Keywords: Cc:

Description

The trunk version of Clozure CL kernel crashes on Ubuntu 10.04 ( Linux kernel 2.6.32-24.41).

% ./lx86cl
zsh: segmentation fault (core dumped)  ./lx86cl
% gdb ./lx86cl core
GNU gdb (GDB) 7.1-ubuntu
Copyright (C) 2010 Free Software Foundation, Inc.
... snip ...
Core was generated by `./lx86cl'.
Program terminated with signal 11, Segmentation fault.
#0  find_foreign_rsp (tcr=0x2ada70, handler=0x80634f0, signum=7, 
    info=0x2acc1c, context=0x2acc9c, return_address=9712656)
    at ../x86-exceptions.c:1549
1549	  if (((BytePtr)rsp < foreign_area->low) ||
(gdb) bt
#0  find_foreign_rsp (tcr=0x2ada70, handler=0x80634f0, signum=7, 
    info=0x2acc1c, context=0x2acc9c, return_address=9712656)
    at ../x86-exceptions.c:1549
#1  handle_signal_on_foreign_stack (tcr=0x2ada70, handler=0x80634f0, signum=7, 
    info=0x2acc1c, context=0x2acc9c, return_address=9712656)
    at ../x86-exceptions.c:1624
#2  0x08061cae in altstack_signal_handler (signum=7, info=0x2acc1c, 
    context=0x2acc9c) at ../x86-exceptions.c:1703
#3  <signal handler called>
#4  create_stack (size=1159168) at ../thread_manager.c:1611
#5  0x080592b2 in allocate_lisp_stack (useable=1048576, softsize=102400, 
    hardsize=4096, softkind=kVSPsoftguard, hardkind=kVSPhardguard, 
    h_p=0x2ad038, base_p=0x2ad03c, softp=0x2ad034, hardp=0x2ad030)
    at ../pmcl-kernel.c:223
#6  0x0805947c in allocate_lisp_stack_area (stack_type=AREA_VSTACK, 
    usable=1048576, softsize=102400, hardsize=4096, softkind=kVSPsoftguard, 
    hardkind=kVSPhardguard) at ../pmcl-kernel.c:279
#7  0x08059565 in allocate_vstack_holding_area_lock (usable=3674112)
    at ../pmcl-kernel.c:326
#8  0x0806529e in new_tcr (vstack_size=1048576, tstack_size=524288)
    at ../thread_manager.c:1342
#9  0x08065bb5 in lisp_thread_entry (param=0xbfcbe390)
    at ../thread_manager.c:1686
#10 0x0057e96e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#11 0x00d9aa4e in clone () from /lib/tls/i686/cmov/libc.so.6
(gdb) 

On  Linux kernel 2.6.32-23.37, it works fine.

Change History

comment:1 Changed 4 years ago by rme

This was noticed on #ccl a few days ago. See  http://ccl.clozure.com/irc-logs/ccl/2010-08/ccl-2010.08.20.txt.

<gbyers> create_stack tries to write to an address that mmap() returns; mmap returns an unmapped address, and wackiness ensues.

It was reported that a .35 kernel didn't have this problem.

comment:2 Changed 4 years ago by gb

  • Status changed from new to closed
  • Resolution set to wontfix
  • Component changed from Compiler to other

As Matt mentioned, someone reported this on the #ccl IRC channel a few days ago.

The short version is that in that version of the Linux kernel, mmap() returns an address that it's neglected to map (it's actually mapped a memory region starting at an address one 4K page greater than the address it returns). CCL tries to write to the (unmapped) returned address and dies spectacularly.

The original reporter noted that a later kernel (2.6.35 ?) didn't have this bug. I can't imagine that CCL is the only application affected by this, and would have expected a newer kernel to have been made available in the Ubuntu repositories by now. As of now (August 27 at about 20:38 UTC) it hasn't been.

I haven't tried too hard, but I haven't yet been able to write a small self-contained C program that demonstrates the problem; when I do, I'll attach it here.

(So, rather than "mmap returns unmapped addresses", the bug could be better described as "under circumstances that are somehow like those in effect during CCL kernel initialization, mmap returns an unmapped address." I doubt very seriously that anyone decided that mmap should behave this way, and there isn't a whole lot that userspace code can do to cause it to misbehave.)

If this problem persists for a long time or affects other distributions, I suppose that we might need to work around it. Otherwise, I'm not really enthused about the idea of intentionally mistrusting mmap's return value.

It seems that removing MAP_GROWSDOWN from the mmap options in the call in MapMemoryForStack() (in lisp-kernel/memory.c) avoids triggering the bug. That isn't too attractive either.

At least for the time being, I'd rather just say "don't use that kernel" and try to come up with a small test case that demonstrates the problem so that the Ubuntu or upstream kernel maintainers can be made aware of it if they aren't already.

comment:3 Changed 4 years ago by llibra

Thank you for your prompt replies. I understood very well.

comment:4 Changed 4 years ago by rigus

  • Priority changed from normal to blocker
  • Resolution wontfix deleted
  • Status changed from closed to reopened
  • Component changed from other to Runtime (threads, GC)

I am experiencing the same behavior with the newest Red Hat and CentOS releases, with lx86cl64, but not with lx86cl. Here is a backtrace of the core dump:


GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5_5.2) ... Reading symbols from /home/paul/lisp/ccl/lx86cl64...done. Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Core was generated by `./lx86cl64'. Program terminated with signal 11, Segmentation fault. #0 find_foreign_rsp (rsp=140733968059424, foreign_area=0x0, tcr=0x2ae8d780a6e0) at ../x86-exceptions.c:1436

warning: Source file is more recent than executable.

1436 if (((BytePtr?)rsp < foreign_area->low)

(gdb) bt #0 find_foreign_rsp (rsp=140733968059424, foreign_area=0x0, tcr=0x2ae8d780a6e0) at ../x86-exceptions.c:1436 #1 0x000000000041be5b in handle_signal_on_foreign_stack (tcr=0x2ae8d780a6e0, handler=0x41dbe0, signum=7, info=0x7fff2e2c8910,

context=0x7fff2e2c87e0, return_address=47179530656528) at ../x86-exceptions.c:1511

#2 <signal handler called> #3 create_stack (size=2412544) at ../thread_manager.c:1574 #4 0x00000000004143b5 in allocate_lisp_stack (useable=<value optimized out>, softsize=2412544, hardsize=7, softkind=4294967295,

hardkind=4294967295, h_p=0x0, base_p=0x7fff2e2c8cc0, softp=0x7fff2e2c8cb0, hardp=0x7fff2e2c8ca8) at ../pmcl-kernel.c:226

#5 0x00000000004145b9 in allocate_lisp_stack_area (stack_type=AREA_VSTACK, usable=0, softsize=102400, hardsize=4096,

softkind=4294967295, hardkind=4294967295) at ../pmcl-kernel.c:282

#6 0x00000000004201fa in new_tcr (vstack_size=2301952, tstack_size=262144) at ../thread_manager.c:1340 #7 0x0000000000414006 in main (argc=<value optimized out>, argv=0x302000000000, envp=<value optimized out>,

aux=<value optimized out>) at ../pmcl-kernel.c:1926

-- Paul

comment:5 Changed 4 years ago by gb

  • Status changed from reopened to closed
  • Resolution set to wontfix

To see if this is the same bug (I suspect that it is):

1) run the lisp kernel under GDB:

$ gdb lx86cl64

2) In gdb, set a breakpoint at MapMemoryForStack, then run until that breakpoint is reached:

(gdb) b MapMemoryForStack
Breakpoint 1 at 0x423340: file ../memory.c, line 293. [or similar output]
(gdb) r

3) At the breakpoint, tell GDB to run until function exit and note the return value:

(gdb) finish
[... other output ...]
Value returned is $1 = (void *) 0x7ffff7d94000
(gdb) 

I don't know that the address returned there is at all predictable; it should be page-aligned (end in 000 in hex).

4) While still at the last GDB prompt, note the process ID of the lisp. (You can do:

(gdb) call getpid()

if that's the easiest way to do it. In another shell, do:

$ cat /proc/<pid>/maps

where <pid> is the integer process ID of the CCL process.

The "maps" pseudofile is a human-readable representation of the process's mapped memory regions. A memory region should start at the address returned by MapMemoryForStack() and printed by GDB above. What I've seen is that that address is unmapped (though a mapped region starts at the returned address + 0x1000).

In other words, under some circumstances mmap() returns an unmapped address.

In other words, that's a (fairly new and very severe) bug in mmap().

On the platform that I can reproduce it on, removing MAP_GROWSDOWN from the options in the call to mmap() in MapMemoryForStack() seems to avoid the problem. MAP_GROWSDOWN is supposed to tell the memory system that pages in the allocated region will be referenced from high addresses to low (like a stack), and since we're allocating a stack here it's possible that that option could improve things on some systems. I don't know of anything else that CCL (or other affected programs) can do to work around what seems pretty clearly to be a Linux kernel bug.

I don't know for sure that you're seeing the same bug, but the backtrace certainly makes that appear to be the case; if you're able to try the experiment suggested here, that would confirm that.

I haven't tried recently, but my initial attempts to demonstrate the problem via a small C program weren't successful, but everything that's happened in CCL to this point is (fairly) simple C code, and it should be possible to file a bug report without using the L word.

comment:6 Changed 4 years ago by rigus

Gary, I did what you suggested. The return value of MapMemoryForStack?() was 0x2aaaaaacd000, and here is the relevant part of the «maps» pseudofile. It seems that the bug is the same: the returned address is equal to the offset (in «maps») and is not mapped, it is off by 0x1000.

2aaaaaaab000-2aaaaaaac000 rwxp 2aaaaaaab000 00:00 0 2aaaaaaca000-2aaaaaacd000 rwxp 2aaaaaaca000 00:00 0 2aaaaaace000-2aaaaad11000 rwxp 2aaaaaacd000 00:00 0 300000000000-3000004e5000 r-xp 0113e000 fd:00 21266435 /usr/local/acl/lisp/ccl/lx86cl64.image 3000004e5000-300040000000 ---p 3000004e5000 00:00 0 300040000000-30004113c000 rwxp 00002000 fd:00 21266435 /usr/local/acl/lisp/ccl/lx86cl64.image

comment:7 Changed 4 years ago by rigus

Sorry for the bad formatting. It should have been

2aaaaaaab000-2aaaaaaac000 rwxp 2aaaaaaab000 00:00 0 
2aaaaaaca000-2aaaaaacd000 rwxp 2aaaaaaca000 00:00 0 
2aaaaaace000-2aaaaad11000 rwxp 2aaaaaacd000 00:00 0 
300000000000-3000004e5000 r-xp 0113e000 fd:00 21266435                   /usr/local/acl/lisp/ccl/lx86cl64.image
3000004e5000-300040000000 ---p 3000004e5000 00:00 0 

comment:8 follow-up: ↓ 9 Changed 4 years ago by rme

I thought that MAP_GROWSDOWN was simply a hint, but on Linux (does MAP_GROWSDOWN appear on any other operating system?), it appears to do something more.

My understanding could be wrong, but it appears that a memory region mapped with MAP_GROWSDOWN will be automatically extended downwards as needed.

The (rejected) patch in  http://lwn.net/Articles/294001/ describes the issue in a comment.

If mmap returns an unmapped address, that would seem to be a kernel bug. On the other hand, it looks like MAP_GROWSDOWN might be doing more than what we want, and it might make sense to omit it anyway.

comment:9 in reply to: ↑ 8 Changed 4 years ago by gb

  • Status changed from closed to reopened
  • Resolution wontfix deleted

Replying to rme:

If mmap returns an unmapped address, that would seem to be a kernel bug. On the other hand, it looks like MAP_GROWSDOWN might be doing more than what we want, and it might make sense to omit it anyway.

Yes; I think that you're right on all points. Omitting MAP_GROWSDOWN wouldn't cause us to lose anything that we want, after all.

comment:10 Changed 4 years ago by gb

  • Owner set to gb
  • Status changed from reopened to new

comment:11 Changed 4 years ago by gb

(In [14233]) Don't use the (Linux-specific ?) MAP_GROWSDOWN mmap option when allocating stacks; it doesn't do what we thought it did and using it seems to trigger a bug in some 2.6.32 Linux kernels. See ticket:731, which this change might fix.

comment:12 Changed 4 years ago by vsedach

Can someone rebuild the Clozure kernel binaries in SVN? I'm at revision 14367 right now, and MapMemoryForStack? no longer has MAP_GROWSDOWN in the source, but lx86cl still segfaults with what appears to be the same problem (I did the gdb and /proc/x/maps check). Cross-compiling Clozure is a bit daunting for me.

PS - This bug still occurs with 2.6.35-22, at least in 32-bit mode (Ubuntu 10.10)

comment:13 Changed 4 years ago by rme

It's easy to rebuild the lisp kernel. It's just a plain C (and assembly language) program; no cross-compilation needed.

cd ccl/lisp-kernel/linuxx8632; make clean; make

Replace linuxx8632 with linuxx8664 for 64-bit.

comment:14 Changed 3 years ago by rme

  • Status changed from new to closed
  • Resolution set to fixed
Note: See TracTickets for help on using tickets.