wiki:MemoryUtils

Version 4 (modified by gz, 4 years ago) (diff)

--

Memory utilities

This section describes functions that could be useful in tracking down memory problems.

Heap objects

[Function]
map-heap-objects function &key area

Calls function on all objects in the heap. If area is specified, it looks only in that area of the heap. area can be one of :readonly, :static, :managed-static, or :dynamic, or a list of areas to look in multiple areas. If area is nil or not specified, looks in all the areas.

function should try to avoid allocating memory. While a little bit of consing may be ok, excessive consing leading to garbage collection will likely cause problems.

[Function]
find-referencers object &optional area

Returns a list of all objects in the heap that contain a direct reference to object. area is as for map-heap-objects.

[Function]
transitive-referencers list-of-objects &key area (verbose t)

Finds all transitive referencers to any of the objects in list-of-objects, i.e. all objects have a path to an object in list-of-objects. Returns a hash table with the referencers as keys and t as value. area may be specified as in find-referencers to limit the referencers to particular areas of the heap. If verbose is true, the default, prints short progress information to *standard-output*.

[Function]
heap-utilization &key (stream *debug-io*) (gc-first t) (area nil) (unit nil) (sort :size) (classes nil) (start nil) (threshold (and classes 0.0005))

heap-utilization walks the lisp heap, collects information about the objects stored on the heap, and prints a report of the results to stream. It shows the number of objects of each type, the logical size (i.e. the size of the data part of the object) and the physical size (the logical size plus any header and padding bytes).

If gc-first is true (the default), does a full gc before scanning the heap.

If classes is true, classifies objects by class rather than just basic type.

area can be used to restrict the walk to one memory area or a list of areas. Some possible values are :dynamic, :static, :managed-static, :readonly. By default, all areas (including stacks) are examined.

sort can be one of :count, :logical-size, or :physical-size to sort output by count or size.

unit can be one of :kb :mb or :gb to show sizes in units other than bytes.

If start is non-nil, it should be an object returned by get-allocation-sentinel; only objects at higher address are scanned (i.e. roughly, only objects allocated after it).

If threshold is non-nil, it should be a number between 0 and 1. All types whose share of the heap is less than threshold will be lumped together in an "All Others" line rather than being listed individually.

Memory maps

[Function]
parse-proc-maps &optional pid

This function is only available on Linux. It returns a list of currently mapped memory sections in the unix process with id pid, which defaults to the current process. Each memory section is represented by a list of 6 elements:

(address perms name section-size unmodified-size modified-size)

where address is the starting address of the section, name is usually the name of the file the section is mapped from, and perms is a 4-character string representing permissions:

  • first character is "r" if readable, "-" otherwise
  • second character is "w" if writable, "-" otherwise
  • third character is "x" if executable, "-" otherwise
  • fourth character is "s" if sharable, "p" if private (copy on write)

section-size is the total amount of address space assigned to the section, unmodified-size is the total size of unmodified (clean) pages in the section, and modified-size is the total size of modified (dirty) pages. (unmodified-size and modified-size may be nil if the operating system doesn't provide this information).

Note that unmodified pages mapped from a file are shared among all processes using them, regardless of the sharable/private permission setting. The permission setting only affects what happens when a page is modified: in sharable sections, any changes are visible to all processes, whereas private sections copy the page on write.

A permission string of "---p" indicates a range of reserved address space using no system resources.

[Function]
proc-maps-diff map1 map2

Compares two lists as returned by parse-proc-maps and returns a list of all changes, in the form of (map1-entry map2-entry) where map1-entry and map2-entry represent the same section (the same start address, permissions and filename) with different size values. map1-entry can be nil indicating a new entry added in map2, and map2-entry can be nil indicating an entry removed in map2.

Examining core files

CCL has a suite of functions for examining core dump files of CCL applications. Currently these functions are only available for x8664 Linux (porting them to another architecture would be a fairly straightfoward learn-CCL-internals project).

A core dump file contains a copy of the Lisp heap of a process. You can open a core file with open-core, and get a summary of the heap contents with core-heap-utilization or idom-heap-utilization. Depending on how the core dump was obtained, you might also be able to examine the stack with core-print-call-history. If this is not enough, there are many functions available that let you manually examine individual objects, map over symbols, etc. Core file objects are represented as integers, encoding the address and the tag of the object, and there are functions available to decode their types and contents.

[Function]
open-core pathname &key image (method :mmap)

Opens a core dump file contained in pathname and establishes it as the current core file used by all the subsequent functions. If image is specified, it should be the name of the CCL image file corresponding to the core file. This is used to get information (e.g. builtin symbol pnames) from read-only sections which are not present in the core file. method determines how the core file is accessed, it can be :mmap or :stream.

[Function]
close-core

Closes the current core file.

[Function]
core-heap-utilization &key (stream *debug-io*) area unit (sort :size) classes (threshold 0.00005)

Examines the lisp heap stored in the core file and reports the types of objects in it. The arguments are as for heap-utilization above, except area can also be :tenured to restrict the computation to just the tenured part of the dynamic heap.

[Function]
idom-heap-utilization &key unit (sort :size) (threshold 0.01) (area :tenured)

This function analyzes the core file heap based on dominance. It takes a while to compute, but often gives more meaningful results than the regular heap utilization report, especially when looking at memory leaks.

An object A is said to dominate another object B if all paths from gc roots to B go through A. A is said to immediately dominate B if A dominates B and every other object that dominates B also dominates A.

Since an object can only have one immediate dominator, all objects that are not dominators themselves can be partitioned into disjoint sets, each consisting of all objects with the same immediate dominator.

If B is in the partition owned by A, i.e. B is not a dominator of anything and is immediately dominated by A, then for all intents and purposes, B is just a part of A: B will be garbage collected whenever A is garbage collected, and there are no pointers directly to B from outside the partition. Nothing about the behavior of the heap would change if B was permanently appended to A instead of being a separate object.

idom-heap-utilization attributes the total size of all the partition members to the immediate dominator. I.e. it computes what core-heap-utilization would report if the heap was flattened by attaching all objects to their immediate dominators, and ignoring all types except the type of the immediate dominator. This tends to emphasize the types of objects that are really responsible for the memory usage.

[Function]
core-print-call-history &key (stream t) origin detailed-p

Attempts to find and describe the stack stored in the core file. The arguments are as for print-call-history.

[Function]
map-core-areas function &key area

Calls function on all objects in the core heap. function receives one argument, an integer representing a lisp object in the core file. area can be used to restrict to certain areas in the heap. It can the name of an area (:readonly, :static, :managed-static or :dynamic), or a list of area names, or the special name :tenured to only consider the tenured section.

[Function]
core-print core-object &optional (stream t) depth

Interpret core-object as an object in the core file, and output its printed representation to stream stream. depth controls how deeply to descend.

[Function]
core-listp core-object

True if core-object is a list

[Function]
core-nullp core-object

True if core-object is null.

[Function]
core-consp core-object

True if core-object is a cons.

[Function]
core-car core-object

core-object must be a cons. Returns its car.

[Function]
core-cdr core-object

core-object must be a cons. Returns its cdr.

[Function]
core-symbolp core-object

True if core-object is a symbol

[Function]
core-functionp core-object

True if core-object is a function

[Function]
core-uvector-p core-object

True if core-object is a uvector.

[Function]
core-uvtype core-object

core-object must be a uvector. Returns a symbol representing core-object's internal vector type (internal types are the types shown in heap-utilization, e.g. ccl::simple-unsigned-word-vector).

[Macro]
core-uvtypep core-object type

Returns true if core-object is a uvector of internal vector type type (internal types are the types shown in heap-utilization, e.g. ccl::simple-unsigned-word-vector).

[Function]
core-uvsize core-object

core-object must be a uvector. Returns its size (number of elements).

[Function]
core-uvref core-object index

core-object must be a uvector. Returns its indexth element.

[Function]
core-object-typecode-type core-object

Returns a symbol representing the internal type of core-object. The same as core-uvtype if core-object is a uvector.

[Function]
core-object-type-key core-object

Returns either a symbol (for built-in types) or a core object representing a symbol that names the type of core object. In either case, the value returned is suitable for use in an eql hash table; use core-type-string to get a consistent printable representation.

[Function]
core-type-string type-key

Given a value returned by core-object-type-key, returns the type name as a string.

[Function]
copy-from-core core-object &key (depth 1)

Attempts to create a lisp copy of the core object core-object, for example if core-object is a string in the core heap, returns a string with the same characters. depth limits how far to descend. At depth 0, only immediate values such as characters and fixnums are copied.

[Function]
core-list core-object

core-object should be a list. Copies the top-level list, returning a list of core objects.

[Function]
core-find-package name &key error

Returns the core object representing a package named name in the core heap. name should be a string or symbol, or a core object which is a string or symbol. If there is no such package in the core heap, returns nil if error is null, else signals an error.

[Function]
core-keyword-package

Returns the address of the keyword package in the core heap. Equivalent to but more efficient than (core-find-package :keyword).

[Function]
core-find-symbol name &optional package

Returns the core object representing the symbol named name in package package in the core heap. name can be a symbol, string, or a core object representing a symbol or string. package can be a symbol, string, package, or core object representing a symbol, string or package. If name is a string or a string core object, then package is required, otherwise package defaults to the package of name.

[Function]
core-package-names core-object

Returns, as a list of strings, the name and nicknames of the core package core-object.

[Function]
core-package-name core-object

Returns, as a string, the name of the core package core-object. The same as (car (core-package-names core-object)) but without consing up the whole list.

[Function]
core-map-symbols function

Map function over all symbols in the core heap. function will be called with one argument, a symbol core object . function will be invoked once for each symbol in the heap, including keywords and uninterned symbols.

[Function]
core-symbol-name core-object

Returns, as a string, the name of the core symbol core-object.

[Function]
core-symbol-value core-object

Returns the core object that is the global value of the core symbol core-object.

[Function]
core-symbol-packagecore-object

Returns the core object that is the package of the core symbol core-object.

[Function]
core-symbol-plist core-object

Returns the core object that is the plist of the core symbol core-object.

[Function]
core-gethash core-object core-hash

Looks up core-object in the core hash table object core-hash. Returns the value, a core object, if found, else nil.

[Function]
core-hash-table-count core-hash

Returns the hash-table-count of the core hash table object core-hash. Signals an error if the table was in the middle of being rehashed.

[Function]
core-lfun-name core-function

Returns the core object that is the contents of the name slot of the core function object core-function.

[Function]
core-lfun-bits core-function

Returns the lfun bits of the core function object core-function.

[Function]
core-nth-immediate core-function index

Returns the core object that is the contents of the indexth immediate slot of the core function object core-function.

[Function]
core-find-class name

name must be a symbol or a symbol core object. Returns a class core object named name, or nil if not found.

[Function]
core-instance-class core-instance

core-instance must be a CLOS instance (i.e. core-object-typecode-type 'ccl::instance). Returns the class core object which is the class of the instance.

[Function]
core-instance-p core-object core-class

Returns true if core-object is a instance of the core class core-class.

[Function]
core-string= core-string string

Returns true if the core string core-string and string are equal. Equivalent to but more efficient than (string= (copy-from-core core-string) string).

[Function]
core-all-processes

Returns a list of core objects representing all processes in the core heap.

[Function]
core-process-name core-process

Returns, as a string, the name of the process core object core-process.

[Function]
core-find-process-for-id lwp

Returns a process core object whose native thread id is lwp, or nil if not found.

[Function]
core-q address &optional offset

Return, as an integer, the pointer-sized contents of the core heap at address (+ address offset).

[Function]
core-l address &optional offset

Return the 32-bit contents of the core heap at address (+ address offset)

[Function]
core-w address &optional offset

Return the 16-bit contents of the core heap at address (+ address offset)

[Function]
core-b address &optional offset

Return the 8-bit contents of the core heap at address (+ address offset)