[[PageOutline]]

== Memory utilities ==

This section describes functions that could be useful in tracking down
memory problems.

=== Heap objects ===

[Function][[BR]]
'''map-heap-objects''' `function &key area`

Calls `function` on all objects in the heap.  If `area` is specified,
it looks only in that area of the heap.  `area` can be one of
`:readonly`, `:static`, `:managed-static`, or `:dynamic`, or a list of
areas to look in multiple areas.  If `area` is `nil` or not specified,
looks in all the areas.

`function` should try to avoid allocating memory.  While a little bit
of consing may be ok, excessive consing leading to garbage collection
will likely cause problems.


[Function][[BR]]
'''find-referencers''' `object &optional area`

Returns a list of all objects in the heap that contain a direct
reference to `object`.  `area` is as for `map-heap-objects`.

[Function][[BR]]
'''transitive-referencers''' `list-of-objects &key area (verbose t)`

Finds all transitive referencers to any of the objects in
`list-of-objects`, i.e. all objects have a path to an object in
`list-of-objects`.  Returns a hash table with the referencers
as keys and `t` as value.  `area` may be specified as in `find-referencers`
to limit the referencers to particular areas of the heap.  If `verbose`
is true, the default, prints short progress information to `*standard-output*`.

[Function][[BR]]
'''object-direct-size''' `object`

Returns the size of `object` (in bytes), including any headers and 
alignment overhead.  Does not descend an object's components.

[Function][[BR]]
'''heap-utilization''' `&key` (''stream'' `*debug-io*`) (''gc-first'' `t`) (''area'' `nil`) (''unit'' `nil`) (''sort'' `:size`) (''classes'' `nil`) (''start'' `nil`) (''threshold'' `(and classes 0.0005)`)


`heap-utilization` walks the lisp heap, collects information about the objects stored on the heap, and prints a report of the results to ''stream''.  It shows the number of objects of each type, the sum of their logical sizes (i.e. the size of the data part of the object) and the sum of their physical sizes (i.e. total size as computed by `object-direct-size`).

If ''gc-first'' is true (the default), does a full gc before scanning the heap.

If ''classes'' is true, classifies objects by class rather than just basic type.

''area'' can be used to restrict the walk to one memory area or a list of areas.  Some possible values are `:dynamic`, `:static`, `:managed-static`, `:readonly`.  By default, all areas (including stacks) are examined.

''sort'' can be one of `:count`, `:logical-size`, or `:physical-size` to sort output by count or size.

''unit'' can be one of `:kb` `:mb` or `:gb` to show sizes in units other than bytes.

If ''start'' is non-nil, it should be an object returned by `get-allocation-sentinel`; only 
objects at higher address are scanned (i.e. roughly, only objects allocated after it).

If ''threshold'' is non-nil, it should be a number between 0 and 1.  All types whose share of the heap is less than ''threshold'' will be lumped together in an "All Others" line rather than being listed individually.


=== Memory maps ===

[Function][[BR]]
'''parse-proc-maps''' `&optional pid`

This function is only available on Linux.  It returns a list of
currently mapped memory sections in the unix process with id `pid`,
which defaults to the current process.  Each memory section is
represented by a list of 6 elements:
   `(address perms name section-size unmodified-size modified-size)`
where `address` is the starting address of the section, `name` is
usually the name of the file the section is mapped from, and `perms`
is a 4-character string representing permissions:

 * first character is `"r"` if readable, `"-"` otherwise
 * second character is `"w"` if writable, `"-"` otherwise
 * third character is `"x"` if executable, `"-"` otherwise
 * fourth character is `"s"` if sharable, `"p"` if private (copy on write)

`section-size` is the total amount of address space assigned to the section,
`unmodified-size` is the total size of unmodified (clean) pages in the
section, and `modified-size` is the total size of modified (dirty) pages.
(`unmodified-size` and `modified-size` may be `nil` if the operating system
doesn't provide this information).

Note that unmodified pages mapped from a file are shared among all
processes using them, regardless of the sharable/private permission
setting. The permission setting only affects what happens when a page
is modified: in sharable sections, any changes are visible to all
processes, whereas private sections copy the page on write.

A permission string of "---p" indicates a range of reserved address
space using no system resources.

[Function][[BR]]
'''proc-maps-diff''' `map1 map2`

Compares two lists as returned by `parse-proc-maps` and returns a list
of all changes, in the form of `(map1-entry map2-entry)` where
`map1-entry` and `map2-entry` represent the same section (the same
start address, permissions and filename) with different size values.
`map1-entry` can be `nil` indicating a new entry added in `map2`,
and `map2-entry` can be `nil` indicating an entry removed in `map2`.

=== Examining core files ===

CCL has a suite of functions for examining core dump files of CCL
applications.  Currently these functions are only available for x8664
Linux (porting them to another architecture would be a fairly
straightfoward learn-CCL-internals project).

A core dump file contains a copy of the Lisp heap of a process.  You
can open a core file with `open-core`, and get a summary of the heap
contents with `core-heap-utilization` or `idom-heap-utilization`.
Depending on how the core dump
was obtained, you might also be able to examine the stack with
`core-print-call-history`.  If this is not enough, there are
many functions available that let you manually examine individual objects,
map over symbols, etc.  Core file objects are represented as
integers, encoding the address and the tag of the object, and there
are functions available to decode their types and contents.

[Function][[BR]]
'''open-core''' `pathname &key image (method :mmap)`

Opens a core dump file contained in `pathname` and establishes it as
the current core file used by all the subsequent functions.  If
`image` is specified, it should be the name of the CCL image file
corresponding to the core file.  This is used to get information
(e.g. builtin symbol pnames) from read-only sections which are not
present in the core file.  `method` determines how the core file is
accessed, it can be `:mmap` or `:stream`.


[Function][[BR]]
'''close-core'''

Closes the current core file.


[Function][[BR]]
'''core-heap-utilization''' `&key (stream *debug-io*) area unit (sort :size) classes (threshold  0.00005)`

Examines the lisp heap stored in the core file and reports the types of objects in it.  The arguments are as for heap-utilization above, except `area` can also be `:tenured` to restrict the computation to just the tenured part of the dynamic heap.

[Function][[BR]]
'''idom-heap-utilization''' `&key unit (sort :size) (threshold 0.01) (area :tenured)`

This function analyzes the core file heap based on dominance.  It takes a while
to compute, but often gives more meaningful results than the regular heap
utilization report, especially when looking at memory leaks.

An object A is said to ''dominate'' another object B if all paths from gc roots
to B go through A.  A is said to ''immediately dominate'' B if A dominates B and
every other object that dominates B also dominates A.

Since an object can only have one immediate dominator, all objects that are not
dominators themselves can be partitioned into disjoint sets, each consisting of
all objects with the same immediate dominator.

If B is in the partition owned by A, i.e. B is not a dominator of anything and
is immediately dominated by A, then for all intents and purposes, B is just a
part of A: B will be garbage collected whenever A is garbage collected, and there
are no pointers directly to B from outside the partition.  Nothing about the behavior
of the heap would change if B was permanently appended to A instead of being a
separate object.

`idom-heap-utilization` attributes the total size of all the partition members
to the immediate dominator.  I.e. it computes what `core-heap-utilization` would report
if the heap was flattened by attaching all objects to their immediate dominators,
and ignoring all types except the type of the immediate dominator.  This
tends to emphasize the types of objects that are really responsible for the memory usage.


[Function][[BR]]
'''core-print-call-history''' `core-process &key (stream t) detailed-p`

Attempts to find and describe the stack for core process `core-process` stored in the core file.  The description is printed to the stream `stream`.  If `detailed-p` is true, lists the values stored in each stack frame.


[Function][[BR]]
'''map-core-areas''' `function &key area`

Calls `function` on all objects in the core heap.  `function` receives one argument, an integer representing a lisp object in the core file.   `area` can be used to restrict to certain areas in the heap.  It can
the name of an area (:readonly, :static, :managed-static or :dynamic), or a list of area names, or the special name `:tenured` to only consider the tenured section.


[Function][[BR]]
'''core-print''' `core-object &optional (stream t) depth`

Interpret `core-object` as an object in the core file, and output its printed
representation to stream `stream`.  `depth` controls how deeply to descend.


[Function][[BR]]
'''core-listp''' `core-object`

True if `core-object` is a list

[Function][[BR]]
'''core-nullp''' `core-object`

True if `core-object` is null.

[Function][[BR]]
'''core-consp''' `core-object`

True if `core-object` is a cons.

[Function][[BR]]
'''core-car''' `core-object`

`core-object` must be a cons.  Returns its car.

[Function][[BR]]
'''core-cdr''' `core-object`

`core-object` must be a cons.  Returns its cdr.

[Function][[BR]]
'''core-symbolp''' `core-object`

True if `core-object` is a symbol

[Function][[BR]]
'''core-functionp''' `core-object`

True if `core-object` is a function

[Function][[BR]]
'''core-uvector-p''' `core-object`

True if `core-object` is a uvector.

[Function][[BR]]
'''core-uvtype''' `core-object`

`core-object` must be a uvector.  Returns a symbol representing `core-object`'s internal vector type (internal
types are the types shown in `heap-utilization`, e.g. `ccl::simple-unsigned-word-vector`).

[Macro][[BR]]
'''core-uvtypep''' `core-object` `type`

Returns true if `core-object` is a uvector of internal vector type `type` (internal
types are the types shown in `heap-utilization`, e.g. `ccl::simple-unsigned-word-vector`).

[Function][[BR]]
'''core-uvsize''' `core-object`

`core-object` must be a uvector.  Returns its size (number of elements).

[Function][[BR]]
'''core-uvref''' `core-object` `index`

`core-object` must be a uvector.  Returns its `index`th element.

[Function][[BR]]
'''core-object-typecode-type''' `core-object`

Returns a symbol representing the internal type of `core-object`.  The
same as `core-uvtype` if `core-object` is a uvector.


[Function][[BR]]
'''core-object-type-key''' `core-object`

Returns either a symbol (for built-in types) or a core object representing
a symbol that names the type of `core object`.  In either case, the value
returned is suitable for use in an eql hash table; use `core-type-string`
to get a consistent printable representation.

[Function][[BR]]
'''core-type-string''' `type-key`

Given a value returned by `core-object-type-key`, returns the type
name as a string.

[Function][[BR]]
'''copy-from-core''' `core-object &key (depth 1)`

Attempts to create a lisp copy of the core object `core-object`, for
example if `core-object` is a string in the core heap, returns a string
with the same characters.  `depth` limits how far to descend.  At depth 0,
only immediate values such as characters and fixnums are copied.

[Function][[BR]]
'''core-list''' `core-object`

`core-object` should be a list.  Copies the top-level list, returning a list of core objects.


[Function][[BR]]
'''core-find-package''' `name &key error`

Returns the core object representing a package named `name` in the core heap.  `name`
should be a string or symbol, or a core object which is a string or symbol.
If there is no such package in the core heap, returns `nil` if `error` is null,
else signals an error.

[Function][[BR]]
'''core-keyword-package'''

Returns the address of the keyword package in the core heap.  Equivalent
to but more efficient than `(core-find-package :keyword)`.


[Function][[BR]]
'''core-find-symbol''' `name &optional package`

Returns the core object representing the symbol named `name` in
package `package` in the core heap.  `name` can be a symbol, string,
or a core object representing a symbol or string. `package` can be a
symbol, string, package, or core object representing a symbol, string or
package. If `name` is a string or a string core object, then
`package` is required, otherwise `package` defaults to the package
of `name`.

[Function][[BR]]
'''core-package-names''' `core-object`

Returns, as a list of strings, the name and nicknames of the core package `core-object`.

[Function][[BR]]
'''core-package-name''' `core-object`

Returns, as a string, the name of the core package `core-object`.  The same as `(car (core-package-names core-object))`
but without consing up the whole list.

[Function][[BR]]
'''core-map-symbols''' `function`

Map `function` over all symbols in the core heap.  `function` will be called with one argument, a
symbol core object .  `function` will be invoked once for each symbol in the heap, including
keywords and uninterned symbols.

[Function][[BR]]
'''core-symbol-name''' `core-object`

Returns, as a string, the name of the core symbol `core-object`.

[Function][[BR]]
'''core-symbol-value''' `core-object`

Returns the core object that is the global value of the core symbol `core-object`.

[Function][[BR]]
'''core-symbol-package'''`core-object`

Returns the core object that is the package of the core symbol `core-object`.

[Function][[BR]]
'''core-symbol-plist''' `core-object`

Returns the core object that is the plist of the core symbol `core-object`.


[Function][[BR]]
'''core-gethash''' `core-object` `core-hash`

Looks up `core-object` in the core hash table object `core-hash`.  Returns the value,
a core object, if found, else nil.


[Function][[BR]]
'''core-hash-table-count''' `core-hash`

Returns the hash-table-count of the core hash table object `core-hash`.  Signals an
error if the table was in the middle of being rehashed.

[Function][[BR]]
'''core-lfun-name''' `core-function`

Returns the core object that is the contents of the name slot of the core function object `core-function`.

[Function][[BR]]
'''core-lfun-bits''' `core-function`

Returns the lfun bits of the core function object `core-function`.


[Function][[BR]]
'''core-nth-immediate''' `core-function` `index`

Returns the core object that is the contents of the `index`th immediate slot of the core function object `core-function`.


[Function][[BR]]
'''core-find-class''' `name`

`name` must be a symbol or a symbol core object.  Returns a class core object named `name`, or nil if not found.


[Function][[BR]]
'''core-instance-class''' `core-instance`

`core-instance` must be a CLOS instance (i.e. `core-object-typecode-type` `'ccl::instance`).  Returns the class core object which is the class of the instance.


[Function][[BR]]
'''core-instance-p''' `core-object` `core-class`

Returns true if `core-object` is a instance of the core class `core-class`.


[Function][[BR]]
'''core-string=''' `core-string string`

Returns true if the core string `core-string` and `string` are equal.  Equivalent to but more efficient than
`(string= (copy-from-core core-string) string)`.


[Function][[BR]]
'''core-all-processes'''

Returns a list of core objects representing all processes in the core heap.

[Function][[BR]]
'''core-process-name''' `core-process`

Returns, as a string, the name of the process core object `core-process`.


[Function][[BR]]
'''core-find-process-for-id''' `lwp`

Returns a process core object whose native thread id is `lwp`, or nil if not found.


[Function][[BR]]
'''core-q''' `address &optional offset`

Return, as an integer, the pointer-sized contents of the core heap at address `(+ address offset)`.

[Function][[BR]]
'''core-l''' `address &optional offset`

Return the 32-bit contents of the core heap at address `(+ address offset)`

[Function][[BR]]
'''core-w''' `address &optional offset`

Return the 16-bit contents of the core heap at address `(+ address offset)`

[Function][[BR]]
'''core-b''' `address &optional offset`

Return the 8-bit contents of the core heap at address `(+ address offset)`