wiki:CCLDocOverview

CCLDoc Documentation system

Overview

CCLDoc is a system for creating lisp documentation. It uses s-expressions to represent document structure, markup, cross references, and contents. It has a small number of basic operators, supports macros for syntax extensions, and supports a simple syntax for embedding expressions in strings for added convenience.

All the symbols referenced in this section without an explicit package prefix are external in the :CCLDOC package.

A CCLDoc document definition is processed by the LOAD-DOCUMENT function to produce a DOCUMENT object. This object can then be processed to create other formats. Currently three output formats are supported: OUTPUT-DOCBOOK generates  docbook output, OUTPUT-TRACWIKI generates  Trac Wiki output, and OUTPUT-CCLDOC generates CCLDoc source output.

By having a small number of relatively high-level operators, CCLDoc allows the creation of consistently formatted documents even when written by many different people over extended periods of time. Additionally, the simple document model enables longevity by making it easy to upgrade and change the back-end tools used to implement the final formatting, without having to convert the document source

Source Syntax

A CCLDoc document is represented by an s-expression with the usual semantics: when the s-expression is a CONS, the CAR of each form is a CCLDoc operator and the CDR contains the arguments. The s-expression can also be a string in which case it has additional internal structure as described in String Syntax. Finally, for convenience, an uninterned symbol is treated the same as a string, providing an alternate syntax (namely #:| ... |) for entering text with many double-quotes.

CCLDoc operators are symbols. Their input package is ignored and they are converted to keywords during processing (so they can be entered with or without an initial colon, as you prefer). There is a set of predefined operators, documented here. You can define additional operators using DEF-EXPANDER.

Basic Operators

(:include-file filename &key in-package external-format) [ccldoc operator]

The file filename should consist of any number of lisp forms followed by CCLDoc forms. The lisp forms are processed as if by LOAD until the first CCLDoc form is encountered, after which all the remaining forms must be CCLDoc forms. The CCLDoc forms are included in the document replacing the include-file form itself.

The in-package parameter can be used to specify the package used to load the file, and is entirely equivalent to placing an in-package form at the start of the included file. external-format specifies the character encoding of the file

(:defsection title &rest subforms) [ccldoc operator]

A named, referenceable, block of text. A defsection can occur at top-level, in which case it defines the document; or it can occur directly within another defsection, in which case it defines a subsection; and finally it can occur directly within the body of a :DEFINITION. Sections can be referred to using the title, and, if necessary to disambiguate, the name of the containing section or definition -- see :REF.

(:index-section title) [ccldoc operator]

Placed at the top-level of the document to specify the title and location of the index

(:glossary-section title &rest glossentries) [ccldoc operator]

Placed at the top-level of the document to specify the title and location of the glossary. It may contain :GLOSSENTRY definitions, although note that :GLOSSENTRY definitions can actually occur anywhere in the document and will be placed in the glossary and sorted automatically

(:block title &rest subforms) [ccldoc operator]

Defines a block of text set off (indented) from the main text. The title can be NIL to create an anonymous block. Note however that even when titled, a block cannot be referred to by name from elsewhere in a document.

(:code-block &rest subforms) [ccldoc operator]

A block suitable for listing of code. A fixed width font will be used to display the content, and whitespace and linebreaks will be shown verbatim. Note however that markup is allowed within code blocks.

(:table title head-row &rest rows) [ccldoc operator]

Defines a table. Tables have a sequence of :ROW clauses, the first of which will be formatted specially as the head row. Tables can be referred to using the title, and if necessary to disambiguate, the name of the containing section -- see :REF.

(:glossentry term &rest subforms) [ccldoc operator]

A glossary definition. term is the term being defined, and the remaining clauses are the explanation of the text. A glossentry can appear anywhere in the source document, but it doesn't get displayed where it appears - all the glossentry forms get collected into the :GLOSSARY-SECTION. Glossary entries can be referred to from elsewhere in the document using the term -- see :REF.

(:definition (type name) signature summary &rest subforms) [ccldoc operator]

A definition block - unit of reference documentation describing a single global lisp name.

type is a symbol specifing the type of the definition, establishing a namespace in which name is interpreted. (As usual, the input package of the type symbol is ignored and the symbol is converted to a keyword during processing). The name can be any lisp object. Definitions can be referred to from elsewhere in the document using their type and name -- see :REF. names of the same type are compared using EQUALP.

signature is the syntactic synopsis of the definition.

Note

Someday I'd like the signature to be a lisp expression that will be automatically converted to text with consistent markup for parameters and values, but that is not implemented right now. For now it might be best for document consistency to simply avoid markup in the signature

summary is a concise one line summary of the functionality being defined.

The subforms provide a more detailed description.

The following type values are supported by default. Additional types can be defined via DEF-DEFINITION-TYPE.

:ccldoc
:hemlock-command
:hemlock-variable
:toplevel-command
:package
:reader-macro
:variable
:method
:lap-macro
:generic-function
:macro
:function
:condition
:class
:type

(:row &rest items) [ccldoc operator]

For use within :TABLE only, defines a single row in the table as a sequence of :ITEM expresssions

(:listing type &rest items) [ccldoc operator]

A listing of items specified by the items forms. type determines how the list will be formatted. type names are symbols whose input package is ignored and converted to keyword during processing. The following type values are supported:

:bullet
A list in which item is marked with a character such as a bullet
:number
A list in which each item is marked with a sequentially incremented label
:definition
A list in which each item consists of a term and an associated description. See :ITEM for the syntax of definition list items
:column
A simple undecorated column, typically of single words or phrases

(:para &rest subforms) [ccldoc operator]

A paragraph.

(:item &rest subforms) [ccldoc operator]

A single item in a :LISTING or :ROW expression. It cannot be used in any other context. In :definition :LISTING items (only), the term and the description should be separated by :=>, for example:

(item "An " (emphasis "important") " item" => "An item which is very important")

As usual, the input package of => is ignored, so it can be entered with or without the colon.

(:index &rest subforms) [ccldoc operator]

Request an index entry. The subforms are displayed in place, but in addition, the phrase is added to the document index with a link to this point.

(:clause &rest subforms) [ccldoc operator]

Much like the lisp PROGN operator, makes a single expressions from a sequence of them.

(:link url &rest subforms) [ccldoc operator]

Makes the text specified by subforms into a link to the url url. If no subforms is specified, the url becomes the text.

(:quote string) [ccldoc operator]

Specifies that string is to be used literally, without being interpreted as described in String Syntax.

(:markup type &rest subforms) [ccldoc operator]

Requests that markup as specified by type should be applied to subforms. The type is named by a symbol whose input package is ignored and converted to keyword during processing. The following types are supported:

:emphasis
Emphasized text, usually rendered in in italics or boldface
:code
Literal lisp code, usually rendered in fixed width font
:param
Parameter name
:sample
Content that should be replaced by a value
:system
Operating system construct, for example a shell command or environment variable

The difference between :param and :sample is subtle and perhaps not worth the trouble... But the intent is that :param be used for a name of the eventual value, whereas :sample be used for an example of such a value. For instance: "(compile-file " (:param "filename") ")"# vs "(compile-file " (:sample "/name/of/file") ")".

(:ref target &rest subforms) [ccldoc operator]

Makes the text specified by subforms into a link to another part of the document as indicated by target. If no subforms are specified, default text is generated based on the target. In addition, if the specified target is a glossary entry or a definition, this reference to it will be listed in the index.

The target name can take one of the following forms:

:document
Targets the document itself. You can't actually link to the document itself, so this is only useful as the parent-target of a section target
(:definition type name)
Targets a :DEFINITION block. type and name are as described in :DEFINITION. In addition, type may be :* (or any interned symbol whose pname is "*"), indicating any type.
(:glossentry term)
Targets a :GLOSSENTRY block defining the given term.
(:section title [:in parent-target ...])
Targets a :DEFSECTION block whose title is title. Optionally a sequence of immediate parents of the section can be named as well, for example (:section "Description" :in "Names and Definitions") might be used in a document containing different subsections named "Description". A fully-qualified unambiguous name of a section might look like (:section "Description" :in "Names and Definitions" :in "Usage" :in :document). The reversed sequence of parent titles can also be included in one string, separated by "::", with the toplevel document indicated by ".". So the equivalent to above would be (:section "Names and Definitions::Description") for the first example, or (:section ".::Usage::Names and Definitions::Descriptions") for the second.
(:table title [:in parent-target ...])
Targets a :TABLE. Syntax is the same as for :section.
:index or (:section :index)
Targets the index section
:glossary or (:section :glossary)
Targets the glossary section
(:chapter title)
Equivalent to (:section title :in :document)

If the document doesn't actual contain a block named by target, no link will be produced, but the clause will still be displayed and, in case of a glossary entry or definition, the reference will be listed in the index.

Convenience Operators

The following operators define more convenient ways of accessing the same functionality as the basic operators described in Basic Operators.

(:document title &rest sections) [ccldoc operator]

Equivalent to (:SECTION title . sections)

(:chapter title &rest sections) [ccldoc operator]

Depending on context, this either defines or references a chapter. When used at top-level of a document, it is equivalent to (:DEFSECTION title . sections). In other contexts, only one argument is allowed, and it is equivalent to (:REF (:chapter title))

(:section title &rest parent-spec) [ccldoc operator]

Equivalent to (:REF (:section title . parent-specs)).

(:variable symbol) [ccldoc operator]

Equivalent to (:REF (:definition :variable symbol))

(:function fspec) [ccldoc operator]

Equivalent to (:REF (:definition :function fspec))

(:macro symbol) [ccldoc operator]

Equivalent to (:REF (:definition :macro symbol))

(:class symbol) [ccldoc operator]

Equivalent to (:REF (:definition :class symbol))

(:type tspec) [ccldoc operator]

Equivalent to (:REF (:definition :type tspec))

(:refdef type name) [ccldoc operator]

Equivalent to (:REF (:definition type name))

(:term string) [ccldoc operator]

Equivalent to (:REF (:glossentry string))

(:emphasis &rest subforms) [ccldoc operator]

Equivalent to (:MARKUP :emphasis . subforms)

(:system &rest subforms) [ccldoc operator]

Equivalent to (:MARKUP :system . subforms)

(:sample &rest subforms) [ccldoc operator]

Equivalent to (:MARKUP :sample . subforms)

(:param &rest subforms) [ccldoc operator]

Equivalent to (:MARKUP :param . subforms)

(:code &rest subforms) [ccldoc operator]

Equivalent to (:MARKUP :code . subforms)

(:lbrace) [ccldoc operator]

Equivalent to (:quote "{")

(:rbrace) [ccldoc operator]

Equivalent to (:quote "}")

String Syntax

Strings are the terminal clauses in CCLDoc expressions and define the text to be output.

For convenience, strings support a syntax for embedding other clauses, in the form {operand arguments}. The parsing of the arguments is up to the operand, but in general consists of some operand-specific number of lisp expressions, with the rest of the arguments forming the body. The processing of the embedded syntax can be avoided by using the :QUOTE operator.

In order to support the embedded syntax, #!\{ and #!\} cannot be entered directly in an unquoted string clause. Use {lbrace} and {rbrace} instead.

Blank lines (i.e. lines consisting only of whitespace characters) in string clauses are treated as paragraph breaks.

API Documentation

(load-document filename &key external-format) => document [Function]

Load a CCLDoc source file and return a DOCUMENT object. The file filename should consist of any number of lisp forms followed by a single CCLDoc form which should be (or expand into) a :DEFSECTION expression. The lisp preceding forms are processed as if by LOAD.

(output-docbook doc filename &key external-format if-exists) [Function]

Takes a DOCUMENT object doc and creates a file in filename containing  docbook source.

(output-tracwiki doc filename &key external-format if-exists) [Function]

Takes a DOCUMENT object doc and creates a file in filename containing  Trac Wiki source.

(output-ccldoc doc filename &key external-format if-exists) [Function]

Takes a DOCUMENT object doc and creates a file filename containing CCLDoc source code, in effect disassembling the document. Note that this output is mainly useful for debugging, since all the macros will have been expanded.

(def-expander name arglist [:parser-types parser-types] &body body) [Macro]

Define a CCLDoc macro. The syntax of arglist and body is the same as for DEFMACRO. When the CCLDoc processor encounters a form whose CAR is an interned symbol with the same pname as name, the body of the expander will be invoked with the form destructured against arglist, and the returned form will be used instead.

The parser-types option can be used to specify the parsing if name is used with the embedded string syntax. It should be an (unquoted) list of some number of :lisp symbols followed by at most one :text symbol. The string parser will read as many arguments as there are instances of :lisp using the lisp READ function, and then the rest of the string will be parsed as CCLDoc forms. For example:

      (def-expander :REFDEF (type name)
        :parser-types (:lisp :lisp)
        `(ref (definition ,type ,name)))

allows the embedded syntax {refdef type name}, where type and name will be parsed using the lisp READ. This example:

      (def-expander my-markup (type &rest subforms)
        :parser-types (:lisp :text)
        `(markup ,type ,@subforms))

allows the embedded syntax {markup type Arbitrary text ...}, where type will be parsed using the lisp READ, and the rest of the form will be processed by CCLDoc and passed in as the subforms parameter to the expander.

(def-definition-type type (&optional parent-type) &key type-name id-prefix function) [Macro]

Adds a new type that can be used to classify name of a :DEFINITION.

type is a symbol naming the type being defined (as usual, the input package of the type symbol is ignored and the symbol is converted to a keyword during processing)

parent-type is, optionally, an existing type from which the new type descends. This is used to inherit some properties, and to help in matching references to their targets (so for example we know that a reference to (:definition :function trace) is applicable to (:definition :macro trace), because the type :macro descends from :function.

type-name the user-visible name of this type, as a string. If not specified, it defaults to the capitalized version of the type.

id-prefix is a prefix to use in generating external id's for definitions of this type. If not specified, it inherits from parent-type, ultimately defaulting to "x_".

function is a function of one argument which will be used to canonicalize names of this type. The returned values will be compared using EQUALP to determine whether the refer to the same object. If not specified, it inherits from parent-type, ultimately defaulting to #'identity.

DOCUMENT Object Model

This section describes the structure of the DOCUMENT object. You only need to read this section if you want to implement a new output format or write a document analysis tool.

A CCLDoc document is a tree of CLAUSE objects, rooted with a DOCUMENT object.

clause [Type]

The type clause is a union of three types:

STRING
a string represents text to be output to the document
CLAUSE-OBJECT
an instance of the CLAUSE-OBJECT class as described below
CONS
a list of STRING or CLAUSE-OBJECT objects (no nested lists), representing a sequence of text

Abstract Classes

clause-object () [Class]

The base class of CCLDoc objects. The following functions are applicable to all clause-object instances:

CLAUSE-PARENT
CLAUSE-DOCUMENT
SECTION-LEVEL
CLAUSE-TEXT

named-clause (clause-object) [Class]

The class of objects which can be linked to from other objects. The following functions are applicable to named-clause instances:

CLAUSE-NAME
CLAUSE-EXTERNAL-ID

clause-with-title (clause-object) [Class]

A mixin class that adds support for the CLAUSE-TITLE property

clause-with-term (clause-object) [Class]

A mixin class that adds support for the CLAUSE-TERM property

clause-with-body (clause-object) [Class]

A mixin class that adds support for the CLAUSE-BODY property

clause-with-items (clause-object) [Class]

A mixin class that adds support for the CLAUSE-ITEMS property

Concrete Classes

document (section) [Class]

Class of the top-level document

index-section (section) [Class]

Class of the index section. The index section has no body, since it's assumed that the backend will take care of producing the section. It contains the desired section title, and its position in the document indicates where the index chapter should go.

glossary-section (section) [Class]

Class of the glossary section. The body of the glossary section consists of GLOSSENTRY objects, sorted according to the term they define

section (named-clause clause-with-title clause-with-body) [Class]

Class of document sections

glossentry (named-clause clause-with-term clause-with-body) [Class]

Class of glossary entries.

code-block (clause-with-body) [Class]

Class of a block to be displayed as code with verbatim line breaks and spacing

block (clause-with-title clause-with-body) [Class]

Class of a block, with an optional title, to be rendered offset from the main text.

para (clause-with-body) [Class]

Class of a paragraph

docerror (clause-object) [Class]

Class of an error message generated during processing of the document source. The text of the error message is returned by CLAUSE-TEXT

link (clause-with-body) [Class]

Class of a link to a url target which is returned LINK-URL. The body of the clause is the text to be displayed. It is guaranteed non-empty (CCLDoc fills in a default if the user didn't provide it).

table (named-clause clause-with-title clause-with-items) [Class]

Class of a table. The items of a table are a vector of ROW objects for the table

row (clause-with-items) [Class]

Class of a table row. row objects occur only in the CLAUSE-ITEMS vector of a TABLE object. The items of a row are a vector of ITEM objects for the columns in the row

listing (clause-with-items) [Class]

Class of a listing. The listing type is returned by LISTING-TYPE. See :LISTING for the list of possible types.

The items of a listing are a vector a ITEM objects, except for :definition listing type, where the items are TERM-ITEM objects.

indexed-clause (clause-with-body) [Class]

Class of text to be both entered in document and added to index

markup (clause-with-body) [Class]

Class of text to have markup applied. The markup type is returned by MARKUP-TYPE. See :MARKUP for the list of possible types

item (clause-with-body) [Class]

Class of text which occurs as an item in a table ROW or a LISTING

item (clause-with-term clause-with-body) [Class]

Class of text which occurs only in LISTING objects of type :definition

xref (indexed-clause) [Class]

Class of a link to another object in the document. The function XREF-TARGET returns the object that is the target of the reference. The clause body is what the user specified as the text of the link. It may be NIL if the user didn't specify any text, in which case the function XREF-DEFAULT-BODY will return default text computed by CCLDoc.

definition (named-clause clause-with-body) [Class]

Class of a definition. The clause body is the full description of the definition. The function DEFINITION-SIGNATURE returns the signature of the definition and DEFINITION-SUMMARY returns the summary. DEFINITION-DISPLAY-NAME returns a user-visible version of the definition name.

Accessor Functions

(clause-text clause) => string [Function]

Returns the text of the clause with all markup removed.

(section-level clause-object) => integer [Function]

Returns the level of the section containing clause-object. The document is level 0

(clause-document clause-object) => document [Function]

Returns the document containing the clause-object

(clause-parent clause-object) => clause-object-or-nil [Function]

Returns the clause containing clause-object, or NIL if clause-object is the document

(clause-name named-clause) => name [Function]

Returns the name of the named-clause. The name is a lisp object uniquely identifying the clause when compared with EQUALP. When named-clause is a GLOSSENTRY, the name is a string, the text of the term being defined by the entry. When named-clause is a SECTION, the name is a CONS whose CAR is the title of the section and whose CDR is the clause-name of the section's parent clause. When named-clause is a DEFINITION, the name is a DSPEC as described in Definition names.

(clause-external-id named-clause) => string [Function]

Returns a string uniquely identifying the named-clause. The string starts with a letter and contains only letters, numbers, underscores, dashes, and periods

(clause-title clause-with-title) => string-or-nil [Function]

Returns the clause title. When clause-with-title is a NAMED-CLAUSE, the title is required and hence the value will not be nil. When clause-with-title is a BLOCK, the title may be nil.

(clause-term clause-with-term) => clause-or-nil [Function]

Returns the term clause

(clause-body clause-with-body) => clause-or-nil [Function]

Returns the body clause

(clause-items clause-with-items) => vector [Function]

Returns a vector. When clause-with-items is a TABLE, the returned vector is a vector of ROW objects. When clause-with-items is a ROW, or it is a LISTING of type other than :definition, then the vector is a vector of ITEM objects. When clause-with-items is a LISTING of type :definition, the vector is a vector of TERM-ITEM objects.

(markup-type markup) => markup-type [Function]

Returns the markup type. See :MARKUP operator for the list of possible types

(listing-type listing) => listing-type [Function]

Returns the listing type. See :LISTING operator for the list of possible types

(definition-display-name definition) => clause [Function]

Returns a user-visible version of the definition name.

(definition-summary definition) => clause [Function]

Returns the definition summary

(definition-signature definition) => clause [Function]

Returns the definition signature

(link-url link) => string [Function]

Returns the url that is the target of this link

(xref-target xref) => named-object [Function]

Returns the target of the reference

(xref-default-body link) => clause [Function]

Returns the default body for the reference that can be used when the user didn't specify one

Definition Names

dspec () [Class]

The type of the value returned by CLAUSE-NAME for DEFINITION objects. Encodes the name and type of the definition

(dspec-type dspec) => definition-type [Function]

The type of the definition named by dspec. See :DEFINITION for a discussion of definition types

(dspec-name dspec) => definition-name [Function]

The name of the definition named by dspec. See :DEFINITION for a discussion of definition names

(dspec-type-name dspec) => string [Function]

A user-visible version of the definition type