Nicolas Martyanoff – Brain dump About

Interactive Common Lisp development

Common Lisp programming is often presented as “interactive”. In most languages, modifications to your program are applied by recompiling it and restarting it. In contrast, Common Lisp lets you incrementally modify your program while it is running.

While this approach is convenient, especially for exploratory programming, it also means that the state of your program during execution does not always reflect the source code. You do not just define new constructs: you look them up, inspect them, modify them or delete them. I had to learn a lot of subtleties the hard way. This article is a compendium of information related to the interactive nature of Common Lisp.

Variables

In Common Lisp variables are identified by symbols. Evaluating (SETQ A 42) creates or updates a variable with the integer 42 as value, and associates it to the A symbol. After the call to SETQ, (BOUNDP 'A) will return T and (SYMBOL-VALUE 'A) will return 42.

You do not delete a variable: instead, you remove the association between the symbol and the variable. You do so with MAKUNBOUND. Following the previous example, (MAKUNBOUND 'A) will remove the association between the A symbol and the variable. And (BOUNDP 'A) returns NIL as expected. As for (SYMBOL-VALUE 'A), it now signals an UNBOUND-VARIABLE error as mandated by the standard.

What about DEFVAR and DEFPARAMETER? They are also used to declare variables (globally defined ones), associating them with symbols. Both define “special” variables (i.e. variables for which all bindings are dynamic; see CLtL2 9.2). The difference is that the initial value passed to DEFVAR is not evaluated if it already has a value. MAKUNBOUND will work on variables declared with DEFVAR or DEFPARAMETER as expected.

DEFCONSTANT is a bit more complicated. CLtL21 5.3.2 states that “once a name has been declared by defconstant to be constant, any further assignment to or binding of that special variable is an error”, but does not clearly define whether MAKUNBOUND should or should not be able to be used on constants. However, CLtL2 5.3.2 also states that “defconstant […] does assert that the value of the variable name is fixed and does license the compiler to build assumptions about the value into programs being compiled”. If the compiler is allowed to rely on the value associated with the variable name, it would make sense not to allow the deletion of the binding. Thus it is recommended to only use constants for values that are guaranteed to never change, e.g. mathematical constants. Most of the time you want DEFPARAMETER.

Note that MAKUNBOUND does not apply to lexical variables.

Functions

Common Lisp is a Lisp-2, meaning that variables and functions are part of two separate namespaces. Despite this clear separation, functions behave similarly to variables.

Using DEFUN will either create or update the global function associated with a symbol. SYMBOL-FUNCTION returns the globally defined function associated with a symbol, and FMAKUNBOUND deletes this association.

Let us point out a common mistake when referencing functions: (QUOTE F) (abbreviated as 'F) yields a symbol while (FUNCTION F) (abbreviated as #'F) yields a function. The function argument of FUNCALL and APPLY can be either a symbol or a function (see CLtL2 7.3) It has two consequences:

First, one can write a function referencing F as (QUOTE F) with the expectation that F will later be bound to a function. The following function definition is perfectly valid even though F has not been defined yet:

(defun foo (a b)
  (funcall 'f a b))

Second, redefining the F function will update its association (or binding) to the F symbol, but the previous function will still be available if it has been referenced somewhere before the update. For example:

(setf (symbol-function 'foo) #'1+)
(let ((old-foo #'foo))
  (setf (symbol-function 'foo) #'1-)
  (funcall old-foo 42))

What about macros? Since macros are a specific kind of functions (CLtL2 5.1.4 “a macro is essentially a function from forms to forms”), it is not surprising that they share the same namespace and can be manipulated in the same way as functions with FBOUNDP, SYMBOL-FUNCTION and FMAKUNBOUND.

Symbols and packages

While functions and variables are familiar concepts to developers, Common Lisp symbols and packages are a bit more peculiar.

A symbol is interned when it is part of a package. The most explicit way to create an interned symbol is to use INTERN, e.g. (INTERN "FOO"). INTERN interns the symbol in the current package by default, but one can pass a package as second argument. After that, (FIND-SYMBOL "FOO") will return our interned symbol as expected.

More surprising, the reader automatically interns symbols. You can test it by evaluating (READ-FROM-STRING "BAR"). After evaluation, BAR is a symbol interned in the current package. This also means that it is very easy to pollute a package with symbols in ways you did not necessarily expect. To clean up, simply use UNINTERN. Remember to refer to the right symbol: to remove the symbol BAR from the package FOO, use (UNINTERN 'FOO::BAR "BAR").

A symbol is either internal or external. EXPORT will make a symbol external to its package while UNEXPORT will make it internal. As for UNINTERN, confusion usually arises around which symbol is affected. (UNEXPORT 'FOO:BAR "FOO") correctly refers to the external symbol in the FOO package and makes it internal again. (UNEXPORT 'BAR "FOO") will signal an error since the BAR symbol is not part of the FOO package (unless of course the current package happens to be FOO).

Packages themselves can be created with MAKE-PACKAGE and destroyed with DELETE-PACKAGE. Developers are usually more familiar with DEFPACKAGE, a macro allowing the creation of a package and its configuration (package use list, imported and exported symbols, etc.) in a declarative way. A surprising and frustrating behavior is that evaluating a DEFPACKAGE form for a package that already exists will result in undefined behavior if the new declaration “is not consistent” (CLtL2 11.7) with the current state of the package. As an example, adding symbols to the export list is perfectly fine. Removing one will result in undefined behavior (usually an error) due to the inconsistency of the export list. Fortunately, Common Lisp offers all the necessary functions to manipulate packages and their symbols: use them!

Classes

The Common Lisp standard includes CLOS, the Common Lisp Object System. Unsurprisingly it provides multiple ways to interact with classes and objects dynamically.

As variables or functions, classes are identified by symbols and FIND-CLASS returns the class associated with a symbol. Class names are part of a separate namespace shared with structures and types.

The DEFCLASS macro is the only way to define or redefine a class. Redefining a class means that instances created afterward with MAKE-INSTANCE will use the new definition. Existing instances are updated: newly added slots are added (either unbound or using the value associated with :INITFORM) and slots that are not defined anymore are deleted. UPDATE-INSTANCE-FOR-REDEFINED-CLASS is particularly interesting: developers can define methods for this generic function in order to control how instances are updated when their class is redefined.

Defining classes may imply implicitly defining methods: the :ACCESSOR, :READER and :WRITER slot keyword arguments will lead to the creation of generic functions. When a class is redefined, methods associated with slots that have been removed will live on.

A limitation of CLOS is that classes cannot be deleted. FIND-CLASS can be used as a place, and (SETF (FIND-CLASS 'FOO) NIL) will remove the association between the FOO symbol and the class, but the class itself and its instances will not disappear. While this limitation may seem strange, ask yourself how an implementation should handle instances of a class that has been deleted.

The class of an instance can be changed with CHANGE-CLASS: slots that exist in the new class will be conserved while those that do not are deleted. New slots are either unbound or set to the value associated with :INITFORM in the new class. In a way similar to UPDATE-INSTANCE-FOR-REDEFINED-CLASS, UPDATE-INSTANCE-FOR-DIFFERENT-CLASS lets developers control precisely the process.

Generics and methods

Generics are functions which can be specialized based on the class (and not type as one could expect) of their arguments and which can have a method combination type.

Generics can be created explicitly with DEFGENERIC or implicitly when DEFMETHOD is called and the list of parameter specializers and method combination does not match any existing generic function. Since generics are functions, FBOUNDP, SYMBOL-FUNCTION and FMAKUNBOUND will work as expected.

Methods themselves are either defined as part of the DEFGENERIC call or separately with DEFMETHOD. Discovering the different methods associated with a generic function is a bit more complicated. There is no standard way to list the methods associated with a generic, but it is at least possible to look up a method with FIND-METHOD. Do remember to pass a function (and not a symbol) as the generic, and to pass classes (and not symbols naming classes) in the list of specializers.

Redefinition is not as obvious as for non-generic functions. When redefining a generic with DEFGENERIC all methods defined as part of the previous DEFGENERIC form are removed and methods defined in the redefinition are added. However, methods defined separately with DEFMETHOD are not affected.

For example, in the following code, the second call to DEFGENERIC will replace the two methods specialized on INTEGER and FLOAT respectively by a single one specialized on a STREAM, but the method specialized on STRING will remain unaffected.

(defgeneric foo (a)
  (:method ((a integer))
    (format nil "~A is an integer" a))
  (:method ((a float))
    (format nil "~A is a float" a)))

(defmethod foo ((a string))
  (format nil "~S is a string" a))

(defgeneric foo (a)
  (:method ((a stream))
    (format nil "~A is a stream" a)))

Note that trying to redefine a generic with a different parameter lambda list will cause the removal of all previously defined methods since none of them can match the new parameters.

Removing a method will require you to find it first using FIND-METHOD and then use REMOVE-METHOD. With the previous example, removing the method specialized on a STRING argument is done with:

(remove-method #'foo (find-method #'foo nil (list (find-class 'string)) nil))

Working with methods is not always easy, and two errors are very common.

First, remember that changing the combinator in a DEFMETHOD will define a new method. If you realize that your :AFTER method should use :AROUND and reevaluate the DEFMETHOD form, remember to delete the method with the :AFTER combinator or you will end up with two methods being called.

Second, when defining a method for a generic from another package, remember to correctly refer to the generic. If you want to define a method on the BAR generic from package FOO, use (DEFMETHOD FOO:BAR (…) …) and not (DEFMETHOD BAR (…) …). In the latter case, you will define a new BAR generic in the current package.

Meta Object Protocol

While CLOS is already quite powerful, various interactions are impossible. One cannot create classes or methods programmatically, introspect classes or instances for example to list their slots or obtain all their superclasses, or list all the methods associated with a generic function.

In addition of an example of a CLOS implementation, The Art of the Metaobject Protocol2 defines multiple extensions to CLOS including metaclasses, metaobjects, dynamic class and generic creation, class introspection and much more.

Most Common Lisp implementations implement at least part of these extensions, usually abbreviated as “MOP”, for “MetaObject Protocol”. The well-known closer-mop system can be used as a compatibility layer for multiple implementations.

Structures

Structures are record constructs defined with DEFSTRUCT. At a glance they may seem very similar to classes, but they have a fundamental limitation: the results of redefining a structure are undefined (CLtL2 19.2).

While this property allows implementations to handle structures in a more efficient way than classes, it makes structures unsuitable for incremental development. As such, they should only be used as a last resort, when a regular class has been proved to be a performance bottleneck.

Conditions

While conditions look very similar to classes the Common Lisp standard does not define them as classes. This is one of the few differences between the standard and CLtL2 which clearly states in 29.3.4 that “Common Lisp condition types are in fact CLOS classes, and condition objects are ordinary CLOS objects”.

This is why one uses DEFINE-CONDITION instead of DEFCLASS and MAKE-CONDITION instead of MAKE-INSTANCE. This also means that one should not use slot-related functions (including the very useful WITH-SLOTS macro) with conditions.

In practice, most modern implementations follow CLtL2 and the CLOS-CONDITIONS:INTEGRATE X3J13 Cleanup Issue and implement conditions as CLOS classes, meaning that conditions can be manipulated and redefined as any other classes. And the same way as any other classes, they cannot be deleted.

Types

Types are identified by symbols and are part of the same namespace as classes (which should not be surprising since defining a class automatically defines a type with the same name).

Types are defined with DEFTYPE, but documentation is surprisingly silent on the effects of type redefinition. This can lead to interesting situations. On some implementations (e.g. SBCL and CCL), if a class slot is defined as having the type FOO, redefining FOO will not be taken into account and the type checking operation (which is not mandated by the standard) will use the previous definition of the type. Infortunatly Common Lisp does not mandate any specific behavior on slot type mismatches (CLtL2 28.1.3.2).

Thus developers should not expect any useful effect from redefining types. Restarting the implementation after substantial type changes is probably best.

In the same vein interactions with types are very limited. You cannot find a type by its symbol or even check whether a type exists or not. Calling TYPE-OF on a value will return a type this value satisfies, but the nature of the type is implementation-dependent (CLtL2 4.9): it could be any supertype. In other words, TYPE-OF could absolutly return T for all values but NIL. At least SUBTYPE-P lets you check whether a type is a subtype of another type.

Going further

Common Lisp is a complex language with a lot of subtleties, way more than what can be covered in a blog post. The curious reader will probably skip the standard (not because you have to buy it, but because it is a low quality scan of a printed document and jump directly to CLtL2 or the Common Lisp HyperSpec. The Art of the Metaobject Protocol is of course the normative reference for the CLOS extensions usually referred to as “MOP”.


  1. Guy L. Steele Jr. Common Lisp the Language, 2nd edition. 1990. ↩︎

  2. Gregor Kiczales, Jim des Rivieres and Daniel G. Bobrow. The Art of the Metaobject Protocol. 1991. ↩︎

Share the word!

Liked my article? Follow me on Twitter or on Mastodon to see what I'm up to.