XML Sucks by Doug Hoyte

Glenn Reid, the inventor of encapsulated postscript and a renowned desktop publishing expert-turned XML heretic, distills it down to these memorable quotes on his site xmlsucks.com:

• XML is not extensible. XML has pre-defined syntax (begin/end tags with attributes that can be set within a tag). As such you can define any tags you want, and add any attributes you want, but that's not extensibility, it's in the original design.
• XML is not a markup language. A markup language is predicated on the idea that the markup is an exception in a river of text. That is, the markup is a departure from the state that existed at the time the markup was encountered. [...] The idea of markup is that you literally mark up a text, "circling" things, if you will, giving instructions to the typesetter (or parser, or other) that this snippet of text is to be treated somehow differently.
• My bet is that 8 times out of 10, XML is not the right format.

A useful provided read macro is #=. #= lets you create self-referential S-expressions. This allows you to do things like represent directed graphs and other interesting data structures with little or no effort.

But most importantly it allows you to serialise data without having to disassemble and reassemble an efficient in-memory data structure where large portions of the data are shared. Here is an example where the 2 lisp lists read in are distinct objects (not eq):

* (defvar not-shared '((t) (t)))

NOT-SHARED
* not-shared

((T) (T))
* (eq (first not-shared) (second not-shared))

NIL

But in the following example, with serialised data using the #= read macro, the 2 lists really are the same list:

* (defvar shared '(#1=(t) #1#))

SHARED
* shared

((T) (T))
* (eq (first shared) (second shared))

T

As another fun example, here's how you can print an infinite list by pointing the cdr of a cons to itself:

* (print '#1=(hello . #1#))

(HELLO HELLO HELLO HELLO HELLO HELLO HELLO HELLO HELLO HELLO HELLO HELLO HELLO
 HELLO HELLO HELLO HELLO HELLO HELLO HELLO HELLO HELLO HELLO HELLO HELLO HELLO
 HELLO HELLO HELLO HELLO HELLO HELLO HELLO HELLO HELLO HELLO HELLO HELLO HELLO
 HELLO HELLO HELLO HELLO HELLO HELLO HELLO HELLO HELLO HELLO HELLO HELLO HELLO
 ...

So unless you want that to happen, be sure you set *print-circle* when serialising data structures with cycles in them as in the following example:

* (let ((*print-circle* t))
    (print '#1=(hello . #1#))
    nil)

#1=(HELLO . #1#)
NIL

What's really interesting about the above example (and some others to come on this page) is the use of CL dynamic scope to "shadow" a global variable just for this one function invocation. This is a remarkably powerful feature of lisp: multiple different environments for reading/printing S-expressions can co-exist (at different times, but transparently) - each with its own unique "shadowed" values for global variables. Even read tables can be shadowed like this since the current read table is also stored in a dynamic scope variable (*readtable*).

Notice that you can't do this in Blub++.NET Enterprise Edition (Interesting thoughts about Scheme):

int global_var = 0;

function do_stuff_that_uses_global_var() {
  ...
}

function whatever() {
  int global_var = 1;
  do_stuff_that_uses_global_var(); // This function will still see global_var as 0!
}

Oh, and don't evaluate this. :)

(progn
  (defun ouch () #1=(progn #1#))
  (compile 'ouch))

Another handy read macro that comes built in with Common Lisp is the #. read-time eval macro. This lets you embed objects into S-expressions that can't be serialised but can be created with a bit of lisp code. One fun example is making S-expressions that become different values each time they're read:

* '(football-game
     (game-started-at #.(get-internal-real-time))
     (coin-flip #.(if (zerop (random 2)) 'heads 'tails)))

(FOOTBALL-GAME (GAME-STARTED-AT 187) (COIN-FLIP HEADS))
* '(football-game
     (game-started-at #.(get-internal-real-time))
     (coin-flip #.(if (zerop (random 2)) 'heads 'tails)))

(FOOTBALL-GAME (GAME-STARTED-AT 309) (COIN-FLIP TAILS))
* (equal * (eval +))

T

Notice that this is not the same as using quasiquote:

* `(football-game
     (game-started-at ,(get-internal-real-time))
     (coin-flip ,(if (zerop (random 2)) 'heads 'tails)))

(FOOTBALL-GAME (GAME-STARTED-AT 791) (COIN-FLIP HEADS))
* (equal * (eval +))

NIL ; unless you're really fast and lucky :)

Let's see this more clearly by stopping evaluation of these forms (in other words, let's quote them):

* ''(football-game
      (game-started-at #.(get-internal-real-time))
      (coin-flip #.(if (zerop (random 2)) 'heads 'tails)))

'(FOOTBALL-GAME (GAME-STARTED-AT 2207) (COIN-FLIP HEADS))
* '`(football-game
      (game-started-at ,(get-internal-real-time))
      (coin-flip ,(if (zerop (random 2)) 'heads 'tails)))

`(FOOTBALL-GAME (GAME-STARTED-AT ,(GET-INTERNAL-REAL-TIME))
  (COIN-FLIP ,(IF (ZEROP (RANDOM 2)) 'HEADS 'TAILS)))

Understand that quasiquote is itself a read macro but that it works differently from read-time evaluation. Quasiquote reads as code that, when evaluated, becomes the desired list. A form handled by #. really reads in as the desired list/object. We can see this more closely by turning off pretty printing for the quasiquote form above:

* (let ((*print-pretty*))
    (print
      '`(football-game
          (game-started-at ,(get-internal-real-time))
          (coin-flip ,(if (zerop (random 2)) 'heads 'tails))))
    nil)

(LISP::BACKQ-LIST
  (QUOTE FOOTBALL-GAME)
  (LISP::BACKQ-LIST
    (QUOTE GAME-STARTED-AT)
    (GET-INTERNAL-REAL-TIME))
  (LISP::BACKQ-LIST
    (QUOTE COIN-FLIP)
    (IF (ZEROP (RANDOM 2)) (QUOTE HEADS) (QUOTE TAILS))))
NIL

LISP::BACKQ-LIST is exactly the same as the function list except for its pretty printing behaviour.

Exercise: Where did the commas go in the above "ugly-printed" form? (Once you understand this, you understand quasiquote)

Now let's really get down and dirty and extend some S-expressions.

Like most languages, lisp normally delimits strings with the " (double quote) character and allows you to include " characters in the string by using the \ (backslash) character as an escape character. But sometimes we might want to include " characters in strings without going to the trouble of escaping them. Here is a handy S-expression extension:

(defun |#"-reader| (stream sub-char numarg)
  (declare (ignore sub-char numarg))
  (let ((chars))
    (do ((prev (read-char stream) curr)
         (curr (read-char stream) (read-char stream)))
        ((and (char= prev #\") (char= curr #\#)))
      (push prev chars))
    (coerce (nreverse chars) 'string)))

(set-dispatch-macro-character #\# #\" #'|#"-reader|)

The above extension adds a read macro called #" to the current read table. The read macro reads until it sees the two terminating characters "# . We can use it like so:

* #"string with " characters. look, no "escapes"."#

"string with \" characters. look, no \"escapes\"."

Another powerful feature of some read macros (though not #") is that they can be nested - an area XML is very weak in. See the #| nestable comment read macro for an example.

This extension is similar to the above but instead is modeled after Perl's >> operator which allows you to quote blocks of text up until a certain string is found in the output:

(defun |#>-reader| (stream sub-char numarg)
  (declare (ignore sub-char numarg))
  (let ((chars))
    (do ((curr (read-char stream)
               (read-char stream)))
        ((char= #\newline curr))
      (push curr chars))
    (let* ((pattern (nreverse chars))
           (pointer pattern)
           (output))
      (do ((curr (read-char stream)
                 (read-char stream)))
          ((null pointer))
        (push curr output)
        (setf pointer
              (if (char= (car pointer) curr)
                (cdr pointer)
                pattern))
        (if (null pointer)
          (return)))
      (coerce
        (nreverse
          (nthcdr (length pattern) output))
        'string))))

(set-dispatch-macro-character
  #\# #\> #'|#>-reader|)

Here is an example use:

* #>END
I can put anything here! ", \, "#, and ># are
no problem! The only thing that will terminate
the reading of this string is...END

"I can put anything here! \", \\, \"#, and ># are
no problem! The only thing that will terminate
the reading of this string is..."

1. Sam Steingold mentioned his excellent S-expression extension which lets you parse XML using the CL reader.
2. Bill Clementson has some interesting code.
3. Michal Czardybon wrote and told me about his data representation language Harpoon.
4. Interesting thoughts of Elliotte Rusty Harold on the future of XML: "I know I said that XML in software development was dead, but maybe a spark of life remains."