Not sure if I'd call it a "pearl", but here's one data point. Hell, I'm not even sure if this counts as a DSL or not, most of what I know of DSLs comes from blog posts and ruby fanatics.

I had a legacy electronic voting system (IIS/VBscript/SQL Server) and wrote some lisp to configure elections. The lisp code output the T-SQL needed to configure the election, and then the VBScript was driven from that data.

Here's an abbreviated example:

(with-election ("3/29/2011 8:00PM" "3/31/2011 7:00PM" "Law") (with-position ("President" :types '("Law Undergrad")) (make-candidates "name 1" "name 2" "name 3")) (make-referendum "Amendment 1" "<strong>Thing To Change</strong> <p>HTML body for the voting UI</p>"))

The :types indicated who was eligible to vote for the position, and there were some other options to those macros. The with-* forms are macros to setup the context, and the make-* functions generate the appropriate SQL strings and write them to a stream (usually a file).

The macros in the sample do a few things:

1. with-election: sets a dynamic variable in lisp for the stream all the SQL strings are written to in make-* functions, allowed easy testing of make-* functions via the REPL 2. with-position: sets a variable in the T-SQL script used by INSERT statements produced by make-candidates. This actually macroexpands to imperative code: (progn (make-position ...) (make-candidates ...)). The macro usage is mostly to get indentation and visually group the relationships

So, not sure if I'd call this a pearl and it's arguably a DSL, but it's sure a lot easier for me to use than direct T-SQL or the ridiculous VBScript web interface.

Other things I've used that might fall into the DSL category:

* cl-who http://weitz.de/cl-who/ * cl-interpol http://weitz.de/cl-interpol/ * clsql's SQL-READER syntax (http://clsql.b9.com/manual/ref-syntax.html)

HTH,

Ryan Davis Acceleration.net Director of Programming Services 2831 NW 41st street, suite B Gainesville, FL 32606

Office: 352-335-6500 x 124 Fax: 352-335-6506

On 7/20/2011 9:32 AM, Didier Verna wrote:

Dear friends,

I'm starting to write a chapter for an upcoming book on domain specific languages. The chapter is called (tentatively):

Extensible languages -- blurring the distinction between DSLs and GPLs

GPL meaning General Purpose Language in this context ;-)

My intention is to demonstrate how the task of implementing a DSL is made easier when it boils down to an extension or subset of your original GPL (hence reusing its infrastructure), instead of being a totally different language, only written on top of the other.

Obviously, I'm going to illustrate this with Common Lisp, and I intend to speak of dynamicity (not only dynamic typing, but in general all things that can be deferred to the run-time), introspection, intersession, structural or procedural reflexivity, meta-object protocols (not sure about this one), macro systems and JIT-compilation. Also, more specifically to Lisp, reader macros (compiler macros maybe?), the condition system (and its ability to *not* unwind) and restarts.

Right now, I would like to know if any of you have DSL "pearls", nice examples of DSLs that you have written in Lisp by using some of its features in a clever or elegant way. I would also gladly accept any point of view or comment on what's important to mention, in terms of design principle or anything else, things that I may have missed in the list above.

Thank you very much in advance!

It seems to me that most people don't understand the rudiments of CL's ability for creating DSL's and live in the land of Blub.

If you're going to discuss creation of DSLs, it might be an idea to first relate the aspects of CL using compiler terminology.

Here are some random thoughts (not all are accurate, but represent an attempt to translate CL concepts into Blub):

- CL is a language that has Lex and Yacc built into it. You can change the syntax of CL to suit your problem.

- Lisp s-expressions are concrete parse trees.

- The CL "reader" is a scanner. You don't need to build a scanner, since CL already provides you with one.

- CL symbols are "tokens".

- The CL scanner can read programs from a number of sources - not just files - e.g. strings, keyboard, streams, etc.

- Physicists commonly create DSL's. They define a notation with which to describe a domain, then use the notation to solve problems in the domain.

- Few other languages can change their own syntax. Examples of self-modifying syntax can be found in term-rewriting systems, such as the language TXL.

- CL macros are parsers.

- CL macros are code emitters.

- CL macros are source-to-source program transformers.

- CL backquote/comma/at-sign notation is used to splice and modify concrete parse trees into other shapes.

- The CL compiler compiles s-expressions into machine code.

- If one creates s-expressions, one can use the CL compiler to compile them into machine code at runtime.

- The CL compiler is a JIT compiler.

- The CL repl interactively compiles s-expressions to machine code, then executes them, then prints the results of the execution.

- The CL language defines a rich library of operators for modifying concrete parse trees.

- To a CL'er, infix notation is just unnecessary syntactic sugar. Many novice CL'ers try to create macros that understand infix notation early on in their careers, but as they become more familiar with the power of CL, they drop the syntatic sugar and use parse trees directly.

- The CL printer contains a notation for displaying structure sharing. This facilitates debugging of optimizers which transform trees into graphs.

- To write a DSL in CL, one first writes a macro that parses s-expressions. Then, the macro is extended with backquotes to emit new s-expressions. The emitted s-expressions are automatically fed to the CL compiler and compiled to machine code.

- Unfortunately, the C language uses the name "macro" for a facility that is far less powerful than that of CL. In C, the macro facility can only parse a syntax that looks like simple symbols and function calls and a special #if syntax that is not even part of the C language definition. In CL, the macro facility can parse any s-expression and utilitize any CL function during code emission.

- With C macros, the programmer does not have the option to change the syntax of the macro language. With CL macros, the programmer has full control of the macro syntax.

- CL has the ability to link/load modules at runtime, "on the fly".

- CL invented the concept of JIT (is this factually true?).

- Forward references in CL are automatically handled by the runtime linker. The linker fixes up references on the fly to improve runtime efficiency (I know that I first saw this facility in UTAH Lisp in 1977).

- The CL scanner, compiler and linker sit in memory all (most) of the time, hence, CL is an all-in-one bundled DSL/compiler toolkit. Some CL's give you the ability to remove the compiler when creating a .exe, if you no longer need it.

- The builtin CL macroexpand function gives the programmer the ability to debug his/her DSL compiler by viewing the emitted code (much like CPP's command line option that leaves the expanded code in a text file).

- CL's rich suite of syntactic and compiler tools came about due to early AI research into automatic code generation.

- CL lambdas (closures) are thunks.

- CL closures are trampolines.

- CL closures and Scheme's call-with-continuation are the ultimate goto's. Any control flow can be modelled and compiled with these facilities.

- CL closures and Scheme's call-with-continuation can be used to create compilers using the concepts of denotational semantics.

- Cl closures are (kind of) like Smalltalk's "blocks".

- CL closures are another example of the rich language-building facilities of CL, not commonly found in most languages.

- CL closures can be used to create anonymous functions, similarly to anonymous classes used in Java.

- The facilities of CL make it possible to model and compile just about any kind of runtime semantics, including very non-linear ones, e.g. PAIP prolog.

- Prolog provides an simple way to build parsers, if efficiency is not of utmost concern. A number of prologs are available to CL programmers, so one can have the best of both worlds. [I use paip prolog to parse diagrammatic syntax.]

hth

pt

On Wed, Jul 20, 2011 at 3:32 PM, Didier Verna didier@lrde.epita.fr wrote:

Dear friends,

I'm starting to write a chapter for an upcoming book on domain specific languages. The chapter is called (tentatively):

Extensible languages -- blurring the distinction between DSLs and GPLs

GPL meaning General Purpose Language in this context ;-)

My intention is to demonstrate how the task of implementing a DSL is made easier when it boils down to an extension or subset of your original GPL (hence reusing its infrastructure), instead of being a totally different language, only written on top of the other.

Obviously, I'm going to illustrate this with Common Lisp, and I intend to speak of dynamicity (not only dynamic typing, but in general all things that can be deferred to the run-time), introspection, intersession, structural or procedural reflexivity, meta-object protocols (not sure about this one), macro systems and JIT-compilation. Also, more specifically to Lisp, reader macros (compiler macros maybe?), the condition system (and its ability to *not* unwind) and restarts.

Right now, I would like to know if any of you have DSL "pearls", nice examples of DSLs that you have written in Lisp by using some of its features in a clever or elegant way. I would also gladly accept any point of view or comment on what's important to mention, in terms of design principle or anything else, things that I may have missed in the list above.

Not really "pearls", but lately I found myself using more and more of Lisp at macroexpansion time, which seems to be what you're looking for. Two examples:

- for my DO+ (doplus) iteration macro - a somewhat iterate-like construct with a simpler implementation that does not use a code walker - I used structures quite heavily. do+ controls iteration via a series of clauses that are macros (either built-in or written by users) that have a peculiar aspect: they don't expand to proper Lisp code, but return structures or lists of structures [1]. do+ macroexpands its clauses (recursively when needed) and uses the returned structures to drive the building of its own expansion. Basically I explicitly used macros as compile-time functions, defining my own limited "AST" (not really a tree but a flat list of structures) which is not Lisp's but is later translated to Lisp, so it's one extra indirection than what is typically found in macros. I think this shows nicely how the distinction between code and data is blurry in Lisp, and as a consequence the distinction between DSL and GPL is blurry as well. This allows the implementation to remain simple: do+ is a little less than 350 lines of code, but only roughly half of them are the actual macro and its supporting code, the rest are built-in clause macros and other minor stuff.

- I also have sketched a HTML generation library - tentatively called tag-pool - where I used CLOS generic functions at macroexpansion time to drive the expansion of tag macros to HTML-outputting code. Contrary to the popular approach, used by CL-WHO among others, to represent HTML as a limited sexp-based DSL where tag names are keywords and a single macro does all the translation work, I have one macro per tag. For CLOS dispatch purposes, to each tag is also associated a class, but it's used only at expansion time, the output code does not instantiate any CLOS object. Tag macros follow a CLOS-based protocol to emit the various bits of HTML like attributes, body, and so on. Users can specialize methods that are part of the protocol to finely control how tag macros are translated, for example providing defaults for certain attributes, or post-processing some attribute's value, etc. As an example, the SCRIPT macro translates its Lisp body to JavaScript using Parenscript, by specializing a single generic function. The body code can itself contain HTML-generating macros and those are translated to HTML-emitting JavaScript as well. Again, a key feature of Lisp (CLOS and generic function dispatch) is used at macroexpansion time to make the implementation of the DSL simpler and more easily extensible.

You can find both examples here: http://code.google.com/p/tapulli/ although only do+ is documented and reasonably complete.

[1] ok, technically a structure *is* proper Lisp code since it is a self-evaluating object and it can be externalized...

Best, Alessio

I like how Lisp allowed me to use almost without any change the spec of Redis to define its commands in cl-redis library ( https://github.com/vseloved/cl-redis/blob/master/commands.lisp). This gives, IMHO, the best possible decoupling from the implementation, which allows for easy refactoring of the underlying mechanisms (performed twice in the course of the library's existence) and seamless addition of new commands.

It's a good example of what Rainer Joswig called "putting parenthesis around the specification and making it run" ( http://lispm.dyndns.org/mov/dsl-in-lisp.mov)

Best regards, Vsevolod

On Wed, Jul 20, 2011 at 4:32 PM, Didier Verna didier@lrde.epita.fr wrote:

Dear friends,

I'm starting to write a chapter for an upcoming book on domain specific languages. The chapter is called (tentatively):

Extensible languages -- blurring the distinction between DSLs and GPLs

GPL meaning General Purpose Language in this context ;-)

My intention is to demonstrate how the task of implementing a DSL is made easier when it boils down to an extension or subset of your original GPL (hence reusing its infrastructure), instead of being a totally different language, only written on top of the other.

Obviously, I'm going to illustrate this with Common Lisp, and I intend to speak of dynamicity (not only dynamic typing, but in general all things that can be deferred to the run-time), introspection, intersession, structural or procedural reflexivity, meta-object protocols (not sure about this one), macro systems and JIT-compilation. Also, more specifically to Lisp, reader macros (compiler macros maybe?), the condition system (and its ability to *not* unwind) and restarts.

Right now, I would like to know if any of you have DSL "pearls", nice examples of DSLs that you have written in Lisp by using some of its features in a clever or elegant way. I would also gladly accept any point of view or comment on what's important to mention, in terms of design principle or anything else, things that I may have missed in the list above.

Thank you very much in advance!

-- Resistance is futile. You will be jazzimilated.

Scientific site: http://www.lrde.epita.fr/~didier Music (Jazz) site: http://www.didierverna.com

---

pro mailing list pro@common-lisp.net http://lists.common-lisp.net/cgi-bin/mailman/listinfo/pro

Didier Verna didier@lrde.epita.fr writes:

Dear friends,

http://groups.google.com/group/comp.lang.lisp/browse_frm/thread/0245342429c6...

Notably, in this type of DSL, you'll note how you can mix Lisp expressions with expressions in the new languages, at several levels.

@[Icals method:(pressPop:(:id)sender) ; resultType:(:id) ; objcl body: ; (declare (ignore sender)) ; (loop ; lisp :for i :from 0 :below 3 ; :do [(aref (icals-stack self) i) setIntValue:[(aref (icals-stack self) (1+ i)) intValue]])] ^^ ^^ || || |+-- lisp |+--- lisp +--- objcl +---- objcl

It might be worthwhile comparing the sources of an early Objective-C front-end such as POC (that includes only the parsing of Objective-C syntax, and generation of the C code for the usual C compiler, and thus is functionnaly equivalent to ObjCL), with the lisp sources of ObjCL.

http://users.telenet.be/stes/compiler.html (objc-3.2.11.tar.gz)

On Jul 20, 2011, at 6:32 AM, Didier Verna wrote:

Right now, I would like to know if any of you have DSL "pearls", nice examples of DSLs that you have written in Lisp by using some of its features in a clever or elegant way. I would also gladly accept any point of view or comment on what's important to mention, in terms of design principle or anything else, things that I may have missed in the list above.

One perspective that I haven't seen addressed yet is: know your audience and related concepts.

Last year, I wrote a tool for processing large quantities of dirty .csv files: more than can fit into RAM, multiple files comprising a single dataset, column sequence changing across files, junk in cell values, different number of columns across files, etc. And extremely short turn-around time.

The idea was to process this data without having to clean it up first, such that one might gain some insights (sums, uniques, bucketing, etc.) helpful in determining whether or not the data was of any value or gain clues on how to proceed further. Proper database ETL would be too time consuming.

My intended audience was a statistical analyst that favored mousing around Excel spreadsheets while letting prior SAS training go unused because of not wanting to write scripts. (That should have been the first clue to be concerned, but oh, what we do for friends!)

Ultimately, the tool was useful to me, and that was sufficient to meet deadlines.

But the DSL code became overly complex due to this extraneous design criteria of accommodating someone who doesn't want to write scripts in the first place.

Instead of a few simple macros such as WITH-MESSY-CSV appropriate to a Lisp programmer, I effectively created a register machine with "simple" commands: include/exclude filtering, sum, count, unique, look-up, group-by, etc.

Otherwise, the approach was sound enough: load relatively small .csv files as look-up tables and iterate over the entire dataset in a single pass, applying lexically scoped blocks of filtering & calculations. Convert only named columns of interest regardless of position changes across files, parse numbers on demand only for those operations that required it, skip unparseable rows as last resort, etc. Some error in results-- but some results with reduced confidence are better than none in this case.

Lessons learned: (a few more while I'm here)

1. Know your audience, and build for the correct users.

2. Build the right tool. (I'm a systems programmer; a good stats person would likely have come up with a better work-flow, likely using R so rich reports could also be generated quickly.)

3. Good language design can be challenging. I would have been better off (perhaps) stealing SQL or XQuery's FLOWR conventions than inventing my own "simple" set of commands. (Syntax is another matter... as you know.)

4. Being adept at backquotes, comma substitution and unrolling lists is not necessarily enough skill to create a good, clean DSL implementation. But keep trying. Do your best to make one for "keeps". Then throw it away, anyway. It's important to not hold anything back in the first version. Ah, experience! (I'll likely go at this one again just for the fun of it.) e.g., unrelated project from years ago: http://play.org/learning-lisp/html.lisp

5. Collaborate: Get input from others. My co-workers who also use Common Lisp were many time-zones and an ocean away, busy with looming deadlines of their own. However, their 10 years CL experience to my 5 (and their far deeper stats familiarity) would certainly have helped here.

-Daniel

-- first name at last name dot com

Speaking of XML, there's another good example of a special-purpose DSL, designed for efficient processing of OSM map data, described here: http://swizard.livejournal.com/142027.html (text in Russian).

On Wed, Jul 20, 2011 at 4:32 PM, Didier Verna didier@lrde.epita.fr wrote:

Dear friends,

I'm starting to write a chapter for an upcoming book on domain specific languages. The chapter is called (tentatively):

Extensible languages -- blurring the distinction between DSLs and GPLs

GPL meaning General Purpose Language in this context ;-)

My intention is to demonstrate how the task of implementing a DSL is made easier when it boils down to an extension or subset of your original GPL (hence reusing its infrastructure), instead of being a totally different language, only written on top of the other.

Obviously, I'm going to illustrate this with Common Lisp, and I intend to speak of dynamicity (not only dynamic typing, but in general all things that can be deferred to the run-time), introspection, intersession, structural or procedural reflexivity, meta-object protocols (not sure about this one), macro systems and JIT-compilation. Also, more specifically to Lisp, reader macros (compiler macros maybe?), the condition system (and its ability to *not* unwind) and restarts.

Right now, I would like to know if any of you have DSL "pearls", nice examples of DSLs that you have written in Lisp by using some of its features in a clever or elegant way. I would also gladly accept any point of view or comment on what's important to mention, in terms of design principle or anything else, things that I may have missed in the list above.

Thank you very much in advance!

-- Resistance is futile. You will be jazzimilated.

Scientific site: http://www.lrde.epita.fr/~didier Music (Jazz) site: http://www.didierverna.com

---

pro mailing list pro@common-lisp.net http://lists.common-lisp.net/cgi-bin/mailman/listinfo/pro