Dear Fellow Parenscripters,
Here I detail a proposed package system for Parenscript to ease modular development. This is a feature intended to make up for the many hacks Javascript programmers use to write libraries. The proposed system takes after and integrates with the Lisp package system. Its implementation will constitute a hearty advancement of the current Parenscript compiler.
The proposed system should introduce new concepts and should be robust enough to lay the foundation for a lot of future work in a coherent way. Different packages can introduce new special forms--that is, modify the language at the lowest level. With this, a "javascript2" package can introduce Parenscript forms that compile to the latest version of ECMAScript without implicating the rest of the project. The additional semantic analysis of Parenscript programs will pave the way for cropped, minified, or obfuscated scripts. In addition, a formal package specification will lead to asdf-installable script libraries with better deployment characteristics than current package systems for web development in other languages.
A Parenscript package (or "script package" or simply "package," as opposed to a "Lisp package") has a few main properties: a name and list of nicknames, an primary associated Lisp package, a list of exported identifiers, and a collection of macros defined in the associated Lisp package. We must also introduce and formalize a two other concepts: Parenscript identifiers (analog of Lisp symbols), and a compilation environment.
A Parenscript package is defined and exists in the context of a compilation environment. A compilation environment simply keeps track of compiler state; when any Parenscript code is compiled, a new compilation environment is created or an existing one is passed to the compiler. The environment is modified to reflect the lexical scope as forms are processed. Specifically, the compilation environment will consist of a stack of Parenscript identifiers (introduced by defun and defvar forms); a stack of macros and symbol macros; a list of defined script packages; and the current script package.
An identifier in Parenscript is analogous to a Lisp symbol, but it only exists during compile time. An identifier has a string value and associated Parenscript package.
Now I will run through a simple example to demonstrate these new developments:
Script packages are primarily defined using a Parenscript form analogous to the Lisp defpackage:
(defpackage friendly-script (:use parenscript psos) (:export hello-world) (:lisp-package friendly) (:documentation "Scripts for issuing greetings."))
This introduces a new package into the compilation environment. We enter the package system after it is defined as in Lisp:
;; enter the friendly-script package ; changes the current package in the compilation environment to :friendly-script (in-package :friendly-script)
;; define the hello-world function, which we export ; adds friendly-script::hello-world to the identifier stack in the compilation env. (defun hello-world () (alert "hello world."))
;; enter the user package ; changes current package in compilation environment (in-package :parenscript-user)
;; call the friendly-script's hello-world function ; the compiler recognizes that friendly-script::hello-world is an e (friendly-script:hello-world)
To give you an idea, this will all compile to something like the following Javascript:
function friendlyScript_helloWorld() { alert("hello world."); }
friendlyScript_helloWorld();
In terms of implementation, the package system will introduce a few new classes (script-package, identifier, and compilation-environment); uproot existing parsing procedures and replant them around the package system; and add a semantic analysis phase into the compilation pipeline. I will fill in details and announce problems with the implementation as it moves forward.
Hopefully this is an exciting prospect for the community. The package system should empower those who want to build large programs or share Parenscript extensions/libraries. It should also preserve the javascript-in-sexps Parenscript we know and love.
Anyhow, I look forward to comments on this proposal. Happy hacking!
Red
It sounds like a great idea! A well thought out package system can gives parenscript another advantage over javascript.
Instead of loading huge js library files I guess some analysis of dependencies between functions and variables will make it possible to assemble a small js file with only the functions needed in the particular situation, right?
If you want alpha testers or help in any way, please let me know.
I have one comment, It would be a good thing if the name-mangling/code generation is implemented so it can be tweaked. Your example suggest to make functions with prefixed names on the global object, like "friendlyScript_helloWorld". But some might want to generate the code in another way, for example make a literal object called friendlyScript with "helloWorld" as one key and the code (function/lambda) as value. It would be nice if both ways and others could be supported. I don't mean you should implement both, but have a generic function or similar interface for generating the function definitions and one for generating the function calls.
Thanks, I'm looking forward to this.
/Henrik Hjelte
On 6/27/07, Red Daly reddaly@gmail.com wrote:
Dear Fellow Parenscripters,
Here I detail a proposed package system for Parenscript to ease modular development. This is a feature intended to make up for the many hacks Javascript programmers use to write libraries. The proposed system takes after and integrates with the Lisp package system. Its implementation will constitute a hearty advancement of the current Parenscript compiler.
The proposed system should introduce new concepts and should be robust enough to lay the foundation for a lot of future work in a coherent way. Different packages can introduce new special forms--that is, modify the language at the lowest level. With this, a "javascript2" package can introduce Parenscript forms that compile to the latest version of ECMAScript without implicating the rest of the project. The additional semantic analysis of Parenscript programs will pave the way for cropped, minified, or obfuscated scripts. In addition, a formal package specification will lead to asdf-installable script libraries with better deployment characteristics than current package systems for web development in other languages.
A Parenscript package (or "script package" or simply "package," as opposed to a "Lisp package") has a few main properties: a name and list of nicknames, an primary associated Lisp package, a list of exported identifiers, and a collection of macros defined in the associated Lisp package. We must also introduce and formalize a two other concepts: Parenscript identifiers (analog of Lisp symbols), and a compilation environment.
A Parenscript package is defined and exists in the context of a compilation environment. A compilation environment simply keeps track of compiler state; when any Parenscript code is compiled, a new compilation environment is created or an existing one is passed to the compiler. The environment is modified to reflect the lexical scope as forms are processed. Specifically, the compilation environment will consist of a stack of Parenscript identifiers (introduced by defun and defvar forms); a stack of macros and symbol macros; a list of defined script packages; and the current script package.
An identifier in Parenscript is analogous to a Lisp symbol, but it only exists during compile time. An identifier has a string value and associated Parenscript package.
Now I will run through a simple example to demonstrate these new developments:
Script packages are primarily defined using a Parenscript form analogous to the Lisp defpackage:
(defpackage friendly-script (:use parenscript psos) (:export hello-world) (:lisp-package friendly) (:documentation "Scripts for issuing greetings."))
This introduces a new package into the compilation environment. We enter the package system after it is defined as in Lisp:
;; enter the friendly-script package ; changes the current package in the compilation environment to :friendly-script (in-package :friendly-script)
;; define the hello-world function, which we export ; adds friendly-script::hello-world to the identifier stack in the compilation env. (defun hello-world () (alert "hello world."))
;; enter the user package ; changes current package in compilation environment (in-package :parenscript-user)
;; call the friendly-script's hello-world function ; the compiler recognizes that friendly-script::hello-world is an e (friendly-script:hello-world)
To give you an idea, this will all compile to something like the following Javascript:
function friendlyScript_helloWorld() { alert("hello world."); }
friendlyScript_helloWorld();
In terms of implementation, the package system will introduce a few new classes (script-package, identifier, and compilation-environment); uproot existing parsing procedures and replant them around the package system; and add a semantic analysis phase into the compilation pipeline. I will fill in details and announce problems with the implementation as it moves forward.
Hopefully this is an exciting prospect for the community. The package system should empower those who want to build large programs or share Parenscript extensions/libraries. It should also preserve the javascript-in-sexps Parenscript we know and love.
Anyhow, I look forward to comments on this proposal. Happy hacking!
Red
parenscript-devel mailing list parenscript-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/parenscript-devel
This sounds like the right way to go. I am also in agreement with Henrik that providing a protocol into the relevant parts of the package system would be a very valuable addition. The one thing I think is critical is that ParenScript's current semantics with respect to naming be preserved if you choose not to use the package system.
Vladimir
On 6/26/07, Red Daly reddaly@gmail.com wrote:
Dear Fellow Parenscripters,
Here I detail a proposed package system for Parenscript to ease modular development. This is a feature intended to make up for the many hacks Javascript programmers use to write libraries. The proposed system takes after and integrates with the Lisp package system. Its implementation will constitute a hearty advancement of the current Parenscript compiler.
The proposed system should introduce new concepts and should be robust enough to lay the foundation for a lot of future work in a coherent way. Different packages can introduce new special forms--that is, modify the language at the lowest level. With this, a "javascript2" package can introduce Parenscript forms that compile to the latest version of ECMAScript without implicating the rest of the project. The additional semantic analysis of Parenscript programs will pave the way for cropped, minified, or obfuscated scripts. In addition, a formal package specification will lead to asdf-installable script libraries with better deployment characteristics than current package systems for web development in other languages.
A Parenscript package (or "script package" or simply "package," as opposed to a "Lisp package") has a few main properties: a name and list of nicknames, an primary associated Lisp package, a list of exported identifiers, and a collection of macros defined in the associated Lisp package. We must also introduce and formalize a two other concepts: Parenscript identifiers (analog of Lisp symbols), and a compilation environment.
A Parenscript package is defined and exists in the context of a compilation environment. A compilation environment simply keeps track of compiler state; when any Parenscript code is compiled, a new compilation environment is created or an existing one is passed to the compiler. The environment is modified to reflect the lexical scope as forms are processed. Specifically, the compilation environment will consist of a stack of Parenscript identifiers (introduced by defun and defvar forms); a stack of macros and symbol macros; a list of defined script packages; and the current script package.
An identifier in Parenscript is analogous to a Lisp symbol, but it only exists during compile time. An identifier has a string value and associated Parenscript package.
Now I will run through a simple example to demonstrate these new developments:
Script packages are primarily defined using a Parenscript form analogous to the Lisp defpackage:
(defpackage friendly-script (:use parenscript psos) (:export hello-world) (:lisp-package friendly) (:documentation "Scripts for issuing greetings."))
This introduces a new package into the compilation environment. We enter the package system after it is defined as in Lisp:
;; enter the friendly-script package ; changes the current package in the compilation environment to :friendly-script (in-package :friendly-script)
;; define the hello-world function, which we export ; adds friendly-script::hello-world to the identifier stack in the compilation env. (defun hello-world () (alert "hello world."))
;; enter the user package ; changes current package in compilation environment (in-package :parenscript-user)
;; call the friendly-script's hello-world function ; the compiler recognizes that friendly-script::hello-world is an e (friendly-script:hello-world)
To give you an idea, this will all compile to something like the following Javascript:
function friendlyScript_helloWorld() { alert("hello world."); }
friendlyScript_helloWorld();
In terms of implementation, the package system will introduce a few new classes (script-package, identifier, and compilation-environment); uproot existing parsing procedures and replant them around the package system; and add a semantic analysis phase into the compilation pipeline. I will fill in details and announce problems with the implementation as it moves forward.
Hopefully this is an exciting prospect for the community. The package system should empower those who want to build large programs or share Parenscript extensions/libraries. It should also preserve the javascript-in-sexps Parenscript we know and love.
Anyhow, I look forward to comments on this proposal. Happy hacking!
Red
parenscript-devel mailing list parenscript-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/parenscript-devel
"Red Daly" reddaly@gmail.com writes:
Here I detail a proposed package system for Parenscript
And in my view an excellent one!
[...]
A Parenscript package (or "script package" or simply "package," as opposed to a "Lisp package") has a few main properties: a name and list of nicknames, an primary associated Lisp package, a list of exported identifiers, and a collection of macros defined in the associated Lisp package. We must also introduce and formalize a two other concepts: Parenscript identifiers (analog of Lisp symbols), and a compilation environment.
A Parenscript package is defined and exists in the context of a compilation environment. A compilation environment simply keeps track of compiler state; when any Parenscript code is compiled, a new compilation environment is created or an existing one is passed to the compiler. The environment is modified to reflect the lexical scope as forms are processed. Specifically, the compilation environment will consist of a stack of Parenscript identifiers (introduced by defun and defvar forms); a stack of macros and symbol macros; a list of defined script packages; and the current script package.
Perhaps compiler macros (or something more powerful like SBCL's deftransform) could be included?
I'd quite like the ability to delay the emission of JavaScript code, so that definitions can be emitted only when the call tree explicitly draws in a particular function or variable. I guess the information for these delayed definitions would belong in the compilation environment. (Maybe a later version ;-)
Having macros and so on fixed to particular ParenScript packages will be a big plus for sharing code. Excellent.
[...]
;; enter the user package ; changes current package in compilation environment (in-package :parenscript-user)
As Henrik said, some people might want to consider a ParenScript package to be JavaScript object. So that here one might emit
with (parenscript-user) { ... }
(Just throwing out random ideas with complicated ramifications. Personally I prefer the prefixed name approach but some JavaScript megalibraries do stash everything in one object.)
[...]
Hopefully this is an exciting prospect for the community. The package system should empower those who want to build large programs or share Parenscript extensions/libraries. It should also preserve the javascript-in-sexps Parenscript we know and love.
It sounds great!
[...]
Please read on for my plea for help formalizing the semantics of Parenscript identifiers.
John Fremlin wrote:
...
Perhaps compiler macros (or something more powerful like SBCL's deftransform) could be included?
I am not intimately familiar with either of these features, though I would appreciate an explanation.
I am certainly interested in opening up and advancing the compilation architecture. Since I am not exactly a seasoned compiler programmer I do not know exactly how to do this effectively. I have found a few articles about the subject:
"An Architecture for an Open Compiler:" http://www2.parc.com/spl/groups/eca/pubs/papers/Lamping-IMSA92/for-web.pdf "What a Metaobject Protocol Based Compiler Can Do For Lisp:" http://www2.parc.com/csl/groups/sda/publications/papers/Kiczales-MOPs-for-Li...
I'd quite like the ability to delay the emission of JavaScript code, so that definitions can be emitted only when the call tree explicitly draws in a particular function or variable. I guess the information for these delayed definitions would belong in the compilation environment. (Maybe a later version ;-)
This certainly seems like a reasonable feature to include in future versions.
;; enter the user package ; changes current package in compilation environment (in-package :parenscript-user)
As Henrik said, some people might want to consider a ParenScript package to be JavaScript object. So that here one might emit
with (parenscript-user) { ... }(Just throwing out random ideas with complicated ramifications. Personally I prefer the prefixed name approach but some JavaScript megalibraries do stash everything in one object.)
[...]
with() opens up a can of worms, but placing function/variable definitions into a single object is in the game plan. Right now, while I try to get something working, I am using a strategy that prefixes variable names.
ATTN: I would like to request some help figuring out how exactly Parenscript identifiers will work. A Parenscript identifier is exactly a lisp symbol. Lisp symbols have associated Lisp packages. Likewise, identifiers should have an associated Parenscript package.
There are two problems right now that make it unclear how to implement identifiers: (1) issues with serialization and (2) complications trying to map lisp symbols to Parenscript identifiers.
1. Serialization-wise, it is unclear what the best semantics are for Parenscript identifiers sometimes. Consider the following code:
(in-package :parenscript-user) (slot-value thing 'property)
If identifiers are to be prefixed before compilation, then this may compile to something like:
thing.parenscriptUser_property
I think this is the correct behavior, but it is confusing. A solution is to introduce a "global" package that compiles without identifier prefixes:
(in-package :parenscript-user) (slot-value thing 'global::property) ; => thing.property
Some syntax sugar might make this more manageable. For example, using keywords instead of global symbols:
(in-package :parenscript-user) (slot-value thing :property) ; => thing.property
If we go the other way and have quoted identifiers serialize without prefixes, we might end up with other confusing semantics:
(setf (slot-value maple 'tree:root) "sprawling.") (setf (slot-value maple 'etymology:root) "Old English.") (alert (slot-value maple 'tree-root)) ; expected: "sprawling"
This would be a confusing compilation:
maple.root = "sprawling" maple.root = "Old English." alert (maple.root)
2. Determining the Parenscript package associated with a given Lisp symbol seems difficult. This is as a result of the ability of Lisp packages to import symbols. Here is the basic functionality for determining the Script package of a Lisp symbol:
(defun *lisp-to-paren-package-table* (make-hash-table) "Maps a lisp package to a script package.")
(defun lisp-to-paren-package (lisp-package) "Gets a script package corresponding to the given Lisp package." (gethash lisp-package *lisp-to-paren-package-table*))
(defun symbol-paren-package (symbol) "Gets the Parenscript package associated with a Lisp symbol." (lisp-to-paren-package (symbol-package symbol))
To me this could introduce some confusing circumstances when a Script package's Lisp package uses symbols from another package. The most common are conflicts from the common-lisp package. Imagine a Parenscript package "paren-psos" has a (defclass) macro--a symbol imported from common-lisp. The standard "parenscript" package has many forms associated with (e.g. defvar, defun)
;; parenscript code: package definitions (defpackage parenscript (:lisp-package :parenscript) (:second-lisp-packages :common-lisp))
(defpackage paren-psos (:lisp-package :paren-psos))
;; lisp code: (in-package :parenscript) (defjsmacro defhappyclass (name slots &rest options) `(progn (paren-psos:defclass ,name ,slots ,@options) (parenscript:setf (slot-value happy-classes ,(string name)) ,name)))
The compiler will need to resolve the script package of the paren-psos:defclass symbol, but will run into problems if that lisp symbol is imported from the common-lisp package. (In lisp, (symbol-package 'paren-psos:defclass) returns COMMON-LISP.) It will end up thinking that defclass is an identifier from the "parenscript" script package. This is confusing. One way to get around it is to make paren-psos shadow some symbols from the packages it uses. This might be sufficient, especially if we add compiler warnings when a parenscript macro is defined and the lisp package of its symbol is different from the current package.
Package resolution is not a problem using a separate Parenscript reader (in fact I have already done it), but using when defining Parenscript macros in Lisp we are confined to the standard Lisp reader.
Any thoughts on these two problems would be appreciated. I am sort of stuck.
Red
Red Daly reddaly@stanford.edu writes:
John Fremlin wrote:
...
Perhaps compiler macros (or something more powerful like SBCL's deftransform) could be included?
I am not intimately familiar with either of these features, though I would appreciate an explanation.
Basically they allow you to specify optimisations to the compiler. deftransform is more flexible and has access to type inference (as far as I understand), but the basic idea is define-compiler-macro.
For a simplified example from a real world use case, in the cl-ppcre (regexp library) if you have something like
`(match "\s*[a-z]+\s" user-input)
then a compiler macro will change that into
`(match ,(create-compiled-regexp-from "\s....") user-input)
so that the regexp is compiled at compile time, massively increasing performance.
The compiler macro would be something like this (off the top of my head)
(define-compiler-macro match (&whole form regexp string &environment env) (if (constantp regexp env) `(match (load-time-value (create-compiled-regexp-from ,regexp)) ,string) form))
(if it returns the same form as it started with then it is a nop)
I am certainly interested in opening up and advancing the compilation architecture. Since I am not exactly a seasoned compiler programmer I do not know exactly how to do this effectively. I have found a few articles about the subject:
I think a lot of the stuff compilers do is quite irrelevant - I mean figuring out which registers to use is not on the agenda. Let's focus on performance problems as they come up ;-)
[...]
(in-package :parenscript-user) (slot-value thing 'property)
If identifiers are to be prefixed before compilation, then this may compile to something like:
thing.parenscriptUser_property
I think this is the correct behavior, but it is confusing. A solution is to introduce a "global" package that compiles without identifier prefixes:
Yes I think that is necessary anyway. I don't think it is confusing, as you have explicitly asked for prefixing.
However, I think that using the same syntax for CL packages as for parenscript packages is a very bad idea, as it means all the parenscript packages have to be defpackage'd in CL so the reader doesn't barf.
I don't know which syntax would be appropriate, as JavaScript already uses . (is that a problem!?) and : is taken. Maybe we could take % or = or something but I don't like either.
(in-package :parenscript-user) (slot-value thing 'global::property) ; => thing.property
Some syntax sugar might make this more manageable. For example, using keywords instead of global symbols:
I disagree! I think that symbol package lookup should occur as in CL and if you :use global and global%property exists and you haven't overridden it, then you should get "thing.property" from (slot-value thing 'property)
(in-package :parenscript-user) (slot-value thing :property) ; => thing.property
If we go the other way and have quoted identifiers serialize without prefixes, we might end up with other confusing semantics:
Yes, it would be terrible
[...]
- Determining the Parenscript package associated with a given Lisp
symbol seems difficult. This is as a result of the ability of Lisp packages to import symbols. Here is the basic functionality for determining the Script package of a Lisp symbol:
If we use the Lisp reader, I think we should use a different syntax for parenscript package designation.
An alternative would be to completely abandon the Lisp reader and use a different one (embeddable with reader macros or something). Would it look very ugly interspersed with normal Lisp?
It sounds a lot better than having stupid package!identifier names or whatever, and if you already have got it working, why not flesh out a proposal so we can see what it would look like?
(At the moment I use this bizarre superquote macro-system I concocted which rather relies on plain s-exps being fed into parenscript so I'm a little chary of the idea.)
[...]
parenscript-devel@common-lisp.net
-
Henrik Hjelte -
John Fremlin -
Red Daly -
Red Daly -
Vladimir Sedach