[Most discussion should take place on the libcl-devel list.]

Hi all,

It has been several months since LibCL saw significant progress. This was a period of small experiments, reflection, and numerous "real-world" distractions.

This past week, I had a bit of free time; and recent rumors of quicklisp focused the need to sieze the moment and put thoughts into action. [Thanks Zach. Honestly, with everything in my head, I couldn't have handled volunteers anyway.]

Here's the result: a revised plan for LibCL, and a skeleton code base ready for contributors.

http://libcl.com/ [main website, as yet unchanged] http://gitorious.org/libcl/world [new git repo]

* User experience

The user experience will see relatively few changes; just more stuff will be available. ASDF 1 to ASDF 2 is the biggest configuration change.

* Developer experience

This is much improved. No more tediously maintaining two git repos for each package. No more harrowing forced conversions of upstream sources into git. No more massive packages.lisp with a metadata defstruct.

Now, each package is defined by a simple package.lisp file. This contains a single plist of information. Many of the keys are inherited from the old packages.lisp file; others are defined in the new metadata.txt. Key new metadata types are :maintainers, :tags, :todo, and :source.

Now a single file describes everything needed to obtain and describe a (nearly) arbitrary upstream package. Helper scripts standardize the process.

A simple framework structures the steps needed for building a release. It also applies local patches until they can be merged upstream. Names aren't finalized, but see workflow.txt. The html index will have a mark denoting local patches, except for patches following a naming convention indicating "trivial" operations like renaming directories and building documentation.

* Expanded vision

History: LibCL began after yet-another-computer/OS-upgrade cycle left me spending months reconstructing my libraries. The current systems just didn't work for me. I started with a bunch of things; but realized that many packages needed to be dropped because they were hard to support (most notably, anything using FFI). Then I moved towards the idea of creating a "core standard library", with a team of developers, etc. That never gelled. People were asking for more; I even found myself needing one-off, non-portable documentation tools to build the docs for LibCL...

LibCL now aims to be a 1-stop shop. If a library isn't in LibCL, please send a metadata file to this mailing list, with a subject line like "[metadata] awesome-package.lisp". Or use a merge request on the gitorious repository. Same thing goes for applications. If/when the downloads start bloating, we can offer variants as needed.

How are non-portable tools handled? Through tags. Only portable, pure CL libraries may be tagged with :pure. Only :pure libraries seeing widespread use may be tagged with :core.

The default invocation of compile-libcl will only compile :core libraries. Calling compile-libcl with other tags allow people to compile things specific to their interests. Selecting a package by name will compile it and all its dependencies. Selecting T will compile everything.

* Upcoming feature

PJB and others had a great idea. After "(compile-libcl t)", walk the packages and generate a master index of all available symbols. Link this index to docstrings, and eventually add links to the full documentation. Good for searching.

This will be included in the next LibCL release.

------------- Call for help

The current libcl-world is a rough skeleton; but key pieces are already in place. Any help would be appreciated; please coordinate on the libcl-devel list; here are some suggested places to start.

Easy tasks: - add metadata for new packages - create/add tags to packages - add maintainer information to packages

Harder tasks: - coordinate metadata and/or the tarball cache with quicklisp? - fix package :todos, and add more as needed - update configure-libcl for ASDF 2 - website stuff - create a cache system for package tarballs and log files - create a comment/voting system for packages - sort through the old upstream/local repos for patches which never made it upstream - design a "fetch on demand" layer a la quicklisp or MiKTeX - hook "fetch on demand" into ASDF - design a system for caching .so and .dll files for FFI bindings

Most changes may be handled through git merge requests. Metadata changes may be sent to libcl-devel with a subject like "[metadata] package.lisp". Contact me directly if you need a shell account for web changes (the current website should be available on gitorious).

Thanks for reading this far. This will be a busy week personally, and I should be offline during September 12-14; but feel free to make progress without me. Let's see where this thing goes.

Later, Daniel