Hi Jan!

Thank you for your patch! I'm going to convert it into loadable patch for most of recent SBCL versions and ship with usocket. Once it's good enough and proved to work, I'll send them to SBCL dev team and let them to consider if IPv6 should be supported officially.

--binghe

On 24/giu/2013, at 16:45, Jan Moringen jmoringe@techfak.uni-bielefeld.de wrote:

Hi.

On Mon, 2013-06-24 at 15:20 +0800, Chun Tian (binghe) wrote:

...

4. Write patches for SBCL and CCL to support IPv6 directly on their exist network interfaces. The purpose is to make unmodified SBCL/CCL working directly with my patches to get the ability to support IPv6 with their exist interfaces. Platforms with such modifications should have :usocket-ipv6 in their *features*.

I once started to write the attached patch for SBCL but then heard some talk about replacing SBCL's stream machinery with iolib and stopped working on that (not sure whether I remember "hearing" that correctly).

But anyway, maybe you have some use for the patch. I managed creating IPv6 connections with it, but the patch is obviously far from finished. It may contain little code fragments from iolib.

Kind regards, Jan <0001-TODO-temp-commit.patch>

Hi all,

just in case it might be somehow interesting, we've had iolib.usocket system for quite some time in http://src.knowledgetools.de/tomas/winapi/index.html It is a simple usocket compatibility layer on top of iolib (also works on Windows winapi 32 and 64 bit). Not sure what the ipv6 status is though as we don't use that. IIRC I still need to implement translation of iolib conditions to usocket ones, but as a precondition for that is unifying conditions from iolib posix and winapi backends. Otherwise, hunchentoot and cl-postgres work well on posix and windows using this compatibility layer.

Cheers,

Tomas

On 06/24/2013 11:12 AM, Anton Vodonosov wrote:

Hello.

24.06.2013, 11:22, "Chun Tian (binghe)" binghe.lisp@gmail.com:

...

To compile usocket with IOlib, user should push :usocket-iolib into their *feature* first.

I would like to propose to use some other solution than conditional controlling compilation with *reatures*.

The disadvantage of the conditional compilation is that when my application loads the usocket as a dependency, the application doesn't know how usocket will work, because it was dediced when usocket was compiled (possible during load of some other application).

If you give little bit more details about he usocket-iolib functions, I can propose more concrete solutions. Very possible the proposal will be a separate ASDF system, usocket-iolib.

Best regards,

• Anton

Hi binghe,

you are welcome. Both iolib and usocket seem to be under the MIT licence so I don't see a problem on that front.

However, shouldn't the code be merged into iolib only rather than merging it into both iolib and usocket? The whole of the iolib winapi port makes it quite a lot of code. It would be a shame to sacrifice the winapi part especially when usocket is all about portability API.

Also, as Anton pointed out, I find a separate system better solution than pushing a feature. After all, usocket is an API. When implemented using iolib, why would there be a need to use any code from the original usocket? At the moment we simply asdf-load :iolib.usocket only and all systems depending on usocket work as desired because the required usocket system and usocket:* stuff is in place.

Cheers,

Tomas

On 06/24/2013 11:33 AM, Chun Tian (binghe) wrote:

Hi Tomas

Thank you! Then I guess you wouldn't mind if I merge your work into usocket as the basis of the new IOlib backend? ^_^

--binghe

On 24/giu/2013, at 17:30, Tomas Hlavaty tomas.hlavaty@knowledgetools.de wrote:

...

Hi all,

just in case it might be somehow interesting, we've had iolib.usocket system for quite some time in http://src.knowledgetools.de/tomas/winapi/index.html It is a simple usocket compatibility layer on top of iolib (also works on Windows winapi 32 and 64 bit). Not sure what the ipv6 status is though as we don't use that. IIRC I still need to implement translation of iolib conditions to usocket ones, but as a precondition for that is unifying conditions from iolib posix and winapi backends. Otherwise, hunchentoot and cl-postgres work well on posix and windows using this compatibility layer.

Cheers,

Tomas

On 06/24/2013 11:12 AM, Anton Vodonosov wrote:

...

Hello.

24.06.2013, 11:22, "Chun Tian (binghe)" binghe.lisp@gmail.com:

...

To compile usocket with IOlib, user should push :usocket-iolib into their *feature* first.

I would like to propose to use some other solution than conditional controlling compilation with *reatures*.

The disadvantage of the conditional compilation is that when my application loads the usocket as a dependency, the application doesn't know how usocket will work, because it was dediced when usocket was compiled (possible during load of some other application).

If you give little bit more details about he usocket-iolib functions, I can propose more concrete solutions. Very possible the proposal will be a separate ASDF system, usocket-iolib.

Best regards,

• Anton

Hi binghe,

the portability layer lives in the package :iolib.usocket and implements the usocket api. How would you implement and api without emulating another package's interface? Isn't the api the package's interface? If the user loads iolib.usocket.asd, what else should he expect instead of usocket with all functionalities of usocket in it?

iolib.usocket.asd creates an alias system and package so that users don't have to make any explicit code changes to their own code base.

yes, usocket is brilliant for covering the platforms where cffi and iolib are not available/desirable.

The problem with the wait-* functions is that they are inherently non-portable (posix specific) and also assume certain programming model which is so far incompatible with wide-spread common lisp libraries. For example, I added a timeout argument to socket-accept so that hunchentoot can work without assuming posix (on winapi). Going forward, it might be worth acknowledging different programming models and embrace them in the usocket api.

Cheers,

Tomas

On 06/24/2013 12:25 PM, Chun Tian (binghe) wrote:

Hi Tomas

I think it's not quite polite to directly emulate another package's interface (and creating other package's Lisp package), so you shouldn't expect after loading IOlib.usocket, you got a package "usocket" with all functionalities of usocket in it. However, with a slightly changed Lisp package name, the whole idea is possible, just like what CFFI tries to emulate UFFI by using a separated system "cffi-uffi.asd". Any way, this is something not cared by me, because I'm not the maintainer of IOlib.

What I cared is to make exist usocket users and applications live better without making any explicit code changes into their own code base. By using IOlib as a backend, usocket's WAIT-FOR-INPUT function now can directly use IOlib's strong I/O multiplexing facility and exceed the 512 fd limitation when using select(). I also care those platforms in which CFFI and IOlib is not available, so a plain usocket implementation is still needed.

On 24/giu/2013, at 17:56, Tomas Hlavaty tomas.hlavaty@knowledgetools.de wrote:

...

Hi binghe,

you are welcome. Both iolib and usocket seem to be under the MIT licence so I don't see a problem on that front.

However, shouldn't the code be merged into iolib only rather than merging it into both iolib and usocket? The whole of the iolib winapi port makes it quite a lot of code. It would be a shame to sacrifice the winapi part especially when usocket is all about portability API.

Also, as Anton pointed out, I find a separate system better solution than pushing a feature. After all, usocket is an API. When implemented using iolib, why would there be a need to use any code from the original usocket? At the moment we simply asdf-load :iolib.usocket only and all systems depending on usocket work as desired because the required usocket system and usocket:* stuff is in place.

Cheers,

Tomas

On 06/24/2013 11:33 AM, Chun Tian (binghe) wrote:

...

Hi Tomas

Thank you! Then I guess you wouldn't mind if I merge your work into usocket as the basis of the new IOlib backend? ^_^

--binghe

On 24/giu/2013, at 17:30, Tomas Hlavaty tomas.hlavaty@knowledgetools.de wrote:

...

Hi all,

just in case it might be somehow interesting, we've had iolib.usocket system for quite some time in http://src.knowledgetools.de/tomas/winapi/index.html It is a simple usocket compatibility layer on top of iolib (also works on Windows winapi 32 and 64 bit). Not sure what the ipv6 status is though as we don't use that. IIRC I still need to implement translation of iolib conditions to usocket ones, but as a precondition for that is unifying conditions from iolib posix and winapi backends. Otherwise, hunchentoot and cl-postgres work well on posix and windows using this compatibility layer.

Cheers,

Tomas

On 06/24/2013 11:12 AM, Anton Vodonosov wrote:

...

Hello.

24.06.2013, 11:22, "Chun Tian (binghe)" binghe.lisp@gmail.com:

...

To compile usocket with IOlib, user should push :usocket-iolib into their *feature* first.

I would like to propose to use some other solution than conditional controlling compilation with *reatures*.

The disadvantage of the conditional compilation is that when my application loads the usocket as a dependency, the application doesn't know how usocket will work, because it was dediced when usocket was compiled (possible during load of some other application).

If you give little bit more details about he usocket-iolib functions, I can propose more concrete solutions. Very possible the proposal will be a separate ASDF system, usocket-iolib.

Best regards,

• Anton

Hi binghe,

I found my old note in :iolib.usocket:

;;; At first, I tried to ;;; implement an iolib backend for usocket but found that was not the ;;; way to go due to heavy posix bias of the usocket backend ;;; interface. Implementing the usocket API directly is better.

The usocket internal backend api was not a good a match for iolib, that's why the iolib.usocket and not usocket.iolib.

On 06/24/2013 01:11 PM, Chun Tian (binghe) wrote:

On 24/giu/2013, at 18:49, Tomas Hlavaty tomas.hlavaty@knowledgetools.de wrote:

...

Hi binghe,

the portability layer lives in the package :iolib.usocket and implements the usocket api. How would you implement and api without emulating another package's interface? Isn't the api the package's interface? If the user loads iolib.usocket.asd, what else should he expect instead of usocket with all functionalities of usocket in it?

It works, but if you try to depend on another package which depend on the real usocket, you'll got conflicts. Therefore such approaches cannot goes into Quicklisp as a common solution, I think.

There are no conflicts. The user either loads iolib.usocket.asd or not. There is no such concept as "trying to depend on another package which depend on the real usocket" because the dependency is on the API, not on the implementation.

...

iolib.usocket.asd creates an alias system and package so that users don't have to make any explicit code changes to their own code base.

yes, usocket is brilliant for covering the platforms where cffi and iolib are not available/desirable.

I should have also mentioned, that the most brilliant part of usocket is that it provides an API on which most common lisp libraries agree:-)

...

The problem with the wait-* functions is that they are inherently non-portable (posix specific) and also assume certain programming model which is so far incompatible with wide-spread common lisp libraries. For example, I added a timeout argument to socket-accept so that hunchentoot can work without assuming posix (on winapi). Going forward, it might be worth acknowledging different programming models and embrace them in the usocket api.

I don't quite under this part. usocket's WAIT-FOR-INPUT, depends on select() on UNIX-like systems, and WSAEventSelect() on Windows. When setting the timeout to zero, it should equals to poll()/epoll(). It's design and firstly implemented by original usocket authors, I think it's quite useful. The blocking approach is also supported when user can live with timeouts.

I guess you're using Hunchentoot with the usocket emulations based on IOlib, but you didn't implement WAIT-FOR-INPUT, right? I think that's because IOlib never port their powerful I/O multiplexing interfaces to Windows, so far it only support three OS-provided functions: select (on general Unix), epoll (on Linux) and kqueue (on BSD systems). Maybe you should encourage IOlib author to support "WSAEventSelect", then you don't need to use a timeout arguemnt to socket-accept any more...

There is only one place in hunchentoot, which depends on a wait-* function and it is to implement timeout on socket-accept because this posix idea of how timeouts are implemented leaks through usocket api iirc. If you look at many common lisp libraries, they dont have a concept of waiting. This can be fixed by adding a timeout to the socket-accept so that library code can clearly state its intent without imposing a specific (non-portable) implementation strategy. (If you really think wait-* is the right thing, why some usocket functions have the timeout argument and some are supposed to use the wait-* function?)

When I was porting iolib to windows, I found it useful to identify several programming models (summarized in the iolib README found in the iolib porting repo).

1 traditional blocking 2 traditional blocking with timeouts 3 select based 4 io completion ports based

Common Lisp libraries have basically code for 1 and 2 which are highly portable and lead to natural programming model. 3 and 4 require radically different programming models and are pretty much not used (except when 3 leaks through a broken api as in case of usocket and thus hunchentoot). Some people try to optimize by programing in 3 but there is a heavy price for that in terms of how one writes code and related consequences. I think 3 also in principle imposes at least one copy of io buffers unlike 4 where data don't have to be copied at all (from optimisation point of view 4 would be superior to posix way of doing it; from programming point of view, code is even worse than 3). Until it is clearer what an api for 3 and 4 should be, I think wait-* should not be in usocket. I actually ported iolib to 4 (io completion port backend) but haven't found a good way of integrating it with lisp so far, and using it just to emulate 1 and 2 seems stupid. 3 could be emulated on winapi I think, but then wouldn't 4 better way to go? Or maybe wouldn't a good api for 3 be a subset of api for 4? So far, there is no lisp code that could use 3 and 4, and it's not clear to me how such code would be made usable/interoperable in the context of common lisp.

For example, how does your code deal with partial input/output (a concept introduced by 3 and 4 programming models)?

There are some options:

a) buffer enough so that the partial io concept disappears above the io loop b) CPS - automatic - manual c) suspendable streams

a) is a good cheat. b) are very bad idea imho. c) is the thing I would like to explore and is the way I wish it works out in the future.

I would be really interested in how do you deal with that to write non-trivial software while exploiting the optimisation of posix select/(e)poll/kqueue. A web server that uses 3 or 4 to send bytes without any need for deep integration with the lisp environment is trivial from conceptual point of view. Every single attempt I've seen to use 3 so far has been of this kind without really addressing and solving the partial io issue. iolib doesn't really address this issue either, it works only for 1 and 2 programming models. if you want to go 3 or 4 with iolib, you get the ffi wrappers for the posix calls but as far as streams are concerned, iolib users are on their own.

Cheers,

Tomas

Hi binghe,

On 06/24/2013 05:25 PM, Chun Tian (binghe) wrote:

On 24/giu/2013, at 21:14, Tomas Hlavaty tomas.hlavaty@knowledgetools.de wrote:

...

...

I guess you're using Hunchentoot with the usocket emulations based on IOlib, but you didn't implement WAIT-FOR-INPUT, right? I think that's because IOlib never port their powerful I/O multiplexing interfaces to Windows, so far it only support three OS-provided functions: select (on general Unix), epoll (on Linux) and kqueue (on BSD systems). Maybe you should encourage IOlib author to support "WSAEventSelect", then you don't need to use a timeout arguemnt to socket-accept any more...

There is only one place in hunchentoot, which depends on a wait-* function and it is to implement timeout on socket-accept because this posix idea of how timeouts are implemented leaks through usocket api iirc. If you look at many common lisp libraries, they dont have a concept of waiting. This can be fixed by adding a timeout to the socket-accept so that library code can clearly state its intent without imposing a specific (non-portable) implementation strategy. (If you really think wait-* is the right thing, why some usocket functions have the timeout argument and some are supposed to use the wait-* function?)

WAIT-* has the ability to check the readiness status of *multiple* sockets at the same time.

which is irrelevant for existing common lisp libraries because none use this feature. There are people who try doing that, but so far write more or less "trivial" code or toys without addressing the real problems with this approach.

Without it you have to create some threads to achieve the same goal.

That's exactly how existing common lisp libraries do it, because it doesn't break the synchronicity of common lisp.

For application which only operate on single socket, there's no need to call WAIT-* at all, just set a read timeout would be enough.

Not true. hunchentoot _needs_ to use the wait-* function exactly because there is currently no other way to timeout on socket-accept.

It is unfortunate that wait-* leaks through the usocket api. API should allow to declare intent (timeout arg) and not to dictate how things are implemented (use posix select to timeout). The current situation seems to be up-side down. I think that the confusion stems from using the posix select for all those timeout related use-cases and event dispatch. For portable lisp code, it would be better not to conflate them.

So we provide timeouts for these simple case. (The question you asked me, can also be directly transferred to OS vendors)

Would it be possible to add timeout to the simple case of socket-accept? (And implement it in usocket using wait_* under the hood so that it doesnt leak and polute user code?) That would be a great step in the right direction.

Of course it would be also good to accomodate posix select style and maybe iocp style but on the other hand, waiting or completion are not specific socket features but rather general io event dispatch mechanisms so forcing it into a socket api seems a bit questionable. Maybe the api should simply expose the underlying fd/handle only?

I'm not an network programming expert, so can't easily answer your other questions. Any way, I don't expect too much from an compatibility layer project like usocket, and I'd better left those complex thing to people who is wring a complex networking software in Lisp, after all they have full access to everything they need from the OS. If CL implementation's exist stream facility can't afford the special need, it's possible to re-implement them by using Gray streams API. My duty is to slowly integrate those proven useful things into this compatibility layer and fill the gaps between different backends.

--binghe

The proven useful things in common lisp are synchronous calls and timeouts, not waiting on multiple fds. If there are any counter-example success stories, it would be great to hear about them:-)

Cheers,

Tomas

Hi binghe,

On 06/25/2013 01:36 AM, Chun Tian (binghe) wrote:

OK, I'll investigate how to add a timeout arg for socket-accept, thank you very much for pointing out this issue!

thank you very much!

Tomas

24.06.2013, 13:30, "Chun Tian (binghe)" binghe.lisp@gmail.com:

Among all those famous ASDF-based packages, I see hunchentoot also used the same approach to compile a special version with SSL support: by pushing :hunchentoot-no-ssl into *features*, hunchentoot can be compiled successfully on platforms (like the old MCL) where CL+SSL is not available yet, or when user simply don't need SSL for their web servers.

Yes, I know how this pattern is used in hunchentoot.

To solve all related issue, I'm going to do some runtime detection on *features*: if last compile time usocket was compiled with or without :usocket-iolib but current load time the feature set is different, ASDF should re-compile all usocket source files instead, not just load previous FASLs.  (I'm not sure if ASDF have already provided such a feature, so let me also copy this mail to Faré, the ASDF maintainer)

I don't like the idea of creating a whole new ASDF system like "usocket-iolib", because that will require other packages to change their system definitions to benefit from this new work.  And the most important thing, whether to depend on IOlib is totally an internal fair of usocket: it doesn't change the programming interface at all.  And the choice on if user want to use native network support of IOlib-based network support on their current platforms, ONLY depends on if they like to load additional DLLs (by CFFI).   I always want usocket  to depend on nothing, so that we can easily patch those 24x7 lisp servers to upgrade the networking support smoothly.

I agree that the usocket clients (application and other libraries) should work via the API and do not depend on particular implementation. What I suggest is to make the implementation switchable at runtime, instead of compile time. I think the solution will be simpler and more flexible solution. Up the the level that we can have at the same time in the same lisp image both IOlib sockets and sockets based on the API provided by the Lisp implementation.

Best regards, - Anton

We have a similar problem with CLSQL; one API with multiple database backends. CLSQL's backend choice is a little different; the backend choice is a user-facing decision whereas usocket choosing iolib is an internal matter, but I thought I'd offer our approach.

We solve it in two ways:

* an ASD files for each backend: clsql-mysql, clsql-sqlite3, etc. (akin to the proposed usocket-iolib) * a generic ASD file that tries to dynamically load more backends on request; if we try to connect to a :sqlite3 database, the connect function checks to see if clsql-sqlite3 is loaded, and if not, issues the load on the spot - see https://github.com/UnwashedMeme/clsql/blob/master/sql/database.lisp#L99

So library users can specify which backend they want via a direct dependency via ASDF, or let the environment take care of it. It's not the cleanest solution in the world and has some restrictions, but it's worked pretty well.

Thanks,

Ryan Davis Director of Programming, Acceleration.net 2837 NW 41st Street, Unit 320 Gainesville, FL 32606 352-335-6500 x124 http://www.acceleration.net

On 06/24/2013 06:27 AM, Chun Tian (binghe) wrote:

Hi Anton

On 24/giu/2013, at 18:10, Anton Vodonosov avodonosov@yandex.ru wrote:

...

...

To solve all related issue, I'm going to do some runtime detection on *features*: if last compile time usocket was compiled with or without :usocket-iolib but current load time the feature set is different, ASDF should re-compile all usocket source files instead, not just load previous FASLs. (I'm not sure if ASDF have already provided such a feature, so let me also copy this mail to Faré, the ASDF maintainer)

I don't like the idea of creating a whole new ASDF system like "usocket-iolib", because that will require other packages to change their system definitions to benefit from this new work. And the most important thing, whether to depend on IOlib is totally an internal fair of usocket: it doesn't change the programming interface at all. And the choice on if user want to use native network support of IOlib-based network support on their current platforms, ONLY depends on if they like to load additional DLLs (by CFFI). I always want usocket to depend on nothing, so that we can easily patch those 24x7 lisp servers to upgrade the networking support smoothly.

I agree that the usocket clients (application and other libraries) should work via the API and do not depend on particular implementation. What I suggest is to make the implementation switchable at runtime, instead of compile time. I think the solution will be simpler and more flexible solution. Up the the level that we can have at the same time in the same lisp image both IOlib sockets and sockets based on the API provided by the Lisp implementation.

It's possible to implement the runtime switches, and I admit this is a good idea when IOlib is being depended by usocket. Now I think it's also possible to provide a standalone, new ASDF system "usocket-iolib", which *explicitly* make sure IOlib is used as backend. But all my previous ideas should still work, there's no conflicts I can see.

...

Best regards,

• Anton

I did some more research to find that nothing has moved in terms of implementation support for IPv6 (in SBCL and CCL, fwiw). I think this should change, in particular as the required changes to sb-bsd-sockets and l1-sockets.lisp are rather small, and it is very embarrassing that the two most popular open source CL implementations still ignore IPv6.

I'm going to come up with a more complete IPv6 update to sb-bsd-sockets in the next few days and try to get it merged into SBCL.

From looking at

http://sourceforge.net/p/sbcl/mailman/sbcl-devel/thread/4E820C9E-33E2-43B2-B..., I see that there has been some work on USOCKET to support IPv6. Has that gone anywhere yet?

Finally, would you consider moving USOCKET development to git? GitHub makes collaboration so much easier.

Cheers, Hans

2014-09-17 9:36 GMT+02:00 Hans Hübner hans.huebner@gmail.com:

Hi Chun,

this discussion thread has kind of ended in nothing and I'm now returning to find out how IPv6 support can be added to Hunchentoot and Drakma. A few options are on the table:

• Make USOCKET support IPv6
• Create IOLib backend for USOCKET
• Add Implementation specific code to Drakma and Hunchentoot

I have given up on the IOLib idea, basically because it uses a shared library that cannot be built using quicklisp.

What is the status of IPv6 support in USOCKET? Is there any hope to see it happen?

Thanks, Hans

2013-06-25 15:12 GMT+02:00 Ryan Davis ryan@acceleration.net:

...

We have a similar problem with CLSQL; one API with multiple database backends. CLSQL's backend choice is a little different; the backend choice is a user-facing decision whereas usocket choosing iolib is an internal matter, but I thought I'd offer our approach.

We solve it in two ways:

• an ASD files for each backend: clsql-mysql, clsql-sqlite3, etc.

(akin to the proposed usocket-iolib) - a generic ASD file that tries to dynamically load more backends on request; if we try to connect to a :sqlite3 database, the connect function checks to see if clsql-sqlite3 is loaded, and if not, issues the load on the spot - see https://github.com/UnwashedMeme/clsql/blob/master/sql/database.lisp#L99

So library users can specify which backend they want via a direct dependency via ASDF, or let the environment take care of it. It's not the cleanest solution in the world and has some restrictions, but it's worked pretty well.

Thanks,

Ryan Davis Director of Programming, Acceleration.net 2837 NW 41st Street, Unit 320 Gainesville, FL 32606 352-335-6500 x124http://www.acceleration.net

On 06/24/2013 06:27 AM, Chun Tian (binghe) wrote:

Hi Anton

On 24/giu/2013, at 18:10, Anton Vodonosov avodonosov@yandex.ru avodonosov@yandex.ru wrote:

To solve all related issue, I'm going to do some runtime detection on *features*: if last compile time usocket was compiled with or without :usocket-iolib but current load time the feature set is different, ASDF should re-compile all usocket source files instead, not just load previous FASLs. (I'm not sure if ASDF have already provided such a feature, so let me also copy this mail to Faré, the ASDF maintainer)

I don't like the idea of creating a whole new ASDF system like "usocket-iolib", because that will require other packages to change their system definitions to benefit from this new work. And the most important thing, whether to depend on IOlib is totally an internal fair of usocket: it doesn't change the programming interface at all. And the choice on if user want to use native network support of IOlib-based network support on their current platforms, ONLY depends on if they like to load additional DLLs (by CFFI). I always want usocket to depend on nothing, so that we can easily patch those 24x7 lisp servers to upgrade the networking support smoothly.

I agree that the usocket clients (application and other libraries) should work via the API and do not depend on particular implementation. What I suggest is to make the implementation switchable at runtime, instead of compile time. I think the solution will be simpler and more flexible solution. Up the the level that we can have at the same time in the same lisp image both IOlib sockets and sockets based on the API provided by the Lisp implementation.

It's possible to implement the runtime switches, and I admit this is a good idea when IOlib is being depended by usocket. Now I think it's also possible to provide a standalone, new ASDF system "usocket-iolib", which *explicitly* make sure IOlib is used as backend. But all my previous ideas should still work, there's no conflicts I can see.

Best regards,

• Anton

Jan,

awesome, I very happy to not have to implement IPv6 myself, and I would love to see it if your patch would make it into SBCL soon. Do you have time to get that going?

Thanks! Hans

2014-09-17 10:58 GMT+02:00 Jan Moringen jmoringe@techfak.uni-bielefeld.de:

On Wed, 2014-09-17 at 10:08 +0200, Hans Hübner wrote:

...

I'm going to come up with a more complete IPv6 update to sb-bsd-sockets in the next few days and try to get it merged into SBCL.

In the meantime, I extended the patch you mentioned to the attached form.

Should I polish it, get it reviewed by other SBCL developers and commit it or would you prefer to write IPv6 support yourself, maybe using the patch as a starting point? (I'm not asking sarcastically since you may have a very different implementation in mind)

Kind regards, Jan

Hi Hans,

Beside other questions, I agree to move USOCKET development to Git. However my exist knowledge to Git cannot do such migration in short time, I hope you can do this for USOCKET, as you’re a maintainer of USOCKET too. (But please don’t put the Git repository on common-lisp.net servers, I think it’s not quite stable. I prefer SourceForge or Github. Once the new Git repository is set up, we’ll do rest of the work on Git instead of SVN.

I’m very sorry that I was not active in the past year. One reason is that I’m keep working for Gensym Corporation to maintain & develop the next version of their flagship Lisp-based product, Gensym G2, and the work is extremely busy. The other reason is that I’m trying to go back to college again for further study, and now I’m in Italy, will shortly be a student of University of Bologna.

For the IPv6 plan, pushing SBCL & CCL community to add the feature is easier for USOCKET and better for future maintenance. On the other side, providing fully loadable IPv6 patches by ourself is hard but provided the possibility to let users on old SBCL/CCL versions directly benefit from IPv6. I’m not sure which approach is the best, but fortunately these approaches are not conflict with each other (so, choosing which plans totally depends on the progress of each). For example, one major commercial CL platform is going to support UDP natively, thus all the UDP loadable patches provided by USOCKET will be obsoleted, but these code will not be removed as they still provide the possibility for users on old versions of that platform to use UDP. (upgrading a commercial platform needs money, upgrading a free platform needs courage, that’s at the same level from my view).

Regards,

Chun Tian (binghe)

Il giorno 17/set/2014, alle ore 10:08, Hans Hübner hans.huebner@gmail.com ha scritto:

I did some more research to find that nothing has moved in terms of implementation support for IPv6 (in SBCL and CCL, fwiw). I think this should change, in particular as the required changes to sb-bsd-sockets and l1-sockets.lisp are rather small, and it is very embarrassing that the two most popular open source CL implementations still ignore IPv6.

I'm going to come up with a more complete IPv6 update to sb-bsd-sockets in the next few days and try to get it merged into SBCL.

From looking at http://sourceforge.net/p/sbcl/mailman/sbcl-devel/thread/4E820C9E-33E2-43B2-B..., I see that there has been some work on USOCKET to support IPv6. Has that gone anywhere yet?

Finally, would you consider moving USOCKET development to git? GitHub makes collaboration so much easier.

Cheers, Hans

2014-09-17 9:36 GMT+02:00 Hans Hübner hans.huebner@gmail.com: Hi Chun,

this discussion thread has kind of ended in nothing and I'm now returning to find out how IPv6 support can be added to Hunchentoot and Drakma. A few options are on the table:

• Make USOCKET support IPv6
• Create IOLib backend for USOCKET
• Add Implementation specific code to Drakma and Hunchentoot

I have given up on the IOLib idea, basically because it uses a shared library that cannot be built using quicklisp.

What is the status of IPv6 support in USOCKET? Is there any hope to see it happen?

Thanks, Hans

2013-06-25 15:12 GMT+02:00 Ryan Davis ryan@acceleration.net: We have a similar problem with CLSQL; one API with multiple database backends. CLSQL's backend choice is a little different; the backend choice is a user-facing decision whereas usocket choosing iolib is an internal matter, but I thought I'd offer our approach.

We solve it in two ways: an ASD files for each backend: clsql-mysql, clsql-sqlite3, etc. (akin to the proposed usocket-iolib) a generic ASD file that tries to dynamically load more backends on request; if we try to connect to a :sqlite3 database, the connect function checks to see if clsql-sqlite3 is loaded, and if not, issues the load on the spot - see https://github.com/UnwashedMeme/clsql/blob/master/sql/database.lisp#L99 So library users can specify which backend they want via a direct dependency via ASDF, or let the environment take care of it. It's not the cleanest solution in the world and has some restrictions, but it's worked pretty well.

Thanks,

Ryan Davis Director of Programming, Acceleration.net 2837 NW 41st Street, Unit 320 Gainesville, FL 32606 352-335-6500 x124 http://www.acceleration.net On 06/24/2013 06:27 AM, Chun Tian (binghe) wrote:

...

Hi Anton

On 24/giu/2013, at 18:10, Anton Vodonosov avodonosov@yandex.ru wrote:

...

...

To solve all related issue, I'm going to do some runtime detection on *features*: if last compile time usocket was compiled with or without :usocket-iolib but current load time the feature set is different, ASDF should re-compile all usocket source files instead, not just load previous FASLs. (I'm not sure if ASDF have already provided such a feature, so let me also copy this mail to Faré, the ASDF maintainer)

I don't like the idea of creating a whole new ASDF system like "usocket-iolib", because that will require other packages to change their system definitions to benefit from this new work. And the most important thing, whether to depend on IOlib is totally an internal fair of usocket: it doesn't change the programming interface at all. And the choice on if user want to use native network support of IOlib-based network support on their current platforms, ONLY depends on if they like to load additional DLLs (by CFFI). I always want usocket to depend on nothing, so that we can easily patch those 24x7 lisp servers to upgrade the networking support smoothly.

I agree that the usocket clients (application and other libraries) should work via the API and do not depend on particular implementation. What I suggest is to make the implementation switchable at runtime, instead of compile time. I think the solution will be simpler and more flexible solution. Up the the level that we can have at the same time in the same lisp image both IOlib sockets and sockets based on the API provided by the Lisp implementation.

It's possible to implement the runtime switches, and I admit this is a good idea when IOlib is being depended by usocket. Now I think it's also possible to provide a standalone, new ASDF system "usocket-iolib", which *explicitly* make sure IOlib is used as backend. But all my previous ideas should still work, there's no conflicts I can see.

...

Best regards,

• Anton

Thank you very much...

And I’d like to add some notes to the exist IPv6 support status of CL platforms currently supported by USOCKET:

AllegroCL supports IPv6 since very early versions [0]; LispWorks supports IPv6 since version 6.1 [1], IPv4 UDP support is provided by USOCKET. CLISP doesn’t support IPv6, neither the socket streams API [2] nor rawsock API [3]; CMUCL doesn’t support IPv6; Scieneer CL (SCL) doesn’t support IPv6 [4], the platform and its networking API is totally derived from CMUCL; SBCL doesn’t support IPv6; ECL doesn’t support IPv6, but its networking support is based on SB-BSD-SOCKETS interface, can be done with SBCL together. Clozure CL (CCL) doesn’t support IPv6 [5]; Armed Bear Common Lisp (ABCL) doesn’t support IPv6, and its whole IPv4 support is provided by USOCKET. Macintosh Common Lisp (MCL) as a dead platform, not to mention, it doesn’t support IPv6 and even doesn’t support Mac OS X 10.7+. ^_^ MOCL, the new iOS CL platform, no access to it, but I think they don’t support IPv6.

Basically, that’s not a shame for SBCL/CCL at all. Among all CL platforms in this world, only Allegro CL and the current latest version of LispWorks (6.1) support IPv6. And technically speaking, the only bottleneck (toward IPv6) in these platforms, is to not treat IP address as a 32-bit integer any more, instead, a 128-bit black-box structure (like what’s done in LispWorks 6.1, structure 'COMM:IPv6-Address) should be accepted by all networking APIs.

Hope this helps,

Chun

[0] http://franz.com/support/documentation/9.0/doc/socket.htm#socket-ipv6-1 [1] http://www.lispworks.com/documentation/lw61/RNIG/html/readme-393.htm#pgfId-9... [2] http://clisp.podval.org/impnotes/socket.html [3] http://clisp.podval.org/impnotes/rawsock.html [4] http://www.scieneer.com/scl/doc/sockets.html [5] http://ccl.clozure.com/ccl-documentation.html#Sockets-Overview

Il giorno 18/set/2014, alle ore 14:21, Hans Hübner hans.huebner@gmail.com ha scritto:

Hi Chun,

thank you for getting back, and no worries: Studying should be great and maybe you'll have some time to return to open source work in the future.

I will be moving usocket's main repository to github as soon as I'm finished with IPv6 support in Clozure CL. I will let y'all know when that has finished.

Personally, I will not be maintaining patches to add IPv6 support to older lisp implementations. We should be able to get native support into the current versions of CCL and SBCL, and it will probably also be possible to support Allegro CL and Lispworks, although that is lower on my priority list. If you or anyone else want to maintain IPv6 patches to older compilers, I'd not be opposed as long as that does not put a burden onto people who use the current versions with native support. It will make sense to wait until we have something for the current versions so that the changes made for that can be used as model or be back ported.

I'll keep you updated on the progress.

-Hans

2014-09-18 13:59 GMT+02:00 Chun Tian (binghe) binghe.lisp@gmail.com: Hi Hans,

Beside other questions, I agree to move USOCKET development to Git. However my exist knowledge to Git cannot do such migration in short time, I hope you can do this for USOCKET, as you’re a maintainer of USOCKET too. (But please don’t put the Git repository on common-lisp.net servers, I think it’s not quite stable. I prefer SourceForge or Github. Once the new Git repository is set up, we’ll do rest of the work on Git instead of SVN.

I’m very sorry that I was not active in the past year. One reason is that I’m keep working for Gensym Corporation to maintain & develop the next version of their flagship Lisp-based product, Gensym G2, and the work is extremely busy. The other reason is that I’m trying to go back to college again for further study, and now I’m in Italy, will shortly be a student of University of Bologna.

For the IPv6 plan, pushing SBCL & CCL community to add the feature is easier for USOCKET and better for future maintenance. On the other side, providing fully loadable IPv6 patches by ourself is hard but provided the possibility to let users on old SBCL/CCL versions directly benefit from IPv6. I’m not sure which approach is the best, but fortunately these approaches are not conflict with each other (so, choosing which plans totally depends on the progress of each). For example, one major commercial CL platform is going to support UDP natively, thus all the UDP loadable patches provided by USOCKET will be obsoleted, but these code will not be removed as they still provide the possibility for users on old versions of that platform to use UDP. (upgrading a commercial platform needs money, upgrading a free platform needs courage, that’s at the same level from my view).

Regards,

Chun Tian (binghe)

Il giorno 17/set/2014, alle ore 10:08, Hans Hübner hans.huebner@gmail.com ha scritto:

...

I did some more research to find that nothing has moved in terms of implementation support for IPv6 (in SBCL and CCL, fwiw). I think this should change, in particular as the required changes to sb-bsd-sockets and l1-sockets.lisp are rather small, and it is very embarrassing that the two most popular open source CL implementations still ignore IPv6.

I'm going to come up with a more complete IPv6 update to sb-bsd-sockets in the next few days and try to get it merged into SBCL.

From looking at http://sourceforge.net/p/sbcl/mailman/sbcl-devel/thread/4E820C9E-33E2-43B2-B..., I see that there has been some work on USOCKET to support IPv6. Has that gone anywhere yet?

Finally, would you consider moving USOCKET development to git? GitHub makes collaboration so much easier.

Cheers, Hans

2014-09-17 9:36 GMT+02:00 Hans Hübner hans.huebner@gmail.com: Hi Chun,

this discussion thread has kind of ended in nothing and I'm now returning to find out how IPv6 support can be added to Hunchentoot and Drakma. A few options are on the table:

• Make USOCKET support IPv6
• Create IOLib backend for USOCKET
• Add Implementation specific code to Drakma and Hunchentoot

I have given up on the IOLib idea, basically because it uses a shared library that cannot be built using quicklisp.

What is the status of IPv6 support in USOCKET? Is there any hope to see it happen?

Thanks, Hans

2013-06-25 15:12 GMT+02:00 Ryan Davis ryan@acceleration.net: We have a similar problem with CLSQL; one API with multiple database backends. CLSQL's backend choice is a little different; the backend choice is a user-facing decision whereas usocket choosing iolib is an internal matter, but I thought I'd offer our approach.

We solve it in two ways: an ASD files for each backend: clsql-mysql, clsql-sqlite3, etc. (akin to the proposed usocket-iolib) a generic ASD file that tries to dynamically load more backends on request; if we try to connect to a :sqlite3 database, the connect function checks to see if clsql-sqlite3 is loaded, and if not, issues the load on the spot - see https://github.com/UnwashedMeme/clsql/blob/master/sql/database.lisp#L99 So library users can specify which backend they want via a direct dependency via ASDF, or let the environment take care of it. It's not the cleanest solution in the world and has some restrictions, but it's worked pretty well.

Thanks,

Ryan Davis Director of Programming, Acceleration.net 2837 NW 41st Street, Unit 320 Gainesville, FL 32606 352-335-6500 x124 http://www.acceleration.net On 06/24/2013 06:27 AM, Chun Tian (binghe) wrote:

...

Hi Anton

On 24/giu/2013, at 18:10, Anton Vodonosov avodonosov@yandex.ru wrote:

...

...

To solve all related issue, I'm going to do some runtime detection on *features*: if last compile time usocket was compiled with or without :usocket-iolib but current load time the feature set is different, ASDF should re-compile all usocket source files instead, not just load previous FASLs. (I'm not sure if ASDF have already provided such a feature, so let me also copy this mail to Faré, the ASDF maintainer)

I don't like the idea of creating a whole new ASDF system like "usocket-iolib", because that will require other packages to change their system definitions to benefit from this new work. And the most important thing, whether to depend on IOlib is totally an internal fair of usocket: it doesn't change the programming interface at all. And the choice on if user want to use native network support of IOlib-based network support on their current platforms, ONLY depends on if they like to load additional DLLs (by CFFI). I always want usocket to depend on nothing, so that we can easily patch those 24x7 lisp servers to upgrade the networking support smoothly.

I agree that the usocket clients (application and other libraries) should work via the API and do not depend on particular implementation. What I suggest is to make the implementation switchable at runtime, instead of compile time. I think the solution will be simpler and more flexible solution. Up the the level that we can have at the same time in the same lisp image both IOlib sockets and sockets based on the API provided by the Lisp implementation.

It's possible to implement the runtime switches, and I admit this is a good idea when IOlib is being depended by usocket. Now I think it's also possible to provide a standalone, new ASDF system "usocket-iolib", which *explicitly* make sure IOlib is used as backend. But all my previous ideas should still work, there's no conflicts I can see.

...

Best regards,

• Anton

Hi Anton,

in theory, it's a noble target. Problem is that there are inherently non-portable things to consider, e.g. how does iolib handle control-c? So for each supported implementation, there needs to be a bit of code that cuts directly into the lowest level of the host lisp environment. It seems doable though. But still, cffi dependency can be undesirable so the original (non-iolib) usocket is still a better solution in those cases.

Also, usocket has heavy posix bias. Ideally, it would be better to identify possible use-cases and create and api that would allow to embrace any of them. For example, the idea of wait-* function makes usocket inherently non-portable to non-posix systems. I am still searching for such unifying all embracing io api:-)

Cheers,

Tomas

On 06/24/2013 12:18 PM, Anton Vodonosov wrote:

I also want to highlight the advantage of IOLib based sockets.

The underlying socket API provided by most of the OSes today is the same - BSD sockets. Various Lisp implementations multiply this basic API by proving their own, distinct lisp versions of socket APIs. On top of it we create a compatibility layer, to unify the API.

OSes: 1 API "BSD sockets" => Lisp impls: 10 APIs => usocket 1 API.

If we use some FFI library like IOLib to access OS socket APIs we avoid all these redundant 10 intermediate APIs.

Best regards,

• Anton

Sure. I don't mean my note as an counter argument to your approach. Just noting the funny situation with multiplied API we have in Lisp. BTW, Win32 socket API is based on BSD sockets.

24.06.2013, 14:37, "Chun Tian (binghe)" binghe.lisp@gmail.com:

You're right, and I want to support as much approaches as possible.

A obvious fact is, there're also platforms not based on BSD sockets, at least not pure.  For example, all implementations on Win32, and MCL (based on OpenTransport), Lisp machines, and ABCL running in JVM.  I want to support all of them.   And for those Win32 users without an installed C compiler, they cannot load IOlib easily, but if they only want to make a TCP stream and send some data, shouldn't the exist usocket package be their best choice?

On 24/giu/2013, at 18:18, Anton Vodonosov avodonosov@yandex.ru wrote:

...

I also want to highlight the advantage of IOLib based sockets.

The underlying socket API provided by most of the OSes today is the same - BSD sockets.  Various Lisp implementations multiply this basic API by proving their own, distinct lisp versions of socket APIs.  On top of it we create a compatibility layer, to unify the API.

OSes: 1 API "BSD sockets"  => Lisp impls: 10 APIs => usocket 1 API.

If we use some FFI library like IOLib to access OS socket APIs we avoid all these redundant 10 intermediate APIs.

Best regards,  - Anton