Thank you... I am aware of the implementation dependent solutions regarding networking. However, if you stick to USOCKET, would it be better to switch to an explicit loop, or is there some other incantation you could use to avoid the hanging read? Thanks Marco On Wed, Dec 17, 2025 at 3:11 PM David McClain <dbm@refined-audiometrics.com> wrote:

It sounds like you are using TCP/IP (the streaming protocol) vs UDP?

I have had tremendous success using the LW Asynchronous Socket interface for socket streams (TCP/IP). Size of sent phrases seems to make no difference to its performance. My transfers have arbitrary sizes and have no predictable length.

I do use a self-synchronizing encoding (Phuang) across the wire, and I think that ultimately becomes a length-prefix followed by data octets. And these packets come in a variety of sizes.

...

On Dec 17, 2025, at 01:01, Marco Antoniotti <marco.antoniotti@unimib.it> wrote:

Hi

I am reading from a uosocket:stream-usocket. My sequence is 100 octets long, but I know the other party is sending UP TO 100 octets.

If the other party sends less than 100 octets, read-sequence/usocket-stream (at least the version on LW) hangs.

Should I just do a loop, checking for the "end marker"?

Thanks

All the best

-- Marco Antoniotti

-- Marco Antoniotti, Professor, Director tel. +39 - 02 64 48 79 01 DISCo, University of Milan-Bicocca U14 2043 http://dcb.disco.unimib.it Viale Sarca 336 I-20126 Milan (MI) ITALY REGAINS: https://regains.disco.unimib.it/

I’ll have to review Paul Phuong’s self-sync protocol to be sure. In my system with self-sync encoding, I do believe that there is a special sequence of octets that indicate a boundary between data packets. But inside the packets is also a length count and checksum, to tell you what to expect. But overall, with self-sync encoding you don’t depend on receiving a prefix count. You simply look for a magic sequence of octets in the stream, whenever they arrive. No assumptions are made on segments being read. You simply read what is available (for the asynchronous case) and look to see (with a state machine) whether the magic octet sequence has been seen.

On Dec 17, 2025, at 08:17, David McClain <dbm@refined-audiometrics.com> wrote:

Well it sounds, from the comments made by you and me, that the key is to send a prefix length in some fixed number of octets, which then describes how many data octets to expect in your next read.

The process only secondarily depends on asynchronous/synchronous behavior. You really should block, somewhere, until you receive the proper number of bytes, or generate a timeout error on missing data. And the prefix length (which itself has a known length) tells you what to expect.

...

On Dec 17, 2025, at 08:11, Marco Antoniotti <marco.antoniotti@unimib.it> wrote:

Thank you...

I am aware of the implementation dependent solutions regarding networking.

However, if you stick to USOCKET, would it be better to switch to an explicit loop, or is there some other incantation you could use to avoid the hanging read?

Thanks

Marco

On Wed, Dec 17, 2025 at 3:11 PM David McClain <dbm@refined-audiometrics.com <mailto:dbm@refined-audiometrics.com>> wrote:

...

It sounds like you are using TCP/IP (the streaming protocol) vs UDP?

I have had tremendous success using the LW Asynchronous Socket interface for socket streams (TCP/IP). Size of sent phrases seems to make no difference to its performance. My transfers have arbitrary sizes and have no predictable length.

I do use a self-synchronizing encoding (Phuang) across the wire, and I think that ultimately becomes a length-prefix followed by data octets. And these packets come in a variety of sizes.

...

On Dec 17, 2025, at 01:01, Marco Antoniotti <marco.antoniotti@unimib.it <mailto:marco.antoniotti@unimib.it>> wrote:

Hi

I am reading from a uosocket:stream-usocket. My sequence is 100 octets long, but I know the other party is sending UP TO 100 octets.

If the other party sends less than 100 octets, read-sequence/usocket-stream (at least the version on LW) hangs.

Should I just do a loop, checking for the "end marker"?

Thanks

All the best

-- Marco Antoniotti

-- Marco Antoniotti, Professor, Director tel. +39 - 02 64 48 79 01 DISCo, University of Milan-Bicocca U14 2043 http://dcb.disco.unimib.it <http://dcb.disco.unimib.it/> Viale Sarca 336 I-20126 Milan (MI) ITALY

REGAINS: https://regains.disco.unimib.it/

But - self sync enables flexibility in data packet sizes, without being prone to stuffing. I use the term stuffing here to denote a kind of network malicious attack. Had I used prefix-length encoding then an attacker could shove a packet with an enormous size and cause DOS by overwhelming my system resources. Self-sync can prevent that by looking only for the magic separators at known locations.

On Dec 17, 2025, at 08:30, David McClain <dbm@refined-audiometrics.com> wrote:

IIRC, the self-sync data format was actually invented to help with read-back of log files which might get clobbered in random locations. TCP/IP is actually error-free transmission in its own right, and you shouldn’t ever see clobbered data arriving. The only error you should see, at the application level, is a timeout for failure to transmit successfully.

So self-sync format is probably redundant in socket streams...

...

On Dec 17, 2025, at 08:23, David McClain <dbm@refined-audiometrics.com> wrote:

I’ll have to review Paul Phuong’s self-sync protocol to be sure. In my system with self-sync encoding, I do believe that there is a special sequence of octets that indicate a boundary between data packets. But inside the packets is also a length count and checksum, to tell you what to expect.

But overall, with self-sync encoding you don’t depend on receiving a prefix count. You simply look for a magic sequence of octets in the stream, whenever they arrive. No assumptions are made on segments being read. You simply read what is available (for the asynchronous case) and look to see (with a state machine) whether the magic octet sequence has been seen.

...

On Dec 17, 2025, at 08:17, David McClain <dbm@refined-audiometrics.com> wrote:

Well it sounds, from the comments made by you and me, that the key is to send a prefix length in some fixed number of octets, which then describes how many data octets to expect in your next read.

The process only secondarily depends on asynchronous/synchronous behavior. You really should block, somewhere, until you receive the proper number of bytes, or generate a timeout error on missing data. And the prefix length (which itself has a known length) tells you what to expect.

...

On Dec 17, 2025, at 08:11, Marco Antoniotti <marco.antoniotti@unimib.it> wrote:

Thank you...

I am aware of the implementation dependent solutions regarding networking.

However, if you stick to USOCKET, would it be better to switch to an explicit loop, or is there some other incantation you could use to avoid the hanging read?

Thanks

Marco

On Wed, Dec 17, 2025 at 3:11 PM David McClain <dbm@refined-audiometrics.com <mailto:dbm@refined-audiometrics.com>> wrote:

...

It sounds like you are using TCP/IP (the streaming protocol) vs UDP?

I have had tremendous success using the LW Asynchronous Socket interface for socket streams (TCP/IP). Size of sent phrases seems to make no difference to its performance. My transfers have arbitrary sizes and have no predictable length.

I do use a self-synchronizing encoding (Phuang) across the wire, and I think that ultimately becomes a length-prefix followed by data octets. And these packets come in a variety of sizes.

...

On Dec 17, 2025, at 01:01, Marco Antoniotti <marco.antoniotti@unimib.it <mailto:marco.antoniotti@unimib.it>> wrote:

Hi

I am reading from a uosocket:stream-usocket. My sequence is 100 octets long, but I know the other party is sending UP TO 100 octets.

If the other party sends less than 100 octets, read-sequence/usocket-stream (at least the version on LW) hangs.

Should I just do a loop, checking for the "end marker"?

Thanks

All the best

-- Marco Antoniotti

-- Marco Antoniotti, Professor, Director tel. +39 - 02 64 48 79 01 DISCo, University of Milan-Bicocca U14 2043 http://dcb.disco.unimib.it <http://dcb.disco.unimib.it/> Viale Sarca 336 I-20126 Milan (MI) ITALY

REGAINS: https://regains.disco.unimib.it/

No, that would only cause a sync error, not a DOS.

On Dec 17, 2025, at 09:17, Stelian Ionescu <sionescu@cddr.org> wrote:

...

Had I used prefix-length encoding then an attacker could shove a packet with an enormous size and cause DOS by overwhelming my system resources. Self-sync can prevent that by looking only for the magic separators at known locations.

Then an attacker will cause DOS by simply ommitting the magic separator :D

-- Stelian Ionescu

DDoS in a way that when the receiver blocks it blocks a thread and either you can’t receive anything anymore on other connections. Or it will consume a ton of other threads which may all block.

??? I don’t understand what you are trying to say here. I cannot imagine a situation in which my Async Sockets get blocked in that manner. And the self-sync protocol will not get tangled up on any inputs. Errors simply cause the self sync to scan forward, dropping all, until it recognizes another start-sync sequence. ——————— FWIW, you care welcome to peruse my implementation of Async Sockets and self-sync protocol here: https://github.com/dbmcclain/Lisp-Actors/tree/main/xTActors/secure-channel

Yes, ok. I really don’t know the mechanism used by LW underneath the Async Sockets. Perhaps a blocking read + periodic timeout? It isn’t important for my purposes to know how it happens. But I can tell you that it is very effective, from the perspective of my application code. I do know that it fires up a separate thread to run a collection manager. Probably does not need one thread per socket. I suspect there is a Unix Select() involved when more than one socket is assigned to an asynchronous collection manager. When I first started using it, the interface seemed a bit arcane. But after a short time, it became second nature. It makes perfect sense to have an asynchronous socket system in my code. I don’t ask for data from any source. But I respond asynchronously to messages posed by any data source.

On Dec 17, 2025, at 11:39, Manfred Bergmann <manfred.bergmann@me.com> wrote:

I meant in case we have a blocking situation. Of course a socket read timeout will eventually release the blocking.

I didn’t look closely at the async implementation. But I’d guess it’s just async over a socket that block on reading?

...

Am 17.12.2025 um 17:57 schrieb David McClain <dbm@refined-audiometrics.com <mailto:dbm@refined-audiometrics.com>>:

...

DDoS in a way that when the receiver blocks it blocks a thread and either you can’t receive anything anymore on other connections. Or it will consume a ton of other threads which may all block.

??? I don’t understand what you are trying to say here.

I cannot imagine a situation in which my Async Sockets get blocked in that manner. And the self-sync protocol will not get tangled up on any inputs. Errors simply cause the self sync to scan forward, dropping all, until it recognizes another start-sync sequence.

——————— FWIW, you care welcome to peruse my implementation of Async Sockets and self-sync protocol here:

https://github.com/dbmcclain/Lisp-Actors/tree/main/xTActors/secure-channel

What did you (MA) mean by "hangs"? If read-sequence/usocket-stream is built upon or named to resemble READ-SEQUENCE, isn't it just fulfilling its contract? As described in the ANS, R-S never returns normally until it has read and modified the entire sequence (modified by :START/:END) unless end of file occurs. Perhaps READ-SEQUENCE is not the correct API for what you need for your communication channel, since the contract of R-S is inappropriate upon which to build a variable-record-length protocol. Several previous comments were obviously constructed on top of this understanding about READ-SEQUENCE, but I didn't see any that made explicit this important point. During the X3J13 process the committee realized that the ancient Lisp I/O functions, which might have been fine for single-char teletype support, would need bulk functions in the language or else each implementation would have to invent its own (which is eventually how it worked out). READ- and WRITE-SEQUENCE were added as the place in the language to hang these things, trusting on mythological implementation competition forces to work out the details, but obviously it wasn't well thought out. As explained at the start of my message, R-S is both over- and under-defined to be the base for efficient communication protocols, modern circa the 1990's. On Wed, Dec 17, 2025 at 11:06 AM David McClain <dbm@refined-audiometrics.com> wrote:

Ahem… the Async Socket system *does not* require a separate thread per socket. I just run one collection manager thread to handle all of the socket connections established with my systems. All of my systems act as either client or server, depending on who initiates a connection. And they all only need one collection manager, which is developed on demand.

On Dec 17, 2025, at 11:59, David McClain <dbm@refined-audiometrics.com> wrote:

Yes, ok. I really don’t know the mechanism used by LW underneath the Async Sockets.

Perhaps a blocking read + periodic timeout? It isn’t important for my purposes to know how it happens. But I can tell you that it is very effective, from the perspective of my application code.

I do know that it fires up a separate thread to run a collection manager. Probably does not need one thread per socket. I suspect there is a Unix Select() involved when more than one socket is assigned to an asynchronous collection manager.

When I first started using it, the interface seemed a bit arcane. But after a short time, it became second nature. It makes perfect sense to have an asynchronous socket system in my code. I don’t ask for data from any source. But I respond asynchronously to messages posed by any data source.

On Dec 17, 2025, at 11:39, Manfred Bergmann <manfred.bergmann@me.com> wrote:

I meant in case we have a blocking situation. Of course a socket read timeout will eventually release the blocking.

I didn’t look closely at the async implementation. But I’d guess it’s just async over a socket that block on reading?

Am 17.12.2025 um 17:57 schrieb David McClain <dbm@refined-audiometrics.com

...

:

DDoS in a way that when the receiver blocks it blocks a thread and either you can’t receive anything anymore on other connections. Or it will consume a ton of other threads which may all block.

??? I don’t understand what you are trying to say here.

I cannot imagine a situation in which my Async Sockets get blocked in that manner. And the self-sync protocol will not get tangled up on any inputs. Errors simply cause the self sync to scan forward, dropping all, until it recognizes another start-sync sequence.

——————— FWIW, you care welcome to peruse my implementation of Async Sockets and self-sync protocol here:

https://github.com/dbmcclain/Lisp-Actors/tree/main/xTActors/secure-channel

Hi Steve

I understand that, and I know about the idiosyncrasies of the bulk operations under different implementations. I am asking what would be the best way to, say, implement a READ-SEQ-NO-HANG (for want of a better name) using USOCKET provided primitives; yes I am working with sockets. Reading a byte at a time and listening would work? Any better ideas? I am fooling around as we speak. Any suggestion will be welcome; the only constraint: using USOCKET. Cheers

Why does it have to be Usocket ? You'd solve this problem by just using HTTP. -- Stelian Ionescu

Sorry Stelian, but that begs the question a bit. Why using CL when using GO solves the problem? Marco Antoniotti https://dcb.disco.unimib.it On Thu, 18 Dec 2025 at 01:56, Stelian Ionescu <sionescu@cddr.org> wrote:

Hi Steve

I understand that, and I know about the idiosyncrasies of the bulk operations under different implementations. I am asking what would be the best way to, say, implement a READ-SEQ-NO-HANG (for want of a better name) using USOCKET provided primitives; yes I am working with sockets. Reading a byte at a time and listening would work? Any better ideas? I am fooling around as we speak. Any suggestion will be welcome; the only constraint: using USOCKET. Cheers

Why does it have to be Usocket ? You'd solve this problem by just using HTTP.

-- Stelian Ionescu

See… the self-sync also uses embedded length codes and checksums. So we know where the next magic octets should appear. And if they are missing - as happens if the log file were clobbered - that allows us to simply skip to the next sync octet sequence. No massive overrun would occur. If someone embeds an extreme length, then we can decide to reject it. Probably we make a convention that large data must be chunked into 64 kB sections or so.

On Dec 17, 2025, at 09:17, Stelian Ionescu <sionescu@cddr.org> wrote:

...

Had I used prefix-length encoding then an attacker could shove a packet with an enormous size and cause DOS by overwhelming my system resources. Self-sync can prevent that by looking only for the magic separators at known locations.

Then an attacker will cause DOS by simply ommitting the magic separator :D

-- Stelian Ionescu

In my system, using Async LW Sockets, I get fragments delivered in arbitrary sizes. I label each arrival with a sequence number, and send the packet to a receiving collector task, which reassembles the packets in sequence order. So delivery to the collector can be arranged in any order and the bytes of a fragment will still be placed into correct position. I don’t use one-by-one reception. And I honestly don’t know the algorithm being used for reading in the LW Async Socket protocol. But I don’t have to know this either. I process the packet fragments serially, as your code indicates, but I receive a buffer full of octets each time.

On Dec 17, 2025, at 08:58, Marco Antoniotti <marco.antoniotti@unimib.it> wrote:

Ok David,

If you are saying that Phuong's self-sync protocol has "boundaries", then that's my case.

So

(loop for b = (read-byte socket nil nil) while (and b (/= b <special-byte>)) collect b)

And you forget READ-SEQUENCE, Of course, I'd bet that ACL READ-VECTOR does the same under the hood.

Cheers

MA

On Wed, Dec 17, 2025 at 4:44 PM David McClain <dbm@refined-audiometrics.com <mailto:dbm@refined-audiometrics.com>> wrote:

...

I’ll have to review Paul Phuong’s self-sync protocol to be sure. In my system with self-sync encoding, I do believe that there is a special sequence of octets that indicate a boundary between data packets. But inside the packets is also a length count and checksum, to tell you what to expect.

But overall, with self-sync encoding you don’t depend on receiving a prefix count. You simply look for a magic sequence of octets in the stream, whenever they arrive. No assumptions are made on segments being read. You simply read what is available (for the asynchronous case) and look to see (with a state machine) whether the magic octet sequence has been seen.

...

On Dec 17, 2025, at 08:17, David McClain <dbm@refined-audiometrics.com <mailto:dbm@refined-audiometrics.com>> wrote:

Well it sounds, from the comments made by you and me, that the key is to send a prefix length in some fixed number of octets, which then describes how many data octets to expect in your next read.

The process only secondarily depends on asynchronous/synchronous behavior. You really should block, somewhere, until you receive the proper number of bytes, or generate a timeout error on missing data. And the prefix length (which itself has a known length) tells you what to expect.

...

On Dec 17, 2025, at 08:11, Marco Antoniotti <marco.antoniotti@unimib.it <mailto:marco.antoniotti@unimib.it>> wrote:

Thank you...

I am aware of the implementation dependent solutions regarding networking.

However, if you stick to USOCKET, would it be better to switch to an explicit loop, or is there some other incantation you could use to avoid the hanging read?

Thanks

Marco

On Wed, Dec 17, 2025 at 3:11 PM David McClain <dbm@refined-audiometrics.com <mailto:dbm@refined-audiometrics.com>> wrote:

...

It sounds like you are using TCP/IP (the streaming protocol) vs UDP?

I have had tremendous success using the LW Asynchronous Socket interface for socket streams (TCP/IP). Size of sent phrases seems to make no difference to its performance. My transfers have arbitrary sizes and have no predictable length.

I do use a self-synchronizing encoding (Phuang) across the wire, and I think that ultimately becomes a length-prefix followed by data octets. And these packets come in a variety of sizes.

...

On Dec 17, 2025, at 01:01, Marco Antoniotti <marco.antoniotti@unimib.it <mailto:marco.antoniotti@unimib.it>> wrote:

Hi

I am reading from a uosocket:stream-usocket. My sequence is 100 octets long, but I know the other party is sending UP TO 100 octets.

If the other party sends less than 100 octets, read-sequence/usocket-stream (at least the version on LW) hangs.

Should I just do a loop, checking for the "end marker"?

Thanks

All the best

-- Marco Antoniotti

-- Marco Antoniotti, Professor, Director tel. +39 - 02 64 48 79 01 DISCo, University of Milan-Bicocca U14 2043 http://dcb.disco.unimib.it <http://dcb.disco.unimib.it/> Viale Sarca 336 I-20126 Milan (MI) ITALY

REGAINS: https://regains.disco.unimib.it/

-- Marco Antoniotti, Professor, Director tel. +39 - 02 64 48 79 01 DISCo, University of Milan-Bicocca U14 2043 http://dcb.disco.unimib.it <http://dcb.disco.unimib.it/> Viale Sarca 336 I-20126 Milan (MI) ITALY

REGAINS: https://regains.disco.unimib.it/

Or use chunked encoding - https://github.com/edicl/chunga Am Do., 18. Dez. 2025 um 09:16 Uhr schrieb Josh Betts < josh@joshua-a-betts.com>:

Hi,

Yes probably.

Or as others have said encode the length at the start and then use read-sequence with :start and :end. If you are worried about a DOS you could come up with some solution to blacklist bad actors on your end, or use whitelists etc.

Regards,

Josh. On 17/12/2025 15:01, Marco Antoniotti wrote:

Hi

I am reading from a uosocket:stream-usocket. My sequence is 100 octets long, but I know the other party is sending UP TO 100 octets.

If the other party sends less than 100 octets, read-sequence/ usocket-stream (at least the version on LW) hangs.

Should I just do a loop, checking for the "end marker"?

Thanks

All the best

-- Marco Antoniotti