I tried changing (make-array buffer-size :element-type 'character) to (make-array buffer-size :element-type 'byte) and I got additional warnings and it took 70 seconds instead of 20.

Thanks, Garrett.

On Fri, Oct 21, 2022 at 1:47 PM Robert Goldman rpgoldman@sift.net wrote:

I don't know what data you are reading but is there any chance that the difference is that when you read text in lisp as ISO-8859-1 lisp is actually processing the text as unicode, but when you are reading it in Java you are just slamming raw bytes into memory?

Maybe this is relevant? https://stackoverflow.com/questions/979932/read-unicode-text-files-with-java

I don't use Java myself, so I can't say, and I don't have access to your data, but it does seem like the Java code is doing something simpler than the Lisp code.

What happens if you change your Lisp code to read-sequence of type byte instead of character?

On 21 Oct 2022, at 13:43, Garrett Dangerfield wrote:

I don't want to cause a firestore here but I was doing some simple benchmarks on file i/o between Java, ABCL, and SBCL and I'm a bit shocked, honestly.

Reading a 2.5M file in 16M chunks in (using iso-8859-1):

• abcl takes a tad over 1 second
• sbcl takes 0.04 seconds

Reading a 5.8G file in 16M chunks in (using iso-8859-1 for Lisp, for Java it's just bytes):

• abcl takes...too long, I gave up
• sbcl takes between 20 and 21 seconds
• Java takes 1.5 seconds

These are all run on the same computer using the same files, etc.

What's up with this? Thoughts? I'd heard that SBCL should be as fast as C under at least some circumstances. I'd wager that C is at least as fast as Java (probably faster).

Thanks, Garrett Dangerfield. (he/him/his)

P.S. Don't get me wrong, I *LOVE* Lisp, I'm trying to get away from Java as fast as I can (the syntax is killing me slowly). I've used ABCL in projects before (it was wonderful, Java doesn't handle XML well).

Lisp code: (with-open-file (stream "/media/danger/OS/temp/jars.txt" :external-format :iso-8859-1) ; great_expectations.iso (let ((size (file-length stream)) (buffer-size (* 16 1024 1024)) ; 16M ) (time (loop with buffer = (make-array buffer-size :element-type 'character) for n-characters = (read-sequence buffer stream) while (< 0 n-characters))) )))

Java code: private static final int BUFFER_SIZE = 16 * 1024 * 1024; try (InputStream in = new FileInputStream("/media/danger/OS/temp/great_expectations.iso"); ) { byte[] buff = new byte[BUFFER_SIZE]; int chunkLen = -1; long start = System.currentTimeMillis(); while ((chunkLen = in.read(buff)) != -1) { System.out.println("chunkLen = " + chunkLen); } double duration = System.currentTimeMillis() - start; duration /= 1000; System.out.println(String.format("it took %,2f secs", duration)); } catch (Exception e) { e.printStackTrace(System.out); } finally { System.out.println("Done."); }

Robert P. Goldman Research Fellow Smart Information Flow Technologies (d/b/a SIFT, LLC)

319 N. First Ave., Suite 400 Minneapolis, MN 55401

Voice: (612) 326-3934 Email: rpgoldman@SIFT.net

Le 21/10/2022 à 23:18, Garrett Dangerfield a écrit :

I tried changing (make-array buffer-size :element-type 'character) to (make-array buffer-size :element-type 'byte) and I got additional warnings and it took 70 seconds instead of 20.

You need to specify a binary file too!

(deftype octet () '(unsigned-byte 8)) (with-open-file (stream #P"~/Downloads/Discord.dmg" :element-type 'octet :external-format :default) (print `(size = ,(file-length stream))) (let ((buffer-size (* 16 1024 1024))) (time (loop with buffer = (make-array buffer-size :element-type 'octet) for n-bytes = (read-sequence buffer stream) while (plusp n-bytes)))))

If you need super fast I/O in ABCL the way to go is to do the direct java calls, same as your Java program. That's what I've done and one gets the expected performance. The JSS syntax makes it fairly painless. I'll have to check but I'm fairly sure no serious optimization has been done for read-sequence. Alan

On Fri, Oct 21, 2022 at 7:54 PM Garrett Dangerfield garrett@dangerimp.com wrote:

Good call. Got the time down with SBCL to 3.5 seconds. So still more than twice Java. ABCL with the same code with the big file (5.8G) is around 110+ seconds.

I'm still surprised by ABCL being SO much slower than SBCL especially with Java being faster than SBCL.

Thoughts?

Thanks, Garrett.

P.S. I also took the "time" out, I am doing SBCL/ABCL the same way as I did in Java with the getting the start time before reading and getting the time after reading. and calculating the duration that way. It didn't make any difference that I could see in the times.

Lisp code: (with-open-file (stream "/media/danger/OS/temp/great_expectations.iso" :element-type '(unsigned-byte 8) :external-format 'iso-8859-1) ; jars.txt iso-8859-1 also tried :default and the time was the same (let ((size (file-length stream)) (buffer-size (* 16 1024 1024)) ; 16M (start (get-internal-real-time)) ) (loop with buffer = (make-array buffer-size :element-type '(unsigned-byte 8)) for n-characters = (read-sequence buffer stream) while (< 0 n-characters)) (format t "took ~,2f secs" (/ (- (get-internal-real-time) start) internal-time-units-per-second)) ))

On Fri, Oct 21, 2022 at 2:52 PM Pascal Bourguignon pjb@informatimago.com wrote:

...

Le 21/10/2022 à 23:18, Garrett Dangerfield a écrit :

...

I tried changing (make-array buffer-size :element-type 'character) to (make-array buffer-size :element-type 'byte) and I got additional warnings and it took 70 seconds instead of 20.

You need to specify a binary file too!

(deftype octet () '(unsigned-byte 8)) (with-open-file (stream #P"~/Downloads/Discord.dmg" :element-type 'octet :external-format :default) (print `(size = ,(file-length stream))) (let ((buffer-size (* 16 1024 1024))) (time (loop with buffer = (make-array buffer-size :element-type 'octet) for n-bytes = (read-sequence buffer stream) while (plusp n-bytes)))))

-- __Pascal Bourguignon__

On Nov 1, 2022, at 02:33, Alan Ruttenberg alanruttenberg@gmail.com wrote:

Here's an example of fast reading in ABCL. The implementation of certain array types use java.nio buffers, which can be directly read into via the java.nio functions.

(defun test-read (path) (let* ((f (new 'RandomAccessFile (namestring (truename path)) "r")) (channel (#"getChannel" f)) (array (make-array (* 16 1024 1024) :element-type '(unsigned-byte 8))) (buffer (get-java-field array "elements" t))) (time (loop for count = (#"read" channel buffer) until (eql count -1) sum count do (#"position" buffer 0) ))))

On my machine, for a 5G file, the SBCL code in an earlier post takes 2.4 seconds. This code takes 1.4 sec. It's fastest if I use 2M buffers - 1.1 seconds. SBCL is also marginally faster with smaller buffer sizes.

In the ABCL source code the files "SimpleArray_*.java" are the implementations of the nio buffer backed array types. See make_array.java where the specific type of underlying array is chosen. There is a global switch for array allocation choosing either direct allocation or nio-buffers, with the default being nio buffers.

(get-java-field 'java$buffers "active" t) -> #<org.armedbear.lisp.Java$Buffers$AllocationPolicy NIO {432E958E}>

Hmmm. That isn’t quite the official interface that I implemented for abcl-1.7. and am rather unsure if setting that really works at the moment. I would need to look closer at the Java implementation to be sure.

The “official” way is to use the additional keywords to CL:MAKE-ARRAY

From the fine manual § 4.10 "Extension to CL:MAKE-ARRARY"

With the NIO feature is present and indicated by the presence of :nio in CL:*FEATURES*, the implementation adds two keyword arguments to CL:MAKE-ARRAY :nio-buffer and :nio-direct.

With the :nio-buffer keyword, the user is able to pass instances of of java.nio.ByteBuffer and its subclasses for the storage of vectors and arrays specialized on the byte-vector types satisfying

(or (unsigned-byte 8) (unsigned-byte 16) (unsigned-byte 32))

As an example, the following would use the :nio-buffer as follows to create a 16 byte vector using the created byte-buffer for storage:

(let* ((length 16) (byte-buffer (java:jstatic "allocate" "java.nio.ByteBuffer" length))) (make-array length :element-type '(unsigned-byte 8) :nio-buffer byte-buffer))

:nio-buffer NIO-BUFFER

Initializes the contents of the new vector or array with the contents of NIO-BUFFER which needs to be a reference to a JAVA-OBJECT of class java.nio.ByteBuffer.

:nio-direct NIO-DIRECT-P

NIO-DIRECT-P is not NIL, constructs a java.nio.Buffer as a ``direct'' buffer. The buffers returned by this method typically have somewhat higher allocation and deallocation costs than non-direct buffers. The contents of direct buffers may reside outside of the normal garbage-collected heap, and so their impact upon the memory footprint of an application might not be obvious. It is therefore recommended that direct buffers be allocated primarily for large, long-lived buffers that are subject to the underlying system's native I/O operations. In general it is best to allocate direct buffers only when they yield a measurable gain in program performance.

I haven't looked into the :element-type 'character case.

Maybe someone who is familiar with the ABCL stream implementation is interested in writing a fast path for read-sequence that uses the nio calls? If so, shout. Otherwise I'll keep it on my procrastinate-by-hacking-abcl list.

I have some experiments with replacing i/o with asynchronous thread pools, for which using specials isn’t going to work, hence it is preferrable to use the CL:MAKE-ARRAY implementation.

After I finish stablizing with abcl-1.9.1 (any day now…). we can start look at optimizing for various read/write patterns. Other than “make it as fast as SBCL” it would be helpful if potential users could give me some usage patterns to optimize for.

Oh, and with suitable elbow grease the ultimate plan is to add a flag to mmap(2) files as well, which should speed up fasl loads immensely.