Hi ABCL-developers,
Is there a recent change (since May?) that would have made compiling files containing large wrapped Java-classes excessively slow? If so, is there any hope of this eventually improving?
Ole
Hi Ole,
On Tue, Aug 23, 2011 at 2:54 PM, Ole Myren Rohne ole.rohne@free.fr wrote:
Hi ABCL-developers,
Is there a recent change (since May?) that would have made compiling files containing large wrapped Java-classes excessively slow? If so, is there any hope of this eventually improving?
Do you have an example of what you're referring to? I'm affraid I'm not fully understanding what you're trying to do. If there's an obvious performance problem, I sure hope we can do something about it. We do not normally accept performance regressions outside our own build times.
That said, I've only seen build and runtimes improve over the last years (I mainly test with the compilation of Maxima, though).
Bye,
Erik.
Hi Erik,
Thank you for your reply! Sorry for confusing subject-line and the brief initial message; here are more details:
I'm using ABCL as a JSR-223 scripting language for the Electric VLSI system. To access the Electric's Java-classes I have a hacked-up version of JFLI, conveniently allowing me to say eg
(def-java-class "com.sun.electric.database.network.Netlist")
to get wrappers for the Java-members of that class. The JFLI-magic obviously happens by introspection (JCLASS-METHODS), basically generating a ton of (lazy) member-wrapping functions.
As I watch the embedded ABCL compile a file #p"tool.lisp" of such DEF-JAVA-CLASS statements, it immediately spits out hundreds of temporary cls-files and a abcl-file,
-rw-r--r-- 1 mrohne staff 1790 Aug 24 08:04 ASDF_TMP_tool_9.cls -rw-r--r-- 1 mrohne staff 1800 Aug 24 08:04 ASDF_TMP_tool_8.cls -rw-r--r-- 1 mrohne staff 1787 Aug 24 08:04 ASDF_TMP_tool_7.cls (...) -rw-r--r-- 1 mrohne staff 1254 Aug 24 08:04 ASDF_TMP_tool_244.cls -rw-r--r-- 1 mrohne staff 1359 Aug 24 08:04 ASDF_TMP_tool_243.cls -rw-r--r-- 1 mrohne staff 1248 Aug 24 08:04 ASDF_TMP_tool_242.cls -rw-r--r-- 1 mrohne staff 1362 Aug 24 08:04 ASDF_TMP_tool_241.cls -rw-r--r-- 1 mrohne staff 108500 Aug 24 08:04 ASDF-TMP-tool.abcl
then goes on chewing on those files for quite a while. Eventually it removes the cls-files and (I assume) renames ASDF-TMP-tool.abcl to tool.abcl. The chewing-time grows non-linearly with the number of cls-files, which in turn seems to depend on the "size" of the Java-class being wrapped, hence my confusing subject-line. A few hundred cls-files finishes in minutes, while more than a thousand easily takes the full morning. A few months back ABCL behaved quite differently, compiling many tens of JFLI-imported classes in a few minutes, so I'm wondering what might have changed?
Ole
On 2011-08-23, at 16:39 , Erik Huelsmann wrote:
Hi Ole,
On Tue, Aug 23, 2011 at 2:54 PM, Ole Myren Rohne ole.rohne@free.fr wrote: Hi ABCL-developers,
Is there a recent change (since May?) that would have made compiling files containing large wrapped Java-classes excessively slow? If so, is there any hope of this eventually improving?
Do you have an example of what you're referring to? I'm affraid I'm not fully understanding what you're trying to do. If there's an obvious performance problem, I sure hope we can do something about it. We do not normally accept performance regressions outside our own build times.
That said, I've only seen build and runtimes improve over the last years (I mainly test with the compilation of Maxima, though).
Bye,
Erik.
Hi Ole,
Thank you for the additional information. Can you tell me which version you were using before and which exact version you're using now? Did something change in your environment?
I'm using ABCL as a JSR-223 scripting language for the Electric VLSI system.
To access the Electric's Java-classes I have a hacked-up version of JFLI, conveniently allowing me to say eg
(def-java-class "com.sun.electric.database.network.Netlist")
to get wrappers for the Java-members of that class. The JFLI-magic obviously happens by introspection (JCLASS-METHODS), basically generating a ton of (lazy) member-wrapping functions.
As I watch the embedded ABCL compile a file #p"tool.lisp" of such DEF-JAVA-CLASS statements, it immediately spits out hundreds of temporary cls-files and a abcl-file,
-rw-r--r-- 1 mrohne staff 1790 Aug 24 08:04 ASDF_TMP_tool_9.cls
-rw-r--r-- 1 mrohne staff 1800 Aug 24 08:04 ASDF_TMP_tool_8.cls
-rw-r--r-- 1 mrohne staff 1787 Aug 24 08:04 ASDF_TMP_tool_7.cls
(...)
-rw-r--r-- 1 mrohne staff 1254 Aug 24 08:04 ASDF_TMP_tool_244.cls
-rw-r--r-- 1 mrohne staff 1359 Aug 24 08:04 ASDF_TMP_tool_243.cls
-rw-r--r-- 1 mrohne staff 1248 Aug 24 08:04 ASDF_TMP_tool_242.cls
-rw-r--r-- 1 mrohne staff 1362 Aug 24 08:04 ASDF_TMP_tool_241.cls
-rw-r--r-- 1 mrohne staff 108500 Aug 24 08:04 ASDF-TMP-tool.abcl
then goes on chewing on those files for quite a while. Eventually it removes the cls-files and (I assume) renames ASDF-TMP-tool.abcl to tool.abcl. The chewing-time grows non-linearly with the number of cls-files, which in turn seems to depend on the "size" of the Java-class being wrapped, hence my confusing subject-line.
My personal experience with classes being compiled -and especially with slowness of it- is that it can be terribly slow over network connections, especially if the network is a busy one.
Last week I added a change which aggrevated the situation, but that aggrevation has again been aleviated a bit this week. There was no change which would cause the numbers to be as bad as the ones you mention.
A few hundred cls-files finishes in minutes, while more than a thousand easily takes the full morning. A few months back ABCL behaved quite differently, compiling many tens of JFLI-imported classes in a few minutes, so I'm wondering what might have changed?
The exponential component in the growth suggests either a very busy network or huge numbers of files in a single directory - with a filesystem not configured to handle it. Eg. the linux filesystems ext2 and ext3 explicitly need "directory indexing" enabled to perform well in situations with "many thousands" of files in a single directory.
246 files should not affect performance, however and would suggest a problem in the network, as I indicated before.
Still: there are solutions which I want to look at which would reduce our dependency on performance of network I/O specifically and maybe even I/O systems in general.
Bye,
Erik.
armedbear-devel@common-lisp.net
-
Erik Huelsmann -
Ole Myren Rohne