Calling into an ECL-compiled .so from C, best practices, how to pass strings, etc...
Hi, I’m trying to compile a common lisp system into a .so which I can then wrap a Python library around to enable easy sharing of the functionality as a Python package (wheel). I’m having trouble finding examples of other C code calling into ECL-compiled shared objects of lisp code. I hope I can solicit some answer and critical code review here, or pointers to exemplar implementations would also be much appreciated. So far I have: 1. A Makefile [1] rule to build the .so from the common lisp package. This part actually seems to be working well (if slow to compile). 2. A C source file [2] wrapping the ECL-compiled shared object. I think this is likely where most of my problems are: specifically: * How does one get a string back out from Common Lisp to C? Currently I’m jumping through hoops that seem like they should be unnecessary [3] and only work some of the time (other times I’m getting multiple copies of the string text). * How does one get a UTF8 string from C (wchar_t*) into Common Lisp? I haven’t found a way so for now I’m limiting my input to ASCII (char*) [4]. * Is the best way to invoke a function really to call c_string_to_object (e.g., [5] among many others)? It seems like there should be a more efficient way than converting a string into a function name for every function dispatch. * Finally, calling init [6] has two problems. First, it takes a very long time every time I load the library. Is there no way to save the state of the lisp image after initialization? Second, it loads ASDF from a path relative to my local system. Can I get ASDF compiled into the shared object to remove as many external dependencies as possible? Any help is much appreciated. Also, thanks to those who answered issues on the ECL GitLab repository to help me get this far. Thanks! Eric [1] https://github.com/GrammaTech/sel/blob/python-library/Makefile#L110-L128 [2] https://github.com/GrammaTech/sel/blob/python-library/software/tree-sitter.c [3] https://github.com/GrammaTech/sel/blob/python-library/software/tree-sitter.c... [4] https://github.com/GrammaTech/sel/blob/python-library/software/tree-sitter.c... [5] https://github.com/GrammaTech/sel/blob/python-library/software/tree-sitter.c... [6] https://github.com/GrammaTech/sel/blob/python-library/software/tree-sitter.c... ________________________________ The information contained in this e-mail and any attachments from GrammaTech, Inc may contain confidential and/or proprietary information, and is intended only for the named recipient to whom it was originally addressed. If you are not the intended recipient, any disclosure, distribution, or copying of this e-mail or its attachments is strictly prohibited. If you have received this e-mail in error, please notify the sender immediately by return e-mail and permanently delete the e-mail and any attachments.
Hi, unfortunately, the links in your email are all dead except for the first one, therefore I can only offer some general comments. 1. Use ecl_base_string_pointer_safe to get a char* pointer to the contents of a base-string. This will signal an error if the argument is not a base-string or is not null-terminated. For converting a string containing unicode characters to an utf-8 encoded base-string, the sequence-streams abstraction (https://common-lisp.net/project/ecl/static/manual/Streams.html#Sequence-Stre...) can be used, e.g. cl_object utf8_encode_base_string(cl_object string) cl_object output = ecl_alloc_adjustable_base_string(ecl_length(string)); cl_object stream = si_make_sequence_output_stream(3, output, ecl_make_keyword("EXTERNAL-FORMAT"), ecl_make_keyword("UTF-8")); cl_write_sequence(2, string, stream); ecl_write_char(0, stream); /* write null terminator */ return output; } 2. Are we talking about a utf-16/32 encoded string with 16/32 bit wide wchar_t or a utf-8 encoded string with ordinary 8-bit char's? In the latter case you may also use sequence streams to convert your utf-8 encoded char* into a Lisp string (see for example https://gitlab.com/spaghettisalat/ecl/-/blob/pathname-encoding/src/c/string.... for a possible implementation. This is from a feature that I'm currently working on so that function will likely land in the next ECL release). 3. Functions from the common-lisp package should be called directly with their exported names. For other functions, the best way is to use ecl_make_symbol and cl_funcall, e.g. cl_funcall(2, /* number of arguments to the funcall function */ ecl_make_symbol("SOME-FUNCTION","SOME-PACKAGE"), /* function to call */ some_argument); 4. You can pass `--with-asdf=builtin` to the configure script to get an ECL executable with ASDF built in. However that won't help with startup time. ECL can't build an image like other lisps can, so we need to load all fasls (or shared libraries which are really the same thing for ECL) at runtime. I recommend not using ASDF in the final executable/shared library if avoidable. ASDF is really a built system mostly meant for development. Best regards, Marius Gerbershagen PS: the API ECL has for converting strings between lisp and C is definitely lacking. Feel free to open merge requests on gitlab if you feel like improving this. Am 09.03.21 um 01:01 schrieb Eric Schulte:
Hi,
I’m trying to compile a common lisp system into a .so which I can then wrap a Python library around to enable easy sharing of the functionality as a Python package (wheel). I’m having trouble finding examples of other C code calling into ECL-compiled shared objects of lisp code. I hope I can solicit some answer and critical code review here, or pointers to exemplar implementations would also be much appreciated.
So far I have:
1. A Makefile [1] rule to build the .so from the common lisp package. This part actually seems to be working well (if slow to compile). 2. A C source file [2] wrapping the ECL-compiled shared object. I think this is likely where most of my problems are: specifically: 1. How does one get a string back out from Common Lisp to C? Currently I’m jumping through hoops that seem like they should be unnecessary [3] and only work some of the time (other times I’m getting multiple copies of the string text). 2. How does one get a UTF8 string from C (wchar_t*) into Common Lisp? I haven’t found a way so for now I’m limiting my input to ASCII (char*) [4]. 3. Is the best way to invoke a function really to call c_string_to_object (e.g., [5] among many others)? It seems like there should be a more efficient way than converting a string into a function name for every function dispatch. 4. Finally, calling init [6] has two problems. First, it takes a very long time every time I load the library. Is there no way to save the state of the lisp image after initialization? Second, it loads ASDF from a path relative to my local system. Can I get ASDF compiled into the shared object to remove as many external dependencies as possible?
Any help is much appreciated. Also, thanks to those who answered issues on the ECL GitLab repository to help me get this far.
Thanks!
Eric
[1] https://github.com/GrammaTech/sel/blob/python-library/Makefile#L110-L128 <https://github.com/GrammaTech/sel/blob/python-library/Makefile#L110-L128>
[2] https://github.com/GrammaTech/sel/blob/python-library/software/tree-sitter.c <https://github.com/GrammaTech/sel/blob/python-library/software/tree-sitter.c>
[3] https://github.com/GrammaTech/sel/blob/python-library/software/tree-sitter.c... <https://github.com/GrammaTech/sel/blob/python-library/software/tree-sitter.c#L14-L20>
[4] https://github.com/GrammaTech/sel/blob/python-library/software/tree-sitter.c... <https://github.com/GrammaTech/sel/blob/python-library/software/tree-sitter.c#L104-L123>
[5] https://github.com/GrammaTech/sel/blob/python-library/software/tree-sitter.c... <https://github.com/GrammaTech/sel/blob/python-library/software/tree-sitter.c#L27>
[6] https://github.com/GrammaTech/sel/blob/python-library/software/tree-sitter.c... <https://github.com/GrammaTech/sel/blob/python-library/software/tree-sitter.c#L6>
------------------------------------------------------------------------ The information contained in this e-mail and any attachments from GrammaTech, Inc may contain confidential and/or proprietary information, and is intended only for the named recipient to whom it was originally addressed. If you are not the intended recipient, any disclosure, distribution, or copying of this e-mail or its attachments is strictly prohibited. If you have received this e-mail in error, please notify the sender immediately by return e-mail and permanently delete the e-mail and any attachments.
Thanks for the reply.
unfortunately, the links in your email are all dead except for the first one, therefore I can only offer some general comments.
Oh, that’s what I get for sending links with a moving branch name in their URL, sorry. Replacing python-library with 20d63dad should work for all links, so: [1] https://github.com/GrammaTech/sel/blob/20d63dad/Makefile#L110-L128 [2] https://github.com/GrammaTech/sel/blob/20d63dad/software/tree-sitter.c [3] https://github.com/GrammaTech/sel/blob/20d63dad/software/tree-sitter.c#L14-L... [4] https://github.com/GrammaTech/sel/blob/20d63dad/software/tree-sitter.c#L104-... [5] https://github.com/GrammaTech/sel/blob/20d63dad/software/tree-sitter.c#L27 [6] https://github.com/GrammaTech/sel/blob/20d63dad/software/tree-sitter.c#L6 The last version of that file is now https://github.com/GrammaTech/sel/blob/a7989247/software/tree-sitter.c. 1. Thanks, ecl_base_string_pointer_safe is definitely what I want to be using. 2. wchar_t 3. Good to know, thanks. 4. When I get back to this I’ll take a look at removing ASDF. In order to make this usable I would have to do something with the startup time (which is ~30 seconds to load our tree-sitter library, maybe because of all of the automatically defined classes?). I’ll also need to profile some of our functions which are considerably slower on ECL than our standard SBCL implementation, maybe there’s something simple we could do that could have a big impact there. Much appreciated, Eric From: Marius Gerbershagen <marius.gerbershagen@gmail.com> Date: Sunday, March 14, 2021 at 7:13 AM To: Eric Schulte <eschulte@grammatech.com>, ecl-devel@common-lisp.net <ecl-devel@common-lisp.net> Subject: [External] - Re: Calling into an ECL-compiled .so from C, best practices, how to pass strings, etc... CAUTION: External Email Hi, unfortunately, the links in your email are all dead except for the first one, therefore I can only offer some general comments. 1. Use ecl_base_string_pointer_safe to get a char* pointer to the contents of a base-string. This will signal an error if the argument is not a base-string or is not null-terminated. For converting a string containing unicode characters to an utf-8 encoded base-string, the sequence-streams abstraction (https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fcommon-li...) can be used, e.g. cl_object utf8_encode_base_string(cl_object string) cl_object output = ecl_alloc_adjustable_base_string(ecl_length(string)); cl_object stream = si_make_sequence_output_stream(3, output, ecl_make_keyword("EXTERNAL-FORMAT"), ecl_make_keyword("UTF-8")); cl_write_sequence(2, string, stream); ecl_write_char(0, stream); /* write null terminator */ return output; } 2. Are we talking about a utf-16/32 encoded string with 16/32 bit wide wchar_t or a utf-8 encoded string with ordinary 8-bit char's? In the latter case you may also use sequence streams to convert your utf-8 encoded char* into a Lisp string (see for example https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgitlab.co... for a possible implementation. This is from a feature that I'm currently working on so that function will likely land in the next ECL release). 3. Functions from the common-lisp package should be called directly with their exported names. For other functions, the best way is to use ecl_make_symbol and cl_funcall, e.g. cl_funcall(2, /* number of arguments to the funcall function */ ecl_make_symbol("SOME-FUNCTION","SOME-PACKAGE"), /* function to call */ some_argument); 4. You can pass `--with-asdf=builtin` to the configure script to get an ECL executable with ASDF built in. However that won't help with startup time. ECL can't build an image like other lisps can, so we need to load all fasls (or shared libraries which are really the same thing for ECL) at runtime. I recommend not using ASDF in the final executable/shared library if avoidable. ASDF is really a built system mostly meant for development. Best regards, Marius Gerbershagen PS: the API ECL has for converting strings between lisp and C is definitely lacking. Feel free to open merge requests on gitlab if you feel like improving this. Am 09.03.21 um 01:01 schrieb Eric Schulte:
Hi,
I’m trying to compile a common lisp system into a .so which I can then wrap a Python library around to enable easy sharing of the functionality as a Python package (wheel). I’m having trouble finding examples of other C code calling into ECL-compiled shared objects of lisp code. I hope I can solicit some answer and critical code review here, or pointers to exemplar implementations would also be much appreciated.
So far I have:
1. A Makefile [1] rule to build the .so from the common lisp package. This part actually seems to be working well (if slow to compile). 2. A C source file [2] wrapping the ECL-compiled shared object. I think this is likely where most of my problems are: specifically: 1. How does one get a string back out from Common Lisp to C? Currently I’m jumping through hoops that seem like they should be unnecessary [3] and only work some of the time (other times I’m getting multiple copies of the string text). 2. How does one get a UTF8 string from C (wchar_t*) into Common Lisp? I haven’t found a way so for now I’m limiting my input to ASCII (char*) [4]. 3. Is the best way to invoke a function really to call c_string_to_object (e.g., [5] among many others)? It seems like there should be a more efficient way than converting a string into a function name for every function dispatch. 4. Finally, calling init [6] has two problems. First, it takes a very long time every time I load the library. Is there no way to save the state of the lisp image after initialization? Second, it loads ASDF from a path relative to my local system. Can I get ASDF compiled into the shared object to remove as many external dependencies as possible?
Any help is much appreciated. Also, thanks to those who answered issues on the ECL GitLab repository to help me get this far.
Thanks!
Eric
[1] https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.co... <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FGrammaTech%2Fsel%2Fblob%2Fpython-library%2FMakefile%23L110-L128&data=04%7C01%7C%7Ca1a7251596b044594d5b08d8e6da1b4a%7C22cbf1b8306c42309e2a81f94e129fa8%7C1%7C1%7C637513171799221198%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=kMKqrucdKrXacsglosfAhrixj%2FTL42a%2FkKrLRmjCEng%3D&reserved=0>
[2] https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.co... <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FGrammaTech%2Fsel%2Fblob%2Fpython-library%2Fsoftware%2Ftree-sitter.c&data=04%7C01%7C%7Ca1a7251596b044594d5b08d8e6da1b4a%7C22cbf1b8306c42309e2a81f94e129fa8%7C1%7C1%7C637513171799231191%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=pOXemyrftgvRqvH9HhHgrpWbw20FGzibnBz2qQpMaUM%3D&reserved=0>
[3] https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.co... <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FGrammaTech%2Fsel%2Fblob%2Fpython-library%2Fsoftware%2Ftree-sitter.c%23L14-L20&data=04%7C01%7C%7Ca1a7251596b044594d5b08d8e6da1b4a%7C22cbf1b8306c42309e2a81f94e129fa8%7C1%7C1%7C637513171799231191%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=6MsQyo2KraI8HrZhM4RgLzlBtgZ%2BRcMbVpuNerZLRHw%3D&reserved=0>
[4] https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.co... <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FGrammaTech%2Fsel%2Fblob%2Fpython-library%2Fsoftware%2Ftree-sitter.c%23L104-L123&data=04%7C01%7C%7Ca1a7251596b044594d5b08d8e6da1b4a%7C22cbf1b8306c42309e2a81f94e129fa8%7C1%7C1%7C637513171799231191%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=Uq374fqBOXTbdrCvWbM%2FOOjLBxl047XwqEgk7R1FNPI%3D&reserved=0>
[5] https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.co... <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FGrammaTech%2Fsel%2Fblob%2Fpython-library%2Fsoftware%2Ftree-sitter.c%23L27&data=04%7C01%7C%7Ca1a7251596b044594d5b08d8e6da1b4a%7C22cbf1b8306c42309e2a81f94e129fa8%7C1%7C1%7C637513171799231191%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=eqOf1k1QCShyKct9rNWMIOqo8WzYJm2OEzcQGujDeKk%3D&reserved=0>
[6] https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.co... <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FGrammaTech%2Fsel%2Fblob%2Fpython-library%2Fsoftware%2Ftree-sitter.c%23L6&data=04%7C01%7C%7Ca1a7251596b044594d5b08d8e6da1b4a%7C22cbf1b8306c42309e2a81f94e129fa8%7C1%7C1%7C637513171799231191%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=WL92c%2FKQEwVqUkT4CeR44Yiw9HW5Ng5TFTpZ6RLor8Q%3D&reserved=0>
------------------------------------------------------------------------ The information contained in this e-mail and any attachments from GrammaTech, Inc may contain confidential and/or proprietary information, and is intended only for the named recipient to whom it was originally addressed. If you are not the intended recipient, any disclosure, distribution, or copying of this e-mail or its attachments is strictly prohibited. If you have received this e-mail in error, please notify the sender immediately by return e-mail and permanently delete the e-mail and any attachments.
The information contained in this e-mail and any attachments from GrammaTech, Inc may contain confidential and/or proprietary information, and is intended only for the named recipient to whom it was originally addressed. If you are not the intended recipient, any disclosure, distribution, or copying of this e-mail or its attachments is strictly prohibited. If you have received this e-mail in error, please notify the sender immediately by return e-mail and permanently delete the e-mail and any attachments.
participants (2)
-
Eric Schulte -
Marius Gerbershagen