Thanks for the reply.

> unfortunately, the links in your email are all dead except for the first one, therefore I can only offer some general comments.

Oh, that’s what I get for sending links with a moving branch name in their URL, sorry. Replacing python-library with 20d63dad should work for all links, so:

[1] https://github.com/GrammaTech/sel/blob/20d63dad/Makefile#L110-L128

[2] https://github.com/GrammaTech/sel/blob/20d63dad/software/tree-sitter.c

[3] https://github.com/GrammaTech/sel/blob/20d63dad/software/tree-sitter.c#L14-L20

[4] https://github.com/GrammaTech/sel/blob/20d63dad/software/tree-sitter.c#L104-L123

[5] https://github.com/GrammaTech/sel/blob/20d63dad/software/tree-sitter.c#L27

[6] https://github.com/GrammaTech/sel/blob/20d63dad/software/tree-sitter.c#L6

The last version of that file is now https://github.com/GrammaTech/sel/blob/a7989247/software/tree-sitter.c.

Thanks, ecl_base_string_pointer_safe is definitely what I want to be using.
wchar_t
Good to know, thanks.
When I get back to this I’ll take a look at removing ASDF. In order to make this usable I would have to do something with the startup time (which is ~30 seconds to load our tree-sitter library, maybe because of all of the automatically defined classes?). I’ll also need to profile some of our functions which are considerably slower on ECL than our standard SBCL implementation, maybe there’s something simple we could do that could have a big impact there.

Much appreciated,

Eric

From: Marius Gerbershagen <marius.gerbershagen@gmail.com>
Date: Sunday, March 14, 2021 at 7:13 AM
To: Eric Schulte <eschulte@grammatech.com>, ecl-devel@common-lisp.net <ecl-devel@common-lisp.net>
Subject: [External] - Re: Calling into an ECL-compiled .so from C, best practices, how to pass strings, etc...

CAUTION: External Email

Hi,

unfortunately, the links in your email are all dead except for the first
one, therefore I can only offer some general comments.

1. Use ecl_base_string_pointer_safe to get a char* pointer to the
contents of a base-string. This will signal an error if the argument is
not a base-string or is not null-terminated. For converting a string
containing unicode characters to an utf-8 encoded base-string, the
sequence-streams abstraction
(https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fcommon-lisp.net%2Fproject%2Fecl%2Fstatic%2Fmanual%2FStreams.html%23Sequence-Streams&data=04%7C01%7C%7Ca1a7251596b044594d5b08d8e6da1b4a%7C22cbf1b8306c42309e2a81f94e129fa8%7C1%7C1%7C637513171799221198%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=yjkF0CVlBM1GKou7QRMHpOUMShuMqMWhLN7dpuwopfQ%3D&reserved=0)
can be used, e.g.

cl_object utf8_encode_base_string(cl_object string)
cl_object output = ecl_alloc_adjustable_base_string(ecl_length(string));
cl_object stream = si_make_sequence_output_stream(3, output,
ecl_make_keyword("EXTERNAL-FORMAT"), ecl_make_keyword("UTF-8"));
cl_write_sequence(2, string, stream);
ecl_write_char(0, stream); /* write null terminator */
return output;
}

2. Are we talking about a utf-16/32 encoded string with 16/32 bit wide
wchar_t or a utf-8 encoded string with ordinary 8-bit char's? In the
latter case you may also use sequence streams to convert your utf-8
encoded char* into a Lisp string (see for example
https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgitlab.com%2Fspaghettisalat%2Fecl%2F-%2Fblob%2Fpathname-encoding%2Fsrc%2Fc%2Fstring.d%23L899&data=04%7C01%7C%7Ca1a7251596b044594d5b08d8e6da1b4a%7C22cbf1b8306c42309e2a81f94e129fa8%7C1%7C1%7C637513171799221198%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=lmFM30MDGgplnrNFz1lM%2FoZlhPzUb3Mb0EyD9rQd6Nw%3D&reserved=0
for a possible implementation. This is from a feature that I'm currently
working on so that function will likely land in the next ECL release).

3. Functions from the common-lisp package should be called directly with
their exported names. For other functions, the best way is to use
ecl_make_symbol and cl_funcall, e.g.

cl_funcall(2, /* number of arguments to the funcall function */
           ecl_make_symbol("SOME-FUNCTION","SOME-PACKAGE"), /* function
to call */
           some_argument);

4. You can pass `--with-asdf=builtin` to the configure script to get an
ECL executable with ASDF built in. However that won't help with startup
time. ECL can't build an image like other lisps can, so we need to load
all fasls (or shared libraries which are really the same thing for ECL)
at runtime. I recommend not using ASDF in the final executable/shared
library if avoidable. ASDF is really a built system mostly meant for
development.

Best regards,
Marius Gerbershagen

PS: the API ECL has for converting strings between lisp and C is
definitely lacking. Feel free to open merge requests on gitlab if you
feel like improving this.

Am 09.03.21 um 01:01 schrieb Eric Schulte:
> Hi,
>
>
>
> I’m trying to compile a common lisp system into a .so which I can then
> wrap a Python library around to enable easy sharing of the functionality
> as a Python package (wheel). I’m having trouble finding examples of
> other C code calling into ECL-compiled shared objects of lisp code. I
> hope I can solicit some answer and critical code review here, or
> pointers to exemplar implementations would also be much appreciated.
>
>
>
> So far I have:
>
> 1. A Makefile [1] rule to build the .so from the common lisp package.
>     This part actually seems to be working well (if slow to compile).
> 2. A C source file [2] wrapping the ECL-compiled shared object. I think
>     this is likely where most of my problems are: specifically:
>      1. How does one get a string back out from Common Lisp to C?
>         Currently I’m jumping through hoops that seem like they should
>         be unnecessary [3] and only work some of the time (other times
>         I’m getting multiple copies of the string text).
>      2. How does one get a UTF8 string from C (wchar_t*) into Common
>         Lisp? I haven’t found a way so for now I’m limiting my input to
>         ASCII (char*) [4].
>      3. Is the best way to invoke a function really to call
>         c_string_to_object (e.g., [5] among many others)? It seems like
>         there should be a more efficient way than converting a string
>         into a function name for every function dispatch.
>      4. Finally, calling init [6] has two problems. First, it takes a
>         very long time every time I load the library. Is there no way
>         to save the state of the lisp image after initialization?
>         Second, it loads ASDF from a path relative to my local system.
>         Can I get ASDF compiled into the shared object to remove as many
>         external dependencies as possible?
>
>
>
> Any help is much appreciated. Also, thanks to those who answered issues
> on the ECL GitLab repository to help me get this far.
>
>
>
> Thanks!
>
> Eric
>
>
>
>
>
> [1]
> https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FGrammaTech%2Fsel%2Fblob%2Fpython-library%2FMakefile%23L110-L128&data=04%7C01%7C%7Ca1a7251596b044594d5b08d8e6da1b4a%7C22cbf1b8306c42309e2a81f94e129fa8%7C1%7C1%7C637513171799221198%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=kMKqrucdKrXacsglosfAhrixj%2FTL42a%2FkKrLRmjCEng%3D&reserved=0
> <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FGrammaTech%2Fsel%2Fblob%2Fpython-library%2FMakefile%23L110-L128&data=04%7C01%7C%7Ca1a7251596b044594d5b08d8e6da1b4a%7C22cbf1b8306c42309e2a81f94e129fa8%7C1%7C1%7C637513171799221198%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=kMKqrucdKrXacsglosfAhrixj%2FTL42a%2FkKrLRmjCEng%3D&reserved=0>
>
> [2]
> https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FGrammaTech%2Fsel%2Fblob%2Fpython-library%2Fsoftware%2Ftree-sitter.c&data=04%7C01%7C%7Ca1a7251596b044594d5b08d8e6da1b4a%7C22cbf1b8306c42309e2a81f94e129fa8%7C1%7C1%7C637513171799221198%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=rvOzv%2BR7nfRAM2NVocrQ2%2FUv0munWpI3FefD%2BUN2bxY%3D&reserved=0
> <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FGrammaTech%2Fsel%2Fblob%2Fpython-library%2Fsoftware%2Ftree-sitter.c&data=04%7C01%7C%7Ca1a7251596b044594d5b08d8e6da1b4a%7C22cbf1b8306c42309e2a81f94e129fa8%7C1%7C1%7C637513171799231191%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=pOXemyrftgvRqvH9HhHgrpWbw20FGzibnBz2qQpMaUM%3D&reserved=0>
>
> [3]
> https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FGrammaTech%2Fsel%2Fblob%2Fpython-library%2Fsoftware%2Ftree-sitter.c%23L14-L20&data=04%7C01%7C%7Ca1a7251596b044594d5b08d8e6da1b4a%7C22cbf1b8306c42309e2a81f94e129fa8%7C1%7C1%7C637513171799231191%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=6MsQyo2KraI8HrZhM4RgLzlBtgZ%2BRcMbVpuNerZLRHw%3D&reserved=0
> <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FGrammaTech%2Fsel%2Fblob%2Fpython-library%2Fsoftware%2Ftree-sitter.c%23L14-L20&data=04%7C01%7C%7Ca1a7251596b044594d5b08d8e6da1b4a%7C22cbf1b8306c42309e2a81f94e129fa8%7C1%7C1%7C637513171799231191%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=6MsQyo2KraI8HrZhM4RgLzlBtgZ%2BRcMbVpuNerZLRHw%3D&reserved=0>
>
> [4]
> https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FGrammaTech%2Fsel%2Fblob%2Fpython-library%2Fsoftware%2Ftree-sitter.c%23L104-L123&data=04%7C01%7C%7Ca1a7251596b044594d5b08d8e6da1b4a%7C22cbf1b8306c42309e2a81f94e129fa8%7C1%7C1%7C637513171799231191%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=Uq374fqBOXTbdrCvWbM%2FOOjLBxl047XwqEgk7R1FNPI%3D&reserved=0
> <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FGrammaTech%2Fsel%2Fblob%2Fpython-library%2Fsoftware%2Ftree-sitter.c%23L104-L123&data=04%7C01%7C%7Ca1a7251596b044594d5b08d8e6da1b4a%7C22cbf1b8306c42309e2a81f94e129fa8%7C1%7C1%7C637513171799231191%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=Uq374fqBOXTbdrCvWbM%2FOOjLBxl047XwqEgk7R1FNPI%3D&reserved=0>
>
> [5]
> https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FGrammaTech%2Fsel%2Fblob%2Fpython-library%2Fsoftware%2Ftree-sitter.c%23L27&data=04%7C01%7C%7Ca1a7251596b044594d5b08d8e6da1b4a%7C22cbf1b8306c42309e2a81f94e129fa8%7C1%7C1%7C637513171799231191%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=eqOf1k1QCShyKct9rNWMIOqo8WzYJm2OEzcQGujDeKk%3D&reserved=0
> <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FGrammaTech%2Fsel%2Fblob%2Fpython-library%2Fsoftware%2Ftree-sitter.c%23L27&data=04%7C01%7C%7Ca1a7251596b044594d5b08d8e6da1b4a%7C22cbf1b8306c42309e2a81f94e129fa8%7C1%7C1%7C637513171799231191%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=eqOf1k1QCShyKct9rNWMIOqo8WzYJm2OEzcQGujDeKk%3D&reserved=0>
>
> [6]
> https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FGrammaTech%2Fsel%2Fblob%2Fpython-library%2Fsoftware%2Ftree-sitter.c%23L6&data=04%7C01%7C%7Ca1a7251596b044594d5b08d8e6da1b4a%7C22cbf1b8306c42309e2a81f94e129fa8%7C1%7C1%7C637513171799231191%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=WL92c%2FKQEwVqUkT4CeR44Yiw9HW5Ng5TFTpZ6RLor8Q%3D&reserved=0
> <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FGrammaTech%2Fsel%2Fblob%2Fpython-library%2Fsoftware%2Ftree-sitter.c%23L6&data=04%7C01%7C%7Ca1a7251596b044594d5b08d8e6da1b4a%7C22cbf1b8306c42309e2a81f94e129fa8%7C1%7C1%7C637513171799231191%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=WL92c%2FKQEwVqUkT4CeR44Yiw9HW5Ng5TFTpZ6RLor8Q%3D&reserved=0>
>
>
>
> ------------------------------------------------------------------------
> The information contained in this e-mail and any attachments from
> GrammaTech, Inc may contain confidential and/or proprietary information,
> and is intended only for the named recipient to whom it was originally
> addressed. If you are not the intended recipient, any disclosure,
> distribution, or copying of this e-mail or its attachments is strictly
> prohibited. If you have received this e-mail in error, please notify the
> sender immediately by return e-mail and permanently delete the e-mail
> and any attachments.