How to manage shared library lifetime through C functions

Greetings,
I hope this is the right group to ask this question; apologies if this
should go the general or some other list.

I have multiple shared libraries that can be called from C that I'd like to
use from a C based postgresql function.

These libraries perform some expensive initialization and they require the
C code to properly release resources when the library is no longer needed.

This means that I need a mechanism to keep a session level pointer to a
library, initialize it when it is called first from a C based function and
dispose the library properly when the session ends (and terminated due to a
problem) I would like to keep the libraries available as long as the
session is alive, so multiple calls are supposed to avoid
initialization/disposal costs every time.

I could probably use a temp table as a container for the initalization and
even pointer values (sounds dirty) but I have no idea how to hook to
session end to clean up when session ends.

What would be a good strategy here?

Regards
Seref

Seref Arikan wrote:
> I hope this is the right group to ask this question; apologies if this should go the general or some
> other list.
>
>
> I have multiple shared libraries that can be called from C that I'd like to use from a C based
> postgresql function.
>
> These libraries perform some expensive initialization and they require the C code to properly release
> resources when the library is no longer needed.
>
>
> This means that I need a mechanism to keep a session level pointer to a library, initialize it when it
> is called first from a C based function and dispose the library properly when the session ends (and
> terminated due to a problem) I would like to keep the libraries available as long as the session is
> alive, so multiple calls are supposed to avoid initialization/disposal costs every time.
>
>
> I could probably use a temp table as a container for the initalization and even pointer values (sounds
> dirty) but I have no idea how to hook to session end to clean up when session ends.
>
>
> What would be a good strategy here?

You could register a callback at process exit with on_proc_exit() from storage/ipc.h.

Yours,
Laurenz Albe

On 08/04/2014 12:54 PM, Seref Arikan wrote:
> Greetings,
> I hope this is the right group to ask this question; apologies if this
> should go the general or some other list.
>
> I have multiple shared libraries that can be called from C that I'd like to
> use from a C based postgresql function.
>
> These libraries perform some expensive initialization and they require the
> C code to properly release resources when the library is no longer needed.
>
> This means that I need a mechanism to keep a session level pointer to a
> library, initialize it when it is called first from a C based function and
> dispose the library properly when the session ends (and terminated due to a
> problem) I would like to keep the libraries available as long as the
> session is alive, so multiple calls are supposed to avoid
> initialization/disposal costs every time.
>
> I could probably use a temp table as a container for the initalization and
> even pointer values (sounds dirty) but I have no idea how to hook to
> session end to clean up when session ends.
>
> What would be a good strategy here?

Define a function called _PG_init() in your C extension. PostgreSQL will
call it once, when the library is loaded into the backend. (The time
it's loaded will depend on whether the library is listed in
shared_preload_libraries, local_preload_libraries, or neither.)

Are you sure you need to do any cleanup? When the session ends, the
backend process will terminate, which will close any open files and
release memory that the library might be holding. If that's not enough,
and the library really needs to do more complicated clean up, then you
can register a callback with on_proc_exit().

Look at the C extensions in the PostgreSQL source tree's contrib
directory for examples of _PG_init() and on_proc_exit().

- Heikki

Thanks a lot Heikki and Albe. Exactly what I was asking for.
Heikki: the libraries are written in languages that have their own runtime
and their documentation insists that both init and dispose calls are
performed when used from C. PG_init() and proc_exit sounds spot on.

Any ideas about keeping some data at session level between calls? Both
calls of the same function and different C functions. (though temp table is
always there as an option)

Best regards
Seref

On Mon, Aug 4, 2014 at 11:17 AM, Heikki Linnakangas <hlinnakangas(at)vmware(dot)com
> wrote:

> On 08/04/2014 12:54 PM, Seref Arikan wrote:
>
>> Greetings,
>> I hope this is the right group to ask this question; apologies if this
>> should go the general or some other list.
>>
>> I have multiple shared libraries that can be called from C that I'd like
>> to
>> use from a C based postgresql function.
>>
>> These libraries perform some expensive initialization and they require the
>> C code to properly release resources when the library is no longer needed.
>>
>> This means that I need a mechanism to keep a session level pointer to a
>> library, initialize it when it is called first from a C based function and
>> dispose the library properly when the session ends (and terminated due to
>> a
>> problem) I would like to keep the libraries available as long as the
>> session is alive, so multiple calls are supposed to avoid
>> initialization/disposal costs every time.
>>
>> I could probably use a temp table as a container for the initalization and
>> even pointer values (sounds dirty) but I have no idea how to hook to
>> session end to clean up when session ends.
>>
>> What would be a good strategy here?
>>
>
> Define a function called _PG_init() in your C extension. PostgreSQL will
> call it once, when the library is loaded into the backend. (The time it's
> loaded will depend on whether the library is listed in
> shared_preload_libraries, local_preload_libraries, or neither.)
>
> Are you sure you need to do any cleanup? When the session ends, the
> backend process will terminate, which will close any open files and release
> memory that the library might be holding. If that's not enough, and the
> library really needs to do more complicated clean up, then you can register
> a callback with on_proc_exit().
>
> Look at the C extensions in the PostgreSQL source tree's contrib directory
> for examples of _PG_init() and on_proc_exit().
>
> - Heikki
>
>

On 08/04/2014 01:31 PM, Seref Arikan wrote:
> Thanks a lot Heikki and Albe. Exactly what I was asking for.
> Heikki: the libraries are written in languages that have their own runtime
> and their documentation insists that both init and dispose calls are
> performed when used from C. PG_init() and proc_exit sounds spot on.

You might also consider creating your own PL handler for those
languages. Or google around if one exists already.

> Any ideas about keeping some data at session level between calls? Both
> calls of the same function and different C functions. (though temp table is
> always there as an option)

You can use a global variable in the C extension. If you need to
allocate memory that survives across function calls, use
"MemoryContextSwitchTo(TopMemoryContext); ... = palloc(...)". I'm sure
you'll find examples of that in the existing extensions too.

- Heikki

On 08/04/2014 06:31 PM, Seref Arikan wrote:
> Thanks a lot Heikki and Albe. Exactly what I was asking for.
> Heikki: the libraries are written in languages that have their own
> runtime and their documentation insists that both init and dispose calls
> are performed when used from C. PG_init() and proc_exit sounds spot on.

That's usually with reference to code that might then load them again,
such as another library. Without specifics it's hard to say.

If these tools depend on the C code terminating cleanly to operate
correctly then they'll also break if the OS crashes, the system loses
power, etc.

Exiting a program without calling cleanup is in many ways just a
particularly friendly kind of crash, and it's often actually much more
efficient than doing cleanup. Why neatly tidy the floor when you're
about to demolish the house? Total waste of time.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Hi,

On Mon, Aug 4, 2014 at 11:58 AM, Heikki Linnakangas <hlinnakangas(at)vmware(dot)com
> wrote:

> On 08/04/2014 01:31 PM, Seref Arikan wrote:
>
>> Thanks a lot Heikki and Albe. Exactly what I was asking for.
>> Heikki: the libraries are written in languages that have their own runtime
>> and their documentation insists that both init and dispose calls are
>> performed when used from C. PG_init() and proc_exit sounds spot on.
>>
>
> You might also consider creating your own PL handler for those languages.
> Or google around if one exists already.

That was what I checked first, but unfortunately none exists at the moment.
I really want to write one,which would probably be a much cleaner way of
doing what I'm going to do, but time pressure does not let me.
At the moment the lowest hanging fruit is to pass values through C
functions to these libraries and get the results back; hence, my questions.

>
>
> Any ideas about keeping some data at session level between calls? Both
>> calls of the same function and different C functions. (though temp table
>> is
>> always there as an option)
>>
>
> You can use a global variable in the C extension. If you need to allocate
> memory that survives across function calls, use "MemoryContextSwitchTo(TopMemoryContext);
> ... = palloc(...)". I'm sure you'll find examples of that in the existing
> extensions too.
>

Thanks, I'll check this out as well.

> - Heikki
>
>

Craig Ringer wrote:
> On 08/04/2014 06:31 PM, Seref Arikan wrote:
>> Thanks a lot Heikki and Albe. Exactly what I was asking for.
>> Heikki: the libraries are written in languages that have their own
>> runtime and their documentation insists that both init and dispose calls
>> are performed when used from C. PG_init() and proc_exit sounds spot on.
>
> That's usually with reference to code that might then load them again,
> such as another library. Without specifics it's hard to say.
>
> If these tools depend on the C code terminating cleanly to operate
> correctly then they'll also break if the OS crashes, the system loses
> power, etc.
>
> Exiting a program without calling cleanup is in many ways just a
> particularly friendly kind of crash, and it's often actually much more
> efficient than doing cleanup. Why neatly tidy the floor when you're
> about to demolish the house? Total waste of time.

There are valid use cases (else the function probably wouldn't exist).
I use it in oracle_fdw to gracefully close any open Oracle connections when
the process exits.

Yours,
Laurenz Albe

On 08/04/2014 09:48 PM, Albe Laurenz wrote:
> There are valid use cases (else the function probably wouldn't exist).
> I use it in oracle_fdw to gracefully close any open Oracle connections when
> the process exits.

True; it's sometimes better to do a clean exit.

It's relying on that always happening that's a problem. In your case
Oracle will clean up dead connections after a while, for example (or
quite promptly as the OS will rapidly TCP RST them). So it's just nicer.

Point taken though.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services