Re: Advice regarding configuration parameters

I would like some configuration parameters to Pl/Java and I would like some
advice. Where should they go?

1. Something similar to postgresql.conf (it's not extendable though, is it?)
2. A Table in the database in the "sqlj" schema
3. Java properties file (cumbersome, must be available prior to starting the
JVM)
4. Some other place that is obvious but not currently known to me

Thanks,

Thomas Hallgren

Am Freitag, 6. Februar 2004 10:27 schrieb Thomas Hallgren:
> I would like some configuration parameters to Pl/Java and I would like some
> advice. Where should they go?
>
> 1. Something similar to postgresql.conf (it's not extendable though, is
> it?)

No, it is not.

> 2. A Table in the database in the "sqlj" schema

That could be an OK solution if you don't need to read the values every time.

> 3. Java properties file (cumbersome, must be available prior to starting
> the JVM)

Yes, probably too cumbersome.

> 4. Some other place that is obvious but not currently known to me

I have been thinking for some time about a generic mechanism to configure
procedural languages. It could be a text array in pg_language that you could
fill at will. That way, it would be easily available to the language handler
or the actual function. Of course, this is still only marginally better than
a regular table in your own schema.

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> Am Freitag, 6. Februar 2004 10:27 schrieb Thomas Hallgren:
>> I would like some configuration parameters to Pl/Java and I would like some
>> advice. Where should they go?
>>
>> 1. Something similar to postgresql.conf (it's not extendable though, is
>> it?)

> No, it is not.

In principle it could be --- the mechanisms already exist in guc.c to
permit outside agents to add variables. The difficulty in having a PL
handler add such variables is that there is no good way to get the
handler to run before postgresql.conf is scanned for the first time,
and if it isn't then GUC will error out on the "unknown" variable name.

> I have been thinking for some time about a generic mechanism to
> configure procedural languages. It could be a text array in
> pg_language that you could fill at will.

If we had a mechanism that allowed "unrecognized" variable names in
postgresql.conf to be saved and reprocessed later, we could allow PLs
and other dynamically-loaded libraries to be configured via ordinary GUC
variables, which would be much nicer than a special-purpose mechanism.
Of course this would have a negative impact on the ability to detect
plain old misspellings in the config file. Perhaps we could have a
compromise that says that specially formed variable names, maybe like
"pljava::myparam", are allowed to escape the normal error check.

regards, tom lane

Tom Lane wrote:
> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
>>Am Freitag, 6. Februar 2004 10:27 schrieb Thomas Hallgren:
>>>1. Something similar to postgresql.conf (it's not extendable though, is
>>>it?)
>
>>No, it is not.
>
> In principle it could be --- the mechanisms already exist in guc.c to
> permit outside agents to add variables. The difficulty in having a PL
> handler add such variables is that there is no good way to get the
> handler to run before postgresql.conf is scanned for the first time,
> and if it isn't then GUC will error out on the "unknown" variable name.

I had started down this road about a year ago (i.e. had a partial
patch), for exactly the same reason Thomas is interested -- I wanted it
for PL/R. But the consensus at the time seemed to be that it wouldn't be
accepted so I gave up on it. I still think it is a good idea and that
the difficulties can be worked out.

Joe

Joe Conway <mail(at)joeconway(dot)com> writes:
> I still think it is a good idea and that
> the difficulties can be worked out.

What do you think of the idea of suppressing the "unknown variable"
error for some class of variable names?

If we had agreement on doing that then I think the rest would be pretty
simple. After a few moments' thought I like the following sketch:

* When a GUC setting read from postgresql.conf (or any other GUC setting
source) contains an unknown variable name that fits the special class,
create a "placeholder" GUC variable and insert it into the GUC variable
array. The placeholder will be a STRING-type variable and will have a
flag set to mark it as a placeholder. This flag would prevent it from
being shown by SHOW ALL, but in other respects it would act like any
other variable.

* Subsequent manipulations, such as overriding of the value from ALTER
DATABASE or other sources, will work normally. Therefore at all times
we will have the correct setting of the placeholder available as a
string. (Actually there are several settings: current, reset, etc,
but we'd know them all.)

* When and if the PL (or other shared library) that uses the variable is
loaded, it will execute an "add_guc_variable" call during its one-time
initialization. This new function either adds a GUC variable (if no
match in the present GUC array) or replaces a placeholder variable (if
found). In event that a placeholder is replaced, we convert the string
value(s) as needed and assign to the newly defined variable.

This scheme could not handle add-on GUC variables with some of the odder
flags, such as GUC_LIST_INPUT, since the correct flag values wouldn't be
known when the input is first seen. And we'd have to think a little
about how to handle variable values that are discovered to be erroneous
when we try to assign them to the variable --- probably we can just drop
them, but does that make the semantics weird at all? But I think it
would solve 99% of the problem for only a small amount of work.

regards, tom lane

Tom Lane wrote:
> What do you think of the idea of suppressing the "unknown variable"
> error for some class of variable names?

I like it. I wonder if we ought to have a way to "register" valid
classes? Maybe a new guc variable in the form of a list of valid
classes. So something like:

custom_variable_class = 'plr,pljava'
plr::foo = '/usr/lib/R'
pljava::baz = 1
plruby::var = true <== this one would generate an error

> If we had agreement on doing that then I think the rest would be
> pretty simple. After a few moments' thought I like the following
> sketch:

[good implementation ideas]

> This scheme could not handle add-on GUC variables with some of the
> odder flags, such as GUC_LIST_INPUT, since the correct flag values
> wouldn't be known when the input is first seen.

I think this is OK.

> And we'd have to think a little about how to handle variable values
> that are discovered to be erroneous when we try to assign them to the
> variable --- probably we can just drop them, but does that make the
> semantics weird at all?

Maybe we can require a default value as a parameter to
add_guc_variable() and use that?

Joe

Joe Conway <mail(at)joeconway(dot)com> writes:
> I like it. I wonder if we ought to have a way to "register" valid
> classes? Maybe a new guc variable in the form of a list of valid
> classes. So something like:

There are some order-of-processing issues there, but maybe. Another
possibility is that after a shlib has finished registering its
variables, it could look for remaining placeholders matching its prefix
and issue WARNINGs about 'em (it can't realistically ERROR, probably,
but a WARNING would surely help). These are actually orthogonal checks
since one addresses the class part and the other the individual variable.

>> And we'd have to think a little about how to handle variable values
>> that are discovered to be erroneous when we try to assign them to the
>> variable --- probably we can just drop them, but does that make the
>> semantics weird at all?

> Maybe we can require a default value as a parameter to
> add_guc_variable() and use that?

Well, the new GUC variable struct would include a default by definition,
and presumably that value would "bubble up" to replace any values that
are found illegal.

The sort of semantic funny I am thinking of is like this:
* postgresql.conf contains pljava::var = somegoodvalue
* ALTER DATABASE SET supplies pljava::var = somebadvalue
For builtin variables the ALTER DATABASE value would be rejected on
sight and the end result would be that the variable contains
'somegoodvalue'. However if we don't yet know the variable at backend
startup, 'somebadvalue' will replace 'somegoodvalue' completely, and
then when the PL actually gets loaded it will get thrown away. End
result is that the variable will have whatever its hardwired default is,
and not 'somegoodvalue' as one would wish. Even more surprising, a
subsequent SIGHUP would make it acquire 'somegoodvalue'.

This particular case could be dealt with by forcing a rescan of
postgresql.conf after new variables are defined (I think we need only do
so if any errors are detected in assigning values), but that will not
handle everything. We don't have any way to get back overridden values
from other sources such as the postmaster command line.

It seems likely to me that such corner cases are sufficiently bizarre to
not bother anyone, but we need to think more to make sure that there
aren't any that would bother someone.

regards, tom lane

Peter Eisentraut wrote:
> Am Freitag, 6. Februar 2004 10:27 schrieb Thomas Hallgren:
> > I would like some configuration parameters to Pl/Java and I would like some
> > advice. Where should they go?
> >
> > 1. Something similar to postgresql.conf (it's not extendable though, is
> > it?)
>
> No, it is not.
>
> > 2. A Table in the database in the "sqlj" schema
>
> That could be an OK solution if you don't need to read the values every time.
>
> > 3. Java properties file (cumbersome, must be available prior to starting
> > the JVM)
>
> Yes, probably too cumbersome.
>
> > 4. Some other place that is obvious but not currently known to me
>
> I have been thinking for some time about a generic mechanism to configure
> procedural languages. It could be a text array in pg_language that you could
> fill at will. That way, it would be easily available to the language handler
> or the actual function. Of course, this is still only marginally better than
> a regular table in your own schema.

One big question is whether the per-language variables are per-server or
per-database. The might determine if you want them in the database or
in a configuration file.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> Peter Eisentraut wrote:
>> I have been thinking for some time about a generic mechanism to
>> configure procedural languages. It could be a text array in
>> pg_language that you could fill at will.

> One big question is whether the per-language variables are per-server or
> per-database. The might determine if you want them in the database or
> in a configuration file.

Of course, there's already a solution for that in GUC.

Given all the work Peter put into GUC (for very good reasons), I was a
tad astonished to read him proposing to develop a non-GUC mechanism
for configuring PLs.

regards, tom lane

Tom Lane wrote:
> Given all the work Peter put into GUC (for very good reasons), I was
> a tad astonished to read him proposing to develop a non-GUC mechanism
> for configuring PLs.

I for one was a tad astonished to read that there is already support for
adding variables at run-time, given that we previously rejected that
notion. But the "namespace" idea for variables added by external
modules sounds interesting.

On 02/06/04:05/5, Tom Lane wrote:
> To: Joe Conway <mail(at)joeconway(dot)com>
> Cc: Peter Eisentraut <peter_e(at)gmx(dot)net>,
> Thomas Hallgren <thhal(at)mailblocks(dot)com>,
> pgsql-hackers(at)postgresql(dot)org
> Subject: Re: [HACKERS] Advice regarding configuration parameters
> Date: Fri, 06 Feb 2004 13:15:02 -0500
> Message-ID: <4515(dot)1076091302(at)sss(dot)pgh(dot)pa(dot)us>
> From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
>
> Joe Conway <mail(at)joeconway(dot)com> writes:
> > I like it. I wonder if we ought to have a way to "register" valid
> > classes? Maybe a new guc variable in the form of a list of valid
> > classes. So something like:
>
> There are some order-of-processing issues there, but maybe. Another
> possibility is that after a shlib has finished registering its
> variables, it could look for remaining placeholders matching its prefix
> and issue WARNINGs about 'em (it can't realistically ERROR, probably,
> but a WARNING would surely help). These are actually orthogonal checks
> since one addresses the class part and the other the individual variable.
>
> >> And we'd have to think a little about how to handle variable values
> >> that are discovered to be erroneous when we try to assign them to the
> >> variable --- probably we can just drop them, but does that make the
> >> semantics weird at all?
>
> > Maybe we can require a default value as a parameter to
> > add_guc_variable() and use that?
>
> Well, the new GUC variable struct would include a default by definition,
> and presumably that value would "bubble up" to replace any values that
> are found illegal.
>
> The sort of semantic funny I am thinking of is like this:
> * postgresql.conf contains pljava::var = somegoodvalue
> * ALTER DATABASE SET supplies pljava::var = somebadvalue
> For builtin variables the ALTER DATABASE value would be rejected on
> sight and the end result would be that the variable contains
> 'somegoodvalue'. However if we don't yet know the variable at backend
> startup, 'somebadvalue' will replace 'somegoodvalue' completely, and
> then when the PL actually gets loaded it will get thrown away. End
> result is that the variable will have whatever its hardwired default is,
> and not 'somegoodvalue' as one would wish. Even more surprising, a
> subsequent SIGHUP would make it acquire 'somegoodvalue'.
>
> This particular case could be dealt with by forcing a rescan of
> postgresql.conf after new variables are defined (I think we need only do
> so if any errors are detected in assigning values), but that will not
> handle everything. We don't have any way to get back overridden values
> from other sources such as the postmaster command line.
>
> It seems likely to me that such corner cases are sufficiently bizarre to
> not bother anyone, but we need to think more to make sure that there
> aren't any that would bother someone.

What about having a designated GUC configuration file directory, one file per
class? When a given class is initialized, it tries to find its corresponding
config file in $DATA/conf/$CLASSFILENAME. $CLASSFILENAME then being able
to differ from the actual classname, perhaps. Also, the variable's classname
would be implied from the filename that it's loaded from. So no need to
"class<some specifier>varname".

I guess postgres.conf should be considered the 'main' or 'postgres' class,
and should probably exist in $DATA/conf/(postgres|main), or whatever people
think is the most descriptive. It doesn't have to be moved, but I figure it
would take away one special case(except on setting a var without classname).

I think this would resolve any out of order issues with one file. No?

Regards,
James William Pye

Some very good suggestions where made here. What happens next? Will this end
up in a TODO list where someone can "claim the task" (I'm trying to learn
how the process works) ?

Kind regards,

Thomas Hallgren

----- Original Message -----
From: "Tom Lane" <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: "Joe Conway" <mail(at)joeconway(dot)com>
Cc: "Peter Eisentraut" <peter_e(at)gmx(dot)net>; "Thomas Hallgren"
<thhal(at)mailblocks(dot)com>; <pgsql-hackers(at)postgresql(dot)org>
Sent: Friday, February 06, 2004 19:15
Subject: Re: [HACKERS] Advice regarding configuration parameters

> Joe Conway <mail(at)joeconway(dot)com> writes:
> > I like it. I wonder if we ought to have a way to "register" valid
> > classes? Maybe a new guc variable in the form of a list of valid
> > classes. So something like:
>
> There are some order-of-processing issues there, but maybe. Another
> possibility is that after a shlib has finished registering its
> variables, it could look for remaining placeholders matching its prefix
> and issue WARNINGs about 'em (it can't realistically ERROR, probably,
> but a WARNING would surely help). These are actually orthogonal checks
> since one addresses the class part and the other the individual variable.
>
> >> And we'd have to think a little about how to handle variable values
> >> that are discovered to be erroneous when we try to assign them to the
> >> variable --- probably we can just drop them, but does that make the
> >> semantics weird at all?
>
> > Maybe we can require a default value as a parameter to
> > add_guc_variable() and use that?
>
> Well, the new GUC variable struct would include a default by definition,
> and presumably that value would "bubble up" to replace any values that
> are found illegal.
>
> The sort of semantic funny I am thinking of is like this:
> * postgresql.conf contains pljava::var = somegoodvalue
> * ALTER DATABASE SET supplies pljava::var = somebadvalue
> For builtin variables the ALTER DATABASE value would be rejected on
> sight and the end result would be that the variable contains
> 'somegoodvalue'. However if we don't yet know the variable at backend
> startup, 'somebadvalue' will replace 'somegoodvalue' completely, and
> then when the PL actually gets loaded it will get thrown away. End
> result is that the variable will have whatever its hardwired default is,
> and not 'somegoodvalue' as one would wish. Even more surprising, a
> subsequent SIGHUP would make it acquire 'somegoodvalue'.
>
> This particular case could be dealt with by forcing a rescan of
> postgresql.conf after new variables are defined (I think we need only do
> so if any errors are detected in assigning values), but that will not
> handle everything. We don't have any way to get back overridden values
> from other sources such as the postmaster command line.
>
> It seems likely to me that such corner cases are sufficiently bizarre to
> not bother anyone, but we need to think more to make sure that there
> aren't any that would bother someone.
>
> regards, tom lane
>

Thomas Hallgren wrote:
> Some very good suggestions where made here. What happens next? Will this end
> up in a TODO list where someone can "claim the task" (I'm trying to learn
> how the process works) ?

If someone doesn't jump right on it and make a "diff -c" proposal, it
probably belongs on the TODO list. If your need is sufficiently high,
and you have the time to take it on, then go for it ;-). If not, I might
someday, but no promises for 7.5.

Joe

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo(at)postgresql(dot)org

Joe Conway wrote:
> Thomas Hallgren wrote:
> > Some very good suggestions where made here. What happens next? Will this end
> > up in a TODO list where someone can "claim the task" (I'm trying to learn
> > how the process works) ?
>
> If someone doesn't jump right on it and make a "diff -c" proposal, it
> probably belongs on the TODO list. If your need is sufficiently high,
> and you have the time to take it on, then go for it ;-). If not, I might
> someday, but no promises for 7.5.

I assume this is regarding disabling of triggers. We already have that
on the TODO list:

* Allow triggers to be disabled [trigger]

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

No, this was not related to triggers at all. The original discussion
concerned GUC functionality to handle configuration settings for pl<lang>
extensions.

- thomas

----- Original Message -----
From: "Bruce Momjian" <pgman(at)candle(dot)pha(dot)pa(dot)us>
To: "Joe Conway" <mail(at)joeconway(dot)com>
Cc: "Thomas Hallgren" <thhal(at)mailblocks(dot)com>; "Tom Lane"
<tgl(at)sss(dot)pgh(dot)pa(dot)us>; "Peter Eisentraut" <peter_e(at)gmx(dot)net>;
<pgsql-hackers(at)postgresql(dot)org>
Sent: Thursday, February 19, 2004 21:00
Subject: Re: [HACKERS] Advice regarding configuration parameters

> Joe Conway wrote:
> > Thomas Hallgren wrote:
> > > Some very good suggestions where made here. What happens next? Will
this end
> > > up in a TODO list where someone can "claim the task" (I'm trying to
learn
> > > how the process works) ?
> >
> > If someone doesn't jump right on it and make a "diff -c" proposal, it
> > probably belongs on the TODO list. If your need is sufficiently high,
> > and you have the time to take it on, then go for it ;-). If not, I might
> > someday, but no promises for 7.5.
>
> I assume this is regarding disabling of triggers. We already have that
> on the TODO list:
>
> * Allow triggers to be disabled [trigger]
>
> --
> Bruce Momjian | http://candle.pha.pa.us
> pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
> + If your life is a hard drive, | 13 Roberts Road
> + Christ can be your backup. | Newtown Square, Pennsylvania
19073
>

Thomas Hallgren wrote:
> No, this was not related to triggers at all. The original discussion
> concerned GUC functionality to handle configuration settings for pl<lang>
> extensions.

OK. If you guys agree on TODO wording, I will add it.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

How about,

"Allow outside agents to extend the GUC variable set"

- thomas

----- Original Message -----
From: "Bruce Momjian" <pgman(at)candle(dot)pha(dot)pa(dot)us>
To: "Thomas Hallgren" <thhal(at)mailblocks(dot)com>
Cc: "Joe Conway" <mail(at)joeconway(dot)com>; "Tom Lane" <tgl(at)sss(dot)pgh(dot)pa(dot)us>;
"Peter Eisentraut" <peter_e(at)gmx(dot)net>; <pgsql-hackers(at)postgresql(dot)org>
Sent: Friday, February 20, 2004 04:39
Subject: Re: [HACKERS] Advice regarding configuration parameters

> Thomas Hallgren wrote:
> > No, this was not related to triggers at all. The original discussion
> > concerned GUC functionality to handle configuration settings for
pl<lang>
> > extensions.
>
> OK. If you guys agree on TODO wording, I will add it.
>
> --
> Bruce Momjian | http://candle.pha.pa.us
> pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
> + If your life is a hard drive, | 13 Roberts Road
> + Christ can be your backup. | Newtown Square, Pennsylvania
19073
>

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo(at)postgresql(dot)org

Thomas Hallgren wrote:
> How about,
>
> "Allow outside agents to extend the GUC variable set"

Added:

* Allow external interfaces to extend the GUC variable set

---------------------------------------------------------------------------

>
> - thomas
>
> ----- Original Message -----
> From: "Bruce Momjian" <pgman(at)candle(dot)pha(dot)pa(dot)us>
> To: "Thomas Hallgren" <thhal(at)mailblocks(dot)com>
> Cc: "Joe Conway" <mail(at)joeconway(dot)com>; "Tom Lane" <tgl(at)sss(dot)pgh(dot)pa(dot)us>;
> "Peter Eisentraut" <peter_e(at)gmx(dot)net>; <pgsql-hackers(at)postgresql(dot)org>
> Sent: Friday, February 20, 2004 04:39
> Subject: Re: [HACKERS] Advice regarding configuration parameters
>
>
> > Thomas Hallgren wrote:
> > > No, this was not related to triggers at all. The original discussion
> > > concerned GUC functionality to handle configuration settings for
> pl<lang>
> > > extensions.
> >
> > OK. If you guys agree on TODO wording, I will add it.
> >
> > --
> > Bruce Momjian | http://candle.pha.pa.us
> > pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
> > + If your life is a hard drive, | 13 Roberts Road
> > + Christ can be your backup. | Newtown Square, Pennsylvania
> 19073
> >
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo(at)postgresql(dot)org
>

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Patch posted on the patches list :-)

Let me know what you think.

- thomas

"Joe Conway" <mail(at)joeconway(dot)com> wrote in message
news:403509B7(dot)1020708(at)joeconway(dot)com(dot)(dot)(dot)
> Thomas Hallgren wrote:
> > Some very good suggestions where made here. What happens next? Will this
end
> > up in a TODO list where someone can "claim the task" (I'm trying to
learn
> > how the process works) ?
>
> If someone doesn't jump right on it and make a "diff -c" proposal, it
> probably belongs on the TODO list. If your need is sufficiently high,
> and you have the time to take it on, then go for it ;-). If not, I might
> someday, but no promises for 7.5.
>
> Joe
>
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo(at)postgresql(dot)org
>

Tom Lane wrote:
> The sort of semantic funny I am thinking of is like this:
> * postgresql.conf contains pljava::var = somegoodvalue
> * ALTER DATABASE SET supplies pljava::var = somebadvalue
> For builtin variables the ALTER DATABASE value would be rejected on
> sight and the end result would be that the variable contains
> 'somegoodvalue'. However if we don't yet know the variable at backend
> startup, 'somebadvalue' will replace 'somegoodvalue' completely, and
> then when the PL actually gets loaded it will get thrown away. End
> result is that the variable will have whatever its hardwired default is,
> and not 'somegoodvalue' as one would wish. Even more surprising, a
> subsequent SIGHUP would make it acquire 'somegoodvalue'.
>
> This particular case could be dealt with by forcing a rescan of
> postgresql.conf after new variables are defined (I think we need only do
> so if any errors are detected in assigning values), but that will not
> handle everything. We don't have any way to get back overridden values
> from other sources such as the postmaster command line.
>
This is the one case that I, after some consideration, decided not to
deal with. I feel that a rescan will only partially solve the problem
anyway. There might have been multiple assignments to a placeholder
prior to when the module is loaded. The only way to implement something
in this direction is to remember all assignments that has been made to
each placeholder variable and then fall back to the last one that has an
OK value once the module loads. Since a warning mechanism is in place
already I felt that such an implementation would be overkill.

Kind regards,

Thomas Hallgren