Re: Attention PL authors: want to be listed in template table?

I've committed changes to implement the cut-down form of this proposal:
http://archives.postgresql.org/pgsql-hackers/2005-08/msg01185.php
discussed here:
http://archives.postgresql.org/pgsql-hackers/2005-09/msg00138.php

Barring further changes, we'll have a hard-wired template list for 8.1
and a real system catalog in 8.2. So there's a choice now for PLs that
are not part of the core distribution: do you want to be listed in the
hard-wired template?

The advantages of being listed are:

1. Reloading old dumps that involve your language should be easier,
since problems like version-specific paths to shared libraries will
go away.

2. Your PL support functions will end up in pg_catalog instead of
the public schema, which will please people who'd like to remove public
from their installations.

The main disadvantage I can see is that you won't easily be able to
change your PL creation parameters (eg, add a validator function)
over the lifespan of the 8.1 release. So depending on your development
roadmap you might think this a bad tradeoff.

If you want to be listed, let me know. What I need to know to list you
is values for this table:

typedef struct
{
char *lanname; /* PL name */
bool lantrusted; /* trusted? */
char *lanhandler; /* name of handler function */
char *lanvalidator; /* name of validator function, or NULL */
char *lanlibrary; /* path of shared library */
} PLTemplate;

As examples, the entries for the core PLs are

{ "plpgsql", true, "plpgsql_call_handler", "plpgsql_validator",
"$libdir/plpgsql" },
{ "pltcl", true, "pltcl_call_handler", NULL,
"$libdir/pltcl" },
{ "pltclu", false, "pltclu_call_handler", NULL,
"$libdir/pltcl" },
{ "plperl", true, "plperl_call_handler", "plperl_validator",
"$libdir/plperl" },
{ "plperlu", false, "plperl_call_handler", "plperl_validator",
"$libdir/plperl" },
{ "plpythonu", false, "plpython_call_handler", NULL,
"$libdir/plpython" },

regards, tom lane

Tom Lane wrote:
> If you want to be listed, let me know. What I need to know to list you
> is values for this table:
>
> typedef struct
> {
> char *lanname; /* PL name */
> bool lantrusted; /* trusted? */
> char *lanhandler; /* name of handler function */
> char *lanvalidator; /* name of validator function, or NULL */
> char *lanlibrary; /* path of shared library */
> } PLTemplate;
>

Hi Tom,

Please include PL/R:

{ "plr", false, "plr_call_handler", NULL, "$libdir/plr" },

Thanks,

Joe

p.s. my (supposedly) static IP changed this summer -- I think my direct
mail might be getting blocked by your spam filters again.

Tom Lane wrote:
> Barring further changes, we'll have a hard-wired template list for
> 8.1 and a real system catalog in 8.2. So there's a choice now for
> PLs that are not part of the core distribution: do you want to be
> listed in the hard-wired template?

Is there any way to create the language and not use the template (an
override option of some kind)?

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> Is there any way to create the language and not use the template (an
> override option of some kind)?

There is deliberately not a way to override (other than using a
different name for the PL). One of the points of this facility is to
fix up broken PL definitions being imported from old dump files, so
believing what the CREATE LANGUAGE command says is exactly what we
don't want.

This will definitely be more palatable with a system catalog that you
could alter, of course. Still, I don't see that there's any big
problem. For instance, if you think you might want a validator later,
you can set up a no-op validator procedure today, and then the template
doesn't need to change when you make the validator do something.
Similarly, you could future-proof yourself against adding a trusted (or
untrusted) variant by putting in some stubs now.

regards, tom lane

Tom Lane wrote:

>For instance, if you think you might want a validator later,
>you can set up a no-op validator procedure today, and then the template
>doesn't need to change when you make the validator do something.
>Similarly, you could future-proof yourself against adding a trusted (or
>untrusted) variant by putting in some stubs now.
>
>
>
>

Only necessary for one release, right?

Anyway, clearly there are cases where a validator make no sense or very
little sense (pl/sh and pl/{j,java} spring to mind).

cheers

andrew

Tom Lane wrote:
> There is deliberately not a way to override (other than using a
> different name for the PL). One of the points of this facility is to
> fix up broken PL definitions being imported from old dump files, so
> believing what the CREATE LANGUAGE command says is exactly what we
> don't want.

I don't doubt that, but I just have a stomach ache with this entire
notion of having a hard-coded exception list of the sort "if the user
requires this, do this instead". Why don't we just put all PLs that we
know of into pg_language to begin with, revoke the permissions, and
just let CREATE LANGUAGE fail when the dump is restored? (The dump
should restore the permissions, shouldn't it?) At least that would let
you alter the "template" using existing means.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Andrew Dunstan wrote:
> Anyway, clearly there are cases where a validator make no sense or
> very little sense (pl/sh and pl/{j,java} spring to mind).

PL/sh has a validator. :-)

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut wrote:

>Andrew Dunstan wrote:
>
>
>>Anyway, clearly there are cases where a validator make no sense or
>>very little sense (pl/sh and pl/{j,java} spring to mind).
>>
>>
>
>PL/sh has a validator. :-)
>
>
>

Really? Curious. I must look more closely.

cheers

andrew

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> I don't doubt that, but I just have a stomach ache with this entire
> notion of having a hard-coded exception list of the sort "if the user
> requires this, do this instead".

I don't see it as an exception list. The direction I see for this is
that the parameters to CREATE LANGUAGE are obsolete and will eventually
be removed altogether, with "CREATE LANGUAGE foo" using an existing
template as the only recommended way to do it.

> Why don't we just put all PLs that we
> know of into pg_language to begin with, revoke the permissions, and
> just let CREATE LANGUAGE fail when the dump is restored?

To do that we'd have to force an initdb, in which case we might as well
add the proposed pltemplate catalog and have done with it.

I'm entirely willing to add the catalog if people are willing to accept
an initdb now. As I said earlier, I don't want to mess with adding a
usage privilege just yet, but we could add the catalog with a
nonfunctional ACL column and write that code later.

regards, tom lane

Tom Lane wrote:
> I don't see it as an exception list. The direction I see for this is
> that the parameters to CREATE LANGUAGE are obsolete and will
> eventually be removed altogether, with "CREATE LANGUAGE foo" using an
> existing template as the only recommended way to do it.

So your proposal is to enable a new language by doing:

1. register a template
2. activate template using CREATE LANGUAGE (which would copy it to
pg_language)

How is this different from

1. register language in pg_language without privileges
2. activate language by granting privileges

This already works and uses only well-known concepts.

> > Why don't we just put all PLs that we
> > know of into pg_language to begin with, revoke the permissions, and
> > just let CREATE LANGUAGE fail when the dump is restored?
>
> To do that we'd have to force an initdb, in which case we might as
> well add the proposed pltemplate catalog and have done with it.

I don't have a strong opinion on initdb, but a difference would be that
this solution would not *require* an initdb but only offer the
improvement if initdb were done while continuing to work as before
without initdb.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> So your proposal is to enable a new language by doing:

> 1. register a template
> 2. activate template using CREATE LANGUAGE (which would copy it to
> pg_language)

That's the long-term idea, yes.

> How is this different from

> 1. register language in pg_language without privileges
> 2. activate language by granting privileges

Because you can't create a language without first creating the support
procedures, which ordinarily requires having the shared library present.
(This is why I proposed text names for the support procedures in
pltemplate, rather than OID references.) This is perhaps not an issue
for entries wired in by initdb, but it's definitely an issue for manual
addition of template entries.

Also, ISTM your proposal is to cause "CREATE LANGUAGE foo" on an
already-existing language to execute "GRANT USAGE ON LANGUAGE foo TO PUBLIC"
instead, rather than erroring out. That doesn't seem to pass the
least-surprise test at all.

regards, tom lane

i am using postgresql 8.0.3 as a single user by running "postgres"

I want to purge all contents in the bufferpool, and I did this by
calling "InitBufTable(256)" (buf_table.c) after each query. However,
this seems not working for each followup query I still get less disk read
(tracked by "smgrread()" in smgr.c) and increased bufferhitcount.

Could anyone tell me which module shall i modify instead?

huaxin zhang <uwcssa(at)gmail(dot)com> writes:
> I want to purge all contents in the bufferpool,

Why do you think that's a good idea? It certainly won't purge the
kernel's disk caches, so if you're hoping to restore the system to
ground zero this won't do it.

> and I did this by
> calling "InitBufTable(256)" (buf_table.c) after each query.

Let's see ... I'd expect that to run out of shared memory pretty soon,
but not till after it's thoroughly corrupted your database ;-)

There is no API exported by bufmgr.c that does what you want. Something
like a combination of FlushRelationBuffers and DropRelFileNodeBuffers,
but processing a whole database (see also DropBuffers) or the whole buffer
cache, might work. You can't drop a buffer that some other process has
pinned, however.

regards, tom lane

Tom,
I assume that the path of the shared library will be somehow relative to
the GUC dynamic_library_path? If not, the lanlibrary should be changed
to "$libdir/libpljava". That requires that PL/Java is installed within
the PostgreSQL distribution.

I also assume that the handler name can be prefixed with a schema name?
All PL/Java support functions reside in the sqlj schema.

If my assumptions are correct, then please add:

{ "java", true, "sqlj.java_call_handler", NULL,
"libpljava" },
{ "javaU", false, "sqlj.javau_call_handler", NULL,
"libpljava" },

The validator for PL/Java will have to wait until 8.2.

Regards,
Thomas Hallgren

Tom Lane wrote:
> I've committed changes to implement the cut-down form of this proposal:
> http://archives.postgresql.org/pgsql-hackers/2005-08/msg01185.php
> discussed here:
> http://archives.postgresql.org/pgsql-hackers/2005-09/msg00138.php
>
> Barring further changes, we'll have a hard-wired template list for 8.1
> and a real system catalog in 8.2. So there's a choice now for PLs that
> are not part of the core distribution: do you want to be listed in the
> hard-wired template?
>
> The advantages of being listed are:
>
> 1. Reloading old dumps that involve your language should be easier,
> since problems like version-specific paths to shared libraries will
> go away.
>
> 2. Your PL support functions will end up in pg_catalog instead of
> the public schema, which will please people who'd like to remove public
> from their installations.
>
> The main disadvantage I can see is that you won't easily be able to
> change your PL creation parameters (eg, add a validator function)
> over the lifespan of the 8.1 release. So depending on your development
> roadmap you might think this a bad tradeoff.
>
> If you want to be listed, let me know. What I need to know to list you
> is values for this table:
>
> typedef struct
> {
> char *lanname; /* PL name */
> bool lantrusted; /* trusted? */
> char *lanhandler; /* name of handler function */
> char *lanvalidator; /* name of validator function, or NULL */
> char *lanlibrary; /* path of shared library */
> } PLTemplate;
>
> As examples, the entries for the core PLs are
>
> { "plpgsql", true, "plpgsql_call_handler", "plpgsql_validator",
> "$libdir/plpgsql" },
> { "pltcl", true, "pltcl_call_handler", NULL,
> "$libdir/pltcl" },
> { "pltclu", false, "pltclu_call_handler", NULL,
> "$libdir/pltcl" },
> { "plperl", true, "plperl_call_handler", "plperl_validator",
> "$libdir/plperl" },
> { "plperlu", false, "plperl_call_handler", "plperl_validator",
> "$libdir/plperl" },
> { "plpythonu", false, "plpython_call_handler", NULL,
> "$libdir/plpython" },
>
>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Have you searched our list archives?
>
> http://archives.postgresql.org
>

Thomas Hallgren <thhal(at)mailblocks(dot)com> writes:
> I assume that the path of the shared library will be somehow relative to
> the GUC dynamic_library_path?

Well, whatever you put in the template is what will be in the probin
field of the support functions. I suppose it does not *have* to use
$libdir, but I would definitely recommend using $libdir rather than
depending on dynamic_library_path.

> I also assume that the handler name can be prefixed with a schema name?
> All PL/Java support functions reside in the sqlj schema.

Not if you use the template facility, they won't. The handler and
validator are hard-wired to live in pg_catalog under this scheme.

> The validator for PL/Java will have to wait until 8.2.

Do you want to drop in a stub? It's only a one-line function.

regards, tom lane

Tom Lane wrote:

>Thomas Hallgren <thhal(at)mailblocks(dot)com> writes:
>
>
>>I assume that the path of the shared library will be somehow relative to
>>the GUC dynamic_library_path?
>>
>>
>
>Well, whatever you put in the template is what will be in the probin
>field of the support functions. I suppose it does not *have* to use
>$libdir, but I would definitely recommend using $libdir rather than
>depending on dynamic_library_path.
>
>
I'm not I understand this. The default setting for the
dynamic_library_path is $libdir, isn't it? So why have another hardwired
setting here? Wouldn't it be better if all PL's used the
dynamic_library_path setting at all times?

>>I also assume that the handler name can be prefixed with a schema name?
>>All PL/Java support functions reside in the sqlj schema.
>>
>>
>
>Not if you use the template facility, they won't. The handler and
>validator are hard-wired to live in pg_catalog under this scheme.
>
>
Ok. That's fine. They're not covered by the SQL standard anyway. I have
a lot of other "support functions" for managing jar files, classpath,
etc. in the database. They all live in the sqlj schema but they will not
be affected by this.

>>The validator for PL/Java will have to wait until 8.2.
>>
>>
>
>Do you want to drop in a stub? It's only a one-line function.
>
>
Yes, that's a good idea. I'll call them "java_validator" and
"javau_validator" respectively.

Regards,
Thomas Hallgren

Thomas Hallgren <thhal(at)mailblocks(dot)com> writes:
> Wouldn't it be better if all PL's used the
> dynamic_library_path setting at all times?

I wouldn't think so at all. That's just another way to shoot yourself
in the foot; $libdir is the correct place by definition, and any other
directory is not the correct place, by definition. This is certainly
true for the PLs that are part of the distribution, but I don't see that
it would be any less true for add-on PLs.

As an example of how to shoot yourself in the foot, consider someone
setting dynamic_library_path to point to a version-specific directory:
/home/postgres/version74/lib
and then updating to a newer release without changing that.

If there were some upside to letting people load alternate versions of
PLs by changing dynamic_library_path then you might have a point ...
but I really don't see any value there, just risk of breakage.

>> Do you want to drop in a stub? It's only a one-line function.
>>
> Yes, that's a good idea. I'll call them "java_validator" and
> "javau_validator" respectively.

OK, I'll add that to the template info.

regards, tom lane

Thomas Hallgren wrote:
> If my assumptions are correct, then please add:
>
> { "java", true, "sqlj.java_call_handler", NULL,
> "libpljava" },
> { "javaU", false, "sqlj.javau_call_handler", NULL,
> "libpljava" },

In the interest of uniformity, please rename your libraries to omit the
"lib" prefix.

The other problem I see emerging here is that in certain environments,
the "java" language may not be trusted, such as when it is compiled
with GCJ. Then, this built-in template will override the CREATE
LANGUAGE specification and introduce a security hole.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Tom Lane wrote:
> > How is this different from
>
> > 1. register language in pg_language without privileges
> > 2. activate language by granting privileges
>
> Because you can't create a language without first creating the support
> procedures, which ordinarily requires having the shared library
> present.

We are only talking about well-known procedural languages. The issue of
completely new languages added by the user is addressed by neither
proposal.

> Also, ISTM your proposal is to cause "CREATE LANGUAGE foo" on an
> already-existing language to execute "GRANT USAGE ON LANGUAGE foo TO
> PUBLIC" instead, rather than erroring out. That doesn't seem to pass
> the least-surprise test at all.

Clearly, there's going to be some surprise element. The surprise
element proposed by you is that the command does something completely
different than specified (which possibly introduces security holes, see
other mail). My proposal is that the command does only a subset of
what it would normally do, which amounts to some sort of implicit "OR
REPLACE", which people are familiar with.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Tom Lane wrote:
> Barring further changes, we'll have a hard-wired template list for
> 8.1 and a real system catalog in 8.2. So there's a choice now for
> PLs that are not part of the core distribution: do you want to be
> listed in the hard-wired template?

Another note: PLs that are not in the core distribution are not created
with createlang in the first place, so the issues that led up to this
proposal don't apply AFAICT, that is:

- "pg_dump dumps them at a pretty low semantic level" --> pg_dump dumps
them at exactly the level they were created at.

- "problem with adding a validator" --> An update of PostgreSQL does not
change the properties of external modules

- "issues with hardwired paths to the shared libraries" --> This is an
issue with all loadable modules and needs a more general solution.
Basically, on a major version upgrade, we need to make the user
recompile all shared libraries.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> We are only talking about well-known procedural languages. The issue of
> completely new languages added by the user is addressed by neither
> proposal.

Sure it is: in my proposal, you can add new languages to the template
catalog. (Which admittedly we don't have today. We could have it
tomorrow, though, if an initdb for beta2 is OK.)

>> Also, ISTM your proposal is to cause "CREATE LANGUAGE foo" on an
>> already-existing language to execute "GRANT USAGE ON LANGUAGE foo TO
>> PUBLIC" instead, rather than erroring out. That doesn't seem to pass
>> the least-surprise test at all.

> Clearly, there's going to be some surprise element. The surprise
> element proposed by you is that the command does something completely
> different than specified (which possibly introduces security holes, see
> other mail).

Haven't seen this other mail yet...

> My proposal is that the command does only a subset of
> what it would normally do, which amounts to some sort of implicit "OR
> REPLACE", which people are familiar with.

No, because what the command would normally do in that situation
(the language already exists) is error out. Silently granting rights
is a security risk. What if the DBA already created the language,
adjusted its permissions the way he wants, and then accidentally did
another CREATE LANGUAGE (or more likely, loaded a dump file containing
same)? Now he's got language usage granted to PUBLIC, and he doesn't
even know it. The CREATE OR REPLACE variants that we have don't muck
with the object's permissions, so I don't see them as good precedent.

regards, tom lane

I wrote:
> Another note: PLs that are not in the core distribution are not
> created with createlang in the first place, so the issues that led up
> to this proposal don't apply AFAICT,

What I'm trying to say here is that it might be OK to hardcode the
properties of core languages in the server, because considering that
the only officially supported way to activate those is
"createlang" (rather than using SQL), the properties of them are in
effect already hardcoded, and from the point of view of a user who is
using createlang, nothing changes. For languages that are not shipped
in the core, we gain pretty much nothing by this approach as currently
implemented and potentially introduce more problems than we solve
(e.g., language is compiled in an untrusted way, but template says it's
trusted; template says language has validator, but user uses old
version that has none; template points to $libdir, user has it
installed elsewhere).

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> What I'm trying to say here is that it might be OK to hardcode the
> properties of core languages in the server, because considering that
> the only officially supported way to activate those is
> "createlang" (rather than using SQL), the properties of them are in
> effect already hardcoded, and from the point of view of a user who is
> using createlang, nothing changes.

Please note that one of the benefits of a template catalog is that
createlang will work for everything in the catalog, not only the core
languages.

> For languages that are not shipped
> in the core, we gain pretty much nothing by this approach as currently
> implemented and potentially introduce more problems than we solve
> (e.g., language is compiled in an untrusted way, but template says it's
> trusted; template says language has validator, but user uses old
> version that has none; template points to $libdir, user has it
> installed elsewhere).

These are straw men. We know we have a problem with hard-coded paths in
old dumps. We know we have a problem with missing validators in old
dumps (which will get worse as more languages acquire validators).
And I think that a template is much more likely to prevent than
introduce the sorts of mis-definition problems you suggest --- all of
them can easily happen now from simple user errors, whereas with a
template it would be right every time.

regards, tom lane

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> The other problem I see emerging here is that in certain environments,
> the "java" language may not be trusted, such as when it is compiled
> with GCJ.

Hmm, is that really the case? I thought Java is Java.

> Then, this built-in template will override the CREATE
> LANGUAGE specification and introduce a security hole.

But it's exactly the same hole the user would create by manually saying
CREATE TRUSTED LANGUAGE in error. I don't think that's a reasonable
argument against the template idea --- it just says that you have to be
aware of what you're doing.

(An appropriate solution, in my mind, would be to drop the trusted call
handler from the shared library if it's built with gcj --- then there's
really no possibility of doing the wrong thing.)

regards, tom lane

Tom Lane wrote:
> These are straw men.

No, these are real problems that exist today.

> We know we have a problem with hard-coded paths
> in old dumps.

This is irrelevant for non-core languages. If I have PL/foo 1.0
installed and upgrade from PostgreSQL 8.0 to 8.1, then I have to
manually recompile PL/foo and install the shared library. So manual
intervention is required in any case and no template system of any kind
will help.

Moreover, the hard-coded paths will not be solved by your proposal. I
don't suppose that in a release or three will start hard coding postgis
parameters to work around hard coded paths in general plug-in modules?

> We know we have a problem with missing validators in
> old dumps

This is irrelevant for non-core languages. If I have PL/foo 1.0
installed and upgrade from PostgreSQL 8.0 to 8.1, the validator in
PL/foo 1.0 will not magically appear or disappear. The information in
the dump is the only accurate information that exists about the
existence of a validator. The information in the template is
completely unrelated to reality. You are assuming that during the
update from PostgreSQL 8.0 to PostgreSQL 8.1 there will also be an
upgrade from PL/foo 1.0 to PL/foo ${latest-at-time-of-8.1-release}, but
there is simply no basis for assuming that.

> And I think that a template is much more likely to
> prevent than introduce the sorts of mis-definition problems you
> suggest

No, the template system introduces misdefinitions that would otherwise
not exist.

> --- all of them can easily happen now from simple user
> errors, whereas with a template it would be right every time.

As you can see above, that is not true.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Tom Lane wrote:
> But it's exactly the same hole the user would create by manually
> saying CREATE TRUSTED LANGUAGE in error. I don't think that's a
> reasonable argument against the template idea --- it just says that
> you have to be aware of what you're doing.
>
> (An appropriate solution, in my mind, would be to drop the trusted
> call handler from the shared library if it's built with gcj --- then
> there's really no possibility of doing the wrong thing.)

The case in reality is this: First of all, the language name "java" is
fixed by the SQL standard, so we ought to allow alternative
implementations to use that name. I'm not sure what kind of interface
the PL/J people are working on, but if they also lay claim to the name
"java", then we have a problem. Second, Java is not, in fact, always
Java, so different quality variants of the same implementations exist.
The Debian package of pljava is compiled using gcj, but it is also
planned to provide an alternative version that is compiled using the
Sun JDK. That way, users can trade off quality/features vs. licensing
freedom. This is the kind of freedom of choice that the current
situation allows. Now the template solution will buy nothing here,
because there has never been a release of PL/Java that (a) used a
hard-coded shared library path, or (b) was ambiguous about its
installation schema in the first place. And the validator upgrade
issue will also not be solved unless there is a major synchronization
effort between the PostgreSQL upstream, the PL/Java upstream, the
PostgreSQL packager, and the PL/Java packager (4 different parties)
(and in worse cases: the GCC packagers, etc.). In other words, it is
not possible and could only be solved effectively by putting PL/Java in
the core both upstream and downstream.

I think you are assuming all the way through this discussion that a
PostgreSQL upgrade will also entail an upgrade of all procedural
languages. People are putting dummy validator functions into their
code right now, but (1) will they also release that new version at the
same time as PostgreSQL 8.1 comes out?, and (2) will users be willing
to upgrade at the same time? Point 1 is questionable at best, and
point 2 is completely out of the question, not only because I've seen
too many PL releases just plain break, but simply because people may
want to upgrade one thing at a time. Again, we could work around both
of these points by putting the PL in question into the core, in which
case I am willing to support the template idea, but for languages out
of the core I don't think the implicitly required synchronization
efforts have been taking into account.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> Tom Lane wrote:
>> We know we have a problem with hard-coded paths
>> in old dumps.

> This is irrelevant for non-core languages. If I have PL/foo 1.0
> installed and upgrade from PostgreSQL 8.0 to 8.1, then I have to
> manually recompile PL/foo and install the shared library. So manual
> intervention is required in any case and no template system of any kind
> will help.

Certainly you have to install new software, but that is a task exactly
equivalent to installing the new version of Postgres itself. The
problem being addressed here is fixing wrong information in your pg_dump
scripts, and that is just as real for non-core as core PLs. The
proposal *will* fix the need for manual intervention while reloading
dump data.

> Moreover, the hard-coded paths will not be solved by your proposal.

On what basis do you assert that? The proposal will replace those paths
by "$libdir/plfoo" (or whatever is in the template), which is something
we should be able to go forward with indefinitely ... and if we can't,
a template update will fix it in some future release.

Now this certainly discriminates against installing PLs outside $libdir,
but I haven't heard any good reason why people would need to do that.

>> We know we have a problem with missing validators in
>> old dumps

> This is irrelevant for non-core languages.

Again, I see no basis for that assertion. It is only true to the extent
that non-core languages haven't created validators yet.

> If I have PL/foo 1.0
> installed and upgrade from PostgreSQL 8.0 to 8.1, the validator in
> PL/foo 1.0 will not magically appear or disappear.

As a general rule, PLs have to at least be recompiled to move to a new
release of the backend, and usually need some source code tweaking as
well. So I think your premise is faulty. You'll need to install the
8.1 version of that PL, and that very possibly will have a validator
where none existed before.

> ... The information in
> the dump is the only accurate information that exists about the
> existence of a validator.

Except that the information in the dump is likely *not* to be accurate.
It tells you the way things were some number of releases ago, and as
we've already seen, people frequently propagate their dumps forward
across multiple releases --- indeed, one would hope that they'd be able
to do so. We need to fix things so that that Just Works in the face of
improvements to PLs.

> You are assuming that during the
> update from PostgreSQL 8.0 to PostgreSQL 8.1 there will also be an
> upgrade from PL/foo 1.0 to PL/foo ${latest-at-time-of-8.1-release}, but
> there is simply no basis for assuming that.

How many of the past major releases have allowed a previous PL to be
used without any changes? Hardly any AFAIR, and 8.1 is certainly not
far behind the previous ones when it comes to making internal API
changes that affect PLs.

In any case, not one of these arguments seems to me to favor the
alternative of preloading definitions into pg_language: that preloaded
definition is going to be just as wrong as the template information, if
either one is wrong. Now it's true that a hardwired template table is
harder to fix than a system catalog, but we can get rid of the hardwired
table at the same cost (viz, a forced initdb) as adding preloaded
definitions. I don't see any other advantage to doing it that way.

One other response to your point about surprise or lack of it: in my
proposal, in a release or so everyone will just be doing "CREATE
LANGUAGE foo" and there will be no surprise that the system knows how to
fill in the low-level details. The handler functions will be relegated
to what they should have been all along: an implementation detail. But
I think that having CREATE LANGUAGE alter the permissions of an existing
entry would be a permanent security risk; it will never not surprise
people, because other forms of CREATE don't do that.

regards, tom lane

Tom Lane wrote:

>Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
>
>
>>The other problem I see emerging here is that in certain environments,
>>the "java" language may not be trusted, such as when it is compiled
>>with GCJ.
>>
>>
Well, yes. But use the word environment in singular please :-) To my
knowledge the security is full-proof with all other VM's since they all
use the standard runtime libraries. The GCJ support is as experimental
as the GCJ in itself and cannot be trusted in production.

>
>Hmm, is that really the case? I thought Java is Java.
>
>
GCJ is a clean house implementation of Java. They don't use the runtime
libraries from Sun and they are not really there yet in their efforts to
copy the functionality. One of the things that lag behind is security.
They hope to have a better security implementation before the year end
but there's no promise.

>>Then, this built-in template will override the CREATE
>>LANGUAGE specification and introduce a security hole.
>>
>>
>
>But it's exactly the same hole the user would create by manually saying
>CREATE TRUSTED LANGUAGE in error. I don't think that's a reasonable
>argument against the template idea --- it just says that you have to be
>aware of what you're doing.
>
>(An appropriate solution, in my mind, would be to drop the trusted call
>handler from the shared library if it's built with gcj --- then there's
>really no possibility of doing the wrong thing.)
>
>
That's a though although I'm not sure we would gain anything. No PL/Java
binaries are provided that run with GCJ. You have to compile from source
using some specific settings. If you are skilled enough to experiment
with that, then there's a good chance you are able to tweak the source
to enable the trusted call handler also.

Regards,
Thomas Hallgren

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> The case in reality is this: First of all, the language name "java" is
> fixed by the SQL standard, so we ought to allow alternative
> implementations to use that name. I'm not sure what kind of interface
> the PL/J people are working on, but if they also lay claim to the name
> "java", then we have a problem. Second, Java is not, in fact, always
> Java, so different quality variants of the same implementations exist.
> The Debian package of pljava is compiled using gcj, but it is also
> planned to provide an alternative version that is compiled using the
> Sun JDK. That way, users can trade off quality/features vs. licensing
> freedom.

Are you seriously suggesting that it's a good idea for the single
language name "java" to mean different things at different
installations? I can't believe that that wouldn't lead to chaos.
In any case, "java" has not been put forward as one of the template
entries, and as long as we don't accept a template for it, we have
not made the situation any worse.

> I think you are assuming all the way through this discussion that a
> PostgreSQL upgrade will also entail an upgrade of all procedural
> languages.

Yes, I am assuming that, and I challenge you to supply examples of PLs
that won't require at least a recompile before there's any hope of their
working on 8.1. In a quick look through the CVS logs, I note that
heap_openr/index_openr are gone, the representation of pg_proc entries
is quite a bit different than it was in 8.0, and there are incompatible
changes in the APIs of spi.c and dynahash.c. The pg_proc changes in
particular practically guarantee a need for a source-code update.

regards, tom lane

Thomas Hallgren <thhal(at)mailblocks(dot)com> writes:
> GCJ is a clean house implementation of Java. They don't use the runtime
> libraries from Sun and they are not really there yet in their efforts to
> copy the functionality. One of the things that lag behind is security.
> They hope to have a better security implementation before the year end
> but there's no promise.

OK, so that is a transient limitation of the GCJ work, not something
fundamental. Thanks for clarifying. In that case I agree that trying
to restrict it mechanically isn't a good idea --- the code restriction
would still be around after the problem was gone.

I still think this is irrelevant to the PL template discussion, however,
since neither our past approach nor either of the proposals will make it
the least bit difficult for a user to mislabel pljava as TRUSTED when
the underlying implementation isn't really trustworthy.

(What the PL template approach *would* do is make it difficult to create
a language that is trusted but named pljavau, or untrusted and named
pljava. Personally I don't see that as a bad thing, however. The
opportunity for confusion is far too great if you go against the
established naming conventions.)

regards, tom lane

Tom Lane wrote:
> Now this certainly discriminates against installing PLs outside
> $libdir, but I haven't heard any good reason why people would need to
> do that.

Development, testing, staging, ...

> > If I have PL/foo 1.0
> > installed and upgrade from PostgreSQL 8.0 to 8.1, the validator in
> > PL/foo 1.0 will not magically appear or disappear.
>
> As a general rule, PLs have to at least be recompiled to move to a
> new release of the backend, and usually need some source code
> tweaking as well.

You keep saying that, but the fact is that most of the non-core PLs will
work just fine across several releases, if only because they have some
#ifdef's. So, no, you do not need to upgrade your PLs when you upgrade
PostgreSQL. You really don't. You just recompile them and put them
back in the same place. There are certainly constraints and
exceptions, but as a general rule it simply is not the case. If it
were, I would not have started this discussion.

Taking a step back and considering this "validator" business in a more
general context: Basically, the validator is just a property that may
or may not be applicable to a C function hidden inside a shared
library. We don't know what's in the shared library, so we (currently)
have to rely on the meta information in the dump to find out. This is
a special case of a more general problem, which is why I mentioned
PostGIS earlier. Any "plug-in" will evolve, and if it's just a bit
sophisticated it will have fairly tight dependencies on the server
version, which is certainly the case for much of the GIS and GiST
stuff. So here we have the same problem. The old dump will have the
meta-information on the old plug-in version. Under your theory, which
is surely true in certain cases, the user will have to upgrade the
plug-in at the same time. So the restore of the dump will have the
old, now wrong information. I don't suppose hard-coding the PostGIS
schema and ignoring special cases of CREATE FUNCTION will be the
answer.

One might object that these cases -- procedural languages and PostGIS
sort of things -- are different, but they are not. The chance that an
upgrade of PostgreSQL will require a version upgrade of the plug-in is
about the same. Certainly, the number of users of PostGIS and the
various GiST modules (FTS et al.) is comparable to, if not higher than
that of some of the PLs. So that means two things:

1. The problem is much worse.
2. The problem is really much less bad because external plug-ins, PL or
not, are often coded to work with multple server versions. And they
should be, because otherwise the work forced upon the development teams
to synchronize and the users to figure all that out will grow more than
linearly.

> In any case, not one of these arguments seems to me to favor the
> alternative of preloading definitions into pg_language:

That is not the alternative I am proposing. Your template idea,
hard-coded even, is just fine for core languages. For non-core
languages I am proposing that we simply do nothing because the problem
we are solving does not exist, or at least has a wildly different
nature.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> Tom Lane wrote:
>> In any case, not one of these arguments seems to me to favor the
>> alternative of preloading definitions into pg_language:

> That is not the alternative I am proposing. Your template idea,
> hard-coded even, is just fine for core languages. For non-core
> languages I am proposing that we simply do nothing because the problem
> we are solving does not exist, or at least has a wildly different
> nature.

Ah. We have been talking at cross-purposes then, because I thought you
were arguing for putting exactly the same template information into a
different place.

Given that, I think we can just agree to disagree: whether a particular
PL could benefit from a template entry is up to the author of that PL
to decide. Depending on the extent of the changes needed for 8.1, it
might or might not make sense to assume that a source code update is
needed, and if there isn't then maybe adding a template is a bad idea.

It does seem though that your arguments indicate a possible need for
local adjustment of the template info ... which means the hard-wired
approach is not good enough, and we have to take the next step of
creating a system catalog.

regards, tom lane

Thomas Hallgren wrote:
> Well, yes. But use the word environment in singular please :-) To my
> knowledge the security is full-proof with all other VM's since they
> all use the standard runtime libraries.

It's not quite as simple as that. There are a bunch of VMs and a bunch
of libraries (and a bunch of compilers), and they can be combined in
many permutations. Not all of them work with PL/Java at the moment,
but we should not hardcode support for just one of them.

> The GCJ support is as
> experimental as the GCJ in itself and cannot be trusted in
> production.

You should not say that too loud when someone from Red Hat is
listening. :-) To my knowledge GCJ is Ready(tm) as of version 4.0.
And it's being used. Distributions such as Fedora and Ubuntu will ship
(or do ship?) with everything compiled using GCJ to the extent
possible. And there are people, in particular at or near Red Hat, who
have been specifically charged for several years now to make sure that
every piece of Java code out there compiles with GCJ.

Regarding the security issue: Word from Andrew Haley of Red Hat is that
it has simply been too much work to implement security up to now. This
should not affect the judgement of the quality of GCJ, it's simply a
missing feature.

Of course, I don't intend to undermine your judgement as the author
about what you consider experimental or not, but you should expect that
if you put your code out there, people will use it in whatever way they
see fit, and in particular with whatever Java toolchain they see fit.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Tom Lane wrote:
> Are you seriously suggesting that it's a good idea for the single
> language name "java" to mean different things at different
> installations? I can't believe that that wouldn't lead to chaos.

There is a standard for the SQL integration of a "java" language, and I
don't see why there can't be multiple implementations of that same
specification. It in fact appears that there are. Of course we would
not want the programming interfaces to differ, but they don't have to.

> In any case, "java" has not been put forward as one of the template
> entries, and as long as we don't accept a template for it, we have
> not made the situation any worse.

Hmm, Thomas Hallgren sent in a template using "java" as name and you
answered "OK", so we're already there if it's already committed.

> Yes, I am assuming that, and I challenge you to supply examples of
> PLs that won't require at least a recompile before there's any hope
> of their working on 8.1.

There is no hope of that, but a mere recompilation does not change the
validator or the schema or any other property that may be under
consideration. The current code will force a *version* upgrade of all
PLs with every PostgreSQL upgrade. I need to download new code and
deal with it. That is currently not required. And considering the
general breakage of PLs out there, I don't think it's acceptable to
require it. What if the new PL/R also requires a new R? What if the
new R requires a new GCC? We don't know that, we can't control that,
we should not interfere in that. I'm not making this up; these
problems are real (although not necessarily in PL/R).

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Tom Lane wrote:
> (What the PL template approach *would* do is make it difficult to
> create a language that is trusted but named pljavau, or untrusted and
> named pljava.

But the latter is exactly what I would like to do.

> Personally I don't see that as a bad thing, however.
> The opportunity for confusion is far too great if you go against the
> established naming conventions.)

Extensibility means you don't control the naming. I guess if you want
to say that this whole idea of extensibility in the language handler
area is hereby withdrawn, doesn't work, never existed, then let's make
that clear. Then we can hardcode everything, tell people, if you want
to write a language handler, you should talk to us so we can arrange
the hooks. That is the direction we're headed in. The PostgreSQL
developers and the language handler authors dictate to the user what
language he can use in what mode. If you don't like it, here's a way
to do manual surgery to change it. Of course you can always change
everything with varying effort. So yeah, that would work, but then it
should be called that. But sacrificing user options to reduce
"confusion" is hardly our game.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> Tom Lane wrote:
>> In any case, "java" has not been put forward as one of the template
>> entries, and as long as we don't accept a template for it, we have
>> not made the situation any worse.

> Hmm, Thomas Hallgren sent in a template using "java" as name and you
> answered "OK", so we're already there if it's already committed.

Oh, I hadn't noticed. That seems like rather a bad idea --- shouldn't
it be "pljava"? ("javaU" isn't going to work either, because of
case_translate_language_name.)

>> Yes, I am assuming that, and I challenge you to supply examples of
>> PLs that won't require at least a recompile before there's any hope
>> of their working on 8.1.

> There is no hope of that, but a mere recompilation does not change the
> validator or the schema or any other property that may be under
> consideration. The current code will force a *version* upgrade of all
> PLs with every PostgreSQL upgrade. I need to download new code and
> deal with it. That is currently not required.

Really? See the oidvector changes. I think that will force at least
minor source changes on every PL. Now there may be people out there who
will prefer making a few small changes by hand to downloading a new
version ... but they can probably manage throwing in a stub validator
function too.

regards, tom lane

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> Extensibility means you don't control the naming. I guess if you want
> to say that this whole idea of extensibility in the language handler
> area is hereby withdrawn, doesn't work, never existed, then let's make
> that clear. Then we can hardcode everything, tell people, if you want
> to write a language handler, you should talk to us so we can arrange
> the hooks. That is the direction we're headed in.

Not at all! The direction we're headed in is that there are two layers
of abstraction instead of only one. What I put forward in my original
proposal was that there would be a superuser-alterable catalog of PL
templates and then pg_language would indicate what's actually available
in a particular database. I do not see that that's noticeably less
flexible than what we have done all along; especially seeing that the
DBA is not required to have a template for any particular PL.

I do concede that you've provided good reasons why a hard-wired template
table is not an adequate stopgap measure. If I go ahead and put in the
originally-proposed system catalog, will you be satisfied?

regards, tom lane

Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:

> Now this certainly discriminates against installing PLs outside $libdir,
> but I haven't heard any good reason why people would need to do that.

Well one example would be distributions that package up some PLs but want to
allow you to install some on your own. The normal model for that is to have a
directory in /usr/lib/<program>/ for the packaged software and a separate
place in /usr/local/lib/<program>/ for the locally installed add-ons.

I'm not sure how that relates to the template work you're discussing. Are all
of the templates you're creating for components that are built out of core and
therefore can be expected to be packaged?

--
greg

Greg Stark <gsstark(at)mit(dot)edu> writes:
> I'm not sure how that relates to the template work you're
> discussing. Are all of the templates you're creating for components
> that are built out of core and therefore can be expected to be
> packaged?

Well, that's exactly the point Peter is arguing: he thinks (if I
understand correctly) that the template mechanism should only be used
for stuff that's included with the core distribution. I disagree;
I have seldom seen any good reason for restricting mechanisms to work
with only core components. It does seem apparent however that we need
the templates to be reconfigurable for local conditions ... so the
hardwired table is out, even as a one-release stopgap. I'll work on
building a catalog tomorrow.

regards, tom lane

Peter Eisentraut wrote:

>Thomas Hallgren wrote:
>
>
>>Well, yes. But use the word environment in singular please :-) To my
>>knowledge the security is full-proof with all other VM's since they
>>all use the standard runtime libraries.
>>
>>
>
>It's not quite as simple as that. There are a bunch of VMs and a bunch
>of libraries (and a bunch of compilers), and they can be combined in
>many permutations. Not all of them work with PL/Java at the moment,
>but we should not hardcode support for just one of them.
>
>
AFAIK, there are only two flavors of the Java Runtime library out there.
The one that Sun provides (and small variants of it, such as the ones
that IBM, HP, and BEA) and the "classpath" clean-room implementation.
All variants of the former are OK with respect to security and only GCJ
has a working environment of the latter. In particular, only GCJ has a
functional standards conformant Java Native Interface (JNI) API to the
latter and PL/Java is built on JNI.

Should however, someone come up with another Java environment built on
"classpath" that has JNI support, then there will be another potential
environment for PL/Java. TMK, there's no such environment and none in
the making. I have serious doubts that there ever will be. IMO it would
be perfectly safe to hard code support for a trusted "java".

>>The GCJ support is as
>>experimental as the GCJ in itself and cannot be trusted in
>>production.
>>
>>
>
>You should not say that too loud when someone from Red Hat is
>listening. :-) To my knowledge GCJ is Ready(tm) as of version 4.0.
>And it's being used. Distributions such as Fedora and Ubuntu will ship
>(or do ship?) with everything compiled using GCJ to the extent
>possible. And there are people, in particular at or near Red Hat, who
>have been specifically charged for several years now to make sure that
>every piece of Java code out there compiles with GCJ.
>
>
Don't get me wrong. I like GCJ and the idea of compiled Java executables
but I try to look at it's potential and usefulness in a realistic way.
If Red Hat wants to tout that it's production ready, that's up to them.
I'm not a marketing guy.

GCJ currently that has limited security. It is 2 years behind mainstream
in versions (they don't have Java 5 yet and their Java 1.4 support is
not complete). It is not stable and the performance is nowhere close to
the commercial implementations. I think the GCJ team is aware of this
and I seriously doubt that it is surprise to the people at Red Hat.

Try using GCJ to run Java applets in a web browser. You can't really
since such applets cannot be trusted. I doubt the browser vendors make
attempts to prevent it though ;-)

>Regarding the security issue: Word from Andrew Haley of Red Hat is that
>it has simply been too much work to implement security up to now. This
>should not affect the judgement of the quality of GCJ, it's simply a
>missing feature.
>
>
Security is some "feature" to "simply miss". Especially if we talk about
a VM.

>Of course, I don't intend to undermine your judgement as the author
>about what you consider experimental or not, but you should expect that
>if you put your code out there, people will use it in whatever way they
>see fit, and in particular with whatever Java toolchain they see fit.
>
>
I do indeed expect that. But the PostgreSQL community cannot take
responsibility for all that may happen when people do that.

PL/Java is designed to run perfectly safe with a JVM that has the
correct features implemented. GCJ has serious issues with security and I
don't see that PL/Java, nor PostgreSQL should make any attempt to fix
them. How safe is PostgreSQL running on an unsafe operating system?

Regards,
Thomas Hallgren

On 8-Sep-05, at 2:18 AM, Thomas Hallgren wrote:

> Peter Eisentraut wrote:
>
>
>> Thomas Hallgren wrote:
>>
>>
>>> Well, yes. But use the word environment in singular please :-) To my
>>> knowledge the security is full-proof with all other VM's since they
>>> all use the standard runtime libraries.
>>>
>>>
>>
>> It's not quite as simple as that. There are a bunch of VMs and a
>> bunch of libraries (and a bunch of compilers), and they can be
>> combined in many permutations. Not all of them work with PL/Java
>> at the moment, but we should not hardcode support for just one of
>> them.
>>
>>
> AFAIK, there are only two flavors of the Java Runtime library out
> there. The one that Sun provides (and small variants of it, such as
> the ones that IBM, HP, and BEA) and the "classpath" clean-room
> implementation. All variants of the former are OK with respect to
> security and only GCJ has a working environment of the latter. In
> particular, only GCJ has a functional standards conformant Java
> Native Interface (JNI) API to the latter and PL/Java is built on JNI.
>
> Should however, someone come up with another Java environment built
> on "classpath" that has JNI support, then there will be another
> potential environment for PL/Java. TMK, there's no such environment
> and none in the making. I have serious doubts that there ever will
> be. IMO it would be perfectly safe to hard code support for a
> trusted "java".
>
Actually the apache guys are doing another one (Harmony), and there
is Kaffe. Hardly relevant to the conversation, just added for completion

>
>>> The GCJ support is as experimental as the GCJ in itself and
>>> cannot be trusted in
>>> production.
>>>
>>>
>>
>> You should not say that too loud when someone from Red Hat is
>> listening. :-) To my knowledge GCJ is Ready(tm) as of version
>> 4.0. And it's being used. Distributions such as Fedora and
>> Ubuntu will ship (or do ship?) with everything compiled using GCJ
>> to the extent possible. And there are people, in particular at or
>> near Red Hat, who have been specifically charged for several years
>> now to make sure that every piece of Java code out there compiles
>> with GCJ.
>>
>>
> Don't get me wrong. I like GCJ and the idea of compiled Java
> executables but I try to look at it's potential and usefulness in a
> realistic way. If Red Hat wants to tout that it's production ready,
> that's up to them. I'm not a marketing guy.
>
> GCJ currently that has limited security. It is 2 years behind
> mainstream in versions (they don't have Java 5 yet and their Java
> 1.4 support is not complete). It is not stable and the performance
> is nowhere close to the commercial implementations. I think the GCJ
> team is aware of this and I seriously doubt that it is surprise to
> the people at Red Hat.
>
> Try using GCJ to run Java applets in a web browser. You can't
> really since such applets cannot be trusted. I doubt the browser
> vendors make attempts to prevent it though ;-)
>
>
>> Regarding the security issue: Word from Andrew Haley of Red Hat is
>> that it has simply been too much work to implement security up to
>> now. This should not affect the judgement of the quality of GCJ,
>> it's simply a missing feature.
>>
>>
> Security is some "feature" to "simply miss". Especially if we talk
> about a VM.
>
>
>> Of course, I don't intend to undermine your judgement as the
>> author about what you consider experimental or not, but you should
>> expect that if you put your code out there, people will use it in
>> whatever way they see fit, and in particular with whatever Java
>> toolchain they see fit.
>>
>>
> I do indeed expect that. But the PostgreSQL community cannot take
> responsibility for all that may happen when people do that.
>
> PL/Java is designed to run perfectly safe with a JVM that has the
> correct features implemented. GCJ has serious issues with security
> and I don't see that PL/Java, nor PostgreSQL should make any
> attempt to fix them. How safe is PostgreSQL running on an unsafe
> operating system?
>
> Regards,
> Thomas Hallgren
>
>
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 5: don't forget to increase your free space map settings
>
>

Dave Cramer wrote:

> Actually the apache guys are doing another one (Harmony), and there
> is Kaffe. Hardly relevant to the conversation, just added for completion

I stand corrected. I forgot Kaffe. It also uses the "classpath" stuff
and have the same issues as GCJ.
Isn't Harmony is just an incubation project at Apache? To my knowledge
they have not produced anything yet and it will be years before they
have something useful.

Regards,
Thomas Hallgren

On 8-Sep-05, at 2:18 AM, Thomas Hallgren wrote:

> Peter Eisentraut wrote:
>
>
>
>> Thomas Hallgren wrote:
>>
>>
>>
>>> Well, yes. But use the word environment in singular please :-) To my
>>> knowledge the security is full-proof with all other VM's since they
>>> all use the standard runtime libraries.
>>>
>>>
>>>
>>
>> It's not quite as simple as that. There are a bunch of VMs and a
>> bunch of libraries (and a bunch of compilers), and they can be
>> combined in many permutations. Not all of them work with PL/Java
>> at the moment, but we should not hardcode support for just one of
>> them.
>>
>>
>>
> AFAIK, there are only two flavors of the Java Runtime library out
> there. The one that Sun provides (and small variants of it, such as
> the ones that IBM, HP, and BEA) and the "classpath" clean-room
> implementation. All variants of the former are OK with respect to
> security and only GCJ has a working environment of the latter. In
> particular, only GCJ has a functional standards conformant Java
> Native Interface (JNI) API to the latter and PL/Java is built on JNI.
>
> Should however, someone come up with another Java environment built
> on "classpath" that has JNI support, then there will be another
> potential environment for PL/Java. TMK, there's no such environment
> and none in the making. I have serious doubts that there ever will
> be. IMO it would be perfectly safe to hard code support for a
> trusted "java".
>
>
Actually the apache guys are doing another one (Harmony), and there
is Kaffe. Hardly relevant to the conversation, just added for completion

>
>
>>> The GCJ support is as experimental as the GCJ in itself and
>>> cannot be trusted in
>>> production.
>>>
>>>
>>>
>>
>> You should not say that too loud when someone from Red Hat is
>> listening. :-) To my knowledge GCJ is Ready(tm) as of version
>> 4.0. And it's being used. Distributions such as Fedora and
>> Ubuntu will ship (or do ship?) with everything compiled using GCJ
>> to the extent possible. And there are people, in particular at or
>> near Red Hat, who have been specifically charged for several years
>> now to make sure that every piece of Java code out there compiles
>> with GCJ.
>>
>>
>>
> Don't get me wrong. I like GCJ and the idea of compiled Java
> executables but I try to look at it's potential and usefulness in a
> realistic way. If Red Hat wants to tout that it's production ready,
> that's up to them. I'm not a marketing guy.
>
> GCJ currently that has limited security. It is 2 years behind
> mainstream in versions (they don't have Java 5 yet and their Java
> 1.4 support is not complete). It is not stable and the performance
> is nowhere close to the commercial implementations. I think the GCJ
> team is aware of this and I seriously doubt that it is surprise to
> the people at Red Hat.
>
> Try using GCJ to run Java applets in a web browser. You can't
> really since such applets cannot be trusted. I doubt the browser
> vendors make attempts to prevent it though ;-)
>
>
>
>> Regarding the security issue: Word from Andrew Haley of Red Hat is
>> that it has simply been too much work to implement security up to
>> now. This should not affect the judgement of the quality of GCJ,
>> it's simply a missing feature.
>>
>>
>>
> Security is some "feature" to "simply miss". Especially if we talk
> about a VM.
>
>
>
>> Of course, I don't intend to undermine your judgement as the
>> author about what you consider experimental or not, but you should
>> expect that if you put your code out there, people will use it in
>> whatever way they see fit, and in particular with whatever Java
>> toolchain they see fit.
>>
>>
>>
> I do indeed expect that. But the PostgreSQL community cannot take
> responsibility for all that may happen when people do that.
>
> PL/Java is designed to run perfectly safe with a JVM that has the
> correct features implemented. GCJ has serious issues with security
> and I don't see that PL/Java, nor PostgreSQL should make any
> attempt to fix them. How safe is PostgreSQL running on an unsafe
> operating system?
>
> Regards,
> Thomas Hallgren
>
>
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 5: don't forget to increase your free space map settings
>
>
>

Well don't forget sablevm, and jamvm. there's quite a few around.

Dave

On 8-Sep-05, at 9:48 AM, Thomas Hallgren wrote:

> Dave Cramer wrote:
>
>
>> Actually the apache guys are doing another one (Harmony), and
>> there is Kaffe. Hardly relevant to the conversation, just added
>> for completion
>>
>
> I stand corrected. I forgot Kaffe. It also uses the "classpath"
> stuff and have the same issues as GCJ.
> Isn't Harmony is just an incubation project at Apache? To my
> knowledge they have not produced anything yet and it will be years
> before they have something useful.
>
> Regards,
> Thomas Hallgren
>
>
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 3: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faq
>
>

Thomas Hallgren wrote:
> GCJ currently that has limited security. It is 2 years behind
> mainstream in versions (they don't have Java 5 yet and their Java 1.4
> support is not complete). It is not stable and the performance is
> nowhere close to the commercial implementations.

Frankly, that is all FUD. A lot of free software is limited or behind
or claimed to be unstable in some way, but that has never stopped
anyone from using it in the appropriate and expanding niches. Just
look at PostgreSQL. Now, if the commercial Java implementations had
acceptable licensing terms, I would consider using them, but they
don't, so I won't.

> PL/Java is designed to run perfectly safe with a JVM that has the
> correct features implemented. GCJ has serious issues with security
> and I don't see that PL/Java, nor PostgreSQL should make any attempt
> to fix them.

Well, we had a similar discussion about the time when the Python
security support was decreed nonexistent by its author. Clearly,
people still use Python, and people still use PL/Python. It's really
easy to spread a panic by claiming that GCJ has "no security". That's
clearly wrong because GCJ can be used safely in many useful situations.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> Thomas Hallgren wrote:
>> PL/Java is designed to run perfectly safe with a JVM that has the
>> correct features implemented. GCJ has serious issues with security
>> and I don't see that PL/Java, nor PostgreSQL should make any attempt
>> to fix them.

> Well, we had a similar discussion about the time when the Python
> security support was decreed nonexistent by its author. Clearly,
> people still use Python, and people still use PL/Python. It's really
> easy to spread a panic by claiming that GCJ has "no security". That's
> clearly wrong because GCJ can be used safely in many useful situations.

Actually, I've just been discussing this with Red Hat's gcj people in
connection with a different project. What they say is that the Java
security manager is completely implemented now, but what is still
missing is that it's possible to bypass Java security if you can execute
untrusted bytecode. So if I understand correctly, a gcj environment is
secure as long as you can prevent hacked-up class files from getting
into your classpath.

regards, tom lane

Peter Eisentraut wrote:

>Thomas Hallgren wrote:
>
>
>>GCJ currently that has limited security. It is 2 years behind
>>mainstream in versions (they don't have Java 5 yet and their Java 1.4
>>support is not complete). It is not stable and the performance is
>>nowhere close to the commercial implementations.
>>
>>
>
>Frankly, that is all FUD.
>
No, that's all facts. At least with the release I'm using (4.0). The
security seems to come along well though. Classpath has all the security
packages fully implemented at this point so this whole discussion is
probably obsolete by the time GCJ does their next release.

> A lot of free software is limited or behind
>or claimed to be unstable in some way, but that has never stopped
>anyone from using it in the appropriate and expanding niches.
>
That's very true. But it doesn't make my statement FUD nevertheless.

>Well, we had a similar discussion about the time when the Python
>security support was decreed nonexistent by its author. Clearly,
>people still use Python, and people still use PL/Python. It's really
>easy to spread a panic by claiming that GCJ has "no security". That's
>clearly wrong because GCJ can be used safely in many useful situations.
>
>
We where discussing a very specific situation here. Not GCJ in general.
As you pointed out yourself (and that's what started this discussion),
GCJ cannot be used for a trusted Java implementation.

Regards,
Thomas Hallgren

Thomas Hallgren wrote:
> >Frankly, that is all FUD.
>
> No, that's all facts.

Those two are not mutually exclusive.

> We where discussing a very specific situation here. Not GCJ in
> general. As you pointed out yourself (and that's what started this
> discussion), GCJ cannot be used for a trusted Java implementation.

The way I was reading your statements was that you concluded from this
sitation that GCJ should not be used at all for real work.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Tom Lane wrote:
> Actually, I've just been discussing this with Red Hat's gcj people in
> connection with a different project. What they say is that the Java
> security manager is completely implemented now, but what is still
> missing is that it's possible to bypass Java security if you can execute
> untrusted bytecode. So if I understand correctly, a gcj environment is
> secure as long as you can prevent hacked-up class files from getting
> into your classpath.
>
That's great news for PL/Java (and for Java in general of course). Did
they mention a release date?

Regards,
Thomas Hallgren

Peter Eisentraut wrote:

>The way I was reading your statements was that you concluded from this
>sitation that GCJ should not be used at all for real work.
>
>
I see your point and my way of expressing it was probably too harsh. I
agree that there are many applications where an unsafe VM can be safely
used.

Regards,
Thomas Hallgren

I wrote:
> Well, that's exactly the point Peter is arguing: he thinks (if I
> understand correctly) that the template mechanism should only be used
> for stuff that's included with the core distribution. I disagree;
> I have seldom seen any good reason for restricting mechanisms to work
> with only core components. It does seem apparent however that we need
> the templates to be reconfigurable for local conditions ... so the
> hardwired table is out, even as a one-release stopgap. I'll work on
> building a catalog tomorrow.

I've committed the changes to have a system catalog in place of the
hard-wired table. In the initial commit, I listed only the languages
included in the core distribution. It's still possible to add
additional entries for non-core languages into the default table
contents, and I'll be happy to do that for any PL authors who want them.
However, given that it is now possible for superusers to alter the
catalog contents, it's not so pressing to have a preloaded entry ---
you could instead suggest that people create an entry while installing
your language.

Were any of the people who already asked for entries persuaded by
Peter's arguments that non-core languages are better off without a
standard template entry? Let me know if you still want one.

regards, tom lane

On 8-Sep-05, at 3:45 PM, Thomas Hallgren wrote:

> Tom Lane wrote:
>
>> Actually, I've just been discussing this with Red Hat's gcj people in
>> connection with a different project. What they say is that the Java
>> security manager is completely implemented now, but what is still
>> missing is that it's possible to bypass Java security if you can
>> execute
>> untrusted bytecode. So if I understand correctly, a gcj
>> environment is
>> secure as long as you can prevent hacked-up class files from getting
>> into your classpath.
>>
Pretty tough to do, since you can read classes in your classpath, and
modify the bytecode on the fly
There's even a library to do it for you.
> That's great news for PL/Java (and for Java in general of course).
> Did they mention a release date?
>
> Regards,
> Thomas Hallgren
>
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 4: Have you searched our list archives?
>
> http://archives.postgresql.org
>
>

Thomas Hallgren <thhal(at)mailblocks(dot)com> writes:
> That's great news for PL/Java (and for Java in general of course). Did
> they mention a release date?

You can have it today if you feel like downloading Fedora Rawhide.
(Actually you might want to wait till tomorrow -- I hear rawhide is
pretty borked at the moment because someone managed to turn off the
setuid bit on "su" ;-))

I'm not sure what the schedule is for releasing Fedora 5, but you
could probably find it out from fedora.redhat.com.

regards, tom lane

"Rupa Schomaker (lists)" <pgsql-hackers(at)lists(dot)rupa(dot)com> writes:
> How are dumps/restores of local modifications to the template going to
> be handled?

pg_dump has no business trying to save or restore the template data;
if it did so then we'd just have moved the original problem of obsolete
data to a different place. See my original proposal.

regards, tom lane

On Thu, 8 Sep 2005, Tom Lane wrote:

> I've committed the changes to have a system catalog in place of the
> hard-wired table. In the initial commit, I listed only the languages
> included in the core distribution.

If I understand this correct you have created a system table that contain
the settings for some of the languages. It basicly have one row for each
language? And in this row there is for example a column that say what the
validator function is called, right? Or what do the current solution look
like?

Doesn't this make it hard for distributions to package up a language in a
rpm (or some other system) and have it just work? If the per language info
was simply stored in a file then this file can be included in a package
and you can install a new language in a simple way.

--
/Dennis Björklund

On 9/8/2005 9:20 PM, Tom Lane wrote:
> "Rupa Schomaker (lists)" <pgsql-hackers(at)lists(dot)rupa(dot)com> writes:
>
>>How are dumps/restores of local modifications to the template going to
>>be handled?
>
>
> pg_dump has no business trying to save or restore the template data;
> if it did so then we'd just have moved the original problem of obsolete
> data to a different place. See my original proposal.
>

I must be missing something.

Situation: A user installs a third-party PL that installs itself by
updating the template catalog and then doing a createlang.

Can the user simply do a dumpall/restore and get back into working
order. Or will the user have to re-run the PL install script?

Situation: A DBA changes something in the catalog as you've suggested is
possible.

Does the DBA now have to perform an extra step after a dump/restore to
get things 'back the way they were?'

The 'whats the point' was sort of implied in my original message.
Rather than say 'whats the point' I was trying to point out a potential
pitfall for end-users with this approach and suggest we should address it.

You removed the pertinent part of your statement about how this catalog
would be useful for more than just core PLs. If we're encouraging this
as a general solution then the solution should be complete.

> regards, tom lane

--
-Rupa

"Rupa Schomaker (lists)" <pgsql-hackers(at)lists(dot)rupa(dot)com> writes:
> Situation: A user installs a third-party PL that installs itself by
> updating the template catalog and then doing a createlang.
> Can the user simply do a dumpall/restore and get back into working
> order. Or will the user have to re-run the PL install script?

So? The dumpall/restore doesn't guarantee that the third-party PL
library is in place, so you still need to re-run the PL's install
script.

The point I think you are missing is that this change is an effort to
separate the existing treatment of PLs into two levels. The handler and
validator and library path properties of PLs are implementation details
that are best left outside the scope of pg_dump. pg_dump has never been
about exactly duplicating the original installation; it's about dumping
the user data in a form that can be transported to other installations
(possibly on different platforms) or to newer releases of Postgres.
Thus for example pg_dump intentionally avoids dumping and restoring any
built-in tables, functions, operators, etc. If it tried to do that
then it'd be useless for moving user data across PG versions, because
it'd wipe out whatever catalog updates the new version had made.

The basic point of the template patch is that the appropriate level of
abstraction for porting procedural languages across PG versions is
simply the language name --- the pg_dump script should say "CREATE
LANGUAGE plfoo", no more and no less. The other details such as the
library path name or the presence of a validator function are items
that can *and should* be allowed to vary across versions. So it
should be up to the new installation to provide valid values.

In the discussion yesterday, Peter provided some reasons why the valid
values might not be the same for every installation; but I continue to
think that they will be the same more often than not. On convenience
grounds it seems to me that it's best to provide built-in template
values for as many PLs as we can. When the destination installation
does have correct template values, the argument that pg_dump failed to
do it becomes moot.

regards, tom lane

Dennis Bjorklund <db(at)zigo(dot)dhs(dot)org> writes:
> Doesn't this make it hard for distributions to package up a language in a
> rpm (or some other system) and have it just work?

No, it makes it easier. Particularly when you think about altering the
implementation details of that language across successive PG versions.

> If the per language info was simply stored in a file then this file
> can be included in a package and you can install a new language in a
> simple way.

That's not any easier from a packager's viewpoint than patching an
appropriate entry into the standard contents of pg_pltemplate.

regards, tom lane