Modifying and solidifying contrib

Hello,

With all the recent discussion on contrib modules etc.. I would like to
offer the following suggestion. I am willing to do a good portion of the
work myself and I can get it done before feature freeze. I will need
help with the global make file stuff however so that is one dependency.

Add directory /modules
Modules are compiled and installed by default but not enabled.
Modules in 8.3 currently are:
pgrowlocks
pg_freespacemap (to be renameed pgfreespacemap to be consistent)
pgstattuple
pgcrypto
xml2
pgbuffercache
initagg

Requirements for /modules
Must go through normal vetting process on -hackers
Must include patches to core documentation in Docbook
Must include test cases? I don't recall if we have regress for all
contrib stuff.

Keep directory contrib
Contrib is not compiled or installed by default
Contrib in 8.3 would be:
start-scripts
pgbench (which I think really should be a foundry project)
vacuumlo (is this even required anymore?)
adminpack
btree_gist
etc...
Requirements for /contrib
Must go through normal vetting process on -hackers
Must includes README
Must include test cases? Same questions for modules

Thoughts, flames?

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

"Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> With all the recent discussion on contrib modules etc.. I would like to
> offer the following suggestion.

AFAICT you're proposing an entirely cosmetic reclassification of /contrib.
Aside from the difficulty of getting agreement on which ones should be
"in" and which "out", what does that really buy us? The thing that
would be really useful to work on is developing a concrete
representation of a "module" that pg_dump would understand, so that you
could e.g. tell it to omit btree_gist from a dump. It might be that
just segregating a contrib module into its own schema would be
sufficient, or maybe that wouldn't work well because of making people
need to deal with long search paths.

regards, tom lane

Tom Lane wrote:
> "Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
>> With all the recent discussion on contrib modules etc.. I would like to
>> offer the following suggestion.
>
> AFAICT you're proposing an entirely cosmetic reclassification of /contrib.

For the most part yes. Perception is reality and all. The exception
being that modules are installed by default.

> Aside from the difficulty of getting agreement on which ones should be
> "in" and which "out", what does that really buy us?

True and I am sure that people with more time to waste than I would like
to spend days creating a thread that is 500 responses long on why their
particular module should be a module or a contrib.

> The thing that
> would be really useful to work on is developing a concrete
> representation of a "module" that pg_dump would understand, so that you
> could e.g. tell it to omit btree_gist from a dump.

I am offering what I can.

> It might be that
> just segregating a contrib module into its own schema would be
> sufficient, or maybe that wouldn't work well because of making people
> need to deal with long search paths.

I do like the contrib schema idea would could easily be melded into this
proposal. I don't like the idea that all of contrib would automatically
be included which is one of the reasons I wanted to split this up.

Sincerely,

Joshua D. Drake

>
> regards, tom lane
>

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

On Sat, Jan 27, 2007 at 08:59:47AM -0800, Joshua D. Drake wrote:
> Hello,
>
> With all the recent discussion on contrib modules etc.. I would like to
> offer the following suggestion. I am willing to do a good portion of the
> work myself and I can get it done before feature freeze. I will need
> help with the global make file stuff however so that is one dependency.
>
> Add directory /modules

Not so great. SQL:2003 has a special meaning for the word "module."

Cheers,
D
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter

Remember to vote!

David Fetter wrote:
> On Sat, Jan 27, 2007 at 08:59:47AM -0800, Joshua D. Drake wrote:
>> Hello,
>>
>> With all the recent discussion on contrib modules etc.. I would like to
>> offer the following suggestion. I am willing to do a good portion of the
>> work myself and I can get it done before feature freeze. I will need
>> help with the global make file stuff however so that is one dependency.
>>
>> Add directory /modules
>
> Not so great. SQL:2003 has a special meaning for the word "module."

Yeah I saw mention of that in another thread, but I really didn't like
the word plugins. Do you have another thought? Extensions?

Sincerely,

Joshua D. Drake

>
> Cheers,
> D

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

On Jan 28, 2007, at 11:25 , Joshua D. Drake wrote:

> David Fetter wrote:
>> Not so great. SQL:2003 has a special meaning for the word "module."
>
> Yeah I saw mention of that in another thread, but I really didn't like
> the word plugins. Do you have another thought? Extensions?

"Extensions" would tie in nicely with its common use in the docs,
especially wrt pgxs:

PostgreSQL can be extended by the user in many ways ...
PostgreSQL also accepts escape string constants, which are an
extension to the SQL standard
To use the infrastructure for your extension ...
Here is an example that builds an extension module ...
They test standard SQL operations as well as the extended
capabilities of PostgreSQL.

The pgxs docs do use "module" as well, but as previously mentioned
module already has a particular meaning in the spec.

Michael Glaesemann
grzm seespotcode net

> PostgreSQL can be extended by the user in many ways ...
> PostgreSQL also accepts escape string constants, which are an extension
> to the SQL standard
> To use the infrastructure for your extension ...
> Here is an example that builds an extension module ...
> They test standard SQL operations as well as the extended capabilities
> of PostgreSQL.
>
> The pgxs docs do use "module" as well, but as previously mentioned
> module already has a particular meaning in the spec.

So what are we thinking here? Along with my suggestion of extensions /
contrib that we modify initdb to load an extensions schema with all
extensions into template1?

Sincerely,

Joshua D. Drake

>
> Michael Glaesemann
> grzm seespotcode net
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo(at)postgresql(dot)org so that your
> message can get through to the mailing list cleanly
>

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

"Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> So what are we thinking here? Along with my suggestion of extensions /
> contrib that we modify initdb to load an extensions schema with all
> extensions into template1?

No, I don't think so. If you do that it's effectively moving all that
stuff into core, especially if you haven't provided a way to turn it off.
I don't plan to hold still for any back-door avenues to suddenly turning
large parts of contrib into core code.

regards, tom lane

Tom Lane wrote:
> "Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
>> So what are we thinking here? Along with my suggestion of extensions /
>> contrib that we modify initdb to load an extensions schema with all
>> extensions into template1?
>
> No, I don't think so. If you do that it's effectively moving all that
> stuff into core, especially if you haven't provided a way to turn it off.

O.k. any thoughts there? What if we didn't make the extensions schema
PUBLIC? Meaning that explicits rights would have to be given for the
extensions to be used by anyone but a super user?

Obviously the initdb switch could also be selective:

initdb --enable-extensions

?

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

On Sat, Jan 27, 2007 at 09:49:25PM -0800, Joshua D. Drake wrote:
> Tom Lane wrote:
> > "Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> >> So what are we thinking here? Along with my suggestion of
> >> extensions / contrib that we modify initdb to load an extensions
> >> schema with all extensions into template1?
> >
> > No, I don't think so. If you do that it's effectively moving all
> > that stuff into core, especially if you haven't provided a way to
> > turn it off.
>
> O.k. any thoughts there? What if we didn't make the extensions
> schema PUBLIC? Meaning that explicits rights would have to be given
> for the extensions to be used by anyone but a super user?

Whether they're auto-installable or not, I'd vote for putting each one
in its own schema by default. That way, people can get an excellent
idea just by looking at what schemas exist what extensions are
installed in a given DB, and it's fairly straight-forward to remove
the thing simply by dropping the schema cascade.

> Obviously the initdb switch could also be selective:
>
> initdb --enable-extensions

If it were an initdb switch, I'd want to have something more like

--enable-extension=earthdistance

Cheers,
D
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter

Remember to vote!

On 1/27/07, Joshua D. Drake <jd(at)commandprompt(dot)com> wrote:
>
> Tom Lane wrote:
> > "Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> >> So what are we thinking here? Along with my suggestion of extensions /
> >> contrib that we modify initdb to load an extensions schema with all
> >> extensions into template1?
> >
> > No, I don't think so. If you do that it's effectively moving all that
> > stuff into core, especially if you haven't provided a way to turn it
> off.
>
> O.k. any thoughts there? What if we didn't make the extensions schema
> PUBLIC? Meaning that explicits rights would have to be given for the
> extensions to be used by anyone but a super user?
>
> Obviously the initdb switch could also be selective:
>
> initdb --enable-extensions
>
> ?
>
> Sincerely,
>
> Joshua D. Drake
>
>
>
I've been listening in on this discussion and it is closely tied to an
extension that I would like to try. The only thing holding me back is that
I am still far to green on the source code to actually propose the work I
would like to do.

It makes sense the you would want to include the contrib into the install,
but I agree with Tom this isn't something you would want in template1.
plpgsql is not even in template1 IIRC. It makes more sense to compile the
libraries and have an activation module that can be represented in a
pg_dump. Maybe the best way to do this would be to place the SQL that
modifies system catalog into a compiled library and call this as a
function. The function call could then be placed into the pg_dump if
needed.

Here is where this ties into what I am working on proposing. I have several
custom C functions in a library that I need in a restore/build. It seems
like at least a couple times every year a new cluster is built and the Admin
forgets to compile the custom functions. He builds the DB and then
"something doesn't work".

The basic idea of my proposal would be to have a function that checks for
the existence of C libraries. The next piece I would need would be a
mechanism for psql to halt the restore SQL script if these checks failed.
Finally I would need to add a switch to tell pg_dump to do the checks first
thereby causing the restore to fail, hopefully with a nice message that
clues the admin that something is wrong.

Yes all of this work so I don't get called after hours yet here I am working
on the fix after hours. :-)

It seems to me like this might be useful in incorporating an
extension/module/whatever into Postgres.

--
Kevin Barnard

"Great Beauty, great strength, and great Riches,
are really and truly of no great Use;
a right Heart exceeds all." -- Benjamin Franklin

Joshua D. Drake wrote:
> Add directory /modules

Moving stuff around in the source code probably won't matter much at all
since few users will actually look into the source code. What we need
is better presentation and accessibility of the installed items, for
example documentation and installation and upgrade handling.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Kevin Barnard wrote:
> The basic idea of my proposal would be to have a function that checks
> for the existence of C libraries. The next piece I would need would
> be a mechanism for psql to halt the restore SQL script if these
> checks failed. Finally I would need to add a switch to tell pg_dump
> to do the checks first thereby causing the restore to fail, hopefully
> with a nice message that clues the admin that something is wrong.

You already get this today. If you create a C function and the library
doesn't exist, you get an error, and if you use the right flags for
psql or pg_restore, the restoration aborts.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

David Fetter wrote:
> On Sat, Jan 27, 2007 at 09:49:25PM -0800, Joshua D. Drake wrote:
>> Tom Lane wrote:
>>> "Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
>>>> So what are we thinking here? Along with my suggestion of
>>>> extensions / contrib that we modify initdb to load an extensions
>>>> schema with all extensions into template1?
>>> No, I don't think so. If you do that it's effectively moving all
>>> that stuff into core, especially if you haven't provided a way to
>>> turn it off.
>> O.k. any thoughts there? What if we didn't make the extensions
>> schema PUBLIC? Meaning that explicits rights would have to be given
>> for the extensions to be used by anyone but a super user?
>
> Whether they're auto-installable or not, I'd vote for putting each one
> in its own schema by default. That way, people can get an excellent
> idea just by looking at what schemas exist what extensions are
> installed in a given DB, and it's fairly straight-forward to remove
> the thing simply by dropping the schema cascade.

Well to me that gets a little messy. I mean:

pg_catalog,public,<user schemas>,xml2,ltree (just to get a could
functions?) etc...

>
>> Obviously the initdb switch could also be selective:
>>
>> initdb --enable-extensions
>
> If it were an initdb switch, I'd want to have something more like
>
> --enable-extension=earthdistance

And have to parse for each extension?

Sincerely,

Joshua D. Drake

>
> Cheers,
> D

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

On Sun, Jan 28, 2007 at 10:10:14AM -0800, Joshua D. Drake wrote:
> David Fetter wrote:
> > On Sat, Jan 27, 2007 at 09:49:25PM -0800, Joshua D. Drake wrote:
> >> Tom Lane wrote:
> >>> "Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> >>>> So what are we thinking here? Along with my suggestion of
> >>>> extensions / contrib that we modify initdb to load an
> >>>> extensions schema with all extensions into template1?
> >>> No, I don't think so. If you do that it's effectively moving
> >>> all that stuff into core, especially if you haven't provided a
> >>> way to turn it off.
> >> O.k. any thoughts there? What if we didn't make the extensions
> >> schema PUBLIC? Meaning that explicits rights would have to be
> >> given for the extensions to be used by anyone but a super user?
> >
> > Whether they're auto-installable or not, I'd vote for putting each
> > one in its own schema by default. That way, people can get an
> > excellent idea just by looking at what schemas exist what
> > extensions are installed in a given DB, and it's fairly
> > straight-forward to remove the thing simply by dropping the schema
> > cascade.
>
> Well to me that gets a little messy. I mean:
>
> pg_catalog,public,<user schemas>,xml2,ltree (just to get a could
> functions?) etc...

Not as messy as trying to drop or re-create a package when there are
already 500 functions in the public schema.

> >> Obviously the initdb switch could also be selective:
> >>
> >> initdb --enable-extensions
> >
> > If it were an initdb switch, I'd want to have something more like
> >
> > --enable-extension=earthdistance
>
> And have to parse for each extension?

I don't see this as a big problem.

Cheers,
D
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter

Remember to vote!

On Sun, Jan 28, 2007 at 04:52:27PM -0500, Andrew Dunstan wrote:
>
>
> Michael Glaesemann wrote:
> >
> >On Jan 28, 2007, at 11:25 , Joshua D. Drake wrote:
> >
> >>David Fetter wrote:
> >>>Not so great. SQL:2003 has a special meaning for the word "module."
> >>
> >>Yeah I saw mention of that in another thread, but I really didn't like
> >>the word plugins. Do you have another thought? Extensions?
> >
> >"Extensions" would tie in nicely with its common use in the docs,
> >especially wrt pgxs:
>
> I don't mind this term, BUT, what we need to get across is not just that
> these are extensions, but that they are *standard* extensions, supplied
> with PostgreSQL core code and supported by the PostgreSQL core team.
> This would be analogous with, say, the standard perl modules (like
> Exporter or IO::Handle) that come with the standard perl source
> distribution. If we can get that idea across then we might lower the
> resistance of people like hosting providers to loading them.

Integrating their docs into the standard PostgreSQL SGML (or XML,
should we go there) docs would go a long, long way toward helping with
this.

Cheers,
D
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter

Remember to vote!

Michael Glaesemann wrote:
>
> On Jan 28, 2007, at 11:25 , Joshua D. Drake wrote:
>
>> David Fetter wrote:
>>> Not so great. SQL:2003 has a special meaning for the word "module."
>>
>> Yeah I saw mention of that in another thread, but I really didn't like
>> the word plugins. Do you have another thought? Extensions?
>
> "Extensions" would tie in nicely with its common use in the docs,
> especially wrt pgxs:
>
>

I don't mind this term, BUT, what we need to get across is not just that
these are extensions, but that they are *standard* extensions, supplied
with PostgreSQL core code and supported by the PostgreSQL core team.
This would be analogous with, say, the standard perl modules (like
Exporter or IO::Handle) that come with the standard perl source
distribution. If we can get that idea across then we might lower the
resistance of people like hosting providers to loading them.

cheers

andrew

>> Well to me that gets a little messy. I mean:
>>
>> pg_catalog,public,<user schemas>,xml2,ltree (just to get a could
>> functions?) etc...
>
> Not as messy as trying to drop or re-create a package when there are
> already 500 functions in the public schema.

I am not sure I understand the correlation. I am not suggesting we
install anything into public. They would all go into a single additional
schema called extensions or some such.

>>>> Obviously the initdb switch could also be selective:
>>>>
>>>> initdb --enable-extensions
>>> If it were an initdb switch, I'd want to have something more like
>>>
>>> --enable-extension=earthdistance
>> And have to parse for each extension?
>
> I don't see this as a big problem.

Well I am not really interesting in this. Someone else is welcome to try
that.

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

>> I don't mind this term, BUT, what we need to get across is not just that
>> these are extensions, but that they are *standard* extensions, supplied
>> with PostgreSQL core code and supported by the PostgreSQL core team.
>> This would be analogous with, say, the standard perl modules (like
>> Exporter or IO::Handle) that come with the standard perl source
>> distribution. If we can get that idea across then we might lower the
>> resistance of people like hosting providers to loading them.

Furthering the perception is reality rule perhaps the schema should be
called std_ext?

>
> Integrating their docs into the standard PostgreSQL SGML (or XML,
> should we go there) docs would go a long, long way toward helping with
> this.

Agreed.

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

On Sun, Jan 28, 2007 at 01:58:38PM -0800, Joshua D. Drake wrote:
>
> >> Well to me that gets a little messy. I mean:
> >>
> >> pg_catalog,public,<user schemas>,xml2,ltree (just to get a could
> >> functions?) etc...
> >
> > Not as messy as trying to drop or re-create a package when there
> > are already 500 functions in the public schema.
>
> I am not sure I understand the correlation. I am not suggesting we
> install anything into public. They would all go into a single
> additional schema called extensions or some such.

I don't think "all or nothing" is a good way to do this. 500
functions in a schema called extensions isn't much more helpful than
500 in public. There's a reason namespaces were invented long ago,
and this is classic use case for same. :)

> >>>> Obviously the initdb switch could also be selective:
> >>>>
> >>>> initdb --enable-extensions
> >>> If it were an initdb switch, I'd want to have something more
> >>> like
> >>>
> >>> --enable-extension=earthdistance
> >> And have to parse for each extension?
> >
> > I don't see this as a big problem.
>
> Well I am not really interesting in this. Someone else is welcome to
> try that.

It's really not hard, even for a C n00b like me. :)

The only trick here, and again it's not a huge one, is to modify the
postgresql.conf to have the correct default search_path.

Cheers,
D
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter

Remember to vote!

>
> I don't think "all or nothing" is a good way to do this. 500
> functions in a schema called extensions isn't much more helpful than
> 500 in public. There's a reason namespaces were invented long ago,
> and this is classic use case for same. :)

I disagree, see my post previously about initializing the extensions
schema to not be accessible initially. It would be there, it would be
loaded, but it would take a superuser to grant ability to access functions.

This allows a clean distinction between the modules while allowing their
access on a case by case basis.

>>>>> --enable-extension=earthdistance
>>>> And have to parse for each extension?
>>> I don't see this as a big problem.
>> Well I am not really interesting in this. Someone else is welcome to
>> try that.
>
> It's really not hard, even for a C n00b like me. :)

I didn't say it was hard. I said I wasn't interested :)

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

On Sun, Jan 28, 2007 at 02:14:36PM -0800, Joshua D. Drake wrote:
> > I don't think "all or nothing" is a good way to do this. 500
> > functions in a schema called extensions isn't much more helpful
> > than 500 in public. There's a reason namespaces were invented
> > long ago, and this is classic use case for same. :)
>
> I disagree, see my post previously about initializing the extensions
> schema to not be accessible initially. It would be there, it would
> be loaded, but it would take a superuser to grant ability to access
> functions.
>
> This allows a clean distinction between the modules while allowing
> their access on a case by case basis.

It's 982 functions as of this writing in CVS TIP's contrib. Do you
not get how wacky it is to have that many functions, none of which
have any collision-prevention built into their install scripts, in a
flat namespace?

Then again, you started the PL/PHP project, so maybe I shouldn't ask ;)

> >>>>> --enable-extension=earthdistance
> >>>> And have to parse for each extension?
> >>> I don't see this as a big problem.
> >> Well I am not really interesting in this. Someone else is welcome
> >> to try that.
> >
> > It's really not hard, even for a C n00b like me. :)
>
> I didn't say it was hard. I said I wasn't interested :)

I think it's necessary to get each in its own schema whether we have
an initdb flag or not.

Cheers,
D
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter

Remember to vote!

David Fetter wrote:
> I think it's necessary to get each in its own schema whether we have
> an initdb flag or not.

In any case, the initdb flag idea is about as much a nonstarter as the
configure flag idea, for the same (packaging) reasons.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

>>>>>>> --enable-extension=earthdistance
>>>>>> And have to parse for each extension?
>>>>> I don't see this as a big problem.
>>>> Well I am not really interesting in this. Someone else is welcome
>>>> to try that.
>>> It's really not hard, even for a C n00b like me. :)
>> I didn't say it was hard. I said I wasn't interested :)
>
> I think it's necessary to get each in its own schema whether we have
> an initdb flag or not.

I can see schemas for certain ones. Tsearch (just as an example as it
appears to be going into core) but ltree?

Sincerely,

Joshua D. Drake

>
> Cheers,
> D

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

Peter Eisentraut wrote:
> David Fetter wrote:
>> I think it's necessary to get each in its own schema whether we have
>> an initdb flag or not.
>
> In any case, the initdb flag idea is about as much a nonstarter as the
> configure flag idea, for the same (packaging) reasons.

I don't see your point but it makes my life simpler. So do you opinions
on the schema idea?

Sincerely,

Joshua D. Drake

>

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

David Fetter wrote:
> On Sun, Jan 28, 2007 at 02:14:36PM -0800, Joshua D. Drake wrote:
> > > I don't think "all or nothing" is a good way to do this. 500
> > > functions in a schema called extensions isn't much more helpful
> > > than 500 in public. There's a reason namespaces were invented
> > > long ago, and this is classic use case for same. :)
> >
> > I disagree, see my post previously about initializing the extensions
> > schema to not be accessible initially. It would be there, it would
> > be loaded, but it would take a superuser to grant ability to access
> > functions.
> >
> > This allows a clean distinction between the modules while allowing
> > their access on a case by case basis.
>
> It's 982 functions as of this writing in CVS TIP's contrib. Do you
> not get how wacky it is to have that many functions, none of which
> have any collision-prevention built into their install scripts, in a
> flat namespace?

We currently have 1695 standard functions. I don't see a problem with
putting the extensions all in one schema, but I also don't see the
point.

--
Bruce Momjian bruce(at)momjian(dot)us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:
> David Fetter wrote:
>
>> It's 982 functions as of this writing in CVS TIP's contrib. Do you
>> not get how wacky it is to have that many functions, none of which
>> have any collision-prevention built into their install scripts, in a
>> flat namespace?
>>
>
> We currently have 1695 standard functions. I don't see a problem with
> putting the extensions all in one schema, but I also don't see the
> point.
>
>

I certainly don't see the point. But I do see a considerable point in
having extensions use their own schemas. The fact that we have 1695
standard functions means we bear the responsibility of ensuring we don't
have name clashes among them. We should encourage extension authors by
example to use the namespace facility to relieve themselves of having to
ensure they don't clash not only with the standard functions but with
other extensions. IOW we should act with respect to the namespace for
extensions we distribute just like we would reasonably expect authors of
third party extensions to behave.

For backwards compatibility, we might be well advised also to distribute
load scripts that put extension objects in the public schema as is done
now, but this should be a deprecated practice, IMNSHO.

cheers

andrew

Keep in mind all contrib loads into public, and I don't remember any
namespace conflict issues in the past.

---------------------------------------------------------------------------

Andrew Dunstan wrote:
> Bruce Momjian wrote:
> > David Fetter wrote:
> >
> >> It's 982 functions as of this writing in CVS TIP's contrib. Do you
> >> not get how wacky it is to have that many functions, none of which
> >> have any collision-prevention built into their install scripts, in a
> >> flat namespace?
> >>
> >
> > We currently have 1695 standard functions. I don't see a problem with
> > putting the extensions all in one schema, but I also don't see the
> > point.
> >
> >
>
> I certainly don't see the point. But I do see a considerable point in
> having extensions use their own schemas. The fact that we have 1695
> standard functions means we bear the responsibility of ensuring we don't
> have name clashes among them. We should encourage extension authors by
> example to use the namespace facility to relieve themselves of having to
> ensure they don't clash not only with the standard functions but with
> other extensions. IOW we should act with respect to the namespace for
> extensions we distribute just like we would reasonably expect authors of
> third party extensions to behave.
>
> For backwards compatibility, we might be well advised also to distribute
> load scripts that put extension objects in the public schema as is done
> now, but this should be a deprecated practice, IMNSHO.
>
> cheers
>
> andrew
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faq

--
Bruce Momjian bruce(at)momjian(dot)us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian <bruce(at)momjian(dot)us> writes:
> Keep in mind all contrib loads into public, and I don't remember any
> namespace conflict issues in the past.

(A) I'm not sure we would have heard about it, and (B) any one user is
probably only using a subset of what has been proposed to be loaded by
default, so the odds of collisions would go way up.

regards, tom lane

Bruce Momjian wrote:
> Keep in mind all contrib loads into public, and I don't remember any
> namespace conflict issues in the past.
>

That is beside the point. Of course there haven't been conflicts -
precisely because a single group controls the whole lot. What I said was
that we should behave as sane third party extension authors would,
namely to use their own namespace to protect themselves from conflicts
with other unknown extensions. It's called setting a good example or
eating your own dog food.

cheers

andrew

Andrew Dunstan wrote:
> Bruce Momjian wrote:
> > Keep in mind all contrib loads into public, and I don't remember any
> > namespace conflict issues in the past.
> >
>
> That is beside the point. Of course there haven't been conflicts -
> precisely because a single group controls the whole lot. What I said was
> that we should behave as sane third party extension authors would,
> namely to use their own namespace to protect themselves from conflicts
> with other unknown extensions. It's called setting a good example or
> eating your own dog food.

The problem I see controlling per-user search_path if +10 extensions are
instlalled, and you want them always to be available by default.

Also, are we going to use per-schema permissions so that the schemas are
not visible by default? That _might_ allow us to install more by
default.

--
Bruce Momjian bruce(at)momjian(dot)us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:
> Andrew Dunstan wrote:
>
>> Bruce Momjian wrote:
>>
>>> Keep in mind all contrib loads into public, and I don't remember any
>>> namespace conflict issues in the past.
>>>
>>>
>> That is beside the point. Of course there haven't been conflicts -
>> precisely because a single group controls the whole lot. What I said was
>> that we should behave as sane third party extension authors would,
>> namely to use their own namespace to protect themselves from conflicts
>> with other unknown extensions. It's called setting a good example or
>> eating your own dog food.
>>
>
> The problem I see controlling per-user search_path if +10 extensions are
> instlalled, and you want them always to be available by default.
>

This suggests maybe we need to look at beefing up a few things. For
example, an alias facility that provided a name in schema X for things
in schema Y would help lots here. (You want everything visible? Just
alias it in public.) ISTR such things in DB2, although it's now many
years since I laid hands on it, so my memory could well be very faulty.

Also, ability to append to the search path rather than just set it could
help, as might ability to add names of non-existent schemas (which would
be ignored at run time when found not to exist).

cheers

andrew

Andrew Dunstan wrote:
> Bruce Momjian wrote:
> > Andrew Dunstan wrote:
> >
> >> Bruce Momjian wrote:
> >>
> >>> Keep in mind all contrib loads into public, and I don't remember any
> >>> namespace conflict issues in the past.
> >>>
> >>>
> >> That is beside the point. Of course there haven't been conflicts -
> >> precisely because a single group controls the whole lot. What I said was
> >> that we should behave as sane third party extension authors would,
> >> namely to use their own namespace to protect themselves from conflicts
> >> with other unknown extensions. It's called setting a good example or
> >> eating your own dog food.
> >>
> >
> > The problem I see controlling per-user search_path if +10 extensions are
> > instlalled, and you want them always to be available by default.
> >
>
> This suggests maybe we need to look at beefing up a few things. For
> example, an alias facility that provided a name in schema X for things
> in schema Y would help lots here. (You want everything visible? Just
> alias it in public.) ISTR such things in DB2, although it's now many
> years since I laid hands on it, so my memory could well be very faulty.

I think that is SYNONYM.

> Also, ability to append to the search path rather than just set it could
> help, as might ability to add names of non-existent schemas (which would
> be ignored at run time when found not to exist).

Agreed.

--
Bruce Momjian bruce(at)momjian(dot)us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Mon, Jan 29, 2007 at 04:44:44PM -0500, Andrew Dunstan wrote:
> Bruce Momjian wrote:
> >Andrew Dunstan wrote:
> >
> >>Bruce Momjian wrote:
> >>
> >>>Keep in mind all contrib loads into public, and I don't remember any
> >>>namespace conflict issues in the past.
> >>>
> >>That is beside the point. Of course there haven't been conflicts -
> >>precisely because a single group controls the whole lot. What I
> >>said was that we should behave as sane third party extension
> >>authors would, namely to use their own namespace to protect
> >>themselves from conflicts with other unknown extensions. It's
> >>called setting a good example or eating your own dog food.
> >
> >The problem I see controlling per-user search_path if +10
> >extensions are instlalled, and you want them always to be available
> >by default.
>
> This suggests maybe we need to look at beefing up a few things. For
> example, an alias facility that provided a name in schema X for
> things in schema Y would help lots here. (You want everything
> visible? Just alias it in public.) ISTR such things in DB2, although
> it's now many years since I laid hands on it, so my memory could
> well be very faulty.
>
> Also, ability to append to the search path rather than just set it
> could help, as might ability to add names of non-existent schemas
> (which would be ignored at run time when found not to exist).

You can already do this via the following (baroque, but idempotent)
method:

UPDATE
pg_catalog.pg_settings
SET
setting =
CASE WHEN 'foo' = ANY(string_to_array(setting, ','))
THEN setting
ELSE setting || ',foo'
END
WHERE
name = 'search_path'
;

Cheers,
D
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter

Remember to vote!

Joshua D. Drake wrote:
> Tom Lane wrote:
>> "Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
>>> So what are we thinking here? Along with my suggestion of extensions /
>>> contrib that we modify initdb to load an extensions schema with all
>>> extensions into template1?
>> No, I don't think so. If you do that it's effectively moving all that
>> stuff into core, especially if you haven't provided a way to turn it off.
>
> O.k. any thoughts there? What if we didn't make the extensions schema
> PUBLIC? Meaning that explicits rights would have to be given for the
> extensions to be used by anyone but a super user?
>
> Obviously the initdb switch could also be selective:

I was thinking last week about what I wanted from a packaging system...

1. Add a new column for all system objects, separate from schema:
"package". So you would have "core", "tsearch", "ltree", etc.
At the very least this lets you manually uninstall a package by
searching for and dropping by package-name. Most of what is currently
considered "PostgreSQL" would be "core" package I suppose, but you could
split out various of the types ("numeric", "text", "internet" etc.) I
suppose). Each project in contrib/ would be a package, as would each
procedural language.

2. All packages have the following attributes:
a. Name, description, version number, supplier name/website
b. Installation script, removal script (as functions - obviously
they'll need some bootstrapping)/
We might want hooks for various upgrade functions too.
c. Test function(s) with predictable names (perhaps just test001()
test002()). These might be dropp-able for a production deployment.
d. A list of dependencies on other packages/versions. This implies
some pg_package_is_installed() function perhaps.
e. A default search_path to override any user search_path settings (so
your package always calls the desired version of foo()).
f. Permissions at a package-level, so that a package can be loaded but
not installed, or installed but not usable by non-superusers.

3. All packages that ship with the standard PG distribution are:
a. built by default
b. installed to a suitable extensions/packages directory.
c. have install/remove functions
d. have some semi-standard naming (pgx_...)
e. have test functions
f. are documented in the main manuals

From my limited knowledge of the code, none of this should require
major surgery except perhaps permissions at the package level and the
bootstrapping for installation.

Bootstrapping could consist of nothing more than a set of SQL scripts
which set up some temporary tables and create and call the _install()
function.

Any of this in the direction that other people were thinking of?

--
Richard Huxton
Archonet Ltd

Richard Huxton <dev(at)archonet(dot)com> writes:
> 1. Add a new column for all system objects, separate from schema:
> "package".

Wouldn't it be a whole lot easier just to drive it off schema, rather
than inventing duplicative parallel infrastructure? That is, say that a
package has one or more schemas and what it "owns" is whatever is in
those schemas. This lets you, for example, use schema permissions to
manage access to the package.

regards, tom lane

Tom Lane wrote:
> Richard Huxton <dev(at)archonet(dot)com> writes:
>> 1. Add a new column for all system objects, separate from schema:
>> "package".
>
> Wouldn't it be a whole lot easier just to drive it off schema, rather
> than inventing duplicative parallel infrastructure? That is, say that a
> package has one or more schemas and what it "owns" is whatever is in
> those schemas. This lets you, for example, use schema permissions to
> manage access to the package.

Four differences:
1. You couldn't have a tsearch package with functions in public. At
least not without some IMPORT TSEARCH.foo() INTO public
2. You can't easily disable access to a whole package if it's spread
over multiple schemas.
3. You can't determine what package various objects belong to - is this
"stopwords" table from tsearch2 or ArchonetSearch17?
4. You can't have one package depending upon another (webstats v2.1
depends on internet_addr v2.3).

With the current search_path functionality I think it's important the
package names are separate. I think I'm right in saying there are two
packaging schemes out there - you either have a single hierarchy
(Perl/Java) or something parallel (at 90 degrees to?) to an existing
setup (RPM/deb). I think our search_path means we're dealing with
something more like a Linux packaging setup.

--
Richard Huxton
Archonet Ltd

Richard Huxton <dev(at)archonet(dot)com> writes:
> Tom Lane wrote:
>> Wouldn't it be a whole lot easier just to drive it off schema, rather
>> than inventing duplicative parallel infrastructure?

> Four differences:
> 1. You couldn't have a tsearch package with functions in public. At
> least not without some IMPORT TSEARCH.foo() INTO public

So? That's what a search path is for.

> 2. You can't easily disable access to a whole package if it's spread
> over multiple schemas.

The main reason I can see for separating a package into more than one
schema is exactly that some parts would be "public" and others
"private". So schema-level access controls are what you want, *not*
package-level.

> 3. You can't determine what package various objects belong to - is this
> "stopwords" table from tsearch2 or ArchonetSearch17?

Sure you can; you look to see what schema it's in.

> 4. You can't have one package depending upon another (webstats v2.1
> depends on internet_addr v2.3).

What's that have to do with it? Perhaps I should be clearer: I agree
with having an explicit representation of a package in some new system
catalog for that purpose. That does not translate to needing to add
package hooks to every other catalog. Indirect links through schemas
seem more than sufficient.

regards, tom lane

Tom Lane wrote:
> I agree
> with having an explicit representation of a package in some new system
> catalog for that purpose. That does not translate to needing to add
> package hooks to every other catalog. Indirect links through schemas
> seem more than sufficient.
>
>

This seems like a good first step in growing a packaging infrastructure.
I'd rather grow it organically than try to design it all up front.

cheers

andrew

>
> This seems like a good first step in growing a packaging
> infrastructure. I'd rather grow it organically than try to design it
> all up front.
>

I am in Denver and have spotty inet access so forgive me. So where does
this above leave us? What are we doing?

Joshua D. Drake

> cheers
>
> andrew
>

Joshua D. Drake wrote:
> >
> > This seems like a good first step in growing a packaging
> > infrastructure. I'd rather grow it organically than try to design it
> > all up front.
> >
>
> I am in Denver and have spotty inet access so forgive me. So where does
> this above leave us? What are we doing?

I was kind of unclear on that too. It seems we are trying to address
several issues: visibility of contrib, installation of contrib, etc.
We discussed whether we put the functions in public, a schema for all
contrib, or a schema for each contrib module, and then there was the
discussion of how to configure someone using ten /contrib modules, or at
least wanting them all to be accessible.

And then there was the idea of allowing schema permissions to control
access, so perhaps we could install more of /contrib by default, and
allow the administrator to just enable/disable them via permissions.
Personally, I think that might be the best approach because it allows us
to eliminate the install process, but doesn't make the database less
secure --- the administrator enables/disables them at runtime, or at
least could.

--
Bruce Momjian bruce(at)momjian(dot)us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:
> Joshua D. Drake wrote:
>
>>> This seems like a good first step in growing a packaging
>>> infrastructure. I'd rather grow it organically than try to design it
>>> all up front.
>>>
>>>
>> I am in Denver and have spotty inet access so forgive me. So where does
>> this above leave us? What are we doing?
>>
>
> I was kind of unclear on that too. It seems we are trying to address
> several issues: visibility of contrib, installation of contrib, etc.
> We discussed whether we put the functions in public, a schema for all
> contrib, or a schema for each contrib module, and then there was the
> discussion of how to configure someone using ten /contrib modules, or at
> least wanting them all to be accessible.
>
> And then there was the idea of allowing schema permissions to control
> access, so perhaps we could install more of /contrib by default, and
> allow the administrator to just enable/disable them via permissions.
> Personally, I think that might be the best approach because it allows us
> to eliminate the install process, but doesn't make the database less
> secure --- the administrator enables/disables them at runtime, or at
> least could.
>
>

The issues I see are:

1. the 'thing" name - the only name I have not seen some objection to is
"extension".
2. namespace - I think the consensus is tending towards one or more per
extension.
3. install/uninstall support: Tom's proposal for an extension->schema
map in the catalog will deal with that nicely, I think.
4. visibility/searchpath issues. I don't think long search paths are a
huge issue, but I think we can make life a bit easier by tweaking
searchpath support a bit (David's clever SQL notwithstanding).
5. legacy support - we need an option to load existing extensions to the
public schema as now, or support for aliases/synonyms (the latter might
be good to have regardless).
6. they all need proper docs. READMEs and the like are nowhere near good
enough.

Richard mentioned special testing requirements, but I don't see why we
can't continue to use our standard regression mechanism.

Mention has also been made of autoloading extensions with initdb. A case
could perhaps be made for doing it in createdb - maybe not every db
needs ltree, say. OTOH, if it's sitting quietly in its own schema than
it's probably not doing any harm either, so maybe initdb should just
load all the extensions it finds, and as you say make one less hoop to
make people jump through. If we do that I think at least we'd need an
option to inhibit autoloading.

cheers

andrew

On 1/30/07, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>
> [...]
> 4. visibility/searchpath issues. I don't think long search paths are a
> huge issue, but I think we can make life a bit easier by tweaking
> searchpath support a bit (David's clever SQL notwithstanding).

As for search_path -- is it really needed to change it? I think it'd be
better to leave default search_path even if we have many extensions each
sitting in its own schema. If DBA/DBD wants, he can change it himself.

The reasons to follow this way are:
1. two or more extensions might have functions with the same name
(actually, that's what schemes/namespaces serve for) => we do not know which
function should have higher priority (what order for schemas to choose?);
2. originally, when I've proposed to use separate schema name for each
contrib module I've forgotten to mention another cause to do it -- this
helps in development because everyone always knows what function is used
(the code becomes a little bit larger, but understanding and code
readability are improved) => so, it's better to not tweak search_path, it's
better to encourage DBD to use full function names (if he wants to avoid
using schema names, he can set search_path himself, resolving possible names
priority issues mentioned above).

Finally, AFAIK other DBMSs use the similar approaches (provide additional
extensions/packages/extensions/... using separate namespaces and do not try
to avoid writing namespace in function calls).

--
Best regards,
Nikolay

On Tue, Jan 30, 2007 at 03:49:14PM -0500, Andrew Dunstan wrote:
> Bruce Momjian wrote:
> >Joshua D. Drake wrote:
> >
> >>>This seems like a good first step in growing a packaging
> >>>infrastructure. I'd rather grow it organically than try to design
> >>>it all up front.
> >>>
> >>I am in Denver and have spotty inet access so forgive me. So where
> >>does this above leave us? What are we doing?
> >
> >I was kind of unclear on that too. It seems we are trying to
> >address several issues: visibility of contrib, installation of
> >contrib, etc. We discussed whether we put the functions in public,
> >a schema for all contrib, or a schema for each contrib module, and
> >then there was the discussion of how to configure someone using ten
> >/contrib modules, or at least wanting them all to be accessible.
> >
> >And then there was the idea of allowing schema permissions to
> >control access, so perhaps we could install more of /contrib by
> >default, and allow the administrator to just enable/disable them
> >via permissions. Personally, I think that might be the best
> >approach because it allows us to eliminate the install process, but
> >doesn't make the database less secure --- the administrator
> >enables/disables them at runtime, or at least could.
>
> The issues I see are:
>
> 1. the 'thing" name - the only name I have not seen some objection
> to is "extension".

+1 for name "extension."

> 2. namespace - I think the consensus is tending towards one or more per
> extension.

+1 here too. I understand that this may result in schemas that have
few functions in them. That's OK :)

> 3. install/uninstall support: Tom's proposal for an extension->schema
> map in the catalog will deal with that nicely, I think.

+1 :)

> 4. visibility/searchpath issues. I don't think long search paths are a
> huge issue, but I think we can make life a bit easier by tweaking
> searchpath support a bit (David's clever SQL notwithstanding).

The only "clever" bit I added was the CASE statement. Credit for the
rest belongs to Andrew at Supernews. It's not a bad thing for people
to keep around, either way. :)

> 5. legacy support - we need an option to load existing extensions to the
> public schema as now, or support for aliases/synonyms (the latter might
> be good to have regardless).

Hrm. This gets tricky. When things are mandated to be in their own
namespace, they need not check what everybody else's things are doing
each time, whereas when they go into the public schema... :P

> 6. they all need proper docs. READMEs and the like are nowhere near good
> enough.

Agreed. I'm thinking a new major section in the SGML docs is in order
with a subsection for each contrib/ piece underneath.

> Richard mentioned special testing requirements, but I don't see why we
> can't continue to use our standard regression mechanism.

A subdirectory in src/tests/regression for each one?

> Mention has also been made of autoloading extensions with initdb. A case
> could perhaps be made for doing it in createdb - maybe not every db
> needs ltree, say. OTOH, if it's sitting quietly in its own schema than
> it's probably not doing any harm either, so maybe initdb should just
> load all the extensions it finds, and as you say make one less hoop to
> make people jump through. If we do that I think at least we'd need an
> option to inhibit autoloading.

I don't think it would be too much trouble to do extensions the way we
now do tables and schemas in pg_dump, i.e. with multiple possible
regular expression entries like

--include-extension=

and

--exclude-extension=

where the includes get evaluated before the excludes.

Just my $.02 :)

Cheers,
D
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter

Remember to vote!

On Jan 31, 2007, at 12:42 , David Fetter wrote:

> On Tue, Jan 30, 2007 at 03:49:14PM -0500, Andrew Dunstan wrote:
>
>> 6. they all need proper docs. READMEs and the like are nowhere
>> near good
>> enough.
>
> Agreed. I'm thinking a new major section in the SGML docs is in order
> with a subsection for each contrib/ piece underneath.

I agree re: new section. Are you thinking that all contrib docs would
be built automatically, even if the individual extensions (neé
contrib modules?) aren't installed? I think that would definitely
raise awareness of the extensions that are available.

I'd also like to see being able to add docs for non-core extensions
(e.g., ip4r) to the main documentation. Not sure what that would
involve: rebuilding the tocs and index, besides the new pages
themselves? Or perhaps just a rebuild of the complete docs? I haven't
had docs building on a local system for a couple of years, so I'm not
it a position currently to play around with this, but it's something
I'd love to learn how to do.

Michael Glaesemann
grzm seespotcode net

David Fetter wrote:
> On Tue, Jan 30, 2007 at 03:49:14PM -0500, Andrew Dunstan wrote:
>
>> 4. visibility/searchpath issues. I don't think long search paths are a
>> huge issue, but I think we can make life a bit easier by tweaking
>> searchpath support a bit (David's clever SQL notwithstanding).
>>
>
> The only "clever" bit I added was the CASE statement. Credit for the
> rest belongs to Andrew at Supernews. It's not a bad thing for people
> to keep around, either way. :)
>

I dislike on principle things that mangle the catalogs directly. As soon
as I see one I think "why aren't we providing an SQL command for that?"
By and large, I think users should be able to work as though the catalog
were not visible, or at least not directly writable.

>> 5. legacy support - we need an option to load existing extensions to the
>> public schema as now, or support for aliases/synonyms (the latter might
>> be good to have regardless).
>>
>
> Hrm. This gets tricky. When things are mandated to be in their own
> namespace, they need not check what everybody else's things are doing
> each time, whereas when they go into the public schema... :P
>

Why is it tricky? This is for legacy only, i.e. for object we know of
today. Any future objects in existing extensions, or objects in new
extensions, should not have this support - they should use their own
namespaces, pure and simple.

>> Richard mentioned special testing requirements, but I don't see why we
>> can't continue to use our standard regression mechanism.
>>
>
> A subdirectory in src/tests/regression for each one?
>

No. One of the reasons for us to maintain some standard extensions is to
act as exemplars. You should be able to build, install and test an
extension without having a complete source tree present. So each
extension should keep its own sql and expected directory as now, I think.

> I don't think it would be too much trouble to do extensions the way we
> now do tables and schemas in pg_dump, i.e. with multiple possible
> regular expression entries like
>
> --include-extension=
>
> and
>
> --exclude-extension=
>
> where the includes get evaluated before the excludes.
>
>

OK, as long as --exclude-extension has an "all" option, or we have a
--no-extensions option also.

cheers

andrew

On Wed, Jan 31, 2007 at 09:31:00AM -0500, Andrew Dunstan wrote:
> David Fetter wrote:
> >On Tue, Jan 30, 2007 at 03:49:14PM -0500, Andrew Dunstan wrote:
> >>4. visibility/searchpath issues. I don't think long search paths
> >>are a huge issue, but I think we can make life a bit easier by
> >>tweaking searchpath support a bit (David's clever SQL
> >>notwithstanding).
> >
> >The only "clever" bit I added was the CASE statement. Credit for
> >the rest belongs to Andrew at Supernews. It's not a bad thing for
> >people to keep around, either way. :)
>
> I dislike on principle things that mangle the catalogs directly. As
> soon as I see one I think "why aren't we providing an SQL command
> for that?" By and large, I think users should be able to work as
> though the catalog were not visible, or at least not directly
> writable.

So are you proposing user-visible functions in pg_catalog like the
following?

append_to_search_path(role NAME, database NAME, paths NAME[])
prepend_to_search_path(role NAME, database NAME, paths NAME[])
remove_from_search_path(role NAME, database NAME, paths NAME[])

The above is how I'm picturing how this would fit in with the TODO of
allowing things to be set on a per-role-and-database basis. There
could be two-argument short-cuts of each of those which would do the
above for the current user.

> >>5. legacy support - we need an option to load existing extensions
> >>to the public schema as now, or support for aliases/synonyms (the
> >>latter might be good to have regardless).
> >
> >Hrm. This gets tricky. When things are mandated to be in their
> >own namespace, they need not check what everybody else's things are
> >doing each time, whereas when they go into the public schema... :P
>
> Why is it tricky? This is for legacy only, i.e. for object we know
> of today. Any future objects in existing extensions, or objects in
> new extensions, should not have this support - they should use their
> own namespaces, pure and simple.

OK

> >>Richard mentioned special testing requirements, but I don't see
> >>why we can't continue to use our standard regression mechanism.
> >
> >A subdirectory in src/tests/regression for each one?
>
> No. One of the reasons for us to maintain some standard extensions
> is to act as exemplars. You should be able to build, install and
> test an extension without having a complete source tree present. So
> each extension should keep its own sql and expected directory as
> now, I think.

Right :)

> >I don't think it would be too much trouble to do extensions the way we
> >now do tables and schemas in pg_dump, i.e. with multiple possible
> >regular expression entries like
> >
> >--include-extension=
> >
> >and
> >
> >--exclude-extension=
> >
> >where the includes get evaluated before the excludes.
>
> OK, as long as --exclude-extension has an "all" option, or we have a
> --no-extensions option also.

While we're at it, both cases should be straight-forward to do.

Cheers,
D
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter

Remember to vote!

There was also mention of having a means to tell pg_dump not to dump
extensions...

On Jan 30, 2007, at 2:49 PM, Andrew Dunstan wrote:

> Bruce Momjian wrote:
>> Joshua D. Drake wrote:
>>
>>>> This seems like a good first step in growing a packaging
>>>> infrastructure. I'd rather grow it organically than try to
>>>> design it all up front.
>>>>
>>>>
>>> I am in Denver and have spotty inet access so forgive me. So
>>> where does this above leave us? What are we doing?
>>>
>>
>> I was kind of unclear on that too. It seems we are trying to address
>> several issues: visibility of contrib, installation of contrib, etc.
>> We discussed whether we put the functions in public, a schema for all
>> contrib, or a schema for each contrib module, and then there was the
>> discussion of how to configure someone using ten /contrib modules,
>> or at
>> least wanting them all to be accessible.
>> And then there was the idea of allowing schema permissions to control
>> access, so perhaps we could install more of /contrib by default, and
>> allow the administrator to just enable/disable them via
>> permissions. Personally, I think that might be the best approach
>> because it allows us
>> to eliminate the install process, but doesn't make the database less
>> secure --- the administrator enables/disables them at runtime, or at
>> least could.
>>
>>
>
> The issues I see are:
>
> 1. the 'thing" name - the only name I have not seen some objection
> to is "extension".
> 2. namespace - I think the consensus is tending towards one or more
> per extension.
> 3. install/uninstall support: Tom's proposal for an extension-
> >schema map in the catalog will deal with that nicely, I think.
> 4. visibility/searchpath issues. I don't think long search paths
> are a huge issue, but I think we can make life a bit easier by
> tweaking searchpath support a bit (David's clever SQL
> notwithstanding).
> 5. legacy support - we need an option to load existing extensions
> to the public schema as now, or support for aliases/synonyms (the
> latter might be good to have regardless).
> 6. they all need proper docs. READMEs and the like are nowhere near
> good enough.
>
> Richard mentioned special testing requirements, but I don't see why
> we can't continue to use our standard regression mechanism.
>
> Mention has also been made of autoloading extensions with initdb. A
> case could perhaps be made for doing it in createdb - maybe not
> every db needs ltree, say. OTOH, if it's sitting quietly in its own
> schema than it's probably not doing any harm either, so maybe
> initdb should just load all the extensions it finds, and as you say
> make one less hoop to make people jump through. If we do that I
> think at least we'd need an option to inhibit autoloading.
>
> cheers
>
> andrew
>
>
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 4: Have you searched our list archives?
>
> http://archives.postgresql.org
>

--
Jim Nasby jim(at)nasby(dot)net
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)

Jim Nasby wrote:
> There was also mention of having a means to tell pg_dump not to dump
> extensions...
>

What's the use case for that? What will we do if there are db objects
that depend on some extensions? Given that there will be some uninstall
support, this one seems less necessary.

I really think we should approach this by not trying to do everything at
once.

cheers

andrew

On Mon, Feb 05, 2007 at 12:19:51PM -0500, Andrew Dunstan wrote:
> Jim Nasby wrote:
> >There was also mention of having a means to tell pg_dump not to dump
> >extensions...
>
> What's the use case for that? What will we do if there are db objects
> that depend on some extensions? Given that there will be some uninstall
> support, this one seems less necessary.

Well, the use case is someone using tsearch2 on version A and wants to
a do a dump to restore into later version B. It would be helpful if
pg_dump compacted the whole tsearch2 infrastrcutre into a single
"INSTALL tsearch2" command. Obviously, the tsearch2 uninstall script
for version B isn't going to work properly for a database restore from
version A. And this way a dump/restore will pickup any new features
added in the later version.

> I really think we should approach this by not trying to do everything at
> once.

That's true, but it's something to keep in mind.

Have a nice day,
--
Martijn van Oosterhout <kleptog(at)svana(dot)org> http://svana.org/kleptog/
> From each according to his ability. To each according to his ability to litigate.

Martijn van Oosterhout wrote:
> On Mon, Feb 05, 2007 at 12:19:51PM -0500, Andrew Dunstan wrote:
>
>> Jim Nasby wrote:
>>
>>> There was also mention of having a means to tell pg_dump not to dump
>>> extensions...
>>>
>> What's the use case for that? What will we do if there are db objects
>> that depend on some extensions? Given that there will be some uninstall
>> support, this one seems less necessary.
>>
>
> Well, the use case is someone using tsearch2 on version A and wants to
> a do a dump to restore into later version B. It would be helpful if
> pg_dump compacted the whole tsearch2 infrastrcutre into a single
> "INSTALL tsearch2" command. Obviously, the tsearch2 uninstall script
> for version B isn't going to work properly for a database restore from
> version A. And this way a dump/restore will pickup any new features
> added in the later version.
>
>
And if there's an API change everything will blow up.

I would suggest we start with what is (I think) simplest and clearest:

. catalog support via a simple extension->schema(s) map
. initdb installs standard extensions if it finds them, unless told not to
. support for adjusting search path.

If that gets done nicely for 8.3 we'll be doing well.

cheers

andrew

On 2/5/07, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
[...]
> I would suggest we start with what is (I think) simplest and clearest:
>
> . catalog support via a simple extension->schema(s) map
> . initdb installs standard extensions if it finds them, unless told not to
> . support for adjusting search path.

Why adjusting search_path is needed at all?

--
Best regards,
Nikolay

On Mon, Jan 29, 2007 at 04:22:43PM -0500, Tom Lane wrote:
>
> (A) I'm not sure we would have heard about it, and (B) any one user is
> probably only using a subset of what has been proposed to be loaded by
> default, so the odds of collisions would go way up.

As a data point, some time ago (7.2 days) I used to do this as a
matter of completeness, and never had a collision.

A

--
Andrew Sullivan | ajs(at)crankycanuck(dot)ca
If they don't do anything, we don't need their acronym.
--Josh Hamilton, on the US FEMA

Andrew Sullivan wrote:
> On Mon, Jan 29, 2007 at 04:22:43PM -0500, Tom Lane wrote:
>
>> (A) I'm not sure we would have heard about it, and (B) any one user is
>> probably only using a subset of what has been proposed to be loaded by
>> default, so the odds of collisions would go way up.
>>
>
> As a data point, some time ago (7.2 days) I used to do this as a
> matter of completeness, and never had a collision.
>
>

(bangs head against brick wall)

of course there isn't a collision.

The point I at least have been trying to make is that extensions
generally (e.g. from pgfoundry) should protect themselves from possible
collisions with core and other unknown extensions that might be loaded,
by using unique namespace(s), and further, that the standard extensions
(i.e. what we now load from contrib) should act as good exemplars by
doing likewise, with some support given for legacy uses that expect them
to use the public schema.

cheers

andrew

On Tue, Feb 06, 2007 at 11:43:24AM -0500, Andrew Dunstan wrote:
> Andrew Sullivan wrote:
> >
> >As a data point, some time ago (7.2 days) I used to do this as a
> >matter of completeness, and never had a collision.
>
> The point I at least have been trying to make is that extensions
> generally (e.g. from pgfoundry) should protect themselves from possible
> collisions with core and other unknown extensions that might be loaded,
> by using unique namespace(s), and further, that the standard extensions

This wasn't a disagreement with your general point. I was just
trying to say that the problem did not ine fact exist at some point,
so the empirical rathole perhaps doesn't need to be explored. The
style question is the only one that is relevant, I think. (I happen
to agree with you on that, and it seems to me that a more complete
proposal for namespace guidelines might be nice.)

A

--
Andrew Sullivan | ajs(at)crankycanuck(dot)ca
I remember when computers were frustrating because they *did* exactly what
you told them to. That actually seems sort of quaint now.
--J.D. Baldwin

Nikolay Samokhvalov wrote:
> On 2/5/07, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
> [...]
>> I would suggest we start with what is (I think) simplest and clearest:
>>
>> . catalog support via a simple extension->schema(s) map
>> . initdb installs standard extensions if it finds them, unless told
>> not to
>> . support for adjusting search path.
>
> Why adjusting search_path is needed at all?
>

If the extension installs everything in dedicated namespace(s), I think
we would want to have an option to add those namespaces easily to search
paths. Right now all we can do is to set a search path. It would be
nice, for example, to have support for appending or prepending something
to the search path.

I suspect most apps/extensions don't currently use namespaces much, or
we might well have seen more demand in this area.

cheers

andrew

On Feb 5, 2007, at 11:19 AM, Andrew Dunstan wrote:
> Jim Nasby wrote:
>> There was also mention of having a means to tell pg_dump not to
>> dump extensions...
> What's the use case for that? What will we do if there are db
> objects that depend on some extensions? Given that there will be
> some uninstall support, this one seems less necessary.

In addition to Martijn's tsearch case, there's also PostGIS. And I
believe this is a pretty big pain for them.

> I really think we should approach this by not trying to do
> everything at once.

That's fine; I just wanted to point out the use case while lists were
being made.
--
Jim Nasby jim(at)nasby(dot)net
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)

On 2/6/07, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
> If the extension installs everything in dedicated namespace(s), I think
> we would want to have an option to add those namespaces easily to search
> paths. Right now all we can do is to set a search path. It would be
> nice, for example, to have support for appending or prepending something
> to the search path.
>
> I suspect most apps/extensions don't currently use namespaces much, or
> we might well have seen more demand in this area.

I still do not understand why is it so needed.
Your argument is "some apps aren't able to call functions as
schemaname.functionname(arg1, arg2, ..)", right? First of all, I do
not think that the number of such apps is huge. Second, this is really
the problem of those apps themselves.

I still think that separate namespaces for extensions is a good idea
while adjusting search_path is not. I've explained my POV in details
several messages ago in this thread...

Separation of extensions with fully specified names
"schemaname.functionname(...)" is good improvement (for simplification
and clarity) and while adjusting search_path should be DBA/DBD's
decision.
--
Best regards,
Nikolay

On 2/7/07, Nikolay Samokhvalov <samokhvalov(at)gmail(dot)com> wrote:
> I still think that separate namespaces for extensions is a good idea
> while adjusting search_path is not. I've explained my POV in details
> several messages ago in this thread...
>
> Separation of extensions with fully specified names
> "schemaname.functionname(...)" is good improvement (for simplification
> and clarity) and while adjusting search_path should be DBA/DBD's
> decision.

Oh, I've just recalled the problem that could arise in this scenario...
We cannot use schema name as prefix for operator calling
("tsearch2.ts_debug(...)" works, while "... tsearch2.@@ ..." doesn't).
This is one specific issue, maybe it's worth to resolve it? Or it's
impossible for some reasons...

--
Best regards,
Nikolay

Nikolay Samokhvalov wrote:
>
> I still do not understand why is it so needed.
> Your argument is "some apps aren't able to call functions as
> schemaname.functionname(arg1, arg2, ..)", right?

wrong.

>
> I still think that separate namespaces for extensions is a good idea
> while adjusting search_path is not. I've explained my POV in details
> several messages ago in this thread...

The difference between us is that I am less inclined to be prescriptive
about such matters than you are. I think that as namespace use expands
we should also probably provide better support for adding things to the
search path (or indeed taking things away). If you don't want to use it
then don't, but I don't see why you are so insistent on denying such
facilities to others.

cheers

andrew

On 2/7/07, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
> Nikolay Samokhvalov wrote:
> >
> > I still do not understand why is it so needed.
> > Your argument is "some apps aren't able to call functions as
> > schemaname.functionname(arg1, arg2, ..)", right?
>
> wrong.
>
> >
> > I still think that separate namespaces for extensions is a good idea
> > while adjusting search_path is not. I've explained my POV in details
> > several messages ago in this thread...
>
>
> The difference between us is that I am less inclined to be prescriptive
> about such matters than you are. I think that as namespace use expands
> we should also probably provide better support for adding things to the
> search path (or indeed taking things away). If you don't want to use it
> then don't, but I don't see why you are so insistent on denying such
> facilities to others.

ok, looks like I've misunderstood your mesages. Sorry for that. Surely
additional capabilities for manipulation with search_path cannot
hinder anybody.

--
Best regards,
Nikolay

"Nikolay Samokhvalov" <samokhvalov(at)gmail(dot)com> writes:
> We cannot use schema name as prefix for operator calling

You can, but it's kind of ugly:

regression=# SELECT 2 OPERATOR(pg_catalog.+) 2;
?column?
----------
4
(1 row)

regards, tom lane

Jim Nasby wrote:

> In addition to Martijn's tsearch case, there's also PostGIS. And I
> believe this is a pretty big pain for them.

Hear hear! It would be nice to dump from an old PostgreSQL/PostGIS
combination and restore to a new version combination, without taking all
the function definitions along for a ride in the dump process. What we
really want is just the data.

--

Paul Ramsey
Refractions Research
http://www.refractions.net
pramsey(at)refractions(dot)net
Phone: 250-383-3022
Cell: 250-885-0632

Are we doing this?

---------------------------------------------------------------------------

Joshua D. Drake wrote:
> Hello,
>
> With all the recent discussion on contrib modules etc.. I would like to
> offer the following suggestion. I am willing to do a good portion of the
> work myself and I can get it done before feature freeze. I will need
> help with the global make file stuff however so that is one dependency.
>
> Add directory /modules
> Modules are compiled and installed by default but not enabled.
> Modules in 8.3 currently are:
> pgrowlocks
> pg_freespacemap (to be renameed pgfreespacemap to be consistent)
> pgstattuple
> pgcrypto
> xml2
> pgbuffercache
> initagg
>
> Requirements for /modules
> Must go through normal vetting process on -hackers
> Must include patches to core documentation in Docbook
> Must include test cases? I don't recall if we have regress for all
> contrib stuff.
>
> Keep directory contrib
> Contrib is not compiled or installed by default
> Contrib in 8.3 would be:
> start-scripts
> pgbench (which I think really should be a foundry project)
> vacuumlo (is this even required anymore?)
> adminpack
> btree_gist
> etc...
> Requirements for /contrib
> Must go through normal vetting process on -hackers
> Must includes README
> Must include test cases? Same questions for modules
>
>
> Thoughts, flames?
>
> Sincerely,
>
> Joshua D. Drake
>
>
>
>
>
>
> --
>
> === The PostgreSQL Company: Command Prompt, Inc. ===
> Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
> Providing the most comprehensive PostgreSQL solutions since 1997
> http://www.commandprompt.com/
>
> Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
> PostgreSQL Replication: http://www.commandprompt.com/products/
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 7: You can help support the PostgreSQL project by donating at
>
> http://www.postgresql.org/about/donate

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +