Core Extensions relocation

Following up on the idea we've been exploring for making some extensions
more prominent, attached is the first rev that I think may be worth
considering seriously. Main improvement from the last is that I
reorganized the docs to break out what I decided to tentatively name
"Core Extensions" into their own chapter. No longer mixed in with the
rest of the contrib modules, and I introduce them a bit differently.
If you want to take a quick look at the new page, I copied it to
http://www.2ndquadrant.us/docs/html/extensions.html

I'm not completely happy on the wordering there yet. The use of both
"modules" and "extensions" is probably worth eliminating, and maybe that
continues on to doing that against the language I swiped from the
contrib intro too. There's also a lot of shared text at the end there,
common wording from that and the contrib page about how to install and
migrate these extensions. Not sure how to refactor it out into another
section cleanly though.

Regression tests came up last time I posted this. Doesn't look like
there are any for the modules I'm suggesting should be promoted. Only
code issue I noticed during another self-review here is that I didn't
rename contrib/pgrowlocks/pgrowlocks--unpackaged--1.0.sql cleanly, may
need to do that one over again to get the commits as clean as possible.

Updated code is at
https://github.com/greg2ndQuadrant/postgres/tree/move-contrib too, and
since this is painful as a patch the compare view at
https://github.com/greg2ndQuadrant/postgres/compare/master...move-contrib
will be easier for browsing the code changes.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

Hello Greg, hello All,

This is my first post at Hackers, so sorry if I am been a noob here, but I
am pretty confused about
how to create the extension pg_buffercache.

First of all, I was trying to create using the old method by calling the
pg_buffercache--1.0.sql directly.
Then I discover the change that occurs recently to use CREATE EXTENSION, but
even now I am getting the weird error:

# select * from pg_available_extensions;
name | default_version | installed_version | comment
----------------+-----------------+-------------------+---------------------------------
plpgsql | 1.0 | 1.0 | PL/pgSQL
procedural language
pg_buffercache | 1.0 | | examine the
shared buffer cache
(2 rows)

postgres=# CREATE EXTENSION pg_buffercache SCHEMA pg_catalog;
ERROR: syntax error at or near "NO"

Right now, talking with some fellows at #postgresql they tell that the error
is NOT occurring for they.
This was about 9.1beta from git.

But even so, I need to ask, because my production is on another versions:

What is the right way to install this contrib at 9.0.1, 9.0.2 and 9.0.4 ?

Many thanks,

Best regards,
vinnix

On Thu, Jun 9, 2011 at 1:14 AM, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:

> Following up on the idea we've been exploring for making some extensions
> more prominent, attached is the first rev that I think may be worth
> considering seriously. Main improvement from the last is that I reorganized
> the docs to break out what I decided to tentatively name "Core Extensions"
> into their own chapter. No longer mixed in with the rest of the contrib
> modules, and I introduce them a bit differently. If you want to take a
> quick look at the new page, I copied it to
> http://www.2ndquadrant.us/docs/html/extensions.html
>
> I'm not completely happy on the wordering there yet. The use of both
> "modules" and "extensions" is probably worth eliminating, and maybe that
> continues on to doing that against the language I swiped from the contrib
> intro too. There's also a lot of shared text at the end there, common
> wording from that and the contrib page about how to install and migrate
> these extensions. Not sure how to refactor it out into another section
> cleanly though.
>
> Regression tests came up last time I posted this. Doesn't look like there
> are any for the modules I'm suggesting should be promoted. Only code issue
> I noticed during another self-review here is that I didn't rename
> contrib/pgrowlocks/pgrowlocks--unpackaged--1.0.sql cleanly, may need to do
> that one over again to get the commits as clean as possible.
>
> Updated code is at
> https://github.com/greg2ndQuadrant/postgres/tree/move-contrib too, and
> since this is painful as a patch the compare view at
> https://github.com/greg2ndQuadrant/postgres/compare/master...move-contribwill be easier for browsing the code changes.
>
> --
> Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
> PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us
>
>
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>
>

--

Vinícius Abrahão Bazana Schmidt
Desenvolvimento
Dextra Sistemas
www.dextra.com.br
+55 19 3256-6722 Ramal 246

Este email é confidencial. Mais informações em:
This message is confidential. More information at:
www.dextra.com.br/confidencial.htm

--
vi[nnix]™
aka: Vinícius Abrahão Bazana Schmidt
vischmidt.wordpress.com
twitter.com/vischmidt

Vinicius Abrahao wrote:
> This is my first post at Hackers, so sorry if I am been a noob here,
> but I am pretty confused about
> how to create the extension pg_buffercache.

This list is for talking about development of new features, normally on
the latest development version of the software (right now 9.1). There
is no such thing as CREATE EXTENSION in versions before that. A
question like "how do I install pg_buffercache for 9.0?" should normally
get sent to one of the other mailing lists; any of pgsql-performance,
pgsql-admin, or pgsql-general would be appropriate to ask that at. This
one really isn't. It's also better to avoid taking someone else's
discussion and replying to it with your questions.

> But even so, I need to ask, because my production is on another versions:
> What is the right way to install this contrib at 9.0.1, 9.0.2 and 9.0.4 ?

But since I happen to know this answer, here's an example from a RedHat
derived Linux system running PostgreSQL 9.0.4, logged in as the postgres
user:

-bash-3.2$ locate pg_buffercache.sql
/usr/pgsql-9.0/share/contrib/pg_buffercache.sql
/usr/pgsql-9.0/share/contrib/uninstall_pg_buffercache.sql
-bash-3.2$ psql -d pgbench -f
/usr/pgsql-9.0/share/contrib/pg_buffercache.sql
SET
CREATE FUNCTION
CREATE VIEW
REVOKE
REVOKE
-bash-3.2$ psql -d pgbench -c "select count(*) from pg_buffercache"
count
-------
4096

The location of the file will be different on other platforms, but
that's the basic idea of how you install it.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

Please do not piggyback on an unrelated thread to ask a question.
Start a new thread.

Vinicius Abrahao <vinnix(dot)bsd(at)gmail(dot)com> writes:
> postgres=# CREATE EXTENSION pg_buffercache SCHEMA pg_catalog;
> ERROR: syntax error at or near "NO"

This looks like a syntax error in the pg_buffercache--1.0.sql file ...
have you tampered with that at all?

I believe BTW that you cannot specify pg_catalog as the target schema
here. When I try that, I get:

regression=# CREATE EXTENSION pg_buffercache SCHEMA pg_catalog;
ERROR: permission denied to create "pg_catalog.pg_buffercache"
DETAIL: System catalog modifications are currently disallowed.

but it goes through fine without the SCHEMA clause.

regards, tom lane

Hi,

Greg Smith <greg(at)2ndquadrant(dot)com> writes:
> Following up on the idea we've been exploring for making some extensions
> more prominent, attached is the first rev that I think may be worth
> considering seriously. Main improvement from the last is that I reorganized
> the docs to break out what I decided to tentatively name "Core Extensions"
> into their own chapter. No longer mixed in with the rest of the contrib
> modules, and I introduce them a bit differently. If you want to take a
> quick look at the new page, I copied it to
> http://www.2ndquadrant.us/docs/html/extensions.html

Thanks a lot for working on this!

I have two remarks here. First, I think that the “core extensions” (+1
for this naming) should not be found in a documentation appendix, but in
the main documentation, as a new Chapter in Part II where we list data
types and operators and system functions. Between current chapters 9
and 10 would be my vote.

Then, I think the angle to use to present “core extensions” would be
that those are things maintained like the core server, but that you
might not need at all. There's no SQL level feature that rely on them,
it's all “extra”, but it's there nonetheless, because you might need it.

Other than that, +1 for your patch. I'd stress out that I support your
idea to split the extensions at the source level, I think that's really
helping to get the message out: that is trustworthy and maintained code.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

On tor, 2011-06-09 at 00:14 -0400, Greg Smith wrote:
> Following up on the idea we've been exploring for making some
> extensions
> more prominent, attached is the first rev that I think may be worth
> considering seriously. Main improvement from the last is that I
> reorganized the docs to break out what I decided to tentatively name
> "Core Extensions" into their own chapter. No longer mixed in with
> the
> rest of the contrib modules, and I introduce them a bit
> differently.

For the directory name, I'd prefer either src/extensions (since there is
more than one), or if you want to go for short somehow, src/ext. (Hmm,
I guess the installation subdirectory is also called "extension". But
it felt wrong on first reading anyway.)

There is some funny business in your new src/extension/Makefile. You
apparently based this on a very old version of contrib/Makefile (if
still contains a CVS keyword header), it uses for loops in make targets
after we just got rid of them, and it references some modules that
aren't there at all. That file needs a complete redo based on current
sources, I think.

Equally, your new extension-global.mk sets MODULEDIR, which is no longer
necessary, and has a CVS header. What version did you branch this
off? :)

Perhaps a small addition to the installation instructions would also be
appropriate, to tell people that certain core extensions, as it were,
are installed by default.

Peter Eisentraut wrote:
> For the directory name, I'd prefer either src/extensions (since there is
> more than one), or if you want to go for short somehow, src/ext. (Hmm,
> I guess the installation subdirectory is also called "extension". But
> it felt wrong on first reading anyway.)
>

I jumped between those two a couple of times myself, settling on
"extension" to match the installation location as you figured out.
Assuming that name shouldn't change at this point, this seemed the best
way to name the new directory, even though I agree it seems weird at first.

> What version did you branch this off? :)
>

Long enough ago that apparently I've missed some major changes; Magnus
already pointed out I needed to revisit how MODULEDIR was used. Looks
like I need to rebuild the first patch in this series yet again, which
shouldn't be too bad. The second time I did that, I made the commits
atomic enough that the inevitable third one would be easy.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

On Sat, Jun 11, 2011 at 12:38 PM, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:
> Peter Eisentraut wrote:
>> For the directory name, I'd prefer either src/extensions (since there is
>> more than one), or if you want to go for short somehow, src/ext.  (Hmm,
>> I guess the installation subdirectory is also called "extension".  But
>> it felt wrong on first reading anyway.)
>
> I jumped between those two a couple of times myself, settling on "extension"
> to match the installation location as you figured out.  Assuming that name
> shouldn't change at this point, this seemed the best way to name the new
> directory, even though I agree it seems weird at first.
>
>> What version did you branch this off? :)
>
> Long enough ago that apparently I've missed some major changes; Magnus
> already pointed out I needed to revisit how MODULEDIR was used.  Looks like
> I need to rebuild the first patch in this series yet again, which shouldn't
> be too bad.  The second time I did that, I made the commits atomic enough
> that the inevitable third one would be easy.

Are you going to do this work for this CommitFest?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Is this going to be done for 9.2?

---------------------------------------------------------------------------

Greg Smith wrote:
> Following up on the idea we've been exploring for making some extensions
> more prominent, attached is the first rev that I think may be worth
> considering seriously. Main improvement from the last is that I
> reorganized the docs to break out what I decided to tentatively name
> "Core Extensions" into their own chapter. No longer mixed in with the
> rest of the contrib modules, and I introduce them a bit differently.
> If you want to take a quick look at the new page, I copied it to
> http://www.2ndquadrant.us/docs/html/extensions.html
>
> I'm not completely happy on the wordering there yet. The use of both
> "modules" and "extensions" is probably worth eliminating, and maybe that
> continues on to doing that against the language I swiped from the
> contrib intro too. There's also a lot of shared text at the end there,
> common wording from that and the contrib page about how to install and
> migrate these extensions. Not sure how to refactor it out into another
> section cleanly though.
>
> Regression tests came up last time I posted this. Doesn't look like
> there are any for the modules I'm suggesting should be promoted. Only
> code issue I noticed during another self-review here is that I didn't
> rename contrib/pgrowlocks/pgrowlocks--unpackaged--1.0.sql cleanly, may
> need to do that one over again to get the commits as clean as possible.
>
> Updated code is at
> https://github.com/greg2ndQuadrant/postgres/tree/move-contrib too, and
> since this is painful as a patch the compare view at
> https://github.com/greg2ndQuadrant/postgres/compare/master...move-contrib
> will be easier for browsing the code changes.
>
> --
> Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
> PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us
>
>

>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On 14 October 2011 17:48, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
>
> Is this going to be done for 9.2?
>
> ---------------------------------------------------------------------------

I didn't spot this thread before. I posted something related
yesterday: http://archives.postgresql.org/pgsql-hackers/2011-10/msg00781.php

--
Thom Brown
Twitter: @darkixion
IRC (freenode): dark_ixion
Registered Linux user: #516935

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 10/14/2011 01:48 PM, Bruce Momjian wrote:
> Is this going to be done for 9.2?
>

Refreshing this patch is on my list of things to finish before the next
CommitFest starts later this month.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

On 11/2/11 8:25 AM, Greg Smith wrote:
> On 10/14/2011 01:48 PM, Bruce Momjian wrote:
>> Is this going to be done for 9.2?
>>
>
> Refreshing this patch is on my list of things to finish before the next
> CommitFest starts later this month.

Put me down as reviewer.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

I've revived the corpose of the patch submitted in May, now that it's a
much less strange time of the development cycle to consider it.
http://archives.postgresql.org/message-id/4DF048BD.8040302@2ndquadrant.com
was the first attempt to move some extensions from contrib/ to a new
src/extension/ directory. I have fixed the main complaints from the
last submit attempt, that I accidentally grabbed some old makesfiles and
CVS junk. The new attempt is attached, and is easiest to follow with
the a diff view that understands "moved a file", like github's:
https://github.com/greg2ndQuadrant/postgres/compare/master...core-extensions

You can also check out the docs changes done so far at
http://www.highperfpostgres.com/docs/html/extensions.html I reorganized
the docs to break out what I decided to tentatively name "Core
Extensions" into their own chapter. They're no longer mixed in with the
rest of the contrib modules, and I introduce them a bit differently.
I'm not completely happy on the wordering there yet. The use of both
"modules" and "extensions" is probably worth eliminating, and maybe that
continues on to doing that against the language I swiped from the
contrib intro too. There's also a lot of shared text at the end there,
common wording from that and the contrib page about how to install and
migrate these extensions. Not sure how to refactor it out into another
section cleanly though.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

On 14 November 2011 09:08, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:
> I've revived the corpose of the patch submitted in May, now that it's a much
> less strange time of the development cycle to consider it.
>  http://archives.postgresql.org/message-id/4DF048BD.8040302@2ndquadrant.com
> was the first attempt to move some extensions from contrib/ to a new
> src/extension/ directory.  I have fixed the main complaints from the last
> submit attempt, that I accidentally grabbed some old makesfiles and CVS
> junk.  The new attempt is attached, and is easiest to follow with the a diff
> view that understands "moved a file", like github's:
>  https://github.com/greg2ndQuadrant/postgres/compare/master...core-extensions
>
> You can also check out the docs changes done so far at
> http://www.highperfpostgres.com/docs/html/extensions.html  I reorganized the
> docs to break out what I decided to tentatively name "Core Extensions" into
> their own chapter.  They're no longer mixed in with the rest of the contrib
> modules, and I introduce them a bit differently.  I'm not completely happy
> on the wordering there yet.  The use of both "modules" and "extensions" is
> probably worth eliminating, and maybe that continues on to doing that
> against the language I swiped from the contrib intro too.  There's also a
> lot of shared text at the end there, common wording from that and the
> contrib page about how to install and migrate these extensions.  Not sure
> how to refactor it out into another section cleanly though.

I'm all for removing all mention of "modules". It's ambiguous and
used inconsistently.

In my previous post in this area
(http://archives.postgresql.org/pgsql-hackers/2011-10/msg00781.php) I
suggested that bundling tools, libraries and extensions together in
the same category is confusing. So those are still a problem for me.
And auto_explain appears in your new "Core Extensions" section, but
it's not an extension in the terminology PostgreSQL uses, so that's
also potentially confusing.

--
Thom Brown
Twitter: @darkixion
IRC (freenode): dark_ixion
Registered Linux user: #516935

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Thom Brown <thom(at)linux(dot)com> writes:
> I'm all for removing all mention of "modules". It's ambiguous and
> used inconsistently.

The module is the shared library object. It should be possible to use
that consistently. And I have some plans on my TODO list about them
anyway, so making them disappear from the manual would not serve my
later plans :)

> And auto_explain appears in your new "Core Extensions" section, but
> it's not an extension in the terminology PostgreSQL uses, so that's
> also potentially confusing.

This is a related problem, we should have a terminology for contrib
tools such as pg_standby or pg_archivecleanup, for modules like the one
you talk about, that provide new features but nothing visible from SQL,
and extensions, that are all about SQL --- and if I can work on my plans
will get even more about SQL in a near future.

It's too late for me today to contribute nice ideas here though.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

> This is a related problem, we should have a terminology for contrib
> tools such as pg_standby or pg_archivecleanup, for modules like the one
> you talk about, that provide new features but nothing visible from SQL,
> and extensions, that are all about SQL --- and if I can work on my plans
> will get even more about SQL in a near future.

I see nothing wrong with "Tools" and "Extensions". I'm not sure that
having one catch-all name for them serves the user.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Greg,

So I'm a bit unclear on why most of the optional data types were
excluded from your list of Core Extensions. I would regard the
following as stable and of general utility:

btree_gin
btree_gist
citext
dblink
file_fdw
fuzzystrmatch
hstore
intarray
isn
ltree
pgcrypto
pg_trgm
unaccent
uuid-ossp

These should, in my opinion, all be Core Extensions. I'd go further to
say that if something is materially an extension (as opposed to a tool
or a code example), and we're shipping it with the core distribution, it
either ought to be a core extension, or it should be kicked out to PGXN.

Am I completely misunderstanding what you're trying to accomplish here?

... also, why is there still a "tsearch2" contrib module around at all?

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On 15 November 2011 00:56, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
> Greg,
>
> So I'm a bit unclear on why most of the optional data types were
> excluded from your list of Core Extensions.  I would regard the
> following as stable and of general utility:
>
> btree_gin
> btree_gist
> citext
> dblink
> file_fdw
> fuzzystrmatch
> hstore
> intarray
> isn
> ltree
> pgcrypto
> pg_trgm
> unaccent
> uuid-ossp

Greg clarified on the core extensions page text:

"These core extensions supply useful features in areas such as
database diagnostics and performance monitoring."

None of those others perform such a role. Instead they add additional
functionality intended to be utilised as part of general data usage,
adding new types, operators, query functions etc. Maybe the term
"core" is inappropriate. Instead we might wish to refer to them as
"utility extensions" or something like that, although that may be just
as vague.

> ... also, why is there still a "tsearch2" contrib module around at all?

Backwards compatibility. No-one will use it except if they're coming
from an older version.

--
Thom Brown
Twitter: @darkixion
IRC (freenode): dark_ixion
Registered Linux user: #516935

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 15 November 2011 00:56, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
> So I'm a bit unclear on why most of the optional data types were
> excluded from your list of Core Extensions.  I would regard the
> following as stable and of general utility:

> isn

I consider contrib/isn to be quite broken. It hard codes ISBN prefixes
for the purposes of sanitising ISBNs, even though their assignment is
actually controlled by a decentralised body of regional authorities.
I'd vote for kicking it out of contrib.

--
Peter Geoghegan       http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services

On 11/14/2011 07:56 PM, Josh Berkus wrote:
> So I'm a bit unclear on why most of the optional data types were
> excluded from your list of Core Extensions.

I was aiming for the extensions that seemed uncontroversial for a first
pass here. One of the tests I applied was "do people sometimes need
this module after going into production with their application?" The
very specific problem I was most concerned about eliminating was people
discovering they needed an extension to troubleshoot performance or
corruption issues, only to discover it wasn't available--because they
hadn't installed the postgresql-contrib package. New package
installation can be a giant pain to get onto a production system in some
places, if it wasn't there during QA etc.

All of the data type extensions fail that test. If you need one of
those, you would have discovered that on your development server, and
made sure the contrib package was available on production too. There
very well may be some types that should be rolled into the core
extensions list, but I didn't want arguments over that to block moving
forward with the set I did suggest. We can always move more of them
later, if this general approach is accepted. It only takes about 5
minutes per extension to move them from contrib to src/extension, once
the new directory tree and doc section is there. But I didn't want to
do the work of moving another 15 of them if the whole idea was going to
get shot down.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

On Mon, Nov 14, 2011 at 8:44 PM, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:
> On 11/14/2011 07:56 PM, Josh Berkus wrote:
>>
>> So I'm a bit unclear on why most of the optional data types were
>> excluded from your list of Core Extensions.
>
> I was aiming for the extensions that seemed uncontroversial for a first pass
> here.  One of the tests I applied was "do people sometimes need this module
> after going into production with their application?"  The very specific
> problem I was most concerned about eliminating was people discovering they
> needed an extension to troubleshoot performance or corruption issues, only
> to discover it wasn't available--because they hadn't installed the
> postgresql-contrib package.  New package installation can be a giant pain to
> get onto a production system in some places, if it wasn't there during QA
> etc.
>
> All of the data type extensions fail that test.  If you need one of those,
> you would have discovered that on your development server, and made sure the
> contrib package was available on production too.  There very well may be
> some types that should be rolled into the core extensions list, but I didn't
> want arguments over that to block moving forward with the set I did suggest.
>  We can always move more of them later, if this general approach is
> accepted.  It only takes about 5 minutes per extension to move them from
> contrib to src/extension, once the new directory tree and doc section is
> there.  But I didn't want to do the work of moving another 15 of them if the
> whole idea was going to get shot down

I continue to think that we should be trying to sort these by subject
matter. The term "core extensions" doesn't convey that these are
server management and debugging tools, hence Josh's confusion.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 11/14/2011 10:09 PM, Robert Haas wrote:
> I continue to think that we should be trying to sort these by subject
> matter. The term "core extensions" doesn't convey that these are
> server management and debugging tools, hence Josh's confusion.
>

I'm not attached to the name, which I just pulled out of the air for the
documentation. Could just as easily call them built-in modules or
extensions. If the objection is that "extensions" isn't technically
correct for auto-explain, you might call them core add-ons instead. My
thinking was that the one exception didn't make it worth the trouble to
introduce a new term altogether here. There's already too many terms
used for talking about this sort of thing, the confusion from using a
word other than "extensions" seemed larger than the confusion sown by
auto-explain not fitting perfectly.

The distinction I care about here is primarily a packaging one. These
are server additions that people should be able to count on having
available, whereas right now they may or may not be installed depending
on if contrib was added. Everything I'm touching requires our RPM and
Debian packagers (at least) make a packaging change, too. I can't
justify why that's worth doing for any of the other extensions, which is
one reason I don't try to tackle them.

The type of finer sorting you and Thom are suggesting seems like it's
mainly a documentation change to me. I'm indifferent to the idea; no
plans to either work on it or object to it. The docs could be made
easier to follow here without any change to the directory tree, and
trying to push out a larger packaging change has downsides. Useful
reminder reading here is
http://wiki.postgresql.org/wiki/PgCon_2011_Developer_Meeting#Moving_Contrib_Around
To quote from there, "Users hate having loads and loads of packages. We
do need to be careful not to oversplit it." There's some useful notes
about dependency issues there too.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

Thom Brown <thom(at)linux(dot)com> writes:
> None of those others perform such a role. Instead they add additional
> functionality intended to be utilised as part of general data usage,
> adding new types, operators, query functions etc. Maybe the term
> "core" is inappropriate. Instead we might wish to refer to them as
> "utility extensions" or something like that, although that may be just
> as vague.

The term “core” here intends to show off that those extensions are
maintained by the core PostgreSQL developer team. If needs be, those
extensions will get updated in minor releases (crash, bugs, security,
etc).

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

Greg,

> I'm not attached to the name, which I just pulled out of the air for
> the
> documentation. Could just as easily call them built-in modules or
> extensions. If the objection is that "extensions" isn't technically
> correct for auto-explain, you might call them core add-ons instead.
> My
> thinking was that the one exception didn't make it worth the trouble
> to
> introduce a new term altogether here. There's already too many terms
> used for talking about this sort of thing, the confusion from using a
> word other than "extensions" seemed larger than the confusion sown by
> auto-explain not fitting perfectly.

Well, I do think it should be *something* Extensions. But Core Extensions implies that the other stuff is just random code, and makes the user wonder why it's included at all. If we're going to rename some of the extensions, then we really need to rename them all or we look like those are being depreciated.

Maybe:

Core Management Extensions
Core Development Extensions
Additional Database Tools
Code Examples
Legacy Modules

I think that covers everything we have in contrib.

Given discussion, is there any point in reporting on the actual patch yet?

--Josh Berkus

Peter,

> I consider contrib/isn to be quite broken. It hard codes ISBN
> prefixes
> for the purposes of sanitising ISBNs, even though their assignment is
> actually controlled by a decentralised body of regional authorities.
> I'd vote for kicking it out of contrib.

Submit a patch to fix it then.

I use ISBN in 2 projects, and it's working fine for me. I'll strongly resist any attempt to "kick it out".

--Josh Berkus

On Tue, Nov 15, 2011 at 12:54 PM, Joshua Berkus <josh(at)agliodbs(dot)com> wrote:
>> I consider contrib/isn to be quite broken. It hard codes ISBN
>> prefixes
>> for the purposes of sanitising ISBNs, even though their assignment is
>> actually controlled by a decentralised body of regional authorities.
>> I'd vote for kicking it out of contrib.
>
> Submit a patch to fix it then.

It's not fixable. The ISBN datatype is the equivalent of having an
SSN datatype that only allows SSNs that have actually been assigned to
a US citizen.

> I use ISBN in 2 projects, and it's working fine for me.  I'll strongly resist any attempt to "kick it out".

That's exactly why contrib is a random amalgamation of really useful
stuff and utter crap: people feel justified in defending the continued
existence of the crap on the sole basis that it's useful to them
personally.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 11/15/2011 12:53 PM, Joshua Berkus wrote:
> Given discussion, is there any point in reporting on the actual patch yet?

I don't expect the discussion will alter the code changes that are the
main chunk of the patch here. The only place the most disputed parts
impact is the documentation.

I like "Management Extensions" as an alternate name for this category
instead, even though it still has the issue that auto_explain isn't
technically an extension. The name does help suggest why they're thrown
into a different directory and package.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

Robert Haas wrote:
> > I use ISBN in 2 projects, and it's working fine for me. ?I'll strongly resist any attempt to "kick it out".
>
> That's exactly why contrib is a random amalgamation of really useful
> stuff and utter crap: people feel justified in defending the continued
> existence of the crap on the sole basis that it's useful to them
> personally.

Agreed. Berkus must have one million customers to have X customers
using every feature we want to remove or change.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> Joshua Berkus <josh(at)agliodbs(dot)com> wrote:

>>> I consider contrib/isn to be quite broken. It hard codes ISBN
>>> prefixes for the purposes of sanitising ISBNs, even though their
>>> assignment is actually controlled by a decentralised body of
>>> regional authorities.

By an international standard which says what numbers are valid in
the "prefix element" and "registration group element" of the ISBN
for each of those regional authorities, and how the check digit is
to be calculated.

>>> I'd vote for kicking it out of contrib.
>>
>> Submit a patch to fix it then.
>
> It's not fixable. The ISBN datatype is the equivalent of having
> an SSN datatype that only allows SSNs that have actually been
> assigned to a US citizen.

Certainly it would make sense to go so far as to support the overall
standard format as described here:

http://www.isbn-international.org/faqs/view/5#q_5

Beyond the broad strokes there, perhaps it would make sense for the
type to be able to digest a RangeMessage.xml file supplied by the
standards organization, so that the current ranges could be plugged
in as needed independently of the PostgreSQL release.

http://www.isbn-international.org/page/ranges
http://www.isbn-international.org/pages/media/Range%20message/RangeMessage.pdf

Hard-coding ranges as of some moment in time seems pretty dubious.

-Kevin

>> Submit a patch to fix it then.
>
> It's not fixable. The ISBN datatype is the equivalent of having an
> SSN datatype that only allows SSNs that have actually been assigned to
> a US citizen.

Nothing is "not fixable". "not fixable without breaking backwards
compatibility" is entirely possible, though. If fixing it led to two
different versions of ISN, then that would be a reason to push it to
PGXN instead of shipping it.

It's not as if ISN is poorly coded. This is a spec issue, which must
have been debated when we first included it. No?

> That's exactly why contrib is a random amalgamation of really useful
> stuff and utter crap: people feel justified in defending the continued
> existence of the crap on the sole basis that it's useful to them
> personally.

Why else would we justify anything? It's very difficult to argue on the
basis of theoretical users. How would we really know what a theoretical
user wants?

Calling something "crap" because it has a spec issue is unwarranted.
We're going to get nowhere in this discussion as long as people are
using extreme and non-descriptive terms.

The thing is, most of the extensions in /contrib have major flaws, or
they would have been folded in to the core code by now. CITEXT doesn't
support multiple collations. INTARRAY and LTREE have inconsistent
operators and many bugs. CUBE lacks documentation. DBlink is an
ongoing battle with security holes. etc.

Picking out one of those and saying "this is crap because of reason X,
but I'll ignore all the flaws in all these other extensions" is
inconsistent and not liable to produce results. Now, if you wanted to
argue that we should kick *all* of the portable extensions out of
/contrib and onto PGXN, then you'd have a much stronger argument.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On 15 November 2011 18:03, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> It's not fixable.  The ISBN datatype is the equivalent of having an
> SSN datatype that only allows SSNs that have actually been assigned to
> a US citizen.

That isn't even the worst part. isn is basically only useful in the US
at the moment, because in every other country there are a number of
bar code symbologies that co-exist within supply chain management for
various reasons. Only in the US can you reasonably assume that all
articles that you come across will be UPC, and even that might be a
shaky assumption these days.

In the E.U. and much of the rest of the world, products that you buy
in the store will bear one of the following symbologies: EAN-8 (for
small articles like chewing gum), UPC (the American one, 12 digits),
EAN-13 and GTIN-14. Some, but not all of these are available from
contrib/isn. There is no datatype that represents "some known barcode
format", even though writing an SQL function that can be used in a
domain check constraint to do just that is next to trivial. I guess
that means that you'd either have to have multiple columns for each
datatype, each existing in case the article in question was of that
particular datatype, or you'd need to make a hard assumption about the
symbology used for all articles that will ever be entered.

The way that these formats maintain backwards compatibility is through
making previous formats valid as the new format by padding zeros to
the left of the older format. So a UPC is actually a valid EAN-13,
just by adding a zero to the start - the US EAN country code is zero,
IIRC, so the rest of the world can pretend that Americans use
EAN-13s/GTIN-14s, even though they think that they use UPCs. The check
digit algorithms for each successive symbology are built such that
this works. This is why a DIY bar code bigint domain can be written so
easily, and also why doing so makes way more sense than using this
contrib module.

To my mind, contrib/isn is a solution looking for a problem, and
that's before we even talk about ISBN prefixes. By including it, we
give users a false sense of security about doing the right thing, when
they're very probably not.

--
Peter Geoghegan       http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services

Excerpts from Robert Haas's message of mar nov 15 15:03:03 -0300 2011:
> On Tue, Nov 15, 2011 at 12:54 PM, Joshua Berkus <josh(at)agliodbs(dot)com> wrote:
> >> I consider contrib/isn to be quite broken. It hard codes ISBN
> >> prefixes
> >> for the purposes of sanitising ISBNs, even though their assignment is
> >> actually controlled by a decentralised body of regional authorities.
> >> I'd vote for kicking it out of contrib.
> >
> > Submit a patch to fix it then.
>
> It's not fixable. The ISBN datatype is the equivalent of having an
> SSN datatype that only allows SSNs that have actually been assigned to
> a US citizen.

Interesting. Isn't it possible to separate it into two parts, one
containing generic input, formatting and check digits verification, and
another one that would do the prefix matching? Presumably, the first
part would still be useful without prefix validation, wouldn't it?
Surely the other validation rules aren't different for the various
prefixes.

Perhaps the prefix matching code should not be in C, or at least use a
lookup table that's not in C code, so that it can be updated in userland
without having to recompile. (BTW, this is very similar to the problem
confronted by the E.164 module, which attempts to do telephone number
validation and formatting; there's a generic body of code that handles
the basic country code validation, but there's supposed to be some
per-CC formatting rules which we're not really clear on where to store.
We punted on it by just having that in a GUC, so that the user can
update it easily; but that's clearly not the best solution).

--
Álvaro Herrera <alvherre(at)commandprompt(dot)com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

On 15 November 2011 19:01, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
> Nothing is "not fixable".

My idea of fixing contrib/isn would be to remove so much of it that it
would obviously make more sense to use simple, flexible SQL. It simply
makes way too many assumptions about the user's business rules for a
generic C module.

> Calling something "crap" because it has a spec issue is unwarranted.
> We're going to get nowhere in this discussion as long as people are
> using extreme and non-descriptive terms.

It is warranted, because contrib/isn is extremely wrong-headed.

> The thing is, most of the extensions in /contrib have major flaws, or
> they would have been folded in to the core code by now.  CITEXT doesn't
> support multiple collations.  INTARRAY and LTREE have inconsistent
> operators and many bugs.  CUBE lacks documentation.  DBlink is an
> ongoing battle with security holes.  etc.

The difference is that it's possible to imagine a scenario under which
I could recommend using any one of those modules, despite their flaws.
I could not contrive a reason for using contrib/isn.

--
Peter Geoghegan       http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services

On Tue, Nov 15, 2011 at 2:01 PM, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
>>> Submit a patch to fix it then.
>>
>> It's not fixable.  The ISBN datatype is the equivalent of having an
>> SSN datatype that only allows SSNs that have actually been assigned to
>> a US citizen.
>
> Nothing is "not fixable".  "not fixable without breaking backwards
> compatibility" is entirely possible, though.  If fixing it led to two
> different versions of ISN, then that would be a reason to push it to
> PGXN instead of shipping it.

Well, the way to fix it would be to publish a new version of
PostgreSQL every time the international authority that assigns ISBN
prefixes allocates a new one, and for everyone to then update their
PostgreSQL installation every time we do that. That doesn't, however,
seem very practical. It just doesn't make sense to define a datatype
where the set of legal values changes over time. The right place to
put such constraints in the application logic, where it doesn't take a
database upgrade to fix the problem.

> It's not as if ISN is poorly coded. This is a spec issue, which must
> have been debated when we first included it. No?

I think our standards for inclusion have changed over the years - a
necessary part of project growth, or we would have 500 contrib modules
instead of 50. The original isbn_issn module was contributed in 1998;
it was replaced by contrib/isn in 2006 because the original module
became obsolete. I think it's fair to say that if this code were
submitted today it would never get accepted into our tree, and the
submitter would be advised not so much to publish on PGXN instead as
to throw it away entirely and rethink their entire approach to the
problem.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> Josh Berkus <josh(at)agliodbs(dot)com> wrote:

>> Nothing is "not fixable". "not fixable without breaking
>> backwards compatibility" is entirely possible, though. If fixing
>> it led to two different versions of ISN, then that would be a
>> reason to push it to PGXN instead of shipping it.
>
> Well, the way to fix it would be to publish a new version of
> PostgreSQL every time the international authority that assigns
> ISBN prefixes allocates a new one, and for everyone to then update
> their PostgreSQL installation every time we do that. That
> doesn't, however, seem very practical.

Having just taken a closer look at contrib/isn, I'm inclined to
think the current implementation is pretty hopeless. ISBN seems
common enough and standardized enough that it could perhaps be
included in contrib with the proviso that ranges would only be
validated through pointing to a copy of the XML provided by the
standards body -- it wouldn't be up to PostgreSQL to supply that.

The other types in contrib/isn are things I don't know enough about
to have an opinion, but it seems a little odd to shove them all
together. It would seem more natural to me to have a distinct type
for each and, if needed, figure out how to get a clean union of the
types.

-Kevin

Josh Berkus <josh(at)agliodbs(dot)com> writes:
> The thing is, most of the extensions in /contrib have major flaws, or
> they would have been folded in to the core code by now. CITEXT doesn't
> support multiple collations. INTARRAY and LTREE have inconsistent
> operators and many bugs. CUBE lacks documentation. DBlink is an
> ongoing battle with security holes. etc.

> Picking out one of those and saying "this is crap because of reason X,
> but I'll ignore all the flaws in all these other extensions" is
> inconsistent and not liable to produce results. Now, if you wanted to
> argue that we should kick *all* of the portable extensions out of
> /contrib and onto PGXN, then you'd have a much stronger argument.

There's a larger issue here, which is that a lot of the stuff in contrib
is useful as (a) coding examples for people to look at, and/or (b) test
cases for core-server functionality. If a module gets kicked out to
PGXN we lose pretty much all the advantages of (b), and some of the
value of (a) because stuff that is in the contrib tree at least gets
maintained when we make server API changes.

So the fact that a module has conceptual flaws that are preventing it
from getting moved into core doesn't really have much to do with whether
we should remove it, IMV. The big question to be asking about ISN is
whether removing it would result in a loss of test coverage for the core
server; and I believe the answer is yes --- ISTR at least one bug that
we found specifically because ISN tickled it.

regards, tom lane

On 15 November 2011 21:53, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> There's a larger issue here, which is that a lot of the stuff in contrib
> is useful as (a) coding examples for people to look at, and/or (b) test
> cases for core-server functionality.  If a module gets kicked out to
> PGXN we lose pretty much all the advantages of (b), and some of the
> value of (a) because stuff that is in the contrib tree at least gets
> maintained when we make server API changes.

The isn module is patently broken. It has the potential to damage the
project's reputation if someone chooses to make an example out of it.
I think that that's more important than any additional test coverage
it may bring. There's only a fairly marginal benefit at the expense of
a bad user experience for anyone who should use isn.

--
Peter Geoghegan       http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services

On Tue, Nov 15, 2011 at 6:44 PM, Peter Geoghegan <peter(at)2ndquadrant(dot)com> wrote:
> On 15 November 2011 21:53, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> There's a larger issue here, which is that a lot of the stuff in contrib
>> is useful as (a) coding examples for people to look at, and/or (b) test
>> cases for core-server functionality.  If a module gets kicked out to
>> PGXN we lose pretty much all the advantages of (b), and some of the
>> value of (a) because stuff that is in the contrib tree at least gets
>> maintained when we make server API changes.
>
> The isn module is patently broken. It has the potential to damage the
> project's reputation if someone chooses to make an example out of it.
> I think that that's more important than any additional test coverage
> it may bring. There's only a fairly marginal benefit at the expense of
> a bad user experience for anyone who should use isn.

I agree. The argument that this code is useful as example code has
been offered before, but the justification is pretty thin when the
example code is an example of a horrible design that no one should
ever copy. I don't see that the isn code is doing anything that is so
unique that one of our add-on data types wouldn't be a suitable
(probably far better) template, but if it is, let's add similar
functionality to some other module, or add a new module that does
whatever that interesting thing is, or shove some code in
src/test/examples. We can't go on complaining one the one hand that
people don't install postgresql-contrib, and then on the other hand
insist on shipping really bad code in contrib. It's asking a lot for
someone who isn't already heavily involved in the project to
distinguish the wheat from the chaff.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 15 November 2011 23:57, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> We can't go on complaining one the one hand that
> people don't install postgresql-contrib, and then on the other hand
> insist on shipping really bad code in contrib.  It's asking a lot for
> someone who isn't already heavily involved in the project to
> distinguish the wheat from the chaff.

ISTM that any sensible sanitisation of data guards against Murphy, not
Machiavelli. Exactly what sort of error is it imagined will be
prevented by enforcing that ISBN prefixes are "valid"? Transcription
and transposition errors are already guarded against very effectively
by the check digit.

If we can't put isn out of its misery, a sensible compromise would be
to rip out the prefix enforcement feature so that, for example, ISBN13
behaved exactly the same as EAN13. That would at least result in
predictable behaviour. The "strike" that separates each part of the
ISBN would be lost, but it is actually not visible on large ISBN
websites, presumably because it's a tar-pit of a problem.

--
Peter Geoghegan       http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services

All,

> I agree. The argument that this code is useful as example code has
> been offered before, but the justification is pretty thin when the
> example code is an example of a horrible design that no one should
> ever copy.

People are already using ISN (or at least ISBN) in production. It's been around for 12 years. So any step we take with contrib/ISN needs to take that into account -- just as we have with Tsearch2 and XML2.

One can certainly argue that some of the stuff in /contrib would be better on PGXN. But in that case, it's not limited to ISN; there are several modules of insufficient quality (including intarray and ltree) or legacy nature which ought to be pushed out. Probably most of them.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
San Francisco

On 16 November 2011 01:09, Joshua Berkus <josh(at)agliodbs(dot)com> wrote:
> People are already using ISN (or at least ISBN) in production.  It's been around for 12 years.

contrib/isn has been around since 2006. The argument "some unknowable
number of people are using this feature in production" could equally
well apply to anything that we might consider deprecating.

I am not arguing for putting isn on PGXN. I'm arguing for actively
warning people against using it, because it is harmful. Any serious
use of the ISBN datatypes can be expected to break unpredictably one
day, and the only thing that someone can do in that situation is to
write their own patch to contrib/isn. They'd then have to wait for
that patch to be accepted if they didn't want to fork, which is a very
bad situation indeed. This already happened once.

--
Peter Geoghegan       http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services

Well, this discussion veering off into ISN has certainly validated my
gut feel that I should touch only the minimum number of things that
kills my pain points, rather than trying any more ambitious
restructuring. I hope that packaged extensions become so popular that
some serious cutting can happen to contrib, especially the data type
additions. If something as big as PostGIS can live happily as an
external project, surely most of these can too.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

Peter Geoghegan <peter(at)2ndquadrant(dot)com> writes:
> If we can't put isn out of its misery, a sensible compromise would be
> to rip out the prefix enforcement feature so that, for example, ISBN13
> behaved exactly the same as EAN13.

That might be a reasonable compromise. Certainly the check-digit
calculation is much more useful for validity checking than the prefix
checking.

regards, tom lane

On Tue, Nov 15, 2011 at 5:50 AM, Dimitri Fontaine
<dimitri(at)2ndquadrant(dot)fr> wrote:
> Thom Brown <thom(at)linux(dot)com> writes:
>> None of those others perform such a role.  Instead they add additional
>> functionality intended to be utilised as part of general data usage,
>> adding new types, operators, query functions etc.  Maybe the term
>> "core" is inappropriate.  Instead we might wish to refer to them as
>> "utility extensions" or something like that, although that may be just
>> as vague.
>
> The term “core” here intends to show off that those extensions are
> maintained by the core PostgreSQL developer team. If needs be, those
> extensions will get updated in minor releases (crash, bugs, security,
> etc).

Everything in contrib meets that definition, more or less.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> The term “core” here intends to show off that those extensions are
>> maintained by the core PostgreSQL developer team. If needs be, those
>> extensions will get updated in minor releases (crash, bugs, security,
>> etc).
>
> Everything in contrib meets that definition, more or less.

Yeah? It would only mean that Josh Berkus complaint about the naming is
to be followed.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

On Wed, Nov 16, 2011 at 9:20 AM, Dimitri Fontaine
<dimitri(at)2ndquadrant(dot)fr> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>>> The term “core” here intends to show off that those extensions are
>>> maintained by the core PostgreSQL developer team. If needs be, those
>>> extensions will get updated in minor releases (crash, bugs, security,
>>> etc).
>>
>> Everything in contrib meets that definition, more or less.
>
> Yeah? It would only mean that Josh Berkus complaint about the naming is
> to be followed.

I am not sure I'm quite following you, but I'm unaware that there are
some contrib modules that we maintain more than others. Bugs and
security vulnerabilities in any of them are typically fixed when
reported. Now, sometimes we might not be able to fix a bug because of
some architectural deficiency, but that also happens in the server -
consider, for example, the recent discussion of creating a table in a
schema that is concurrently being dropped, which is likely to require
far more invasive fixing than we are probably willing to do anywhere
other than master.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 11/15/11 7:40 PM, Tom Lane wrote:
> Peter Geoghegan <peter(at)2ndquadrant(dot)com> writes:
>> If we can't put isn out of its misery, a sensible compromise would be
>> to rip out the prefix enforcement feature so that, for example, ISBN13
>> behaved exactly the same as EAN13.
>
> That might be a reasonable compromise. Certainly the check-digit
> calculation is much more useful for validity checking than the prefix
> checking.

Sounds good to me. FWIW, I know that ISBN is being used for some
library software, so a backwards-compatible fix would be very desirable.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Josh Berkus <josh(at)agliodbs(dot)com> writes:
> On 11/15/11 7:40 PM, Tom Lane wrote:
>> Peter Geoghegan <peter(at)2ndquadrant(dot)com> writes:
>>> If we can't put isn out of its misery, a sensible compromise would be
>>> to rip out the prefix enforcement feature so that, for example, ISBN13
>>> behaved exactly the same as EAN13.

>> That might be a reasonable compromise. Certainly the check-digit
>> calculation is much more useful for validity checking than the prefix
>> checking.

> Sounds good to me. FWIW, I know that ISBN is being used for some
> library software, so a backwards-compatible fix would be very desirable.

How backwards-compatible do you mean? Do you think we need to keep the
existing prefix-check logic as an option, or can we just drop it and be
done?

(I'm a bit concerned about the angle that the code has some smarts
currently about where to put hyphens in output. If we rip that out
it could definitely break application code, whereas dropping a check
shouldn't.)

regards, tom lane

> (I'm a bit concerned about the angle that the code has some smarts
> currently about where to put hyphens in output. If we rip that out
> it could definitely break application code, whereas dropping a check
> shouldn't.)

Right. Do we need to dump the hyphen logic?

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On 17 November 2011 02:32, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
>
>> (I'm a bit concerned about the angle that the code has some smarts
>> currently about where to put hyphens in output.  If we rip that out
>> it could definitely break application code, whereas dropping a check
>> shouldn't.)
>
> Right.  Do we need to dump the hyphen logic?

Only if you think that it's better to have something that covers many
cases but is basically broke. Perhaps it's different for code that is
already committed, but in the case of new submissions it tends to be
better to have something that is limited in a well-understood way
rather than less limited in a way that is unpredictable or difficult
to reason about.

--
Peter Geoghegan       http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services

Peter Geoghegan <peter(at)2ndquadrant(dot)com> writes:
> On 17 November 2011 02:32, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
>> Right. Do we need to dump the hyphen logic?

> Only if you think that it's better to have something that covers many
> cases but is basically broke. Perhaps it's different for code that is
> already committed, but in the case of new submissions it tends to be
> better to have something that is limited in a well-understood way
> rather than less limited in a way that is unpredictable or difficult
> to reason about.

Well, as was stated upthread, we might have bounced this module in toto
if it were submitted today. But contrib/isn has been there since 2006,
and its predecessor contrib/isbn_issn was there since 1998, and both of
those submissions came from (different) people who needed the
functionality bad enough to write it. It's not reasonable to suppose
that nobody is using it today. Ergo, we can't just summarily break
backwards compatibility on the grounds that we don't like the design.
Heck, we don't even have a field bug report that the design limitation
is causing any real problems for real users ... so IMO, the claims that
this is dangerously broken are a bit overblown.

regards, tom lane

On 11/16/11 7:54 PM, Tom Lane wrote:
> Heck, we don't even have a field bug report that the design limitation
> is causing any real problems for real users ... so IMO, the claims that
> this is dangerously broken are a bit overblown.

This is why I mentioned clients using ISBN in production. I have yet to
see any actual breakage in the field. Granted, both clients are
US-only. I don't believe either of these clients is depending on the
prefix-check, and that's the sort of thing we could announce in the
release notes.

I do get the feeling that Peter got burned by ISN, given his vehemence
in erasing it from the face of the earth. So that's one bug report. ;-)

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On 17 November 2011 03:54, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> It's not reasonable to suppose
> that nobody is using it today.

I didn't suppose that no one is using it, but that those that are
using it are unaware of the risks with prefix validation, and that
there will be a rude awakening for them.

> Ergo, we can't just summarily break
> backwards compatibility on the grounds that we don't like the design.
> Heck, we don't even have a field bug report that the design limitation
> is causing any real problems for real users ... so IMO, the claims that
> this is dangerously broken are a bit overblown.

I think that's it's rather unlikely that removing hyphenation and
prefix validation would adversely affect anyone, provided that it was
well documented and wasn't applied to stable branches. If it were up
to me, I might remove validation from stable branches but keep
hyphenation, while removing both for 9.2 . After all, hyphenation will
break anyway, so they're worse off continuing to rely on hyphenation
when it cannot actually be relied on.

On 17 November 2011 05:03, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
> I do get the feeling that Peter got burned by ISN, given his vehemence
> in erasing it from the face of the earth.  So that's one bug report.  ;-)

Actually, I reviewed a band-aid patch about a year ago, and I was
fairly concerned at the obvious wrong-headedness of something in our
tree. I have a certain amount of domain knowledge here, but I've never
actually attempted to use it in production. For all its faults, it is
at least obviously broken to someone that knows about GS1 standards
(having separate bar code datatypes is just not useful at all),
although that tends to not be Americans. This may account for why
we've heard so few complaints. It's also really easy and effective to
create your own domain, and the flexibility that this affords tends to
make an off-the-shelf solution unattractive (I've done things like
store "compacted" bar codes that will subsequently be used to match a
full bar code with an embedded price - I didn't want to enforce a
check digit for the compacted representation).

If I had a lot of time to work on fixing contrib/isn, I still
wouldn't, because the correct thing to do is to produce your own
domain.

--
Peter Geoghegan       http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services

On Thu, Nov 17, 2011 at 10:44 AM, Peter Geoghegan <peter(at)2ndquadrant(dot)com> wrote:
> I think that's it's rather unlikely that removing hyphenation and
> prefix validation would adversely affect anyone, provided that it was
> well documented and wasn't applied to stable branches. If it were up
> to me, I might remove validation from stable branches but keep
> hyphenation, while removing both for 9.2 . After all, hyphenation will
> break anyway, so they're worse off continuing to rely on hyphenation
> when it cannot actually be relied on.

Well, keep in mind that most people test their code. It seems likely
that it actually DOES work pretty well for the people who are using
the module. The ones for whom it didn't work presumably would have
complained (and, mostly, they haven't) or abandoned the module (in
which case they're irrelevant to the discussion of who might be hurt
by this change). I'd be willing to bet a nickle that we'll get
complaints if we rip that hyphenation behavior out.

At the same time, I still think we should push this out to PGXN or
pgfoundry or something. The fact that it's useful to some people does
not mean that it's a good example for other people to follow, or that
we want the core distribution to be in the process of tracking ISBN
prefixes on behalf of PostgreSQL users everywhere.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> At the same time, I still think we should push this out to PGXN or
> pgfoundry or something. The fact that it's useful to some people does
> not mean that it's a good example for other people to follow, or that
> we want the core distribution to be in the process of tracking ISBN
> prefixes on behalf of PostgreSQL users everywhere.

I wouldn't object to that as long as we replaced it with some other
module that had a similar relationship to core types. We'd never have
realized the need for CREATE TYPE's LIKE option, until it was far too
late, if we'd not had contrib/isn to show up the problem (cf
commit 3f936aacc057e4b391ab953fea2ffb689a12a8bc).

But really I think this discussion is making a mountain out of a
molehill. If tracking ISBN prefix changes is such a time-consuming
activity, why have we not seen a steady stream of update patches from
users? By my count there's been just one such patch since 2006
(commit 6d1af7b2180719102a907bd3e35d218b43e76ad1).

regards, tom lane

On mån, 2011-11-14 at 20:44 -0500, Greg Smith wrote:
> The very specific problem I was most concerned about eliminating was
> people discovering they needed an extension to troubleshoot
> performance or corruption issues, only to discover it wasn't
> available--because they hadn't installed the postgresql-contrib
> package.

Who's to say that after this, the core extensions won't end up in a new
separate package postgresql-extensions (or similar) or might even stay
in postgresql-contrib, for compatibility?

On 11/17/2011 03:18 PM, Peter Eisentraut wrote:
> Who's to say that after this, the core extensions won't end up in a new
> separate package postgresql-extensions (or similar) or might even stay
> in postgresql-contrib, for compatibility?
>

I don't know why packagers would make an active decision that will make
their lives more difficult, with no benefit to them and a regression
against project recommendations for their users. The last thing anyone
packaging PostgreSQL wants is more packages to deal with; there are
already too many. Each of the current sub-packages has a legitimate
technical or distribution standard reason to exist--guidelines like
"break out client and server components" or "minimize the package
dependencies for the main server". I can't think of any good reason
that would inspire the sort of drift you're concerned about.

There's little compatibility argument beyond consistency with the
previous packages. The reason why this is suddenly feasible now is that
the under the hood change are almost all hidden by CREATE EXTENSION.

And if some wanted to wander this way, they'll end up having to maintain
a doc patch to address the fact that they've broken with project
recommendations. This text in what I submitted will no longer be true:

"This appendix contains information regarding core extensions that are
built and included with a standard installation of PostgreSQL."

One of the reasons I picked the name I did was to contrast with the
existing description of contrib:

"porting tools, analysis utilities, and plug-in features that are not
part of the core PostgreSQL system, mainly because they address a
limited audience or are too experimental to be part of the main source
tree."

That says it's perfectly fine to make these optional in another
package--they're not "part of the core". That scary wording is
practically telling packagers to separate them, so it's easy to keep the
experimental stuff away from the production quality components.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

Greg Smith <greg(at)2ndQuadrant(dot)com> writes:
> On 11/17/2011 03:18 PM, Peter Eisentraut wrote:
>> Who's to say that after this, the core extensions won't end up in a new
>> separate package postgresql-extensions (or similar) or might even stay
>> in postgresql-contrib, for compatibility?

> I don't know why packagers would make an active decision that will make
> their lives more difficult, with no benefit to them and a regression
> against project recommendations for their users.

Why do you figure that, exactly? The path of least resistance will
be precisely to leave everything packaged as it is, in a single
postgresql-contrib module. I'm pretty likely to do that myself for
Fedora and RHEL. Subdividing/rearranging contrib makes the packager's
life more complicated, *and* makes his users' lives more complicated,
if only because things aren't where they were before. It seems unlikely
to happen, at least in the near term.

> And if some wanted to wander this way, they'll end up having to maintain
> a doc patch to address the fact that they've broken with project
> recommendations. This text in what I submitted will no longer be true:

You're assuming anybody will notice or care about that text, if indeed
it gets committed/released with that wording at all.

The upstream project can't force these decisions on packagers, and it
doesn't help to go about under the illusion that we can.

regards, tom lane

Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:
> Why do you figure that, exactly? The path of least resistance will
> be precisely to leave everything packaged as it is, in a single
> postgresql-contrib module. I'm pretty likely to do that myself for
> Fedora and RHEL. Subdividing/rearranging contrib makes the packager's
> life more complicated, *and* makes his users' lives more complicated,
> if only because things aren't where they were before. It seems unlikely
> to happen, at least in the near term.

Then if we want packagers to move, what about removing all the
extensions not listed by Greg from the contrib/ directory and inventing
another place where to manage them, which is not automatically built,
but still part of buildfarm tests, if at all possible.

Then the only change we suggest to packagers is to have the main
PostgreSQL package install the contrib one by means of dependencies.

I don't much like this solution, but that's how I read your email. The
status quo is not a good place to live in.

> The upstream project can't force these decisions on packagers, and it
> doesn't help to go about under the illusion that we can.

Really? You are packaging for RHEL, Dave is responsible for the windows
packaging, Devrim is covering the other RPM systems (apart from SuSE
maybe and I'm not even sure) and Martin is caring for debian and ubuntu
and following along. We're missing BSD ports packagers, and we're
covering like 90% or more of the servers and developers installs.

We can't force everybody hands to doing it our way, but I'm pretty sure
we can talk to them and see what they think about the usefulness of this
proposal and how they intend to react. We're not *that* disconnected.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

On 11/18/11 12:27 PM, Dimitri Fontaine wrote:
> Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:
>> Why do you figure that, exactly? The path of least resistance will
>> be precisely to leave everything packaged as it is, in a single
>> postgresql-contrib module. I'm pretty likely to do that myself for
>> Fedora and RHEL. Subdividing/rearranging contrib makes the packager's
>> life more complicated, *and* makes his users' lives more complicated,
>> if only because things aren't where they were before. It seems unlikely
>> to happen, at least in the near term.
>
> Then if we want packagers to move, what about removing all the
> extensions not listed by Greg from the contrib/ directory and inventing
> another place where to manage them, which is not automatically built,
> but still part of buildfarm tests, if at all possible.

Actually, the whole idea is that the "Core Management Extensions" should
move from the -contrib module to the -server module. That is, those
extensions should always get installed with any server.

Of course, packagers may then reasonably ask why that code is not just
part of the core?

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On 11/18/2011 03:36 PM, Josh Berkus wrote:
> Of course, packagers may then reasonably ask why that code is not just
> part of the core?
>

Let me step back from the implementation ideas for a minute and talk
about this, and how it ties into release philosophy. The extensions
infrastructure for PostgreSQL is one of its strongest features. We can
use it as a competitive advantage over other databases, one that can
make this database stable and continuously innovating at the same time.
But that's not happening enough yet; I feel this change is a major step
in that direction. There's no demonstration that extensions are edible
dog food like the core database visibly eating a can.

To see why this matters so much, let's compare two stereotypes of
PostgreSQL users at different extremes of upgrade tolerance. First we
have the classic enterprise DBA. Relative to this person's
expectations, PostgreSQL releases are way too fast. They can't upgrade
their database every year; that's madness. This is the person who we
yell at about how they should upgrade to the latest minor point release,
because once they have a working system they touch *nothing*. For this
user, the long beta period of new PostgreSQL releases, and its general
conservative development model, are key components to PostgreSQL being
suitable for them.

At the other extreme, we have the software developer with a frantic
development/release schedule, the one who's running the latest stable
version of every tool they use. This person expects some bugs in them,
and the first reaction to running into one is to ask "is this fixed in
the next version?" You'll find at least one component in their stack
that's labeled "compiled from the latest checkout" because that was the
only way to get a working version. And to them, the yearly release
cycle of PostgreSQL is glacial. When they run into a limitation that is
impacting a user base that's doubling every few months, they can't wait
a year for a fix; they could easily go out of business by then.

The key to satisfying both these extremes at once is a strong extensions
infrastructure, led by the example of serious tools that are provided
that way in the PostgreSQL core. For this to catch on, we need the
classic DBAs to trust core extensions enough to load them into
production. They don't do that now because the current contrib
description sounds too scary, and they may not even have loaded that
package onto the server. And we need people who want more frequent
database core changes to see that extensions are a viable way to build
some pretty extensive server hacks.

I've submitted two changes to this CommitFest that are enhancing
features in this "core extensions" set. Right now I have multiple
customers who are desperate for both of those features. With
extensions, I can give them changes that solve their immediate crisis
right now, almost a full year before they could possibly appear in a
proper release of PostgreSQL. And then I can push those back toward
community PostgreSQL, with any luck landing in the next major version.
Immediate gratification for the person funding development, and peer
reviewed code that goes through a long beta and release cycle. That's
the vision I have for a PostgreSQL that is simultaneously stable and
agile. The easiest way to get there it is to lead by example--by having
extensions that provide necessary, visible components to core, while
still being obviously separate components. That's the best approach for
proving this development model works and is suitable for everyone.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

On 11/18/2011 09:35 AM, Tom Lane wrote:
> Subdividing/rearranging contrib makes the packager's
> life more complicated, *and* makes his users' lives more complicated,
> if only because things aren't where they were before. It seems unlikely
> to happen, at least in the near term.
>

Users are visibly suffering by the current packaging. Production DBAs
are afraid to install contrib because it's described as untrustworthy.
They are hit by emergencies that the inspection tools would help with,
but cannot get contrib installed easily without root permissions. They
have performance issues that the contrib modules I'm trying to relocate
into the server package would help with, but company policies related to
post-deployment installation mean they cannot use them. They have to
always be installed to make this class of problem go away.

If you feel there are more users that would be negatively impacted by
some directories moving than what I'm describing above, we are a very
fundamental disagreement here. The status quote for all of these
extensions is widely understood to be unusable in common corporate
environments. Packagers should be trying to improve the PostgreSQL
experience, and I'm trying to help with that.

In the face of pushback from packagers, I can roll my own packages that
are designed without this problem; I'm being pushed into doing that
instead of working on community PostgreSQL already. But I'd really
prefer to see this very common problem identified as so important it
should get fixed everywhere instead.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

Greg Smith wrote:
> I've submitted two changes to this CommitFest that are enhancing
> features in this "core extensions" set. Right now I have multiple
> customers who are desperate for both of those features. With
> extensions, I can give them changes that solve their immediate crisis
> right now, almost a full year before they could possibly appear in a
> proper release of PostgreSQL. And then I can push those back toward
> community PostgreSQL, with any luck landing in the next major version.
> Immediate gratification for the person funding development, and peer
> reviewed code that goes through a long beta and release cycle. That's
> the vision I have for a PostgreSQL that is simultaneously stable and
> agile. The easiest way to get there it is to lead by example--by having
> extensions that provide necessary, visible components to core, while
> still being obviously separate components. That's the best approach for
> proving this development model works and is suitable for everyone.

I think a question is how often people are waiting for features that
actually can be addressed in a contrib/plugin way. My gut feeling is
that most missing features have to be added to the server core (e.g.
index-only scans) and are not possible to add in a contrib/plugin way.

I am not saying this would not help, but I am saying that this is going
to address only a small part of the goal of getting features to users
quicker.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

Peter Geoghegan wrote:
> On 17 November 2011 03:54, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> >?It's not reasonable to suppose
> > that nobody is using it today.
>
> I didn't suppose that no one is using it, but that those that are
> using it are unaware of the risks with prefix validation, and that
> there will be a rude awakening for them.
>
> > Ergo, we can't just summarily break
> > backwards compatibility on the grounds that we don't like the design.
> > Heck, we don't even have a field bug report that the design limitation
> > is causing any real problems for real users ... so IMO, the claims that
> > this is dangerously broken are a bit overblown.
>
> I think that's it's rather unlikely that removing hyphenation and
> prefix validation would adversely affect anyone, provided that it was
> well documented and wasn't applied to stable branches. If it were up
> to me, I might remove validation from stable branches but keep
> hyphenation, while removing both for 9.2 . After all, hyphenation will
> break anyway, so they're worse off continuing to rely on hyphenation
> when it cannot actually be relied on.

Clarification: Our policy for patching back-branches is that the bug
has to affect many users, be serious, and the fix has to be easily
tested.

For a user-visible change (which this would be), the criteria is even
more strict.

I don't see any of this reaching the level that it needs to be
backpatched, so I think we have to accept that this will be 9.2-only
change.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

Bruce,

> I don't see any of this reaching the level that it needs to be
> backpatched, so I think we have to accept that this will be 9.2-only
> change.

Agreed. If users encounter issues with the prefix in the field, it will be easy enough for them to back-patch. But we don't want to be responsible for it as a project.

--Josh

On 11/21/2011 11:40 AM, Bruce Momjian wrote:
> I think a question is how often people are waiting for features that
> actually can be addressed in a contrib/plugin way. My gut feeling is
> that most missing features have to be added to the server core (e.g.
> index-only scans) and are not possible to add in a contrib/plugin way.
>

Good question; let's talk about 9.0. We were building/distributing
three things for that version that poked into the server:

-Replication monitoring tools that slipped from the 9.0 schedule,
similar to what became pg_stat_replication in 9.1
-An early version of what became hot_standby_feedback in 9.1.
-pg_streamrecv

While these weren't all packaged as extensions per se, all of them used
the PGXS interface. And they all provided deployment blocking features
to early adopters before those features were available in core, in some
cases after the issues they address had been encountered in production
deployments. As I was ranting on my blog recently, I'm seeing more
complaints recently about monitoring and management features--exactly
the sort of thing that you can improve as an extension, and that the
extensions I've proposed provide--than I am over missing big features.

Index-only scans are a good example, as one of the most requested
performance feature you can only get in core (I'd put them at #2 behind
materialized views for the customers I talk to). I wouldn't bet that
they are considered more important by a typical deployment than good
built-in query profiling though. I get complaints about query
monitoring from every single PostgreSQL install, while complaints about
not having index-only scans only come from the bigger installations.
Note how demand is high enough that we have two pg_stat_statements
replacements submitted right now.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

Greg Smith wrote:
> On 11/21/2011 11:40 AM, Bruce Momjian wrote:
> > I think a question is how often people are waiting for features that
> > actually can be addressed in a contrib/plugin way. My gut feeling is
> > that most missing features have to be added to the server core (e.g.
> > index-only scans) and are not possible to add in a contrib/plugin way.
> >
>
> Good question; let's talk about 9.0. We were building/distributing
> three things for that version that poked into the server:
>
> -Replication monitoring tools that slipped from the 9.0 schedule,
> similar to what became pg_stat_replication in 9.1
> -An early version of what became hot_standby_feedback in 9.1.
> -pg_streamrecv
>
> While these weren't all packaged as extensions per se, all of them used
> the PGXS interface. And they all provided deployment blocking features
> to early adopters before those features were available in core, in some
> cases after the issues they address had been encountered in production
> deployments. As I was ranting on my blog recently, I'm seeing more
> complaints recently about monitoring and management features--exactly
> the sort of thing that you can improve as an extension, and that the
> extensions I've proposed provide--than I am over missing big features.
>
> Index-only scans are a good example, as one of the most requested
> performance feature you can only get in core (I'd put them at #2 behind
> materialized views for the customers I talk to). I wouldn't bet that
> they are considered more important by a typical deployment than good
> built-in query profiling though. I get complaints about query
> monitoring from every single PostgreSQL install, while complaints about
> not having index-only scans only come from the bigger installations.
> Note how demand is high enough that we have two pg_stat_statements
> replacements submitted right now.

Agreed much of the edge stuff, e.g. monitoring, can be done as plugins.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On Fri, Nov 18, 2011 at 9:35 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Why do you figure that, exactly?  The path of least resistance will
> be precisely to leave everything packaged as it is, in a single
> postgresql-contrib module.  I'm pretty likely to do that myself for
> Fedora and RHEL.  Subdividing/rearranging contrib makes the packager's
> life more complicated, *and* makes his users' lives more complicated,
> if only because things aren't where they were before.  It seems unlikely
> to happen, at least in the near term.

When we discussed this topic at the developer's meeting, I thought we
had general consensus that it would be a good idea to package a
limited number of important and stable debugging tools with the core
server, and I had the impression that you (Tom) thought this was a
reasonable thing to do. If you don't, or if you did then but don't
now, then it seems to me that this conversation is dead in the water
for so long as you're the one packaging for Red Hat, and we should
just move on; you pretty much have unassailable personal veto power on
this issue. But let's not pretend that the conversation is about what
packagers in general will do, because I don't think it is. I think
it's about what you personally will do.

I think that if we move a few things into src/extension and set things
up such that they get installed even if you just do "make install"
rather than requiring "make install-world", packagers who don't have
any terribly strong personal agenda will decide that means they ought
to be shipped with the server. However, if you're personally
committed to making sure that all of that stuff remains in
postgresql-contrib in Red Hat/Fedora, regardless of where we move it
to on our end, then that's where it's going to be, at least on all Red
Hat-derived systems, which is a big enough chunk of the world to
matter quite a lot. Note that I'm not necessarily saying anything
about whether your reasons for such a decision might be good or bad;
I'm just pointing out that a good deal of our ability to make a change
in this area is within your personal control.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Tom,

> I think that if we move a few things into src/extension and set things
> up such that they get installed even if you just do "make install"
> rather than requiring "make install-world", packagers who don't have
> any terribly strong personal agenda will decide that means they ought
> to be shipped with the server. However, if you're personally
> committed to making sure that all of that stuff remains in
> postgresql-contrib in Red Hat/Fedora, regardless of where we move it
> to on our end, then that's where it's going to be, at least on all Red
> Hat-derived systems, which is a big enough chunk of the world to
> matter quite a lot. Note that I'm not necessarily saying anything
> about whether your reasons for such a decision might be good or bad;
> I'm just pointing out that a good deal of our ability to make a change
> in this area is within your personal control.

Any response to this before I take it to the other packagers?

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

All,

This is currently awaiting a check by gsmith that the 7 named extensions
do not add any new dependancies.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Greg,

> This is currently awaiting a check by gsmith that the 7 named extensions
> do not add any new dependancies.

Are you going to investigate this? If not, I'll give it a try this weekend.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

All,

Andrew ran crake on these modules, and they do not add any links not
added by core postgres already.

As such, can we proceed with this patch? Greg, do you have an updated
version to run against HEAD?

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com