Re: Why not install pgstattuple by default?

Hackers,

I've run into a couple of occasions lately where I really wanted
pgstattuple on a production server in order to check table/index bloat.
However, in the production environment at a large site installing a
contrib module can involve a process which takes days or weeks.

Is there some reason why the stattuple functions aren't just available
as core functions? Are they unsafe somehow?

--
-- Josh Berkus
---------------------------------------------------------
Josh Berkus PostgreSQL Experts Inc.
CEO database professionals
josh(dot)berkus(at)pgexperts(dot)com www.pgexperts.com
1-888-743-9778 x.508 San Francisco

On Fri, May 6, 2011 at 00:34, Josh Berkus <josh(dot)berkus(at)pgexperts(dot)com> wrote:
> Hackers,
>
> I've run into a couple of occasions lately where I really wanted
> pgstattuple on a production server in order to check table/index bloat.
>  However, in the production environment at a large site installing a
> contrib module can involve a process which takes days or weeks.

That can be said for a lot of things in contrib. pg_standby in 8.4 for
example. Or adminpack. Or dblink. Or hstore. There's a mix of "example
stuff" and "actually pretty darn useful in production stuff". I'm sure
you can find a couple of hundred emails in the archives on this very
topic.

From 9.1, it'll be a simple CREATE EXTENSION command - so much of the
problem goes away. Well. It doesn't go away, but it gets a lot more
neatly swept under the rug.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Em 06-05-2011 05:06, Magnus Hagander escreveu:
> On Fri, May 6, 2011 at 00:34, Josh Berkus<josh(dot)berkus(at)pgexperts(dot)com> wrote:
>> Hackers,
>>
>> I've run into a couple of occasions lately where I really wanted
>> pgstattuple on a production server in order to check table/index bloat.
>> However, in the production environment at a large site installing a
>> contrib module can involve a process which takes days or weeks.
>
I already faced that problem too.

> From 9.1, it'll be a simple CREATE EXTENSION command - so much of the
> problem goes away. Well. It doesn't go away, but it gets a lot more
> neatly swept under the rug.
>
That's half of the history. Admin needs to install postgresql-contrib package.
Sometimes it takes too much time to convince clients that some additional
supplied modules are useful for them.

Now that we have extensions, why not build and package the contrib modules by
default? 'make world' is not the answer. There is not an option for "install
all pieces of software". Let's install pg+contrib and leave only 'CREATE
EXTENSION foo' for the admins.

--
Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

On Fri, May 6, 2011 at 18:22, Euler Taveira de Oliveira
<euler(at)timbira(dot)com> wrote:
> Em 06-05-2011 05:06, Magnus Hagander escreveu:
>>
>> On Fri, May 6, 2011 at 00:34, Josh Berkus<josh(dot)berkus(at)pgexperts(dot)com>
>>  wrote:
>>>
>>> Hackers,
>>>
>>> I've run into a couple of occasions lately where I really wanted
>>> pgstattuple on a production server in order to check table/index bloat.
>>>  However, in the production environment at a large site installing a
>>> contrib module can involve a process which takes days or weeks.
>>
> I already faced that problem too.
>
>>  From 9.1, it'll be a simple CREATE EXTENSION command - so much of the
>> problem goes away. Well. It doesn't go away, but it gets a lot more
>> neatly swept under the rug.
>>
> That's half of the history. Admin needs to install postgresql-contrib
> package. Sometimes it takes too much time to convince clients that some
> additional supplied modules are useful for them.
>
> Now that we have extensions, why not build and package the contrib modules
> by default? 'make world' is not the answer. There is not an option for
> "install all pieces of software". Let's install pg+contrib and leave only
> 'CREATE EXTENSION foo' for the admins.

That's mostly an issue to be solved by the packagers. Some contrib
modules add dependencies, but those that don't could easily be
packaged in the main server package.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Fri, May 6, 2011 at 1:32 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> On Fri, May 6, 2011 at 18:22, Euler Taveira de Oliveira
> <euler(at)timbira(dot)com> wrote:
>> Em 06-05-2011 05:06, Magnus Hagander escreveu:
>>>
>>> On Fri, May 6, 2011 at 00:34, Josh Berkus<josh(dot)berkus(at)pgexperts(dot)com>
>>>  wrote:
>>>>
>>>> Hackers,
>>>>
>>>> I've run into a couple of occasions lately where I really wanted
>>>> pgstattuple on a production server in order to check table/index bloat.
>>>>  However, in the production environment at a large site installing a
>>>> contrib module can involve a process which takes days or weeks.
>>>
>> I already faced that problem too.
>>
>>>  From 9.1, it'll be a simple CREATE EXTENSION command - so much of the
>>> problem goes away. Well. It doesn't go away, but it gets a lot more
>>> neatly swept under the rug.
>>>
>> That's half of the history. Admin needs to install postgresql-contrib
>> package. Sometimes it takes too much time to convince clients that some
>> additional supplied modules are useful for them.
>>
>> Now that we have extensions, why not build and package the contrib modules
>> by default? 'make world' is not the answer. There is not an option for
>> "install all pieces of software". Let's install pg+contrib and leave only
>> 'CREATE EXTENSION foo' for the admins.
>
> That's mostly an issue to be solved by the packagers. Some contrib
> modules add dependencies, but those that don't could easily be
> packaged in the main server package.

It seems to me that there's something of a "packaging policy" question to this.

A long time ago, on a pre-buildfarm planet, far, far away, it was
pretty uncertain what contrib modules could be hoped to run on what
platform.

At Afilias, we used to have to be *really* picky, because the subset
that ran on Solaris and AIX were not even close to all of them.
pgstattuples *was* one that the DBAs always wanted, but what would
compile was alway hit-and-miss.

Once we got AIX running a buildfarm node, that led to getting *ALL* of
contrib working there, and I'm pretty sure that similar happened with
other platforms at around the same time (I'm thinking this was 7.4,
but it might have been 8.0)

Be that all as it may, there has been a "sea change", where we have
moved from "sporadic usability of contrib" to it being *continually*
tested on *all* buildfarm platforms, which certainly adds to the
confidence level.

But people are evidently still setting packaging policies based on how
things were back in 7.3, even though that perhaps isn't necessary
anymore.

Certainly it's not a huge amount of code; less than 2MB these days.
-> % wc `dpkg -L postgresql-contrib-9.0` | tail -1
15952 67555 1770987 total

I'm getting "paper cuts" quite a bit these days over the differences
between what different packaging systems decide to install. The one
*I* get notably bit on, of late, is that I have written code that
expects to have pg_config to do some degree of self-discovery, only to
find production folk complaining that they only have "psql" available
in their environment.

I don't expect the extension system to help with any of this, since if
"production folk" try to install minimal sets of packages, they're
liable to consciously exclude extension support. The "improvement"
would come from drawing contrib a bit closer to core, and encouraging
packagers (dpkg, rpm, ports) to fold contrib into "base" rather than
separating it. I'm sure that would get some pushback, though.
--
When confronted by a difficult problem, solve it by reducing it to the
question, "How would the Lone Ranger handle this?"

On 05/06/2011 01:55 PM, Christopher Browne wrote:
>
> Once we got AIX running a buildfarm node, that led to getting *ALL* of
> contrib working there, and I'm pretty sure that similar happened with
> other platforms at around the same time (I'm thinking this was 7.4,
> but it might have been 8.0)

FYI, the buildfarm started in late 2004, near the end of the 8.0
development cycle. It quickly led to a number of contrib fixes.

Time flies when you're having fun ...

cheers

andrew

Em 06-05-2011 14:55, Christopher Browne escreveu:
> The "improvement"
> would come from drawing contrib a bit closer to core, and encouraging
> packagers (dpkg, rpm, ports) to fold contrib into "base" rather than
> separating it. I'm sure that would get some pushback, though.
>
I'm in favor of find out what are the popular extensions and make them into
"base"; the other ones could be moved to PGXN.

--
Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

Christopher Browne wrote:
> I'm getting "paper cuts" quite a bit these days over the differences
> between what different packaging systems decide to install. The one
> *I* get notably bit on, of late, is that I have written code that
> expects to have pg_config to do some degree of self-discovery, only to
> find production folk complaining that they only have "psql" available
> in their environment.
>

Given the other improvements in being able to build extensions in 9.1,
we really should push packagers to move pg_config from the PostgreSQL
development package into the main one starting in that version. I've
gotten bit by this plenty of times.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

On Fri, May 6, 2011 at 2:32 PM, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:
> Christopher Browne wrote:
>>
>> I'm getting "paper cuts" quite a bit these days over the differences
>> between what different packaging systems decide to install.  The one
>> *I* get notably bit on, of late, is that I have written code that
>> expects to have pg_config to do some degree of self-discovery, only to
>> find production folk complaining that they only have "psql" available
>> in their environment.
>
> Given the other improvements in being able to build extensions in 9.1, we
> really should push packagers to move pg_config from the PostgreSQL
> development package into the main one starting in that version.  I've gotten
> bit by this plenty of times.

I'm agreeable to that, in general.

If there's a "server" package and a "client" package, it likely only
fits with the "server" package. On a host where only the "client" is
installed, they won't be able to install extensions, so it's pretty
futile to have it there.
--
When confronted by a difficult problem, solve it by reducing it to the
question, "How would the Lone Ranger handle this?"

On 05/06/2011 03:14 PM, Christopher Browne wrote:
> On Fri, May 6, 2011 at 2:32 PM, Greg Smith<greg(at)2ndquadrant(dot)com> wrote:
>> Christopher Browne wrote:
>>> I'm getting "paper cuts" quite a bit these days over the differences
>>> between what different packaging systems decide to install. The one
>>> *I* get notably bit on, of late, is that I have written code that
>>> expects to have pg_config to do some degree of self-discovery, only to
>>> find production folk complaining that they only have "psql" available
>>> in their environment.
>> Given the other improvements in being able to build extensions in 9.1, we
>> really should push packagers to move pg_config from the PostgreSQL
>> development package into the main one starting in that version. I've gotten
>> bit by this plenty of times.
> I'm agreeable to that, in general.
>
> If there's a "server" package and a "client" package, it likely only
> fits with the "server" package. On a host where only the "client" is
> installed, they won't be able to install extensions, so it's pretty
> futile to have it there.

I don't agree. It can be useful even there, to see how the libraries are
configured, for example. I'd be inclined to bundle it with
postgresql-libs or the moral equivalent.

cheers

andrew

On Fri, May 6, 2011 at 21:19, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>
>
> On 05/06/2011 03:14 PM, Christopher Browne wrote:
>>
>> On Fri, May 6, 2011 at 2:32 PM, Greg Smith<greg(at)2ndquadrant(dot)com>  wrote:
>>>
>>> Christopher Browne wrote:
>>>>
>>>> I'm getting "paper cuts" quite a bit these days over the differences
>>>> between what different packaging systems decide to install.  The one
>>>> *I* get notably bit on, of late, is that I have written code that
>>>> expects to have pg_config to do some degree of self-discovery, only to
>>>> find production folk complaining that they only have "psql" available
>>>> in their environment.
>>>
>>> Given the other improvements in being able to build extensions in 9.1, we
>>> really should push packagers to move pg_config from the PostgreSQL
>>> development package into the main one starting in that version.  I've
>>> gotten
>>> bit by this plenty of times.
>>
>> I'm agreeable to that, in general.
>>
>> If there's a "server" package and a "client" package, it likely only
>> fits with the "server" package.  On a host where only the "client" is
>> installed, they won't be able to install extensions, so it's pretty
>> futile to have it there.
>
> I don't agree. It can be useful even there, to see how the libraries are
> configured, for example. I'd be inclined to bundle it with postgresql-libs
> or the moral equivalent.

+1.

And it's not like it wastes huge amount of space...

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> On Fri, May 6, 2011 at 21:19, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>> On 05/06/2011 03:14 PM, Christopher Browne wrote:
>>> If there's a "server" package and a "client" package, it likely only
>>> fits with the "server" package. On a host where only the "client" is
>>> installed, they won't be able to install extensions, so it's pretty
>>> futile to have it there.

>> I don't agree. It can be useful even there, to see how the libraries are
>> configured, for example. I'd be inclined to bundle it with postgresql-libs
>> or the moral equivalent.

> +1.

Well, actually, I think packagers have generally put it into a -devel
subpackage. If it were in either a "server" or "client" package there
would be much less of an issue.

Bundling pg_config into a -libs package is probably not going to happen,
at least not on Red Hat systems, because it would create multilib issues
(ie, you're supposed to be able to install 32-bit and 64-bit libraries
concurrently, but there's noplace to put a /usr/bin file without causing
a conflict).

FWIW, I did move pg_config from -devel to the "main" (really client)
postgresql package in Fedora, as of 9.0. That will ensure it's present
in either client or server installations. Eventually that packaging
will reach RHEL ...

regards, tom lane

On 05/06/2011 04:06 PM, Tom Lane wrote:
> Magnus Hagander<magnus(at)hagander(dot)net> writes:
>> On Fri, May 6, 2011 at 21:19, Andrew Dunstan<andrew(at)dunslane(dot)net> wrote:
>>> On 05/06/2011 03:14 PM, Christopher Browne wrote:
>>>> If there's a "server" package and a "client" package, it likely only
>>>> fits with the "server" package. On a host where only the "client" is
>>>> installed, they won't be able to install extensions, so it's pretty
>>>> futile to have it there.
>>> I don't agree. It can be useful even there, to see how the libraries are
>>> configured, for example. I'd be inclined to bundle it with postgresql-libs
>>> or the moral equivalent.
>> +1.
> Well, actually, I think packagers have generally put it into a -devel
> subpackage. If it were in either a "server" or "client" package there
> would be much less of an issue.
>
> Bundling pg_config into a -libs package is probably not going to happen,
> at least not on Red Hat systems, because it would create multilib issues
> (ie, you're supposed to be able to install 32-bit and 64-bit libraries
> concurrently, but there's noplace to put a /usr/bin file without causing
> a conflict).
>
> FWIW, I did move pg_config from -devel to the "main" (really client)
> postgresql package in Fedora, as of 9.0. That will ensure it's present
> in either client or server installations. Eventually that packaging
> will reach RHEL ...
>
>

That's reasonable, and certainly better than having it in -devel.

cheers

andrew

Christopher Browne <cbbrowne(at)gmail(dot)com> writes:
> But people are evidently still setting packaging policies based on how
> things were back in 7.3, even though that perhaps isn't necessary
> anymore.

FWIW, once you get past the client versus server distinction, I think
most subpackaging decisions are based on either the idea that "only a
minority of people will want this", or a desire to limit how many
dependencies are pulled in by the main package(s). Both of those
concerns apply to various subsets of -contrib, which means it's going
to be hard to persuade packagers to fold -contrib into the -server
package altogether. Nor would you gain their approval by trying to
pre-empt the decision.

We might get somewhere by trying to identify a small set of particularly
popular contrib modules that don't add any extra dependencies, and then
recommending to packagers that those ones get bundled into the main
server package.

> Certainly it's not a huge amount of code; less than 2MB these days.
> -> % wc `dpkg -L postgresql-contrib-9.0` | tail -1
> 15952 67555 1770987 total

Well, to add some concrete facts rather than generalities to my own post,
here are the sizes of the built RPMs from my last build for Fedora:

-rw-r--r--. 1 tgl tgl 3839458 Apr 18 10:50 postgresql-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl 490788 Apr 18 10:50 postgresql-contrib-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl 27337677 Apr 18 10:51 postgresql-debuginfo-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl 961660 Apr 18 10:50 postgresql-devel-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl 7569048 Apr 18 10:50 postgresql-docs-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl 246506 Apr 18 10:50 postgresql-libs-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl 64940 Apr 18 10:50 postgresql-plperl-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl 65776 Apr 18 10:50 postgresql-plpython-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl 45941 Apr 18 10:50 postgresql-pltcl-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl 5302117 Apr 18 10:50 postgresql-server-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl 1370509 Apr 18 10:50 postgresql-test-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl 3644113 Apr 18 10:50 postgresql-upgrade-9.0.4-1.fc13.x86_64.rpm

The separate debuginfo package is distro policy enforced by toolchain;
I couldn't do anything about that even if I wanted to. The separate
-libs subpackage is also hard to avoid because of distro policy about
multilib installations. Separating devel support files (such as
headers) is also standard practice. The other subdivisions are either
my fault or those of my predecessors. plperl, plpython, and pltcl are
split out for dependency reasons, ie to not have the -server package
require you to install those languages and their respective ecosystems.
I think the separation of the -docs, -test, and -upgrade subpackages is
also pretty easy to defend on the grounds that "they're big and not
everyone wants 'em, especially not in production".

That leaves us with these three subpackages about which there's room
for argument:

-rw-r--r--. 1 tgl tgl 3839458 Apr 18 10:50 postgresql-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl 490788 Apr 18 10:50 postgresql-contrib-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl 5302117 Apr 18 10:50 postgresql-server-9.0.4-1.fc13.x86_64.rpm

Merging -contrib into the server package would increase the size of the
latter by almost 10%, which is enough to bother people. Also, a bit of
dependency extraction shows that -contrib has these dependencies beyond
the ones in the two main packages:

libcrypt.so.1
libossp-uuid.so.16
libxslt.so.1

That's not a particularly large list, I guess, but they're still the
sorts of dependencies that don't win any friends when it's time to get
the distro to fit on a DVD.

Bottom line is that I'd rather have a smaller postgresql-server package
that gets included in the shipping DVD than a complete one that gets
kicked off because it's too large and pulls in too many other non-core
dependencies.

So, again, some selective migration of contrib modules into the main
-server package might be doable, but the key word there is selective.

regards, tom lane

All,

> We might get somewhere by trying to identify a small set of particularly
> popular contrib modules that don't add any extra dependencies, and then
> recommending to packagers that those ones get bundled into the main
> server package.

Yeah, I wasn't thinking of including all of contrib. There's a lot of
reasons not to do that. I was asking about pgstattuple in particular,
since it's:
(a) small
(b) has no external dependancies
(c) adds no stability risk or performance overhead
(d) is usually needed on production systems when it's needed at all

It's possible that we have one or two other diagnostic utilities which
meet the above profile. pageinspect, maybe?

The reason why this is such an issue is that for big users with
high-demand production environments, installing any software, even
postgresql-devel or postgresql-contrib packages, are big major IT deals
which require weeks of advance scheduling. As a result, diagnostic
tools from contrib tend not to be used because the problem they need to
diagnose is much more urgent than that.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On Fri, May 6, 2011 at 5:58 PM, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
> Yeah, I wasn't thinking of including all of contrib.  There's a lot of
> reasons not to do that.

Slightly off-topic, but I really think we would benefit from trying to
divide up contrib. Right now it's a mixture of (a) debugging and
instrumentation tools (e.g. pgstattuple, pageinspect, pgrowlocks,
pg_freespacemap, pg_buffercache), (b) server functionality that is
generally useful but considered worth including in core (e.g. hstore,
citext, pg_trgm), (c) deprecated modules that we keep around mostly
for hysterical reasons (tsearch2, xml2, intagg), and (d) examples and
regression test support (dummy_seclabel, spi, start-scripts). I think
it would make things a lot easier for both packagers and actual users
if we separated these things into different directories, e.g.:

debugging and instrumentation tools -> src/debug
server functionality -> contrib
server functionality (deprecated) -> contrib/deprecated
examples & regression test suport -> src/test/examples

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 05/06/2011 05:58 PM, Josh Berkus wrote:
> Yeah, I wasn't thinking of including all of contrib. There's a lot of
> reasons not to do that. I was asking about pgstattuple in particular,
> since it's:
> (a) small
> (b) has no external dependancies
> (c) adds no stability risk or performance overhead
> (d) is usually needed on production systems when it's needed at all
>
> It's possible that we have one or two other diagnostic utilities which
> meet the above profile. pageinspect, maybe?
>

I use pgstattuple, pageinspect, pg_freespacemap, and pg_buffercache
regularly enough that I wish they were more common. Throw in pgrowlocks
and you've got the whole group Robert put into the debug set. It makes
me sad every time I finish a utility using one of these and realize I'll
have to include the whole "make sure you have the contrib modules
installed" disclaimer in its documentation again.

These are the only ones I'd care about moving into a more likely place.
The rest of the contrib modules are the sort where if you need them, you
realize that early and get them installed. These are different by
virtue of their need popping up most often during emergencies. The fact
that I believe they all match the low impact criteria too makes it even
easier to consider.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> On Fri, May 6, 2011 at 5:58 PM, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
>> Yeah, I wasn't thinking of including all of contrib. There's a lot of
>> reasons not to do that.

> Slightly off-topic, but I really think we would benefit from trying to
> divide up contrib. [ snip ]
> I think
> it would make things a lot easier for both packagers and actual users
> if we separated these things into different directories, e.g.:

> debugging and instrumentation tools -> src/debug
> server functionality -> contrib
> server functionality (deprecated) -> contrib/deprecated
> examples & regression test suport -> src/test/examples

From a packager's standpoint, that would be entirely worthless. The
source tree's just a source tree, they don't care what lives where
within it. I was just thinking about what it'd take to actually
repackage things for Fedora, and the main problem is here:

%files contrib
...
%{_datadir}/pgsql/contrib/
...

If you're not adept at reading RPM specfiles, what that is saying
is that everything that "make install" has stuck under
${prefix}/share/pgsql/contrib/ is to be included in the contrib RPM.
To selectively move some stuff to the server RPM, I'd have to replace
that one line with a file-by-file list of *everything* in share/contrib,
and then move some of those lines to the "%files server" section, and
then look forward to having to maintain that list in future versions.
I'm already maintaining a file-by-file list of contrib's .so's, and I
can tell you it's a PITA.

As a packager, what I'd really want to see from a division into
recommended and not-so-recommended packages is that they get installed
into different subdirectories by "make install". Then I could just
point RPM at those directories and I'd be done.

I don't know how practical this is from our development standpoint,
nor from a user's standpoint --- I doubt we want to ask people to use
different CREATE EXTENSION commands depending on the preferredness of
the extension.

A possibly workable compromise would be to provide two separate makefile
installation targets for preferred and less preferred modules. The RPM
script could then do something like

make install-contrib-preferred
ls -R .../sharedir >contrib.files.for.server-package
make install-contrib-second-class-citizens
ls -R .../sharedir >all.contrib.files
... and then some magic with "comm" to separate out the contrib
... files not mentioned in contrib.files.for.server-package ...

Pretty grotty but it would work. Anyway my point is that this is all
driven off the *installed* file tree. A specfile writer doesn't know
nor want to know where "make install" is getting things from in the
source tree.

regards, tom lane

On 5/6/11 3:19 PM, Robert Haas wrote:
> Slightly off-topic, but I really think we would benefit from trying to
> divide up contrib.

I don't agree, unless by "divide up" you mean "move several things to
extensions".

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

> These are the only ones I'd care about moving into a more likely place.
> The rest of the contrib modules are the sort where if you need them, you
> realize that early and get them installed. These are different by
> virtue of their need popping up most often during emergencies. The fact
> that I believe they all match the low impact criteria too makes it even
> easier to consider.

Yes, precisely. If I need intarray, I'm going to need it for a
development push, which is planned well in advance. But if I need
pageinspect, it's almost certainly because an emergency has arisen.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On Fri, May 6, 2011 at 11:32 PM, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:
> I use pgstattuple, pageinspect, pg_freespacemap, and pg_buffercache
> regularly enough that I wish they were more common.  Throw in pgrowlocks and
> you've got the whole group Robert put into the debug set.  It makes me sad
> every time I finish a utility using one of these and realize I'll have to
> include the whole "make sure you have the contrib modules installed"
> disclaimer in its documentation again.

Well the lightweight way to achieve what you want is to just move
these functions into core. There's a pretty good argument to be made
for debugging tools being considered an integral part of a base
system. I remember making the same argument when Sun first made the
radical move for a Unix vendor to stop shipping a working C compiler
and debugger as part of the base Solaris packages.

The only argument I see as particularly frightening on that front is
people playing the sekurity card. A naive attacker who obtains access
to the postgres account could do more damage than they might be able
to do without these modules installed. Of course an attacker with
"postgres" can do just about anything but it's not entirely baseless
-- we don't set up the database with modules like plsh installed by
default for example.

The only actual security issue I can think of is that the pageinspect
module would let users look at deleted records more easily. It would
be pretty tricky, but not impossible, to do that without it.

--
greg

On Fri, May 6, 2011 at 6:47 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> As a packager, what I'd really want to see from a division into
> recommended and not-so-recommended packages is that they get installed
> into different subdirectories by "make install".  Then I could just
> point RPM at those directories and I'd be done.

Well, that might be good, too. But, right now, if someone pulls up
our documentation, or our source tree, they could easily be forgiven
for thinking that hstore and dummy_seclabel are comparable, and they
aren't.

> I don't know how practical this is from our development standpoint,
> nor from a user's standpoint --- I doubt we want to ask people to use
> different CREATE EXTENSION commands depending on the preferredness of
> the extension.

Certainly not.

> A possibly workable compromise would be to provide two separate makefile
> installation targets for preferred and less preferred modules.  The RPM
> script could then do something like
>
>        make install-contrib-preferred
>        ls -R .../sharedir >contrib.files.for.server-package
>        make install-contrib-second-class-citizens
>        ls -R .../sharedir >all.contrib.files
>        ... and then some magic with "comm" to separate out the contrib
>        ... files not mentioned in contrib.files.for.server-package ...
>
> Pretty grotty but it would work.  Anyway my point is that this is all
> driven off the *installed* file tree.  A specfile writer doesn't know
> nor want to know where "make install" is getting things from in the
> source tree.

This isn't any uglier than some other RPM hacks I've seen, and less
ugly than some, but you'd have a better sense of that than I do. At
any rate, having the various categories separated in the source tree
can't possibly hurt the effort to make something like this work, and
might make it somewhat easier.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> On Fri, May 6, 2011 at 6:47 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> As a packager, what I'd really want to see from a division into
>> recommended and not-so-recommended packages is that they get installed
>> into different subdirectories by "make install".

> Well, that might be good, too. But, right now, if someone pulls up
> our documentation, or our source tree, they could easily be forgiven
> for thinking that hstore and dummy_seclabel are comparable, and they
> aren't.

Sure, but that's a documentation issue, which again is not going to be
helped by a source-tree rearrangement.

As somebody who spends a lot of time on back-patching, I'm not excited
in the least by suggestions to rearrange the source tree for marginal
cosmetic benefits, which is all that I see here.

regards, tom lane

On Fri, May 6, 2011 at 9:17 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> On Fri, May 6, 2011 at 6:47 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> As a packager, what I'd really want to see from a division into
>>> recommended and not-so-recommended packages is that they get installed
>>> into different subdirectories by "make install".
>
>> Well, that might be good, too.  But, right now, if someone pulls up
>> our documentation, or our source tree, they could easily be forgiven
>> for thinking that hstore and dummy_seclabel are comparable, and they
>> aren't.
>
> Sure, but that's a documentation issue, which again is not going to be
> helped by a source-tree rearrangement.

I disagree - I think it would be helpful to rearrange both things.

> As somebody who spends a lot of time on back-patching, I'm not excited
> in the least by suggestions to rearrange the source tree for marginal
> cosmetic benefits, which is all that I see here.

I understand, but we have back-patched only 32 patches that touch
contrib into REL9_0_STABLE since its creation, of which 9 were done by
you, and only 4 of those would have required adjustment under the
separation criteria I proposed. I think, therefore, that the impact
would be bearable. Source-code rearrangement is never going to be
completely free, but that seems like a tolerable level of annoyance.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Christopher Browne <cbbrowne(at)gmail(dot)com> writes:
> I don't expect the extension system to help with any of this, since if
> "production folk" try to install minimal sets of packages, they're
> liable to consciously exclude extension support. The "improvement"
> would come from drawing contrib a bit closer to core, and encouraging
> packagers (dpkg, rpm, ports) to fold contrib into "base" rather than
> separating it. I'm sure that would get some pushback, though.

What I think the next step is here is to better classify contribs/ into
what we consider production ready core extension (adminpack, hstore,
ltree, pgstattuple, you name it — that's the trick), code example (spi,
some more I guess) and extensibility examples (for hooks or whatnot).

We've been talking about renaming contrib for a long time, but that will
not cut it. Classifying it and agreeing to maintain some parts of it
the same way we maintain the core is what's asked here. Is there a will
to go there?

If there's a will to maintain some contribs the way core is maintained
itself, we have to pick a new name for that, and to pick a list of
current contribs to move in there. Then packagers will either include
that in the main package or have the main package depend on the new one.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

On Sat, May 7, 2011 at 8:54 AM, Dimitri Fontaine <dimitri(at)2ndquadrant(dot)fr> wrote:
> We've been talking about renaming contrib for a long time, but that will
> not cut it.  Classifying it and agreeing to maintain some parts of it
> the same way we maintain the core is what's asked here.  Is there a will
> to go there?

I'm game. I doubt it'll be a lot more maintenance; we already pretty
much patch contrib when we patch everything else. It'll just be
easier to sort the wheat from the chaff.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Fri, 06 May 2011 20:06:04 -0000, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> Bundling pg_config into a -libs package is probably not going to happen,
> at least not on Red Hat systems, because it would create multilib issues
> (ie, you're supposed to be able to install 32-bit and 64-bit libraries
> concurrently, but there's noplace to put a /usr/bin file without causing
> a conflict).

There is no separate directory for 64bit binaries like /usr/bin and
/usr/bin/amd64 on Solaris? And even in the absence of such a convention
I'd still like to have both 32bit and 64bit binaries of the server/client
available.

I have a feeling this topic is digressing a bit.

--
Johann Oskarsson http://www.2ndquadrant.com/ |[]
PostgreSQL Development, 24x7 Support, Training and Services --+--
|
Blog: http://my.opera.com/myrkraverk/blog/

Greg Stark <gsstark(at)mit(dot)edu> writes:
> On Fri, May 6, 2011 at 11:32 PM, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:
>> I use pgstattuple, pageinspect, pg_freespacemap, and pg_buffercache
>> regularly enough that I wish they were more common. Throw in pgrowlocks and
>> you've got the whole group Robert put into the debug set. It makes me sad
>> every time I finish a utility using one of these and realize I'll have to
>> include the whole "make sure you have the contrib modules installed"
>> disclaimer in its documentation again.

> Well the lightweight way to achieve what you want is to just move
> these functions into core.

I'm completely not in favor of that. We have spent man-years upon
man-years on making Postgres an extensible system. If we can't actually
*use* the extension features then that was all a waste of effort.

If anything I'd rather see us looking at what parts of the current core
system could be pushed out to extensions. The geometric types are a
pretty obvious candidate, for example.

> The only argument I see as particularly frightening on that front is
> people playing the sekurity card.

Yeah, and it's a reasonable argument. Even if it's not a reasonable
argument, you won't win any friends from the other side of the fence
by taking away their ability to choose.

regards, tom lane

On fre, 2011-05-06 at 14:32 -0400, Greg Smith wrote:
> Given the other improvements in being able to build extensions in 9.1,
> we really should push packagers to move pg_config from the PostgreSQL
> development package into the main one starting in that version. I've
> gotten bit by this plenty of times.

Do you need pg_config to install extensions?

Em 07-05-2011 13:42, Peter Eisentraut escreveu:
> Do you need pg_config to install extensions?
>
No. But we need it to build other extensions.

--
Euler Taveira de Oliveira - Timbira http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

On lör, 2011-05-07 at 17:35 -0300, Euler Taveira de Oliveira wrote:
> Em 07-05-2011 13:42, Peter Eisentraut escreveu:
> > Do you need pg_config to install extensions?
> >
> No. But we need it to build other extensions.

But for that you need the -dev[el] package anyway, so there would be no
point in moving pg_config out of it.

On 05/07/2011 12:42 PM, Peter Eisentraut wrote:
> On fre, 2011-05-06 at 14:32 -0400, Greg Smith wrote:
>
>> Given the other improvements in being able to build extensions in 9.1,
>> we really should push packagers to move pg_config from the PostgreSQL
>> development package into the main one starting in that version. I've
>> gotten bit by this plenty of times.
>>
> Do you need pg_config to install extensions?
>

No, but you still need it to build them. PGXN is a source code
distribution method, not a binary one. It presumes users can build
modules they download using PGXS. No pg_config, no working PGXS, no
working PGXN. For such a small binary to ripple out to that impact is bad.

The repmgr program we distribute has the same problem, so I've been
getting first-hand reports of just how many people are likely to run
into this recently. You have to install postgresql-devel with RPM and
on Debian, the very non-obvious postgresql-server-dev-$version

Anyway, didn't want to hijack this thread beyond pointing out that if
there any package reshuffling that happens for contrib changes, it
should check for and resolve this problem too.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

On 05/07/2011 04:43 PM, Peter Eisentraut wrote:
> On lör, 2011-05-07 at 17:35 -0300, Euler Taveira de Oliveira wrote:
>> Em 07-05-2011 13:42, Peter Eisentraut escreveu:
>>> Do you need pg_config to install extensions?
>>>
>> No. But we need it to build other extensions.
> But for that you need the -dev[el] package anyway, so there would be no
> point in moving pg_config out of it.
>

pg_config is useful quite apart from its use in building things, as was
discussed upthread.

cheers

andrew

On lör, 2011-05-07 at 17:16 -0400, Andrew Dunstan wrote:
> pg_config is useful quite apart from its use in building things, as was
> discussed upthread.

Link please.

On lör, 2011-05-07 at 17:06 -0400, Greg Smith wrote:
> The repmgr program we distribute has the same problem, so I've been
> getting first-hand reports of just how many people are likely to run
> into this recently. You have to install postgresql-devel with RPM and
> on Debian, the very non-obvious postgresql-server-dev-$version

I'm pretty sure that for installing modules that need compilation from
CPAN or PyPI, you will need the corresponding p*-dev* package installed.
Nothing new here. I don't think you will get distributors to abandon
the concept of -dev(el) packages for this. Just saying.

On 05/07/2011 05:26 PM, Peter Eisentraut wrote:
> On lör, 2011-05-07 at 17:16 -0400, Andrew Dunstan wrote:
>> pg_config is useful quite apart from its use in building things, as was
>> discussed upthread.
> Link please.
>

<http://archives.postgresql.org/pgsql-hackers/2011-05/msg00275.php>

cheers

andrew

On 05/06/2011 04:06 PM, Tom Lane wrote:
> FWIW, I did move pg_config from -devel to the "main" (really client)
> postgresql package in Fedora, as of 9.0. That will ensure it's present
> in either client or server installations. Eventually that packaging
> will reach RHEL ...
>

We should make sure that the PGDG packages adopt that for 9.1 then, so
it starts catching on more. Unless Devrim changed to catch up since I
last installed an RPM set, in that 9.0 it's still in the same place:

$ rpm -qf /usr/pgsql-9.0/bin/pg_config
postgresql90-devel-9.0.2-2PGDG.rhel5

While Peter's question about whether it's really all that useful is
reasonable, I'd at least like to get a better error message when you
don't have everything needed to compile extensions. I think the
shortest path to that is making pg_config more likely to be installed,
then to check whether the file "pg_config --pgxs" references exists.
I'll see if I can turn that idea into an actual change to propose.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

Attached patch is a first cut at what moving one contrib module (in this
case pg_buffercache) to a new directory structure might look like. The
idea is that src/extension could become a place for "first-class"
extensions to live. Those are ones community is committed to providing
in core, but are just better implemented as extensions than in-database
functions, for reasons that include security. This idea has been shared
by a lot of people for a while, only problem is that it wasn't really
practical to implement cleanly until the extensions code hit. I think
it is now, this attempts to prove it.

Since patches involving file renaming are clunky, the changes might be
easier to see at
https://github.com/greg2ndQuadrant/postgres/commit/507923e21e963c873a84f1b850d64e895776574f
where I just pushed this change too. The install step for the modules
looks like this now:

gsmith(at)grace:~/pgwork/src/move-contrib/src/extension/pg_buffercache$
make install
/bin/mkdir -p '/home/gsmith/pgwork/inst/move-contrib/lib/postgresql'
/bin/mkdir -p
'/home/gsmith/pgwork/inst/move-contrib/share/postgresql/extension'
/bin/sh ../../../config/install-sh -c -m 755 pg_buffercache.so
'/home/gsmith/pgwork/inst/move-contrib/lib/postgresql/pg_buffercache.so'
/bin/sh ../../../config/install-sh -c -m 644 ./pg_buffercache.control
'/home/gsmith/pgwork/inst/move-contrib/share/postgresql/extension/'
/bin/sh ../../../config/install-sh -c -m 644 ./pg_buffercache--1.0.sql
./pg_buffercache--unpackaged--1.0.sql
'/home/gsmith/pgwork/inst/move-contrib/share/postgresql/extension/'
$ psql -c "create extension pg_buffercache"
CREATE EXTENSION

The only clunky bit I wasn't really happy with is the amount of code
duplication that comes from having a src/extension/Makefile that looks
almost, but not quite, identical to contrib/Makefile. The rest of the
changes don't seem too bad to me, and even that's really only 36 lines
that aren't touched often. Yes, the paths are different, so backports
won't happen without an extra step. But the code changes required were
easier than I was expecting, due to the general good modularity of the
extensions infrastructure. So long as the result ends up in
share/postgresql/extension/ , whether they started in contrib/<module>
or src/extension/<module> doesn't really matter to CREATE EXTENSION.
But having them broke out this way makes it easy for the default
Makefile to build and install them all. (I recognize I didn't do that
last step yet though)

I'll happily go covert pgstattuple and the rest of the internal
diagnostics modules to this scheme, and do the doc cleanups, this
upcoming week if it means I'll be able to use those things without
installing all of contrib one day. Ditto for proposing RPM and Debian
packaging changes that match them. All that work will get paid back the
first time I don't have to fill out a bunch of paperwork (again) at a
customer site justifying why they need to install the contrib [RPM|deb]
package (which has some scary stuff in it) on all their servers, just so
I can get some bloat or buffer inspection module.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

On lör, 2011-05-07 at 17:38 -0400, Andrew Dunstan wrote:
>
> On 05/07/2011 05:26 PM, Peter Eisentraut wrote:
> > On lör, 2011-05-07 at 17:16 -0400, Andrew Dunstan wrote:
> >> pg_config is useful quite apart from its use in building things, as was
> >> discussed upthread.
> > Link please.
> >
>
> <http://archives.postgresql.org/pgsql-hackers/2011-05/msg00275.php>

That thread just asserts that it might be useful, and I responded by
asking for what.

On 05/08/2011 05:24 AM, Peter Eisentraut wrote:
> On lör, 2011-05-07 at 17:38 -0400, Andrew Dunstan wrote:
>> On 05/07/2011 05:26 PM, Peter Eisentraut wrote:
>>> On lör, 2011-05-07 at 17:16 -0400, Andrew Dunstan wrote:
>>>> pg_config is useful quite apart from its use in building things, as was
>>>> discussed upthread.
>>> Link please.
>>>
>> <http://archives.postgresql.org/pgsql-hackers/2011-05/msg00275.php>
> That thread just asserts that it might be useful, and I responded by
> asking for what.

As I said there: "to see how the libraries are configured, for example."

Just the other day I wanted to know what compilation options had been
used for a particular installation. pg_config wasn't installed because
the -devel package wasn't installed, and it would have saved me quite
some time if pg_config had been available.

Another example is to find out what the installation is using for
shares, the service directory and so on.

cheers

andrew

On sön, 2011-05-08 at 07:21 -0400, Andrew Dunstan wrote:
> As I said there: "to see how the libraries are configured, for example."
>
> Just the other day I wanted to know what compilation options had been
> used for a particular installation. pg_config wasn't installed because
> the -devel package wasn't installed, and it would have saved me quite
> some time if pg_config had been available.
>
> Another example is to find out what the installation is using for
> shares, the service directory and so on.

Yeah, those are decent reasons.

My example is of doing "self-discovery" to see if all needful database
components seem to be properly installed.

E.g. - the app needs pgcrypto, intarray, and a custom data type. The
install script can consequently inform the production folk either "looks
good", or, alternately, "seems problematic!"

Actually, I haven't coded a sample of the "look for custom SPI & types"
part, but it's a natural extension of what I have.

Of course, it only provides a legitimate test when run on the database
server, which isn't always how production folk want to do it, but that's
part of a different argument...

On Sun, May 8, 2011 at 12:02 AM, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:
> Attached patch is a first cut at what moving one contrib module (in this
> case pg_buffercache) to a new directory structure might look like.  The idea
> is that src/extension could become a place for "first-class" extensions to
> live.  Those are ones community is committed to providing in core, but are
> just better implemented as extensions than in-database functions, for
> reasons that include security.  This idea has been shared by a lot of people
> for a while, only problem is that it wasn't really practical to implement
> cleanly until the extensions code hit.  I think it is now, this attempts to
> prove it.
>
> Since patches involving file renaming are clunky, the changes might be
> easier to see at
> https://github.com/greg2ndQuadrant/postgres/commit/507923e21e963c873a84f1b850d64e895776574f
> where I just pushed this change too.  The install step for the modules looks
> like this now:
>
> gsmith(at)grace:~/pgwork/src/move-contrib/src/extension/pg_buffercache$ make
> install
> /bin/mkdir -p '/home/gsmith/pgwork/inst/move-contrib/lib/postgresql'
> /bin/mkdir -p
> '/home/gsmith/pgwork/inst/move-contrib/share/postgresql/extension'
> /bin/sh ../../../config/install-sh -c -m 755  pg_buffercache.so
> '/home/gsmith/pgwork/inst/move-contrib/lib/postgresql/pg_buffercache.so'
> /bin/sh ../../../config/install-sh -c -m 644 ./pg_buffercache.control
> '/home/gsmith/pgwork/inst/move-contrib/share/postgresql/extension/'
> /bin/sh ../../../config/install-sh -c -m 644 ./pg_buffercache--1.0.sql
> ./pg_buffercache--unpackaged--1.0.sql
>  '/home/gsmith/pgwork/inst/move-contrib/share/postgresql/extension/'
> $ psql -c "create extension pg_buffercache"
> CREATE EXTENSION
>
> The only clunky bit I wasn't really happy with is the amount of code
> duplication that comes from having a src/extension/Makefile that looks
> almost, but not quite, identical to contrib/Makefile.  The rest of the
> changes don't seem too bad to me, and even that's really only 36 lines that
> aren't touched often.  Yes, the paths are different, so backports won't
> happen without an extra step.  But the code changes required were easier
> than I was expecting, due to the general good modularity of the extensions
> infrastructure.  So long as the result ends up in
> share/postgresql/extension/ , whether they started in contrib/<module> or
> src/extension/<module> doesn't really matter to CREATE EXTENSION.  But
> having them broke out this way makes it easy for the default Makefile to
> build and install them all.  (I recognize I didn't do that last step yet
> though)
>
> I'll happily go covert pgstattuple and the rest of the internal diagnostics
> modules to this scheme, and do the doc cleanups, this upcoming week if it
> means I'll be able to use those things without installing all of contrib one
> day.  Ditto for proposing RPM and Debian packaging changes that match them.
>  All that work will get paid back the first time I don't have to fill out a
> bunch of paperwork (again) at a customer site justifying why they need to
> install the contrib [RPM|deb] package (which has some scary stuff in it) on
> all their servers, just so I can get some bloat or buffer inspection module.

I would really like to see us try to group things by topic, and not
just by whether or not we can all agree that the extension is
important enough to be first-class (which is bound to be a bit
tendentious). We probably can't completely avoid some bikeshedding on
that topic, but even there it strikes me that sorting by topic might
make things a bit more clear. For example, if we could somehow group
together all the diagnostic tools, maybe something like the list
below, I think that would be a start. Now then we might go on to
argue about which are the more useful diagnostic tools, but I think
it's easier to argue about that category than it is to argue in the
abstract about whether you'd rather have hstore or pgstattuple, to
which the answer can only be "that depends".

auto_explain
oid2name
pageinspect
pg_buffercache
pg_freespacemap
pg_stat_statements
pg_test_fsync (perhaps)
pgrowlocks
pgstattuple

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 05/09/2011 10:53 AM, Robert Haas wrote:
> I would really like to see us try to group things by topic, and not
> just by whether or not we can all agree that the extension is
> important enough to be first-class (which is bound to be a bit
> tendentious).

Having played around with the prototype, I think it doesn't actually
matter if there's a further division below the new one I introduced.
The main thing I think is worth pointing out is that I only feel
extensions with no external dependencies are worth the trouble of
re-classifying here. If it were worth reorganizing contrib just for the
sake of categorizing it, that would have been done years ago. The new
thing is that extensions make it really easy to make some tools
available in the server's extensions subdirectly, without actually
activating them in the default install.

Looking at your list:

> auto_explain
> oid2name
> pageinspect
> pg_buffercache
> pg_freespacemap
> pg_stat_statements
> pg_test_fsync (perhaps)
> pgrowlocks
> pgstattuple
>

oid2name and pg_test_fsync would be out because those are real
executables. I'd rather not introduce the risk/complexity of playing
around with moving standalone utilities of such marginal value. Whereas
I think it sets an excellent precedent if the server is shipping with
some standard add-ons, built using the same extension mechanism
available to external code, in the core server package. I'd certainly
be happy to add auto_explain and pg_stat_statements (also extremely
popular things to install for me) to that list.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

On Mon, May 9, 2011 at 1:14 PM, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:
> On 05/09/2011 10:53 AM, Robert Haas wrote:
>>
>> I would really like to see us try to group things by topic, and not
>> just by whether or not we can all agree that the extension is
>> important enough to be first-class (which is bound to be a bit
>> tendentious).
>
> Having played around with the prototype, I think it doesn't actually matter
> if there's a further division below the new one I introduced.  The main
> thing I think is worth pointing out is that I only feel extensions with no
> external dependencies are worth the trouble of re-classifying here.  If it
> were worth reorganizing contrib just for the sake of categorizing it, that
> would have been done years ago.  The new thing is that extensions make it
> really easy to make some tools available in the server's extensions
> subdirectly, without actually activating them in the default install.
>
> Looking at your list:
>
>> auto_explain
>> oid2name
>> pageinspect
>> pg_buffercache
>> pg_freespacemap
>> pg_stat_statements
>> pg_test_fsync (perhaps)
>> pgrowlocks
>> pgstattuple
>>
>
> oid2name and pg_test_fsync would be out because those are real executables.
>  I'd rather not introduce the risk/complexity of playing around with moving
> standalone utilities of such marginal value.  Whereas I think it sets an
> excellent precedent if the server is shipping with some standard add-ons,
> built using the same extension mechanism available to external code, in the
> core server package.  I'd certainly be happy to add auto_explain and
> pg_stat_statements (also extremely popular things to install for me) to that
> list.

I'm happy enough with that set of guidelines: namely, that we'd use
src/extension only for things that don't require additional
dependencies, and not for things that build standalone executables.
If we're going to move things around, I think we should take the
trouble to categorize them along the way, and your idea of inserting
one more subdirectory under src/extension for grouping seems fine to
me.

I don't think we should be too obstinate about trying to twist the arm
of packagers who (as Tom points out) will do whatever they want in
spite of us, but the current state of contrib, with all sorts of
things of varying type, complexity, and value mixed together cannot
possibly be a good thing. Even if the effect of all this is that some
distributions end up with postgresql-server-instrumentation and
postgresql-server-datatypes packages, rather than putting everything
in postgresql-server, I still think that'd be better than having a
monolithic lump called postgresql-contrib. Heaven only knows what
could be in there (says the sys admin)...

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Excerpts from Robert Haas's message of lun may 09 14:31:33 -0400 2011:

> I'm happy enough with that set of guidelines: namely, that we'd use
> src/extension only for things that don't require additional
> dependencies, and not for things that build standalone executables.
> If we're going to move things around, I think we should take the
> trouble to categorize them along the way, and your idea of inserting
> one more subdirectory under src/extension for grouping seems fine to
> me.

For executables we already have src/bin. Do we really need a separate
place for, say, pg_standby or pg_upgrade?

--
Álvaro Herrera <alvherre(at)commandprompt(dot)com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
> Excerpts from Robert Haas's message of lun may 09 14:31:33 -0400 2011:
>> I'm happy enough with that set of guidelines: namely, that we'd use
>> src/extension only for things that don't require additional
>> dependencies, and not for things that build standalone executables.
>> If we're going to move things around, I think we should take the
>> trouble to categorize them along the way, and your idea of inserting
>> one more subdirectory under src/extension for grouping seems fine to
>> me.

> For executables we already have src/bin. Do we really need a separate
> place for, say, pg_standby or pg_upgrade?

Putting them in there implies we think they are of core-code quality.
I'm definitely *not* ready to grant that status to pg_upgrade, for
instance.

regards, tom lane

Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:
> Sure, but that's a documentation issue, which again is not going to be
> helped by a source-tree rearrangement.

So we have several problem to solve here, and I agree that source code
rearrangement is fixing none of them. Maybe it would ease maintaining
down the road, though, but I'll leave that choice up to you.

Which contribs are ready (safe) for production? We could handle that in
the version numbers, having most of contrib at version 0.9.1 (say) and
some of them at version 1.0. We could also stop distributing examples
in binary form, only ship them in source package.

Then we need to include inspection extensions into the core packaging,
but still as extensions. That's more of a packager problem, except that
they need a clear and strong message about it. Maybe have a new
Makefile and build those extensions as part of the server build, and
install them as part as the "normal" install.

Another mix of those ideas is to ship inspection extensions and ready
for production ones all into a new package postgresql-server-extensions
that the main server would depend on, and the ones that are adding more
dependencies still in contrib, where not-ready for production extensions
would not get built.

This kind of ideas would also allow to quite easily remove things from
the main server and have them as extensions, available on any install
but a CREATE EXTENSION (WITH SCHEMA pg_catalog) command away. And still
maintained at the same quality level in the source tree.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

On 05/09/2011 03:31 PM, Alvaro Herrera wrote:
> For executables we already have src/bin. Do we really need a separate
> place for, say, pg_standby or pg_upgrade?
>

There's really no executables in contrib that I find myself regularly
desperate for/angry at because they're not installed as an integral part
of the server, the way I regularly find myself cursing some things that
are now extensions. The only contrib executable I use often is pgbench,
and that's normally early in server life--when it's still possible to
get things installed easily most places. And it's something that can be
removed when finished, in cases where people are nervous about the
contrib package.

Situations where pg_standby or pg_upgrade suddenly pop up as emergency
needs seem unlikely too, which is also the case with oid2name,
pg_test_fsync, pg_archivecleanup, and vacuumlo. I've had that happen
with pg_archivecleanup exactly once since it appeared--running out of
space and that was the easiest way to make the problem go away
immediately and permanently--but since it was on an 8.4 server we had to
download the source and build anyway.

Also, my experience is that people are not that paranoid about running
external binaries, even though they could potentially do harm to the
database. Can always do a backup beforehand. But the idea of loading a
piece of code that lives in the server all the time freaks them out.
The way the word contrib implies (and sometimes is meant to mean) low
quality, while stuff that ships with the main server package does not,
has been beaten up here for years already. It's only a few cases where
that's not fully justified, and the program can easily become an
extension, that I feel are really worth changing here. There are 49
directories in contrib/ ; at best maybe 20% of them will ever fall into
that category.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

On 05/09/2011 02:31 PM, Robert Haas wrote:
> I don't think we should be too obstinate about trying to twist the arm
> of packagers who (as Tom points out) will do whatever they want in
> spite of us, but the current state of contrib, with all sorts of
> things of varying type, complexity, and value mixed together cannot
> possibly be a good thing.

I think the idea I'm running with for now means that packagers won't
actually have to do anything. I'd expect typical packaging for 9.1 to
include share/postgresql/extension from the build results without being
more specific. You need to grab 3 files from there to get the plpgsql
extension, and I can't imagine any packager listing them all by name.
So if I dump the converted contrib extensions to put files under there,
and remove them from the contrib build area destination, I suspect they
will magically jump from the contrib to the extensions area of the
server package at next package build; no packager level changes
required. The more I look at this, the less obtrusive of a change it
seems to be. Only people who will really notice are users who discover
more in the basic server package, and of course committers with
backporting to do.

Since packaged builds of 9.1 current with beta1 seem to be in short
supply still, this theory looks hard to prove just yet. I'm very
excited that it's packaging week here however (rolls eyes), so I'll
check it myself. I'll incorporate the suggestions made since I posted
that test patch and do a bigger round of this next, end to end with an
RPM set as the output. It sounds like everyone who has a strong opinion
on what this change might look like has sketched a similar looking
bikeshed. Once a reasonable implementation is hammered out, I'd rather
jump to the big argument between not liking change vs. the advocacy
benefits to PostgreSQL of doing this; they are considerable in my mind.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

Tom Lane wrote:
> Christopher Browne <cbbrowne(at)gmail(dot)com> writes:
> > But people are evidently still setting packaging policies based on how
> > things were back in 7.3, even though that perhaps isn't necessary
> > anymore.
>
> FWIW, once you get past the client versus server distinction, I think
> most subpackaging decisions are based on either the idea that "only a
> minority of people will want this", or a desire to limit how many
> dependencies are pulled in by the main package(s). Both of those
> concerns apply to various subsets of -contrib, which means it's going
> to be hard to persuade packagers to fold -contrib into the -server
> package altogether. Nor would you gain their approval by trying to
> pre-empt the decision.
>
> We might get somewhere by trying to identify a small set of particularly
> popular contrib modules that don't add any extra dependencies, and then
> recommending to packagers that those ones get bundled into the main
> server package.
>
> > Certainly it's not a huge amount of code; less than 2MB these days.
> > -> % wc `dpkg -L postgresql-contrib-9.0` | tail -1
> > 15952 67555 1770987 total
>
> Well, to add some concrete facts rather than generalities to my own post,
> here are the sizes of the built RPMs from my last build for Fedora:
>
> -rw-r--r--. 1 tgl tgl 3839458 Apr 18 10:50 postgresql-9.0.4-1.fc13.x86_64.rpm
> -rw-r--r--. 1 tgl tgl 490788 Apr 18 10:50 postgresql-contrib-9.0.4-1.fc13.x86_64.rpm
> -rw-r--r--. 1 tgl tgl 27337677 Apr 18 10:51 postgresql-debuginfo-9.0.4-1.fc13.x86_64.rpm
> -rw-r--r--. 1 tgl tgl 961660 Apr 18 10:50 postgresql-devel-9.0.4-1.fc13.x86_64.rpm
> -rw-r--r--. 1 tgl tgl 7569048 Apr 18 10:50 postgresql-docs-9.0.4-1.fc13.x86_64.rpm
> -rw-r--r--. 1 tgl tgl 246506 Apr 18 10:50 postgresql-libs-9.0.4-1.fc13.x86_64.rpm
> -rw-r--r--. 1 tgl tgl 64940 Apr 18 10:50 postgresql-plperl-9.0.4-1.fc13.x86_64.rpm
> -rw-r--r--. 1 tgl tgl 65776 Apr 18 10:50 postgresql-plpython-9.0.4-1.fc13.x86_64.rpm
> -rw-r--r--. 1 tgl tgl 45941 Apr 18 10:50 postgresql-pltcl-9.0.4-1.fc13.x86_64.rpm
> -rw-r--r--. 1 tgl tgl 5302117 Apr 18 10:50 postgresql-server-9.0.4-1.fc13.x86_64.rpm
> -rw-r--r--. 1 tgl tgl 1370509 Apr 18 10:50 postgresql-test-9.0.4-1.fc13.x86_64.rpm
> -rw-r--r--. 1 tgl tgl 3644113 Apr 18 10:50 postgresql-upgrade-9.0.4-1.fc13.x86_64.rpm

Is that last one pg_upgrade? It seems very big.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

Bruce Momjian <bruce(at)momjian(dot)us> writes:
> Tom Lane wrote:
>> here are the sizes of the built RPMs from my last build for Fedora:
>>
>> -rw-r--r--. 1 tgl tgl 3839458 Apr 18 10:50 postgresql-9.0.4-1.fc13.x86_64.rpm
>> -rw-r--r--. 1 tgl tgl 490788 Apr 18 10:50 postgresql-contrib-9.0.4-1.fc13.x86_64.rpm
>> -rw-r--r--. 1 tgl tgl 27337677 Apr 18 10:51 postgresql-debuginfo-9.0.4-1.fc13.x86_64.rpm
>> -rw-r--r--. 1 tgl tgl 961660 Apr 18 10:50 postgresql-devel-9.0.4-1.fc13.x86_64.rpm
>> -rw-r--r--. 1 tgl tgl 7569048 Apr 18 10:50 postgresql-docs-9.0.4-1.fc13.x86_64.rpm
>> -rw-r--r--. 1 tgl tgl 246506 Apr 18 10:50 postgresql-libs-9.0.4-1.fc13.x86_64.rpm
>> -rw-r--r--. 1 tgl tgl 64940 Apr 18 10:50 postgresql-plperl-9.0.4-1.fc13.x86_64.rpm
>> -rw-r--r--. 1 tgl tgl 65776 Apr 18 10:50 postgresql-plpython-9.0.4-1.fc13.x86_64.rpm
>> -rw-r--r--. 1 tgl tgl 45941 Apr 18 10:50 postgresql-pltcl-9.0.4-1.fc13.x86_64.rpm
>> -rw-r--r--. 1 tgl tgl 5302117 Apr 18 10:50 postgresql-server-9.0.4-1.fc13.x86_64.rpm
>> -rw-r--r--. 1 tgl tgl 1370509 Apr 18 10:50 postgresql-test-9.0.4-1.fc13.x86_64.rpm
>> -rw-r--r--. 1 tgl tgl 3644113 Apr 18 10:50 postgresql-upgrade-9.0.4-1.fc13.x86_64.rpm

> Is that last one pg_upgrade? It seems very big.

pg_upgrade plus a supporting set of 8.4 files ...

regards, tom lane

Tom Lane wrote:
> Bruce Momjian <bruce(at)momjian(dot)us> writes:
> > Tom Lane wrote:
> >> here are the sizes of the built RPMs from my last build for Fedora:
> >>
> >> -rw-r--r--. 1 tgl tgl 3839458 Apr 18 10:50 postgresql-9.0.4-1.fc13.x86_64.rpm
> >> -rw-r--r--. 1 tgl tgl 490788 Apr 18 10:50 postgresql-contrib-9.0.4-1.fc13.x86_64.rpm
> >> -rw-r--r--. 1 tgl tgl 27337677 Apr 18 10:51 postgresql-debuginfo-9.0.4-1.fc13.x86_64.rpm
> >> -rw-r--r--. 1 tgl tgl 961660 Apr 18 10:50 postgresql-devel-9.0.4-1.fc13.x86_64.rpm
> >> -rw-r--r--. 1 tgl tgl 7569048 Apr 18 10:50 postgresql-docs-9.0.4-1.fc13.x86_64.rpm
> >> -rw-r--r--. 1 tgl tgl 246506 Apr 18 10:50 postgresql-libs-9.0.4-1.fc13.x86_64.rpm
> >> -rw-r--r--. 1 tgl tgl 64940 Apr 18 10:50 postgresql-plperl-9.0.4-1.fc13.x86_64.rpm
> >> -rw-r--r--. 1 tgl tgl 65776 Apr 18 10:50 postgresql-plpython-9.0.4-1.fc13.x86_64.rpm
> >> -rw-r--r--. 1 tgl tgl 45941 Apr 18 10:50 postgresql-pltcl-9.0.4-1.fc13.x86_64.rpm
> >> -rw-r--r--. 1 tgl tgl 5302117 Apr 18 10:50 postgresql-server-9.0.4-1.fc13.x86_64.rpm
> >> -rw-r--r--. 1 tgl tgl 1370509 Apr 18 10:50 postgresql-test-9.0.4-1.fc13.x86_64.rpm
> >> -rw-r--r--. 1 tgl tgl 3644113 Apr 18 10:50 postgresql-upgrade-9.0.4-1.fc13.x86_64.rpm
>
> > Is that last one pg_upgrade? It seems very big.
>
> pg_upgrade plus a supporting set of 8.4 files ...

OK, where do I get to dance around that pg_upgrade is packaged in Fedora
thanks to you? At PGCon? LOL

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On Sat, 2011-05-07 at 21:47 -0400, Greg Smith wrote:
> On 05/06/2011 04:06 PM, Tom Lane wrote:
> > FWIW, I did move pg_config from -devel to the "main" (really client)
> > postgresql package in Fedora, as of 9.0. That will ensure it's
> present
> > in either client or server installations. Eventually that packaging
> > will reach RHEL ...
> >
>
> We should make sure that the PGDG packages adopt that for 9.1 then, so
> it starts catching on more. Unless Devrim changed to catch up since I
> last installed an RPM set, in that 9.0 it's still in the same place:
>
> $ rpm -qf /usr/pgsql-9.0/bin/pg_config
> postgresql90-devel-9.0.2-2PGDG.rhel5

I'm not sure that I can move it to main package in 9.0 package set, I
need to make sure that I won't break anything. But it is pretty doable
for 9.1.

Regards,
--
Devrim GÜNDÜZ
Principal Systems Engineer @ EnterpriseDB: http://www.enterprisedb.com
PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer
Community: devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org Twitter: http://twitter.com/devrimgunduz

Devrim =?ISO-8859-1?Q?G=DCND=DCZ?= <devrim(at)gunduz(dot)org> writes:
> I'm not sure that I can move it to main package in 9.0 package set, I
> need to make sure that I won't break anything. But it is pretty doable
> for 9.1.

It should be okay to move, since the -devel subpackage requires the main
one. Therefore there is no configuration in which pg_config would be
present before and missing after the change.

regards, tom lane

On Thu, 2011-05-12 at 19:37 -0400, Tom Lane wrote:
>
>
> It should be okay to move, since the -devel subpackage requires the
> main one. Therefore there is no configuration in which pg_config
> would be present before and missing after the change.

Thanks Tom. I can make this change in next build set.

Regards,

--
Devrim GÜNDÜZ
Principal Systems Engineer @ EnterpriseDB: http://www.enterprisedb.com
PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer
Community: devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org Twitter: http://twitter.com/devrimgunduz

Attached is a second patch to move a number of extensions from contrib/
to src/test/. Extensions there are built by the default built target,
making installation of the postgresql-XX-contrib package unnecessary for
them to be available.

This request--making some of these additions available without the
"contrib" name/package being involved--has popped up many times before,
and it turys out to be really easy to resolve with the new extensions
infrastructure. I think it's even a reasonable change to consider
applying now, between 9.1 Beta 1 and Beta 2. The documentation
adjustments are the only serious bit left here that I've been able to
find, the code changes here are all internal to the build process and easy.

I moved the following extensions:

auto_explain pageinspect pg_buffercache pg_freespacemap pgrowlocks
pg_stat_statements pgstattuple

My criteria was picking extensions that:

1) Don't have any special dependencies
2) Are in contrib mainly because they don't need to be internal
functions, not because their code quality is demo/early
3) Tend to be installed on a production server for troubleshooting
problems, rather than being required by development.
4) Regularly pop up as necessary/helpful in production deployment

Some of my personal discussions of this topic have suggested that some
other popular extensions like pgcrypto and hstore get converted too. I
think those all fail test (3), and I'm not actually sure where pgcrypto
adds any special dependency/distribution issues were it to be moved to
the main database package. If this general idea catches on, a wider
discussion of what else should get "promoted" to this extensions area
would be appropriate. The ones I picked seemed the easiest to justify
by this criteria set.

Any packager who grabs the shared/postgresql/extension directory in 9.1,
which I expect to be all of them, shouldn't need any changes to pick up
this adjustment. For example, pgstattuple installs these files:

share/postgresql/extension/pgstattuple--1.0.sql
share/postgresql/extension/pgstattuple--unpackaged--1.0.sql
share/postgresql/extension/pgstattuple.control

And these are the same locations they were already at. The location of
the source and which target built it is the change here, the result
isn't any different. This means that this change won't even break
extensions already installed.

Once the basic directory plumbing is in place, conversion of a single
extension from contrib/ to src/test/ is, trivial. The diff view

I did five of them in an hour once I figured out what was needed.
Easiest to view the changes at
https://github.com/greg2ndQuadrant/postgres/commits/move-contrib , the
patch file is huge because of all the renames.
https://github.com/greg2ndQuadrant/postgres/commit/d647091b18c4448c5a582d423f8839ef0c717e91
show a good example of one convert, that changes pg_freespacemap. There
are more changes to the comments listing the name of the file than to
any code. (Yes, I know there are some whitespace issues I introduced in
the new Makefile, they should be fixed by a later commit in the series)

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

Greg Smith wrote:
> Attached is a second patch to move a number of extensions from
> contrib/ to src/test/. Extensions there are built by the default
> built target, making installation of the postgresql-XX-contrib package
> unnecessary for them to be available.

That was supposed to be contrib/ to src/extension , no idea where that
test bit came from.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

On Wed, May 18, 2011 at 10:25, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:
> Attached is a second patch to move a number of extensions from contrib/ to
> src/test/.  Extensions there are built by the default built target, making
> installation of the postgresql-XX-contrib package unnecessary for them to be
> available.

+1 in general on the concept :-)

> This request--making some of these additions available without the "contrib"
> name/package being involved--has popped up many times before, and it turys
> out to be really easy to resolve with the new extensions infrastructure.  I
> think it's even a reasonable change to consider applying now, between 9.1
> Beta 1 and Beta 2.  The documentation adjustments are the only serious bit
> left here that I've been able to find, the code changes here are all
> internal to the build process and easy.

Does this include regression tests? Or will they need some mods?

> I moved the following extensions:
>
> auto_explain pageinspect pg_buffercache pg_freespacemap pgrowlocks
> pg_stat_statements pgstattuple
>
> My criteria was picking extensions that:
>
> 1) Don't have any special dependencies
> 2) Are in contrib mainly because they don't need to be internal functions,
> not because their code quality is demo/early
> 3) Tend to be installed on a production server for troubleshooting problems,
> rather than being required by development.
> 4) Regularly pop up as necessary/helpful in production deployment

These seem like reasonable criteria.

> Some of my personal discussions of this topic have suggested that some other
> popular extensions like pgcrypto and hstore get converted too.  I think
> those all fail test (3), and I'm not actually sure where pgcrypto adds any
> special dependency/distribution issues were it to be moved to the main
> database package.  If this general idea catches on, a wider discussion of
> what else should get "promoted" to this extensions area would be
> appropriate.  The ones I picked seemed the easiest to justify by this
> criteria set.

pgcrypto would cause trouble for any builds *without* SSL. I don't
think any packagers do that, but people doing manual builds would
certainly get different results.

> Any packager who grabs the shared/postgresql/extension directory in 9.1,
> which I expect to be all of them, shouldn't need any changes to pick up this
> adjustment.  For example, pgstattuple installs these files:
>
> share/postgresql/extension/pgstattuple--1.0.sql
> share/postgresql/extension/pgstattuple--unpackaged--1.0.sql
> share/postgresql/extension/pgstattuple.control
>
> And these are the same locations they were already at.  The location of the
> source and which target built it is the change here, the result isn't any
> different.  This means that this change won't even break extensions already
> installed.
>
> Once the basic directory plumbing is in place, conversion of a single
> extension from contrib/ to src/test/ is, trivial.  The diff view
>
> I did five of them in an hour once I figured out what was needed.  Easiest
> to view the changes at
> https://github.com/greg2ndQuadrant/postgres/commits/move-contrib , the patch
> file is huge because of all the renames.
> https://github.com/greg2ndQuadrant/postgres/commit/d647091b18c4448c5a582d423f8839ef0c717e91
> show a good example of one convert, that changes pg_freespacemap.  There are
> more changes to the comments listing the name of the file than to any code.
>  (Yes, I know there are some whitespace issues I introduced in the new
> Makefile, they should be fixed by a later commit in the series)

This is where the compare view rocks:

https://github.com/greg2ndQuadrant/postgres/compare/postgres:master...greg2ndQuadrant:move-contrib

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Wed, May 18, 2011 at 2:57 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> On Wed, May 18, 2011 at 10:25, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:
>> Some of my personal discussions of this topic have suggested that some other
>> popular extensions like pgcrypto and hstore get converted too.  I think
>> those all fail test (3), and I'm not actually sure where pgcrypto adds any
>> special dependency/distribution issues were it to be moved to the main
>> database package.  If this general idea catches on, a wider discussion of
>> what else should get "promoted" to this extensions area would be
>> appropriate.  The ones I picked seemed the easiest to justify by this
>> criteria set.
>
> pgcrypto would cause trouble for any builds *without* SSL. I don't
> think any packagers do that, but people doing manual builds would
> certainly get different results.

What kind of trouble? It should work fine without SSL.

--
marko

On Wed, May 18, 2011 at 15:29, Marko Kreen <markokr(at)gmail(dot)com> wrote:
> On Wed, May 18, 2011 at 2:57 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>> On Wed, May 18, 2011 at 10:25, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:
>>> Some of my personal discussions of this topic have suggested that some other
>>> popular extensions like pgcrypto and hstore get converted too.  I think
>>> those all fail test (3), and I'm not actually sure where pgcrypto adds any
>>> special dependency/distribution issues were it to be moved to the main
>>> database package.  If this general idea catches on, a wider discussion of
>>> what else should get "promoted" to this extensions area would be
>>> appropriate.  The ones I picked seemed the easiest to justify by this
>>> criteria set.
>>
>> pgcrypto would cause trouble for any builds *without* SSL. I don't
>> think any packagers do that, but people doing manual builds would
>> certainly get different results.
>
> What kind of trouble?  It should work fine without SSL.

Oh, you're right - it does. But it does provide different
functionalties? Or does it actually do exactly the same stuff, just in
different ways?

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Greg Smith wrote:
> Any packager who grabs the shared/postgresql/extension directory in
> 9.1, which I expect to be all of them, shouldn't need any changes to
> pick up this adjustment. For example, pgstattuple installs these files:
>
> share/postgresql/extension/pgstattuple--1.0.sql
> share/postgresql/extension/pgstattuple--unpackaged--1.0.sql
> share/postgresql/extension/pgstattuple.control
>
> And these are the same locations they were already at.

...and the bit I missed here is that there's a fourth file here:

lib/postgresql/pgstattuple.so

If you look at a 9.1 spec file, such as
http://svn.pgrpms.org/browser/rpm/redhat/9.1/postgresql/EL-6/postgresql-9.1.spec
, you'll find:

%files contrib
...
%{pgbaseinstdir}/lib/pgstattuple.so

Which *does* require a packager change to relocate from the
postgresql-91-package to the main server one. So the theory that a
change here might happen without pushing a repackaging suggestion toward
packagers is busted. This does highlight that some packaging guidelines
would be needed here to completely this work.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

On Wed, May 18, 2011 at 3:37 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> On Wed, May 18, 2011 at 15:29, Marko Kreen <markokr(at)gmail(dot)com> wrote:
>> On Wed, May 18, 2011 at 2:57 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>>> On Wed, May 18, 2011 at 10:25, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:
>>>> Some of my personal discussions of this topic have suggested that some other
>>>> popular extensions like pgcrypto and hstore get converted too.  I think
>>>> those all fail test (3), and I'm not actually sure where pgcrypto adds any
>>>> special dependency/distribution issues were it to be moved to the main
>>>> database package.  If this general idea catches on, a wider discussion of
>>>> what else should get "promoted" to this extensions area would be
>>>> appropriate.  The ones I picked seemed the easiest to justify by this
>>>> criteria set.
>>>
>>> pgcrypto would cause trouble for any builds *without* SSL. I don't
>>> think any packagers do that, but people doing manual builds would
>>> certainly get different results.
>>
>> What kind of trouble?  It should work fine without SSL.
>
> Oh, you're right - it does. But it does provide different
> functionalties? Or does it actually do exactly the same stuff, just in
> different ways?

Same stuff, assuming you use recommended algorithms
(Blowfish, AES, MD5, SHA1, SHA2)

OpenSSL brings in more speedy implementations (maybe),
and additional algorithms (ripemd160, 3des, cast5, twofish).

--
marko