Re: PostgreSQL 9.3 beta breaks some extensions "make install"

Hi list,

While testing out PostgreSQL 9.3beta1, I stumbled upon a problem
installing some extensions (pgTAP and semver among others):

% make
[...]
% make DESTDIR=/tmp/foo install
[...]
/usr/bin/install -c -m 644 ./sql/semver--unpackaged--0.2.1.sql
./sql/semver--0.2.4--0.3.0.sql ./sql/semver--0.2.1--0.2.4.sql
./sql/semver--0.3.0.sql ./sql/semver--0.3.0.sql
'/tmp/foo/usr/share/postgresql/extension/'
/usr/bin/install: will not overwrite just-created
‘/tmp/foo/usr/share/postgresql/extension/semver--0.3.0.sql’ with
‘./sql/semver--0.3.0.sql’
make: *** [install] Error 1

I traced the problem down to commit
9db7ccae2000524b72a4052352cbb5407fb53b02 "Use system install program
when available and usable". It turns out that 'install' from coreutils
8.21 complains when it's told to install the same source file twice.

It's caused by this common pattern in extension makefiles:
DATA = $(wildcard sql/*--*.sql) sql/$(EXTENSION)--$(EXTVERSION).sql

The wildcard will match semver--0.3.0.sql too, in addition to the
direct reference, causing it to be duplicated (unless "make install"
is called from a clean directory).

The attached 1st patch fixes this case by putting $(sort ...) around
the DATA install command, which drops duplicate filenames. While there
are other install commands, the chances of them containing duplicate
names seem lower.

I'm not sure if this is worth worrying about, but there's still a
problem when DATA and DATA_built are separated, e.g:
DATA = $(wildcard sql/*--*.sql)
DATA_built = sql/$(EXTENSION)--$(EXTVERSION).sql

This can't be solved with a sort since DATA_built doesn't use
$(addprefix). But the install could be split into 2 separate commands
(2nd patch)

Regards,
Marti

Marti Raudsepp <marti(at)juffo(dot)org> writes:
> While testing out PostgreSQL 9.3beta1, I stumbled upon a problem
> installing some extensions (pgTAP and semver among others):
> ...
> I traced the problem down to commit
> 9db7ccae2000524b72a4052352cbb5407fb53b02 "Use system install program
> when available and usable". It turns out that 'install' from coreutils
> 8.21 complains when it's told to install the same source file twice.

TBH, I thought that was a dangerous idea from the get-go. My vote for
fixing this would be to revert that change, not try to hack all the
makefiles to work around it.

regards, tom lane

On Tue, 2013-05-14 at 04:12 +0300, Marti Raudsepp wrote:
> It's caused by this common pattern in extension makefiles:
> DATA = $(wildcard sql/*--*.sql) sql/$(EXTENSION)--$(EXTVERSION).sql
>
What is the point of this? Why have the wildcard and then the
non-wildcard term?

I think using wildcard is bad style and you should fix it.

On 05/13/2013 10:27 PM, Peter Eisentraut wrote:
> On Tue, 2013-05-14 at 04:12 +0300, Marti Raudsepp wrote:
>> It's caused by this common pattern in extension makefiles:
>> DATA = $(wildcard sql/*--*.sql) sql/$(EXTENSION)--$(EXTVERSION).sql
>>
> What is the point of this? Why have the wildcard and then the
> non-wildcard term?
>
> I think using wildcard is bad style and you should fix it.
>

I've been caught by it. I'm not sure why the wildcard is a bad idea -
don't we want any present update sql files to be installed? . I don't
see any reason to have the redundant addition of the current version's
sql file, though.

cheers

andrew

On Tue, May 14, 2013 at 5:27 AM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
> On Tue, 2013-05-14 at 04:12 +0300, Marti Raudsepp wrote:
>> It's caused by this common pattern in extension makefiles:
>> DATA = $(wildcard sql/*--*.sql) sql/$(EXTENSION)--$(EXTVERSION).sql

> What is the point of this? Why have the wildcard and then the
> non-wildcard term?

Because the non-wildcard file is built by the same Makefile (it's
copied from the sql/$(EXTENSION).sql file). If it wasn't there, a
"make install" from a clean checkout would miss this file.

all: sql/$(EXTENSION)--$(EXTVERSION).sql

sql/$(EXTENSION)--$(EXTVERSION).sql: sql/$(EXTENSION).sql
cp $< $@

> I think using wildcard is bad style and you should fix it.

Perhaps, but fixing the extensions is not a solution at this point. A
large number of extensions use this exact code (it comes from David
Wheeler's template AFAIK). We might stand a chance in fixing the
public extensions on PGXN, but this would presumably still break
non-public extensions that people have written.

For example:
David Wheeler's pgTAP https://github.com/theory/pgTAP/blob/master/Makefile#L91
Jan Urbański's first_last_agg
https://github.com/wulczer/first_last_agg/blob/master/Makefile
Andrew Dunstan's json_build
https://github.com/pgexperts/json_build/blob/master/Makefile
Theo Schlossnagle's pg_amqp
http://api.pgxn.org/src/pg_amqp/pg_amqp-0.3.0/Makefile
OmniTI's pg_jobmon https://github.com/omniti-labs/pg_jobmon/blob/master/Makefile

Regards,
Marti

Le mardi 14 mai 2013 10:17:13, Marti Raudsepp a écrit :
> On Tue, May 14, 2013 at 5:27 AM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
> > On Tue, 2013-05-14 at 04:12 +0300, Marti Raudsepp wrote:
> >> It's caused by this common pattern in extension makefiles:
> >> DATA = $(wildcard sql/*--*.sql) sql/$(EXTENSION)--$(EXTVERSION).sql
> >
> > What is the point of this? Why have the wildcard and then the
> > non-wildcard term?
>
> Because the non-wildcard file is built by the same Makefile (it's
> copied from the sql/$(EXTENSION).sql file). If it wasn't there, a
> "make install" from a clean checkout would miss this file.
>
> all: sql/$(EXTENSION)--$(EXTVERSION).sql
>
> sql/$(EXTENSION)--$(EXTVERSION).sql: sql/$(EXTENSION).sql
> cp $< $@
>
> > I think using wildcard is bad style and you should fix it.
>
> Perhaps, but fixing the extensions is not a solution at this point. A
> large number of extensions use this exact code (it comes from David
> Wheeler's template AFAIK). We might stand a chance in fixing the
> public extensions on PGXN, but this would presumably still break
> non-public extensions that people have written.

My understanding is that the offending commit reveals possible bad written
instruction in some Makefile. What's wrong ?
--
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

Marti Raudsepp <marti(at)juffo(dot)org> writes:
> all: sql/$(EXTENSION)--$(EXTVERSION).sql
>
> sql/$(EXTENSION)--$(EXTVERSION).sql: sql/$(EXTENSION).sql
> cp $< $@

That's a recipe for problems. Each time I meet with such a construct in
an extension's Makefile I propose an hard coded alternative. We're
speaking of maintaining less than half a dozen file names here, with one
or two additions per year, in most cases.

Really, that trick is a recipe for problems. Use at your own risk.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

PS: yes I've been saying that all along, ever since David first cooked
that, and I've blogged about it, and I've worked on alternatives and
tools, in particular pg_buildext, part of the postgresql-common
debian package.

On 5/13/13 10:58 PM, Andrew Dunstan wrote:
> I'm not sure why the wildcard is a bad idea - don't we want any present
> update sql files to be installed?

Generally, wildcards in makefiles are a bad idea because you will then
not discover any broken tarballs and checkouts. The users will just
install nothing or a subset of the files you had intended.

It might also break vpath builds. That can probably be worked around,
though.

On 5/14/13 4:17 AM, Marti Raudsepp wrote:
> On Tue, May 14, 2013 at 5:27 AM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
>> On Tue, 2013-05-14 at 04:12 +0300, Marti Raudsepp wrote:
>>> It's caused by this common pattern in extension makefiles:
>>> DATA = $(wildcard sql/*--*.sql) sql/$(EXTENSION)--$(EXTVERSION).sql
>
>> What is the point of this? Why have the wildcard and then the
>> non-wildcard term?
>
> Because the non-wildcard file is built by the same Makefile (it's
> copied from the sql/$(EXTENSION).sql file). If it wasn't there, a
> "make install" from a clean checkout would miss this file.

If it's built, then it should be listed in DATA_built.

> Perhaps, but fixing the extensions is not a solution at this point. A
> large number of extensions use this exact code (it comes from David
> Wheeler's template AFAIK).

So far, the number is still less than the number of extensions broken by
the htup header refactoring, so I'm not worried about it.

Also, substituting a custom install program has always been supported,
so this was already broken anyway, now we just know about it earlier.

On 5/14/13 7:50 AM, Dimitri Fontaine wrote:
> Marti Raudsepp <marti(at)juffo(dot)org> writes:
>> > all: sql/$(EXTENSION)--$(EXTVERSION).sql
>> >
>> > sql/$(EXTENSION)--$(EXTVERSION).sql: sql/$(EXTENSION).sql
>> > cp $< $@
> That's a recipe for problems. Each time I meet with such a construct in
> an extension's Makefile I propose an hard coded alternative. We're
> speaking of maintaining less than half a dozen file names here, with one
> or two additions per year, in most cases.

I don't think this in particular is a problem. I use something like
that myself. But you should list all the older extension upgrade files
explicitly (to which your point applies).

On 05/14/2013 07:59 AM, Peter Eisentraut wrote:
> On 5/14/13 4:17 AM, Marti Raudsepp wrote:
>> On Tue, May 14, 2013 at 5:27 AM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
>>> On Tue, 2013-05-14 at 04:12 +0300, Marti Raudsepp wrote:
>>>> It's caused by this common pattern in extension makefiles:
>>>> DATA = $(wildcard sql/*--*.sql) sql/$(EXTENSION)--$(EXTVERSION).sql
>>> What is the point of this? Why have the wildcard and then the
>>> non-wildcard term?
>> Because the non-wildcard file is built by the same Makefile (it's
>> copied from the sql/$(EXTENSION).sql file). If it wasn't there, a
>> "make install" from a clean checkout would miss this file.
> If it's built, then it should be listed in DATA_built.

So, AIUI, leaving aside stylistic arguments about use of wildcards, one
solution could be:

DATA_built = sql/$(EXTENSION)--$(EXTVERSION).sql
DATA = $(filter-out sql/$(EXTENSION)--$(EXTVERSION).sql, $(wildcard
sql/*--*.sql))

Is that right?

cheers

andrew

Le mardi 14 mai 2013 10:17:13, Marti Raudsepp a écrit :
> On Tue, May 14, 2013 at 5:27 AM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
> > On Tue, 2013-05-14 at 04:12 +0300, Marti Raudsepp wrote:
> >> It's caused by this common pattern in extension makefiles:
> >> DATA = $(wildcard sql/*--*.sql) sql/$(EXTENSION)--$(EXTVERSION).sql
> >
> > What is the point of this? Why have the wildcard and then the
> > non-wildcard term?
>
> Because the non-wildcard file is built by the same Makefile (it's
> copied from the sql/$(EXTENSION).sql file). If it wasn't there, a
> "make install" from a clean checkout would miss this file.
>
> all: sql/$(EXTENSION)--$(EXTVERSION).sql
>
> sql/$(EXTENSION)--$(EXTVERSION).sql: sql/$(EXTENSION).sql
> cp $< $@

There is also something else here, the addition of the all:.... target.

I believe it should be removed and the added target(s) be written after the
include of pgxs makefile.

If I follow your example, then I would rewrite http://manager.pgxn.org/howto

From
=====
PG91 = $(shell $(PG_CONFIG) --version | grep -qE " 8\.| 9\.0" && echo no ||
echo yes)

ifeq ($(PG91),yes)
all: sql/$(EXTENSION)--$(EXTVERSION).sql

sql/$(EXTENSION)--$(EXTVERSION).sql: sql/$(EXTENSION).sql
cp $< $@

DATA = $(wildcard sql/*--*.sql) sql/$(EXTENSION)--$(EXTVERSION).sql
EXTRA_CLEAN = sql/$(EXTENSION)--$(EXTVERSION).sql
endif

PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)
====

To

====
PG91 = $(shell $(PG_CONFIG) --version | grep -qE " 8\.| 9\.0" && echo no ||
echo yes)

ifeq ($(PG91),yes)
DATA = $(wildcard sql/*--*.sql) sql/$(EXTENSION)--$(EXTVERSION).sql
EXTRA_CLEAN = sql/$(EXTENSION)--$(EXTVERSION).sql
endif

PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)

ifeq ($(PG91),yes)
sql/$(EXTENSION)--$(EXTVERSION).sql: sql/$(EXTENSION).sql
cp $< $@
endif
====

because the default target from the PostgreSQL Makefile is «all:» and the
addition of a target before inclusion of PostgreSQL makefile change the default
(see DEFAULT_GOAL variable)

Thought ?
--
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

On Tue, May 14, 2013 at 2:59 PM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
>> Perhaps, but fixing the extensions is not a solution at this point. A
>> large number of extensions use this exact code (it comes from David
>> Wheeler's template AFAIK).
>
> So far, the number is still less than the number of extensions broken by
> the htup header refactoring, so I'm not worried about it.

I think that's fair. The C API has always been in flux and there are
no stability guarantees. We expect C extension writers to know what
they're doing.

But I think we should be more forgiving to extensions written in
"stable" languages like PL/pgSQL, especially if they break *because*
the writer was following PGXN packaging best practices
(http://manager.pgxn.org/howto) and not a fault of their own.

I did a quick and dirty survey of extensions on PGXN and found that
the install change causes problems for (at least) 22% of extensions
there. I think it's well worth the time to implement a workaround,
rather than hassle extension writers.

---
I downloaded and tried to build the latest version of each package on
PGXN (103 total). Now obviously I can't build everything because I
don't have all the necessary dependencies -- which means it's probably
skewed against C extensions. So don't take these results too
seriously. But a it's data point regardless:

9.2 9.3beta1
70 61 <-- "make" succeeds
65 33 <-- "make install" succeeds
0 23 <-- outputs "install: will not overwrite just-created"
0 7 <-- outputs "implicit declaration of function 'heap_"

Note that the htup change causes just a compile-time warning (it will
fail at extension load time), so some of these are counted as
successful builds.

Script and some instructions to replicate this test are attached.

Regards,
Marti

Marti Raudsepp <marti(at)juffo(dot)org> writes:
> I did a quick and dirty survey of extensions on PGXN and found that
> the install change causes problems for (at least) 22% of extensions
> there. I think it's well worth the time to implement a workaround,
> rather than hassle extension writers.

What's really worrying me about this is that beta1 has been out for
less than 48 hours and we've already found one way in which a lot
of system-provided install scripts vary from the one we provide.
How many more compatibility problems will there be?

We changed to using install-sh unconditionally back in 2001 because
we had too many problems with system-provided scripts that didn't do
what we expected. I see very little reason to believe that the
compatibility problems have disappeared since then, and in fact this
complaint seems to me to be sufficient to refute that thesis.

I still think we should revert 9db7ccae2000524b72a4052352cbb5407fb53b02.
The argument that the system-provided program might be faster carries
very little weight for me --- "make install" is fast enough already.
It's not worth making a bunch of extension authors jump through hoops,
whether their style was bad or not.

regards, tom lane

* Tom Lane (tgl(at)sss(dot)pgh(dot)pa(dot)us) wrote:
> I still think we should revert 9db7ccae2000524b72a4052352cbb5407fb53b02.
> The argument that the system-provided program might be faster carries
> very little weight for me --- "make install" is fast enough already.
> It's not worth making a bunch of extension authors jump through hoops,
> whether their style was bad or not.

+1.

Thanks,

Stephen

On Tue, May 14, 2013 at 10:30 PM, Stephen Frost <sfrost(at)snowman(dot)net> wrote:
> * Tom Lane (tgl(at)sss(dot)pgh(dot)pa(dot)us) wrote:
>> I still think we should revert 9db7ccae2000524b72a4052352cbb5407fb53b02.
>> The argument that the system-provided program might be faster carries
>> very little weight for me --- "make install" is fast enough already.
>> It's not worth making a bunch of extension authors jump through hoops,
>> whether their style was bad or not.
>
> +1.

+1.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 5/14/13 5:45 PM, Tom Lane wrote:
> We changed to using install-sh unconditionally back in 2001 because
> we had too many problems with system-provided scripts that didn't do
> what we expected. I see very little reason to believe that the
> compatibility problems have disappeared since then, and in fact this
> complaint seems to me to be sufficient to refute that thesis.

The compatibility issues in 2001 were completely different ones and were
explicitly resolved with a new configure check (which is used worldwide
by thousands of packages, note). Let's not confuse the issue.

> The argument that the system-provided program might be faster carries
> very little weight for me --- "make install" is fast enough already.

It is not for me. Note also that make install is part of test runs.

> It's not worth making a bunch of extension authors jump through hoops,
> whether their style was bad or not.

Well, I consider that this is not a style issue. It's an issue of
wide-spread bugginess caused by uninformed copy-and-paste, and I'm glad
we found it. Considering the widespread crappiness in PostgreSQL
extension makefiles, I don't consider it a problem that a few things
need to be fixed.

That said, I'm obviously outnumbered here. What about the following
compromise: Use the configure-selected install program inside
PostgreSQL (which we can test easily), and use install-sh under
USE_PGXS? Admittedly, the make install time of extensions is probably
not an issue.

(The affected extensions will still be buggy because users can still
substitute their own install programs. We're just hiding the issue for
a while.)

On 5/14/13 10:38 AM, Cédric Villemain wrote:
> If I follow your example, then I would rewrite http://manager.pgxn.org/howto

Oh that's where that is coming from? Well that example has all kinds of
problems.

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> That said, I'm obviously outnumbered here. What about the following
> compromise: Use the configure-selected install program inside
> PostgreSQL (which we can test easily), and use install-sh under
> USE_PGXS? Admittedly, the make install time of extensions is probably
> not an issue.

That works for me, since as you say we can easily fix any such bugs
in the core code. The scary thing about this for extension authors
is that they may very well see no bug in their own testing, only to
have their packages fall over in the wild. We shouldn't make each
author who's copied that code rediscover the problem for themselves
that expensively.

> (The affected extensions will still be buggy because users can still
> substitute their own install programs. We're just hiding the issue for
> a while.)

I'm not following this argument. The old makefile coding explicitly set

INSTALL = $(SHELL) $(top_srcdir)/config/install-sh -c

The only way I can see that that would be overridden is an explicit

make INSTALL=/my/random/script install

and surely we cannot guarantee that things don't break when you do
something like that. There are hundreds of ways you can break
the build if you mess with makefile variables. So I'm not prepared
to call it a bug if that doesn't work.

regards, tom lane

Le mardi 14 mai 2013 15:08:42, Andrew Dunstan a écrit :
> On 05/14/2013 07:59 AM, Peter Eisentraut wrote:
> > On 5/14/13 4:17 AM, Marti Raudsepp wrote:
> >> On Tue, May 14, 2013 at 5:27 AM, Peter Eisentraut <peter_e(at)gmx(dot)net>
wrote:
> >>> On Tue, 2013-05-14 at 04:12 +0300, Marti Raudsepp wrote:
> >>>> It's caused by this common pattern in extension makefiles:
> >>>> DATA = $(wildcard sql/*--*.sql) sql/$(EXTENSION)--$(EXTVERSION).sql
> >>>
> >>> What is the point of this? Why have the wildcard and then the
> >>> non-wildcard term?
> >>
> >> Because the non-wildcard file is built by the same Makefile (it's
> >> copied from the sql/$(EXTENSION).sql file). If it wasn't there, a
> >> "make install" from a clean checkout would miss this file.
> >
> > If it's built, then it should be listed in DATA_built.
>
> So, AIUI, leaving aside stylistic arguments about use of wildcards, one
> solution could be:
>
> DATA_built = sql/$(EXTENSION)--$(EXTVERSION).sql
> DATA = $(filter-out sql/$(EXTENSION)--$(EXTVERSION).sql, $(wildcard
> sql/*--*.sql))
>
> Is that right?

Yes.
--
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

On 05/15/2013 10:05 AM, Tom Lane wrote:
> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
>> That said, I'm obviously outnumbered here. What about the following
>> compromise: Use the configure-selected install program inside
>> PostgreSQL (which we can test easily), and use install-sh under
>> USE_PGXS? Admittedly, the make install time of extensions is probably
>> not an issue.
> That works for me, since as you say we can easily fix any such bugs
> in the core code. The scary thing about this for extension authors
> is that they may very well see no bug in their own testing, only to
> have their packages fall over in the wild. We shouldn't make each
> author who's copied that code rediscover the problem for themselves
> that expensively.

+1, although I will be renovating the Makefiles for all my extensions
along the lines of my previous email.

cheers

andrew

On Wed, May 15, 2013 at 09:51:15AM -0400, Peter Eisentraut wrote:
> On 5/14/13 10:38 AM, Cédric Villemain wrote:
> > If I follow your example, then I would rewrite http://manager.pgxn.org/howto
>
> Oh that's where that is coming from? Well that example has all kinds of
> problems.

Would you be so kind as to point out same, or better still, to write
up what you think of as a better example intended for the same
audience? I'm sure the PGXN people would be delighted to put
something better up there.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

Le mercredi 15 mai 2013 16:43:17, Andrew Dunstan a écrit :
> On 05/15/2013 10:05 AM, Tom Lane wrote:
> > Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> >> That said, I'm obviously outnumbered here. What about the following
> >> compromise: Use the configure-selected install program inside
> >> PostgreSQL (which we can test easily), and use install-sh under
> >> USE_PGXS? Admittedly, the make install time of extensions is probably
> >> not an issue.
> >
> > That works for me, since as you say we can easily fix any such bugs
> > in the core code. The scary thing about this for extension authors
> > is that they may very well see no bug in their own testing, only to
> > have their packages fall over in the wild. We shouldn't make each
> > author who's copied that code rediscover the problem for themselves
> > that expensively.
>
> +1, although I will be renovating the Makefiles for all my extensions
> along the lines of my previous email.

pgxs.mk has some old rules to replace $libdir (and some few other maybe).
Maybe we can try to find what make sense for most of the extension authors and
add rules/target to pgxs.mk to reduce the useless copy/paste in the Makefile of
extensions.

So what's desirable ?
--
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> DATA_built = sql/$(EXTENSION)--$(EXTVERSION).sql
> DATA = $(filter-out sql/$(EXTENSION)--$(EXTVERSION).sql, $(wildcard
> sql/*--*.sql))
>
> Is that right?

I think that's still breaking VPATH builds because the widlcard call
happens in the current tree, not in the VPATH (source) tree.

And VPATH building of extension is crucially important for me, as the
easiest way I've found to build and package a given extension against
all currently supported version of PostgreSQL.

I couldn't figure out how to allow for wildcard and whatnot in the
extensions Makefile and have it work in all cases, including VPATH, so
my current advice is to get rid of any dynamic code here and just plain
copy and paste the file names you're interested into.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

On 05/16/2013 05:41 AM, Dimitri Fontaine wrote:
> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
>> DATA_built = sql/$(EXTENSION)--$(EXTVERSION).sql
>> DATA = $(filter-out sql/$(EXTENSION)--$(EXTVERSION).sql, $(wildcard
>> sql/*--*.sql))
>>
>> Is that right?
> I think that's still breaking VPATH builds because the widlcard call
> happens in the current tree, not in the VPATH (source) tree.
>
> And VPATH building of extension is crucially important for me, as the
> easiest way I've found to build and package a given extension against
> all currently supported version of PostgreSQL.
>
> I couldn't figure out how to allow for wildcard and whatnot in the
> extensions Makefile and have it work in all cases, including VPATH, so
> my current advice is to get rid of any dynamic code here and just plain
> copy and paste the file names you're interested into.
>

Is there documented support for VPATH builds? I know of an external
script that promises such support, but AFAIK there is nothing in our
code that supports it. If we're going to support this I suggest we do it
properly. I don't consider myself on the hook to support some external
build tool.

As for supporting multiple versions, I have given up supporting multiple
Postgres versions from a single set of sources, for FDWs at least. For
those I use a git branch that mirrors the corresponding Postgres branch.

cheers

andrew

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> On 05/16/2013 05:41 AM, Dimitri Fontaine wrote:
>> And VPATH building of extension is crucially important for me, as the
>> easiest way I've found to build and package a given extension against
>> all currently supported version of PostgreSQL.

> Is there documented support for VPATH builds?

The core code certainly builds okay in VPATH mode, and I would agree
with Dimitri that pgxs builds should as well. But this is more of an
autoconf+make feature than ours, so I'm not sure why you'd expect us
to document it.

Having said that, if a particular extension's makefile contains features
that cause it to not build VPATH, then presumably that extension's
author doesn't care.

regards, tom lane

Le jeudi 16 mai 2013 15:51:48, Tom Lane a écrit :
> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> > On 05/16/2013 05:41 AM, Dimitri Fontaine wrote:
> >> And VPATH building of extension is crucially important for me, as the
> >> easiest way I've found to build and package a given extension against
> >> all currently supported version of PostgreSQL.
> >
> > Is there documented support for VPATH builds?
>
> The core code certainly builds okay in VPATH mode, and I would agree
> with Dimitri that pgxs builds should as well. But this is more of an
> autoconf+make feature than ours, so I'm not sure why you'd expect us
> to document it.

Extension does not support VPATH at 100% (well, pgxs.mk).
There is a minor hack for some REGRESS files but that's all.

I think at least DOCS, DATA and REGRESS needs some addition on pgxs.mk to help
extension author allows build out-of-tree (postgresql been built out or not).

I'll work on a patch for that.
--
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

On 05/16/2013 10:39 AM, Cédric Villemain wrote:
> Le jeudi 16 mai 2013 15:51:48, Tom Lane a écrit :
>> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
>>> On 05/16/2013 05:41 AM, Dimitri Fontaine wrote:
>>>> And VPATH building of extension is crucially important for me, as the
>>>> easiest way I've found to build and package a given extension against
>>>> all currently supported version of PostgreSQL.
>>> Is there documented support for VPATH builds?
>> The core code certainly builds okay in VPATH mode, and I would agree
>> with Dimitri that pgxs builds should as well. But this is more of an
>> autoconf+make feature than ours, so I'm not sure why you'd expect us
>> to document it.
> Extension does not support VPATH at 100% (well, pgxs.mk).
> There is a minor hack for some REGRESS files but that's all.

Right. My impression is that pgxs.mk actively works against vpath builds.

>
> I think at least DOCS, DATA and REGRESS needs some addition on pgxs.mk to help
> extension author allows build out-of-tree (postgresql been built out or not).
>
> I'll work on a patch for that.

OK, great.

cheers

andrew

Andrew Dunstan wrote:
>
> On 05/16/2013 10:39 AM, Cédric Villemain wrote:
> >Le jeudi 16 mai 2013 15:51:48, Tom Lane a écrit :
> >>Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> >>>On 05/16/2013 05:41 AM, Dimitri Fontaine wrote:
> >>>>And VPATH building of extension is crucially important for me, as the
> >>>>easiest way I've found to build and package a given extension against
> >>>>all currently supported version of PostgreSQL.
> >>>Is there documented support for VPATH builds?
> >>The core code certainly builds okay in VPATH mode, and I would agree
> >>with Dimitri that pgxs builds should as well. But this is more of an
> >>autoconf+make feature than ours, so I'm not sure why you'd expect us
> >>to document it.
> >Extension does not support VPATH at 100% (well, pgxs.mk).
> >There is a minor hack for some REGRESS files but that's all.
>
> Right. My impression is that pgxs.mk actively works against vpath builds.

That's my experience, yes. It would be great to get it fixed.

Then there's the outright weird stuff using ancient makefiles ..
*grumble* pg_filedump *grumble*

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Alvaro Herrera <alvherre(at)2ndquadrant(dot)com> writes:
> Then there's the outright weird stuff using ancient makefiles ..
> *grumble* pg_filedump *grumble*

I've never made any effort to improve the original makefile for that.
Want to send a patch?

regards, tom lane

Tom Lane wrote:
> Alvaro Herrera <alvherre(at)2ndquadrant(dot)com> writes:
> > Then there's the outright weird stuff using ancient makefiles ..
> > *grumble* pg_filedump *grumble*
>
> I've never made any effort to improve the original makefile for that.
> Want to send a patch?

Not right away, but I will get to it eventually, thanks.

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Le jeudi 16 mai 2013 18:53:19, Alvaro Herrera a écrit :
> Andrew Dunstan wrote:
> > On 05/16/2013 10:39 AM, Cédric Villemain wrote:
> > >Le jeudi 16 mai 2013 15:51:48, Tom Lane a écrit :
> > >>Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> > >>>On 05/16/2013 05:41 AM, Dimitri Fontaine wrote:
> > >>>>And VPATH building of extension is crucially important for me, as the
> > >>>>easiest way I've found to build and package a given extension against
> > >>>>all currently supported version of PostgreSQL.
> > >>>
> > >>>Is there documented support for VPATH builds?
> > >>
> > >>The core code certainly builds okay in VPATH mode, and I would agree
> > >>with Dimitri that pgxs builds should as well. But this is more of an
> > >>autoconf+make feature than ours, so I'm not sure why you'd expect us
> > >>to document it.
> > >
> > >Extension does not support VPATH at 100% (well, pgxs.mk).
> > >There is a minor hack for some REGRESS files but that's all.
> >
> > Right. My impression is that pgxs.mk actively works against vpath builds.
>
> That's my experience, yes. It would be great to get it fixed.

OK. I've reduce that to the attached (WIP) patch .
Problem is usage of $(srcdir) in the PGXS case. I try to remove it in favor of
recipes for each case (DATA, DOCS, ...)

It's a quick hack... It does not handle some cases yet (for example it does
not find .o when come the time to build .so if the .o was already in VPATH),
this later issue is a separate one I think.

If it seems to be on the right way, I'll keep fixing EXTENSION building with
VPATH.

Comments ?
--
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

Re: Cédric Villemain 2013-05-17 <201305171642(dot)59241(dot)cedric(at)2ndquadrant(dot)com>
> If it seems to be on the right way, I'll keep fixing EXTENSION building with
> VPATH.

I haven't tried the patch, but let me just say that Debian (and
apt.postgresql.org) would very much like the VPATH situation getting
improved. At the moment we seem to have to invent a new build recipe
for every extension around.

Thanks for working on this.

Christoph
--
cb(at)df7cb(dot)de | http://www.df7cb.de/

> > If it seems to be on the right way, I'll keep fixing EXTENSION building
> > with VPATH.
>
> I haven't tried the patch, but let me just say that Debian (and
> apt.postgresql.org) would very much like the VPATH situation getting
> improved. At the moment we seem to have to invent a new build recipe
> for every extension around.

I have been busy the last week.
I just took time to inspect our contribs, USE_PGXS is not supported by all of
them atm because of SHLIB_PREREQS (it used submake) I have a patch pending
here to fix that. Once all our contribs can build with USE_PGXS I fix the VPATH.

The last step is interesting: installcheck/REGRESS. For this one, if I can
know exactly what's required (for debian build for example), then I can also
fix this target.

--
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

Re: Cédric Villemain 2013-05-25 <201305251641(dot)28401(dot)cedric(at)2ndquadrant(dot)com>
> I just took time to inspect our contribs, USE_PGXS is not supported by all of
> them atm because of SHLIB_PREREQS (it used submake) I have a patch pending
> here to fix that. Once all our contribs can build with USE_PGXS I fix the VPATH.

The evil part of the problem is that you'd need to fix it all the way
back to 8.4 (or 9.0 once 8.4 is EOL) if we want it to build extensions
on apt.pg.org. (Or find a way that is compatible with as many majors
as possible.)

> The last step is interesting: installcheck/REGRESS. For this one, if I can
> know exactly what's required (for debian build for example), then I can also
> fix this target.

Debian builds don't have root access, so "make installcheck" would
need to install into a temp dir (probably DESTDIR). Currently this
fails because the server will still insist on reading the extension
control files from PGSHAREDIR. (See the thread starting at
http://www.postgresql.org/message-id/20120221101903.GA15647@gnash for
details.)

Otherwise than that, and my pending "pg_regress --host /tmp" patch,
things should probably ok for buildds.

Christoph
--
cb(at)df7cb(dot)de | http://www.df7cb.de/

> I just took time to inspect our contribs, USE_PGXS is not supported by all
> of them atm because of SHLIB_PREREQS (it used submake) I have a patch
> pending here to fix that.

Attached patch fix SHLIB_PREREQS when building with USE_PGXS

commit 19e231b introduced SHLIB_PREREQS but failed to port that to PGXS build.

The issue is that "submake-*" can not be built with PGXS, in this case they
must check that expected files are present (and installed).
Maybe it is better to only check if they have been built ?

This fix the build of dblink and postgres_fdw (make USE_PGXS=1)

--
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

> Once all our contribs can build with USE_PGXS I
> fix the VPATH.
>
> The last step is interesting: installcheck/REGRESS. For this one, if I can
> know exactly what's required (for debian build for example), then I can
> also fix this target.

There is a hack to link the regression data files from the srcdir
to the builddir when doing 'make VPATH'. but it failed when used in
conjunction with USE_PGXS and out-of-tree build of an extension.

Issue is the absence of the data/ directory in the builddir.

Attached patch fix that.
--
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

> Once all our contribs can build with USE_PGXS I
> fix the VPATH.

Attached patch set VPATH for out-of-tree extension builds

If the makefile is not in the current directory (where we launch 'make')
then assume we are building out-of-src tree and set the VPATH to the
directory of the *first* makefile...

Thus it fixes:
mkdir /tmp/a && cd /tmp/a
make -f extension_src/Makefile USE_PGXS=1

Note that the patch fix things. Still I am not really happy with the rule to
get the srcdir.

--
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

Le mardi 28 mai 2013 14:16:38, Cédric Villemain a écrit :
> > Once all our contribs can build with USE_PGXS I
> > fix the VPATH.
> >
> > The last step is interesting: installcheck/REGRESS. For this one, if I
> > can know exactly what's required (for debian build for example), then I
> > can also fix this target.
>
> There is a hack to link the regression data files from the srcdir
> to the builddir when doing 'make VPATH'. but it failed when used in
> conjunction with USE_PGXS and out-of-tree build of an extension.
>
> Issue is the absence of the data/ directory in the builddir.
>
> Attached patch fix that.

use $(MKDIR_P) instead of mkdir -p ....

--
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

Le samedi 25 mai 2013 16:41:24, Cédric Villemain a écrit :
> > > If it seems to be on the right way, I'll keep fixing EXTENSION building
> > > with VPATH.
> >
> > I haven't tried the patch, but let me just say that Debian (and
> > apt.postgresql.org) would very much like the VPATH situation getting
> > improved. At the moment we seem to have to invent a new build recipe
> > for every extension around.

Attached patch adds support for VPATH with USE_PGXS
It just change recipe for install: in pgxs.mk.

I am unsure automatic variables can be used this way with all UNIX variation
of make...

I also didn't touch MODULE and PROGRAM (yet)
--
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

> I just took time to inspect our contribs, USE_PGXS is not supported by all
> of them atm because of SHLIB_PREREQS (it used submake) I have a patch
> pending here to fix that. Once all our contribs can build with USE_PGXS I
> fix the VPATH.

I've added 'most' of the patches to the commitfest... (I am not sure it is
required, as this is more bugfix than anything else IMHO)
See
https://commitfest.postgresql.org/action/patch_view?id=1122
https://commitfest.postgresql.org/action/patch_view?id=1123
https://commitfest.postgresql.org/action/patch_view?id=1124

I stopped trying to add new item after too many failures from
https://commitfest.postgresql.org/action/patch_form
So one patch is not in the commitfest yet (fix_install_ext_vpath.patch)

--
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

On 05/29/2013 06:08 PM, Cédric Villemain wrote:
>> I just took time to inspect our contribs, USE_PGXS is not supported by all
>> of them atm because of SHLIB_PREREQS (it used submake) I have a patch
>> pending here to fix that. Once all our contribs can build with USE_PGXS I
>> fix the VPATH.
>
> I've added 'most' of the patches to the commitfest... (I am not sure it is
> required, as this is more bugfix than anything else IMHO)
> See
> https://commitfest.postgresql.org/action/patch_view?id=1122
> https://commitfest.postgresql.org/action/patch_view?id=1123
> https://commitfest.postgresql.org/action/patch_view?id=1124
>
>
> I stopped trying to add new item after too many failures from
> https://commitfest.postgresql.org/action/patch_form
> So one patch is not in the commitfest yet (fix_install_ext_vpath.patch)

"failures"? what kind of issues did you experience?

Stefan

Le jeudi 30 mai 2013 14:32:46, Stefan Kaltenbrunner a écrit :
> On 05/29/2013 06:08 PM, Cédric Villemain wrote:
> >> I just took time to inspect our contribs, USE_PGXS is not supported by
> >> all of them atm because of SHLIB_PREREQS (it used submake) I have a
> >> patch pending here to fix that. Once all our contribs can build with
> >> USE_PGXS I fix the VPATH.
> >
> > I've added 'most' of the patches to the commitfest... (I am not sure it
> > is required, as this is more bugfix than anything else IMHO)
> > See
> > https://commitfest.postgresql.org/action/patch_view?id=1122
> > https://commitfest.postgresql.org/action/patch_view?id=1123
> > https://commitfest.postgresql.org/action/patch_view?id=1124
> >
> >
> > I stopped trying to add new item after too many failures from
> > https://commitfest.postgresql.org/action/patch_form
> > So one patch is not in the commitfest yet (fix_install_ext_vpath.patch)
>
> "failures"? what kind of issues did you experience?

I didn't sent too much details as I am not sure if it was my setup which
breaks things or not.

[...]

Just tested with Stephen, looks like a problem with something on my side,
sorry for the noise. (rekonq 0 - chromium 1)

--
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

Le mardi 28 mai 2013 14:10:14, Cédric Villemain a écrit :
> > I just took time to inspect our contribs, USE_PGXS is not supported by
> > all of them atm because of SHLIB_PREREQS (it used submake) I have a
> > patch pending here to fix that.
>
> Attached patch fix SHLIB_PREREQS when building with USE_PGXS
>
> commit 19e231b introduced SHLIB_PREREQS but failed to port that to PGXS
> build.
>
> The issue is that "submake-*" can not be built with PGXS, in this case they
> must check that expected files are present (and installed).
> Maybe it is better to only check if they have been built ?
>
> This fix the build of dblink and postgres_fdw (make USE_PGXS=1)

This patch is a bugfix and IMO should be applied in 9.1 and 9.2.
Else we should remove the PGXS support from the Makefile of the contribs dblink
and postgres_fdw because it is not working currently.
--
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

Le mardi 28 mai 2013 15:15:55, Cédric Villemain a écrit :
> Le samedi 25 mai 2013 16:41:24, Cédric Villemain a écrit :
> > > > If it seems to be on the right way, I'll keep fixing EXTENSION
> > > > building with VPATH.
> > >
> > > I haven't tried the patch, but let me just say that Debian (and
> > > apt.postgresql.org) would very much like the VPATH situation getting
> > > improved. At the moment we seem to have to invent a new build recipe
> > > for every extension around.
>
> Attached patch adds support for VPATH with USE_PGXS
> It just change recipe for install: in pgxs.mk.
>
> I am unsure automatic variables can be used this way with all UNIX
> variation of make...
>
> I also didn't touch MODULE and PROGRAM (yet)

This patch can also be seen as a bugfix.
The problem is that in case like this one:
===
FOO=bar.control
installcontrol: $(FOO)
$(INSTALL_DATA) $(FOO) \
'$(DESTDIR)$(datadir)/extension/'
===

INSTALL_DATA will install the file defined by FOO (=bar.control).

But in the next case (the fix):
===
FOO=bar.control
installcontrol: $(FOO)
$(INSTALL_DATA) $< '$(DESTDIR)$(datadir)/extension/'
===

the $< contains the *filepath* where 'make' found the FOO file.
I believe it is because recipes are read once target and prerequisite are set,
so $< contains the full path to FOO, but FOO will still contain exactly what
has been assigned to FOO.

I choose to add targets for the variables that can be set when using PGXS. And
say 'install:' target to depends on each.

Maybe there is a smarter way to do it... my skills in Makefile are limited.

So far the feedback is good for the set of patches: pg_buildext (the debian
postgresql builder tool) is still working as expected and can be simplified. I
can build each contrib provided with PostgreSQL and some others have been
tried (like plr).
IMHO this is mostly bugfixing and it just outline that we can support VPATH and
PGXS build at the same time.

--
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

On Tue, May 14, 2013 at 4:12 AM, Marti Raudsepp <marti(at)juffo(dot)org> wrote:
> While testing out PostgreSQL 9.3beta1, I stumbled upon a problem

> % make DESTDIR=/tmp/foo install

> /usr/bin/install: will not overwrite just-created
> ‘/tmp/foo/usr/share/postgresql/extension/semver--0.3.0.sql’ with
> ‘./sql/semver--0.3.0.sql’
> make: *** [install] Error 1

On Wed, May 15, 2013 at 4:49 PM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
> That said, I'm obviously outnumbered here. What about the following
> compromise: Use the configure-selected install program inside
> PostgreSQL (which we can test easily), and use install-sh under
> USE_PGXS? Admittedly, the make install time of extensions is probably
> not an issue.

Did we ever do anything about this? It looks like the thread got
distracted with VPATH builds and now I'm seeing this problem in 9.3.0.
:(

This occurs in Arch Linux, but for some odd reason not on Ubuntu when
using apt.postgresql.org. Somehow the pgxs.mk supplied by
apt.postgresql.org differs from the one shipped in PostgreSQL.

Is there a chance of getting this resolved in PostgreSQL or should we
get extension writers to fix their makefiles instead?

Regards,
Marti

Marti Raudsepp <marti(at)juffo(dot)org> a écrit :
>On Tue, May 14, 2013 at 4:12 AM, Marti Raudsepp <marti(at)juffo(dot)org>
>wrote:
>> While testing out PostgreSQL 9.3beta1, I stumbled upon a problem
>
>> % make DESTDIR=/tmp/foo install
>
>> /usr/bin/install: will not overwrite just-created
>> ‘/tmp/foo/usr/share/postgresql/extension/semver--0.3.0.sql’ with
>> ‘./sql/semver--0.3.0.sql’
>> make: *** [install] Error 1
>
>On Wed, May 15, 2013 at 4:49 PM, Peter Eisentraut <peter_e(at)gmx(dot)net>
>wrote:
>> That said, I'm obviously outnumbered here. What about the following
>> compromise: Use the configure-selected install program inside
>> PostgreSQL (which we can test easily), and use install-sh under
>> USE_PGXS? Admittedly, the make install time of extensions is
>probably
>> not an issue.
>
>Did we ever do anything about this? It looks like the thread got
>distracted with VPATH builds and now I'm seeing this problem in 9.3.0.
>:(
>
>This occurs in Arch Linux, but for some odd reason not on Ubuntu when
>using apt.postgresql.org. Somehow the pgxs.mk supplied by
>apt.postgresql.org differs from the one shipped in PostgreSQL.
>
>Is there a chance of getting this resolved in PostgreSQL or should we
>get extension writers to fix their makefiles instead?

Apt.pgdg got the patch present in postgresql head applyed.
Andrew is about to commit (well...I hope) a doc patch about that and also a little fix.
Imho this is a bugfix so I hope it will be applyed in older branches.

--
Envoyé de mon téléphone. Excusez la brièveté.

On Fri, Sep 13, 2013 at 6:42 PM, Cédric Villemain
<cedric(at)2ndquadrant(dot)com> wrote:
> Marti Raudsepp <marti(at)juffo(dot)org> a écrit :
>>Did we ever do anything about this? It looks like the thread got
>>distracted with VPATH builds and now I'm seeing this problem in 9.3.0.

> Andrew is about to commit (well...I hope) a doc patch about that and also a little fix.
> Imho this is a bugfix so I hope it will be applyed in older branches.

Oh I see, indeed commit 6697aa2bc25c83b88d6165340348a31328c35de6
"Improve support for building PGXS modules with VPATH" fixes the
problem and I see it's not present in REL9_3_0.

Andrew and others, does this seem safe enough to backport to 9.3.1?

> Apt.pgdg got the patch present in postgresql head applyed.

Erm, isn't apt.postgresql.org supposed to ship the *official*
PostgreSQL versions? Given that this issue affects all distros, I
don't see why Ubuntu/Debian need to be patched separately.

Does anyone else think this is problematic? By slipping patches into
distro-specific packages, you're confusing users (like me) and
bypassing the PostgreSQL QA process.

PS: Where are the sources used to build packages on apt.postgresql.org?

Regards,
Marti

> > Apt.pgdg got the patch present in postgresql head applyed.
>
> Erm, isn't apt.postgresql.org supposed to ship the *official*
> PostgreSQL versions? Given that this issue affects all distros, I
> don't see why Ubuntu/Debian need to be patched separately.

Well, the patches are applyed on the debian packages (not only in
apt.pgdg.org).
The packages provided by apt.postgresql.org are based on the 'official
packages' from debian. (if you allow me this circle)

> Does anyone else think this is problematic? By slipping patches into
> distro-specific packages, you're confusing users (like me) and
> bypassing the PostgreSQL QA process.

The QA is about distribution of packages here. Debian applies 14 patches
on 9.3 builds, not only the things about vpath we're talking about.

> PS: Where are the sources used to build packages on
> apt.postgresql.org?

9.3:
http://anonscm.debian.org/loggerhead/pkg-postgresql/postgresql-9.3/trunk/changes

--
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

On Tue, Sep 17, 2013 at 10:37 AM, Cédric Villemain
<cedric(at)2ndquadrant(dot)com> wrote:
>> Erm, isn't apt.postgresql.org supposed to ship the *official*
>> PostgreSQL versions? Given that this issue affects all distros, I
>> don't see why Ubuntu/Debian need to be patched separately.

> Well, the patches are applyed on the debian packages (not only in
> apt.pgdg.org).
> The packages provided by apt.postgresql.org are based on the 'official
> packages' from debian. (if you allow me this circle)

Oh I see, that makes sense.

>> PS: Where are the sources used to build packages on
> 9.3:
> http://anonscm.debian.org/loggerhead/pkg-postgresql/postgresql-9.3/trunk/changes

Thanks!

Regards,
Marti

On Fri, Sep 13, 2013 at 8:17 PM, Marti Raudsepp <marti(at)juffo(dot)org> wrote:
> On Fri, Sep 13, 2013 at 6:42 PM, Cédric Villemain <cedric(at)2ndquadrant(dot)com> wrote:
>> Andrew is about to commit (well...I hope) a doc patch about that and also a little fix.
>> Imho this is a bugfix so I hope it will be applyed in older branches.
>
> Oh I see, indeed commit 6697aa2bc25c83b88d6165340348a31328c35de6
> "Improve support for building PGXS modules with VPATH" fixes the
> problem and I see it's not present in REL9_3_0.
>
> Andrew and others, does this seem safe enough to backport to 9.3.1?

Ping? Will this be backported to 9.3 or should I report to extension
authors to fix their Makefiles?

I understand that the 9.3.1 release might still be weeks away, I'd
just like to get a vague confirmation about what committers think.

Note that this patch is already applied to Debian/Ubuntu packages
(including those on apt.postgresql.org).

Regards,
Marti

Marti Raudsepp wrote:
> On Fri, Sep 13, 2013 at 8:17 PM, Marti Raudsepp <marti(at)juffo(dot)org> wrote:

> > Oh I see, indeed commit 6697aa2bc25c83b88d6165340348a31328c35de6
> > "Improve support for building PGXS modules with VPATH" fixes the
> > problem and I see it's not present in REL9_3_0.
> >
> > Andrew and others, does this seem safe enough to backport to 9.3.1?
>
> Ping? Will this be backported to 9.3 or should I report to extension
> authors to fix their Makefiles?

I think this should be backpatched.

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 09/23/2013 11:31 AM, Alvaro Herrera wrote:
> Marti Raudsepp wrote:
>> On Fri, Sep 13, 2013 at 8:17 PM, Marti Raudsepp <marti(at)juffo(dot)org> wrote:
>>> Oh I see, indeed commit 6697aa2bc25c83b88d6165340348a31328c35de6
>>> "Improve support for building PGXS modules with VPATH" fixes the
>>> problem and I see it's not present in REL9_3_0.
>>>
>>> Andrew and others, does this seem safe enough to backport to 9.3.1?
>> Ping? Will this be backported to 9.3 or should I report to extension
>> authors to fix their Makefiles?
> I think this should be backpatched.
>

I'm working on it. It appears to have a slight problem or two I want to
fix at the same time, rather than backpatch something broken.

cheers

andrew

Le lundi 23 septembre 2013 11:43:13 Andrew Dunstan a écrit :
> On 09/23/2013 11:31 AM, Alvaro Herrera wrote:
> > Marti Raudsepp wrote:
> >> On Fri, Sep 13, 2013 at 8:17 PM, Marti Raudsepp <marti(at)juffo(dot)org>
wrote:
> >>> Oh I see, indeed commit 6697aa2bc25c83b88d6165340348a31328c35de6
> >>> "Improve support for building PGXS modules with VPATH" fixes the
> >>> problem and I see it's not present in REL9_3_0.
> >>>
> >>> Andrew and others, does this seem safe enough to backport to
> >>> 9.3.1?
> >>
> >> Ping? Will this be backported to 9.3 or should I report to
> >> extension
> >> authors to fix their Makefiles?
> >
> > I think this should be backpatched.
>
> I'm working on it. It appears to have a slight problem or two I want
> to fix at the same time, rather than backpatch something broken.

Would you mind sharing the problems you are facing ?
You've noticed the problem about installdirs, I suppose. The attached
patch is the fix currently applyed at debian.

--
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

On 09/23/2013 12:15 PM, Cédric Villemain wrote:

>> I'm working on it. It appears to have a slight problem or two I want
>> to fix at the same time, rather than backpatch something broken.
> Would you mind sharing the problems you are facing ?
> You've noticed the problem about installdirs, I suppose. The attached
> patch is the fix currently applyed at debian.
>

I will when I'm sure it's not a case of pilot error on my part.

cheers

andrew

Hi Andrew,

On Mon, Sep 23, 2013 at 6:43 PM, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
> I'm working on it. It appears to have a slight problem or two I want to fix
> at the same time, rather than backpatch something broken.

Any progress on this? I notice that the fixes didn't make it into 9.3.1.

Regards,
Marti