Re: PostgreSQL-9.0alpha: jade required?

Tom -

Didn't realize I was arm waving - was I? (Sometimes email falls well short...)

We've managed a build of PostgreSQL 9.0-alpha4 - nice! However, the # make install command apparently(?) hiccups
on a dependency on Jade (we ain't usin' it!)

Here's the exact make install output:

- - - snip
make[3]: Entering directory `/media/SOURCES/postgresql-9.0alpha4/doc/src/sgml'
{ \
echo "<!entity version \"9.0alpha4\">"; \
echo "<!entity majorversion \"9.0\">"; \
} >version.sgml
"/usr/local/bin/perl" ./mk_feature_tables.pl YES ../../../src/backend/catalog/sql_feature_packages.txt ../../../src/backend/catalog/sql_features.txt > features-supported.sgml
"/usr/local/bin/perl" ./mk_feature_tables.pl NO ../../../src/backend/catalog/sql_feature_packages.txt ../../../src/backend/catalog/sql_features.txt > features-unsupported.sgml
jade -wall -wno-unused-param -wno-empty -wfully-tagged -D . -D . -c /usr/share/sgml/docbook/dsssl-stylesheets/catalog -d stylesheet.dsl -t sgml -i output-html -V html-index postgres.sgml
make[3]: jade: Command not found
make[3]: *** [HTML.index] Error 127
make[3]: Leaving directory `/media/SOURCES/postgresql-9.0alpha4/doc/src/sgml'
make[2]: *** [install] Error 2
make[2]: Leaving directory `/media/SOURCES/postgresql-9.0alpha4/doc/src'
make[1]: *** [install] Error 2
make[1]: Leaving directory `/media/SOURCES/postgresql-9.0alpha4/doc'
make: *** [install] Error 2
- - - snip

Looking at the Error 127; chown'd all files to root, then tried the make install again - as root - with same result.

Got around this by commenting out the $DOC line in GNUmakefile's make install section, and seem to have installed rest of the build AOK…

Would it make sense: to have more granular control over docs build in the config script - specific kinds of docs/no docs at all/etc?

Thanks for any help, and for the great work! We love PostgreSQL!

Lou

----- Original Message -----
From: "Tom Lane" <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: "Lou Picciano" <loupicciano(at)comcast(dot)net>
Cc: pgsql-bugs(at)postgresql(dot)org
Sent: Wednesday, February 24, 2010 11:08:19 AM GMT -05:00 US/Canada Eastern
Subject: Re: [BUGS] PostgreSQL-9.0alpha: jade required?

Lou Picciano <loupicciano(at)comcast(dot)net> writes:
> Not sure it's exactly a bug, but in attempting a compile of PostgreSQL-9.0alpha, we are getting a choke on jade (don't have jade on this system)
> Can the config script test for jade or, better yet, allow an option to turn off build of documentation?

Would you show exactly what happened rather than arm-waving? I'm sure
that the majority of the buildfarm machines don't have jade installed,
but they're getting through their builds OK.

(And yes, there *is* a configure check.)

regards, tom lane

On 02/24/2010 08:43 AM, Lou Picciano wrote:
> Tom -
>
> Didn't realize I was arm waving - was I? (Sometimes email falls well
> short...)
>
> We've managed a build of PostgreSQL 9.0-alpha4 - nice! However, the #
> make install command apparently(?) hiccups
> on a dependency on Jade (we ain't usin' it!)

I had forgotten to report it, but we saw this exact case at the Postgres
booth at SCaLE on someone's laptop, and did the same workaround. We
started with the alpha tarball. I believe the machine was Fedora, but
not sure which. I tried to repeat the issue in a fresh CentOS VM when I
got home but did not see the problem (perhaps because jade was part of
the install -- will have to check).

Related to this I have noticed in recent weeks on my own development
machine that "make install" takes *much* longer, but only sporadically,
due to the docs building.

Joe

Joe Conway <mail(at)joeconway(dot)com> writes:
> Related to this I have noticed in recent weeks on my own development
> machine that "make install" takes *much* longer, but only sporadically,
> due to the docs building.

This might be related to Peter's changes to the docs build procedure.
The way things work now is that if you've built the docs in the past,
and haven't cleaned them out with "make maintainer-clean" (distclean
doesn't cut it), then an ordinary "make install" will install the
html files --- and first it will update them if they're out of date
relative to the source SGML files.

This bites me regularly in my own source tree, since a "cvs update"
will often bring in new SGML files. My usual work flow is
"make distclean; cvs update; configure; make; make install"
and the lack of removal of previously built docs triggers an unwanted
rebuild. I've complained about this but I have to admit that I
don't see a very clean solution, since we can't have make distclean
removing the built docs. I'll probably have to adjust my workflow.

That doesn't in itself explain a problem with building from the
alpha tarball though. Is it possible there's a clock skew problem
in the tarball's file timestamps?

regards, tom lane

I wrote:
> That doesn't in itself explain a problem with building from the
> alpha tarball though. Is it possible there's a clock skew problem
> in the tarball's file timestamps?

I poked around in the alpha4 tarball and didn't find clock skew.
What I found out was that there's some fundamental fuzzy thinking
in the new docs build process:

* install-html depends on html depends on $(GENERATED_SGML)

* $(GENERATED_SGML) is removed by make clean, therefore also by
make distclean

Ergo, this type of failure is *guaranteed* when trying to build
from a distribution tarball. This needs to be rethought.

It might be sufficient to not clean $(GENERATED_SGML) except in
make maintainer-clean, but I'm not convinced that's a nice solution.

regards, tom lane

I wrote:
> * $(GENERATED_SGML) is removed by make clean, therefore also by
> make distclean
> Ergo, this type of failure is *guaranteed* when trying to build
> from a distribution tarball. This needs to be rethought.

I looked at this some more, and this time I noticed that the makefile
has

.SECONDARY: postgres.xml $(GENERATED_SGML) HTML.index

which puts the lie to the above theory. Also, in some simple testing
here I've not been able to reproduce the behavior of make wanting to
rebuild the HTML doc files when working from the alpha4 tarball. So
I'm feeling baffled again.

I can think of a couple of possible theories at this point:

* those reporting problems are using versions of gmake that have bugs in
handling .SECONDARY files.

* those reporting problems have re-autoconf'd. Since version.sgml
is declared to depend on $(top_srcdir)/configure, this would result
in a forced docs rebuild. It might help a bit to make it depend on
configure.in instead; though I'm far from sure this explains the
complaints.

regards, tom lane

Tom Lane wrote:
> I wrote:
>> * $(GENERATED_SGML) is removed by make clean, therefore also by
>> make distclean
>> Ergo, this type of failure is *guaranteed* when trying to build
>> from a distribution tarball. This needs to be rethought.
>
> I looked at this some more, and this time I noticed that the makefile
> has
>
> .SECONDARY: postgres.xml $(GENERATED_SGML) HTML.index
>
> which puts the lie to the above theory. Also, in some simple testing
> here I've not been able to reproduce the behavior of make wanting to
> rebuild the HTML doc files when working from the alpha4 tarball. So
> I'm feeling baffled again.

I have tested a few different ways on a fresh CentOS vm and have been
unable to reproduce the issue either (including make clean prior to
build, mv openjade and jade so they are not found by configure)

> I can think of a couple of possible theories at this point:
>
> * those reporting problems are using versions of gmake that have bugs in
> handling .SECONDARY files.

I added Deepak on whose machine I witnessed the problem to the cc list
so that he can tell us what OS it was in his case.

> * those reporting problems have re-autoconf'd. Since version.sgml
> is declared to depend on $(top_srcdir)/configure, this would result
> in a forced docs rebuild. It might help a bit to make it depend on
> configure.in instead; though I'm far from sure this explains the
> complaints.

I'm reasonably certain he did not do this before seeing the problem. I
think it was a simple untar, configure, make, make install...

Joe

Now, you've reminded me of something: That one or more versions of tar have trouble with very long file/directory names

I've run into this with one of the source trees we've been working in - was it here in PostgreSQL? Could this be a culprit?

----- Original Message -----
From: "Joseph Conway" <mail(at)joeconway(dot)com>
To: "Tom Lane" <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: "Peter Eisentraut" <peter_e(at)gmx(dot)net>, "Lou Picciano" <loupicciano(at)comcast(dot)net>, pgsql-bugs(at)postgresql(dot)org, bsdeepu(at)gmail(dot)com
Sent: Friday, February 26, 2010 12:29:43 AM GMT -05:00 US/Canada Eastern
Subject: Re: [BUGS] PostgreSQL-9.0alpha: jade required?

Tom Lane wrote:
> I wrote:
>> * $(GENERATED_SGML) is removed by make clean, therefore also by
>> make distclean
>> Ergo, this type of failure is *guaranteed* when trying to build
>> from a distribution tarball. This needs to be rethought.
>
> I looked at this some more, and this time I noticed that the makefile
> has
>
> .SECONDARY: postgres.xml $(GENERATED_SGML) HTML.index
>
> which puts the lie to the above theory. Also, in some simple testing
> here I've not been able to reproduce the behavior of make wanting to
> rebuild the HTML doc files when working from the alpha4 tarball. So
> I'm feeling baffled again.

I have tested a few different ways on a fresh CentOS vm and have been
unable to reproduce the issue either (including make clean prior to
build, mv openjade and jade so they are not found by configure)

> I can think of a couple of possible theories at this point:
>
> * those reporting problems are using versions of gmake that have bugs in
> handling .SECONDARY files.

I added Deepak on whose machine I witnessed the problem to the cc list
so that he can tell us what OS it was in his case.

> * those reporting problems have re-autoconf'd. Since version.sgml
> is declared to depend on $(top_srcdir)/configure, this would result
> in a forced docs rebuild. It might help a bit to make it depend on
> configure.in instead; though I'm far from sure this explains the
> complaints.

I'm reasonably certain he did not do this before seeing the problem. I
think it was a simple untar, configure, make, make install...

Joe

Lou Picciano <loupicciano(at)comcast(dot)net> writes:
> Now, you've reminded me of something: That one or more versions of tar have trouble with very long file/directory names
> I've run into this with one of the source trees we've been working in - was it here in PostgreSQL? Could this be a culprit?

I believe we dealt with that, by renaming some files shorter. It wasn't
anywhere near the docs files anyway.

regards, tom lane

On ons, 2010-02-24 at 12:16 -0500, Tom Lane wrote:
> Joe Conway <mail(at)joeconway(dot)com> writes:
> > Related to this I have noticed in recent weeks on my own development
> > machine that "make install" takes *much* longer, but only sporadically,
> > due to the docs building.
>
> This might be related to Peter's changes to the docs build procedure.
> The way things work now is that if you've built the docs in the past,
> and haven't cleaned them out with "make maintainer-clean" (distclean
> doesn't cut it), then an ordinary "make install" will install the
> html files --- and first it will update them if they're out of date
> relative to the source SGML files.

One possible change I have been pondering is to move that logic to the
"all" target. That is, if you have built the docs in the past and "make
install" would rebuild them, then "make all" will also rebuild them.
That doesn't fix the underlying issue for some people's workflows, but
at least it removes the element of surprise from "make install", which
is expected to be fast compared to "make all".

The problem with that approach, however, is that issuing "make all" in
the doc/src/sgml/ directory would never build anything at all before you
have build things the first time using some other route such as "make
html" and "make man".

One could probably also make this logic conditional upon being called in
a recursive make, which might actually work but will surely confuse the
next guy who tries out something off the normal path.

On Thu, Mar 4, 2010 at 9:15 AM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
> On ons, 2010-02-24 at 12:16 -0500, Tom Lane wrote:
>> Joe Conway <mail(at)joeconway(dot)com> writes:
>> > Related to this I have noticed in recent weeks on my own development
>> > machine that "make install" takes *much* longer, but only sporadically,
>> > due to the docs building.
>>
>> This might be related to Peter's changes to the docs build procedure.
>> The way things work now is that if you've built the docs in the past,
>> and haven't cleaned them out with "make maintainer-clean" (distclean
>> doesn't cut it), then an ordinary "make install" will install the
>> html files --- and first it will update them if they're out of date
>> relative to the source SGML files.
>
> One possible change I have been pondering is to move that logic to the
> "all" target.  That is, if you have built the docs in the past and "make
> install" would rebuild them, then "make all" will also rebuild them.
> That doesn't fix the underlying issue for some people's workflows, but
> at least it removes the element of surprise from "make install", which
> is expected to be fast compared to "make all".
>
> The problem with that approach, however, is that issuing "make all" in
> the doc/src/sgml/ directory would never build anything at all before you
> have build things the first time using some other route such as "make
> html" and "make man".
>
> One could probably also make this logic conditional upon being called in
> a recursive make, which might actually work but will surely confuse the
> next guy who tries out something off the normal path.

I think that the whole idea of make targets building different things
depending on what you've built previously is confusing,
counterintuitive, and illogical. make all should either build the
docs, or not; trying to guess what the user wants is, IMO, nuts.

</rant>

...Robert

May I add 2 cents?

While I'd agree a 'make all' should, uh... make _all_, and that make building based on lots of guessing is counterintuitive,
an option to configure like:

./configure --no-docs or
./configure --with-htmldocs-only

- with some of the obvious variants - might be very useful. Especially if docs are available elsewhere, and for those of us who may not have the inclination to retain/host all docs for all packages we build, or may prefer a specific format anyway, and who are really looking for the core software in any case...

Lou Picciano

----- Original Message -----
From: "Robert Haas" <robertmhaas(at)gmail(dot)com>
To: "Peter Eisentraut" <peter_e(at)gmx(dot)net>
Cc: "Tom Lane" <tgl(at)sss(dot)pgh(dot)pa(dot)us>, "Joe Conway" <mail(at)joeconway(dot)com>, "Lou Picciano" <loupicciano(at)comcast(dot)net>, pgsql-bugs(at)postgresql(dot)org
Sent: Thursday, March 4, 2010 12:09:00 PM GMT -05:00 US/Canada Eastern
Subject: Re: [BUGS] PostgreSQL-9.0alpha: jade required?

On Thu, Mar 4, 2010 at 9:15 AM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
> On ons, 2010-02-24 at 12:16 -0500, Tom Lane wrote:
>> Joe Conway <mail(at)joeconway(dot)com> writes:
>> > Related to this I have noticed in recent weeks on my own development
>> > machine that "make install" takes *much* longer, but only sporadically,
>> > due to the docs building.
>>
>> This might be related to Peter's changes to the docs build procedure.
>> The way things work now is that if you've built the docs in the past,
>> and haven't cleaned them out with "make maintainer-clean" (distclean
>> doesn't cut it), then an ordinary "make install" will install the
>> html files --- and first it will update them if they're out of date
>> relative to the source SGML files.
>
> One possible change I have been pondering is to move that logic to the
> "all" target. That is, if you have built the docs in the past and "make
> install" would rebuild them, then "make all" will also rebuild them.
> That doesn't fix the underlying issue for some people's workflows, but
> at least it removes the element of surprise from "make install", which
> is expected to be fast compared to "make all".
>
> The problem with that approach, however, is that issuing "make all" in
> the doc/src/sgml/ directory would never build anything at all before you
> have build things the first time using some other route such as "make
> html" and "make man".
>
> One could probably also make this logic conditional upon being called in
> a recursive make, which might actually work but will surely confuse the
> next guy who tries out something off the normal path.

I think that the whole idea of make targets building different things
depending on what you've built previously is confusing,
counterintuitive, and illogical. make all should either build the
docs, or not; trying to guess what the user wants is, IMO, nuts.

</rant>

...Robert

On tor, 2010-03-04 at 12:09 -0500, Robert Haas wrote:
> I think that the whole idea of make targets building different things
> depending on what you've built previously is confusing,
> counterintuitive, and illogical. make all should either build the
> docs, or not; trying to guess what the user wants is, IMO, nuts.

The consequence of that would effectively be to make the SGML tools
mandatory for a CVS checkout.

On tor, 2010-03-04 at 17:53 +0000, Lou Picciano wrote:
> While I'd agree a 'make all' should, uh... make _all_, and that make
> building based on lots of guessing is counterintuitive, an option to
> configure like:
>
> ./configure --no-docs or ./configure --with-htmldocs-only
>
> - with some of the obvious variants - might be very useful.
> Especially if docs are available elsewhere, and for those of us who
> may not have the inclination to retain/host all docs for all packages
> we build, or may prefer a specific format anyway, and who are really
> looking for the core software in any case...

But that would be a negative regression for end users, who we want to
have the docs available by default, so they can read them.

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> On tor, 2010-03-04 at 17:53 +0000, Lou Picciano wrote:
>> ./configure --no-docs or ./configure --with-htmldocs-only

> But that would be a negative regression for end users, who we want to
> have the docs available by default, so they can read them.

"End users" in that sense would almost certainly be working from a
distribution tarball, if not a prepackaged distro. I don't think
this discussion is about them; it's about what is most convenient
for developers. As a developer, I don't find the current arrangement
convenient in the least.

What I'd be for is breaking the docs out as a separate top-level target,
ie "make docs", "make install-docs". I don't much care for Lou's
suggestion of tying it to a configure option because that imposes the
significant additional cost of re-configuring when I change my mind.
I do need to be *able* to build the docs, I just don't want it happening
by surprise.

regards, tom lane

Tom's suggestion is much better than mine. I concur fully.

My issue, essentially, is that I want to get on with the business of actually testing these alphas...

The 'Jade effect' has become an obstacle; time wasted.

And, though I do want the documentation - ultimately - I'd like the option of make-ing a non Jade-dependent version of them.
Don't give the proverbial 'two hoots' about Jade.

Regards All, Lou

----- Original Message -----
From: "Tom Lane" <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: "Peter Eisentraut" <peter_e(at)gmx(dot)net>
Cc: "Lou Picciano" <loupicciano(at)comcast(dot)net>, "Robert Haas" <robertmhaas(at)gmail(dot)com>, "Joe Conway" <mail(at)joeconway(dot)com>, pgsql-bugs(at)postgresql(dot)org
Sent: Friday, March 5, 2010 8:09:54 AM GMT -05:00 US/Canada Eastern
Subject: Re: [BUGS] PostgreSQL-9.0alpha: jade required?

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> On tor, 2010-03-04 at 17:53 +0000, Lou Picciano wrote:
>> ./configure --no-docs or ./configure --with-htmldocs-only

> But that would be a negative regression for end users, who we want to
> have the docs available by default, so they can read them.

"End users" in that sense would almost certainly be working from a
distribution tarball, if not a prepackaged distro. I don't think
this discussion is about them; it's about what is most convenient
for developers. As a developer, I don't find the current arrangement
convenient in the least.

What I'd be for is breaking the docs out as a separate top-level target,
ie "make docs", "make install-docs". I don't much care for Lou's
suggestion of tying it to a configure option because that imposes the
significant additional cost of re-configuring when I change my mind.
I do need to be *able* to build the docs, I just don't want it happening
by surprise.

regards, tom lane

On fre, 2010-03-05 at 08:09 -0500, Tom Lane wrote:
> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> > On tor, 2010-03-04 at 17:53 +0000, Lou Picciano wrote:
> >> ./configure --no-docs or ./configure --with-htmldocs-only
>
> > But that would be a negative regression for end users, who we want to
> > have the docs available by default, so they can read them.
>
> "End users" in that sense would almost certainly be working from a
> distribution tarball, if not a prepackaged distro. I don't think
> this discussion is about them; it's about what is most convenient
> for developers. As a developer, I don't find the current arrangement
> convenient in the least.
>
> What I'd be for is breaking the docs out as a separate top-level target,
> ie "make docs", "make install-docs". I don't much care for Lou's
> suggestion of tying it to a configure option because that imposes the
> significant additional cost of re-configuring when I change my mind.
> I do need to be *able* to build the docs, I just don't want it happening
> by surprise.

What this discussion is about is balancing between the needs of
developers and the (perceived) needs of end users. The problem is, as
is becoming apparent, that you can't really do that without making a
mess of the makefiles or breaking things for one side or the other.

By splitting out the doc building into a separate target, we will have
less users installing the documentation. Lou's suggestion is
essentially equivalent in that regard. I don't know if that's where we
want to go, but if so, it's a trivial and clean solution.

I don't know if "make world" is targeted at end users or developers or
the build farm. But the doc build is there anyway.

On Fri, Mar 5, 2010 at 8:09 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
>> On tor, 2010-03-04 at 17:53 +0000, Lou Picciano wrote:
>>> ./configure --no-docs   or ./configure --with-htmldocs-only
>
>> But that would be a negative regression for end users, who we want to
>> have the docs available by default, so they can read them.
>
> "End users" in that sense would almost certainly be working from a
> distribution tarball, if not a prepackaged distro.  I don't think
> this discussion is about them; it's about what is most convenient
> for developers.  As a developer, I don't find the current arrangement
> convenient in the least.
>
> What I'd be for is breaking the docs out as a separate top-level target,
> ie "make docs", "make install-docs".  I don't much care for Lou's
> suggestion of tying it to a configure option because that imposes the
> significant additional cost of re-configuring when I change my mind.
> I do need to be *able* to build the docs, I just don't want it happening
> by surprise.

Agreed. I think that is a much better solution.

...Robert

On Fri, Mar 5, 2010 at 9:21 AM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
> By splitting out the doc building into a separate target, we will have
> less users installing the documentation.

I don't see why. And even if it's true, it just means some people
were installing the docs "by accident" before even though they didn't
really want them.

...Robert

Robert Haas escribió:
> On Fri, Mar 5, 2010 at 8:09 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> > What I'd be for is breaking the docs out as a separate top-level target,
> > ie "make docs", "make install-docs".  I don't much care for Lou's
> > suggestion of tying it to a configure option because that imposes the
> > significant additional cost of re-configuring when I change my mind.
> > I do need to be *able* to build the docs, I just don't want it happening
> > by surprise.
>
> Agreed. I think that is a much better solution.

+1, and I think docs should be a dependency of world, too.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

On Thu, Mar 11, 2010 at 8:50 PM, Alvaro Herrera
<alvherre(at)commandprompt(dot)com> wrote:
> Robert Haas escribió:
>> On Fri, Mar 5, 2010 at 8:09 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
>> > What I'd be for is breaking the docs out as a separate top-level target,
>> > ie "make docs", "make install-docs".  I don't much care for Lou's
>> > suggestion of tying it to a configure option because that imposes the
>> > significant additional cost of re-configuring when I change my mind.
>> > I do need to be *able* to build the docs, I just don't want it happening
>> > by surprise.
>>
>> Agreed.  I think that is a much better solution.
>
> +1, and I think docs should be a dependency of world, too.

Definitely.

...Robert