Patch: Remove all declarations from pg_attribute.h, consolidate BKI scripts

Greetings,

Following up on my experimental patch last month to revamp the BKI
infrastructure, I am proposing a less invasive set of changes with the
hope of offering something committable. Some of these were discussed
by Robert Haas and others last summer.

1. Remove all DATA() declarations from pg_attribute.h, since they are
easily generated. Introduce a new BKI pseudo-command
BKI_NAILED_IN_CACHE, which indicates that relcache.c needs a
Schema_pg_foo declaration for that catalog. Place these declarations
in a new header schemapg.h. This will reduce the effort to add or
change critical tables.

2. Use identical scripts on Posix and Windows systems, using Perl 5.6
(no CPAN modules needed). The grepping of the catalog headers is done
by Catalog.pm, which gives the scripts gen_bki.pl and gen_fmgr.pl a
structured interface to the data. The pg_type info is saved so that
the relevant fields can be copied into those of pg_attribute.

3. Make the BKI files, fmgrtab.c, fmgroids.h, and schemapg.h distprep
targets, so distribution tarballs can still be built without Perl on
Posix systems.

Feedback on the Makefile changes would be appreciated, since that was
the hardest part for me. The MSVC changes are untested.

John Naylor

On Sun, Dec 20, 2009 at 7:20 PM, John Naylor <jcnaylor(at)gmail(dot)com> wrote:
> Greetings,
>
> Following up on my experimental patch last month to revamp the BKI
> infrastructure, I am proposing a less invasive set of changes with the
> hope of offering something committable. Some of these were discussed
> by Robert Haas and others last summer.
>
> 1. Remove all DATA() declarations from pg_attribute.h, since they are
> easily generated. Introduce a new BKI pseudo-command
> BKI_NAILED_IN_CACHE, which indicates that relcache.c needs a
> Schema_pg_foo declaration for that catalog. Place these declarations
> in a new header schemapg.h. This will reduce the effort to add or
> change critical tables.
>
> 2. Use identical scripts on Posix and Windows systems, using Perl 5.6
> (no CPAN modules needed). The grepping of the catalog headers is done
> by Catalog.pm, which gives the scripts gen_bki.pl and gen_fmgr.pl a
> structured interface to the data. The pg_type info is saved so that
> the relevant fields can be copied into those of pg_attribute.
>
> 3. Make the BKI files, fmgrtab.c, fmgroids.h, and schemapg.h distprep
> targets, so distribution tarballs can still be built without Perl on
> Posix systems.
>
> Feedback on the Makefile changes would be appreciated, since that was
> the hardest part for me. The MSVC changes are untested.

This is really nice. I haven't done a full review, but it seems as if
you've eliminated some of the more controversial aspects of what I did
before, as well as done some good cleanup and refactoring. One minor
nit is that I think we should end up with genbki.pl rather than
gen_bki.pl, in the interest of changing nothing without a good reason.

A more important point is whether we really need to make this
dependent on Perl 5.6 or later. What features are we using here that
actually require Perl 5.6? I suspect the answer is "none, but we
don't like writing the code in a way that is backward compatible to
crufty, ancient versions of Perl", to which my response is "get over
it". :-) I always use the three-argument form of open() in all new
code, but for fixed strings the two-argument form is just as good and
has been supported for far longer. Any suggestions that we should
prefer clean code over portability are, in my opinion, non-starters.

Again, thank you for your work on this!

...Robert

Hi,

On Monday 21 December 2009 02:23:39 Robert Haas wrote:
> A more important point is whether we really need to make this
> dependent on Perl 5.6 or later. What features are we using here that
> actually require Perl 5.6? I suspect the answer is "none, but we
> don't like writing the code in a way that is backward compatible to
> crufty, ancient versions of Perl", to which my response is "get over
> it". :-) I always use the three-argument form of open() in all new
> code, but for fixed strings the two-argument form is just as good and
> has been supported for far longer. Any suggestions that we should
> prefer clean code over portability are, in my opinion, non-starters.
I dont see a platform without perl 5.6 where a new enough flex/bison is
available...

Andres

Andres Freund <andres(at)anarazel(dot)de> writes:
> On Monday 21 December 2009 02:23:39 Robert Haas wrote:
>> A more important point is whether we really need to make this
>> dependent on Perl 5.6 or later.

> I dont see a platform without perl 5.6 where a new enough flex/bison is
> available...

That argument only holds water if you are willing to follow the same
rules as we use for flex/bison, ie, they are not needed to build from
a source tarball. Otherwise this *is* moving the goalposts on required
tool support.

regards, tom lane

On Monday 21 December 2009 04:23:57 Tom Lane wrote:
> Andres Freund <andres(at)anarazel(dot)de> writes:
> > On Monday 21 December 2009 02:23:39 Robert Haas wrote:
> >> A more important point is whether we really need to make this
> >> dependent on Perl 5.6 or later.
> >
> > I dont see a platform without perl 5.6 where a new enough flex/bison is
> > available...
>
> That argument only holds water if you are willing to follow the same
> rules as we use for flex/bison, ie, they are not needed to build from
> a source tarball. Otherwise this *is* moving the goalposts on required
> tool support.
The patch already does that if I understood John correctly.

Andres

On Sun, Dec 20, 2009 at 7:24 PM, Andres Freund <andres(at)anarazel(dot)de> wrote:
> On Monday 21 December 2009 04:23:57 Tom Lane wrote:
>> Andres Freund <andres(at)anarazel(dot)de> writes:
>> > On Monday 21 December 2009 02:23:39 Robert Haas wrote:
>> >> A more important point is whether we really need to make this
>> >> dependent on Perl 5.6 or later.
>> >
>> > I dont see a platform without perl 5.6 where a new enough flex/bison is
>> > available...
>>
>> That argument only holds water if you are willing to follow the same
>> rules as we use for flex/bison, ie, they are not needed to build from
>> a source tarball.  Otherwise this *is* moving the goalposts on required
>> tool support.
> The patch already does that if I understood John correctly.

Yes, everything output by Perl in my patch is a distprep target.

Some minor changes would enable it to work on 5.0, but I'd like to
make sure that's necessary.

So, what Perl version should be targeted for those building from CVS,
and is that already documented somewhere? Does anyone know the
earliest version on the build farm?

John

On Sun, Dec 20, 2009 at 10:23 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Andres Freund <andres(at)anarazel(dot)de> writes:
>> On Monday 21 December 2009 02:23:39 Robert Haas wrote:
>>> A more important point is whether we really need to make this
>>> dependent on Perl 5.6 or later.
>
>> I dont see a platform without perl 5.6 where a new enough flex/bison is
>> available...
>
> That argument only holds water if you are willing to follow the same
> rules as we use for flex/bison, ie, they are not needed to build from
> a source tarball.  Otherwise this *is* moving the goalposts on required
> tool support.

I believe that we have long had agreement on making the relevant files
distprep targets, so this will not be an issue. Anyway, the whole
thing is a silly argument anyway: we can certainly make this
compatible back even as far as Perl 5.0 if need be for very little
extra work.

What is worth a little bit of effort to establish is exactly what
version of Perl we're already depending on, so that we can document
that for the benefit of future tool writers. There's no reason why
this particular thing needs to be compatible further back than what is
already required otherwise.

...Robert

On Sun, Dec 20, 2009 at 10:55:55PM -0500, Robert Haas wrote:
> On Sun, Dec 20, 2009 at 10:23 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > Andres Freund <andres(at)anarazel(dot)de> writes:
> >> On Monday 21 December 2009 02:23:39 Robert Haas wrote:
> >>> A more important point is whether we really need to make this
> >>> dependent on Perl 5.6 or later.
> >
> >> I dont see a platform without perl 5.6 where a new enough flex/bison is
> >> available...
> >
> > That argument only holds water if you are willing to follow the same
> > rules as we use for flex/bison, ie, they are not needed to build from
> > a source tarball.  Otherwise this *is* moving the goalposts on required
> > tool support.
>
> I believe that we have long had agreement on making the relevant files
> distprep targets, so this will not be an issue. Anyway, the whole
> thing is a silly argument anyway: we can certainly make this
> compatible back even as far as Perl 5.0 if need be for very little
> extra work.

FYI Perl 5.6.0 was released in March 2000. 5.6.2 in November 2003.

The general perception in the perl community is that 5.8.x is the oldest
perl that's widely used and that gets regular automated testing:
http://stats.cpantesters.org/pmatrix-wide.html
There's very little use of 5.6.x (typically 5.6.2).
There's almost no use of earlier versions.

> What is worth a little bit of effort to establish is exactly what
> version of Perl we're already depending on, so that we can document
> that for the benefit of future tool writers. There's no reason why
> this particular thing needs to be compatible further back than what is
> already required otherwise.

I'd like to know the earliest version of perl that's supported for
PL/Perl, if that's ever been determined (I've not seen it in the docs).

plperl requires Safe v2.09, released in Oct 2003 and included in 5.8.1.
That version, and later versions, have only been tested back to perl 5.6.

I'd recommend Perl 5.6(.2) as a minimum for threads/multiplicity.
Perl 5.5 had complelely different, and unsafe, code for threads
and no concept of multiplicity.

I'd recommend Perl 5.8(.1) as a minimum for UTF-8 databases.
Perl 5.7 had a complelely different, and flawed, concept of
Unicode operation.

Tim.

p.s. I'll test PL/Perl with perl 5.6.2 (assuming I can still build it on
my system) as part of testing the PL/Perl patches I'm working on.

On Mon, Dec 21, 2009 at 5:45 AM, Tim Bunce <Tim(dot)Bunce(at)pobox(dot)com> wrote:
> FYI Perl 5.6.0 was released in March 2000. 5.6.2 in November 2003.

Gosh, I feel old. I started on Perl 4.036.

>> What is worth a little bit of effort to establish is exactly what
>> version of Perl we're already depending on, so that we can document
>> that for the benefit of future tool writers.  There's no reason why
>> this particular thing needs to be compatible further back than what is
>> already required otherwise.
>
> I'd like to know the earliest version of perl that's supported for
> PL/Perl, if that's ever been determined (I've not seen it in the docs).
>
> plperl requires Safe v2.09, released in Oct 2003 and included in 5.8.1.
> That version, and later versions, have only been tested back to perl 5.6.
>
> I'd recommend Perl 5.6(.2) as a minimum for threads/multiplicity.
> Perl 5.5 had complelely different, and unsafe, code for threads
> and no concept of multiplicity.
>
> I'd recommend Perl 5.8(.1) as a minimum for UTF-8 databases.
> Perl 5.7 had a complelely different, and flawed, concept of
> Unicode operation.
>
> Tim.
>
> p.s. I'll test PL/Perl with perl 5.6.2 (assuming I can still build it on
> my system) as part of testing the PL/Perl patches I'm working on.

It's possible that we might support building the release in general
with an older version of Perl than what we support for PL/perl, but it
sounds like we may have already burned this bridge if the MSVC stuff
expects something newer. Maybe we should just document that 5.6 is
the minimum supported version to build from source and be done with it
(we should clarify what MSVC requires). If someone is running a 1990s
version of Perl in 2010 when 8.5 comes out, they can always make
distprep on another machine. There can't be many people who will want
a brand-new PostgreSQL and an ancient Perl, I think.

...Robert

2009/12/21 Robert Haas <robertmhaas(at)gmail(dot)com>:
> On Mon, Dec 21, 2009 at 5:45 AM, Tim Bunce <Tim(dot)Bunce(at)pobox(dot)com> wrote:
>> FYI Perl 5.6.0 was released in March 2000. 5.6.2 in November 2003.
>
> Gosh, I feel old.  I started on Perl 4.036.
>
>>> What is worth a little bit of effort to establish is exactly what
>>> version of Perl we're already depending on, so that we can document
>>> that for the benefit of future tool writers.  There's no reason why
>>> this particular thing needs to be compatible further back than what is
>>> already required otherwise.
>>
>> I'd like to know the earliest version of perl that's supported for
>> PL/Perl, if that's ever been determined (I've not seen it in the docs).
>>
>> plperl requires Safe v2.09, released in Oct 2003 and included in 5.8.1.
>> That version, and later versions, have only been tested back to perl 5.6.
>>
>> I'd recommend Perl 5.6(.2) as a minimum for threads/multiplicity.
>> Perl 5.5 had complelely different, and unsafe, code for threads
>> and no concept of multiplicity.
>>
>> I'd recommend Perl 5.8(.1) as a minimum for UTF-8 databases.
>> Perl 5.7 had a complelely different, and flawed, concept of
>> Unicode operation.
>>
>> Tim.
>>
>> p.s. I'll test PL/Perl with perl 5.6.2 (assuming I can still build it on
>> my system) as part of testing the PL/Perl patches I'm working on.
>
> It's possible that we might support building the release in general
> with an older version of Perl than what we support for PL/perl, but it
> sounds like we may have already burned this bridge if the MSVC stuff
> expects something newer.  Maybe we should just document that 5.6 is

The MSVC stuff is only tested on 5.8.

> the minimum supported version to build from source and be done with it
> (we should clarify what MSVC requires).  If someone is running a 1990s

You mean somehting like
http://www.postgresql.org/docs/8.4/static/install-win32-full.html#AEN23266

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On Dec 21, 2009, at 7:46 AM, Magnus Hagander <magnus(at)hagander(dot)net>
wrote:
> 2009/12/21 Robert Haas <robertmhaas(at)gmail(dot)com>:
>> On Mon, Dec 21, 2009 at 5:45 AM, Tim Bunce <Tim(dot)Bunce(at)pobox(dot)com>
>> wrote:
>>> FYI Perl 5.6.0 was released in March 2000. 5.6.2 in November 2003.
>>
>> Gosh, I feel old. I started on Perl 4.036.
>>
>>>> What is worth a little bit of effort to establish is exactly what
>>>> version of Perl we're already depending on, so that we can document
>>>> that for the benefit of future tool writers. There's no reason why
>>>> this particular thing needs to be compatible further back than
>>>> what is
>>>> already required otherwise.
>>>
>>> I'd like to know the earliest version of perl that's supported for
>>> PL/Perl, if that's ever been determined (I've not seen it in the
>>> docs).
>>>
>>> plperl requires Safe v2.09, released in Oct 2003 and included in
>>> 5.8.1.
>>> That version, and later versions, have only been tested back to
>>> perl 5.6.
>>>
>>> I'd recommend Perl 5.6(.2) as a minimum for threads/multiplicity.
>>> Perl 5.5 had complelely different, and unsafe, code for threads
>>> and no concept of multiplicity.
>>>
>>> I'd recommend Perl 5.8(.1) as a minimum for UTF-8 databases.
>>> Perl 5.7 had a complelely different, and flawed, concept of
>>> Unicode operation.
>>>
>>> Tim.
>>>
>>> p.s. I'll test PL/Perl with perl 5.6.2 (assuming I can still build
>>> it on
>>> my system) as part of testing the PL/Perl patches I'm working on.
>>
>> It's possible that we might support building the release in general
>> with an older version of Perl than what we support for PL/perl, but
>> it
>> sounds like we may have already burned this bridge if the MSVC stuff
>> expects something newer. Maybe we should just document that 5.6 is
>
> The MSVC stuff is only tested on 5.8.
>
>> the minimum supported version to build from source and be done with
>> it
>> (we should clarify what MSVC requires). If someone is running a
>> 1990s
>
> You mean somehting like
> http://www.postgresql.org/docs/8.4/static/install-win32-full.html#AEN23266

Yeah, very much like that. :-)

...Robert

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> 2009/12/21 Robert Haas <robertmhaas(at)gmail(dot)com>:
>> On Mon, Dec 21, 2009 at 5:45 AM, Tim Bunce <Tim(dot)Bunce(at)pobox(dot)com> wrote:
>>> plperl requires Safe v2.09, released in Oct 2003 and included in 5.8.1.
>>> That version, and later versions, have only been tested back to perl 5.6.

>>> I'd recommend Perl 5.8(.1) as a minimum for UTF-8 databases.
>>> Perl 5.7 had a complelely different, and flawed, concept of
>>> Unicode operation.

> The MSVC stuff is only tested on 5.8.

Given the above three things it seems like we could define 5.8.1 as the
minimum required version.

regards, tom lane

On Mon, Dec 21, 2009 at 10:09:58AM -0500, Tom Lane wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
> > 2009/12/21 Robert Haas <robertmhaas(at)gmail(dot)com>:
> >> On Mon, Dec 21, 2009 at 5:45 AM, Tim Bunce <Tim(dot)Bunce(at)pobox(dot)com> wrote:
> >>> plperl requires Safe v2.09, released in Oct 2003 and included in 5.8.1.
> >>> That version, and later versions, have only been tested back to perl 5.6.
>
> >>> I'd recommend Perl 5.8(.1) as a minimum for UTF-8 databases.
> >>> Perl 5.7 had a complelely different, and flawed, concept of
> >>> Unicode operation.
>
> > The MSVC stuff is only tested on 5.8.
>
> Given the above three things it seems like we could define 5.8.1 as the
> minimum required version.

I'd be delighted with that.

Tim.

On Dec 21, 2009, at 7:18 AM, Tim Bunce wrote:

>> Given the above three things it seems like we could define 5.8.1 as the
>> minimum required version.
>
> I'd be delighted with that.

+1

BTW Tim, have you tested with 5.11 yet?

Best,

David

On Mon, Dec 21, 2009 at 08:22:54AM -0800, David E. Wheeler wrote:
> On Dec 21, 2009, at 7:18 AM, Tim Bunce wrote:
>
> >> Given the above three things it seems like we could define 5.8.1 as the
> >> minimum required version.
> >
> > I'd be delighted with that.
>
> +1
>
> BTW Tim, have you tested with 5.11 yet?

Not recently. I'm putting the finishing touches on my updated feature
patch now. I hope to post it in the next few days. I'll test with 5.8.1
and 5.11 before I do.

Tim.