Re: Perl 5.12 complains about ecpg parser-hacking scripts

On Fedora 14 I see these warnings while building ecpg's preproc.y:

Use of assignment to $[ is deprecated at ./parse.pl line 21.
Use of assignment to $[ is deprecated at ./check_rules.pl line 18.

Any Perl experts want to make those go away?

regards, tom lane

On 01/22/2011 06:06 PM, Tom Lane wrote:
> On Fedora 14 I see these warnings while building ecpg's preproc.y:
>
> Use of assignment to $[ is deprecated at ./parse.pl line 21.
> Use of assignment to $[ is deprecated at ./check_rules.pl line 18.
>
> Any Perl experts want to make those go away?
>
>

I think these really need to be rewritten from scratch. They look like
they were written by someone who never heard of Perl 5 (it's only about
16 years old).

We could probably silence the warnings by putting:

use warnings; no warnings qw(deprecate);

at the top of the file, but that would be rather like putting a bandaid
on a massive chest wound.

cheers

andrew

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> On 01/22/2011 06:06 PM, Tom Lane wrote:
>> On Fedora 14 I see these warnings while building ecpg's preproc.y:
>>
>> Use of assignment to $[ is deprecated at ./parse.pl line 21.
>> Use of assignment to $[ is deprecated at ./check_rules.pl line 18.
>>
>> Any Perl experts want to make those go away?

> I think these really need to be rewritten from scratch.

I noticed they didn't "use strict" either ... have at it.

> We could probably silence the warnings by putting:
> use warnings; no warnings qw(deprecate);

Doesn't seem like a step forward.

regards, tom lane

On 01/22/2011 08:48 PM, Tom Lane wrote:
> Andrew Dunstan<andrew(at)dunslane(dot)net> writes:
>> On 01/22/2011 06:06 PM, Tom Lane wrote:
>>> On Fedora 14 I see these warnings while building ecpg's preproc.y:
>>>
>>> Use of assignment to $[ is deprecated at ./parse.pl line 21.
>>> Use of assignment to $[ is deprecated at ./check_rules.pl line 18.
>>>
>>> Any Perl experts want to make those go away?
>> I think these really need to be rewritten from scratch.
> I noticed they didn't "use strict" either ... have at it.
>

If I do it's unlikely to be any time soon. I wasn't putting my hand up
for the job, just pointing out that fixing these would barely be
scratching the surface of what's needed.

But there are quite a few perlheads around. ISTR Matt Trout was
muttering about these scripts on IRC recently.

cheers

andrew

On 23 January 2011 13:14, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
> But there are quite a few perlheads around. ISTR Matt Trout was muttering
> about these scripts on IRC recently.

A quick cleanup of the check_rules.pl...

It's a starting point at least.

On Sat, Jan 22, 2011 at 08:40:13PM -0500, Andrew Dunstan wrote:
> I think these really need to be rewritten from scratch. They look
> like they were written by someone who never heard of Perl 5 (it's
> only about 16 years old).

You might remember that we had this discussion before. The script was written
in awk and then translated to perl by a2p. We knew that this code was less than
optimal, but at least it works.

As I already said when the script was introduced, I would love to have a real
perl solution, but I'm not a perl programmer by any means.

Michael
--
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org
Jabber: michael.meskes at googlemail dot com
VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL

On 01/22/2011 09:28 PM, kris(at)shannon(dot)id(dot)au wrote:
> On 23 January 2011 13:14, Andrew Dunstan<andrew(at)dunslane(dot)net> wrote:
>> But there are quite a few perlheads around. ISTR Matt Trout was muttering
>> about these scripts on IRC recently.
>
> A quick cleanup of the check_rules.pl...
>
> It's a starting point at least.
>
>
>
>

Oh! Perl is my favorite. Kris, if you're not going to, I'd love to work on this.

-Andy

On 01/23/2011 06:11 AM, Michael Meskes wrote:
> On Sat, Jan 22, 2011 at 08:40:13PM -0500, Andrew Dunstan wrote:
>> I think these really need to be rewritten from scratch. They look
>> like they were written by someone who never heard of Perl 5 (it's
>> only about 16 years old).
> You might remember that we had this discussion before. The script was written
> in awk and then translated to perl by a2p. We knew that this code was less than
> optimal, but at least it works.

Oh, I'm sorry, I'd totally forgotten that. At least it explains why it
looks like it's from a time warp. I bet the perl guys haven't touched
a2p for many many years.

cheers

andrew

On 01/23/2011 08:29 AM, Andy Colson wrote:
> On 01/22/2011 09:28 PM, kris(at)shannon(dot)id(dot)au wrote:
>> On 23 January 2011 13:14, Andrew Dunstan<andrew(at)dunslane(dot)net> wrote:
>>> But there are quite a few perlheads around. ISTR Matt Trout was muttering
>>> about these scripts on IRC recently.

Ok, so I've figured out what its purpose is.

Are there other problems with this script? Does it not pull out the rule names correct all the time or something? What problem was Matt having with it?

I think rewriting from scratch is overkill, unless this script is just failing. The code to pull the rule names out is a bit complex, and it seems to work, so I'd rather not touch it.

Are there other things you wished this script did? (reports, counts, etc)

-Andy

On 01/23/2011 10:16 AM, Andy Colson wrote:
> On 01/23/2011 08:29 AM, Andy Colson wrote:
>> On 01/22/2011 09:28 PM, kris(at)shannon(dot)id(dot)au wrote:
>>> On 23 January 2011 13:14, Andrew Dunstan<andrew(at)dunslane(dot)net> wrote:
>>>> But there are quite a few perlheads around. ISTR Matt Trout was
>>>> muttering
>>>> about these scripts on IRC recently.
>
> Ok, so I've figured out what its purpose is.
>
> Are there other problems with this script? Does it not pull out the
> rule names correct all the time or something? What problem was Matt
> having with it?
>
> I think rewriting from scratch is overkill, unless this script is just
> failing. The code to pull the rule names out is a bit complex, and it
> seems to work, so I'd rather not touch it.
>
> Are there other things you wished this script did? (reports, counts,
> etc)
>
>

It's doing the right thing. But it's really spaghetti code, generated by
a2p. Matt was just (rightly) offended by the $[ setting, IIRC.

The point is that it's close to being totally unmaintainable.

But if you think you can remedy it without rewriting it, go for it.

I think minimum requirements would be:

* avoid setting $[
* use strict;
* comments saying what it's actually doing

We want to be in a situation where of it ever does break because of some
external change, we're not left having to wade through the crap to find
out how to fix it.

cheers

andrew

On Sun, Jan 23, 2011 at 09:16:33AM -0600, Andy Colson wrote:
> Are there other problems with this script? Does it not pull out the rule names correct all the time or something? What problem was Matt having with it?

No, afaict it works correctly but throws some warnings.

Michael
--
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org
Jabber: michael.meskes at googlemail dot com
VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL

On 01/23/2011 10:06 AM, Andrew Dunstan wrote:
>
>
> On 01/23/2011 10:16 AM, Andy Colson wrote:
>> On 01/23/2011 08:29 AM, Andy Colson wrote:
>>> On 01/22/2011 09:28 PM, kris(at)shannon(dot)id(dot)au wrote:
>>>> On 23 January 2011 13:14, Andrew Dunstan<andrew(at)dunslane(dot)net> wrote:
>>>>> But there are quite a few perlheads around. ISTR Matt Trout was muttering
>>>>> about these scripts on IRC recently.
>>
>> Ok, so I've figured out what its purpose is.
>>
>> Are there other problems with this script? Does it not pull out the rule names correct all the time or something? What problem was Matt having with it?
>>
>> I think rewriting from scratch is overkill, unless this script is just failing. The code to pull the rule names out is a bit complex, and it seems to work, so I'd rather not touch it.
>>
>> Are there other things you wished this script did? (reports, counts, etc)
>>
>>
>
>
> It's doing the right thing. But it's really spaghetti code, generated by a2p. Matt was just (rightly) offended by the $[ setting, IIRC.
>
> The point is that it's close to being totally unmaintainable.
>
> But if you think you can remedy it without rewriting it, go for it.
>
> I think minimum requirements would be:
>
> * avoid setting $[
> * use strict;
> * comments saying what it's actually doing
>
>
> We want to be in a situation where of it ever does break because of some external change, we're not left having to wade through the crap to find out how to fix it.
>
> cheers
>
> andrew

** OOPS, forenote: I only noticed check_rules.pl. That's the only file I looked at. I'll go look at parse.pl right now. Below refers to check_rules.pl only.

I can comment the code. $[ can safely be removed (from check_rules.pl) because we are not even using arrays.

I disagree (check_rules.pl) is not spaghetti code. There are two loops, one thru gram.y to pick out the rule names, and one through ecpg.addons to make sure they are in use.

Unmaintainable is another story. Parsing gram.y for rule names, then combining them to make up the same name in ecpg.addons is a little complex. But then I think it would be a little complex in any language. A little change to the syntax/layout of either file and it could go bad, but you'd have that same problem with any other language too.

Is there anyway to make bison/yacc/gcc/etc spit out the rule names?

I'm not sure rule name is the proper words. In gram.y we have:

stmt :
AlterDatabaseStmt
| AlterDatabaseSetStmt

We pull out and create:

stmtAlterDatabaseStmt
and
stmtAlterDatabaseSetStmt

-Andy

On 24 January 2011 03:06, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
> It's doing the right thing. But it's really spaghetti code, generated by
> a2p. Matt was just (rightly) offended by the $[ setting, IIRC.
>
> The point is that it's close to being totally unmaintainable.
>
> But if you think you can remedy it without rewriting it, go for it.
>
> I think minimum requirements would be:
>
>   * avoid setting $[
>   * use strict;
>   * comments saying what it's actually doing
>
>
> We want to be in a situation where of it ever does break because of some
> external change, we're not left having to wade through the crap to find out
> how to fix it.
>
> cheers
>
> andrew
>

Well here's an absolutely minimal patch to remove the $[ assignments
and fix up the array accesses.
I have tested that parse.pl produces identical output for both HEAD
and the 65 commits which touch ecpg.*
or gram.y since the introduction on check_rules.pl

git log 81a82a13^..HEAD -- src/backend/parser/gram.y
src/interfaces/ecpg/preproc/{ecpg.??*,*.pl}

I've also tested that the new check_rules.pl dies when I add an extra
unused rule to ecpg.addons.

I'll try and make a proper rewrite sometime this week.

What is the minimal perl version that we are requiring these days?

Andy Colson <andy(at)squeakycode(dot)net> writes:
> Is there anyway to make bison/yacc/gcc/etc spit out the rule names?

bison -v produces a debug output file that includes nicely cleaned-up
versions of all the rules. But it includes a lot of other stuff too,
and I'm not at all sure that the file format is stable across bison
versions, so maybe depending on that isn't a hot idea.

> I'm not sure rule name is the proper words. In gram.y we have:

"Production" is the standard technical name, but on the other hand the
bison documentation seems to consistently use the word "rule", so
there's probably nothing wrong with doing so here too.

regards, tom lane

On 23 January 2011 14:28, Kris Shannon <kris(at)shannon(dot)id(dot)au> wrote:
> On 23 January 2011 13:14, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>> But there are quite a few perlheads around. ISTR Matt Trout was muttering
>> about these scripts on IRC recently.
>
> A quick cleanup of the check_rules.pl...
>
> It's a starting point at least.
>

And I should have at least tried to run this before I submitted... :(
New minimal tested submission momentarily.

Kris Shannon <kris(at)shannon(dot)id(dot)au> writes:
> What is the minimal perl version that we are requiring these days?

5.8 according to configure.

regards, tom lane

On sön, 2011-01-23 at 12:23 -0500, Tom Lane wrote:
> Andy Colson <andy(at)squeakycode(dot)net> writes:
> > Is there anyway to make bison/yacc/gcc/etc spit out the rule names?
>
> bison -v produces a debug output file that includes nicely cleaned-up
> versions of all the rules. But it includes a lot of other stuff too,
> and I'm not at all sure that the file format is stable across bison
> versions, so maybe depending on that isn't a hot idea.

Having maintained a gram.y -> keywords.sgml script based on that idea
for N years, I can report that this is not stable enough to work here.

On 1/23/2011 5:11 AM, Michael Meskes wrote:
> On Sat, Jan 22, 2011 at 08:40:13PM -0500, Andrew Dunstan wrote:
>> I think these really need to be rewritten from scratch. They look
>> like they were written by someone who never heard of Perl 5 (it's
>> only about 16 years old).
>
> You might remember that we had this discussion before. The script was written
> in awk and then translated to perl by a2p. We knew that this code was less than
> optimal, but at least it works.
>
> As I already said when the script was introduced, I would love to have a real
> perl solution, but I'm not a perl programmer by any means.
>
> Michael

I thought Kris was going to work on this, but saw no progress, and I was
bored the other day, so I started working on it.

Here is a parse.pl, with some major refactoring.

I named it with a 2 so I could run it beside the original and diff em:

time perl parse.pl . < ../../../backend/parser/gram.y > preproc.y

real 0m10.636s
user 0m8.793s
sys 0m0.050s

time perl parse2.pl . < ../../../backend/parser/gram.y > preproc2.y

real 0m0.289s
user 0m0.214s
sys 0m0.009s

diff preproc.y preproc2.y

I am sure there are new bugs. I have not run it on anything but 9.0.1.
Are there other .y files you might feed it? (something other than
backend/parser/gram.y?)

I touched pretty much everything, and I broke a few things and had to go
back and fix 'em, so it would not surprise me at all if there were still
a few bugs.

Anyway, feedback, suggestions, etc are welcomed. (oh, yeah, I probably
need to go add some documentation)

-Andy

Andy Colson <andy(at)squeakycode(dot)net> writes:
> Here is a parse.pl, with some major refactoring.

> I am sure there are new bugs. I have not run it on anything but 9.0.1.
> Are there other .y files you might feed it? (something other than
> backend/parser/gram.y?)

That's the only file it has to work for. You could try it against 8.4
and HEAD versions as well as 9.0, but I'm not sure how much additional
test coverage that will get you.

regards, tom lane

On Wed, Mar 02, 2011 at 01:33:35PM -0600, Andy Colson wrote:
> I thought Kris was going to work on this, but saw no progress, and I
> was bored the other day, so I started working on it.
>
> Here is a parse.pl, with some major refactoring.
>
> I named it with a 2 so I could run it beside the original and diff em:

Thanks for all the work.

> I am sure there are new bugs. I have not run it on anything but
> 9.0.1. Are there other .y files you might feed it? (something other
> than backend/parser/gram.y?)

I ran it against several versions and it always gave the right output. So i
decided to just commit it to the archive so we can see if it breaks anything.
The old script is still in there so in case of a major problem that I cannot
foresee we can simply change the Makefile back to using parse.pl.

Michael

--
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org
Jabber: michael.meskes at googlemail dot com
VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL

On 3/3/2011 6:49 AM, Michael Meskes wrote:
> On Wed, Mar 02, 2011 at 01:33:35PM -0600, Andy Colson wrote:
>> I thought Kris was going to work on this, but saw no progress, and I
>> was bored the other day, so I started working on it.
>>
>> Here is a parse.pl, with some major refactoring.
>>
>> I named it with a 2 so I could run it beside the original and diff em:
>
> Thanks for all the work.
>
>> I am sure there are new bugs. I have not run it on anything but
>> 9.0.1. Are there other .y files you might feed it? (something other
>> than backend/parser/gram.y?)
>
> I ran it against several versions and it always gave the right output. So i
> decided to just commit it to the archive so we can see if it breaks anything.
> The old script is still in there so in case of a major problem that I cannot
> foresee we can simply change the Makefile back to using parse.pl.
>
> Michael
>

And here is check_rules. Much less change, but I did run it through
perl tidy, so I'll bet most of the lines changed.

-Andy

On 3 March 2011 06:33, Andy Colson <andy(at)squeakycode(dot)net> wrote:
> On 1/23/2011 5:11 AM, Michael Meskes wrote:
>> As I already said when the script was introduced, I would love to have a
>> real
>> perl solution, but I'm not a perl programmer by any means.
>>
>> Michael
>
> I thought Kris was going to work on this, but saw no progress, and I was
> bored the other day, so I started working on it.

Yeah, sorry about that. I got as far as check_rules.pl with perl5.8 and
running it against all the commits which changed the files it depends on
and comparing the results with the old script; and then a major meltdown
at work - primary SAN crashed :( - got in the way and I totally forgot
about it...

Thanks for picking up the ball and running with it.