Re: [PATCH] Re: Adding XMLEXISTS to the grammar

Hi,

I've been working to improve the syntax of the XMLEXISTS function that I
put a patch forward for and have been attempting to get my head around
how you modify the grammar. I admit I'm not getting much anywhere
probably as I don't know bison but I'm starting to wonder if it's worth
the pain given recent comments on this list about not changing the
grammar for JSON support. At this point I can see a way of implementing
the following abridged syntax (abridged as I'm not doing full XQuery
support at this stage) in a conventional plain function call by handling
the PG_FUNCTION_ARGS approriately, but would this acceptable?

XMLEXISTS
(
xpath_expression
[
PASSING BY REF xml_expression [BY REF]
]
)

In case it isn't, and indeed to help me with the XML schema validation
work I'm doing, I would still like some help on how the grammar works.
From what I've greped and seen in the comments you need to modify the
following files:

- src/backend/parser/gram.y
- src/backend/parser/parse_expr.c
- src/backend/utils/ruleutils.c
- src/include/parser/kwlist.h

From what I can tell, you add the keywords to the lists in gram.y and
kwlist.h. At the appropriate place in gram.y you define the syntax and
pull out what you need and stuff it into a node (in my case using the
makeXmlExpr). You then modify parse_expr.c and ruleutils.c to handle the
new values in the fields of the XmlExpr node. Assuming I'm right so far,
the step I'm failing to figure out is where the actual c function that
implements the function gets called/associated within the grammar. What
am I missing?

Thanks in advance,

--
Mike Fowler
Registered Linux user: 379787

On Wed, Jun 9, 2010 at 6:32 AM, Mike Fowler <mike(at)mlfowler(dot)com> wrote:
> I've been working to improve the syntax of the XMLEXISTS function that I put
> a patch forward for and have been attempting to get my head around how you
> modify the grammar. I admit I'm not getting much anywhere probably as I
> don't know bison but I'm starting to wonder if it's worth the pain given
> recent comments on this list about not changing the grammar for JSON
> support.

I think we're willing to change the parser to comply with the SQL
standard, but not for add-on datatypes.

> At this point I can see a way of implementing the following
> abridged syntax (abridged as I'm not doing full XQuery support at this
> stage) in a conventional plain function call by handling the
> PG_FUNCTION_ARGS approriately, but would this acceptable?
>
> XMLEXISTS
> (
> xpath_expression
>  [
>  PASSING BY REF xml_expression [BY REF]
>  ]
> )

I don't see how you're going to make this work without parser changes,
and even if you can I think it would be too ugly to consider.

> In case it isn't, and indeed to help me with the XML schema validation work
> I'm doing, I would still like some help on how the grammar works. From what
> I've greped and seen in the comments you need to modify the following files:
>
> - src/backend/parser/gram.y
> - src/backend/parser/parse_expr.c
> - src/backend/utils/ruleutils.c
> - src/include/parser/kwlist.h
>
> From what I can tell, you add the keywords to the lists in gram.y and
> kwlist.h. At the appropriate place in gram.y you define the syntax and pull
> out what you need and stuff it into a node (in my case using the
> makeXmlExpr). You then modify parse_expr.c and ruleutils.c to handle the new
> values in the fields of the XmlExpr node. Assuming I'm right so far, the
> step I'm failing to figure out is where the actual c function that
> implements the function gets called/associated within the grammar. What am I
> missing?

Look at how the POSITION() pseudofunction is defined around gram.y
line 9651. Essentially any special syntax of this type gets converted
to a regular function call internally. So in your case I think there
will be some function that gets called something ike this:

xmlexists(xpath_expression, xml_expression)

...but the grammar can be modified to allow a different syntax for
that function call.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> Look at how the POSITION() pseudofunction is defined around gram.y
> line 9651. Essentially any special syntax of this type gets converted
> to a regular function call internally. So in your case I think there
> will be some function that gets called something ike this:

> xmlexists(xpath_expression, xml_expression)

> ...but the grammar can be modified to allow a different syntax for
> that function call.

Note also that we typically try to allow the function to be called with
the generic comma-separated syntax as well as the keyword-based syntax
that the SQL committee has such weird love for. This makes life easier
for users, and it also means that we don't need special cases in
ruleutils.c.

regards, tom lane

> Look at how the POSITION() pseudofunction is defined around gram.y
> line 9651. Essentially any special syntax of this type gets converted
> to a regular function call internally. So in your case I think there
> will be some function that gets called something ike this:
>
> xmlexists(xpath_expression, xml_expression)
>
> ...but the grammar can be modified to allow a different syntax for
> that function call.
>
I've finally managed to get gram.y to parse the syntax correctly. After
progressing from a segmentation fault that occured when the grammar was
correct I'm now left with a cryptic error:

xmltest=# SELECT COUNT(id) FROM xmltest WHERE xmlexists('/menu/beers'
PASSING BY REF data);
ERROR: unrecognized node type: 1852140847

At a guess there is another step that I need to do after modifying
gram.y. One mailing list posting I found mentioned copyfuncs.c but
really I'm unsure as to what next. Anyone know what the missing step is?

Regards,

--
Mike Fowler
Registered Linux user: 379787

On Tue, Jun 22, 2010 at 1:17 PM, Mike Fowler <mike(at)mlfowler(dot)com> wrote:
>
>> Look at how the POSITION() pseudofunction is defined around gram.y
>> line 9651.  Essentially any special syntax of this type gets converted
>> to a regular function call internally.  So in your case I think there
>> will be some function that gets called something ike this:
>>
>> xmlexists(xpath_expression, xml_expression)
>>
>> ...but the grammar can be modified to allow a different syntax for
>> that function call.
>>
>
> I've finally managed to get gram.y to parse the syntax correctly. After
> progressing from a segmentation fault that occured when the grammar was
> correct I'm now left with a cryptic error:
>
> xmltest=# SELECT COUNT(id) FROM xmltest WHERE xmlexists('/menu/beers'
> PASSING BY REF data);
> ERROR:  unrecognized node type: 1852140847
>
> At a guess there is another step that I need to do after modifying gram.y.
> One mailing list posting I found mentioned copyfuncs.c but really I'm unsure
> as to what next. Anyone know what the missing step is?

I usually troubleshoot things like this by setting a breakpoint in
elog_start or elog_finish. Then you can see where it's blowing up.
Off the top of my head, I would guess you've added a node type whose
structure definition doesn't begin with NodeTag, or else you've got a
memory clobber.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

Robert Haas wrote:
>
> I usually troubleshoot things like this by setting a breakpoint in
> elog_start or elog_finish. Then you can see where it's blowing up.
> Off the top of my head, I would guess you've added a node type whose
> structure definition doesn't begin with NodeTag, or else you've got a
> memory clobber.

Thanks Robert, I've managed to resolve this make making a type cast
inside gram.y. However, it now seems that the function itself can not be
found. I've made an entry in pg_proc.h, but when running psql I'm
getting the following:

xmltest=# SELECT COUNT(id) FROM xmltest WHERE xmlexists('/menu/beers'
PASSING BY REF data);
ERROR: function pg_catalog.xml_exists(text, xml) does not exist
LINE 1: SELECT COUNT(id) FROM xmltest WHERE xmlexists('/menu/beers' ...
^
HINT: No function matches the given name and argument types. You might
need to add explicit type casts.

In gram.y I've got:

FuncCall *n = makeNode(FuncCall);
n->funcname = SystemFuncName("xml_exists");

(also tried SystemFuncName("xmlexists");)

In xml.h:

extern bool xml_exists(text *xpath_expr_text, xmltype *data);

I've also tried

bool xml_exists(PG_FUNCTION_ARGS) {

and finally in pg_proc.h I have:

DATA(insert OID = 3037 ( xmlexists PGNSP PGUID 12 1 0 0 f f f t f i
3 0 16 "25 142" _null_ _null_ _null_ _null_ xml_exists _null_ _null_
_null_ ));
DESCR("evaluate XPath expression in a boolean context");

(also tried ( xml_exists PGNSP....))

After each attempt, I've blown away the installation, made clean and
installed, initialised a fresh database and restored my sample database.
I've had a grep around using position and it's target function textpos
as examples but I fail to see any other file that they live in other
than their implementation. As far as I can tell, I'm not doing anything
different from position. Any thoughts?

Regards,

--
Mike Fowler
Registered Linux user: 379787

On Thu, Jun 24, 2010 at 2:37 PM, Mike Fowler <mike(at)mlfowler(dot)com> wrote:
> Thanks Robert, I've managed to resolve this make making a type cast inside
> gram.y. However, it now seems that the function itself can not be found.
> I've made an entry in pg_proc.h, but when running psql I'm getting the
> following:
>
> xmltest=# SELECT COUNT(id) FROM xmltest WHERE xmlexists('/menu/beers'
> PASSING BY REF data);
> ERROR:  function pg_catalog.xml_exists(text, xml) does not exist
> LINE 1: SELECT COUNT(id) FROM xmltest WHERE xmlexists('/menu/beers' ...
>                                           ^
> HINT:  No function matches the given name and argument types. You might need
> to add explicit type casts.
>
> In gram.y I've got:
>
> FuncCall *n = makeNode(FuncCall);
> n->funcname = SystemFuncName("xml_exists");
>
> (also tried SystemFuncName("xmlexists");)
>
> In xml.h:
>
> extern bool xml_exists(text *xpath_expr_text, xmltype *data);
>
> I've also tried
>
> bool xml_exists(PG_FUNCTION_ARGS) {
>
> and finally in pg_proc.h I have:
>
> DATA(insert OID = 3037 (  xmlexists     PGNSP PGUID 12 1 0 0 f f f t f i 3 0
> 16 "25 142" _null_ _null_ _null_ _null_ xml_exists _null_ _null_ _null_ ));
> DESCR("evaluate XPath expression in a boolean context");
>
> (also tried ( xml_exists   PGNSP....))
>
> After each attempt, I've blown away the installation, made clean and
> installed, initialised a fresh database and restored my sample database.
> I've had a grep around using position and it's target function textpos as
> examples but I fail to see any other file that they live in other than their
> implementation. As far as I can tell, I'm not doing anything different from
> position. Any thoughts?

It looks like the pg_proc entry is creating an SQL function called
xmlexists referencing a C function called xml_exists, and the gram.y
changes want there to be an SQL function called xml_exists. I think
you should rip out all the catalog and parser changes for starters,
and just try to get it working as a regular old function. Once you
have that working, you can add the syntax support back in. I'd
suggest making the C and SQL function names the same as each other,
but different from the keyword you're planning to use (xmlexists).

As for declaring the function, I believe you want this:

Datum
your_function_name(PG_FUNCTION_ARGS)
{

}

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

>> and finally in pg_proc.h I have:
>>
>> DATA(insert OID = 3037 ( xmlexists PGNSP PGUID 12 1 0 0 f f f t f i 3 0
>> 16 "25 142" _null_ _null_ _null_ _null_ xml_exists _null_ _null_ _null_ ));
>> DESCR("evaluate XPath expression in a boolean context");
>>
> It looks like the pg_proc entry is creating an SQL function called
> xmlexists referencing a C function called xml_exists, and the gram.y
> changes want there to be an SQL function called xml_exists. I think
> you should rip out all the catalog and parser changes for starters,
> and just try to get it working as a regular old function. Once you
> have that working, you can add the syntax support back in. I'd
> suggest making the C and SQL function names the same as each other,
> but different from the keyword you're planning to use (xmlexists).
>
Thanks again for your help Robert, turns out the fault was in the
pg_proc entry (the 3 up there should've been a two!). Once I took the
grammar out it was quickly obvious where I'd gone wrong.

Attached is a patch with the revised XMLEXISTS function, complete with
grammar support and regression tests. The implemented grammar is:

XMLEXISTS ( xpath_expression PASSING BY REF xml_value [BY REF] )

Though the full grammar makes everything after the xpath_expression
optional, I've left it has mandatory simply to avoid lots of rework of
the function (would need new null checks, memory handling would need
reworking).

--
Mike Fowler
Registered Linux user: 379787

On Sun, Jun 27, 2010 at 12:04 PM, Mike Fowler <mike(at)mlfowler(dot)com> wrote:
> Thanks again for your help Robert, turns out the fault was in the pg_proc
> entry (the 3 up there should've been a two!). Once I took the grammar out it
> was quickly obvious where I'd gone wrong.

Glad it was a helpful suggestion.

> Attached is a patch with the revised XMLEXISTS function, complete with
> grammar support and regression tests. The implemented grammar is:
>
> XMLEXISTS ( xpath_expression PASSING BY REF xml_value [BY REF] )
>
> Though the full grammar makes everything after the xpath_expression
> optional, I've left it has mandatory simply to avoid lots of rework of the
> function (would need new null checks, memory handling would need reworking).

So if you don't specify the xml_value, what does the xpath_expression
get applied to?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

Robert Haas wrote:
> On Sun, Jun 27, 2010 at 12:04 PM, Mike Fowler <mike(at)mlfowler(dot)com> wrote:
>
>> Thanks again for your help Robert, turns out the fault was in the pg_proc
>> entry (the 3 up there should've been a two!). Once I took the grammar out it
>> was quickly obvious where I'd gone wrong.
>>
>
> Glad it was a helpful suggestion.
>
>
>> Attached is a patch with the revised XMLEXISTS function, complete with
>> grammar support and regression tests. The implemented grammar is:
>>
>> XMLEXISTS ( xpath_expression PASSING BY REF xml_value [BY REF] )
>>
>> Though the full grammar makes everything after the xpath_expression
>> optional, I've left it has mandatory simply to avoid lots of rework of the
>> function (would need new null checks, memory handling would need reworking).
>>
>
> So if you don't specify the xml_value, what does the xpath_expression
> get applied to?
>
From what I can gather the xpath_expression would be evalutated against
an empty document thereby returning false for every xpath_expression
except for 'true()'. Apache Derby has made the xml_value mandatory as
well (though I'll stress my conclusion wasn't based on this fact). If
you think it would better to adhere more closely to the standard I can
certainly look to do so. From a cursory glance at libxml's API I think
it should be straight forward to query against an empty document such
that I wouldn't need ot code for the exceptional case (or cases if I've
missed others).

Regards,

--
Mike Fowler
Registered Linux user: 379787

Mike Fowler wrote:
> Thanks again for your help Robert, turns out the fault was in the
> pg_proc entry (the 3 up there should've been a two!). Once I took the
> grammar out it was quickly obvious where I'd gone wrong.
>
> Attached is a patch with the revised XMLEXISTS function, complete with
> grammar support and regression tests. The implemented grammar is:
>
> XMLEXISTS ( xpath_expression PASSING BY REF xml_value [BY REF] )
>
> Though the full grammar makes everything after the xpath_expression
> optional, I've left it has mandatory simply to avoid lots of rework of
> the function (would need new null checks, memory handling would need
> reworking).
>

As with the xpath_exists patch I've now added the SGML documentation
detailing this function and extended the regression test a little to
test XML literals.

Regards,

--
Mike Fowler
Registered Linux user: 379787

On tis, 2010-06-29 at 12:22 +0100, Mike Fowler wrote:
> Mike Fowler wrote:
> > Thanks again for your help Robert, turns out the fault was in the
> > pg_proc entry (the 3 up there should've been a two!). Once I took the
> > grammar out it was quickly obvious where I'd gone wrong.
> >
> > Attached is a patch with the revised XMLEXISTS function, complete with
> > grammar support and regression tests. The implemented grammar is:
> >
> > XMLEXISTS ( xpath_expression PASSING BY REF xml_value [BY REF] )
> >
> > Though the full grammar makes everything after the xpath_expression
> > optional, I've left it has mandatory simply to avoid lots of rework of
> > the function (would need new null checks, memory handling would need
> > reworking).
> >
>
> As with the xpath_exists patch I've now added the SGML documentation
> detailing this function and extended the regression test a little to
> test XML literals.

Some thoughts, mostly nitpicks:

The snippet of documentation could be clearer. It says "if the xml
satisifies the xpath". Not sure what that means exactly. An XPath
expression, by definition, returns a value. How is that value used to
determine the result?

Naming of parser symbols: xmlexists_list isn't actually a list of
xmlexists's. That particular rule can probably be done away with anyway
and the code be put directly into the XMLEXISTS rule.

Why is the first argument AexprConst instead of a_expr? The SQL
standard says it's a character string literal, but I think we can very
well allow arbitrary expressions.

xmlexists_query_argument_list should be optional.

The rules xml_default_passing_mechanism and xml_passing_mechanism are
pretty useless to have a separate rules. Just mention the tokens where
they are used.

Why c_expr?

Call the C-level function xmlexists for consistency.

Hi Peter,

Thanks for your feedback.

On 20/07/10 19:54, Peter Eisentraut wrote:
>>> Attached is a patch with the revised XMLEXISTS function, complete with
>>> grammar support and regression tests. The implemented grammar is:
>>>
>>> XMLEXISTS ( xpath_expression PASSING BY REF xml_value [BY REF] )
>>>
>>> Though the full grammar makes everything after the xpath_expression
>>> optional, I've left it has mandatory simply to avoid lots of rework of
>>> the function (would need new null checks, memory handling would need
>>> reworking).
> Some thoughts, mostly nitpicks:
>
> The snippet of documentation could be clearer. It says "if the xml
> satisifies the xpath". Not sure what that means exactly. An XPath
> expression, by definition, returns a value. How is that value used to
> determine the result?
>

I'll rephrase it: The function xmlexists returns true if the xpath
returns any nodes and false otherwise.

> Naming of parser symbols: xmlexists_list isn't actually a list of
> xmlexists's. That particular rule can probably be done away with anyway
> and the code be put directly into the XMLEXISTS rule.
>
> Why is the first argument AexprConst instead of a_expr? The SQL
> standard says it's a character string literal, but I think we can very
> well allow arbitrary expressions.
>

Yes, it was AexprConst because of the specification. I also found that
using it solved my shift/reduce problems, but I can change it a_expr as
see if I can work them out in a different way.

> xmlexists_query_argument_list should be optional.
>

OK, I'll change it.

> The rules xml_default_passing_mechanism and xml_passing_mechanism are
> pretty useless to have a separate rules. Just mention the tokens where
> they are used.
>

Again, I'll change that too.

> Why c_expr?
>

As with the AexprConst, it's choice was partially influenced by the fact
it solved the shift/reduce errors I was getting. I'm guessing than that
I should really use a_expr and resolve the shift/reduce problem differently?

> Call the C-level function xmlexists for consistency.
>

Sure. I'll look to get a patch addressing these concerns out in the next
day or two, work/family/sleep permitting! :)

Regards,

--
Mike Fowler
Registered Linux user: 379787

On 21/07/10 08:33, Mike Fowler wrote:

>> Why is the first argument AexprConst instead of a_expr? The SQL
>> standard says it's a character string literal, but I think we can very
>> well allow arbitrary expressions.
>
> Yes, it was AexprConst because of the specification. I also found that
> using it solved my shift/reduce problems, but I can change it a_expr as
> see if I can work them out in a different way.

[snip]

>> Why c_expr?
>
> As with the AexprConst, it's choice was partially influenced by the fact
> it solved the shift/reduce errors I was getting. I'm guessing than that
> I should really use a_expr and resolve the shift/reduce problem
> differently?
>

Attached is the revised version of the patch addressing all the issues
raised in the review, except for the use of AexprConst and c_expr. With
my limited knowledge of bison I've failed to resolve the shift/reduce
errors that are introduced by using a_expr. I'm open to suggestions as
my desk is getting annoyed with me beating it in frustration!

Thanks again for taking the time to review my work.

Regards,

--
Mike Fowler
Registered Linux user: 379787

On lör, 2010-07-24 at 20:32 +0100, Mike Fowler wrote:
> Attached is the revised version of the patch addressing all the
> issues
> raised in the review, except for the use of AexprConst and c_expr.
> With
> my limited knowledge of bison I've failed to resolve the shift/reduce
> errors that are introduced by using a_expr. I'm open to suggestions
> as
> my desk is getting annoyed with me beating it in frustration!

It's committed now. I ended up refactoring this a little bit so that
xpath() and xmlexists() can share more of the code. Also, I relaxed the
grammar a bit for better compatibility with DB2, Oracle, and Derby.