Re: regression test for extended query protocol

[Sorry for a top post. For unknown reason I couldn' get mails from
pgsql-hackers since this morning and I have to check the archive]

>> In my understanding, we don't have any regression test for protocol
>> level prepared query (we do have SQL level prepared query tests,
>> though). Shouldn't we add those tests to the regression test suites?
>
> I thought that ECPG was covering a portion of that. Right?
> --
> Michael

In my understanding, ECPG calls libpq, thus the test cases are limited
to the cases which are only possible with libpq (or it may be even
limited to the cases where ECPG deal with.)

Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp

Tatsuo Ishii <ishii(at)postgresql(dot)org> writes:
>>> In my understanding, we don't have any regression test for protocol
>>> level prepared query (we do have SQL level prepared query tests,
>>> though). Shouldn't we add those tests to the regression test suites?

>> I thought that ECPG was covering a portion of that. Right?

> In my understanding, ECPG calls libpq, thus the test cases are limited
> to the cases which are only possible with libpq (or it may be even
> limited to the cases where ECPG deal with.)

I do not think it's very practical for the core tests to try to cover
any behavior that's not reachable via libpq. Even testing stuff that's
not reached through ecpg would require a whole slew of new infrastructure
with no use except testing.

I think realistically a better approach to this would be to get some
buildfarm members running the JDBC regression tests (I assume there
are some ...)

regards, tom lane

On 3 August 2016 at 10:53, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> Tatsuo Ishii <ishii(at)postgresql(dot)org> writes:
> >>> In my understanding, we don't have any regression test for protocol
> >>> level prepared query (we do have SQL level prepared query tests,
> >>> though). Shouldn't we add those tests to the regression test suites?
>
> >> I thought that ECPG was covering a portion of that. Right?
>
> > In my understanding, ECPG calls libpq, thus the test cases are limited
> > to the cases which are only possible with libpq (or it may be even
> > limited to the cases where ECPG deal with.)
>
> I do not think it's very practical for the core tests to try to cover
> any behavior that's not reachable via libpq. Even testing stuff that's
> not reached through ecpg would require a whole slew of new infrastructure
> with no use except testing.
>
> I think realistically a better approach to this would be to get some
> buildfarm members running the JDBC regression tests (I assume there
> are some ...)
>
> regards, tom lane
>
> We currently run tests every time a PR is created on github, but I don't
think there are any animals running the JDBC test suite

We can add tests, what exactly do we want to test. Then setting up an
animal to run the tests would be fairly straight forward.

Dave Cramer

davec(at)postgresintl(dot)com
www.postgresintl.com

On Thu, Aug 4, 2016 at 12:14 AM, Dave Cramer <pg(at)fastcrypt(dot)com> wrote:
> We currently run tests every time a PR is created on github, but I don't
> think there are any animals running the JDBC test suite
>
> We can add tests, what exactly do we want to test. Then setting up an animal
> to run the tests would be fairly straight forward.

It may be interesting to implement that as a module in the buildfarm
client I think that any animal could include at will. Just a thought.
--
Michael

Michael Paquier wrote:
> On Thu, Aug 4, 2016 at 12:14 AM, Dave Cramer <pg(at)fastcrypt(dot)com> wrote:
> > We currently run tests every time a PR is created on github, but I don't
> > think there are any animals running the JDBC test suite
> >
> > We can add tests, what exactly do we want to test. Then setting up an animal
> > to run the tests would be fairly straight forward.
>
> It may be interesting to implement that as a module in the buildfarm
> client I think that any animal could include at will. Just a thought.

Implementing things as buildfarm modules gets boring, though -- consider
for instance that src/test/recovery is only being run by hamster, and
only because you patched the buildfarm client; only Andrew can influence
buildfarm animals into running that test by integrating the module in
the client script, and even that is limited by having to request animal
caretakers to upgrade that script.

If somebody had some spare time to devote to this, I would suggest to
implement something in core that can be used to specify a list of
commands to run, and a list of files-to-be-saved-in-bf-log emitted by
each command. We could have one such base file in the core repo that
specifies some tests to run (such as "make -C src/test/recovery check"),
and an additional file can be given by buildfarm animals to run custom
tests, without having to write BF modules for each thing. With that,
pgsql committers could simply add a new test to be run by all buildfarm
animals by adding it to the list in core.

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Fri, Aug 5, 2016 at 12:21 AM, Alvaro Herrera
<alvherre(at)2ndquadrant(dot)com> wrote:
> If somebody had some spare time to devote to this, I would suggest to
> implement something in core that can be used to specify a list of
> commands to run, and a list of files-to-be-saved-in-bf-log emitted by
> each command. We could have one such base file in the core repo that
> specifies some tests to run (such as "make -C src/test/recovery check"),
> and an additional file can be given by buildfarm animals to run custom
> tests, without having to write BF modules for each thing. With that,
> pgsql committers could simply add a new test to be run by all buildfarm
> animals by adding it to the list in core.

Do you think about using a special makefile target to run those
commands, say in src/test? At the end we are going to need to patch
the buildfarm client code at least once, at least that would be worth
it in the long term..
--
Michael

Michael Paquier wrote:
> On Fri, Aug 5, 2016 at 12:21 AM, Alvaro Herrera
> <alvherre(at)2ndquadrant(dot)com> wrote:
> > If somebody had some spare time to devote to this, I would suggest to
> > implement something in core that can be used to specify a list of
> > commands to run, and a list of files-to-be-saved-in-bf-log emitted by
> > each command. We could have one such base file in the core repo that
> > specifies some tests to run (such as "make -C src/test/recovery check"),
> > and an additional file can be given by buildfarm animals to run custom
> > tests, without having to write BF modules for each thing. With that,
> > pgsql committers could simply add a new test to be run by all buildfarm
> > animals by adding it to the list in core.
>
> Do you think about using a special makefile target to run those
> commands, say in src/test? At the end we are going to need to patch
> the buildfarm client code at least once, at least that would be worth
> it in the long term..

Sure. Some time ago I proposed something like a JSON file, something
like

test_foo => {
"command" : "make -C src/test/blah check",
"save_output" : [ "src/test/blah/server.log", "src/test/blah/regress*.log" ]
}

as I recall, Andrew said that he didn't like JSON very much but that the
idea made sense to him.

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Thu, Aug 11, 2016 at 5:33 AM, Alvaro Herrera
<alvherre(at)2ndquadrant(dot)com> wrote:
> Michael Paquier wrote:
>> On Fri, Aug 5, 2016 at 12:21 AM, Alvaro Herrera
>> <alvherre(at)2ndquadrant(dot)com> wrote:
>> > If somebody had some spare time to devote to this, I would suggest to
>> > implement something in core that can be used to specify a list of
>> > commands to run, and a list of files-to-be-saved-in-bf-log emitted by
>> > each command. We could have one such base file in the core repo that
>> > specifies some tests to run (such as "make -C src/test/recovery check"),
>> > and an additional file can be given by buildfarm animals to run custom
>> > tests, without having to write BF modules for each thing. With that,
>> > pgsql committers could simply add a new test to be run by all buildfarm
>> > animals by adding it to the list in core.
>>
>> Do you think about using a special makefile target to run those
>> commands, say in src/test? At the end we are going to need to patch
>> the buildfarm client code at least once, at least that would be worth
>> it in the long term..
>
> Sure. Some time ago I proposed something like a JSON file, something
> like
>
> test_foo => {
> "command" : "make -C src/test/blah check",
> "save_output" : [ "src/test/blah/server.log", "src/test/blah/regress*.log" ]
> }
>
> as I recall, Andrew said that he didn't like JSON very much but that the
> idea made sense to him.

As long as the buildfarm client has native support to parse that, that
sounds good. I think that we had better add a TODO item now, I won't
be able to tackle that in the short term.
--
Michael