Proposal - libpq Type System beta-0.8a (was PGparam)

We previously tried to send this proposal/patch, but it never showed up.
We tried twice in a 16 hour period. It might be something with the
attachment, who knows. This time, we are providing a URL instead of
attaching the patch. Please disregard previous emails if they come back
from never-never-land.

http://www.esilo.com/projects/postgresql/libpq/typesys-beta-0.8a.tar.gz

What was previously called PGparam is now called libpq Type System.
PGparam is just one structure within the Type System.

This is an updated proposal/patch for a Type System in libpq. Basically,
it allows applications to send binary formatted paramters "put" and
receive text or binary formatted results "get" through a printf-style
interface. It also adds the ability to register user-defined types,
sub-classes of existing types as well as composites. There is full
support for arrays, composites, composite arrays and nested composites.

There are four documents included within the tar:

type-system-api.txt - Documents the API functions

type-specifiers.txt - This explains the type specifier syntax
(printf-style). It also documents how to construct parameters and get
result values for every supported type.

composites-arrays.txt - Documents the use of arrays, composites and
arrays of composites. This proposed API has full support for nested
arrays or composites.

type-handlers.txt - Explains how to register and implement a libpq type
handler. Types can be sub-classed.

The patch is named typesys.patch. There is a regression test named
regression-test.c and a makefile for it named makefile.typesys.

Andrew & Merlin
eSilo

Hi,

Andrew Chernow wrote:
> It might be something with the
> attachment, who knows.

Most probably that was the case, yes. The -hackers list is limited,
please use -patches to send patches. ;-)

Regards

Markus

Markus Schiltknecht wrote:
> Hi,
>
> Andrew Chernow wrote:
>> It might be something with the attachment, who knows.
>
> Most probably that was the case, yes. The -hackers list is limited,
> please use -patches to send patches. ;-)
>
> Regards
>
> Markus
>
>

Noted.

In our case, its a little ambiguos whether -hackers or -patches is the
correct place. We are really posting a proposal that happens to have a
working implementation. We are looking for feedback and/or a discussion.

andrew

Andrew Chernow wrote:
> Markus Schiltknecht wrote:
>> Hi,
>>
>> Andrew Chernow wrote:
>>> It might be something with the attachment, who knows.
>>
>> Most probably that was the case, yes. The -hackers list is limited,
>> please use -patches to send patches. ;-)
>>
>> Regards
>>
>> Markus
>>
>>
>
> Noted.
>
> In our case, its a little ambiguos whether -hackers or -patches is the
> correct place. We are really posting a proposal that happens to have
> a working implementation. We are looking for feedback and/or a
> discussion.
>
>

You should split it and send the proposal to -hackers. Ideally, you
would have had a proposal discussed before you wrote a line of code.

cheers

andrew

Andrew Dunstan wrote:
>
>
> Andrew Chernow wrote:
>> Markus Schiltknecht wrote:
>>> Hi,
>>>
>>> Andrew Chernow wrote:
>>>> It might be something with the attachment, who knows.
>>>
>>> Most probably that was the case, yes. The -hackers list is limited,
>>> please use -patches to send patches. ;-)
>>>
>>> Regards
>>>
>>> Markus
>>>
>>>
>>
>> Noted.
>>
>> In our case, its a little ambiguos whether -hackers or -patches is the
>> correct place. We are really posting a proposal that happens to have
>> a working implementation. We are looking for feedback and/or a
>> discussion.
>>
>>
>
> You should split it and send the proposal to -hackers. Ideally, you
> would have had a proposal discussed before you wrote a line of code.
>
> cheers
>
> andrew
>
>

>>proposal discussed before you wrote a line of code
Yeah, we realize that. In our situation, we use this code internally
which is why it exists. Back in Aug 2007, we packaged it up and
proposed it because we thought it would be useful to others. Since
then, we have submitted several versions. Feedback was minimal.

In fact, only Tom has made suggestions, which we have taken into
consideration and adjusted the spec accordingly.

We are interested in having a discussion about the beta-0.8a proposal
and concept, not the implementation or submitting procedures. We
provided the code in case someone wants to take a test drive.

Andrew

On Jan 8, 2008 12:57 PM, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
> You should split it and send the proposal to -hackers. Ideally, you
> would have had a proposal discussed before you wrote a line of code.

This is the latest in a long series of submissions...check the
archives. We are (and have been) sensitive to the other pressures
resulting from the 8.3 release.

That said, we have been a little dismayed in the lack of comment. We
attributed this to either lack of interest or just general business
(we perhaps optimistically guessed the latter). We needed our
extensions for our own projects and are willing to maintain them
outside of the project if we have to...that is for the community to
decide. For the record, we are extremely excited about the libpq
changes and think others well be as well.

Tom noted the lack of documentation as well as a number of technical
issues. We addressed those issues (within the scope of what we wanted
to accomplish).

merlin

Merlin,

> That said, we have been a little dismayed in the lack of comment.

I think most people can't really follow what functionality this would allow
users & driver authors to access, and what the spec for that functionality
would. I know I'm not clear on it.

A high-level proposal would arouse more general interest. Otherwise,
you'll just get a critique and eventually it'll either get applied or
rejected without much comment.

--
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco

"Merlin Moncure" <mmoncure(at)gmail(dot)com> writes:
> That said, we have been a little dismayed in the lack of comment.

Personally, I'm ignoring this thread, as well as the one on
partitioning, because I'm up to my arse in 8.3-release alligators.
I'm going to try hard not to think about any 8.4 development issues
until 8.3 is actually out the door and we branch for 8.4 development.

I can't say how many other people are equally pressed for time ...
but seeing the minimal attention that seems to be getting paid to
open 8.3 issues, it doesn't look to me like the community as a whole
has a lot of spare cycles right now.

regards, tom lane

On Jan 8, 2008 4:31 PM, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
> Merlin,
>
> > That said, we have been a little dismayed in the lack of comment.
>
> I think most people can't really follow what functionality this would allow
> users & driver authors to access, and what the spec for that functionality
> would. I know I'm not clear on it.
>
> A high-level proposal would arouse more general interest. Otherwise,
> you'll just get a critique and eventually it'll either get applied or
> rejected without much comment.

There are over 1500 lines of documentation attached in the patch.
That's a good place to start learning (or the attached regression
test, if you want to get right to it). There's a lot of material to
cover. Here's a very high level overview of the functionality:

Goal:
The ultimate goal is to be able to put parameters for command
execution and get results in a consistent way. The wire format
(either text or binary) is completely abstracted. We added many
functions to facilitate this, but the core functionality comes from
two varargs style functions, PQgetf and PQputf, and the PGparam, which
complements PGresult.

Features:
*) Binary transfer of all built in types is supported in both
directions. Basically, we marshal backend wire format to/from C types
(some native, some were introduced). We also support text results so
you can pull data in a consistent interface.
*) For user types (with custom send/recv functions), functions can be
registered to marshal them through a type registration interface.
*) Arrays and composites are supported automatically (composites have
to be 'registered').
*) Client side handlers can be aliased (domains) or subclassed...type
handlers can chained together for special handling and/or conversion
to exotic application types.

Here is a short example which demonstrates some of the major features.
There are many other examples and discussions of minutia in the
documentation.

int resfmt = 1; /* binary format */

/* Put an int4 and a text */
PGparam *param = PQparamCreate(conn);
PQputf(param, "%int4 %text", 2000, "foobar");
PQparamExec(conn, param, resfmt,
"insert into foo(id, user) values ($1, $2)");

/* The above as a one liner, internally 'puts' for you */
res = PQexecParamsf(conn, resfmt,
"insert into foo(id, user) values (%int4, %text)",
2000, "foobar");

int i4;
char *text;
PGresult *res = PQparamExec(conn, NULL, resfmt, "select * from foo limit 1");

/* From tuple 0, get an int4 at field 0 and a text
* from the user field
* '%' denotes by field num, '#' by field name
*/
PQgetf(res, 0, "%int4 #text*", 0, &i4, "user", &text);

note the above line is not wired to binary, text results would be fine as well.

/* let's get an array */

PGresult *res = PQparamExec(conn, NULL, resfmt, "select current_schemas(true)");

/* pop an array object out of the result. it creates a new result
* with one field and one 'tuple' for each array element.
*
* arrays of composites return one field for each attribute of the composite.
*/
PGarray array;
PQgetf(res, 0, "%name[]", 0, &array);
PQclear(res);

for (i = 0; i < PQntuples(array.res); i++)
{
char *name;
PQgetf(array.res, i, "%name*", 0, &name);
printf("%s\n", name);
}
PQclear(array.res);

/* return data from composite type which we create and register */
CREATE TYPE merlin as (a int, t text); -- on server
PGresult *merlin;
PQregisterTypeHandler(conn, "merlin", NULL, NULL);
res = PQparamExec(conn, NULL, resfmt, "select (1, 'abc')::merlin");
PQgetf(res, 0, "%merlin", 0, &merlin);
PQclear(res);
PQgetf(merlin, 0, "%int4 #text*", 0, &i4, "t", &text);
PQclear(merlin);

merlin

Tom Lane wrote:
> "Merlin Moncure" <mmoncure(at)gmail(dot)com> writes:
>> That said, we have been a little dismayed in the lack of comment.
>
> Personally, I'm ignoring this thread, as well as the one on
> partitioning, because I'm up to my arse in 8.3-release alligators.
> I'm going to try hard not to think about any 8.4 development issues
> until 8.3 is actually out the door and we branch for 8.4 development.
>
> I can't say how many other people are equally pressed for time ...
> but seeing the minimal attention that seems to be getting paid to
> open 8.3 issues, it doesn't look to me like the community as a whole
> has a lot of spare cycles right now.
>
> regards, tom lane
>
>

Thanks for the heads up. We had a feeling the 8.3 crunch was a factor.
We don't want to slow that down because we were are waiting for 8.3 to
perform yet another mysql to postgresql migration :)

andrew

On Tue, Jan 08, 2008 at 05:33:51PM -0500, Merlin Moncure wrote:
> Here is a short example which demonstrates some of the major features.
> There are many other examples and discussions of minutia in the
> documentation.

I havn't looked at the source but FWIW I think it's an awesome idea.

Have a nice day,
--
Martijn van Oosterhout <kleptog(at)svana(dot)org> http://svana.org/kleptog/
> Those who make peaceful revolution impossible will make violent revolution inevitable.
> -- John F Kennedy