Re: nested hstore patch

Hi!

Attatched patch adds nesting feature, types (string, boll and numeric values),
arrays and scalar to hstore type.

All new features are described in PGConf.EU talk
http://www.sai.msu.su/~megera/postgres/talks/hstore-dublin-2013.pdf (since PGCon
some features was added).

Patch includes:
1 implementaion SRF_RETURN_NEXT_NULL()
2 contrib/hstore changes
3 docs of new hstore module (many thanks to David E. Wheeler
<david(dot)wheeler(at)pgexperts(dot)com>)

In current state patch is in WIP status, for short period I plan to move support
of binary nested structure to core to share binary representation for hstore and
json types.

--
Teodor Sigaev E-mail: teodor(at)sigaev(dot)ru
WWW: http://www.sigaev.ru/

On 11/12/2013 01:35 PM, Teodor Sigaev wrote:
> Hi!
>
> Attatched patch adds nesting feature, types (string, boll and numeric
> values), arrays and scalar to hstore type.
>
> All new features are described in PGConf.EU talk
> http://www.sai.msu.su/~megera/postgres/talks/hstore-dublin-2013.pdf
> (since PGCon some features was added).
>
> Patch includes:
> 1 implementaion SRF_RETURN_NEXT_NULL()
> 2 contrib/hstore changes
> 3 docs of new hstore module (many thanks to David E. Wheeler
> <david(dot)wheeler(at)pgexperts(dot)com>)
>
> In current state patch is in WIP status, for short period I plan to
> move support of binary nested structure to core to share binary
> representation for hstore and json types.
>
>
>

Thanks, Teodor.

As soon as we have that shared binary representation available, I will
be working on adapting it to JSON.

cheers

andrew

On 11/12/13, 1:35 PM, Teodor Sigaev wrote:
> Attatched patch adds nesting feature, types (string, boll and numeric
> values), arrays and scalar to hstore type.

Could you check your email client for next time? It's sending
Content-Type: application/x-tar for a *.patch.gz file.

On 11/13/2013 01:37 AM, Andrew Dunstan wrote:
>
> On 11/12/2013 01:35 PM, Teodor Sigaev wrote:
>> Hi!
>>
>> Attatched patch adds nesting feature, types (string, boll and numeric
>> values), arrays and scalar to hstore type.
>>
>> All new features are described in PGConf.EU talk
>> http://www.sai.msu.su/~megera/postgres/talks/hstore-dublin-2013.pdf
>> (since PGCon some features was added).
>>
>> Patch includes:
>> 1 implementaion SRF_RETURN_NEXT_NULL()
>> 2 contrib/hstore changes
>> 3 docs of new hstore module (many thanks to David E. Wheeler
>> <david(dot)wheeler(at)pgexperts(dot)com>)
>>
>> In current state patch is in WIP status, for short period I plan to
>> move support of binary nested structure to core to share binary
>> representation for hstore and json types.
>>
>>
>>
>
> Thanks, Teodor.
>
> As soon as we have that shared binary representation available, I will
> be working on adapting it to JSON.
As I remember from earlier discussions, current json has some
artefacts that some people want to preserve and which are incompatible
with hstore approach where you have actual object behind the serialisation.

I remember strong voices in support of *not* normalising json, so that
things like

{"a":1,"a":true, "a":"b", "a":none}

would go through the system unaltered, for claimed standard usage of
json as
"processing instructions". That is as source code which can possibly
converted
to JavaScript Object and not something that would come out of
serialising of
any existing JavaScript Object.

I suggest we add another type, maybe jsobj, which has input and output
as standard
"JSON" but which is defined from the start to be equivalent of existing
object
and not "preservable source code" to such object.

Cheers

--
Hannu Krosing
PostgreSQL Consultant
Performance, Scalability and High Availability
2ndQuadrant Nordic OÜ

On Nov 13, 2013, at 3:59 PM, Hannu Krosing <hannu(at)2ndquadrant(dot)com> wrote:

> I remember strong voices in support of *not* normalising json, so that
> things like
>
> {"a":1,"a":true, "a":"b", "a":none}
>
> would go through the system unaltered, for claimed standard usage of
> json as
> "processing instructions". That is as source code which can possibly
> converted
> to JavaScript Object and not something that would come out of
> serialising of
> any existing JavaScript Object.

My recollection from PGCon was that there was consensus to normalize on the way in -- or at least, if we switched to a binary representation as proposed by Oleg & Teodor, it was not worth the hassle to try to keep it.

> I suggest we add another type, maybe jsobj, which has input and output
> as standard
> "JSON" but which is defined from the start to be equivalent of existing
> object
> and not "preservable source code" to such object.

-1 Let's try to keep this simple. See also VARCHAR and VARCHAR2 on Oracle.

Best,

David

On 11/14/2013 01:32 AM, David E. Wheeler wrote:
> On Nov 13, 2013, at 3:59 PM, Hannu Krosing <hannu(at)2ndquadrant(dot)com> wrote:
>
>> I remember strong voices in support of *not* normalising json, so that
>> things like
>>
>> {"a":1,"a":true, "a":"b", "a":none}
>>
>> would go through the system unaltered, for claimed standard usage of
>> json as
>> "processing instructions". That is as source code which can possibly
>> converted
>> to JavaScript Object and not something that would come out of
>> serialising of
>> any existing JavaScript Object.
> My recollection from PGCon was that there was consensus to normalize on
> the way in --
Great news! I remember advocating this approach in the mailing lists
but having been out-voted based on "current real-world usage out there" :)
> or at least, if we switched to a binary representation as proposed by
> Oleg & Teodor, it was not worth the hassle to try to keep it.
Very much agree. For the source code approach I'd recommend
text type with maybe a check that it is possible to convert it to json.

--
Hannu Krosing
PostgreSQL Consultant
Performance, Scalability and High Availability
2ndQuadrant Nordic OÜ

On 11/14/2013 03:21 AM, Hannu Krosing wrote:
> On 11/14/2013 01:32 AM, David E. Wheeler wrote:
>> On Nov 13, 2013, at 3:59 PM, Hannu Krosing <hannu(at)2ndquadrant(dot)com> wrote:
>>
>>> I remember strong voices in support of *not* normalising json, so that
>>> things like
>>>
>>> {"a":1,"a":true, "a":"b", "a":none}
>>>
>>> would go through the system unaltered, for claimed standard usage of
>>> json as
>>> "processing instructions". That is as source code which can possibly
>>> converted
>>> to JavaScript Object and not something that would come out of
>>> serialising of
>>> any existing JavaScript Object.
>> My recollection from PGCon was that there was consensus to normalize on
>> the way in --
> Great news! I remember advocating this approach in the mailing lists
> but having been out-voted based on "current real-world usage out there" :)
>> or at least, if we switched to a binary representation as proposed by
>> Oleg & Teodor, it was not worth the hassle to try to keep it.
> Very much agree. For the source code approach I'd recommend
> text type with maybe a check that it is possible to convert it to json.
>

I don't think you and David are saying the same thing. AIUI he wants one
JSON type and is prepared to discard text preservation (duplicate keys
and key order). You want two json types, one of which would feature text
preservation.

Correct me if I'm wrong.

cheers

andrew

>

On 11/14/2013 01:47 PM, Andrew Dunstan wrote:
>
> On 11/14/2013 03:21 AM, Hannu Krosing wrote:
>> On 11/14/2013 01:32 AM, David E. Wheeler wrote:
>>> On Nov 13, 2013, at 3:59 PM, Hannu Krosing <hannu(at)2ndquadrant(dot)com>
>>> wrote:
>>>
>>>> I remember strong voices in support of *not* normalising json, so that
>>>> things like
>>>>
>>>> {"a":1,"a":true, "a":"b", "a":none}
>>>>
>>>> would go through the system unaltered, for claimed standard usage of
>>>> json as
>>>> "processing instructions". That is as source code which can possibly
>>>> converted
>>>> to JavaScript Object and not something that would come out of
>>>> serialising of
>>>> any existing JavaScript Object.
>>> My recollection from PGCon was that there was consensus to normalize on
>>> the way in --
>> Great news! I remember advocating this approach in the mailing lists
>> but having been out-voted based on "current real-world usage out
>> there" :)
>>> or at least, if we switched to a binary representation as proposed by
>>> Oleg & Teodor, it was not worth the hassle to try to keep it.
>> Very much agree. For the source code approach I'd recommend
>> text type with maybe a check that it is possible to convert it to json.
>>
>
>
> I don't think you and David are saying the same thing. AIUI he wants
> one JSON
> type and is prepared to discard text preservation (duplicate keys and
> key order).
> You want two json types, one of which would feature text preservation.
I actually *want* the same thing that David wants, but I think that
Merlin has
valid concerns about backwards compatibility.

If we have let this behaviour in, it is not nice to break several uses
of it now.

If we could somehow turn "old json" into a text domain with json syntax
check
(which it really is up to 9.3) via pg_upgrade that would be great.

It would be the required for pg_dump to have some swicth to output
different typename in CREATE TABLE and similar.
>
> Correct me if I'm wrong.
>
> cheers
>
> andrew
>
>
>>
>

--
Hannu Krosing
PostgreSQL Consultant
Performance, Scalability and High Availability
2ndQuadrant Nordic OÜ

Hi,

On 2013-11-12 22:35:31 +0400, Teodor Sigaev wrote:
> Attatched patch adds nesting feature, types (string, boll and numeric
> values), arrays and scalar to hstore type.

I took a quick peek at this:
* You cannot simply catch and ignore errors by doing
+ PG_TRY();
+ {
+ n = DatumGetNumeric(DirectFunctionCall3(numeric_in, CStringGetDatum(s->val), 0, -1));
+ }
+ PG_CATCH();
+ {
+ n = NULL;
+ }
+ PG_END_TRY();
That skips cleanup and might ignore some errors (think memory allocation
failures). But why do you even want to silently ignore errors there?
* Shouldn't the checks for v->size be done before filling the
datastructures in makeHStoreValueArray() and makeHStoreValuePairs()?
* could you make ORDER_PAIRS() a function instead of a macro? It's
pretty long and there's no reason not to use a function.
* You call numeric_recv via recvHStoreValue via recvHStore without
checks on the input length. That seems - without having checked it in
detail - a good way to read unrelated memory. Generally ISTM the input
needs to be more carefully checked in the whole recv function.
* There's quite some new new, completely uncommented, code. Especially
in hstore_op.c.
* the _PG_init you added should probably do a EmitWarningsOnPlaceholders().
* why does hstore need it's own atoi?
* shouldn't all the function prototypes be marked as externs?
* Lots of trailing whitespaces, quite some long lines, cuddly braces,
...
* I think hstore_compat.c's header should be updated.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Thu, Nov 14, 2013 at 05:50:02PM +0100, Hannu Krosing wrote:
> If we could somehow turn "old json" into a text domain with json syntax
> check
> (which it really is up to 9.3) via pg_upgrade that would be great.
>
> It would be the required for pg_dump to have some swicth to output
> different typename in CREATE TABLE and similar.

I don't think pg_upgrade isn't in a position to handle this. I think it
would require a script to be run after pg_upgrade completes.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +

On Wed, Nov 13, 2013 at 6:59 PM, Hannu Krosing <hannu(at)2ndquadrant(dot)com> wrote:
> I remember strong voices in support of *not* normalising json, so that
> things like
>
> {"a":1,"a":true, "a":"b", "a":none}
>
> would go through the system unaltered, for claimed standard usage of
> json as
> "processing instructions". That is as source code which can possibly
> converted
> to JavaScript Object and not something that would come out of
> serialising of
> any existing JavaScript Object.

Yeah, as the guy who wrote the original version of the JSON type,
which works just exactly like the XML type does, I stronly object to
changing the behavior. And doubly so now that it's released, as we
would be breaking backward compatibility.

> I suggest we add another type, maybe jsobj, which has input and output
> as standard
> "JSON" but which is defined from the start to be equivalent of existing
> object
> and not "preservable source code" to such object.

I think this was the consensus solution when this was last discussed,
and I support it. There is similar space for a binary XML data type
if someone feels like implementing it. I think the names that were
proposed previously were something like jsonb and xmlb.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, Nov 19, 2013 at 10:51:06AM -0500, Robert Haas wrote:
> I think this was the consensus solution when this was last discussed,
> and I support it. There is similar space for a binary XML data type
> if someone feels like implementing it. I think the names that were
> proposed previously were something like jsonb and xmlb.

The natural name is OBJSON, meaning object JSON, because as PostgreSQL
people, we have to double-use letters wherever possible. ;-)

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +

On 11/19/2013 10:51 AM, Robert Haas wrote:
>
>> I suggest we add another type, maybe jsobj, which has input and output
>> as standard
>> "JSON" but which is defined from the start to be equivalent of existing
>> object
>> and not "preservable source code" to such object.
> I think this was the consensus solution when this was last discussed,
> and I support it. There is similar space for a binary XML data type
> if someone feels like implementing it. I think the names that were
> proposed previously were something like jsonb and xmlb.
>

I think that's the consensus position on a strategy.

JSONB seems to be the current winner min the name stakes.

cheers

andrew

On Tue, Nov 19, 2013 at 10:55 AM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> On Tue, Nov 19, 2013 at 10:51:06AM -0500, Robert Haas wrote:
>> I think this was the consensus solution when this was last discussed,
>> and I support it. There is similar space for a binary XML data type
>> if someone feels like implementing it. I think the names that were
>> proposed previously were something like jsonb and xmlb.
>
> The natural name is OBJSON, meaning object JSON, because as PostgreSQL
> people, we have to double-use letters wherever possible. ;-)

Personally, I think the patch author should just run ps auxww | md5 |
sed 's/^[^0-9]//' | cut -c1-8 and call the new data type by the
resulting name.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 11/19/2013 11:00 AM, Robert Haas wrote:
> On Tue, Nov 19, 2013 at 10:55 AM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
>> On Tue, Nov 19, 2013 at 10:51:06AM -0500, Robert Haas wrote:
>>> I think this was the consensus solution when this was last discussed,
>>> and I support it. There is similar space for a binary XML data type
>>> if someone feels like implementing it. I think the names that were
>>> proposed previously were something like jsonb and xmlb.
>> The natural name is OBJSON, meaning object JSON, because as PostgreSQL
>> people, we have to double-use letters wherever possible. ;-)
> Personally, I think the patch author should just run ps auxww | md5 |
> sed 's/^[^0-9]//' | cut -c1-8 and call the new data type by the
> resulting name.
>

My personal vote goes for "marmaduke". I've been dying to call some
builtin object that for ages.

cheers

andrew

On 11/12/13, 1:35 PM, Teodor Sigaev wrote:
> Hi!
>
> Attatched patch adds nesting feature, types (string, boll and numeric
> values), arrays and scalar to hstore type.

Documentation doesn't build:

openjade:hstore.sgml:206:16:E: document type does not allow element "VARLISTENTRY" here; assuming missing "VARIABLELIST" start-tag

Compiler warnings:

hstore_io.c: In function 'array_to_hstore':
hstore_io.c:1736:29: error: 'result' may be used uninitialized in this function [-Werror=maybe-uninitialized]

On Nov 20, 2013, at 6:19 AM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:

> openjade:hstore.sgml:206:16:E: document type does not allow element "VARLISTENTRY" here; assuming missing "VARIABLELIST" start-tag

Thanks, I fixed this one.

David

On Nov 12, 2013, at 10:35 AM, Teodor Sigaev <teodor(at)sigaev(dot)ru> wrote:

> Hi!
>
> Attatched patch adds nesting feature, types (string, boll and numeric values), arrays and scalar to hstore type.

My apologies for not getting to this sooner, work has been a bit nutty. The truth is that I reviewed this patch quite a bit a month back, mostly so I could write documentation, the results of which are included in this patch. And I'm super excited for what's to come in the next iteration, as I hear that Teodor and Andrew are hard at work adding jsonb as a binary-compatible JSON data type.

Meanwhile, for this version, a quick overview of what has changed since 9.2.

Contents & Purpose
==================

Improved Data Type Support
--------------------------

* Added data type support for values. Previously they could only be strings or NULL, but with this patch they can also be numbers or booleans.

* Added array support. Values can be arrays of other values. The format for arrays is a bracketed, comma-delimited list.

* Added nesting support. hstore values can themselves be hstores. Nested hstores are wrapped in braces, but the root-level hstore is not (for compatibility with the format of previous versions of hstore).

* An hstore value is no longer required to be an hstore object. It can now be any scalar value.

These three items make the basic format feature-complete with JSON. Here's an example where the values are scalars:

=% SELECT 'foo'::hstore, '"hi \"bob\""'::hstore, '1.0'::hstore, 'true'::hstore, NULL::hstore;
hstore | hstore | hstore | hstore | hstore
--------+--------------+--------+--------+--------
"foo" | "hi \"bob\"" | 1.0 | t |

And here are a couple of arrays with strings, numbers, booleans, and NULLs:

SELECT '[k,v]'::hstore, '[1.0, "hi there", false, null]'::hstore;
hstore | hstore
------------+----------------------------
["k", "v"] | [1.0, "hi there", f, NULL]

Here's a complicated example formatted with `hstore.pretty_print` enabled.

=% SET hstore.pretty_print=true;
=% SELECT '{
"type" => "Feature",
"bbox" => [-180.0, -90.0, 180.0, 90.0],
"geometry" => {
"type" => "Polygon",
"coordinates" => [[
[-180.0, 10.0], [20.0, 90.0], [180.0, -5.0], [-30.0, -90.0]
]]
}
}'::hstore;
hstore
--------------------------
"bbox"=> +
[ +
-180.0, +
-90.0, +
180.0, +
90.0 +
], +
"type"=>"Feature", +
"geometry"=> +
{ +
"type"=>"Polygon", +
"coordinates"=> +
[ +
[ +
[ +
-180.0, +
10.0 +
], +
[ +
20.0, +
90.0 +
], +
[ +
180.0, +
-5.0 +
], +
[ +
-30.0, +
-90.0 +
] +
] +
] +
}

So, exact feature parity with the JSON data type.

* hstore.pretty_print is a new GUC, specifically to allow an HSTORE value to be pretty-printed. There is also a function to pretty-print, so we might be able to just do away with the GUC.

Interface
---------

* New operators:
+ `hstore -> int`: Get string value at array index (starting at 0)
+ `hstore ^> text`: Get numeric value for key
+ `hstore ^> int`: Get numeric value at array index
+ `hstore ?> text`: Get boolean value for key
+ `hstore ?> int`: Get boolean value at array index
+ `hstore #> text[]`: Get string value for key path
+ `hstore #^> text[]`: Get numeric value for key path
+ `hstore #?> text[]`: Get boolean value for key path
+ `hstore %> text`: Get hstore value for key
+ `hstore %> int`: Get hstore value at array index
+ `hstore #%> text[]`: Get hstore value for key path
+ `hstore ? int`: Does hstore contain array index
+ `hstore #? text[]`: Does hstore contain key path
+ `hstore - int`: Delete index from left operand
+ `hstore #- text[]`: Delete key path from left operand

* New functions:
+ `hstore(text)`: Make a text scalar hstore
+ `hstore(numeric)`: Make a numeric scalar hstore
+ `hstore(boolean)`: Make a boolean scalar hstore
+ `hstore(text, hstore)`: Make a nested hstore
+ `hstore(text, numeric)`: Make an hstore with a key and numeric value
+ `hstore(text, boolean)`: Make an hstore with a key and boolean value
+ `array_to_hstore(anyarray): Make an array hstore from an SQL array
+ `hvals(hstore)` Get values as a set of hstore values
+ `json_to_hstore(json)` Convert JSON to hstore
+ `each_hstore(hstore)` Get set of hstore key/value pairs
+ `hstore_typeof(hstore)` Return text name for the hstore type (hash, array, text, numeric, etc.)
+ `replace(hstore,text[],hstore)`: Replace value at specified path
+ `concat_path(hstore,text[],hstore)`: Concatenate hstore value at specified path
+ `hstore_print(hstore, params)`: Format hstore as text

The hstore_print() function has a number of optional boolean parameters to affect how the resulting text is formatted. They all default to false:

- pretty_print
- array_curly_braces: use {} instead of [] for arrays
- root_hash_decorated: Use {} for the root hash
- json: Format as JSON
- loose: Try to parse numbers and booleans from text values

Other Changes
-------------

* New casts: JSON and HSTORE can be cast to each other. I don't think they're implicit, though the forthcoming jsonb data type might support explicit casting to and from hstore, since internally they will be identical.

* The internal representation has been changed, but should be backward (and pg_upgrade) compatible, just as Andrew Gierth's change from 8.4 to 9.0 was. One can do an in-place update to rewrite all records at once. Of course, nested and/or non-hash hstore values dumped from 9.4 will not be able to be loaded into 9.3.

* GIN indexing is now supported. This is actually pretty amazing. For an hstore value, even hash keys are considered values, as far as the index is concerned. This makes it efficient to find hstore values that contain a key. I wrote an example in this blog post:

http://theory.so/pg/2013/10/25/indexing-nested-hstore/

Submission review
=================
* Is the patch in a patch format which has context? Yes.
* Does it apply cleanly to the current git master? It did for me, though I think Peter has found an issue or two since.
* Does it include reasonable tests, necessary doc patches, etc? Yes.

Usability review
================

* Does the patch actually implement what it says it does? Yes.
* Do we want that? OH yes.
* Do we already have it? No.
* Does it follow SQL spec, or the community-agreed behavior? Yes, though want jsonb, too.
* Does it include pg_dump support? Yes
* Are there dangers? Could break backward compatibility, though I don't think it does.
* Have all the bases been covered? I think so

Feature test
============

* Does the feature work as advertised? Yes.
* Are there corner cases the author has failed to consider? All I noticed were promptly fixed.
* Are there any assertion failures or crashes? No

Performance review
==================

* Does the patch slow down simple tests? No
* If it claims to improve performance, does it? Yes, with GIN index support. Loading hstore values is slower than loading JSON, but everything else is faster than JSON.
* Does it slow down other things? No.

Coding review
=============

* Does it follow the project guidelines? Yes.
* Are there portability issues? Unknown
* Will it work on Windows/BSD etc? Tested on OS X only.
* Are the comments sufficient and accurate? Yes.
* Does it do what it says, correctly? As best I can tell, yes.
* Does it produce compiler warnings? No.
* Can you make it crash? No, but I did find a bug or two that was promptly fixed.

Architecture review
===================

* Is everything done in a way that fits together coherently with other features/modules? yes.
* Are there interdependencies that can cause problems? No

Conclusion
==========

I love where nested hstore is going, especially since it will be used for jsonb, too. The nesting, data type, and GIN index support is really great, and the new constructors provide a nice SQL API that make it easy to use. I think that the next version of this patch will be full of win for the project.

This was considered a WIP patch, since the jsonb support is still forthcoming, so it's appropriate to leave it marked “Returned with feedback”. As Andrew is doing much of that work, the code itself will get a much closer examination from him. But for the hstore feature itself, I think the current interface and features are ready to go.

Best,

David

On 2013-12-20 15:16:30 -0800, David E. Wheeler wrote:
> But for the hstore feature itself, I think the current interface and features are ready to go.

I think this patch needs significant amount of work because it can be
considered ready for committer. I found the list of issues in
http://archives.postgresql.org/message-id/20131118163633.GE20305%40awork2.anarazel.de
within 10 minutes, indicating that it clearly cannot be ready yet.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Fri, Dec 20, 2013 at 6:16 PM, David E. Wheeler <david(at)justatheory(dot)com> wrote:
> * New operators:
> + `hstore -> int`: Get string value at array index (starting at 0)
> + `hstore ^> text`: Get numeric value for key
> + `hstore ^> int`: Get numeric value at array index
> + `hstore ?> text`: Get boolean value for key
> + `hstore ?> int`: Get boolean value at array index
> + `hstore #> text[]`: Get string value for key path
> + `hstore #^> text[]`: Get numeric value for key path
> + `hstore #?> text[]`: Get boolean value for key path
> + `hstore %> text`: Get hstore value for key
> + `hstore %> int`: Get hstore value at array index
> + `hstore #%> text[]`: Get hstore value for key path
> + `hstore ? int`: Does hstore contain array index
> + `hstore #? text[]`: Does hstore contain key path
> + `hstore - int`: Delete index from left operand
> + `hstore #- text[]`: Delete key path from left operand

Although in some ways there's a certain elegance to this, it also
sorta looks like punctuation soup. I can't help wondering whether
we'd be better off sticking to function names.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 12/23/2013 12:28 PM, Robert Haas wrote:
> On Fri, Dec 20, 2013 at 6:16 PM, David E. Wheeler <david(at)justatheory(dot)com> wrote:
>> * New operators:
>> + `hstore -> int`: Get string value at array index (starting at 0)
>> + `hstore ^> text`: Get numeric value for key
>> + `hstore ^> int`: Get numeric value at array index
>> + `hstore ?> text`: Get boolean value for key
>> + `hstore ?> int`: Get boolean value at array index
>> + `hstore #> text[]`: Get string value for key path
>> + `hstore #^> text[]`: Get numeric value for key path
>> + `hstore #?> text[]`: Get boolean value for key path
>> + `hstore %> text`: Get hstore value for key
>> + `hstore %> int`: Get hstore value at array index
>> + `hstore #%> text[]`: Get hstore value for key path
>> + `hstore ? int`: Does hstore contain array index
>> + `hstore #? text[]`: Does hstore contain key path
>> + `hstore - int`: Delete index from left operand
>> + `hstore #- text[]`: Delete key path from left operand
> Although in some ways there's a certain elegance to this, it also
> sorta looks like punctuation soup. I can't help wondering whether
> we'd be better off sticking to function names.
>
Has anybody looked into how hard it would be to add "method" notation
to postgreSQL, so that instead of calling

getString(hstorevalue, n)

we could use

hstorevalue.getString(n)

--
Hannu Krosing
PostgreSQL Consultant
Performance, Scalability and High Availability
2ndQuadrant Nordic OÜ

Hello

2013/12/23 Hannu Krosing <hannu(at)2ndquadrant(dot)com>

> On 12/23/2013 12:28 PM, Robert Haas wrote:
> > On Fri, Dec 20, 2013 at 6:16 PM, David E. Wheeler <david(at)justatheory(dot)com>
> wrote:
> >> * New operators:
> >> + `hstore -> int`: Get string value at array index (starting at 0)
> >> + `hstore ^> text`: Get numeric value for key
> >> + `hstore ^> int`: Get numeric value at array index
> >> + `hstore ?> text`: Get boolean value for key
> >> + `hstore ?> int`: Get boolean value at array index
> >> + `hstore #> text[]`: Get string value for key path
> >> + `hstore #^> text[]`: Get numeric value for key path
> >> + `hstore #?> text[]`: Get boolean value for key path
> >> + `hstore %> text`: Get hstore value for key
> >> + `hstore %> int`: Get hstore value at array index
> >> + `hstore #%> text[]`: Get hstore value for key path
> >> + `hstore ? int`: Does hstore contain array index
> >> + `hstore #? text[]`: Does hstore contain key path
> >> + `hstore - int`: Delete index from left operand
> >> + `hstore #- text[]`: Delete key path from left operand
> > Although in some ways there's a certain elegance to this, it also
> > sorta looks like punctuation soup. I can't help wondering whether
> > we'd be better off sticking to function names.
> >
> Has anybody looked into how hard it would be to add "method" notation
> to postgreSQL, so that instead of calling
>
> getString(hstorevalue, n)
>
> we could use
>
> hstorevalue.getString(n)
>

yes, I played with it some years ago. I ended early, there was a problem
with parser - when I tried append a new rule. And because there was not
simple solution, I didn't continue.

But it can be nice feature - minimally for plpgsql coders.

Regards

Pavel

>
> --
> Hannu Krosing
> PostgreSQL Consultant
> Performance, Scalability and High Availability
> 2ndQuadrant Nordic OÜ
>
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>

On Dec 23, 2013, at 6:28 AM, Robert Haas wrote:

> On Fri, Dec 20, 2013 at 6:16 PM, David E. Wheeler <david(at)justatheory(dot)com> wrote:
>> * New operators:
>> + `hstore -> int`: Get string value at array index (starting at 0)
>> + `hstore ^> text`: Get numeric value for key
>> + `hstore ^> int`: Get numeric value at array index
>> + `hstore ?> text`: Get boolean value for key
>> + `hstore ?> int`: Get boolean value at array index
>> + `hstore #> text[]`: Get string value for key path
>> + `hstore #^> text[]`: Get numeric value for key path
>> + `hstore #?> text[]`: Get boolean value for key path
>> + `hstore %> text`: Get hstore value for key
>> + `hstore %> int`: Get hstore value at array index
>> + `hstore #%> text[]`: Get hstore value for key path
>> + `hstore ? int`: Does hstore contain array index
>> + `hstore #? text[]`: Does hstore contain key path
>> + `hstore - int`: Delete index from left operand
>> + `hstore #- text[]`: Delete key path from left operand
>
> Although in some ways there's a certain elegance to this, it also
> sorta looks like punctuation soup. I can't help wondering whether
> we'd be better off sticking to function names.

The key thing is making it easy for people to easily chain calls to their nested hstore objects, and I think these operators accomplish that.

Some of them are fairly intuitive, and I think as a community if we have a) good docs, b) good blog posts on how to use nested hstore, and c) provides clear instructions @ PG events on how to use it, it would be okay, though some things, i.e. extracting the key by a path, might be better being in a function anyway. However, having it as an operator might encourage more usage, only because people tend to think that "functions will slow my query down."

My only concern is the consistency with the generally accepted standard of JSON and with the upcoming jsonb type. I'm not sure if the jsonb API has been defined yet, but it would be great to keep consistency between nested hstore and jsonb so people don't have to learn two different access systems. Data extraction from JSON is often done by the dot operator in implementations, and depending on the language you are in, there are ways to add / test existence / remove objects from the JSON blob.

Being able to extract objects from nested hstore / JSON using the dot operator would be simple and intuitive and general well-understood, but of course there are challenges with doing that in PG and well, proper SQL.

Jonathan

Attached is a new version of patch, which addresses most issues raised
by Andres.

It's long holidays in Russia now and it happened that Teodor is
traveling with family, so Teodor asked me to reply. Comments in code
will be added asap.

Oleg

On Mon, Nov 18, 2013 at 8:36 PM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> Hi,
>
> On 2013-11-12 22:35:31 +0400, Teodor Sigaev wrote:
>> Attatched patch adds nesting feature, types (string, boll and numeric
>> values), arrays and scalar to hstore type.
>
> I took a quick peek at this:
> * You cannot simply catch and ignore errors by doing
> + PG_TRY();
> + {
> + n = DatumGetNumeric(DirectFunctionCall3(numeric_in, CStringGetDatum(s->val), 0, -1));
> + }
> + PG_CATCH();
> + {
> + n = NULL;
> + }
> + PG_END_TRY();
> That skips cleanup and might ignore some errors (think memory allocation
> failures). But why do you even want to silently ignore errors there?

Fixed: now it's ignore only ERRCODE_INVALID_TEXT_REPRESENTATION error.
Our stringIsNumber() could be too optimistic.

> * Shouldn't the checks for v->size be done before filling the
> datastructures in makeHStoreValueArray() and makeHStoreValuePairs()?

Why check before filling? The result will be the same (throwing an
error) but it saves a loop over array/hash. String values aren't
copied during parse process.

> * could you make ORDER_PAIRS() a function instead of a macro? It's
> pretty long and there's no reason not to use a function.

fixed - macro and delaction argument was used for development.

> * You call numeric_recv via recvHStoreValue via recvHStore without
> checks on the input length. That seems - without having checked it in
> detail - a good way to read unrelated memory. Generally ISTM the input
> needs to be more carefully checked in the whole recv function.

numeric_recv checks this, otherwise we can say that numeric_recv
could not check its input. numeric_recv() gets a StringInfo buffer.

> * There's quite some new new, completely uncommented, code. Especially
> in hstore_op.c.

We'll comment asap

> * the _PG_init you added should probably do a EmitWarningsOnPlaceholders().

fixed

> * why does hstore need it's own atoi?

because:
* our pg_atoi should work with non-C-string (not finished with \0)
* doesn't throw an exception on error

> * shouldn't all the function prototypes be marked as externs?

fixed

> * Lots of trailing whitespaces, quite some long lines, cuddly braces,

tried to fix

> ...
> * I think hstore_compat.c's header should be updated.

yes, we'll do with

>
> Greetings,
>
> Andres Freund
>
> --
> Andres Freund http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Training & Services
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, January 8, 2014 22:29, Oleg Bartunov wrote:
> Attached is a new version of patch, which addresses most issues raised
> by Andres.

> [ nested_hstore-0.42.patch.gz ]

Building documentation fails:

openjade:hstore.sgml:1010:18:E: end tag for element "A" which is not open
openjade:hstore.sgml:1011:13:E: document type does not allow element "TYPE" here
openjade:hstore.sgml:1012:8:E: document type does not allow element "TYPE" here
openjade:hstore.sgml:1012:27:E: document type does not allow element "TYPE" here
openjade:hstore.sgml:1013:15:E: document type does not allow element "PROGRAMLISTING" here
openjade:hstore.sgml:1024:8:E: end tag for "TYPE" omitted, but OMITTAG NO was specified
openjade:hstore.sgml:1010:3: start tag was here
make: *** [HTML.index] Error 1
make: *** Deleting file `HTML.index'

This is caused by a small tag typo.

The attached fixes that hstore.sgml typo.

thanks,

Erikjan

On 12/23/13, 9:47 AM, Pavel Stehule wrote:
> Has anybody looked into how hard it would be to add "method" notation
> to postgreSQL, so that instead of calling
>
> getString(hstorevalue, n)
>
> we could use
>
> hstorevalue.getString(n)
>
>
> yes, I played with it some years ago. I ended early, there was a problem with parser - when I tried append a new rule. And because there was not simple solution, I didn't continue.
>
> But it can be nice feature - minimally for plpgsql coders.

Isn't there also some major problem with differentiating between schema/table/field with that too? I recall discussion along those lines, though maybe it was for the idea of recursive schemas.
--
Jim C. Nasby, Data Architect jim(at)nasby(dot)net
512.569.9461 (cell) http://jim.nasby.net

On 01/08/2014 04:29 PM, Oleg Bartunov wrote:
> Attached is a new version of patch, which addresses most issues raised
> by Andres.
>
> It's long holidays in Russia now and it happened that Teodor is
> traveling with family, so Teodor asked me to reply. Comments in code
> will be added asap.

Oleg,

Please merge in the jsonb work and resubmit. See
<https://github.com/feodor/postgres/commits/jsonb_and_hstore> I not that
this repo does not apparently contain any of your latest changes.

cheers

andrew

On 01/09/2014 06:12 AM, Andrew Dunstan wrote:
> Oleg,
>
> Please merge in the jsonb work and resubmit. See
> <https://github.com/feodor/postgres/commits/jsonb_and_hstore> I note that
> this repo does not apparently contain any of your latest changes.

I'll go further and say that if the Hstore2 patch doesn't support JSONB
for 9.4, we should postpone it to 9.5. We really don't want to get into
a situation where we need an Hstore3 because we accepted an Hstore2
which needs to be rev'd for JSON.

Especially since there's no good reason for the JSON changes not to be
merged already.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On 01/09/2014 02:11 PM, Josh Berkus wrote:
> On 01/09/2014 06:12 AM, Andrew Dunstan wrote:
>> Oleg,
>>
>> Please merge in the jsonb work and resubmit. See
>> <https://github.com/feodor/postgres/commits/jsonb_and_hstore> I note that
>> this repo does not apparently contain any of your latest changes.
> I'll go further and say that if the Hstore2 patch doesn't support JSONB
> for 9.4, we should postpone it to 9.5. We really don't want to get into
> a situation where we need an Hstore3 because we accepted an Hstore2
> which needs to be rev'd for JSON.
>
> Especially since there's no good reason for the JSON changes not to be
> merged already.
>

After some work by Oleg, for which I'm grateful, and a little more by
me, here is a combined patch for the jsonb and nested hstore work.

Outstanding issues with the jsonb stuff:

* I have replicated all the json processing functions for jsonb
(although not the json generating functions, such as to_json). Most
of these currently work by turning the jsonb back into json and then
processing as before. I am sorting out some technical issues and
hope to have all of these rewritten to use the native jsonb API in a
few days time.
* We still need to document jsonb. That too I hope will be done quite
shortly.
* The jsonb regression test currently contains U+ABCD - I guess we'd
better use some hex encoding or whatever for that - unlike json, the
jsonb de-serializer dissolves unicode escapes.

cheers

andrew

I moved patch to the January commitfest
(https://commitfest.postgresql.org/action/patch_view?id=1289) .

Oleg

PS.

Kudos to Teodor and his mobile phone, which he used to synchronize
branches on github.

On Fri, Jan 10, 2014 at 2:08 AM, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>
> On 01/09/2014 02:11 PM, Josh Berkus wrote:
>>
>> On 01/09/2014 06:12 AM, Andrew Dunstan wrote:
>>>
>>> Oleg,
>>>
>>> Please merge in the jsonb work and resubmit. See
>>> <https://github.com/feodor/postgres/commits/jsonb_and_hstore> I note that
>>> this repo does not apparently contain any of your latest changes.
>>
>> I'll go further and say that if the Hstore2 patch doesn't support JSONB
>> for 9.4, we should postpone it to 9.5. We really don't want to get into
>> a situation where we need an Hstore3 because we accepted an Hstore2
>> which needs to be rev'd for JSON.
>>
>> Especially since there's no good reason for the JSON changes not to be
>> merged already.
>>
>
>
> After some work by Oleg, for which I'm grateful, and a little more by me,
> here is a combined patch for the jsonb and nested hstore work.
>
> Outstanding issues with the jsonb stuff:
>
> * I have replicated all the json processing functions for jsonb
> (although not the json generating functions, such as to_json). Most
> of these currently work by turning the jsonb back into json and then
> processing as before. I am sorting out some technical issues and
> hope to have all of these rewritten to use the native jsonb API in a
> few days time.
> * We still need to document jsonb. That too I hope will be done quite
> shortly.
> * The jsonb regression test currently contains U+ABCD - I guess we'd
> better use some hex encoding or whatever for that - unlike json, the
> jsonb de-serializer dissolves unicode escapes.
>
>
> cheers
>
> andrew
>
>
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>

On Thu, Jan 9, 2014 at 5:08 PM, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
> * I have replicated all the json processing functions for jsonb
> (although not the json generating functions, such as to_json). Most
> of these currently work by turning the jsonb back into json and then
> processing as before. I am sorting out some technical issues and
> hope to have all of these rewritten to use the native jsonb API in a
> few days time.
> * We still need to document jsonb. That too I hope will be done quite
> shortly.
> * The jsonb regression test currently contains U+ABCD - I guess we'd
> better use some hex encoding or whatever for that - unlike json, the
> jsonb de-serializer dissolves unicode escapes.

How does that work if the server encoding isn't UTF-8?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 01/10/2014 01:29 PM, Robert Haas wrote:
> On Thu, Jan 9, 2014 at 5:08 PM, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>> * The jsonb regression test currently contains U+ABCD - I guess we'd
>> better use some hex encoding or whatever for that - unlike json, the
>> jsonb de-serializer dissolves unicode escapes.
> How does that work if the server encoding isn't UTF-8?
>

There is a jsonb_1.out file for the non-utf8 case, just as there is a
json_1.out for the same case. Unicode escapes for non-ascii characters
are forbidden in jsonb as they are in json, if the encoding isn't utf8.

FYI, we are actually using the json lexing and parsing mechanism, so
that these types will accept exactly the same inputs. However, since
we're not storing json text in jsonb, but instead the decomposed
elements, the unicode escapes are resolved in the stored values.

I already have a fix for the point above (see
<https://github.com/feodor/postgres/commit/7d5b8f12747b4a75e8b32914340d07617f1af302>)
and it will be included in the next version of the patch.

cheers

andrew

The documentation doesn't build.

On Sat, January 11, 2014 20:30, Peter Eisentraut wrote:
> The documentation doesn't build.

corrective patch is here:

http://www.postgresql.org/message-id/37b9f104d5a838eec9b75f3668517aa5.squirrel@webmail.xs4all.nl

On 01/11/2014 03:03 PM, Erik Rijkers wrote:
> On Sat, January 11, 2014 20:30, Peter Eisentraut wrote:
>> The documentation doesn't build.
> corrective patch is here:
>
> http://www.postgresql.org/message-id/37b9f104d5a838eec9b75f3668517aa5.squirrel@webmail.xs4all.nl
>

It's been committed at
<https://github.com/feodor/postgres/commit/a21a4be55a5b12c4bd89b6ab2f77cf32e319de31>.
It will be in the next version of the patch posted.

cheers

andrew

On Jan 11, 2014, at 1:47 PM, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:

> It's been committed at <https://github.com/feodor/postgres/commit/a21a4be55a5b12c4bd89b6ab2f77cf32e319de31>. It will be in the next version of the patch posted.

Bah! Sorry about that. Habit from decades of typing HTML.

David

On Sat, January 11, 2014 22:47, Andrew Dunstan wrote:
>
> On 01/11/2014 03:03 PM, Erik Rijkers wrote:
>> On Sat, January 11, 2014 20:30, Peter Eisentraut wrote:
>>> The documentation doesn't build.
>> corrective patch is here:
>>
>> http://www.postgresql.org/message-id/37b9f104d5a838eec9b75f3668517aa5.squirrel@webmail.xs4all.nl
>>
> It will be in the next version of the patch posted.

Attached is another handful of doc-fixes...

On Mon, January 13, 2014 00:24, Erik Rijkers wrote:
> On Sat, January 11, 2014 22:47, Andrew Dunstan wrote:
>>
>> On 01/11/2014 03:03 PM, Erik Rijkers wrote:
>>> On Sat, January 11, 2014 20:30, Peter Eisentraut wrote:
>>>> The documentation doesn't build.
>>> corrective patch is here:
>>>
>>> http://www.postgresql.org/message-id/37b9f104d5a838eec9b75f3668517aa5.squirrel@webmail.xs4all.nl
>>>
>> It will be in the next version of the patch posted.
>
>
> Attached is another handful of doc-fixes...
>

There are errors in the example expressions in "Table F-6. hstore Operators".

Attached is a cumulative doc-patch (which includes the changes I sent earlier) which fixes these.

I also attach an test perl program that shows the (small) differences in output between what's in that doc table and what
one actually gets. (I found these too insignificant to change but perhaps you have a different opinion.)

thanks,

Erik Rijkers

Thank you, Erik !

Oleg

On Mon, Jan 13, 2014 at 12:25 PM, Erik Rijkers <er(at)xs4all(dot)nl> wrote:
> On Mon, January 13, 2014 00:24, Erik Rijkers wrote:
>> On Sat, January 11, 2014 22:47, Andrew Dunstan wrote:
>>>
>>> On 01/11/2014 03:03 PM, Erik Rijkers wrote:
>>>> On Sat, January 11, 2014 20:30, Peter Eisentraut wrote:
>>>>> The documentation doesn't build.
>>>> corrective patch is here:
>>>>
>>>> http://www.postgresql.org/message-id/37b9f104d5a838eec9b75f3668517aa5.squirrel@webmail.xs4all.nl
>>>>
>>> It will be in the next version of the patch posted.
>>
>>
>> Attached is another handful of doc-fixes...
>>
>
> There are errors in the example expressions in "Table F-6. hstore Operators".
>
> Attached is a cumulative doc-patch (which includes the changes I sent earlier) which fixes these.
>
> I also attach an test perl program that shows the (small) differences in output between what's in that doc table and what
> one actually gets. (I found these too insignificant to change but perhaps you have a different opinion.)
>
>
> thanks,
>
> Erik Rijkers
>
>
>
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>

On 01/13/2014 03:25 AM, Erik Rijkers wrote:

> There are errors in the example expressions in "Table F-6. hstore Operators".
>
> Attached is a cumulative doc-patch (which includes the changes I sent earlier) which fixes these.
>
> I also attach an test perl program that shows the (small) differences in output between what's in that doc table and what
> one actually gets. (I found these too insignificant to change but perhaps you have a different opinion.)
>
>

A new version of the patch is attached. It includes all of Erik's docs
fixes and a small fix by Alexander Korotkov for hstore hash ops.

cheers

andrew

Andrew Dunstan wrote:
>
> On 01/13/2014 03:25 AM, Erik Rijkers wrote:
>
> >There are errors in the example expressions in "Table F-6. hstore Operators".
> >
> >Attached is a cumulative doc-patch (which includes the changes I sent earlier) which fixes these.
> >
> >I also attach an test perl program that shows the (small) differences in output between what's in that doc table and what
> >one actually gets. (I found these too insignificant to change but perhaps you have a different opinion.)
>
>
> A new version of the patch is attached. It includes all of Erik's
> docs fixes and a small fix by Alexander Korotkov for hstore hash
> ops.

Interestingly, this also include transaction_commit event triggers.

There are also a few PANIC elogs, probably not what's intended.

(I was just giving this a quick skim to see if there's support to build
JSON objects incrementally from C source, i.e. not have to call
functions using the fmgr interface. Apparently that's not the case, but
if I'm wrong please let me know.)

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Andrew,

did you run perl script ? Actually, I found, that operator table needs
to be fixed.

Oleg

On Mon, Jan 13, 2014 at 7:36 PM, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>
> On 01/13/2014 03:25 AM, Erik Rijkers wrote:
>
>> There are errors in the example expressions in "Table F-6. hstore
>> Operators".
>>
>> Attached is a cumulative doc-patch (which includes the changes I sent
>> earlier) which fixes these.
>>
>> I also attach an test perl program that shows the (small) differences in
>> output between what's in that doc table and what
>> one actually gets. (I found these too insignificant to change but perhaps
>> you have a different opinion.)
>>
>>
>
>
>
> A new version of the patch is attached. It includes all of Erik's docs fixes
> and a small fix by Alexander Korotkov for hstore hash ops.
>
> cheers
>
> andrew
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>

On 01/13/2014 11:03 AM, Alvaro Herrera wrote:
> Andrew Dunstan wrote:
>> On 01/13/2014 03:25 AM, Erik Rijkers wrote:
>>
>>> There are errors in the example expressions in "Table F-6. hstore Operators".
>>>
>>> Attached is a cumulative doc-patch (which includes the changes I sent earlier) which fixes these.
>>>
>>> I also attach an test perl program that shows the (small) differences in output between what's in that doc table and what
>>> one actually gets. (I found these too insignificant to change but perhaps you have a different opinion.)
>>
>> A new version of the patch is attached. It includes all of Erik's
>> docs fixes and a small fix by Alexander Korotkov for hstore hash
>> ops.
> Interestingly, this also include transaction_commit event triggers.

Oh, wow, really? git really did something horrible, or I did
inadvertently. This is what comes from using the same directory for
multiple development lines :-(

Will fix

>
> There are also a few PANIC elogs, probably not what's intended.

Oleg, Teodor, please address.

>
> (I was just giving this a quick skim to see if there's support to build
> JSON objects incrementally from C source, i.e. not have to call
> functions using the fmgr interface. Apparently that's not the case, but
> if I'm wrong please let me know.)

Erm, maybe you need the other json patch:
<http://www.postgresql.org/message-id/52C76B33.1050808@dunslane.net>

If we need to adjust some of that a bit to make it more friendly for
internal use I'm happy to try to do that. Unfortunately, I don't think
that's terribly easy for VARIADIC "any" functions like these.

cheers

andrew

On 01/13/2014 11:16 AM, Oleg Bartunov wrote:
> Andrew,
>
> did you run perl script ? Actually, I found, that operator table needs
> to be fixed.
>
>

No. My build machine doesn't actually have DBD::Pg installed. Can you
send me a patch if you don't want to push it yourself, or maybe Erik can
send a pacth top adjust the table.

cheers

andrew

On Mon, January 13, 2014 18:30, Andrew Dunstan wrote:
>
>
> On 01/13/2014 11:16 AM, Oleg Bartunov wrote:
>> Andrew,
>>
>> did you run perl script ? Actually, I found, that operator table needs
>> to be fixed.
>>
>
> No. My build machine doesn't actually have DBD::Pg installed. Can you
> send me a patch if you don't want to push it yourself, or maybe Erik can
> send a pacth top adjust the table.
>

> [ nested_hstore_and_jsonb-2.patch ]

( centos 6.5, gcc 4.8.2. )

The patch applies & compiles with warnings (see below).

The opr_sanity test fails during make check: regression.diffs attached.

Also attached are changes to hstore.sgml, to operator + functions table, plus some typos.

Thanks,
Erik Rijkers

make

jsonfuncs.c: In function ‘each_object_field_end_jsonb’:
jsonfuncs.c:1328:7: warning: assignment from incompatible pointer type [enabled by default]
val = DatumGetPointer(DirectFunctionCall1(jsonb_in, CStringGetDatum(cstr)));
^
jsonfuncs.c: In function ‘elements_array_element_end_jsonb’:
jsonfuncs.c:1530:8: warning: assignment from incompatible pointer type [enabled by default]
jbval = DatumGetPointer(DirectFunctionCall1(jsonb_in, CStringGetDatum(cstr)));
^

make contrib:

hstore_io.c: In function ‘array_to_hstore’:
hstore_io.c:1694:30: warning: ‘result’ may be used uninitialized in this function [-Wmaybe-uninitialized]
PG_RETURN_POINTER(hstoreDump(result));

Erik,

thanks for docs fixes, we have even more :)

Oleg

On Tue, Jan 14, 2014 at 4:18 AM, Erik Rijkers <er(at)xs4all(dot)nl> wrote:
> On Mon, January 13, 2014 18:30, Andrew Dunstan wrote:
>>
>>
>> On 01/13/2014 11:16 AM, Oleg Bartunov wrote:
>>> Andrew,
>>>
>>> did you run perl script ? Actually, I found, that operator table needs
>>> to be fixed.
>>>
>>
>> No. My build machine doesn't actually have DBD::Pg installed. Can you
>> send me a patch if you don't want to push it yourself, or maybe Erik can
>> send a pacth top adjust the table.
>>
>
>> [ nested_hstore_and_jsonb-2.patch ]
>
> ( centos 6.5, gcc 4.8.2. )
>
> The patch applies & compiles with warnings (see below).
>
> The opr_sanity test fails during make check: regression.diffs attached.
>
> Also attached are changes to hstore.sgml, to operator + functions table, plus some typos.
>
> Thanks,
> Erik Rijkers
>
>
> make
>
> jsonfuncs.c: In function ‘each_object_field_end_jsonb’:
> jsonfuncs.c:1328:7: warning: assignment from incompatible pointer type [enabled by default]
> val = DatumGetPointer(DirectFunctionCall1(jsonb_in, CStringGetDatum(cstr)));
> ^
> jsonfuncs.c: In function ‘elements_array_element_end_jsonb’:
> jsonfuncs.c:1530:8: warning: assignment from incompatible pointer type [enabled by default]
> jbval = DatumGetPointer(DirectFunctionCall1(jsonb_in, CStringGetDatum(cstr)));
> ^
>
>
> make contrib:
>
> hstore_io.c: In function ‘array_to_hstore’:
> hstore_io.c:1694:30: warning: ‘result’ may be used uninitialized in this function [-Wmaybe-uninitialized]
> PG_RETURN_POINTER(hstoreDump(result));
>
>
>
>
>

On Mon, January 13, 2014 16:36, Andrew Dunstan wrote:
> A new version of the patch is attached. It includes all of Erik's docs

> [ nested_hstore_and_jsonb-2.patch ]

This crashes the server:

testdb=# select 'x' || ('a=>"1"':: hstore) ;
The connection to the server was lost. Attempting reset: Failed.

logging:
TRAP: FailedAssertion("!(value->array.nelems == 1)", File: "jsonb_support.c", Line: 904)
2014-01-15 00:32:01.854 CET 1206 LOG: server process (PID 3918) was terminated by signal 6: Aborted
2014-01-15 00:32:01.854 CET 1206 DETAIL: Failed process was running: select 'x' || ('a=>"1"':: hstore) ;

Btw, I find it strange that:

testdb=# select ('a=>""':: hstore) #%> '{a}' ;
?column?
----------
""
(1 row)

so that:

Time: 0.641 ms
testdb=# select ( ('a=>""':: hstore) #%> '{a}' ) = '' ;
?column?
----------
f
(1 row)

testdb=# select ( ('a=>""':: hstore) #%> '{a}' ) = '""' ;
?column?
----------
t
(1 row)

Maybe there is a rationale, but it seems to me that
('a=>""':: hstore) #%> '{a}'
should deliver the empty string '', and not two double quotes.

Thanks,

Erik Rijkers

It doesn't crashed in the last version in our repository.

=# select 'x'::hstore || ('a=>"1"':: hstore) ;
?column?
---------------
"x", "a", "1"
(1 row)

On Wed, Jan 15, 2014 at 3:53 AM, Erik Rijkers <er(at)xs4all(dot)nl> wrote:
> On Mon, January 13, 2014 16:36, Andrew Dunstan wrote:
>> A new version of the patch is attached. It includes all of Erik's docs
>
>> [ nested_hstore_and_jsonb-2.patch ]
>
>
> This crashes the server:
>
> testdb=# select 'x' || ('a=>"1"':: hstore) ;
> The connection to the server was lost. Attempting reset: Failed.
>
> logging:
> TRAP: FailedAssertion("!(value->array.nelems == 1)", File: "jsonb_support.c", Line: 904)
> 2014-01-15 00:32:01.854 CET 1206 LOG: server process (PID 3918) was terminated by signal 6: Aborted
> 2014-01-15 00:32:01.854 CET 1206 DETAIL: Failed process was running: select 'x' || ('a=>"1"':: hstore) ;
>
>
> Btw, I find it strange that:
>
> testdb=# select ('a=>""':: hstore) #%> '{a}' ;
> ?column?
> ----------
> ""
> (1 row)
>
> so that:
>
> Time: 0.641 ms
> testdb=# select ( ('a=>""':: hstore) #%> '{a}' ) = '' ;
> ?column?
> ----------
> f
> (1 row)
>
> testdb=# select ( ('a=>""':: hstore) #%> '{a}' ) = '""' ;
> ?column?
> ----------
> t
> (1 row)
>
> Maybe there is a rationale, but it seems to me that
> ('a=>""':: hstore) #%> '{a}'
> should deliver the empty string '', and not two double quotes.
>
>
> Thanks,
>
> Erik Rijkers
>
>
>
>
>
>
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, January 15, 2014 08:01, Oleg Bartunov wrote:
> It doesn't crashed in the last version in our repository.
>
> =# select 'x'::hstore || ('a=>"1"':: hstore) ;
> ?column?
> ---------------
> "x", "a", "1"
> (1 row)
>

OK, shall I use that repository instead of the latest posted patch?
No point in testing old code ( I used nested_hstore_and_jsonb-2.patch ).

Could you send a link to where that repository is?

( btw, your query is not quite the same as the one I used:
select 'x' || ('a=>"1"':: hstore)
but your query also crashes my server here so I suppose
it triggers the same bug )

thanks,

Erik Rijkers

https://github.com/feodor/postgres

On Wed, Jan 15, 2014 at 11:10 AM, Erik Rijkers <er(at)xs4all(dot)nl> wrote:
> On Wed, January 15, 2014 08:01, Oleg Bartunov wrote:
>> It doesn't crashed in the last version in our repository.
>>
>> =# select 'x'::hstore || ('a=>"1"':: hstore) ;
>> ?column?
>> ---------------
>> "x", "a", "1"
>> (1 row)
>>
>
> OK, shall I use that repository instead of the latest posted patch?
> No point in testing old code ( I used nested_hstore_and_jsonb-2.patch ).
>
> Could you send a link to where that repository is?
>
> ( btw, your query is not quite the same as the one I used:
> select 'x' || ('a=>"1"':: hstore)
> but your query also crashes my server here so I suppose
> it triggers the same bug )
>
>
> thanks,
>
> Erik Rijkers
>

On Wed, January 15, 2014 09:46, Oleg Bartunov wrote:
>> On Wed, January 15, 2014 08:01, Oleg Bartunov wrote:
>>> It doesn't crashed in the last version in our repository.
>>>
>>> =# select 'x'::hstore || ('a=>"1"':: hstore) ;
>>> ?column?
>>> ---------------
>>> "x", "a", "1"
>>> (1 row)
>>>
>>
>> OK, shall I use that repository instead of the latest posted patch?

I now installed from:
https://github.com/feodor/postgres

and compiled both a 'fast' and a 'debug' server (=with --enable-cassert see [1])

It turns out that the statement does not crash on a server compiled without --enable-cassert.

But a compile with --enable-cassert shows that a bug is still lurking:

testdb=# select 'x'::hstore || ('a=>"1"':: hstore) ;
The connection to the server was lost. Attempting reset: Failed.
!>

TRAP: FailedAssertion("!(value->array.nelems == 1)", File: "hstore_support.c", Line: 896)

Not good.

( please note that the assert is in a different file ('hstore_support.c') from the earlier assert error that I posted )

Thanks,

Erik Rijkers

[1]
pg_config:
'--prefix=/home/aardvark/pg_stuff/pg_installations/pgsql.nested_hstore_url'
'--bindir=/home/aardvark/pg_stuff/pg_installations/pgsql.nested_hstore_url/bin'
'--libdir=/home/aardvark/pg_stuff/pg_installations/pgsql.nested_hstore_url/lib' '--with-pgport=46541' '--enable-depend'
'--enable-cassert' '--enable-debug' '--with-openssl' '--with-perl' '--with-libxml' '--with-libxslt' '--with-zlib'

On 01/15/2014 05:32 AM, Erik Rijkers wrote:
> On Wed, January 15, 2014 09:46, Oleg Bartunov wrote:
>>> On Wed, January 15, 2014 08:01, Oleg Bartunov wrote:
>>>> It doesn't crashed in the last version in our repository.
>>>>
>>>> =# select 'x'::hstore || ('a=>"1"':: hstore) ;
>>>> ?column?
>>>> ---------------
>>>> "x", "a", "1"
>>>> (1 row)
>>>>
>>> OK, shall I use that repository instead of the latest posted patch?

You can always check the jsonb_and_hstore branch on the repo for the
latest and greatest.

But here is the latest patch from that.

cheers

andrew

On 01/15/2014 11:25 AM, Andrew Dunstan wrote:
>
>
> But here is the latest patch from that.

And one more.

here is the current state of the jsonb part of this:

all but two sets of functions (jsonb_extract_path* and the associated
operators #> and #>>, and
jsonb_populate_record/jsonb_populate_record_set) have been reimplemented
using the faster processing of jsonb. Those two sets still fall back on
processing json text. These will be fixed very shortly. I'm waiting on
some promised documentation. There is also some code cleanup (add more
comments, use ereport() instead of elog() and the like) that needs to be
done. However, it all works, and is just pretty much completely
consistent with the way json works.

I'm sure Oleg, Teodor and I will be able to get this whole thing whipped
into shape pretty quickly.

cheers

andrew