Re: JsonbValue to Jsonb conversion

Hi all,

I'm faced with some troubles about the jsonb implementation, and I hope
I'll get little advice =)
If I understand correctly, an abstract function for jsonb modification
should have the following stages:

Jsonb -> JsonbValue -> Modification -> JsonbValue -> Jsonb

One can convert the *JsonbValue* to the *Jsonb* only by *JsonbValueToJsonb*
function. So, my question is can be *JsonbValue*, that contains few
*jbvBinary* elements, converted to *Jsonb* by this function? It will be
very useful, if you want modify only small part of your JsonbValue (e.g.
replace value of some key). But when I'm trying to do this, an exception
"unknown type of jsonb container" appears. Maybe I missed something? Or is
there another approach to do this conversion?

On 09/23/2014 12:23 PM, Dmitry Dolgov wrote:
> Hi all,
>
> I'm faced with some troubles about the jsonb implementation, and I
> hope I'll get little advice =)
> If I understand correctly, an abstract function for jsonb modification
> should have the following stages:
>
> Jsonb -> JsonbValue -> Modification -> JsonbValue -> Jsonb
>
> One can convert the *JsonbValue* to the *Jsonb* only by
> *JsonbValueToJsonb* function. So, my question is can be *JsonbValue*,
> that contains few *jbvBinary* elements, converted to *Jsonb* by this
> function? It will be very useful, if you want modify only small part
> of your JsonbValue (e.g. replace value of some key). But when I'm
> trying to do this, an exception "unknown type of jsonb container"
> appears. Maybe I missed something? Or is there another approach to do
> this conversion?

If you can come up with a way of handling the jbvBinary values then by
all means send a patch.

But this problem is fairly easily worked around by using an iterator
over the binary value. The attached patch, which is work in progress for
adding in the currently missing json functions for jsonb, contains a
sort of example of doing this in jsonb_agg_transfn.

cheers

andrew

Hi

I am working on review of this patch.

There is new warnings:

jsonb.c: In function ‘jsonb_agg_transfn’:
jsonb.c:1540:20: warning: assignment makes pointer from integer without a
cast
v.val.numeric = DirectFunctionCall1(numeric_uplus,
NumericGetDatum(v.val.numeric));
^
jsonb.c: In function ‘jsonb_object_agg_transfn’:
jsonb.c:1745:20: warning: assignment makes pointer from integer without a
cast
v.val.numeric = DirectFunctionCall1(numeric_uplus,
NumericGetDatum(v.val.numeric));

[pavel(at)localhost postgresql]$ gcc --version
gcc (GCC) 4.9.1 20140930 (Red Hat 4.9.1-11)

Check fails

parallel group (19 tests): alter_table plancache temp domain prepare limit
plpgsql conversion sequence copy2 rangefuncs returning truncate xml with
without_oid largeobject polymorphism rowtypes
plancache ... FAILED (test process exited with exit
code 2)
limit ... FAILED (test process exited with exit
code 2)
plpgsql ... FAILED (test process exited with exit
code 2)
copy2 ... FAILED (test process exited with exit
code 2)
temp ... FAILED (test process exited with exit
code 2)
domain ... FAILED (test process exited with exit
code 2)
rangefuncs ... FAILED (test process exited with exit
code 2)
prepare ... FAILED (test process exited with exit
code 2)
without_oid ... FAILED (test process exited with exit
code 2)
conversion ... FAILED (test process exited with exit
code 2)
truncate ... FAILED (test process exited with exit
code 2)
alter_table ... FAILED (test process exited with exit
code 2)
sequence ... FAILED (test process exited with exit
code 2)
polymorphism ... FAILED (test process exited with exit
code 2)
rowtypes ... FAILED (test process exited with exit
code 2)
returning ... FAILED (test process exited with exit
code 2)
largeobject ... FAILED (test process exited with exit
code 2)
with ... FAILED (test process exited with exit
code 2)
xml ... FAILED (test process exited with exit
code 2)
test stats ... FAILED (test process exited with exit
code 2)

[pavel(at)localhost postgresql]$ uname -a
Linux localhost.localdomain 3.16.3-302.fc21.x86_64 #1 SMP Fri Sep 26
14:27:20 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

backtrace

Core was generated by `postgres: pavel regression [local]
SELECT '.
Program terminated with signal SIGSEGV, Segmentation fault.

(gdb) bt
#0 0x0000000001e95300 in ?? ()
#1 0x00000000007c048b in parse_object_field (lex=0x1ede9d8,
sem=0x7fff3c3c4660) at json.c:398
#2 0x00000000007c0524 in parse_object (lex=0x1ede9d8, sem=0x7fff3c3c4660)
at json.c:430
#3 0x00000000007c0214 in pg_parse_json (lex=0x1ede9d8, sem=0x7fff3c3c4660)
at json.c:297
#4 0x00000000007c5d91 in datum_to_jsonb (val=32118224, is_null=0 '\000',
result=0x7fff3c3c4800, tcategory=JSONBTYPE_JSON,
outfuncoid=322, key_scalar=0 '\000') at jsonb.c:789
#5 0x00000000007c68be in add_jsonb (val=32118224, is_null=0 '\000',
result=0x7fff3c3c4800, val_type=114, key_scalar=0 '\000')
at jsonb.c:1050
#6 0x00000000007c6d08 in jsonb_build_object (fcinfo=0x1edcb80) at
jsonb.c:1155
#7 0x000000000060bfc5 in ExecMakeFunctionResultNoSets (fcache=0x1edcb10,
econtext=0x1edc920, isNull=0x1edd568 "", isDone=0x1edd680)
at execQual.c:1992
#8 0x000000000060c8bc in ExecEvalFunc (fcache=0x1edcb10,
econtext=0x1edc920, isNull=0x1edd568 "", isDone=0x1edd680)
at execQual.c:2383
#9 0x0000000000612869 in ExecTargetList (targetlist=0x1edd650,
econtext=0x1edc920, values=0x1edd550, isnull=0x1edd568 "",
itemIsDone=0x1edd680, isDone=0x7fff3c3c4a84) at execQual.c:5265
#10 0x0000000000612e9d in ExecProject (projInfo=0x1edd580,
isDone=0x7fff3c3c4a84) at execQual.c:5480
#11 0x000000000062c046 in ExecResult (node=0x1edc810) at nodeResult.c:155
#12 0x0000000000608997 in ExecProcNode (node=0x1edc810) at
execProcnode.c:373
#13 0x000000000060696e in ExecutePlan (estate=0x1edc700,
planstate=0x1edc810, operation=CMD_SELECT, sendTuples=1 '\001',
numberTuples=0, direction=ForwardScanDirection, dest=0x1ea18b0) at
execMain.c:1481
#14 0x0000000000604de8 in standard_ExecutorRun (queryDesc=0x1edc2f0,
direction=ForwardScanDirection, count=0) at execMain.c:308
#15 0x0000000000604ce5 in ExecutorRun (queryDesc=0x1edc2f0,
direction=ForwardScanDirection, count=0) at execMain.c:256
#16 0x000000000075615a in PortalRunSelect (portal=0x1eda2e0, forward=1
'\001', count=0, dest=0x1ea18b0) at pquery.c:946
#17 0x0000000000755e34 in PortalRun (portal=0x1eda2e0,
count=9223372036854775807, isTopLevel=1 '\001', dest=0x1ea18b0,
altdest=0x1ea18b0, completionTag=0x7fff3c3c4dc0 "") at pquery.c:790
#18 0x00000000007502c2 in exec_simple_query (
query_string=0x1ea0050 "SELECT jsonb_build_object('e',json '{\"x\": 3,
\"y\": [1,2,3]}');") at postgres.c:1045
#19 0x0000000000754284 in PostgresMain (argc=1, argv=0x1e45448,
dbname=0x1e452f8 "postgres", username=0x1e452e0 "pavel")
at postgres.c:4010
#20 0x00000000006e7954 in BackendRun (port=0x1e64d20) at postmaster.c:4118
#21 0x00000000006e70ac in BackendStartup (port=0x1e64d20) at
postmaster.c:3793
#22 0x00000000006e3ba9 in ServerLoop () at postmaster.c:1572
#23 0x00000000006e327c in PostmasterMain (argc=3, argv=0x1e44540) at
postmaster.c:1219

Problematic statement:

SELECT jsonb_build_array('a',1,'b',1.2,'c',true,'d',null,'e',json '{"x": 3,
"y": [1,2,3]}');

or SELECT jsonb_build_object('e',json '{"x": 3, "y": [1,2,3]}');

Tested on HEAD

Regards

Pavel

2014-09-24 2:20 GMT+02:00 Andrew Dunstan <andrew(at)dunslane(dot)net>:

>
> On 09/23/2014 12:23 PM, Dmitry Dolgov wrote:
>
>> Hi all,
>>
>> I'm faced with some troubles about the jsonb implementation, and I hope
>> I'll get little advice =)
>> If I understand correctly, an abstract function for jsonb modification
>> should have the following stages:
>>
>> Jsonb -> JsonbValue -> Modification -> JsonbValue -> Jsonb
>>
>> One can convert the *JsonbValue* to the *Jsonb* only by
>> *JsonbValueToJsonb* function. So, my question is can be *JsonbValue*, that
>> contains few *jbvBinary* elements, converted to *Jsonb* by this function?
>> It will be very useful, if you want modify only small part of your
>> JsonbValue (e.g. replace value of some key). But when I'm trying to do
>> this, an exception "unknown type of jsonb container" appears. Maybe I
>> missed something? Or is there another approach to do this conversion?
>>
>
>
> If you can come up with a way of handling the jbvBinary values then by all
> means send a patch.
>
> But this problem is fairly easily worked around by using an iterator over
> the binary value. The attached patch, which is work in progress for adding
> in the currently missing json functions for jsonb, contains a sort of
> example of doing this in jsonb_agg_transfn.
>
> cheers
>
> andrew
>
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>
>

On 10/13/2014 06:41 AM, Pavel Stehule wrote:
> Hi
>
> I am working on review of this patch.

The patch attached to the message you are replying to was never intended
to be reviewed. It was only given by way of illustration of a technique.

The original patch to be reviewed is on the message
<http://www.postgresql.org/message-id/5425D277.4030804@dunslane.net> as
shown on the commitfest app. I have just submitted a revised patch to
fix the compiler warnings you complained of, at
<http://www.postgresql.org/message-id/543BD598.4020809@dunslane.net> I
have not found any segfaults in the regression tests.

And please don't top-post on the PostgreSQL lists.

cheers

andrew

2014-10-13 15:45 GMT+02:00 Andrew Dunstan <andrew(at)dunslane(dot)net>:

>
> On 10/13/2014 06:41 AM, Pavel Stehule wrote:
>
>> Hi
>>
>> I am working on review of this patch.
>>
>
>
> The patch attached to the message you are replying to was never intended
> to be reviewed. It was only given by way of illustration of a technique.
>
> The original patch to be reviewed is on the message <
> http://www.postgresql.org/message-id/5425D277.4030804@dunslane.net> as
> shown on the commitfest app. I have just submitted a revised patch to fix
> the compiler warnings you complained of, at <http://www.postgresql.org/
> message-id/543BD598(dot)4020809(at)dunslane(dot)net> I have not found any segfaults
> in the regression tests.
>

I checked this last version - warning is out, but SIGFAULT on jsonb test is
there .. I rechecked it with clang compiler, but result is same

I try to search why

>
> And please don't top-post on the PostgreSQL lists.
>

I am sorry

Regards

Pavel

>
> cheers
>
> andrew
>
>
>

Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com> writes:
> I checked this last version - warning is out, but SIGFAULT on jsonb test is
> there .. I rechecked it with clang compiler, but result is same

Stack trace please?

regards, tom lane

2014-10-13 16:19 GMT+02:00 Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>:

> Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com> writes:
> > I checked this last version - warning is out, but SIGFAULT on jsonb test
> is
> > there .. I rechecked it with clang compiler, but result is same
>
> Stack trace please?
>

(gdb) bt
#0 0x0000000000000072 in ?? ()
#1 0x000000000087d598 in parse_array_element (lex=0x2880118,
sem=0x7fffb4f02508) at json.c:461
#2 0x0000000000878da7 in parse_array (lex=0x2880118, sem=0x7fffb4f02508)
at json.c:505
#3 0x000000000087d837 in parse_object_field (lex=0x2880118,
sem=0x7fffb4f02508) at json.c:391
#4 0x0000000000878cb2 in parse_object (lex=0x2880118, sem=0x7fffb4f02508)
at json.c:432
#5 0x000000000087831c in pg_parse_json (lex=0x2880118, sem=0x7fffb4f02508)
at json.c:297
#6 0x000000000087f484 in datum_to_jsonb (val=42202912, is_null=0 '\000',
result=0x7fffb4f02800,
tcategory=JSONBTYPE_JSON, outfuncoid=322, key_scalar=0 '\000') at
jsonb.c:789
#7 0x000000000087fce7 in add_jsonb (val=42202912, is_null=0 '\000',
result=0x7fffb4f02800, val_type=114,
key_scalar=0 '\000') at jsonb.c:1050
#8 0x000000000087fbcc in jsonb_build_object (fcinfo=0x287e2c0) at
jsonb.c:1155
#9 0x000000000066d179 in ExecMakeFunctionResultNoSets (fcache=0x287e250,
econtext=0x287e060, isNull=0x287eca8 "",
isDone=0x287edc0) at execQual.c:1992
#10 0x000000000066776f in ExecEvalFunc (fcache=0x287e250,
econtext=0x287e060, isNull=0x287eca8 "", isDone=0x287edc0)
at execQual.c:2383
#11 0x000000000066c3bb in ExecTargetList (targetlist=0x287ed90,
econtext=0x287e060, values=0x287ec90,
isnull=0x287eca8 "", itemIsDone=0x287edc0, isDone=0x7fffb4f02aac) at
execQual.c:5265
#12 0x000000000066c2c2 in ExecProject (projInfo=0x287ecc0,
isDone=0x7fffb4f02aac) at execQual.c:5480
#13 0x0000000000689ceb in ExecResult (node=0x287df50) at nodeResult.c:155
#14 0x0000000000661987 in ExecProcNode (node=0x287df50) at
execProcnode.c:373
#15 0x000000000065dd46 in ExecutePlan (estate=0x287de40,
planstate=0x287df50, operation=CMD_SELECT,
sendTuples=1 '\001', numberTuples=0, direction=ForwardScanDirection,
dest=0x283fa00) at execMain.c:1481
#16 0x000000000065dc70 in standard_ExecutorRun (queryDesc=0x2809d50,
direction=ForwardScanDirection, count=0)
at execMain.c:308
#17 0x000000000065db3f in ExecutorRun (queryDesc=0x2809d50,
direction=ForwardScanDirection, count=0)
at execMain.c:256
#18 0x00000000007ec70c in PortalRunSelect (portal=0x2807bc0, forward=1
'\001', count=0, dest=0x283fa00)
at pquery.c:946
#19 0x00000000007ec229 in PortalRun (portal=0x2807bc0,
count=9223372036854775807, isTopLevel=1 '\001',
dest=0x283fa00, altdest=0x283fa00, completionTag=0x7fffb4f02ec0 "") at
pquery.c:790
#20 0x00000000007e7f7c in exec_simple_query (
query_string=0x283e1a0 "SELECT jsonb_build_object('e',json '{\"x\": 3,
\"y\": [1,2,3]}');") at postgres.c:1045
#21 0x00000000007e72cb in PostgresMain (argc=1, argv=0x27e5838,
dbname=0x27e56e8 "postgres",
---Type <return> to continue, or q <return> to quit---q
username=0x27e56d0 "paveQuit

Regards

Pavel

>
> regards, tom lane
>

Hi

A JsonSemAction sem is not well initialized

a array_element_start is not initialized and enforces sigfault on my comp

*** ./utils/adt/jsonb.c.orig 2014-10-13 16:37:00.479708142 +0200
--- ./utils/adt/jsonb.c 2014-10-13 16:36:33.704650644 +0200
***************
*** 786,791 ****
--- 786,793 ----
sem.scalar = jsonb_in_scalar;
sem.object_field_start = jsonb_in_object_field_start;

+ sem.array_element_start = NULL;
+
pg_parse_json(lex, &sem);

}

I am not sure, if this fix is valid, but all tests are passed now

Regards

Pavel

2014-10-13 16:21 GMT+02:00 Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>:

>
>
> 2014-10-13 16:19 GMT+02:00 Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>:
>
>> Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com> writes:
>> > I checked this last version - warning is out, but SIGFAULT on jsonb
>> test is
>> > there .. I rechecked it with clang compiler, but result is same
>>
>> Stack trace please?
>>
>
> (gdb) bt
> #0 0x0000000000000072 in ?? ()
> #1 0x000000000087d598 in parse_array_element (lex=0x2880118,
> sem=0x7fffb4f02508) at json.c:461
> #2 0x0000000000878da7 in parse_array (lex=0x2880118, sem=0x7fffb4f02508)
> at json.c:505
> #3 0x000000000087d837 in parse_object_field (lex=0x2880118,
> sem=0x7fffb4f02508) at json.c:391
> #4 0x0000000000878cb2 in parse_object (lex=0x2880118, sem=0x7fffb4f02508)
> at json.c:432
> #5 0x000000000087831c in pg_parse_json (lex=0x2880118,
> sem=0x7fffb4f02508) at json.c:297
> #6 0x000000000087f484 in datum_to_jsonb (val=42202912, is_null=0 '\000',
> result=0x7fffb4f02800,
> tcategory=JSONBTYPE_JSON, outfuncoid=322, key_scalar=0 '\000') at
> jsonb.c:789
> #7 0x000000000087fce7 in add_jsonb (val=42202912, is_null=0 '\000',
> result=0x7fffb4f02800, val_type=114,
> key_scalar=0 '\000') at jsonb.c:1050
> #8 0x000000000087fbcc in jsonb_build_object (fcinfo=0x287e2c0) at
> jsonb.c:1155
> #9 0x000000000066d179 in ExecMakeFunctionResultNoSets (fcache=0x287e250,
> econtext=0x287e060, isNull=0x287eca8 "",
> isDone=0x287edc0) at execQual.c:1992
> #10 0x000000000066776f in ExecEvalFunc (fcache=0x287e250,
> econtext=0x287e060, isNull=0x287eca8 "", isDone=0x287edc0)
> at execQual.c:2383
> #11 0x000000000066c3bb in ExecTargetList (targetlist=0x287ed90,
> econtext=0x287e060, values=0x287ec90,
> isnull=0x287eca8 "", itemIsDone=0x287edc0, isDone=0x7fffb4f02aac) at
> execQual.c:5265
> #12 0x000000000066c2c2 in ExecProject (projInfo=0x287ecc0,
> isDone=0x7fffb4f02aac) at execQual.c:5480
> #13 0x0000000000689ceb in ExecResult (node=0x287df50) at nodeResult.c:155
> #14 0x0000000000661987 in ExecProcNode (node=0x287df50) at
> execProcnode.c:373
> #15 0x000000000065dd46 in ExecutePlan (estate=0x287de40,
> planstate=0x287df50, operation=CMD_SELECT,
> sendTuples=1 '\001', numberTuples=0, direction=ForwardScanDirection,
> dest=0x283fa00) at execMain.c:1481
> #16 0x000000000065dc70 in standard_ExecutorRun (queryDesc=0x2809d50,
> direction=ForwardScanDirection, count=0)
> at execMain.c:308
> #17 0x000000000065db3f in ExecutorRun (queryDesc=0x2809d50,
> direction=ForwardScanDirection, count=0)
> at execMain.c:256
> #18 0x00000000007ec70c in PortalRunSelect (portal=0x2807bc0, forward=1
> '\001', count=0, dest=0x283fa00)
> at pquery.c:946
> #19 0x00000000007ec229 in PortalRun (portal=0x2807bc0,
> count=9223372036854775807, isTopLevel=1 '\001',
> dest=0x283fa00, altdest=0x283fa00, completionTag=0x7fffb4f02ec0 "") at
> pquery.c:790
> #20 0x00000000007e7f7c in exec_simple_query (
> query_string=0x283e1a0 "SELECT jsonb_build_object('e',json '{\"x\": 3,
> \"y\": [1,2,3]}');") at postgres.c:1045
> #21 0x00000000007e72cb in PostgresMain (argc=1, argv=0x27e5838,
> dbname=0x27e56e8 "postgres",
> ---Type <return> to continue, or q <return> to quit---q
> username=0x27e56d0 "paveQuit
>
> Regards
>
> Pavel
>
>
>>
>> regards, tom lane
>>
>
>

2014-10-13 15:45 GMT+02:00 Andrew Dunstan <andrew(at)dunslane(dot)net>:

>
> On 10/13/2014 06:41 AM, Pavel Stehule wrote:
>
>> Hi
>>
>> I am working on review of this patch.
>>
>
>
> The patch attached to the message you are replying to was never intended
> to be reviewed. It was only given by way of illustration of a technique.
>
> The original patch to be reviewed is on the message <
> http://www.postgresql.org/message-id/5425D277.4030804@dunslane.net> as
> shown on the commitfest app. I have just submitted a revised patch to fix
> the compiler warnings you complained of, at <http://www.postgresql.org/
> message-id/543BD598(dot)4020809(at)dunslane(dot)net> I have not found any segfaults
> in the regression tests.
>
> And please don't top-post on the PostgreSQL lists.
>

Attached small fix of uninitialized local variable

Regards

Pavel

>
> cheers
>
> andrew
>
>
>

On 10/13/2014 10:39 AM, Pavel Stehule wrote:
> Hi
>
> A JsonSemAction sem is not well initialized
>
> a array_element_start is not initialized and enforces sigfault on my comp
>
> *** ./utils/adt/jsonb.c.orig 2014-10-13 16:37:00.479708142 +0200
> --- ./utils/adt/jsonb.c 2014-10-13 16:36:33.704650644 +0200
> ***************
> *** 786,791 ****
> --- 786,793 ----
> sem.scalar = jsonb_in_scalar;
> sem.object_field_start =
> jsonb_in_object_field_start;
>
> + sem.array_element_start = NULL;
> +
> pg_parse_json(lex, &sem);
>
> }
>
> I am not sure, if this fix is valid, but all tests are passed now
>
>

Good find. I think what we should probably do is initialize the whole
thing with:

memset(&sem, 0, sizeof(JsonSemAction));

before assigning anything to its fields. That would be consistent with
what we do elsewhere.

I'll make that change and submit a new patch.

Please stop using this thread, however. It's inappropriate for reviewing
this patch.

cheers

andrew