Re: Variable length varlena headers redux

"Tom Lane" <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:

> Greg Stark <gsstark(at)mit(dot)edu> writes:
>> Bruce Momjian <bruce(at)momjian(dot)us> writes:
>>> I know it is kind of odd to have a data type that is only used on disk,
>>> and not in memory, but I see this as a baby varlena type, used only to
>>> store and get varlena values using less disk space.
>
>> I was leaning toward generating the short varlena headers primarily in
>> heap_form*tuple and just having the datatype specific code generate 4-byte
>> headers much as you describe.
>
> I thought we had a solution for all this, namely to make the short-form
> headers be essentially a TOAST-compressed representation. The format
> with 4-byte headers is still legal but just not compressed. Anyone who
> fails to detoast an input argument is already broken, so there's no code
> compatibility hit taken.

It's not just input arguments though. A function could call
DirectFunctionCall* and rightfully expect the return value not to need
detoasting.

I suppose this leads me to *only* generate short headers at heap_form*tuple
time. Then DirectFunctionCall isn't relevant and most of the user code is
perfectly safe.

There could still be cases where a heaptuple is passed around in pl_exec.c or
somewhere but if it's subsequently deformed whoever looks at it hopefully
wouldn't be too surprised for it to be mandatory that they go through
pg_detoast_datum. It'll happen as long as they use the DatumGetFoo macros
anyways.

It does mean that anyone going through a heap_form*tuple/heap_deform*tuple
cycle may generate more copies and memory allocations than they expected.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com