Re: [RFC] indirect toast tuple support

On Sat, Feb 16, 2013 at 11:42 AM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> Given that there have been wishes to support something like b) for quite
> some time, independent from logical decoding, it seems like a good idea
> to add support for it. Its e.g. useful for avoiding repeated detoasting
> or decompression of tuples.
>
> The problem with b) is that there is no space in varlena's flag bits to
> directly denote that a varlena points into memory instead of either
> directly containing the data or a varattrib_1b_e containing a
> varatt_external pointing to an on-disk toasted tuple.

So the other way that we could do this is to use something that's the
same size as a TOAST pointer but has different content - the
seemingly-obvious choice being va_toastrelid == 0. I'd be a little
reluctant to do it the way you propose because we might, at some
point, want to try to reduce the size of toast pointers. If you have
a tuple with many attributes, the size of the TOAST pointers
themselves starts to add up. It would be nice to be able to have 8
byte or even 4 byte toast pointers to handle those situations. If we
steal one or both of those lengths to mean "the data is cached in
memory somewhere" then we can't use those lengths in a smaller on-disk
representation, which would seem a shame.

But having said that, +1 on the general idea of getting something like
this done. We really need a better infrastructure to avoid copying
large values around repeatedly in memory - a gigabyte is a lot of data
to be slinging around.

Of course, you will not be surprised to hear that I think this is 9.4 material.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company