Re: extensible external toast tuple support & snappy prototype

On 2013-06-07 12:16:48 -0400, Stephen Frost wrote:
> * Andres Freund (andres(at)2ndquadrant(dot)com) wrote:
> > Currently on a little endian system the pglz header contains the length
> > in the first four bytes as:
> > [dddddddd][dddddddd][dddddddd][xxdddddd]
> > Where dd are valid length bits for pglz and xx are the two bits which
> > are always zero since we only ever store up to 1GB. We can redefine 'xx'
> > to mean whatever we want but we cannot change it's place.
>
> I'm not thrilled with the idea of using those 2 bits from the length
> integer. I understand the point of it and that we'd be able to have
> binary compatibility from it but is it necessary to track at the
> per-tuple level..? What about possibly supporting >1GB objects at some
> point (yes, I know there's a lot of other issues there, but still).
> We've also got complexity around the size of the length integer already.

I am open for different suggestions, but I don't know of any realistic
ones.
Note that the 1GB limitation is already pretty heavily baked in into
varlenas itself (which is not the length we are talking about here!)
since we use the two remaining bits discern between the 4 types of
varlenas we have.

* short (1b)
* short, pointing to a ondisk tuple (1b_e)
* long (4b)
* long compressed (4b_c)

Since long compressed ones always need to be convertible to long ones we
can't ever have a 'rawsize' (which is what's proposed to be used for
this) that's larger than 1GB.

So, breaking the 1GB limit will not be stopped by this, but much much
earlier. And it will require a break in on-disk compatibility.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services