Re: extensible external toast tuple support & snappy prototype

On 2013-06-07 10:44:24 -0400, Robert Haas wrote:
> On Fri, Jun 7, 2013 at 10:30 AM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> > Turns out the benefits are imo big enough to make it worth pursuing
> > further.
>
> Yeah, those were nifty numbers.
>
> > The problem is that to discern from pglz on little endian the byte with
> > the two high bits unset is actually the fourth byte in a toast datum. So
> > we would need to store it in the 5th byte or invent some more
> > complicated encoding scheme.
> >
> > So I think we should just define '00' as pglz, '01' as xxx, '10' as yyy
> > and '11' as storing the schema in the next byte.
>
> Not totally following, but I'm fine with that.

Currently on a little endian system the pglz header contains the length
in the first four bytes as:
[dddddddd][dddddddd][dddddddd][xxdddddd]
Where dd are valid length bits for pglz and xx are the two bits which
are always zero since we only ever store up to 1GB. We can redefine 'xx'
to mean whatever we want but we cannot change it's place.

> >> > 3) Surely choosing the compression algorithm via GUC ala SET
> >> > toast_compression_algo = ... isn't the way to go. I'd say a storage
> >> > attribute is more appropriate?
> >>
> >> The way we do caching right now supposes that attoptions will be
> >> needed only occasionally. It might need to be revised if we're going
> >> to need it all the time. Or else we might need to use a dedicated
> >> pg_class column.
> >
> > Good point. It probably belongs right besides attstorage, seems to be
> > the most consistent choice anyway.
>
> Possibly, we could even store it in attstorage. We're really only
> using two bits of that byte right now, so just invent some more
> letters.

Hm. Possible, but I don't think that's worth it. There's a padding byte
before attinhcount anyway.
Storing the preferred location in attstorage (plain, preferred-internal,
external, preferred-external) separately from the compression seems to
make sense to me.

> > Alternatively, if we only add one form of compression, we can just
> > always store in snappy/lz4/....
>
> Not following.

I mean, we don't necessarily need to make it configurable if we just add
one canonical new "better" compression format. I am not sure that's
sufficient since I can see usecases for 'very fast but not too well
compressed' and 'very well compressed but slow', but I am personally not
really interested in the second case, so ...

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services