Re: [REVIEW] Re: Compression of full-page-writes

On Tue, Aug 5, 2014 at 6:25 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:

>
>
> This gradual approach looks good to me. And, if the additional compression
> algorithm like lz4 is always better than pglz for every scenarios, we can
> just
> change the code so that the additional algorithm is always used. Which
> would
> make the code simpler.
>
>
Right.

> > 3. Compressing one block vs all blocks:
> > Andres suggested that compressing all backup blocks in one go may give us
> > better compression ratio. This is worth trying. I'm wondering what would
> the
> > best way to do so without minimal changes to the xlog insertion code.
> Today,
> > we add more rdata items for backup block header(s) and backup blocks
> > themselves (if there is a "hole" then 2 per backup block) beyond what the
> > caller has supplied. If we have to compress all the backup blocks
> together,
> > then one approach is to copy the backup block headers and the blocks to a
> > temp buffer, compress that and replace the rdata entries added previously
> > with a single rdata.
>
> Basically sounds reasonable. But, how does this logic work if there are
> multiple rdata and only some of them are backup blocks?
>
>
My idea is to just make a pass over the rdata entries past the
rdt_lastnormal element after processing the backup blocks and making
additional entries in the chain. These additional rdata entries correspond
to the backup blocks and their headers. So we can copy the rdata->data of
these elements in a temp buffer and compress the entire thing in one go. We
can then replace the rdata chain past the rdt_lastnormal with a single
rdata with data pointing to the compressed data. Recovery code just needs
to decompress this data the record header indicates that the backup data is
compressed. Sure the exact mechanism to indicate if the data is compressed
(and by which algorithm) can be worked out.

> If a "hole" is not copied to that temp buffer, ISTM that we should
> change backup block header so that it contains the info for a
> "hole", e.g., location that a "hole" starts. No?
>
>
AFAICS its not required if we compress the stream of BkpBlock and the block
data. The current mechanism of constructing the additional rdata chain
items takes care of hole anyways.

Thanks,
Pavan

--
Pavan Deolasee
http://www.linkedin.com/in/pavandeolasee