Re: Cost of XLogInsert CRC calculations

On Fri, 2005-03-11 at 19:31 +0100, Hans-Jürgen Schönig wrote:
> > One of the things I was thinking about was whether we could use up those
> > cycles more effectively. If we were to include a compression routine
> > before we calculated the CRC that would
> > - reduce the size of the blocks to be written, hence reduce size of xlog
> > - reduce the following CRC calculation
> >
> > I was thinking about using a simple run-length encoding to massively
> > shrink half-empty blocks with lots of zero padding, but we've already
> > got code to LZW the data down also.

> I think having a compression routine in there could make real sense.
> We have done some major I/O testing involving compression for a large
> customer some time ago. We have seen that compressing / decompressing on
> the fly is in MOST cases much faster than uncompressed I/O (try a simple
> "cat file | ..." vs." zcat file.gz | ...") - the zcat version will be
> faster on all platforms we have tried (Linux, AIX, Sun on some SAN
> system, etc. ...).
> Also, when building up a large database within one transaction the xlog
> will eat a lot of storage - this can be quite annoying when you have to
> deal with a lot of data).

Agreed.

> Are there any technical reasons which would prevent somebody from
> implementing compression?

Not currently, that I'm aware of. But the way additional blocks are
added to xlog records would need to be changed to allow for variable
length output.

It's on my list...

Best Regards, Simon Riggs