Re: [REVIEW] Re: Compression of full-page-writes

On 2014-09-13 08:52:33 +0300, Ants Aasma wrote:
> On Sat, Sep 13, 2014 at 6:59 AM, Arthur Silva <arthurprs(at)gmail(dot)com> wrote:
> > That's not entirely true. CRC-32C beats pretty much everything with the same
> > length quality-wise and has both hardware implementations and highly
> > optimized software versions.
>
> For better or for worse CRC is biased by detecting all single bit
> errors, the detection capability of larger errors is slightly
> diminished. The quality of the other algorithms I mentioned is also
> very good, while producing uniformly varying output.

There's also much more literature about the various CRCs in comparison
to some of these hash allgorithms. Pretty much everything tests how well
they're suited for hashtables, but that's not really what we need
(although it might not hurt *at all* to have something faster there...).

I do think we need to think about the types of errors we really have to
detect. It's not all that clear that either the typical guarantees/tests
for CRCs nor for checksums (smhasher, whatever) are very
representative...

> CRC has exactly
> one hardware implementation in general purpose CPU's and Intel has a
> patent on the techniques they used to implement it. The fact that AMD
> hasn't yet implemented this instruction shows that this patent is
> non-trivial to work around.

I think AMD has implemeded SSE4.2 with bulldozer. It's still only recent
x86 though. So I think there's good reasons for moving away from it.

How one could get patents on exposing hardware CRC implementations -
hard to find a computing device without one - as a instruction is beyond
me...

I think it's pretty clear by now that we should move to lz4 for a couple
things - which bundles xxhash with it. So that has one argument for it.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services