Re: Bit data type header reduction in some cases

Hi,

It's regarding a Todo item of "Bit data type header reduction" in some
cases. The header contains two parts. 1) The varlena header is
automatically converted to 1 byte header from 4 bytes in case of small
data. 2) The bit length header called "bit_len" to store the actual bit
length which is of 4 bytes in size. I want to reduce this bit_len size to 1
byte in some cases as similar to varlena header. With this change the size
of the column reduced by 3 bytes, thus shows very good decrease in disk
usage.

I am planning to the change it based on the total length of bits data. If
the number of bits is less than 0x7F then one byte header can be chosen
instead of 4 byte header. During reading of the data, the similar way it
can be calculated.

The problem I am thinking is, as it can cause problems to old databases
having bit columns when they
upgrade to a newer version without using pg_dumpall. Is there anyway to
handle this problem? Or Is there any better way to handle the problem
itself? please let me know your suggestions.

Regards,
Hari Babu
Fujitsu Australia

On 02/25/2014 08:23 AM, Haribabu Kommi wrote:
> It's regarding a Todo item of "Bit data type header reduction" in some
> cases. The header contains two parts. 1) The varlena header is
> automatically converted to 1 byte header from 4 bytes in case of small
> data. 2) The bit length header called "bit_len" to store the actual bit
> length which is of 4 bytes in size. I want to reduce this bit_len size to 1
> byte in some cases as similar to varlena header. With this change the size
> of the column reduced by 3 bytes, thus shows very good decrease in disk
> usage.
>
> I am planning to the change it based on the total length of bits data. If
> the number of bits is less than 0x7F then one byte header can be chosen
> instead of 4 byte header. During reading of the data, the similar way it
> can be calculated.

Since we're designing a new format, how about using bit padding instead
of an explicit length field? Add one 1-bit to the end of the bit data,
followed by zeros. That's even more compact. See
https://en.wikipedia.org/wiki/Padding_%28cryptography%29#Bit_padding

> The problem I am thinking is, as it can cause problems to old databases
> having bit columns when they
> upgrade to a newer version without using pg_dumpall. Is there anyway to
> handle this problem? Or Is there any better way to handle the problem
> itself? please let me know your suggestions.

On a little-endian system, you could easily use the most-significant bit
(sign bit) of the bit_len field to indicate that it's a new-format datum:

/*
* Little-endian
*/
typedef struct
{
int32 vl_len_;
int32 bit_len;
bits8 bit_dat[1];
} VarBit_Old;

typedef struct
{
int32 vl_len_;
uint8 bit_len;
bits8 bit_dat[1]; /* var len */
} VarBit_New;

#define IS_NEW_FORMAT(((VarBit_New *) x)->bit_len & 0x80)

On a big-endian system that's more difficult, because the MSB would not
be the first byte of the field. You could still make it work if the new
format would have the length byte in fourth byte of the Datum. Something
like this:

/*
* Big-endian
*/
typedef struct
{
int32 vl_len_;
int32 bit_len;
bits8 bit_dat[1];
} VarBit_Old;

typedef struct
{
int32 vl_len_;
bits8 bit_dat_first_three[3];
uint8 bit_len;
bits8 bit_dat_rest[1]; /* var len */
} VarBit_New;

#define IS_NEW_FORMAT(((VarBit_New *) x)->bit_len & 0x80)

That's not a complete solution yet, you also need a special case for a
new-format value with less than 4 bytes of bits. Such a field would
presumably not have the bit_len field at the fourth byte, because it
would be shorter than that, but you could distinguish it by looking at
the length in the varsize header. So in big-endian, you'd have one more
format for very small varbits:

/* Big-endian, less than 4 bytes of bits data */
typedef struct
{
int32 vl_len_;
uint8 bit_len;
bits8 bit_dat[1]; /* var len */
} VarBit_New_Tiny;

The same basic idea also works with bit padding.

- Heikki

Heikki Linnakangas <hlinnakangas(at)vmware(dot)com> writes:
> On 02/25/2014 08:23 AM, Haribabu Kommi wrote:
>> It's regarding a Todo item of "Bit data type header reduction" in some
>> cases. The header contains two parts. 1) The varlena header is
>> automatically converted to 1 byte header from 4 bytes in case of small
>> data. 2) The bit length header called "bit_len" to store the actual bit
>> length which is of 4 bytes in size. I want to reduce this bit_len size to 1
>> byte in some cases as similar to varlena header. With this change the size
>> of the column reduced by 3 bytes, thus shows very good decrease in disk
>> usage.

> [ various contorted schemes for doing that backward-compatibly ]

TBH, this sounds like a huge amount of effort and a significant risk of
new bugs in return for not darn much. Who uses bit values anyway?
They were removed from the SQL spec more than 10 years ago. And of that
population, who cares about a byte or two per value? The field demand
for this is not only zero, it's probably negative. (The thread referenced
by the TODO entry shows no evidence of user demand.)

regards, tom lane

On Tue, Feb 25, 2014 at 09:32:55AM -0500, Tom Lane wrote:
> Heikki Linnakangas <hlinnakangas(at)vmware(dot)com> writes:
> > On 02/25/2014 08:23 AM, Haribabu Kommi wrote:
> >> It's regarding a Todo item of "Bit data type header reduction" in some
> >> cases. The header contains two parts. 1) The varlena header is
> >> automatically converted to 1 byte header from 4 bytes in case of small
> >> data. 2) The bit length header called "bit_len" to store the actual bit
> >> length which is of 4 bytes in size. I want to reduce this bit_len size to 1
> >> byte in some cases as similar to varlena header. With this change the size
> >> of the column reduced by 3 bytes, thus shows very good decrease in disk
> >> usage.
>
> > [ various contorted schemes for doing that backward-compatibly ]
>
> TBH, this sounds like a huge amount of effort and a significant risk of
> new bugs in return for not darn much. Who uses bit values anyway?
> They were removed from the SQL spec more than 10 years ago. And of that
> population, who cares about a byte or two per value? The field demand
> for this is not only zero, it's probably negative. (The thread referenced
> by the TODO entry shows no evidence of user demand.)

OK, TODO item removed.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +