Re: charset/collation in values

I've looked into storing charset/collation in the string values. This
means that we change varchar/text/BpChar to be structures that have a
charset oid field and a collation oid field, the rest of the Datum is the
string data.

Coercability I think one don't need to put in the Datum and it can be
stored in the Nodes. Charset/Collation need to be in the Datum since we
send that into functions as arguments.

Since we are changing what's stored in the Datum and the normal code saves
that on disk then we will end up with charset/collation stored on disk for
each value. If we want to avoid storing charset/collation both in the
column type and in each row, we would need an extra layer that transforms
the Datums before they are stored. As a first implementation it's easier
to just store everything.

For each type we need to have convertion functions to and from strings.
Any suggestion of how to represent these as strings now when it's a string
plus two oid's? This is a though one..

I have more comments/questions later on, but these are enough for one
mail.

--
/Dennis Björklund

Dennis Bjorklund wrote:
> I've looked into storing charset/collation in the string values. This
> means that we change varchar/text/BpChar to be structures that have a
> charset oid field and a collation oid field, the rest of the Datum is the
> string data.
>
I think the number of charset/collation combinations will be relatively
few so perhaps it would be space efficient to maintain a table where
each combination is given an oid and have string values store that
rather than two separate oid's?

Regards,
Thomas Hallgren

Thomas Hallgren <thhal(at)mailblocks(dot)com> writes:
> I think the number of charset/collation combinations will be relatively
> few so perhaps it would be space efficient to maintain a table where
> each combination is given an oid and have string values store that
> rather than two separate oid's?

In fact, we should do our best to get the overhead down to 1 or 2 bytes.
Two OIDs (8 bytes) is ridiculous.

I'm not sure if 1 byte is enough or not --- there might be more than 256
charsets/collations to support. 2 ought to be plenty though.

regards, tom lane

On Mon, 1 Nov 2004, Tom Lane wrote:

> > I think the number of charset/collation combinations will be relatively
> > few so perhaps it would be space efficient to maintain a table where
> > each combination is given an oid and have string values store that
> > rather than two separate oid's?
>
> In fact, we should do our best to get the overhead down to 1 or 2 bytes.
> Two OIDs (8 bytes) is ridiculous.

Just to be clear, we don't want to store it on disk no matter what since
it should be enough to store it once for each column. As a first solution
we could store it just to keep it simple until we have tried it out.

--
/Dennis Björklund

> On Mon, 1 Nov 2004, Tom Lane wrote:
>
> > > I think the number of charset/collation combinations will be relatively
> > > few so perhaps it would be space efficient to maintain a table where
> > > each combination is given an oid and have string values store that
> > > rather than two separate oid's?
> >
> > In fact, we should do our best to get the overhead down to 1 or 2 bytes.
> > Two OIDs (8 bytes) is ridiculous.
>
> Just to be clear, we don't want to store it on disk no matter what since
> it should be enough to store it once for each column. As a first solution
> we could store it just to keep it simple until we have tried it out.

Right. AFAIK nobody has proposed charsets/collations onto disk.
--
Tatsuo Ishii

Tatsuo Ishii wrote:
> Right. AFAIK nobody has proposed charsets/collations onto disk.
> --
My apologies in that case. I triggered on Dennis wording "If we want to
avoid storing charset/collation both in the column type and in each row,
we would need an extra layer that transforms the Datums before they are
stored. As a first implementation it's easier to just store everything."

Regards,
Thomas Hallgren

Tatsuo Ishii <t-ishii(at)sra(dot)co(dot)jp> writes:
> Right. AFAIK nobody has proposed charsets/collations onto disk.

Oh?

Personally, I'd much sooner eat those few bytes than try to impose a
regime where in-memory representation is different from on-disk.

regards, tom lane

Am Montag, 1. November 2004 07:41 schrieb Dennis Bjorklund:
> For each type we need to have convertion functions to and from strings.
> Any suggestion of how to represent these as strings now when it's a string
> plus two oid's? This is a though one..

A collation implies a character set, so you only need to store one piece of
information anyway.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

On Tue, 2 Nov 2004, Peter Eisentraut wrote:

> A collation implies a character set, so you only need to store one piece of
> information anyway.

No, a collation implies a character repertoire like UCS (unicode), it can
apply to several character sets like UTF8 and UTF16.

One can enumerate all combinations if one want to, as suggested
previously.

--
/Dennis Björklund

> Am Montag, 1. November 2004 07:41 schrieb Dennis Bjorklund:
> > For each type we need to have convertion functions to and from strings.
> > Any suggestion of how to represent these as strings now when it's a string
> > plus two oid's? This is a though one..
>
> A collation implies a character set, so you only need to store one piece of
> information anyway.

In my understanding the relation between charset and collation is
1:N. Thus storing only a collation is sufficient to determine the
charset. However a charset cannot determine a collation.
--
Tatsuo Ishii

Am Dienstag, 2. November 2004 13:15 schrieb Dennis Bjorklund:
> On Tue, 2 Nov 2004, Peter Eisentraut wrote:
> > A collation implies a character set, so you only need to store one piece
> > of information anyway.
>
> No, a collation implies a character repertoire like UCS (unicode), it can
> apply to several character sets like UTF8 and UTF16.

For the theoretical specification of a collation, it might suffice to know the
character repertoire. But I think in practice, the implementation of a
collation will require knowing the specific character encoding.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Am Dienstag, 2. November 2004 13:53 schrieb Tatsuo Ishii:
> In my understanding the relation between charset and collation is
> 1:N. Thus storing only a collation is sufficient to determine the
> charset. However a charset cannot determine a collation.

Exactly.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

On Tue, 2 Nov 2004, Peter Eisentraut wrote:

> For the theoretical specification of a collation, it might suffice to
> know the character repertoire. But I think in practice, the
> implementation of a collation will require knowing the specific
> character encoding.

The named entity that is called a collation works for a character
repertoire. It would need to handle different charsets for that repertoire
of course. So there would be one collation called say ucs_sv and not
utf8_sv, utf16_sv, utf32_sv.

Anyway, this is not a problem.

--
/Dennis Björklund

Dennis Bjorklund wrote:
> The named entity that is called a collation works for a character
> repertoire. It would need to handle different charsets for that
> repertoire of course. So there would be one collation called say
> ucs_sv and not utf8_sv, utf16_sv, utf32_sv.

Again, theoretically, this might work, but I doubt that this is a
practical implementation. Moreover, since Unicode is more or less the
only chararacter repertoire that have more than one encoding in use,
and neither UTF-16 nor UTF-32 can be used inside the PostgreSQL server
(embedded zero bytes etc.), this is really a nonissue.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

> Dennis Bjorklund wrote:
> > The named entity that is called a collation works for a character
> > repertoire. It would need to handle different charsets for that
> > repertoire of course. So there would be one collation called say
> > ucs_sv and not utf8_sv, utf16_sv, utf32_sv.
>
> Again, theoretically, this might work, but I doubt that this is a
> practical implementation. Moreover, since Unicode is more or less the
> only chararacter repertoire that have more than one encoding in use,
> and neither UTF-16 nor UTF-32 can be used inside the PostgreSQL server
> (embedded zero bytes etc.), this is really a nonissue.

I agree with Peter.
--
Tatsuo Ishii