Re: thousands comma numeric formatting in psql

Hello,

This is my first post to this list.
Sorry if my english it's not so good. It's not my native language.

I'm interrested to now if someone think that having a feature like
'thousands comma delimited numeric formatting' in psql it's a
usefull thing.

I've made a patch for psql that adds this feature, so issuing a
select like this:

my_database=> select 1234567.89;

results in:

?column?
--------------
1,234,567.89

This feature is toggle on/off with a backslash command ('\n'):

my_database=> \n
Numeric formatting is off.

my_database=> select 1234567.89;

?column?
--------------
1234567.89

For me, psql it's still the best client for postgres, faster and
flexible than graphic ones. And having tables with many numeric
columns witch contain huge numbers make difficult to read this
numbers.

One solution to deal with this is to use to_char function, but for
complex selects against multiple tables it's not a good option.

Another one is to make a custom function that works like this:

select my_function(... complex subselect ...);

but this seems ugly to me.

By adding the '\n' switch to psql I can make any complex select and
have all numeric fields in result formatted in easy readable form.

I'm not an expert in postgresql, so if someone thinks there is an
easier way to deal with numeric fields, please share.

If my idea is considered usefull I can post the patch to this list.

Best regards,
Eugen.

On Tue, Jun 21, 2005 at 08:42:16 +0300,
Eugen Nedelcu <eugen(at)sifolt(dot)ro> wrote:
>
> One solution to deal with this is to use to_char function, but for
> complex selects against multiple tables it's not a good option.

Why not? You only have to apply it to the output expressions that
need it.

Note that if you output numbers like this, you also need to be able to
read them back in. I don't think adding complexity for doing that is
worth not having to add a few to_char calls in your select queries.

On Tue, Jun 21, 2005 at 06:59:38AM -0500, Bruno Wolff III wrote:
> On Tue, Jun 21, 2005 at 08:42:16 +0300,
> Eugen Nedelcu <eugen(at)sifolt(dot)ro> wrote:
> >
> > One solution to deal with this is to use to_char function, but for
> > complex selects against multiple tables it's not a good option.
>
> Why not? You only have to apply it to the output expressions that
> need it.

I think this:

select * from table_with_text_and_numeric_fields;

is much,much easier than:

select text_field1,text_field2,to_char(numeric_field1, '99G999G999'),
to_char(numeric_field2, '9G999G999G999'), text_field3,
text_field4, to_char(numeric_field3, 'MI90G999D99') from
table_with_text_and_numeric_fields;

>
> Note that if you output numbers like this, you also need to be able to
> read them back in. I don't think adding complexity for doing that is
> worth not having to add a few to_char calls in your select queries.
>

I don't know what 'read them back in' means to you.
This formatting is only done when the number is output to the screen.

Something like:
fputs(thousands_comma_number, fout)
instead of:
fputs(original_number, fout)

If you want to output to some file for reading back later, you could
turn the feature off with the backslash switch '\n'.

This is a patch for psql client and not for the backend. It's role
is to output numbers to screen in easy readable form (2,345,675,454,543
is much easier to read then 2345675454543.456). I think graphical
clients like pgAdmin or phppgadmin have a way to do this. I don't know
this for sure but I will investigate it.

Best Regards,
Eugen

On Tue, Jun 21, 2005 at 04:03:43PM +0300, Eugen Nedelcu wrote:

> This is a patch for psql client and not for the backend. It's role
> is to output numbers to screen in easy readable form (2,345,675,454,543
> is much easier to read then 2345675454543.456). I think graphical
> clients like pgAdmin or phppgadmin have a way to do this. I don't know
> this for sure but I will investigate it.

I think it would be much nicer if it was a backend setting. If it
depended on the locale setting (LC_NUMERIC, I think) and worked for both
input and output, it would be very good.

--
Alvaro Herrera (<alvherre[a]surnet.cl>)
"Uno puede defenderse de los ataques; contra los elogios se esta indefenso"

On Tue, Jun 21, 2005 at 16:03:43 +0300,
Eugen Nedelcu <eugen(at)sifolt(dot)ro> wrote:
>
> This is a patch for psql client and not for the backend. It's role
> is to output numbers to screen in easy readable form (2,345,675,454,543
> is much easier to read then 2345675454543.456). I think graphical
> clients like pgAdmin or phppgadmin have a way to do this. I don't know
> this for sure but I will investigate it.

I did miss that this was a psql only feature. I still don't see a great
need, but it isn't going to cause a problem for copy or pg_dump that way.

Alvaro Herrera <alvherre(at)surnet(dot)cl> writes:
> On Tue, Jun 21, 2005 at 04:03:43PM +0300, Eugen Nedelcu wrote:
>> This is a patch for psql client and not for the backend.

> I think it would be much nicer if it was a backend setting.

Doing this as a backend setting has been proposed and rejected before.
The risk of breaking client code seems to outweigh any possible value.

As a psql setting, though, it seems relatively harmless --- certainly
no more dangerous than \x or the other pset formatting parameters.

> If it depended on the locale setting (LC_NUMERIC, I think)

Yes, I wanted to ask that too --- if the patch can cope with locales that
exchange the roles of comma and period, that would make a lot of people
very happy.

regards, tom lane

Hello,

I have included my patch attached to this mail.

I have made the changes to deal with locale settings from client
environment. So now you can start psql like this:

(export LC_ALL=ro_RO; psql -U user db)

and have numeric formatting with '.' as thousands separator and
',' as decimal point, or

(export LC_ALL=en_US; psql -U user db)

and have numeric formatting with ',' as thousands separator and
'.' as decimal point. This formatting is default when locale is 'C'

You can set any locale and numeric formatting code will take it in
consideration.

This patch is for version 7.3.2. The steps for install is:

1) cp thousands_comma.diff $POSTGRES_DIR/src/bin/psql
2) cd $POSTGRES_DIR/src/bin/psql
3) patch -p0 < thousands_comma.diff
4) ../../../configure && make

Best Regards,
Eugen

Eugen Nedelcu wrote:
> Hello,
>
> I have included my patch attached to this mail.
>
> I have made the changes to deal with locale settings from client
> environment. So now you can start psql like this:
>
> (export LC_ALL=ro_RO; psql -U user db)
>
> and have numeric formatting with '.' as thousands separator and
> ',' as decimal point, or
>
> (export LC_ALL=en_US; psql -U user db)
>
> and have numeric formatting with ',' as thousands separator and
> '.' as decimal point. This formatting is default when locale is 'C'
>
> You can set any locale and numeric formatting code will take it in
> consideration.
>
> This patch is for version 7.3.2. The steps for install is:
>
> 1) cp thousands_comma.diff $POSTGRES_DIR/src/bin/psql
> 2) cd $POSTGRES_DIR/src/bin/psql
> 3) patch -p0 < thousands_comma.diff
> 4) ../../../configure && make

I have heavily modified your patch and have applied it. Instead of
using langinfo, I used a \pset variable numericsep. (We can talk about
adding langinfo detection later.) By default, it is off, ''. If you
set it to '.', the decimal marker will be ','. This also allows
separators like ' ' too so numebers can appear as 100 000.

I have also added documentation.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:
> I have heavily modified your patch and have applied it. Instead of
> using langinfo, I used a \pset variable numericsep.

Why?

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut wrote:
> Bruce Momjian wrote:
> > I have heavily modified your patch and have applied it. Instead of
> > using langinfo, I used a \pset variable numericsep.
>
> Why?

Because I don't have langinfo on my system, so I can't test it, nor add
configure code for it. It also prevents us from using space as the
separator, which is the international standard.

If you think we should use langinfo, we can discuss it, but at this
point, it is not used.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Am Dienstag, 12. Juli 2005 05:29 schrieb Bruce Momjian:
> > Bruce Momjian wrote:
> > > I have heavily modified your patch and have applied it. Instead of
> > > using langinfo, I used a \pset variable numericsep.

> Because I don't have langinfo on my system, so I can't test it, nor add
> configure code for it. It also prevents us from using space as the
> separator, which is the international standard.

OK, so let's discuss what we really want here.

The original submitter wanted locale-dependent output, which seems reasonable,
because that's what locale settings are intended for.

The current state allows users to manually set the format, or actually only
one aspect of the format? Where can you set the decimal separator and the
size of the grouping (3 digits or 4 digits)? Is this capability even useful
to get localized behavior?

My other concern is that if we allow manual specification of the output format
of some data type, then eventually someone will want to specify the format of
some other data type as well, such as the date/time types. We should think
about whether we want to be consistent here.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut wrote:
> Am Dienstag, 12. Juli 2005 05:29 schrieb Bruce Momjian:
> > > Bruce Momjian wrote:
> > > > I have heavily modified your patch and have applied it. Instead of
> > > > using langinfo, I used a \pset variable numericsep.
>
> > Because I don't have langinfo on my system, so I can't test it, nor add
> > configure code for it. It also prevents us from using space as the
> > separator, which is the international standard.
>
> OK, so let's discuss what we really want here.
>
> The original submitter wanted locale-dependent output, which seems reasonable,
> because that's what locale settings are intended for.

True, but there were no locale configure tests, and I don't have
langinfo, so I just made it a \pset 'string' variable. Also, when I
learned that the international system uses a space to separate
trigroups, a separate setting seemed best:

http://en.wikipedia.org/wiki/Comma_%28punctuation%29#Numbers

One thing we could do is to have a special value like 'locale' that uses
langinfo, and all other values are taken literally.

> The current state allows users to manually set the format, or actually only
> one aspect of the format? Where can you set the decimal separator and the
> size of the grouping (3 digits or 4 digits)? Is this capability even useful
> to get localized behavior?

You can't change the grouping from three, and the decimal is only be
changed by setting the numeric separator to a period. We could add a
new \pset for the decimal mark, but then do we auto-set it when the
separator is a period?

> My other concern is that if we allow manual specification of the output format
> of some data type, then eventually someone will want to specify the format of
> some other data type as well, such as the date/time types. We should think
> about whether we want to be consistent here.

We do allow MDY and DMY specification, but that controls both input and
output in the server, while this just controls psql display. It is a
good question how other settings should be handled, but I don't know the
answers. Anyone?

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> We do allow MDY and DMY specification, but that controls both input and
> output in the server, while this just controls psql display. It is a
> good question how other settings should be handled, but I don't know the
> answers. Anyone?

This patch sounds quite a lot different from what I thought we had
agreed to, which was a way to specify that the client-side LC_NUMERIC
locale settings should be used for formatting numbers. I think the only
pset should be "locale on" or "locale not on". This business about
space is a red herring: if you want that, create a locale spec that
specifies it.

regards, tom lane

Tom Lane wrote:
> Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> > We do allow MDY and DMY specification, but that controls both input and
> > output in the server, while this just controls psql display. It is a
> > good question how other settings should be handled, but I don't know the
> > answers. Anyone?
>
> This patch sounds quite a lot different from what I thought we had
> agreed to, which was a way to specify that the client-side LC_NUMERIC
> locale settings should be used for formatting numbers. I think the only
> pset should be "locale on" or "locale not on". This business about
> space is a red herring: if you want that, create a locale spec that
> specifies it.

Well, how many people have langinfo? I don't. If this is what the
group wants, Someone with langinfo is going to have to code the
configure tests and convert it to a binary setting, or remove the patch.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> Tom Lane wrote:
>> This patch sounds quite a lot different from what I thought we had
>> agreed to, which was a way to specify that the client-side LC_NUMERIC
>> locale settings should be used for formatting numbers. I think the only
>> pset should be "locale on" or "locale not on". This business about
>> space is a red herring: if you want that, create a locale spec that
>> specifies it.

> Well, how many people have langinfo? I don't.

What's langinfo got to do with it? To minimize our portability
exposure, the patch should use the same facilities that
src/backend/utils/adt/pg_locale.c already depends on, namely
setlocale() and localeconv().

regards, tom lane

Tom Lane wrote:
> Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> > Tom Lane wrote:
> >> This patch sounds quite a lot different from what I thought we had
> >> agreed to, which was a way to specify that the client-side LC_NUMERIC
> >> locale settings should be used for formatting numbers. I think the only
> >> pset should be "locale on" or "locale not on". This business about
> >> space is a red herring: if you want that, create a locale spec that
> >> specifies it.
>
> > Well, how many people have langinfo? I don't.
>
> What's langinfo got to do with it? To minimize our portability
> exposure, the patch should use the same facilities that
> src/backend/utils/adt/pg_locale.c already depends on, namely
> setlocale() and localeconv().

The original patch included "langinfo.h" and used this call to set the
default separator:

char *dec_point = nl_langinfo(__DECIMAL_POINT);

I don't see __DECIMAL_POINT defined anywhere on the FreeBSD 4.11 box we
use for CVS.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

On Tue, Jul 12, 2005 at 10:48:40AM -0400, Bruce Momjian wrote:
> Peter Eisentraut wrote:
> > Am Dienstag, 12. Juli 2005 05:29 schrieb Bruce Momjian:
> > > > Bruce Momjian wrote:
>
> > The current state allows users to manually set the format, or
> > actually only one aspect of the format? Where can you set the
> > decimal separator and the size of the grouping (3 digits or 4
> > digits)? Is this capability even useful to get localized
> > behavior?
>
> You can't change the grouping from three,

This makes it a non-starter, IMHO. One set of people who use 4 digits
is this little ethnic group called the Chinese.

> > My other concern is that if we allow manual specification of the
> > output format of some data type, then eventually someone will want
> > to specify the format of some other data type as well, such as the
> > date/time types. We should think about whether we want to be
> > consistent here.
>
> We do allow MDY and DMY specification, but that controls both input
> and output in the server, while this just controls psql display. It
> is a good question how other settings should be handled, but I don't
> know the answers. Anyone?

Hrm. I think we can safely tackle input and output as separate
features here. What do we do for to_char() with such separators?

Cheers,
D
--
David Fetter david(at)fetter(dot)org http://fetter.org/
phone: +1 510 893 6100 mobile: +1 415 235 3778

Remember to vote!

David Fetter wrote:
> On Tue, Jul 12, 2005 at 10:48:40AM -0400, Bruce Momjian wrote:
> > Peter Eisentraut wrote:
> > > Am Dienstag, 12. Juli 2005 05:29 schrieb Bruce Momjian:
> > > > > Bruce Momjian wrote:
> >
> > > The current state allows users to manually set the format, or
> > > actually only one aspect of the format? Where can you set the
> > > decimal separator and the size of the grouping (3 digits or 4
> > > digits)? Is this capability even useful to get localized
> > > behavior?
> >
> > You can't change the grouping from three,
>
> This makes it a non-starter, IMHO. One set of people who use 4 digits
> is this little ethnic group called the Chinese.

I didn't know that.

> > > My other concern is that if we allow manual specification of the
> > > output format of some data type, then eventually someone will want
> > > to specify the format of some other data type as well, such as the
> > > date/time types. We should think about whether we want to be
> > > consistent here.
> >
> > We do allow MDY and DMY specification, but that controls both input
> > and output in the server, while this just controls psql display. It
> > is a good question how other settings should be handled, but I don't
> > know the answers. Anyone?
>
> Hrm. I think we can safely tackle input and output as separate
> features here. What do we do for to_char() with such separators?

Well, it isn't so much the input/output issue, but that the output
control is only in psql. Is that OK? I don't image we could ever put
input functionality in psql, only in the server.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073