All caught up

Hi all,

When I tried to psql command, I found that the result of command seems to be
different from explanation of the manual. Please tell me which is
correct performance, the result of command or the explanation of manual.
I show the details as follows.

[Events］
The encoding that psql command sets has difference between the result of command
and the explanation of manual.

The following is the explanation of encoding that psql command sets,
which I made extracts from the 9.2 manual.
---------------------------------------------------------------------------
VI. Reference / II. PostgreSQL Client Applications / psql / Connecting to a Database
"If at least one of standard input or standard output are a terminal,
then psql sets the client encoding to "auto",
which will detect the appropriate client encoding from the locale settings
(LC_CTYPE environment variable on Unix systems).
If this doesn't work out as expected,
the client encoding can be overridden using the environment variable PGCLIENTENCODING."
----------------------------------------------------------------------------
Above explanation says that "If at least one of standard input or standard output
are a terminal, then psql sets the client encoding to "auto",
which will detect the appropriate client encoding from the locale settings
(LC_CTYPE environment variable on Unix systems)".
But according to the result from psql command, it seems that the appropriate
client encoding is NOT detected from the locale settings.

The result of psql command is following.

[Precondition of test]
- PostgreSQL VL：9.2.4
- Locale settings encoding : ja_JP.eucJP (LANG)
- Client encoding：UTF8
- client_encoding(paramater of postgresql.conf) : None specified
- "test.txt" is psql's input file which is written "\encoding".

[Result]
1) In case both standard input and standard output are a terminal
$ psql postgres -f test.txt -o result.txt
EUC_JP
→ Locale setting encoding

2) In case only standard input is a terminal
$ psql postgres -f test.txt > result.txt
$ cat result.txt
UTF8
→Client encoding

3) In case only standard output is a terminal
$ psql postgres -o result.txt < test.txt
UTF8
→Client encoding

4) In case both standard input and standard output are not a terminal
$ psql postgres < test.txt > result.txt
$ cat result.txt
UTF8
→Client encoding

1) and 4) performed just as manual, but 2) and 3) do not.
If 2) and 3) performed just as munual, I think that their encoding should be EUC_JP
that is locale setting encoding.

If manual has mistake, I think that correct explanation is following.

"If BOTH standard input AND standard output are a terminal, then psql sets
the client encoding to "auto", which will detect the appropriate client encoding
from the locale settings (LC_CTYPE environment variable on Unix systems)."

What do you think?

Warmest Regards.

On 02/05/2014 11:43 PM, 坂本 翼 wrote:
> Hi all,
>
> When I tried to psql command, I found that the result of command seems to be
> different from explanation of the manual. Please tell me which is
> correct performance, the result of command or the explanation of manual.
> I show the details as follows.
>
> [Events］
> The encoding that psql command sets has difference between the result of command
> and the explanation of manual.
>
> The following is the explanation of encoding that psql command sets,
> which I made extracts from the 9.2 manual.
> ---------------------------------------------------------------------------
> VI. Reference / II. PostgreSQL Client Applications / psql / Connecting to a Database
> "If at least one of standard input or standard output are a terminal,
> then psql sets the client encoding to "auto",
> which will detect the appropriate client encoding from the locale settings
> (LC_CTYPE environment variable on Unix systems).
> If this doesn't work out as expected,
> the client encoding can be overridden using the environment variable PGCLIENTENCODING."
> ----------------------------------------------------------------------------
> Above explanation says that "If at least one of standard input or standard output
> are a terminal, then psql sets the client encoding to "auto",
> which will detect the appropriate client encoding from the locale settings
> (LC_CTYPE environment variable on Unix systems)".
> But according to the result from psql command, it seems that the appropriate
> client encoding is NOT detected from the locale settings.
>
> The result of psql command is following.
>
> [Precondition of test]
> - PostgreSQL VL：9.2.4
> - Locale settings encoding : ja_JP.eucJP (LANG)

Not sure that it makes a difference but the docs say psql looks at
LC_CTYPE not LANG for Unix systems. You did not say what OS you are
working on though from the examples I am guessing some form of Unix.

> - Client encoding：UTF8
> - client_encoding(paramater of postgresql.conf) : None specified
> - "test.txt" is psql's input file which is written "\encoding".
>
> [Result]
> 1) In case both standard input and standard output are a terminal
> $ psql postgres -f test.txt -o result.txt
> EUC_JP
> → Locale setting encoding
>
> 2) In case only standard input is a terminal
> $ psql postgres -f test.txt > result.txt
> $ cat result.txt
> UTF8
> →Client encoding
>
> 3) In case only standard output is a terminal
> $ psql postgres -o result.txt < test.txt
> UTF8
> →Client encoding
>
> 4) In case both standard input and standard output are not a terminal
> $ psql postgres < test.txt > result.txt
> $ cat result.txt
> UTF8
> →Client encoding
>
> 1) and 4) performed just as manual, but 2) and 3) do not.
> If 2) and 3) performed just as munual, I think that their encoding should be EUC_JP
> that is locale setting encoding.
>
> If manual has mistake, I think that correct explanation is following.
>
> "If BOTH standard input AND standard output are a terminal, then psql sets
> the client encoding to "auto", which will detect the appropriate client encoding
> from the locale settings (LC_CTYPE environment variable on Unix systems)."
>
> What do you think?
>
> Warmest Regards.
>
>
>
>
>
>
>

--
Adrian Klaver
adrian(dot)klaver(at)gmail(dot)com

> Not sure that it makes a difference but the docs say psql looks at
> LC_CTYPE not LANG for Unix systems. You did not say what OS you are
> working on though from the examples I am guessing some form of Unix.

Thank you for the response.
Sorry, I had not indicated OS information.
OS information is the following:
Red Hat Enterprise Linux Server release 6.2 Kernel 2.6.32-220.el6.x86_64 on an x86_64

The reason which was being verified using the LANG environment variable, I thought that the value of a LANG environment variable was set as LC_CTYPE when the LC_CTYPE environment variable and the LC_ALL environment variable are not set up.

The LC_CTYPE environment variable was set up and re-verified.
The result of psql command is following.

[Result]
% setenv LC_CTYPE ja_JP.eucJP

% psql postgres -f test.txt -o result.txt EUC_JP

% psql postgres -f test.txt > result.txt
UTF8

% psql postgres -o result.txt < test.txt
UTF8

% psql postgres < test.txt > result.txt
UTF8

Even when a LC_CTYPE environment variable was set up, the result did not change.
What do you think?

[CC'ed -hackers]

Tsubasa Sakamoto wrote:
>> Not sure that it makes a difference but the docs say psql looks at
>> LC_CTYPE not LANG for Unix systems. You did not say what OS you are
>> working on though from the examples I am guessing some form of Unix.

> The LC_CTYPE environment variable was set up and re-verified.
> The result of psql command is following.
>
> [Result]
> % setenv LC_CTYPE ja_JP.eucJP
>
> % psql postgres -f test.txt -o result.txt EUC_JP
>
> % psql postgres -f test.txt > result.txt
> UTF8
>
> % psql postgres -o result.txt < test.txt
> UTF8
>
> % psql postgres < test.txt > result.txt
> UTF8
>
> Even when a LC_CTYPE environment variable was set up, the result did not change.
> What do you think?

I think that the documentation contradicts the code.

In bin/psql/settings.h:

typedef struct _psqlSettings
{
[...]
bool notty; /* stdin or stdout is not a tty (as determined
* on startup) */
[...]
} PsqlSettings;

extern PsqlSettings pset;

In bin/psql/command.c and bin/psql/startup.c:

keywords[6] = "client_encoding";
values[6] = (pset.notty || getenv("PGCLIENTENCODING")) ? NULL : "auto";

That matches your observations: client_encoding=auto is only
used if both stdin and stdout are attached to a tty.

I suggest the attached documentation fix.

Yours,
Laurenz Albe

Thanks for your support.

Best regards,
Tsubasa Sakamoto

> -----Original Message-----
> From: Albe Laurenz [mailto:laurenz(dot)albe(at)wien(dot)gv(dot)at]
> Sent: Friday, February 07, 2014 6:12 PM
> To: Tsubasa Sakamoto *EXTERN*; 'Adrian Klaver'
> Cc: pgsql-general(at)postgresql(dot)org; pgsql-hackers(at)postgresql(dot)org
> Subject: RE: [GENERAL] client encoding that psql command sets
>
> [CC'ed -hackers]
>
> Tsubasa Sakamoto wrote:
> >> Not sure that it makes a difference but the docs say psql looks at
> >> LC_CTYPE not LANG for Unix systems. You did not say what OS you are
> >> working on though from the examples I am guessing some form of Unix.
>
> > The LC_CTYPE environment variable was set up and re-verified.
> > The result of psql command is following.
> >
> > [Result]
> > % setenv LC_CTYPE ja_JP.eucJP
> >
> > % psql postgres -f test.txt -o result.txt EUC_JP
> >
> > % psql postgres -f test.txt > result.txt
> > UTF8
> >
> > % psql postgres -o result.txt < test.txt
> > UTF8
> >
> > % psql postgres < test.txt > result.txt
> > UTF8
> >
> > Even when a LC_CTYPE environment variable was set up, the result did not
> change.
> > What do you think?
>
> I think that the documentation contradicts the code.
>
> In bin/psql/settings.h:
>
> typedef struct _psqlSettings
> {
> [...]
> bool notty; /* stdin or stdout is not a tty (as
> determined
> * on startup) */ [...] } PsqlSettings;
>
> extern PsqlSettings pset;
>
> In bin/psql/command.c and bin/psql/startup.c:
>
> keywords[6] = "client_encoding";
> values[6] = (pset.notty || getenv("PGCLIENTENCODING")) ? NULL :
> "auto";
>
> That matches your observations: client_encoding=auto is only used if both
> stdin and stdout are attached to a tty.
>
> I suggest the attached documentation fix.
>
> Yours,
> Laurenz Albe

On Fri, Feb 7, 2014 at 09:12:10AM +0000, Albe Laurenz wrote:
> > Even when a LC_CTYPE environment variable was set up, the result did not change.
> > What do you think?
>
> I think that the documentation contradicts the code.
>
> In bin/psql/settings.h:
>
> typedef struct _psqlSettings
> {
> [...]
> bool notty; /* stdin or stdout is not a tty (as determined
> * on startup) */
> [...]
> } PsqlSettings;
>
> extern PsqlSettings pset;
>
> In bin/psql/command.c and bin/psql/startup.c:
>
> keywords[6] = "client_encoding";
> values[6] = (pset.notty || getenv("PGCLIENTENCODING")) ? NULL : "auto";
>
> That matches your observations: client_encoding=auto is only
> used if both stdin and stdout are attached to a tty.
>
> I suggest the attached documentation fix.

Patch applied and backpatched to 9.3. Thanks.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +

Bruce Momjian wrote:
>> I suggest the attached documentation fix.
>
> Patch applied and backpatched to 9.3. Thanks.

What would PostgreSQL do without Bruce who undertakes the
Herculean task of making sure that nothing gets forgotten
and slips through the cracks?

Thanks!

Yours,
Laurenz Albe

On Fri, Apr 18, 2014 at 07:42:06AM +0000, Albe Laurenz wrote:
> Bruce Momjian wrote:
> >> I suggest the attached documentation fix.
> >
> > Patch applied and backpatched to 9.3. Thanks.
>
> What would PostgreSQL do without Bruce who undertakes the
> Herculean task of making sure that nothing gets forgotten
> and slips through the cracks?

Thanks.

FYI, I finished going through all the unresolved email threads
yesterday, so I have done as much as I can to prepare for beta. I will
start the 9.4 major release notes now.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +