Re: Proposed patch - psql wraps at window width

bruce wrote:
> Tom Lane wrote:
> > Bruce Momjian <bruce(at)momjian(dot)us> writes:
> > > You are going to need to find community members who support your
> > > analysis if you want to make any headway in changing the patch.
> >
> > Let's turn that around, shall we? I think at this point it's *you*
> > that are standing alone and need to find someone who agrees with
> > your approach.
>
> I am confused exactly what people want changed in the patch. Some want
> no control over wrapping in file output, and others want $COLUMN to
> control column file output. The only person I am seeing code from is
> Greg Stark, but I think most don't want his changes.

Here is an email showing two people who want some way of getting wrapped
output into a file:

http://archives.postgresql.org/pgsql-patches/2008-04/msg00344.php

I think the API in the patch is the best I am going to do to keep
everyone happy --- 'wrapped' doesn't affect file/pipe output unless you
also tell it the width you want. Most interactive users are going to
set 'wrapped' and never set the width so it is automatically determined.

Some want wrapped to not affect file/pipe at all, while others (Greg
only?) want $COLUMNS to affect file/pipe output.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

"Bruce Momjian" <bruce(at)momjian(dot)us> writes:

> I think the API in the patch is the best I am going to do to keep
> everyone happy --- 'wrapped' doesn't affect file/pipe output unless you
> also tell it the width you want. Most interactive users are going to
> set 'wrapped' and never set the width so it is automatically determined.

Aaah, I think that's the key to where we're going wrong.

Trying to use the columns logic to encode two separate decisions. "wrapped or
not wrapped" and "how wide".

I think this also clarifies Tom's objection. He's worried about people
configuring psql for interactive use and being surprised when their automated
scripts fail to parse the resulting output.

We do need a way to specifically request wrapped format, but if we want a way
to say "wrapped format only on a terminal" then we should have a mode for that
too.

But once we're in wrapped format we should stick to it and always follow the
same logic to determine the width.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's Slony Replication support!

Gregory Stark wrote:
> "Bruce Momjian" <bruce(at)momjian(dot)us> writes:
>
> > I think the API in the patch is the best I am going to do to keep
> > everyone happy --- 'wrapped' doesn't affect file/pipe output unless you
> > also tell it the width you want. Most interactive users are going to
> > set 'wrapped' and never set the width so it is automatically determined.
>
> Aaah, I think that's the key to where we're going wrong.
>
> Trying to use the columns logic to encode two separate decisions. "wrapped or
> not wrapped" and "how wide".

Well, they kind of fit because there is no good way to know the width
they would want for file/pipe output (no the terminal width isn't very
helpful in most cases). Bottom line is we are going to need a way to
specify the width for wrapped file/pipe (COLUMNS is not something that
is easily set), and \pset columns seems to allow both the setting of the
width and saying we want wrapping for file/pipe.

I am not excited about 'wrapped-interactive' and 'wrapped-all' formats.
Do you have some other idea in mind?

> I think this also clarifies Tom's objection. He's worried about people
> configuring psql for interactive use and being surprised when their automated
> scripts fail to parse the resulting output.
>
> We do need a way to specifically request wrapped format, but if we want a way
> to say "wrapped format only on a terminal" then we should have a mode for that
> too.
>
> But once we're in wrapped format we should stick to it and always follow the
> same logic to determine the width.

I can't think of any cases where we have one setting for interactive and
another for all uses.

I do think we might be adding an 'auto' format the does
aligned/wrapped/expanded based on the table width, but only for
interactive use.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

"Bruce Momjian" <bruce(at)momjian(dot)us> writes:

> I am not excited about 'wrapped-interactive' and 'wrapped-all' formats.
> Do you have some other idea in mind?

Some other idea for the name of it? Not particularly. I like your later
suggestion of "auto". Perhaps just "wrapped" and "auto"?

> I can't think of any cases where we have one setting for interactive and
> another for all uses.

In general I don't like such things but it seems to be what you're trying to
do. And also what Tom seems to be yearning for when he says that it shouldn't
affect file output. And if we do it we should do it properly and not hobble
both modes.

Note that basing it on the window size ioctl doesn't actually dtrt either.
What you really want is for it to depend on isatty(). If isatty(fout) is true
then you want to try to take the interactive default even if the ioctl fails.

Trying to join the two decisions means that nobody will be happy. If you ssh
in you won't get wrapped format, if you redirect to a file and specify wrapped
format explicitly you'll be frustrated that you're still not getting it. And
if you do want those things so you \pset columns you'll find it starts doing
it even when you -f file.dmp -- which doesn't seem bad to me but it means it
isn't satisfying your desires.

> I do think we might be adding an 'auto' format the does
> aligned/wrapped/expanded based on the table width, but only for
> interactive use.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's On-Demand Production Tuning

Gregory Stark wrote:
> "Bruce Momjian" <bruce(at)momjian(dot)us> writes:
>
> > I am not excited about 'wrapped-interactive' and 'wrapped-all' formats.
> > Do you have some other idea in mind?
>
> Some other idea for the name of it? Not particularly. I like your later
> suggestion of "auto". Perhaps just "wrapped" and "auto"?

Well, I was going to bring up changes to the default after the patch was
applied but I will bring it up now. I think there is some real
attractivness to having long values wrap to fit on your screen in
interactive mode. In fact, it is hard to justify our current
interactive behavior of a row just overflowing the screen width and
moving to the next line. We have gotten used to it, but it is certainly
not very user-friendly.

We have discussed having a formatting mode where aligned output switches
to expanded output when the row is too wide. One idea would be to
create an 'auto' mode that would display in aligned, or wrapped if that
doesn't fit, or expanded if that doesn't fit.

However, this idea really doesn't help us with the file/pipe idea. One
crazy idea would be for formatting to take two values, like:

\pset formatting wrapped aligned

where the first is for interactive use and the second is for file/pipe
output. Now, if they only specify one value, what does that control?

> > I can't think of any cases where we have one setting for interactive and
> > another for all uses.
>
> In general I don't like such things but it seems to be what you're trying to
> do. And also what Tom seems to be yearning for when he says that it shouldn't
> affect file output. And if we do it we should do it properly and not hobble
> both modes.
>
> Note that basing it on the window size ioctl doesn't actually dtrt either.
> What you really want is for it to depend on isatty(). If isatty(fout) is true
> then you want to try to take the interactive default even if the ioctl fails.

Yes, we do that now with $COLUMNS.

> Trying to join the two decisions means that nobody will be happy. If you ssh
> in you won't get wrapped format, if you redirect to a file and specify wrapped
> format explicitly you'll be frustrated that you're still not getting it. And
> if you do want those things so you \pset columns you'll find it starts doing
> it even when you -f file.dmp -- which doesn't seem bad to me but it means it
> isn't satisfying your desires.

I think we need to honor $COLUMNS if ioctl() fails.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:
> > Trying to join the two decisions means that nobody will be happy. If you ssh
> > in you won't get wrapped format, if you redirect to a file and specify wrapped
> > format explicitly you'll be frustrated that you're still not getting it. And
> > if you do want those things so you \pset columns you'll find it starts doing
> > it even when you -f file.dmp -- which doesn't seem bad to me but it means it
> > isn't satisfying your desires.
>
> I think we need to honor $COLUMNS if ioctl() fails.

Clarification, honor $COLUMNS if ioctl() fails but only for interactive
mode.

If we can't make everyone happy we could just not add the patch. It
sounds crazy but we have had cases in the past where we couldn't get
concensus and we did nothing. Of course nothing is usually worse than
either option but groups behave in strange ways sometimes.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:

> interactive mode. In fact, it is hard to justify our current
> interactive behavior of a row just overflowing the screen width and
> moving to the next line. We have gotten used to it, but it is certainly
> not very user-friendly.

+1

>
> We have discussed having a formatting mode where aligned output switches
> to expanded output when the row is too wide. One idea would be to
> create an 'auto' mode that would display in aligned, or wrapped if that
> doesn't fit, or expanded if that doesn't fit.
>
> However, this idea really doesn't help us with the file/pipe idea. One
> crazy idea would be for formatting to take two values, like:
>
> \pset formatting wrapped aligned

IMO we should never wrap file or pipe output. Once it leaves psql, it
should be up to whatever you pushed it through (file/pipe/script) to
handle formatting.

Sincerely,

Joshua D. Drake

Joshua D. Drake wrote:
> Bruce Momjian wrote:
> > We have discussed having a formatting mode where aligned output switches
> > to expanded output when the row is too wide. One idea would be to
> > create an 'auto' mode that would display in aligned, or wrapped if that
> > doesn't fit, or expanded if that doesn't fit.
> >
> > However, this idea really doesn't help us with the file/pipe idea. One
> > crazy idea would be for formatting to take two values, like:
> >
> > \pset formatting wrapped aligned
>
> IMO we should never wrap file or pipe output. Once it leaves psql, it
> should be up to whatever you pushed it through (file/pipe/script) to
> handle formatting.

Yes, I understand that argument, but the way we wrap values is something
an external tool is going to have great trouble duplicating.

For example, if I want to send wide psql output in email, right now I
just send it wide or use \x. With 'wrapped' I can set it to 72 columns
and get something I can email to people. I could copy it from my
screen, but if the output is more than a screen full it is much harder
to capture. (Yea, I could use 'script', but that isn't for the novice,
and I have no idea how that would be done on Win32.)

We could go with wrap only handling interactive and then see what
feedback we get from the field. For short output people can cut/paste
into files if they want wrapped output. That would eliminate the need
for \pset columns. We could just do ioctl() and check $COLUMNS.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:

> For example, if I want to send wide psql output in email, right now I
> just send it wide or use \x. With 'wrapped' I can set it to 72 columns
> and get something I can email to people. I could copy it from my
> screen, but if the output is more than a screen full it is much harder
> to capture. (Yea, I could use 'script', but that isn't for the novice,
> and I have no idea how that would be done on Win32.)

I think your idea of novice is a little different than mine. A novice
isn't going to use this feature anyway, and by the time they start to
consider it (if they ever do), they will have already wrapped it in Perl
or C#.

Sincerely,

Joshua D. Drake

* Bruce Momjian <bruce(at)momjian(dot)us> [080424 23:14]:

> Well, I was going to bring up changes to the default after the patch was
> applied but I will bring it up now. I think there is some real
> attractivness to having long values wrap to fit on your screen in
> interactive mode. In fact, it is hard to justify our current
> interactive behavior of a row just overflowing the screen width and
> moving to the next line. We have gotten used to it, but it is certainly
> not very user-friendly.

-1.

Personally, I don't ever want to see that wrapped format.

<opinion valid="maybe not">
That wrapped format is more user-unfriendly than the long lines, which
are at least easily understandable.
</opinion>

So add a wrapped format, because obviously some people want it, but in a
similar vien to the making psql have no banner by default, I think
making wrapped format the default will cause much user consternation.

In Bruce's sample, even with the rules "right there", I still had to
look at the query to try and figure out what each data piece actually
was.

But I have to admit, trying to get a generic multi-column "wrapped"
format is a hard task. I couldn't come up with a scheme for showing it
easily myself on my (discarded) attempt to reply to the sample output.
Anything that *doesn't* have a wrapped line end with \ and then continue
with a > is going to feel awkward to me, and I'ld rather just have the
long lines, because having a one-off, complicated display format in
psql, even though it could technically define exactly what the data is
it's trying to display, if it's different from everything else means I'm
going to avoid it as much as possible.

a.
--
Aidan Van Dyk Create like a god,
aidan(at)highrise(dot)ca command like a king,
http://www.highrise.ca/ work like a slave.

Bruce Momjian wrote:
> We have discussed having a formatting mode where aligned output switches
> to expanded output when the row is too wide. One idea would be to
> create an 'auto' mode that would display in aligned, or wrapped if that
> doesn't fit, or expanded if that doesn't fit.

I haven't heard any new ideas of how to resolve this issue in the past
few hours so I will throw out two new ideas.

Have a 'format=auto' mode that does aligned/wrapped/expanded, but only
for screen output --- file/pipe would still use aligned. And have
'format=wrapped' affect file/pipe by requiring the user to specify the
width, or use a default of 72.

Another idea is to require the user to specify the file/pipe output
width when they define format=wrapped, e.g. format=wrapped:75. If they
don't specify the width, 'wrapped' doesn't affect file/pipe. (Perhaps
the width controls screen width too.) (That is effectively what \pset
columns does, but this is more integrated.)

I have heard why people want 'wrapped' to affect file/pipe output, but I
have not heard a clear explanation of why people don't want that. I
personally think that having the screen width affect the width of
file/pipe is odd, but if the user specifies the width, it seems fine to
me. What is the objection?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian escribió:

> Have a 'format=auto' mode that does aligned/wrapped/expanded, but only
> for screen output --- file/pipe would still use aligned. And have
> 'format=wrapped' affect file/pipe by requiring the user to specify the
> width, or use a default of 72.

I have a different question. Why are we mixing file and pipe output? I
think the use cases are different and perhaps we should use different
defaults.

For example, most people I've seen writing shell scripts involving psql
output have to pass the -A flag all the time. Perhaps we could change
the default output to unaligned for pipes?

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

On Fri, 25 Apr 2008 10:45:01 -0400
Alvaro Herrera <alvherre(at)commandprompt(dot)com> wrote:

> I have a different question. Why are we mixing file and pipe
> output? I think the use cases are different and perhaps we should
> use different defaults.
>
> For example, most people I've seen writing shell scripts involving
> psql output have to pass the -A flag all the time. Perhaps we could
> change the default output to unaligned for pipes?

and -t

Sincerely,

Joshua D. Drake

--
The PostgreSQL Company since 1997: http://www.commandprompt.com/
PostgreSQL Community Conference: http://www.postgresqlconference.org/
United States PostgreSQL Association: http://www.postgresql.us/
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate

Alvaro Herrera wrote:
> Bruce Momjian escribi?:
>
> > Have a 'format=auto' mode that does aligned/wrapped/expanded, but only
> > for screen output --- file/pipe would still use aligned. And have
> > 'format=wrapped' affect file/pipe by requiring the user to specify the
> > width, or use a default of 72.
>
> I have a different question. Why are we mixing file and pipe output? I
> think the use cases are different and perhaps we should use different
> defaults.
>
> For example, most people I've seen writing shell scripts involving psql
> output have to pass the -A flag all the time. Perhaps we could change
> the default output to unaligned for pipes?

Yes, that would make sense to me.

I think the people wanting wrapped to control file/pipe don't want it as
the default, but want _some_ way of getting wrapped output into a file.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:
> Alvaro Herrera wrote:
> > Bruce Momjian escribi?:
> >
> > > Have a 'format=auto' mode that does aligned/wrapped/expanded, but only
> > > for screen output --- file/pipe would still use aligned. And have
> > > 'format=wrapped' affect file/pipe by requiring the user to specify the
> > > width, or use a default of 72.
> >
> > I have a different question. Why are we mixing file and pipe output? I
> > think the use cases are different and perhaps we should use different
> > defaults.
> >
> > For example, most people I've seen writing shell scripts involving psql
> > output have to pass the -A flag all the time. Perhaps we could change
> > the default output to unaligned for pipes?
>
> Yes, that would make sense to me.
>
> I think the people wanting wrapped to control file/pipe don't want it as
> the default, but want _some_ way of getting wrapped output into a file.

Let me add that the patch as it was posted does not have wrapping
affecting file/pipe output unless you also specify a column width with
\pset. This seemed the most logical and tried to satisfy the most
people. You can see this in the documentation changes of the patch:

ftp://momjian.us/pub/postgresql/mypatches/wrap

I have now clarified the documentation in the patch.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

"Bruce Momjian" <bruce(at)momjian(dot)us> writes:

>> I think the people wanting wrapped to control file/pipe don't want it as
>> the default, but want _some_ way of getting wrapped output into a file.
>
> Let me add that the patch as it was posted does not have wrapping
> affecting file/pipe output unless you also specify a column width with
> \pset. This seemed the most logical and tried to satisfy the most
> people. You can see this in the documentation changes of the patch:

If you specify format=wrapped and get something other than wrapped it's a bug
and people will undoubtedly report it as such.

If you want a way to specify "wrapped on a terminal but not to a non-terminal"
then you should make that explicit and separate from the column-width
determination.

I'm done with this thread now.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's 24x7 Postgres support!

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sat, Apr 26, 2008 at 4:46 AM, Gregory Stark wrote:
> If you specify format=wrapped and get something other than wrapped it's a bug
> and people will undoubtedly report it as such.
>

Agree. If I tell psql that I want wrapped output and it gives me
something else when I output to a file, I'm going to be confused.

In particular, I often work up a query in psql, making revisions with
\e, and when I've got the results I want, I do a \g out.txt to dump
the query output into a file.

I expect that the contents of out.txt will be exactly the same as the
query output I've just been crafting in psql.

If I want to output the query in a machine-readable form to a file, I
just use \a to switch to unaligned output.

Aside: \a should probably be changed to toggle between "unaligned" and
"aligned"/"wrapped", whichever of the two you had selected last.

Cheers,
BJ

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: http://getfiregpg.org

iD8DBQFIEixE5YBsbHkuyV0RArhoAKDUH/Svt84xPFn7BGRkq7zEWtonpwCg/pOS
66uQ6mKWvxnZLICfAcJzdLw=
=ibbj
-----END PGP SIGNATURE-----

Brendan Jurd wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Sat, Apr 26, 2008 at 4:46 AM, Gregory Stark wrote:
> > If you specify format=wrapped and get something other than wrapped it's a bug
> > and people will undoubtedly report it as such.
> >
>
> Agree. If I tell psql that I want wrapped output and it gives me
> something else when I output to a file, I'm going to be confused.
>
> In particular, I often work up a query in psql, making revisions with
> \e, and when I've got the results I want, I do a \g out.txt to dump
> the query output into a file.
>
> I expect that the contents of out.txt will be exactly the same as the
> query output I've just been crafting in psql.

Obviously you have expections of how wrapping should behave. Please
name me an application that has a wrapped mode that has the output to a
file wrap based on the screen width? It isn't 'ls -C'.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian escribió:

> Obviously you have expections of how wrapping should behave. Please
> name me an application that has a wrapped mode that has the output to a
> file wrap based on the screen width? It isn't 'ls -C'.

Why would we need to imitate what other apps do? What we need to
investigate is use cases, and how do we cater for each one, making it
easy for the most common while at the same time making it not impossible
for the most obscure.

There is no point in doing things in a certain way just because others
do the same. Are you going to argue that we need to make the server
crash from time to time because other systems do that too?

We came up with dollar quoting which is a completely novel idea AFAIK.
Why can't we come up with other useful, novel designs?

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
> There is no point in doing things in a certain way just because others
> do the same. Are you going to argue that we need to make the server
> crash from time to time because other systems do that too?

> We came up with dollar quoting which is a completely novel idea AFAIK.
> Why can't we come up with other useful, novel designs?

I don't think there's a need to invent a totally new way of dealing with
window wrapping. If we did, it'd be highly unlikely to preserve the
principle of least surprise.

regards, tom lane

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sat, Apr 26, 2008 at 5:21 AM, Bruce Momjian
wrote:
> Obviously you have expections of how wrapping should behave. Please
> name me an application that has a wrapped mode that has the output to a
> file wrap based on the screen width? It isn't 'ls -C'.
>

Trying to draw analogies between psql and other command-line tools is
always going to be a bit of a stretch. Name me any application at all
that tries to produce both human- and machine- readable
representations of arbitrary tabular data.

All I'm saying is, if the user has exlicitly told psql that he wants
wrapped, human-readable output, we should give it to him regardless of
where he wants to put it.

If the user hasn't specified any format at all, then it's fine to play
guessing games and try to select the best format automatically for
him, based on factors like the destination. But IMO once the user
makes a determination about the output format, that's the end of the
story. You toe that line.

Cheers,
BJ

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: http://getfiregpg.org

iD8DBQFIEju85YBsbHkuyV0RAgNTAJ4ghWB1SlvuQhsH3ltrbVM5LoMBwACgoTdT
B6UDw4oG5tlwTlcLcJA4xic=
=2lmV
-----END PGP SIGNATURE-----

>>> On Fri, Apr 25, 2008 at 3:14 PM, in message
<37ed240d0804251314y72461b8fy8d56606f362f91c6(at)mail(dot)gmail(dot)com>, "Brendan
Jurd"
<direvus(at)gmail(dot)com> wrote:

> If the user hasn't specified any format at all, then it's fine to
play
> guessing games and try to select the best format automatically for
> him, based on factors like the destination. But IMO once the user
> makes a determination about the output format, that's the end of the
> story. You toe that line.

I would go further, and say that it would be surprising and
troublesome for psql to guess at whether I want wrapping or unaligned
output. A given set of command line switches and a given set of
inputs should give a consistent output format, regardless of whether
it's going into a pipe or a disk file or out to the local console or
back through ssh. Like a previous poster, I often use an interactive
session to refine something that will be run against a list of servers
with xargs or will be run from crontab. If the interactive output is
big enough to cause it to go through "less", then I still want to see
the format it would have if it didn't. If I save to a file from
"less" or copy and paste from a ssh window when it was a few long
lines, I want it to match what I will get if I run directly to disk.

I consider current behavior pretty darned friendly to the way I work.
Some of the suggestions in this thread sound downright nightmarish to
me. I hope that wrapping never happens without an explicit command
line option or a backslash command.

-Kevin

Alvaro Herrera wrote:
> Bruce Momjian escribi?:
>
> > Obviously you have expections of how wrapping should behave. Please
> > name me an application that has a wrapped mode that has the output to a
> > file wrap based on the screen width? It isn't 'ls -C'.
>
> Why would we need to imitate what other apps do? What we need to
> investigate is use cases, and how do we cater for each one, making it
> easy for the most common while at the same time making it not impossible
> for the most obscure.
>
> There is no point in doing things in a certain way just because others
> do the same. Are you going to argue that we need to make the server
> crash from time to time because other systems do that too?
>
> We came up with dollar quoting which is a completely novel idea AFAIK.
> Why can't we come up with other useful, novel designs?

Your argument about crashing above seems like reductio ad absurdum
(http://en.wikipedia.org/wiki/Proof_by_contradiction).

My point was that the poster was saying he expected the file/pipe output
to honor the screen width on output, so I asked him for an example of
why he had that expectation --- that seems logical. Perhaps it is only
to be consistent with other psql behavior.

FYI, ls -C actually wraps to 72(?) unless you specify another width, so
one possible behavior would be for \pset wrapped to wrap to 72 for
file/pipe unless you set \pset columns. That might make the "I want it
always to wrap" group happier, but not the "wrapped shouldn't affect
file/pipe". I have not heard anyone explain why the later behavior is
bad, especially if we default to a width of 72 rather than the screen
width.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Am Samstag, 26. April 2008 schrieb Bryce Nesbitt:
> But that leaves a big hole: what does the setting in .psqlrc refer to?  
> Do we need separate controls in .psql?
>
>    \pset format_terminal wrap [auto|nnn|off]
>    \pset format_terminal html
>    \pset format_stream wrap [auto|nnn|off]
>    \pset format_stream html

I think one of the weirdest behaviors in psql is that it reads the startup
file in noninteractive mode. Unix shells don't do this for good reasons. If
we could work out a change that moves the psql startup file behavior more in
line with that of Unix shells, perhaps meaning separate startup files, then
users could put all the crazy formatting settings they like in .psqlrc
without affecting noninteractive output. And there would still be the
possibility to format noninteractive output the way you want by explicit
intervention. This is basically what your pseudoproposal above would
accomplish, just on a more global scale.

As the originator of the "psql wraps at window width" patch, I'd like to
set a matter or two straight:

The ioctl() function does not fail under ssh, contrary to the assertion
made several times. Nor does $COLUMNS remain static if the window size
changes. $COLUMNS is not a property of a bash, you'll find it is set by
the readline library. $COLUMNS is not fully cross-platform, though
$MANWIDTH should be fine. Please supply counter examples if needed, but
this is pretty well traveled ground. I think the original patch is fat
and happy as far as interactive terminal use.

But I agree it's not desirable to wrap file any sort of stream output,
by default, as that would break just about any script known to mankind.
Yet wrapping is a very user-friendly default for interactive terminals.
This is potentially an irreconcilable inconsistency.

I weigh in on the side of leaving it inconsistent, but making it really
easy to force the behavior you want with something like:
\pset format wrap [auto|nnn|off]

But that leaves a big hole: what does the setting in .psqlrc refer to?
Do we need separate controls in .psql?

\pset format_terminal wrap [auto|nnn|off]
\pset format_terminal html
\pset format_stream wrap [auto|nnn|off]
\pset format_stream html

Where, on a stream, auto and off would have the same meaning, and \pset
format would set both?

Bryce Nesbitt wrote:
> But I agree it's not desirable to wrap file any sort of stream output,
> by default, as that would break just about any script known to mankind.
> Yet wrapping is a very user-friendly default for interactive terminals.
> This is potentially an irreconcilable inconsistency.
>
> I weigh in on the side of leaving it inconsistent, but making it really
> easy to force the behavior you want with something like:
> \pset format wrap [auto|nnn|off]
>
> But that leaves a big hole: what does the setting in .psqlrc refer to?
> Do we need separate controls in .psql?
>
> \pset format_terminal wrap [auto|nnn|off]
> \pset format_terminal html
> \pset format_stream wrap [auto|nnn|off]
> \pset format_stream html
>
> Where, on a stream, auto and off would have the same meaning, and \pset
> format would set both?

Hey, I can work with this idea. First, there really is no 'off' mode
for wrapped because that is aligned. What we could do is to have:

\pset format wrapped display

affect only output to the screen, using the screen width, and:

\pset format wrapped nnn

affect output to the screen and file/pipes.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

[Just when I thought I was out, they pull me back in -- argh, I'm weak]

"Bruce Momjian" <bruce(at)momjian(dot)us> writes:

> FYI, ls -C actually wraps to 72(?) unless you specify another width,

I told you exactly what ls did, at least GNU ls. It uses -w if specified, if
not then it uses the ioctl if that succeeds, if it fails it uses COLUMNS, and
if that's unavailable it uses a constant.

> one possible behavior would be for \pset wrapped to wrap to 72 for
> file/pipe unless you set \pset columns.

You can't use ls to justify having different rules for screen width for
"file/pipe":

$ COLUMNS=80 ls -C | cat
distmp3.rh3280 gconfd-stark orbit-stark purpleNMN49T ssh-WdHPsk4277

$ COLUMNS=60 ls -C | cat
distmp3.rh3280 orbit-stark ssh-WdHPsk4277
gconfd-stark purpleNMN49T

$ COLUMNS=40 ls -C | cat
distmp3.rh3280 purpleNMN49T
gconfd-stark ssh-WdHPsk4277
orbit-stark

$ COLUMNS=20 ls -C | cat
distmp3.rh3280
gconfd-stark
orbit-stark
purpleNMN49T
ssh-WdHPsk4277

> That might make the "I want it always to wrap" group happier, but not the
> "wrapped shouldn't affect file/pipe". I have not heard anyone explain why
> the later behavior is bad, especially if we default to a width of 72 rather
> than the screen width.

Presumably they're concerned that scripts which dump out data and then try to
parse it will have trouble parsing wrapped output. In any case that should be
based on whether isatty() is true, which is related to but not the same as
whether the window size ioctl succeeds.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's Slony Replication support!

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sat, Apr 26, 2008 at 5:08 PM, Bryce Nesbitt
wrote:
> Well, come to think of it, "wrapped" is not really a new output format in
> the sense of "html" or "latex". It could build on aligned:
>
> \pset format aligned [autowrap|nowrap|nnn]
>

I agree that wrapped is more a variant of aligned mode than a format
mode in its own right. Expressing it that way with \pset has the nice
bonus that you don't lose your preference for wrapped mode when
switching between aligned and unaligned with \a.

> But there's still the issue of wanting separate defaults for tty and stream
> outputs. The last thing you want is an admin deciding on wrapping, and then
> subtly breaking scripts. My personal desired defaults are:
>

Well, if we pursue Peter's suggestion that psql abstain from reading
the startup file in noninteractive mode, then this problem goes away.
An admin could opt to wrap in his interactive sessions without it ever
affecting the behaviour of scripts ... which is exactly what you would
want.

Cheers,
BJ
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)
Comment: http://getfiregpg.org

iD8DBQFIEs3x5YBsbHkuyV0RAiPsAJ0QIhWmq4s622dTZNP4MknxWTm30wCfaMTP
kY9qEW0GB3rJb3Xq5F92geY=
=GbOb
-----END PGP SIGNATURE-----

Bruce Momjian wrote:
> Hey, I can work with this idea. First, there really is no 'off' mode
> for wrapped because that is aligned...
Well, come to think of it, "wrapped" is not really a new output format
in the sense of "html" or "latex". It could build on aligned:

\pset format aligned [autowrap|nowrap|nnn]

But there's still the issue of wanting separate defaults for tty and
stream outputs. The last thing you want is an admin deciding on
wrapping, and then subtly breaking scripts. My personal desired
defaults are:

* Terminals autowrap.
* Streams don't wrap, except in the rare case when I want to force a
specific width (e.g. 79 for a newsgroup posting).

-Bryce

Gregory Stark wrote:
>
> [Just when I thought I was out, they pull me back in -- argh, I'm weak]
>
> "Bruce Momjian" <bruce(at)momjian(dot)us> writes:
>
> > FYI, ls -C actually wraps to 72(?) unless you specify another width,
>
> I told you exactly what ls did, at least GNU ls. It uses -w if specified, if
> not then it uses the ioctl if that succeeds, if it fails it uses COLUMNS, and
> if that's unavailable it uses a constant.

> $ COLUMNS=40 ls -C | cat
> distmp3.rh3280 purpleNMN49T
> gconfd-stark ssh-WdHPsk4277
> orbit-stark

I don't see that behavior here on Ubuntu 7.10:

$ COLUMNNS=120 ls -C |cat
archive cd initrd lost+found proc srv usr
basement.usr dev initrd.img media root sys var
bin etc laptop mnt rtmp tmp vmlinuz
boot home lib opt sbin u win
$ ls --version
ls (GNU coreutils) 5.97

That is not a 120 width. 'ls' seems to ignore columns for pipe output.

> > That might make the "I want it always to wrap" group happier, but not the
> > "wrapped shouldn't affect file/pipe". I have not heard anyone explain why
> > the later behavior is bad, especially if we default to a width of 72 rather
> > than the screen width.
>
> Presumably they're concerned that scripts which dump out data and then try to
> parse it will have trouble parsing wrapped output. In any case that should be
> based on whether isatty() is true, which is related to but not the same as
> whether the window size ioctl succeeds.

Right now we honor $COLUMNS only when isatty() is true.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:
> Hey, I can work with this idea. First, there really is no 'off' mode
> for wrapped because that is aligned. What we could do is to have:
>
> \pset format wrapped display
>
> affect only output to the screen, using the screen width, and:
>
> \pset format wrapped nnn
>
> affect output to the screen and file/pipes.

A new idea would be for a wrap value of zero to be special:

\pset format wrapped 0

to wrap to screen width for terminal and file/pipe output.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:
> Gregory Stark wrote:
> I don't see that behavior here on Ubuntu 7.10:
>
> $ COLUMNNS=120 ls -C |cat
> archive cd initrd lost+found proc srv usr
> basement.usr dev initrd.img media root sys var
> bin etc laptop mnt rtmp tmp vmlinuz
> boot home lib opt sbin u win
> $ ls --version
> ls (GNU coreutils) 5.97
>
> That is not a 120 width. 'ls' seems to ignore columns for pipe output.

Oops, Alvaro pointed out I typo'ed the variable name COLUMNS as
COLUMNNS. I see now that 'ls -C' does honor columns. See my later
posting about '\pset wrapped 0' as a special case where we could honor
the ioctl/COLUMNS case.

My real confusion is this:

$ echo $COLUMNS
146

$ ls -C|less
archive cd initrd lost+found proc srv usr
basement.usr dev initrd.img media root sys var
bin etc laptop mnt rtmp tmp vmlinuz
boot home lib opt sbin u win

$ COLUMNS=120 ls -C|less
archive bin cd etc initrd laptop lost+found mnt proc rtmp srv tmp usr vmlinuz
basement.usr boot dev home initrd.img lib media opt root sbin sys u var win

Why does the first 'ls' not honor columns while the second does? How
does 'ls' detect that the COLUMNS=120 is somehow different from the
default COLUMNS value?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

* Bruce Momjian <bruce(at)momjian(dot)us> [080426 09:44]:

> Why does the first 'ls' not honor columns while the second does? How
> does 'ls' detect that the COLUMNS=120 is somehow different from the
> default COLUMNS value?

I would hazard a guess that COLUMNS isn't "exported" from your
shell environment in the first case. In the other cases, the explicit:
VAR=... command
the shell is told to set VAR explicitly before starting command, in
addition to any exported vars.

a.

--
Aidan Van Dyk Create like a god,
aidan(at)highrise(dot)ca command like a king,
http://www.highrise.ca/ work like a slave.

Bruce Momjian wrote:
> Oops, Alvaro pointed out I typo'ed the variable name COLUMNS as
> COLUMNNS. I see now that 'ls -C' does honor columns. See my later
> posting about '\pset wrapped 0' as a special case where we could honor
> the ioctl/COLUMNS case.
>
> My real confusion is this:
>
> $ echo $COLUMNS
> 146
>
> $ ls -C|less
> archive cd initrd lost+found proc srv usr
> basement.usr dev initrd.img media root sys var
> bin etc laptop mnt rtmp tmp vmlinuz
> boot home lib opt sbin u win
>
> $ COLUMNS=120 ls -C|less
> archive bin cd etc initrd laptop lost+found mnt proc rtmp srv tmp usr vmlinuz
> basement.usr boot dev home initrd.img lib media opt root sbin sys u var win
>
> Why does the first 'ls' not honor columns while the second does? How
> does 'ls' detect that the COLUMNS=120 is somehow different from the
> default COLUMNS value?

Ah, I see now. $COLUMNS isn't exported to subshells, hence the previous
comment that readline needs to be called before it has a value. It
seems psql does have COLUMNS set if readline is defined, which means we
can't detect of $COLUMNS was passed to psql or was detected. More
interesting, it doesn't seem psql sets $COLUMNS in batch mode:

psql -c '\echo `echo $COLUMNS`' test
{blank line}

COLUMNS=120 sql -c '\echo `echo $COLUMNS`' test
120

so we could argue that COLUMNS should be honored but again this would
affect \g filename. The issue with 'ls' is that it knows it isn't going
to be getting new commands from the user that change where its output is
going, while psql can.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian <bruce(at)momjian(dot)us> writes:
> I don't see that behavior here on Ubuntu 7.10:

> $ COLUMNNS=120 ls -C |cat
> archive cd initrd lost+found proc srv usr
> basement.usr dev initrd.img media root sys var
> bin etc laptop mnt rtmp tmp vmlinuz
> boot home lib opt sbin u win
> $ ls --version
> ls (GNU coreutils) 5.97

> That is not a 120 width. 'ls' seems to ignore columns for pipe output.

Well, it's *certainly* gonna ignore "COLUMNNS".

regards, tom lane

Tom Lane wrote:
> Bruce Momjian <bruce(at)momjian(dot)us> writes:
>
>> I don't see that behavior here on Ubuntu 7.10:
>> $ COLUMNNS=120 ls -C |cat
>> archive cd initrd lost+found proc srv usr
>> basement.usr dev initrd.img media root sys var
>> bin etc laptop mnt rtmp tmp vmlinuz
>> boot home lib opt sbin u win
>> $ ls --version
>> ls (GNU coreutils) 5.97
>>
> Well, it's *certainly* gonna ignore "COLUMNNS".
>
> regards, tom lane
>
I'm having trouble seeing the relevance, either way. First many shells
don't set $COLUMNS at all (readline does it for psql). And most shells
that set $COLUMNS don't export it. So most people get different output
from:

# ls -C
# ls -C | cat

Unless they are in the habit of doing:

# COLUMNS=$COLUMNS ls -C |cat

I think it's too weird to default pipes to whatever the terminal width
happens to be. So that leaves you with an explicit command to set the
width for pipes.

-Bryce

Echo $MANWIDTH

"Bryce Nesbitt" <bryce2(at)obviously(dot)com> writes:

> Unless they are in the habit of doing:
>
> # COLUMNS=$COLUMNS ls -C |cat

Some of us are actually in the habit of doing that because it's easier to use
the standard interface than remembering the different command-line option for
each command. I quite often do precisely that with dpkg, for example.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's 24x7 Postgres support!

Gregory Stark wrote:
> "Bryce Nesbitt" <bryce2(at)obviously(dot)com> writes:
>
> > Unless they are in the habit of doing:
> >
> > # COLUMNS=$COLUMNS ls -C |cat
>
> Some of us are actually in the habit of doing that because it's easier to use
> the standard interface than remembering the different command-line option for
> each command. I quite often do precisely that with dpkg, for example.

Yes, this is true, but it assume the application is not going to set
$COLUMNS itself, like psql does in interactive mode:

test=> \echo `echo $COLUMNS`
127

$ sql -c '\echo `echo $COLUMNS`' test
(empty)

Now, we could get fancy and honor $COLUMNS only in non-interactive mode,
but that seems confusing.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

"Bruce Momjian" <bruce(at)momjian(dot)us> writes:

> Gregory Stark wrote:
>> "Bryce Nesbitt" <bryce2(at)obviously(dot)com> writes:
>>
>> > Unless they are in the habit of doing:
>> >
>> > # COLUMNS=$COLUMNS ls -C |cat
>>
>> Some of us are actually in the habit of doing that because it's easier to use
>> the standard interface than remembering the different command-line option for
>> each command. I quite often do precisely that with dpkg, for example.
>
> Yes, this is true, but it assume the application is not going to set
> $COLUMNS itself, like psql does in interactive mode:
>
> test=> \echo `echo $COLUMNS`
> 127
>
> $ sql -c '\echo `echo $COLUMNS`' test
> (empty)
>
> Now, we could get fancy and honor $COLUMNS only in non-interactive mode,
> but that seems confusing.

We could always read COLUMNS early on before readline is initialized and stash
the value away in a variable. But...

We would only look at COLUMNS if the ioctl for window size failed. Does
psql/readline do anything to COLUMNS in that case?

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's RemoteDBA services!

Gregory Stark wrote:
> > Now, we could get fancy and honor $COLUMNS only in non-interactive mode,
> > but that seems confusing.
>
> We could always read COLUMNS early on before readline is initialized and stash
> the value away in a variable. But...
>
> We would only look at COLUMNS if the ioctl for window size failed. Does
> psql/readline do anything to COLUMNS in that case?

We do look at COLUMNS if the ioctl() fails, but not for file/pipe
output.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

"Bruce Momjian" <bruce(at)momjian(dot)us> writes:

> Gregory Stark wrote:
>> > Now, we could get fancy and honor $COLUMNS only in non-interactive mode,
>> > but that seems confusing.
>>
>> We could always read COLUMNS early on before readline is initialized and stash
>> the value away in a variable. But...
>>
>> We would only look at COLUMNS if the ioctl for window size failed. Does
>> psql/readline do anything to COLUMNS in that case?
>
> We do look at COLUMNS if the ioctl() fails, but not for file/pipe
> output.

Yeah, it looks like your most recent patch still has the bug that if the user
specifies wrapped there are some complicated rules creating cases where it
will ignore the user's request and use un-wrapped output instead.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Get trained by Bruce Momjian - ask me about EnterpriseDB's PostgreSQL training!

Am Dienstag, 29. April 2008 schrieb Bruce Momjian:
> We do look at COLUMNS if the ioctl() fails, but not for file/pipe
> output.

This is quite a useless complication. Readline uses exactly the same ioctl()
call to determine the columns, so if ioctl() were to fail, then COLUMNS would
be unset or wrong as well.

"Peter Eisentraut" <peter_e(at)gmx(dot)net> writes:

> Am Dienstag, 29. April 2008 schrieb Bruce Momjian:
>> We do look at COLUMNS if the ioctl() fails, but not for file/pipe
>> output.
>
> This is quite a useless complication. Readline uses exactly the same ioctl()
> call to determine the columns, so if ioctl() were to fail, then COLUMNS would
> be unset or wrong as well.

COLUMNS is just a regular environment variable. The user is free to set it and
many people have dotfiles, aliases, or scripts which do just that.

Consider, for example, someone with a cron job which runs several commands
such as "ls -C", "dpkg -l", and of course "psql -Pformat=wrapped" to generate
various reports and wants it all formatted to 72 columns. They would normally
just set COLUMNS=72 and run their commands and get an email all formatted to
72 columns.

But your point is valid, that's why I'm not too worried about cases where
COLUMNS is set to the desired width but readline interferes with it. In those
cases we would be using the ioctl value anyways. It would probably still be a
good idea to getenv(COLUMNS) early on before readline is initialized though.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Get trained by Bruce Momjian - ask me about EnterpriseDB's PostgreSQL training!

Peter Eisentraut wrote:
> Am Dienstag, 29. April 2008 schrieb Bruce Momjian:
> > We do look at COLUMNS if the ioctl() fails, but not for file/pipe
> > output.
>
> This is quite a useless complication. Readline uses exactly the same ioctl()
> call to determine the columns, so if ioctl() were to fail, then COLUMNS would
> be unset or wrong as well.

I was thinking about Win32 or binaries that don't have readline.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Gregory Stark wrote:
> "Bruce Momjian" <bruce(at)momjian(dot)us> writes:
>
> > Gregory Stark wrote:
> >> > Now, we could get fancy and honor $COLUMNS only in non-interactive mode,
> >> > but that seems confusing.
> >>
> >> We could always read COLUMNS early on before readline is initialized and stash
> >> the value away in a variable. But...
> >>
> >> We would only look at COLUMNS if the ioctl for window size failed. Does
> >> psql/readline do anything to COLUMNS in that case?
> >
> > We do look at COLUMNS if the ioctl() fails, but not for file/pipe
> > output.
>
> Yeah, it looks like your most recent patch still has the bug that if the user
> specifies wrapped there are some complicated rules creating cases where it
> will ignore the user's request and use un-wrapped output instead.

Can you be more specific? You mean if the headings don't fit? Yea,
that is true. I am thinking of adding a \pset auto format to \x in
those cases, but that if for later.

Also, I thiink you could do in your .psqlrc:

\pset columns `echo $COLUMNS`

to get the behavior you want.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

"Bruce Momjian" <bruce(at)momjian(dot)us> writes:

> Gregory Stark wrote:
>> "Bruce Momjian" <bruce(at)momjian(dot)us> writes:
>>
>> > We do look at COLUMNS if the ioctl() fails, but not for file/pipe
>> > output.
>>
>> Yeah, it looks like your most recent patch still has the bug that if the user
>> specifies wrapped there are some complicated rules creating cases where it
>> will ignore the user's request and use un-wrapped output instead.
>
> Can you be more specific? You mean if the headings don't fit? Yea,
> that is true. I am thinking of adding a \pset auto format to \x in
> those cases, but that if for later.

[No I wasn't thinking of that, that's an interesting case too though I think
we might need to think a bit harder about cases that wrap poorly. If you have
long column headings we could wrap those too. But what if you have enough
space for just a few characters per column and you have long text fields in
those columns?]

I just meant the same thing I've been on about all week. Currently the
decision about whether to use wrapped mode is tied up with the decision on
what width to use and the result is that we ignore -Pformat=wrapped according
to some arcane set of rules.

The cases where we ignore the user's selected format are quite complex and not
accurately described in the documentation. They're also not accurately
described by your "not for file/pipe output" description either.

An accurate description would appear to be something like:

<quote>Wrapped</quote> is like <literal>aligned</> but wraps to a target
width of <literal>\pset columns</> or the width of the screen (unless screen
size determination fails or output has been redirected using -o or \o in
which case it is ignored and psql uses normal aligned mode unless \pset
columns is used).

It's confusing and inconsistent. I think it's better to pick a simple set of
general principles and code to that. Trying to code to presumed use cases
often ends up with code which handles corner cases poorly or inconsistently.

I think the behaviour should be simply:

format=auto
isatty(fout) ? format := wrapped : format := aligned
format=wrapped
columns := \pset columns || ioctl(fout) || getenv(COLUMNS) || 79

[Note in the above that the ioctl is on fout, not stdout!]

That would be easy to explain in the documentation as two simple consistent
rules. And as a bonus it would be consistent with other programs which use
these variables.

So the description I would code to is simply:

"Wrapped" is like aligned but wraps to \pset columns or an automatically
determined screen size. The screen size is determined automatically if output
is to a terminal which supports that, if that fails then by checking the
COLUMNS environment variable, and if that's unset then by defaulting to 79.

"Auto" selects "wrapped" format when output is a terminal and "aligned"
format otherwise.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's Slony Replication support!

Am Dienstag, 29. April 2008 schrieb Bruce Momjian:
> Peter Eisentraut wrote:
> > Am Dienstag, 29. April 2008 schrieb Bruce Momjian:
> > > We do look at COLUMNS if the ioctl() fails, but not for file/pipe
> > > output.
> >
> > This is quite a useless complication. Readline uses exactly the same
> > ioctl() call to determine the columns, so if ioctl() were to fail, then
> > COLUMNS would be unset or wrong as well.
>
> I was thinking about Win32 or binaries that don't have readline.

These rules don't seem very consistent. You are mixing platform dependencies,
build options, theoretical, unproven failures of kernel calls, none of which
have anything to do with each other. For example, if readline weren't
installed, then there would be no one who sets COLUMNS, so why look at it?
If you want to allow users to set COLUMNS manually (possibly useful, see Greg
Stark's arguments), then it should have priority over ioctl(), not the other
way around.

Peter Eisentraut wrote:
> Am Dienstag, 29. April 2008 schrieb Bruce Momjian:
> > Peter Eisentraut wrote:
> > > Am Dienstag, 29. April 2008 schrieb Bruce Momjian:
> > > > We do look at COLUMNS if the ioctl() fails, but not for file/pipe
> > > > output.
> > >
> > > This is quite a useless complication. Readline uses exactly the same
> > > ioctl() call to determine the columns, so if ioctl() were to fail, then
> > > COLUMNS would be unset or wrong as well.
> >
> > I was thinking about Win32 or binaries that don't have readline.
>
> These rules don't seem very consistent. You are mixing platform dependencies,
> build options, theoretical, unproven failures of kernel calls, none of which
> have anything to do with each other. For example, if readline weren't
> installed, then there would be no one who sets COLUMNS, so why look at it?
> If you want to allow users to set COLUMNS manually (possibly useful, see Greg
> Stark's arguments), then it should have priority over ioctl(), not the other
> way around.

OK, two people like it, no one has objected. :-) I will work on making
those changes. Thanks.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

"Bruce Momjian" <bruce(at)momjian(dot)us> writes:

> Peter Eisentraut wrote:
>> Am Dienstag, 29. April 2008 schrieb Bruce Momjian:
>> > Peter Eisentraut wrote:
>> > > Am Dienstag, 29. April 2008 schrieb Bruce Momjian:
>> > > > We do look at COLUMNS if the ioctl() fails, but not for file/pipe
>> > > > output.
>> > >
>> > > This is quite a useless complication. Readline uses exactly the same
>> > > ioctl() call to determine the columns, so if ioctl() were to fail, then
>> > > COLUMNS would be unset or wrong as well.
>> >
>> > I was thinking about Win32 or binaries that don't have readline.
>>
>> These rules don't seem very consistent. You are mixing platform dependencies,
>> build options, theoretical, unproven failures of kernel calls, none of which
>> have anything to do with each other. For example, if readline weren't
>> installed, then there would be no one who sets COLUMNS, so why look at it?
>> If you want to allow users to set COLUMNS manually (possibly useful, see Greg
>> Stark's arguments), then it should have priority over ioctl(), not the other
>> way around.
>
> OK, two people like it, no one has objected. :-) I will work on making
> those changes. Thanks.

Uh, precisely what changes are you referring to now? "These rules don't seem
very consistent" doesn't sound like "no one has objected" to me.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's PostGIS support!

This patch is blocking other work -- for instance, the PrintTable API
and two patches that depend on it.

Could we get the main hunks committed soon, with the policy bits
stripped out? That way, discussion on the behavior can continue until
we reach an agreement, and others can work on other patches.

("Policy bits stripped out" could mean selecting the wrapped behavior
only explicitly and selecting the column width only on a \pset variable.
All the hard bits about when to use $COLUMNS and the ioctl(), which are
the controversial parts, can be deferred and are pretty localized
changes AFAICS.)

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
> This patch is blocking other work -- for instance, the PrintTable API
> and two patches that depend on it.

> Could we get the main hunks committed soon, with the policy bits
> stripped out? That way, discussion on the behavior can continue until
> we reach an agreement, and others can work on other patches.

This patch seems sufficiently controversial that "commit now" is the
very last thing that should happen to it.

I suggest committing the PrintTable stuff and not worrying about whether
that breaks the wrap patch.

regards, tom lane

Tom Lane wrote:
> This patch seems sufficiently controversial that "commit now" is the
> very last thing that should happen to it.
>
> I suggest committing the PrintTable stuff and not worrying about whether
> that breaks the wrap patch.
>
> regards, tom lane\
>
AFIK, the only thing that's controversial about the patch is how to turn
it on and off -- the actual core code appears to be inflaming no
passions. And it's the core code that presents merge issues.

Could it be committed with a hidden enable syntax, for the interim? Or
even no enable syntax, just to have it in the code base?

\pset format wrap-beta-test on

Bryce Nesbitt <bryce2(at)obviously(dot)com> writes:
> AFIK, the only thing that's controversial about the patch is how to turn
> it on and off -- the actual core code appears to be inflaming no
> passions. And it's the core code that presents merge issues.

Well, personally I haven't read the core code yet, since it's not commit
fest yet ;-). I don't know whether there are any issues there, but
it wouldn't surprise me given the number of issues in the control code.

regards, tom lane

Gregory Stark wrote:
> [No I wasn't thinking of that, that's an interesting case too though I think
> we might need to think a bit harder about cases that wrap poorly. If you have
> long column headings we could wrap those too. But what if you have enough
> space for just a few characters per column and you have long text fields in
> those columns?]
I've been using this patch for months.

After the first week, I switched to the behavior above (the wrap code
gives up at a certain point). It is a lot better, try it for yourself.
If you try to cram too much stuff onto a line it's going to become an
even more unreadable mess of very very tall columns.

Wrapping column headings only gets you so far, delaying the inevitable a
little bit. A patch to squeeze out extra header space, or abbreviate
headers would be an advance.
-Bryce

\pset format aligned autowrap
\pset format aligned 80

Gregory Stark wrote:
> [No I wasn't thinking of that, that's an interesting case too though I think
> we might need to think a bit harder about cases that wrap poorly. If you have
> long column headings we could wrap those too. But what if you have enough
> space for just a few characters per column and you have long text fields in
> those columns?]

I've been using this patch for months.

After the first week, I switched to the behavior above (the wrap code
gives up at a certain point). It is a lot better, try it for yourself.
If you try to cram too much stuff onto a line it's going to become an
even more unreadable mess of very very tall columns.

Wrapping column headings only gets you so far, delaying the inevitable a
little bit. A patch to squeeze out extra header space, or abbreviate
headers would be an advance.
-Bryce

\pset format aligned autowrap
\pset format aligned 80

Tom Lane wrote:
> Well, personally I haven't read the core code yet, since it's not commit fest yet ;-). I don't know whether there are any issues there, but it wouldn't surprise me given the number of issues in the control code.
>
> regards, tom lane
>

I'm biased because I wrote the patch. However -- I think the core is
commitable (it won't break anything else, or have complex dependencies
or bloat).

The issue of $COLUMNS was identified as unresolved at the time the patch
was submitted, and I think it's the only bit in serious question.

Bryce Nesbitt wrote:
> Tom Lane wrote:
> > Well, personally I haven't read the core code yet, since it's not commit fest yet ;-). I don't know whether there are any issues there, but it wouldn't surprise me given the number of issues in the control code.
> >
> > regards, tom lane
> >
>
> I'm biased because I wrote the patch. However -- I think the core is
> commitable (it won't break anything else, or have complex dependencies
> or bloat).
>
> The issue of $COLUMNS was identified as unresolved at the time the patch
> was submitted, and I think it's the only bit in serious question.

This code is complicated. There is no need to rush and have to revisit
it later.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:
> Peter Eisentraut wrote:
> > Am Dienstag, 29. April 2008 schrieb Bruce Momjian:
> > > Peter Eisentraut wrote:
> > > > Am Dienstag, 29. April 2008 schrieb Bruce Momjian:
> > > > > We do look at COLUMNS if the ioctl() fails, but not for file/pipe
> > > > > output.
> > > >
> > > > This is quite a useless complication. Readline uses exactly the same
> > > > ioctl() call to determine the columns, so if ioctl() were to fail, then
> > > > COLUMNS would be unset or wrong as well.
> > >
> > > I was thinking about Win32 or binaries that don't have readline.
> >
> > These rules don't seem very consistent. You are mixing platform dependencies,
> > build options, theoretical, unproven failures of kernel calls, none of which
> > have anything to do with each other. For example, if readline weren't
> > installed, then there would be no one who sets COLUMNS, so why look at it?
> > If you want to allow users to set COLUMNS manually (possibly useful, see Greg
> > Stark's arguments), then it should have priority over ioctl(), not the other
> > way around.
>
> OK, two people like it, no one has objected. :-) I will work on making
> those changes. Thanks.

OK, so COLUMNS should take precedence. I assume this is going to
require us to read the COLUMNS enviroment variable in psql _before_
readline sets it, and that COLUMNS will only affect screen output, like
ioctl(). Is that consistent?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

> OK, so COLUMNS should take precedence. I assume this is going to
> require us to read the COLUMNS enviroment variable in psql _before_
> readline sets it, and that COLUMNS will only affect screen output, like
> ioctl(). Is that consistent?
>
This whole thing is confusing enough at the point, I think a complete
proposal needs to be articulated. It is hard to comment on a fragment of
an idea.

The main cases to cover are: (1) how to specify wrap for tty's (2) how
to specify wrap for pipes (3) how to get wraped on platforms that don't
have the ioctl (presumably windows without cygwin) (4) how to set up
different defaults for tty's and pipes (e.g. wrap interactive tty's, but
leave output aligned for scripts).

And perhaps, as a bonus comment on (5) the idea of having psql NOT
source .psqlrc

I hope at some point someone will actually try the actual core wrapping
code, and comment on it.

-Bryce

"Bruce Momjian" <bruce(at)momjian(dot)us> writes:

> OK, so COLUMNS should take precedence. I assume this is going to
> require us to read the COLUMNS enviroment variable in psql _before_
> readline sets it, and that COLUMNS will only affect screen output, like
> ioctl(). Is that consistent?

What are you talking about? "screen output"?

Are you even getting my emails? Please confirm that you receive this email.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's On-Demand Production Tuning

Bryce Nesbitt wrote:
>
> > OK, so COLUMNS should take precedence. I assume this is going to
> > require us to read the COLUMNS environment variable in psql _before_
> > readline sets it, and that COLUMNS will only affect screen output, like
> > ioctl(). Is that consistent?
> >
> This whole thing is confusing enough at the point, I think a complete
> proposal needs to be articulated. It is hard to comment on a fragment of
> an idea.
>
> The main cases to cover are: (1) how to specify wrap for tty's

In order of precedence:

\pset columns
$COLUMNS
iotcl()

> (2) how to specify wrap for pipes

\pset columns

> (3) how to get wrapped on platforms that don't
> have the ioctl (presumably windows without cygwin)

\pset columns
$COLUMNS

> (4) how to set up
> different defaults for tty's and pipes (e.g. wrap interactive tty's, but
> leave output aligned for scripts).
>
> And perhaps, as a bonus comment on (5) the idea of having psql NOT
> source .psqlrc

-X
--no-psqlrc

> I hope at some point someone will actually try the actual core wrapping
> code, and comment on it.

I tested it and it worked well once I modified it.

Updated patch with clearer documentation that matches the above
behavior:

ftp://momjian.us/pub/postgresql/mypatches/wrap

FYI, I looked into 'ls -C' hanlding a little more and ls (GNU coreutils)
5.97 honors COLUMNS _only_ in file/pipe output, not for screen output.
What the C code does is to read COLUMNS, then overwrite that value with
ioctl() if it works.

Do we want to follow that behavior? ls has a '-w' option to specify the
width, like our \pset columns. However, the manual page seems backwards:

-w, --width=COLS
assume screen width instead of current value

The GNU 'ls' manual does not mention COLUMNS.

BSD 'ls' used COLUMNS only when outputing to the screen, and ioctl
fails. However, the BSD manual says:

COLUMNS If this variable contains a string representing a decimal
integer, it is used as the column position width for
displaying multiple-text-column output. The ls utility
calculates how many pathname text columns to display based
on the width pro- vided. (See -C.)

Again, I think the manual is wrong.

So it seem GNU ls and BSD ls are inconsistent, which I think means we
should design our API the best we can, rather than rely on how others
interpret COLUMNS.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Gregory Stark wrote:
> "Bruce Momjian" <bruce(at)momjian(dot)us> writes:
>
> > OK, so COLUMNS should take precedence. I assume this is going to
> > require us to read the COLUMNS enviroment variable in psql _before_
> > readline sets it, and that COLUMNS will only affect screen output, like
> > ioctl(). Is that consistent?
>
> What are you talking about? "screen output"?
>
> Are you even getting my emails? Please confirm that you receive this email.

Yes, I am getting your emails, but I have more than you to please.

Are others OK with $COLUMNS controlling screen output and file/pipe, or
perhaps COLUMNS controlling only file/pipe, as GNU ls does? I have
heard a few people say they only want \pset columns to control
file/pipe.

I have outlined the GNU ls behavior in a email I just sent.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:
> Gregory Stark wrote:
> > "Bruce Momjian" <bruce(at)momjian(dot)us> writes:
> >
> > > OK, so COLUMNS should take precedence. I assume this is going to
> > > require us to read the COLUMNS enviroment variable in psql _before_
> > > readline sets it, and that COLUMNS will only affect screen output, like
> > > ioctl(). Is that consistent?
> >
> > What are you talking about? "screen output"?
> >
> > Are you even getting my emails? Please confirm that you receive this email.
>
> Yes, I am getting your emails, but I have more than you to please.
>
> Are others OK with $COLUMNS controlling screen output and file/pipe, or
> perhaps COLUMNS controlling only file/pipe, as GNU ls does? I have
> heard a few people say they only want \pset columns to control
> file/pipe.
>
> I have outlined the GNU ls behavior in a email I just sent.

I talked to Greg on IM and he had a new idea, related to my 'auto' idea
of using aligned/wrapped/expanded mode automatically.

His idea is that 'auto' mode (aligned/wrapped/expanded) is only for
output to the screen, and 'wrapped' is always wrapped. 'Wrapped' would
be affected by \pset columns, COLUMNS, or the terminal width for output
to screen, file, or pipe. If no width could be derrmined for wrapped,
it would default to 72.

This gives us 'wrapped' that always wraps (if it can fit the headings),
and most people will use 'auto' in their psqlrc, and it affects only
terminal output.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian <bruce(at)momjian(dot)us> writes:
> Are others OK with $COLUMNS controlling screen output and file/pipe, or
> perhaps COLUMNS controlling only file/pipe, as GNU ls does? I have
> heard a few people say they only want \pset columns to control
> file/pipe.

I agree with the latter. Anyone who is setting COLUMNS is going to set
it to reflect the *screen* width they want; it has zero to do with
what should happen for output going somewhere else than the screen.

regards, tom lane

Tom Lane wrote:
> Bruce Momjian <bruce(at)momjian(dot)us> writes:
> > Are others OK with $COLUMNS controlling screen output and file/pipe, or
> > perhaps COLUMNS controlling only file/pipe, as GNU ls does? I have
> > heard a few people say they only want \pset columns to control
> > file/pipe.
>
> I agree with the latter. Anyone who is setting COLUMNS is going to set
> it to reflect the *screen* width they want; it has zero to do with
> what should happen for output going somewhere else than the screen.

OK, that's what I thought, and what I think Peter wants, and what the
posted patch does.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

"Bruce Momjian" <bruce(at)momjian(dot)us> writes:

> FYI, I looked into 'ls -C' hanlding a little more and ls (GNU coreutils)
> 5.97 honors COLUMNS _only_ in file/pipe output, not for screen output.
> What the C code does is to read COLUMNS, then overwrite that value with
> ioctl() if it works.

Well saying "file/pipe" and "screen output" is making a lot of assumptions.
The pipe may still be ending up on the screen if it's being piped to a pager.
And a "file" could still be a terminal. Also, not all terminals support the
ioctl().

What ls does is use the same logic for all cases. When the ioctl works that
takes precedence regardless of whether it's "file/pipe" or "screen output".
When it fails it uses COLUMNS instead if set.

We really must stop thinking in terms of specific use cases. You will tie
yourselves in knots if you try to enumerate all the types of terminals, places
people redirect output, and uses people might make for the output. And it will
result in very complex, hard to explain behaviour which will surprise and
disappoint users.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's 24x7 Postgres support!

"Tom Lane" <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:

> Bruce Momjian <bruce(at)momjian(dot)us> writes:
>> Are others OK with $COLUMNS controlling screen output and file/pipe, or
>> perhaps COLUMNS controlling only file/pipe, as GNU ls does? I have
>> heard a few people say they only want \pset columns to control
>> file/pipe.
>
> I agree with the latter. Anyone who is setting COLUMNS is going to set
> it to reflect the *screen* width they want; it has zero to do with
> what should happen for output going somewhere else than the screen.

That's just not true. Consider someone writing a cron job script who wants
their output formatted to 72 columns. They would set COLUMNS to direct all the
programs in the cron job script and be annoyed if some of them ignored it.
Consider also an application like pgadmin which might want to run a script in
a window and control how wide the output is.

I think you need a stronger reason to disregard the user's explicit settings
than assuming that you know what the user's planning to do with the resulting
data. Just because the output has been redirected doesn't mean it wasn't
redirected to something like "more" or "head" or whatever with the result
*still* showing up on the screen.

It's one thing to change the default behaviour. It's another to disregard
user-requested options.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's 24x7 Postgres support!

Gregory Stark wrote:
> "Tom Lane" <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:
>
> > Bruce Momjian <bruce(at)momjian(dot)us> writes:
> >> Are others OK with $COLUMNS controlling screen output and file/pipe, or
> >> perhaps COLUMNS controlling only file/pipe, as GNU ls does? I have
> >> heard a few people say they only want \pset columns to control
> >> file/pipe.
> >
> > I agree with the latter. Anyone who is setting COLUMNS is going to set
> > it to reflect the *screen* width they want; it has zero to do with
> > what should happen for output going somewhere else than the screen.
>
> That's just not true. Consider someone writing a cron job script who wants
> their output formatted to 72 columns. They would set COLUMNS to direct all the
> programs in the cron job script and be annoyed if some of them ignored it.
> Consider also an application like pgadmin which might want to run a script in
> a window and control how wide the output is.

Well, if they are surprised, they are easily surprised. :-)

What logic is there that GNU ls honors COLUMNS only in non-terminal
output? And the use of COLUMNS isn't even documented in the GNU ls
manual page. And BSD ls honors COLUMNS only for terminal output when
the ioctl fails(). It is hard to see that there is some major
expectation of how COLUMNS should behave that would make our usage
"surprising".

If the user wants to set the wrap width for all output, they have to use
\pset columns.

Also, because we run on some platforms that don't have that ioctl(), we
are using COLUMNS as a way of providing the width only for screen
output, like iotcl() does.

> I think you need a stronger reason to disregard the user's explicit settings
> than assuming that you know what the user's planning to do with the resulting
> data. Just because the output has been redirected doesn't mean it wasn't
> redirected to something like "more" or "head" or whatever with the result
> *still* showing up on the screen.
>
> It's one thing to change the default behaviour. It's another to disregard
> user-requested options.

We are at least correctly documenting our behavior, which is more than
the two 'ls' versions I saw did. If users want to set something, they
better consult the documentation to find out how to do it.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Gregory Stark wrote:
> "Bruce Momjian" <bruce(at)momjian(dot)us> writes:
>
> > FYI, I looked into 'ls -C' hanlding a little more and ls (GNU coreutils)
> > 5.97 honors COLUMNS _only_ in file/pipe output, not for screen output.
> > What the C code does is to read COLUMNS, then overwrite that value with
> > ioctl() if it works.
>
> Well saying "file/pipe" and "screen output" is making a lot of assumptions.
> The pipe may still be ending up on the screen if it's being piped to a pager.
> And a "file" could still be a terminal. Also, not all terminals support the
> ioctl().

True, but I can't think of a cleaner way to specify it, and I haven't
see wording that is clearer. I think most people understand this
distinction. If they don't they will learn soon enough when they try it.

> What ls does is use the same logic for all cases. When the ioctl works that
> takes precedence regardless of whether it's "file/pipe" or "screen output".
> When it fails it uses COLUMNS instead if set.
>
> We really must stop thinking in terms of specific use cases. You will tie
> yourselves in knots if you try to enumerate all the types of terminals, places
> people redirect output, and uses people might make for the output. And it will
> result in very complex, hard to explain behaviour which will surprise and
> disappoint users.

I think we need to focus on usability. Using consistent logic is nice,
but if it doesn't do what most people would want, then we have to adjust.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

* Bruce Momjian <bruce(at)momjian(dot)us> [080506 10:56]:

> What logic is there that GNU ls honors COLUMNS only in non-terminal
> output? And the use of COLUMNS isn't even documented in the GNU ls
> manual page. And BSD ls honors COLUMNS only for terminal output when
> the ioctl fails(). It is hard to see that there is some major
> expectation of how COLUMNS should behave that would make our usage
> "surprising".
>
> If the user wants to set the wrap width for all output, they have to use
> \pset columns.
>
> Also, because we run on some platforms that don't have that ioctl(), we
> are using COLUMNS as a way of providing the width only for screen
> output, like iotcl() does.

I have to admit to using the COLUMNS=... <command> trick myself.

I do have COLUMNS exported in my terminal, and often to stuff like:

ls -C | less

and I expect it to wrap at $COLUMNS (my terminal width) in my pager.

And since the GNU coreutils is pretty consistent in this regard, I often
export COLUMNS=<xxx> in scripts for cron jobs, reports, etc, to get
output that formats nicely for emails, etc.

If I ever wanted the psql wrapped format, I guess *I* would hope that
psql would work that way, simply because that's the way the other tools
I use do it. But I can see that if the other tools work differently,
then there is going to have to be some people (maybe everybody) always
double-checking the psql man page to find out again how it formats.

But since I'm not a user wanting the wrapped format, don't cater to my
hypothetical wants.

But one of the interesting things is that psql has an is *interactive*
mode (something the GNU utils don't have to worry about). So *when* you
choose to figure out your columns is important, and really impacts
behaviour too.

For instance, if I was doing a query, I often to it interactively first:
SELECT [...] FROM [....] LIMIT 50;
And when I'm sure I have the right values,expressions, column aliases,
etc, I do:
\o /tmp/output
SELECT [...] FROM [...];
\o
And in this case, I would expect that /tmp/output would have identical
formatting to the LIMITed query I just ran interactively, not matter
what setting I had for format/wrapped/auto/$COLUMNS.

> We are at least correctly documenting our behavior, which is more than
> the two 'ls' versions I saw did. If users want to set something, they
> better consult the documentation to find out how to do it.

Correctly documenting it is good, just as the version of GNU ls I have
on Debian, where I see:

`-1'
`--format=single-column'
List one file per line. This is the default for `ls' when standard
output is not a terminal.

`-C'
`--format=vertical'
List files in columns, sorted vertically. This is the default for
`ls' if standard output is a terminal. It is always the default
for the `dir' program. GNU `ls' uses variable width columns to
display as many files as possible in the fewest lines.

`-w'
`--width=COLS'
Assume the screen is COLS columns wide. The default is taken from
the terminal settings if possible; otherwise the environment
variable `COLUMNS' is used if it is set; otherwise the default is
80.

`--color [=WHEN]'
Specify whether to use color for distinguishing file types. WHEN
may be omitted, or one of:
* none - Do not use color at all. This is the default.
* auto - Only use color if standard output is a terminal.
* always - Always use color.
Specifying `--color' and no WHEN is equivalent to `--color=always'.
Piping a colorized listing through a pager like `more' or `less'
usually produces unreadable results. However, using `more -f'
does seem to work.

That's pretty straight forward, pretty explicit, and matches the
description of what Greg has been saying all along.

a.

--
Aidan Van Dyk Create like a god,
aidan(at)highrise(dot)ca command like a king,
http://www.highrise.ca/ work like a slave.

Aidan Van Dyk <aidan(at)highrise(dot)ca> writes:
> But one of the interesting things is that psql has an is *interactive*
> mode (something the GNU utils don't have to worry about). So *when* you
> choose to figure out your columns is important, and really impacts
> behaviour too.

Well, COLUMNS has no hope of tracking on-the-fly changes of window size,
which is why the ioctl should take precedence over it.

> Correctly documenting it is good, just as the version of GNU ls I have
> on Debian, where I see:
> `-w'
> `--width=COLS'
> Assume the screen is COLS columns wide. The default is taken from
> the terminal settings if possible; otherwise the environment
> variable `COLUMNS' is used if it is set; otherwise the default is
> 80.

Fedora 8 has the same wording in "info ls". Possibly Bruce was reading
the man page, which is not as complete (and admits it).

Experimentation on OS X (BSD clone) shows that its "ls" believes COLUMNS
in preference to ioctl, so there's clearly scope for argument; but on
the whole I think we should follow the GNU behavior. The BSD version's
behavior is full of enormous amounts of historical cruft (and its man
page admits that) --- I suspect the behavior on this point "just grew"
instead of being carefully thought about.

regards, tom lane

Aidan Van Dyk wrote:
> I have to admit to using the COLUMNS=... <command> trick myself.
>
> I do have COLUMNS exported in my terminal, and often to stuff like:
>
> ls -C | less
>
> and I expect it to wrap at $COLUMNS (my terminal width) in my pager.
>
> And since the GNU coreutils is pretty consistent in this regard, I often
> export COLUMNS=<xxx> in scripts for cron jobs, reports, etc, to get
> output that formats nicely for emails, etc.

Interesting.

> If I ever wanted the psql wrapped format, I guess *I* would hope that
> psql would work that way, simply because that's the way the other tools
> I use do it. But I can see that if the other tools work differently,
> then there is going to have to be some people (maybe everybody) always
> double-checking the psql man page to find out again how it formats.

True.

> But since I'm not a user wanting the wrapped format, don't cater to my
> hypothetical wants.
>
> But one of the interesting things is that psql has an is *interactive*
> mode (something the GNU utils don't have to worry about). So *when* you
> choose to figure out your columns is important, and really impacts
> behaviour too.
>
> For instance, if I was doing a query, I often to it interactively first:
> SELECT [...] FROM [....] LIMIT 50;
> And when I'm sure I have the right values,expressions, column aliases,
> etc, I do:
> \o /tmp/output
> SELECT [...] FROM [...];
> \o
> And in this case, I would expect that /tmp/output would have identical
> formatting to the LIMITed query I just ran interactively, not matter
> what setting I had for format/wrapped/auto/$COLUMNS.

The only thing we could do there perhaps is to have psql wrap file
output to the terminal width if outputting to a pipe/file, but only from
an interactive session, but that is just too odd. If we make psql too
automatic it will be hard to explain and have more surprises.

> `-w'
> `--width=COLS'
> Assume the screen is COLS columns wide. The default is taken from
> the terminal settings if possible; otherwise the environment
> variable `COLUMNS' is used if it is set; otherwise the default is
> 80.

I just looked at coreutils-6.9 and 5.97 and neither manual has a mention
of COLUMNS. Seems this is some Debian manual addition or something. I
don't see it on Ubuntu 7.10 either.

> That's pretty straight forward, pretty explicit, and matches the
> description of what Greg has been saying all along.

Agreed. The problem is I have a pretty even split on how this feature
should behave so we have go to with the majority and adjust as we get
feedback from the field.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Am Dienstag, 6. Mai 2008 schrieb Tom Lane:
> Well, COLUMNS has no hope of tracking on-the-fly changes of window size,
> which is why the ioctl should take precedence over it.

Readline changes the value of COLUMNS on the fly.

Tom Lane wrote:
> Aidan Van Dyk <aidan(at)highrise(dot)ca> writes:
> > But one of the interesting things is that psql has an is *interactive*
> > mode (something the GNU utils don't have to worry about). So *when* you
> > choose to figure out your columns is important, and really impacts
> > behaviour too.
>
> Well, COLUMNS has no hope of tracking on-the-fly changes of window size,
> which is why the ioctl should take precedence over it.

True, but Peter wanted COLUMNS to be honored over ioctl():

http://archives.postgresql.org/pgsql-hackers/2008-04/msg01853.php

> > Correctly documenting it is good, just as the version of GNU ls I have
> > on Debian, where I see:
> > `-w'
> > `--width=COLS'
> > Assume the screen is COLS columns wide. The default is taken from
> > the terminal settings if possible; otherwise the environment
> > variable `COLUMNS' is used if it is set; otherwise the default is
> > 80.
>
> Fedora 8 has the same wording in "info ls". Possibly Bruce was reading
> the man page, which is not as complete (and admits it).

Oh, you are looking at info. That makes sense now.

> Experimentation on OS X (BSD clone) shows that its "ls" believes COLUMNS
> in preference to ioctl, so there's clearly scope for argument; but on
> the whole I think we should follow the GNU behavior. The BSD version's
> behavior is full of enormous amounts of historical cruft (and its man
> page admits that) --- I suspect the behavior on this point "just grew"
> instead of being carefully thought about.

Well, the GNU 'ls' behavior, because it does not check for isatty(),
will use COLUMNS mostly for file/pipe output because those are the cases
where ioctl() fails. Is that what you want?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian <bruce(at)momjian(dot)us> writes:
> I just looked at coreutils-6.9 and 5.97 and neither manual has a mention
> of COLUMNS. Seems this is some Debian manual addition or something. I
> don't see it on Ubuntu 7.10 either.

You're looking in the wrong place. See "info ls".

regards, tom lane

Peter Eisentraut wrote:
> Am Dienstag, 6. Mai 2008 schrieb Tom Lane:
> > Well, COLUMNS has no hope of tracking on-the-fly changes of window size,
> > which is why the ioctl should take precedence over it.
>
> Readline changes the value of COLUMNS on the fly.

Yes, but my patch grabs COLUMNS before we call readline(), so we use the
COLUMN value we were invoked with, rather than tracking the changes
readline() makes to it.

If we use COLUMNS as set by readline, there is no way to pass a COLUMNS
value into psql reliably.

'ls' doesn't use readline so it is safe from changes while it is doing
its output.

I can pull COLUMNS as modified by readline() but I thought no one wanted
that.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> Am Dienstag, 6. Mai 2008 schrieb Tom Lane:
>> Well, COLUMNS has no hope of tracking on-the-fly changes of window size,
>> which is why the ioctl should take precedence over it.

> Readline changes the value of COLUMNS on the fly.

... from the ioctl's results, presumably, so what you are saying is that
if readline is active then this discussion is all moot.

In any case I'd be happier if our behavior on this point was not
dependent on whether readline is built in. A normal person would expect
readline to affect only input behavior, not output behavior.

regards, tom lane

* Bruce Momjian <bruce(at)momjian(dot)us> [080506 11:59]:

> > But one of the interesting things is that psql has an is *interactive*
> > mode (something the GNU utils don't have to worry about). So *when* you
> > choose to figure out your columns is important, and really impacts
> > behaviour too.
> >
> > For instance, if I was doing a query, I often to it interactively first:
> > SELECT [...] FROM [....] LIMIT 50;
> > And when I'm sure I have the right values,expressions, column aliases,
> > etc, I do:
> > \o /tmp/output
> > SELECT [...] FROM [...];
> > \o
> > And in this case, I would expect that /tmp/output would have identical
> > formatting to the LIMITed query I just ran interactively, not matter
> > what setting I had for format/wrapped/auto/$COLUMNS.
>
> The only thing we could do there perhaps is to have psql wrap file
> output to the terminal width if outputting to a pipe/file, but only from
> an interactive session, but that is just too odd. If we make psql too
> automatic it will be hard to explain and have more surprises.

Yes, and *this* is the tough part. And *I* think that the control of
wrapping/width should be based on psql's output fd/$COLUMNS, not
necessarily the query output buffer fd, because I want the interactive
output to be identical when I run the query and let psql automatically
pipe it through $PAGER and when I run the query and tell psql \o it to
pipe it somewhere specific.

Of course, all over-ridden by some specific \pset to make it all more
complicated ;-)

> I just looked at coreutils-6.9 and 5.97 and neither manual has a mention
> of COLUMNS. Seems this is some Debian manual addition or something. I
> don't see it on Ubuntu 7.10 either.

Yes, the man pages point to the Texinfo manual for full documentation:
http://www.gnu.org/software/coreutils/manual/html_node/General-output-formatting.html#General-output-formatting

a.

--
Aidan Van Dyk Create like a god,
aidan(at)highrise(dot)ca command like a king,
http://www.highrise.ca/ work like a slave.

Bruce Momjian wrote:
> Updated patch with clearer documentation that matches the above
> behavior:
>
> ftp://momjian.us/pub/postgresql/mypatches/wrap

I found a bug in my patch, particularly related to wrapping to pipes.
Turns out if psql uses the pager internally:

\pset format wrapped

SELECT 1, 2, repeat('a', 80), repeat('b', 80), E'a\nb\nc\nd', 1
FROM generate_series(1,50);

it does not wrap to the screen width because of our default behavior of
not wrapping pipe output by default. I had to add an is_pager boolean
parameter to print_aligned_text(). (I tried passing is_pager via
printTableOpt but it is a const.)

I have updated the ftp URL to fix this.

One item not addressed is that the 'wrapped' format wrapping could force
output off the page without the pager being used if it is _near_ a full
screen before wrapping. Of course we don't consider additional lines
that wrap by extending past the right margin for non-wrapped mode, so
perhaps it is OK.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Patch applied (yea!).

---------------------------------------------------------------------------

Bruce Momjian wrote:
> Bruce Momjian wrote:
> > Updated patch with clearer documentation that matches the above
> > behavior:
> >
> > ftp://momjian.us/pub/postgresql/mypatches/wrap
>
> I found a bug in my patch, particularly related to wrapping to pipes.
> Turns out if psql uses the pager internally:
>
> \pset format wrapped
>
> SELECT 1, 2, repeat('a', 80), repeat('b', 80), E'a\nb\nc\nd', 1
> FROM generate_series(1,50);
>
> it does not wrap to the screen width because of our default behavior of
> not wrapping pipe output by default. I had to add an is_pager boolean
> parameter to print_aligned_text(). (I tried passing is_pager via
> printTableOpt but it is a const.)
>
> I have updated the ftp URL to fix this.
>
> One item not addressed is that the 'wrapped' format wrapping could force
> output off the page without the pager being used if it is _near_ a full
> screen before wrapping. Of course we don't consider additional lines
> that wrap by extending past the right margin for non-wrapped mode, so
> perhaps it is OK.
>
> --
> Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
> EnterpriseDB http://enterprisedb.com
>
> + If your life is a hard drive, Christ can be your backup. +
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +