Re: libedit memory stomp is apparently fixed in OS X Lion

We've had several complaints about that tab-completion bug in the
Apple-supplied version of libedit, most recently here:
http://archives.postgresql.org/pgsql-bugs/2011-06/msg00119.php

I had a bug filed with Apple about that, and today I got some auto-mail
indicating they'd fixed that bug as of OS X 10.7 (Lion). I don't have
Lion installed here, but I grabbed the libedit sources from
www.opensource.apple.com and indeed it looks fixed. So, if any early
adopters want to try it out ...

regards, tom lane

On 22 July 2011 03:24, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> I had a bug filed with Apple about that, and today I got some auto-mail
> indicating they'd fixed that bug as of OS X 10.7 (Lion).  I don't have
> Lion installed here, but I grabbed the libedit sources from
> www.opensource.apple.com and indeed it looks fixed.  So, if any early
> adopters want to try it out ...

I'll add that I've heard reports that Lion Server comes with
PostgreSQL as standard, and Lion Desktop comes with psql.

--
Peter Geoghegan       http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services

On Fri, Jul 22, 2011 at 5:01 PM, Peter Geoghegan <peter(at)2ndquadrant(dot)com> wrote:
> On 22 July 2011 03:24, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> I had a bug filed with Apple about that, and today I got some auto-mail
>> indicating they'd fixed that bug as of OS X 10.7 (Lion).  I don't have
>> Lion installed here, but I grabbed the libedit sources from
>> www.opensource.apple.com and indeed it looks fixed.  So, if any early
>> adopters want to try it out ...
>
> I'll add that I've heard reports that Lion Server comes with
> PostgreSQL as standard, and Lion Desktop comes with psql.

Yup:

raptor:PEM dpage$ uname -a
Darwin raptor.local 11.0.0 Darwin Kernel Version 11.0.0: Sat Jun 18
12:56:35 PDT 2011; root:xnu-1699.22.73~1/RELEASE_X86_64 x86_64
raptor:PEM dpage$ which psql
/usr/bin/psql
raptor:PEM dpage$ psql --version
psql (PostgreSQL) 9.0.4
contains support for command-line editing

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Dave Page <dpage(at)pgadmin(dot)org> writes:
> On Fri, Jul 22, 2011 at 5:01 PM, Peter Geoghegan <peter(at)2ndquadrant(dot)com> wrote:
>> On 22 July 2011 03:24, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> I had a bug filed with Apple about that, and today I got some auto-mail
> indicating they'd fixed that bug as of OS X 10.7 (Lion). I don't have
> Lion installed here, but I grabbed the libedit sources from
> www.opensource.apple.com and indeed it looks fixed. So, if any early
> adopters want to try it out ...

>> I'll add that I've heard reports that Lion Server comes with
>> PostgreSQL as standard, and Lion Desktop comes with psql.

> Yup:
> raptor:PEM dpage$ psql --version
> psql (PostgreSQL) 9.0.4
> contains support for command-line editing

So, is it linked to their libedit, and if so does it crash when you try
to tab-complete a case with exactly 9 completions?

regards, tom lane

On Fri, Jul 22, 2011 at 5:16 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Dave Page <dpage(at)pgadmin(dot)org> writes:
>> On Fri, Jul 22, 2011 at 5:01 PM, Peter Geoghegan <peter(at)2ndquadrant(dot)com> wrote:
>>> On 22 July 2011 03:24, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> I had a bug filed with Apple about that, and today I got some auto-mail
>> indicating they'd fixed that bug as of OS X 10.7 (Lion).  I don't have
>> Lion installed here, but I grabbed the libedit sources from
>> www.opensource.apple.com and indeed it looks fixed.  So, if any early
>> adopters want to try it out ...
>
>>> I'll add that I've heard reports that Lion Server comes with
>>> PostgreSQL as standard, and Lion Desktop comes with psql.
>
>> Yup:
>> raptor:PEM dpage$ psql --version
>> psql (PostgreSQL) 9.0.4
>> contains support for command-line editing
>
> So, is it linked to their libedit,

Yes.

> and if so does it crash when you try
> to tab-complete a case with exactly 9 completions?

Tab complete doesn't seem to work at all (nor does it in my builds
with libedit). No crashes or anything - it just spews a blank line and
returns to the prompt.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Jul 22, 2011, at 11:01 AM, Peter Geoghegan wrote:
> On 22 July 2011 03:24, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> I had a bug filed with Apple about that, and today I got some auto-mail
>> indicating they'd fixed that bug as of OS X 10.7 (Lion). I don't have
>> Lion installed here, but I grabbed the libedit sources from
>> www.opensource.apple.com and indeed it looks fixed. So, if any early
>> adopters want to try it out ...
>
> I'll add that I've heard reports that Lion Server comes with
> PostgreSQL as standard, and Lion Desktop comes with psql.

Interesting... I assume that they're using it for something internal?

IIRC this has actually caused some issues in the past... they had some rather old version installed that was being used by an internal tool... if you tried to install your own version some rather interesting issues could then crop up.
--
Jim C. Nasby, Database Architect jim(at)nasby(dot)net
512.569.9461 (cell) http://jim.nasby.net

[ ... having now installed OS X Lion ... ]

Dave Page <dpage(at)pgadmin(dot)org> writes:
> On Fri, Jul 22, 2011 at 5:16 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> and if so does it crash when you try
>> to tab-complete a case with exactly 9 completions?

> Tab complete doesn't seem to work at all (nor does it in my builds
> with libedit). No crashes or anything - it just spews a blank line and
> returns to the prompt.

Same here. Investigation discloses that Apple took a snapshot of
libedit at a time when rl_line_buffer was absolutely, utterly,
completely broken: it doesn't get updated when the current-line
buffer is realloc'd, meaning it's generally pointing at garbage,
meaning we cannot see any of the words on the current line before
the current word, meaning that any aspect of tab completion that
requires preceding context does not work. So for example you can
get it to tab-complete "sel" to "SELECT", but no case where a table
name needs to be completed will work, because all of those depend on
seeing some previous words that cue us to think a table name is needed.

This appears to have been fixed in netbsd libedit in christo's commit
of Sat Aug 28 15:44:59 2010 UTC (readline.c v1.91 and other files of
even date). However, Lion's copy of libedit is at least a month
older than that (readline.c is v1.89, for instance).

I'm off to file a bug report with Apple, and anybody else who cares
about this should do likewise. But I would not hold my breath for
a fix appearing before OS X 10.8 :-(.

I think you had better plan on incorporating GNU readline into installer
builds for Lion.

regards, tom lane

On Sat, Jul 30, 2011 at 8:25 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> I think you had better plan on incorporating GNU readline into installer
> builds for Lion.

Unfortunately the licence makes that a non-starter.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Excerpts from Dave Page's message of sáb jul 30 15:32:03 -0400 2011:
> On Sat, Jul 30, 2011 at 8:25 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > I think you had better plan on incorporating GNU readline into installer
> > builds for Lion.
>
> Unfortunately the licence makes that a non-starter.

A fixed version of libedit, then?

--
Álvaro Herrera <alvherre(at)commandprompt(dot)com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

On Sun, Jul 31, 2011 at 6:45 PM, Alvaro Herrera
<alvherre(at)commandprompt(dot)com> wrote:
> Excerpts from Dave Page's message of sáb jul 30 15:32:03 -0400 2011:
>> On Sat, Jul 30, 2011 at 8:25 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> > I think you had better plan on incorporating GNU readline into installer
>> > builds for Lion.
>>
>> Unfortunately the licence makes that a non-starter.
>
> A fixed version of libedit, then?

Yes, that's the best we can do.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Sat, Jul 30, 2011 at 08:32:03PM +0100, Dave Page wrote:
> On Sat, Jul 30, 2011 at 8:25 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > I think you had better plan on incorporating GNU readline into
> > installer builds for Lion.
>
> Unfortunately the licence makes that a non-starter.

What is it about a GPLed psql client that is a non-starter for the
installer? I'm not a fan of the GPL, but in this case, the effects of
linking it in are quite limited in scope, i.e. they pertain to exactly
one binary.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

On Mon, Aug 1, 2011 at 6:46 PM, David Fetter <david(at)fetter(dot)org> wrote:
> On Sat, Jul 30, 2011 at 08:32:03PM +0100, Dave Page wrote:
>> On Sat, Jul 30, 2011 at 8:25 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> > I think you had better plan on incorporating GNU readline into
>> > installer builds for Lion.
>>
>> Unfortunately the licence makes that a non-starter.
>
> What is it about a GPLed psql client that is a non-starter for the
> installer?  I'm not a fan of the GPL, but in this case, the effects of
> linking it in are quite limited in scope, i.e. they pertain to exactly
> one binary.

The scope or number of files is irrelevant. Having just a single GPL'd
file in those installers makes it impossible for ISVs to bundle them
with their products, unless they open source them under a GPL
compatible licence.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 08/01/2011 02:17 PM, Dave Page wrote:
> On Mon, Aug 1, 2011 at 6:46 PM, David Fetter<david(at)fetter(dot)org> wrote:
>> On Sat, Jul 30, 2011 at 08:32:03PM +0100, Dave Page wrote:
>>> On Sat, Jul 30, 2011 at 8:25 PM, Tom Lane<tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>>> I think you had better plan on incorporating GNU readline into
>>>> installer builds for Lion.
>>> Unfortunately the licence makes that a non-starter.
>> What is it about a GPLed psql client that is a non-starter for the
>> installer? I'm not a fan of the GPL, but in this case, the effects of
>> linking it in are quite limited in scope, i.e. they pertain to exactly
>> one binary.
> The scope or number of files is irrelevant. Having just a single GPL'd
> file in those installers makes it impossible for ISVs to bundle them
> with their products, unless they open source them under a GPL
> compatible licence.

IIRC, there is also a question of whether it is legal at all to
distribute a binary linked with both libreadline and openssl.

cheers

andrew

On Mon, Aug 01, 2011 at 07:17:00PM +0100, Dave Page wrote:
> On Mon, Aug 1, 2011 at 6:46 PM, David Fetter <david(at)fetter(dot)org> wrote:
> > On Sat, Jul 30, 2011 at 08:32:03PM +0100, Dave Page wrote:
> >> On Sat, Jul 30, 2011 at 8:25 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> >> > I think you had better plan on incorporating GNU readline into
> >> > installer builds for Lion.
> >>
> >> Unfortunately the licence makes that a non-starter.
> >
> > What is it about a GPLed psql client that is a non-starter for the
> > installer?  I'm not a fan of the GPL, but in this case, the
> > effects of linking it in are quite limited in scope, i.e. they
> > pertain to exactly one binary.
>
> The scope or number of files is irrelevant. Having just a single
> GPL'd file in those installers makes it impossible for ISVs to
> bundle them with their products, unless they open source them under
> a GPL compatible licence.

I know of no one, not even the wackiest AGPL loony-tune, who claims
that the GPL is infectious to the degree you're describing. Are you
*sure* the reason you state is the actual reason?

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

On Monday, August 1, 2011, David Fetter <david(at)fetter(dot)org> wrote:
> On Mon, Aug 01, 2011 at 07:17:00PM +0100, Dave Page wrote:
>> On Mon, Aug 1, 2011 at 6:46 PM, David Fetter <david(at)fetter(dot)org> wrote:
>> > On Sat, Jul 30, 2011 at 08:32:03PM +0100, Dave Page wrote:
>> >> On Sat, Jul 30, 2011 at 8:25 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> >> > I think you had better plan on incorporating GNU readline into
>> >> > installer builds for Lion.
>> >>
>> >> Unfortunately the licence makes that a non-starter.
>> >
>> > What is it about a GPLed psql client that is a non-starter for the
>> > installer? I'm not a fan of the GPL, but in this case, the
>> > effects of linking it in are quite limited in scope, i.e. they
>> > pertain to exactly one binary.
>>
>> The scope or number of files is irrelevant. Having just a single
>> GPL'd file in those installers makes it impossible for ISVs to
>> bundle them with their products, unless they open source them under
>> a GPL compatible licence.
>
> I know of no one, not even the wackiest AGPL loony-tune, who claims
> that the GPL is infectious to the degree you're describing. Are you
> *sure* the reason you state is the actual reason?

Well, lets find out. Try shipping a closed-source product which includes the
GPL MySQL ODBC driver, and see how long you can hold out against Oracle's
attack lawyers.

Preventing open source code being used in closed source products is the
whole point of the GPL.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 08/01/2011 04:23 PM, David Fetter wrote:
>>
>> The scope or number of files is irrelevant. Having just a single
>> GPL'd file in those installers makes it impossible for ISVs to
>> bundle them with their products, unless they open source them under
>> a GPL compatible licence.
> I know of no one, not even the wackiest AGPL loony-tune, who claims
> that the GPL is infectious to the degree you're describing.
>
>

Years ago I was told exactly that by some perhaps over-enthusiastic
MySQL Sales staff (in the pre-Sun days). They also had a *very*
expansive notion of what constituted "distribution". These guys are thus
indirectly responsible to some extent for my involvement over the last
eight years or so with PostgreSQL, proof that good may come of evil, if
you like :-)

cheers

andrew

David Fetter <david(at)fetter(dot)org> writes:
> On Mon, Aug 01, 2011 at 07:17:00PM +0100, Dave Page wrote:
>> The scope or number of files is irrelevant. Having just a single
>> GPL'd file in those installers makes it impossible for ISVs to
>> bundle them with their products, unless they open source them under
>> a GPL compatible licence.

> I know of no one, not even the wackiest AGPL loony-tune, who claims
> that the GPL is infectious to the degree you're describing.

Really? You must have a very small sample. The *entire reason* that
there is an LGPL as well as straight GPL is that the latter is thought
to have exactly this effect when used for a library. And the readline
guys have deliberately chosen to use GPL, not LGPL, because that is the
effect they want.

This doesn't affect most Linux distributions because the entire distro
is shipped under GPL-compliant conditions (eg you can get all the
source). Or at least that's the interpretation Red Hat's lawyers use.
But it *would* affect somebody who wanted to bundle readline-containing
psql into a larger proprietary product. We use BSD-ish licensing in
part because that scenario is okay with us ... and Dave is concerned
about not losing that property for the installer builds.

regards, tom lane

On Mon, Aug 01, 2011 at 09:38:30PM +0100, Dave Page wrote:
> On Monday, August 1, 2011, David Fetter <david(at)fetter(dot)org> wrote:
> > On Mon, Aug 01, 2011 at 07:17:00PM +0100, Dave Page wrote:
> >> On Mon, Aug 1, 2011 at 6:46 PM, David Fetter <david(at)fetter(dot)org> wrote:
> >> > On Sat, Jul 30, 2011 at 08:32:03PM +0100, Dave Page wrote:
> >> >> On Sat, Jul 30, 2011 at 8:25 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> >> >> > I think you had better plan on incorporating GNU readline
> >> >> > into installer builds for Lion.
> >> >>
> >> >> Unfortunately the licence makes that a non-starter.
> >> >
> >> > What is it about a GPLed psql client that is a non-starter for
> >> > the installer? I'm not a fan of the GPL, but in this case, the
> >> > effects of linking it in are quite limited in scope, i.e. they
> >> > pertain to exactly one binary.
> >>
> >> The scope or number of files is irrelevant. Having just a single
> >> GPL'd file in those installers makes it impossible for ISVs to
> >> bundle them with their products, unless they open source them
> >> under a GPL compatible licence.
> >
> > I know of no one, not even the wackiest AGPL loony-tune, who
> > claims that the GPL is infectious to the degree you're describing.
> > Are you *sure* the reason you state is the actual reason?
>
> Well, lets find out. Try shipping a closed-source product which
> includes the GPL MySQL ODBC driver, and see how long you can hold
> out against Oracle's attack lawyers.

So long as you make the MySQL ODBC driver source code available, you
have no problem. The GPL infection kicks in at the step where you
link, which you're not doing when you ship that driver.

> Preventing open source code being used in closed source products is
> the whole point of the GPL.

You're smarter than I if you've figured out what motivates Stallman
and Moglen, and the text of the license is far from clear on this
matter.

This business about incompatibility between libssl and the GPL seems
somewhat more plausible. Ideological opposition to shipping GPL
software is also a plausible reason. There's nothing wrong with any
of this. I just don't find the stated reason terribly plausible.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

* David Fetter:

> What is it about a GPLed psql client that is a non-starter for the
> installer? I'm not a fan of the GPL, but in this case, the effects of
> linking it in are quite limited in scope, i.e. they pertain to exactly
> one binary.

And the usual showstopper, OpenSSL, might well fall under the system
library exception on MacOS X. Curiously, the GPL favors proprietary
operating systems in this area.

--
Florian Weimer <fweimer(at)bfk(dot)de>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99