Re: Confusion over Python drivers {license}

I saw this on reddit and thought I might drop a line.

I went through this same issue, trying to find a postgresql driver. Mostly, I had no intention of using psycopg because of its copyleft licensing. (Of course, I don't need to go into why.)

Anyways, here's some info that might help on three alternatives:
Py-postgresql, PyGreSQL, and PG3000

Py-postgresql is active (but I only see 1 person committing)
MIT/BSD style license

It uses some some C for speed.

It has DB-API and PG-API bindings.

Problem: is it Python 3 only!

PyGreSQL (looking to not be very active in the last few months)
MIT/BSD style license
I'm vague on it's features, whether it has C optimizations, or if it is a version limited to Python 3 or 2.x

PG8000 is active at the moment
MIT/BSD style license
It works on 2.x and Python 3

Problem: it is pure Python, so it is slower

If you plan on pursuing this, I'd recommend talking to people on on 3 projects. Here's what I would describe as ideal in a project:

* MIT/BSD style license
* works on Python 2.x and 3
* C optimizations for speed (or whatever for speed)

------------------
That's that part. Now for some problems that would need solving in order for the db api to be extremely useful/userfriendly in the Python world:
------------------
* SQLAlchemy bindings/support -- as of right now only psycopyg works in
SQLAlchemy without problems. PG8000 will work, but you need an
experimental version of boht PG8000 and SQLAlchemy. The other two do
not work. SQLAlchemy support is probably most needed of anything. SQLAlchemy support should not only be possible but default.
* Django bindings (these are unique to Django and nobody but psycopg has
them, which is not good). Django support should not only be possible but default.

* Needs 2.x and Python 3 compatability
* It is inevitable that if the library uses C optimizations, then that means they need to provide lots of different binaries -- think 32bit vs 64bit and then 2.4, 2.5, 3.0 etc.... Some combinations will often be left out making it hard for someone to use/install it.

Of course all of this is from the perspective of Python users. Of course, you have your own features that you want from your end (from PostgreSQL's perspective). Perhaps this info would help you to know which avenue to pursue.

_________________________________________________________________
Your E-mail and More On-the-Go. Get Windows Live Hotmail Free.
http://clk.atdmt.com/GBL/go/201469229/direct/01/

Kevin,

> Of course all of this is from the perspective of Python users. Of
> course, you have your own features that you want from your end (from
> PostgreSQL's perspective). Perhaps this info would help you to know
> which avenue to pursue.

That's invaluable. Thanks for chiming in!

--Josh Berkus

kevinar18(at)hotmail(dot)com (Kevin Ar18) writes:
> Of course all of this is from the perspective of Python users. Of
> course, you have your own features that you want from your end (from
> PostgreSQL's perspective). Perhaps this info would help you to know
> which avenue to pursue.

No, those seem like fine ways of getting a good perspective on the
issue.

I happen not to use Python much, so there's a certain aspect of "don't
care" on my part... but that doesn't imply that my "PostgreSQL
perspective" would tend to override yours. Instead, I think that the
"Python users' perspective" *is* a mighty important thing.

The interface needs aspects of "cleanness" on both sides of the
interface...

- On the Python side, it needs to "play well" in a variety of ways
that you seem to have described nicely, some technical, some
licensing oriented. Some relating to interfacing to further bits
of Python and to applications and frameworks written in Python.

- On the PostgreSQL side, there's certainly a preference for
"licensing simplicity."

Note that most of the issues there really lie on the Python side, which
underlines the importance of "Python users' perspective."

Further, the "ideal" and "issues/problems" that you point out all seem
reasonable. The good seems good and the bad seems like things that do
indeed need to be accepted as consequences of the good.

It will doubtless help guide assistance.
--
output = reverse("moc.liamg" "@" "enworbbc")
"...as a robotics designer once told me, you don't really appreciate
how smart a moron is until you try to design a robot..."
-- Jerry Pournelle

I hope people don't mind my asking about this on the list... as I hinted at before, I don't really follow the development of PostgreSQL, I was just interested in the Python driver project that I heard about.

Anyways, as I understand it, the current goal is to use psycopg and get it changed to LGPL (assuming all the contributors of psycopg agree and confirm they did not use GPL code from any other location). Is this correct?

When I first heard about the endeavor, I thought the goal was to take one or several of the non-copyleft projects, which were rather unfocused, and work with those teams to produce a really good implementation for Python. However, as I understand it (based on what Greg told me) the license is not really an issue as long as it is not GPL; instead, the PostgreSQL team would mostly prefer something that is nearly done, so as to have to do much more work. Is this a correct assessment?

Based on that, I guess my question is what would it have taken to have picked one of BSD/MIT projects and working with those people instead? In other words, what key things affected the decision for psycopg? What areas is it so far ahead in or that would have just been too much work to fix in the other implementations?

Anyways, I hope this message doesn't come across as bad form. It's unfortunate for me that there was not a good enough BSD/MIT project; but I can live without right? :) Still, I just thought I might ask and find out a little more about what the team was looking for in a PostgreSQL implementation, and maybe do a little research myself (to see if anything was missed).

_________________________________________________________________
Hotmail: Free, trusted and rich email service.
http://clk.atdmt.com/GBL/go/201469228/direct/01/

Kevin Ar18 <kevinar18(at)hotmail(dot)com> writes:
> When I first heard about the endeavor, I thought the goal was to take
> one or several of the non-copyleft projects, which were rather
> unfocused, and work with those teams to produce a really good
> implementation for Python. However, as I understand it (based on what
> Greg told me) the license is not really an issue as long as it is not
> GPL; instead, the PostgreSQL team would mostly prefer something that
> is nearly done, so as to have to do much more work. Is this a correct
> assessment?

Well, all else being equal we'd certainly prefer a library that was
licensed more like the core Postgres database. However, we don't have
infinite resources, and an LGPL license is not a showstopper (at least
not to the people who seem to be willing to work on this problem).
The attractiveness of the license has to be balanced against how much
work we'd have to put in and how long it will take to get results.

Not being a python user myself, I wasn't paying all that close attention
to the discussion, but that's my sense of how the decision went.

If you feel that a BSD/MIT license is a must-have for your purposes,
you're certainly free to push development of one of the other driver
projects instead, and to try to organize some other people to help.
I don't believe anyone is trying to funnel all development effort into
psycopg2.

regards, tom lane

> Well, all else being equal we'd certainly prefer a library that was
> licensed more like the core Postgres database. However, we don't have
> infinite resources, and an LGPL license is not a showstopper (at least
> not to the people who seem to be willing to work on this problem).
> The attractiveness of the license has to be balanced against how much
> work we'd have to put in and how long it will take to get results.
>
> Not being a python user myself, I wasn't paying all that close attention
> to the discussion, but that's my sense of how the decision went.
>
> If you feel that a BSD/MIT license is a must-have for your purposes,
> you're certainly free to push development of one of the other driver
> projects instead, and to try to organize some other people to help.
> I don't believe anyone is trying to funnel all development effort into
> psycopg2.
Thanks for the reply.

I guess that's good advice; I suppose I should just do that and talk to some of the teams about it. It would probably help a lot to focus on just one implementation instead of several, even if it's not the same one as what the PostgreSQL team works on. :)

_________________________________________________________________
Hotmail: Trusted email with Microsoft’s powerful SPAM protection.
http://clk.atdmt.com/GBL/go/201469226/direct/01/

Kevin Ar18 wrote:
>
> Based on that, I guess my question is what would it have taken to have
> picked one of BSD/MIT projects and working with those people instead?
> In other words, what key things affected the decision for psycopg?
> What areas is it so far ahead in or that would have just been too much
> work to fix in the other implementations?

A lot of it as I see it is plain old fashioned popularity resulting from
long sustained development. psycopg has been around since 2001, the
current version is already compatible with SQLAlchemy, Django, Zope,
PyDO, and it's integral to the large Python/PostgreSQL toolchain from
Skype including tools like Londiste. When something is supported in
that many places, and the main complaints are its license and
documentation rather than its code, I know I'm not going to start by
assuming I should just hack on somebody else's code to try and replace
it just because their license is a little better. And I'd consider
BSD/MIT only a little better than the LGPL.

That sort of bad decision making is how we got to so many abandoned
Python drivers in the first place. If everybody who decided they were
going to write their own from scratch had decided to work on carefully
and painfully refactoring and improving PyGreSQL instead, in an
incremental way that grew its existing community along the way, we might
have a BSD driver with enough features and users to be a valid
competitor to psycopg2. But writing something shiny new from scratch is
fun, while worrying about backward compatibility and implementing all
the messy parts you need to really be complete on a project like this
isn't, so instead we have a stack of not quite right drivers without any
broad application support.

(Aside: I have a bit of vocabulary I regularly use for this now.
Open-source projects that just ignore working on an existing stack of
working implementations, to instead start from scratch to build
something of questionably improved fanciness instead regardless of its
impact on backward compatibility and the existing userbase, have in my
terminology "PulseAudioed" the older projects).

The gauntlet I would throw down for anyone who thinks there's a better
approach here is to demonstrate a driver that has working support going
back to Python 2.4 for any two of the apps on my list above. Get even
that much of a foothold, and maybe the fact that yours is more Pythonic,
cleaner, or better licensed matters. Until then, working apps have to
be the primary motivation for what to work on here, unless there's a
really terrible problem with the driver. The existing psycopg license
and the web site issues were in combination enough to reach that level
of total issues for a number of people. With the news from Federico
that a license improvement is approaching and signs of major
documentation improvements, that problem seems like it will soon be
solved as far as I'm concerned. When someone manages to make their
alternative driver a prerequisite for an app I need, only at that point
will that driver show back up on my radar.

--
Greg Smith 2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
greg(at)2ndQuadrant(dot)com www.2ndQuadrant.com

Tom Lane wrote:
> If you feel that a BSD/MIT license is a must-have for your purposes,
> you're certainly free to push development of one of the other driver
> projects instead, and to try to organize some other people to help.
> I don't believe anyone is trying to funnel all development effort into
> psycopg2.
>

Certainly not, and I hope no one has gotten the impression that there's
anything "official" being recognized about psycopg happening here
because it hasn't. Anointing a "one true driver" (a phrase that seems
to keep popping up in external discussions of this topic) isn't the sort
of thing the PostgreSQL core does. And all that happens to people who
ignore what I tell them to do is that they receive a steady stream of
long, sarcastic e-mails for a while afterwards.

I just updated
http://wiki.postgresql.org/wiki/Python_PostgreSQL_Driver_TODO to reflect
a few corrections I noted while researching (everybody else is welcome
to edit that page too you know), and to include some of the recent
feedback showing up on this list.

I know I was just looking for a Python driver that is compatible with
the apps I most often run into, documented well enough that I can write
my own if I feel like it, fast enough, and free enough that I can deploy
the result wherever I want. That seemed similar to the priorities other
people who had an opinion here suggested too. Pragmatically, psycopg2
just seemed to have the shortest path toward being something useful to
the largest userbase in that sort of context, and we've unofficially
rolled down that path a bit.

This rabble-rousing seems to have nudged both development communities
toward being more closely aligned in the future in a couple of ways,
which is great, but I wouldn't read much more into things than that.
Other projects will continue to grow and shrink, and the pure Python
ones in particular continue to be quite valuable because they fill a
niche that psycopg2 doesn't target at all. I'd sure like to see all
three of those projects merge into one big one though. My bet has to be
on pg8000, since it has the perfect license, supports all the necessary
Python versions, and it's been around long enough (almost two years)
that support for it is already in the latest SQLAlchemy beta.

We seem to have revitalized discussion around modernizing PyGreSQL too,
so I wouldn't discount that one completely yet either. For those who
feel a true BSD license is vital, I direct you toward
http://mailman.vex.net/pipermail/pygresql/2010-February/002315.html to
learn more about what direction they could use some help in going. But
whatever you do, don't start another project instead.

--
Greg Smith 2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
greg(at)2ndQuadrant(dot)com www.2ndQuadrant.com

On Wed, 2010-02-10 at 23:13 -0500, Greg Smith wrote:
> Until then, working apps have to
> be the primary motivation for what to work on here, unless there's a
> really terrible problem with the driver. The existing psycopg license
> and the web site issues were in combination enough to reach that level
> of total issues for a number of people. With the news from Federico
> that a license improvement is approaching and signs of major
> documentation improvements, that problem seems like it will soon be
> solved as far as I'm concerned. When someone manages to make their
> alternative driver a prerequisite for an app I need, only at that point
> will that driver show back up on my radar.

Keep in mind that backwards compatibility is not the only issue here;
forwards compatibility matters as well*. A lot of the encoding issues I
wrote up ( http://wiki.postgresql.org/wiki/Driver_development ) will
probably be real bugs in a python3 application using a driver that
doesn't understand encoding properly.

Obviously this is less urgent than having a driver that works now, but
it's still important. I think we would attract some goodwill from the
python community if we were helping them move to python3, rather than
sitting around waiting 'til they've already moved and decided that they
can't use postgresql.

Regards,
Jeff Davis

* I tried installing psycopg2-2.0.13 and the build system apparently
doesn't support python3.1, so I assume that psycopg2 doesn't support
python3 at all.

>Obviously this is less urgent than having a driver that works now, but
>it's still important. I think we would attract some goodwill from the
>python community if we were helping them move to python3, rather than
>sitting around waiting 'til they've already moved and decided that they
>can't use postgresql.

It's very, very difficult (but not impossible) to support both python 2
and 3 simultaneously, particularly if you have non-trivial C extension
code. Even the python gods will admit that it's still early days.

--
Andrew McNamara, Senior Developer, Object Craft
http://www.object-craft.com.au/

On 12/02/2010 01:00, Jeff Davis wrote:
> * I tried installing psycopg2-2.0.13 and the build system apparently
> doesn't support python3.1, so I assume that psycopg2 doesn't support
> python3 at all.

python3 was almost completely supported some months ago but then I had
to fix some bugs and almost 99% of psycopg users are still with python2
so the python2 branch is in a better shape. But most of the work is done
so, after the next release, I'll start porting changes to the python3
branch ("master" on git) and finish the work.

federico

--
Federico Di Gregorio fog(at)initd(dot)org
Viva il pollo! Viva il pollo! -- sisterconfusion
Continuo a preferire il coniglio -- vodka

On Fri, 2010-02-12 at 10:38 +0100, Federico Di Gregorio wrote:
> On 12/02/2010 01:00, Jeff Davis wrote:
> > * I tried installing psycopg2-2.0.13 and the build system apparently
> > doesn't support python3.1, so I assume that psycopg2 doesn't support
> > python3 at all.
>
> python3 was almost completely supported some months ago but then I had
> to fix some bugs and almost 99% of psycopg users are still with python2
> so the python2 branch is in a better shape. But most of the work is done
> so, after the next release, I'll start porting changes to the python3
> branch ("master" on git) and finish the work.

Ok, sounds great.

Do you attempt to handle the encoding issues when working with python3?
I'd be interested to see your approach, because I didn't get much
feedback on the doc I wrote.

Regards,
Jeff Davis

Jeff Davis wrote:
> Keep in mind that backwards compatibility is not the only issue here;
> forwards compatibility matters as well*. A lot of the encoding issues I
> wrote up ( http://wiki.postgresql.org/wiki/Driver_development ) will
> probably be real bugs in a python3 application using a driver that
> doesn't understand encoding properly.
>

Sure, but the benefit of being the de-facto standard to some problem,
because you're already supported by everybody, is that it's easier to
attract the most help to move forward too. The disclaimers on
investments always read "past performance is not indicative of future
results". I don't think that's really true in the open-source project
case. Generally, I'd rather trust a project that's been keeping up with
things for a long time already to continue to do so, particularly one
that's built up a community around it already, rather than risk
switching to one without a known good track record in that record.

(Exercise for the reader: consider how what I just said applies in both
a positive and negative way toward MySQL and its associated development
model)

--
Greg Smith 2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
greg(at)2ndQuadrant(dot)com www.2ndQuadrant.com