Extensions Dependency Checking

Hackers,

I wanted to get a (ok, not so) quick note in about this before the beta dropped. I've been thinking about the "requires" parameter on extensions control files. Right now it just lists the names of extensions that are required for the extension to run:

requires = 'foo, bar'

But I'm assuming that at some point there's going to be something a bit more robust: specifically, requiring a minimum version, perhaps something like:

requires = 'foo 1.0, bar 0.31.4'

I know that Dimitri has thought about this a bit, and I suspect we can all agree that most extensible systems have some way to at least specify the minimum version of a dependency. Is this something that's likely to come down the pike, perhaps in 9.2 or later?

If so, we return to the issue of comparing version strings in order to resolve/verify dependencies.

One way to solve the problem would be either for a developer to specify a version order in the control file, or else to ensure that the upgrade scripts are there for every single version.

But frankly, I would oppose something like this, as it once again puts the onus on the extension developer to take care of what is essentially a bookkeeping detail. And if I'm any example, there will often be mistakes, typos, etc.

I think a better solution would be to mandate a version string format for extensions *today*. This would remove the onus on the developer, while making version comparisons inside the database efficient and predictable.

Now, I know that ideas has been rejected in the past (at least for *right now*), but I'm here to tell you that not solving this problem up front is fodder for a nightmare later. Perl, for example, has only recently standardized on a version numbering scheme, but thanks to a 15 year history of none at all, it has to bend over backwards to try to manage any version scheme in existence. I mean, just look at this mess:

http://www.dagolden.com/index.php/369/version-numbers-should-be-boring/

I definitely think that version numbers should be boring. A few simple rules and then forget about it, it should just work.

I certainly didn't want to deal with this issue in PGXN, where dependency checking is planned for the command-line client. So I decided to use semantic versions. They have now been implemented as a native type in PostgreSQL, thanks to Sam Vilain porting the PL/pgSQL-based domain I originally wrote. The code is here:

http://master.pgxn.org/dist/semver/0.2.0/

The reason I went with semantic versions, rather than just a numeric, as I'd originally planned, is because they are almost exactly the same format as PostgreSQL version numbers. So they were useful for specifying a dependency on a particular version of PostgreSQL. So by letting the core project set the standard, with just a minor modification of that standard, I get something that just works but is much more flexible than a numeric version. FWIW, the only difference that I've noticed between semantic versions and PostgreSQL versions is in developer/alpha/beta releases. So where PostgreSQL has released 9.1alpha5, the semantic version is 9.1.0alpha5. (And the semver data type can convert from the former to the latter).

So, to summarize:

* I think we're going to need a formal version string spec for extensions.

* If that's true, I think it should be specified *now*, before extensions are in the wild, so that we don't end up with the legacy version string nightmares that Perl modules suffer from.

* If we do adopt a spec, I think it should reflect the PostgreSQL core version strings as closely as possible, and should be fully compatible with them.

* So it might be worth looking at semver or something similar to integrate.

Thoughts?

Best,

David

On Fri, Apr 1, 2011 at 11:45 AM, David E. Wheeler <david(at)kineticode(dot)com> wrote:
> But I'm assuming that at some point there's going to be something a bit more robust: specifically, requiring a minimum version, perhaps something like:
>
>    requires = 'foo 1.0, bar 0.31.4'

Or maybe:

requires = 'foo = 1.0, bar >= 0.31.4'

> * I think we're going to need a formal version string spec for extensions.

I agree.

> * If that's true, I think it should be specified *now*, before extensions are in the wild, so that we don't end up with the legacy version string nightmares that Perl modules suffer from.

I think I agree with this, too.

> * If we do adopt a spec, I think it should reflect the PostgreSQL core version strings as closely as possible, and should be fully compatible with them.

I am less sure about this one.

> * So it might be worth looking at semver or something similar to integrate.

No. It's too late to be monkeying with this. I think for 9.1 we will
need to content ourselves with setting a good precedent, rather than
enforcing it programatically. It's not going to work to insist on all
numeric version strings anyway, because we've already got this 'FROM
unpackaged' bit floating around.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> On Fri, Apr 1, 2011 at 11:45 AM, David E. Wheeler <david(at)kineticode(dot)com> wrote:
>> * I think we're going to need a formal version string spec for extensions.

> I agree.

I don't. We deliberately decided *not* to have any wired-in
interpretation of extension numbers, and I don't think that decision
needs to be reversed. David can choose to enforce something for stuff
distributed through PGXN if he wishes, but that's no concern of the core
server's. In particular I'm really skeptical of the theory that we need
or should want version restrictions in Requires references. The
equivalent feature in RPM is deprecated for Fedora/RedHat packaging use,
and I see no reason why we'd need it more than they do.

>> * So it might be worth looking at semver or something similar to integrate.

> No. It's too late to be monkeying with this. I think for 9.1 we will
> need to content ourselves with setting a good precedent, rather than
> enforcing it programatically. It's not going to work to insist on all
> numeric version strings anyway, because we've already got this 'FROM
> unpackaged' bit floating around.

Once 9.1 is out, it'll probably be too late to dictate any semantics for
version numbers, because somebody will have done something incompatible
with it before 9.2 is released. If we are going to try to insist on
this, now is the time. But I don't agree with that position.

regards, tom lane

On Mon, Apr 4, 2011 at 5:48 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> On Fri, Apr 1, 2011 at 11:45 AM, David E. Wheeler <david(at)kineticode(dot)com> wrote:
>>> * I think we're going to need a formal version string spec for extensions.
>
>> I agree.
>
> I don't.  We deliberately decided *not* to have any wired-in
> interpretation of extension numbers, and I don't think that decision
> needs to be reversed.  David can choose to enforce something for stuff
> distributed through PGXN if he wishes, but that's no concern of the core
> server's.  In particular I'm really skeptical of the theory that we need
> or should want version restrictions in Requires references.  The
> equivalent feature in RPM is deprecated for Fedora/RedHat packaging use,
> and I see no reason why we'd need it more than they do.

Oh, really? How can you possibly get by without it? Dependencies of
this type are all over the place.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Apr 4, 2011, at 2:48 PM, Tom Lane wrote:

> Once 9.1 is out, it'll probably be too late to dictate any semantics for
> version numbers, because somebody will have done something incompatible
> with it before 9.2 is released. If we are going to try to insist on
> this, now is the time.

Yes, exactly my point. If it's going to be done, it should be now.

> But I don't agree with that position.

Why not?

Best.

David

On Mon, Apr 4, 2011 at 6:06 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:

>> I don't.  We deliberately decided *not* to have any wired-in
>> interpretation of extension numbers, and I don't think that decision
>> needs to be reversed.  David can choose to enforce something for stuff
>> distributed through PGXN if he wishes, but that's no concern of the core
>> server's.  In particular I'm really skeptical of the theory that we need
>> or should want version restrictions in Requires references.  The
>> equivalent feature in RPM is deprecated for Fedora/RedHat packaging use,
>> and I see no reason why we'd need it more than they do.
>
> Oh, really?  How can you possibly get by without it?  Dependencies of
> this type are all over the place.

I think the general movement is toward *feature* dependancies. So for
intstance, an extension can specify what *feature* it requires, and
difference "versions" of an extension can provide different
"features".

In that case, you don't really need extenson "foo" > 2.1, you need the
feature that foo 2.1.x provides, maybe "foo-api-2" (note that 2 would
be part of a "name", not any comparison aware version.

I'm already going to be "naming" my extensions with "major versions"
as part of the name (like all the distro postgresql packages) so my
versions will only ever be simple integers of exactly compatable
objects.

But checking http://developer.postgresql.org/pgdocs/postgres/extend-extensions.html,
I don't see any "provides" mechanism. That might be something
actually needed if we are trying to avoid "version" comparisons and
want to be describing actual dependencies...

--
Aidan Van Dyk                                             Create like a god,
aidan(at)highrise(dot)ca                                       command like a king,
http://www.highrise.ca/                                   work like a slave.

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> On Mon, Apr 4, 2011 at 5:48 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> ... In particular I'm really skeptical of the theory that we need
>> or should want version restrictions in Requires references. The
>> equivalent feature in RPM is deprecated for Fedora/RedHat packaging use,
>> and I see no reason why we'd need it more than they do.

> Oh, really? How can you possibly get by without it? Dependencies of
> this type are all over the place.

Yeah, but the proposed solution doesn't actually fix the problem.
There are various issues, but the fundamental objection is that when you
really need it the information is usually wrong/obsolete, especially if
it's maintained by hand. As an example, if we have foo depending on
bar and it says "Requires: bar >= 2.0", that's all well and good, but
it tells us nothing whatsoever about whether this version of foo will
work with bar 3.0, especially if 3.0 postdates the time when the foo
author put that into his specfile. Maybe we should really interpret
that as "Requires: bar >= 2.0 && < 3.0", or maybe not. It's also very
easy for foo's author to start using some nice_bar_function() without
realizing that that was actually only added in bar 2.1.

If you want automatic dependency checking to actually do something
useful, you need something very substantially finer-grained and more
complex than this. In the Fedora world they're mostly depending on
per-shared-library sonames and per-exported-symbol symbol versioning,
and even then it doesn't always do what you wished. But I'll note
that neither of those mechanisms bothers to infer any version ordering:
either you have a match, or you don't.

The equivalent thing of what's now considered good packaging practice
would probably be to extract lists of specific objects provided or
required by a module, preferably automatically. The requires side of
that might be doable just by inspecting pg_depend. The provides side
would be harder, mainly because you'd want some way to omit objects
that are considered purely internal to a module.

Anyway, I'm uneager to adopt a practice that was discredited years ago
among real packaging systems. I think it'll just delay somebody
deciding to fix it properly.

regards, tom lane

Aidan Van Dyk <aidan(at)highrise(dot)ca> writes:
> On Mon, Apr 4, 2011 at 6:06 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> Oh, really? How can you possibly get by without it? Dependencies of
>> this type are all over the place.

> I think the general movement is toward *feature* dependancies. So for
> intstance, an extension can specify what *feature* it requires, and
> difference "versions" of an extension can provide different
> "features".

Right.

> But checking http://developer.postgresql.org/pgdocs/postgres/extend-extensions.html,
> I don't see any "provides" mechanism.

Yes, some sort of manual Provides: (in addition to automatically
extracted Provides:) would likely be part of any serious solution.
We're not there yet, and we're not going to get there in time for 9.1.
But in any case, mechanisms that involve version ordering comparisons
seem to be on their way out for deciding whether package A is
compatible with package B.

regards, tom lane

Aidan Van Dyk <aidan(at)highrise(dot)ca> writes:
> I think the general movement is toward *feature* dependancies. So for
> intstance, an extension can specify what *feature* it requires, and
> difference "versions" of an extension can provide different
> "features".

That sounds like what Emacs is doing too.

> But checking http://developer.postgresql.org/pgdocs/postgres/extend-extensions.html,
> I don't see any "provides" mechanism. That might be something
> actually needed if we are trying to avoid "version" comparisons and
> want to be describing actual dependencies...

The 'provides' mechanism can be added later I think. It's like saying
that in 9.1 an extension 'foo' provides the feature 'foo' and you can't
control that, whereas in future version you will be able to control what
your extension (and its specific version) provides.

Also we will be able to list what the PostgreSQL server provides, maybe,
so that compile time options can be depended on by extensions.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

On Apr 4, 2011, at 3:57 PM, Tom Lane wrote:

>> I think the general movement is toward *feature* dependancies. So for
>> intstance, an extension can specify what *feature* it requires, and
>> difference "versions" of an extension can provide different
>> "features".
>
> Right.

Sounds like a book-keeping nightmare for extension developers. It will discourage large or rapidly-evolving extensions like pgTAP because it will be a PITA to specify features.

>> But checking http://developer.postgresql.org/pgdocs/postgres/extend-extensions.html,
>> I don't see any "provides" mechanism.
>
> Yes, some sort of manual Provides: (in addition to automatically
> extracted Provides:) would likely be part of any serious solution.

<shed type="bike">I'd like to request "Features:" instead of "Provides:".</shed>

> We're not there yet, and we're not going to get there in time for 9.1.
> But in any case, mechanisms that involve version ordering comparisons
> seem to be on their way out for deciding whether package A is
> compatible with package B.

This is news to me, frankly, and the bookkeeping requirements seem potentially awful.

If it's possible that it won't work out this way, that those arguing for version dependency resolution end up getting the consensus, not having a version string format is going to be a nightmare. On the other hand, if we added one now, and feature dependency tracking won the day, well, a version string format could always be loosened later.

Best,

David

On Tue, Apr 5, 2011 at 4:20 PM, David E. Wheeler <david(at)kineticode(dot)com> wrote:
> On Apr 4, 2011, at 3:57 PM, Tom Lane wrote:
>
>>> I think the general movement is toward *feature* dependancies.  So for
>>> intstance, an extension can specify what *feature* it requires, and
>>> difference "versions" of an extension can provide different
>>> "features".
>>
>> Right.
>
> Sounds like a book-keeping nightmare for extension developers. It will discourage large or rapidly-evolving extensions like pgTAP because it will be a PITA to specify features.

Sure, but if you want, the "feature" you can provide can be something like:
pgtap-1.0 (or any of pgtap-0.2{0,1,2,3,4}).

And if your package is backwards compatable, it could even provide:
pgtap-0.25
pgtap-0.24
pgtap-0.23

And that also means that you don't have to screw every body over when
some future pgtap-123.45 is no longer compatible, and the extensions
have relied on "$VERSION > 0.23" meaning they'll work with it.

I mean, PG itself is an example. Does pg > 8.4 mean your code will
work with all future (or even past, but > 8.4) PG versions?

>> We're not there yet, and we're not going to get there in time for 9.1.
>> But in any case, mechanisms that involve version ordering comparisons
>> seem to be on their way out for deciding whether package A is
>> compatible with package B.
>
> This is news to me, frankly, and the bookkeeping requirements seem potentially awful.
>
> If it's possible that it won't work out this way, that those arguing for version dependency resolution end up getting the consensus, not having a version string format is going to be a nightmare. On the other hand, if we added one now, and feature dependency tracking won the day, well, a version string format could always be loosened later.

As someone who has had to try and deal with "package versions" for
dependencies in RPM and DEB, and been through the hell that is open
source package variants, all with the ability to turn on/off features
at configure/compile time, a just versions even with <, <=, =, >=, >
all mapped correctly isn't good enough.

Of course, I'ld love for extension in 9.1 to provide a basic
provides/features for my extension to give, but if that train has
already left the station, I don't have much choice ;-(

a.

--
Aidan Van Dyk                                             Create like a god,
aidan(at)highrise(dot)ca                                       command like a king,
http://www.highrise.ca/                                   work like a slave.

On Apr 5, 2011, at 1:42 PM, Aidan Van Dyk wrote:

> Sure, but if you want, the "feature" you can provide can be something like:
> pgtap-1.0 (or any of pgtap-0.2{0,1,2,3,4}).
>
> And if your package is backwards compatable, it could even provide:
> pgtap-0.25
> pgtap-0.24
> pgtap-0.23

I see, I get it.

> And that also means that you don't have to screw every body over when
> some future pgtap-123.45 is no longer compatible, and the extensions
> have relied on "$VERSION > 0.23" meaning they'll work with it.

I see.

> I mean, PG itself is an example. Does pg > 8.4 mean your code will
> work with all future (or even past, but > 8.4) PG versions?

I see. So the extension author can more easily tell users when compatibility has been dropped for something. That makes sense.

> As someone who has had to try and deal with "package versions" for
> dependencies in RPM and DEB, and been through the hell that is open
> source package variants, all with the ability to turn on/off features
> at configure/compile time, a just versions even with <, <=, =, >=, >
> all mapped correctly isn't good enough.

Yeah. The use of an implicit >= in CPAN modules has been a decent 90% solution, but it does cause headaches for people that they can't express things better. The ability to do so would require a mini-language with more operators, precedence, grouping, etc.

> Of course, I'ld love for extension in 9.1 to provide a basic
> provides/features for my extension to give, but if that train has
> already left the station, I don't have much choice ;-(

Yeah, but the way it is doesn't break the ability to do it later. I suspect that Dim and Tom will be thinking about it for 9.2.

Anyway, your post helps me to understand things better, and so I'm less insistent about imposing a version numbering scheme now (though I still think it would be more useful to have one than not).

Best,

David

On Tue, Apr 5, 2011 at 4:51 PM, David E. Wheeler <david(at)kineticode(dot)com> wrote:

>> Of course, I'ld love for extension in 9.1 to provide a basic
>> provides/features for my extension to give, but if that train has
>> already left the station, I don't have much choice ;-(
>
> Yeah, but the way it is doesn't break the ability to do it later. I suspect that Dim and Tom will be thinking about it for 9.2.
>
> Anyway, your post helps me to understand things better, and so I'm less insistent about imposing a version numbering scheme now (though I still think it would be more useful to have one than not).

Versions are useful for figuring out if I should upgrade packages or
not. But I believe the extension framework has explicitly made the
"upgrade" problem a manual one at this point, either taking
destination versions from the control, or the alter command.

So for PGXN's problem, I see the point of versions being required.
But for installation the dependancy graph, "provides/features" rather
than versions are much more useful. And automatic feature/provides
(like library so, and symbol versions in the OS package world,
"objects" in PG world) would definitely be nice, but my Makefile can
build those for me for now until 9.2 (or 9.3, 9.3, etc), if only I had
a way to track them with my installed extension ;-) </stop begging>

a.

--
Aidan Van Dyk                                             Create like a god,
aidan(at)highrise(dot)ca                                       command like a king,
http://www.highrise.ca/                                   work like a slave.

On Apr 5, 2011, at 1:59 PM, Aidan Van Dyk wrote:

> Versions are useful for figuring out if I should upgrade packages or
> not. But I believe the extension framework has explicitly made the
> "upgrade" problem a manual one at this point, either taking
> destination versions from the control, or the alter command.
>
> So for PGXN's problem, I see the point of versions being required.

Okay, thanks, I'm convinced (and relieved; I hated to be bringing something like this up so close to launch).

> But for installation the dependancy graph, "provides/features" rather
> than versions are much more useful. And automatic feature/provides
> (like library so, and symbol versions in the OS package world,
> "objects" in PG world) would definitely be nice, but my Makefile can
> build those for me for now until 9.2 (or 9.3, 9.3, etc), if only I had
> a way to track them with my installed extension ;-) </stop begging>

I'm sure a patch would be welcomed. ;-P

Best,

David