Re: Plugin system like Firefox

Matthew T. O'Connor wrote:
> A few random thoughts...
>
> The application that comes to mind first for me when you talk plugins is
> Firefox. They make it very easy to browse for plugins and to install,
> update, remove them. Their plug-in system also tries to account for
> Firefox version and OS platform which we would need to do also.
>
> Perhaps one thing that would help PostgreSQL plug-ins is a nice GUI
> plug-in browser and management application. The logical place to add
> this IMHO is PGAdmin since it is GUI, already talks to the DB and is
> cross platform. I'm not saying a GUI should be required to manage
> plug-ins, a fully CLI option should be made available too.

I feel the above comment about Firefox is the crux of the plugin issue.
Having per-operating system options for each plugin is never going to
scale.

Having users compile/install these plugins works, but the effort
required discourages their use, and the fewer people that try them the
fewer people use them and contribute back to the Postgres plugin
ecosystem.

This is similar to the problem of installing Postgres before we had
per-OS installs and Postgres software bundles like the Win32 installer
and Postgres Plus.

The Firefox plugins are successful because it is easy to install stuff.
I run Ubuntu on my laptop so I can easily install software --- I can
compile/install software from source, but I prefer to use the Ubuntu
software install system so I can concentrate on my major task.

So, ideally, if we do a plug-in system, I think we need some way to have
these plugins be very easily installed, perhaps by choosing object files
pre-compile by the build farm for each operating system.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Tue, Aug 12, 2008 at 4:13 AM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> So, ideally, if we do a plug-in system, I think we need some way to have
> these plugins be very easily installed, perhaps by choosing object files
> pre-compile by the build farm for each operating system.

Hmm, that idea sounds remarkably familiar <thinks back to a meeting in
NJ about 2 years ago>

:-)

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

Hi,

Bruce Momjian wrote:
> So, ideally, if we do a plug-in system, I think we need some way to have
> these plugins be very easily installed, perhaps by choosing object files
> pre-compile by the build farm for each operating system.

IIRC the main problem isn't compilation and packaging of plugins. On
your Ubuntu system, for example, you can easily install
'postgresql-contrib-8.x' and get most of the contrib modules as *.so
libraries. You'll even find the install_*.sql and uninstall_*.sql
scripts in '/usr/share/postgresql/8.x/contrib'.

The main problem I see is ease of installation and backup/restore
capability of these modules. Installing the tsearch2 module with:

psql $YOURDB < /usr/share/postgresql/8.x/contrib/tsearch2.sql

..is already quite easy to do. But beware of the trouble when trying to
back up and restore that database later on!

At the moment, we distinguish between system objects, initdb objects and
user objects by OID:

#define FirstBootstrapObjectId 10000
#define FirstNormalObjectId 16384

But objects created by modules are indistinguishable from user objects.
IMO, that's the main problem. Maybe we can solve this by reserving yet
another range of OIDs for modules. Then treat these objects as 'system'
objects during a pg_dump? I'm not sure if such a limited range of OIDs
is a good idea, but IMO the ability to tell apart system, module and
user objects is the key feature required here.

Regards

Markus Wanner

Am Tuesday, 12. August 2008 schrieb Bruce Momjian:
> I feel the above comment about Firefox is the crux of the plugin issue.

The difference is that Firefox is an interactive eye-candy system whereas
PostgreSQL is a functionality-based server system. A PostgreSQL database
system needs to have a set up method that is robust, reproducible, and
noninteractive. Easy of use and click & drool is all nice, but overall I
think the requirements profile for a PostgreSQL plugin system is quite
different from Firefox.

Markus Wanner wrote:
> Hi,
>
> Bruce Momjian wrote:
> > So, ideally, if we do a plug-in system, I think we need some way to have
> > these plugins be very easily installed, perhaps by choosing object files
> > pre-compile by the build farm for each operating system.
>
> IIRC the main problem isn't compilation and packaging of plugins. On
> your Ubuntu system, for example, you can easily install
> 'postgresql-contrib-8.x' and get most of the contrib modules as *.so
> libraries. You'll even find the install_*.sql and uninstall_*.sql
> scripts in '/usr/share/postgresql/8.x/contrib'.
>
> The main problem I see is ease of installation and backup/restore
> capability of these modules. Installing the tsearch2 module with:
>
> psql $YOURDB < /usr/share/postgresql/8.x/contrib/tsearch2.sql
>
> ..is already quite easy to do. But beware of the trouble when trying to
> back up and restore that database later on!

Well, dump/restore is part of the problem for /contrib but we need
something that handles pgfoundry stuff too, which I doubt is ever going
to be covered sufficiently by any OS-specific solutions.

The bottom line is that for software to be successful we need a critical
mass, and as long as we are doing OS-specific plugins we aren't going to
reach that critical mass because the OS-specific aspect splits up
demand.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Peter Eisentraut wrote:
> Am Tuesday, 12. August 2008 schrieb Bruce Momjian:
> > I feel the above comment about Firefox is the crux of the plugin issue.
>
> The difference is that Firefox is an interactive eye-candy system whereas
> PostgreSQL is a functionality-based server system. A PostgreSQL database
> system needs to have a set up method that is robust, reproducible, and
> noninteractive. Easy of use and click & drool is all nice, but overall I
> think the requirements profile for a PostgreSQL plugin system is quite
> different from Firefox.

True, but I don't see having eye-candy and robust as mutually exclusive.
This is one of the areas where Postgres can go beyond what other
databases can offer, and if we can make it easier to use, we will get
more benefit from it.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Hi,

Bruce Momjian wrote:
> The bottom line is that for software to be successful we need a critical
> mass, and as long as we are doing OS-specific plugins we aren't going to
> reach that critical mass because the OS-specific aspect splits up
> demand.

What about the pgxs toolchain? Doesn't that provide enough
OS-abstraction to be usable for projects outside of Postgres itself? If
no, why not? (I've currently only used it for extensions under different
Linuxen, so I can't tell much about OS-specific aspects of it).

Are you proposing that we should introduce our own packaging system for
such add-on components? Like CP(g)AN (which has been proposed before..)?

Regards

Markus Wanner

Markus Wanner wrote:
> Hi,
>
> Bruce Momjian wrote:
> > The bottom line is that for software to be successful we need a critical
> > mass, and as long as we are doing OS-specific plugins we aren't going to
> > reach that critical mass because the OS-specific aspect splits up
> > demand.
>
> What about the pgxs toolchain? Doesn't that provide enough
> OS-abstraction to be usable for projects outside of Postgres itself? If
> no, why not? (I've currently only used it for extensions under different
> Linuxen, so I can't tell much about OS-specific aspects of it).

pgxs is very good, but it is for interpolating server information to be
used during a compile; it does not deal with packaging.

> Are you proposing that we should introduce our own packaging system for
> such add-on components? Like CP(g)AN (which has been proposed before..)?

Yes, pretty much. I imagine some kind of web interface or Java app (did
I just say that?) that lists all plugins and when you choose one it
downloads an object file appropriate for your operating system plus SQL
scripts and somehow automatically installs them in the desired database.

That is the kind of capability we need to really advance things. We
would still allow source installs but for people wanting to try things
out, I see no other alternative, and "try things out" == adoption.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

I can't speak for anyone else, but I much prefer packages that make
use of my operating system's package management system rather than
rolling their own. If I need a perl package that I can't get through
yum, I build my own RPMs rather than installing through CPAN.

I actually think that the biggest hurdle to the adoption of non-core
code is knowing that it's out there to begin with. Firefox advertises
the availability of add-ons VERY prominently, and that is why people
give them a try. www.postgresql.org has a link to pgfoundry on the
home page, but nothing about contrib, and pgfoundry is not as
end-user-friendly as addons.mozilla.org.

And the core documentation doesn't mention contrib or pgfoundry much
either. There is a section on contrib in the docs, but that only
helps you if your question is "What does contrib module X do?". If
your question is "Is there a contrib module X that addresses my
problem?", that's not so easy. For example, if you''re reading the
page on string functions[1], it will tell you that you can do regular
expression matching and substring matching, but they WON'T tell you
that if you're looking for soundex matching, you should look to the
fuzzystrmatch[2] package.

...Robert

[1] http://www.postgresql.org/docs/8.3/interactive/functions-string.html
[2] http://www.postgresql.org/docs/8.3/interactive/fuzzystrmatch.html

On Tue, Aug 12, 2008 at 12:41 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> Markus Wanner wrote:
>> Hi,
>>
>> Bruce Momjian wrote:
>> > The bottom line is that for software to be successful we need a critical
>> > mass, and as long as we are doing OS-specific plugins we aren't going to
>> > reach that critical mass because the OS-specific aspect splits up
>> > demand.
>>
>> What about the pgxs toolchain? Doesn't that provide enough
>> OS-abstraction to be usable for projects outside of Postgres itself? If
>> no, why not? (I've currently only used it for extensions under different
>> Linuxen, so I can't tell much about OS-specific aspects of it).
>
> pgxs is very good, but it is for interpolating server information to be
> used during a compile; it does not deal with packaging.
>
>> Are you proposing that we should introduce our own packaging system for
>> such add-on components? Like CP(g)AN (which has been proposed before..)?
>
> Yes, pretty much. I imagine some kind of web interface or Java app (did
> I just say that?) that lists all plugins and when you choose one it
> downloads an object file appropriate for your operating system plus SQL
> scripts and somehow automatically installs them in the desired database.
>
> That is the kind of capability we need to really advance things. We
> would still allow source installs but for people wanting to try things
> out, I see no other alternative, and "try things out" == adoption.
>
> --
> Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
> EnterpriseDB http://enterprisedb.com
>
> + If your life is a hard drive, Christ can be your backup. +
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>

Hi,

Robert Haas wrote:
> I can't speak for anyone else, but I much prefer packages that make
> use of my operating system's package management system rather than
> rolling their own. If I need a perl package that I can't get through
> yum, I build my own RPMs rather than installing through CPAN.

I very much agree to that (well, s/RPM/DEB/).

But AFAIK we also need to provide packages for OSes without a package
management system. Windows being the most popular such OS.

Which probably means we should provide something that can work on its
own *or* through another package management system (very much like CPAN
and others, again).

Regards

Markus Wanner

Markus Wanner wrote:
> Hi,
>
> Robert Haas wrote:
> > I can't speak for anyone else, but I much prefer packages that make
> > use of my operating system's package management system rather than
> > rolling their own. If I need a perl package that I can't get through
> > yum, I build my own RPMs rather than installing through CPAN.
>
> I very much agree to that (well, s/RPM/DEB/).
>
> But AFAIK we also need to provide packages for OSes without a package
> management system. Windows being the most popular such OS.
>
> Which probably means we should provide something that can work on its
> own *or* through another package management system (very much like CPAN
> and others, again).

We don't get economies of scale without an OS-agnostic way of installing
packages. I realize many prefer their OS-native packaging system, but
that isn't the target audience of a packaging system that will increase
adoption.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Hi,

Bruce Momjian wrote:
> .. an OS-agnostic way of installing packages.

Uh.. I don't think such a thing exists.

> I realize many prefer their OS-native packaging system, but
> that isn't the target audience of a packaging system that will increase
> adoption.

I'm just arguing that such a packaging system should not get in the way
of a pre-existing packaging system (whether it is OS-native or not,
whether it is an open and well known format or a proprietary
SomethingSuperPerfect(tm) Packaging System for Windows).

Regards

Markus Wanner

Markus Wanner wrote:
> Hi,
>
> Bruce Momjian wrote:
> > .. an OS-agnostic way of installing packages.
>
> Uh.. I don't think such a thing exists.

Seems to in Firefox.

> > I realize many prefer their OS-native packaging system, but
> > that isn't the target audience of a packaging system that will increase
> > adoption.
>
> I'm just arguing that such a packaging system should not get in the way
> of a pre-existing packaging system (whether it is OS-native or not,
> whether it is an open and well known format or a proprietary
> SomethingSuperPerfect(tm) Packaging System for Windows).

I never suggested interfering with our existing distribution methods.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Aug 12, 2008, at 13:16, Bruce Momjian wrote:

>>> .. an OS-agnostic way of installing packages.
>>
>> Uh.. I don't think such a thing exists.
>
> Seems to in Firefox.

And Perl's CPAN repository and installation module.

Best,

David

On Aug 12, 2008, at 12:42 PM, Bruce Momjian wrote:

> Markus Wanner wrote:
>> Hi,
>>
>> Robert Haas wrote:
>>> I can't speak for anyone else, but I much prefer packages that make
>>> use of my operating system's package management system rather than
>>> rolling their own. If I need a perl package that I can't get
>>> through
>>> yum, I build my own RPMs rather than installing through CPAN.
>>
>> I very much agree to that (well, s/RPM/DEB/).
>>
>> But AFAIK we also need to provide packages for OSes without a package
>> management system. Windows being the most popular such OS.
>>
>> Which probably means we should provide something that can work on its
>> own *or* through another package management system (very much like
>> CPAN
>> and others, again).
>
> We don't get economies of scale without an OS-agnostic way of
> installing
> packages. I realize many prefer their OS-native packaging system, but
> that isn't the target audience of a packaging system that will
> increase
> adoption.

If the OS-agnostic version is designed appropriately then turning
it (mechanically) into an OS-specific one should be possible.

Cheers,
Steve

>>>> .. an OS-agnostic way of installing packages.
>>>
>>> Uh.. I don't think such a thing exists.
>>
>> Seems to in Firefox.
>
> And Perl's CPAN repository and installation module.

Don't forget the command line installation of packages for the R
programming language.

>> > .. an OS-agnostic way of installing packages.
>> Uh.. I don't think such a thing exists.
> Seems to in Firefox.

I think nearly all Firefox extensions are written in XML and
Javascript. It's quite a bit easier to have an OS-agnostic way of
installing packages when the packages are platform-independent.

>> > I realize many prefer their OS-native packaging system, but
>> > that isn't the target audience of a packaging system that will increase
>> > adoption.

Really? Who is it?

...Robert

On Aug 12, 2008, at 18:12, Robert Haas wrote:

>>>> I realize many prefer their OS-native packaging system, but
>>>> that isn't the target audience of a packaging system that will
>>>> increase
>>>> adoption.
>
> Really? Who is it?

Those of us who don't use packaging systems. And, we are legion.

Best,

David

Robert Haas wrote:
> >> > .. an OS-agnostic way of installing packages.
> >> Uh.. I don't think such a thing exists.
> > Seems to in Firefox.
>
> I think nearly all Firefox extensions are written in XML and
> Javascript. It's quite a bit easier to have an OS-agnostic way of
> installing packages when the packages are platform-independent.
>
> >> > I realize many prefer their OS-native packaging system, but
> >> > that isn't the target audience of a packaging system that will increase
> >> > adoption.
>
> Really? Who is it?

People who want to try a database and its plugins to see if they like
it.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

> -----Original Message-----
> From: pgsql-hackers-owner(at)postgresql(dot)org [mailto:pgsql-hackers-
> owner(at)postgresql(dot)org] On Behalf Of Bruce Momjian
> Sent: Tuesday, August 12, 2008 9:04 PM
> To: Robert Haas
> Cc: Markus Wanner; Matthew T. O'Connor; Tom Lane; Greg Sabino Mullane;
> pgsql-hackers(at)postgresql(dot)org
> Subject: Re: [HACKERS] Plugin system like Firefox
>
> Robert Haas wrote:
> > >> > .. an OS-agnostic way of installing packages.
> > >> Uh.. I don't think such a thing exists.
> > > Seems to in Firefox.
> >
> > I think nearly all Firefox extensions are written in XML and
> > Javascript. It's quite a bit easier to have an OS-agnostic way of
> > installing packages when the packages are platform-independent.
> >
> > >> > I realize many prefer their OS-native packaging system, but
> > >> > that isn't the target audience of a packaging system that will
> increase
> > >> > adoption.
> >
> > Really? Who is it?
>
> People who want to try a database and its plugins to see if they like
> it.

For what it's worth, I am rather fond of the firefox plugin for sqlite.

Now, SQLite has very little in the way of administrative tools, so that
tilts the field, but I guess other people might toy with PostgreSQL if a
plugin were available and might not otherwise just because one is
available.

On Aug 12, 2008, at 2:26 AM, Dave Page wrote:
> On Tue, Aug 12, 2008 at 4:13 AM, Bruce Momjian <bruce(at)momjian(dot)us>
> wrote:
>> So, ideally, if we do a plug-in system, I think we need some way
>> to have
>> these plugins be very easily installed, perhaps by choosing object
>> files
>> pre-compile by the build farm for each operating system.
>
> Hmm, that idea sounds remarkably familiar <thinks back to a meeting in
> NJ about 2 years ago>

...and one at Pervasive even before that...
--
Decibel!, aka Jim C. Nasby, Database Architect decibel(at)decibel(dot)org
Give your computer some brain candy! www.distributed.net Team #1828

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

Le 12 août 08 à 18:41, Bruce Momjian a écrit :
>> Are you proposing that we should introduce our own packaging system
>> for
>> such add-on components? Like CP(g)AN (which has been proposed
>> before..)?
>
> Yes, pretty much. I imagine some kind of web interface or Java app
> (did
> I just say that?) that lists all plugins and when you choose one it
> downloads an object file appropriate for your operating system plus
> SQL
> scripts and somehow automatically installs them in the desired
> database.
>
> That is the kind of capability we need to really advance things. We
> would still allow source installs but for people wanting to try things
> out, I see no other alternative, and "try things out" == adoption.

What is wrong with my proposal here:
http://archives.postgresql.org/pgsql-hackers/2008-07/msg01098.php
http://archives.postgresql.org/pgsql-hackers/2008-07/msg01116.php

I tried to address binary distribution, PostgreSQL stamped packages,
distributions packaging compliance (deb and rpm and etc), and ease of
use at user level (a package embeds .so file(s) with any SQL you need).

The proposal mention SQL level interface (create/drop/install package)
and command-line tools to fetch from internet, build with pgxs and
install the prepared package into the given database. The binary
distribution is left to people in charge in my proposal, they are
debian packagers, RPM ones like Devrim, or Win32 Installer team.

Please tell me how far from a workable proposal I am...

Regards,
- --
dim

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iEYEARECAAYFAkinRGoACgkQlBXRlnbh1bkTOgCfT7xerFH6i5Q2g35djcYoYGLI
7bMAn2/WqGyzVwf/8KAacpo9QCuYxv+G
=lZqK
-----END PGP SIGNATURE-----