Re: PLUGINS Functionlity in Win32 build scripts

Hi,
I am currently working in EnterpriseDB (www.enterprisedb.com). During the integration of pldebugger ( http://pgfoundry.org/projects/edb-debugger ) with postgres on windows I faced a problem that plugins are not being copied to the lib/plugins directory. Plugins should be copied in (Installation dir)lib/plugins to work properly.
To solve this problem I added PLUGINS logic in the Windows Perl build scripts of PostgreSQL 8.3.3. It searches for PLUGINS variable in the contrib Makefile and processes "PLUGINS" as "MODULES" and copies the generated plugin library to the (Installation dir)lib/plugins.
Please find the attached plugin.patch file. Thanks.

Best Regards,
Asif Naeem
www.enterprisedb.com

_________________________________________________________________
News, entertainment and everything you care about at Live.com. Get it now!
http://www.live.com/getstarted.aspx

MUHAMMAD ASIF wrote:
> During the integration of pldebugger ( http://pgfoundry.org/projects/edb-debugger ) with postgres on windows I faced a problem that plugins are not being copied to the lib/plugins directory. Plugins should be copied in (Installation dir)lib/plugins to work properly.
> To solve this problem I added PLUGINS logic in the Windows Perl build scripts of PostgreSQL 8.3.3. It searches for PLUGINS variable in the contrib Makefile and processes "PLUGINS" as "MODULES" and copies the generated plugin library to the (Installation dir)lib/plugins.
> Please find the attached plugin.patch file. Thanks.

We'll need the same logic on Unix-platforms as well.

I've added this to the November commitfest Wiki page.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On Tue, Sep 9, 2008 at 3:04 AM, MUHAMMAD ASIF <anaeem(dot)it(at)hotmail(dot)com> wrote:
> I am currently working in EnterpriseDB (www.enterprisedb.com). During the integration of pldebugger ( http://pgfoundry.org/projects/edb-debugger ) with postgres on windows I faced a problem that plugins are not being copied to the lib/plugins directory. Plugins should be copied in (Installation dir)lib/plugins to work properly.
> To solve this problem I added PLUGINS logic in the Windows Perl build scripts of PostgreSQL 8.3.3. It searches for PLUGINS variable in the contrib Makefile and processes "PLUGINS" as "MODULES" and copies the generated plugin library to the (Installation dir)lib/plugins.
> Please find the attached plugin.patch file. Thanks.

Hi,

I've been assigned to review this patch, but I confess I'm a little
murky on what problem it's trying to solve. Can you explain what I
need to do to recreate the problem?

There's also this comment on the Wiki (not sure why it wasn't posted
to the mailing list...): "Dave Page says: This doesn't work with the
edb-debugger plugin, which is the only such plugin around AFAIK. It
needs to ignore comments on the PLUGINS line, and handle multiple
targets (plugin_debugger, pldbgapi, targetinfo etc). Not sure if we
want that complexity though." Any comments?

Thanks,

...Robert

> I've been assigned to review this patch, but I confess I'm a little
> murky on what problem it's trying to solve. Can you explain what I
> need to do to recreate the problem?
>
> There's also this comment on the Wiki (not sure why it wasn't posted
> to the mailing list...): "Dave Page says: This doesn't work with the
> edb-debugger plugin, which is the only such plugin around AFAIK. It
> needs to ignore comments on the PLUGINS line, and handle multiple
> targets (plugin_debugger, pldbgapi, targetinfo etc). Not sure if we
> want that complexity though." Any comments?

Since there has been no response to this email, I am going to move
this patch to the "Rejected" section on the commitfest Wiki.

...Robert

Robert,
I am sorry about this, This missed my attention. Here is my reply to your mail.
> Date: Fri, 28 Nov 2008 08:42:39 -0500> From: robertmhaas(at)gmail(dot)com> To: anaeem(dot)it(at)hotmail(dot)com> Subject: Re: [HACKERS] PLUGINS Functionlity in Win32 build scripts> CC: pgsql-hackers(at)postgresql(dot)org> > On Tue, Sep 9, 2008 at 3:04 AM, MUHAMMAD ASIF <anaeem(dot)it(at)hotmail(dot)com> wrote:> > I am currently working in EnterpriseDB (www.enterprisedb.com). During the integration of pldebugger ( http://pgfoundry.org/projects/edb-debugger ) with postgres on windows I faced a problem that plugins are not being copied to the lib/plugins directory. Plugins should be copied in (Installation dir)lib/plugins to work properly.> > To solve this problem I added PLUGINS logic in the Windows Perl build scripts of PostgreSQL 8.3.3. It searches for PLUGINS variable in the contrib Makefile and processes "PLUGINS" as "MODULES" and copies the generated plugin library to the (Installation dir)lib/plugins.> > Please find the attached plugin.patch file. Thanks.> > Hi,> > I've been assigned to review this patch, but I confess I'm a little> murky on what problem it's trying to solve. Can you explain what I> need to do to recreate the problem?
In postgreSQL, Plugin modules should be installed in (Installation dir)lib/plugins to run properly. There is logic in src/makefiles/pgxs.mk for handling shared objects implemented as e.g. "Modules" variable that are installed in (Installation dir)/lib. There should be some way to handle plugins as well. We faced this issue during the integration of contrib module pldebugger ( http://pgfoundry.org/frs/?group_id=1000175 ) with the edb-postgresql code. pldebugger/Makefile handles plugins by itself, Unix/Linux don’t mind it because it build through make. On windows, perl scripts are used to build the system that relies on Makefiles that uses pgxs.mk variables e.g "Modules" etc. It fails to build pldebugger that implements its own plugin build logic in its Makefile. Right now for us there is only one plugin module but in future there may be more.Instead of any workaround we come up to a better solution to handle plugins automatically by pgxs.mk as other shared objects are being handled on Unix and Windows. We added PLUGIN logic in pgxs.mk and windows perl build scripts so that there is no need to handle plugins separately by new contrib modules.

> > There's also this comment on the Wiki (not sure why it wasn't posted> to the mailing list...): "Dave Page says: This doesn't work with the> edb-debugger plugin, which is the only such plugin around AFAIK. It> needs to ignore comments on the PLUGINS line, and handle multiple> targets (plugin_debugger, pldbgapi, targetinfo etc). Not sure if we> want that complexity though." Any comments?
Dave was refering pldebugger code that uses comments in its Makefile.
pldebugger/Makefile : Line # 36
PLUGINS = plugin_debugger # plugin_profiler plugin_tracer

Make tool ignores the comments but windows perl scripts don’t understand "#" in process of extracting module names.
It is a separate functionality to enhance the scripts so that it can understand commented code with "#".I would love to make these changes separately but it don’t seems to be in the scope of PLUGINS functionality.

> > Thanks,> > ...Robert> > -- > Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)> To make changes to your subscription:> http://www.postgresql.org/mailpref/pgsql-hackers
I am sending you patches that i tested with postgresql-8.3.4. I am looking forward to your quries :).Thanks.

Best Regards,
M. Asif Naeem
www.enterprisedb.com

_________________________________________________________________
Discover the new Windows Vista
http://search.msn.com/results.aspx?q=windows+vista&mkt=en-US&form=QBRE

>> I've been assigned to review this patch, but I confess I'm a little
>> murky on what problem it's trying to solve. Can you explain what I
>> need to do to recreate the problem?
> In postgreSQL, Plugin modules should be installed in (Installation
> dir)lib/plugins to run properly. There is logic in src/makefiles/pgxs.mk for
> handling shared objects implemented as e.g. "Modules" variable that are
> installed in (Installation dir)/lib. There should be some way to handle
> plugins as well. We faced this issue during the integration of contrib
> module pldebugger ( http://pgfoundry.org/frs/?group_id=1000175 ) with the
> edb-postgresql code. pldebugger/Makefile handles plugins by itself,
> Unix/Linux don't mind it because it build through make. On windows, perl
> scripts are used to build the system that relies on Makefiles that uses
> pgxs.mk variables e.g "Modules" etc. It fails to build pldebugger that
> implements its own plugin build logic in its Makefile. Right now for us
> there is only one plugin module but in future there may be more.Instead of
> any workaround we come up to a better solution to handle plugins
> automatically by pgxs.mk as other shared objects are being handled on Unix
> and Windows. We added PLUGIN logic in pgxs.mk and windows perl build scripts
> so that there is no need to handle plugins separately by new contrib
> modules.

I am a bit dubious about the idea of adding supposedly generic
functionality to cater to a single client, but the bigger problem is
that even after reading this I still don't really know what I'm
supposed to be looking at. I think you're saying that pldebugger
contains some code that could be made simpler and less easily broken
if this patch were applied. Is that correct? If so, please tell me
where to download the pldebugger code and which file(s) to look in for
the code that could be improved.

...Robert

I am on it , I understand your concern. I will send you the sample code and pldebugger details to test the logic ASAP .Thanks.> Date: Tue, 9 Dec 2008 23:36:42 -0500> From: robertmhaas(at)gmail(dot)com> To: anaeem(dot)it(at)hotmail(dot)com> Subject: Re: [HACKERS] PLUGINS Functionlity in Win32 build scripts> CC: pgsql-hackers(at)postgresql(dot)org> > >> I've been assigned to review this patch, but I confess I'm a little> >> murky on what problem it's trying to solve. Can you explain what I> >> need to do to recreate the problem?> > In postgreSQL, Plugin modules should be installed in (Installation> > dir)lib/plugins to run properly. There is logic in src/makefiles/pgxs.mk for> > handling shared objects implemented as e.g. "Modules" variable that are> > installed in (Installation dir)/lib. There should be some way to handle> > plugins as well. We faced this issue during the integration of contrib> > module pldebugger ( http://pgfoundry.org/frs/?group_id=1000175 ) with the> > edb-postgresql code. pldebugger/Makefile handles plugins by itself,> > Unix/Linux don't mind it because it build through make. On windows, perl> > scripts are used to build the system that relies on Makefiles that uses> > pgxs.mk variables e.g "Modules" etc. It fails to build pldebugger that> > implements its own plugin build logic in its Makefile. Right now for us> > there is only one plugin module but in future there may be more.Instead of> > any workaround we come up to a better solution to handle plugins> > automatically by pgxs.mk as other shared objects are being handled on Unix> > and Windows. We added PLUGIN logic in pgxs.mk and windows perl build scripts> > so that there is no need to handle plugins separately by new contrib> > modules.> > I am a bit dubious about the idea of adding supposedly generic> functionality to cater to a single client, but the bigger problem is> that even after reading this I still don't really know what I'm> supposed to be looking at. I think you're saying that pldebugger> contains some code that could be made simpler and less easily broken> if this patch were applied. Is that correct? If so, please tell me> where to download the pldebugger code and which file(s) to look in for> the code that could be improved.> > ...Robert
_________________________________________________________________
Invite your mail contacts to join your friends list with Windows Live Spaces. It's easy!
http://spaces.live.com/spacesapi.aspx?wx_action=create&wx_url=/friends.aspx&mkt=en-us

Purpose of this Plugins change is to add support in postgresql source to build plugin and install in (install dir)/lib/plugins. We made these changes for a plugin module i.e. pldebugger and we found this a useful feature that can help postgresql in adding plugins easily.
Attached file (foo.zip) is a test contrib module code that can be used to test PLUGINS logic ( patches.zip patch I have sent you in the last email). Following are the steps to build foo.zip i.e
a. Apply patches plugins_make.patch and plugins_msvc.patch.
b. Unzip and copy the foo directory in the contrib directory
c. Build and Install the system
It will build foo library and place it in lib/plugins directory.You can test it on both windows and linux/unix platforms.Thanks.

Best Regards,
Asif Naeem

> Date: Tue, 9 Dec 2008 23:36:42 -0500
> From: robertmhaas(at)gmail(dot)com
> To: anaeem(dot)it(at)hotmail(dot)com
> Subject: Re: [HACKERS] PLUGINS Functionlity in Win32 build scripts
> CC: pgsql-hackers(at)postgresql(dot)org
>
>>> I've been assigned to review this patch, but I confess I'm a little
>>> murky on what problem it's trying to solve. Can you explain what I
>>> need to do to recreate the problem?
>> In postgreSQL, Plugin modules should be installed in (Installation
>> dir)lib/plugins to run properly. There is logic in src/makefiles/pgxs.mk for
>> handling shared objects implemented as e.g. "Modules" variable that are
>> installed in (Installation dir)/lib. There should be some way to handle
>> plugins as well. We faced this issue during the integration of contrib
>> module pldebugger ( http://pgfoundry.org/frs/?group_id=1000175 ) with the
>> edb-postgresql code. pldebugger/Makefile handles plugins by itself,
>> Unix/Linux don't mind it because it build through make. On windows, perl
>> scripts are used to build the system that relies on Makefiles that uses
>> pgxs.mk variables e.g "Modules" etc. It fails to build pldebugger that
>> implements its own plugin build logic in its Makefile. Right now for us
>> there is only one plugin module but in future there may be more.Instead of
>> any workaround we come up to a better solution to handle plugins
>> automatically by pgxs.mk as other shared objects are being handled on Unix
>> and Windows. We added PLUGIN logic in pgxs.mk and windows perl build scripts
>> so that there is no need to handle plugins separately by new contrib
>> modules.
>
> I am a bit dubious about the idea of adding supposedly generic
> functionality to cater to a single client, but the bigger problem is
> that even after reading this I still don't really know what I'm
> supposed to be looking at. I think you're saying that pldebugger
> contains some code that could be made simpler and less easily broken
> if this patch were applied. Is that correct? If so, please tell me
> where to download the pldebugger code and which file(s) to look in for
> the code that could be improved.
>
> ...Robert
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers

_________________________________________________________________
Discover the new Windows Vista
http://search.msn.com/results.aspx?q=windows+vista&mkt=en-US&form=QBRE

Pldebugger uses its own PLUGIN implementation that is ok for make utility but when we try to compile the code on Windows Perl build scripts fail because it don’t understand Makefile without pgxs.mk variables. If we add plugin logic in pgxs.mk and windows Perl build scripts , it would be simple to add a plugin module as contrib without much effort. Actually pldebugger gave us this idea that there should be some built in logic to build plugins in postgresql code. You can find pldebugger latest code at http://pgfoundry.org/frs/?group_id=1000175. PLUGINS logic is a partial thing, There are some other steps involved to build pldebugger on linux and windows if you like I can send you the more details on that. Thanks.

> From: anaeem(dot)it(at)hotmail(dot)com
> To: robertmhaas(at)gmail(dot)com
> CC: pgsql-hackers(at)postgresql(dot)org
> Subject: Re: [HACKERS] PLUGINS Functionlity in Win32 build scripts
> Date: Thu, 18 Dec 2008 00:24:46 +0500
>
>
> Purpose of this Plugins change is to add support in postgresql source to build plugin and install in (install dir)/lib/plugins. We made these changes for a plugin module i.e. pldebugger and we found this a useful feature that can help postgresql in adding plugins easily.
> Attached file (foo.zip) is a test contrib module code that can be used to test PLUGINS logic ( patches.zip patch I have sent you in the last email). Following are the steps to build foo.zip i.e
> a. Apply patches plugins_make.patch and plugins_msvc.patch.
> b. Unzip and copy the foo directory in the contrib directory
> c. Build and Install the system
> It will build foo library and place it in lib/plugins directory.You can test it on both windows and linux/unix platforms.Thanks.
>
> Best Regards,
> Asif Naeem
>
>> Date: Tue, 9 Dec 2008 23:36:42 -0500
>> From: robertmhaas(at)gmail(dot)com
>> To: anaeem(dot)it(at)hotmail(dot)com
>> Subject: Re: [HACKERS] PLUGINS Functionlity in Win32 build scripts
>> CC: pgsql-hackers(at)postgresql(dot)org
>>
>>>> I've been assigned to review this patch, but I confess I'm a little
>>>> murky on what problem it's trying to solve. Can you explain what I
>>>> need to do to recreate the problem?
>>> In postgreSQL, Plugin modules should be installed in (Installation
>>> dir)lib/plugins to run properly. There is logic in src/makefiles/pgxs.mk for
>>> handling shared objects implemented as e.g. "Modules" variable that are
>>> installed in (Installation dir)/lib. There should be some way to handle
>>> plugins as well. We faced this issue during the integration of contrib
>>> module pldebugger ( http://pgfoundry.org/frs/?group_id=1000175 ) with the
>>> edb-postgresql code. pldebugger/Makefile handles plugins by itself,
>>> Unix/Linux don't mind it because it build through make. On windows, perl
>>> scripts are used to build the system that relies on Makefiles that uses
>>> pgxs.mk variables e.g "Modules" etc. It fails to build pldebugger that
>>> implements its own plugin build logic in its Makefile. Right now for us
>>> there is only one plugin module but in future there may be more.Instead of
>>> any workaround we come up to a better solution to handle plugins
>>> automatically by pgxs.mk as other shared objects are being handled on Unix
>>> and Windows. We added PLUGIN logic in pgxs.mk and windows perl build scripts
>>> so that there is no need to handle plugins separately by new contrib
>>> modules.
>>
>> I am a bit dubious about the idea of adding supposedly generic
>> functionality to cater to a single client, but the bigger problem is
>> that even after reading this I still don't really know what I'm
>> supposed to be looking at. I think you're saying that pldebugger
>> contains some code that could be made simpler and less easily broken
>> if this patch were applied. Is that correct? If so, please tell me
>> where to download the pldebugger code and which file(s) to look in for
>> the code that could be improved.
>>
>> ...Robert
>>
>> --
>> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-hackers
>
> _________________________________________________________________
> Discover the new Windows Vista
> http://search.msn.com/results.aspx?q=windows+vista&mkt=en-US&form=QBRE

_________________________________________________________________
Explore the seven wonders of the world
http://search.msn.com/results.aspx?q=7+wonders+world&mkt=en-US&form=QBRE

MUHAMMAD ASIF <anaeem(dot)it(at)hotmail(dot)com> writes:
> Purpose of this Plugins change is to add support in postgresql source
> to build plugin and install in (install dir)/lib/plugins. We made
> these changes for a plugin module i.e. pldebugger and we found this a
> useful feature that can help postgresql in adding plugins easily.

AFAICT, the idea of this patch is to make shared libraries that
automatically install in $libdir/plugins/ rather than $libdir/.

This is actually in direct contradiction to the original intent of the
plugins/ subdirectory, which was that it only contain libraries that the
local administrator had decided to consider safe and put there manually.
Since the normal superuser-only restrictions for library loading are
bypassed for stuff in plugins/, there's a nontrivial risk of security
problems if stuff just gets put there willy-nilly.

If we want to change this security policy, let's have a discussion about
it in those terms. It's a policy, not a bug to be patched around.

regards, tom lane

Tom Lane escribió:

> AFAICT, the idea of this patch is to make shared libraries that
> automatically install in $libdir/plugins/ rather than $libdir/.
>
> This is actually in direct contradiction to the original intent of the
> plugins/ subdirectory, which was that it only contain libraries that the
> local administrator had decided to consider safe and put there manually.

I think this is fantasy -- the plpgsql debugger already has a makefile
that writes the files directly to the plugins subdir. The admin does
not have to know what's happening, or take any concious decision about
it. He just installs the module (or worse: he just has the module
installed automatically by the "one-click installer").

The only thing this patch does is enable the MSVC build to do it as
easily as the Make-based installation. I don't see the security problem
as this patch's fault.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
> Tom Lane escribi:
>> This is actually in direct contradiction to the original intent of the
>> plugins/ subdirectory, which was that it only contain libraries that the
>> local administrator had decided to consider safe and put there manually.

> I think this is fantasy -- the plpgsql debugger already has a makefile
> that writes the files directly to the plugins subdir.

Well, so are we to take one violation of a security policy as meaning
we should encourage more?

regards, tom lane

I understand you concern about the security issues.Actully it just give us the ease to automate build and installation of plugin modules on both the platforms Unix and Windows. We can use configure options to allow/disallow installation of plugin binaries. I am sure that it will enhance the build process.Thanks.

> To: alvherre(at)commandprompt(dot)com
> CC: anaeem(dot)it(at)hotmail(dot)com; robertmhaas(at)gmail(dot)com; pgsql-hackers(at)postgresql(dot)org
> Subject: Re: [HACKERS] PLUGINS Functionlity in Win32 build scripts
> Date: Thu, 18 Dec 2008 15:43:19 -0500
> From: tgl(at)sss(dot)pgh(dot)pa(dot)us
>
> Alvaro Herrera writes:
>> Tom Lane escribió:
>>> This is actually in direct contradiction to the original intent of the
>>> plugins/ subdirectory, which was that it only contain libraries that the
>>> local administrator had decided to consider safe and put there manually.
>
>> I think this is fantasy -- the plpgsql debugger already has a makefile
>> that writes the files directly to the plugins subdir.
>
> Well, so are we to take one violation of a security policy as meaning
> we should encourage more?
>
> regards, tom lane
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
_________________________________________________________________
News, entertainment and everything you care about at Live.com. Get it now!
http://www.live.com/getstarted.aspx

Tom Lane wrote:
> This is actually in direct contradiction to the original intent of the
> plugins/ subdirectory, which was that it only contain libraries that the
> local administrator had decided to consider safe and put there manually.
> Since the normal superuser-only restrictions for library loading are
> bypassed for stuff in plugins/, there's a nontrivial risk of security
> problems if stuff just gets put there willy-nilly.

By what process or criteria is a local administrator supposed to
evaluate whether a module is safe? (I could make up one, but does one
exist now?)

Moreover, this mechanism appears to be pretty evil towards packaging
systems. You don't really want to make administrators move files around
that are under package manager control. A system table or variable that
lists safe modules would be friendlier, iff you really want to have this
under local administrator control.