Re: [patch] build issues on Win32

I've run across a couple of stumbling blocks when building on Win32
(specifically, XP + MinGW):

1. PostgreSQL's private versions of inet_aton etc. can conflict with
similar functions in other libraries (in my case, PostgreSQL's
inet_aton conflicts with libavformat's).

2. On Win32, PostgreSQL uses the SSPI authentication interface. This
interface is implemented in secur32.dll, which needs to be added to
the linker command line. This is apparently only an issue when
building a static libpq.

3. src/template/win32 sets LDFLAGS unconditionally, overriding anything
the user may have specified in the environment or on the command
line (such as -L/path/to/my/lib/dir).

The attached patch addresses these issues by:

1. #defining the function names as appropriate

2. adding -lsecur32 to LIBS in src/Makefile.global.in when PORTNAME is
win32.

3. adding ${LDFLAGS} at the front of the LDFLAGS redefinition in
src/template/win32.

The patch was developed and tested on 8.3.9, because that's what my
customer uses. I have verified that it applies cleanly (albeit with
offsets) to 8.4.2.

BTW, IWBNI there were a quick and easy way to build and install only
libpq. I use this sequence of commands (after configure):

$ make -C src/port all
$ make -C src/backend utils/fmgroids.h
$ make -C src/backend ../../src/include/utils/fmgroids.h
$ make -C src/include all install
$ make -C src/interfaces/libpq all install
$ make -C src/bin/pg_config all install

DES
--
Dag-Erling Smørgrav - des(at)des(dot)no

=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= <des(at)des(dot)no> writes:
> I've run across a couple of stumbling blocks when building on Win32
> (specifically, XP + MinGW):

> 1. PostgreSQL's private versions of inet_aton etc. can conflict with
> similar functions in other libraries (in my case, PostgreSQL's
> inet_aton conflicts with libavformat's).

So what? We don't link with those libraries. The proposed #defines
seem like a completely bad idea, especially since as-presented they
would affect every platform not only yours. We don't need the
maintenance/debugging headaches of routines that don't have the names
they appear to have.

> ifeq ($(PORTNAME),win32)
> -LIBS += -lws2_32 -lshfolder
> +LIBS += -lws2_32 -lshfolder -lsecur32
> endif

Surely this bit would need to be conditional on whether libsecur32
is available?

> -LDFLAGS="-Wl,--allow-multiple-definition"
> +LDFLAGS="${LDFLAGS} -Wl,--allow-multiple-definition"

That bit seems sane.

regards, tom lane

Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:
> "Dag-Erling Smørgrav" <des(at)des(dot)no> writes:
> > 1. PostgreSQL's private versions of inet_aton etc. can conflict with
> > similar functions in other libraries (in my case, PostgreSQL's
> > inet_aton conflicts with libavformat's).
> So what? We don't link with those libraries.

Your users might need to link with both. I'm working on an application
that generates animations (specifically, animated weather forecasts)
based on data retrieved from a PostgreSQL database.

> The proposed #defines seem like a completely bad idea, especially
> since as-presented they would affect every platform not only yours.

Yes. That's the idea. This is a common idiom - the canonical way, if
you will, of solving this problem, which is not exclusive to PostgreSQL.
There are even cases in which you have no other choice, e.g. when the
function you want to use is available but does not work properly or does
not implement a particular feature that you need.

> We don't need the maintenance/debugging headaches of routines that
> don't have the names they appear to have.

Honestly, I don't see the problem.

> > ifeq ($(PORTNAME),win32)
> > -LIBS += -lws2_32 -lshfolder
> > +LIBS += -lws2_32 -lshfolder -lsecur32
> > endif
> Surely this bit would need to be conditional on whether libsecur32
> is available?

It's just as much part of Win32 as ws2_32 (winsock).

DES
--
Dag-Erling Smørgrav - des(at)des(dot)no

2010/3/10 Dag-Erling Smørgrav <des(at)des(dot)no>:
> Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:
>> "Dag-Erling Smørgrav" <des(at)des(dot)no> writes:
>> >  1. PostgreSQL's private versions of inet_aton etc. can conflict with
>> >     similar functions in other libraries (in my case, PostgreSQL's
>> >     inet_aton conflicts with libavformat's).
>> So what?  We don't link with those libraries.
>
> Your users might need to link with both.  I'm working on an application
> that generates animations (specifically, animated weather forecasts)
> based on data retrieved from a PostgreSQL database.

This shows up only with static links of libpq, correct? Or are you
building a backend function?

>> The proposed #defines seem like a completely bad idea, especially
>> since as-presented they would affect every platform not only yours.
>
> Yes.  That's the idea.  This is a common idiom - the canonical way, if
> you will, of solving this problem, which is not exclusive to PostgreSQL.
> There are even cases in which you have no other choice, e.g. when the
> function you want to use is available but does not work properly or does
> not implement a particular feature that you need.
>
>> We don't need the maintenance/debugging headaches of routines that
>> don't have the names they appear to have.
>
> Honestly, I don't see the problem.
>
>> >  ifeq ($(PORTNAME),win32)
>> > -LIBS += -lws2_32 -lshfolder
>> > +LIBS += -lws2_32 -lshfolder -lsecur32
>> >  endif
>> Surely this bit would need to be conditional on whether libsecur32
>> is available?
>
> It's just as much part of Win32 as ws2_32 (winsock).

Yeah, it's a standard win32 library so that's not a problem.

But the fix seems wrong. If you are using a static libpq, the library
should be added whenever you link that library into your client
application. Not for every single EXE and DLL that postgres produces.
The makefiles already add it to the places in PostgreSQL where it's
needed - namely postgres.exe and libpq.dll.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= <des(at)des(dot)no> writes:
> Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:
>> "Dag-Erling Smrgrav" <des(at)des(dot)no> writes:
>>> 1. PostgreSQL's private versions of inet_aton etc. can conflict with
>>> similar functions in other libraries (in my case, PostgreSQL's
>>> inet_aton conflicts with libavformat's).

>> So what? We don't link with those libraries.

> Your users might need to link with both.

We don't support linking the backend into other applications. If you're
complaining about libpq, the right thing for that is to use a platform
that can suppress non-exported symbols from a shared library. Maybe
what we need to do is teach the mingw build path how to respect the
exports list for libpq?

regards, tom lane

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> Dag-Erling Smørgrav <des(at)des(dot)no> writes:
> > Your users might need to link with both.  I'm working on an
> > application that generates animations (specifically, animated
> > weather forecasts) based on data retrieved from a PostgreSQL
> > database.
> This shows up only with static links of libpq, correct?

Yes.

> But the fix seems wrong. If you are using a static libpq, the library
> should be added whenever you link that library into your client
> application. Not for every single EXE and DLL that postgres produces.

Without this patch, pg_ctl fails to build... and I don't see the logic
of including ws2_32 in LIBS but not secur32. remember that LIBS
directly affects what pg_config reports, so if you don't add secur32 to
LIBS, consumers (applications) don't know that they need it.

DES
--
Dag-Erling Smørgrav - des(at)des(dot)no

Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:
> We don't support linking the backend into other applications.

libpq uses this as well.

> If you're complaining about libpq, the right thing for that is to use
> a platform that can suppress non-exported symbols from a shared
> library. Maybe what we need to do is teach the mingw build path how
> to respect the exports list for libpq?

If that works, I'm all for it. I have no idea how to do it, though.

DES
--
Dag-Erling Smørgrav - des(at)des(dot)no

=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= <des(at)des(dot)no> writes:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>> But the fix seems wrong. If you are using a static libpq, the library
>> should be added whenever you link that library into your client
>> application. Not for every single EXE and DLL that postgres produces.

> Without this patch, pg_ctl fails to build...

It builds for everybody else (and we do have multiple mingw machines in
the buildfarm, so it's not like this doesn't get tested). I think there
is some other factor involved here, and you need to identify what that
is.

regards, tom lane

Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:
> "Dag-Erling Smørgrav" <des(at)des(dot)no> writes:
> > Without this patch, pg_ctl fails to build...
> It builds for everybody else (and we do have multiple mingw machines in
> the buildfarm, so it's not like this doesn't get tested). I think there
> is some other factor involved here, and you need to identify what that
> is.

--disable-shared, as previously mentioned.

DES
--
Dag-Erling Smørgrav - des(at)des(dot)no

=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= <des(at)des(dot)no> writes:
> Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:
>> It builds for everybody else (and we do have multiple mingw machines in
>> the buildfarm, so it's not like this doesn't get tested). I think there
>> is some other factor involved here, and you need to identify what that
>> is.

> --disable-shared, as previously mentioned.

Oh. Well, we don't really support that, and there is a proposal on the
table to remove it altogether from the configure script. I don't think
we're going to contort our source code in order to make a marginal
improvement in the ability to coexist with random other code that is
also polluting the link-time namespace.

regards, tom lane

On Wed, Mar 10, 2010 at 01:28:55PM -0500, Tom Lane wrote:
> =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= <des(at)des(dot)no> writes:
> > Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:
> >> It builds for everybody else (and we do have multiple mingw
> >> machines in the buildfarm, so it's not like this doesn't get
> >> tested). I think there is some other factor involved here, and
> >> you need to identify what that is.
>
> > --disable-shared, as previously mentioned.
>
> Oh. Well, we don't really support that, and there is a proposal on
> the table to remove it altogether from the configure script. I
> don't think we're going to contort our source code in order to make
> a marginal improvement in the ability to coexist with random other
> code that is also polluting the link-time namespace.

+1 for de-supporting this option.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

2010/3/10 David Fetter <david(at)fetter(dot)org>:
>> > --disable-shared, as previously mentioned.
>>
>> Oh.  Well, we don't really support that, and there is a proposal on
>> the table to remove it altogether from the configure script.  I
>> don't think we're going to contort our source code in order to make
>> a marginal improvement in the ability to coexist with random other
>> code that is also polluting the link-time namespace.
>
> +1 for de-supporting this option.

I would be sad about this. It seems likely there are platforms where
it's important. But I'm not really about to spend the effort to fix it
up myself and I agree it wouldn't be worth hacking the source to get
it to work. I'm a bit puzzled why the symbol conflicts occur only with
static linking though -- it seems like static linking would give more
opportunity to isolate symbols than dynamic linking, not less. Perhaps
our static linking rules are broken?

--
greg

Greg Stark <gsstark(at)mit(dot)edu> writes:
> I would be sad about this. It seems likely there are platforms where
> it's important. But I'm not really about to spend the effort to fix it
> up myself and I agree it wouldn't be worth hacking the source to get
> it to work. I'm a bit puzzled why the symbol conflicts occur only with
> static linking though -- it seems like static linking would give more
> opportunity to isolate symbols than dynamic linking, not less. Perhaps
> our static linking rules are broken?

A dynamic library is a single entity with a certain degree of isolation.
You can hide symbols so they are only visible within that library.

A static library is basically just a tarball of individual relocatable
objects.

The GNU toolchain (and probably others too) allows you to combine
several relocatable objects together into one and hide some of the
symbols, as if the entire thing had been a single C file with some of
the functions declared as static. However, when you do that, you lose
an important advantage of static libraries: the ability to link only
what you need.

DES
--
Dag-Erling Smørgrav - des(at)des(dot)no

2010/3/11 Dag-Erling Smørgrav <des(at)des(dot)no>:
> Greg Stark <gsstark(at)mit(dot)edu> writes:
>> I would be sad about this. It seems likely there are platforms where
>> it's important. But I'm not really about to spend the effort to fix it
>> up myself and I agree it wouldn't be worth hacking the source to get
>> it to work. I'm a bit puzzled why the symbol conflicts occur only with
>> static linking though -- it seems like static linking would give more
>> opportunity to isolate symbols than dynamic linking, not less. Perhaps
>> our static linking rules are broken?
>
> A dynamic library is a single entity with a certain degree of isolation.
> You can hide symbols so they are only visible within that library.

In particular, libpq only exports a fixed subset of symbols on any
platform that supports that (which includes Windows).

AFAIK, there is no way to make that restriction on static libraries,
at least not on Windows.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> In particular, libpq only exports a fixed subset of symbols on any
> platform that supports that (which includes Windows).

> AFAIK, there is no way to make that restriction on static libraries,
> at least not on Windows.

Right, and not anyplace else either except by pre-linking the contained
.o files, which eliminates one of the claimed advantages of a .a library.

So one of the reasons not to support a static version of libpq is
precisely this one: it means that every non-static symbol inside libpq
is subject to conflicts against calling applications and other libraries
that they choose to link with. I don't particularly care to deal with
that, and I especially don't choose to accept the position that any such
conflicts are *our* problem to fix.

regards, tom lane

Greg Stark <gsstark(at)mit(dot)edu> writes:
> 2010/3/10 David Fetter <david(at)fetter(dot)org>:
>>>> --disable-shared, as previously mentioned.
>>
>> +1 for de-supporting this option.

> I would be sad about this. It seems likely there are platforms where
> it's important.

Any such platform would already be contending with plpgsql not working,
encoding conversion not working, etc etc. It's barely conceivable that
a client-only installation would be useful. But given that nobody has
actually proposed supporting such a platform in the past ten years,
I don't think one's likely to come out of the woodwork now. AFAICT
the only case where anyone tries to do this is they have a personal
preference to avoid shared libraries, for generally-pretty-darn-dubious
reasons.

Red Hat has developed a pretty strict policy against even shipping
static libraries, because it's such a PITA to deal with updates.
Let me give you a fresh-in-mind example: there is an open security bug
against libpng (which I package for Red Hat in my copious spare time).
I was distressed to find that fixing the bug left firefox still failing
against a test web page. Investigation disclosed that the reason for
this is that firefox is using a private static copy of libpng. That was
a stupid decision on their part, as it means they're going to have to be
involved in fixing this issue, not to mention past and future issues.

Now libpq doesn't often have critical security bugs filed against it,
but it certainly has bugs. Do you really want to have to remember to
rebuild every piece of dependent software when you update it? The OP's
case apparently involves multiple independent libraries that he wants to
link statically, which makes the problem multiple times worse.

So my position is that static linking has enough negatives that you
need a lot more than a hypothetical use-case to justify it.

regards, tom lane

2010/3/11 Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>:
> Now libpq doesn't often have critical security bugs filed against it,
> but it certainly has bugs.  Do you really want to have to remember to
> rebuild every piece of dependent software when you update it?

I absolutely agree that linking statically is a terrible idea for
distributions. Debian has a similar prohibition against linking
statically against any other package.

But it does ship static libraries because commercial binary-only
packages often require them. They have to be packaged to work on
multiple OS distributions including some which don't provide
especially smooth dependency systems. You don't want to get calls from
customers running your software against random library versions you've
never tested.

The two (separate) goals which are useful are 1) Provide a library
others can link against to produce a binary which has no run-time
dependency on your library. In this case the only library they might
want to link against would be libpq. The encoding libraries etc aren't
things they're going to link agains. And 2) build binaries which have
no dependencies on system libraries so someone can ship them and
expect them to run on any system regardless of the run-time
environment.

I agree that these are both over-used but they are sometimes the least
bad option. On the other hand, the third goal "avoid using the shared
library facilities" is pointless, I see no reason to avoid building
and loading the encoding code or the contrib modules. They're using
the same technology as shared libraries but they're not really shared
libraries in the sense of being shipped separately from the binaries
using them.

--
greg

Tom Lane wrote:
> Now libpq doesn't often have critical security bugs filed against it,
> but it certainly has bugs. Do you really want to have to remember to
> rebuild every piece of dependent software when you update it? The OP's
> case apparently involves multiple independent libraries that he wants to
> link statically, which makes the problem multiple times worse.
>
> So my position is that static linking has enough negatives that you
> need a lot more than a hypothetical use-case to justify it.
>
>

+1.

cheers

andrew

Greg Stark <gsstark(at)mit(dot)edu> writes:
> The two (separate) goals which are useful are 1) Provide a library
> others can link against to produce a binary which has no run-time
> dependency on your library. In this case the only library they might
> want to link against would be libpq. The encoding libraries etc aren't
> things they're going to link agains. And 2) build binaries which have
> no dependencies on system libraries so someone can ship them and
> expect them to run on any system regardless of the run-time
> environment.

> I agree that these are both over-used but they are sometimes the least
> bad option. On the other hand, the third goal "avoid using the shared
> library facilities" is pointless, I see no reason to avoid building
> and loading the encoding code or the contrib modules. They're using
> the same technology as shared libraries but they're not really shared
> libraries in the sense of being shipped separately from the binaries
> using them.

True. That still makes the current --disable-shared configure option
useless, but it doesn't go as far as suggesting that we ought to
implement --disable-static, which would be the logical conclusion from
my position.

I don't actually want to do --disable-static, but my feeling about it
is "if it breaks you get to keep both pieces". I'm not interested in
providing workarounds to let a static libpq be used in combination
with any-random-other-static-library. That's up to the user of the
thing to deal with.

BTW, I'm not sure I buy the argument that commercial software requires
static linking. Red Hat would be as interested in that market as
anybody, and as I said, they don't think it's necessary to ship static
libraries (with a *very* short list of exceptions).

regards, tom lane

On Mar 11, 2010, at 1:06 PM, Tom Lane wrote:
>
>
> BTW, I'm not sure I buy the argument that commercial software requires
> static linking. Red Hat would be as interested in that market as
> anybody, and as I said, they don't think it's necessary to ship static
> libraries (with a *very* short list of exceptions).

Trying to support a binary distributed[1] application on a dozen or more
different distributions is fairly painful (I'm building on four Debians,
eight Ubuntus, several RHELs, three Solarises and a few others
right now), and the biggest problem is the different distributions
don't package all the libraries you need, and when they do they
include a wide range of versions, often woefully obsolete.

It's a serious pain. The "woefully obsolete" bit also means that you
have to code to the lowest common denominator - you cannot
use any feature more recent than the obsolete library that the
oldest version of RHEL you support has, nor any feature that's
been deprecated in the latest bleeding edge Ubuntu release

(There are some APIs where an old feature has been replaced
by a new feature, and the old one deprecated and removed
over a timeframe short enough that conservative distributions
only have the old one and more bleeding edge distributions
only have the new one. libpq is stable enough and backwards
compatible enough that it's never been an issue there).

Building applications mostly statically linked avoids the vast
majority of those problems. It's not the only way to do it, and
I don't think it's ever the best way to do it as it can lead to
a bunch of other problems but it's often the easiest fix for
getting cross-platform support working right now, and I
can understand why people want to be able to do it.

Cheers,
Steve

[1] Binary distributed is not the same as commercial, though
a lot of commercial code is distributed as binaries so it's
one place where the problem comes up. Commercial
software is also more limited in it's ability to just blame
the OS vendor and ignore the issue.

On Thu, Mar 11, 2010 at 01:34:34PM -0800, Steve Atkins wrote:
> On Mar 11, 2010, at 1:06 PM, Tom Lane wrote:
> >
> > BTW, I'm not sure I buy the argument that commercial software
> > requires static linking. Red Hat would be as interested in that
> > market as anybody, and as I said, they don't think it's necessary
> > to ship static libraries (with a *very* short list of exceptions).
>
> Trying to support a binary distributed[1] application on a dozen or
> more different distributions is fairly painful (I'm building on four
> Debians, eight Ubuntus, several RHELs, three Solarises and a few
> others right now), and the biggest problem is the different
> distributions don't package all the libraries you need, and when
> they do they include a wide range of versions, often woefully
> obsolete.

I've found it easiest to ship all of PostgreSQL rather than deal with
this kind of stuff.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:
> Any such platform would already be contending with plpgsql not working,
> encoding conversion not working, etc etc. It's barely conceivable that
> a client-only installation would be useful.

Uh, I don't understand why it's so hard to conceive that someone might
need the client library to access a database running on one machine from
an application running on another.

> AFAICT the only case where anyone tries to do this is they have a
> personal preference to avoid shared libraries, for generally-pretty-
> darn-dubious reasons.

Well, statically linked binaries are much easier to distribute, for one;
and there are platforms where shared libraries simply do not exist, or
where the entire system runs in a single namespace. Netware is still
alive and kicking.

DES
--
Dag-Erling Smørgrav - des(at)des(dot)no

2010/3/12 Dag-Erling Smørgrav <des(at)des(dot)no>:
> Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:
>> Any such platform would already be contending with plpgsql not working,
>> encoding conversion not working, etc etc.  It's barely conceivable that
>> a client-only installation would be useful.
>
> Uh, I don't understand why it's so hard to conceive that someone might
> need the client library to access a database running on one machine from
> an application running on another.

Well, in this specific case, we're talking Windows, right? Why not
just distribute a libpq.dll along with the application, and you're
done?

It's not that people don't realize it may be useful. Just that it's
not useful enough to enough people to put lots of workarounds in th
ecode to make it work.

>> AFAICT the only case where anyone tries to do this is they have a
>> personal preference to avoid shared libraries, for generally-pretty-
>> darn-dubious reasons.
>
> Well, statically linked binaries are much easier to distribute, for one;
> and there are platforms where shared libraries simply do not exist, or
> where the entire system runs in a single namespace.  Netware is still
> alive and kicking.

Do we even support netware on liboq? :-)

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On Wed, Mar 10, 2010 at 10:51:23AM -0500, Tom Lane wrote:
> =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= <des(at)des(dot)no> writes:
> > -LDFLAGS="-Wl,--allow-multiple-definition"
> > +LDFLAGS="${LDFLAGS} -Wl,--allow-multiple-definition"
>
> That bit seems sane.

I've committed that part. Thanks.

--
Noah Misch
EnterpriseDB http://www.enterprisedb.com