Re: Compiling extensions on Windows

<p dir="ltr">If libintl.h and any headers it in turn includes are bundled, there is no longer an issue with NLS. </p>
<p dir="ltr">That was just a workaround for building exts when Pg's headers tried to refer to nonexistent headers when NLS was enabled.</p>
<div class="quote">On 6 Jan 2014 18:57, Sandeep Thakkar &lt;sandeep(dot)thakkar(at)enterprisedb(dot)com&gt; wrote:<br type='attribution'><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Sure. I&#39;ll make the changes so that the next available Windows installers include lbintl.h in $Installdir/include. How about the changes with respect to NLS?<br><div class="gmail_extra"><br><br><div class="gmail_quote">
On Mon, Jan 6, 2014 at 2:44 PM, Dave Page <span dir="ltr">&lt;<a href="mailto:dpage(at)pgadmin(dot)org" target="_blank">dpage(at)pgadmin(dot)org</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On Mon, Jan 6, 2014 at 3:32 AM, Craig Ringer &lt;<a href="mailto:craig(at)2ndquadrant(dot)com">craig(at)2ndquadrant(dot)com</a>&gt; wrote:<br>
&gt; Hi all<br>
&gt;<br>
&gt; Out of personal interest (in pain and suffering) I was recently looking<br>
&gt; into how to compile extensions out-of-tree on Windows using Visual<br>
&gt; Studio (i.e. no PGXS).<br>
&gt;<br>
&gt; It looks like the conventional answer to this is &quot;Do a source build of<br>
&gt; PG, compile your ext in-tree in contrib/, and hope the result is binary<br>
&gt; compatible with release PostgreSQL builds for Windows&quot;. Certainly that&#39;s<br>
&gt; how I&#39;ve been doing it to date.<br>
&gt;<br>
&gt; How about everyone else here? Does anyone actually build and distribute<br>
&gt; extensions out of tree at all?<br>
&gt;<br>
&gt; I&#39;m interested in making the Windows installer distributions a bit more<br>
&gt; extension dev friendly. In particular, I&#39;d really like to see EDB&#39;s<br>
&gt; Windows installers include the libintl.h for the included libintl, since<br>
&gt; its omission, combined with Pg being built with ENABLE_NLS, tends to<br>
&gt; break things horribly. Users can always undefine ENABLE_NLS, but it&#39;s an<br>
&gt; unnecessary roadblock.<br>
<br>
</div>Sandeep, can you work on fixing this please?<br>
<br>
Thanks.<br>
<div class="HOEnZb"><div class="h5"><br>
&gt; Are there any objections from -hackers to including 3rd party headers<br>
&gt; for libs we expose in our public headers in the binary distribution?<br>
&gt;<br>
&gt; Other than bundling 3rd party headers, any ideas/suggestions for how we<br>
&gt; might make ext building saner on Windows?<br>
&gt;<br>
&gt; --<br>
&gt;  Craig Ringer                   <a href="http://www.2ndQuadrant.com/" target="_blank">http://www.2ndQuadrant.com/</a><br>
&gt;  PostgreSQL Development, 24x7 Support, Training &amp; Services<br>
&gt;<br>
&gt;<br>
</div></div><span class="HOEnZb"><font color="#888888">&gt; --<br>
&gt; Sent via pgsql-hackers mailing list (<a href="mailto:pgsql-hackers(at)postgresql(dot)org">pgsql-hackers(at)postgresql(dot)org</a>)<br>
&gt; To make changes to your subscription:<br>
&gt; <a href="http://www.postgresql.org/mailpref/pgsql-hackers" target="_blank">http://www.postgresql.org/mailpref/pgsql-hackers</a><br>
<br>
<br>
<br>
--<br>
Dave Page<br>
Blog: <a href="http://pgsnake.blogspot.com" target="_blank">http://pgsnake.blogspot.com</a><br>
Twitter: @pgsnake<br>
<br>
EnterpriseDB UK: <a href="http://www.enterprisedb.com" target="_blank">http://www.enterprisedb.com</a><br>
The Enterprise PostgreSQL Company<br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Sandeep Thakkar<div><br></div></div>
</div></div>
</blockquote></div>

Okay.

BTW, I just checked that Windows 32bit installer ships the libintl.h. Did
you try if it works for you? Or it requires more headers? Somehow, Windows
64bit installer installer missed it. I'll fix the build script.

On Mon, Jan 6, 2014 at 4:55 PM, Craig Ringer <craig(at)2ndquadrant(dot)com> wrote:

> If libintl.h and any headers it in turn includes are bundled, there is no
> longer an issue with NLS.
>
> That was just a workaround for building exts when Pg's headers tried to
> refer to nonexistent headers when NLS was enabled.
> On 6 Jan 2014 18:57, Sandeep Thakkar <sandeep(dot)thakkar(at)enterprisedb(dot)com>
> wrote:
>
> Sure. I'll make the changes so that the next available Windows installers
> include lbintl.h in $Installdir/include. How about the changes with respect
> to NLS?
>
>
> On Mon, Jan 6, 2014 at 2:44 PM, Dave Page <dpage(at)pgadmin(dot)org> wrote:
>
>> On Mon, Jan 6, 2014 at 3:32 AM, Craig Ringer <craig(at)2ndquadrant(dot)com>
>> wrote:
>> > Hi all
>> >
>> > Out of personal interest (in pain and suffering) I was recently looking
>> > into how to compile extensions out-of-tree on Windows using Visual
>> > Studio (i.e. no PGXS).
>> >
>> > It looks like the conventional answer to this is "Do a source build of
>> > PG, compile your ext in-tree in contrib/, and hope the result is binary
>> > compatible with release PostgreSQL builds for Windows". Certainly that's
>> > how I've been doing it to date.
>> >
>> > How about everyone else here? Does anyone actually build and distribute
>> > extensions out of tree at all?
>> >
>> > I'm interested in making the Windows installer distributions a bit more
>> > extension dev friendly. In particular, I'd really like to see EDB's
>> > Windows installers include the libintl.h for the included libintl, since
>> > its omission, combined with Pg being built with ENABLE_NLS, tends to
>> > break things horribly. Users can always undefine ENABLE_NLS, but it's an
>> > unnecessary roadblock.
>>
>> Sandeep, can you work on fixing this please?
>>
>> Thanks.
>>
>> > Are there any objections from -hackers to including 3rd party headers
>> > for libs we expose in our public headers in the binary distribution?
>> >
>> > Other than bundling 3rd party headers, any ideas/suggestions for how we
>> > might make ext building saner on Windows?
>> >
>> > --
>> > Craig Ringer http://www.2ndQuadrant.com/
>> > PostgreSQL Development, 24x7 Support, Training & Services
>> >
>> >
>> > --
>> > Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
>> > To make changes to your subscription:
>> > http://www.postgresql.org/mailpref/pgsql-hackers
>>
>>
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>>
>
>
>
> --
> Sandeep Thakkar
>
>

--
Sandeep Thakkar

On 01/06/2014 07:44 PM, Sandeep Thakkar wrote:
> Okay.
>
> BTW, I just checked that Windows 32bit installer ships the libintl.h.
> Did you try if it works for you? Or it requires more headers? Somehow,
> Windows 64bit installer installer missed it. I'll fix the build script.

That appears to be the only header missing for simple compilation. I did
some testing to verify that it works.

http://blog.2ndquadrant.com/compiling-postgresql-extensions-visual-studio-windows/

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 01/06/2014 07:44 PM, Sandeep Thakkar wrote:
> Okay.
>
> BTW, I just checked that Windows 32bit installer ships the libintl.h.
> Did you try if it works for you? Or it requires more headers? Somehow,
> Windows 64bit installer installer missed it. I'll fix the build script.

Actually, on second thoughts there were two other issues. One I reported
separately (double-inclusion of pg_config_os.h due to _WIN32 vs WIN32).
The other is worth looking at here.

We don't set __declspec(dllexport) on extension functions automatically
when building stand-alone on Windows. So it's necessary to explicitly
specify PGDLLEXPORT for each function. We seem to work around this in
the Perl build toolchain by forcing export of everything not explicitly
static (which is, btw, a pretty crappy thing we should revisit in favour
of using PGDLLEXPORT and -fvisibility=hidden on Linux).

Instead we should perhaps be adding this automatically via a prototype
generated by PG_FUNCTION_INFO_V1, or adding PGDLLEXPORT to all our
example documentation. I think the latter is preferable because if we
start generating a prototype for the base function in PG_FUNCTION_INFO
when we didn't before it could break existing code.

Comments?

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Craig Ringer <craig(at)2ndquadrant(dot)com> writes:
> We don't set __declspec(dllexport) on extension functions automatically
> when building stand-alone on Windows. So it's necessary to explicitly
> specify PGDLLEXPORT for each function.

I'm not sure I believe this. I don't see any PGDLLEXPORT symbols in any
of the standard contrib modules; how is it that they work?

> Instead we should perhaps be adding this automatically via a prototype
> generated by PG_FUNCTION_INFO_V1, or adding PGDLLEXPORT to all our
> example documentation. I think the latter is preferable because if we
> start generating a prototype for the base function in PG_FUNCTION_INFO
> when we didn't before it could break existing code.

> Comments?

One of the things I've always found particularly vile about Microsoft
is the way that they seem to think it's fine to make people sprinkle
Windows-only droppings throughout code that's supposed to be portable.
I'm not in favor of asking people to write out PGDLLEXPORT manually
on every function unless it's *absolutely* necessary, and the available
evidence suggests to me that it isn't.

So if it's really necessary to change anything here, I'd rather see us
take the approach of hiding it in PG_FUNCTION_INFO_V1. What happens
if we do that and there's also a manually-written prototype?

regards, tom lane

On Sat, Jan 11, 2014 at 5:00 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> Craig Ringer <craig(at)2ndquadrant(dot)com> writes:
> > We don't set __declspec(dllexport) on extension functions automatically
> > when building stand-alone on Windows. So it's necessary to explicitly
> > specify PGDLLEXPORT for each function.
>
> I'm not sure I believe this. I don't see any PGDLLEXPORT symbols in any
> of the standard contrib modules; how is it that they work?
>

They are built through our perl toolkit, which enables exporting of *all*
symbols, regardless of flags in the code.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> On Sat, Jan 11, 2014 at 5:00 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> I'm not sure I believe this. I don't see any PGDLLEXPORT symbols in any
>> of the standard contrib modules; how is it that they work?

> They are built through our perl toolkit, which enables exporting of *all*
> symbols, regardless of flags in the code.

That seems like a perfectly reasonable solution, given the way we use
loadable modules. Excess symbols in the module shouldn't really do
any harm. Can't we just document the flags to use for this, if you're
building in some other way?

regards, tom lane

On Sat, Jan 11, 2014 at 7:05 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> Magnus Hagander <magnus(at)hagander(dot)net> writes:
> > On Sat, Jan 11, 2014 at 5:00 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> >> I'm not sure I believe this. I don't see any PGDLLEXPORT symbols in any
> >> of the standard contrib modules; how is it that they work?
>
> > They are built through our perl toolkit, which enables exporting of *all*
> > symbols, regardless of flags in the code.
>
> That seems like a perfectly reasonable solution, given the way we use
> loadable modules. Excess symbols in the module shouldn't really do
> any harm. Can't we just document the flags to use for this, if you're
> building in some other way?
>

It's not a build flag, and that's the main problem. It's the src/tools/msvc/
gendef.pl script that builds the export list. And what Craig is after here
is being able to build extensions using standard tools without needing our
full build infrastructure.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On 01/11/2014 01:55 PM, Magnus Hagander wrote:
> On Sat, Jan 11, 2014 at 7:05 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us
> <mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us>> wrote:
>
> Magnus Hagander <magnus(at)hagander(dot)net <mailto:magnus(at)hagander(dot)net>>
> writes:
> > On Sat, Jan 11, 2014 at 5:00 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us
> <mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us>> wrote:
> >> I'm not sure I believe this. I don't see any PGDLLEXPORT
> symbols in any
> >> of the standard contrib modules; how is it that they work?
>
> > They are built through our perl toolkit, which enables exporting
> of *all*
> > symbols, regardless of flags in the code.
>
> That seems like a perfectly reasonable solution, given the way we use
> loadable modules. Excess symbols in the module shouldn't really do
> any harm. Can't we just document the flags to use for this, if you're
> building in some other way?
>
>
> It's not a build flag, and that's the main problem. It's the
> src/tools/msvc/gendef.pl <http://gendef.pl> script that builds the
> export list. And what Craig is after here is being able to build
> extensions using standard tools without needing our full build
> infrastructure.
>
>

What I'd like is something that would use or mimic our msvc build tools
but for extensions. (And no, I don't have time to build it.)

cheers

andrew

On 01/12/2014 12:00 AM, Tom Lane wrote:
> Craig Ringer <craig(at)2ndquadrant(dot)com> writes:
>> > We don't set __declspec(dllexport) on extension functions automatically
>> > when building stand-alone on Windows. So it's necessary to explicitly
>> > specify PGDLLEXPORT for each function.
> I'm not sure I believe this. I don't see any PGDLLEXPORT symbols in any
> of the standard contrib modules; how is it that they work?
>

We force all symbols to be exported using the project file generation
tools in the Pg build system, so PGDLLEXPORT is not required there.

That doesn't help stand alone builds. We don't have anything like PGXS
for Windows, so the only real option for building and distributing
reliably compatible extensions is your own Visual Studio project.

"Just build in contrib/ in a source tree" is not an acceptable answer,
because extensions need binary compatibility with public installer
distributions like the one EDB maintains. Unless you're sure your build
is set up just the same way I'm not at all convinced you're going to get
that, though you may get something that works *most* of the time. This
hasn't historically been a project keen on "mostly" solutions though.

> So if it's really necessary to change anything here, I'd rather see us
> take the approach of hiding it in PG_FUNCTION_INFO_V1. What happens
> if we do that and there's also a manually-written prototype?

I've run out of weekend and have to go full-speed on upd. sb. views,
BDR, and event triggers tomorrow, so I don't think I'll have a chance to
test it immediately. I'll check after the CF deadline. That's certainly
one option, anyway, and one that'll solve the immediate issue with
extensions, and would be the most practical short term solution if it works.

Otherwise we can just add DLLEXPORT macros on public functions in
contrib/ and the docs. They're harmless on other platforms - and in fact
can be quite handy.

Those "windows droppings" can actually be useful: They annotate sources
with a useful piece of information we don't currently use on *nix, but
probably should. They specify a shared library or executable's public
interface.

Imagine that the macro was called "PGAPI" or "PG_PUBLIC_API", not
"PGDLLEXPORT". That's what it actually *means*, really.

On *nix we treat the executable and shared lib APIs as the union of the
exported symbols of all compilation units in the object. That's rather
crude, as it means that something very internal to Pg must be public if
it's to be visible across more than one compilation unit, and that if we
wish to hide it we have to squash things together into bigger
compilation units than we might otherwise want.

That's a pain for backward compat, as theoretically all of PostgreSQL's
exported symbols are public API. We don't really offer a good way to
tell what's public and what's not anyway.

Instead, we can and (IMO) should be marking our public API explicitly.
If it's supposed to be usable from a contrib / extension / fdw / etc,
mark it PGDLLEXPORT. Or rename that PG_PUBLIC_API and label it that.
Whatever. On Windows, these expand to __declspec(dllexport) and cause
symbol export. On gcc, we switch to building with -fvisiblity=hidden and
have these macros expand to __attribute__ ((dllexport)) . Access to
non-exported symbols becomes a link error. On ancient GCCs and on other
toolchains we continue to export all symbols by default, as now.

This'd reduce symbol table sizes a bit - though it's really only a big
win for C++ code with monster symbol tables. More importantly, it'd make
it easier to tell what's supposed to be reasonably stable across
versions as public API, and what's totally internal.

(It might be that we have enough exts delving deep in to Pg's guts
already that this is impractical. It'd be fun to see.)

Related references:

http://gcc.gnu.org/wiki/Visibility
http://people.redhat.com/drepper/dsohowto.pdf

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 01/12/2014 04:54 PM, Craig Ringer wrote:
> On 01/12/2014 12:00 AM, Tom Lane wrote:
>> So if it's really necessary to change anything here, I'd rather see us
>> take the approach of hiding it in PG_FUNCTION_INFO_V1. What happens
>> if we do that and there's also a manually-written prototype?
>
> That's certainly
> one option, anyway, and one that'll solve the immediate issue with
> extensions, and would be the most practical short term solution if it works.

... which it kind-of does.

Turned out to be trivial to test. If the prototype with PGDLLEXPORT
appears *first*, then all is well. If the prototype with PGDLLEXPORT
appears AFTER a user-provided prototype it fails with:

1>DemoExtension.c(16): error C2375: 'add_one' : redefinition; different
linkage
1> DemoExtension.c(14) : see declaration of 'add_one'

Two copies of the prototype, both with PGDLLEXPORT, work fine.

So really the question is: Do we care? The current usage of extensions
built outside the Pg tree on Windows is likely to be nearly zero, and is
already fiddly. I'm not too fussed if we make people fix up their
prototypes.

I think we can just emit a prototype for the function from
PG_FUNCTION_INFO_V1 . The only things it's going to break is the odd bit
of weirdly-built, not really supported extensions on Windows.

Are there any platforms that object to prototype redefinition? If not,
we can just emit a prototype on all platforms, not just Windows.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Craig Ringer <craig(at)2ndquadrant(dot)com> writes:
> Turned out to be trivial to test. If the prototype with PGDLLEXPORT
> appears *first*, then all is well. If the prototype with PGDLLEXPORT
> appears AFTER a user-provided prototype it fails with:

That's sort of what I thought would happen. It's problematic because
putting the extern before the PG_FUNCTION_INFO_V1 is standard practice,
especially if you have the extern in a header file.

> I think we can just emit a prototype for the function from
> PG_FUNCTION_INFO_V1.

Doesn't sound like it; we'll still be forced to modify or remove
manually-written externs in basically all contrib and third-party code,
if we want it to build on Windows. Which, as I said before, does not
seem to me to be a workable solution path at all. It would take
years to track down all the third-party code and get it fixed, if
the authors don't themselves build/test on Windows.

And I continue to maintain that it's not acceptable for the Windows port
to require this anyway. If any other platform came to us and said "hey
guys, you need to plaster this non-ANSI-C decoration on every exported
function", what response do you think they'd get?

One last point is that "automatically export every external symbol" is
exactly what happens for these modules on Unix-ish platforms. If it
doesn't work the same way on Windows, we're just opening ourselves up to
subtle portability issues.

This needs to be fixed in the Windows build system, not the source code.

regards, tom lane

On 01/13/2014 02:04 AM, Tom Lane wrote:
> Craig Ringer <craig(at)2ndquadrant(dot)com> writes:
>
>> I think we can just emit a prototype for the function from
>> PG_FUNCTION_INFO_V1.
>
> Doesn't sound like it.

On second thought, agreed. The externs tending to appear in headers
kills that.

In that case - after the rush for the CF close, expect a patch from me
against post-9.4 that adds PGDLLEXPORT to the documentation examples,
and another that adds them for contribs (to help people using them as
examples). Nothing else needed, and it doesn't have to affect the *nix
build process or server/ code in any way.

I'll also add a note in the docs explaining what's wrong if you get an
error about an obviously-present function being missing in your
extension on Windows when it works on other platforms.

> And I continue to maintain that it's not acceptable for the Windows port
> to require this anyway. If any other platform came to us and said "hey
> guys, you need to plaster this non-ANSI-C decoration on every exported
> function", what response do you think they'd get?
>
> One last point is that "automatically export every external symbol" is
> exactly what happens for these modules on Unix-ish platforms. If it
> doesn't work the same way on Windows, we're just opening ourselves up to
> subtle portability issues.

The portability issues are easily dealt with by using gcc's symbol
visibility features on *nix, which will produce slightly smaller
binaries with faster linkage too. I'll demo that once I've got the
current work off my plate. Platforms w/o gcc visibility don't need to
care, nothing changes, they just don't get the benefits. Everyone
develops on platforms that have gcc and visibility features so they'll
spot any issues introduced.

As described earlier, doing this helps differentiate "internal stuff"
from "public API" if we choose to, as well.

If we don't want to attempt to treat anything as "internal,
non-public-use, non-stable API", then there's no point using visibility
- everything not "static" is public API and should be exported to the
world. That's how things are now. In that case I'd stick with just
improving the docs to cover PGDLLEXPORT, and maybe the contribs.

I do think we should think about actually hiding private internal API
post-9.4, though.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services