Re: BUG #3818: Cross compilation problems

The following bug has been logged online:

Bug reference: 3818
Logged by: Richard Evans
Email address: richard(dot)evans(at)blueallegro(dot)net
PostgreSQL version: 8.3beta
Operating system: Linux/windows
Description: Cross compilation problems
Details:

I've been investigating cross-compiling for mingw32 on a linux build system.
I used the current snapshot since it has a fix for the gettimeofday problem
with new mingw installs.

Mostly it worked, but there were a few problems which I think can all be
fixed in the configure script:

1. The configure script does not check for cross-compile installs of ar,
dlltool, dllwrap and windres. It manages to find the cross compile gcc
(mingw32-gcc in my install), but does not check for mingw32-dlltool, etc.
You work round this by setting AR= DLLTOOL= and DLLWRAP= on the make line
but ideally it should be done in the configure stage.

2. The Makefile in pgevent/bin refers to dllwrap directly, not $(DLLWRAP).

3. Several makefiles use windres directly; there is no $(WINDRES) variable
which can be set.

4. The zic problem in make install - see bug #1311. Simple solution here is
to have a configure option which allows the use of the build systems own zic
command.

I may try coming up with some patches to configure to check for dlltool, etc
but I haven't edited configure files before.

Once I worked round these problems I was able to cross compile a working
postgres for mingw32 on a linux host.

Richard Evans wrote:
> The following bug has been logged online:
>
> Bug reference: 3818
> Logged by: Richard Evans
> Email address: richard(dot)evans(at)blueallegro(dot)net
> PostgreSQL version: 8.3beta
> Operating system: Linux/windows
> Description: Cross compilation problems
> Details:
>
> I've been investigating cross-compiling for mingw32 on a linux build system.
> I used the current snapshot since it has a fix for the gettimeofday problem
> with new mingw installs.

Out of curiosity, why?

> Mostly it worked, but there were a few problems which I think can all be
> fixed in the configure script:
>
> 1. The configure script does not check for cross-compile installs of ar,
> dlltool, dllwrap and windres. It manages to find the cross compile gcc
> (mingw32-gcc in my install), but does not check for mingw32-dlltool, etc.
> You work round this by setting AR= DLLTOOL= and DLLWRAP= on the make line
> but ideally it should be done in the configure stage.
>
> 2. The Makefile in pgevent/bin refers to dllwrap directly, not $(DLLWRAP).

This is a definite oversight, so I've fixed this in HEAD.

> 3. Several makefiles use windres directly; there is no $(WINDRES) variable
> which can be set.
>
> 4. The zic problem in make install - see bug #1311. Simple solution here is
> to have a configure option which allows the use of the build systems own zic
> command.
>
> I may try coming up with some patches to configure to check for dlltool, etc
> but I haven't edited configure files before.

The rest of this is definitely not 8.3 material. But if you come up with
patches for 8.4, I'm sure we'd be interested ;-) If you have any
specific questions about the autoconf stuff (or any other parts of it),
just post it to -hackers.

> Once I worked round these problems I was able to cross compile a working
> postgres for mingw32 on a linux host.

Cool!

//Magnus

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> Richard Evans wrote:
>> 4. The zic problem in make install - see bug #1311. Simple solution here is
>> to have a configure option which allows the use of the build systems own zic
>> command.

> The rest of this is definitely not 8.3 material. But if you come up with
> patches for 8.4, I'm sure we'd be interested ;-)

--with-system-tzdata already solves the zic problem.

regards, tom lane

If the target system does not have system tzdata, (mingw for example),
you still need to run zic during the cross compile process. I just
thought that using the build systems own zic was the simplest way to do
this.

In my tests I edited the Makefile in src/timezone and changed ./zic to
just zic and it seemed to work fine.

Richard

> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>
>> Richard Evans wrote:
>>
>>> 4. The zic problem in make install - see bug #1311. Simple solution here is
>>> to have a configure option which allows the use of the build systems own zic
>>> command.
>>>
>
>
>> The rest of this is definitely not 8.3 material. But if you come up with
>> patches for 8.4, I'm sure we'd be interested ;-)
>>
>
> --with-system-tzdata already solves the zic problem.
>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster
>
>

Richard Evans wrote:
>> The following bug has been logged online:
>>
>> Bug reference: 3818
>> Logged by: Richard Evans
>> Email address: richard(dot)evans(at)blueallegro(dot)net
>> PostgreSQL version: 8.3beta
>> Operating system: Linux/windows
>> Description: Cross compilation problems
>> Details:
>>
>> I've been investigating cross-compiling for mingw32 on a linux build system.
>> I used the current snapshot since it has a fix for the gettimeofday problem
>> with new mingw installs.
>>
>
> Out of curiosity, why?
>
I want to set up a build system on a single linux host to build for a
number of different targets - linux and win32 at least.
>
>
>> Mostly it worked, but there were a few problems which I think can all be
>> fixed in the configure script:
>>
>> 1. The configure script does not check for cross-compile installs of ar,
>> dlltool, dllwrap and windres. It manages to find the cross compile gcc
>> (mingw32-gcc in my install), but does not check for mingw32-dlltool, etc.
>> You work round this by setting AR= DLLTOOL= and DLLWRAP= on the make line
>> but ideally it should be done in the configure stage.
>>
>> 2. The Makefile in pgevent/bin refers to dllwrap directly, not $(DLLWRAP).
>>
>
> This is a definite oversight, so I've fixed this in HEAD.
>
>
>
>> 3. Several makefiles use windres directly; there is no $(WINDRES) variable
>> which can be set.
>>
>> 4. The zic problem in make install - see bug #1311. Simple solution here is
>> to have a configure option which allows the use of the build systems own zic
>> command.
>>
>> I may try coming up with some patches to configure to check for dlltool, etc
>> but I haven't edited configure files before.
>>
>
> The rest of this is definitely not 8.3 material. But if you come up with
> patches for 8.4, I'm sure we'd be interested ;-) If you have any
> specific questions about the autoconf stuff (or any other parts of it),
> just post it to -hackers.
>
>
>
>> Once I worked round these problems I was able to cross compile a working
>> postgres for mingw32 on a linux host.
>>
>
> Cool!
>
> //Magnus
>
>

Richard Evans <richard(dot)evans(at)blueallegro(dot)net> writes:
>> --with-system-tzdata already solves the zic problem.

> If the target system does not have system tzdata, (mingw for example),
> you still need to run zic during the cross compile process.

No, you just need to physically copy a tzdata tree onto the target at
some point. If you're assuming that the host system has a compatible
zic, then it surely has got some installed tzdata files too.

regards, tom lane

Richard, would you send us patches for the cross compile variables you
needed changed? Thanks.

---------------------------------------------------------------------------

Richard Evans wrote:
>
> The following bug has been logged online:
>
> Bug reference: 3818
> Logged by: Richard Evans
> Email address: richard(dot)evans(at)blueallegro(dot)net
> PostgreSQL version: 8.3beta
> Operating system: Linux/windows
> Description: Cross compilation problems
> Details:
>
> I've been investigating cross-compiling for mingw32 on a linux build system.
> I used the current snapshot since it has a fix for the gettimeofday problem
> with new mingw installs.
>
> Mostly it worked, but there were a few problems which I think can all be
> fixed in the configure script:
>
> 1. The configure script does not check for cross-compile installs of ar,
> dlltool, dllwrap and windres. It manages to find the cross compile gcc
> (mingw32-gcc in my install), but does not check for mingw32-dlltool, etc.
> You work round this by setting AR= DLLTOOL= and DLLWRAP= on the make line
> but ideally it should be done in the configure stage.
>
> 2. The Makefile in pgevent/bin refers to dllwrap directly, not $(DLLWRAP).
>
> 3. Several makefiles use windres directly; there is no $(WINDRES) variable
> which can be set.
>
> 4. The zic problem in make install - see bug #1311. Simple solution here is
> to have a configure option which allows the use of the build systems own zic
> command.
>
> I may try coming up with some patches to configure to check for dlltool, etc
> but I haven't edited configure files before.
>
> Once I worked round these problems I was able to cross compile a working
> postgres for mingw32 on a linux host.
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo(at)postgresql(dot)org so that your
> message can get through to the mailing list cleanly

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Sorry about the delay on this. Just managed to find some time to look
into it again.

I've made a couple of patches, for 8.3.3 and 8.2.9, which enable cross
compilation for windows (mingw32) from a unix platform.

The changes are in these areas:

1. configure.in detecting the target system versions of dllwrap,
dlltool, windres and ar
2. in various makefiles, checking BUILDOS not PORTNAME when deciding
whether to use 'pwd -W' or just 'pwd'
3. Using the local system's 'zic' command when cross compiling, adding a
--with-zic option to configure.in in case zic is not in the path (on my
fedora 9 system it is in /usr/sbin/zic).
4. Some path tweaks in the make files to allow the cross compile build
to occur outside the the source directory.
5. Working round a binutils-2.18 bug in windres when the rc file has /
or \ in the path

If I apply the patches and run autoconf then I can do a full cross
compile and install (to a distribution dir) for 8.3.3 and 8.2.9 using a
configure like:

PATH=/usr/local/xc/bin:$PATH ../postgresql-8.3.3/configure
--host=mingw32 --without-zlib \
--prefix=somedir --libdir=somedir/bin --with-zic=/usr/sbin/zic

I have my cross compilation tool set in /usr/local/xc; I set libdir so
that the dlls end up in the bin dir which makes running on windows
easier ...

I've also tested native builds on a mingw32 system using the patched files.

Hope this helps.

Richard
> Richard, would you send us patches for the cross compile variables you
> needed changed? Thanks.
>
> ---------------------------------------------------------------------------
>
> Richard Evans wrote:
>
>> The following bug has been logged online:
>>
>> Bug reference: 3818
>> Logged by: Richard Evans
>> Email address: richard(dot)evans(at)blueallegro(dot)net
>> PostgreSQL version: 8.3beta
>> Operating system: Linux/windows
>> Description: Cross compilation problems
>> Details:
>>
>> I've been investigating cross-compiling for mingw32 on a linux build system.
>> I used the current snapshot since it has a fix for the gettimeofday problem
>> with new mingw installs.
>>
>> Mostly it worked, but there were a few problems which I think can all be
>> fixed in the configure script:
>>
>> 1. The configure script does not check for cross-compile installs of ar,
>> dlltool, dllwrap and windres. It manages to find the cross compile gcc
>> (mingw32-gcc in my install), but does not check for mingw32-dlltool, etc.
>> You work round this by setting AR= DLLTOOL= and DLLWRAP= on the make line
>> but ideally it should be done in the configure stage.
>>
>> 2. The Makefile in pgevent/bin refers to dllwrap directly, not $(DLLWRAP).
>>
>> 3. Several makefiles use windres directly; there is no $(WINDRES) variable
>> which can be set.
>>
>> 4. The zic problem in make install - see bug #1311. Simple solution here is
>> to have a configure option which allows the use of the build systems own zic
>> command.
>>
>> I may try coming up with some patches to configure to check for dlltool, etc
>> but I haven't edited configure files before.
>>
>> Once I worked round these problems I was able to cross compile a working
>> postgres for mingw32 on a linux host.
>>
>> ---------------------------(end of broadcast)---------------------------
>> TIP 1: if posting/reading through Usenet, please send an appropriate
>> subscribe-nomail command to majordomo(at)postgresql(dot)org so that your
>> message can get through to the mailing list cleanly
>>
>
>

Richard Evans wrote:

> 1. configure.in detecting the target system versions of dllwrap,
> dlltool, windres and ar
> 2. in various makefiles, checking BUILDOS not PORTNAME when deciding
> whether to use 'pwd -W' or just 'pwd'
> 3. Using the local system's 'zic' command when cross compiling, adding a
> --with-zic option to configure.in in case zic is not in the path (on my
> fedora 9 system it is in /usr/sbin/zic).
> 4. Some path tweaks in the make files to allow the cross compile build
> to occur outside the the source directory.
> 5. Working round a binutils-2.18 bug in windres when the rc file has /
> or \ in the path

I think part (4) makes plenty of sense just to enable VPATH building. I
guess nobody has tried VPATH on cygwin.

I can't comment on the rest of the changes, but most of them seem to be
reasonable. I think the "pwd" stuff should be done in a makefile
template.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Am Saturday, 9. August 2008 schrieb Richard Evans:
> I've made a couple of patches, for 8.3.3 and 8.2.9, which enable cross
> compilation for windows (mingw32) from a unix platform.

This looks good, with a couple of tweaks. I don't think we are going to be
making these kinds of changes in the 8.2 and 8.3 branches, so I suggest you
prepare a patch for 8.4. Some of the makefiles have changed, so a bit of
work might be involved.

> 2. in various makefiles, checking BUILDOS not PORTNAME when deciding
> whether to use 'pwd -W' or just 'pwd'

I was wondering, what is the difference between these? And couldn't we just
use $(CURDIR) instead?

> 5. Working round a binutils-2.18 bug in windres when the rc file has /
> or \ in the path

What is the status of this bug? Shouldn't it be fixed instead?

> Am Saturday, 9. August 2008 schrieb Richard Evans:
>> I've made a couple of patches, for 8.3.3 and 8.2.9, which enable cross
>> compilation for windows (mingw32) from a unix platform.
>
> This looks good, with a couple of tweaks. I don't think we are going
to be
> making these kinds of changes in the 8.2 and 8.3 branches, so I
suggest you
> prepare a patch for 8.4. Some of the makefiles have changed, so a
bit of
> work might be involved.

This is from the source repository? I'll take a look.

>
>> 2. in various makefiles, checking BUILDOS not PORTNAME when deciding
>> whether to use 'pwd -W' or just 'pwd'
>
> I was wondering, what is the difference between these? And couldn't
we just
> use $(CURDIR) instead?

I'm not sure why the makefiles need the current directory. pwd -W is
specific to mingw, I think it gives the directory in windows format. It
has to be changed for cross compilation otherwise you get errors.

>
>> 5. Working round a binutils-2.18 bug in windres when the rc file has /
>> or \ in the path
>
> What is the status of this bug? Shouldn't it be fixed instead?
>

The bug is fixed in the binutils source repository, so presumably will
be fixed in 2.19. But it seems better to support the current version
rather than wait for the next.

Richard

Richard Evans wrote:
> I'm not sure why the makefiles need the current directory. pwd -W is
> specific to mingw, I think it gives the directory in windows format.  It
> has to be changed for cross compilation otherwise you get errors.

What does $(CURDIR) resolve to on mingw? Try the following makefile:

default:
@echo $(shell pwd)
@echo $(shell pwd -W)
@echo $(CURDIR)

Running make inside the msys shell, I get:

/c/misc/mingw/bin
c:/misc/mingw/bin
/c/misc/mingw/bin

running inside a plain DOS command window, I get:

/mingw/bin
c:/misc/mingw/bin
/mingw/bin

Msys/mingw have these funny 'system' mounts. However it looks like that
/pwd -W/ gives the most sensible output in both cases. Must be why it's
there.

Richard

> Richard Evans wrote:
>
>> I'm not sure why the makefiles need the current directory. pwd -W is
>> specific to mingw, I think it gives the directory in windows format. It
>> has to be changed for cross compilation otherwise you get errors.
>>
>
> What does $(CURDIR) resolve to on mingw? Try the following makefile:
>
> default:
> @echo $(shell pwd)
> @echo $(shell pwd -W)
> @echo $(CURDIR)
>
>

Where are we on this?

---------------------------------------------------------------------------

Richard Evans wrote:
> Running make inside the msys shell, I get:
>
> /c/misc/mingw/bin
> c:/misc/mingw/bin
> /c/misc/mingw/bin
>
> running inside a plain DOS command window, I get:
>
> /mingw/bin
> c:/misc/mingw/bin
> /mingw/bin
>
> Msys/mingw have these funny 'system' mounts. However it looks like that
> /pwd -W/ gives the most sensible output in both cases. Must be why it's
> there.
>
> Richard
>
> > Richard Evans wrote:
> >
> >> I'm not sure why the makefiles need the current directory. pwd -W is
> >> specific to mingw, I think it gives the directory in windows format. It
> >> has to be changed for cross compilation otherwise you get errors.
> >>
> >
> > What does $(CURDIR) resolve to on mingw? Try the following makefile:
> >
> > default:
> > @echo $(shell pwd)
> > @echo $(shell pwd -W)
> > @echo $(CURDIR)
> >
> >

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Sunday 07 December 2008 01:02:05 Bruce Momjian wrote:
> Where are we on this?

Some of this has been fixed. But a lot of the code has been moved around
between 8.3 and 8.4, so we can't just take take the patches as is. If
Richard is still interested, I suggest he try out 8.4 and then submit any
remaining desired changes.

I've taken a look at the current development snapshot ane made a new
patch. This is against the snapshot source dated 2008-12-30.

I have omitted the workround for the bug in binutils 2.18 since 2.19 is
now available and fixes the problem.

I tested the cross compilation for mingw32 using gcc 4.3.2 and binutils
2.19, with the latest mingw runtime and w32api from mingw.org.

I verified that the build works on a Windows 2000 system.

If the patch goes in, I can provide some notes on the cross compilation
process.

Richard
> On Sunday 07 December 2008 01:02:05 Bruce Momjian wrote:
>
>> Where are we on this?
>>
>
> Some of this has been fixed. But a lot of the code has been moved around
> between 8.3 and 8.4, so we can't just take take the patches as is. If
> Richard is still interested, I suggest he try out 8.4 and then submit any
> remaining desired changes.
>
>

On Tuesday 30 December 2008 17:49:16 Richard Evans wrote:
> I've taken a look at the current development snapshot ane made a new
> patch. This is against the snapshot source dated 2008-12-30.

Half of this patch appears to attempt to fix not cross-compilation problems,
but out-of-tree builds (vpath builds). Considering that other developers
regularly exercise this, it looks a bit suspicious. Can you clarify your
intent?

Peter Eisentraut wrote:
> On Tuesday 30 December 2008 17:49:16 Richard Evans wrote:
> > I've taken a look at the current development snapshot ane made a new
> > patch. This is against the snapshot source dated 2008-12-30.
>
> Half of this patch appears to attempt to fix not cross-compilation problems,
> but out-of-tree builds (vpath builds). Considering that other developers
> regularly exercise this, it looks a bit suspicious. Can you clarify your
> intent?

Hmm, but does anyone test the VPATH build on mingw? It looks like the
rules being modified are part of that port exclusively.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Yes, that's true. There are also some changes to support general
cross-compilation.

Some more details:

configure.in: if you not using system timezone data,
src/timezone/Makefile builds a zic executable and then runs it to
generate tzdata files. If you are cross compiling this won't work so I
added an option to identify the build system's zic command and use that
instead. You need this for a mingw32 cross compile since there is no
system tzdata on Windows. tzdata files are architecture independent.

The alternative would be to build a host system zic alongside or instead
of the target system zic but that seemed more complex since the
configure script would need to set up for local and target compilation.

src/bin/pgevent/Makefile: fix source and include path for vpath mingw32
compilation.

src/interfaces/ecpg/test/Makefile: use pwd -W if the build system is
mingw32, as opposed to the target system being win32.

src/interfaces/libpq/Makefile: fix include path for vpath mingw32
compilation (to find pthread-win32.h).

src/Makefile.global.in: set ZIC and BUILD_OS variables (the latter to
test for cross-compilation).

src/makefiles/Makefile.win32: fix windres include path for vpath mingw32
compilation.

src/Makefile.shlib: fix value of DLL_DEFFILE so that .def file is
generated in the build tree, not the source tree.

src/timezone/Makefile: use system zic command when cross-compiling

Richard
> Peter Eisentraut wrote:
>
>> On Tuesday 30 December 2008 17:49:16 Richard Evans wrote:
>>
>>> I've taken a look at the current development snapshot ane made a new
>>> patch. This is against the snapshot source dated 2008-12-30.
>>>
>> Half of this patch appears to attempt to fix not cross-compilation problems,
>> but out-of-tree builds (vpath builds). Considering that other developers
>> regularly exercise this, it looks a bit suspicious. Can you clarify your
>> intent?
>>
>
> Hmm, but does anyone test the VPATH build on mingw? It looks like the
> rules being modified are part of that port exclusively.
>
>

Richard Evans wrote:
> I've taken a look at the current development snapshot ane made a new
> patch. This is against the snapshot source dated 2008-12-30.

This is now committed.