Re: [bug fix] postgres.exe fails to start on Windows Server 2012 due to ASLR

Hello,

My customer reported the following problem on Windows. I'm afraid this is a
serious problem, and I think we should provide a fix in the next minor
release. I've attached a fix, and I would wish it to be back-ported.

[Problem]
The customer is using 64-bit PostgreSQL 9.1.x on Windows Server 2012 R2. He
found the following messages output repeatedly (once per about 3 minutes) in
the event log.

LOG: could not reserve shared memory region (addr=0000000001270000) for
child 0000000000001340: 487
LOG: could not fork autovacuum worker process: A blocking operation was
interrupted by a call to WSACancelBlockingCall.

I found the same problem was reported by another user here:

Re: PG 924, Windows 2012, error code 487
http://www.postgresql.org/message-id/2C0926ABD16BB641A8E2F11A549200425471425A21@PHXCCRPRD01.adprod.bmc.com

In my customer's case, the failed process was autovacuum worker, so there
was no visible impact on their applications. However, as the above mail
suggests, this can happen with any backend process.

[Cause]
This is caused by the improved ASLR (Address Space Layout Randomization) in
Windows 8/2012. That is analyzed in the following mail last year, but the
problem was not addressed:

windows 8 RTM compatibility issue (could not reserve shared memory region
for child)
http://www.postgresql.org/message-id/5046CAEB.4010600@grammatech.com

[Fix]
Disable ASLR when building PostgreSQL modules. This eliminated the log
output in my customer's environment.

I'll add this to the CommitFest if necessary.

Regards
MauMau

On 01/24/2014 06:42 PM, MauMau wrote:
> Hello,
>
> My customer reported the following problem on Windows. I'm afraid this
> is a serious problem, and I think we should provide a fix in the next
> minor release. I've attached a fix, and I would wish it to be back-ported.
>
>
> [Problem]
> The customer is using 64-bit PostgreSQL 9.1.x

Which "x"?

Does this issue also occur on 9.3.2, or in 9.4 HEAD, when tested on Win2k12?

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

I'm sorry for the late reply. I was unable to access email.

From: "Craig Ringer" <craig(at)2ndquadrant(dot)com>
> On 01/24/2014 06:42 PM, MauMau wrote:
>> The customer is using 64-bit PostgreSQL 9.1.x
>
> Which "x"?

9.1.6.

> Does this issue also occur on 9.3.2, or in 9.4 HEAD, when tested on
> Win2k12?

I'm sure it should. The release note doesn't have any reference to this
issue. Another user who reported this issue in pgsql-general experienced
this with 9.2.4.

In addition, Another customer reported to me that psql failed to connect to
PostgreSQL 9.1.9 running on Windows Server 2012 R2, about once out of 7
times. I believe this needs immediate fix.

Regards
MauMau

On 01/30/2014 08:14 PM, MauMau wrote:
>> Does this issue also occur on 9.3.2, or in 9.4 HEAD, when tested on
>> Win2k12?
>
> I'm sure it should. The release note doesn't have any reference to this
> issue. Another user who reported this issue in pgsql-general
> experienced this with 9.2.4.

> In addition, Another customer reported to me that psql failed to connect
> to PostgreSQL 9.1.9 running on Windows Server 2012 R2, about once out of
> 7 times. I believe this needs immediate fix.

I'm reasonably persuaded that there's a need for this, though IFEO (see
below) can be used at or after install-time as a workaround.

It looks like your patch sets the msbuild equivalent of the
/DYNAMICBASE:NO flag
(http://msdn.microsoft.com/en-us/library/bb384887.aspx). Is this known
to work (or be silently ignored) back to Windows SDK 7.1? (I'll test it
today and see).

I don't think we need to worry about Force ASLR
(http://support.microsoft.com/kb/2639308) as it seems it only applies
when an application loads modules, unless an admin goes playing in the
registry.

Users facing this problem can work around it without code changes by
setting IFEO in the registry to disable ASLR for postgres.exe. See:

http://msdn.microsoft.com/en-us/library/windows/desktop/ff190925(v=vs.85).aspx

http://support.microsoft.com/kb/2639308

http://msdn.microsoft.com/en-us/library/bb430720.aspx

It may be reasonable for EDB to consider releasing updated installers
that set IFEO flags to disable ASLR on postgres.exe .

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

From: "Craig Ringer" <craig(at)2ndquadrant(dot)com>
> I'm reasonably persuaded that there's a need for this, though IFEO (see
> below) can be used at or after install-time as a workaround.

I see. And I also found it effective as another workaround to set the below
registry key. This disables ASLR for all executables, so it would be
disliked.

HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory
Management\MoveImages

> It looks like your patch sets the msbuild equivalent of the
> /DYNAMICBASE:NO flag
> (http://msdn.microsoft.com/en-us/library/bb384887.aspx). Is this known
> to work (or be silently ignored) back to Windows SDK 7.1? (I'll test it
> today and see).

OK, I tried these versions at hand:

* Windows SDK 7.1, which happens to be associated with Visual Studio 2010 on
my PC
link.exe accepts /dynamicbase:no, which takes effect. Without
/dynamicbase:no, dumpbin /headers displays the lines:

8140 DLL characteristics
Dynamic base
NX compatible
Terminal Server Aware

On the other hand, with /dynamicbase:no, the second line above disappeared.

* Visual Studio 2008 Express
link.exe seems to silently ignore /dynamicbase:no. dumpbin /headers does
not display "Dynamic base" regardless of whether /dynamicbase:no is
specified.

> I don't think we need to worry about Force ASLR
> (http://support.microsoft.com/kb/2639308) as it seems it only applies
> when an application loads modules, unless an admin goes playing in the
> registry.
>
>
> Users facing this problem can work around it without code changes by
> setting IFEO in the registry to disable ASLR for postgres.exe. See:

The problem is that users cannot discover the workaround. As more users use
PostgreSQL on Windows 8/2012, similar questions would be asked in
pgsql-general and pgsql-bugs.

> It may be reasonable for EDB to consider releasing updated installers
> that set IFEO flags to disable ASLR on postgres.exe .

Not all users use PostgreSQL built by EnterpriseDB, so I think
src\tools\msvc\build.bat should produce modules that are not affected by
ASLR.

Regards
MauMau

On 02/01/2014 09:53 PM, MauMau wrote:
>
> Not all users use PostgreSQL built by EnterpriseDB, so I think
> src\tools\msvc\build.bat should produce modules that are not affected by
> ASLR.

I completely agree; just saying that any installer can set the key.

I'm convinced that setting this flag is appropriate, at least while Pg
relies on having the shared memory segment mapped in the same zone in
every executable. Just pointing out that there's a workaround in the
mean time.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

From: "Craig Ringer" <craig(at)2ndquadrant(dot)com>
> I completely agree; just saying that any installer can set the key.
>
> I'm convinced that setting this flag is appropriate, at least while Pg
> relies on having the shared memory segment mapped in the same zone in
> every executable. Just pointing out that there's a workaround in the
> mean time.

Please don't mind, I didn't misunderstand your intent. I think we should
apply this in the next minor release to avoid unnecessary confusion -- more
new users would use PostgreSQL on Windows 8/2012 and hit this problem.

I added this patch to the CommitFest.

Regards
MauMau

On 02/04/2014 07:28 PM, MauMau wrote:
>>
>
> Please don't mind, I didn't misunderstand your intent. I think we
> should apply this in the next minor release to avoid unnecessary
> confusion -- more new users would use PostgreSQL on Windows 8/2012 and
> hit this problem.
>
> I added this patch to the CommitFest.

It's really a bugfix suitable for backpatching IMO.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Tue, Feb 4, 2014 at 12:57 PM, Craig Ringer <craig(at)2ndquadrant(dot)com> wrote:

> On 02/04/2014 07:28 PM, MauMau wrote:
> >>
> >
> > Please don't mind, I didn't misunderstand your intent. I think we
> > should apply this in the next minor release to avoid unnecessary
> > confusion -- more new users would use PostgreSQL on Windows 8/2012 and
> > hit this problem.
> >
> > I added this patch to the CommitFest.
>
> It's really a bugfix suitable for backpatching IMO.
>
>
Unfortunately we missed the releases that have just been wrapped.

I've applied this and backpatched to 9.3 and 9.2, which is as far back as
it goes cleanly.

In 9.1 the build system looked significantly different, which makes it
strange since the original report in this thread was about 9.1 but the
patch supplied clearly wasn't for that.

Does somebody want to look at backpatching this to 9.1 and earlier, or
should we just say that it's not fully supported on those Windows versions
unless you apply the registry workaround?

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> I've applied this and backpatched to 9.3 and 9.2, which is as far back as
> it goes cleanly.

> In 9.1 the build system looked significantly different, which makes it
> strange since the original report in this thread was about 9.1 but the
> patch supplied clearly wasn't for that.

> Does somebody want to look at backpatching this to 9.1 and earlier, or
> should we just say that it's not fully supported on those Windows versions
> unless you apply the registry workaround?

The question I think needs to be asked is not that, but "what about the
mingw and cygwin builds?".

regards, tom lane

On Tue, Feb 18, 2014 at 3:28 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> Magnus Hagander <magnus(at)hagander(dot)net> writes:
> > I've applied this and backpatched to 9.3 and 9.2, which is as far back as
> > it goes cleanly.
>
> > In 9.1 the build system looked significantly different, which makes it
> > strange since the original report in this thread was about 9.1 but the
> > patch supplied clearly wasn't for that.
>
> > Does somebody want to look at backpatching this to 9.1 and earlier, or
> > should we just say that it's not fully supported on those Windows
> versions
> > unless you apply the registry workaround?
>
> The question I think needs to be asked is not that, but "what about the
> mingw and cygwin builds?".
>

Cygwin has their own implementation of fork(). So the fix there has to be
done in cygwin and not us - presumably they have either already fixed it,
or will fix it in order to get general compatibility with the new ASLR
versions.

Mingw is a different story. But some googling indicates that mingw disables
ASLR by default. You need to specify --dynamicbase if you want to to be
dynamic, which we don't. (Being mingw I've not managed to find any actual
docs, but for example
http://lists.cypherpunks.ca/pipermail/otr-dev/2012-August/001373.html seems
to indicate it has to be enabled).

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

From: "Magnus Hagander" <magnus(at)hagander(dot)net>
> Unfortunately we missed the releases that have just been wrapped.

It's really unfortunate... I hope the next minor release will be soon.

> I've applied this and backpatched to 9.3 and 9.2, which is as far back as
> it goes cleanly.

Thank you.

> In 9.1 the build system looked significantly different, which makes it
> strange since the original report in this thread was about 9.1 but the
> patch supplied clearly wasn't for that.
>
> Does somebody want to look at backpatching this to 9.1 and earlier, or
> should we just say that it's not fully supported on those Windows versions
> unless you apply the registry workaround?

I'll try it at least for 9.1 in a few days when time permits. The fix
should be adding only one line in Project.pm.

Regards
MauMau

From: "Magnus Hagander" <magnus(at)hagander(dot)net>
> Does somebody want to look at backpatching this to 9.1 and earlier, or
> should we just say that it's not fully supported on those Windows versions
> unless you apply the registry workaround?

Please use the attached patch. It applies cleanly to both 9.1 and 9.0.

We don't need to consider 8.4, because ASLR became enabled by default in
Visual Studio 2008 and 8.4 doesn't support building with 2008.

I tested with Visual Studio 2008 Express. You can check if ASLR is disabled
using dumpbin. "dumpbin /headers <EXE_or_DLL_file>" outputs the following
lines. If the "Dynamic base" line is displayed, ASLR is enabled. If this
line disappears, ASLR is disabled.

8140 DLL characteristics
Dynamic base
NX compatible
Terminal Server Aware

Regards
MauMau

On Sat, Feb 22, 2014 at 5:44 AM, MauMau <maumau307(at)gmail(dot)com> wrote:

> From: "Magnus Hagander" <magnus(at)hagander(dot)net>
>
> Does somebody want to look at backpatching this to 9.1 and earlier, or
>> should we just say that it's not fully supported on those Windows versions
>> unless you apply the registry workaround?
>>
>
> Please use the attached patch. It applies cleanly to both 9.1 and 9.0.
>

Applied, apologies for the delay.

We don't need to consider 8.4, because ASLR became enabled by default in
> Visual Studio 2008 and 8.4 doesn't support building with 2008.
>

Ok, thanks for confirming!

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/