Re: [9.4 bug] The database server hangs with write-heavy workload on Windows

Hello,

One user reported a hang problem with 9.4 beta2 on Windows. The PostgreSQL
is 64-bit version. I couldn't find the cause, but want to solve the
problem. Could you help with this?

I heard that the user had run 16 concurrent psql sessions which executes
INSERT and UPDATE statements, which is a write-intensive stress test. He
encountered the hang phenomenon twice, one of which occured several hours
after the start of the test, and the other occured about an hour after the
test launch.

The user gave me the stack traces, which I attached at the end of this mail.
The problem appears to be related to the xlog insert scaling. But I can't
figure out where the root cause lies --- WAL slot handling, spinlock on
Windows, or PGSemaphoreLock/UnLock on Windows?

The place I suspect is S_UNLOCK(). It doesn't use any memory barrier. Is
this correct on Intel64 processors?

#define S_UNLOCK(lock) (*((volatile slock_t *) (lock)) = 0)

The rest of this mail is the stack trace:

00000000`0043e0a8 00007ff8`213d12ee : 00000000`00000002 00000000`00000002
00000000`00000001 00000000`00000000 : ntdll!ZwWaitForMultipleObjects+0xa
00000000`0043e0b0 00000001`401de68e : 00000000`00000000 00007ff5`ffffe000
00000000`00000000 00000000`04fb6b40 :
KERNELBASE!WaitForMultipleObjectsEx+0xe1
00000000`0043e390 00000001`4023cf11 : 00000000`02a55500 00000000`1c117410
80605042`36ad2501 00000001`405546e0 : postgres!PGSemaphoreLock+0x6e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\port\win32_sema.c @
145]
00000000`0043e3e0 00000001`4006203b : 00000000`f9017d56 00000000`00000022
00000000`00000000 00000000`00000400 : postgres!LWLockAcquireCommon+0x121
[d:\pginstaller.auto\postgres.windows-x64\src\backend\storage\lmgr\lwlock.c
@ 625]
00000000`0043e430 00000001`4002c182 : 00000000`00000005 00000000`00000000
00000000`004e2f00 00000000`00000000 : postgres!XLogInsert+0x62b
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1110]
00000000`0043e700 00000001`400323b6 : 00000000`00000000 00000000`00000000
00000000`00000a63 00000000`0289de10 : postgres!log_heap_clean+0x102
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
6561]
00000000`0043e7e0 00000001`400320e8 : 00000000`040ec5c0 00000000`00000a63
00000000`0043f340 00000000`040ec5c0 : postgres!heap_page_prune+0x2a6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\pruneheap.c
@ 261]
00000000`0043f2f0 00000001`4002dc40 : 00000000`0057ea30 00000000`00000000
00000000`00000000 00000000`028d1810 : postgres!heap_page_prune_opt+0x148
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\pruneheap.c
@ 150]
00000000`0043f340 00000001`4002e7da : 00000000`028d1800 00000000`00000d26
00000000`00000005 00000000`0057ea30 : postgres!heapgetpage+0xa0
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
355]
00000000`0043f3e0 00000001`4002802c : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!heapgettup_pagemode+0x40a
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
944]
00000000`0043f460 00000001`40126507 : 00000000`00000000 00000000`0000001d
00000000`00000000 00000000`0000001d : postgres!heap_getnext+0x1c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
1478]
00000000`0043f490 00000001`401137f5 : 00000000`028d05b0 00000000`028d06c0
00000000`00000000 00000000`028a3d30 : postgres!SeqNext+0x27
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodeseqscan.c
@ 76]
00000000`0043f4c0 00000001`4010c7b2 : 00000000`0058dba0 00000000`028d05b0
00000000`00000000 00000000`00000000 : postgres!ExecScan+0xd5
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execscan.c @
167]
00000000`0043f520 00000001`4012448d : 00000000`028d02e0 00000000`028d02d8
00000000`028d02e0 00000000`00585a00 : postgres!ExecProcNode+0xd2
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execprocnode.c
@ 400]
00000000`0043f550 00000001`4010c772 : 00000000`00587bc0 00000000`028d0110
00000000`00000000 00000000`028d0258 : postgres!ExecModifyTable+0x10d
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodemodifytable.c
@ 926]
00000000`0043f610 00000001`4010bb6d : 00000000`028d0110 00000000`00587bc0
00000000`00000000 00000000`0056c740 : postgres!ExecProcNode+0x92
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execprocnode.c
@ 377]
00000000`0043f640 00000001`401099d8 : 00000000`00570ff0 00000000`0051e400
00000000`028d0110 00000000`005831f0 : postgres!ExecutePlan+0x5d
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
1481]
00000000`0043f680 00000001`4024f813 : 00000000`00570ff0 00000000`0051e468
00000000`0051c530 00000000`005831f0 : postgres!standard_ExecutorRun+0xa8
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
319]
00000000`0043f6f0 00000001`4024ff5a : 00000000`0043f9f0 00000000`028d2120
00000000`0051e468 00000000`00000000 : postgres!ProcessQuery+0x93
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 190]
00000000`0043f740 00000001`4024f32a : 00000000`028d2120 00000001`4024ef01
00000000`00000000 00000000`0051c530 : postgres!PortalRunMulti+0xfa
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 1293]
00000000`0043f7a0 00000001`4024cf83 : 00000000`00582f00 00000000`00000000
00000000`00000002 00000000`00000000 : postgres!PortalRun+0x1da
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 820]
00000000`0043f980 00000001`4024b272 : 00000000`004b6f30 00000000`0051c530
00000000`0043fb70 00000000`004b6f50 : postgres!exec_simple_query+0x563
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
1052]
00000000`0043fa70 00000001`401f052a : 00000001`40557a00 00000001`4041f930
00000001`0000100d 00000001`40420868 : postgres!PostgresMain+0x602
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
4016]
00000000`0043fc60 00000001`401ed886 : 00000001`00000001 00000000`00056ab8
00000000`1bc8cf84 00000000`00000000 : postgres!BackendRun+0x1ca
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4118]
00000000`0043fcc0 00000001`40141ad0 : 00000000`00000000 00000000`00000003
00000000`00000003 00000000`004ab2c1 : postgres!SubPostmasterMain+0x2a6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4623]
00000000`0043ff00 00000001`40380bc7 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!main+0x210
[d:\pginstaller.auto\postgres.windows-x64\src\backend\main\main.c @ 215]
00000000`0043ff30 00007ff8`22bd15cd : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!__tmainCRTStartup+0x10f
[f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 626]
00000000`0043ff60 00007ff8`23cc43d1 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0043ff90 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

00000000`0043f618 00007ff8`213d12ee : 00001498`468bd12c 00000000`00000002
00000000`00000001 00000000`00000001 : ntdll!ZwWaitForMultipleObjects+0xa
00000000`0043f620 00000001`401de68e : 00000000`00000000 00007ff5`ffffe000
01cfe37b`00000000 00000000`04fba600 :
KERNELBASE!WaitForMultipleObjectsEx+0xe1
00000000`0043f900 00000001`4023cf11 : 00000000`02a55700 00000000`00000000
01cfe37b`ea074020 01cfe37b`ea074020 : postgres!PGSemaphoreLock+0x6e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\port\win32_sema.c @
145]
00000000`0043f950 00000001`400708df : 00000000`00000004 00000000`0043fa69
00000000`00000020 00000000`00000000 : postgres!LWLockAcquireCommon+0x121
[d:\pginstaller.auto\postgres.windows-x64\src\backend\storage\lmgr\lwlock.c
@ 625]
00000000`0043f9a0 00000001`40067580 : 00000000`0043fa69 00000000`54361300
00000000`00000000 00000000`00000020 :
postgres!WALInsertLockAcquireExclusive+0x2f
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1618]
00000000`0043f9d0 00000001`401e3f3d : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!CreateCheckPoint+0x1d0
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 8001]
00000000`0043fad0 00000001`400777d8 : 00000000`ffffffff 00000000`0064b2d4
00000000`0064b280 00000001`4039aac8 : postgres!CheckpointerMain+0x45d
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\checkpointer.c
@ 503]
00000000`0043fc60 00000001`401ed8fe : 00000001`40557810 00000000`00000000
00000000`00000000 00000000`00000040 : postgres!AuxiliaryProcessMain+0x4a8
[d:\pginstaller.auto\postgres.windows-x64\src\backend\bootstrap\bootstrap.c
@ 427]
00000000`0043fcc0 00000001`40141ad0 : 00000000`00000000 00000000`00000004
00000000`00000004 00000000`0064b2c9 : postgres!SubPostmasterMain+0x31e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4639]
00000000`0043ff00 00000001`40380bc7 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!main+0x210
[d:\pginstaller.auto\postgres.windows-x64\src\backend\main\main.c @ 215]
00000000`0043ff30 00007ff8`22bd15cd : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!__tmainCRTStartup+0x10f
[f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 626]
00000000`0043ff60 00007ff8`23cc43d1 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0043ff90 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

00000000`0043eb28 00007ff8`213d12ee : 03f00000`000001f0 00000000`00000002
00000000`00000001 00000000`00000000 : ntdll!ZwWaitForMultipleObjects+0xa
00000000`0043eb30 00000001`401de68e : 00000000`00000000 00007ff5`ffffb000
00000000`00000000 00000000`04fb5f80 :
KERNELBASE!WaitForMultipleObjectsEx+0xe1
00000000`0043ee10 00000001`4023cf11 : 00000000`02a55500 00000000`00000001
00000000`0066e940 00000000`00000000 : postgres!PGSemaphoreLock+0x6e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\port\win32_sema.c @
145]
00000000`0043ee60 00000001`4006203b : 00000000`05245640 00000000`00000204
00000000`00000000 00000000`625b2275 : postgres!LWLockAcquireCommon+0x121
[d:\pginstaller.auto\postgres.windows-x64\src\backend\storage\lmgr\lwlock.c
@ 625]
00000000`0043eeb0 00000001`4002cf3e : 00000000`00000000 00000000`006343e0
00000000`00000000 00000000`00000070 : postgres!XLogInsert+0x62b
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1110]
00000000`0043f180 00000001`4002a569 : 00000000`00697828 00000000`00000dc2
00000000`00000000 00000000`028b0001 : postgres!log_heap_update+0x5ee
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
6919]
00000000`0043f370 00000001`401252bc : 00000000`02896200 00000000`02896300
00000000`028b00d0 00000000`00000000 : postgres!heap_update+0xb29
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
3768]
00000000`0043f480 00000001`401246bc : 00000000`02896310 00000000`00000000
00000000`02897884 00000000`028976b0 : postgres!ExecUpdate+0x1ac
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodemodifytable.c
@ 681]
00000000`0043f550 00000001`4010c772 : 00000000`00689100 0000000b`042e0000
00000000`0043f618 00000000`02896288 : postgres!ExecModifyTable+0x33c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodemodifytable.c
@ 1029]
00000000`0043f610 00000001`4010bb6d : 00000000`02896140 00000000`006891d0
00000000`00000000 00000000`0066cd40 : postgres!ExecProcNode+0x92
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execprocnode.c
@ 377]
00000000`0043f640 00000001`401099d8 : 00000000`006715f0 00000000`0061ec00
00000000`02896140 00000000`006837f0 : postgres!ExecutePlan+0x5d
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
1481]
00000000`0043f680 00000001`4024f813 : 00000000`006715f0 00000000`0061ec98
00000000`0061cd60 00000000`006837f0 : postgres!standard_ExecutorRun+0xa8
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
319]
00000000`0043f6f0 00000001`4024ff5a : 00000000`0043f9f0 00000000`02892120
00000000`0061ec98 00000000`00000000 : postgres!ProcessQuery+0x93
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 190]
00000000`0043f740 00000001`4024f32a : 00000000`02892120 00000001`4024ef01
00000000`00000000 00000000`0061cd60 : postgres!PortalRunMulti+0xfa
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 1293]
00000000`0043f7a0 00000001`4024cf83 : 00000000`00683500 00000000`00000000
00000000`00000002 00000000`00000000 : postgres!PortalRun+0x1da
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 820]
00000000`0043f980 00000001`4024b272 : 00000000`005b6f30 00000000`0061cd60
00000000`0043fb70 00000000`005b6f50 : postgres!exec_simple_query+0x563
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
1052]
00000000`0043fa70 00000001`401f052a : 00000001`40557a00 00000001`4041f930
00000001`0000100d 00000001`40420868 : postgres!PostgresMain+0x602
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
4016]
00000000`0043fc60 00000001`401ed886 : 00000001`00000001 00000000`00047888
00000000`1bc8cf84 00000000`00000000 : postgres!BackendRun+0x1ca
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4118]
00000000`0043fcc0 00000001`40141ad0 : 00000000`00000000 00000000`00000003
00000000`00000003 00000000`005ab2c1 : postgres!SubPostmasterMain+0x2a6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4623]
00000000`0043ff00 00000001`40380bc7 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!main+0x210
[d:\pginstaller.auto\postgres.windows-x64\src\backend\main\main.c @ 215]
00000000`0043ff30 00007ff8`22bd15cd : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!__tmainCRTStartup+0x10f
[f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 626]
00000000`0043ff60 00007ff8`23cc43d1 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0043ff90 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

00000000`0043df68 00007ff8`213d12ee : 00000064`00000000 00000000`00000002
00000000`00000001 00000000`00000000 : ntdll!ZwWaitForMultipleObjects+0xa
00000000`0043df70 00000001`401de68e : 00000000`00000000 00007ff5`ffffe000
00000000`00000000 00000000`02a55700 :
KERNELBASE!WaitForMultipleObjectsEx+0xe1
00000000`0043e250 00000001`4023c921 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000016 : postgres!PGSemaphoreLock+0x6e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\port\win32_sema.c @
145]
00000000`0043e2a0 00000001`4007050c : 00000000`1c1f0000 00000000`00000200
00000000`00000004 00000000`000001a0 : postgres!LWLockWaitForVar+0xb1
[d:\pginstaller.auto\postgres.windows-x64\src\backend\storage\lmgr\lwlock.c
@ 1006]
00000000`0043e2f0 00000001`4006bff7 : 00000000`02a54aa0 00000000`1c1ee000
00000000`00000000 00000000`1c1f0000 :
postgres!WaitXLogInsertionsToFinish+0x10c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1749]
00000000`0043e330 00000001`4007061f : 00000000`02a54aa0 00000000`0000e177
00000000`00000077 00000000`000001a0 : postgres!AdvanceXLInsertBuffer+0x127
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 2153]
00000000`0043e3a0 00000001`40070077 : 00c0906c`513cedb3 00000000`1c2ee208
00000001`405546e0 00000000`1c2ee1a0 : postgres!GetXLogBuffer+0xcf
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1843]
00000000`0043e3d0 00000001`40062567 : 00000000`00000068 00000000`00000000
00000000`0043e480 00000000`253fb4eb : postgres!CopyXLogRecordToWAL+0x37
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1449]
00000000`0043e430 00000001`4002c182 : 00000000`00000008 00000000`00000000
00000000`00592f00 00000000`00000000 : postgres!XLogInsert+0xb57
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1206]
00000000`0043e700 00000001`400323b6 : 00000000`00000008 00000000`00000008
00000000`00000385 00000000`007fde10 : postgres!log_heap_clean+0x102
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
6561]
00000000`0043e7e0 00000001`400320e8 : 00000000`033305c0 00000000`00000385
00000000`0043f340 00000000`033305c0 : postgres!heap_page_prune+0x2a6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\pruneheap.c
@ 261]
00000000`0043f2f0 00000001`4002dc40 : 00000000`0062f920 00000000`00000000
00000000`00000000 00000000`00831810 : postgres!heap_page_prune_opt+0x148
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\pruneheap.c
@ 150]
00000000`0043f340 00000001`4002e7da : 00000000`00831800 00000000`0000003e
00000000`00000010 00000000`0062f920 : postgres!heapgetpage+0xa0
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
355]
00000000`0043f3e0 00000001`4002802c : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!heapgettup_pagemode+0x40a
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
944]
00000000`0043f460 00000001`40126507 : 00000000`00000000 00000000`0000001d
00000000`00000000 00000000`0000001d : postgres!heap_getnext+0x1c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
1478]
00000000`0043f490 00000001`401137f5 : 00000000`008305b0 00000000`008306c0
00000000`00000000 00000000`00803d30 : postgres!SeqNext+0x27
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodeseqscan.c
@ 76]
00000000`0043f4c0 00000001`4010c7b2 : 00000000`00643b10 00000000`008305b0
00000000`00000000 00000000`00000000 : postgres!ExecScan+0xd5
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execscan.c @
167]
00000000`0043f520 00000001`4012448d : 00000000`008302e0 00000000`008302d8
00000000`008302e0 00000000`00638110 : postgres!ExecProcNode+0xd2
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execprocnode.c
@ 400]
00000000`0043f550 00000001`4010c772 : 00000000`00639ac0 00000000`00830110
00000000`00000000 00000000`00830258 : postgres!ExecModifyTable+0x10d
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodemodifytable.c
@ 926]
00000000`0043f610 00000001`4010bb6d : 00000000`00830110 00000000`00639ac0
00000000`00000000 00000000`0061d630 : postgres!ExecProcNode+0x92
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execprocnode.c
@ 377]
00000000`0043f640 00000001`401099d8 : 00000000`00620ea0 00000000`005cf700
00000000`00830110 00000000`006340e0 : postgres!ExecutePlan+0x5d
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
1481]
00000000`0043f680 00000001`4024f813 : 00000000`00620ea0 00000000`005cf768
00000000`005cd830 00000000`006340e0 : postgres!standard_ExecutorRun+0xa8
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
319]
00000000`0043f6f0 00000001`4024ff5a : 00000000`0043f9f0 00000000`00836140
00000000`005cf768 00000000`00000000 : postgres!ProcessQuery+0x93
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 190]
00000000`0043f740 00000001`4024f32a : 00000000`00836140 00000001`4024ef01
00000000`00000000 00000000`005cd830 : postgres!PortalRunMulti+0xfa
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 1293]
00000000`0043f7a0 00000001`4024cf83 : 00000000`00633e00 00000000`00000000
00000000`00000002 00000000`00000000 : postgres!PortalRun+0x1da
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 820]
00000000`0043f980 00000001`4024b272 : 00000000`00566f30 00000000`005cd830
00000000`0043fb70 00000000`00566f50 : postgres!exec_simple_query+0x563
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
1052]
00000000`0043fa70 00000001`401f052a : 00000001`40557a00 00000001`4041f930
00000001`0000100d 00000001`40420868 : postgres!PostgresMain+0x602
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
4016]
00000000`0043fc60 00000001`401ed886 : 00000001`00000001 00000000`0008c230
00000000`1bc8cf84 00000000`00000000 : postgres!BackendRun+0x1ca
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4118]
00000000`0043fcc0 00000001`40141ad0 : 00000000`00000000 00000000`00000003
00000000`00000003 00000000`0055b2c1 : postgres!SubPostmasterMain+0x2a6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4623]
00000000`0043ff00 00000001`40380bc7 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!main+0x210
[d:\pginstaller.auto\postgres.windows-x64\src\backend\main\main.c @ 215]
00000000`0043ff30 00007ff8`22bd15cd : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!__tmainCRTStartup+0x10f
[f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 626]
00000000`0043ff60 00007ff8`23cc43d1 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0043ff90 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

00000000`0043e9e8 00007ff8`213d12ee : 00000000`00000000 00000000`00000003
00000000`00000001 00000000`00000102 : ntdll!ZwWaitForMultipleObjects+0xa
00000000`0043e9f0 00000001`401dce97 : 00000000`00000002 00007ff5`ffffe000
00000000`00000000 00000000`00000000 :
KERNELBASE!WaitForMultipleObjectsEx+0xe1
00000000`0043ecd0 00000001`401f06b6 : 00000000`00000002 00000000`00000000
00000000`006270e0 00000000`00000002 : postgres!pgwin32_select+0x3c7
[d:\pginstaller.auto\postgres.windows-x64\src\backend\port\win32\socket.c @
569]
00000000`0043fa00 00000001`401ed3dc : 00000000`00000002 00000000`00000000
00000000`00000000 00000000`005fb231 : postgres!ServerLoop+0x166
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 1530]
00000000`0043fe90 00000001`40141bab : 00000000`00000003 00000000`00624858
00000000`005fb1e0 00000000`00000002 : postgres!PostmasterMain+0xd9c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 1228]
00000000`0043ff00 00000001`40380bc7 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!main+0x2eb
[d:\pginstaller.auto\postgres.windows-x64\src\backend\main\main.c @ 228]
00000000`0043ff30 00007ff8`22bd15cd : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!__tmainCRTStartup+0x10f
[f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 626]
00000000`0043ff60 00007ff8`23cc43d1 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0043ff90 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

00000000`0043e9e8 00007ff8`213d12ee : 00000000`02c20e00 00000000`00000002
00000000`00000001 00000000`00000000 : ntdll!ZwWaitForMultipleObjects+0xa
00000000`0043e9f0 00000001`401de68e : 00000000`00000000 00007ff5`ffffe000
00000000`00000000 00000000`02a55580 :
KERNELBASE!WaitForMultipleObjectsEx+0xe1
00000000`0043ecd0 00000001`4023c921 : 00000000`00000000 00000000`047385c0
0000f12b`61c612a3 00000000`00000113 : postgres!PGSemaphoreLock+0x6e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\port\win32_sema.c @
145]
00000000`0043ed20 00000001`4007050c : 00000000`1c0f0000 00000000`00000080
00000000`00000001 00000000`00000000 : postgres!LWLockWaitForVar+0xb1
[d:\pginstaller.auto\postgres.windows-x64\src\backend\storage\lmgr\lwlock.c
@ 1006]
00000000`0043ed70 00000001`4006bff7 : 00000000`02a54aa0 00000000`00000000
00000000`00000000 00000000`1c0f0000 :
postgres!WaitXLogInsertionsToFinish+0x10c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1749]
00000000`0043edb0 00000001`4007061f : 00000000`02a54aa0 00000000`0000e0f7
00000000`00000077 00000000`00000000 : postgres!AdvanceXLInsertBuffer+0x127
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 2153]
00000000`0043ee20 00000001`400700e7 : 00000000`02b43eb3 00000000`0000014d
00000000`00000410 00000000`1c1ee000 : postgres!GetXLogBuffer+0xcf
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1843]
00000000`0043ee50 00000001`40062567 : 00000000`000005a8 00000000`00000500
00000000`0043f260 00000000`93a75654 : postgres!CopyXLogRecordToWAL+0xa7
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1486]
00000000`0043eeb0 00000001`4002cf3e : 00000000`00000000 00000000`005a4250
00000000`00000000 00000000`00000070 : postgres!XLogInsert+0xb57
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1206]
00000000`0043f180 00000001`4002a569 : 00000000`005fef70 00000000`00000e21
00000000`00000000 00000000`005fef01 : postgres!log_heap_update+0x5ee
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
6919]
00000000`0043f370 00000001`401252bc : 00000000`00838200 00000000`00838300
00000000`005fef70 00000000`00000000 : postgres!heap_update+0xb29
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
3768]
00000000`0043f480 00000001`401246bc : 00000000`00838320 00000000`00000000
00000000`00839894 00000000`008396c0 : postgres!ExecUpdate+0x1ac
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodemodifytable.c
@ 681]
00000000`0043f550 00000001`4010c772 : 00000000`005f7b00 00000003`0d910000
00000000`0043f618 00000000`00838298 : postgres!ExecModifyTable+0x33c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodemodifytable.c
@ 1029]
00000000`0043f610 00000001`4010bb6d : 00000000`00838150 00000000`005f7bb0
00000000`00000000 00000000`005dc730 : postgres!ExecProcNode+0x92
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execprocnode.c
@ 377]
00000000`0043f640 00000001`401099d8 : 00000000`005deb60 00000000`0058e400
00000000`00838150 00000000`005f31e0 : postgres!ExecutePlan+0x5d
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
1481]
00000000`0043f680 00000001`4024f813 : 00000000`005deb60 00000000`0058e468
00000000`0058c530 00000000`005f31e0 : postgres!standard_ExecutorRun+0xa8
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
319]
00000000`0043f6f0 00000001`4024ff5a : 00000000`0043f9f0 00000000`00834130
00000000`0058e468 00000000`00000000 : postgres!ProcessQuery+0x93
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 190]
00000000`0043f740 00000001`4024f32a : 00000000`00834130 00000001`4024ef01
00000000`00000000 00000000`0058c530 : postgres!PortalRunMulti+0xfa
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 1293]
00000000`0043f7a0 00000001`4024cf83 : 00000000`005f2f00 00000000`00000000
00000000`00000002 00000000`00000000 : postgres!PortalRun+0x1da
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 820]
00000000`0043f980 00000001`4024b272 : 00000000`00526f30 00000000`0058c530
00000000`0043fb70 00000000`00526f50 : postgres!exec_simple_query+0x563
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
1052]
00000000`0043fa70 00000001`401f052a : 00000001`40557a00 00000001`4041f930
00000001`0000100d 00000001`40420868 : postgres!PostgresMain+0x602
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
4016]
00000000`0043fc60 00000001`401ed886 : 00000001`00000001 00000000`00047888
00000000`1bc8cf84 00000000`00000000 : postgres!BackendRun+0x1ca
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4118]
00000000`0043fcc0 00000001`40141ad0 : 00000000`00000000 00000000`00000003
00000000`00000003 00000000`0051b2c1 : postgres!SubPostmasterMain+0x2a6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4623]
00000000`0043ff00 00000001`40380bc7 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!main+0x210
[d:\pginstaller.auto\postgres.windows-x64\src\backend\main\main.c @ 215]
00000000`0043ff30 00007ff8`22bd15cd : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!__tmainCRTStartup+0x10f
[f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 626]
00000000`0043ff60 00007ff8`23cc43d1 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0043ff90 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

00000000`0043f7d8 00007ff8`213d12ee : 00007375`74617473 00000000`00000000
2f676f6c`785f6770 5f657669`68637261 : ntdll!ZwWaitForMultipleObjects+0xa
00000000`0043f7e0 00007ff8`22bd134f : 00000001`40557558 00007ff5`ffffe000
00000000`00000000 00000000`00000019 :
KERNELBASE!WaitForMultipleObjectsEx+0xe1
00000000`0043fac0 00000001`401de1ac : 00000000`00000000 00000000`00000019
00000000`0060a2b0 00000001`401dd467 : kernel32!WaitForMultipleObjects+0xf
00000000`0043fb00 00000001`401e5248 : 01cfe37d`bafbea94 00000000`54361610
00000000`00000000 00000000`00000000 : postgres!WaitLatchOrSocket+0x22c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\port\win32_latch.c @
203]
00000000`0043fc10 00000001`401e4e96 : 00000001`40557810 00000000`00000000
00000000`0000003f 00000000`00000040 : postgres!pgarch_MainLoop+0x208
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\pgarch.c @
419]
00000000`0043fc90 00000001`401edaf5 : 00000000`00000003 00000000`00000000
00000000`000000af 00000000`00000003 : postgres!PgArchiverMain+0x116
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\pgarch.c @
269]
00000000`0043fcc0 00000001`40141ad0 : 00000000`00000000 00000000`00000003
00000000`00000003 00000000`0060a301 : postgres!SubPostmasterMain+0x515
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4700]
00000000`0043ff00 00000001`40380bc7 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!main+0x210
[d:\pginstaller.auto\postgres.windows-x64\src\backend\main\main.c @ 215]
00000000`0043ff30 00007ff8`22bd15cd : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!__tmainCRTStartup+0x10f
[f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 626]
00000000`0043ff60 00007ff8`23cc43d1 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0043ff90 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

00000000`0043f708 00007ff8`213d12ee : 00000000`00000060 00000000`00000000
00000000`00000000 00000000`00000040 : ntdll!ZwWaitForMultipleObjects+0xa
00000000`0043f710 00007ff8`22bd134f : 00000000`04fb8bb0 00007ff5`ffffe000
00000000`00000000 00000000`00000019 :
KERNELBASE!WaitForMultipleObjectsEx+0xe1
00000000`0043f9f0 00000001`401de1ac : 00000000`00000000 00000000`0062b270
00000000`00000000 00000000`00000000 : kernel32!WaitForMultipleObjects+0xf
00000000`0043fa30 00000001`401df8f8 : 0001a7f5`0e6c0cf8 0001a7f5`0e6c0c00
00000000`00000000 00000001`40557a10 : postgres!WaitLatchOrSocket+0x22c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\port\win32_latch.c @
203]
00000000`0043fb40 00000001`401ed97e : 00000001`40557810 00000000`00000040
00000000`00000000 00000000`00000003 : postgres!AutoVacLauncherMain+0x418
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\autovacuum.c
@ 615]
00000000`0043fcc0 00000001`40141ad0 : 00000000`00000000 00000000`00000003
00000000`00000003 00000000`0062b2c1 : postgres!SubPostmasterMain+0x39e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4655]
00000000`0043ff00 00000001`40380bc7 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!main+0x210
[d:\pginstaller.auto\postgres.windows-x64\src\backend\main\main.c @ 215]
00000000`0043ff30 00007ff8`22bd15cd : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!__tmainCRTStartup+0x10f
[f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 626]
00000000`0043ff60 00007ff8`23cc43d1 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0043ff90 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

00000000`0043e988 00007ff8`213d12ee : 00000000`000003ff 00007ff8`00000080
00000000`00000007 02cd0000`00000001 : ntdll!ZwWaitForMultipleObjects+0xa
00000000`0043e990 00000001`401de68e : 00000000`00000000 00007ff5`ffffe000
00000000`00000000 00000000`04fb82c0 :
KERNELBASE!WaitForMultipleObjectsEx+0xe1
00000000`0043ec70 00000001`4023cf11 : 00000000`02a24920 00000000`00000000
00000000`0043ed10 00000001`40220e36 : postgres!PGSemaphoreLock+0x6e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\port\win32_sema.c @
145]
00000000`0043ecc0 00000001`4002dc4e : 00000000`01de7c80 00000000`01de7c30
00000000`000004d8 00000000`01de7c80 : postgres!LWLockAcquireCommon+0x121
[d:\pginstaller.auto\postgres.windows-x64\src\backend\storage\lmgr\lwlock.c
@ 625]
00000000`0043ed10 00000001`4002e7da : 00000000`01de19c0 00000000`00000006
00000000`005f6998 00000000`01de7c30 : postgres!heapgetpage+0xae
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
357]
00000000`0043edb0 00000001`4002802c : 00000000`01de7c00 00000000`00000001
00000000`00000000 00000000`00000000 : postgres!heapgettup_pagemode+0x40a
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
944]
00000000`0043ee30 00000001`401eacae : 00000000`01de7c80 00000000`01de7c30
00000000`005b5b50 00000001`40557a10 : postgres!heap_getnext+0x1c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
1478]
00000000`0043ee60 00000001`401e71b9 : 00000000`00000000 00000000`00002dd4
00000000`01de1930 00007ff8`13278a8e : postgres!pgstat_collect_oids+0x12e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\pgstat.c @
1151]
00000000`0043ef00 00000001`401e0e0f : 00000000`00004000 00000000`00000000
00000001`40557a10 00000000`00000000 : postgres!pgstat_vacuum_stat+0x129
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\pgstat.c @
1003]
00000000`0043f750 00000001`401dff74 : 00000000`00004000 00000000`00000000
00000001`40557a10 00000001`4041dd30 : postgres!do_autovacuum+0x9f
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\autovacuum.c
@ 1955]
00000000`0043fb30 00000001`401ed9fe : 00000001`40557810 00000000`00000000
00000000`00000000 00000000`00000003 : postgres!AutoVacWorkerMain+0x364
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\autovacuum.c
@ 1687]
00000000`0043fcc0 00000001`40141ad0 : 00000000`00000000 00000000`00000003
00000000`00000003 00000000`0050b2c1 : postgres!SubPostmasterMain+0x41e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4671]
00000000`0043ff00 00000001`40380bc7 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!main+0x210
[d:\pginstaller.auto\postgres.windows-x64\src\backend\main\main.c @ 215]
00000000`0043ff30 00007ff8`22bd15cd : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!__tmainCRTStartup+0x10f
[f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 626]
00000000`0043ff60 00007ff8`23cc43d1 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0043ff90 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

00000000`0043f5b8 00007ff8`213d12ee : 00000000`c0000000 00000007`00000000
00000000`00000000 00000000`00000001 : ntdll!ZwWaitForMultipleObjects+0xa
00000000`0043f5c0 00000001`401de68e : 00000000`00000000 00007ff5`ffffe000
00000000`00000000 00000000`02a55700 :
KERNELBASE!WaitForMultipleObjectsEx+0xe1
00000000`0043f8a0 00000001`4023c921 : 00000000`00000000 00000000`00000001
00000000`0043f940 00000000`000000de : postgres!PGSemaphoreLock+0x6e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\port\win32_sema.c @
145]
00000000`0043f8f0 00000001`4007050c : 00000000`1c2ec228 00000000`00000200
00000000`00000004 00000000`004f7360 : postgres!LWLockWaitForVar+0xb1
[d:\pginstaller.auto\postgres.windows-x64\src\backend\storage\lmgr\lwlock.c
@ 1006]
00000000`0043f940 00000001`40062806 : 00000000`02a54aa0 00000000`1c1ee000
00000000`1c2ec228 00000000`1c0f28e8 :
postgres!WaitXLogInsertionsToFinish+0x10c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1749]
00000000`0043f980 00000001`40223af7 : 00000000`02c09980 00000000`00000001
00000000`00000015 00000000`00000015 : postgres!XLogFlush+0xd6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 2799]
00000000`0043f9f0 00000001`40223209 : 00000000`00000001 00000000`00000015
00000000`00000002 01cfe37b`ea47a480 : postgres!FlushBuffer+0xf7
[d:\pginstaller.auto\postgres.windows-x64\src\backend\storage\buffer\bufmgr.c
@ 1987]
00000000`0043fa80 00000001`402227e4 : 00000000`00000101 00000000`00000be2
00000000`00000c72 00000000`00000015 : postgres!SyncOneBuffer+0xc9
[d:\pginstaller.auto\postgres.windows-x64\src\backend\storage\buffer\bufmgr.c
@ 1696]
00000000`0043fab0 00000001`401e38d7 : 000c5440`000000a7 0001a7f4`00000000
00000000`00000008 00000000`00000000 : postgres!BgBufferSync+0x1f4
[d:\pginstaller.auto\postgres.windows-x64\src\backend\storage\buffer\bufmgr.c
@ 1578]
00000000`0043fb00 00000001`400777c9 : 00000000`ffffffff 00000000`004ab2d4
00000000`004ab280 00000001`4039aac8 : postgres!BackgroundWriterMain+0x227
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\bgwriter.c
@ 284]
00000000`0043fc60 00000001`401ed8fe : 00000001`40557810 00000000`00000000
00000000`00000000 00000000`00000040 : postgres!AuxiliaryProcessMain+0x499
[d:\pginstaller.auto\postgres.windows-x64\src\backend\bootstrap\bootstrap.c
@ 422]
00000000`0043fcc0 00000001`40141ad0 : 00000000`00000000 00000000`00000004
00000000`00000004 00000000`004ab2c9 : postgres!SubPostmasterMain+0x31e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4639]
00000000`0043ff00 00000001`40380bc7 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!main+0x210
[d:\pginstaller.auto\postgres.windows-x64\src\backend\main\main.c @ 215]
00000000`0043ff30 00007ff8`22bd15cd : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!__tmainCRTStartup+0x10f
[f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 626]
00000000`0043ff60 00007ff8`23cc43d1 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0043ff90 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

00000000`0043e0a8 00007ff8`213d12ee : 00000000`00000002 00000000`00000002
00000000`00000001 00000000`00000000 : ntdll!ZwWaitForMultipleObjects+0xa
00000000`0043e0b0 00000001`401de68e : 00000000`00000000 00007ff5`ffffe000
00000000`00000000 00000000`04fb7fd0 :
KERNELBASE!WaitForMultipleObjectsEx+0xe1
00000000`0043e390 00000001`4023cf11 : 00000000`02a55680 00000000`0043e468
00000000`00000212 00000001`4005297d : postgres!PGSemaphoreLock+0x6e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\port\win32_sema.c @
145]
00000000`0043e3e0 00000001`4006203b : 00000000`0c6e89c5 00000000`00000048
00000000`00000000 00000000`00000100 : postgres!LWLockAcquireCommon+0x121
[d:\pginstaller.auto\postgres.windows-x64\src\backend\storage\lmgr\lwlock.c
@ 625]
00000000`0043e430 00000001`4002c182 : 00000000`00000008 00000000`00000000
00000056`00640101 0000002b`00000010 : postgres!XLogInsert+0x62b
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1110]
00000000`0043e700 00000001`400323b6 : 00000000`00000008 00000000`00000008
00000000`0000029d 00000000`0089de10 : postgres!log_heap_clean+0x102
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
6561]
00000000`0043e7e0 00000001`400320e8 : 00000000`031605c0 00000000`0000029d
00000000`0043f340 00000000`031605c0 : postgres!heap_page_prune+0x2a6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\pruneheap.c
@ 261]
00000000`0043f2f0 00000001`4002dc40 : 00000000`006df920 00000000`00000000
00000000`00000000 00000000`008d1810 : postgres!heap_page_prune_opt+0x148
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\pruneheap.c
@ 150]
00000000`0043f340 00000001`4002e7da : 00000000`008d1800 00000000`0000003e
00000000`00000010 00000000`006df920 : postgres!heapgetpage+0xa0
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
355]
00000000`0043f3e0 00000001`4002802c : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!heapgettup_pagemode+0x40a
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
944]
00000000`0043f460 00000001`40126507 : 00000000`00000000 00000000`0000001d
00000000`00000000 00000000`0000001d : postgres!heap_getnext+0x1c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
1478]
00000000`0043f490 00000001`401137f5 : 00000000`008d05b0 00000000`008d06c0
00000000`00000000 00000000`008a3d30 : postgres!SeqNext+0x27
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodeseqscan.c
@ 76]
00000000`0043f4c0 00000001`4010c7b2 : 00000000`006f63d0 00000000`008d05b0
00000000`00000000 00000000`00000000 : postgres!ExecScan+0xd5
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execscan.c @
167]
00000000`0043f520 00000001`4012448d : 00000000`008d02e0 00000000`008d02d8
00000000`008d02e0 00000000`006e8110 : postgres!ExecProcNode+0xd2
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execprocnode.c
@ 400]
00000000`0043f550 00000001`4010c772 : 00000000`006e9ac0 00000000`008d0110
00000000`00000000 00000000`008d0258 : postgres!ExecModifyTable+0x10d
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodemodifytable.c
@ 926]
00000000`0043f610 00000001`4010bb6d : 00000000`008d0110 00000000`006e9ac0
00000000`00000000 00000000`006cd630 : postgres!ExecProcNode+0x92
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execprocnode.c
@ 377]
00000000`0043f640 00000001`401099d8 : 00000000`006d2f20 00000000`0067f700
00000000`008d0110 00000000`006e40e0 : postgres!ExecutePlan+0x5d
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
1481]
00000000`0043f680 00000001`4024f813 : 00000000`006d2f20 00000000`0067f768
00000000`0067d830 00000000`006e40e0 : postgres!standard_ExecutorRun+0xa8
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
319]
00000000`0043f6f0 00000001`4024ff5a : 00000000`0043f9f0 00000000`008d6140
00000000`0067f768 00000000`00000000 : postgres!ProcessQuery+0x93
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 190]
00000000`0043f740 00000001`4024f32a : 00000000`008d6140 00000001`4024ef01
00000000`00000000 00000000`0067d830 : postgres!PortalRunMulti+0xfa
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 1293]
00000000`0043f7a0 00000001`4024cf83 : 00000000`006e3e00 00000000`00000000
00000000`00000002 00000000`00000000 : postgres!PortalRun+0x1da
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 820]
00000000`0043f980 00000001`4024b272 : 00000000`00616f30 00000000`0067d830
00000000`0043fb70 00000000`00616f50 : postgres!exec_simple_query+0x563
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
1052]
00000000`0043fa70 00000001`401f052a : 00000001`40557a00 00000001`4041f930
00000001`0000100d 00000001`40420868 : postgres!PostgresMain+0x602
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
4016]
00000000`0043fc60 00000001`401ed886 : 00000001`00000001 00000000`00080a98
00000000`1bc8cf84 00000000`00000000 : postgres!BackendRun+0x1ca
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4118]
00000000`0043fcc0 00000001`40141ad0 : 00000000`00000000 00000000`00000003
00000000`00000003 00000000`0060b2c1 : postgres!SubPostmasterMain+0x2a6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4623]
00000000`0043ff00 00000001`40380bc7 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!main+0x210
[d:\pginstaller.auto\postgres.windows-x64\src\backend\main\main.c @ 215]
00000000`0043ff30 00007ff8`22bd15cd : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!__tmainCRTStartup+0x10f
[f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 626]
00000000`0043ff60 00007ff8`23cc43d1 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0043ff90 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

00000000`0043da78 00007ff8`213d12ee : 00000000`02a106c0 00000001`40220af2
00000000`0043dc70 00000001`40222e8b : ntdll!ZwWaitForMultipleObjects+0xa
00000000`0043da80 00000001`401de68e : 00000000`00000000 00007ff5`ffffe000
00000000`00000000 00000000`04fb88a0 :
KERNELBASE!WaitForMultipleObjectsEx+0xe1
00000000`0043dd60 00000001`4023cf11 : 00000000`02a55800 00000000`0043de38
00000000`00000015 00000001`4006bc7e : postgres!PGSemaphoreLock+0x6e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\port\win32_sema.c @
145]
00000000`0043ddb0 00000001`4006203b : 00000000`beff0611 00000000`00000018
00000000`00000000 00000000`00000100 : postgres!LWLockAcquireCommon+0x121
[d:\pginstaller.auto\postgres.windows-x64\src\backend\storage\lmgr\lwlock.c
@ 625]
00000000`0043de00 00000001`4002c182 : 00000000`00000001 00000000`00000002
00000000`00000000 00000001`400403c1 : postgres!XLogInsert+0x62b
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1110]
00000000`0043e0d0 00000001`400323b6 : 00000000`00000000 00000000`00000000
00000000`000004d8 00000000`006e6010 : postgres!log_heap_clean+0x102
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
6561]
00000000`0043e1b0 00000001`400320e8 : 00000000`035d65c0 00000000`000004d8
00000000`0043ed10 00000000`035d65c0 : postgres!heap_page_prune+0x2a6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\pruneheap.c
@ 261]
00000000`0043ecc0 00000001`4002dc40 : 00000000`028d3c10 00000000`00000000
00000000`00000027 00000000`028d3c60 : postgres!heap_page_prune_opt+0x148
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\pruneheap.c
@ 150]
00000000`0043ed10 00000001`4002e7da : 00000000`028d9a00 00000000`00000006
00000000`00726e58 00000000`028d3c10 : postgres!heapgetpage+0xa0
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
355]
00000000`0043edb0 00000001`4002802c : 00000000`028d3c00 00000000`00000001
00000000`00000000 00000000`00000000 : postgres!heapgettup_pagemode+0x40a
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
944]
00000000`0043ee30 00000001`401eacae : 00000000`028d3c60 00000000`028d3c10
00000000`006e6010 00000001`40557a10 : postgres!heap_getnext+0x1c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
1478]
00000000`0043ee60 00000001`401e71b9 : 00000000`00000000 00000000`00002dd4
00000000`028d5950 00007ff8`13278a8e : postgres!pgstat_collect_oids+0x12e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\pgstat.c @
1151]
00000000`0043ef00 00000001`401e0e0f : 00000000`00004000 00000000`00000000
00000001`40557a10 00000000`00000000 : postgres!pgstat_vacuum_stat+0x129
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\pgstat.c @
1003]
00000000`0043f750 00000001`401dff74 : 00000000`00004000 00000000`00000000
00000001`40557a10 00000001`4041dd30 : postgres!do_autovacuum+0x9f
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\autovacuum.c
@ 1955]
00000000`0043fb30 00000001`401ed9fe : 00000001`40557810 00000000`00000000
00000000`00000000 00000000`00000003 : postgres!AutoVacWorkerMain+0x364
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\autovacuum.c
@ 1687]
00000000`0043fcc0 00000001`40141ad0 : 00000000`00000000 00000000`00000003
00000000`00000003 00000000`0063b2c1 : postgres!SubPostmasterMain+0x41e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4671]
00000000`0043ff00 00000001`40380bc7 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!main+0x210
[d:\pginstaller.auto\postgres.windows-x64\src\backend\main\main.c @ 215]
00000000`0043ff30 00007ff8`22bd15cd : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!__tmainCRTStartup+0x10f
[f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 626]
00000000`0043ff60 00007ff8`23cc43d1 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0043ff90 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

00000000`0043e0a8 00007ff8`213d12ee : 00000000`00000002 00000000`00000002
00000000`00000001 00000000`00000000 : ntdll!ZwWaitForMultipleObjects+0xa
00000000`0043e0b0 00000001`401de68e : 00000000`00000000 00007ff5`ffffe000
00000000`00000000 00000000`04fb6e30 :
KERNELBASE!WaitForMultipleObjectsEx+0xe1
00000000`0043e390 00000001`4023cf11 : 00000000`02a55780 00000000`0043e468
00000000`00000212 00000001`4005297d : postgres!PGSemaphoreLock+0x6e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\port\win32_sema.c @
145]
00000000`0043e3e0 00000001`4006203b : 00000000`280a7857 00000000`00000022
00000000`00000000 00000000`00000400 : postgres!LWLockAcquireCommon+0x121
[d:\pginstaller.auto\postgres.windows-x64\src\backend\storage\lmgr\lwlock.c
@ 625]
00000000`0043e430 00000001`4002c182 : 00000000`00000005 00000000`00000000
00000000`005d3f00 00000000`00000000 : postgres!XLogInsert+0x62b
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1110]
00000000`0043e700 00000001`400323b6 : 00000000`00000000 00000000`00000000
00000000`000003a6 00000000`01dede10 : postgres!log_heap_clean+0x102
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
6561]
00000000`0043e7e0 00000001`400320e8 : 00000000`033725c0 00000000`000003a6
00000000`0043f340 00000000`033725c0 : postgres!heap_page_prune+0x2a6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\pruneheap.c
@ 261]
00000000`0043f2f0 00000001`4002dc40 : 00000000`00673250 00000000`00000000
00000000`00000000 00000000`01e27840 : postgres!heap_page_prune_opt+0x148
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\pruneheap.c
@ 150]
00000000`0043f340 00000001`4002e7da : 00000000`01e27800 00000000`00000c80
00000000`00000005 00000000`00673250 : postgres!heapgetpage+0xa0
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
355]
00000000`0043f3e0 00000001`4002802c : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!heapgettup_pagemode+0x40a
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
944]
00000000`0043f460 00000001`40126507 : 00000000`00000000 00000000`0000001d
00000000`00000000 00000000`0000001d : postgres!heap_getnext+0x1c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
1478]
00000000`0043f490 00000001`401137f5 : 00000000`01e265e0 00000000`01e266f0
00000000`00000000 00000000`01df3d30 : postgres!SeqNext+0x27
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodeseqscan.c
@ 76]
00000000`0043f4c0 00000001`4010c7b2 : 00000000`006823c0 00000000`01e265e0
00000000`00000000 00000000`00000000 : postgres!ExecScan+0xd5
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execscan.c @
167]
00000000`0043f520 00000001`4012448d : 00000000`01e26310 00000000`01e26308
00000000`01e26310 00000000`0067a220 : postgres!ExecProcNode+0xd2
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execprocnode.c
@ 400]
00000000`0043f550 00000001`4010c772 : 00000000`0067c3e0 00000000`01e26140
00000000`00000000 00000000`01e26288 : postgres!ExecModifyTable+0x10d
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodemodifytable.c
@ 926]
00000000`0043f610 00000001`4010bb6d : 00000000`01e26140 00000000`0067c3e0
00000000`00000000 00000000`00660f60 : postgres!ExecProcNode+0x92
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execprocnode.c
@ 377]
00000000`0043f640 00000001`401099d8 : 00000000`00663ba0 00000000`00613000
00000000`01e26140 00000000`00677a10 : postgres!ExecutePlan+0x5d
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
1481]
00000000`0043f680 00000001`4024f813 : 00000000`00663ba0 00000000`00613078
00000000`00611140 00000000`00677a10 : postgres!standard_ExecutorRun+0xa8
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
319]
00000000`0043f6f0 00000001`4024ff5a : 00000000`0043f9f0 00000000`01e24130
00000000`00613078 00000000`00000000 : postgres!ProcessQuery+0x93
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 190]
00000000`0043f740 00000001`4024f32a : 00000000`01e24130 00000001`4024ef01
00000000`00000000 00000000`00611140 : postgres!PortalRunMulti+0xfa
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 1293]
00000000`0043f7a0 00000001`4024cf83 : 00000000`00677700 00000000`00000000
00000000`00000002 00000000`00000000 : postgres!PortalRun+0x1da
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 820]
00000000`0043f980 00000001`4024b272 : 00000000`005a6f30 00000000`00611140
00000000`0043fb70 00000000`005a6f50 : postgres!exec_simple_query+0x563
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
1052]
00000000`0043fa70 00000001`401f052a : 00000001`40557a00 00000001`4041f930
00000001`0000100d 00000001`40420868 : postgres!PostgresMain+0x602
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
4016]
00000000`0043fc60 00000001`401ed886 : 00000001`00000001 00000000`00093b48
00000000`1bc8cf84 00000000`00000000 : postgres!BackendRun+0x1ca
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4118]
00000000`0043fcc0 00000001`40141ad0 : 00000000`00000000 00000000`00000003
00000000`00000003 00000000`0059b2c1 : postgres!SubPostmasterMain+0x2a6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4623]
00000000`0043ff00 00000001`40380bc7 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!main+0x210
[d:\pginstaller.auto\postgres.windows-x64\src\backend\main\main.c @ 215]
00000000`0043ff30 00007ff8`22bd15cd : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!__tmainCRTStartup+0x10f
[f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 626]
00000000`0043ff60 00007ff8`23cc43d1 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0043ff90 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

00000000`0043e0a8 00007ff8`213d12ee : 00000000`00000002 00000000`00000002
00000000`00000001 00000000`00000000 : ntdll!ZwWaitForMultipleObjects+0xa
00000000`0043e0b0 00000001`401de68e : 00000000`00000000 00007ff5`ffffe000
00000000`00000000 00000000`04fb6850 :
KERNELBASE!WaitForMultipleObjectsEx+0xe1
00000000`0043e390 00000001`4023cf11 : 00000000`02a55680 00000000`0043e468
00000000`00000212 00000001`4005297d : postgres!PGSemaphoreLock+0x6e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\port\win32_sema.c @
145]
00000000`0043e3e0 00000001`4006203b : 00000000`360a2ce5 00000000`00000022
00000000`00000000 00000000`00000400 : postgres!LWLockAcquireCommon+0x121
[d:\pginstaller.auto\postgres.windows-x64\src\backend\storage\lmgr\lwlock.c
@ 625]
00000000`0043e430 00000001`4002c182 : 00000000`00000005 00007ff8`23c958a7
00000000`00699000 00000000`00000000 : postgres!XLogInsert+0x62b
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1110]
00000000`0043e700 00000001`400323b6 : 00000000`00000000 00000000`00000000
00000000`000001f4 00000000`008ade10 : postgres!log_heap_clean+0x102
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
6561]
00000000`0043e7e0 00000001`400320e8 : 00000000`0300e5c0 00000000`000001f4
00000000`0043f340 00000000`0300e5c0 : postgres!heap_page_prune+0x2a6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\pruneheap.c
@ 261]
00000000`0043f2f0 00000001`4002dc40 : 00000000`006ef710 00000000`00000000
00000000`00000000 00000000`008e7840 : postgres!heap_page_prune_opt+0x148
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\pruneheap.c
@ 150]
00000000`0043f340 00000001`4002e7da : 00000000`008e7800 00000000`00000c80
00000000`00000005 00000000`006ef710 : postgres!heapgetpage+0xa0
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
355]
00000000`0043f3e0 00000001`4002802c : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!heapgettup_pagemode+0x40a
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
944]
00000000`0043f460 00000001`40126507 : 00000000`00000000 00000000`0000001d
00000000`00000000 00000000`0000001d : postgres!heap_getnext+0x1c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
1478]
00000000`0043f490 00000001`401137f5 : 00000000`008e65e0 00000000`008e66f0
00000000`00000000 00000000`008b3d30 : postgres!SeqNext+0x27
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodeseqscan.c
@ 76]
00000000`0043f4c0 00000001`4010c7b2 : 00000000`006fe880 00000000`008e65e0
00000000`00000000 00000000`00000000 : postgres!ExecScan+0xd5
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execscan.c @
167]
00000000`0043f520 00000001`4012448d : 00000000`008e6310 00000000`008e6308
00000000`008e6310 00000000`006f66e0 : postgres!ExecProcNode+0xd2
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execprocnode.c
@ 400]
00000000`0043f550 00000001`4010c772 : 00000000`006f88a0 00000000`008e6140
00000000`00000000 00000000`008e6288 : postgres!ExecModifyTable+0x10d
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodemodifytable.c
@ 926]
00000000`0043f610 00000001`4010bb6d : 00000000`008e6140 00000000`006f88a0
00000000`00000000 00000000`006dd420 : postgres!ExecProcNode+0x92
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execprocnode.c
@ 377]
00000000`0043f640 00000001`401099d8 : 00000000`006e0060 00000000`0068ec00
00000000`008e6140 00000000`006f3ed0 : postgres!ExecutePlan+0x5d
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
1481]
00000000`0043f680 00000001`4024f813 : 00000000`006e0060 00000000`0068ec98
00000000`0068cd60 00000000`006f3ed0 : postgres!standard_ExecutorRun+0xa8
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
319]
00000000`0043f6f0 00000001`4024ff5a : 00000000`0043f9f0 00000000`008e0110
00000000`0068ec98 00000000`00000000 : postgres!ProcessQuery+0x93
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 190]
00000000`0043f740 00000001`4024f32a : 00000000`008e0110 00000001`4024ef01
00000000`00000000 00000000`0068cd60 : postgres!PortalRunMulti+0xfa
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 1293]
00000000`0043f7a0 00000001`4024cf83 : 00000000`006f3c00 00000000`00000000
00000000`00000002 00000000`00000000 : postgres!PortalRun+0x1da
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 820]
00000000`0043f980 00000001`4024b272 : 00000000`00626f30 00000000`0068cd60
00000000`0043fb70 00000000`00626f50 : postgres!exec_simple_query+0x563
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
1052]
00000000`0043fa70 00000001`401f052a : 00000001`40557a00 00000001`4041f930
00000001`0000100d 00000001`40420868 : postgres!PostgresMain+0x602
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
4016]
00000000`0043fc60 00000001`401ed886 : 00000001`00000001 00000000`0007cc18
00000000`1bc8cf84 00000000`00000000 : postgres!BackendRun+0x1ca
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4118]
00000000`0043fcc0 00000001`40141ad0 : 00000000`00000000 00000000`00000003
00000000`00000003 00000000`0061b2c1 : postgres!SubPostmasterMain+0x2a6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4623]
00000000`0043ff00 00000001`40380bc7 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!main+0x210
[d:\pginstaller.auto\postgres.windows-x64\src\backend\main\main.c @ 215]
00000000`0043ff30 00007ff8`22bd15cd : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!__tmainCRTStartup+0x10f
[f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 626]
00000000`0043ff60 00007ff8`23cc43d1 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0043ff90 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

00000000`0043f408 00007ff8`213d12ee : 00000000`00000000 00000000`00000009
00000000`004b0000 00000000`00000000 : ntdll!ZwWaitForMultipleObjects+0xa
00000000`0043f410 00007ff8`22bd134f : 00000000`00000021 00007ff5`ffffe000
00000000`00000000 00000000`0000001b :
KERNELBASE!WaitForMultipleObjectsEx+0xe1
00000000`0043f6f0 00000001`401de1ac : 00000000`00000000 00000000`000007d0
00000000`0000001b 00007ff8`00000000 : kernel32!WaitForMultipleObjects+0xf
00000000`0043f730 00000001`401e6d20 : 00000000`01a5bc9c 00000000`004f4928
00000000`0043f940 00000000`00000000 : postgres!WaitLatchOrSocket+0x22c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\port\win32_latch.c @
203]
00000000`0043f840 00000001`401edb52 : 00000001`40557810 00000000`00000000
00000001`40557a10 00000000`004ba2b0 : postgres!PgstatCollectorMain+0x770
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\pgstat.c @
3409]
00000000`0043fcc0 00000001`40141ad0 : 00000000`00000000 00000000`00000003
00000000`00000003 00000000`004ba301 : postgres!SubPostmasterMain+0x572
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4709]
00000000`0043ff00 00000001`40380bc7 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!main+0x210
[d:\pginstaller.auto\postgres.windows-x64\src\backend\main\main.c @ 215]
00000000`0043ff30 00007ff8`22bd15cd : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!__tmainCRTStartup+0x10f
[f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 626]
00000000`0043ff60 00007ff8`23cc43d1 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0043ff90 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

00000000`0043f6e8 00007ff8`213d12ee : 00000000`0043f710 00000000`00000000
00000003`00000000 00000000`00000102 : ntdll!ZwWaitForMultipleObjects+0xa
00000000`0043f6f0 00000001`401de68e : 00000000`00000000 00007ff5`ffffe000
00000000`00000000 00000000`02a55700 :
KERNELBASE!WaitForMultipleObjectsEx+0xe1
00000000`0043f9d0 00000001`4023c921 : 00000000`00000000 00000000`00000001
00000000`000004ac 00000001`4006c0a9 : postgres!PGSemaphoreLock+0x6e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\port\win32_sema.c @
145]
00000000`0043fa20 00000001`4007050c : 00000000`1c2ec000 00000000`00000200
00000000`00000004 00000000`000004ac : postgres!LWLockWaitForVar+0xb1
[d:\pginstaller.auto\postgres.windows-x64\src\backend\storage\lmgr\lwlock.c
@ 1006]
00000000`0043fa70 00000001`40062a8a : 00000000`1c2ec000 00000000`1c1ee000
00000000`0051b200 00000000`00000000 :
postgres!WaitXLogInsertionsToFinish+0x10c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1749]
00000000`0043fab0 00000001`401f5b00 : 00000000`00000032 00000000`00000000
00000001`40000000 00000000`0051b2d4 : postgres!XLogBackgroundFlush+0x11a
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 2971]
00000000`0043fb00 00000001`400777ec : 00000000`ffffffff 00000000`0051b280
00000000`0051b2d4 00000001`4039aac8 : postgres!WalWriterMain+0x260
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\walwriter.c
@ 288]
00000000`0043fc60 00000001`401ed8fe : 00000001`40557810 00000000`00000000
00000000`00000000 00000000`00000040 : postgres!AuxiliaryProcessMain+0x4bc
[d:\pginstaller.auto\postgres.windows-x64\src\backend\bootstrap\bootstrap.c
@ 433]
00000000`0043fcc0 00000001`40141ad0 : 00000000`00000000 00000000`00000004
00000000`00000004 00000000`0051b2c9 : postgres!SubPostmasterMain+0x31e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4639]
00000000`0043ff00 00000001`40380bc7 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!main+0x210
[d:\pginstaller.auto\postgres.windows-x64\src\backend\main\main.c @ 215]
00000000`0043ff30 00007ff8`22bd15cd : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!__tmainCRTStartup+0x10f
[f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 626]

00000000`0043e988 00007ff8`213d12ee : 00000000`000003ff 00007ff8`00000080
00000000`00000007 02940000`00000001 : ntdll!ZwWaitForMultipleObjects+0xa
00000000`0043e990 00000001`401de68e : 00000000`00000000 00007ff5`ffffe000
00000000`00000000 00000000`04fb85b0 :
KERNELBASE!WaitForMultipleObjectsEx+0xe1
00000000`0043ec70 00000001`4023cf11 : 00000000`02a24920 00000000`00000000
00000000`0043ed10 00000001`40220e36 : postgres!PGSemaphoreLock+0x6e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\port\win32_sema.c @
145]
00000000`0043ecc0 00000001`4002dc4e : 00000000`008e7c80 00000000`008e7c30
00000000`000004d8 00000000`008e7c80 : postgres!LWLockAcquireCommon+0x121
[d:\pginstaller.auto\postgres.windows-x64\src\backend\storage\lmgr\lwlock.c
@ 625]
00000000`0043ed10 00000001`4002e7da : 00000000`008e39d0 00000000`00000006
00000000`00706d28 00000000`008e7c30 : postgres!heapgetpage+0xae
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
357]
00000000`0043edb0 00000001`4002802c : 00000000`008e7c00 00000000`00000001
00000000`00000000 00000000`00000000 : postgres!heapgettup_pagemode+0x40a
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
944]
00000000`0043ee30 00000001`401eacae : 00000000`008e7c80 00000000`008e7c30
00000000`006c5ee0 00000001`40557a10 : postgres!heap_getnext+0x1c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
1478]
00000000`0043ee60 00000001`401e71b9 : 00000000`00000000 00000000`00002dd4
00000000`008e3940 00007ff8`13278a8e : postgres!pgstat_collect_oids+0x12e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\pgstat.c @
1151]
00000000`0043ef00 00000001`401e0e0f : 00000000`00004000 00000000`00000000
00000001`40557a10 00000000`00000000 : postgres!pgstat_vacuum_stat+0x129
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\pgstat.c @
1003]
00000000`0043f750 00000001`401dff74 : 00000000`00004000 00000000`00000000
00000001`40557a10 00000001`4041dd30 : postgres!do_autovacuum+0x9f
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\autovacuum.c
@ 1955]
00000000`0043fb30 00000001`401ed9fe : 00000001`40557810 00000000`00000000
00000000`00000000 00000000`00000003 : postgres!AutoVacWorkerMain+0x364
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\autovacuum.c
@ 1687]
00000000`0043fcc0 00000001`40141ad0 : 00000000`00000000 00000000`00000003
00000000`00000003 00000000`0061b2c1 : postgres!SubPostmasterMain+0x41e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4671]
00000000`0043ff00 00000001`40380bc7 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!main+0x210
[d:\pginstaller.auto\postgres.windows-x64\src\backend\main\main.c @ 215]
00000000`0043ff30 00007ff8`22bd15cd : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!__tmainCRTStartup+0x10f
[f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 626]
00000000`0043ff60 00007ff8`23cc43d1 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0043ff90 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

00000000`0043df68 00007ff8`213d12ee : 00000918`1d4b494e 00000000`00000002
00000000`00000001 00000000`00000000 : ntdll!ZwWaitForMultipleObjects+0xa
00000000`0043df70 00000001`401de68e : 00000000`00000000 00007ff5`ffffe000
00000000`00000000 00000000`02a55700 :
KERNELBASE!WaitForMultipleObjectsEx+0xe1
00000000`0043e250 00000001`4023c921 : 00000000`00000000 00000000`04fb7120
00000000`00000001 00000000`00000016 : postgres!PGSemaphoreLock+0x6e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\port\win32_sema.c @
145]
00000000`0043e2a0 00000001`4007050c : 00000000`1c1f0000 00000000`00000200
00000000`00000004 00000000`00000158 : postgres!LWLockWaitForVar+0xb1
[d:\pginstaller.auto\postgres.windows-x64\src\backend\storage\lmgr\lwlock.c
@ 1006]
00000000`0043e2f0 00000001`4006bff7 : 00000000`02a54aa0 00000000`1c1ee000
00000000`00000000 00000000`1c1f0000 :
postgres!WaitXLogInsertionsToFinish+0x10c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1749]
00000000`0043e330 00000001`4007061f : 00000000`02a54aa0 00000000`0000e177
00000000`00000077 00000000`00000158 : postgres!AdvanceXLInsertBuffer+0x127
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 2153]
00000000`0043e3a0 00000001`40070077 : 00c0906c`513cedb3 00000000`1c2ee1a0
00000001`405546e0 00000000`1c2ee158 : postgres!GetXLogBuffer+0xcf
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1843]
00000000`0043e3d0 00000001`40062567 : 00000000`00000042 00000000`00000000
00000000`0043e480 00000000`992954e8 : postgres!CopyXLogRecordToWAL+0x37
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1449]
00000000`0043e430 00000001`4002c182 : 00000000`00000005 00000000`00000000
00000000`00000070 00000000`00000000 : postgres!XLogInsert+0xb57
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1206]
00000000`0043e700 00000001`400323b6 : 00000000`00000000 00000000`00000000
00000000`0000010f 00000000`01e5de10 : postgres!log_heap_clean+0x102
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
6561]
00000000`0043e7e0 00000001`400320e8 : 00000000`02e445c0 00000000`0000010f
00000000`0043f340 00000000`02e445c0 : postgres!heap_page_prune+0x2a6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\pruneheap.c
@ 261]
00000000`0043f2f0 00000001`4002dc40 : 00000000`0070eec0 00000000`00000000
00000000`00000000 00000000`01e99850 : postgres!heap_page_prune_opt+0x148
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\pruneheap.c
@ 150]
00000000`0043f340 00000001`4002e7da : 00000000`01e99800 00000000`00000ccd
00000000`00000005 00000000`0070eec0 : postgres!heapgetpage+0xa0
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
355]
00000000`0043f3e0 00000001`4002802c : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!heapgettup_pagemode+0x40a
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
944]
00000000`0043f460 00000001`40126507 : 00000000`00000000 00000000`0000001d
00000000`00000000 00000000`0000001d : postgres!heap_getnext+0x1c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
1478]
00000000`0043f490 00000001`401137f5 : 00000000`01e985f0 00000000`01e98700
00000000`00000000 00000000`01e63d30 : postgres!SeqNext+0x27
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodeseqscan.c
@ 76]
00000000`0043f4c0 00000001`4010c7b2 : 00000000`0071e030 00000000`01e985f0
00000000`00000000 00000000`00000000 : postgres!ExecScan+0xd5
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execscan.c @
167]
00000000`0043f520 00000001`4012448d : 00000000`01e98320 00000000`01e98318
00000000`01e98320 00000000`00715e90 : postgres!ExecProcNode+0xd2
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execprocnode.c
@ 400]
00000000`0043f550 00000001`4010c772 : 00000000`00718050 00000000`01e98150
00000000`00000000 00000000`01e98298 : postgres!ExecModifyTable+0x10d
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodemodifytable.c
@ 926]
00000000`0043f610 00000001`4010bb6d : 00000000`01e98150 00000000`00718050
00000000`00000000 00000000`006fcbd0 : postgres!ExecProcNode+0x92
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execprocnode.c
@ 377]
00000000`0043f640 00000001`401099d8 : 00000000`00700030 00000000`006ae400
00000000`01e98150 00000000`00713680 : postgres!ExecutePlan+0x5d
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
1481]
00000000`0043f680 00000001`4024f813 : 00000000`00700030 00000000`006ae468
00000000`006ac530 00000000`00713680 : postgres!standard_ExecutorRun+0xa8
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
319]
00000000`0043f6f0 00000001`4024ff5a : 00000000`0043f9f0 00000000`01e96140
00000000`006ae468 00000000`00000000 : postgres!ProcessQuery+0x93
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 190]
00000000`0043f740 00000001`4024f32a : 00000000`01e96140 00000001`4024ef01
00000000`00000000 00000000`006ac530 : postgres!PortalRunMulti+0xfa
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 1293]
00000000`0043f7a0 00000001`4024cf83 : 00000000`00713300 00000000`00000000
00000000`00000002 00000000`00000000 : postgres!PortalRun+0x1da
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 820]
00000000`0043f980 00000001`4024b272 : 00000000`00646f30 00000000`006ac530
00000000`0043fb70 00000000`00646f50 : postgres!exec_simple_query+0x563
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
1052]
00000000`0043fa70 00000001`401f052a : 00000001`40557a00 00000001`4041f930
00000001`0000100d 00000001`40420868 : postgres!PostgresMain+0x602
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
4016]
00000000`0043fc60 00000001`401ed886 : 00000001`00000001 00000000`0006d9e8
00000000`1bc8cf84 00000000`00000000 : postgres!BackendRun+0x1ca
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4118]
00000000`0043fcc0 00000001`40141ad0 : 00000000`00000000 00000000`00000003
00000000`00000003 00000000`0063b2c1 : postgres!SubPostmasterMain+0x2a6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4623]
00000000`0043ff00 00000001`40380bc7 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!main+0x210
[d:\pginstaller.auto\postgres.windows-x64\src\backend\main\main.c @ 215]
00000000`0043ff30 00007ff8`22bd15cd : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!__tmainCRTStartup+0x10f
[f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 626]
00000000`0043ff60 00007ff8`23cc43d1 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0043ff90 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

00000000`0043f818 00007ff8`213d12ee : 00000000`00000000 00000000`00000000
00007ff5`ffecc000 00000000`00001500 : ntdll!ZwWaitForMultipleObjects+0xa
00000000`0043f820 00007ff8`22bd134f : 00000001`40557e98 00007ff5`ffffe000
00000000`00000000 00000000`00000009 :
KERNELBASE!WaitForMultipleObjectsEx+0xe1
00000000`0043fb00 00000001`401de1ac : 00000000`00000000 00000001`4037ade5
00000000`00000002 00000001`4037a1f5 : kernel32!WaitForMultipleObjects+0xf
00000000`0043fb40 00000001`401f4a80 : 00000000`0232ca38 00000000`00000009
00000000`00000007 00000000`00000400 : postgres!WaitLatchOrSocket+0x22c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\port\win32_latch.c @
203]
00000000`0043fc50 00000001`401edba9 : 00000001`40557a10 00000000`00000000
00000000`543612dd 00000000`00000004 : postgres!SysLoggerMain+0x5d0
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\syslogger.c
@ 502]
00000000`0043fcc0 00000001`40141ad0 : 00000000`00000000 00000000`00000004
00000000`00000004 00000000`0048b2c9 : postgres!SubPostmasterMain+0x5c9
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4719]
00000000`0043ff00 00000001`40380bc7 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!main+0x210
[d:\pginstaller.auto\postgres.windows-x64\src\backend\main\main.c @ 215]
00000000`0043ff30 00007ff8`22bd15cd : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!__tmainCRTStartup+0x10f
[f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 626]
00000000`0043ff60 00007ff8`23cc43d1 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0043ff90 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

00000000`0043e0a8 00007ff8`213d12ee : 00000000`00000002 00000000`00000002
00000000`00000001 00000000`00000000 : ntdll!ZwWaitForMultipleObjects+0xa
00000000`0043e0b0 00000001`401de68e : 00000000`00000000 00007ff5`ffffe000
00000000`00000000 00000000`04fb7700 :
KERNELBASE!WaitForMultipleObjectsEx+0xe1
00000000`0043e390 00000001`4023cf11 : 00000000`02a55500 00000000`0043e468
00000000`00000212 00000001`4005297d : postgres!PGSemaphoreLock+0x6e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\port\win32_sema.c @
145]
00000000`0043e3e0 00000001`4006203b : 00000000`476c7789 00000000`00000048
00000000`00000000 00000000`00000100 : postgres!LWLockAcquireCommon+0x121
[d:\pginstaller.auto\postgres.windows-x64\src\backend\storage\lmgr\lwlock.c
@ 625]
00000000`0043e430 00000001`4002c182 : 00000000`00000008 00000000`00000000
00000000`00633f00 00000000`00000000 : postgres!XLogInsert+0x62b
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1110]
00000000`0043e700 00000001`400323b6 : 00000000`00000008 00000000`00000008
00000000`0000042f 00000000`01e1de10 : postgres!log_heap_clean+0x102
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
6561]
00000000`0043e7e0 00000001`400320e8 : 00000000`034845c0 00000000`0000042f
00000000`0043f340 00000000`034845c0 : postgres!heap_page_prune+0x2a6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\pruneheap.c
@ 261]
00000000`0043f2f0 00000001`4002dc40 : 00000000`006ce3f0 00000000`00000000
00000000`00000000 00000000`01e5b790 : postgres!heap_page_prune_opt+0x148
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\pruneheap.c
@ 150]
00000000`0043f340 00000001`4002e7da : 00000000`01e5b700 00000000`0000003e
00000000`00000010 00000000`006ce3f0 : postgres!heapgetpage+0xa0
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
355]
00000000`0043f3e0 00000001`4002802c : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!heapgettup_pagemode+0x40a
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
944]
00000000`0043f460 00000001`40126507 : 00000000`00000000 00000000`0000001d
00000000`00000000 00000000`0000001d : postgres!heap_getnext+0x1c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
1478]
00000000`0043f490 00000001`401137f5 : 00000000`01e5a530 00000000`01e5a640
00000000`00000000 00000000`01e23d30 : postgres!SeqNext+0x27
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodeseqscan.c
@ 76]
00000000`0043f4c0 00000001`4010c7b2 : 00000000`006e7660 00000000`01e5a530
00000000`00000000 00000000`00000000 : postgres!ExecScan+0xd5
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execscan.c @
167]
00000000`0043f520 00000001`4012448d : 00000000`01e5a260 00000000`01e5a258
00000000`01e5a260 00000000`006d9420 : postgres!ExecProcNode+0xd2
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execprocnode.c
@ 400]
00000000`0043f550 00000001`4010c772 : 00000000`006dadd0 00000000`01e5a090
00000000`00000000 00000000`01e5a1d8 : postgres!ExecModifyTable+0x10d
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodemodifytable.c
@ 926]
00000000`0043f610 00000001`4010bb6d : 00000000`01e5a090 00000000`006dadd0
00000000`00000000 00000000`006bd930 : postgres!ExecProcNode+0x92
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execprocnode.c
@ 377]
00000000`0043f640 00000001`401099d8 : 00000000`006bfd70 00000000`0066f600
00000000`01e5a090 00000000`006d2bb0 : postgres!ExecutePlan+0x5d
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
1481]
00000000`0043f680 00000001`4024f813 : 00000000`006bfd70 00000000`0066f658
00000000`0066d720 00000000`006d2bb0 : postgres!standard_ExecutorRun+0xa8
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
319]
00000000`0043f6f0 00000001`4024ff5a : 00000000`0043f9f0 00000000`01e56070
00000000`0066f658 00000000`00000000 : postgres!ProcessQuery+0x93
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 190]
00000000`0043f740 00000001`4024f32a : 00000000`01e56070 00000001`4024ef01
00000000`00000000 00000000`0066d720 : postgres!PortalRunMulti+0xfa
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 1293]
00000000`0043f7a0 00000001`4024cf83 : 00000000`006d2800 00000000`00000000
00000000`00000002 00000000`00000000 : postgres!PortalRun+0x1da
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 820]
00000000`0043f980 00000001`4024b272 : 00000000`00606f30 00000000`0066d720
00000000`0043fb70 00000000`00606f50 : postgres!exec_simple_query+0x563
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
1052]
00000000`0043fa70 00000001`401f052a : 00000001`40557a00 00000001`4041f930
00000001`0000100d 00000001`40420868 : postgres!PostgresMain+0x602
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
4016]
00000000`0043fc60 00000001`401ed886 : 00000001`00000001 00000000`000979c8
00000000`1bc8cf84 00000000`00000000 : postgres!BackendRun+0x1ca
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4118]
00000000`0043fcc0 00000001`40141ad0 : 00000000`00000000 00000000`00000003
00000000`00000003 00000000`005fb2c1 : postgres!SubPostmasterMain+0x2a6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4623]
00000000`0043ff00 00000001`40380bc7 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!main+0x210
[d:\pginstaller.auto\postgres.windows-x64\src\backend\main\main.c @ 215]
00000000`0043ff30 00007ff8`22bd15cd : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!__tmainCRTStartup+0x10f
[f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 626]
00000000`0043ff60 00007ff8`23cc43d1 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0043ff90 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

00000000`0043e0a8 00007ff8`213d12ee : 00000000`00000002 00000000`00000002
00000000`00000001 00000000`00000000 : ntdll!ZwWaitForMultipleObjects+0xa
00000000`0043e0b0 00000001`401de68e : 00000000`00000000 00007ff5`ffffc000
00000000`00000000 00000000`04fb6270 :
KERNELBASE!WaitForMultipleObjectsEx+0xe1
00000000`0043e390 00000001`4023cf11 : 00000000`02a55580 00000000`1c1ede10
80605042`36ad2501 00000001`405546e0 : postgres!PGSemaphoreLock+0x6e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\port\win32_sema.c @
145]
00000000`0043e3e0 00000001`4006203b : 00000000`fb22ca99 00000000`00000038
00000000`00000000 00000000`00000400 : postgres!LWLockAcquireCommon+0x121
[d:\pginstaller.auto\postgres.windows-x64\src\backend\storage\lmgr\lwlock.c
@ 625]
00000000`0043e430 00000001`4002c182 : 00000000`00000010 00007ff8`23cbcf62
00000000`00000000 00000000`00000000 : postgres!XLogInsert+0x62b
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1110]
00000000`0043e700 00000001`400323b6 : 00000000`00000000 00000000`00000000
00000000`00000e25 00000000`0288de10 : postgres!log_heap_clean+0x102
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
6561]
00000000`0043e7e0 00000001`400320e8 : 00000000`048705c0 00000000`00000e25
00000000`0043f340 00000000`048705c0 : postgres!heap_page_prune+0x2a6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\pruneheap.c
@ 261]
00000000`0043f2f0 00000001`4002dc40 : 00000000`0066db60 00000000`00000000
00000000`00000000 00000000`028cb790 : postgres!heap_page_prune_opt+0x148
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\pruneheap.c
@ 150]
00000000`0043f340 00000001`4002e7da : 00000000`028cb700 0288e001`00000403
00000000`00000010 00000000`0066db60 : postgres!heapgetpage+0xa0
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
355]
00000000`0043f3e0 00000001`4002802c : 00000000`028ca000 00000000`028cb600
00000000`00000000 00000000`00000000 : postgres!heapgettup_pagemode+0x40a
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
944]
00000000`0043f460 00000001`40126507 : 00000000`006899a0 00000000`00000000
00000000`006899a0 00000000`00000000 : postgres!heap_getnext+0x1c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
1478]
00000000`0043f490 00000001`401137f5 : 00000000`028ca530 00000000`028ca640
00000000`00000002 00000000`028ca300 : postgres!SeqNext+0x27
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodeseqscan.c
@ 76]
00000000`0043f4c0 00000001`4010c7b2 : 00000000`00000002 00000000`028ca530
00000000`00000000 00000000`00000000 : postgres!ExecScan+0xd5
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execscan.c @
167]
00000000`0043f520 00000001`4012448d : 00000000`028ca260 00000000`00000000
00000000`028cb7d4 00000000`028cb600 : postgres!ExecProcNode+0xd2
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execprocnode.c
@ 400]
00000000`0043f550 00000001`4010c772 : 00000000`0067a500 00000020`003e0000
00000000`0043f618 00000000`028ca1d8 : postgres!ExecModifyTable+0x10d
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodemodifytable.c
@ 926]
00000000`0043f610 00000001`4010bb6d : 00000000`028ca090 00000000`0067a540
00000000`00000000 00000000`0065c890 : postgres!ExecProcNode+0x92
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execprocnode.c
@ 377]
00000000`0043f640 00000001`401099d8 : 00000000`0065ecc0 00000000`0060ed00
00000000`028ca090 00000000`00672320 : postgres!ExecutePlan+0x5d
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
1481]
00000000`0043f680 00000001`4024f813 : 00000000`0065ecc0 00000000`0060edb8
00000000`0060ce80 00000000`00672320 : postgres!standard_ExecutorRun+0xa8
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
319]
00000000`0043f6f0 00000001`4024ff5a : 00000000`0043f9f0 00000000`028c8080
00000000`0060edb8 00000000`00000000 : postgres!ProcessQuery+0x93
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 190]
00000000`0043f740 00000001`4024f32a : 00000000`028c8080 00000001`4024ef01
00000000`00000000 00000000`0060ce80 : postgres!PortalRunMulti+0xfa
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 1293]
00000000`0043f7a0 00000001`4024cf83 : 00000000`00672000 00000000`00000000
00000000`00000002 00000000`00000000 : postgres!PortalRun+0x1da
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 820]
00000000`0043f980 00000001`4024b272 : 00000000`005a6f30 00000000`0060ce80
00000000`0043fb70 00000000`005a6f50 : postgres!exec_simple_query+0x563
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
1052]
00000000`0043fa70 00000001`401f052a : 00000001`40557a00 00000001`4041f930
00000001`0000100d 00000001`40420868 : postgres!PostgresMain+0x602
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
4016]
00000000`0043fc60 00000001`401ed886 : 00000001`00000001 00000000`00062250
00000000`1bc8cf84 00000000`00000000 : postgres!BackendRun+0x1ca
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4118]
00000000`0043fcc0 00000001`40141ad0 : 00000000`00000000 00000000`00000003
00000000`00000003 00000000`0059b2c1 : postgres!SubPostmasterMain+0x2a6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4623]
00000000`0043ff00 00000001`40380bc7 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!main+0x210
[d:\pginstaller.auto\postgres.windows-x64\src\backend\main\main.c @ 215]
00000000`0043ff30 00007ff8`22bd15cd : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!__tmainCRTStartup+0x10f
[f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 626]
00000000`0043ff60 00007ff8`23cc43d1 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0043ff90 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

00000000`0043e9e8 00007ff8`213d12ee : 00000000`00000000 00000000`00000002
00000000`00000001 00000000`00000000 : ntdll!ZwWaitForMultipleObjects+0xa
00000000`0043e9f0 00000001`401de68e : 00000000`00000000 00007ff5`ffffd000
00000000`00000000 00000000`02a55700 :
KERNELBASE!WaitForMultipleObjectsEx+0xe1
00000000`0043ecd0 00000001`4023c921 : 00000000`00000000 00000000`02d205c0
00009732`7a0a4e67 00000000`00000104 : postgres!PGSemaphoreLock+0x6e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\port\win32_sema.c @
145]
00000000`0043ed20 00000001`4007050c : 00000000`1c1f0000 00000000`00000200
00000000`00000004 00000000`00000000 : postgres!LWLockWaitForVar+0xb1
[d:\pginstaller.auto\postgres.windows-x64\src\backend\storage\lmgr\lwlock.c
@ 1006]
00000000`0043ed70 00000001`4006bff7 : 00000000`02a54aa0 00000000`1c1ee000
00000000`00000000 00000000`1c1f0000 :
postgres!WaitXLogInsertionsToFinish+0x10c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1749]
00000000`0043edb0 00000001`4007061f : 00000000`02a54aa0 00000000`0000e177
00000000`00000077 00000000`00000000 : postgres!AdvanceXLInsertBuffer+0x127
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 2153]
00000000`0043ee20 00000001`400700e7 : 00000000`02b43be3 00000000`0000041d
00000000`00000140 00000000`1c2ee000 : postgres!GetXLogBuffer+0xcf
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1843]
00000000`0043ee50 00000001`40062567 : 00000000`000005a8 00000000`00000500
00000000`0043f260 00000000`1c13d4d0 : postgres!CopyXLogRecordToWAL+0xa7
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1486]
00000000`0043eeb0 00000001`4002cf3e : 00000000`00000000 00000000`00593980
00000000`00000000 00000000`00000070 : postgres!XLogInsert+0xb57
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1206]
00000000`0043f180 00000001`4002a569 : 00000000`005eef80 00000000`0000010c
00000000`00000000 00000000`005eef01 : postgres!log_heap_update+0x5ee
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
6919]
00000000`0043f370 00000001`401252bc : 00000000`02908200 00000000`02908300
00000000`005eef80 00000000`00000000 : postgres!heap_update+0xb29
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
3768]
00000000`0043f480 00000001`401246bc : 00000000`02908320 00000000`00000000
00000000`02909894 00000000`029096c0 : postgres!ExecUpdate+0x1ac
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodemodifytable.c
@ 681]
00000000`0043f550 00000001`4010c772 : 00000000`005e7b00 00000002`0dff0000
00000000`0043f618 00000000`02908298 : postgres!ExecModifyTable+0x33c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodemodifytable.c
@ 1029]
00000000`0043f610 00000001`4010bb6d : 00000000`02908150 00000000`005e7bc0
00000000`00000000 00000000`005cc740 : postgres!ExecProcNode+0x92
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execprocnode.c
@ 377]
00000000`0043f640 00000001`401099d8 : 00000000`005d1810 00000000`0057e400
00000000`02908150 00000000`005e31f0 : postgres!ExecutePlan+0x5d
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
1481]
00000000`0043f680 00000001`4024f813 : 00000000`005d1810 00000000`0057e468
00000000`0057c530 00000000`005e31f0 : postgres!standard_ExecutorRun+0xa8
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
319]
00000000`0043f6f0 00000001`4024ff5a : 00000000`0043f9f0 00000000`02900110
00000000`0057e468 00000000`00000000 : postgres!ProcessQuery+0x93
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 190]
00000000`0043f740 00000001`4024f32a : 00000000`02900110 00000001`4024ef01
00000000`00000000 00000000`0057c530 : postgres!PortalRunMulti+0xfa
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 1293]
00000000`0043f7a0 00000001`4024cf83 : 00000000`005e2f00 00000000`00000000
00000000`00000002 00000000`00000000 : postgres!PortalRun+0x1da
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 820]
00000000`0043f980 00000001`4024b272 : 00000000`00516f30 00000000`0057c530
00000000`0043fb70 00000000`00516f50 : postgres!exec_simple_query+0x563
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
1052]
00000000`0043fa70 00000001`401f052a : 00000001`40557a00 00000001`4041f930
00000001`0000100d 00000001`40420868 : postgres!PostgresMain+0x602
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
4016]
00000000`0043fc60 00000001`401ed886 : 00000001`00000001 00000000`00038270
00000000`1bc8cf84 00000000`00000000 : postgres!BackendRun+0x1ca
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4118]
00000000`0043fcc0 00000001`40141ad0 : 00000000`00000000 00000000`00000003
00000000`00000003 00000000`0050b2c1 : postgres!SubPostmasterMain+0x2a6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4623]
00000000`0043ff00 00000001`40380bc7 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!main+0x210
[d:\pginstaller.auto\postgres.windows-x64\src\backend\main\main.c @ 215]
00000000`0043ff30 00007ff8`22bd15cd : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!__tmainCRTStartup+0x10f
[f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 626]
00000000`0043ff60 00007ff8`23cc43d1 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0043ff90 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

00000000`0043e0a8 00007ff8`213d12ee : 00000000`00000002 00000000`00000002
00000000`00000001 00000000`00000000 : ntdll!ZwWaitForMultipleObjects+0xa
00000000`0043e0b0 00000001`401de68e : 00000000`00000000 00007ff5`ffffe000
00000000`00000000 00000000`04fb7410 :
KERNELBASE!WaitForMultipleObjectsEx+0xe1
00000000`0043e390 00000001`4023cf11 : 00000000`02a55880 00000000`0043e468
00000000`00000212 00000001`4005297d : postgres!PGSemaphoreLock+0x6e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\port\win32_sema.c @
145]
00000000`0043e3e0 00000001`4006203b : 00000000`e4f47e1a 00000000`00000022
00000000`00000000 00000000`00000400 : postgres!LWLockAcquireCommon+0x121
[d:\pginstaller.auto\postgres.windows-x64\src\backend\storage\lmgr\lwlock.c
@ 625]
00000000`0043e430 00000001`4002c182 : 00000000`00000005 00000000`00000000
00000056`00690101 0000002b`00000010 : postgres!XLogInsert+0x62b
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1110]
00000000`0043e700 00000001`400323b6 : 00000000`00000000 00000000`00000000
00000000`000002fd 00000000`028bde10 : postgres!log_heap_clean+0x102
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
6561]
00000000`0043e7e0 00000001`400320e8 : 00000000`032205c0 00000000`000002fd
00000000`0043f340 00000000`032205c0 : postgres!heap_page_prune+0x2a6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\pruneheap.c
@ 261]
00000000`0043f2f0 00000001`4002dc40 : 00000000`0072f960 00000000`00000000
00000000`00000000 00000000`028f5830 : postgres!heap_page_prune_opt+0x148
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\pruneheap.c
@ 150]
00000000`0043f340 00000001`4002e7da : 00000000`028f5800 00000000`00000c80
00000000`00000005 00000000`0072f960 : postgres!heapgetpage+0xa0
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
355]
00000000`0043f3e0 00000001`4002802c : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!heapgettup_pagemode+0x40a
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
944]
00000000`0043f460 00000001`40126507 : 00000000`00000000 00000000`0000001d
00000000`00000000 00000000`0000001d : postgres!heap_getnext+0x1c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
1478]
00000000`0043f490 00000001`401137f5 : 00000000`028f45d0 00000000`028f46e0
00000000`00000000 00000000`028c3d30 : postgres!SeqNext+0x27
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodeseqscan.c
@ 76]
00000000`0043f4c0 00000001`4010c7b2 : 00000000`0073ead0 00000000`028f45d0
00000000`00000000 00000000`00000000 : postgres!ExecScan+0xd5
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execscan.c @
167]
00000000`0043f520 00000001`4012448d : 00000000`028f4300 00000000`028f42f8
00000000`028f4300 00000000`00736930 : postgres!ExecProcNode+0xd2
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execprocnode.c
@ 400]
00000000`0043f550 00000001`4010c772 : 00000000`00738af0 00000000`028f4130
00000000`00000000 00000000`028f4278 : postgres!ExecModifyTable+0x10d
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodemodifytable.c
@ 926]
00000000`0043f610 00000001`4010bb6d : 00000000`028f4130 00000000`00738af0
00000000`00000000 00000000`0071d670 : postgres!ExecProcNode+0x92
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execprocnode.c
@ 377]
00000000`0043f640 00000001`401099d8 : 00000000`0071f680 00000000`006cee00
00000000`028f4130 00000000`00734120 : postgres!ExecutePlan+0x5d
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
1481]
00000000`0043f680 00000001`4024f813 : 00000000`0071f680 00000000`006ceef8
00000000`006ccfc0 00000000`00734120 : postgres!standard_ExecutorRun+0xa8
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
319]
00000000`0043f6f0 00000001`4024ff5a : 00000000`0043f9f0 00000000`028f6140
00000000`006ceef8 00000000`00000000 : postgres!ProcessQuery+0x93
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 190]
00000000`0043f740 00000001`4024f32a : 00000000`028f6140 00000001`4024ef01
00000000`00000000 00000000`006ccfc0 : postgres!PortalRunMulti+0xfa
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 1293]
00000000`0043f7a0 00000001`4024cf83 : 00000000`00733e00 00000000`00000000
00000000`00000002 00000000`00000000 : postgres!PortalRun+0x1da
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 820]
00000000`0043f980 00000001`4024b272 : 00000000`00666f30 00000000`006ccfc0
00000000`0043fb70 00000000`00666f50 : postgres!exec_simple_query+0x563
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
1052]
00000000`0043fa70 00000001`401f052a : 00000001`40557a00 00000001`4041f930
00000001`0000100d 00000001`40420868 : postgres!PostgresMain+0x602
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
4016]
00000000`0043fc60 00000001`401ed886 : 00000001`00000001 00000000`000883b0
00000000`1bc8cf84 00000000`00000000 : postgres!BackendRun+0x1ca
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4118]
00000000`0043fcc0 00000001`40141ad0 : 00000000`00000000 00000000`00000003
00000000`00000003 00000000`0065b2c1 : postgres!SubPostmasterMain+0x2a6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4623]
00000000`0043ff00 00000001`40380bc7 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!main+0x210
[d:\pginstaller.auto\postgres.windows-x64\src\backend\main\main.c @ 215]
00000000`0043ff30 00007ff8`22bd15cd : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!__tmainCRTStartup+0x10f
[f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 626]
00000000`0043ff60 00007ff8`23cc43d1 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0043ff90 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

00000000`0043e0a8 00007ff8`213d12ee : 00000000`00000002 00000000`00000002
00000000`00000001 00000000`00000000 : ntdll!ZwWaitForMultipleObjects+0xa
00000000`0043e0b0 00000001`401de68e : 00000000`00000000 00007ff5`ffffd000
00000000`00000000 00000000`04fb53c0 :
KERNELBASE!WaitForMultipleObjectsEx+0xe1
00000000`0043e390 00000001`4023cf11 : 00000000`02a55700 00000000`0043e468
00000000`00000212 00000001`4005297d : postgres!PGSemaphoreLock+0x6e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\port\win32_sema.c @
145]
00000000`0043e3e0 00000001`4006203b : 00000000`c44a1328 00000000`00000048
00000000`00000000 00000000`00000100 : postgres!LWLockAcquireCommon+0x121
[d:\pginstaller.auto\postgres.windows-x64\src\backend\storage\lmgr\lwlock.c
@ 625]
00000000`0043e430 00000001`4002c182 : 00000000`00000008 00000000`00000000
00000000`00692f00 00000000`00000000 : postgres!XLogInsert+0x62b
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1110]
00000000`0043e700 00000001`400323b6 : 00000000`00000008 00000000`00000008
00000000`0000015b 00000000`028bde10 : postgres!log_heap_clean+0x102
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
6561]
00000000`0043e7e0 00000001`400320e8 : 00000000`02edc5c0 00000000`0000015b
00000000`0043f340 00000000`02edc5c0 : postgres!heap_page_prune+0x2a6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\pruneheap.c
@ 261]
00000000`0043f2f0 00000001`4002dc40 : 00000000`0072e3f0 00000000`00000000
00000000`00000000 00000000`028f5760 : postgres!heap_page_prune_opt+0x148
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\pruneheap.c
@ 150]
00000000`0043f340 00000001`4002e7da : 00000000`028f5700 00000000`0000003e
00000000`00000010 00000000`0072e3f0 : postgres!heapgetpage+0xa0
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
355]
00000000`0043f3e0 00000001`4002802c : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!heapgettup_pagemode+0x40a
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
944]
00000000`0043f460 00000001`40126507 : 00000000`00000000 00000000`0000001d
00000000`00000000 00000000`0000001d : postgres!heap_getnext+0x1c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
1478]
00000000`0043f490 00000001`401137f5 : 00000000`028f4500 00000000`028f4610
00000000`00000000 00000000`028c3d30 : postgres!SeqNext+0x27
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodeseqscan.c
@ 76]
00000000`0043f4c0 00000001`4010c7b2 : 00000000`00747660 00000000`028f4500
00000000`00000000 00000000`00000000 : postgres!ExecScan+0xd5
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execscan.c @
167]
00000000`0043f520 00000001`4012448d : 00000000`028f4230 00000000`028f4228
00000000`028f4230 00000000`00739420 : postgres!ExecProcNode+0xd2
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execprocnode.c
@ 400]
00000000`0043f550 00000001`4010c772 : 00000000`0073add0 00000000`028f4060
00000000`00000000 00000000`028f41a8 : postgres!ExecModifyTable+0x10d
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodemodifytable.c
@ 926]
00000000`0043f610 00000001`4010bb6d : 00000000`028f4060 00000000`0073add0
00000000`00000000 00000000`0071d930 : postgres!ExecProcNode+0x92
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execprocnode.c
@ 377]
00000000`0043f640 00000001`401099d8 : 00000000`00721df0 00000000`006cf600
00000000`028f4060 00000000`00732bb0 : postgres!ExecutePlan+0x5d
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
1481]
00000000`0043f680 00000001`4024f813 : 00000000`00721df0 00000000`006cf658
00000000`006cd720 00000000`00732bb0 : postgres!standard_ExecutorRun+0xa8
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
319]
00000000`0043f6f0 00000001`4024ff5a : 00000000`0043f9f0 00000000`028fa090
00000000`006cf658 00000000`00000000 : postgres!ProcessQuery+0x93
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 190]
00000000`0043f740 00000001`4024f32a : 00000000`028fa090 00000001`4024ef01
00000000`00000000 00000000`006cd720 : postgres!PortalRunMulti+0xfa
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 1293]
00000000`0043f7a0 00000001`4024cf83 : 00000000`00732800 00000000`00000000
00000000`00000002 00000000`00000000 : postgres!PortalRun+0x1da
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 820]
00000000`0043f980 00000001`4024b272 : 00000000`00666f30 00000000`006cd720
00000000`0043fb70 00000000`00666f50 : postgres!exec_simple_query+0x563
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
1052]
00000000`0043fa70 00000001`401f052a : 00000001`40557a00 00000001`4041f930
00000001`0000100d 00000001`40420868 : postgres!PostgresMain+0x602
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
4016]
00000000`0043fc60 00000001`401ed886 : 00000001`00000001 00000000`00075300
00000000`1bc8cf84 00000000`00000000 : postgres!BackendRun+0x1ca
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4118]
00000000`0043fcc0 00000001`40141ad0 : 00000000`00000000 00000000`00000003
00000000`00000003 00000000`0065b2c1 : postgres!SubPostmasterMain+0x2a6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4623]
00000000`0043ff00 00000001`40380bc7 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!main+0x210
[d:\pginstaller.auto\postgres.windows-x64\src\backend\main\main.c @ 215]
00000000`0043ff30 00007ff8`22bd15cd : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!__tmainCRTStartup+0x10f
[f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 626]
00000000`0043ff60 00007ff8`23cc43d1 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0043ff90 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

00000000`0043eb28 00007ff8`213d12ee : 00000000`00000000 00000000`00000002
00000000`00000001 00000000`00000000 : ntdll!ZwWaitForMultipleObjects+0xa
00000000`0043eb30 00000001`401de68e : 00000000`00000000 00007ff5`ffffd000
00000000`00000000 00000000`04fb6560 :
KERNELBASE!WaitForMultipleObjectsEx+0xe1
00000000`0043ee10 00000001`4023cf11 : 00000000`02a55680 00000000`00000016
00000000`00000016 00000000`00676f08 : postgres!PGSemaphoreLock+0x6e
[d:\pginstaller.auto\postgres.windows-x64\src\backend\port\win32_sema.c @
145]
00000000`0043ee60 00000001`4006203b : 00000000`834333d0 00000000`00000204
00000000`00000000 00000000`00000080 : postgres!LWLockAcquireCommon+0x121
[d:\pginstaller.auto\postgres.windows-x64\src\backend\storage\lmgr\lwlock.c
@ 625]
00000000`0043eeb0 00000001`4002cf3e : 00000000`00000000 00000001`40347470
00000000`00000000 00000000`005c91f0 : postgres!XLogInsert+0x62b
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\transam\xlog.c
@ 1110]
00000000`0043f180 00000001`4002a569 : 00000000`00676a18 00000000`00000e2a
00000000`00000000 00000000`05628901 : postgres!log_heap_update+0x5ee
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
6919]
00000000`0043f370 00000001`401252bc : 00000000`05600200 00000000`05600300
00000000`05628980 00000000`00000000 : postgres!heap_update+0xb29
[d:\pginstaller.auto\postgres.windows-x64\src\backend\access\heap\heapam.c @
3768]
00000000`0043f480 00000001`401246bc : 00000000`056002e0 00000000`00000000
00000000`05601854 00000000`05601680 : postgres!ExecUpdate+0x1ac
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodemodifytable.c
@ 681]
00000000`0043f550 00000001`4010c772 : 00000000`00668b00 00000001`042e0000
00000000`0043f618 00000000`05600258 : postgres!ExecModifyTable+0x33c
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\nodemodifytable.c
@ 1029]
00000000`0043f610 00000001`4010bb6d : 00000000`05600110 00000000`00668bd0
00000000`00000000 00000000`0064c740 : postgres!ExecProcNode+0x92
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execprocnode.c
@ 377]
00000000`0043f640 00000001`401099d8 : 00000000`00650be0 00000000`005fe400
00000000`05600110 00000000`006631f0 : postgres!ExecutePlan+0x5d
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
1481]
00000000`0043f680 00000001`4024f813 : 00000000`00650be0 00000000`005fe468
00000000`005fc530 00000000`006631f0 : postgres!standard_ExecutorRun+0xa8
[d:\pginstaller.auto\postgres.windows-x64\src\backend\executor\execmain.c @
319]
00000000`0043f6f0 00000001`4024ff5a : 00000000`0043f9f0 00000000`05606140
00000000`005fe468 00000000`00000000 : postgres!ProcessQuery+0x93
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 190]
00000000`0043f740 00000001`4024f32a : 00000000`05606140 00000001`4024ef01
00000000`00000000 00000000`005fc530 : postgres!PortalRunMulti+0xfa
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 1293]
00000000`0043f7a0 00000001`4024cf83 : 00000000`00662f00 00000000`00000000
00000000`00000002 00000000`00000000 : postgres!PortalRun+0x1da
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\pquery.c @ 820]
00000000`0043f980 00000001`4024b272 : 00000000`00596f30 00000000`005fc530
00000000`0043fb70 00000000`00596f50 : postgres!exec_simple_query+0x563
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
1052]
00000000`0043fa70 00000001`401f052a : 00000001`40557a00 00000001`4041f930
00000001`0000100d 00000001`40420868 : postgres!PostgresMain+0x602
[d:\pginstaller.auto\postgres.windows-x64\src\backend\tcop\postgres.c @
4016]
00000000`0043fc60 00000001`401ed886 : 00000001`00000001 00000000`0004f1a0
00000000`1bc8cf84 00000000`00000000 : postgres!BackendRun+0x1ca
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4118]
00000000`0043fcc0 00000001`40141ad0 : 00000000`00000000 00000000`00000003
00000000`00000003 00000000`0058b2c1 : postgres!SubPostmasterMain+0x2a6
[d:\pginstaller.auto\postgres.windows-x64\src\backend\postmaster\postmaster.c
@ 4623]
00000000`0043ff00 00000001`40380bc7 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!main+0x210
[d:\pginstaller.auto\postgres.windows-x64\src\backend\main\main.c @ 215]
00000000`0043ff30 00007ff8`22bd15cd : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : postgres!__tmainCRTStartup+0x10f
[f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c @ 626]
00000000`0043ff60 00007ff8`23cc43d1 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0043ff90 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

Regards
MauMau

On 10/09/2014 09:47 PM, MauMau wrote:
>
> I heard that the user had run 16 concurrent psql sessions which executes
> INSERT and UPDATE statements, which is a write-intensive stress test.
> He encountered the hang phenomenon twice, one of which occured several
> hours after the start of the test, and the other occured about an hour
> after the test launch.

It'd be interesting and useful to run this test on a debug build of
PostgreSQL, i.e. one compiled against the debug version of the C library
and with full debuginfo not just minimal .pdb.

How were the stacks captured - what tool?

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 2014-10-09 22:47:48 +0900, MauMau wrote:
> Hello,
>
> One user reported a hang problem with 9.4 beta2 on Windows. The PostgreSQL
> is 64-bit version. I couldn't find the cause, but want to solve the
> problem. Could you help with this?
>
> I heard that the user had run 16 concurrent psql sessions which executes
> INSERT and UPDATE statements, which is a write-intensive stress test. He
> encountered the hang phenomenon twice, one of which occured several hours
> after the start of the test, and the other occured about an hour after the
> test launch.
>
> The user gave me the stack traces, which I attached at the end of this mail.
> The problem appears to be related to the xlog insert scaling. But I can't
> figure out where the root cause lies --- WAL slot handling, spinlock on
> Windows, or PGSemaphoreLock/UnLock on Windows?
>
> The place I suspect is S_UNLOCK(). It doesn't use any memory barrier. Is
> this correct on Intel64 processors?

What precisely do you mean with "Intel64"? 64bit x86 or Itanium?

Also, what's the precise workload? Can you reproduce the problem?

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

From: "Craig Ringer" <craig(at)2ndquadrant(dot)com>
> It'd be interesting and useful to run this test on a debug build of
> PostgreSQL, i.e. one compiled against the debug version of the C library
> and with full debuginfo not just minimal .pdb.

Although I'm not sure the user can do this now, I'll ask him anyway.

> How were the stacks captured - what tool?

According to his mail, Windbg or userdump.exe. I'll ask him about this.

Regards
MauMau

From: "Andres Freund" <andres(at)2ndquadrant(dot)com>
> What precisely do you mean with "Intel64"? 64bit x86 or Itanium?

64-bit x86, i.e. x86-64.

> Also, what's the precise workload? Can you reproduce the problem?

IIUC, each client inserts 1000 records into one table, then repeats updating
all those records. I'll ask him again.

Regards
MauMau

On 10/10/2014 04:16 AM, MauMau wrote:
> From: "Craig Ringer" <craig(at)2ndquadrant(dot)com>
>> It'd be interesting and useful to run this test on a debug build of
>> PostgreSQL, i.e. one compiled against the debug version of the C library
>> and with full debuginfo not just minimal .pdb.
>
> Although I'm not sure the user can do this now, I'll ask him anyway.

It sounds like they've produced a test case, so they should be able to
with a bit of luck.

Or even better, send you the test case.

>> How were the stacks captured - what tool?
>
> According to his mail, Windbg or userdump.exe. I'll ask him about this.

Thanks. The stack trace looks fairly sane, i.e. there's nothing
obviously out of whack at a glance, but I tend to get more informative
traces from Visual Studio debugging sessions.

Your next step here really needs to be to make this reproducible against
a debug build. Then see if reverting the xlog scalability work actually
changes the behaviour, given that you hypothesised that it could be
involved.

As I said off-list, if you can narrow the test case down to something
that can be reproduced more quickly, you could also git-bisect to seek
the commit at fault. Even if the test case takes an hour, that's still
viable:

$ git bisect start
$ git bisect bad
$ git bisect good REL9_3_RC1
Bisecting: a merge base must be tested
[e472b921406407794bab911c64655b8b82375196] Avoid deadlocks during
insertion into SP-GiST indexes.
$ git bisect good
Bisecting: 1026 revisions left to test after this (roughly 10 steps)
...

Thanks to the magic of binary search.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

From: "Craig Ringer" <craig(at)2ndquadrant(dot)com>
> It sounds like they've produced a test case, so they should be able to
> with a bit of luck.
>
> Or even better, send you the test case.

I asked the user about this. It sounds like the relevant test case consists
of many scripts. He explained to me that the simplified test steps are:

1. initdb
2. pg_ctl start
3. Create 16 tables. Each of those tables consist of around 10 columns.
4. Insert 1000 rows into each of those 16 tables.
5. Launch 16 psql sessions concurrently. Each session updates all 1000 rows
of one table, e.g., session 1 updates table 1, session 2 updates table 2,
and so on.
6. Repeat step 5 50 times.

This sounds a bit complicated, but I understood that the core part is 16
concurrent updates, which should lead to contention on xlog insert slots
and/or spinlocks.

> Your next step here really needs to be to make this reproducible against
> a debug build. Then see if reverting the xlog scalability work actually
> changes the behaviour, given that you hypothesised that it could be
> involved.

Thank you, but that may be labor-intensive and time-consuming. In addition,
the user uses a machine with multiple CPU cores, while I only have a desktop
PC with two CPU cores. So I doubt I can reproduce the problem on my PC.

I asked the user to change S_UNLOCK to something like the following and run
the test during this weekend (the next Monday is a national holiday in
Japan).

#define S_UNLOCK(lock) InterlockedExchange(lock, 0)

FYI, the user reported today that the problem didn't occur when he ran the
same test for 24 hours on 9.3.5. Do you see something relevant in 9.4?

Regards
MauMau

On 2014-10-10 23:08:34 +0900, MauMau wrote:
> From: "Craig Ringer" <craig(at)2ndquadrant(dot)com>
> >It sounds like they've produced a test case, so they should be able to
> >with a bit of luck.
> >
> >Or even better, send you the test case.
>
> I asked the user about this. It sounds like the relevant test case consists
> of many scripts. He explained to me that the simplified test steps are:
>
> 1. initdb
> 2. pg_ctl start
> 3. Create 16 tables. Each of those tables consist of around 10 columns.
> 4. Insert 1000 rows into each of those 16 tables.
> 5. Launch 16 psql sessions concurrently. Each session updates all 1000 rows
> of one table, e.g., session 1 updates table 1, session 2 updates table 2,
> and so on.
> 6. Repeat step 5 50 times.
>
> This sounds a bit complicated, but I understood that the core part is 16
> concurrent updates, which should lead to contention on xlog insert slots
> and/or spinlocks.

Hm. I've run similar loads on linux for long enough that I'm relatively
sure I'd have seen this.

Could you get them to print out the content's of the lwlock all these
processes are waiting for?

> >Your next step here really needs to be to make this reproducible against
> >a debug build. Then see if reverting the xlog scalability work actually
> >changes the behaviour, given that you hypothesised that it could be
> >involved.

I don't think you can trivially revert the xlog scalability stuff.

> Thank you, but that may be labor-intensive and time-consuming. In addition,
> the user uses a machine with multiple CPU cores, while I only have a desktop
> PC with two CPU cores. So I doubt I can reproduce the problem on my PC.

Well, it'll also be labor intensive for the community to debug.

> I asked the user to change S_UNLOCK to something like the following and run
> the test during this weekend (the next Monday is a national holiday in
> Japan).
>
> #define S_UNLOCK(lock) InterlockedExchange(lock, 0)

That shouldn't be required. For one, on 9.4 (not 9.5!) spinlock releases
only need to prevent reordering on the CPU level. As x86 is a TSO
architecture (total store order) that doesn't require doing anything
special. And even if it'd require more, on msvc volatile reads/stores
act as acquire/release fences unless you monkey with the compiler settings.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 10/10/2014 05:08 PM, MauMau wrote:
> From: "Craig Ringer" <craig(at)2ndquadrant(dot)com>
>> It sounds like they've produced a test case, so they should be able to
>> with a bit of luck.
>>
>> Or even better, send you the test case.
>
> I asked the user about this. It sounds like the relevant test case consists
> of many scripts. He explained to me that the simplified test steps are:
>
> 1. initdb
> 2. pg_ctl start
> 3. Create 16 tables. Each of those tables consist of around 10 columns.
> 4. Insert 1000 rows into each of those 16 tables.
> 5. Launch 16 psql sessions concurrently. Each session updates all 1000 rows
> of one table, e.g., session 1 updates table 1, session 2 updates table 2,
> and so on.
> 6. Repeat step 5 50 times.
>
> This sounds a bit complicated, but I understood that the core part is 16
> concurrent updates, which should lead to contention on xlog insert slots
> and/or spinlocks.

I was able to reproduce this. I reduced wal_buffers to 64kB, and
NUM_XLOGINSERT_LOCKS to 4 to increase the probability of the deadlock,
and ran a test case as above on my laptop for several hours, and it
finally hung. Will investigate...

- Heikki

On 10/13/2014 10:47 AM, Heikki Linnakangas wrote:
> On 10/10/2014 05:08 PM, MauMau wrote:
>> From: "Craig Ringer" <craig(at)2ndquadrant(dot)com>
>>> It sounds like they've produced a test case, so they should be able to
>>> with a bit of luck.
>>>
>>> Or even better, send you the test case.
>>
>> I asked the user about this. It sounds like the relevant test case consists
>> of many scripts. He explained to me that the simplified test steps are:
>>
>> 1. initdb
>> 2. pg_ctl start
>> 3. Create 16 tables. Each of those tables consist of around 10 columns.
>> 4. Insert 1000 rows into each of those 16 tables.
>> 5. Launch 16 psql sessions concurrently. Each session updates all 1000 rows
>> of one table, e.g., session 1 updates table 1, session 2 updates table 2,
>> and so on.
>> 6. Repeat step 5 50 times.
>>
>> This sounds a bit complicated, but I understood that the core part is 16
>> concurrent updates, which should lead to contention on xlog insert slots
>> and/or spinlocks.
>
> I was able to reproduce this. I reduced wal_buffers to 64kB, and
> NUM_XLOGINSERT_LOCKS to 4 to increase the probability of the deadlock,
> and ran a test case as above on my laptop for several hours, and it
> finally hung. Will investigate...

Ok, I tracked the bug down to the way LWLockAcquireWithVar,
LWLockRelease, and LWLockWaitForVar work. Here's a simplified model of
how this happens:

Three backends are needed to cause the deadlock. Let's call them A, B
and C. There are two locks, and one of them protects a variable, i.e. is
used with LWLockAcquireWithVar et al. The backends run these operations:

A: (checkpointer does this in xlog.c)
LWLockAcquireWithVar(lock1, value1ptr, 0xFF)
LWLockAcquire(lock2);
LWLockRelease(lock1);
LWLockRelease(lock2);

B:
LWLockAcquire(lock2, LW_EXCLUSIVE)
LWLockWaitForVar(lock1, value1ptr, 0, &newval);
LWLockRelease(lock2);

C:
LWLockAcquireWithVar(lock1, value1ptr, 0);
LWLockRelease(lock1)

So, A acquire both locks, in order lock1, lock2. B acquires lock2, and
then waits for lock1 to become free or have a non-zero value in the
variable. So A and B operate on the locks in opposite order, but this is
not supposed to deadlock, because A sets the variable to non-zero, and B
waits for it to become non-zero. Then there is a third action, C, that
just acquire lock1, with zero value.

This is the sequence that leads to the deadlock:

(both locks are free in the beginning)
C: LWLockAcquireWithVar(lock1, 0). Gets the lock.
A: LWLockAcquireWithVar(lock1, 0xFF). Blocks.
B: LWLockAcuire(lock2). Gets the lock.
B: LWLockWaitForVar(lock 1, 0). Blocks.

C: LWLockRelease(lock1). Wakes up A and B. Sets releaseOK=false because
A is waiting for the lock in exclusive mode.
C: LWLockAcquireWithVar(lock1, 0). Steals the lock back before A or B
have had a chance to run.

B: Wakes up. Observes the lock is still taken, with val 0. Adds itself
back to wait queue and goes back to sleep.
C: Releases lock 1, releaseOK is false because A has not run yet. Does
not wake up anyone.
A: Wakes up. Acquires lock 1 with val 0xFF...
A: Blocks waiting on lock 2.

So the gist of the problem is that LWLockRelease doesn't wake up
LW_WAIT_UNTIL_FREE waiters, when releaseOK == false. It should, because
a LW_WAIT_UNTIL FREE waiter is now free to run if the variable has
changed in value, and it won't steal the lock from the other backend
that's waiting to get the lock in exclusive mode, anyway.

I noticed another potential bug: LWLockAcquireCommon doesn't use a
volatile pointer when it sets the value of the protected variable:

> /* If there's a variable associated with this lock, initialize it */
> if (valptr)
> *valptr = val;
>
> /* We are done updating shared state of the lock itself. */
> SpinLockRelease(&lock->mutex);

If the compiler or CPU decides to reorder those two, so that the
variable is set after releasing the spinlock, things will break.

The attached patch should fix these two bugs. It is for REL9_4_STABLE;
needs to be forward-patched ot master too. This fixes the deadlock for
me. Anyone see any issues with this?

Thanks MauMau for the testing!

- Heikki

On Mon, Oct 13, 2014 at 10:56 AM, Heikki Linnakangas
<hlinnakangas(at)vmware(dot)com> wrote:
> I noticed another potential bug: LWLockAcquireCommon doesn't use a volatile
> pointer when it sets the value of the protected variable:
>
>> /* If there's a variable associated with this lock, initialize it
>> */
>> if (valptr)
>> *valptr = val;
>>
>> /* We are done updating shared state of the lock itself. */
>> SpinLockRelease(&lock->mutex);
>
>
> If the compiler or CPU decides to reorder those two, so that the variable is
> set after releasing the spinlock, things will break.
>
>
> The attached patch should fix these two bugs. It is for REL9_4_STABLE; needs
> to be forward-patched ot master too. This fixes the deadlock for me. Anyone
> see any issues with this?

The volatile-ization portion of this should not be forward-ported to
master; in master, spinlock operations have compiler barrier
semantics, and we've accordingly started removing volatile qualifiers
from various parts of the code, including lwlock.c.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2014-10-13 17:56:10 +0300, Heikki Linnakangas wrote:
> So the gist of the problem is that LWLockRelease doesn't wake up
> LW_WAIT_UNTIL_FREE waiters, when releaseOK == false. It should, because a
> LW_WAIT_UNTIL FREE waiter is now free to run if the variable has changed in
> value, and it won't steal the lock from the other backend that's waiting to
> get the lock in exclusive mode, anyway.

I'm not a big fan of that change. Right now we don't iterate the waiters
if releaseOK isn't set. Which is good for the normal lwlock code because
it avoids pointer indirections (of stuff likely residing on another
cpu). Wouldn't it be more sensible to reset releaseOK in *UpdateVar()? I
might just miss something here.

>
> I noticed another potential bug: LWLockAcquireCommon doesn't use a volatile
> pointer when it sets the value of the protected variable:
>
> > /* If there's a variable associated with this lock, initialize it */
> > if (valptr)
> > *valptr = val;
> >
> > /* We are done updating shared state of the lock itself. */
> > SpinLockRelease(&lock->mutex);
>
> If the compiler or CPU decides to reorder those two, so that the variable is
> set after releasing the spinlock, things will break.

Good catch. As Robert says that should be fine with master, but 9.4
obviously needs it.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 10/13/2014 06:26 PM, Andres Freund wrote:
> On 2014-10-13 17:56:10 +0300, Heikki Linnakangas wrote:
>> So the gist of the problem is that LWLockRelease doesn't wake up
>> LW_WAIT_UNTIL_FREE waiters, when releaseOK == false. It should, because a
>> LW_WAIT_UNTIL FREE waiter is now free to run if the variable has changed in
>> value, and it won't steal the lock from the other backend that's waiting to
>> get the lock in exclusive mode, anyway.
>
> I'm not a big fan of that change. Right now we don't iterate the waiters
> if releaseOK isn't set. Which is good for the normal lwlock code because
> it avoids pointer indirections (of stuff likely residing on another
> cpu).

I can't get excited about that. It's pretty rare for releaseOK to be
false, and when it's true, you iterate the waiters anyway.

> Wouldn't it be more sensible to reset releaseOK in *UpdateVar()? I
> might just miss something here.

That's not enough. There's no LWLockUpdateVar involved in the example I
gave. And LWLockUpdateVar() already wakes up all LW_WAIT_UNTIL_FREE
waiters, regardless of releaseOK.

Hmm, we could set releaseOK in LWLockWaitForVar(), though, when it
(re-)queues the backend. That would be less invasive, for sure
(attached). I like this better.

BTW, attached is a little test program I wrote to reproduce this more
easily. It exercises the LWLock* calls directly. To run, make and
install, and do "CREATE EXTENSION lwlocktest". Then launch three
backends concurrently that run "select lwlocktest(1)", "select
lwlocktest(2)" and "select lwlocktest(3)". It will deadlock within seconds.

- Heikki

On 10/13/2014 06:57 PM, Heikki Linnakangas wrote:
> Hmm, we could set releaseOK in LWLockWaitForVar(), though, when it
> (re-)queues the backend. That would be less invasive, for sure
> (attached). I like this better.

Committed this.

- Heikki

From: "Heikki Linnakangas" <hlinnakangas(at)vmware(dot)com>
> Committed this.

Thank you very much. I didn't anticipate such a difficult complicated
cause. The user agreed to try the patch tonight. I'll report back the
result as soon as I got it from him.

BTW, in LWLockWaitForVar(), the first line of the following code fragment is
not necessary, because lwWaitLink is set to head immediately. I think it
would be good to eliminate as much unnecessary code as possible from the
spinlock section.

proc->lwWaitLink = NULL;

/* waiters are added to the front of the queue */
proc->lwWaitLink = lock->head;

Regards
MauMau

From: "MauMau" <maumau307(at)gmail(dot)com>
> Thank you very much. I didn't anticipate such a difficult complicated
> cause. The user agreed to try the patch tonight. I'll report back the
> result as soon as I got it from him.

The test ran successfully without hang for 24 hours. It was run with your
patch + the following:

> BTW, in LWLockWaitForVar(), the first line of the following code fragment
> is not necessary, because lwWaitLink is set to head immediately. I think
> it would be good to eliminate as much unnecessary code as possible from
> the spinlock section.
>
> proc->lwWaitLink = NULL;
>
> /* waiters are added to the front of the queue */
> proc->lwWaitLink = lock->head;
>

Regards
MauMau

On 10/14/2014 03:59 PM, MauMau wrote:
> BTW, in LWLockWaitForVar(), the first line of the following code fragment is
> not necessary, because lwWaitLink is set to head immediately. I think it
> would be good to eliminate as much unnecessary code as possible from the
> spinlock section.
>
> proc->lwWaitLink = NULL;
>
> /* waiters are added to the front of the queue */
> proc->lwWaitLink = lock->head;

Thanks, fixed!

- Heikki