Re: 9.1.3 backends getting stuck in 'startup'

On 04/27/12 12:27, Tom Lane wrote:
> Jeff Frost <jeff(at)pgexperts(dot)com> writes:
>> Any idea what I should be looking for in the backtraces?
>> I would imagine I can ignore any that are in InitPostgres, but that still
>> leaves quite a few to look through.
> I think you can probably skip all that are blocked in LWLockAcquire
> called from bufmgr.c:531, at least for a first pass. Calls from
> elsewhere in bufmgr.c might be more interesting, and anything that's not
> blocked at an LWLockAcquire at all might be even more interesting.
>
> regards, tom lane

Here we go. One that's not blocked on acquiring the lock, but does have
bufmgr.c in it:

#0 0x00007f252f601a50 in __read_nocancel () at
../sysdeps/unix/syscall-template.S:82
#1 0x00007f253163bced in read (file=<value optimized out>,
buffer=0x7f251f3a1fa0 ":\n", amount=<value optimized out>) at
/usr/include/bits/unistd.h:45
#2 FileRead (file=<value optimized out>, buffer=0x7f251f3a1fa0 ":\n",
amount=<value optimized out>) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/storage/file/fd.c:1191
#3 0x00007f25316547a6 in mdread (reln=<value optimized out>, forknum=<value
optimized out>, blocknum=5825, buffer=0x7f251f3a1fa0 ":\n") at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/storage/smgr/md.c:640
#4 0x00007f25316387f1 in ReadBuffer_common (smgr=0x7f2532d38310,
relpersistence=<value optimized out>, forkNum=<value optimized out>,
blockNum=<value optimized out>, mode=<value optimized out>, strategy=<value
optimized out>, hit=0x7fffea5fb47f "") at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/storage/buffer/bufmgr.c:438
#5 0x00007f25316391e4 in ReadBufferExtended (reln=0x7f2532cf6990,
forkNum=MAIN_FORKNUM, blockNum=5825, mode=RBM_NORMAL, strategy=0x4ff74188) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/storage/buffer/bufmgr.c:246
#6 0x00007f25314817bf in heapgetpage (scan=0x7f2532d335d0, page=5825) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/access/heap/heapam.c:223
#7 0x00007f25314821ba in heapgettup (scan=0x7f2532d335d0, dir=<value
optimized out>, nkeys=<value optimized out>, key=<value optimized out>) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/access/heap/heapam.c:556
#8 0x00007f25314827bc in heap_getnext (scan=0x7f2532d335d0,
direction=523902880) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/access/heap/heapam.c:1345
#9 0x00007f2531711570 in RelationBuildTupleDesc (targetRelId=<value optimized
out>, insertIt=<value optimized out>) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/utils/cache/relcache.c:468
#10 RelationBuildDesc (targetRelId=<value optimized out>, insertIt=<value
optimized out>) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/utils/cache/relcache.c:882
#11 0x00007f2531712f26 in RelationIdGetRelation (relationId=2616) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/utils/cache/relcache.c:1568
#12 0x00007f2531485aab in relation_open (relationId=2616, lockmode=523902880)
at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/access/heap/heapam.c:907
#13 0x00007f2531485b33 in heap_open (relationId=6, lockmode=523902880) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/access/heap/heapam.c:1077
#14 0x00007f25317106d7 in LookupOpclassInfo (operatorClassOid=10035,
numSupport=1) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/utils/cache/relcache.c:1278
#15 0x00007f2531710c48 in IndexSupportInitialize (relation=<value optimized
out>) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/utils/cache/relcache.c:1159
#16 RelationInitIndexAccessInfo (relation=<value optimized out>) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/utils/cache/relcache.c:1102
#17 0x00007f2531711d91 in RelationBuildDesc (targetRelId=<value optimized
out>, insertIt=<value optimized out>) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/utils/cache/relcache.c:904
#18 0x00007f253171201d in load_critical_index (indexoid=2693, heapoid=2618) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/utils/cache/relcache.c:3080
#19 0x00007f25317124d2 in RelationCacheInitializePhase3 () at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/utils/cache/relcache.c:2892
#20 0x00007f25317293e9 in InitPostgres (in_dbname=<value optimized out>,
dboid=<value optimized out>, username=<value optimized out>, out_dbname=<value
optimized out>) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/utils/init/postinit.c:816
#21 0x00007f253165a908 in PostgresMain (argc=828737552, argv=<value optimized
out>, username=0x7f2532b90820 "monitoring") at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/tcop/postgres.c:3650
#22 0x00007f2531619ad3 in BackendRun () at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/postmaster/postmaster.c:3606
#23 BackendStartup () at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/postmaster/postmaster.c:3291
#24 ServerLoop () at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/postmaster/postmaster.c:1455
#25 0x00007f253161c57c in PostmasterMain (argc=851231024, argv=0x7f2532b8c8f0)
at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/postmaster/postmaster.c:1116
#26 0x00007f25315baec3 in main (argc=3, argv=0x7f2532b8c8e0) at
/build/buildd/postgresql-9.1-9.1.3/build/../src/backend/main/main.c:199

--
Jeff Frost <jeff(at)pgexperts(dot)com>
CTO, PostgreSQL Experts, Inc.
Phone: 1-888-PG-EXPRT x506
FAX: 415-762-5122
http://www.pgexperts.com/