Re: bgwriter never dies

I noticed while doing some debugging this morning that if the postmaster
crashes for some reason (eg kill -9) the bgwriter process never goes
away. Backends will eventually exit when their clients quit, and the
stats collection processes shut down nicely, but the bgwriter process
has to be killed by hand. This doesn't seem like a real good thing.
Maybe there should be a provision similar to the stats collector's
check-for-read-ready-from-a-pipe?

regards, tom lane

Tom Lane wrote:
> I noticed while doing some debugging this morning that if the postmaster
> crashes for some reason (eg kill -9) the bgwriter process never goes
> away. Backends will eventually exit when their clients quit, and the
> stats collection processes shut down nicely, but the bgwriter process
> has to be killed by hand. This doesn't seem like a real good thing.
> Maybe there should be a provision similar to the stats collector's
> check-for-read-ready-from-a-pipe?

Hmmmm,

the case of the bgwriter is a bit of a twist here. In contrast to the
collectors it is connected to the shared memory. So it can keep
resources and also even worse, it could write() after the postmaster died.

Maybe there is a chance to create a watchdog for free here. Do we
currently create our own process group, with all processes under the
postmaster belonging to it? If the bgwriter would at the times it naps
check if its parent process is init, (Win32 note, check if the
postmaster does not exist any more instead), it could kill the entire
process group on behalf of the dead postmaster. This is one more system
call at a time, where the bgwriter does a system call with a timeout to
nap anyway.

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck(at)Yahoo(dot)com #

Jan Wieck <JanWieck(at)Yahoo(dot)com> writes:
> Tom Lane wrote:
>> Maybe there should be a provision similar to the stats collector's
>> check-for-read-ready-from-a-pipe?

> the case of the bgwriter is a bit of a twist here. In contrast to the
> collectors it is connected to the shared memory. So it can keep
> resources and also even worse, it could write() after the postmaster died.

That's not "worse", really. Any backends that are still alive are
committing real live transactions --- they're telling their clients
they committed, so we'd better commit. I don't mind if performance gets
worse or if we lose pg_stats statistics, but we'd better not adopt the
attitude that transaction correctness no longer matters after a
postmaster crash.

So one thing we ought to think about here is whether system correctness
depends on the bgwriter continuing to run until the last backend is
gone. AFAICS that is not true now --- the bgwriter just improves
performance --- but we'd better decide what our plan for the future is.

> Maybe there is a chance to create a watchdog for free here. Do we
> currently create our own process group, with all processes under the
> postmaster belonging to it?

We do not; I'm not sure the notion of a process group is even portable,
and I am pretty sure that the API to control process grouping isn't.

> If the bgwriter would at the times it naps
> check if its parent process is init, (Win32 note, check if the
> postmaster does not exist any more instead), it could kill the entire
> process group on behalf of the dead postmaster.

I don't think we want that. IMHO the preferred behavior if the
postmaster crashes should be like a "smart shutdown" --- you don't spawn
any more backends (obviously) but existing backends should be allowed to
run until their clients exit. That's how things have always worked
anyway...

[ thinks ... ] If we do want it we don't need any process-group
assumptions. The bgwriter is connected to shmem so it can scan the
PGPROC array and issue kill() against each sibling.

We oughta debate the desired behavior first though.

regards, tom lane

Tom Lane wrote:
> Jan Wieck <JanWieck(at)Yahoo(dot)com> writes:
>> Tom Lane wrote:
>>> Maybe there should be a provision similar to the stats collector's
>>> check-for-read-ready-from-a-pipe?
>
>> the case of the bgwriter is a bit of a twist here. In contrast to the
>> collectors it is connected to the shared memory. So it can keep
>> resources and also even worse, it could write() after the postmaster died.
>
> That's not "worse", really. Any backends that are still alive are
> committing real live transactions --- they're telling their clients
> they committed, so we'd better commit. I don't mind if performance gets
> worse or if we lose pg_stats statistics, but we'd better not adopt the
> attitude that transaction correctness no longer matters after a
> postmaster crash.
>
> So one thing we ought to think about here is whether system correctness
> depends on the bgwriter continuing to run until the last backend is
> gone. AFAICS that is not true now --- the bgwriter just improves
> performance --- but we'd better decide what our plan for the future is.
>
>> Maybe there is a chance to create a watchdog for free here. Do we
>> currently create our own process group, with all processes under the
>> postmaster belonging to it?
>
> We do not; I'm not sure the notion of a process group is even portable,
> and I am pretty sure that the API to control process grouping isn't.
>
>> If the bgwriter would at the times it naps
>> check if its parent process is init, (Win32 note, check if the
>> postmaster does not exist any more instead), it could kill the entire
>> process group on behalf of the dead postmaster.
>
> I don't think we want that. IMHO the preferred behavior if the
> postmaster crashes should be like a "smart shutdown" --- you don't spawn
> any more backends (obviously) but existing backends should be allowed to
> run until their clients exit. That's how things have always worked
> anyway...
>
> [ thinks ... ] If we do want it we don't need any process-group
> assumptions. The bgwriter is connected to shmem so it can scan the
> PGPROC array and issue kill() against each sibling.

Right. Which can change the backend behaviour from a smart shutdown to
an immediate shutdown. In the case of a postmaster crash, I think
something in the system is so wrong that I'd prefer an immediate shutdown.

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck(at)Yahoo(dot)com #

Jan Wieck <JanWieck(at)Yahoo(dot)com> writes:
> Tom Lane wrote:
>> I don't think we want that. IMHO the preferred behavior if the
>> postmaster crashes should be like a "smart shutdown" --- you don't spawn
>> any more backends (obviously) but existing backends should be allowed to
>> run until their clients exit. That's how things have always worked
>> anyway...

> ... In the case of a postmaster crash, I think
> something in the system is so wrong that I'd prefer an immediate shutdown.

Surely some other people have opinions on this? Hello out there?

regards, tom lane

> > I don't think we want that. IMHO the preferred behavior if the
> > postmaster crashes should be like a "smart shutdown" --- you don't spawn
> > any more backends (obviously) but existing backends should be allowed to
> > run until their clients exit. That's how things have always worked
> > anyway...
> >
> > [ thinks ... ] If we do want it we don't need any process-group
> > assumptions. The bgwriter is connected to shmem so it can scan the
> > PGPROC array and issue kill() against each sibling.
>
> Right. Which can change the backend behaviour from a smart shutdown to
> an immediate shutdown. In the case of a postmaster crash, I think
> something in the system is so wrong that I'd prefer an immediate shutdown.

I agree that if the postmaster dies something bad is definately happening.
However, there will be a period of time X between the postmaster dying and
the bgwriter (or another process, perhaps) discovering this. Which means
that the bug/hardware problem/condition which killed the postmaster may
affect other live backends. Hmmm. Still, if we can minimise impact then
we're probably assisting. We could always add a GUC variable ;-).

Gavin

Jan Wieck <JanWieck(at)Yahoo(dot)com> writes:
> In the case of a postmaster crash, I think something in the system
> is so wrong that I'd prefer an immediate shutdown.

I agree. Allowing existing backends to commit transactions after the
postmaster has died doesn't strike me as being that useful, and is
probably more confusing than anything else.

That said, if it takes some period of time between the death of the
postmaster and the shutdown of any backends, we *need* to ensure that
any transactions committed during that period still make it to durable
storage.

-Neil

On Tuesday 24 February 2004 23:47, Neil Conway wrote:
> Jan Wieck <JanWieck(at)Yahoo(dot)com> writes:
> > In the case of a postmaster crash, I think something in the system
> > is so wrong that I'd prefer an immediate shutdown.
>
> I agree. Allowing existing backends to commit transactions after the
> postmaster has died doesn't strike me as being that useful, and is
> probably more confusing than anything else.
>
> That said, if it takes some period of time between the death of the
> postmaster and the shutdown of any backends, we *need* to ensure that
> any transactions committed during that period still make it to durable
> storage.
>

Yes, roll back any existing/uncommited transactions and shutdown those
connections, but make sure that committed transactions are stored on disk
before exiting completly.

Robert Treat
--
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL

At 12:19 AM 26/02/2004, Robert Treat wrote:
>Yes, roll back any existing/uncommited transactions and shutdown

I'm not event sure I'd go with the rollback; whatever killed the PM may
make the rest of the system unstable. I'd prefer to see the transactions
rolled back (if necessary) as part of the log recovery on PM startup, not
by possibly dying PG proceses.

----------------------------------------------------------------
Philip Warner | __---_____
Albatross Consulting Pty. Ltd. |----/ - \
(A.B.N. 75 008 659 498) | /(@) ______---_
Tel: (+61) 0500 83 82 81 | _________ \
Fax: (+61) 03 5330 3172 | ___________ |
Http://www.rhyme.com.au | / \|
| --________--
PGP key available upon request, | /
and from pgp.mit.edu:11371 |/

Philip Warner <pjw(at)rhyme(dot)com(dot)au> writes:
> I'm not event sure I'd go with the rollback; whatever killed the PM may
> make the rest of the system unstable. I'd prefer to see the transactions
> rolled back (if necessary) as part of the log recovery on PM startup, not
> by possibly dying PG proceses.

Well, in the first place "rollback" is not an explicit action in
Postgres; you're thinking of Oracle or some other old-line technology.
There's nothing that has to happen to undo the effects of a failed
transaction.

But my real problem with the above line of reasoning is that there is
no basis for assuming that a postmaster failure has anything to do with
problems at the backend level. We have always gone out of our way to
ensure that the postmaster is disconnected from backend failure causes
--- it doesn't touch any but the simplest shared-memory datastructures,
for example. This design rule exists mostly to try to ensure that the
postmaster will survive backend crashes, but the effects cut both ways:
there is no reason that a backend won't survive a postmaster crash.
In practice, the few postmaster crashes I've seen have been due to
localized bugs in postmaster-only code or a Linux kernel randomly
seizing on the postmaster as the victim for an out-of-memory kill.
I have never seen the postmaster crash as a result of backend-level
problems, and if I did I'd be out to fix it immediately.

So my opinion is that "kill all the backends when the postmaster
crashes" is a bad idea that will only result in a net reduction in
system reliability. There is no point in building insulated independent
components if you then put in logic to force the system uptime to be the
minimum of the component uptimes.

regards, tom lane

At 04:01 PM 26/02/2004, Tom Lane wrote:
>there is no basis for assuming that a postmaster failure has
>anything to do with problems at the backend level....So my
>opinion is that "kill all the backends when the postmaster
>crashes" is a bad idea

Sounds fine. Then a system that will allow a new PM to start ASAP and serve
other connections would be great. I assume that means an orderly shutdown &
restart, but I can't see a way to make the restart work.

----------------------------------------------------------------
Philip Warner | __---_____
Albatross Consulting Pty. Ltd. |----/ - \
(A.B.N. 75 008 659 498) | /(@) ______---_
Tel: (+61) 0500 83 82 81 | _________ \
Fax: (+61) 03 5330 3172 | ___________ |
Http://www.rhyme.com.au | / \|
| --________--
PGP key available upon request, | /
and from pgp.mit.edu:11371 |/

>Tom Lane
> Jan Wieck <JanWieck(at)Yahoo(dot)com> writes:
> > Tom Lane wrote:
> >> I don't think we want that. IMHO the preferred behavior if the
> >> postmaster crashes should be like a "smart shutdown" --- you don't
> spawn
> >> any more backends (obviously) but existing backends should be
allowed
> to
> >> run until their clients exit. That's how things have always worked
> >> anyway...
>
> > ... In the case of a postmaster crash, I think
> > something in the system is so wrong that I'd prefer an immediate
> shutdown.
>
> Surely some other people have opinions on this? Hello out there?
>

I would prefer that all backends are allowed (how would you stop them?)
should carry on working on their current transaction only, then quit.
But that sounds like each backened would need to check postmaster status
at end of every transaction - yuk! Is there another way to get that
behaviour?

Your comments about least reliable component bringing rest down is
appropriate. You should assume that the backends are doing something
very important and should never be interfered with - like a very long
running transaction that is mere seconds away from committing. Besides,
this might encourage some kind of denial of service attacks... Overall,
my feeling is that a broken postmaster could be bad, but so could a
malfunctioning "fail safe" feature, so immediate shutdown wouldn't
necessarily get you out of the **** in the way that it seems it might.

If the postmaster crashes, then you might get the situation that you
have one person still connected, yet are unable to connect others. That
would be very annoying with many connected users - admittedly not much
problem if you're using external session pooling. You can't restart the
postmaster with one backend still up can you? I hope not, that sounds
bad: convince me! But if you can't its in everybody else's interests for
that last guy to stop cleanly, but earlier than their own convenience,
to allow the whole system to be restarted.

Oracle uses a PMON process to monitor for this situation: Oracle SMON
process is similar to postmaster/bg_writer. If SMON dies, PMON will
restart it. Should we have a pgmon process that watches the postmaster
and restarts it if required?

Overall, we need a clear statement of how this works, so things like the
archiver process for PITR knows when to stop/start etc. My suggestion
would be to draw out the finite state machine, so there's never a case
when we accidentally turn off archiving when there's some part of pg
still up.

Best regards, Simon Riggs

"Simon Riggs" <simon(at)2ndquadrant(dot)com> writes:
> Should we have a pgmon process that watches the postmaster
> and restarts it if required?

I doubt it; in practice the postmaster is *very* reliable (because it
doesn't do much), and so I'm not sure that adding a watchdog is going to
increase the net reliability of the system. The sorts of things that
could take out the postmaster are likely to take out a watchdog too.

regards, tom lane

On Thu, Feb 26, 2004 at 17:09:21 -0500,
Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> "Simon Riggs" <simon(at)2ndquadrant(dot)com> writes:
> > Should we have a pgmon process that watches the postmaster
> > and restarts it if required?
>
> I doubt it; in practice the postmaster is *very* reliable (because it
> doesn't do much), and so I'm not sure that adding a watchdog is going to
> increase the net reliability of the system. The sorts of things that
> could take out the postmaster are likely to take out a watchdog too.

Running postgres under daemontools is an easy way to accomplish this.
The one case it won't handle is if some process gets the process id
from the old postgres process before the new one starts up.