Re: Abnormal Exit Quandary

I'm getting this message on Postgres 7.4.1 (Mac OS 10.3)

2004-05-24 15:20:15 WARNING: terminating connection because of crash
of another server process
DETAIL: The postmaster has commanded this server process to roll back
the current transaction and exit, because another server process exited
abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and
repeat your command.

This message repeats itself for each back-end that shuts down.

This is the second time in two days I've got this message in my log.
The server has been running fine for about 5 months straight up until
now. The load on the server has increased somewhat in the last few
days, not in the number of back-end processes, but in the number of
SELECTs each back-end performs.

Can someone give me any pointers on what troubleshooting steps I can
take to get my system stable again?

Joe Lester <joe_lester(at)sweetwater(dot)com> writes:
> I'm getting this message on Postgres 7.4.1 (Mac OS 10.3)
> 2004-05-24 15:20:15 WARNING: terminating connection because of crash
> of another server process

> Can someone give me any pointers on what troubleshooting steps I can
> take to get my system stable again?

Well, for starters you could tell us about the actual crash, and not the
subsequent shutdown/restart. The messages you've quoted tell nothing at
all. What happened just before that?

regards, tom lane

Is it crashing around the time a pg_dump is executed? There is a similar problem with 7.4.1 on cygwin / windows.

There were some patches for this applied to 7.4.2 if that is the case.

On Mon, May 24, 2004 at 04:25:30PM -0500, Joe Lester wrote:
> I'm getting this message on Postgres 7.4.1 (Mac OS 10.3)
>
> 2004-05-24 15:20:15 WARNING: terminating connection because of crash
> of another server process
> DETAIL: The postmaster has commanded this server process to roll back
> the current transaction and exit, because another server process exited
> abnormally and possibly corrupted shared memory.
> HINT: In a moment you should be able to reconnect to the database and
> repeat your command.
>
> This message repeats itself for each back-end that shuts down.
>
> This is the second time in two days I've got this message in my log.
> The server has been running fine for about 5 months straight up until
> now. The load on the server has increased somewhat in the last few
> days, not in the number of back-end processes, but in the number of
> SELECTs each back-end performs.
>
> Can someone give me any pointers on what troubleshooting steps I can
> take to get my system stable again?
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo(at)postgresql(dot)org so that your
> message can get through to the mailing list cleanly

No. I dump the database every night, but this problem has happened
twice in the afternoon around 3:00 PM. There's nothing special running
at that time. I'll respond to Tom and hopefully provide some more
detail about my case. Thanks for the response.

On May 24, 2004, at 9:19 PM, Mike G wrote:

> Is it crashing around the time a pg_dump is executed? There is a
> similar problem with 7.4.1 on cygwin / windows.
>
> There were some patches for this applied to 7.4.2 if that is the case.
>
> On Mon, May 24, 2004 at 04:25:30PM -0500, Joe Lester wrote:
>> I'm getting this message on Postgres 7.4.1 (Mac OS 10.3)
>>
>> 2004-05-24 15:20:15 WARNING: terminating connection because of crash
>> of another server process
>> DETAIL: The postmaster has commanded this server process to roll back
>> the current transaction and exit, because another server process
>> exited
>> abnormally and possibly corrupted shared memory.
>> HINT: In a moment you should be able to reconnect to the database and
>> repeat your command.
>>
>> This message repeats itself for each back-end that shuts down.
>>
>> This is the second time in two days I've got this message in my log.
>> The server has been running fine for about 5 months straight up until
>> now. The load on the server has increased somewhat in the last few
>> days, not in the number of back-end processes, but in the number of
>> SELECTs each back-end performs.
>>
>> Can someone give me any pointers on what troubleshooting steps I can
>> take to get my system stable again?
>>
>>
>>
>> ---------------------------(end of
>> broadcast)---------------------------
>> TIP 3: if posting/reading through Usenet, please send an appropriate
>> subscribe-nomail command to majordomo(at)postgresql(dot)org so that your
>> message can get through to the mailing list cleanly
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo(at)postgresql(dot)org so that your
> message can get through to the mailing list cleanly
>

Thank you for pointing that out Tom. This is much more revealing. I
think I can take it from here. I'll check max_files_per_process in
postgresql.conf and I'll check my kernal's settings. Thanks!

2004-05-24 15:02:16 LOG: out of file descriptors: Too many open files
in system; release and retry
2004-05-24 15:02:16 LOG: out of file descriptors: Too many open files
in system; release and retry
2004-05-24 15:20:12 LOG: out of file descriptors: Too many open files
in system; release and retry
2004-05-24 15:20:13 PANIC: could not open control file
"/Library/PostgreSQL/data/global/pg_control": Too many open files in
system
2004-05-24 15:20:13 LOG: checkpoint process (PID 1188) was terminated
by signal 6
2004-05-24 15:20:13 LOG: terminating any other active server processes
2004-05-24 15:20:13 WARNING: terminating connection because of crash
of another server process
DETAIL: The postmaster has commanded this server process to roll back
the current transaction and exit, because another server process exited
abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and
repeat your command.

On May 24, 2004, at 8:31 PM, Tom Lane wrote:

> Joe Lester <joe_lester(at)sweetwater(dot)com> writes:
>> I'm getting this message on Postgres 7.4.1 (Mac OS 10.3)
>> 2004-05-24 15:20:15 WARNING: terminating connection because of crash
>> of another server process
>
> Well, for starters you could tell us about the actual crash, and not
> the
> subsequent shutdown/restart. The messages you've quoted tell nothing
> at
> all. What happened just before that?
>
> regards, tom lane
>