Re: psql: FATAL: the database system is in recovery mode

Hi All,

I am doing some database related queries and this is working fine at fedora
core 4.
But the same database queries giving the FATAL error on fedora 9.

If I restarts the database on fedora core 9 then this is perfectlry working
without giving any error.

My postgres version is
On Fedora core-9[FC9] Machine:
$ rpm -qa|grep postgres
postgresql-server-8.3.1-1.fc9.i386
postgresql-devel-8.3.1-1.fc9.i386
postgresql-python-8.3.1-1.fc9.i386
postgresql-8.3.1-1.fc9.i386
postgresql-libs-8.3.1-1.fc9.i386

On Fedora core-4[FC49] Machine:
$ rpm -qa|grep postgres
postgresql-server-8.0.3-1
postgresql-libs-8.0.3-1
postgresql-odbc-08.00.0100-1
postgresql-jdbc-8.0.3-1
postgresql-docs-8.0.3-1
postgresql-tcl-8.0.3-1
postgresql-test-8.0.3-1
postgresql-devel-8.0.3-1
postgresql-8.0.3-1
gnucash-backend-postgres-1.8.11-3
freeradius-postgresql-1.0.2-2
postgresql-python-8.0.3-1
postgresql-contrib-8.0.3-1
postgresql-pl-8.0.3-1

Problem is:
----------------
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

connection to server was lost

psql: FATAL: the database system is in recovery mode

psql: FATAL: the database system is in recovery mode

psql: FATAL: the database system is in recovery mode

psql: FATAL: the database system is in recovery mode

psql: FATAL: the database system is in recovery mode

psql: FATAL: the database system is in recovery mode

psql: FATAL: the database system is in recovery mode

psql: FATAL: the database system is in recovery mode

psql: FATAL: the database system is in recovery mode

psql: FATAL: the database system is in recovery mode

psql: FATAL: the database system is in recovery mode

server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
connection to server was lost
psql: FATAL: the database system is in recovery mode

--------------

Is this database bug or there is any versioning incopatability.
Can anyone please help me in this regards.

--
With Regards,
Bhushan

Bhushan Verma wrote:
> I am doing some database related queries and this is working fine at fedora
> core 4.
> But the same database queries giving the FATAL error on fedora 9.
>
> If I restarts the database on fedora core 9 then this is perfectlry working
> without giving any error.
>
> My postgres version is
> On Fedora core-9[FC9] Machine:
> $ rpm -qa|grep postgres
> postgresql-server-8.3.1-1.fc9.i386
> postgresql-devel-8.3.1-1.fc9.i386
> postgresql-python-8.3.1-1.fc9.i386
> postgresql-8.3.1-1.fc9.i386
> postgresql-libs-8.3.1-1.fc9.i386

It probably won't solve this problem, but you need to upgrade. The
latest 8.3 release is 8.3.7. See
http://www.postgresql.org/support/versioning

> On Fedora core-4[FC49] Machine:
> $ rpm -qa|grep postgres
> postgresql-server-8.0.3-1
> ...

Same here. Latest 8.0 minor release is 8.0.21

> Problem is:
> ----------------
> server closed the connection unexpectedly
> This probably means the server terminated abnormally
> before or while processing the request.
>
> connection to server was lost
>
> psql: FATAL: the database system is in recovery mode
> ...
>
> server closed the connection unexpectedly
> This probably means the server terminated abnormally
> before or while processing the request.
> connection to server was lost
> psql: FATAL: the database system is in recovery mode
>
> --------------
>
> Is this database bug or there is any versioning incopatability.

You'll have to give more details or no-one will be able to help you.
Please share the query that caused the crash, and CREATE statements of
all the tables involved in the query. Is there any triggers or anything
else involved?

Can you get a core dump and post stack trace from it? Something along
the lines of:
1. ulimit -c unlimited
2. pg_ctl start
3. <induce crash>
4. gdb /usr/bin/postgres <datadir>/core
5. bt

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Bhushan Verma wrote:

> server closed the connection unexpectedly
> This probably means the server terminated abnormally
> before or while processing the request.
> connection to server was lost

Look in the PostgreSQL server logs for more information. It's probably
in /var/log/postgresql or something like that - it depends on exactly
how the distro packaged PostgreSQL.

--
Craig Ringer

Thanks for your response.
As follows is log details

ERROR: table "transactionrecord_5917" does not exist
STATEMENT: drop table TransactionRecord_5917;
ERROR: table "pagerecord_5917" does not exist
STATEMENT: drop table PageRecord_5917;
ERROR: table "urlrecord_5917" does not exist
STATEMENT: drop table URLRecord_5917;
LOG: server process (PID 29225) was terminated by signal 11: Segmentation
fault
LOG: terminating any other active server processes
LOG: all server processes terminated; reinitializing
FATAL: the database system is in recovery mode
LOG: database system was interrupted; last known up at 2009-06-23 12:49:49
IST
FATAL: the database system is in recovery mode
FATAL: the database system is in recovery mode
LOG: database system was not properly shut down; automatic recovery in
progress
FATAL: the database system is in recovery mode
LOG: redo starts at 0/185239F8
FATAL: the database system is in recovery mode
FATAL: the database system is in recovery mode
FATAL: the database system is in recovery mode
LOG: record with zero length at 0/185A67FC
LOG: redo done at 0/185A67D0
LOG: last completed transaction was at log time 2009-06-23
14:49:10.37602+05:30
FATAL: the database system is in recovery mode
LOG: autovacuum launcher started
LOG: database system is ready to accept connections

With Regards,
Bhushan
On 6/24/09, Craig Ringer <craig(at)postnewspapers(dot)com(dot)au> wrote:
>
> Bhushan Verma wrote:
>
> > server closed the connection unexpectedly
> > This probably means the server terminated abnormally
> > before or while processing the request.
> > connection to server was lost
>
>
> Look in the PostgreSQL server logs for more information. It's probably
> in /var/log/postgresql or something like that - it depends on exactly
> how the distro packaged PostgreSQL.
>
> --
>
> Craig Ringer
>

--
With Regards,
Bhushan

Hi,

Thanks for your response.

I am not able to find out the core file.
but in log file its giving the message like:

LOG: server process (PID 29225) was terminated by signal 11: Segmentation
fault
LOG: terminating any other active server processes
LOG: all server processes terminated; reinitializing
FATAL: the database system is in recovery mode
LOG: database system was interrupted; last known up at 2009-06-23 12:49:49
IST
FATAL: the database system is in recovery mode
FATAL: the database system is in recovery mode

I have searched for the core file for this pid.
Is there any spcefic path for the postgres executable?

I have already checked for ulimit -c unlimited etc.
On my system core file for some other application are generating properly.

With Regards,
Bhushan

On 6/24/09, Heikki Linnakangas <heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
>
> Bhushan Verma wrote:
> > I am doing some database related queries and this is working fine at
> fedora
> > core 4.
> > But the same database queries giving the FATAL error on fedora 9.
> >
> > If I restarts the database on fedora core 9 then this is perfectlry
> working
> > without giving any error.
> >
> > My postgres version is
> > On Fedora core-9[FC9] Machine:
> > $ rpm -qa|grep postgres
> > postgresql-server-8.3.1-1.fc9.i386
> > postgresql-devel-8.3.1-1.fc9.i386
> > postgresql-python-8.3.1-1.fc9.i386
> > postgresql-8.3.1-1.fc9.i386
> > postgresql-libs-8.3.1-1.fc9.i386
>
>
> It probably won't solve this problem, but you need to upgrade. The
> latest 8.3 release is 8.3.7. See
> http://www.postgresql.org/support/versioning
>
>
> > On Fedora core-4[FC49] Machine:
> > $ rpm -qa|grep postgres
> > postgresql-server-8.0.3-1
>
> > ...
>
> Same here. Latest 8.0 minor release is 8.0.21
>
>
> > Problem is:
> > ----------------
> > server closed the connection unexpectedly
> > This probably means the server terminated abnormally
> > before or while processing the request.
> >
> > connection to server was lost
> >
> > psql: FATAL: the database system is in recovery mode
>
> > ...
>
> >
> > server closed the connection unexpectedly
> > This probably means the server terminated abnormally
> > before or while processing the request.
> > connection to server was lost
> > psql: FATAL: the database system is in recovery mode
> >
> > --------------
> >
> > Is this database bug or there is any versioning incopatability.
>
>
> You'll have to give more details or no-one will be able to help you.
> Please share the query that caused the crash, and CREATE statements of
> all the tables involved in the query. Is there any triggers or anything
> else involved?
>
> Can you get a core dump and post stack trace from it? Something along
> the lines of:
> 1. ulimit -c unlimited
> 2. pg_ctl start
> 3. <induce crash>
> 4. gdb /usr/bin/postgres <datadir>/core
> 5. bt
>
>
> --
> Heikki Linnakangas
> EnterpriseDB http://www.enterprisedb.com
>

--
With Regards,
Bhushan

Bhushan Verma wrote:
> I have searched for the core file for this pid.
> Is there any spcefic path for the postgres executable?
>
> I have already checked for ulimit -c unlimited etc.
> On my system core file for some other application are generating properly.

The core file is generated in the data directory. I believe the default
location on Fedora is /var/lib/pgsql/data

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

>The core file is generated in the data directory. I believe the default
>location on Fedora is /var/lib/pgsql/dat

Yes I also think the same.
But there is no core file in /var/lib/pgsql/dat on my system.
I am trying to find out why this is not generating core while log message
"LOG: server process (PID 14255) was terminated by signal 11: Segmentation
fault" is wriiten in the log file.

With Regards,
Bhushan

On 6/24/09, Heikki Linnakangas <heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
>
> Bhushan Verma wrote:
> > I have searched for the core file for this pid.
> > Is there any spcefic path for the postgres executable?
> >
> > I have already checked for ulimit -c unlimited etc.
> > On my system core file for some other application are generating
> properly.
>
>
> The core file is generated in the data directory. I believe the default
> location on Fedora is /var/lib/pgsql/data
>
>
> --
>
> Heikki Linnakangas
> EnterpriseDB http://www.enterprisedb.com
>

--
With Regards,
Bhushan

Bhushan Verma wrote:
>> The core file is generated in the data directory. I believe the default
>> location on Fedora is /var/lib/pgsql/dat
>
> Yes I also think the same.
> But there is no core file in /var/lib/pgsql/dat on my system.
> I am trying to find out why this is not generating core while log message
> "LOG: server process (PID 14255) was terminated by signal 11: Segmentation
> fault" is wriiten in the log file.

Perhaps the ulimit is not in effect after all? Try starting postmaster
directly, without pg_ctl:

ulimit -c unlimited
postmaster -D /var/lib/pgsql/data

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Bhushan Verma wrote:
>>> postmaster -D /var/lib/pgsql/data
> I am using the same command as you said.

I'm afraid I'm out of ideas on how to get the core dump then.

You could also try attaching gdb to the backend process before it
crashes, and get the backtrace from there. Something along the lines of:

postmaster -D /var/lib/pgsql/data
psql postgres ...
ps ax | grep postgres # check the PID of the backend process psql is
connected to.
gdb
gdb> attach <pid of backend>
gdb> cont
<run the query in psql that crashes>
gdb> bt

You still haven't posted the offending query, BTW. Is it a particular
one, or does it crash at random?

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Hi

Thanks one again for your response.

>>You still haven't posted the offending query,
I am running on shell script that contains various select statment,
one of the example is as follows;
----------
SELECT PageIndex,
(SUM(ConnectDoneTime - StartTime) * 100) / SUM(EndTime - StartTime)
AS "ConnectTimePct",
(SUM(WriteCompleTime - ConnectDoneTime) * 100) / SUM(EndTime -
StartTime) AS "RequestSentTimePct",
(SUM(FirstByteRcdTime - WriteCompleTime) * 100) / SUM(EndTime -
StartTime) AS "FirstByteTimePct",
(SUM(EndTime - FirstByteRcdTime) * 100) / SUM(EndTime - StartTime) AS
"DownloadTimePct"
FROM UrlRecord_$1
WHERE EndTime <> 0
AND ConnectDoneTime <> 0
AND WriteCompleTime <> 0
AND FirstByteRcdTime <> 0
GROUP By PageIndex
ORDER By PageIndex;

-----------

I want to tell you that its happening after sometime ie at random. time is
not fixed.

>>Is it a particular one, or does it crash at random?

its crash at random.

On 6/24/09, Heikki Linnakangas <heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
>
> Bhushan Verma wrote:
> >>> postmaster -D /var/lib/pgsql/data
> > I am using the same command as you said.
>
>
> I'm afraid I'm out of ideas on how to get the core dump then.
>
> You could also try attaching gdb to the backend process before it
> crashes, and get the backtrace from there. Something along the lines of:
>
>
> postmaster -D /var/lib/pgsql/data
>
> psql postgres ...
> ps ax | grep postgres # check the PID of the backend process psql is
> connected to.
> gdb
> gdb> attach <pid of backend>
> gdb> cont
> <run the query in psql that crashes>
> gdb> bt
>
>
> You still haven't posted the offending query, BTW. Is it a particular
> one, or does it crash at random?
>
>
> --
>
> Heikki Linnakangas
> EnterpriseDB http://www.enterprisedb.com
>

--
With Regards,
Bhushan

Bhushan Verma wrote:

>>> Is it a particular one, or does it crash at random?
>
> its crash at random.

Random segfaults easily by application bugs ( memory corruption,
accessing uninitialized memory and dereferencing pointers there, etc )
or hardware issues like bad CPU/CPU cache/memory, overheating, etc.

Are you having issues with any other applications?

If you can afford the load, consider running something quite CPU/memory
intensive for a while - say, a big compile job - and make sure it all
goes smoothly.

It'd also be interesting to know whether a separate PostgreSQL instance
with a fresh database had issues, but that's probably not practical for
you to test.

Most likely it's some kind of corrupt database triggering a subtle bug
somewhere in Pg, anyway...

--
Craig Ringer