performance issues with DBI module when data too big

From: "Nicolas Nolst" <nnolst(at)hotmail(dot)com>
To: pgsql-admin(at)postgresql(dot)org, pgsql-general(at)postgresql(dot)org
Subject: performance issues with DBI module when data too big
Date: 2002-06-03 16:56:59
Message-ID: F141dJRh2GcClDcNMAP00011eb3@hotmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-admin pgsql-general

<html><div style='background-color:'><DIV>
<DIV>
<DIV></DIV>
<DIV></DIV>
<P>Hi all,</P>
<P>I have developped a perl script to populate a database with two tables: sessions and actions.</P>
<P>the table actions contains the following columns: session_id, url, timestamp. The column session_id references to the table sessions.&nbsp;</P>
<P>the table sessions contains the following columns: session_id, remote_ip, phone_type, phone_number. The column session_id is serial.</P>
<P>The lines of the table actions which are part of the same session have the same session_id.</P>
<P>There are then more lines in the table actions than in the table session.</P>
<P><BR>To fill the two tables, I first need to know if the session already exists for a certain phone_type, a certain remote_ip and a certain phone_number:</P>
<P>SELECT&nbsp;session_id&nbsp;FROM sessions&nbsp;WHERE (phone_number = ?)&nbsp;AND (remote_ip&nbsp; = ?)&nbsp;AND (phone_type = ?)&nbsp;ORDER BY&nbsp;session_id;</P>
<P>I also need to apply a criteria to&nbsp;know if I have to&nbsp;add a new entry in the table sessions or not:</P>
<P>SELECT&nbsp;(max(timestamp) + ?)&lt;?&nbsp;FROM actions&nbsp;WHERE (session_id = ?); &nbsp; </P>
<P>&nbsp;</P>
<P>If the session already exists I add a line in the table actions with a INSERT</P>
<P>If the session doesn't exist or if the criteria is true, I add a line in the table sessions with an INSERT, retrieve the session_id of the line just added with the following request</P>
<P>SELECT&nbsp;session_id&nbsp;FROM sessions where (msisdn=?)&nbsp;AND (remote_ip=?)&nbsp;AND (user_agent=?)&nbsp;ORDER&nbsp;BY session_id&nbsp;DESC&nbsp;LIMIT 1</P>
<P>and the add with a INSERT a line in the table actions.</P>
<P>I have put indexes on sessions(session_id), sessions(msisdn), actions(session_id).</P>
<P>My problem is that populating my database is slower&nbsp; when the data gets bigger and the performance falls dramatically. I thought that is would be improve with my indexes but the problem still persists.</P>
<P>Could you please give me some clues that could solve this issue.</P>
<P>Thanks.</P>
<P><BR>&nbsp;<BR><BR>Nicolas Nolst </P>
<DIV></DIV>
<DIV></DIV><IMG src="http://www.3dflags/World/Gif/belgium_gs.gif">
<DIV></DIV></DIV></DIV></div><br clear=all><hr>Join the worlds largest e-mail service with MSN Hotmail. <a href='http://g.msn.com/1HM300901/158'>Click Here</a><br></html>

Attachment Content-Type Size
unknown_filename text/html 2.3 KB

Responses

Browse pgsql-admin by date

  From Date Subject
Next Message Jan Wieck 2002-06-03 17:50:17 Re: [GENERAL] performance issues with DBI module when data too big
Previous Message Henry House 2002-06-03 16:40:44 Re: drop table failed to drop associated sequence

Browse pgsql-general by date

  From Date Subject
Next Message Jan Wieck 2002-06-03 17:17:21 Re: tid scan - is it ever used?
Previous Message Jan Wieck 2002-06-03 16:25:54 Re: What popular, large commercial websites run