Re: Web + Slicing/Paging datas

Hi!

In a mod_py application I wanna write a wrapper that handle all PSQL
data view with paging/slicing.

For example:
I have 1.500 records. I wanna show only N (f. ex: 15) records in the
view, other records are accessable with a pager (links):

[First, P-2, P-1, P, P+1 P+2, Last]
F. Ex: First, 5, 6, {7}, 8, 9, Last

Ok, I can realize this with count, and next to define the select's
start/and limit parameters.

But I heard about the count(*) is slow in PG.

This paging is a typical problem: I need to paid two times for the datas.
First time I get all data, but I only count them. Next time I get only
the slice of records I need.

As I saw, some systems with less data do this:

1.)
Inserts all records to a temp table.
Check the affected rows (as count).
Slicing the records.
Fetch the slice records.
Destroy temp table.

2.)
Select all records.
Fetching all records.
Dropping all not needed elements.
Return needed records.
Close cursor.

Every solution is slow, the 1.) is because of storing the records (bulk
insert) 2.) is because of fetching not needed records (network speed).

So I wanna ask: what are you doing if you wanna use paging/slicing of
records?
The first (count/slicing) solution is enough fast for you?

Thanks for your help:
dd

durumdara wrote:
> Hi!
>
> In a mod_py application I wanna write a wrapper that handle all PSQL
> data view with paging/slicing.
>
> ..
> Inserts all records to a temp table.
> Check the affected rows (as count).
> Slicing the records.
> Fetch the slice records.
> Destroy temp table.

how about KEEPING the data in this slice format, and keeping the count
somewhere ? you'd need to update the count and slice info when new data
gets added, but perhaps thats better than doing it every time you want
to view a slice?

Hi!

2009.04.23. 9:23 keltezéssel, John R Pierce írta:
> durumdara wrote:
>> Hi!
>>
>> In a mod_py application I wanna write a wrapper that handle all PSQL
>> data view with paging/slicing.
>>
>> ..
>> Inserts all records to a temp table.
>> Check the affected rows (as count).
>> Slicing the records.
>> Fetch the slice records.
>> Destroy temp table.
>
>
> how about KEEPING the data in this slice format, and keeping the count
> somewhere ? you'd need to update the count and slice info when new
> data gets added, but perhaps thats better than doing it every time you
> want to view a slice?
>
>
This is a complicated thing.
These datas are provided by a query, with some user's conditions.
For example:
Text = Free String Search
Live = This data is living
Category = Some category
SubCategory = Some subcategory
...
etc.

So your way is possible working with this pseudo code:

def Slicing(PageIndex, Conditions):
# Del recs that have older than 30 minutes
delsql = "delete from pagingtable where inserted < %s" %
IncWithMinutes(Now, -30)
ExecSQL(delsql)
# The Query
datasql = BuildSQL(Conditions)
# Have same query in the pool?
checksql = "select count from pagingtable where sql='%s'" %
SafeEscape(datasql)
records = ExecSQL(checksql)
if records:
# Yes
count = records[0]['COUNT']
else:
# No, we must run a count sql to check
countsql = BuildSQL(Conditions, Count = 1)
datarecords = ExecSQL(countsql)
datarecord = datarecords[0]
count = datarecord['COUNT']
# We must insert it to the paging table
InsertPagingRecord(datasql, count)
...
# Use the count
...
finalsql = BuildSQL(Conditions, WithPaging = PageIndex)
finalrecords = ExecSQL(finalsql)
...

But possible it is too complex and I fear that it cause more pain than I
winning in it... (deadlocks?)...

Thanks for your help:
dd

On 2009-04-23, durumdara <durumdara(at)gmail(dot)com> wrote:
> Hi!
>
> In a mod_py application I wanna write a wrapper that handle all PSQL
> data view with paging/slicing.
>
> For example:
> I have 1.500 records. I wanna show only N (f. ex: 15) records in the
> view, other records are accessable with a pager (links):

For small numbers of records you can use

SELECT ...
... LIMIT how_many OFFSET where_to_start

If there are a large number of results this may be inefficient.