Re: Improve ODBC Throughput ?

What kinds of indexes do you have on your tables? I think the way
Access works to limit transfer it first pulls the indexes and probably
the primary one first to determine how to fetch and how many to fetch.
I suspect if you have no indexes performance would be really bad.

Also which version of psqlODBC driver are you using?

-----Original Message-----
From: pgsql-odbc-owner(at)postgresql(dot)org
[mailto:pgsql-odbc-owner(at)postgresql(dot)org] On Behalf Of Arnaud Lesauvage
Sent: Thursday, October 19, 2006 8:55 AM
To: pgsql-odbc(at)postgresql(dot)org
Subject: [ODBC] Improve ODBC Throughput ?

Hi list !

I mostly use my PostgreSQL as a backend DB with Access2000 frontends.
The tables are linked via psqlODBC.
The problem I have is that even though queries / views run very fast,
the data take a long time to arrive to the client.
In the best situations, data is sent/received at ~500kB per second (for
instance, tables with only one column, which is a very large text field
>10.000 bytes per field).
In bad situations, I am at ~50kB per second.
If the queries / views return large resultsets, it can seconds for the
table to open., even though the query ran in milliseconds on the server
(explain analyze proves this).

What can I do to improve this throughput ? Either on the server side, or
on the ODBC client side ?

As a side note : network congestion, server acitvity is not a good
explanation. This server also is a file server, and I transfer files at
10MB per second all the time.

Thanks for your advices !
--
Arnaud

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

-----------------------------------------
The substance of this message, including any attachments, may be
confidential, legally
privileged and/or exempt from disclosure pursuant to Massachusetts
law. It is intended
solely for the addressee. If you received this in error, please
contact the sender and
delete the material from any computer.

Obe, Regina a écrit :
> What kinds of indexes do you have on your tables? I think the way
> Access works to limit transfer it first pulls the indexes and probably
> the primary one first to determine how to fetch and how many to fetch.
> I suspect if you have no indexes performance would be really bad.
>
> Also which version of psqlODBC driver are you using?

I have integer primary keys on most tables, if not all of them.
Even my views have an integer primary key (at least,
something that can be used as an integer pkey in Access).

My version of psqlODBC is 8.1.2.0.

I forgot to mention something important : I obtain the best
performances (500kB / sec, cf my first post) when I am NOT
in access.
In a VBA module (Autocad, not Access), I open an ADO
connection to PostgreSQL and directly query the server (no
linked table involved here). But still, this is "only"
500kB/sec...

You say that you have a performance issue.

1. Cache Size
I would guess that the problem may be in how many records you retrieve at a
time.
In VBA code with ADO , it is sometimes helpful to use the Recordset
CacheSize property to pull more than one row at a time.

2.Declare Fetch
In Access, you do not have this fine control. Some psqlODBC user have
experimented with the DECLARE/FETCH option in the psqlODBC settngs.I am not
entirely sure how Declare/Fetch works though, I have never used it.

3. General design for performance
I don't know much about your PostgreSQL database and Access setup, but I
can imagine a situation where you are trying to open a linked table which
has millions of rows and some of the fields are of type TEXT, essentially
BLOBs that Access must treat as MEMO. Simply trying to open something like
that whole is a pretty bad idea for a transactional application. It is
usually best to filter,...retrieve the fewest rows to suit the users need,
and to do other transactions explicitly one records at a time on primary
keys.

Lastly, you could turn on the ODBC trace or the psqlODBC MyLog to looks for
less obvious issues, since you have ruled out the network.

Much Luck to you.

Greg Campbell ENG-ASE/Michelin US5
Lexington, South Carolina
803-951-5561, x75561
Fax: 803-951-5531
greg(dot)campbell(at)us(dot)michelin(dot)com

Arnaud Lesauvage
<thewild(at)freesurf
.fr> To
Sent by: "Obe, Regina"
pgsql-odbc-owner@ <robe(dot)dnd(at)cityofboston(dot)gov>
postgresql.org cc
pgsql-odbc(at)postgresql(dot)org
Subject
10/19/2006 10:24 Re: [ODBC] Improve ODBC Throughput
?

Obe, Regina a écrit :
> What kinds of indexes do you have on your tables? I think the way
> Access works to limit transfer it first pulls the indexes and probably
> the primary one first to determine how to fetch and how many to fetch.
> I suspect if you have no indexes performance would be really bad.
>
> Also which version of psqlODBC driver are you using?

I have integer primary keys on most tables, if not all of them.
Even my views have an integer primary key (at least,
something that can be used as an integer pkey in Access).

My version of psqlODBC is 8.1.2.0.

I forgot to mention something important : I obtain the best
performances (500kB / sec, cf my first post) when I am NOT
in access.
In a VBA module (Autocad, not Access), I open an ADO
connection to PostgreSQL and directly query the server (no
linked table involved here). But still, this is "only"
500kB/sec...

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo(at)postgresql(dot)org so that your
message can get through to the mailing list cleanly

> In a VBA module (Autocad, not Access), I open an ADO
> connection to PostgreSQL and directly query the server (no
> linked table involved here).

Sorry if this is slightly off topic from the thread.

I am curious, what functionality are you getting when you connect Autocad to Postgresql?

Regards,

Richard Broersma Jr.

greg(dot)campbell(at)us(dot)michelin(dot)com a écrit :
> You say that you have a performance issue.
>
> 1. Cache Size
> I would guess that the problem may be in how many records you retrieve at a
> time.
> In VBA code with ADO , it is sometimes helpful to use the Recordset
> CacheSize property to pull more than one row at a time.

I will try this !
I never used this setting, thansk for pointing this out !

> 2.Declare Fetch
> In Access, you do not have this fine control. Some psqlODBC user have
> experimented with the DECLARE/FETCH option in the psqlODBC settngs.I am not
> entirely sure how Declare/Fetch works though, I have never used it.

My tables are linked with these options :
UseDeclareFetch=1
UseServerSidePrepare=1
Fetch=10000
TrueIsMinus1=1
BoolsAsChar=0
TextAsLongVarchar=0
RowVersioning=1
RecognizeUniqueIndexes=1

> 3. General design for performance
> I don't know much about your PostgreSQL database and Access setup, but I
> can imagine a situation where you are trying to open a linked table which
> has millions of rows and some of the fields are of type TEXT, essentially
> BLOBs that Access must treat as MEMO. Simply trying to open something like
> that whole is a pretty bad idea for a transactional application. It is
> usually best to filter,...retrieve the fewest rows to suit the users need,
> and to do other transactions explicitly one records at a time on primary
> keys.

All my text fields are maximum 254 characters long (to be
recognized as varchar in Access).
I only have very few memo fields, and only when they are
compulsorey.
I think the DB design is OK in my case.

> Lastly, you could turn on the ODBC trace or the psqlODBC MyLog to looks for
> less obvious issues, since you have ruled out the network.

I'll do that as a last solution, I don't have much time to
dig into these huge logs. But I will if I don't find
anything else.

> Much Luck to you.

Thanks, I might need luck ! ;)

Richard Broersma Jr a écrit :
>> In a VBA module (Autocad, not Access), I open an ADO
>> connection to PostgreSQL and directly query the server (no
>> linked table involved here).
>
> Sorry if this is slightly off topic from the thread.
>
> I am curious, what functionality are you getting when you connect Autocad to Postgresql?

I retrieve the WKT representation of PostGIS geometries. I
wrote some very simple Autocad functions to draw them, and
some simple functions to convert Autocad entities to WKT.
Nothing like a connector, but it works quite well
Not perfect yet, I haven't found a simple way to make
multipolygons in Autocad. (any Multi obect in fact).

> >2.Declare Fetch
> >In Access, you do not have this fine control. Some psqlODBC user have
> >experimented with the DECLARE/FETCH option in the psqlODBC settngs.I am not
> >entirely sure how Declare/Fetch works though, I have never used it.
>
> My tables are linked with these options :
> UseDeclareFetch=1
> UseServerSidePrepare=1
> Fetch=10000

UseDeclareFetch and UseServerSidePrepare are disjoint options.
If I remember it the right way the UseServerSidePrepare disallow
UseDeclareFetch. Please try UseServerSidePrepare=0 when you want
using DeclareFetch.

Why you set Fetch so high? How many rows do you expect?
The UseDeclareFetch declare the select statement as cursor (using
declare statement) and then get the rows using fetch statement
(see postgresql manual).
I see no usage for UseDeclareFetch with so high Fetch.

The UseServerSidePrepare use prepare and execute statement (also
see postgresql manual).

Regards,

Luf

Ludek Finstrle a écrit :
>> >2.Declare Fetch
>> >In Access, you do not have this fine control. Some psqlODBC user have
>> >experimented with the DECLARE/FETCH option in the psqlODBC settngs.I am not
>> >entirely sure how Declare/Fetch works though, I have never used it.
>>
>> My tables are linked with these options :
>> UseDeclareFetch=1
>> UseServerSidePrepare=1
>> Fetch=10000
>
> UseDeclareFetch and UseServerSidePrepare are disjoint options.
> If I remember it the right way the UseServerSidePrepare disallow
> UseDeclareFetch. Please try UseServerSidePrepare=0 when you want
> using DeclareFetch.
>
> Why you set Fetch so high? How many rows do you expect?
> The UseDeclareFetch declare the select statement as cursor (using
> declare statement) and then get the rows using fetch statement
> (see postgresql manual).
> I see no usage for UseDeclareFetch with so high Fetch.
>
> The UseServerSidePrepare use prepare and execute statement (also
> see postgresql manual).

I thought that since memory consumption was not an issue, I
could fetch a lot of rows at once. I read the ODBC
documentation about that but not PostgreSQL's, but I'll
correct this ASAP.

What is your advice in fact ?

UseDeclareFetch=0
UseServerSidePrepare=1

Or maybe

UseDeclareFetch=1
UseServerSidePrepare=0
Fetch=100

I might just try both in fact (don't have the time right
now, maybe later today though).

Hello,

> >UseDeclareFetch and UseServerSidePrepare are disjoint options.
> >If I remember it the right way the UseServerSidePrepare disallow
> >UseDeclareFetch. Please try UseServerSidePrepare=0 when you want
> >using DeclareFetch.
> >
> >Why you set Fetch so high? How many rows do you expect?
> >The UseDeclareFetch declare the select statement as cursor (using
> >declare statement) and then get the rows using fetch statement
> >(see postgresql manual).
> >I see no usage for UseDeclareFetch with so high Fetch.
> >
> >The UseServerSidePrepare use prepare and execute statement (also
> >see postgresql manual).
>
> I thought that since memory consumption was not an issue, I
> could fetch a lot of rows at once. I read the ODBC

Yes, it could. When you have UseDeclareFetch=0 it get all rows at once.
But sometimes it take too long when you need few of them ASAP and
others later (so it's good time to use UseDeclareFetch=1).

> documentation about that but not PostgreSQL's, but I'll
> correct this ASAP.

That's no problem. I only point you to the documentation to fully
understand what psqlODBC makes for you internally ;o) It's always
the good start point when you need some tuning.
BTW the start reference point is:
http://www.postgresql.org/docs/8.1/interactive/sql-commands.html

> What is your advice in fact ?

I don't know what you really want. Sometimes is better UseDeclareFetch=1,
sometimes is better UseServerSidePrepare=1 and sometimes is better
UseDeclareFetch=0 and UseServerSidePrepare=0.

I don't know how internally works Access or CAD so I have no advice.

Regards,

Luf

Arnaud Lesauvage wrote:
> Ludek Finstrle a écrit :
>>
>> UseDeclareFetch and UseServerSidePrepare are disjoint options.
>> If I remember it the right way the UseServerSidePrepare disallow
>> UseDeclareFetch. Please try UseServerSidePrepare=0 when you want
>> using DeclareFetch.

It's not true.
Though the driver doesn't use PREPARE functionality for SELECT statements
under UseDeclareFetch mode, it uses PREPARE functionality for other DML
commands.

regards,
Hiroshi Inoue