Re: Is there any method to keep table in memory at startup

Lists: pgsql-hackers
From: vinayj(at)ncst(dot)ernet(dot)in
To: pgsql-hackers(at)postgresql(dot)org
Subject: Is there any method to keep table in memory at startup
Date: 2004-04-21 04:13:33
Message-ID: 1082520813.4085f4ed6b6ff@www.ncst.ernet.in
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Hi
I am working on a project in postgres..in which i designed customized data type
and operations on it.it requires a look up table..
I have three options regarding this table...
1. Every time a query is executed it creates table assigns values and after
execution destroys it...which is overhead..

2. store table on disk in database and access it whenever required but it
degrades the performance

3. whenever psql starts it can load the table in memory from database which is
efficient way to do

but i don't know how to load table in memory when psql starts up ...
please guide me..
thanks in advance
regards
vinay

---------------------------------------------------------------
This mail is sent through IMP: http://horde.org/imp/
Used as the Webmail Interface at C-DAC, Mumbai: http://www.ncst.ernet.in


From: "scott(dot)marlowe" <scott(dot)marlowe(at)ihs(dot)com>
To: <vinayj(at)ncst(dot)ernet(dot)in>
Cc: <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Is there any method to keep table in memory at startup
Date: 2004-04-27 20:52:03
Message-ID: Pine.LNX.4.33.0404271450380.6234-100000@css120.ihs.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Wed, 21 Apr 2004 vinayj(at)ncst(dot)ernet(dot)in wrote:

> Hi
> I am working on a project in postgres..in which i designed customized data type
> and operations on it.it requires a look up table..
> I have three options regarding this table...
> 1. Every time a query is executed it creates table assigns values and after
> execution destroys it...which is overhead..
>
> 2. store table on disk in database and access it whenever required but it
> degrades the performance
>
> 3. whenever psql starts it can load the table in memory from database which is
> efficient way to do

PostgreSQL has no facility to put tables into memory.

Assuming this lookup table will be hit quite often, it WILL be in memory
for selects. updates / deletes / inserts will have to get flushed out to
disk of course.

the Linux and BSD kernels are both quite good at keeping commonly used
data in memory. I think you are mistaken in assuming that an "on disk"
table will be significantly slower than if it was fixed in memory due to
the very efficient cachine of the most common unix kernels.


From: Vinay Jain <vinayj(at)sarathi(dot)ncst(dot)ernet(dot)in>
To: "scott(dot)marlowe" <scott(dot)marlowe(at)ihs(dot)com>
Cc: vinayj(at)ncst(dot)ernet(dot)in, pgsql-hackers(at)postgresql(dot)org
Subject: Re: Is there any method to keep table in memory at startup
Date: 2004-04-28 03:53:46
Message-ID: 408F2ACA.6060306@sarathi.ncst.ernet.in
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Hi
thank you for such a useful information...
but actually in my case if i keep table in disk it significantly
degrades performance and even for a table of 10 rows it takes 1-2
minutes I think u r not beliving it ! am i right
for example
I create a table in which i use my customized data type say student
create table student
(Name INDCHAR //INDCHAR is customized data type
age integer);
now i give query like this
select * from student order by name;
it will search for it's comparator operator (<) and related function...
in that function there is one lookup table if that table is in memory no
problem! (oh but it can't be) if it is in disk my program makes
connection to database and execute query which is just a select
statement on a simple where condition of equality. then closes connection
so every time less than operator(<) is called it does the same task..
what i feel in table of 10 rows how many times the < operator will be
called(NO idea but must be > 10 times)
is there any solution..
thanks in advance
regards
vinay
scott.marlowe wrote:

>On Wed, 21 Apr 2004 vinayj(at)ncst(dot)ernet(dot)in wrote:
>
>
>
>>Hi
>>I am working on a project in postgres..in which i designed customized data type
>>and operations on it.it requires a look up table..
>>I have three options regarding this table...
>>1. Every time a query is executed it creates table assigns values and after
>>execution destroys it...which is overhead..
>>
>>2. store table on disk in database and access it whenever required but it
>>degrades the performance
>>
>>3. whenever psql starts it can load the table in memory from database which is
>>efficient way to do
>>
>>
>
>PostgreSQL has no facility to put tables into memory.
>
>Assuming this lookup table will be hit quite often, it WILL be in memory
>for selects. updates / deletes / inserts will have to get flushed out to
>disk of course.
>
>the Linux and BSD kernels are both quite good at keeping commonly used
>data in memory. I think you are mistaken in assuming that an "on disk"
>table will be significantly slower than if it was fixed in memory due to
>the very efficient cachine of the most common unix kernels.
>
>
>---------------------------(end of broadcast)---------------------------
>TIP 6: Have you searched our list archives?
>
> http://archives.postgresql.org
>
>
>


From: Andrew Hammond <ahammond(at)ca(dot)afilias(dot)info>
To: vinayj(at)ncst(dot)ernet(dot)in
Cc: "scott(dot)marlowe" <scott(dot)marlowe(at)ihs(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: Is there any method to keep table in memory at startup
Date: 2004-04-28 13:28:08
Message-ID: 408FB168.7020500@ca.afilias.info
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Vinay Jain wrote:
> Hi
> thank you for such a useful information...
> but actually in my case if i keep table in disk it significantly
> degrades performance and even for a table of 10 rows it takes 1-2
> minutes I think u r not beliving it ! am i right
> for example
> I create a table in which i use my customized data type say student
> create table student
> (Name INDCHAR //INDCHAR is customized data type
> age integer);
> now i give query like this
> select * from student order by name;
> it will search for it's comparator operator (<) and related function...
> in that function there is one lookup table if that table is in memory no
> problem! (oh but it can't be) if it is in disk my program makes
> connection to database and execute query which is just a select
> statement on a simple where condition of equality. then closes connection

There's your problem. Creating database connections is an expensive
operation. They are not intended to be opened and closed often or
quickly. Open your database connection at the beginning of your program,
and close it at the end.

You could also throw an index on the column you're using in your order
by clause, but that won't make a difference until your table get a
little bigger.

Please take further questions of this nature to the pgsql-novice list.

> so every time less than operator(<) is called it does the same task..
> what i feel in table of 10 rows how many times the < operator will be
> called(NO idea but must be > 10 times)
> is there any solution..
> thanks in advance
> regards
> vinay


From: Vinay Jain <vinayj(at)sarathi(dot)ncst(dot)ernet(dot)in>
To: Andrew Hammond <ahammond(at)ca(dot)afilias(dot)info>
Cc: vinayj(at)ncst(dot)ernet(dot)in, "scott(dot)marlowe" <scott(dot)marlowe(at)ihs(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: Is there any method to keep table in memory at startup
Date: 2004-05-01 05:24:47
Message-ID: 4093349F.4050709@sarathi.ncst.ernet.in
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Hi
thanx and sorry that I asked such a simple question in postgres-hackers
list....
but the complexity which i feel on that basis ....please allow me to
explain my problem further.....
As i am working on sorting order , length and substring functions for
Hindi text(Indian Language)...
Here is the problem which i found in postgresql...
after setting collating sequence in proper way(i.e. C) the order was on
basis of unicode values...but in Hindi Language some of combined unicode
values makes a single character
similarly length is not appropriate for these reasons & hence substring
operations
so i designed a customized data type called IndChar....and operations on it
in order by statement the only function called is indchar_lt(defined for
< operator)......
Now please guide me where is starting(where i can open connection to
database) and ending of my program....I feel only in indchar_lt function
which will be called many times in order by statement causing
performance degradation..as i am not much experienced this assumption
may be wrong...
so my question remains as it is that is there any such thing which can
be called at startup of psql.........to make connection to database

regards
Vinay

Andrew Hammond wrote:

> Vinay Jain wrote:
>
>> Hi
>> thank you for such a useful information...
>> but actually in my case if i keep table in disk it significantly
>> degrades performance and even for a table of 10 rows it takes 1-2
>> minutes I think u r not beliving it ! am i right
>> for example
>> I create a table in which i use my customized data type say student
>> create table student
>> (Name INDCHAR //INDCHAR is customized data type
>> age integer);
>> now i give query like this
>> select * from student order by name;
>> it will search for it's comparator operator (<) and related function...
>> in that function there is one lookup table if that table is in memory
>> no problem! (oh but it can't be) if it is in disk my program makes
>> connection to database and execute query which is just a select
>> statement on a simple where condition of equality. then closes
>> connection
>
>
> There's your problem. Creating database connections is an expensive
> operation. They are not intended to be opened and closed often or
> quickly. Open your database connection at the beginning of your
> program, and close it at the end.
>
> You could also throw an index on the column you're using in your order
> by clause, but that won't make a difference until your table get a
> little bigger.
>
> Please take further questions of this nature to the pgsql-novice list.
>
>> so every time less than operator(<) is called it does the same task..
>> what i feel in table of 10 rows how many times the < operator will be
>> called(NO idea but must be > 10 times)
>> is there any solution..
>> thanks in advance
>> regards
>> vinay
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faqs/FAQ.html
>


From: Andrew Hammond <ahammond(at)ca(dot)afilias(dot)info>
To: Vinay Jain <vinayj(at)sarathi(dot)ncst(dot)ernet(dot)in>
Cc: vinayj(at)ncst(dot)ernet(dot)in, pgsql-hackers(at)postgresql(dot)org
Subject: Re: Is there any method to keep table in memory at startup
Date: 2004-05-03 15:08:03
Message-ID: 40966053.7090801@ca.afilias.info
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Vinay Jain wrote:
> Hi
> thanx and sorry that I asked such a simple question in postgres-hackers
> list....
> but the complexity which i feel on that basis ....please allow me to
> explain my problem further.....
> As i am working on sorting order , length and substring functions for
> Hindi text(Indian Language)...
> Here is the problem which i found in postgresql...
> after setting collating sequence in proper way(i.e. C) the order was on
> basis of unicode values...but in Hindi Language some of combined unicode
> values makes a single character
> similarly length is not appropriate for these reasons & hence substring
> operations
> so i designed a customized data type called IndChar....and operations on it
> in order by statement the only function called is indchar_lt(defined for
> < operator)......

Is your indchar_lt function declared IMMUTABLE? That would allow it's
results to be cached instead of re-calculated every time.

> Now please guide me where is starting(where i can open connection to
> database) and ending of my program....I feel only in indchar_lt function
> which will be called many times in order by statement causing
> performance degradation..

Have you created an index on that column? That would be a usual way to
speed up an ORDER BY. NB, the function involved must be IMMUTABLE to be
used in an index.

> as i am not much experienced this assumption may be wrong...

My professor at University used to always say "measure measure measure".
Postgres makes it easy to measure. Try putting EXPLAIN ANALYZE before
your SELECT statement. Run the SELECT a couple of times first so that
the OS can get stuff cached, then:

do an EXPLAIN ANALYZE on the query, save the results

then ANALYZE the tables involved and to another EXPLAIN ANALYZE on the
query, save the results

add the necessary index, ANALYZE then EXPLAIN ANALYZE.

> so my question remains as it is that is there any such thing which can
> be called at startup of psql.........to make connection to database

I'm really not sure what you mean by this. psql connects to the database
on startup.

> regards
> Vinay
>
>
>
>
> Andrew Hammond wrote:
>
>> Vinay Jain wrote:
>>
>>> Hi
>>> thank you for such a useful information...
>>> but actually in my case if i keep table in disk it significantly
>>> degrades performance and even for a table of 10 rows it takes 1-2
>>> minutes I think u r not beliving it ! am i right
>>> for example
>>> I create a table in which i use my customized data type say student
>>> create table student
>>> (Name INDCHAR //INDCHAR is customized data type
>>> age integer);
>>> now i give query like this
>>> select * from student order by name;
>>> it will search for it's comparator operator (<) and related function...
>>> in that function there is one lookup table if that table is in memory
>>> no problem! (oh but it can't be) if it is in disk my program makes
>>> connection to database and execute query which is just a select
>>> statement on a simple where condition of equality. then closes
>>> connection
>>
>>
>>
>> There's your problem. Creating database connections is an expensive
>> operation. They are not intended to be opened and closed often or
>> quickly. Open your database connection at the beginning of your
>> program, and close it at the end.
>>
>> You could also throw an index on the column you're using in your order
>> by clause, but that won't make a difference until your table get a
>> little bigger.
>>
>> Please take further questions of this nature to the pgsql-novice list.
>>
>>> so every time less than operator(<) is called it does the same task..
>>> what i feel in table of 10 rows how many times the < operator will be
>>> called(NO idea but must be > 10 times)
>>> is there any solution..
>>> thanks in advance
>>> regards
>>> vinay
>>
>>
>>
>>
>> ---------------------------(end of broadcast)---------------------------
>> TIP 5: Have you checked our extensive FAQ?
>>
>> http://www.postgresql.org/docs/faqs/FAQ.html
>>
>
>
>


From: Vinay Jain <vinayj(at)sarathi(dot)ncst(dot)ernet(dot)in>
To: Andrew Hammond <ahammond(at)ca(dot)afilias(dot)info>
Cc: vinayj(at)ncst(dot)ernet(dot)in, pgsql-hackers(at)postgresql(dot)org
Subject: Re: Is there any method to keep table in memory at startup
Date: 2004-05-06 08:47:49
Message-ID: 4099FBB5.7070809@sarathi.ncst.ernet.in
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

hi
ya function is declared immutable and strict... also made btree index..
one mistake that i was doing that I was using libpq rather than
SPI..but even after using it though no doubt the performance is
increased still not up to the mark
if i use index than it's explain analyze is like this :

// For 11 rows

template1=# explain ANALYZE select * from temp_hindi_copy3 order by name;
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------
Index Scan using temp_hindi_copy3_pkey on temp_hindi_copy3
(cost=0.00..52.00 rows=1000 width=36) (actual time=0.06..0.12 rows=11
loops=1)
Total runtime: 0.24 msec
which is good and obvious but when i have to use other functions with
it like

// 11 Rows

template1=# explain ANALYZE select name, length(name), substr(name, 3,
2) from temp_hindi_copy3 order by name;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------
Index Scan using temp_hindi_copy3_pkey on temp_hindi_copy3
(cost=0.00..52.00 rows=1000 width=32) (actual time=88.72..920.18 rows=11
loops=1)
Total runtime: 920.37 msec
(2 rows)
which is not good
similarly for Non Key order by:

//136 Rows
template1=# explain ANALYZE select * from temp_hindi_copy order by name;
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------
Sort (cost=69.83..72.33 rows=1000 width=36) (actual
time=26282.57..26282.68 rows=136 loops=1)
Sort Key: name
-> Seq Scan on temp_hindi_copy (cost=0.00..20.00 rows=1000
width=36) (actual time=0.22..0.60 rows=136 loops=1)
Total runtime: 26282.92 msec
(4 rows)

Which takes a lot of time the reason is that in every indchar_lt
function call SPI connection is made and destroyed...
and order by clause calls it n (> no. of rows) times..
I tried a more than 10 times still the output was same.......... also in
one order by statement the same query is called more than n times....
but performance is same....
so is there any way to just make connection once and keep data in
memory...and when psql is exited the connection is closed...

When we start psql it makes connection with database is correct...
but not from program ....so have to Make SPI connection...

Andrew Hammond wrote:

> Vinay Jain wrote:
>
>> Hi
>> thanx and sorry that I asked such a simple question in
>> postgres-hackers list....
>> but the complexity which i feel on that basis ....please allow me to
>> explain my problem further.....
>> As i am working on sorting order , length and substring functions for
>> Hindi text(Indian Language)...
>> Here is the problem which i found in postgresql...
>> after setting collating sequence in proper way(i.e. C) the order was
>> on basis of unicode values...but in Hindi Language some of combined
>> unicode values makes a single character
>> similarly length is not appropriate for these reasons & hence
>> substring operations
>> so i designed a customized data type called IndChar....and operations
>> on it
>> in order by statement the only function called is indchar_lt(defined
>> for < operator)......
>
>
is your indchar_lt function declared IMMUTABLE? That would allow it's
results to be cached instead of re-calculated every time.

>

>
>> Now please guide me where is starting(where i can open connection to
>> database) and ending of my program....I feel only in indchar_lt
>> function which will be called many times in order by statement
>> causing performance degradation..
>
>
> Have you created an index on that column? That would be a usual way to
> speed up an ORDER BY. NB, the function involved must be IMMUTABLE to
> be used in an index.
>
>> as i am not much experienced this assumption may be wrong...
>
>
> My professor at University used to always say "measure measure
> measure". Postgres makes it easy to measure. Try putting EXPLAIN
> ANALYZE before your SELECT statement. Run the SELECT a couple of times
> first so that the OS can get stuff cached, then:
>
> do an EXPLAIN ANALYZE on the query, save the results
>
> then ANALYZE the tables involved and to another EXPLAIN ANALYZE on the
> query, save the results
>
> add the necessary index, ANALYZE then EXPLAIN ANALYZE.
>
>> so my question remains as it is that is there any such thing which
>> can be called at startup of psql.........to make connection to database
>
>
> I'm really not sure what you mean by this. psql connects to the
> database on startup.
>
>> regards
>> Vinay
>>
>>
>>
>>
>> Andrew Hammond wrote:
>>
>>> Vinay Jain wrote:
>>>
>>>> Hi
>>>> thank you for such a useful information...
>>>> but actually in my case if i keep table in disk it
>>>> significantly degrades performance and even for a table of 10 rows
>>>> it takes 1-2 minutes I think u r not beliving it ! am i right
>>>> for example
>>>> I create a table in which i use my customized data type say student
>>>> create table student
>>>> (Name INDCHAR //INDCHAR is customized data type
>>>> age integer);
>>>> now i give query like this
>>>> select * from student order by name;
>>>> it will search for it's comparator operator (<) and related
>>>> function...
>>>> in that function there is one lookup table if that table is in
>>>> memory no problem! (oh but it can't be) if it is in disk my
>>>> program makes connection to database and execute query which is
>>>> just a select statement on a simple where condition of equality.
>>>> then closes connection
>>>
>>>
>>>
>>>
>>> There's your problem. Creating database connections is an expensive
>>> operation. They are not intended to be opened and closed often or
>>> quickly. Open your database connection at the beginning of your
>>> program, and close it at the end.
>>>
>>> You could also throw an index on the column you're using in your
>>> order by clause, but that won't make a difference until your table
>>> get a little bigger.
>>>
>>> Please take further questions of this nature to the pgsql-novice list.
>>>
>>>> so every time less than operator(<) is called it does the same task..
>>>> what i feel in table of 10 rows how many times the < operator will
>>>> be called(NO idea but must be > 10 times)
>>>> is there any solution..
>>>> thanks in advance
>>>> regards
>>>> vinay
>>>
>>>
>>>
>>>
>>>
>>> ---------------------------(end of
>>> broadcast)---------------------------
>>> TIP 5: Have you checked our extensive FAQ?
>>>
>>> http://www.postgresql.org/docs/faqs/FAQ.html
>>>
>>
>>
>>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faqs/FAQ.html
>


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Vinay Jain <vinayj(at)sarathi(dot)ncst(dot)ernet(dot)in>
Cc: Andrew Hammond <ahammond(at)ca(dot)afilias(dot)info>, vinayj(at)ncst(dot)ernet(dot)in, pgsql-hackers(at)postgresql(dot)org
Subject: Re: Is there any method to keep table in memory at startup
Date: 2004-05-06 12:40:47
Message-ID: 26232.1083847247@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Vinay Jain <vinayj(at)sarathi(dot)ncst(dot)ernet(dot)in> writes:
> Which takes a lot of time the reason is that in every indchar_lt
> function call SPI connection is made and destroyed...

I cannot imagine how you'd think that that would be practical from a
performance standpoint.

> Here is the problem which i found in postgresql...
> after setting collating sequence in proper way(i.e. C) the order was
> on basis of unicode values...but in Hindi Language some of combined
> unicode values makes a single character
> similarly length is not appropriate for these reasons & hence
> substring operations

Why don't you just use a Hindi locale? The infrastructure should all be
there already for you.

regards, tom lane


From: "Andrew Dunstan" <andrew(at)dunslane(dot)net>
To: <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Is there any method to keep table in memory at startup
Date: 2004-05-06 13:06:57
Message-ID: 1771.24.211.141.25.1083848817.squirrel@www.dunslane.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Vinay Jain said:
> hi
> ya function is declared immutable and strict... also made btree
> index..

The docs state this about immutable functions:

IMMUTABLE indicates that the function always returns the same result when
given the same argument values; that is, it does not do database lookups
or otherwise use information not directly present in its argument list.

So it appears you are lying to postgres when you declare your function to
be immutable, because you are doing a database lookup.

Why not just code your lookup table as static data immediately available
to your function, and look it up from C directly, rather than using a
database table? Then your function could be genuinely immutable (and fast).

cheers

andrew


From: Vinay Jain <vinayj(at)sarathi(dot)ncst(dot)ernet(dot)in>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Andrew Hammond <ahammond(at)ca(dot)afilias(dot)info>, vinayj(at)ncst(dot)ernet(dot)in, pgsql-hackers(at)postgresql(dot)org
Subject: Re: Is there any method to keep table in memory at startup
Date: 2004-05-06 13:25:47
Message-ID: 409A3CDB.1000906@sarathi.ncst.ernet.in
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Tom Lane wrote:

>Vinay Jain <vinayj(at)sarathi(dot)ncst(dot)ernet(dot)in> writes:
>
>
>>Which takes a lot of time the reason is that in every indchar_lt
>>function call SPI connection is made and destroyed...
>>
>>
>
>I cannot imagine how you'd think that that would be practical from a
>performance standpoint.
>
>
>
>>Here is the problem which i found in postgresql...
>>after setting collating sequence in proper way(i.e. C) the order was
>>on basis of unicode values...but in Hindi Language some of combined
>>unicode values makes a single character
>>similarly length is not appropriate for these reasons & hence
>>substring operations
>>
>>
>
>Why don't you just use a Hindi locale? The infrastructure should all be
>there already for you.
>
> regards, tom lane
>
Hi
Not actually even in Hindi Locale the output was incorrect..i.e. sort
order was wrong
and also length and substring operations
which are not based on syllables.

>
>
>


From: Vinay Jain <vinayj(at)sarathi(dot)ncst(dot)ernet(dot)in>
To: Andrew Dunstan <andrew(at)dunslane(dot)net>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: Is there any method to keep table in memory at startup
Date: 2004-05-06 13:33:06
Message-ID: 409A3E92.8000307@sarathi.ncst.ernet.in
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Andrew Dunstan wrote:

>Vinay Jain said:
>
>
>>hi
>>ya function is declared immutable and strict... also made btree
>>index..
>>
>>
>
>
>The docs state this about immutable functions:
>
>
>IMMUTABLE indicates that the function always returns the same result when
>given the same argument values; that is, it does not do database lookups
>or otherwise use information not directly present in its argument list.
>
>
>So it appears you are lying to postgres when you declare your function to
>be immutable, because you are doing a database lookup.
>
>Why not just code your lookup table as static data immediately available
>to your function, and look it up from C directly, rather than using a
>database table? Then your function could be genuinely immutable (and fast).
>
>cheers
>
>andrew
>
>Hi
>
You mean that I should create a static table in C program itself and use
it...if i am not wrong
Ya for the time being i am doing this thing but actually table is bigger
(around 5000 rows with 6 columns)
also this table is also created and destroyed in each indchar_lt call
which is called many times in order by clause
one more thing i want it generalized so that I can include other indian
languages also..without changing code..
If there is not any method to get results fast using database table i
will have to opt this option only..
regards
Vinay

>
>
>
>---------------------------(end of broadcast)---------------------------
>TIP 6: Have you searched our list archives?
>
> http://archives.postgresql.org
>
>
>


From: Andrew Dunstan <andrew(at)dunslane(dot)net>
To: pgsql-hackers(at)postgresql(dot)org
Subject: Re: Is there any method to keep table in memory at startup
Date: 2004-05-06 14:28:30
Message-ID: 409A4B8E.2010309@dunslane.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Vinay Jain wrote:

> You mean that I should create a static table in C program itself and
> use it...if i am not wrong
> Ya for the time being i am doing this thing but actually table is
> bigger (around 5000 rows with 6 columns)

bigger than what?

> also this table is also created and destroyed in each indchar_lt call
> which is called many times in order by clause

uh ... maybe you need to look in your C manual about the effect of a
"static" declaration. The object will be created once.

> one more thing i want it generalized so that I can include other
> indian languages also..without changing code..
> If there is not any method to get results fast using database table i
> will have to opt this option only..

Not for what you want - you are pursuing a chimera, IMNSHO.

cheers

andrew


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Vinay Jain <vinayj(at)sarathi(dot)ncst(dot)ernet(dot)in>
Cc: Andrew Hammond <ahammond(at)ca(dot)afilias(dot)info>, vinayj(at)ncst(dot)ernet(dot)in, pgsql-hackers(at)postgresql(dot)org
Subject: Re: Is there any method to keep table in memory at startup
Date: 2004-05-06 14:32:21
Message-ID: 28455.1083853941@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Vinay Jain <vinayj(at)sarathi(dot)ncst(dot)ernet(dot)in> writes:
> Not actually even in Hindi Locale the output was incorrect..i.e. sort
> order was wrong
> and also length and substring operations
> which are not based on syllables.

Hm, possibly you weren't using the same character set encoding that the
locale was expecting? It's not very well documented, but every locale
setting works only with a specific encoding.

If the locale definition really is wrong for your purposes, it seems
like what you want to do is write a new locale definition that does what
you want. Then you could use it with any Unix program, not only
Postgres. (I've never done this, but I can't see that it would be any
harder than writing C code inside Postgres to do it ...)

regards, tom lane


From: Vinay Jain <vinayj(at)sarathi(dot)ncst(dot)ernet(dot)in>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Andrew Hammond <ahammond(at)ca(dot)afilias(dot)info>, vinayj(at)ncst(dot)ernet(dot)in, pgsql-hackers(at)postgresql(dot)org
Subject: Re: Is there any method to keep table in memory at startup
Date: 2004-05-07 03:54:17
Message-ID: 409B0869.7000001@sarathi.ncst.ernet.in
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers


Tom Lane wrote:

>Vinay Jain <vinayj(at)sarathi(dot)ncst(dot)ernet(dot)in> writes:
>
>
>>Not actually even in Hindi Locale the output was incorrect..i.e. sort
>>order was wrong
>>and also length and substring operations
>>which are not based on syllables.
>>
>>
>
>Hm, possibly you weren't using the same character set encoding that the
>locale was expecting? It's not very well documented, but every locale
>setting works only with a specific encoding.
>
I kept server side encoding to hi_IN.UTF-8...(Hindi Locale)
also tested changing my OS encoding to the same.....
so i think that I was using right character set encoding..

>
>If the locale definition really is wrong for your purposes, it seems
>like what you want to do is write a new locale definition that does what
>you want. Then you could use it with any Unix program, not only
>Postgres. (I've never done this, but I can't see that it would be any
>harder than writing C code inside Postgres to do it ...)
>
>Locale defination is not wrong for my purpose only ordering , length and substring operations are incorrect (in postgres these operations are based on either character or unicodes not on syllables)which i corrected using my own data type and operations
>
and this is not the case with Postgres only even in MS-SQL the problem
is same...Research work is going on Ordering issues of Hindi and this
project is part of it
I opted PostgreSQL because It gives flexibility to design own data type
and operations in a nice way...
only problem is performance.........because I have to look up a table
while doing comparision and want to keep this table in DataBase

> regards, tom lane
>
>regards
>
Vinay Jain

>
>
>


From: Vinay Jain <vinayj(at)sarathi(dot)ncst(dot)ernet(dot)in>
To: Andrew Dunstan <andrew(at)dunslane(dot)net>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: Is there any method to keep table in memory at startup
Date: 2004-05-07 06:47:03
Message-ID: 409B30E7.2090303@sarathi.ncst.ernet.in
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Andrew Dunstan wrote:

> Vinay Jain wrote:
>
>> You mean that I should create a static table in C program itself and
>> use it...if i am not wrong
>> Ya for the time being i am doing this thing but actually table is
>> bigger (around 5000 rows with 6 columns)
>
>
>
> bigger than what?
>
>> also this table is also created and destroyed in each indchar_lt call
>> which is called many times in order by clause
>
>
>
> uh ... maybe you need to look in your C manual about the effect of a
> "static" declaration. The object will be created once.
>
> yup I know the effect of Static but this is also fact that when
> program terminates and restarts object will be created again...(not
> persistant)..

one thing strikes in my mind is that indchar_lt is function in shared
object file indchar.so.....if this file is not unloaded during Order by
call than this static defination can work for me..
Not sure but I think this file is loaded each time indchar_lt is called
in order by clause...

>
>> one more thing i want it generalized so that I can include other
>> indian languages also..without changing code..
>> If there is not any method to get results fast using database table i
>> will have to opt this option only..
>
>
>
> Not for what you want - you are pursuing a chimera, IMNSHO.
>
> cheers
>
> andrew
> regards

Vinay jain

>
> ---------------------------(end of broadcast)---------------------------
> TIP 9: the planner will ignore your desire to choose an index scan if
> your
> joining column's datatypes do not match
>


From: "Andrew Dunstan" <andrew(at)dunslane(dot)net>
To: <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Is there any method to keep table in memory at startup
Date: 2004-05-07 11:41:56
Message-ID: 2338.24.211.141.25.1083930116.squirrel@www.dunslane.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Vinay Jain said:
> Andrew Dunstan wrote:
>
>>
>> uh ... maybe you need to look in your C manual about the effect of a
>> "static" declaration. The object will be created once.
>>
> yup I know the effect of Static but this is also fact that when
> program terminates and restarts object will be created again...(not
> persistant)..
>

you can have it preloaded and persistent to the end of the server run, see
http://www.postgresql.org/docs/current/static/runtime-config.html#RUNTIME-
CONFIG-RESOURCE

> one thing strikes in my mind is that indchar_lt is function in shared
> object file indchar.so.....if this file is not unloaded during Order by
> call than this static defination can work for me..
> Not sure but I think this file is loaded each time indchar_lt is called
> in order by clause...
>

If course it is not loaded each time. That would be insane. If not
preloaded it is loaded when first called in each process, and then kept.
It is never unloaded (except by the termination of the process that loaded
it).

It seems you have been laboring under a misapprehension.

cheers

andrew


From: Vinay Jain <vinayj(at)sarathi(dot)ncst(dot)ernet(dot)in>
To: Andrew Dunstan <andrew(at)dunslane(dot)net>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: Is there any method to keep table in memory at startup
Date: 2004-05-07 12:05:12
Message-ID: 409B7B78.8080905@sarathi.ncst.ernet.in
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Andrew Dunstan wrote:

>Vinay Jain said:
>
>
>>Andrew Dunstan wrote:
>>
>>
>>
>>>uh ... maybe you need to look in your C manual about the effect of a
>>>"static" declaration. The object will be created once.
>>>
>>>
>>>
>>yup I know the effect of Static but this is also fact that when
>>program terminates and restarts object will be created again...(not
>>persistant)..
>>
>>
>>
>
>you can have it preloaded and persistent to the end of the server run, see
>http://www.postgresql.org/docs/current/static/runtime-config.html#RUNTIME-
>CONFIG-RESOURCE
>
>
>
>
>>one thing strikes in my mind is that indchar_lt is function in shared
>>object file indchar.so.....if this file is not unloaded during Order by
>> call than this static defination can work for me..
>>Not sure but I think this file is loaded each time indchar_lt is called
>> in order by clause...
>>
>>
>>
>
>If course it is not loaded each time. That would be insane. If not
>preloaded it is loaded when first called in each process, and then kept.
>It is never unloaded (except by the termination of the process that loaded
>it).
>
>It seems you have been laboring under a misapprehension.
>
If this is the case than I can make connection to data base in starting
of indchar.so file and close connection at end of it and it should work....
bingo!!
it would solve my problem

>
>cheers
>
>andrew
>
>thanx
>
>Vinay Jain
>---------------------------(end of broadcast)---------------------------
>TIP 1: subscribe and unsubscribe commands go to majordomo(at)postgresql(dot)org
>
>
>


From: "Andrew Dunstan" <andrew(at)dunslane(dot)net>
To: <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Is there any method to keep table in memory at startup
Date: 2004-05-07 12:25:28
Message-ID: 2592.24.211.141.25.1083932728.squirrel@www.dunslane.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Vinay Jain said:
> Andrew Dunstan wrote:
>
>>
>>If course it is not loaded each time. That would be insane. If not
>>preloaded it is loaded when first called in each process, and then
>>kept. It is never unloaded (except by the termination of the process
>>that loaded it).
>>
>>It seems you have been laboring under a misapprehension.
>>
> If this is the case than I can make connection to data base in starting
> of indchar.so file and close connection at end of it and it should
> work.... bingo!!
> it would solve my problem
>

*sigh*

You are not getting it.

An immutable function MUST NOT DEPEND ON DATA IN THE DATABASE. Sorry to
shout but you really need to understand this.

Use static C data, not database tables. In addition to it being pure, it
will also be enormously faster than getting data from the database. Yes it
means that if you want to change the lookup data you need to recompile
your C function library, and to redo any indexes etc. that depend on the
function. That's just the way it is, I'm afraid.

cheers

andrew


From: Vinay Jain <vinayj(at)sarathi(dot)ncst(dot)ernet(dot)in>
To: Andrew Dunstan <andrew(at)dunslane(dot)net>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: Is there any method to keep table in memory at startup
Date: 2004-05-07 12:51:25
Message-ID: 409B864D.5040406@sarathi.ncst.ernet.in
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Andrew Dunstan wrote:

>Vinay Jain said:
>
>
>>Andrew Dunstan wrote:
>>
>>
>>
>>>If course it is not loaded each time. That would be insane. If not
>>>preloaded it is loaded when first called in each process, and then
>>>kept. It is never unloaded (except by the termination of the process
>>>that loaded it).
>>>
>>>It seems you have been laboring under a misapprehension.
>>>
>>>
>>>
>>If this is the case than I can make connection to data base in starting
>> of indchar.so file and close connection at end of it and it should
>>work.... bingo!!
>>it would solve my problem
>>
>>
>>
>
>*sigh*
>
>You are not getting it.
>
>An immutable function MUST NOT DEPEND ON DATA IN THE DATABASE. Sorry to
>shout but you really need to understand this.
>
>Use static C data, not database tables. In addition to it being pure, it
>will also be enormously faster than getting data from the database. Yes it
>means that if you want to change the lookup data you need to recompile
>your C function library, and to redo any indexes etc. that depend on the
>function. That's just the way it is, I'm afraid.
>
>cheers
>
>andrew
>Hi
>
ya you are right but i wanted to escape from compilation and stuff....

I will think on declaring function stable rather than immutable and test
performance....
for the time being i will follow your advice...
thanks
regards
Vinay

>
>
>---------------------------(end of broadcast)---------------------------
>TIP 8: explain analyze is your friend
>
>
>