Duplicate key insert question

Lists: pgsql-general
From: Jean-Christian Imbeault <jc(at)mega-bucks(dot)co(dot)jp>
To: pgsql-general(at)postgresql(dot)org
Subject: Duplicate key insert question
Date: 2003-07-02 00:10:14
Message-ID: 3F0222E6.2010706@mega-bucks.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

I have a table with a primary field and a few other fields. What is the
fastest way to do an insert into that table assuming that sometimes I
might try to insert a record with a duplicate primary key and want that
to fail?

I know that if I try a plain insert this will work, but in the case
where I am trying to insert a duplicate key, the insert fails (as it
should) and an error is logged.

I could first do a check to see if there is already an entry with the
same key as the one I am trying to insert but that would mean quite a
few extra operations.

Is there a quick and clean way of doing something like:

INSERT into table values(...) IF there isn't already a row with pkey=x

Thanks,

Jean-Christian Imbeault

PS The reason I am asking is that an open source project is using MySQL
as their DB and they have a port to PG that isn't very clean b/c the DB
code keeps trying to insert duplicate primary keys. According to them
MySQL doesn't complain and just drops the insert whereas PG (as is
right) complains. I've offered to clean up their PG insertion code but
they say that they don't want too many extra checks as their app writes
to the DB a *lot* and any extra check is going to slow down the
application noticeably ...


From: "Reuben D(dot) Budiardja" <techlist(at)voyager(dot)phys(dot)utk(dot)edu>
To: Jean-Christian Imbeault <jc(at)mega-bucks(dot)co(dot)jp>, pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 00:43:59
Message-ID: 200307012043.59837.techlist@voyager.phys.utk.edu
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

On Tuesday 01 July 2003 08:10 pm, Jean-Christian Imbeault wrote:
> I have a table with a primary field and a few other fields. What is the
> fastest way to do an insert into that table assuming that sometimes I
> might try to insert a record with a duplicate primary key and want that
> to fail?
>
> I know that if I try a plain insert this will work, but in the case
> where I am trying to insert a duplicate key, the insert fails (as it
> should) and an error is logged.
>
> I could first do a check to see if there is already an entry with the
> same key as the one I am trying to insert but that would mean quite a
> few extra operations.
>
> Is there a quick and clean way of doing something like:
>
> INSERT into table values(...) IF there isn't already a row with pkey=x
>
> Thanks,
>
> Jean-Christian Imbeault

Hi, not sure if this is answering your question, but I just asked similar
questions here. I asked about using INSERT WHERE NOT EXISTS (which you can do
in PostgreSQL). Here is what you can do:

INSERT INTO mytable
SELECT 'value1', 'value2'
WHERE NOT EXISTS
(SELECT NULL FROM mytable
WHERE mycondition)

This will just return 0 when fails, but it does check first. Don't know if you
can really afford that. Just for reference, this brought up some discussion
here. Here is a link to the archive:
http://marc.theaimsgroup.com/?l=postgresql-general&w=2&r=1&s=WHERE+NOT+EXISTS&q=b

Hope that helps.
RDB

--
Reuben D. Budiardja
Department of Physics and Astronomy
The University of Tennessee, Knoxville, TN
-------------------------------------------------
/"\ ASCII Ribbon Campaign against HTML
\ / email and proprietary format
X attachments.
/ \
-------------------------------------------------
Have you been used by Microsoft today?
Choose your life. Choose freedom.
Choose LINUX.
-------------------------------------------------


From: Jean-Christian Imbeault <jc(at)mega-bucks(dot)co(dot)jp>
To: techlist(at)voyager(dot)phys(dot)utk(dot)edu
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 00:45:23
Message-ID: 3F022B23.9030202@mega-bucks.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

Reuben D. Budiardja wrote:
>
> Hi, not sure if this is answering your question, but I just asked similar
> questions here. I asked about using INSERT WHERE NOT EXISTS (which you can do
> in PostgreSQL). Here is what you can do:
>
> INSERT INTO mytable
> SELECT 'value1', 'value2'
> WHERE NOT EXISTS
> (SELECT NULL FROM mytable
> WHERE mycondition)
>
> http://marc.theaimsgroup.com/?l=postgresql-general&w=2&r=1&s=WHERE+NOT+EXISTS&q=b

Thanks for the link!

I read the thread and it looks like even the above solution is not
perfect because of a possible race condition where two inserts trying to
insert a row with a pk not in the table will both get think it is ok to
do so, try it and then both will fail?

If I followed all the arguments correctly according to the thread there
is *no* way to do what I (and you ;) want in one simple query.

Do you agree? Or did I miss something?

Jean-Christian Imbeault


From: Alvaro Herrera <alvherre(at)dcc(dot)uchile(dot)cl>
To: Jean-Christian Imbeault <jc(at)mega-bucks(dot)co(dot)jp>
Cc: techlist(at)voyager(dot)phys(dot)utk(dot)edu, pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 00:51:11
Message-ID: 20030702005111.GK27363@dcc.uchile.cl
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

On Wed, Jul 02, 2003 at 09:45:23AM +0900, Jean-Christian Imbeault wrote:
> Reuben D. Budiardja wrote:

> > http://marc.theaimsgroup.com/?l=postgresql-general&w=2&r=1&s=WHERE+NOT+EXISTS&q=b
>
> Thanks for the link!
>
> I read the thread and it looks like even the above solution is not
> perfect because of a possible race condition where two inserts trying to
> insert a row with a pk not in the table will both get think it is ok to
> do so, try it and then both will fail?

No, only the "second" one will fail (though it's difficult which one is
the second)

> If I followed all the arguments correctly according to the thread there
> is *no* way to do what I (and you ;) want in one simple query.

No, there's not. You should check the returned value from the insertion
function to see if it succeeded or not. Sadly, an error will cause the
whole transaction to abort, but if they come from the MySQL side it will
hardly matter. But you should try to use a sequence if at all possible
to avoid all these problems.

--
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"A wizard is never late, Frodo Baggins, nor is he early.
He arrives precisely when he means to." (Gandalf, en LoTR FoTR)


From: Jean-Christian Imbeault <jc(at)mega-bucks(dot)co(dot)jp>
To: Alvaro Herrera <alvherre(at)dcc(dot)uchile(dot)cl>
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 00:58:28
Message-ID: 3F022E34.1040808@mega-bucks.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

Alvaro Herrera wrote:
>
> No, only the "second" one will fail (though it's difficult which one is
> the second)

From:

http://marc.theaimsgroup.com/?l=postgresql-general&m=105656988915991&w=2

Ian Barwick wrote:

[...]

I proposed that same solution 3 years ago. Tom shoots it down:

[...]

I couldn't get the link to work so I couldn't read why Tom shot it down.
But if Tom shot down this idea down ... then it mustn't be correct.

>>If I followed all the arguments correctly according to the thread there
>>is *no* way to do what I (and you ;) want in one simple query.
>
>
> No, there's not.

You say no, but at first you say that the proposed method works. The
proposed method, if it is correct, is simple enough for me. By simple I
mean all can be done with one query.

> You should check the returned value from the insertion
> function to see if it succeeded or not.

No, what I want if to have one query that will *always* insert if there
is no record with this primary key and *always* do nothing (not fail,
not generate an error) if there is already a record with this primary
key. I don't want to check return values :)

Thanks,

Jean-Christian Imbeault


From: Alvaro Herrera <alvherre(at)dcc(dot)uchile(dot)cl>
To: Jean-Christian Imbeault <jc(at)mega-bucks(dot)co(dot)jp>
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 01:16:02
Message-ID: 20030702011602.GL27363@dcc.uchile.cl
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

On Wed, Jul 02, 2003 at 09:58:28AM +0900, Jean-Christian Imbeault wrote:
> Alvaro Herrera wrote:
> >
> > No, only the "second" one will fail (though it's difficult which one is
> > the second)
>
> I couldn't get the link to work so I couldn't read why Tom shot it down.
> But if Tom shot down this idea down ... then it mustn't be correct.

The thread is here:
http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&threadm=3A4D6116.1A613402%40mascari.com&rnum=1&prev=/groups%3Fq%3DMike%2BMascari%2BINSERT%2BNOT%2BEXISTS%26ie%3D

The solution is not correct in that there _is_ a race condition.

> > You should check the returned value from the insertion
> > function to see if it succeeded or not.
>
> No, what I want if to have one query that will *always* insert if there
> is no record with this primary key and *always* do nothing (not fail,
> not generate an error) if there is already a record with this primary
> key. I don't want to check return values :)

No way.

--
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"No es bueno caminar con un hombre muerto"


From: Jean-Christian Imbeault <jc(at)mega-bucks(dot)co(dot)jp>
To: Alvaro Herrera <alvherre(at)dcc(dot)uchile(dot)cl>
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 01:23:22
Message-ID: 3F02340A.9050205@mega-bucks.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

Alvaro Herrera wrote:
>
> The thread is here:
> http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&threadm=3A4D6116.1A613402%40mascari.com&rnum=1&prev=/groups%3Fq%3DMike%2BMascari%2BINSERT%2BNOT%2BEXISTS%26ie%3D

Thanks!

> The solution is not correct in that there _is_ a race condition.

I thought so :(

>>No, what I want if to have one query that will *always* insert if there
>>is no record with this primary key and *always* do nothing (not fail,
>>not generate an error) if there is already a record with this primary
>>key. I don't want to check return values :)
>
>
> No way.

I was beginning to think so. Thanks for confirming my suspicions.

In your opinion what is the best solution, if we define best as not
generating any error messages and executing as quickly as possible?

Thanks,

Jean-Christian Imbeault


From: Jean-Christian Imbeault <jc(at)mega-bucks(dot)co(dot)jp>
To: techlist(at)voyager(dot)phys(dot)utk(dot)edu
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 01:25:54
Message-ID: 3F0234A2.4030407@mega-bucks.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

Reuben D. Budiardja wrote:
>
> No, onlu *one* of them will fail, but yes, the other will then generate error.
> So it really is a trade off. Another way would be to lock the table, as other
> has suggested. But then there is disadvantages to that also.

Really? I just got a post form Alvaro Herrera saying;

"The solution is not correct in that there _is_ a race condition."

Maybe I misunderstood, but "not correct" doesn't sound good :)

Jean-Christian Imbeault


From: "Reuben D(dot) Budiardja" <techlist(at)voyager(dot)phys(dot)utk(dot)edu>
To: Jean-Christian Imbeault <jc(at)mega-bucks(dot)co(dot)jp>
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 01:32:59
Message-ID: 200307012132.59118.techlist@voyager.phys.utk.edu
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

On Tuesday 01 July 2003 08:45 pm, Jean-Christian Imbeault wrote:
> Reuben D. Budiardja wrote:
> > Hi, not sure if this is answering your question, but I just asked similar
> > questions here. I asked about using INSERT WHERE NOT EXISTS (which you
> > can do in PostgreSQL). Here is what you can do:
> >
> > INSERT INTO mytable
> > SELECT 'value1', 'value2'
> > WHERE NOT EXISTS
> > (SELECT NULL FROM mytable
> > WHERE mycondition)
> >
> > http://marc.theaimsgroup.com/?l=postgresql-general&w=2&r=1&s=WHERE+NOT+EX
> >ISTS&q=b
>
> Thanks for the link!
>
> I read the thread and it looks like even the above solution is not
> perfect because of a possible race condition where two inserts trying to
> insert a row with a pk not in the table will both get think it is ok to
> do so, try it and then both will fail?

No, onlu *one* of them will fail, but yes, the other will then generate error.
So it really is a trade off. Another way would be to lock the table, as other
has suggested. But then there is disadvantages to that also.

RDB

--
Reuben D. Budiardja
Department of Physics and Astronomy
The University of Tennessee, Knoxville, TN
-------------------------------------------------
/"\ ASCII Ribbon Campaign against HTML
\ / email and proprietary format
X attachments.
/ \
-------------------------------------------------
Have you been used by Microsoft today?
Choose your life. Choose freedom.
Choose LINUX.
-------------------------------------------------


From: Alvaro Herrera <alvherre(at)dcc(dot)uchile(dot)cl>
To: Jean-Christian Imbeault <jc(at)mega-bucks(dot)co(dot)jp>
Cc: techlist(at)voyager(dot)phys(dot)utk(dot)edu, pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 01:36:36
Message-ID: 20030702013636.GN27363@dcc.uchile.cl
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

On Wed, Jul 02, 2003 at 10:25:54AM +0900, Jean-Christian Imbeault wrote:
> Reuben D. Budiardja wrote:
> >
> > No, onlu *one* of them will fail, but yes, the other will then generate error.
> > So it really is a trade off. Another way would be to lock the table, as other
> > has suggested. But then there is disadvantages to that also.
>
> Really? I just got a post form Alvaro Herrera saying;
>
> "The solution is not correct in that there _is_ a race condition."
>
> Maybe I misunderstood, but "not correct" doesn't sound good :)

Well, he is right. One will fail, the other will not. The race
condition is for the application. If you want to ignore it, you can do
that, but there _will_ be an ERROR thrown and the transaction will be
aborted. The other transaction _will_ insert the tuple, though, and it
won't be aborted.

Note that for the race condition to show there has to be a race, i.e.
two backends trying to insert the same primary key at the same time. If
one finishes half a second before the other, they will behave that way
you want, i.e. there will one tuple inserted and no error generated.

--
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"No deja de ser humillante para una persona de ingenio saber
que no hay tonto que no le pueda enseƱar algo." (Jean B. Say)


From: Jean-Christian Imbeault <jc(at)mega-bucks(dot)co(dot)jp>
To: Alvaro Herrera <alvherre(at)dcc(dot)uchile(dot)cl>
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 01:43:48
Message-ID: 3F0238D4.1030307@mega-bucks.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

Alvaro Herrera wrote:
>
> Well, he is right. One will fail, the other will not. The race
> condition is for the application. If you want to ignore it, you can do
> that, but there _will_ be an ERROR thrown and the transaction will be
> aborted.

Ah ... then maybe this solution is 'good enough'. It will still generate
an error message some of the time (when there is a race condition) but
will definitely generate fewer error messages than the current method
used which is just to do the insert and let it fail if there is already
a record with the same primary key.

Thanks for the help!

Jean-Christian Imbeault


From: "Reuben D(dot) Budiardja" <techlist(at)voyager(dot)phys(dot)utk(dot)edu>
To: Jean-Christian Imbeault <jc(at)mega-bucks(dot)co(dot)jp>
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 03:07:46
Message-ID: 200307012307.46754.techlist@voyager.phys.utk.edu
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

On Tuesday 01 July 2003 09:25 pm, Jean-Christian Imbeault wrote:
> Reuben D. Budiardja wrote:
> > No, onlu *one* of them will fail, but yes, the other will then generate
> > error. So it really is a trade off. Another way would be to lock the
> > table, as other has suggested. But then there is disadvantages to that
> > also.
>
> Really? I just got a post form Alvaro Herrera saying;
>
> "The solution is not correct in that there _is_ a race condition."
>
> Maybe I misunderstood, but "not correct" doesn't sound good :)
>
If you want to avoid the race condition as well, then use the locking
mechanism for transaction. Combine it with the previous INSERT ... SELECT ...
WHERE NOT EXISTS, it should give you what you want. I suspect it's slower
though. Eg:

BEGIN WORK;

INSERT INTO mytable
SELECT 'value1', 'value2'
WHERE NOT EXISTS
(SELECT NULL FROM mytable
WHERE mycondition)

COMMIT WORK;

This should solve the Race Condition, since other transaction have to wait.
But if the PK already exists, this will quit without error.

RDB


From: Jean-Christian Imbeault <jc(at)mega-bucks(dot)co(dot)jp>
To: techlist(at)voyager(dot)phys(dot)utk(dot)edu
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 03:08:56
Message-ID: 3F024CC8.8000403@mega-bucks.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

Reuben D. Budiardja wrote:
>
> INSERT INTO mytable
> SELECT 'value1', 'value2'
> WHERE NOT EXISTS
> (SELECT NULL FROM mytable
> WHERE mycondition)

Thank you to everyone who helped out on my question. I am trying to
implement the above solution but I'm having problems getting this to
work when I want to insert more than one value:

TAL=# create table b (a text primary key, b text);
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index 'b_pkey'
for table 'b'
CREATE TABLE
TAL=# insert into b select 'a';
INSERT 335311 1
TAL=# insert into b select 'b', select 'b';
ERROR: parser: parse error at or near "select" at character 27

Did I get the syntax wrong?

Thanks,

Jean-Christian Imbeault


From: "Reuben D(dot) Budiardja" <techlist(at)voyager(dot)phys(dot)utk(dot)edu>
To: Jean-Christian Imbeault <jc(at)mega-bucks(dot)co(dot)jp>
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 03:26:46
Message-ID: 200307012326.46173.techlist@voyager.phys.utk.edu
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

On Tuesday 01 July 2003 11:08 pm, Jean-Christian Imbeault wrote:
> Reuben D. Budiardja wrote:
> > INSERT INTO mytable
> > SELECT 'value1', 'value2'
> > WHERE NOT EXISTS
> > (SELECT NULL FROM mytable
> > WHERE mycondition)
>
> Thank you to everyone who helped out on my question. I am trying to
> implement the above solution but I'm having problems getting this to
> work when I want to insert more than one value:
>
> TAL=# create table b (a text primary key, b text);
> NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index 'b_pkey'
> for table 'b'
> CREATE TABLE
> TAL=# insert into b select 'a';
> INSERT 335311 1
> TAL=# insert into b select 'b', select 'b';
> ERROR: parser: parse error at or near "select" at character 27
>

I don't see what you're trying to do. Why do you have two select ?

RDB


From: Bruno Wolff III <bruno(at)wolff(dot)to>
To: Jean-Christian Imbeault <jc(at)mega-bucks(dot)co(dot)jp>
Cc: techlist(at)voyager(dot)phys(dot)utk(dot)edu, pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 03:32:40
Message-ID: 20030702033240.GA8524@wolff.to
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

On Wed, Jul 02, 2003 at 12:08:56 +0900,
Jean-Christian Imbeault <jc(at)mega-bucks(dot)co(dot)jp> wrote:
> TAL=# insert into b select 'b', select 'b';
> ERROR: parser: parse error at or near "select" at character 27

You probably want:
insert into b select 'b', 'b';


From: Ian Barwick <barwick(at)gmx(dot)net>
To: Jean-Christian Imbeault <jc(at)mega-bucks(dot)co(dot)jp>, Alvaro Herrera <alvherre(at)dcc(dot)uchile(dot)cl>
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 04:34:55
Message-ID: 200307020634.55147.barwick@gmx.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

On Wednesday 02 July 2003 02:58, Jean-Christian Imbeault wrote:
> Alvaro Herrera wrote:
> > No, only the "second" one will fail (though it's difficult which one is
> > the second)
>
> From:
>
> http://marc.theaimsgroup.com/?l=postgresql-general&m=105656988915991&w=2
>
> Ian Barwick wrote:
>
> [...]
>
> I proposed that same solution 3 years ago. Tom shoots it down:

(This quote is not from Mike Mascari, not me)

> [...]
>
> I couldn't get the link to work so I couldn't read why Tom shot it down.
> But if Tom shot down this idea down ... then it mustn't be correct.

see:
http://archives.postgresql.org/pgsql-general/2000-12/msg00970.php

entire thread:
http://archives.postgresql.org/pgsql-general/2000-12/msg00947.php

Ian Barwick
barwick(at)gmx(dot)net


From: "Nigel J(dot) Andrews" <nandrews(at)investsystems(dot)co(dot)uk>
To: Jean-Christian Imbeault <jc(at)mega-bucks(dot)co(dot)jp>
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 07:13:49
Message-ID: Pine.LNX.4.21.0307020811200.29474-100000@ponder.fairway2k.co.uk
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

On Wed, 2 Jul 2003, Jean-Christian Imbeault wrote:

> I have a table with a primary field and a few other fields. What is the
> fastest way to do an insert into that table assuming that sometimes I
> might try to insert a record with a duplicate primary key and want that
> to fail?
>
> I know that if I try a plain insert this will work, but in the case
> where I am trying to insert a duplicate key, the insert fails (as it
> should) and an error is logged.
>
> I could first do a check to see if there is already an entry with the
> same key as the one I am trying to insert but that would mean quite a
> few extra operations.
>
> Is there a quick and clean way of doing something like:
>
> INSERT into table values(...) IF there isn't already a row with pkey=x
>
> Thanks,
>
> Jean-Christian Imbeault
>
> PS The reason I am asking is that an open source project is using MySQL
> as their DB and they have a port to PG that isn't very clean b/c the DB
> code keeps trying to insert duplicate primary keys. According to them
> MySQL doesn't complain and just drops the insert whereas PG (as is
> right) complains. I've offered to clean up their PG insertion code but
> they say that they don't want too many extra checks as their app writes
> to the DB a *lot* and any extra check is going to slow down the
> application noticeably ...

Skipping ahead without reading all the messages in this thread (got to rush
out) what about using a before insert trigger, doing the check in there,
returning null if the insert would fail and see if they complain about the slow
down :)

Of course it's still got the race condition for the application unless you also
lock the table and it'll screw up any use of currval(sequence) afterwards that
expects to get the id of the row inserted with a id obtained from
nextval(sequence)

Nigel Andrews


From: Jean-Christian Imbeault <jc(at)mega-bucks(dot)co(dot)jp>
To: "Nigel J(dot) Andrews" <nandrews(at)investsystems(dot)co(dot)uk>
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 07:19:10
Message-ID: 3F02876E.7030707@mega-bucks.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

Nigel J. Andrews wrote:
>
> Skipping ahead without reading all the messages in this thread (got to rush
> out) what about using a before insert trigger, doing the check in there,
> returning null if the insert would fail and see if they complain about the slow
> down :)

Oooh! I think I like that .... there would be no penalty for the MySQL
version since this would be a trigger and only in the postgres database
version of the code.

Will a trigger still allow one insert to succeed if there are multiple
backends trying to insert the same primary key in a table? There must be
no 'lost' inserts ....

> Of course it's still got the race condition for the application unless you also
> lock the table and it'll screw up any use of currval(sequence) afterwards that
> expects to get the id of the row inserted with a id obtained from
> nextval(sequence)

I just want two things from any valid solution:

1- if there is an insert and there is not row with the new insert's
primary key then the insert is made. If there are multiple inserts one
succeeds. i.e. No valid inserts will be 'lost'.

2- reduce the number of error messages logged as a result of
'collisions' between two backends trying to insert duplicate primary key
rows.

I am not very familiar with triggers so I would very much appreciate any
feedback. But I think the use of a trigger might safe?

Thanks,

Jean-Christian Imbeault


From: "Shridhar Daithankar" <shridhar_daithankar(at)persistent(dot)co(dot)in>
To: pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 07:30:44
Message-ID: 3F02D77C.3125.4B161F4@localhost
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

On 2 Jul 2003 at 16:19, Jean-Christian Imbeault wrote:
> I just want two things from any valid solution:
>
> 1- if there is an insert and there is not row with the new insert's
> primary key then the insert is made. If there are multiple inserts one
> succeeds. i.e. No valid inserts will be 'lost'.

With postgresql, if you use transacations, no valid inserts are ever lost
unless there is a deadlock.
>
> 2- reduce the number of error messages logged as a result of
> 'collisions' between two backends trying to insert duplicate primary key
> rows.

Log of what? Postgresql will log all the errors. If you don't want to see them,
tune the logging option.

If your code is doing any checking for return value etc., check if it is a
duplicate message and discard it. That way you can minimize your application
logging.

Otherwise use select/insert behaviour referred earlier.

Besides if you are going to insert a duplicate rarely, why worry so much about
performance? And if your dulicates are bit too frequent for comfort, you might
have some more issues w.r.t database table design to look at.

Bye
Shridhar

--
And 1.1.81 is officially BugFree(tm), so if you receive any bug-reportson it,
you know they are just evil lies."(By Linus Torvalds,
Linus(dot)Torvalds(at)cs(dot)helsinki(dot)fi)


From: Jean-Christian Imbeault <jc(at)mega-bucks(dot)co(dot)jp>
To: shridhar_daithankar(at)persistent(dot)co(dot)in
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 07:52:33
Message-ID: 3F028F41.2060102@mega-bucks.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

Shridhar Daithankar wrote:

Again, please let me preface my email. The application in question is
not written by me. I am trying to find a simple and easily implementable
way of making the current implementation more postgres friendly :)

> With postgresql, if you use transacations, no valid inserts are ever lost
> unless there is a deadlock.

Just needed to make sure as I don't know much about triggers.

>>2- reduce the number of error messages logged as a result of
>>'collisions' between two backends trying to insert duplicate primary key
>>rows.
>
> Log of what? Postgresql will log all the errors. If you don't want to see them,
> tune the logging option.

I don't want to turn all error logging off, actually I don't want to
turn any error logging off :) If errors are logged it usually means
there is a bug some somewhere ...

> If your code is doing any checking for return value etc., check if it is a
> duplicate message and discard it. That way you can minimize your application
> logging.

It's not my application logging, it's postgres logging the fact that a
duplicate insertion was attempted:

ERROR: Cannot insert a duplicate key into unique index pot_documents_pkey

> Otherwise use select/insert behaviour referred earlier.

But the trigger is nicer since none of the application code would need
to be changed at all. I would just add a trigger to the database
creation script.

> And if your dulicates are bit too frequent for comfort, you might
> have some more issues w.r.t database table design to look at.

They are frequent, and I agree that there are issues with the database
design. But the application developers don't want to address them (if
it's broke why fix it kind of thing) ...

I just noticed that their app was generating a *lot* of log entries when
used with a postgres database and I offered to try and find a way of
reducing the number of errors generated.

Just trying to find a way to make *their* app play nice with postgres so
that people who do use their app will use it with postgres as a DB and
not that *other* DB ...

Jean-Christian Imbeault


From: "Shridhar Daithankar" <shridhar_daithankar(at)persistent(dot)co(dot)in>
To: pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 08:05:20
Message-ID: 3F02DF98.6553.4D10EA8@localhost
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

On 2 Jul 2003 at 16:52, Jean-Christian Imbeault wrote:
> They are frequent, and I agree that there are issues with the database
> design. But the application developers don't want to address them (if
> it's broke why fix it kind of thing) ...
>
> I just noticed that their app was generating a *lot* of log entries when
> used with a postgres database and I offered to try and find a way of
> reducing the number of errors generated.
>
> Just trying to find a way to make *their* app play nice with postgres so
> that people who do use their app will use it with postgres as a DB and
> not that *other* DB ...

Your error message is

ERROR: Cannot insert a duplicate key into unique index pot_documents_pkey

Just restart the postmaster and redirect log to script which ignores this
message. A simple pipe to grep -v would do the trick. No need to modify the
app, no need to nodify the db as well.

HTH

Bye
Shridhar

--
COBOL: An exercise in Artificial Inelegance.


From: Jean-Christian Imbeault <jc(at)mega-bucks(dot)co(dot)jp>
To: shridhar_daithankar(at)persistent(dot)co(dot)in
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 08:09:50
Message-ID: 3F02934E.4000608@mega-bucks.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

Shridhar Daithankar wrote:
>
> Just restart the postmaster and redirect log to script which ignores this
> message. A simple pipe to grep -v would do the trick. No need to
modify the
> app, no need to nodify the db as well.

True but each and every user would need to redirect their log, pipe to a
filter, etc ...

With a a trigger it's built into the app. You install the app and that's
it. It's a lot more user friendly, no?

Jean-Christian Imbeault


From: "Shridhar Daithankar" <shridhar_daithankar(at)persistent(dot)co(dot)in>
To: pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 08:17:49
Message-ID: 3F02E285.3763.4DC7CFD@localhost
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

On 2 Jul 2003 at 17:09, Jean-Christian Imbeault wrote:

> Shridhar Daithankar wrote:
> >
> > Just restart the postmaster and redirect log to script which ignores this
> > message. A simple pipe to grep -v would do the trick. No need to
> modify the
> > app, no need to nodify the db as well.
>
> True but each and every user would need to redirect their log, pipe to a
> filter, etc ...

No. It's just your server log.

> With a a trigger it's built into the app. You install the app and that's
> it. It's a lot more user friendly, no?

Triggers are not built into app. They are built in database so every app. would
automatically get them without restarting database or application.

Bye
Shridhar

--
Vulcans never bluff. -- Spock, "The Doomsday Machine", stardate 4202.1


From: Jean-Christian Imbeault <jc(at)mega-bucks(dot)co(dot)jp>
To: shridhar_daithankar(at)persistent(dot)co(dot)in
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 08:20:56
Message-ID: 3F0295E8.60600@mega-bucks.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

Shridhar Daithankar wrote:
>
> No. It's just your server log.

I was assuming that any users of this app who want it using postgres as
a database would want to stop these error messages from being logged.
(The developers know that these error messages are being logged, but
they don't see them as real errors)

> Triggers are not built into app. They are built in database so every
app. would
> automatically get them without restarting database or application.

Sorry I forgot to mention that the app comes with a script to built the
necessary tables, etc. So I would add to that script to create the
necessary trigger.

Jc


From: "Shridhar Daithankar" <shridhar_daithankar(at)persistent(dot)co(dot)in>
To: pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 08:23:17
Message-ID: 3F02E3CD.20970.4E1811B@localhost
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

On 2 Jul 2003 at 17:20, Jean-Christian Imbeault wrote:

> Shridhar Daithankar wrote:
> >
> > No. It's just your server log.
>
> I was assuming that any users of this app who want it using postgres as
> a database would want to stop these error messages from being logged.
> (The developers know that these error messages are being logged, but
> they don't see them as real errors)

Well, with grep -v in log pipe, you are just preventing these messages from
reaching log file. If a client does error checking. He would get the message
anyway.

Bye
Shridhar

--
One-Shot Case Study, n.: The scientific equivalent of the four-leaf clover, from which it is concluded all clovers possess four leaves and are sometimes green.


From: "Nigel J(dot) Andrews" <nandrews(at)investsystems(dot)co(dot)uk>
To: Shridhar Daithankar <shridhar_daithankar(at)persistent(dot)co(dot)in>
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 08:30:34
Message-ID: Pine.LNX.4.21.0307020929090.29474-100000@ponder.fairway2k.co.uk
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

On Wed, 2 Jul 2003, Shridhar Daithankar wrote:

> On 2 Jul 2003 at 17:20, Jean-Christian Imbeault wrote:
>
> > Shridhar Daithankar wrote:
> > >
> > > No. It's just your server log.
> >
> > I was assuming that any users of this app who want it using postgres as
> > a database would want to stop these error messages from being logged.
> > (The developers know that these error messages are being logged, but
> > they don't see them as real errors)
>
> Well, with grep -v in log pipe, you are just preventing these messages from
> reaching log file. If a client does error checking. He would get the message
> anyway.

You've also got the issue of the same pgsql cluster being used for more than
one database and the potential for filtering out ERROR messages from other dbs
in the cluster that really shouldn't be filtered out.

--
Nigel J. Andrews


From: "Shridhar Daithankar" <shridhar_daithankar(at)persistent(dot)co(dot)in>
To: pgsql-general(at)postgresql(dot)org
Subject: Re: Duplicate key insert question
Date: 2003-07-02 08:34:36
Message-ID: 3F02E674.27715.4EBDD15@localhost
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

On 2 Jul 2003 at 9:30, Nigel J. Andrews wrote:

> > Well, with grep -v in log pipe, you are just preventing these messages from
> > reaching log file. If a client does error checking. He would get the message
> > anyway.
> You've also got the issue of the same pgsql cluster being used for more than
> one database and the potential for filtering out ERROR messages from other dbs
> in the cluster that really shouldn't be filtered out.

Well, if you put the entire error message in grep string, that should not
affect much as it contains index name. Of course, unless you have same index in
two different databases. Tough luck then..

Bye
Shridhar

--
Fun experiments: Get a can of shaving cream, throw it in a freezer for about a
week. Then take it out, peel the metal off and put it where you want...
bedroom, car, etc. As it thaws, it expands an unbelievable amount.