Re: new version of contrib-intarray

Mark,

we prepared new version of contrib-intarray -
index support for 1-D integer arrays using GiST.

Changes:

- Improved regression test
- Current implementation provides index support for one-dimensional
array of int4's - gist__int_ops, suitable for small and medium size
of arrays, and gist__intbig_ops for indexing large arrays
(we use superimposed signature with length of 4096 bits to represent sets,
see Sven Helmer,1997).

Archive is available from
http://www.sai.msu.su/~megera/postgres/gist/code/7.1/contrib-intarray.tar.gz

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg(at)sai(dot)msu(dot)su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Oleg, do you want this in /contrib for 7.1?

> Mark,
>
> we prepared new version of contrib-intarray -
> index support for 1-D integer arrays using GiST.
>
> Changes:
>
>
> - Improved regression test
> - Current implementation provides index support for one-dimensional
> array of int4's - gist__int_ops, suitable for small and medium size
> of arrays, and gist__intbig_ops for indexing large arrays
> (we use superimposed signature with length of 4096 bits to represent sets,
> see Sven Helmer,1997).
>
> Archive is available from
> http://www.sai.msu.su/~megera/postgres/gist/code/7.1/contrib-intarray.tar.gz
>
> Regards,
> Oleg
> _____________________________________________________________
> Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
> Sternberg Astronomical Institute, Moscow University (Russia)
> Internet: oleg(at)sai(dot)msu(dot)su, http://www.sai.msu.su/~megera/
> phone: +007(095)939-16-83, +007(095)939-23-83
>
>

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

On Sat, 27 Jan 2001, Bruce Momjian wrote:

>
> Oleg, do you want this in /contrib for 7.1?

yes, if it's possible.
btw, is there way to specify default ops for index ?
We have two methods of index creation for intarrays and
would like to define which should be used by default

>
> > Mark,
> >
> > we prepared new version of contrib-intarray -
> > index support for 1-D integer arrays using GiST.
> >
> > Changes:
> >
> >
> > - Improved regression test
> > - Current implementation provides index support for one-dimensional
> > array of int4's - gist__int_ops, suitable for small and medium size
> > of arrays, and gist__intbig_ops for indexing large arrays
> > (we use superimposed signature with length of 4096 bits to represent sets,
> > see Sven Helmer,1997).
> >
> > Archive is available from
> > http://www.sai.msu.su/~megera/postgres/gist/code/7.1/contrib-intarray.tar.gz
> >
> > Regards,
> > Oleg
> > _____________________________________________________________
> > Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
> > Sternberg Astronomical Institute, Moscow University (Russia)
> > Internet: oleg(at)sai(dot)msu(dot)su, http://www.sai.msu.su/~megera/
> > phone: +007(095)939-16-83, +007(095)939-23-83
> >
> >
>
>
>

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg(at)sai(dot)msu(dot)su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Oleg Bartunov <oleg(at)sai(dot)msu(dot)su> writes:
> btw, is there way to specify default ops for index ?

Sure, that's what pg_opclass is for. Just insert the opclass name
and the OID of the type you want it to be the default index opclass
for.

regards, tom lane

On Sat, 27 Jan 2001, Tom Lane wrote:

> Oleg Bartunov <oleg(at)sai(dot)msu(dot)su> writes:
> > btw, is there way to specify default ops for index ?
>
> Sure, that's what pg_opclass is for. Just insert the opclass name
> and the OID of the type you want it to be the default index opclass
> for.

Tom, we already did this. Here is what we have in pg_opclass:

gist__int_ops | 1007
gist__intbig_ops | 1007
(32 rows)

we want gist__int_ops to be default index opclass.
If we delete gist__intbig_ops entry from opclass, then we couldn't use
gist__intbig_ops !

Regards,
Oleg

>
> regards, tom lane
>

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg(at)sai(dot)msu(dot)su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Oleg Bartunov <oleg(at)sai(dot)msu(dot)su> writes:
> gist__int_ops | 1007
> gist__intbig_ops | 1007

> we want gist__int_ops to be default index opclass.
> If we delete gist__intbig_ops entry from opclass, then we couldn't use
> gist__intbig_ops !

Put in gist__intbig_ops with zero for the default type. You should
never have more than one entry in pg_opclass claiming to be the default
for a given type OID.

regards, tom lane

Seems we have an older version in CVS. I will update it now. I assume
/contrib is available for changes up until release, as usual.

> Mark,
>
> we prepared new version of contrib-intarray -
> index support for 1-D integer arrays using GiST.
>
> Changes:
>
>
> - Improved regression test
> - Current implementation provides index support for one-dimensional
> array of int4's - gist__int_ops, suitable for small and medium size
> of arrays, and gist__intbig_ops for indexing large arrays
> (we use superimposed signature with length of 4096 bits to represent sets,
> see Sven Helmer,1997).
>
> Archive is available from
> http://www.sai.msu.su/~megera/postgres/gist/code/7.1/contrib-intarray.tar.gz
>
> Regards,
> Oleg
> _____________________________________________________________
> Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
> Sternberg Astronomical Institute, Moscow University (Russia)
> Internet: oleg(at)sai(dot)msu(dot)su, http://www.sai.msu.su/~megera/
> phone: +007(095)939-16-83, +007(095)939-23-83
>
>

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Installed in CVS. Thanks.

> Mark,
>
> we prepared new version of contrib-intarray -
> index support for 1-D integer arrays using GiST.
>
> Changes:
>
>
> - Improved regression test
> - Current implementation provides index support for one-dimensional
> array of int4's - gist__int_ops, suitable for small and medium size
> of arrays, and gist__intbig_ops for indexing large arrays
> (we use superimposed signature with length of 4096 bits to represent sets,
> see Sven Helmer,1997).
>
> Archive is available from
> http://www.sai.msu.su/~megera/postgres/gist/code/7.1/contrib-intarray.tar.gz
>
> Regards,
> Oleg
> _____________________________________________________________
> Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
> Sternberg Astronomical Institute, Moscow University (Russia)
> Internet: oleg(at)sai(dot)msu(dot)su, http://www.sai.msu.su/~megera/
> phone: +007(095)939-16-83, +007(095)939-23-83
>
>

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Bruce Momjian writes:

> Installed in CVS. Thanks.

You overwrote the changes that other people have made meanwhile.

>
> > Mark,
> >
> > we prepared new version of contrib-intarray -
> > index support for 1-D integer arrays using GiST.
> >
> > Changes:
> >
> >
> > - Improved regression test
> > - Current implementation provides index support for one-dimensional
> > array of int4's - gist__int_ops, suitable for small and medium size
> > of arrays, and gist__intbig_ops for indexing large arrays
> > (we use superimposed signature with length of 4096 bits to represent sets,
> > see Sven Helmer,1997).
> >
> > Archive is available from
> > http://www.sai.msu.su/~megera/postgres/gist/code/7.1/contrib-intarray.tar.gz
> >
> > Regards,
> > Oleg
> > _____________________________________________________________
> > Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
> > Sternberg Astronomical Institute, Moscow University (Russia)
> > Internet: oleg(at)sai(dot)msu(dot)su, http://www.sai.msu.su/~megera/
> > phone: +007(095)939-16-83, +007(095)939-23-83
> >
> >
>
>
>

--
Peter Eisentraut peter_e(at)gmx(dot)net http://yi.org/peter-e/

I checked README.intarray to see what the most recent date was, and it
was Jan 10, so I knew that this version was newer. I then did a diff -c
against the current CVS and I didn't see anything unusual in the
changes.

Attached is the CVS diff command line showing me all the changes made:

cvs diff -c -D '2001-01-13 00:00:00' -D'2001-03-16 00:00:00' .

I see change of += in CFLAGS (harmless), movement of #include
<postgres.h>, and removal of // comments, which don't appear anymore in
the code.

Do you see anything else? This one was easy to track because it was
installed only recently, but other /contrib stuff is much tougher
because you never know what the original install date was.

I usually only look at Makefile changes in /contrib because that is
where most of the customization is, and I don't see any changes made to
Makefile by the patch. It doesn't even touch the CFLAGS += change.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Bruce Momjian writes:

> I see change of += in CFLAGS (harmless),

Not.

> movement of #include
> <postgres.h>, and removal of // comments, which don't appear anymore in
> the code.

I only saw that the Makefile is back to how it looked at rev 1.1 before I
did some work on it. AFAICT the Makefile should be reverted back to the
previous revision, since the code change does not require any changes to
the Makefile.

--
Peter Eisentraut peter_e(at)gmx(dot)net http://yi.org/peter-e/

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> I only saw that the Makefile is back to how it looked at rev 1.1 before I
> did some work on it. AFAICT the Makefile should be reverted back to the
> previous revision, since the code change does not require any changes to
> the Makefile.

I did this, also reinstalled the include-file changes I had made, and
then spent several fruitless hours trying to find why the "intbig" index
operators fail selftest here (on HP-PA). I suppose it's a portability
problem, since presumably they pass for Oleg ... but I don't see it.

Who else finds that the new contrib/intarray code passes or fails its
selftest, and on what platforms?

regards, tom lane

I wrote:
> I did this, also reinstalled the include-file changes I had made, and
> then spent several fruitless hours trying to find why the "intbig" index
> operators fail selftest here (on HP-PA). I suppose it's a portability
> problem, since presumably they pass for Oleg ... but I don't see it.

Further experimentation shows that intbig fails selftest on ALL
platforms under 7.1. I see the problem: the intarray operators are
mostly unprepared to cope with TOASTed input arrays. In particular,
_intbig_union() generates an erroneous "null" result for a compressed
input array, leading to completely incorrect GiST index trees in the
self-test example.

A somewhat-related error in this code is that some routines feel free
to scribble on their input. This is tres uncool, because they may be
scribbling on disk buffers. Example:

regression=# create table foo(f1 int4[]);
CREATE
regression=# insert into foo values ('{10,1,2,1,4}');
INSERT 150265 1
regression=# select * from foo;
f1
--------------
{10,1,2,1,4}
(1 row)

regression=# select * from foo where f1 && '{4}';
f1
--------------
{1,1,2,4,10}
(1 row)

regression=# select * from foo;
f1
--------------
{1,1,2,4,10}
(1 row)

And you thought SELECT was a read-only operation ...

I do not have time to work on this stuff now, but as it stands the
contrib/intarray code is unusable in 7.1. Unless Oleg can find the
time to fix these issues before release, I will recommend that we
not ship contrib/intarray in 7.1.

regards, tom lane

I just returned from vacation and identified the problem.
We'll fix it.

Regards,
Oleg
On Sun, 18 Mar 2001, Tom Lane wrote:

> I wrote:
> > I did this, also reinstalled the include-file changes I had made, and
> > then spent several fruitless hours trying to find why the "intbig" index
> > operators fail selftest here (on HP-PA). I suppose it's a portability
> > problem, since presumably they pass for Oleg ... but I don't see it.
>
> Further experimentation shows that intbig fails selftest on ALL
> platforms under 7.1. I see the problem: the intarray operators are
> mostly unprepared to cope with TOASTed input arrays. In particular,
> _intbig_union() generates an erroneous "null" result for a compressed
> input array, leading to completely incorrect GiST index trees in the
> self-test example.
>
> A somewhat-related error in this code is that some routines feel free
> to scribble on their input. This is tres uncool, because they may be
> scribbling on disk buffers. Example:
>
>
> regression=# create table foo(f1 int4[]);
> CREATE
> regression=# insert into foo values ('{10,1,2,1,4}');
> INSERT 150265 1
> regression=# select * from foo;
> f1
> --------------
> {10,1,2,1,4}
> (1 row)
>
> regression=# select * from foo where f1 && '{4}';
> f1
> --------------
> {1,1,2,4,10}
> (1 row)
>
> regression=# select * from foo;
> f1
> --------------
> {1,1,2,4,10}
> (1 row)
>
>
> And you thought SELECT was a read-only operation ...
>
> I do not have time to work on this stuff now, but as it stands the
> contrib/intarray code is unusable in 7.1. Unless Oleg can find the
> time to fix these issues before release, I will recommend that we
> not ship contrib/intarray in 7.1.
>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/users-lounge/docs/faq.html
>

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg(at)sai(dot)msu(dot)su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Hi,

New version of contrib-intarray is available from
http://www.sai.msu.su/~megera/postgres/gist/code/7.1/contrib-intarray.tar.gz

>From README.intarray:

March 19, 2001
1. Added support for toastable keys
2. Improved split algorithm for intbig (selection speedup is about 30%)

Regards,
Oleg

On Mon, 19 Mar 2001, Oleg Bartunov wrote:

> I just returned from vacation and identified the problem.
> We'll fix it.
>
> Regards,
> Oleg
> On Sun, 18 Mar 2001, Tom Lane wrote:
>
> > I wrote:
> > > I did this, also reinstalled the include-file changes I had made, and
> > > then spent several fruitless hours trying to find why the "intbig" index
> > > operators fail selftest here (on HP-PA). I suppose it's a portability
> > > problem, since presumably they pass for Oleg ... but I don't see it.
> >
> > Further experimentation shows that intbig fails selftest on ALL
> > platforms under 7.1. I see the problem: the intarray operators are
> > mostly unprepared to cope with TOASTed input arrays. In particular,
> > _intbig_union() generates an erroneous "null" result for a compressed
> > input array, leading to completely incorrect GiST index trees in the
> > self-test example.
> >
> > A somewhat-related error in this code is that some routines feel free
> > to scribble on their input. This is tres uncool, because they may be
> > scribbling on disk buffers. Example:
> >
> >
> > regression=# create table foo(f1 int4[]);
> > CREATE
> > regression=# insert into foo values ('{10,1,2,1,4}');
> > INSERT 150265 1
> > regression=# select * from foo;
> > f1
> > --------------
> > {10,1,2,1,4}
> > (1 row)
> >
> > regression=# select * from foo where f1 && '{4}';
> > f1
> > --------------
> > {1,1,2,4,10}
> > (1 row)
> >
> > regression=# select * from foo;
> > f1
> > --------------
> > {1,1,2,4,10}
> > (1 row)
> >
> >
> > And you thought SELECT was a read-only operation ...
> >
> > I do not have time to work on this stuff now, but as it stands the
> > contrib/intarray code is unusable in 7.1. Unless Oleg can find the
> > time to fix these issues before release, I will recommend that we
> > not ship contrib/intarray in 7.1.
> >
> > regards, tom lane
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 5: Have you checked our extensive FAQ?
> >
> > http://www.postgresql.org/users-lounge/docs/faq.html
> >
>
> Regards,
> Oleg
> _____________________________________________________________
> Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
> Sternberg Astronomical Institute, Moscow University (Russia)
> Internet: oleg(at)sai(dot)msu(dot)su, http://www.sai.msu.su/~megera/
> phone: +007(095)939-16-83, +007(095)939-23-83
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/users-lounge/docs/faq.html
>

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg(at)sai(dot)msu(dot)su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Sorry, I have again messed up this Makefile, and I am glad Tom has put
things back.

Seems I am no longer capable of understanding the affects of such
changes as this:

! override CPPFLAGS := -I$(srcdir) $(CPPFLAGS) -DPGSQL71

! override CPPFLAGS += -I$(srcdir) -DPGSQL71

Having $(srcdir) before $(CPPFLAGS) must be significant.

I am going to make a proposal that we partition off certain areas to be
handled by certain people so I don't mess things up again. Look for my
email in a few minutes.

Bruce Momjian writes:
>
> > I see change of += in CFLAGS (harmless),
>
> Not.
>
> > movement of #include
> > <postgres.h>, and removal of // comments, which don't appear anymore in
> > the code.
>
> I only saw that the Makefile is back to how it looked at rev 1.1 before I
> did some work on it. AFAICT the Makefile should be reverted back to the
> previous revision, since the code change does not require any changes to
> the Makefile.
>
> --
> Peter Eisentraut peter_e(at)gmx(dot)net http://yi.org/peter-e/
>
>

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Oleg Bartunov <oleg(at)sai(dot)msu(dot)su> writes:
> New version of contrib-intarray is available from
> http://www.sai.msu.su/~megera/postgres/gist/code/7.1/contrib-intarray.tar.gz

Got it, will review it ASAP ...

regards, tom lane

> Bruce Momjian writes:
>
> > Seems I am no longer capable of understanding the affects of such
> > changes as this:
> >
> > ! override CPPFLAGS := -I$(srcdir) $(CPPFLAGS) -DPGSQL71
> >
> > ! override CPPFLAGS += -I$(srcdir) -DPGSQL71
> >
> > Having $(srcdir) before $(CPPFLAGS) must be significant.
>
> The CVS log message says:
>
> : Make sure -L and -I's for our source tree are always before system include
> : or library directories on the command line.
>
> That's all this does.

Makes sense. See my new email about patches. I think I need to rely
more on you and others to help me evaluate these changes. We have much
more expertice in the group than I can hope to comprehend.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Bruce Momjian writes:

> Seems I am no longer capable of understanding the affects of such
> changes as this:
>
> ! override CPPFLAGS := -I$(srcdir) $(CPPFLAGS) -DPGSQL71
>
> ! override CPPFLAGS += -I$(srcdir) -DPGSQL71
>
> Having $(srcdir) before $(CPPFLAGS) must be significant.

The CVS log message says:

: Make sure -L and -I's for our source tree are always before system include
: or library directories on the command line.

That's all this does.

--
Peter Eisentraut peter_e(at)gmx(dot)net http://yi.org/peter-e/

Oleg Bartunov <oleg(at)sai(dot)msu(dot)su> writes:
> New version of contrib-intarray is available from
> http://www.sai.msu.su/~megera/postgres/gist/code/7.1/contrib-intarray.tar.gz

I have made some further changes (you weren't quite there on not
scribbling on input datums, for example) and committed the updates.
Thanks!

regards, tom lane

Oleg, can you grab the CVS copy and use that for further patches?
Thanks.

> Oleg Bartunov <oleg(at)sai(dot)msu(dot)su> writes:
> > New version of contrib-intarray is available from
> > http://www.sai.msu.su/~megera/postgres/gist/code/7.1/contrib-intarray.tar.gz
>
> I have made some further changes (you weren't quite there on not
> scribbling on input datums, for example) and committed the updates.
> Thanks!
>
> regards, tom lane
>

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026