Re: APR 1.0 released

Hi all,
now that Apache Portable Runtime was release why don't
use it on Postgres?

Regards
Gaetano Mendola

Gaetano Mendola wrote:

> Hi all,
> now that Apache Portable Runtime was release why don't
> use it on Postgres?
>

Now that we have discovered the formula for green cheese, why don't we
remake the moon out of it?

cheers

andrew

On Sat, 4 Sep 2004, Gaetano Mendola wrote:

> Hi all,
> now that Apache Portable Runtime was release why don't
> use it on Postgres?

Short question: why? what does it give us, other then potential reliance
on another project to build ... ?

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy(at)hub(dot)org Yahoo!: yscrappy ICQ: 7615664

Gaetano Mendola <mendola(at)bigfoot(dot)com> writes:
> now that Apache Portable Runtime was release why don't
> use it on Postgres?

The sense of the question is backwards. Why *should* we use it?

regards, tom lane

Centuries ago, Nostradamus foresaw when scrappy(at)postgresql(dot)org ("Marc G. Fournier") would write:
> On Sat, 4 Sep 2004, Gaetano Mendola wrote:
>> Hi all,
>> now that Apache Portable Runtime was release why don't
>> use it on Postgres?
>
> Short question: why? what does it give us, other then potential
> reliance on another project to build ... ?

It would allow reopening all the threads about why PostgreSQL doesn't
use threading...

That being said, there are places where there would be merit to using
threading in PostgreSQL, though NOT where the usual futile discussions
ask for it. Notably, on an SMP system, it would be a neat idea for
complex queries involving joins to split themselves so that different
parts run in separate threads.

The other Way, Way Cool part would be for queries that are scanning
big tables to split the scans into unions of partial scans, so that on
an 8 CPU box you'd take the "Big 4GB Table" and have 8 threads
simultaneously scanning different parts of it. (And making ARC all
the more important :-).)

But that would, however it happened, involve BIG, SCARY changes...

... And since APR isn't BSD licensed, that would probably cause a
problem.
--
(format nil "~S(at)~S" "cbbrowne" "acm.org")
http://www3.sympatico.ca/cbbrowne/wp.html
Of course, unless one has a theory, one cannot expect much help from a
computer unless _it_ has a theory)... -- Marvin Minsky

Tom Lane wrote:

> Gaetano Mendola <mendola(at)bigfoot(dot)com> writes:
>
>>now that Apache Portable Runtime was release why don't
>>use it on Postgres?
>
>
> The sense of the question is backwards. Why *should* we use it?

In order to avoid all the annoyance that someone else had in
write code portable. I mean, how much time ( I'm not a postgres
developer, I like to think, for lack of time ) was spent in order
to port postgres to win32 ? Don't you think that use of APR could
save time ?

Andrew: about the green cheese, why not remake the moon with it
if this have some benefit ?

Marc: you are not obliged to change APR version each eye blink.
Don't you think that use a portable library could save time ?

One example for all: actually postgres is multi process, some time
I see my server with 3 CPU in holiday and one overloaded to sort
thousand rows. Don't you think in some cases spawn a couple of
thread could improve it ? Let me dream that you agree on this and
may be in years someone start to do it ( I'm using postgres since
when "create or replace function" or "table functions" was a blasphemy
so I'm sure that will happen). What are you going to do? Reinvent
the hell and create a sort of framework to work with thread dealing
with Win32 world ? I don't know if APR provide a spin lock mechanism,
tell me how many times did you see discussion here on hackers about
on how make the spin lock effective?
In my experience I'm a C++ developer and each time I have to do
something I full rely on STL, BOOST, XALAN, XERCES and may be I'll
use the APR now that seem stable enough and I swear each time my
colleagues are reinventing the list, queue, thread interactions....

Regards
Gaetano Mendola

Gaetano Mendola <mendola(at)bigfoot(dot)com> writes:
> Don't you think that use of APR could save time ?

No, because we've already *done* the work it would purport to save.
It would cost us work to adapt our code to sit on top of APR, and
it's not clear to me that we'd be getting anything for it.

IIRC, this was proposed before and we looked at APR in some detail,
and came to the conclusion that it wouldn't be worth changing. See
the archives.

> Don't you think in some cases spawn a couple of
> thread could improve it ?

The fact that we were on top of APR would not automagically mean that
we could thread-ize the backend, nor that we would want to.

> I don't know if APR provide a spin lock mechanism,

You don't even know that, but you're confident that we can throw away
our spinlock work and use APR anyway. You're wasting our time. Get
some evidence if you want to propose this.

regards, tom lane

Christopher Browne wrote:

> ... And since APR isn't BSD licensed, that would probably cause a
> problem.

They are changin license for APR and I'll be not surprised if they
adopth the BSD one.

Regards
Gaetano Mendola

Gaetano Mendola wrote:

> Tom Lane wrote:
>
>> Gaetano Mendola <mendola(at)bigfoot(dot)com> writes:
>>
>>> now that Apache Portable Runtime was release why don't
>>> use it on Postgres?
>>
>>
>>
>> The sense of the question is backwards. Why *should* we use it?
>
>
> In order to avoid all the annoyance that someone else had in
> write code portable. I mean, how much time ( I'm not a postgres
> developer, I like to think, for lack of time ) was spent in order
> to port postgres to win32 ? Don't you think that use of APR could
> save time ?
>
> Andrew: about the green cheese, why not remake the moon with it
> if this have some benefit ?
>
>

Go and study the history of how long it took the Apache people to get
APR done. Look at the history of the various MPMs. By contrast, we got
our Windows port done in rather less than a year, partly by *not* going
down ratholes like APR. Now it's true that they had a different (and
harder) set of problems to deal with - in particular scaling to huge
numbers of very short-lived connections. Even so, it took them a very
long time (years and years) to get right, and they still use a different
MPM by default on Windows from what they use on Unix - and you have to
choose it at configure time. I am not crtiticizing the Apache people - I
am just saying there is no evidence that using APR would have any
benefit at all for PostgreSQL - and it would be massively invasive and
require huge effort to do so.

cheers

andrew

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tom Lane wrote:

| Gaetano Mendola <mendola(at)bigfoot(dot)com> writes:
|>I don't know if APR provide a spin lock mechanism,
|
|
| You don't even know that, but you're confident that we can throw away
| our spinlock work and use APR anyway. You're wasting our time. Get
| some evidence if you want to propose this.

I'm sorry to have wasted your time.

With the experience you have I don't think you need my evidences.

Regards
Gaetano Mendola

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (MingW32)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBOlKM7UpzwH2SGd4RAshqAJ9L1MeBsWlU/G2I1DRoA1/GifMj9ACg/RWX
IInlsOU+Z78LSyyL3maBC3Q=
=BcID
-----END PGP SIGNATURE-----

On Sun, 5 Sep 2004, Gaetano Mendola wrote:

> Christopher Browne wrote:
>
>
>> ... And since APR isn't BSD licensed, that would probably cause a
>> problem.
>
> They are changin license for APR and I'll be not surprised if they
> adopth the BSD one.

Since Apache has developed their own license, and I've seen at least one
non-Apache project (Spamassassin) switch over to it so far, I would be
very surprised is APR switched a non-Apache license ...

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy(at)hub(dot)org Yahoo!: yscrappy ICQ: 7615664

Gaetano Mendola wrote:
[ PGP not available, raw data follows ]
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Tom Lane wrote:
>
> | Gaetano Mendola <mendola(at)bigfoot(dot)com> writes:
> |>I don't know if APR provide a spin lock mechanism,
> |
> |
> | You don't even know that, but you're confident that we can throw away
> | our spinlock work and use APR anyway. You're wasting our time. Get
> | some evidence if you want to propose this.
>
> I'm sorry to have wasted your time.
>
> With the experience you have I don't think you need my evidences.

I looked at the APR code to get some ideas for the Win32 port. Some of
the ideas were good, but in other places like rename they didn't do very
well we were better off doing it ourselves and getting it right.

I remember looking at their code to fix the rename/unlink while the file
is open problem, and they didn't seem to have a fix for that so we
developed our own method that works like Unix.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Bruce Momjian schrieb:
> I looked at the APR code to get some ideas for the Win32 port. Some of
> the ideas were good, but in other places like rename they didn't do very
> well we were better off doing it ourselves and getting it right.
>
> I remember looking at their code to fix the rename/unlink while the file
> is open problem, and they didn't seem to have a fix for that so we
> developed our own method that works like Unix.

sorry, but your rename doesn't work on cygwin. maybe it works with mingw.

cygwin has it's own and working way of doing rename's.
maybe you should have looked at the cygwin sources instead.
(src/winsup/cygwin/syscalls.cc)

first doing a WinAPI MoveFileEx and then after a failure trying the
cygwin version, which will also try its own MoveFile loop, will not
work. they are conflicting.

same with unlink, but at least the mingw and cygwin unlink versions
don't conflict here. here you don't stack two conflicting loops together.
nevertheless cygwin's unlink is much better than pgunlink in case of
locking problems. it does its own sort of delayed removal then.

IMHO port/dirmod.c is a dirty and half-baked hack, which works for mingw
only.
--
Reini Urban
http://xarch.tu-graz.ac.at/home/rurban/

Reini Urban said:
> Bruce Momjian schrieb:
>> I looked at the APR code to get some ideas for the Win32 port. Some
>> of the ideas were good, but in other places like rename they didn't do
>> very well we were better off doing it ourselves and getting it right.
>>
>> I remember looking at their code to fix the rename/unlink while the
>> file is open problem, and they didn't seem to have a fix for that so
>> we developed our own method that works like Unix.
>
> sorry, but your rename doesn't work on cygwin. maybe it works with
> mingw.
>
> cygwin has it's own and working way of doing rename's.
> maybe you should have looked at the cygwin sources instead.
> (src/winsup/cygwin/syscalls.cc)
>
> first doing a WinAPI MoveFileEx and then after a failure trying the
> cygwin version, which will also try its own MoveFile loop, will not
> work. they are conflicting.
>
> same with unlink, but at least the mingw and cygwin unlink versions
> don't conflict here. here you don't stack two conflicting loops
> together. nevertheless cygwin's unlink is much better than pgunlink in
> case of locking problems. it does its own sort of delayed removal
> then.
>
> IMHO port/dirmod.c is a dirty and half-baked hack, which works for
> mingw only.

Are you sure you are reading this code correctly? Your reading would only be
correct if WIN32 is defined on Cygwin - it isn't IIRC (don't have a
convenient way to test ATM). The relevant code is this:

#ifdef WIN32
while (!MoveFileEx(from, to, MOVEFILE_REPLACE_EXISTING))
#endif
#ifdef __CYGWIN__
while (rename(from, to) < 0)
#endif

If the code doesn't work, please submit empirical proof, rather than make
assertions of "half-baked hack". If it's broken we'll fix it. Bruce's point
about the usefulness of APR remains, nonetheless.

cheers

andrew

Andrew Dunstan schrieb:
> Reini Urban said:
>>Bruce Momjian schrieb:
>>
>>>I looked at the APR code to get some ideas for the Win32 port. Some
>>>of the ideas were good, but in other places like rename they didn't do
>>>very well we were better off doing it ourselves and getting it right.
>>>
>>>I remember looking at their code to fix the rename/unlink while the
>>>file is open problem, and they didn't seem to have a fix for that so
>>>we developed our own method that works like Unix.
>>
>>sorry, but your rename doesn't work on cygwin. maybe it works with
>>mingw.
>>
>>cygwin has it's own and working way of doing rename's.
>>maybe you should have looked at the cygwin sources instead.
>>(src/winsup/cygwin/syscalls.cc)
>>
>>first doing a WinAPI MoveFileEx and then after a failure trying the
>>cygwin version, which will also try its own MoveFile loop, will not
>>work. they are conflicting.
>>
>>same with unlink, but at least the mingw and cygwin unlink versions
>>don't conflict here. here you don't stack two conflicting loops
>>together. nevertheless cygwin's unlink is much better than pgunlink in
>>case of locking problems. it does its own sort of delayed removal
>>then.
>>
>>IMHO port/dirmod.c is a dirty and half-baked hack, which works for
>>mingw only.
>
>
>
> Are you sure you are reading this code correctly? Your reading would only be
> correct if WIN32 is defined on Cygwin - it isn't IIRC (don't have a
> convenient way to test ATM). The relevant code is this:
>
> #ifdef WIN32
> while (!MoveFileEx(from, to, MOVEFILE_REPLACE_EXISTING))
> #endif
> #ifdef __CYGWIN__
> while (rename(from, to) < 0)
> #endif
>
> If the code doesn't work, please submit empirical proof, rather than make
> assertions of "half-baked hack". If it's broken we'll fix it. Bruce's point
> about the usefulness of APR remains, nonetheless.

I already posted my needed patches to make beta2 work on cygwin.
But on the pqsql-cygwin mailinglist:
http://xarch.tu-graz.ac.at/home/rurban/software/cygwin/postgresql/postgresql-8.0.0beta2-1
Only a plperl problem is pending. BTW: plperl never worked on cygwin before.

FYI: WIN32 is also defined because <windows.h> is included.
(/usr/incluse/w32api/windef.h)
If you want this or that, do proper nesting, and use #else.

#ifdef __CYGWIN__
while (rename(from, to) < 0)
#else
#ifdef WIN32
while (!MoveFileEx(from, to, MOVEFILE_REPLACE_EXISTING))
#endif
#endif

You cannot safely assume WIN32 is only defined on mingw, but not on
__CYGWIN__. And you need <windows.h> because of some winapi calls below.
The same false assumption was also in src/include/utils/palloc.h

But the whole pgrename #ifdef is fragile and a mess.
cygwin rename works good enough, and I just #ifdef'ed it away.
The two #undef need to be inserted before #include <unistd.h>, otherwise
pgrename will be declared, but rename not.

gcc -E -I../include dirmod-orig.c:
int
pgrename(const char *from, const char *to)
{
int loops = 0;
while (!MoveFileExA(from, to, 1))
while (rename(from, to) < 0)
{
if (GetLastError() != 5L)
if ((*__errno()) != 13)
return -1;
pg_usleep(100000);
if (loops == 30)
errstart(0, "dirmod-orig.c", 87,
__func__), elog_finish(15, "could not rename \"%s\" to \"%s\",
continuing to try",
from, to);
loops++;
}
if (loops > 30)
errstart(0, "dirmod-orig.c", 98, __func__),
elog_finish(15, "completed rename of \"%s\" to \"%s\"", from, to);
return 0;
}

--
Reini Urban
http://xarch.tu-graz.ac.at/home/rurban/

Reini Urban wrote:

>
>
> FYI: WIN32 is also defined because <windows.h> is included.
> (/usr/incluse/w32api/windef.h)
> If you want this or that, do proper nesting, and use #else.
>
>

Ugh, yes. A little experimentation shows that __WIN32__ is defined for
MinGW only, but WIN32 is for both. I wonder how we missed that in
various places. Maybe we need a little audit of the use of WIN32.

cheers

andrew

OK, care to submit a patch. As I remember the fix for rename/unlink
also includes how the file is opened with flags. Anyway, we spent a lot
of time on this so you will have to go back in the archvies to find it
and determine how it can be improved.

Your track record for Cygwin diagnosis isn't 100%. I am going to need
complete research before changing anything at this point in beta.

---------------------------------------------------------------------------

Reini Urban wrote:
> Bruce Momjian schrieb:
> > I looked at the APR code to get some ideas for the Win32 port. Some of
> > the ideas were good, but in other places like rename they didn't do very
> > well we were better off doing it ourselves and getting it right.
> >
> > I remember looking at their code to fix the rename/unlink while the file
> > is open problem, and they didn't seem to have a fix for that so we
> > developed our own method that works like Unix.
>
> sorry, but your rename doesn't work on cygwin. maybe it works with mingw.
>
> cygwin has it's own and working way of doing rename's.
> maybe you should have looked at the cygwin sources instead.
> (src/winsup/cygwin/syscalls.cc)
>
> first doing a WinAPI MoveFileEx and then after a failure trying the
> cygwin version, which will also try its own MoveFile loop, will not
> work. they are conflicting.
>
> same with unlink, but at least the mingw and cygwin unlink versions
> don't conflict here. here you don't stack two conflicting loops together.
> nevertheless cygwin's unlink is much better than pgunlink in case of
> locking problems. it does its own sort of delayed removal then.
>
> IMHO port/dirmod.c is a dirty and half-baked hack, which works for mingw
> only.
> --
> Reini Urban
> http://xarch.tu-graz.ac.at/home/rurban/
>

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Bruce Momjian schrieb:
> OK, care to submit a patch. As I remember the fix for rename/unlink
> also includes how the file is opened with flags. Anyway, we spent a lot
> of time on this so you will have to go back in the archvies to find it
> and determine how it can be improved.
>
> Your track record for Cygwin diagnosis isn't 100%. I am going to need
> complete research before changing anything at this point in beta.

Ok, I'll do an analysis and patch which will have a chance to be accepted.
Keeping pgrename in CYGWIN is probably a good idea.
At least for consistent error reporting (which helped me in finding the
problem)
Personally I don't think that any rename()-usleep loop is necessary.
I'll check the archives.

> ---------------------------------------------------------------------------
> Reini Urban wrote:
>>Bruce Momjian schrieb:
>>
>>>I looked at the APR code to get some ideas for the Win32 port. Some of
>>>the ideas were good, but in other places like rename they didn't do very
>>>well we were better off doing it ourselves and getting it right.
>>>
>>>I remember looking at their code to fix the rename/unlink while the file
>>>is open problem, and they didn't seem to have a fix for that so we
>>>developed our own method that works like Unix.
>>
>>sorry, but your rename doesn't work on cygwin. maybe it works with mingw.
>>
>>cygwin has it's own and working way of doing rename's.
>>maybe you should have looked at the cygwin sources instead.
>>(src/winsup/cygwin/syscalls.cc)
>>
>>first doing a WinAPI MoveFileEx and then after a failure trying the
>>cygwin version, which will also try its own MoveFile loop, will not
>>work. they are conflicting.
>>
>>same with unlink, but at least the mingw and cygwin unlink versions
>>don't conflict here. here you don't stack two conflicting loops together.
>>nevertheless cygwin's unlink is much better than pgunlink in case of
>>locking problems. it does its own sort of delayed removal then.
>>
>>IMHO port/dirmod.c is a dirty and half-baked hack, which works for mingw
>>only.
--
Reini Urban
http://xarch.tu-graz.ac.at/home/rurban/

Andrew Dunstan wrote:
>
>
> Reini Urban wrote:
>
> >
> >
> > FYI: WIN32 is also defined because <windows.h> is included.
> > (/usr/incluse/w32api/windef.h)
> > If you want this or that, do proper nesting, and use #else.
> >
> >
>
> Ugh, yes. A little experimentation shows that __WIN32__ is defined for
> MinGW only, but WIN32 is for both. I wonder how we missed that in
> various places. Maybe we need a little audit of the use of WIN32.

Done, and patch attached and applied. Hopefully at least Cygwin will
compile dirmod.c now. (Most of the patch is consistency
reorganization.) We still need a review of that file.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Bruce Momjian said:
> Andrew Dunstan wrote:
>>
>>
>> Reini Urban wrote:
>>
>> >
>> >
>> > FYI: WIN32 is also defined because <windows.h> is included.
>> > (/usr/incluse/w32api/windef.h)
>> > If you want this or that, do proper nesting, and use #else.
>> >
>> >
>>
>> Ugh, yes. A little experimentation shows that __WIN32__ is defined for
>> MinGW only, but WIN32 is for both. I wonder how we missed that in
>> various places. Maybe we need a little audit of the use of WIN32.
>
> Done, and patch attached and applied. Hopefully at least Cygwin will
> compile dirmod.c now. (Most of the patch is consistency
> reorganization.) We still need a review of that file.
>

I don't understand most of this patch. What difference does changing the
preprocessor test order make?

cheers

andrew

"Andrew Dunstan" <andrew(at)dunslane(dot)net> writes:
> I don't understand most of this patch. What difference does changing the
> preprocessor test order make?

I think Bruce was mostly trying to make all the similar tests look
alike. Also I agree that "if a && !b" is clearer than "if !b && a";
the latter requires a bit more thought to parse the extent of the !
operator...

However, per Michael's report there's some oversight in this patch.
I'm not presently ready to update to CVS tip; who can find the problem?

regards, tom lane

Tom Lane wrote:
> "Andrew Dunstan" <andrew(at)dunslane(dot)net> writes:
> > I don't understand most of this patch. What difference does changing the
> > preprocessor test order make?
>
> I think Bruce was mostly trying to make all the similar tests look
> alike. Also I agree that "if a && !b" is clearer than "if !b && a";
> the latter requires a bit more thought to parse the extent of the !
> operator...

Right, just consistency.

> However, per Michael's report there's some oversight in this patch.
> I'm not presently ready to update to CVS tip; who can find the problem?

I have not seen the report yet. I had no plan to change the behavior
except for Cygwin.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:

>Tom Lane wrote:
>
>
>>"Andrew Dunstan" <andrew(at)dunslane(dot)net> writes:
>>
>>
>>>I don't understand most of this patch. What difference does changing the
>>>preprocessor test order make?
>>>
>>>
>>I think Bruce was mostly trying to make all the similar tests look
>>alike. Also I agree that "if a && !b" is clearer than "if !b && a";
>>the latter requires a bit more thought to parse the extent of the !
>>operator...
>>
>>
>
>Right, just consistency.
>
>

Ok. I understand now.

I'm not sure exactly what Bruce checked, so I just spent a few cycles
making sure that we did not inadvertantly pick up a define of WIN32 from
windows.h anywhere else. I *think* we are OK on that. However, ISTM this
is a foot just waiting to be shot - in retrospect using WIN32 as our
marker for native Windows, which we do in a great many places (around
300 by my count) was a less than stellar choice, given that it is
defined by windows.h, and especially since we use that header for Cygwin
as well as for Windows native in a few places.

cheers

andrew

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> I'm not sure exactly what Bruce checked, so I just spent a few cycles
> making sure that we did not inadvertantly pick up a define of WIN32 from
> windows.h anywhere else. I *think* we are OK on that. However, ISTM this
> is a foot just waiting to be shot - in retrospect using WIN32 as our
> marker for native Windows, which we do in a great many places (around
> 300 by my count) was a less than stellar choice, given that it is
> defined by windows.h, and especially since we use that header for Cygwin
> as well as for Windows native in a few places.

Well, it's easily changed, if all that's needed is a search-and-replace.
Suggestions for a better name?

regards, tom lane

Tom Lane schrieb:

> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
>
>>I'm not sure exactly what Bruce checked, so I just spent a few cycles
>>making sure that we did not inadvertantly pick up a define of WIN32 from
>>windows.h anywhere else. I *think* we are OK on that. However, ISTM this
>>is a foot just waiting to be shot - in retrospect using WIN32 as our
>>marker for native Windows, which we do in a great many places (around
>>300 by my count) was a less than stellar choice, given that it is
>>defined by windows.h, and especially since we use that header for Cygwin
>>as well as for Windows native in a few places.
>
>
> Well, it's easily changed, if all that's needed is a search-and-replace.
> Suggestions for a better name?

MINGW32

--
Reini Urban
http://xarch.tu-graz.ac.at/home/rurban/

Andrew Dunstan wrote:
> >>I think Bruce was mostly trying to make all the similar tests look
> >>alike. Also I agree that "if a && !b" is clearer than "if !b && a";
> >>the latter requires a bit more thought to parse the extent of the !
> >>operator...
> >>
> >>
> >
> >Right, just consistency.
> >
> >
>
>
> Ok. I understand now.
>
> I'm not sure exactly what Bruce checked, so I just spent a few cycles
> making sure that we did not inadvertantly pick up a define of WIN32 from
> windows.h anywhere else. I *think* we are OK on that. However, ISTM this
> is a foot just waiting to be shot - in retrospect using WIN32 as our
> marker for native Windows, which we do in a great many places (around
> 300 by my count) was a less than stellar choice, given that it is
> defined by windows.h, and especially since we use that header for Cygwin
> as well as for Windows native in a few places.

The use of WIN32 was because it usually does mean MinGW and Cygwin. We
had lots of Cygwin-specific defines in there already so Win32 just means
both Mingw and Cygwin. You will see only a few cases where we want
Mingw and not Cygwin, but in those case we often also want MSVC and
Borland, so it really is WIN32 && ! __CYGWIN__. We do have one or two
tests for __MINGW32__ where we really do want just that.

Would you look around and see if this can be improved. I can't see any.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Reini Urban wrote:
> Bruce Momjian schrieb:
> > OK, care to submit a patch. As I remember the fix for rename/unlink
> > also includes how the file is opened with flags. Anyway, we spent a lot
> > of time on this so you will have to go back in the archvies to find it
> > and determine how it can be improved.
> >
> > Your track record for Cygwin diagnosis isn't 100%. I am going to need
> > complete research before changing anything at this point in beta.
>
> Ok, I'll do an analysis and patch which will have a chance to be accepted.
> Keeping pgrename in CYGWIN is probably a good idea.
> At least for consistent error reporting (which helped me in finding the
> problem)
> Personally I don't think that any rename()-usleep loop is necessary.
> I'll check the archives.

I agree the rename loop seems unnecessary. I kept it in case we hadn't
dealt with all the failure places. Should we remove them now or wait
for 8.1? Seems we should keep them in and see if we get reports from
users of looping forever, and if not we can remove them in 8.1.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Bruce Momjian schrieb:

> Reini Urban wrote:
>
>>Bruce Momjian schrieb:
>>
>>>OK, care to submit a patch. As I remember the fix for rename/unlink
>>>also includes how the file is opened with flags. Anyway, we spent a lot
>>>of time on this so you will have to go back in the archvies to find it
>>>and determine how it can be improved.
>>>
>>>Your track record for Cygwin diagnosis isn't 100%. I am going to need
>>>complete research before changing anything at this point in beta.
>>
>>Ok, I'll do an analysis and patch which will have a chance to be accepted.
>>Keeping pgrename in CYGWIN is probably a good idea.
>>At least for consistent error reporting (which helped me in finding the
>>problem)
>>Personally I don't think that any rename()-usleep loop is necessary.
>>I'll check the archives.
>
>
> I agree the rename loop seems unnecessary. I kept it in case we hadn't
> dealt with all the failure places. Should we remove them now or wait
> for 8.1? Seems we should keep them in and see if we get reports from
> users of looping forever, and if not we can remove them in 8.1.

we at CYGWIN had similar problems with windows locks on unlink.

if unlink fails with ERROR_SHARING_VIOLATION or ERROR_ACCESS_DENIED,
unlinking is deferred, put into a delqueue. we do no busy waiting then.
it's done on exit.
The most common problem is the "delete on close" semantics to handle
removing a file which may be open.

rename only fails on open files. we try first MoveFile, if that fails we
try MoveFileEx MOVEFILE_REPLACE_EXISTING, but no loop on rename.

http://sources.redhat.com/cgi-bin/cvsweb.cgi/src/winsup/cygwin/syscalls.cc?cvsroot=src
http://sources.redhat.com/cgi-bin/cvsweb.cgi/src/winsup/cygwin/delqueue.cc?cvsroot=src

--
Reini Urban
http://xarch.tu-graz.ac.at/home/rurban/

Bruce Momjian wrote:

>Andrew Dunstan wrote:
>
>
>>
>>I'm not sure exactly what Bruce checked, so I just spent a few cycles
>>making sure that we did not inadvertantly pick up a define of WIN32 from
>>windows.h anywhere else. I *think* we are OK on that. However, ISTM this
>>is a foot just waiting to be shot - in retrospect using WIN32 as our
>>marker for native Windows, which we do in a great many places (around
>>300 by my count) was a less than stellar choice, given that it is
>>defined by windows.h, and especially since we use that header for Cygwin
>>as well as for Windows native in a few places.
>>
>>
>
>The use of WIN32 was because it usually does mean MinGW and Cygwin.
>

But it doesn't. On MinGW WIN32 is a builtin compiler-defined value, and
on Cygwin it isn't. To see this, do:

touch empty.c; cpp -dM empty.c | grep WIN32

WIN32 *is* defined by windows.h, but in most cases we only include it if
WIN32 is *already* defined. windows.h is included unconditionally in our
win32.h, but again in most cases we only include that if WIN32 is
already defined. So in most cases where we use it it isn't for Cygwin.
But there are a few system include files on Cygwin that include it, so
it's not guaranteed, although I don't think those affect us.

> We
>had lots of Cygwin-specific defines in there already so Win32 just means
>both Mingw and Cygwin. You will see only a few cases where we want
>Mingw and not Cygwin, but in those case we often also want MSVC and
>Borland, so it really is WIN32 && ! __CYGWIN__. We do have one or two
>tests for __MINGW32__ where we really do want just that.
>
>Would you look around and see if this can be improved. I can't see any.
>
>
>

As I said, I did look at all the include cases. That was based on the
assumption that we actually wanted what I thought was the intention,
namely that WIN32 was for Windows native only. If that's not the case we
would need to review every one of the ~300 cases where WIN32 is used in
#ifdef and friends.

Bottom line - this is something of a mess. If we can make sure Cygwin
isn't broken, we can probably live with what have for now. Personally, I
would have configure work out something cleaner, like, say, defining
WINDOWS_ALL for both Windows native and Cygwin. Then we could use that
for cases meant to cover both, and __CYGWIN__ and __MINGW32__ for the
specific cases, without worrying what the compiler and/or the system
header files might have defined for us.

cheers

andrew

Andrew Dunstan schrieb:
> Bruce Momjian wrote:
>> Andrew Dunstan wrote:
>>> I'm not sure exactly what Bruce checked, so I just spent a few cycles
>>> making sure that we did not inadvertantly pick up a define of WIN32
>>> from windows.h anywhere else. I *think* we are OK on that. However,
>>> ISTM this is a foot just waiting to be shot - in retrospect using
>>> WIN32 as our marker for native Windows, which we do in a great many
>>> places (around 300 by my count) was a less than stellar choice, given
>>> that it is defined by windows.h, and especially since we use that
>>> header for Cygwin as well as for Windows native in a few places.
>>>
>>
>>
>> The use of WIN32 was because it usually does mean MinGW and Cygwin.
>
>
> But it doesn't. On MinGW WIN32 is a builtin compiler-defined value, and
> on Cygwin it isn't. To see this, do:
>
> touch empty.c; cpp -dM empty.c | grep WIN32
>
> WIN32 *is* defined by windows.h, but in most cases we only include it if
> WIN32 is *already* defined. windows.h is included unconditionally in our
> win32.h, but again in most cases we only include that if WIN32 is
> already defined. So in most cases where we use it it isn't for Cygwin.
> But there are a few system include files on Cygwin that include it, so
> it's not guaranteed, although I don't think those affect us.
>
>
>
>> We
>> had lots of Cygwin-specific defines in there already so Win32 just means
>> both Mingw and Cygwin. You will see only a few cases where we want
>> Mingw and not Cygwin, but in those case we often also want MSVC and
>> Borland, so it really is WIN32 && ! __CYGWIN__. We do have one or two
>> tests for __MINGW32__ where we really do want just that.
>>
>> Would you look around and see if this can be improved. I can't see any.
>
> As I said, I did look at all the include cases. That was based on the
> assumption that we actually wanted what I thought was the intention,
> namely that WIN32 was for Windows native only. If that's not the case we
> would need to review every one of the ~300 cases where WIN32 is used in
> #ifdef and friends.
>
> Bottom line - this is something of a mess. If we can make sure Cygwin
> isn't broken, we can probably live with what have for now. Personally, I
> would have configure work out something cleaner, like, say, defining
> WINDOWS_ALL for both Windows native and Cygwin. Then we could use that
> for cases meant to cover both, and __CYGWIN__ and __MINGW32__ for the
> specific cases, without worrying what the compiler and/or the system
> header files might have defined for us.

Most of the ~300 cases are ok for CYGWIN. And probably for MINGW also.
But I don't do MINGW countertests. I assume you do :)

Just palloc misses some pending fixes for CYGWIN. cvs head didn't has
this fixed.
I'll come with a new patch to cvs HEAD soon.
I'm quite busy with apache and php porting also.
And I want to be careful not to break the FRONTEND section.

At least beta2 needed this patch:
--- postgresql-8.0.0beta2/src/include/utils/palloc.h.orig 2004-08-29
05:13:11.000000000 +0100
+++ postgresql-8.0.0beta2/src/include/utils/palloc.h 2004-09-03
14:03:50.279562100 +0100
@@ -80,7 +80,7 @@

#define pstrdup(str) MemoryContextStrdup(CurrentMemoryContext, (str))

-#ifdef WIN32
+#if defined(WIN32) || defined(__CYGWIN__)
extern void *pgport_palloc(Size sz);
extern char *pgport_pstrdup(const char *str);
extern void pgport_pfree(void *pointer);

--
Reini Urban
http://xarch.tu-graz.ac.at/home/rurban/

Reini Urban wrote:

> Andrew Dunstan schrieb:
>
>>> We
>>> had lots of Cygwin-specific defines in there already so Win32 just
>>> means
>>> both Mingw and Cygwin. You will see only a few cases where we want
>>> Mingw and not Cygwin, but in those case we often also want MSVC and
>>> Borland, so it really is WIN32 && ! __CYGWIN__. We do have one or two
>>> tests for __MINGW32__ where we really do want just that.
>>>
>>> Would you look around and see if this can be improved. I can't see
>>> any.
>>
>>
>> As I said, I did look at all the include cases. That was based on the
>> assumption that we actually wanted what I thought was the intention,
>> namely that WIN32 was for Windows native only. If that's not the case
>> we would need to review every one of the ~300 cases where WIN32 is
>> used in #ifdef and friends.
>>
>> Bottom line - this is something of a mess. If we can make sure Cygwin
>> isn't broken, we can probably live with what have for now.
>> Personally, I would have configure work out something cleaner, like,
>> say, defining WINDOWS_ALL for both Windows native and Cygwin. Then we
>> could use that for cases meant to cover both, and __CYGWIN__ and
>> __MINGW32__ for the specific cases, without worrying what the
>> compiler and/or the system header files might have defined for us.
>
>
> Most of the ~300 cases are ok for CYGWIN. And probably for MINGW also.
> But I don't do MINGW countertests. I assume you do :)
>
>

Cygwin is the likely point of failure here, since we know WIN32 is
always defined on MinGW.

cheers

andrew

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> Bottom line - this is something of a mess. If we can make sure Cygwin
> isn't broken, we can probably live with what have for now. Personally, I
> would have configure work out something cleaner, like, say, defining
> WINDOWS_ALL for both Windows native and Cygwin. Then we could use that
> for cases meant to cover both, and __CYGWIN__ and __MINGW32__ for the
> specific cases, without worrying what the compiler and/or the system
> header files might have defined for us.

I agree that this is a good idea, partly because I do not care for the
assumption that MINGW is the only compilation environment we'll ever
support for the Windows-native port.

I'm not in a position to work out or test the required changes, but I'll
be happy to apply a patch if you do the legwork ...

regards, tom lane

Tom Lane wrote:

>Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
>
>
>>Bottom line - this is something of a mess. If we can make sure Cygwin
>>isn't broken, we can probably live with what have for now. Personally, I
>>would have configure work out something cleaner, like, say, defining
>>WINDOWS_ALL for both Windows native and Cygwin. Then we could use that
>>for cases meant to cover both, and __CYGWIN__ and __MINGW32__ for the
>>specific cases, without worrying what the compiler and/or the system
>>header files might have defined for us.
>>
>>
>
>I agree that this is a good idea, partly because I do not care for the
>assumption that MINGW is the only compilation environment we'll ever
>support for the Windows-native port.
>
>I'm not in a position to work out or test the required changes, but I'll
>be happy to apply a patch if you do the legwork ...
>
>
>
>

Too big a task for my current time budget :-( - currently my work does
not involve any PostgreSQL component, and I am flat out delivering what
I am paid for.

Unless someone else steps up to the plate it will have to go on the TODO
list.

cheers

andrew

Andrew Dunstan wrote:
> >I agree that this is a good idea, partly because I do not care for the
> >assumption that MINGW is the only compilation environment we'll ever
> >support for the Windows-native port.
> >
> >I'm not in a position to work out or test the required changes, but I'll
> >be happy to apply a patch if you do the legwork ...
> >
> >
> >
> >
>
> Too big a task for my current time budget :-( - currently my work does
> not involve any PostgreSQL component, and I am flat out delivering what
> I am paid for.

I will do it.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

OK, change made. Thanks.

---------------------------------------------------------------------------

> Most of the ~300 cases are ok for CYGWIN. And probably for MINGW also.
> But I don't do MINGW countertests. I assume you do :)
>
> Just palloc misses some pending fixes for CYGWIN. cvs head didn't has
> this fixed.
> I'll come with a new patch to cvs HEAD soon.
> I'm quite busy with apache and php porting also.
> And I want to be careful not to break the FRONTEND section.
>
> At least beta2 needed this patch:
> --- postgresql-8.0.0beta2/src/include/utils/palloc.h.orig 2004-08-29
> 05:13:11.000000000 +0100
> +++ postgresql-8.0.0beta2/src/include/utils/palloc.h 2004-09-03
> 14:03:50.279562100 +0100
> @@ -80,7 +80,7 @@
>
> #define pstrdup(str) MemoryContextStrdup(CurrentMemoryContext, (str))
>
> -#ifdef WIN32
> +#if defined(WIN32) || defined(__CYGWIN__)
> extern void *pgport_palloc(Size sz);
> extern char *pgport_pstrdup(const char *str);
> extern void pgport_pfree(void *pointer);
>
>
> --
> Reini Urban
> http://xarch.tu-graz.ac.at/home/rurban/
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faqs/FAQ.html
>

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

>>>>> "CB" == Christopher Browne <cbbrowne(at)acm(dot)org> writes:

CB> futile discussions ask for it. Notably, on an SMP system, it
CB> would be a neat idea for complex queries involving joins to
CB> split themselves so that different parts run in separate
CB> threads.

You don't really need threads for this. All you need is to have
multiple backends and use queues to exchange tuples at specific
points. This is much like the Exchange operator in Volcano.

CB> The other Way, Way Cool part would be for queries that are
CB> scanning big tables to split the scans into unions of partial
CB> scans, so that on an 8 CPU box you'd take the "Big 4GB Table"
CB> and have 8 threads simultaneously scanning different parts of
CB> it. (And making ARC all the more important :-).)

Again this can be done without threads .. you just need inter-process
communication.

(BTW, there is at least one commercial system that follows exactly
this model).

--
Pip-pip
Sailesh
http://www.cs.berkeley.edu/~sailesh

Any chance of having query parallelization added to TODO? I'm guessing
it will be a huge job, but it's also one of the places where the 'big 3'
have a huge advantage in scalability.

On Mon, Sep 13, 2004 at 10:24:05AM -0700, Sailesh Krishnamurthy wrote:
> >>>>> "CB" == Christopher Browne <cbbrowne(at)acm(dot)org> writes:
>
> CB> futile discussions ask for it. Notably, on an SMP system, it
> CB> would be a neat idea for complex queries involving joins to
> CB> split themselves so that different parts run in separate
> CB> threads.
>
> You don't really need threads for this. All you need is to have
> multiple backends and use queues to exchange tuples at specific
> points. This is much like the Exchange operator in Volcano.
>
> CB> The other Way, Way Cool part would be for queries that are
> CB> scanning big tables to split the scans into unions of partial
> CB> scans, so that on an 8 CPU box you'd take the "Big 4GB Table"
> CB> and have 8 threads simultaneously scanning different parts of
> CB> it. (And making ARC all the more important :-).)
>
> Again this can be done without threads .. you just need inter-process
> communication.
>
> (BTW, there is at least one commercial system that follows exactly
> this model).
>
> --
> Pip-pip
> Sailesh
> http://www.cs.berkeley.edu/~sailesh
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
>

--
Jim C. Nasby, Database Consultant decibel(at)decibel(dot)org
Give your computer some brain candy! www.distributed.net Team #1828

Windows: "Where do you want to go today?"
Linux: "Where do you want to go tomorrow?"
FreeBSD: "Are you guys coming, or what?"

Added to TODO:

* Consider parallel processing a single query

This would involve using multiple threads or processes to do optimization,
sorting, or execution of single query. The major advantage of such a
feature would be to allow multiple CPUs to work together to process a
single query.

---------------------------------------------------------------------------

Jim C. Nasby wrote:
> Any chance of having query parallelization added to TODO? I'm guessing
> it will be a huge job, but it's also one of the places where the 'big 3'
> have a huge advantage in scalability.
>
> On Mon, Sep 13, 2004 at 10:24:05AM -0700, Sailesh Krishnamurthy wrote:
> > >>>>> "CB" == Christopher Browne <cbbrowne(at)acm(dot)org> writes:
> >
> > CB> futile discussions ask for it. Notably, on an SMP system, it
> > CB> would be a neat idea for complex queries involving joins to
> > CB> split themselves so that different parts run in separate
> > CB> threads.
> >
> > You don't really need threads for this. All you need is to have
> > multiple backends and use queues to exchange tuples at specific
> > points. This is much like the Exchange operator in Volcano.
> >
> > CB> The other Way, Way Cool part would be for queries that are
> > CB> scanning big tables to split the scans into unions of partial
> > CB> scans, so that on an 8 CPU box you'd take the "Big 4GB Table"
> > CB> and have 8 threads simultaneously scanning different parts of
> > CB> it. (And making ARC all the more important :-).)
> >
> > Again this can be done without threads .. you just need inter-process
> > communication.
> >
> > (BTW, there is at least one commercial system that follows exactly
> > this model).
> >
> > --
> > Pip-pip
> > Sailesh
> > http://www.cs.berkeley.edu/~sailesh
> >
> >
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 4: Don't 'kill -9' the postmaster
> >
>
> --
> Jim C. Nasby, Database Consultant decibel(at)decibel(dot)org
> Give your computer some brain candy! www.distributed.net Team #1828
>
> Windows: "Where do you want to go today?"
> Linux: "Where do you want to go tomorrow?"
> FreeBSD: "Are you guys coming, or what?"
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faqs/FAQ.html
>

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

On Thu, 7 Oct 2004, Bruce Momjian wrote:

>
> Added to TODO:
>
> * Consider parallel processing a single query
>
> This would involve using multiple threads or processes to do optimization,
> sorting, or execution of single query. The major advantage of such a
> feature would be to allow multiple CPUs to work together to process a
> single query.

Do we have 'make backend thread safe' listed yet? As I recall it, until
that gets done, parallelization of anything was considered to be a
relatively onerous task, no?

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy(at)hub(dot)org Yahoo!: yscrappy ICQ: 7615664

Marc G. Fournier wrote:
> On Thu, 7 Oct 2004, Bruce Momjian wrote:
>
> >
> > Added to TODO:
> >
> > * Consider parallel processing a single query
> >
> > This would involve using multiple threads or processes to do optimization,
> > sorting, or execution of single query. The major advantage of such a
> > feature would be to allow multiple CPUs to work together to process a
> > single query.
>
> Do we have 'make backend thread safe' listed yet? As I recall it, until
> that gets done, parallelization of anything was considered to be a
> relatively onerous task, no?

Well, not really. We could perhaps make sorting be thread-safe without
doing the entire backend itself. The only trick would be making modules
used by sorting thread-safe, not the whole thing.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Marc G. Fournier wrote:
> Do we have 'make backend thread safe' listed yet? As I recall it, until
> that gets done, parallelization of anything was considered to be a
> relatively onerous task, no?

ISTM there's no reason we couldn't parallelize query execution using the
same IPC techniques that we use now. What would be the advantage of
using threads instead?

-Neil

Neil Conway wrote:
> Marc G. Fournier wrote:
> > Do we have 'make backend thread safe' listed yet? As I recall it, until
> > that gets done, parallelization of anything was considered to be a
> > relatively onerous task, no?
>
> ISTM there's no reason we couldn't parallelize query execution using the
> same IPC techniques that we use now. What would be the advantage of
> using threads instead?

Separate processes. Yes, we could do that too and the item mentions that.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

A while back I was looking the backend code in preparation to start
beginning to look at parallelization techniques for PG ;)... My
thought was instead of trying to parallelize each individual plan node
(multi-process sort, etc.) I would look at creating worker
threads/processes for each plan node as a whole. For example, take a
plan that looks like this:

QUERY
PLAN
---------------------------------------------------------------------------------------------------------------------------------------------
Subquery Scan metarecord_field_entry_view (cost=5.32..4038.80
rows=21 width=112)
-> Append (cost=5.32..4038.59 rows=21 width=112)
-> Subquery Scan "*SELECT* 1" (cost=5.32..5.33 rows=1 width=74)
-> HashAggregate (cost=5.32..5.32 rows=1 width=74)
-> Index Scan using tmr_fe_field on
metarecord_title_field_entry (cost=0.00..5.31 rows=1 width=74)
Index Cond: (field = 'added_entry_author'::text)
Filter: ((field_class = 'title'::text) AND
(value ~~* '% joe %'::text))
-> Subquery Scan "*SELECT* 2" (cost=4031.02..4031.20
rows=18 width=62)
-> HashAggregate (cost=4031.02..4031.02 rows=18 width=62)
-> Seq Scan on metarecord_author_field_entry
(cost=0.00..4030.79 rows=18 width=62)
Filter: ((field_class = 'author'::text) AND
(field = 'added_entry_author'::text) AND (value ~~* '% joe %'::text))
-> Subquery Scan "*SELECT* 3" (cost=2.03..2.04 rows=1 width=81)
-> HashAggregate (cost=2.03..2.03 rows=1 width=81)
-> Index Scan using smr_fe_field on
metarecord_subject_field_entry (cost=0.00..2.02 rows=1 width=81)
Index Cond: (field = 'added_entry_author'::text)
Filter: ((field_class = 'subject'::text)
AND (value ~~* '% joe %'::text))
-> Subquery Scan "*SELECT* 4" (cost=0.01..0.02 rows=1 width=112)
-> HashAggregate (cost=0.01..0.01 rows=1 width=112)
-> Seq Scan on metarecord_misc_field_entry
(cost=0.00..0.00 rows=1 width=112)
Filter: ((field_class = 'misc'::text) AND
(field = 'added_entry_author'::text) AND (value ~~* '% joe %'::text))

The optimizer would look at each node as it walked down the tree and
see that 'Append' node has multiple peer child nodes. It would look
at the cost estimate of the child nodes and if that cost is greater
that the total average cost across all nodes it would spin off a
worker thread/process to handle gathering the sub-resultset.

In any case, I've no time to even *start* looking into something like
that. But even if I did, am I all wet?

--miker

On Fri, 8 Oct 2004 11:56:27 -0400 (EDT), Bruce Momjian
<pgman(at)candle(dot)pha(dot)pa(dot)us> wrote:
> Neil Conway wrote:
> > Marc G. Fournier wrote:
> > > Do we have 'make backend thread safe' listed yet? As I recall it, until
> > > that gets done, parallelization of anything was considered to be a
> > > relatively onerous task, no?
> >
> > ISTM there's no reason we couldn't parallelize query execution using the
> > same IPC techniques that we use now. What would be the advantage of
> > using threads instead?
>
> Separate processes. Yes, we could do that too and the item mentions that.
>
> --
> Bruce Momjian | http://candle.pha.pa.us
> pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
> + If your life is a hard drive, | 13 Roberts Road
> + Christ can be your backup. | Newtown Square, Pennsylvania 19073
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
> (send "unregister YourEmailAddressHere" to majordomo(at)postgresql(dot)org)
>

>>>>> "Marc" == Marc G Fournier <scrappy(at)postgresql(dot)org> writes:

Marc> On Thu, 7 Oct 2004, Bruce Momjian wrote:
>> Added to TODO:
>>
>> * Consider parallel processing a single query
>>
>> This would involve using multiple threads or processes to do
>> optimization, sorting, or execution of single query. The major
>> advantage of such a feature would be to allow multiple CPUs to
>> work together to process a single query.

Marc> Do we have 'make backend thread safe' listed yet? As I
Marc> recall it, until that gets done, parallelization of anything
Marc> was considered to be a relatively onerous task, no?

You don't really need to parallelize in separate threads .. you can
have more than one process working on one query. This is in fact the
model that exploits SMPs in at least one commercial RDBMS.

--
Pip-pip
Sailesh
http://www.cs.berkeley.edu/~sailesh

IMHO the best references to parallelizing query plans are in the
Volcano papers. The Exchange operator is a really clean abstraction -
the idea is to place the Exchange operator in query plans and that way
you don't have to paralellize any other operator. Exchange takes care
of managing the IPC queues and also worries about whether or not you
have to, say, "rehash the data", or "broadcast the data to all other
processes" or "direct the data to a single node" ...

I'd suggest reading the following paper:

"Encapsulation of parallelism in the Volcano query processing system"

By Goetz Graefe in SIGMOD 1990.

Link: http://portal.acm.org/citation.cfm?id=98720

The above link also has references to Gamma but I really like the
exposition in the Volcano/Exchange work much better.

--
Pip-pip
Sailesh
http://www.cs.berkeley.edu/~sailesh