Roadmap for a Win32 port

Yeah: ST is designed for network apps, and its for network bound apps that
you
gain the most performance - but by using it to allow
a child process to hold multiple connections and accept/return data to
those connections simultaneously, I forsaw a potential performance
improvement...
*shrug* Most connections remain idle most of thier life... yes?

the SGI folks developed this library as part of a project to make apache
faster (http://aap.sourceforge.net/) - multiple child
processes as normal, but allowed multiple connections per child.

And although the performance improvements they got were greatest on irix,
performance was improved upto 70% on linux. Some of this was from QSC
(http://aap.sourceforge.net/mod_qsc.html) , however...

just some food for thought.

----- Original Message -----
From: "Neil Conway" <nconway(at)klamath(dot)dyndns(dot)org>
To: "Jon Franz" <coventry(at)one(dot)net>
Cc: <pgsql-hackers(at)postgresql(dot)org>
Sent: Wednesday, June 05, 2002 8:05 PM
Subject: Re: [HACKERS] Roadmap for a Win32 port

> On Wed, 5 Jun 2002 18:50:46 -0400
> "Jon Franz" <coventry(at)one(dot)net> wrote:
> > One note: SGI developers discovered they could get amazing performance
using
> > as hybrid threaded and forked-process model with apache - we might want
to
> > look into this. They even have a library for network-communication
> > utilizing thier 'state threads' model.
>
> I think ST is designed for network I/O-bound apps -- last I checked,
> disk I/O will still block an entire ST process. While you can get around
> that by using another process to do disk I/O, it sounds like ST won't be
> that useful.
>
> However, Chris KL. (I believe) raised the idea of using POSIX AIO for
> PostgreSQL. Without having looked into it extensively, this technique
> sounds promising. Perhaps someone who has looked into this further
> (e.g. someone from Redhat) can comment?
>
> Cheers,
>
> Neil
>
> --
> Neil Conway <neilconway(at)rogers(dot)com>
> PGP Key ID: DB3C29FC
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster

Hi all,

I followed the various threads regarding this for some time now. My current
situation is:

I'm working at a company which does industrial automation, and does it's own
custom products. We try to be cross-platform, but it's a windoze world, as
far as most measurement devices or PLCs are concerned. We also employ
databases for various tasks (including simple ones as holding configuration
data, but also hammering production data into it at a rate of several hundred
records/sec.)
Well, we would *love* to use PostgreSQL in most our projects and products,
(and we do already use it in some), because it has proven to be very reliable
and quite fast.

So, I'm faced with using PostgreSQL on windows also (you can't always put a
Linux box besides). We do this using cygwin, but it's a bit painful ;-)
(although it works!).

Thinking about the hreads I read, it seems there are 2 obstacles to native PG
on W:

1.) no fork,
2.) no SYSV IPC

Ok, 1.) is an issue, but there's a fork() in MinGW, so it's 'just' going to
be a bit slow on new connections to the DB, right?? But this could be sorted
out once we *have* a native WIN32 build.

The second one's a bit harder, but... I'm currently trying to find time to do
a minimal implementation of SYSV IPC on WIN32 calls, just enough to get PG up
(doesn't need msg*() for example, right?).
As far as I understand it, we would not need to have IPC items around *after*
all backends and postmaster have gone away, or? Then there's no need for a
'daemon' process like in cygwin.

So, my route would be to get it to run *somehow* without paying attention to
speed and not to change much of the existing code, THEN see how we could get
rid of fork() on windows.

What do you guys think? Anyone up to join efforts? (I'll start the IPC thingy
anyway, as an exercise, and see where I'll end).

Greetings,
Joerg

P.s.: thanks for a great database system!!
--
Leading SW developer - S.E.A GmbH
Mail: joerg(dot)hessdoerfer(at)sea-gmbh(dot)com
WWW: http://www.sea-gmbh.com

Actually, take a look at the thread starting at:

http://archives.postgresql.org/pgsql-hackers/2002-05/msg00665.php

Right now, IMHO, the big show stopper is passing global variables to the
child processes in Windows ... the above thread talks about a method of
pulling together the global variables *cleanly* that Tom seems to feel
wouldn't add much in the way of long term maintenance headaches ... *and*,
as I understand it, would provide us with a means to use threading in
future developments if deemed appropriate ...

From what I read by those 'in the know' about Windows programming, if we
could centralize the global variables somewhat, using CreateProcess in
Windows shouldn't be a big deal, eliminiating the whole fork() headache
...

On Thu, 16 May 2002, Joerg Hessdoerfer wrote:

> Hi all,
>
> I followed the various threads regarding this for some time now. My current
> situation is:
>
> I'm working at a company which does industrial automation, and does it's own
> custom products. We try to be cross-platform, but it's a windoze world, as
> far as most measurement devices or PLCs are concerned. We also employ
> databases for various tasks (including simple ones as holding configuration
> data, but also hammering production data into it at a rate of several hundred
> records/sec.)
> Well, we would *love* to use PostgreSQL in most our projects and products,
> (and we do already use it in some), because it has proven to be very reliable
> and quite fast.
>
> So, I'm faced with using PostgreSQL on windows also (you can't always put a
> Linux box besides). We do this using cygwin, but it's a bit painful ;-)
> (although it works!).
>
> Thinking about the hreads I read, it seems there are 2 obstacles to native PG
> on W:
>
> 1.) no fork,
> 2.) no SYSV IPC
>
> Ok, 1.) is an issue, but there's a fork() in MinGW, so it's 'just' going to
> be a bit slow on new connections to the DB, right?? But this could be sorted
> out once we *have* a native WIN32 build.
>
> The second one's a bit harder, but... I'm currently trying to find time to do
> a minimal implementation of SYSV IPC on WIN32 calls, just enough to get PG up
> (doesn't need msg*() for example, right?).
> As far as I understand it, we would not need to have IPC items around *after*
> all backends and postmaster have gone away, or? Then there's no need for a
> 'daemon' process like in cygwin.
>
> So, my route would be to get it to run *somehow* without paying attention to
> speed and not to change much of the existing code, THEN see how we could get
> rid of fork() on windows.
>
> What do you guys think? Anyone up to join efforts? (I'll start the IPC thingy
> anyway, as an exercise, and see where I'll end).
>
> Greetings,
> Joerg
>
> P.s.: thanks for a great database system!!
> --
> Leading SW developer - S.E.A GmbH
> Mail: joerg(dot)hessdoerfer(at)sea-gmbh(dot)com
> WWW: http://www.sea-gmbh.com
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
> (send "unregister YourEmailAddressHere" to majordomo(at)postgresql(dot)org)
>

On Thu, 2002-05-16 at 13:47, Joerg Hessdoerfer wrote:
> So, my route would be to get it to run *somehow* without paying attention to
> speed and not to change much of the existing code, THEN see how we could get
> rid of fork() on windows.

Getting it to compile and then "somehow" run on MinGW seems a good first
step on road to full native Win32 PG.

----------
Hannu

> On Thu, 2002-05-16 at 13:47, Joerg Hessdoerfer wrote:
> > So, my route would be to get it to run *somehow* without paying
> > attention to speed and not to change much of the existing code,
> > THEN see how we could get rid of fork() on windows.
>

What is the biggest problem here?
The Shmem/IPC stuff, or the fork() stuff?
I'm think that we could do a fork() implementation in usermode by copying the memory allocations.
How fast that would be regarding the context switches, i don't know, but i'm willing to experiment some to see how feesible this is...

Anyone tried this before?

Magnus

On Thursday 16 May 2002 22:10, you wrote:
[...]
>
> What is the biggest problem here?
> The Shmem/IPC stuff, or the fork() stuff?
> I'm think that we could do a fork() implementation in usermode by copying
> the memory allocations. How fast that would be regarding the context
> switches, i don't know, but i'm willing to experiment some to see how
> feesible this is...
>
> Anyone tried this before?
>
> Magnus
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster

The problem is not the fork() call itself, this has been done (MinGW and
cygwin I know of, possibly others) but the speed of fork() on windows, it's
creepingly slow (due to usermode copy, I assume ;-).

IPC needs to be done, I'm just about to start...

Greetings,
Joerg
--
Leading SW developer - S.E.A GmbH
Mail: joerg(dot)hessdoerfer(at)sea-gmbh(dot)com
WWW: http://www.sea-gmbh.com

Joerg Hessdoerfer <Joerg(dot)Hessdoerfer(at)sea-gmbh(dot)com> wrote:
[snip]
> The problem is not the fork() call itself, this has been done (MinGW
> and cygwin I know of, possibly others) but the speed of fork() on
> windows, it's creepingly slow (due to usermode copy, I assume ;-).
>
> IPC needs to be done, I'm just about to start...
>

I'm not so familiar with the win32 kernel mode stuff.
But i've seen programs using .vxd (kernelmode, ring X ?) helpers for getting more privileges, maybe cross process ones.
Well, i'll look into this sometime if it's possible to reduce the context switches by going vxd.
There must be some way to read protection of the pages and map them as COW or RO in the new process to get rid of much of the copy, but then again, we're talking microsoft here :)
I once did a .exe loader that used the MapViewOfFile (win32 mmap) of the .exe itself to accomplish shared loadable modules that worked on x86 linux and win32 without recompile (might be something like the XFree86 binary gfx card drivers).

Good luck on the IPC work!

Magnus

--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Programmer/Networker [|] Magnus Naeslund
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Maybe Vince could set up a Win32 porting project page, and since we now seem
to have a few interested parties willing to code on a native Win32 version,
they should have their own project page. This could make communication
easier for them and make sure the project doesn't die...

Chris

> -----Original Message-----
> From: pgsql-hackers-owner(at)postgresql(dot)org
> [mailto:pgsql-hackers-owner(at)postgresql(dot)org]On Behalf Of Joerg
> Hessdoerfer
> Sent: Friday, 17 May 2002 4:36 AM
> To: Magnus Naeslund(f)
> Cc: pgsql-hackers(at)postgresql(dot)org
> Subject: Re: [HACKERS] WIN32 native ... lets start?!?
>
>
> On Thursday 16 May 2002 22:10, you wrote:
> [...]
> >
> > What is the biggest problem here?
> > The Shmem/IPC stuff, or the fork() stuff?
> > I'm think that we could do a fork() implementation in usermode
> by copying
> > the memory allocations. How fast that would be regarding the context
> > switches, i don't know, but i'm willing to experiment some to see how
> > feesible this is...
> >
> > Anyone tried this before?
> >
> > Magnus
> >
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 4: Don't 'kill -9' the postmaster
>
> The problem is not the fork() call itself, this has been done (MinGW and
> cygwin I know of, possibly others) but the speed of fork() on
> windows, it's
> creepingly slow (due to usermode copy, I assume ;-).
>
> IPC needs to be done, I'm just about to start...
>
> Greetings,
> Joerg
> --
> Leading SW developer - S.E.A GmbH
> Mail: joerg(dot)hessdoerfer(at)sea-gmbh(dot)com
> WWW: http://www.sea-gmbh.com
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo(at)postgresql(dot)org
>

On Fri, 17 May 2002, Christopher Kings-Lynne wrote:

> Maybe Vince could set up a Win32 porting project page, and since we now seem
> to have a few interested parties willing to code on a native Win32 version,
> they should have their own project page. This could make communication
> easier for them and make sure the project doesn't die...

Might be an idea to create a pgsql-hackers-win32 list also? Or just
pgsql-win32?

>
> Chris
>
> > -----Original Message-----
> > From: pgsql-hackers-owner(at)postgresql(dot)org
> > [mailto:pgsql-hackers-owner(at)postgresql(dot)org]On Behalf Of Joerg
> > Hessdoerfer
> > Sent: Friday, 17 May 2002 4:36 AM
> > To: Magnus Naeslund(f)
> > Cc: pgsql-hackers(at)postgresql(dot)org
> > Subject: Re: [HACKERS] WIN32 native ... lets start?!?
> >
> >
> > On Thursday 16 May 2002 22:10, you wrote:
> > [...]
> > >
> > > What is the biggest problem here?
> > > The Shmem/IPC stuff, or the fork() stuff?
> > > I'm think that we could do a fork() implementation in usermode
> > by copying
> > > the memory allocations. How fast that would be regarding the context
> > > switches, i don't know, but i'm willing to experiment some to see how
> > > feesible this is...
> > >
> > > Anyone tried this before?
> > >
> > > Magnus
> > >
> > >
> > > ---------------------------(end of broadcast)---------------------------
> > > TIP 4: Don't 'kill -9' the postmaster
> >
> > The problem is not the fork() call itself, this has been done (MinGW and
> > cygwin I know of, possibly others) but the speed of fork() on
> > windows, it's
> > creepingly slow (due to usermode copy, I assume ;-).
> >
> > IPC needs to be done, I'm just about to start...
> >
> > Greetings,
> > Joerg
> > --
> > Leading SW developer - S.E.A GmbH
> > Mail: joerg(dot)hessdoerfer(at)sea-gmbh(dot)com
> > WWW: http://www.sea-gmbh.com
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 1: subscribe and unsubscribe commands go to majordomo(at)postgresql(dot)org
> >
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
>

"Marc G. Fournier" <scrappy(at)hub(dot)org> writes:
> Might be an idea to create a pgsql-hackers-win32 list also? Or just
> pgsql-win32?

Actually, I think that'd be a bad idea. The very last thing we need is
for these discussions to get fragmented. The issues affect the whole
backend AFAICS.

regards, tom lane

On Friday 17 May 2002 22:16, you wrote:
> "Marc G. Fournier" <scrappy(at)hub(dot)org> writes:
> > Might be an idea to create a pgsql-hackers-win32 list also? Or just
> > pgsql-win32?
>
> Actually, I think that'd be a bad idea. The very last thing we need is
> for these discussions to get fragmented. The issues affect the whole
> backend AFAICS.
>
> regards, tom lane

Yes, indeed. I would also like to discuss matters on this list, as one get's
a 'heads up' from people in the know much easier.

BTW, I'm in the process of doing the 'really only what is necessary for pg'
ipc-stuff, and was wondering if anybody already did some configuration of the
source tree towards MinGW?? How should we go about that? I would rather like
not using cygwin's sh for that ;-), and we have no 'ln' !!

Greetings,
Joerg
--
Leading SW developer - S.E.A GmbH
Mail: joerg(dot)hessdoerfer(at)sea-gmbh(dot)com
WWW: http://www.sea-gmbh.com

OK, I think I am now caught up on the Win32/cygwin discussion, and would
like to make some remarks.

First, are we doing enough to support the Win32 platform? I think the
answer is clearly "no". There are 3-5 groups/companies working on Win32
ports of PostgreSQL. We always said there would not be PostgreSQL forks
if we were doing our job to meet user needs. Well, obviously, a number
of groups see a need for a better Win32 port and we aren't meeting that
need, so they are. I believe this is one of the few cases where groups
are going out on their own because we are falling behind.

So, there is no question in my mind we need to do more to encourage
Win32 ports. Now, on to the details.

INSTALLER
---------

We clearly need an installer that is zero-hassle for users. We need to
decide on a direction for this.

GUI
---

We need a slick GUI. pgadmin2 seems to be everyone's favorite, with
pgaccess on Win32 also an option. What else do we need here?

BINARY
------

This is the big daddy. It is broken down into several sections:

FORK()

How do we handle fork()? Do we use the cygwin method that copies the
whole data segment, or put the global data in shared memory and copy
that small part manually after we create a new process?

THREADING

Related to fork(), do we implement an optionally threaded postmaster,
which eliminates CreateProcess() entirely? I don't think we will have
superior performance on Win32 without it. (This would greatly help
Solaris as well.)

IPC

We can use Cygwin, MinGW, Apache, or our own code for this. Are there
other options?

ENVIRONMENT

Lots of our code requires a unix shell and utilities. Will we continue
using cygwin for this?

---------------------------------------------------------------------------

As a roadmap, it would be good to get consensus on as many of these
items as possible so people can start working in these areas. We can
keep a web page of decisions we have made to help rally developers to
the project.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

On Wed, 2002-06-05 at 16:33, Bruce Momjian wrote:
> OK, I think I am now caught up on the Win32/cygwin discussion, and would
> like to make some remarks.
>
> First, are we doing enough to support the Win32 platform? I think the
> answer is clearly "no". There are 3-5 groups/companies working on Win32
> ports of PostgreSQL. We always said there would not be PostgreSQL forks
> if we were doing our job to meet user needs. Well, obviously, a number
> of groups see a need for a better Win32 port and we aren't meeting that
> need, so they are. I believe this is one of the few cases where groups
> are going out on their own because we are falling behind.
>
> So, there is no question in my mind we need to do more to encourage
> Win32 ports. Now, on to the details.
>
> INSTALLER
> ---------
>
> We clearly need an installer that is zero-hassle for users. We need to
> decide on a direction for this.
>
> GUI
> ---
>
> We need a slick GUI. pgadmin2 seems to be everyone's favorite, with
> pgaccess on Win32 also an option. What else do we need here?
>
> BINARY
> ------
>
> This is the big daddy. It is broken down into several sections:
>
> FORK()
>
> How do we handle fork()? Do we use the cygwin method that copies the
> whole data segment, or put the global data in shared memory and copy
> that small part manually after we create a new process?
>
> THREADING
>
> Related to fork(), do we implement an optionally threaded postmaster,
> which eliminates CreateProcess() entirely? I don't think we will have
> superior performance on Win32 without it. (This would greatly help
> Solaris as well.)
>
> IPC
>
> We can use Cygwin, MinGW, Apache, or our own code for this. Are there
> other options?
>
> ENVIRONMENT
>
> Lots of our code requires a unix shell and utilities. Will we continue
> using cygwin for this?
>
> ---------------------------------------------------------------------------
>
> As a roadmap, it would be good to get consensus on as many of these
> items as possible so people can start working in these areas. We can
> keep a web page of decisions we have made to help rally developers to
> the project.
>
> --
> Bruce Momjian | http://candle.pha.pa.us
> pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
> + If your life is a hard drive, | 830 Blythe Avenue
> + Christ can be your backup. | Drexel Hill, Pennsylvania 19026
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/users-lounge/docs/faq.html
>
Is it worth looking at how the mysql crowd did their win32 port -
(or is that intrinsically a _bad_thing_ to do..) ?

(I am guessing that is why their sources requires c++ ....)

regards

Mark

Mark kirkwood wrote:
> Is it worth looking at how the mysql crowd did their win32 port -
> (or is that intrinsically a _bad_thing_ to do..) ?
>
> (I am guessing that is why their sources requires c++ ....)

Absolutely worth seeing how MySQL does it. They use cygwin, and I
assume they aren't seeing the fork() issue because they are threaded,
and perhaps they don't use SysV IPC like we do.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

I might be naive here, but would not proper threading model remove the need
for fork() altogether? On both Unix and Win32? Should not be too hard to
come up with abstraction which encapsulates POSIX, BeOS and Win32 threads...
I am not sure how universal POSIX threads are by now. Any important Unix
platforms which don't support them yet?

This has downside of letting any bug to kill the whole thing. On the bright
side, performance should be better on some platforms (note however, Apache
group still can't come up with implementation of threaded model which would
provide better performance than forked or other models). The need to deal
with possibility of 'alien' postmaster running along with orphaned backends
would also be removed since there would be only one process.

Issue of thread safety of code will come up undoubtedly and some things will
probably have to be revamped. But in long term this is probably best way if
you want to have efficient and uniform Unix AND Win32 implementations.

I am not too familiar with Win32. Speaking about POSIX threads, it would be
something like a thread pool with low & high watermarks. Main thread would
handle thread pool and hand over requests to worker threads (blocked on
condvar). How does that sound?

-- igor

----- Original Message -----
From: "Bruce Momjian" <pgman(at)candle(dot)pha(dot)pa(dot)us>
To: "PostgreSQL-development" <pgsql-hackers(at)postgresql(dot)org>
Sent: Tuesday, June 04, 2002 11:33 PM
Subject: [HACKERS] Roadmap for a Win32 port

> OK, I think I am now caught up on the Win32/cygwin discussion, and would
> like to make some remarks.
>
> First, are we doing enough to support the Win32 platform? I think the
> answer is clearly "no". There are 3-5 groups/companies working on Win32
> ports of PostgreSQL. We always said there would not be PostgreSQL forks
> if we were doing our job to meet user needs. Well, obviously, a number
> of groups see a need for a better Win32 port and we aren't meeting that
> need, so they are. I believe this is one of the few cases where groups
> are going out on their own because we are falling behind.
>
> So, there is no question in my mind we need to do more to encourage
> Win32 ports. Now, on to the details.
>
> INSTALLER
> ---------
>
> We clearly need an installer that is zero-hassle for users. We need to
> decide on a direction for this.
>
> GUI
> ---
>
> We need a slick GUI. pgadmin2 seems to be everyone's favorite, with
> pgaccess on Win32 also an option. What else do we need here?
>
> BINARY
> ------
>
> This is the big daddy. It is broken down into several sections:
>
> FORK()
>
> How do we handle fork()? Do we use the cygwin method that copies the
> whole data segment, or put the global data in shared memory and copy
> that small part manually after we create a new process?
>
> THREADING
>
> Related to fork(), do we implement an optionally threaded postmaster,
> which eliminates CreateProcess() entirely? I don't think we will have
> superior performance on Win32 without it. (This would greatly help
> Solaris as well.)
>
> IPC
>
> We can use Cygwin, MinGW, Apache, or our own code for this. Are there
> other options?
>
> ENVIRONMENT
>
> Lots of our code requires a unix shell and utilities. Will we continue
> using cygwin for this?
>
> --------------------------------------------------------------------------
-
>
> As a roadmap, it would be good to get consensus on as many of these
> items as possible so people can start working in these areas. We can
> keep a web page of decisions we have made to help rally developers to
> the project.
>
> --
> Bruce Momjian | http://candle.pha.pa.us
> pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
> + If your life is a hard drive, | 830 Blythe Avenue
> + Christ can be your backup. | Drexel Hill, Pennsylvania 19026
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/users-lounge/docs/faq.html
>

Igor Kovalenko wrote:
> I might be naive here, but would not proper threading model remove the need
> for fork() altogether? On both Unix and Win32? Should not be too hard to
> come up with abstraction which encapsulates POSIX, BeOS and Win32 threads...
> I am not sure how universal POSIX threads are by now. Any important Unix
> platforms which don't support them yet?
>
> This has downside of letting any bug to kill the whole thing. On the bright
> side, performance should be better on some platforms (note however, Apache
> group still can't come up with implementation of threaded model which would
> provide better performance than forked or other models). The need to deal
> with possibility of 'alien' postmaster running along with orphaned backends
> would also be removed since there would be only one process.
>
> Issue of thread safety of code will come up undoubtedly and some things will
> probably have to be revamped. But in long term this is probably best way if
> you want to have efficient and uniform Unix AND Win32 implementations.
>
> I am not too familiar with Win32. Speaking about POSIX threads, it would be
> something like a thread pool with low & high watermarks. Main thread would
> handle thread pool and hand over requests to worker threads (blocked on
> condvar). How does that sound?

Good summary. I think we would support both threaded and fork()
operation, and users can control which they prefer. For a web backend
where many sessions are a single query, people may want to give up the
stability of fork() and go with threads, even on Unix.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

...
> Good summary. I think we would support both threaded and fork()
> operation, and users can control which they prefer. For a web backend
> where many sessions are a single query, people may want to give up the
> stability of fork() and go with threads, even on Unix.

I would think that we would build on our strengths of having a fork/exec
model for separate clients. A threaded model *could* benefit individual
clients who are doing queries on multiprocessor servers, and I would be
supportive of efforts to enable that.

But the requirements for that may be less severe than for managing
multiple clients within the same process, and imho there is not strong
requirement to enable the latter for our current crop of well supported
targets. If it came for free then great, but if it came with a high cost
then the choice is not as obvious. It is also not a *requirement* if we
were instead able to do the multiple threads for a single client
scenerio first.

- Thomas

One note: SGI developers discovered they could get amazing performance using
as hybrid threaded and forked-process model with apache - we might want to
look into this. They even have a library for network-communication
utilizing thier 'state threads' model. Please see:

http://state-threads.sourceforge.net/docs/st.html

Thus, on platforms where it can be supported, we should keep in mind that a
hybrid multiprocess/multithreaded postgresql might be the fastest
solution...

----- Original Message -----
From: "Bruce Momjian" <pgman(at)candle(dot)pha(dot)pa(dot)us>
To: "Igor Kovalenko" <Igor(dot)Kovalenko(at)motorola(dot)com>
Cc: "PostgreSQL-development" <pgsql-hackers(at)postgresql(dot)org>
Sent: Wednesday, June 05, 2002 4:05 PM
Subject: Re: [HACKERS] Roadmap for a Win32 port

> Igor Kovalenko wrote:
> > I might be naive here, but would not proper threading model remove the
need
> > for fork() altogether? On both Unix and Win32? Should not be too hard to
> > come up with abstraction which encapsulates POSIX, BeOS and Win32
threads...
> > I am not sure how universal POSIX threads are by now. Any important Unix
> > platforms which don't support them yet?
> >
> > This has downside of letting any bug to kill the whole thing. On the
bright
> > side, performance should be better on some platforms (note however,
Apache
> > group still can't come up with implementation of threaded model which
would
> > provide better performance than forked or other models). The need to
deal
> > with possibility of 'alien' postmaster running along with orphaned
backends
> > would also be removed since there would be only one process.
> >
> > Issue of thread safety of code will come up undoubtedly and some things
will
> > probably have to be revamped. But in long term this is probably best way
if
> > you want to have efficient and uniform Unix AND Win32 implementations.
> >
> > I am not too familiar with Win32. Speaking about POSIX threads, it would
be
> > something like a thread pool with low & high watermarks. Main thread
would
> > handle thread pool and hand over requests to worker threads (blocked on
> > condvar). How does that sound?
>
> Good summary. I think we would support both threaded and fork()
> operation, and users can control which they prefer. For a web backend
> where many sessions are a single query, people may want to give up the
> stability of fork() and go with threads, even on Unix.
>
> --
> Bruce Momjian | http://candle.pha.pa.us
> pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
> + If your life is a hard drive, | 830 Blythe Avenue
> + Christ can be your backup. | Drexel Hill, Pennsylvania 19026
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo(at)postgresql(dot)org

On Wed, 5 Jun 2002 18:50:46 -0400
"Jon Franz" <coventry(at)one(dot)net> wrote:
> One note: SGI developers discovered they could get amazing performance using
> as hybrid threaded and forked-process model with apache - we might want to
> look into this. They even have a library for network-communication
> utilizing thier 'state threads' model.

I think ST is designed for network I/O-bound apps -- last I checked,
disk I/O will still block an entire ST process. While you can get around
that by using another process to do disk I/O, it sounds like ST won't be
that useful.

However, Chris KL. (I believe) raised the idea of using POSIX AIO for
PostgreSQL. Without having looked into it extensively, this technique
sounds promising. Perhaps someone who has looked into this further
(e.g. someone from Redhat) can comment?

Cheers,

Neil

--
Neil Conway <neilconway(at)rogers(dot)com>
PGP Key ID: DB3C29FC

Yes I proposed to use the GNU Pth library instead. It's an event
demultiplexer just like the sgi library, but has a posix thread interface.
This architecture is actually the more robust and also the more scalable. On
a single processor server, you don't have the multi-thread synchronization
and context switching overhead and you also take full advantage of
multi-processor servers when you create several processes. Plus you have
much less concern about global variables.

Also for those concerned about the licence of this library here is an
abstract of it:
"The author places this library under the LGPL to make sure that it
can be used both commercially and non-commercially provided that
modifications to the code base are always donated back to the official
code base under the same license conditions. Please keep in mind that
especially using this library in code not staying under the GPL or
the LGPL _is_ allowed and that any taint or license creap into code
that uses the library is not the authors intention. It is just the
case that _including_ this library into the source tree of other
applications is a little bit more inconvinient because of the LGPL.
But it has to be this way for good reasons. And keep in mind that
inconvinient doesn't mean not allowed or even impossible."

So it can be used in both commercial and non commercial project.

----- Original Message -----
From: "Jon Franz" <coventry(at)one(dot)net>
To: <pgsql-hackers(at)postgresql(dot)org>
Sent: Thursday, June 06, 2002 8:50 AM
Subject: Re: [HACKERS] Roadmap for a Win32 port

> One note: SGI developers discovered they could get amazing performance
using
> as hybrid threaded and forked-process model with apache - we might want to
> look into this. They even have a library for network-communication
> utilizing thier 'state threads' model. Please see:
>
> http://state-threads.sourceforge.net/docs/st.html
>
> Thus, on platforms where it can be supported, we should keep in mind that
a
> hybrid multiprocess/multithreaded postgresql might be the fastest
> solution...
>
>
> ----- Original Message -----
> From: "Bruce Momjian" <pgman(at)candle(dot)pha(dot)pa(dot)us>
> To: "Igor Kovalenko" <Igor(dot)Kovalenko(at)motorola(dot)com>
> Cc: "PostgreSQL-development" <pgsql-hackers(at)postgresql(dot)org>
> Sent: Wednesday, June 05, 2002 4:05 PM
> Subject: Re: [HACKERS] Roadmap for a Win32 port
>
>
> > Igor Kovalenko wrote:
> > > I might be naive here, but would not proper threading model remove the
> need
> > > for fork() altogether? On both Unix and Win32? Should not be too hard
to
> > > come up with abstraction which encapsulates POSIX, BeOS and Win32
> threads...
> > > I am not sure how universal POSIX threads are by now. Any important
Unix
> > > platforms which don't support them yet?
> > >
> > > This has downside of letting any bug to kill the whole thing. On the
> bright
> > > side, performance should be better on some platforms (note however,
> Apache
> > > group still can't come up with implementation of threaded model which
> would
> > > provide better performance than forked or other models). The need to
> deal
> > > with possibility of 'alien' postmaster running along with orphaned
> backends
> > > would also be removed since there would be only one process.
> > >
> > > Issue of thread safety of code will come up undoubtedly and some
things
> will
> > > probably have to be revamped. But in long term this is probably best
way
> if
> > > you want to have efficient and uniform Unix AND Win32 implementations.
> > >
> > > I am not too familiar with Win32. Speaking about POSIX threads, it
would
> be
> > > something like a thread pool with low & high watermarks. Main thread
> would
> > > handle thread pool and hand over requests to worker threads (blocked
on
> > > condvar). How does that sound?
> >
> > Good summary. I think we would support both threaded and fork()
> > operation, and users can control which they prefer. For a web backend
> > where many sessions are a single query, people may want to give up the
> > stability of fork() and go with threads, even on Unix.
> >
> > --
> > Bruce Momjian | http://candle.pha.pa.us
> > pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
> > + If your life is a hard drive, | 830 Blythe Avenue
> > + Christ can be your backup. | Drexel Hill, Pennsylvania
19026
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 1: subscribe and unsubscribe commands go to majordomo(at)postgresql(dot)org
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo(at)postgresql(dot)org
>
>

Neil Conway wrote:
> On Wed, 5 Jun 2002 18:50:46 -0400
> "Jon Franz" <coventry(at)one(dot)net> wrote:
> > One note: SGI developers discovered they could get amazing performance using
> > as hybrid threaded and forked-process model with apache - we might want to
> > look into this. They even have a library for network-communication
> > utilizing thier 'state threads' model.
>
> I think ST is designed for network I/O-bound apps -- last I checked,
> disk I/O will still block an entire ST process. While you can get around
> that by using another process to do disk I/O, it sounds like ST won't be
> that useful.
>
> However, Chris KL. (I believe) raised the idea of using POSIX AIO for
> PostgreSQL. Without having looked into it extensively, this technique
> sounds promising. Perhaps someone who has looked into this further
> (e.g. someone from Redhat) can comment?

I know Red Hat is interested in AIO. Only a few OS's support it so it
was hard to get exited about it at the time, but with threading, a
AIO-specific module could be attempted.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Hello Thomas,

Wednesday, June 5, 2002, 7:02:33 PM, you wrote:

TL> ...
>> Good summary. I think we would support both threaded and fork()
>> operation, and users can control which they prefer. For a web backend
>> where many sessions are a single query, people may want to give up the
>> stability of fork() and go with threads, even on Unix.

TL> I would think that we would build on our strengths of having a fork/exec
TL> model for separate clients. A threaded model *could* benefit individual
TL> clients who are doing queries on multiprocessor servers, and I would be
TL> supportive of efforts to enable that.
Just a note - this is also the solution adopted by Interbase/Firebird
and it seems interesting. They already had the same problems
PostgreSQL has been under today.
Those interested in read about Interbase's architeture, please refer
to http://community.borland.com/article/0,1410,23217,00.html.
"Classic" is the fork() model, and the "SuperServer" is the threaded
model.
-------------
Best regards,
Steve Howe mailto:howe(at)carcass(dot)dhs(dot)org

Gnu Pth also supports AIO
----- Original Message -----
From: "Nicolas Bazin" <nbazin(at)ingenico(dot)com(dot)au>
To: "Jon Franz" <coventry(at)one(dot)net>; <pgsql-hackers(at)postgresql(dot)org>
Sent: Thursday, June 06, 2002 10:50 AM
Subject: Re: [HACKERS] Roadmap for a Win32 port

> Yes I proposed to use the GNU Pth library instead. It's an event
> demultiplexer just like the sgi library, but has a posix thread interface.
> This architecture is actually the more robust and also the more scalable.
On
> a single processor server, you don't have the multi-thread synchronization
> and context switching overhead and you also take full advantage of
> multi-processor servers when you create several processes. Plus you have
> much less concern about global variables.
>
> Also for those concerned about the licence of this library here is an
> abstract of it:
> "The author places this library under the LGPL to make sure that it
> can be used both commercially and non-commercially provided that
> modifications to the code base are always donated back to the official
> code base under the same license conditions. Please keep in mind that
> especially using this library in code not staying under the GPL or
> the LGPL _is_ allowed and that any taint or license creap into code
> that uses the library is not the authors intention. It is just the
> case that _including_ this library into the source tree of other
> applications is a little bit more inconvinient because of the LGPL.
> But it has to be this way for good reasons. And keep in mind that
> inconvinient doesn't mean not allowed or even impossible."
>
> So it can be used in both commercial and non commercial project.
>
>
> ----- Original Message -----
> From: "Jon Franz" <coventry(at)one(dot)net>
> To: <pgsql-hackers(at)postgresql(dot)org>
> Sent: Thursday, June 06, 2002 8:50 AM
> Subject: Re: [HACKERS] Roadmap for a Win32 port
>
>
> > One note: SGI developers discovered they could get amazing performance
> using
> > as hybrid threaded and forked-process model with apache - we might want
to
> > look into this. They even have a library for network-communication
> > utilizing thier 'state threads' model. Please see:
> >
> > http://state-threads.sourceforge.net/docs/st.html
> >
> > Thus, on platforms where it can be supported, we should keep in mind
that
> a
> > hybrid multiprocess/multithreaded postgresql might be the fastest
> > solution...
> >
> >
> > ----- Original Message -----
> > From: "Bruce Momjian" <pgman(at)candle(dot)pha(dot)pa(dot)us>
> > To: "Igor Kovalenko" <Igor(dot)Kovalenko(at)motorola(dot)com>
> > Cc: "PostgreSQL-development" <pgsql-hackers(at)postgresql(dot)org>
> > Sent: Wednesday, June 05, 2002 4:05 PM
> > Subject: Re: [HACKERS] Roadmap for a Win32 port
> >
> >
> > > Igor Kovalenko wrote:
> > > > I might be naive here, but would not proper threading model remove
the
> > need
> > > > for fork() altogether? On both Unix and Win32? Should not be too
hard
> to
> > > > come up with abstraction which encapsulates POSIX, BeOS and Win32
> > threads...
> > > > I am not sure how universal POSIX threads are by now. Any important
> Unix
> > > > platforms which don't support them yet?
> > > >
> > > > This has downside of letting any bug to kill the whole thing. On the
> > bright
> > > > side, performance should be better on some platforms (note however,
> > Apache
> > > > group still can't come up with implementation of threaded model
which
> > would
> > > > provide better performance than forked or other models). The need to
> > deal
> > > > with possibility of 'alien' postmaster running along with orphaned
> > backends
> > > > would also be removed since there would be only one process.
> > > >
> > > > Issue of thread safety of code will come up undoubtedly and some
> things
> > will
> > > > probably have to be revamped. But in long term this is probably best
> way
> > if
> > > > you want to have efficient and uniform Unix AND Win32
implementations.
> > > >
> > > > I am not too familiar with Win32. Speaking about POSIX threads, it
> would
> > be
> > > > something like a thread pool with low & high watermarks. Main thread
> > would
> > > > handle thread pool and hand over requests to worker threads (blocked
> on
> > > > condvar). How does that sound?
> > >
> > > Good summary. I think we would support both threaded and fork()
> > > operation, and users can control which they prefer. For a web backend
> > > where many sessions are a single query, people may want to give up the
> > > stability of fork() and go with threads, even on Unix.
> > >
> > > --
> > > Bruce Momjian | http://candle.pha.pa.us
> > > pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
> > > + If your life is a hard drive, | 830 Blythe Avenue
> > > + Christ can be your backup. | Drexel Hill, Pennsylvania
> 19026
> > >
> > > ---------------------------(end of
broadcast)---------------------------
> > > TIP 1: subscribe and unsubscribe commands go to
majordomo(at)postgresql(dot)org
> >
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 1: subscribe and unsubscribe commands go to majordomo(at)postgresql(dot)org
> >
> >
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
>
>

Hello Bruce,

Wednesday, June 5, 2002, 1:33:44 AM, you wrote:

BM> INSTALLER
BM> ---------

BM> We clearly need an installer that is zero-hassle for users. We need to
BM> decide on a direction for this.
I suggest Nullsoft install system
(http://www.nullsoft.com/free/nsis/). It's real good and very simple
to use. I can help on this if you want.

BM> ENVIRONMENT

BM> Lots of our code requires a unix shell and utilities. Will we continue
BM> using cygwin for this?
There are other ports ( http://unxutils.sourceforge.net/ ) that won't
require Cygwin but they won't provide an environment so complete as
Cygwin does.

I also would like to empathize that probably a small GUI for
controlling the PostgreSQL service/application would be nice. I think
about something sitting in the system tray like MSSQL, Oracle,
Interbase, etc. does.
I could code this in Delphi if you like. I don't have experience in
writing GUI apps in C. There is an open source versions of Delphi so
it won't be a problem compiling it.
Also coming with this, a code for starting the PostgreSQL as a service
would be really nice. For those from UNIX world that don't know what a
service is, think about it as a daemon for Windows. A service can be
automatically started when the machine boots up.

-------------
Best regards,
Steve Howe mailto:howe(at)carcass(dot)dhs(dot)org

I think SGI gets amazing performance because they have very good (efficient)
synchronisation primitives on SGI. Some proprietary light-weight mutexes.
Using threaded or mixed model just by itself is not going to do a miracle.
Threads will save you some context switch time, but that will probably
translate into lower CPU usage rather than performance boost. And if your
mutexes are not fast or awkwardly implemented (say Linux), it might be even
worse. Apache is not all that fast on Linux as on SGI, whatever model you
chose. I also doubt that purely threaded model would be slower than mixed
one.

Now about the AIO model. It is useful when you need to do something else
while I/O requests are being processed as long as platform does it in some
useful way. If all you can do is to submit requests and keep sitting in
select/poll then AIO does not buy you anything you can't get by just using
threaded model. However, if you can tag the requests and set up
notifications, then few I/O threads could handle relatively large number of
requests from different clients. Note, this means you don't have any
association between clients and servers at all, there is pool of generic I/O
threads which serve requests from whoever they come. It saves system
resources and scales very well. It also provides interesting possibilities
for fault recovery - since handlers are generic all the state information
would have to be kept in some kind of global context area. That area can be
saved into persistent memory or dumped onto disk and *recovered* after a
forced restart. Server and library could be designed in such a way that
clients may continue where they left with a recoverable error.

In POSIX AIO model you can tag requests and set up notifications via
synchronous signals. You wait for them *synchronously* in 'waiter' thread
via sigwaitinfo() and avoid the headache of asynchronous signals hitting you
any time... Unfortunately on some platforms (Solaris) the depth of
synchronous signal queue is fixed at magic value 32 (and not adjustable).
This may not be a problem if you're sure that waiting thread will be able to
drain the queue faster than it gets filled with notifications... but I'm not
sure there is a portable way to guarantee that, so you need to check for
overloads and handle them... that complicates things. On Solaris you also
need a mile of compiler/linker switches to even get this scheme to work and
I am afraid other platforms may not support it at all (but then again, they
may not support AIO to begin with).

And speaking about getting best of all worlds. Note how Apache spent nearly
3 years developing their portable Multi-Processing Modules scheme. What they
got for that is handful of models neither of which perform noticeably better
than original pre-fork() model. Trying to swallow all possible ways to
handle things on all possible platforms usually does not produce very fast
code. It tends to produce very complex code with mediocre performance and
introduces extra complexity into configuration process. If you consider all
that was done mostly to support Win32, one might doubt if it was worth the
while.

What I am trying to say is, extra complexity in model to squeeze few percent
of performance is not a wise investment of time and efforts. On Win32 you
don't really compete in terms of performance. You compete in terms of
easyness and features. Spend 3 years trying to support Windows and Unix in
most optimal way including all subvariants of Unix ... meanwhile MSFT will
come up with some bundled SQL server. It probably will have more features
since they will spend time doing features rather than inventing a model to
support gazillion of platforms. Chances are, it will be faster too - due to
better integration with OS and better compiler.

I am not in position to tell you what to do guys. But if I was asked, I'd
say supporting Win32 is only worth it if it comes as a natural result of a
simple, coherent and uniform model applied to Unix. Threaded model may not
have as much inherent stability as forked/mixed, but it has inherent
simplicity and better Unix/Windows/BeOS portability. It can be done faster
and simpler code will make work on features easier.

Regards,
- Igor

"There are 2 ways to design an efficient system - first is to design it so
complex that there are no obvious deficiencies, second is to design it so
simple that there are obviously no deficiencies. Second way is much harder"
(author unknown to me)

----- Original Message -----
From: "Neil Conway" <nconway(at)klamath(dot)dyndns(dot)org>
To: "Jon Franz" <coventry(at)one(dot)net>
Cc: <pgsql-hackers(at)postgresql(dot)org>
Sent: Wednesday, June 05, 2002 7:05 PM
Subject: Re: [HACKERS] Roadmap for a Win32 port

> On Wed, 5 Jun 2002 18:50:46 -0400
> "Jon Franz" <coventry(at)one(dot)net> wrote:
> > One note: SGI developers discovered they could get amazing performance
using
> > as hybrid threaded and forked-process model with apache - we might want
to
> > look into this. They even have a library for network-communication
> > utilizing thier 'state threads' model.
>
> I think ST is designed for network I/O-bound apps -- last I checked,
> disk I/O will still block an entire ST process. While you can get around
> that by using another process to do disk I/O, it sounds like ST won't be
> that useful.
>
> However, Chris KL. (I believe) raised the idea of using POSIX AIO for
> PostgreSQL. Without having looked into it extensively, this technique
> sounds promising. Perhaps someone who has looked into this further
> (e.g. someone from Redhat) can comment?
>
> Cheers,
>
> Neil
>
> --
> Neil Conway <neilconway(at)rogers(dot)com>
> PGP Key ID: DB3C29FC
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
>

Here is a summary of the responses to my Win32 roadmap. I hope this
will allow further discussion.

---------------------------------------------------------------------------

INSTALLER
---------
Cygwin Setup.exe http://cygwin.com
Nullsoft http://www.nullsoft.com/free/nsis/

GUI
---
pgAdmin2 http://pgadmin.postgresql.org/pgadmin2.php?ContentID=1
pgaccess http://pgaccess.org/
Java admin (to be written)
Dev-C++ admin (to be written) http://sourceforge.net/projects/dev-cpp/

BINARY
------

FORK()

cygwin fork() http://cygwin.com
CreateProcess() and copy global area

THREADING

Posix threads
Gnu pth http://www.gnu.org/software/pth/
ST http://state-threads.sourceforge.net/docs/st.html
(single-session multi-threading possible)
(Posix AIO is possible)

IPC

Cygwin http://cygwin.com
MinGW http://www.mingw.org/
ACE http://www.cs.wustl.edu/~schmidt/ACE.html
APR http://apr.apache.org/
Our own

ENVIRONMENT

Cygwin http://cygwin.com
UnxUtils http://unxutils.sourceforge.net/
Write own initdb

IMPLEMENTATIONS
---------------
PostgreSQLe http://hp.vector.co.jp/authors/VA023283/PostgreSQLe.html
Dbexperts http://www.dbexperts.net/postgresql
Connx http://www.connx.com/
gborg http://gborg.postgresql.org/project/winpackage/projdisplay.php
Interbase http://community.borland.com/article/0,1410,23217,00.html

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Igor Kovalenko wrote:
> I think SGI gets amazing performance because they have very good (efficient)
> synchronization primitives on SGI. Some proprietary light-weight mutexes.
> Using threaded or mixed model just by itself is not going to do a miracle.
> Threads will save you some context switch time, but that will probably
> translate into lower CPU usage rather than performance boost. And if your
> mutexes are not fast or awkwardly implemented (say Linux), it might be even
> worse. Apache is not all that fast on Linux as on SGI, whatever model you
> chose. I also doubt that purely threaded model would be slower than mixed
> one.

Let me throw out an idea. I have been mentioning full fork, light
fork(copy globals only), and threading as possible solutions.

Another idea uses neither threading nor copying. It is the old system
we used before I removed exec() from our code. We used to pass the
database name as an argument to an exec'ed postgres binary that
continued with the database connection.

We removed the exec, then started moving what we could into the
postmaster so each backend didn't need to do the initialization.

One solution is to return to that for Win32 only, so instead of doing:

initialization()
want for connection()
fork backend()

we do for Win32:

want for connection()
exec backend()
initialization()

It wouldn't be hard to do. We would still do CreateProcess rather than
CreateThread, but it eliminates the fork/threading issues. We don't
know the database before the connection arrives, so we don't do a whole
lot of initialization.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Hi Bruce,

You obviosuly missed my recent posting advertising the homepage
of Konstantin Knizhnik?

Make sure to have a look: http://www.garret.ru/~knizhnik/

You find there -everything- concerning multiplatform IPC,
threading and even some extraordinary, complete database
backends that are superior to the database backends
previously available as open source (including PostgreSQL,
I'm afraid...). The licensing of all of this stuff is -public domain-.
I think this should really be worth a look/discussion/mentioning.

Here an excerpt of my last email, describing the furios list
of features abailable in GOODS:

Some core features of the GOODS backend (as they come to my mind):
-> full ACID transaction support, incl. distributed transactions
-> Multiple stoarge servers distributed over a TCP/ID network
-> multible reader/single writer (MVCC)
-> dual client side object cache
-> online backup (snapshot backup AND permanent backup)
-> nested transactions on object level
-> transaction isolation levels on object level
-> object level shared and exclusive locks
-> excellent C++ programming interface
-> WAL
-> garbage collection for no longer reference database objects (online VACUUM)
-> fully thread safe client interface
-> JAVA client API
-> very high performance as a result of a lot of fine tuning (better
perfomance than berkeley db in my benchmarks!!!)
-> asyncrous event notification on object instance modification
-> extremly high code quality
-> a one person effort, hence a very clean design
-> the most relevant platforms are supported out of the box
-> complete build is done in less than a minute on my machine
-> it's documented
-> it's tested and found to be working for a while now
...

kind regards,
Robert Schrem

On Thursday 06 June 2002 04:57, you wrote:
> Here is a summary of the responses to my Win32 roadmap. I hope this
> will allow further discussion.
>
> ---------------------------------------------------------------------------
>
> INSTALLER
> ---------
> Cygwin Setup.exe http://cygwin.com
> Nullsoft http://www.nullsoft.com/free/nsis/
>
> GUI
> ---
> pgAdmin2
> http://pgadmin.postgresql.org/pgadmin2.php?ContentID=1 pgaccess
> http://pgaccess.org/
> Java admin (to be written)
> Dev-C++ admin (to be written)
> http://sourceforge.net/projects/dev-cpp/
>
> BINARY
> ------
>
>
> FORK()
>
> cygwin fork() http://cygwin.com
> CreateProcess() and copy global area
>
> THREADING
>
> Posix threads
> Gnu pth http://www.gnu.org/software/pth/
> ST
> http://state-threads.sourceforge.net/docs/st.html (single-session
> multi-threading possible)
> (Posix AIO is possible)
>
> IPC
>
> Cygwin http://cygwin.com
> MinGW http://www.mingw.org/
> ACE
> http://www.cs.wustl.edu/~schmidt/ACE.html APR
> http://apr.apache.org/
> Our own
>
> ENVIRONMENT
>
> Cygwin http://cygwin.com
> UnxUtils http://unxutils.sourceforge.net/
> Write own initdb
>
>
> IMPLEMENTATIONS
> ---------------
> PostgreSQLe
> http://hp.vector.co.jp/authors/VA023283/PostgreSQLe.html Dbexperts
> http://www.dbexperts.net/postgresql Connx
> http://www.connx.com/
> gborg
> http://gborg.postgresql.org/project/winpackage/projdisplay.php Interbase
>
> http://community.borland.com/article/0,1410,23217,00.html

Bruce Momjian wrote:
>
> Let me throw out an idea. I have been mentioning full fork, light
> fork(copy globals only), and threading as possible solutions.
>
> Another idea uses neither threading nor copying. It is the old system
> we used before I removed exec() from our code. We used to pass the
> database name as an argument to an exec'ed postgres binary that
> continued with the database connection.
>
> We removed the exec, then started moving what we could into the
> postmaster so each backend didn't need to do the initialization.
>
> One solution is to return to that for Win32 only, so instead of doing:
>
> initialization()
> want for connection()
> fork backend()
>
> we do for Win32:
>
> want for connection()
> exec backend()
> initialization()

Summarizes pretty much what we discussed Monday on the phone.
Except that the postmaster still has to initialize the shared
memory and other stuff. It's just that the backends and
helper processes need to reinitialize themself (attach).

> It wouldn't be hard to do. We would still do CreateProcess rather than
> CreateThread, but it eliminates the fork/threading issues. We don't
> know the database before the connection arrives, so we don't do a whole
> lot of initialization.

All I see so far is the reading of the postgresql.conf, the
pg_hba.conf and the password files. Nothing fancy and the
postmaster could easily write out a binary content only file
that the backends then read, eliminating the parsing
overhead.

The bad news is that Tom is right. We did a terrible job in
using the new side effect, that the shared memory segment is
at the same address in all forked processes, after removing
the need to reattach.

In detail the XLog code, the FreeSpaceMap code and the
"shared memory" hashtable code now use pointers, located in
shared memory. For the XLog and FreeSpace code this is
understandable, because they where developed under the fork()
only model. But the dynahash code used offsets only until
v7.1!

All three (no claim that that's all) make it impossible to
ever have someone attaching to the shared memory from the
outside. So with these moves we made the shared memory a
"Postmaster and children" only thing. Raises the question,
why we need an IPC key at all any more.

Anyhow, looks as if I can get that fork() vs. fork()+exec()
feature done pretty soon. It'll be controlled by another
Postmaster commandline switch. After cleaning up the mess I
did to get it working quick, I'll provide a patch for
discussion.

Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck(at)Yahoo(dot)com #

Added to the list. Thanks.

---------------------------------------------------------------------------

Robert Schrem wrote:
> Hi Bruce,
>
> You obviosuly missed my recent posting advertising the homepage
> of Konstantin Knizhnik?
>
> Make sure to have a look: http://www.garret.ru/~knizhnik/
>
> You find there -everything- concerning multiplatform IPC,
> threading and even some extraordinary, complete database
> backends that are superior to the database backends
> previously available as open source (including PostgreSQL,
> I'm afraid...). The licensing of all of this stuff is -public domain-.
> I think this should really be worth a look/discussion/mentioning.
>
> Here an excerpt of my last email, describing the furios list
> of features abailable in GOODS:
>
> Some core features of the GOODS backend (as they come to my mind):
> -> full ACID transaction support, incl. distributed transactions
> -> Multiple stoarge servers distributed over a TCP/ID network
> -> multible reader/single writer (MVCC)
> -> dual client side object cache
> -> online backup (snapshot backup AND permanent backup)
> -> nested transactions on object level
> -> transaction isolation levels on object level
> -> object level shared and exclusive locks
> -> excellent C++ programming interface
> -> WAL
> -> garbage collection for no longer reference database objects (online VACUUM)
> -> fully thread safe client interface
> -> JAVA client API
> -> very high performance as a result of a lot of fine tuning (better
> perfomance than berkeley db in my benchmarks!!!)
> -> asyncrous event notification on object instance modification
> -> extremly high code quality
> -> a one person effort, hence a very clean design
> -> the most relevant platforms are supported out of the box
> -> complete build is done in less than a minute on my machine
> -> it's documented
> -> it's tested and found to be working for a while now
> ...
>
> kind regards,
> Robert Schrem
>
>
> On Thursday 06 June 2002 04:57, you wrote:
> > Here is a summary of the responses to my Win32 roadmap. I hope this
> > will allow further discussion.
> >
> > ---------------------------------------------------------------------------
> >
> > INSTALLER
> > ---------
> > Cygwin Setup.exe http://cygwin.com
> > Nullsoft http://www.nullsoft.com/free/nsis/
> >
> > GUI
> > ---
> > pgAdmin2
> > http://pgadmin.postgresql.org/pgadmin2.php?ContentID=1 pgaccess
> > http://pgaccess.org/
> > Java admin (to be written)
> > Dev-C++ admin (to be written)
> > http://sourceforge.net/projects/dev-cpp/
> >
> > BINARY
> > ------
> >
> >
> > FORK()
> >
> > cygwin fork() http://cygwin.com
> > CreateProcess() and copy global area
> >
> > THREADING
> >
> > Posix threads
> > Gnu pth http://www.gnu.org/software/pth/
> > ST
> > http://state-threads.sourceforge.net/docs/st.html (single-session
> > multi-threading possible)
> > (Posix AIO is possible)
> >
> > IPC
> >
> > Cygwin http://cygwin.com
> > MinGW http://www.mingw.org/
> > ACE
> > http://www.cs.wustl.edu/~schmidt/ACE.html APR
> > http://apr.apache.org/
> > Our own
> >
> > ENVIRONMENT
> >
> > Cygwin http://cygwin.com
> > UnxUtils http://unxutils.sourceforge.net/
> > Write own initdb
> >
> >
> > IMPLEMENTATIONS
> > ---------------
> > PostgreSQLe
> > http://hp.vector.co.jp/authors/VA023283/PostgreSQLe.html Dbexperts
> > http://www.dbexperts.net/postgresql Connx
> > http://www.connx.com/
> > gborg
> > http://gborg.postgresql.org/project/winpackage/projdisplay.php Interbase
> >
> > http://community.borland.com/article/0,1410,23217,00.html
>
>

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Jan Wieck wrote:
> > One solution is to return to that for Win32 only, so instead of doing:
> >
> > initialization()
> > want for connection()
> > fork backend()
> >
> > we do for Win32:
> >
> > want for connection()
> > exec backend()
> > initialization()
>
> Summarizes pretty much what we discussed Monday on the phone.
> Except that the postmaster still has to initialize the shared
> memory and other stuff. It's just that the backends and
> helper processes need to reinitialize themself (attach).

Yes, obviously I simplified, and I do believe our optimizations are
helping on Unix. It is just that I think for Win32 the fork is more
harmful than removing those optimizations.

One thing that may not have been clear is that we don't need to play
with globals at all. We just pass whatever info we want to the child
via command-line arguments, rather than shared memory.

> > It wouldn't be hard to do. We would still do CreateProcess rather than
> > CreateThread, but it eliminates the fork/threading issues. We don't
> > know the database before the connection arrives, so we don't do a whole
> > lot of initialization.
>
> All I see so far is the reading of the postgresql.conf, the
> pg_hba.conf and the password files. Nothing fancy and the
> postmaster could easily write out a binary content only file
> that the backends then read, eliminating the parsing
> overhead.

Yes, that is clearly possible. Another option is to just write out a
no-comments, no-whitespace version of each file and just have the
backends read those. The advantage is that we can use the same code to
read them, and I don't think it would be any slower than a binary file.

> The bad news is that Tom is right. We did a terrible job in
> using the new side effect, that the shared memory segment is
> at the same address in all forked processes, after removing
> the need to reattach.
>
> In detail the XLog code, the FreeSpaceMap code and the
> "shared memory" hashtable code now use pointers, located in
> shared memory. For the XLog and FreeSpace code this is
> understandable, because they where developed under the fork()
> only model. But the dynahash code used offsets only until
> v7.1!
>
> All three (no claim that that's all) make it impossible to
> ever have someone attaching to the shared memory from the
> outside. So with these moves we made the shared memory a
> "Postmaster and children" only thing. Raises the question,
> why we need an IPC key at all any more.

Well, we could force shmat() to bind to the same address, but I suspect
that might fail in some cases.

> Anyhow, looks as if I can get that fork() vs. fork()+exec()
> feature done pretty soon. It'll be controlled by another
> Postmaster commandline switch. After cleaning up the mess I
> did to get it working quick, I'll provide a patch for
> discussion.

Yes, very little impact. We then need someone to do some Win32 timings
to see if things have improved. As Tom mentioned, we need some hard
numbers for these things. In fact, I would like a Win32 test that takes
our code and compares fork(), then exit(), with CreateProcess(), exit().
It doesn't have create a db session, but I would like to see some
timings to know what we are gaining. Heck, time CreateThread too and
let's see what that shows.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Bruce Momjian writes:

> Lots of our code requires a unix shell and utilities. Will we continue
> using cygwin for this?

We should probably get rid of using shell scripts for application programs
altogether, for a number of reasons besides this one, such as the
inability to properly handle input values with spaces, commas, etc. (we
probably don't handle very long values either on some platforms), the
inability to maintain open database connections so that createlang needs
to prompt for the same password thrice, general portable scripting
headaches, and the lack of internationalization facilities.

I'd even volunteer to do this. Comments?

--
Peter Eisentraut peter_e(at)gmx(dot)net

Peter Eisentraut wrote:
> Bruce Momjian writes:
>
> > Lots of our code requires a unix shell and utilities. Will we continue
> > using cygwin for this?
>
> We should probably get rid of using shell scripts for application programs
> altogether, for a number of reasons besides this one, such as the
> inability to properly handle input values with spaces, commas, etc. (we
> probably don't handle very long values either on some platforms), the
> inability to maintain open database connections so that createlang needs
> to prompt for the same password thrice, general portable scripting
> headaches, and the lack of internationalization facilities.
>
> I'd even volunteer to do this. Comments?

I know I have discouraged it because I think shell script language has a
good toolset for those applications. I have fixed all the spacing
issues.

What language where you thinking of using? C?

Also, it seems Win32 doesn't need these scripts, except initdb.
PostgreSQLe didn't use the, it just did initdb, and the rest were done
using a GUI. However, initdb would remain a problem. PostgreSQLe wrote
its own.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Bruce Momjian writes:

> GUI
> ---
> pgAdmin2 http://pgadmin.postgresql.org/pgadmin2.php?ContentID=1
> pgaccess http://pgaccess.org/
> Java admin (to be written)
> Dev-C++ admin (to be written) http://sourceforge.net/projects/dev-cpp/

Surely Unix folks would like a GUI as well?

--
Peter Eisentraut peter_e(at)gmx(dot)net

How about a SOAP interface and a web-based front end that provides the cross
platform support? My company's TIBET framework would provide a solid
foundation for this kind of admin suite. In fact, we're already in the
planning stages on doing just that.

ss

Scott Shattuck
Technical Pursuit Inc.

----- Original Message -----
From: "Peter Eisentraut" <peter_e(at)gmx(dot)net>
To: "Bruce Momjian" <pgman(at)candle(dot)pha(dot)pa(dot)us>
Cc: "PostgreSQL-development" <pgsql-hackers(at)postgreSQL(dot)org>
Sent: Friday, June 07, 2002 11:42 AM
Subject: Re: [HACKERS] Roadmap for a Win32 port

> Bruce Momjian writes:
>
> > GUI
> > ---
> > pgAdmin2
http://pgadmin.postgresql.org/pgadmin2.php?ContentID=1
> > pgaccess http://pgaccess.org/
> > Java admin (to be written)
> > Dev-C++ admin (to be written)
http://sourceforge.net/projects/dev-cpp/
>
> Surely Unix folks would like a GUI as well?
>
> --
> Peter Eisentraut peter_e(at)gmx(dot)net
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo(at)postgresql(dot)org

Bruce Momjian writes:

> I know I have discouraged it because I think shell script language has a
> good toolset for those applications. I have fixed all the spacing
> issues.

My point is that it is not, for the reasons that I listed. Handling
spaces is a small part of one of the several problems, there are problems
with newlines, tabs, commas, slashes, quotes -- everytime you call sed or
read you lose one character.

> What language where you thinking of using? C?

Yes, that way we can share code (pg_dumpall<->pg_dump, initdb<->postgres),
use the established internationalization facilities, and use libpq
directly in create* and drop*.

> Also, it seems Win32 doesn't need these scripts, except initdb.

The utility of these programs is independent of the platform. If we think
pg_dumpall is not useful, then let's remove it.

--
Peter Eisentraut peter_e(at)gmx(dot)net

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
>> Also, it seems Win32 doesn't need these scripts, except initdb.

> The utility of these programs is independent of the platform. If we think
> pg_dumpall is not useful, then let's remove it.

I have been seriously considering converting pg_dumpall to C anyway,
because it's already *very* messy, and I don't see any reasonable
way to make it support dumping per-database and per-user config
settings. (Do you really want to try to parse array values in a
shell script?)

(I'd actually consider making pg_dumpall a part of the pg_dump
executable; then it could invoke pg_dump as a subroutine call...)

If Peter's got the time/energy to convert 'em all, I'm for it.

regards, tom lane

Tom Lane wrote:
> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> >> Also, it seems Win32 doesn't need these scripts, except initdb.
>
> > The utility of these programs is independent of the platform. If we think
> > pg_dumpall is not useful, then let's remove it.
>
> I have been seriously considering converting pg_dumpall to C anyway,
> because it's already *very* messy, and I don't see any reasonable
> way to make it support dumping per-database and per-user config
> settings. (Do you really want to try to parse array values in a
> shell script?)
>
> (I'd actually consider making pg_dumpall a part of the pg_dump
> executable; then it could invoke pg_dump as a subroutine call...)
>
> If Peter's got the time/energy to convert 'em all, I'm for it.

Yea, shame it will now take 15 lines of C code to do what we could do in
1 line of shell script but I don't see any other option.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> Yea, shame it will now take 15 lines of C code to do what we could do in
> 1 line of shell script but I don't see any other option.

In places we are using 15 lines of shell to do what would take 1 line
in C ;-). Yes, it'll probably be bigger overall, but I think you are
overstating the penalty.

regards, tom lane

Peter Eisentraut wrote:
> Bruce Momjian writes:
>
> > I know I have discouraged it because I think shell script language has a
> > good toolset for those applications. I have fixed all the spacing
> > issues.
>
> My point is that it is not, for the reasons that I listed. Handling
> spaces is a small part of one of the several problems, there are problems
> with newlines, tabs, commas, slashes, quotes -- everytime you call sed or
> read you lose one character.
>
> > What language where you thinking of using? C?
>
> Yes, that way we can share code (pg_dumpall<->pg_dump, initdb<->postgres),
> use the established internationalization facilities, and use libpq
> directly in create* and drop*.
>
> > Also, it seems Win32 doesn't need these scripts, except initdb.
>
> The utility of these programs is independent of the platform. If we think
> pg_dumpall is not useful, then let's remove it.

I think the first two targets for C-ification would be pg_dumpall and
initdb. The others have SQL equivalents. Maybe pg_ctl too.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Scott,

I just started a java admin tool project on sf called
www.sf.net/projects/jpgadmin, which should be able to handle web based
interfaces, the idea being to seperate the model and view so that we can
support a swing or web interface.

Dave

On Fri, 2002-06-07 at 18:05, Scott Shattuck wrote:
> How about a SOAP interface and a web-based front end that provides the cross
> platform support? My company's TIBET framework would provide a solid
> foundation for this kind of admin suite. In fact, we're already in the
> planning stages on doing just that.
>
> ss
>
> Scott Shattuck
> Technical Pursuit Inc.
>
>
> ----- Original Message -----
> From: "Peter Eisentraut" <peter_e(at)gmx(dot)net>
> To: "Bruce Momjian" <pgman(at)candle(dot)pha(dot)pa(dot)us>
> Cc: "PostgreSQL-development" <pgsql-hackers(at)postgreSQL(dot)org>
> Sent: Friday, June 07, 2002 11:42 AM
> Subject: Re: [HACKERS] Roadmap for a Win32 port
>
>
> > Bruce Momjian writes:
> >
> > > GUI
> > > ---
> > > pgAdmin2
> http://pgadmin.postgresql.org/pgadmin2.php?ContentID=1
> > > pgaccess http://pgaccess.org/
> > > Java admin (to be written)
> > > Dev-C++ admin (to be written)
> http://sourceforge.net/projects/dev-cpp/
> >
> > Surely Unix folks would like a GUI as well?
> >
> > --
> > Peter Eisentraut peter_e(at)gmx(dot)net
> >
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 1: subscribe and unsubscribe commands go to majordomo(at)postgresql(dot)org
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo(at)postgresql(dot)org
>
>

Bruce Momjian writes:

> I think the first two targets for C-ification would be pg_dumpall and
> initdb. The others have SQL equivalents. Maybe pg_ctl too.

I think eventually pg_ctl should be folded into the postmaster executable.
This would remove a great amount of possible misunderstandings between the
two programs.

--
Peter Eisentraut peter_e(at)gmx(dot)net

Peter Eisentraut wrote:
> Bruce Momjian writes:
>
> > I think the first two targets for C-ification would be pg_dumpall and
> > initdb. The others have SQL equivalents. Maybe pg_ctl too.
>
> I think eventually pg_ctl should be folded into the postmaster executable.
> This would remove a great amount of possible misunderstandings between the
> two programs.

And pg_ctl will be run with a symlink to postmaster like postgres,
right? Makes sense.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Jan Wieck wrote:
> > And pg_ctl will be run with a symlink to postmaster like postgres,
> > right? Makes sense.
>
> No symlink. Windows doesn't have symlinks, the "link" stuff you
> see is just some file with a special meaning for the Windows
> explorer. There is absolutely no support built into the OS. They
> really haven't learned alot since the DOS times, when they added
> "." and ".." entries to directories to "look" similar to UNIX.
> Actually, they never really understood what a hardlink is in the
> first place, so why do you expect them to know how to implement
> symbolic ones?
>
> It will be at least another copy of the postmaster (dot exe).

Yea, I just liked the idea of the postmaster binary somehow reporting
the postmaster status. Seems it is in a better position to do that than
a shell script.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Bruce Momjian wrote:
>
> Peter Eisentraut wrote:
> > Bruce Momjian writes:
> >
> > > I think the first two targets for C-ification would be pg_dumpall and
> > > initdb. The others have SQL equivalents. Maybe pg_ctl too.
> >
> > I think eventually pg_ctl should be folded into the postmaster executable.
> > This would remove a great amount of possible misunderstandings between the
> > two programs.
>
> And pg_ctl will be run with a symlink to postmaster like postgres,
> right? Makes sense.

No symlink. Windows doesn't have symlinks, the "link" stuff you
see is just some file with a special meaning for the Windows
explorer. There is absolutely no support built into the OS. They
really haven't learned alot since the DOS times, when they added
"." and ".." entries to directories to "look" similar to UNIX.
Actually, they never really understood what a hardlink is in the
first place, so why do you expect them to know how to implement
symbolic ones?

It will be at least another copy of the postmaster (dot exe).

Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being
right. #
# Let's break this rule - forgive
me. #
#==================================================
JanWieck(at)Yahoo(dot)com #

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> I think eventually pg_ctl should be folded into the postmaster executable.
> This would remove a great amount of possible misunderstandings between the
> two programs.

Like what?

The thing pg_ctl needs to know is where PGDATA is, and that
unfortunately isn't going to be known any better just by sharing
executables.

I don't object to C-ifying pg_ctl, but I don't see that it will
automatically improve pg_ctl's robustness materially.

regards, tom lane

Tom Lane writes:

> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> > I think eventually pg_ctl should be folded into the postmaster executable.
> > This would remove a great amount of possible misunderstandings between the
> > two programs.
>
> Like what?

The biggie is that pg_ctl reports the postmaster to have started
successfully without ever checking. And the "wait" option is broken and
not trivial to fix.

Other problems are the matching of the port numbers and the requirement
that admins should be able to enter a password when the server starts (for
SSL).

The luring prerequisite here is that the postmaster would have to be able
to log directly to a file, which now that all communication is guaranteed
to go through elog() should be less complicated, at least compared to
fixing the "wait" option. In fact I'm hoping that the Windows porters
will run into this same requirement just about pretty soon.

--
Peter Eisentraut peter_e(at)gmx(dot)net

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
>> Like what?

> The biggie is that pg_ctl reports the postmaster to have started
> successfully without ever checking. And the "wait" option is broken and
> not trivial to fix.

Indeed, but how will it help to merge the two executables into one?
I don't think you can simply postpone the fork() until all setup
is complete --- that would mean you don't know the final postmaster
PID until much too late.

regards, tom lane