postgresql in FreeBSD jails: proposal

Lists: pgsql-adminpgsql-bugspgsql-committerspgsql-generalpgsql-hackerspgsql-jdbcpgsql-odbcpgsql-patches
From: Mischa Sandberg <mischa_sandberg(at)telus(dot)net>
To: pgsql-hackers(at)postgresql(dot)org
Subject: postgresql in FreeBSD jails: proposal
Date: 2008-01-16 16:41:42
Message-ID: 1200501702.478e33c61b74e@webmail.telus.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-admin pgsql-bugs pgsql-committers pgsql-general pgsql-hackers pgsql-jdbc pgsql-odbc pgsql-patches

Here (@sophos.com) we run machine cluster tests using FreeBSD jails. A
jail is halfway between a chroot and a VM. Jails blow a number of
assumptions about a unix environment: sysv ipc's are global to all
jails; but a process can only "see" other processes also running in the
jail. In fact, the quickest way to tell whether you're running in a jail
is to test for process 1.

PGSharedMemoryCreate chooses/reuses an ipc key in a reasonable way to
cover previous postmasters crashing and leaving a shm seg behind,
possibly with some backends still running.

Unfortunately, with multiple jails running PG servers and (due to app
limitations) all servers having same PGPORT, you get the situation that
when jail#2 (,jail#3,...) server comes up, it:
- detects that there is a shm seg with ipc key 5432001
- checks whether the associated postmaster process exists (with kill -0)
- overwrites the segment created and being used by jail #1

There's a workaround (there always is) other than this patch, involving
NAT translation so that the postmasters listen on different ports, but
the outside world sees them each listening on 5432. But that seems
somewhat circuitous.

I've hacked sysv_shmem.c (in PG 8.0.9) to handle this problem. Given the
trouble that postmaster goes to, to stop shm seg leakage, I'd like to
solicit any opinions on the wisdom of this edge case. If this patch IS
useful, what would be the right level of compile-time restriction
("#ifdef __FreeBSD__" ???)

@@ -319,7 +319,8 @@

if (makePrivate) /* a standalone backend
shouldn't do this */
continue;
-
+ /* In a FreeBSD jail, you can't "kill -0" a postmaster
+ * running in a different jail, so the shm seg might
+ * still be in use. Safer to test nattch ?
+ */
+ if (kill(1,0) && errno == ESRCH &&
!PGSharedMemoryIsInUse(0,NextShmSegID))
+ continue;
if ((memAddress = PGSharedMemoryAttach(NextShmemSegID,
&shmid)) == NULL)
continue; /* can't attach,
not one of mine */

End of Patch.


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Mischa Sandberg <mischa_sandberg(at)telus(dot)net>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: postgresql in FreeBSD jails: proposal
Date: 2008-01-16 17:09:55
Message-ID: 25135.1200503395@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-admin pgsql-bugs pgsql-committers pgsql-general pgsql-hackers pgsql-jdbc pgsql-odbc pgsql-patches

Mischa Sandberg <mischa_sandberg(at)telus(dot)net> writes:
> + /* In a FreeBSD jail, you can't "kill -0" a postmaster
> + * running in a different jail, so the shm seg might
> + * still be in use. Safer to test nattch ?
> + */
> + if (kill(1,0) && errno == ESRCH && !PGSharedMemoryIsInUse(0,NextShmSegID))
> + continue;

Isn't the last part of that test backward? If it isn't, I don't
understand what it's for at all.

regards, tom lane


From: Mischa Sandberg <mischa_sandberg(at)telus(dot)net>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: postgresql in FreeBSD jails: proposal
Date: 2008-01-16 17:13:23
Message-ID: 1200503603.478e3b33346ca@webmail.telus.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-admin pgsql-bugs pgsql-committers pgsql-general pgsql-hackers pgsql-jdbc pgsql-odbc pgsql-patches

Quoting Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>:

> Mischa Sandberg <mischa_sandberg(at)telus(dot)net> writes:
> > + /* In a FreeBSD jail, you can't "kill -0" a
> postmaster
> > + * running in a different jail, so the shm seg
> might
> > + * still be in use. Safer to test nattch ?
> > + */
> > + if (kill(1,0) && errno == ESRCH &&
> PGSharedMemoryIsInUse(0,NextShmemSegID))
> > + continue;
>
> Isn't the last part of that test backward? If it isn't, I don't
> understand what it's for at all.

Serious blush here. Yes.


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Mischa Sandberg <mischa_sandberg(at)telus(dot)net>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: postgresql in FreeBSD jails: proposal
Date: 2008-01-16 17:33:32
Message-ID: 25481.1200504812@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-admin pgsql-bugs pgsql-committers pgsql-general pgsql-hackers pgsql-jdbc pgsql-odbc pgsql-patches

Mischa Sandberg <mischa_sandberg(at)telus(dot)net> writes:
> Quoting Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>:
>> Mischa Sandberg <mischa_sandberg(at)telus(dot)net> writes:
>>> + if (kill(1,0) && errno == ESRCH && PGSharedMemoryIsInUse(0,NextShmemSegID))
>>> + continue;
>>
>> Isn't the last part of that test backward? If it isn't, I don't
>> understand what it's for at all.

> Serious blush here. Yes.

Actually, after re-reading what PGSharedMemoryIsInUse does, I don't
think you want to use it: it goes to considerable lengths to avoid
returning a false positive, whereas in this context I believe we
*do* need to avoid segments that belong to other data directories.
So you probably need a separate chunk of code that only does the
nattch test.

regards, tom lane


From: Stephen Frost <sfrost(at)snowman(dot)net>
To: Mischa Sandberg <mischa_sandberg(at)telus(dot)net>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: postgresql in FreeBSD jails: proposal
Date: 2008-01-16 17:50:41
Message-ID: 20080116175041.GQ5031@tamriel.snowman.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-admin pgsql-bugs pgsql-committers pgsql-general pgsql-hackers pgsql-jdbc pgsql-odbc pgsql-patches

* Mischa Sandberg (mischa_sandberg(at)telus(dot)net) wrote:
> Here (@sophos.com) we run machine cluster tests using FreeBSD jails. A
> jail is halfway between a chroot and a VM. Jails blow a number of
> assumptions about a unix environment: sysv ipc's are global to all
> jails; but a process can only "see" other processes also running in the
> jail. In fact, the quickest way to tell whether you're running in a jail
> is to test for process 1.

I've got a couple of concerns about this-

#1: Having the shared memory be global is a rather large problem when it
comes to something like PG which can have a fair bit of data going
through that area that could be sensitive.
#2: Isn't there already a uid check that's done? Wouldn't this make
more sense anyway (and hopefully minimize the impact of a bad person
getting control of the PG database/user in a given jail)?
#3: At least in the linux-equivilant to jails (linux-vservers, imv
anyway), they started w/o an init process and eventually decided it
made sense to have one, so I'm not sure that this test will always
work and the result might catch someone by suprise at some later
date. Is there a better/more explicit test?

Thanks,

Stephen


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Stephen Frost <sfrost(at)snowman(dot)net>
Cc: Mischa Sandberg <mischa_sandberg(at)telus(dot)net>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: postgresql in FreeBSD jails: proposal
Date: 2008-01-16 20:51:16
Message-ID: 27984.1200516676@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-admin pgsql-bugs pgsql-committers pgsql-general pgsql-hackers pgsql-jdbc pgsql-odbc pgsql-patches

Stephen Frost <sfrost(at)snowman(dot)net> writes:
> I've got a couple of concerns about this-

> #1: Having the shared memory be global is a rather large problem when it
> comes to something like PG which can have a fair bit of data going
> through that area that could be sensitive.

Well, you'd have to talk to the FreeBSD kernel hackers about changing
that, but I imagine it's still true that userid permissions checking
applies. Whether to run the postmasters that are in different jails
under different userids is a separate questions.

> #3: At least in the linux-equivilant to jails (linux-vservers, imv
> anyway), they started w/o an init process and eventually decided it
> made sense to have one, so I'm not sure that this test will always
> work and the result might catch someone by suprise at some later
> date. Is there a better/more explicit test?

We could just leave out the kill(1,0) part. In fact I wonder whether
we shouldn't do something like this on all platforms not only FreeBSD.
Quite aside from any considerations of jails, it seems like a pretty
bad idea to try to zap a shmem segment that has any attached processes.

Consider a system that normally runs multiple postmasters, in which one
postmaster has died but left orphaned backends behind, and we are trying
to start an unrelated postmaster. The current code seems capable of
deciding to zap the segment with those orphaned backends attached.
This'd require a shmem key collision which seems pretty improbable given
our key assignments, but not quite impossible. If it did happen then
the net effect would be to clear the segment's ID (since it can't
actually go away till the connected processes do). The bad thing about
that is that if the dead postmaster were then restarted, it wouldn't
recognize the segment as being its own, and would happily start up
despite the orphaned backends. Result: exactly the kind of conflicts
and data corruption that all these interlocks are trying to prevent.

So unless I'm missing something here, adding a check for nattch = 0
is a good idea, quite aside from making FreeBSD jails safer.

I think the worrisome question that follows on from Stephen's is really
whether FreeBSD will ever decide to lie about nattch (ie, exclude
processes in other jails from that count).

regards, tom lane


From: "Marc G(dot) Fournier" <scrappy(at)hub(dot)org>
To: pgadmin-devteam(at)postgresql(dot)org(dot)pgadmin-hackers(at)postgresql(dot)org(dot)pgadmin-support(at)postgresql(dot)org(dot)pgsql-admin(at)postgresql(dot)org(dot)pgsql-advocacy(at)postgresql(dot)org(dot)pgsql-announce(at)postgresql(dot)org(dot)pgsql-benchmarks(at)postgresql(dot)org(dot)pgsql-bugs(at)postgresql(dot)org(dot)pgsql-chat(at)postg
Subject: Re: postgresql in FreeBSD jails: proposal
Date: 2008-01-17 05:52:50
Message-ID: fmmqfh$buq$3@news.hub.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-admin pgsql-bugs pgsql-committers pgsql-general pgsql-hackers pgsql-jdbc pgsql-odbc pgsql-patches

mischa_sandberg(at)telus(dot)net (Mischa Sandberg) writes:

>Unfortunately, with multiple jails running PG servers and (due to app
>limitations) all servers having same PGPORT, you get the situation that
>when jail#2 (,jail#3,...) server comes up, it:
>- detects that there is a shm seg with ipc key 5432001
>- checks whether the associated postmaster process exists (with kill -0)
>- overwrites the segment created and being used by jail #1

Easiest fix: change the UID of the user running the postmaster (ie. pgsql) so
that each runs as a distinct UID (instead of distinct PGPORT) ... been doing
this since moving to FreeBSD 6.x ... no patches required ...
--
----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: "Marc G(dot) Fournier" <scrappy(at)hub(dot)org>
Cc: pgsql-hackers(at)postgreSQL(dot)org
Subject: Re: [ADMIN] postgresql in FreeBSD jails: proposal
Date: 2008-01-17 06:12:54
Message-ID: 4089.1200550374@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-admin pgsql-bugs pgsql-committers pgsql-general pgsql-hackers pgsql-jdbc pgsql-odbc pgsql-patches

"Marc G. Fournier" <scrappy(at)hub(dot)org> writes:
> mischa_sandberg(at)telus(dot)net (Mischa Sandberg) writes:
>> Unfortunately, with multiple jails running PG servers and (due to app
>> limitations) all servers having same PGPORT, you get the situation that
>> when jail#2 (,jail#3,...) server comes up, it:
>> - detects that there is a shm seg with ipc key 5432001
>> - checks whether the associated postmaster process exists (with kill -0)
>> - overwrites the segment created and being used by jail #1

> Easiest fix: change the UID of the user running the postmaster (ie. pgsql) so
> that each runs as a distinct UID (instead of distinct PGPORT) ... been doing
> this since moving to FreeBSD 6.x ... no patches required ...

Sure, but in the spirit of "belt and suspenders too", I'd think that
doing that *and* something like Mischa's proposal wouldn't be bad.

(BTW, as far as I saw the original post only went to -hackers
... there's something messed up about your reply.)

regards, tom lane


From: "Marc G(dot) Fournier" <scrappy(at)hub(dot)org>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: pgsql-hackers(at)postgreSQL(dot)org
Subject: Re: [ADMIN] postgresql in FreeBSD jails: proposal
Date: 2008-01-17 06:17:53
Message-ID: B72666942EA7D6CFBC0B7C37@ganymede.hub.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-admin pgsql-bugs pgsql-committers pgsql-general pgsql-hackers pgsql-jdbc pgsql-odbc pgsql-patches

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

- --On Thursday, January 17, 2008 01:12:54 -0500 Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
wrote:

> "Marc G. Fournier" <scrappy(at)hub(dot)org> writes:
>> mischa_sandberg(at)telus(dot)net (Mischa Sandberg) writes:
>>> Unfortunately, with multiple jails running PG servers and (due to app
>>> limitations) all servers having same PGPORT, you get the situation that
>>> when jail#2 (,jail#3,...) server comes up, it:
>>> - detects that there is a shm seg with ipc key 5432001
>>> - checks whether the associated postmaster process exists (with kill -0)
>>> - overwrites the segment created and being used by jail #1
>
>> Easiest fix: change the UID of the user running the postmaster (ie. pgsql) so
>> that each runs as a distinct UID (instead of distinct PGPORT) ... been doing
>> this since moving to FreeBSD 6.x ... no patches required ...
>
> Sure, but in the spirit of "belt and suspenders too", I'd think that
> doing that *and* something like Mischa's proposal wouldn't be bad.

No arguments here, just pointing out that changing PGPORT isn't/wasnt' the only
way of addressing this problem ... if we can do something more 'internal', it
would definitely make life alot easier ...

- ----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4 (FreeBSD)

iD8DBQFHjvMR4QvfyHIvDvMRAjJuAKCAYGkyvDOMkA6wqeQ7nAqoA1mkRQCdG+5n
aD1uG+zUtevdJGJ3BsqeDAs=
=Y0DY
-----END PGP SIGNATURE-----


From: Achilleas Mantzios <achill(at)matrix(dot)gatewaynet(dot)com>
To: pgsql-jdbc(at)postgresql(dot)org
Cc: "Marc G(dot) Fournier" <scrappy(at)hub(dot)org>, "pgadmin-devteam(at)postgresql(dot)org(dot)pgadmin-hackers(at)postgresql(dot)org(dot)pgadmin-support(at)postgresql(dot)org(dot)pgsql-admin(at)postgresql(dot)org(dot)pgsql-advocacy(at)postgresql(dot)org(dot)pgsql-announce(at)postgresql(dot)org(dot)pgsql-benchmarks(at)postgresql(dot)org(dot)pgsql-bugs(at)postgresql(dot)org(dot)pgsql-chat"(at)pos
Subject: Re: postgresql in FreeBSD jails: proposal
Date: 2008-01-17 08:11:30
Message-ID: 200801171011.30904.achill@matrix.gatewaynet.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-admin pgsql-bugs pgsql-committers pgsql-general pgsql-hackers pgsql-jdbc pgsql-odbc pgsql-patches

Στις Thursday 17 January 2008 07:52:50 ο/η Marc G. Fournier έγραψε:
> mischa_sandberg(at)telus(dot)net (Mischa Sandberg) writes:
>
> >Unfortunately, with multiple jails running PG servers and (due to app
> >limitations) all servers having same PGPORT, you get the situation that
> >when jail#2 (,jail#3,...) server comes up, it:
> >- detects that there is a shm seg with ipc key 5432001
> >- checks whether the associated postmaster process exists (with kill -0)
> >- overwrites the segment created and being used by jail #1
>
> Easiest fix: change the UID of the user running the postmaster (ie. pgsql) so
> that each runs as a distinct UID (instead of distinct PGPORT) ... been doing
> this since moving to FreeBSD 6.x ... no patches required ...

Hi Marc,
what happened with the patches at http://docs.freevps.com/doku.php?id=freebsd:changelog?
Those patches are supposed to make IPC run without collisions in many jails, thus solving the
classic problem with postgresql, allowing the admin to use the same uid, PGPORT in all pgsql
server instances.
Do you know if any of this stuff make it anywhere in official FreeBSD?

--
Achilleas Mantzios


From: Achilleas Mantzios <achill(at)matrix(dot)gatewaynet(dot)com>
To: pgsql-admin(at)postgresql(dot)org
Subject: Re: postgresql in FreeBSD jails: proposal
Date: 2008-01-17 08:21:33
Message-ID: 200801171021.33707.achill@matrix.gatewaynet.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-admin pgsql-bugs pgsql-committers pgsql-general pgsql-hackers pgsql-jdbc pgsql-odbc pgsql-patches

Στις Thursday 17 January 2008 07:52:50 ο/η Marc G. Fournier έγραψε:
> mischa_sandberg(at)telus(dot)net (Mischa Sandberg) writes:
>
> >Unfortunately, with multiple jails running PG servers and (due to app
> >limitations) all servers having same PGPORT, you get the situation that
> >when jail#2 (,jail#3,...) server comes up, it:
> >- detects that there is a shm seg with ipc key 5432001
> >- checks whether the associated postmaster process exists (with kill -0)
> >- overwrites the segment created and being used by jail #1
>
> Easiest fix: change the UID of the user running the postmaster (ie. pgsql) so
> that each runs as a distinct UID (instead of distinct PGPORT) ... been doing
> this since moving to FreeBSD 6.x ... no patches required ...

Hi Marc,
what happened with the patches at http://docs.freevps.com/doku.php?id=freebsd:changelog?
Those patches are supposed to make IPC run without collisions in many jails, thus solving the
classic problem with postgresql, allowing the admin to use the same uid, PGPORT in all pgsql
server instances.
Do you know if any of this stuff make it anywhere in official FreeBSD?

--
Achilleas Mantzios


From: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
To: "Marc G(dot) Fournier" <scrappy(at)hub(dot)org>
Cc: Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: [ADMIN] postgresql in FreeBSD jails: proposal
Date: 2008-01-17 13:48:14
Message-ID: 20080117134814.GD17828@alvh.no-ip.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-admin pgsql-bugs pgsql-committers pgsql-general pgsql-hackers pgsql-jdbc pgsql-odbc pgsql-patches


Cc: "pgadmin-devteam(at)postgresql(dot)org(dot)pgadmin-hackers@postgresql(dot)org(dot)pgadmin-support(at)postgresql(dot)org(dot)pgsql-admin@postgresql(dot)org(dot)pgsql-advocacy(at)postgresql(dot)org(dot)pgsql-announce@postgresql(dot)org(dot)pgsql-benchmarks(at)postgresql(dot)org(dot)pgsql-bugs@postgresql.org.pgsql-chat"@post

Hey, this is exactly the sort of weird "Cc:" line I saw in the recent
spam surge. Since I suspect you are using the news server to post, I
suggest you take a long and careful look at the gateway's configuration.
It seems there's something very broken here.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support


From: "Dave Page" <dpage(at)postgresql(dot)org>
To: "Alvaro Herrera" <alvherre(at)commandprompt(dot)com>
Cc: "Marc G(dot) Fournier" <scrappy(at)hub(dot)org>, Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: [ADMIN] postgresql in FreeBSD jails: proposal
Date: 2008-01-17 13:58:36
Message-ID: 937d27e10801170558hd6753a2i5221c3a8438f1aa1@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-admin pgsql-bugs pgsql-committers pgsql-general pgsql-hackers pgsql-jdbc pgsql-odbc pgsql-patches

On 17/01/2008, Alvaro Herrera <alvherre(at)commandprompt(dot)com> wrote:
>
> Cc: "pgadmin-devteam(at)postgresql(dot)org(dot)pgadmin-hackers@postgresql(dot)org(dot)pgadmin-support(at)postgresql(dot)org(dot)pgsql-admin@postgresql(dot)org(dot)pgsql-advocacy(at)postgresql(dot)org(dot)pgsql-announce@postgresql(dot)org(dot)pgsql-benchmarks(at)postgresql(dot)org(dot)pgsql-bugs@postgresql.org.pgsql-chat"@post
>
> Hey, this is exactly the sort of weird "Cc:" line I saw in the recent
> spam surge. Since I suspect you are using the news server to post, I
> suggest you take a long and careful look at the gateway's configuration.
> It seems there's something very broken here.

I sure hope that pgadmin-devteam isn't going out on the newserver -
thats the pgAdmin equivalent to pgsql-core :-O

/D


From: Stephen Frost <sfrost(at)snowman(dot)net>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: "Marc G(dot) Fournier" <scrappy(at)hub(dot)org>, pgsql-hackers(at)postgreSQL(dot)org
Subject: Re: [ADMIN] postgresql in FreeBSD jails: proposal
Date: 2008-01-17 14:37:29
Message-ID: 20080117143729.GR5031@tamriel.snowman.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-admin pgsql-bugs pgsql-committers pgsql-general pgsql-hackers pgsql-jdbc pgsql-odbc pgsql-patches

* Tom Lane (tgl(at)sss(dot)pgh(dot)pa(dot)us) wrote:
> "Marc G. Fournier" <scrappy(at)hub(dot)org> writes:
> > Easiest fix: change the UID of the user running the postmaster (ie. pgsql) so
> > that each runs as a distinct UID (instead of distinct PGPORT) ... been doing
> > this since moving to FreeBSD 6.x ... no patches required ...
>
> Sure, but in the spirit of "belt and suspenders too", I'd think that
> doing that *and* something like Mischa's proposal wouldn't be bad.

I agree that we should try to be careful about stepping on segments that
might still be in use, but I would also discourage jail users from using
the same uid for multiple PG clusters since the jail doesn't protect the
shmem segment. We use seperate uids even w/ linux-vservers where shmem
and everything *is* seperate, following the same 'belt and suspenders
too' spirit for security.

Thanks,

Stephen


From: "Marc G(dot) Fournier" <scrappy(at)hub(dot)org>
To: Dave Page <dpage(at)postgresql(dot)org>, Alvaro Herrera <alvherre(at)commandprompt(dot)com>
Cc: Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: [ADMIN] postgresql in FreeBSD jails: proposal
Date: 2008-01-17 22:19:14
Message-ID: E2F6188AEDD0AB8BAA5FD820@ganymede.hub.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-admin pgsql-bugs pgsql-committers pgsql-general pgsql-hackers pgsql-jdbc pgsql-odbc pgsql-patches

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

- --On Thursday, January 17, 2008 13:58:36 +0000 Dave Page
<dpage(at)postgresql(dot)org>
wrote:

> On 17/01/2008, Alvaro Herrera <alvherre(at)commandprompt(dot)com> wrote:
>>
>> Cc:
>> "pgadmin-devteam(at)postgresql(dot)org(dot)pgadmin-hackers@postgresql.org.pgadmin-suppo
>> rt(at)postgresql(dot)org(dot)pgsql-admin@postgresql(dot)org(dot)pgsql-advocacy(at)postgresql(dot)org(dot)p
>> gsql-announce(at)postgresql(dot)org(dot)pgsql-benchmarks@postgresql(dot)org(dot)pgsql-bugs(at)post
>> gresql.org.pgsql-chat"@post
>>
>> Hey, this is exactly the sort of weird "Cc:" line I saw in the recent
>> spam surge. Since I suspect you are using the news server to post, I
>> suggest you take a long and careful look at the gateway's configuration.
>> It seems there's something very broken here.
>
> I sure hope that pgadmin-devteam isn't going out on the newserver -
> thats the pgAdmin equivalent to pgsql-core :-O

Just checked the subscriber list for that list, and I don't see news listed on
it ...and no such newsgroup either:

%grep pgadmin db/active
pgsql.interfaces.pgadmin.hackers 0000004826 0000000001 y
pgsql.interfaces.pgadmin.support 0000002946 0000000001 y

- ----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4 (FreeBSD)

iD8DBQFHj9Ri4QvfyHIvDvMRAq/SAJ433rjmQjHG5OiR1PJ3BOq/93kPBwCg4an3
QaqGiypV6Jp0Bovi/O7EADs=
=KKZB
-----END PGP SIGNATURE-----


From: Mischa Sandberg <mischa_sandberg(at)telus(dot)net>
To: Stephen Frost <sfrost(at)snowman(dot)net>
Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, "Marc G(dot) Fournier" <scrappy(at)hub(dot)org>, pgsql-hackers(at)postgreSQL(dot)org
Subject: Re: [ADMIN] postgresql in FreeBSD jails: proposal
Date: 2008-01-17 22:52:03
Message-ID: 1200610323.478fdc131bb3a@webmail.telus.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-admin pgsql-bugs pgsql-committers pgsql-general pgsql-hackers pgsql-jdbc pgsql-odbc pgsql-patches

Quoting Stephen Frost <sfrost(at)snowman(dot)net>:

> * Tom Lane (tgl(at)sss(dot)pgh(dot)pa(dot)us) wrote:
> > "Marc G. Fournier" <scrappy(at)hub(dot)org> writes:
> > > Easiest fix: change the UID of the user running the postmaster
> (ie. pgsql) so
> > > that each runs as a distinct UID (instead of distinct PGPORT) ...
> been doing
> > > this since moving to FreeBSD 6.x ... no patches required ...
> >
> > Sure, but in the spirit of "belt and suspenders too", I'd think
> that
> > doing that *and* something like Mischa's proposal wouldn't be bad.
>
> I agree that we should try to be careful about stepping on
> segments that might still be in use, but I would also discourage
> jail users from using the same uid for multiple PG clusters
> since the jail doesn't protect the shmem segment.
> We use seperate uids even w/ linux-vservers where shmem
> and everything *is* seperate, following the same
> 'belt and suspenders too' spirit for security.

Thanks for all the input. Fixing freebsd might get answered
on a different channel :-)

Unfortunately, different uid's is not even an option here;
but serious security in this sitch is not relevant, either.

We have a freebsd core guy here, and he says that there's no
pressing incentive for jails to handle sysv ipc, given mmap
and file locking :-( And given his other comments, I wouldn't
consider jails a "secure" environment, just a modest and
convenient way to emulate multiple machines with caveats.
.........................................................
So, given Tom's comment, that it's antisocial to zap a shm seg
that other processes have attached ...

I'm going to skip the kill(1,0) test and depend on nattch only,
with a function that PGSharedMemoryIsInUse() can also use.
(For a healthy server, nattch is never less than 2, right?)
If no unpleasant edge cases come out of this in our test
framework, I'd like to submit that as a patch.
Talked with our Linux guys about vserver, and they see no issues.
Mr. Solaris here is currently a long way ooto ... opinions?

Afaics the change in behaviour is, if a degraded server exited
with some backend hanging, the second server will create a new
segment after bumping the ipc key; if system shm limits do not
allow for two such shm segments, the second server will bail.
For production systems, ensuring no orphan shm segs
is not left to heuristic clean-up by server re-start.

Hope that makes sense for the generic Postgres world.

If anyone is interested in creating hung backends, you can
create a named pipe, and tell the server to COPY from it.
---
Engineers think that equations approximate reality.
Physicists think that reality approximates the equations.
Mathematicians never make the connection.


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Mischa Sandberg <mischa_sandberg(at)telus(dot)net>
Cc: Stephen Frost <sfrost(at)snowman(dot)net>, "Marc G(dot) Fournier" <scrappy(at)hub(dot)org>, pgsql-hackers(at)postgreSQL(dot)org
Subject: Re: [ADMIN] postgresql in FreeBSD jails: proposal
Date: 2008-01-17 23:10:16
Message-ID: 16615.1200611416@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-admin pgsql-bugs pgsql-committers pgsql-general pgsql-hackers pgsql-jdbc pgsql-odbc pgsql-patches

Mischa Sandberg <mischa_sandberg(at)telus(dot)net> writes:
> If anyone is interested in creating hung backends, you can
> create a named pipe, and tell the server to COPY from it.

That won't create a problematic situation though, until/unless you
SIGQUIT the parent postmaster.

Personally I think of this as "what happens after someone
kill -9's a postmaster that has live children".

regards, tom lane


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Mischa Sandberg <mischa_sandberg(at)telus(dot)net>
Cc: Stephen Frost <sfrost(at)snowman(dot)net>, "Marc G(dot) Fournier" <scrappy(at)hub(dot)org>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: [ADMIN] postgresql in FreeBSD jails: proposal
Date: 2008-01-17 23:49:32
Message-ID: 17061.1200613772@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-admin pgsql-bugs pgsql-committers pgsql-general pgsql-hackers pgsql-jdbc pgsql-odbc pgsql-patches

Mischa Sandberg <mischa_sandberg(at)telus(dot)net> writes:
> I'm going to skip the kill(1,0) test and depend on nattch only,
> with a function that PGSharedMemoryIsInUse() can also use.
> (For a healthy server, nattch is never less than 2, right?)

Oh, forgot to mention: healthy servers are not the point here.
You should make the code keep its hands off any segment with
nonzero nattch, because even one orphaned backend is enough
to cause trouble.

regards, tom lane


From: Mischa Sandberg <mischa_sandberg(at)telus(dot)net>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Stephen Frost <sfrost(at)snowman(dot)net>, "Marc G(dot) Fournier" <scrappy(at)hub(dot)org>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: [ADMIN] postgresql in FreeBSD jails: proposal
Date: 2008-01-17 23:52:33
Message-ID: 1200613953.478fea419c929@webmail.telus.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-admin pgsql-bugs pgsql-committers pgsql-general pgsql-hackers pgsql-jdbc pgsql-odbc pgsql-patches

Quoting Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>:

> Mischa Sandberg <mischa_sandberg(at)telus(dot)net> writes:
> > I'm going to skip the kill(1,0) test and depend on nattch only,
> > with a function that PGSharedMemoryIsInUse() can also use.
> > (For a healthy server, nattch is never less than 2, right?)
>
> Oh, forgot to mention: healthy servers are not the point here.
> You should make the code keep its hands off any segment with
> nonzero nattch, because even one orphaned backend is enough
> to cause trouble.

Note taken. Worth putting a warning in the log, too?

Engineers think that equations approximate reality.
Physicists think that reality approximates the equations.
Mathematicians never make the connection.


From: Bruce Momjian <bruce(at)momjian(dot)us>
To: Mischa Sandberg <mischa_sandberg(at)telus(dot)net>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: postgresql in FreeBSD jails: proposal
Date: 2008-03-25 02:11:31
Message-ID: 200803250211.m2P2BVn13115@momjian.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-admin pgsql-bugs pgsql-committers pgsql-general pgsql-hackers pgsql-jdbc pgsql-odbc pgsql-patches


Added to TODO:

* Improve detection of shared memory segments being used by other
FreeBSD jails

http://archives.postgresql.org/pgsql-hackers/2008-01/msg00656.php

---------------------------------------------------------------------------

Mischa Sandberg wrote:
> Here (@sophos.com) we run machine cluster tests using FreeBSD jails. A
> jail is halfway between a chroot and a VM. Jails blow a number of
> assumptions about a unix environment: sysv ipc's are global to all
> jails; but a process can only "see" other processes also running in the
> jail. In fact, the quickest way to tell whether you're running in a jail
> is to test for process 1.
>
> PGSharedMemoryCreate chooses/reuses an ipc key in a reasonable way to
> cover previous postmasters crashing and leaving a shm seg behind,
> possibly with some backends still running.
>
> Unfortunately, with multiple jails running PG servers and (due to app
> limitations) all servers having same PGPORT, you get the situation that
> when jail#2 (,jail#3,...) server comes up, it:
> - detects that there is a shm seg with ipc key 5432001
> - checks whether the associated postmaster process exists (with kill -0)
> - overwrites the segment created and being used by jail #1
>
> There's a workaround (there always is) other than this patch, involving
> NAT translation so that the postmasters listen on different ports, but
> the outside world sees them each listening on 5432. But that seems
> somewhat circuitous.
>
> I've hacked sysv_shmem.c (in PG 8.0.9) to handle this problem. Given the
> trouble that postmaster goes to, to stop shm seg leakage, I'd like to
> solicit any opinions on the wisdom of this edge case. If this patch IS
> useful, what would be the right level of compile-time restriction
> ("#ifdef __FreeBSD__" ???)
>
> @@ -319,7 +319,8 @@
>
> if (makePrivate) /* a standalone backend
> shouldn't do this */
> continue;
> -
> + /* In a FreeBSD jail, you can't "kill -0" a postmaster
> + * running in a different jail, so the shm seg might
> + * still be in use. Safer to test nattch ?
> + */
> + if (kill(1,0) && errno == ESRCH &&
> !PGSharedMemoryIsInUse(0,NextShmSegID))
> + continue;
> if ((memAddress = PGSharedMemoryAttach(NextShmemSegID,
> &shmid)) == NULL)
> continue; /* can't attach,
> not one of mine */
>
> End of Patch.
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo(at)postgresql(dot)org so that your
> message can get through to the mailing list cleanly

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +


From: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
To: Bruce Momjian <bruce(at)momjian(dot)us>
Cc: Mischa Sandberg <mischa_sandberg(at)telus(dot)net>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: postgresql in FreeBSD jails: proposal
Date: 2008-03-25 14:47:24
Message-ID: 20080325144724.GG9815@alvh.no-ip.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-admin pgsql-bugs pgsql-committers pgsql-general pgsql-hackers pgsql-jdbc pgsql-odbc pgsql-patches

Bruce Momjian wrote:
>
> Added to TODO:
>
> * Improve detection of shared memory segments being used by other
> FreeBSD jails
>
> http://archives.postgresql.org/pgsql-hackers/2008-01/msg00656.php

There's a bit more than that to it -- see
http://archives.postgresql.org/pgsql-hackers/2008-01/msg00673.php

In short, it's not just a FreeBSD issue, but something a bit more
general.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.


From: Bruce Momjian <bruce(at)momjian(dot)us>
To: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
Cc: Mischa Sandberg <mischa_sandberg(at)telus(dot)net>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: postgresql in FreeBSD jails: proposal
Date: 2008-05-08 02:24:13
Message-ID: 200805080224.m482ODd09399@momjian.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-admin pgsql-bugs pgsql-committers pgsql-general pgsql-hackers pgsql-jdbc pgsql-odbc pgsql-patches

Alvaro Herrera wrote:
> Bruce Momjian wrote:
> >
> > Added to TODO:
> >
> > * Improve detection of shared memory segments being used by other
> > FreeBSD jails
> >
> > http://archives.postgresql.org/pgsql-hackers/2008-01/msg00656.php
>
> There's a bit more than that to it -- see
> http://archives.postgresql.org/pgsql-hackers/2008-01/msg00673.php
>
> In short, it's not just a FreeBSD issue, but something a bit more
> general.

Added to TODO:

* Improve detection of shared memory segments being used by others
by checking the SysV shared memory field 'nattch'

http://archives.postgresql.org/pgsql-hackers/2008-01/msg00656.php
http://archives.postgresql.org/pgsql-hackers/2008-01/msg00673.php

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +