wiki.postgresql.org is awfully slow this evening

Is anyone else seeing verrrry slllooowww response from the PG wiki?

regards, tom lane

Tom Lane wrote:
> Is anyone else seeing verrrry slllooowww response from the PG wiki?

Yep. I've given up on assigning reviewers for the evening. Keeps
timing out.

--Josh

Josh Berkus wrote:
> Tom Lane wrote:
>> Is anyone else seeing verrrry slllooowww response from the PG wiki?
>
> Yep. I've given up on assigning reviewers for the evening. Keeps
> timing out.

I have received Nagios timeouts today on the French boxen. I believe the
wiki is hosted over there. I am sure they will report on the status
shortly; they are pretty good about that.

Sincerely,

Joshua D. Drake

>
> --Josh
>

Joshua D. Drake wrote:
> Josh Berkus wrote:
>> Tom Lane wrote:
>>> Is anyone else seeing verrrry slllooowww response from the PG wiki?
>>
>> Yep. I've given up on assigning reviewers for the evening. Keeps
>> timing out.
>
> I have received Nagios timeouts today on the French boxen. I believe the
> wiki is hosted over there. I am sure they will report on the status
> shortly; they are pretty good about that.

while I cannot confirm an actual network related issue (latency tracking
from remus which is in austria shows no problem for the last 24h) we had
two runaway cvsps processes in the git jail(which is on the same
physical host as the wiki) that more or less hogged all the CPU on the box.

Stefan

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

> from remus which is in austria shows no problem for the last 24h) we had
> two runaway cvsps processes in the git jail(which is on the same
> physical host as the wiki) that more or less hogged all the CPU on the box.

Oh for Pete's sake, can we please get away from jails and just use
dedicated servers? We've had enough people volunteer hardware and
time to make this happen.

- --
Greg Sabino Mullane greg(at)turnstep(dot)com
PGP Key: 0x14964AC8 200809041007
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
-----BEGIN PGP SIGNATURE-----

iEYEAREDAAYFAki/66gACgkQvJuQZxSWSsgFbACg38bPRwFMKFoixPg80QVtiVpO
sgsAnRu2S3lKJ3udfQTKl0gFujVKA/sD
=UT0M
-----END PGP SIGNATURE-----

On Thu, Sep 4, 2008 at 3:08 PM, Greg Sabino Mullane <greg(at)turnstep(dot)com> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: RIPEMD160
>
>
>> from remus which is in austria shows no problem for the last 24h) we had
>> two runaway cvsps processes in the git jail(which is on the same
>> physical host as the wiki) that more or less hogged all the CPU on the box.
>
> Oh for Pete's sake, can we please get away from jails and just use
> dedicated servers? We've had enough people volunteer hardware and
> time to make this happen.

No we haven't - not even remotely. At a rough count we'd need at least
another 17 servers (based on the current number of VMs), and we'd lose
the ability to move services between hardware quickly and easily. Oh,
and we'd have the management headache of dealing with a bunch more
hosting providers, as I doubt the current ones will give us that many
boxes.

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

Dave Page wrote:
> On Thu, Sep 4, 2008 at 3:08 PM, Greg Sabino Mullane <greg(at)turnstep(dot)com> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: RIPEMD160
>>
>>
>>> from remus which is in austria shows no problem for the last 24h) we had
>>> two runaway cvsps processes in the git jail(which is on the same
>>> physical host as the wiki) that more or less hogged all the CPU on the box.
>> Oh for Pete's sake, can we please get away from jails and just use
>> dedicated servers? We've had enough people volunteer hardware and
>> time to make this happen.
>
> No we haven't - not even remotely. At a rough count we'd need at least
> another 17 servers (based on the current number of VMs), and we'd lose
> the ability to move services between hardware quickly and easily. Oh,
> and we'd have the management headache of dealing with a bunch more
> hosting providers, as I doubt the current ones will give us that many
> boxes.

*cough*

Yes I think they would.

That being said :) I think its a mistake. It would be a complete waste
of resources to go to dedicated machines. Some of the machines we have
we hardly use at this point.

Sincerely,

Joshua D. Drake

On Thu, Sep 4, 2008 at 3:32 PM, Joshua D. Drake <jd(at)commandprompt(dot)com> wrote:
>> No we haven't - not even remotely. At a rough count we'd need at least
>> another 17 servers (based on the current number of VMs), and we'd lose
>> the ability to move services between hardware quickly and easily. Oh,
>> and we'd have the management headache of dealing with a bunch more
>> hosting providers, as I doubt the current ones will give us that many
>> boxes.
>
> *cough*
>
> Yes I think they would.

Ya think? I'm struggling to see which of our 6 providers would pony up
more than a couple more machines.

> That being said :) I think its a mistake. It would be a complete waste of
> resources to go to dedicated machines. Some of the machines we have we
> hardly use at this point.

I think most are certainly used, but some require much fewer resources
than others.

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

Dave Page wrote:

> No we haven't - not even remotely. At a rough count we'd need at least
> another 17 servers (based on the current number of VMs), and we'd lose
> the ability to move services between hardware quickly and easily. Oh,
> and we'd have the management headache of dealing with a bunch more
> hosting providers, as I doubt the current ones will give us that many
> boxes.

Well, maybe we don't need to replace all 17, just some of the more
active ones.

Joshua points out:

> That being said :) I think its a mistake. It would be a complete waste
> of resources to go to dedicated machines. Some of the machines we have
> we hardly use at this point.

Fair enough, I withdraw the dedicated box request. Can we perhaps separate
the wiki then, so we don't have a repeat of yesterday? Maybe put wiki
or git onto one of the more lightly loaded physical boxes?

- --
Greg Sabino Mullane greg(at)turnstep(dot)com
PGP Key: 0x14964AC8 200809041059
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
-----BEGIN PGP SIGNATURE-----

iEYEAREDAAYFAki/+DoACgkQvJuQZxSWSsj22QCfZlZQ4XoxYxo+UOxXAjNeKdio
UecAoOPRoRN8EPKhgTGRINzjnAxXOJNd
=MNYh
-----END PGP SIGNATURE-----

"Greg Sabino Mullane" <greg(at)turnstep(dot)com> writes:
> Fair enough, I withdraw the dedicated box request. Can we perhaps separate
> the wiki then, so we don't have a repeat of yesterday? Maybe put wiki
> or git onto one of the more lightly loaded physical boxes?

I think the real bottom line here is that our (meaning the developers')
use of and dependence on the wiki has increased enormously over the past
few months, meaning that it's time to put it on beefier hardware. The
VM/jail setup was intended to make exactly that sort of thing easy, no?

(I'm not sure if the above statement is actually true, or if this was
just a one-time occurrence. But I'd certainly vote for moving the wiki
if it's getting at all resource-starved.)

regards, tom lane

Tom Lane wrote:
> "Greg Sabino Mullane" <greg(at)turnstep(dot)com> writes:
>> Fair enough, I withdraw the dedicated box request. Can we perhaps separate
>> the wiki then, so we don't have a repeat of yesterday? Maybe put wiki
>> or git onto one of the more lightly loaded physical boxes?
>
> I think the real bottom line here is that our (meaning the developers')
> use of and dependence on the wiki has increased enormously over the past
> few months, meaning that it's time to put it on beefier hardware. The
> VM/jail setup was intended to make exactly that sort of thing easy, no?
>
> (I'm not sure if the above statement is actually true, or if this was
> just a one-time occurrence. But I'd certainly vote for moving the wiki
> if it's getting at all resource-starved.)

this was clearly a one-time incident(and I'm fairly sure there was a
network issue too) - we only have two jails on alderaan at all and it is
more than fast enough to handle the wiki and git.

http://www.kaltenbrunner.cc/files/alderaan.postgresql.org-cpu-day.png

shows pretty nicely that after I killed the two rough processes load
diminished - what is more interesting here actually why our current
monitoring failed to detect that situation which is what I'm current
investigating ...

Stefan

Stefan, all,

> while I cannot confirm an actual network related issue (latency tracking
> from remus which is in austria shows no problem for the last 24h) we had
> two runaway cvsps processes in the git jail(which is on the same
> physical host as the wiki) that more or less hogged all the CPU on the
> box.

Hmmmph. Doesn't FreeBSD have some way to cap CPU/RAM usage per jail yet?
I thought we got that with 7. No?

--
--Josh

Josh Berkus
PostgreSQL
San Francisco