Scheduled maintenance affecting gitmaster

Hi!

The PostgreSQL Infrastructure Team will be performing scheduled
maintenance on the server that is hosting gitmaster.postgresql.org the
coming sunday, Feb 13th. While we expect this to only cause a very
short downtime for the service, one can never be entirely sure with
remote maintenance.. For this reason, please avoid using the service
during this time. Note that only the git *master* server is affected,
not the git.postgresql.org anonymous mirror is on another host and
will not be affected.

We will post a note here when the maintenance has been completed.

Date: 2011-02-13
Start: 12:00 UTC
End: ~13:00 UTC, but may drag out
Affected services: gitmaster.postgresql.org and misc. internal services

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Tue, Feb 8, 2011 at 19:52, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> Hi!
>
> The PostgreSQL Infrastructure Team will be performing scheduled
> maintenance on the server that is hosting gitmaster.postgresql.org the
> coming sunday, Feb 13th. While we expect this to only cause a very
> short downtime for the service, one can never be entirely sure with
> remote maintenance.. For this reason, please avoid using the service
> during this time. Note that only the git *master* server is affected,
> not the git.postgresql.org anonymous mirror is on another host and
> will not be affected.
>
> We will post a note here when the maintenance has been completed.
>
> Date: 2011-02-13
> Start: 12:00 UTC
> End: ~13:00 UTC, but may drag out
> Affected services: gitmaster.postgresql.org and misc. internal services

Unfortunately, one of the worst-case scenarios appears to have
happened - a machine did not come back up after a reboot. We are
working with the hosting company to figure out what went wrong and get
it back up as soon as possible, but obviously this will not happen
within the window we were hoping for.

We have a backup option which is to bring the VM up on a separate
machine, but given the amount of extra work and possible further
issues with this, we are going to hold out for a while to see if we
can get the current machine back into working state, given that it's
only "internal services".

We'll get back to you with more information as soon as we have it.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> Unfortunately, one of the worst-case scenarios appears to have
> happened - a machine did not come back up after a reboot.
> ...
> We'll get back to you with more information as soon as we have it.

I didn't see any followup to this?

gitmaster seems to be responding as of now, is it safe to push?

regards, tom lane

On 02/14/2011 01:27 AM, Tom Lane wrote:
> Magnus Hagander<magnus(at)hagander(dot)net> writes:
>> Unfortunately, one of the worst-case scenarios appears to have
>> happened - a machine did not come back up after a reboot.
>> ...
>> We'll get back to you with more information as soon as we have it.
>
> I didn't see any followup to this?

yeah - the hosting company managed to reboot the box for us which
brought it back to life in the middle of the night (with both magnus and
me asleep).

>
> gitmaster seems to be responding as of now, is it safe to push?

yes it is - however we will need to schedule another maintenance window
soon to finish the stuff we actually wanted to do.

Stefan

On Mon, Feb 14, 2011 at 07:13, Stefan Kaltenbrunner
<stefan(at)kaltenbrunner(dot)cc> wrote:
> On 02/14/2011 01:27 AM, Tom Lane wrote:
>>
>> Magnus Hagander<magnus(at)hagander(dot)net>  writes:
>>>
>>> Unfortunately, one of the worst-case scenarios appears to have
>>> happened - a machine did not come back up after a reboot.
>>> ...
>>> We'll get back to you with more information as soon as we have it.
>>
>> I didn't see any followup to this?
>
> yeah - the hosting company managed to reboot the box for us which brought it
> back to life in the middle of the night (with both magnus and me asleep).

Indeed. But the good news is that once it came back up, the VM with
the git server started ok :-)

>> gitmaster seems to be responding as of now, is it safe to push?
>
> yes it is - however we will need to schedule another maintenance window soon
> to finish the stuff we actually wanted to do.

So, after some discussion with Stefan, we (well, I guess I) decided we
should just go ahead and declare the maintenance window not closed
yet, and finish off the upgrade right now :-) Given that the majority
of our commits don't happen now, we'll hopefully have it done by the
time the US folks wake up again.

So, maintenance window again, starting now, and we'll let you know as
soon as we're done. And we're definitely hoping for the machine to
come back up properly this time :-)

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On 02/14/2011 10:09 AM, Magnus Hagander wrote:
> On Mon, Feb 14, 2011 at 07:13, Stefan Kaltenbrunner
> <stefan(at)kaltenbrunner(dot)cc> wrote:
>> On 02/14/2011 01:27 AM, Tom Lane wrote:
>>>
>>> Magnus Hagander<magnus(at)hagander(dot)net> writes:
>>>>
>>>> Unfortunately, one of the worst-case scenarios appears to have
>>>> happened - a machine did not come back up after a reboot.
>>>> ...
>>>> We'll get back to you with more information as soon as we have it.
>>>
>>> I didn't see any followup to this?
>>
>> yeah - the hosting company managed to reboot the box for us which brought it
>> back to life in the middle of the night (with both magnus and me asleep).
>
> Indeed. But the good news is that once it came back up, the VM with
> the git server started ok :-)
>
>
>>> gitmaster seems to be responding as of now, is it safe to push?
>>
>> yes it is - however we will need to schedule another maintenance window soon
>> to finish the stuff we actually wanted to do.
>
> So, after some discussion with Stefan, we (well, I guess I) decided we
> should just go ahead and declare the maintenance window not closed
> yet, and finish off the upgrade right now :-) Given that the majority
> of our commits don't happen now, we'll hopefully have it done by the
> time the US folks wake up again.
>
> So, maintenance window again, starting now, and we'll let you know as
> soon as we're done. And we're definitely hoping for the machine to
> come back up properly this time :-)

and it did not... We are trying to figure out what the actual problem
here really is because it seems to boot just fine when powercycled just
not with a software initiated reboot.
We will notify once we have more information...

Stefan

2011/2/14 Stefan Kaltenbrunner <Stefan(at)kaltenbrunner(dot)cc>:
> On 02/14/2011 10:09 AM, Magnus Hagander wrote:
>> On Mon, Feb 14, 2011 at 07:13, Stefan Kaltenbrunner
>> <stefan(at)kaltenbrunner(dot)cc> wrote:
>>> On 02/14/2011 01:27 AM, Tom Lane wrote:
>>>>
>>>> Magnus Hagander<magnus(at)hagander(dot)net>  writes:
>>>>>
>>>>> Unfortunately, one of the worst-case scenarios appears to have
>>>>> happened - a machine did not come back up after a reboot.
>>>>> ...
>>>>> We'll get back to you with more information as soon as we have it.
>>>>
>>>> I didn't see any followup to this?
>>>
>>> yeah - the hosting company managed to reboot the box for us which brought it
>>> back to life in the middle of the night (with both magnus and me asleep).
>>
>> Indeed. But the good news is that once it came back up, the VM with
>> the git server started ok :-)
>>
>>
>>>> gitmaster seems to be responding as of now, is it safe to push?
>>>
>>> yes it is - however we will need to schedule another maintenance window soon
>>> to finish the stuff we actually wanted to do.
>>
>> So, after some discussion with Stefan, we (well, I guess I) decided we
>> should just go ahead and declare the maintenance window not closed
>> yet, and finish off the upgrade right now :-) Given that the majority
>> of our commits don't happen now, we'll hopefully have it done by the
>> time the US folks wake up again.
>>
>> So, maintenance window again, starting now, and we'll let you know as
>> soon as we're done. And we're definitely hoping for the machine to
>> come back up properly this time :-)
>
> and it did not... We are trying to figure out what the actual problem
> here really is because it seems to boot just fine when powercycled just
> not with a software initiated reboot.
> We will notify once we have more information...
>

Does it make sense to get some console link or ipmi set up for those
crucial parts of the infrastructure ?

--
Cédric Villemain               2ndQuadrant
http://2ndQuadrant.fr/     PostgreSQL : Expertise, Formation et Support

On Mon, Feb 14, 2011 at 11:46, Cédric Villemain
<cedric(dot)villemain(dot)debian(at)gmail(dot)com> wrote:
> 2011/2/14 Stefan Kaltenbrunner <Stefan(at)kaltenbrunner(dot)cc>:
>> On 02/14/2011 10:09 AM, Magnus Hagander wrote:
>>> On Mon, Feb 14, 2011 at 07:13, Stefan Kaltenbrunner
>>> <stefan(at)kaltenbrunner(dot)cc> wrote:
>>>> On 02/14/2011 01:27 AM, Tom Lane wrote:
>>>>>
>>>>> Magnus Hagander<magnus(at)hagander(dot)net>  writes:
>>>>>>
>>>>>> Unfortunately, one of the worst-case scenarios appears to have
>>>>>> happened - a machine did not come back up after a reboot.
>>>>>> ...
>>>>>> We'll get back to you with more information as soon as we have it.
>>>>>
>>>>> I didn't see any followup to this?
>>>>
>>>> yeah - the hosting company managed to reboot the box for us which brought it
>>>> back to life in the middle of the night (with both magnus and me asleep).
>>>
>>> Indeed. But the good news is that once it came back up, the VM with
>>> the git server started ok :-)
>>>
>>>
>>>>> gitmaster seems to be responding as of now, is it safe to push?
>>>>
>>>> yes it is - however we will need to schedule another maintenance window soon
>>>> to finish the stuff we actually wanted to do.
>>>
>>> So, after some discussion with Stefan, we (well, I guess I) decided we
>>> should just go ahead and declare the maintenance window not closed
>>> yet, and finish off the upgrade right now :-) Given that the majority
>>> of our commits don't happen now, we'll hopefully have it done by the
>>> time the US folks wake up again.
>>>
>>> So, maintenance window again, starting now, and we'll let you know as
>>> soon as we're done. And we're definitely hoping for the machine to
>>> come back up properly this time :-)
>>
>> and it did not... We are trying to figure out what the actual problem
>> here really is because it seems to boot just fine when powercycled just
>> not with a software initiated reboot.
>> We will notify once we have more information...
>>
>
> Does it make sense to get some console link or ipmi set up for those
> crucial parts of the infrastructure ?

This is production servers, of course they are equipped with remove consoles.

However, these consoles are only accessible from the hosting companys
internal company network or VPN, so we cannot access them directly.

It is something we are discussing with them...

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

2011/2/14 Magnus Hagander <magnus(at)hagander(dot)net>:
> On Mon, Feb 14, 2011 at 11:46, Cédric Villemain
> <cedric(dot)villemain(dot)debian(at)gmail(dot)com> wrote:
>> 2011/2/14 Stefan Kaltenbrunner <Stefan(at)kaltenbrunner(dot)cc>:
>>> On 02/14/2011 10:09 AM, Magnus Hagander wrote:
>>>> On Mon, Feb 14, 2011 at 07:13, Stefan Kaltenbrunner
>>>> <stefan(at)kaltenbrunner(dot)cc> wrote:
>>>>> On 02/14/2011 01:27 AM, Tom Lane wrote:
>>>>>>
>>>>>> Magnus Hagander<magnus(at)hagander(dot)net>  writes:
>>>>>>>
>>>>>>> Unfortunately, one of the worst-case scenarios appears to have
>>>>>>> happened - a machine did not come back up after a reboot.
>>>>>>> ...
>>>>>>> We'll get back to you with more information as soon as we have it.
>>>>>>
>>>>>> I didn't see any followup to this?
>>>>>
>>>>> yeah - the hosting company managed to reboot the box for us which brought it
>>>>> back to life in the middle of the night (with both magnus and me asleep).
>>>>
>>>> Indeed. But the good news is that once it came back up, the VM with
>>>> the git server started ok :-)
>>>>
>>>>
>>>>>> gitmaster seems to be responding as of now, is it safe to push?
>>>>>
>>>>> yes it is - however we will need to schedule another maintenance window soon
>>>>> to finish the stuff we actually wanted to do.
>>>>
>>>> So, after some discussion with Stefan, we (well, I guess I) decided we
>>>> should just go ahead and declare the maintenance window not closed
>>>> yet, and finish off the upgrade right now :-) Given that the majority
>>>> of our commits don't happen now, we'll hopefully have it done by the
>>>> time the US folks wake up again.
>>>>
>>>> So, maintenance window again, starting now, and we'll let you know as
>>>> soon as we're done. And we're definitely hoping for the machine to
>>>> come back up properly this time :-)
>>>
>>> and it did not... We are trying to figure out what the actual problem
>>> here really is because it seems to boot just fine when powercycled just
>>> not with a software initiated reboot.
>>> We will notify once we have more information...
>>>
>>
>> Does it make sense to get some console link or ipmi set up for those
>> crucial parts of the infrastructure ?
>
> This is production servers, of course they are equipped with remove consoles.
>
> However, these consoles are only accessible from the hosting companys
> internal company network or VPN, so we cannot access them directly.
>
> It is something we are discussing with them...

ok. Not the top priority here I believe, but those kind of crisis
period usually help (to have it set up quickly, as the topic is hot )

Thank you for your time and effort spent,
--
Cédric Villemain               2ndQuadrant
http://2ndQuadrant.fr/     PostgreSQL : Expertise, Formation et Support

On Mon, Feb 14, 2011 at 10:39, Stefan Kaltenbrunner
<Stefan(at)kaltenbrunner(dot)cc> wrote:
> On 02/14/2011 10:09 AM, Magnus Hagander wrote:
>> On Mon, Feb 14, 2011 at 07:13, Stefan Kaltenbrunner
>> <stefan(at)kaltenbrunner(dot)cc> wrote:
>>> On 02/14/2011 01:27 AM, Tom Lane wrote:
>>>>
>>>> Magnus Hagander<magnus(at)hagander(dot)net>  writes:
>>>>>
>>>>> Unfortunately, one of the worst-case scenarios appears to have
>>>>> happened - a machine did not come back up after a reboot.
>>>>> ...
>>>>> We'll get back to you with more information as soon as we have it.
>>>>
>>>> I didn't see any followup to this?
>>>
>>> yeah - the hosting company managed to reboot the box for us which brought it
>>> back to life in the middle of the night (with both magnus and me asleep).
>>
>> Indeed. But the good news is that once it came back up, the VM with
>> the git server started ok :-)
>>
>>
>>>> gitmaster seems to be responding as of now, is it safe to push?
>>>
>>> yes it is - however we will need to schedule another maintenance window soon
>>> to finish the stuff we actually wanted to do.
>>
>> So, after some discussion with Stefan, we (well, I guess I) decided we
>> should just go ahead and declare the maintenance window not closed
>> yet, and finish off the upgrade right now :-) Given that the majority
>> of our commits don't happen now, we'll hopefully have it done by the
>> time the US folks wake up again.
>>
>> So, maintenance window again, starting now, and we'll let you know as
>> soon as we're done. And we're definitely hoping for the machine to
>> come back up properly this time :-)
>
> and it did not... We are trying to figure out what the actual problem
> here really is because it seems to boot just fine when powercycled just
> not with a software initiated reboot.
> We will notify once we have more information...

Status update on this - Stefan is currently working with the
datacenter people on getting this fixed (they are now available
on-site), since we are now having an actual issue with the machine
(GRUB failure on boot) rather than just a failure to shut down.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Mon, Feb 14, 2011 at 16:15, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> On Mon, Feb 14, 2011 at 10:39, Stefan Kaltenbrunner
> <Stefan(at)kaltenbrunner(dot)cc> wrote:
>> On 02/14/2011 10:09 AM, Magnus Hagander wrote:
>>> On Mon, Feb 14, 2011 at 07:13, Stefan Kaltenbrunner
>>> <stefan(at)kaltenbrunner(dot)cc> wrote:
>>>> On 02/14/2011 01:27 AM, Tom Lane wrote:
>>>>>
>>>>> Magnus Hagander<magnus(at)hagander(dot)net>  writes:
>>>>>>
>>>>>> Unfortunately, one of the worst-case scenarios appears to have
>>>>>> happened - a machine did not come back up after a reboot.
>>>>>> ...
>>>>>> We'll get back to you with more information as soon as we have it.
>>>>>
>>>>> I didn't see any followup to this?
>>>>
>>>> yeah - the hosting company managed to reboot the box for us which brought it
>>>> back to life in the middle of the night (with both magnus and me asleep).
>>>
>>> Indeed. But the good news is that once it came back up, the VM with
>>> the git server started ok :-)
>>>
>>>
>>>>> gitmaster seems to be responding as of now, is it safe to push?
>>>>
>>>> yes it is - however we will need to schedule another maintenance window soon
>>>> to finish the stuff we actually wanted to do.
>>>
>>> So, after some discussion with Stefan, we (well, I guess I) decided we
>>> should just go ahead and declare the maintenance window not closed
>>> yet, and finish off the upgrade right now :-) Given that the majority
>>> of our commits don't happen now, we'll hopefully have it done by the
>>> time the US folks wake up again.
>>>
>>> So, maintenance window again, starting now, and we'll let you know as
>>> soon as we're done. And we're definitely hoping for the machine to
>>> come back up properly this time :-)
>>
>> and it did not... We are trying to figure out what the actual problem
>> here really is because it seems to boot just fine when powercycled just
>> not with a software initiated reboot.
>> We will notify once we have more information...
>
> Status update on this - Stefan is currently working with the
> datacenter people on getting this fixed (they are now available
> on-site), since we are now having an actual issue with the machine
> (GRUB failure on boot) rather than just a failure to shut down.

We are still having issues with this box. For that reason, we have
moved the gitmaster server over to another box, where it's now up and
running. DNS has been updated, but it will take some time for it to
sync out. For those of you who want access now, you ca nreach the new
master server at 98.129.198.116.

Note, however, that mail relaying from this machine does not currently
work, so commit messages will be out for a bit longer while we work on
clearing things up.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/