Re: marking old branches as no longer maintained

I'd like to add a feature to the buildfarm that automatically picks up
any new branch and automatically stops building any branch we're not
maintaining any more. To do the latter, I need some way for the client
to detect that we are or aren't interested in a branch. What I'd like to
do is add a file to the old back branches (say from 7.4 to 8.1 currently
- I can grandfather the rest) called "end-of-life-reached" or some such,
with some appropriate text. As a branch reaches its end of life, i.e.
right before the last release we make, we should add that file to the
branch.

I think this would possibly be useful anyway, regardless of buildfarm
utility - I still hear of people mistakenly grabbing and building
releases past EOL, and maybe this will give one or two the extra clue
they need that this is less than a good idea.

Comments?

cheers

andrew

On Tue, Jun 28, 2011 at 19:46, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>
> I'd like to add a feature to the buildfarm that automatically picks up any
> new branch and automatically stops building any branch we're not maintaining
> any more. To do the latter, I need some way for the client to detect that we
> are or aren't interested in a branch. What I'd like to do is add a file to
> the old back branches (say from 7.4 to 8.1 currently - I can grandfather the
> rest) called "end-of-life-reached" or some such, with some appropriate text.
> As a branch reaches its end of life, i.e. right before the last release we
> make, we should add that file to the branch.

Does this need to be driven out of the main tree? Couldn't you just
have a "blacklist" in the buildfarm code or site? (disclaimer: I
haven't looked at how it works so that may be a completely useless
idea..)

Another way would be to just not run bulids if there are no commits in
<n> days on a branch. Don't we already not run builds on branches with
no comments? Maybe just put a limit on how long we allow an override
of that?

> I think this would possibly be useful anyway, regardless of buildfarm
> utility - I still hear of people mistakenly grabbing and building releases
> past EOL, and maybe this will give one or two the extra clue they need that
> this is less than a good idea.

If you want that to actually work, you probably need to do something
to the point of breaking the configure script. There's zero chance of
people who're not reading the information about which releases are
supported are actually going read a random file somewhere in the
source tree, regardless of where you place it and what you name it.

You could reqiure something like "./configure
--yes-i-know-what-i-am-doing" or something like that, I guess...

> Comments?

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On 06/28/2011 01:54 PM, Magnus Hagander wrote:
> On Tue, Jun 28, 2011 at 19:46, Andrew Dunstan<andrew(at)dunslane(dot)net> wrote:
>> I'd like to add a feature to the buildfarm that automatically picks up any
>> new branch and automatically stops building any branch we're not maintaining
>> any more. To do the latter, I need some way for the client to detect that we
>> are or aren't interested in a branch. What I'd like to do is add a file to
>> the old back branches (say from 7.4 to 8.1 currently - I can grandfather the
>> rest) called "end-of-life-reached" or some such, with some appropriate text.
>> As a branch reaches its end of life, i.e. right before the last release we
>> make, we should add that file to the branch.
> Does this need to be driven out of the main tree? Couldn't you just
> have a "blacklist" in the buildfarm code or site? (disclaimer: I
> haven't looked at how it works so that may be a completely useless
> idea..)

Not very easily, mainly because of difficulties with MinGW SDK perl.
Building it into the code would defeat the purpose.

If it's contentious I won't bother. We've managed OK for years, and can
go on managing as we are.

cheers

andrew

On Tue, Jun 28, 2011 at 20:38, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>
>
> On 06/28/2011 01:54 PM, Magnus Hagander wrote:
>>
>> On Tue, Jun 28, 2011 at 19:46, Andrew Dunstan<andrew(at)dunslane(dot)net>  wrote:
>>>
>>> I'd like to add a feature to the buildfarm that automatically picks up
>>> any
>>> new branch and automatically stops building any branch we're not
>>> maintaining
>>> any more. To do the latter, I need some way for the client to detect that
>>> we
>>> are or aren't interested in a branch. What I'd like to do is add a file
>>> to
>>> the old back branches (say from 7.4 to 8.1 currently - I can grandfather
>>> the
>>> rest) called "end-of-life-reached" or some such, with some appropriate
>>> text.
>>> As a branch reaches its end of life, i.e. right before the last release
>>> we
>>> make, we should add that file to the branch.
>>
>> Does this need to be driven out of the main tree? Couldn't you just
>> have a "blacklist" in the buildfarm code or site? (disclaimer: I
>> haven't looked at how it works so that may be a completely useless
>> idea..)
>
>
> Not very easily, mainly because of difficulties with MinGW SDK perl.
> Building it into the code would defeat the purpose.

Drat.

> If it's contentious I won't bother. We've managed OK for years, and can go
> on managing as we are.

If we can find a good way to do it, I think having BF animals
automatically picking up new branches is a very good thing to have. So
don't give up so easily :D If adding a more or less random file to
back branches is the only way to do it, I'm for doing that - I'd just
like to find some method that feels cleaner. But maybe I'm just
bikeshedding for no real use here.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Tue, Jun 28, 2011 at 8:02 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>
> If we can find a good way to do it, I think having BF animals
> automatically picking up new branches is a very good thing to have. So
> don't give up so easily :D If adding a more or less random file to
> back branches is the only way to do it, I'm for doing that - I'd just
> like to find some method that feels cleaner. But maybe I'm just
> bikeshedding for no real use here.

Adding new branches automatically would be great, but it'll need some
work from the animal herders as well as some careful design - for
example, my Windows animals have separate schedules for each branch
(some running more frequently than others), whilst my Solaris ones now
use a runner script that cycles through the list of branches on each
of a couple of animals.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 06/28/2011 03:02 PM, Magnus Hagander wrote:
>
>> If it's contentious I won't bother. We've managed OK for years, and can go
>> on managing as we are.
> If we can find a good way to do it, I think having BF animals
> automatically picking up new branches is a very good thing to have. So
> don't give up so easily :D If adding a more or less random file to
> back branches is the only way to do it, I'm for doing that - I'd just
> like to find some method that feels cleaner. But maybe I'm just
> bikeshedding for no real use here.

Another way would be for us to have a file on master with the names of
the branches we care about.

cheers

andrew

On 06/28/2011 03:17 PM, Dave Page wrote:
> On Tue, Jun 28, 2011 at 8:02 PM, Magnus Hagander<magnus(at)hagander(dot)net> wrote:
>> If we can find a good way to do it, I think having BF animals
>> automatically picking up new branches is a very good thing to have. So
>> don't give up so easily :D If adding a more or less random file to
>> back branches is the only way to do it, I'm for doing that - I'd just
>> like to find some method that feels cleaner. But maybe I'm just
>> bikeshedding for no real use here.
> Adding new branches automatically would be great, but it'll need some
> work from the animal herders as well as some careful design - for
> example, my Windows animals have separate schedules for each branch
> (some running more frequently than others), whilst my Solaris ones now
> use a runner script that cycles through the list of branches on each
> of a couple of animals.

Modern buildfarm code has a wrapper builtin. So my crontab usually just
looks like this:

27 * * * * cd bf && ./run_branches.pl --config=nightjar.conf --run-all

The buildfarm.conf has a section like this:

if ($branch eq 'global')
{
$conf{branches_to_build} = [qw( HEAD REL9_1_STABLE
REL9_0_STABLE REL8_4_STABLE REL8_3_STABLE REL8_2_STABLE)];
}

What I'd like to do is to allow this to read:

if ($branch eq 'global')
{
$conf{branches_to_build} = 'ALL';
}

and have it choose the right set for you.

But if you want to run some more frequently you'd still be stuck having
to manage that yourself. There's actually not a lot of point in doing it
that way, though. We don't build unless there have been changes on the
branch, unless told otherwise, so you might as well run frequently and
test all the branches - for the most part only HEAD (i.e. master) will
be built because it gets far more changes than the back branches.

cheers

andrew

On Tuesday, June 28, 2011, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>
>
> On 06/28/2011 03:17 PM, Dave Page wrote:
>
> On Tue, Jun 28, 2011 at 8:02 PM, Magnus Hagander<magnus(at)hagander(dot)net>  wrote:
>
> If we can find a good way to do it, I think having BF animals
> automatically picking up new branches is a very good thing to have. So
> don't give up so easily :D If adding a more or less random file to
> back branches is the only way to do it, I'm for doing that - I'd just
> like to find some method that feels cleaner. But maybe I'm just
> bikeshedding for no real use here.
>
> Adding new branches automatically would be great, but it'll need some
> work from the animal herders as well as some careful design - for
> example, my Windows animals have separate schedules for each branch
> (some running more frequently than others), whilst my Solaris ones now
> use a runner script that cycles through the list of branches on each
> of a couple of animals.
>
>
> Modern buildfarm code has a wrapper builtin. So my crontab usually just looks like this:
>
>   27 * * * * cd bf && ./run_branches.pl --config=nightjar.conf --run-all
>
> The buildfarm.conf has a section like this:
>
>   if ($branch eq 'global')
>   {
>        $conf{branches_to_build} = [qw( HEAD REL9_1_STABLE
>   REL9_0_STABLE REL8_4_STABLE REL8_3_STABLE REL8_2_STABLE)];
>   }
>
> What I'd like to do is to allow this to read:
>
>   if ($branch eq 'global')
>   {
>        $conf{branches_to_build} = 'ALL';
>   }
>
> and have it choose the right set for you.

Oh, cool. Guess I'll be reconfiguring my animals soon :-)

> But if you want to run some more frequently you'd still be stuck having to manage that yourself. There's actually not a lot of point in doing it that way, though. We don't build unless there have been changes on the branch, unless told otherwise, so you might as well run frequently and test all the branches - for the most part only HEAD (i.e. master) will be built because it gets far more changes than the back branches.

It was something Tom asked for ages ago, so he could see if the
Windows build got broken more promptly. I didn't want multiple
branches running with increased frequency as I run a number of animals
on a single machine with vmware, and a back patched change could cause
a lot of extra work.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 06/28/2011 03:48 PM, Dave Page wrote:
>
>> But if you want to run some more frequently you'd still be stuck having to manage that yourself. There's actually not a lot of point in doing it that way, though. We don't build unless there have been changes on the branch, unless told otherwise, so you might as well run frequently and test all the branches - for the most part only HEAD (i.e. master) will be built because it gets far more changes than the back branches.
> It was something Tom asked for ages ago, so he could see if the
> Windows build got broken more promptly. I didn't want multiple
> branches running with increased frequency as I run a number of animals
> on a single machine with vmware, and a back patched change could cause
> a lot of extra work.

Oh, so maybe we need to have some sort of throttle. Probably just for
non-head or non-(head-or-latest) would suffice.

cheers

andrew

On tis, 2011-06-28 at 15:37 -0400, Andrew Dunstan wrote:
> What I'd like to do is to allow this to read:
>
> if ($branch eq 'global')
> {
> $conf{branches_to_build} = 'ALL';
> }
>
> and have it choose the right set for you.

It seems to me that if you put a marker file into old branches, you'd
still have to check out and poll the old branches, which could become
somewhat expensive as the number of old branches grows.

Couldn't you just put a text file on the build farm server with
recommended branches?

On 06/28/2011 04:51 PM, Peter Eisentraut wrote:
> On tis, 2011-06-28 at 15:37 -0400, Andrew Dunstan wrote:
>> What I'd like to do is to allow this to read:
>>
>> if ($branch eq 'global')
>> {
>> $conf{branches_to_build} = 'ALL';
>> }
>>
>> and have it choose the right set for you.
> It seems to me that if you put a marker file into old branches, you'd
> still have to check out and poll the old branches, which could become
> somewhat expensive as the number of old branches grows.

No, not really. I'd use 'git ls-tree $branchname $filetolookfor". I have
tested it and this works just fine, takes a second or so.

> Couldn't you just put a text file on the build farm server with
> recommended branches?

As I told Magnus, that gets ugly because of limitations in MinGW's SDK
perl. I suppose I could just not implement the feature for MinGW, but
I've tried damn hard not to make those sorts of compromises and I'm not
keen to start.

cheers

andrew

On tis, 2011-06-28 at 17:05 -0400, Andrew Dunstan wrote:
> > Couldn't you just put a text file on the build farm server with
> > recommended branches?
>
> As I told Magnus, that gets ugly because of limitations in MinGW's SDK
> perl. I suppose I could just not implement the feature for MinGW, but
> I've tried damn hard not to make those sorts of compromises and I'm not
> keen to start.

The buildfarm code can upload the build result via HTTP; why can't it
download a file via HTTP?

On 06/28/2011 05:31 PM, Peter Eisentraut wrote:
> On tis, 2011-06-28 at 17:05 -0400, Andrew Dunstan wrote:
>>> Couldn't you just put a text file on the build farm server with
>>> recommended branches?
>> As I told Magnus, that gets ugly because of limitations in MinGW's SDK
>> perl. I suppose I could just not implement the feature for MinGW, but
>> I've tried damn hard not to make those sorts of compromises and I'm not
>> keen to start.
> The buildfarm code can upload the build result via HTTP; why can't it
> download a file via HTTP?

It has to use a separate script to do that. I don't really want to add
another one just for this.

(thinks a bit) I suppose I can make it do:

my $url = "http://buildfarm.postgresql.org/branches_of_interest.txt";
my $branches_of_interest = `perl -MLWP::Simple -e "getprint(q{$url})"`;

Maybe that's the best option. It's certainly going to be less code than
anything else :-)

cheers

andrew

Andrew Dunstan wrote:
>
>
> On 06/28/2011 05:31 PM, Peter Eisentraut wrote:
> > On tis, 2011-06-28 at 17:05 -0400, Andrew Dunstan wrote:
> >>> Couldn't you just put a text file on the build farm server with
> >>> recommended branches?
> >> As I told Magnus, that gets ugly because of limitations in MinGW's SDK
> >> perl. I suppose I could just not implement the feature for MinGW, but
> >> I've tried damn hard not to make those sorts of compromises and I'm not
> >> keen to start.
> > The buildfarm code can upload the build result via HTTP; why can't it
> > download a file via HTTP?
>
>
> It has to use a separate script to do that. I don't really want to add
> another one just for this.
>
> (thinks a bit) I suppose I can make it do:
>
> my $url = "http://buildfarm.postgresql.org/branches_of_interest.txt";
> my $branches_of_interest = `perl -MLWP::Simple -e "getprint(q{$url})"`;
>
> Maybe that's the best option. It's certainly going to be less code than
> anything else :-)

Could you pull the list of active branches from our web site HTML?
\
--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On 07/11/2011 07:59 PM, Bruce Momjian wrote:
> Andrew Dunstan wrote:
>>
>> On 06/28/2011 05:31 PM, Peter Eisentraut wrote:
>>> On tis, 2011-06-28 at 17:05 -0400, Andrew Dunstan wrote:
>>>>> Couldn't you just put a text file on the build farm server with
>>>>> recommended branches?
>>>> As I told Magnus, that gets ugly because of limitations in MinGW's SDK
>>>> perl. I suppose I could just not implement the feature for MinGW, but
>>>> I've tried damn hard not to make those sorts of compromises and I'm not
>>>> keen to start.
>>> The buildfarm code can upload the build result via HTTP; why can't it
>>> download a file via HTTP?
>>
>> It has to use a separate script to do that. I don't really want to add
>> another one just for this.
>>
>> (thinks a bit) I suppose I can make it do:
>>
>> my $url = "http://buildfarm.postgresql.org/branches_of_interest.txt";
>> my $branches_of_interest = `perl -MLWP::Simple -e "getprint(q{$url})"`;
>>
>> Maybe that's the best option. It's certainly going to be less code than
>> anything else :-)
> Could you pull the list of active branches from our web site HTML?
>

I can, but I'm not that keen on having to do web scraping. Currently my
test machine (crake) is using the above scheme and it's working fine.
It's not a huge burden to maintain, after all.

cheers

andrew

On Tue, Jul 12, 2011 at 01:10, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>
>
> On 07/11/2011 07:59 PM, Bruce Momjian wrote:
>>
>> Andrew Dunstan wrote:
>>>
>>> On 06/28/2011 05:31 PM, Peter Eisentraut wrote:
>>>>
>>>> On tis, 2011-06-28 at 17:05 -0400, Andrew Dunstan wrote:
>>>>>>
>>>>>> Couldn't you just put a text file on the build farm server with
>>>>>> recommended branches?
>>>>>
>>>>> As I told Magnus, that gets ugly because of limitations in MinGW's SDK
>>>>> perl. I suppose I could just not implement the feature for MinGW, but
>>>>> I've tried damn hard not to make those sorts of compromises and I'm not
>>>>> keen to start.
>>>>
>>>> The buildfarm code can upload the build result via HTTP; why can't it
>>>> download a file via HTTP?
>>>
>>> It has to use a separate script to do that. I don't really want to add
>>> another one just for this.
>>>
>>> (thinks a bit) I suppose I can make it do:
>>>
>>>     my $url = "http://buildfarm.postgresql.org/branches_of_interest.txt";
>>>     my $branches_of_interest = `perl -MLWP::Simple -e
>>> "getprint(q{$url})"`;
>>>
>>> Maybe that's the best option. It's certainly going to be less code than
>>> anything else :-)
>>
>> Could you pull the list of active branches from our web site HTML?
>>
>
> I can, but I'm not that keen on having to do web scraping. Currently my test
> machine (crake) is using the above scheme and it's working fine. It's not a
> huge burden to maintain, after all.

You don't actually need to resort to web scraping - it's available as
well formatted xml (http://www.postgresql.org/versions.rss).

That said, I agree that it's not a huge burden, and probably a better
idea, to do it your current way.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/