Re: snapbuild woes

On 2017-05-01 12:32:07 -0400, Tom Lane wrote:
> Andres Freund <andres(at)anarazel(dot)de> writes:
> > On 2017-05-01 08:46:47 -0400, Tom Lane wrote:
> >> 30sec is kind of a big lump from a buildfarm standpoint, especially if
> >> you mean "it runs for 30s on my honkin' fast workstation". I'm fine
> >> with individual tests that run for ~ 1sec.
>
> > I was more thinking of pgench -T$XX, rather than constant number of
> > iterations. I currently can reproduce the issues within like 3-4
> > minutes, so 5s is probably not quite sufficient to get decent coverage.
>
> Adding a five-minute pgbench run to the buildfarm sequence is definitely
> going to get you ridden out of town on a rail.

Right - that was referring to Noah's comment upthread:

On 2017-04-29 14:42:01 -0700, Noah Misch wrote:
> If the probabilistic test catches the bug even 5% of the time in typical
> configurations, the buildfarm will rapidly identify any regression. I'd
> choose a 7s test that detects the bug 5% of the time over a 30s test that
> detects it 99% of the time. (When I wrote src/bin/pgbench/t/001_pgbench.pl
> for a probabilistic bug, I sized that test to finish in 1s and catch its bug
> half the time. In its case, only two buildfarm members were able to
> demonstrate the original bug, so 5% detection would have been too low.)

and I suspect that you'd not find these within 5s within sufficient
time, because the detection rate would be too low.

> But quite aside from the question of whether we can afford the cycles,
> it seems like the wrong approach. IMO the buildfarm is mainly for
> verifying portability, not for trying to prove that race-like
> conditions don't exist. In most situations we're going out of our way
> to ensure reproduceability of tests we add to the buildfarm sequence;
> but it seems like this is looking for irreproducible results.

Yea, I wondered about that upthread as well. But the tests are quite
useful nonetheless. Wonder about adding them simply as a separate
target.

- Andres