Re: visibility map and reltuples

It appears that the visibility map patch is causing pg_class.reltuples to be
set improperly after a vacuum. For example, it is set to 0 if the map
indicated that no pages in the heap needed to be scanned.

Perhaps reltuples should not be updated unless every page was scanned during
the vacuum?

--
Ned T. Crigler

Ned T. Crigler wrote:
> It appears that the visibility map patch is causing pg_class.reltuples to be
> set improperly after a vacuum. For example, it is set to 0 if the map
> indicated that no pages in the heap needed to be scanned.
>
> Perhaps reltuples should not be updated unless every page was scanned during
> the vacuum?

Yeah, vacuum shouldn't overwrite reltuples if it hasn't scanned all pages.

The interplay of vacuum and analyze in VACUUM ANALYZE needs to be
changed too. Currently, the analyze after vacuum doesn't overwrite
reltuples, because the one calculated by vacuum is based on scanning all
pages, and is thus more accurate than the one estimated from the sample
(which is not true anymore, as you pointed out). I think the vacuum
needs to somehow tell analyze whether it updated reltuples or not, so
that analyze can update reltuples if the vacuum didn't scan all pages.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Heikki Linnakangas wrote:
> Ned T. Crigler wrote:
>> It appears that the visibility map patch is causing pg_class.reltuples
>> to be
>> set improperly after a vacuum. For example, it is set to 0 if the map
>> indicated that no pages in the heap needed to be scanned.
>>
>> Perhaps reltuples should not be updated unless every page was scanned
>> during
>> the vacuum?
>
> Yeah, vacuum shouldn't overwrite reltuples if it hasn't scanned all pages.

Because we use reltuples divided by relpages in the planner, we probably
shouldn't update relpages either if we don't update reltuples.
Otherwise, if the table has grown a lot since we last updated reltuples,
the reltuples / relpages ratio would be less, not more, accurate, if
relpages is updated to a new higher value but reltuples is not.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

I wonder if we should switch to keeping reltuplesperpage instead. Then
a partial vacuum could update it by taking the average number of
tuples per page forbthe pages it saw. Perhaps adjusting it to the
weights average between the old value and the new value based on how
many pages were seen.

I suppose there's no reason we can't update reltuples using that same
logic though it would be a big opaque.

--
Greg

On 15 Dec 2008, at 04:01, Heikki Linnakangas <heikki(dot)linnakangas(at)enterprisedb(dot)com
> wrote:

> Heikki Linnakangas wrote:
>> Ned T. Crigler wrote:
>>> It appears that the visibility map patch is causing
>>> pg_class.reltuples to be
>>> set improperly after a vacuum. For example, it is set to 0 if the
>>> map
>>> indicated that no pages in the heap needed to be scanned.
>>>
>>> Perhaps reltuples should not be updated unless every page was
>>> scanned during
>>> the vacuum?
>> Yeah, vacuum shouldn't overwrite reltuples if it hasn't scanned all
>> pages.
>
> Because we use reltuples divided by relpages in the planner, we
> probably shouldn't update relpages either if we don't update
> reltuples. Otherwise, if the table has grown a lot since we last
> updated reltuples, the reltuples / relpages ratio would be less, not
> more, accurate, if relpages is updated to a new higher value but
> reltuples is not.
>
> --
> Heikki Linnakangas
> EnterpriseDB http://www.enterprisedb.com
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers

Greg Stark wrote:
> I wonder if we should switch to keeping reltuplesperpage instead. Then a
> partial vacuum could update it by taking the average number of tuples
> per page forbthe pages it saw. Perhaps adjusting it to the weights
> average between the old value and the new value based on how many pages
> were seen.

The pages scanned by a partial vacuum isn't a random sample of pages in
the table. That would bias the reltuplesperpage value towards those
pages that are updated more.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Heikki Linnakangas <heikki(dot)linnakangas(at)enterprisedb(dot)com> writes:
> Greg Stark wrote:
>> I wonder if we should switch to keeping reltuplesperpage instead. Then a
>> partial vacuum could update it by taking the average number of tuples
>> per page forbthe pages it saw. Perhaps adjusting it to the weights
>> average between the old value and the new value based on how many pages
>> were seen.

> The pages scanned by a partial vacuum isn't a random sample of pages in
> the table. That would bias the reltuplesperpage value towards those
> pages that are updated more.

Yeah ... and it's highly likely that repeatedly-updated pages would have
more dead space than never-updated ones, so there'd be a systematic
creep towards underestimation of the total tuple count.

I think your previous sketch is right: suppress update of reltuples (and
relpages) from a partial vacuum scan, and ensure that the analyze phase
is allowed to do it instead if it happens during VACUUM ANALYZE.

regards, tom lane

On Mon, 15 Dec 2008, Greg Stark wrote:

> I wonder if we should switch to keeping reltuplesperpage instead.

It would be preferrable to not touch the user side of reltuples if
possible, since it's the only instant way to get a good estimate of the
number of rows in a table right now. That's been a regular application
technique for at least two years now, since
http://www.varlena.com/GeneralBits/120.php popularized it.

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD

Tom Lane wrote:
> I think your previous sketch is right: suppress update of reltuples (and
> relpages) from a partial vacuum scan, and ensure that the analyze phase
> is allowed to do it instead if it happens during VACUUM ANALYZE.

We also mustn't reset n_live_tuples in pgstat in partial vacuum.
Committed a patch to do that.

Thanks for the report!

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com