Re: autovacuum and reloptions

Lists: pgsql-hackers
From: Alvaro Herrera <alvherre(at)alvh(dot)no-ip(dot)org>
To: Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: autovacuum and reloptions
Date: 2008-10-08 14:21:12
Message-ID: 20081008142111.GA4167@alvh.no-ip.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

So I gave up waiting for someone else to do the reloptions patch for
autovacuum and started work on it myself. What I soon discovered is
that on first blush it seems a lot easier than I had expected.

On second look, however, the problem is that I seem to be
forced to declare all the autovacuum-related options and their parsing
properties in reloptions.c. This seems very ugly. I'd very much prefer
to be able to declare the options in autovacuum.c and let the rest of
the code just ignore them.

However, parseRelOptions seems inclined to barf about options it doesn't
know about. Maybe that's fine with the current usage, but I think it
would be better to leave the options in StdRdOptions alone, and have the
autovacuum options defined elsewhere, which seems to require an API
change to parseRelOptions -- though I'm not sure what's appropriate.

AFAICS this is completely uncharted territory -- the current code
understands only fillfactor as a valid option. If we were down the
route of just adding the new options just like fillfactor is currently
dealt with, the API would get really ugly very quickly.

It seems to me we should provide a way to "register" valid options, so
that autovacuum.c could inform reloptions.c what are the valid keys that
a normal option parsing should just ignore (and, conversely, what
options should it ignore when parsing for autovacuum). Thinking more
about it, it seems to me that the treatment that fillfactor currently
gets should be ripped out in favor of being registered too, somehow ...

Before we waste too much time thinking how this registering is to be
done, does anybody think that the current approach is OK and thus I
should just add the autovacuum options directly into StdRdOptions and
default_reloptions?

--
Alvaro Herrera Developer, http://www.PostgreSQL.org/
"El Maquinismo fue proscrito so pena de cosquilleo hasta la muerte"
(Ijon Tichy en Viajes, Stanislaw Lem)


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Alvaro Herrera <alvherre(at)alvh(dot)no-ip(dot)org>
Cc: Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2008-10-08 14:57:53
Message-ID: 29896.1223477873@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Alvaro Herrera <alvherre(at)alvh(dot)no-ip(dot)org> writes:
> On second look, however, the problem is that I seem to be
> forced to declare all the autovacuum-related options and their parsing
> properties in reloptions.c. This seems very ugly.

That was in fact the intended design, and is why the functions exist in
a separate file reloptions.c rather than having been dumped into some
existing place like heapam.c. I don't see anything very wrong with
having autovacuum options in StdRdOptions and default_reloptions().

There might at some point be a good case for inventing a plugin approach
here, but since autovacuum is a pretty much core component now, I don't
see the need to do so for it.

regards, tom lane


From: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2008-10-08 15:10:22
Message-ID: 20081008151022.GB4164@alvh.no-ip.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Tom Lane wrote:
> Alvaro Herrera <alvherre(at)alvh(dot)no-ip(dot)org> writes:
> > On second look, however, the problem is that I seem to be
> > forced to declare all the autovacuum-related options and their parsing
> > properties in reloptions.c. This seems very ugly.
>
> That was in fact the intended design, and is why the functions exist in
> a separate file reloptions.c rather than having been dumped into some
> existing place like heapam.c. I don't see anything very wrong with
> having autovacuum options in StdRdOptions and default_reloptions().

Hmm, OK. However, given that the various AMs amoptions also use
default_reloptions, they are going to accept the autovacuum options too.
Is that OK?

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
Cc: Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2008-10-08 15:34:40
Message-ID: 6748.1223480080@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
> Hmm, OK. However, given that the various AMs amoptions also use
> default_reloptions, they are going to accept the autovacuum options too.
> Is that OK?

Well, we might want to split default_reloptions into two versions,
one for heaps and the other for indexes. But it doesn't bother me a
whole lot if we don't.

regards, tom lane


From: Gregory Stark <stark(at)enterprisedb(dot)com>
To: Alvaro Herrera <alvherre(at)alvh(dot)no-ip(dot)org>
Cc: Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2008-10-08 16:08:22
Message-ID: 87d4ibqdc9.fsf@oxford.xeocode.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Alvaro Herrera <alvherre(at)alvh(dot)no-ip(dot)org> writes:

> Before we waste too much time thinking how this registering is to be
> done, does anybody think that the current approach is OK and thus I
> should just add the autovacuum options directly into StdRdOptions and
> default_reloptions?

Given Simon's suggestion that i/o parameters should be per-tablespace I think
we might need to refactor this further.

I wonder if we could piggy-back on guc parameters. So you would register a guc
variable with a flag saying it's sensible to be set per-tablespace or
per-table.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's PostGIS support!


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Gregory Stark <stark(at)enterprisedb(dot)com>
Cc: Alvaro Herrera <alvherre(at)alvh(dot)no-ip(dot)org>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2008-10-08 16:13:02
Message-ID: 11054.1223482382@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Gregory Stark <stark(at)enterprisedb(dot)com> writes:
> I wonder if we could piggy-back on guc parameters.

God, no. GUC is hopelessly complex already, we should *not* try to make
it track different values of a parameter for different tables.

Attaching a reloptions-like column to pg_tablespace might not be
unreasonable ... but I think that has little to do with Alvaro's
immediate problem.

regards, tom lane


From: ITAGAKI Takahiro <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>
To: Alvaro Herrera <alvherre(at)alvh(dot)no-ip(dot)org>
Cc: Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2008-10-09 02:41:19
Message-ID: 20081009113015.78FF.52131E4D@oss.ntt.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Alvaro Herrera <alvherre(at)alvh(dot)no-ip(dot)org> wrote:

> So I gave up waiting for someone else to do the reloptions patch for
> autovacuum and started work on it myself.

Is it needed to keep backward compatibility?

I'd like to suggest to keep pg_catalog.pg_autovacuum as a system view
even after the options is put into reloptons, and the view to be
updatable using RULEs if possible.

Current pg_autovacuum-table approach has a benefit that
we can configure options by rule, for example:

INSERT INTO pg_autovacuu SELECT ... FROM pg_class WHERE ...;

But we will not able to do that if the settings will be in reloptions
because ALTER TABLE SET cannot be used with JOINs.

Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: ITAGAKI Takahiro <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>
Cc: Alvaro Herrera <alvherre(at)alvh(dot)no-ip(dot)org>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2008-10-09 12:47:08
Message-ID: 13123.1223556428@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

ITAGAKI Takahiro <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp> writes:
> Alvaro Herrera <alvherre(at)alvh(dot)no-ip(dot)org> wrote:
>> So I gave up waiting for someone else to do the reloptions patch for
>> autovacuum and started work on it myself.

> Is it needed to keep backward compatibility?

> I'd like to suggest to keep pg_catalog.pg_autovacuum as a system view
> even after the options is put into reloptons, and the view to be
> updatable using RULEs if possible.

Ugh. No. It has been explicitly stated all along that pg_autovacuum
was a temporary API and that anyone depending on it could expect future
trouble.

> But we will not able to do that if the settings will be in reloptions
> because ALTER TABLE SET cannot be used with JOINs.

Any mechanism that a rule might use to set reloptions would be just
as usable in a join as the rule itself ...

regards, tom lane


From: Peter Eisentraut <peter_e(at)gmx(dot)net>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Gregory Stark <stark(at)enterprisedb(dot)com>, Alvaro Herrera <alvherre(at)alvh(dot)no-ip(dot)org>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2008-10-14 15:57:07
Message-ID: 48F4C153.2050102@gmx.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Tom Lane wrote:
> Gregory Stark <stark(at)enterprisedb(dot)com> writes:
>> I wonder if we could piggy-back on guc parameters.
>
> God, no. GUC is hopelessly complex already, we should *not* try to make
> it track different values of a parameter for different tables.

Are there any more specific reasons than "it's very complex"? After
all, all the autovacuum options already exist as GUC parameters, so you
don't have to repeat all the validation code, for example.


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Peter Eisentraut <peter_e(at)gmx(dot)net>
Cc: Gregory Stark <stark(at)enterprisedb(dot)com>, Alvaro Herrera <alvherre(at)alvh(dot)no-ip(dot)org>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2008-10-14 16:32:15
Message-ID: 7780.1224001935@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> Tom Lane wrote:
>> God, no. GUC is hopelessly complex already, we should *not* try to make
>> it track different values of a parameter for different tables.

> Are there any more specific reasons than "it's very complex"?

That one seems quite sufficient to me; but consider dump/restore if you
need more.

regards, tom lane


From: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
To: Alvaro Herrera <alvherre(at)alvh(dot)no-ip(dot)org>
Cc: Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2008-10-15 00:52:16
Message-ID: 48F53EC0.3020004@timbira.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Alvaro Herrera escreveu:
> So I gave up waiting for someone else to do the reloptions patch for
> autovacuum and started work on it myself. What I soon discovered is
> that on first blush it seems a lot easier than I had expected.
>
Sorry about that. :( I was swamped with PGCon Brasil and then I took
some days to rest. I'm expecting to finish it before next CF.

What did I already do? I refactored reloptions.c to support multiple
options. I tried to follow up the same way GUC do (of course, it is much
simpler). I'm thinking about removing (replacing?) StdRdOptions 'cause
we need a different struct to store reloptions. Suggestions?

I'm attaching the WIP patch so you can comment on it. I want to continue
working on it but I'm afraid you already did more than I do (in this
case, let me know for not duplicating efforts).

--
Euler Taveira de Oliveira
http://www.timbira.com/

Attachment Content-Type Size
relopt5.diff text/plain 21.5 KB

From: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
To: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
Cc: Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2008-10-15 16:23:18
Message-ID: 20081015162318.GD6012@alvh.no-ip.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Euler Taveira de Oliveira wrote:

> What did I already do? I refactored reloptions.c to support multiple
> options. I tried to follow up the same way GUC do (of course, it is much
> simpler). I'm thinking about removing (replacing?) StdRdOptions 'cause
> we need a different struct to store reloptions. Suggestions?
>
> I'm attaching the WIP patch so you can comment on it. I want to continue
> working on it but I'm afraid you already did more than I do (in this
> case, let me know for not duplicating efforts).

Interesting. The main problem with this patch is that it loses the
ability to pass to parseRelOptions the set of options that are valid for
each context. Right now all callers use the same list comprising only
fillfactor, but my guess is that once we add new options it will make
sense to start differentiating. It makes no sense for indexes to have
autovacuum options, for example.

This is why I suggested in the email that started this thread we needed
some sort of registering capability (which was thrown down). I think
the main idea in your patch is fine, and better than what I was doing
which was just adding every option in default_reloptions. However it
needs to be adjusted somehow so that it doesn't mix all the options.
Maybe we could add a "category" bitmask for which each option would be
valid.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.


From: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
To: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
Cc: Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2008-10-15 16:45:58
Message-ID: 48F61E46.1050300@timbira.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Alvaro Herrera escreveu:

> Maybe we could add a "category" bitmask for which each option would be
> valid.
>
The category tests are fine, that's why I introduced relopt_gen.kind but
I'm not using it yet. I'll try to refactor it to use bitmask (some
options could be used to both -- fillfactor) and to add it in the
validation code.

--
Euler Taveira de Oliveira
http://www.timbira.com/


From: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
To: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
Cc: Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2008-10-15 22:46:30
Message-ID: 20081015224630.GI6012@alvh.no-ip.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Euler Taveira de Oliveira wrote:
> Alvaro Herrera escreveu:
>
> > Maybe we could add a "category" bitmask for which each option would be
> > valid.
> >
> The category tests are fine, that's why I introduced relopt_gen.kind but
> I'm not using it yet.

Ah, right, I hadn't noticed the kind stuff, maybe because it's unused
;-)

> I'll try to refactor it to use bitmask (some options could be used to
> both -- fillfactor) and to add it in the validation code.

Right, that's why I suggested using a bitmask.

Okay, so I'll let you deal with this for a while yet. A minor
suggestion: label the enum members distinctively, i.e. instead of
RO_BOOL perhaps use RELOPT_TYPE_BOOL, and RO_HEAP should be
RELOPT_KIND_HEAP (note this cannot be a plain enum, each category needs
a separate bit).

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
Cc: Euler Taveira de Oliveira <euler(at)timbira(dot)com>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2008-10-16 02:58:21
Message-ID: 2631.1224125901@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
> Okay, so I'll let you deal with this for a while yet. A minor
> suggestion: label the enum members distinctively, i.e. instead of
> RO_BOOL perhaps use RELOPT_TYPE_BOOL, and RO_HEAP should be
> RELOPT_KIND_HEAP (note this cannot be a plain enum, each category needs
> a separate bit).

My first reaction was that the categories should just be the different
possible values of relkind. However, it also seems possible that there
could be index-AM-specific reloptions. So maybe what we need is a
categorization that is like relkind but breaks down RELKIND_INDEX into a
category per AM.

regards, tom lane


From: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
To: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
Cc: Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2008-11-03 16:22:48
Message-ID: 20081103162248.GL4509@alvh.no-ip.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Euler Taveira de Oliveira wrote:
> Alvaro Herrera escreveu:
> > So I gave up waiting for someone else to do the reloptions patch for
> > autovacuum and started work on it myself. What I soon discovered is
> > that on first blush it seems a lot easier than I had expected.
> >
> Sorry about that. :( I was swamped with PGCon Brasil and then I took
> some days to rest. I'm expecting to finish it before next CF.

So did this go anywhere?

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.


From: ITAGAKI Takahiro <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>
To: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
Cc: Euler Taveira de Oliveira <euler(at)timbira(dot)com>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2008-11-11 03:28:20
Message-ID: 20081111121845.7BE9.52131E4D@oss.ntt.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

I have a comment about reloptions of autovacuum parameters:
I'd like to have an easier way to extract individual parameters.

Alvaro Herrera <alvherre(at)commandprompt(dot)com> wrote:

> Euler Taveira de Oliveira wrote:
> > What did I already do? I refactored reloptions.c to support multiple
> > options. I tried to follow up the same way GUC do (of course, it is much
> > simpler). I'm thinking about removing (replacing?) StdRdOptions 'cause
> > we need a different struct to store reloptions. Suggestions?
> Interesting.

We store reloptions as an array of 'key=value' text, but there is
no official way to read each parameter. I always create an user
defined function to extract fillfactors, but it would be better
if we have a standard method to do that.

---- [brute force way]
CREATE FUNCTION pg_fillfactor(reloptions text[], relam OID)
RETURNS integer AS
$$
SELECT (regexp_matches(array_to_string($1, '/'),
'fillfactor=([0-9]+)'))[1]::integer AS fillfactor
UNION ALL
SELECT CASE $2
WHEN 0 THEN 100 -- heap
WHEN 403 THEN 90 -- btree
WHEN 405 THEN 70 -- hash
WHEN 783 THEN 90 -- gist
WHEN 2742 THEN 100 -- gin
END
LIMIT 1;
$$
LANGUAGE sql STABLE;
----

Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center


From: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
To: ITAGAKI Takahiro <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>
Cc: Alvaro Herrera <alvherre(at)commandprompt(dot)com>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2008-11-11 13:13:29
Message-ID: 491984F9.8070203@timbira.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

ITAGAKI Takahiro escreveu:

[Sorry for the delay. I'm preparing the final patch and in a day or so
I'll post it.]

> I have a comment about reloptions of autovacuum parameters:
> I'd like to have an easier way to extract individual parameters.
>
> Alvaro Herrera <alvherre(at)commandprompt(dot)com> wrote:
>
>> Euler Taveira de Oliveira wrote:
>>> What did I already do? I refactored reloptions.c to support multiple
>>> options. I tried to follow up the same way GUC do (of course, it is much
>>> simpler). I'm thinking about removing (replacing?) StdRdOptions 'cause
>>> we need a different struct to store reloptions. Suggestions?
>> Interesting.
>
> We store reloptions as an array of 'key=value' text, but there is
> no official way to read each parameter. I always create an user
> defined function to extract fillfactors, but it would be better
> if we have a standard method to do that.
>
We will have an official function (getRelOptions()?) to extract the
reloption (autovacuum) parameters because we need to transform
pg_autovacuum catalog table in a view to maintain backward compatibility.

--
Euler Taveira de Oliveira
http://www.timbira.com/


From: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
To: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
Cc: ITAGAKI Takahiro <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2008-11-11 14:29:34
Message-ID: 20081111142934.GG4272@alvh.no-ip.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Euler Taveira de Oliveira wrote:

> We will have an official function (getRelOptions()?) to extract the
> reloption (autovacuum) parameters because we need to transform
> pg_autovacuum catalog table in a view to maintain backward compatibility.

Why? I'd say don't waste your time. If anything, it can be done after
the initial patch has been committed.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support


From: ITAGAKI Takahiro <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>
To: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
Cc: Alvaro Herrera <alvherre(at)commandprompt(dot)com>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2008-11-12 02:24:20
Message-ID: 20081112111746.9AFA.52131E4D@oss.ntt.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers


Euler Taveira de Oliveira <euler(at)timbira(dot)com> wrote:

> We will have an official function (getRelOptions()?) to extract the
> reloption (autovacuum) parameters

Yeah, I want it, but...

> because we need to transform
> pg_autovacuum catalog table in a view to maintain backward compatibility.

I don't think we need pg_autovacuum in 8.4 because it is a deprecated API.
However, we should provide some easier methods to create an user-defined
pg_autovacuum view in case that users really need it.

Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center


From: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
To: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
Cc: ITAGAKI Takahiro <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>, Alvaro Herrera <alvherre(at)commandprompt(dot)com>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2008-11-21 05:23:07
Message-ID: 492645BB.8000805@timbira.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Euler Taveira de Oliveira escreveu:

> [Sorry for the delay. I'm preparing the final patch and in a day or so
> I'll post it.]
>
Here is the patch that replace pg_autovaccum catalog with reloptions. I
refactored the reloptions.c to support multiple parameters and made the
action of adding a new option an easy task. I'm testing it yet, so don't
expect it to work properly. I'll prepare docs as soon as I finish the
tests. Do i have to prepare some regression tests?

I don't provide a pg_autovacuum view as suggested by Itagari-san [1] but
if others agree that we need it, I will work on it. I don't if we need a
function (wrapper around getRelOption()) to get an option from
reloptions array.

I add an ugly-hack to \d+ foo. IMHO, it'll be good to know what options
are used by table/index foo (we already do it for oids) but I'm not
happy with my suggestion.

I move RelationGet*() functions from rel.h. That's because we need some
knowledge that's only in reloptions.c (getRelOptions). But I want to
avoid including reloptions.h at some files.

Comments?

PS> don't forget to remove include/catalog/pg_autovacuum.h

[1] http://archives.postgresql.org/pgsql-hackers/2008-11/msg00830.php

--
Euler Taveira de Oliveira
http://www.timbira.com/

Attachment Content-Type Size
relopt27.diff.gz application/x-gzip 15.1 KB

From: "Robert Haas" <robertmhaas(at)gmail(dot)com>
To: "Euler Taveira de Oliveira" <euler(at)timbira(dot)com>
Cc: "ITAGAKI Takahiro" <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>, "Alvaro Herrera" <alvherre(at)commandprompt(dot)com>, "Pg Hackers" <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2009-01-11 02:33:24
Message-ID: 603c8f070901101833j3f5e52daxf54414e2dd6f51b3@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

> Here is the patch that replace pg_autovaccum catalog with reloptions. I
> refactored the reloptions.c to support multiple parameters and made the
> action of adding a new option an easy task. I'm testing it yet, so don't
> expect it to work properly. I'll prepare docs as soon as I finish the
> tests. Do i have to prepare some regression tests?
>
> I don't provide a pg_autovacuum view as suggested by Itagari-san [1] but
> if others agree that we need it, I will work on it. I don't if we need a
> function (wrapper around getRelOption()) to get an option from
> reloptions array.
>
> I add an ugly-hack to \d+ foo. IMHO, it'll be good to know what options
> are used by table/index foo (we already do it for oids) but I'm not
> happy with my suggestion.
>
> I move RelationGet*() functions from rel.h. That's because we need some
> knowledge that's only in reloptions.c (getRelOptions). But I want to
> avoid including reloptions.h at some files.
>
> Comments?
>
> PS> don't forget to remove include/catalog/pg_autovacuum.h

Several things related to this patch have been committed:

http://archives.postgresql.org/message-id/20081219143958.6F2DD7563FE@cvs.postgresql.org
http://archives.postgresql.org/message-id/20090105171428.77B29754A17@cvs.postgresql.org
(and several follow-on commits)

What still remains to be done for 8.4?

...Robert


From: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
To: Robert Haas <robertmhaas(at)gmail(dot)com>
Cc: ITAGAKI Takahiro <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>, Alvaro Herrera <alvherre(at)commandprompt(dot)com>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2009-01-11 23:47:40
Message-ID: 496A851C.6020308@timbira.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Robert Haas escreveu:
> Several things related to this patch have been committed:
>
> http://archives.postgresql.org/message-id/20081219143958.6F2DD7563FE@cvs.postgresql.org
> http://archives.postgresql.org/message-id/20090105171428.77B29754A17@cvs.postgresql.org
> (and several follow-on commits)
>
> What still remains to be done for 8.4?
>
autovacuum part?

--
Euler Taveira de Oliveira
http://www.timbira.com/


From: "Robert Haas" <robertmhaas(at)gmail(dot)com>
To: "Euler Taveira de Oliveira" <euler(at)timbira(dot)com>
Cc: "ITAGAKI Takahiro" <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>, "Alvaro Herrera" <alvherre(at)commandprompt(dot)com>, "Pg Hackers" <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2009-01-12 01:40:08
Message-ID: 603c8f070901111740y634192edh5bbbaa76fe9f08f0@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Sun, Jan 11, 2009 at 6:47 PM, Euler Taveira de Oliveira
<euler(at)timbira(dot)com> wrote:
> Robert Haas escreveu:
>> Several things related to this patch have been committed:
>>
>> http://archives.postgresql.org/message-id/20081219143958.6F2DD7563FE@cvs.postgresql.org
>> http://archives.postgresql.org/message-id/20090105171428.77B29754A17@cvs.postgresql.org
>> (and several follow-on commits)
>>
>> What still remains to be done for 8.4?
>>
> autovacuum part?

I guess Alvaro is working on that? Or are you?

...Robert


From: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
To: Robert Haas <robertmhaas(at)gmail(dot)com>
Cc: Euler Taveira de Oliveira <euler(at)timbira(dot)com>, ITAGAKI Takahiro <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2009-01-12 11:15:33
Message-ID: 20090112111533.GA4919@alvh.no-ip.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Robert Haas escribió:
> On Sun, Jan 11, 2009 at 6:47 PM, Euler Taveira de Oliveira
> <euler(at)timbira(dot)com> wrote:
> > Robert Haas escreveu:
> >> Several things related to this patch have been committed:
> >>
> >> http://archives.postgresql.org/message-id/20081219143958.6F2DD7563FE@cvs.postgresql.org
> >> http://archives.postgresql.org/message-id/20090105171428.77B29754A17@cvs.postgresql.org
> >> (and several follow-on commits)
> >>
> >> What still remains to be done for 8.4?
> >>
> > autovacuum part?
>
> I guess Alvaro is working on that? Or are you?

I have a followup patch that allows one to use a namespace in ALTER
TABLE SET commands, like this:

alter table foo set (toast.fillfactor = 50)

This is still WIP because it has some minor annoyances. I will finish
and commit the table-based amoptions infrastructure and then be back on
the namespace patch.

This namespace patch is a prerequisite of the autovacuum work, because
without it, it is impossible to change autovacuum settings for toast
tables, which would be a regression.

I have a separate branch on which I keep the old patch from Euler
updated to the current reloptions code; so it is probably very similar
to what Euler just sent. (I'll have a look at that soon anyway.)

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support


From: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
To: Robert Haas <robertmhaas(at)gmail(dot)com>
Cc: Euler Taveira de Oliveira <euler(at)timbira(dot)com>, ITAGAKI Takahiro <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2009-01-12 13:01:52
Message-ID: 20090112130152.GF4919@alvh.no-ip.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Alvaro Herrera escribió:

> I have a separate branch on which I keep the old patch from Euler
> updated to the current reloptions code; so it is probably very similar
> to what Euler just sent. (I'll have a look at that soon anyway.)

Huh, nevermind -- I thought that Euler had just sent an updated version
of his patch, but only now I noticed that the message I was looking at
is dated Nov. 21st.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support


From: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
To: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
Cc: Robert Haas <robertmhaas(at)gmail(dot)com>, ITAGAKI Takahiro <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2009-01-12 18:14:14
Message-ID: 496B8876.7060609@timbira.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Alvaro Herrera escreveu:
> Alvaro Herrera escribió:
>
>> I have a separate branch on which I keep the old patch from Euler
>> updated to the current reloptions code; so it is probably very similar
>> to what Euler just sent. (I'll have a look at that soon anyway.)
>
> Huh, nevermind -- I thought that Euler had just sent an updated version
> of his patch, but only now I noticed that the message I was looking at
> is dated Nov. 21st.
>
And I did it (It is even listed in the wiki). Last version was [1]. I fixed
some problems and added some docs.

[1] http://archives.postgresql.org/pgsql-hackers/2008-12/msg00292.php

--
Euler Taveira de Oliveira
http://www.timbira.com/


From: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
To: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
Cc: Robert Haas <robertmhaas(at)gmail(dot)com>, ITAGAKI Takahiro <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2009-01-12 18:41:13
Message-ID: 20090112184113.GI4919@alvh.no-ip.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Euler Taveira de Oliveira escribió:
> Alvaro Herrera escreveu:
> > Alvaro Herrera escribió:
> >
> >> I have a separate branch on which I keep the old patch from Euler
> >> updated to the current reloptions code; so it is probably very similar
> >> to what Euler just sent. (I'll have a look at that soon anyway.)
> >
> > Huh, nevermind -- I thought that Euler had just sent an updated version
> > of his patch, but only now I noticed that the message I was looking at
> > is dated Nov. 21st.
> >
> And I did it (It is even listed in the wiki). Last version was [1]. I fixed
> some problems and added some docs.

Yes, the autovacuum patch I have is loosely based on that one IIRC. (I
modified it heavily though.)

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support


From: Peter Eisentraut <peter_e(at)gmx(dot)net>
To: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
Cc: Euler Taveira de Oliveira <euler(at)timbira(dot)com>, Robert Haas <robertmhaas(at)gmail(dot)com>, ITAGAKI Takahiro <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2009-01-23 08:25:30
Message-ID: 49797EFA.3020009@gmx.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Alvaro Herrera wrote:
> Euler Taveira de Oliveira escribió:
>> Alvaro Herrera escreveu:
>>> Alvaro Herrera escribió:
>>>
>>>> I have a separate branch on which I keep the old patch from Euler
>>>> updated to the current reloptions code; so it is probably very similar
>>>> to what Euler just sent. (I'll have a look at that soon anyway.)
>>> Huh, nevermind -- I thought that Euler had just sent an updated version
>>> of his patch, but only now I noticed that the message I was looking at
>>> is dated Nov. 21st.
>>>
>> And I did it (It is even listed in the wiki). Last version was [1]. I fixed
>> some problems and added some docs.
>
> Yes, the autovacuum patch I have is loosely based on that one IIRC. (I
> modified it heavily though.)

Could someone udpate the status of this item in the commitfest wiki?


From: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
To: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
Cc: Robert Haas <robertmhaas(at)gmail(dot)com>, ITAGAKI Takahiro <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2009-01-23 16:17:35
Message-ID: 20090123161735.GH4047@alvh.no-ip.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Alvaro Herrera escribió:
> Euler Taveira de Oliveira escribió:

> > And I did it (It is even listed in the wiki). Last version was [1]. I fixed
> > some problems and added some docs.
>
> Yes, the autovacuum patch I have is loosely based on that one IIRC. (I
> modified it heavily though.)

Here's my proposed patch. There is a bug in the handling of TOAST
tables; I'm sending this as a WIP to add it to the commitfest status
page for this patch.

This also needs to be merged with my other patch to handle reloptions
namespaces, which will allow us to set reloptions for TOAST tables.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Attachment Content-Type Size
reloptions-autovacuum-2.patch text/x-diff 67.9 KB

From: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
To: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
Cc: Robert Haas <robertmhaas(at)gmail(dot)com>, ITAGAKI Takahiro <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2009-01-23 18:08:05
Message-ID: 20090123180805.GM4047@alvh.no-ip.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Alvaro Herrera escribió:

> Here's my proposed patch. There is a bug in the handling of TOAST
> tables; I'm sending this as a WIP to add it to the commitfest status
> page for this patch.

Sorry, that was a really stupid bug.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Attachment Content-Type Size
reloptions-autovacuum-2.1.patch text/x-diff 514 bytes

From: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
To: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
Cc: Robert Haas <robertmhaas(at)gmail(dot)com>, ITAGAKI Takahiro <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2009-02-05 21:49:01
Message-ID: 20090205214901.GH3064@alvh.no-ip.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

So here's what looks like a committable patch.

Note to self: remember to remove src/include/catalog/pg_autovacuum.h and
to bump catversion.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Attachment Content-Type Size
reloptions-autovacuum-4.1.patch text/x-diff 62.0 KB

From: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
To: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
Cc: Robert Haas <robertmhaas(at)gmail(dot)com>, ITAGAKI Takahiro <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2009-02-06 02:25:47
Message-ID: 498B9FAB.8040002@timbira.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Alvaro Herrera escreveu:
> So here's what looks like a committable patch.
>
> Note to self: remember to remove src/include/catalog/pg_autovacuum.h and
> to bump catversion.
>
>
Works for me. Just a few comments.

(i) I don't like this construction "by entries by changing storage
parameters". I prefer "by changing storage parameters" or "by entries in
pg_class.reloptions";

(ii) I think we should change the expression "storage parameters" for
something else because autovacuum is related to maintenance. My suggestion is
a general expression like "relation parameters";

(iii) I noticed that GUC defaults and relopt defaults are different
(autovacuum_cost_delay and autovacuum_cost_limit). Is there any reason for not
using -1?

(iv) Maybe we should document that pg_dump will only dump reloptions like
toast.foo iff the relation has an associated TOAST table. This seems obvious
but ...

--
Euler Taveira de Oliveira
http://www.timbira.com/


From: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
To: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
Cc: Robert Haas <robertmhaas(at)gmail(dot)com>, ITAGAKI Takahiro <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2009-02-06 17:30:33
Message-ID: 20090206173033.GB3089@alvh.no-ip.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Euler Taveira de Oliveira escribió:

> (i) I don't like this construction "by entries by changing storage
> parameters". I prefer "by changing storage parameters" or "by entries in
> pg_class.reloptions";

Heh, obvious "by entries by" is a cut'n'pasto; fixed.

Maybe an xref would be better there. I attach the doc patch again; I
fiddled a bit with it this morning. Comments?

> (ii) I think we should change the expression "storage parameters" for
> something else because autovacuum is related to maintenance. My suggestion is
> a general expression like "relation parameters";

I'm not sure I agree with this idea, because the term "storage
parameter" has been used for several releases already. This would be a
relatively major terminology change.

> (iii) I noticed that GUC defaults and relopt defaults are different
> (autovacuum_cost_delay and autovacuum_cost_limit). Is there any reason for not
> using -1?

Yeah, -1 does not make sense. It made sense in pg_autovacuum because
otherwise you didn't have to specify "skip this setting and use the
default". In reloptions, if you don't want to specify the setting, just
don't specify it.

> (iv) Maybe we should document that pg_dump will only dump reloptions like
> toast.foo iff the relation has an associated TOAST table. This seems obvious
> but ...

Well, if it doesn't have a toast table, then there's no need for toast
settings, is there? I mean, you could construe it as a gotcha: if you
CREATE TABLE with only fixed-length columns and specify some reloptions,
and later add a column that requires a toast table, it won't have the
options you set at CREATE time. However, it seems to me to be very low
in the importance scale.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Attachment Content-Type Size
reloptions-autovacuum-docs.patch text/x-diff 27.6 KB

From: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
To: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
Cc: Robert Haas <robertmhaas(at)gmail(dot)com>, ITAGAKI Takahiro <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2009-02-06 22:45:22
Message-ID: 498CBD82.1000308@timbira.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Alvaro Herrera escreveu:
>> (ii) I think we should change the expression "storage parameters" for
>> something else because autovacuum is related to maintenance. My suggestion is
>> a general expression like "relation parameters";
>
> I'm not sure I agree with this idea, because the term "storage
> parameter" has been used for several releases already. This would be a
> relatively major terminology change.
>
I don't buy your argument. 'fillfactor' is a _storage_ parameter but
'autovacuum_enabled' is not. I don't like terminology changes too but in this
case it sounds strange calling autovacuum_* as storage parameters.

--
Euler Taveira de Oliveira
http://www.timbira.com/


From: Jaime Casanova <jcasanov(at)systemguards(dot)com(dot)ec>
To: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
Cc: Alvaro Herrera <alvherre(at)commandprompt(dot)com>, Robert Haas <robertmhaas(at)gmail(dot)com>, ITAGAKI Takahiro <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: autovacuum and reloptions
Date: 2009-02-07 05:10:24
Message-ID: 3073cc9b0902062110l35159095od47a200c9fbb7194@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Fri, Feb 6, 2009 at 5:45 PM, Euler Taveira de Oliveira
<euler(at)timbira(dot)com> wrote:
> Alvaro Herrera escreveu:
>>> (ii) I think we should change the expression "storage parameters" for
>>> something else because autovacuum is related to maintenance. My suggestion is
>>> a general expression like "relation parameters";
>>
>> I'm not sure I agree with this idea, because the term "storage
>> parameter" has been used for several releases already. This would be a
>> relatively major terminology change.
>>
> I don't buy your argument. 'fillfactor' is a _storage_ parameter but
> 'autovacuum_enabled' is not. I don't like terminology changes too but in this
> case it sounds strange calling autovacuum_* as storage parameters.
>

+1

--
Atentamente,
Jaime Casanova
Soporte y capacitación de PostgreSQL
Asesoría y desarrollo de sistemas
Guayaquil - Ecuador
Cel. +59387171157