From: | Bruce Momjian <bruce(at)momjian(dot)us> |
---|---|
To: | Robert Haas <robertmhaas(at)gmail(dot)com> |
Cc: | PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>, Josh Berkus <josh(at)agliodbs(dot)com> |
Subject: | Re: Auto-tuning work_mem and maintenance_work_mem |
Date: | 2013-10-10 03:13:14 |
Message-ID: | 20131010031314.GJ7092@momjian.us |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
On Wed, Oct 9, 2013 at 09:34:16PM -0400, Robert Haas wrote:
> On Wed, Oct 9, 2013 at 9:11 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> > On Wed, Oct 9, 2013 at 08:55:33PM -0400, Robert Haas wrote:
> >> On Wed, Oct 9, 2013 at 4:10 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> >> > I disagree. I think we can get a forumla that is certainly better than
> >> > a fixed value. I think the examples I have shown do have better value
> >> > than a default fixed value. I am open to whatever forumula people think
> >> > is best, but I can't see how a fixed value is a win in general.
> >>
> >> To really do auto-tuning correctly, we need to add a GUC, or some
> >> platform-dependent code, or both, for the amount of memory on the
> >> machine, which is not and should not be assumed to have anything to do
> >> with shared_buffers, which is often set to very small values like
> >> 256MB on Windows, and even on Linux, may not be more than 2GB even on
> >> a very large machine. With that, we could set a much better value for
> >> effective_cache_size, and it would help here, too.
> >
> > If you are setting shared_buffers low, you probably want the others low
> > too,
>
> I don't think that's true. People set shared_buffers low because when
> they set it high, they get write I/O storms that cripple their system
> at checkpoint time, or because they need to minimize double-buffering.
If people are doing such changes, they are obviously capable of knowing
their workload and setting these things to non-default values.
> > or can change them.
>
> That is obviously true, but it's true now, too.
And that comment is helpful how?
> >> to know why this is better than setting work_mem to 4MB and calling it
> >> good. I accept that the current default is too low; I do not accept
> >
> > For servers that are not dedicated, a fixed value can easily be too
> > large, and for a larger server, the value can easily be too small. Not
> > sure how you can argue that a fixed value could be better.
>
> But your auto-tuned value can easily be too low or too high, too.
My option is better, not perfect --- I don't know how many times I can
say something again and again. Fortunately there are enough people who
understand that on the lists.
> Consider someone with a system that has 64GB of RAM. EnterpriseDB
> has had customers who have found that with, say, a 40GB database, it's
> best to set shared_buffers to 40GB so that the database remains fully
> cached. Your latest formula will auto-tune work_mem to roughly 100MB.
> On the other hand, if the same customer has a 400GB database, which
> can't be fully cached no matter what, a much lower setting for
> shared_buffers, like maybe 8GB, is apt to perform better. Your
> formula will auto-tune shared_buffers to roughly 20MB.
You mean work_mem?
> In other words, when there's only 24GB of memory available for
> everything-except-shared-buffers, your formula sets work_mem five
> times higher than when there's 48GB of memory available for
> everything-except-shared-buffers. That surely can't be right.
Again, IT ISN'T PERFECT, AND NOTHING WILL BE PERFECT, EVENT HAND TUNING.
This is about improvement for a typical workload.
> >> that the correct value has anything to do with the size of
> >> shared_buffers.
> >
> > Well, an open item is to add an available_memory GUC and base everything
> > on that, including shared_buffers. That would allow Windows-specific
> > adjustments for the default.
>
> That seems considerably more principled than this patch.
That was Josh Berkus's idea. I am fine writing 20x more lines of code
to improve this, but I am determined this will be improved.
--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ Everyone has their own god. +
From | Date | Subject | |
---|---|---|---|
Next Message | Peter Geoghegan | 2013-10-10 03:15:44 | Re: Auto-tuning work_mem and maintenance_work_mem |
Previous Message | Bruce Momjian | 2013-10-10 03:02:13 | Re: Auto-tuning work_mem and maintenance_work_mem |