Re: Fixed xloginsert_locks for 9.4

I did a fair dive into double-checking the decision to just leave
xloginsert_locks fixed at 8 for 9.4. My conclusion: good call, move
along. Further improvements beyond what the 8-way split gives sure are
possible. But my guess from chasing them a little is that additional
places will pop up as things that must also be tweaked, before you'll
see those gains turn significant.

I'd like to see that box re-opened at one point. But if we do that, I'm
comfortable that could end with a xloginsert_locks that tunes itself
reasonably on large servers in the end, similar to wal_buffers. There's
nothing about this that makes feel like it needs a GUC. I barely needed
an exposed knob to do this evaluation.

= Baseline =

I rolled back a few commits to just before the GUC was removed and
tested against that point in git time. Starting with the 4 client test
case Heikki provided, the fastest runs on my 24 core server looked like
this:

tps = 56.691855 (including connections establishing)

Repeat runs do need to drop the table and rebuild, because eventually AV
kicks in on things in a big way, and then your test is toast until it's
done. Attached is what I settled on for a test harness. Nothing here
was so subtle I felt a more complicated harness was needed.

Standard practice for me is to give pgbench more workers when worrying
about any scalability tests. That gives a tiny improvement, to where
this is typical with 4 clients and 4 workers:

tps = 60.942537 (including connections establishing)

Increasing to 24 clients plus 24 workers gives roughly the same numbers,
suggesting that the bottleneck here is certainly not the client count,
and that the suggestion of 4 was high enough:

tps = 56.731581 (including connections establishing)

Decreasing xloginsert_locks to 1, so back to the original problem, the
rate normally looks like this instead:

tps = 25.384708 (including connections establishing)

So the big return you get just fine with the default tuning; great. I'm
happy to see it ship like this as good enough for 9.4.

= More locks =

For the next phase, I stuck to 24 clients and 24 workers. If I then
bump up xloginsert_locks to something much larger, there is an
additional small gain to be had. With 24 locks, so basically ever
client has their own, instead of 57-60 TPS, I managed to get as high as
this:

tps = 66.790968 (including connections establishing)

However, the minute I get into this territory, there's an obvious
bottleneck shift going on in there too. The rate of creating new
checkpoint segments becomes troublesome as one example, with messages
like this:

LOG: checkpoints are occurring too frequently (1 second apart)
HINT: Consider increasing the configuration parameter
"checkpoint_segments".

When 9.4 is already giving a more than 100% gain on this targeted test
case, I can't see that chasing after maybe an extra 10% is worth having
yet another GUC around. Especially when it will probably take multiple
tuning steps before you're done anyway; we don't really know the rest of
them yet; and when we do, we probably won't need a GUC to cope with them
in the end anyway.

--
Greg Smith greg(dot)smith(at)crunchydatasolutions(dot)com
Chief PostgreSQL Evangelist - http://crunchydatasolutions.com/

On Thu, Oct 2, 2014 at 5:08 PM, Greg Smith
<greg(dot)smith(at)crunchydatasolutions(dot)com> wrote:
> When 9.4 is already giving a more than 100% gain on this targeted test case,
> I can't see that chasing after maybe an extra 10% is worth having yet
> another GUC around. Especially when it will probably take multiple tuning
> steps before you're done anyway; we don't really know the rest of them yet;
> and when we do, we probably won't need a GUC to cope with them in the end
> anyway.

Agreed. I think that prior to 9.4, the logging performance of Postgres
was very competitive when compared to other systems. At this stage,
it's probably extremely fast by any standard. Amit's work on only
WAL-logging the modified portion of UPDATEs helps here too.

I tend to believe that the next big round of performance gains can be
had by working on the buffer manager, and B-Tree indexes. At some
point we should work on prefix compression within B-Tree leaf pages.
We should also work on adding "abbreviated keys" to B-Tree internal
pages. Doing so should almost remove the benefit of using the C
locale, because most comparisons needed for index scans can use
comparisons implemented as nothing more than a memcmp() (note that
internal pages have values that are naturally heterogeneous, so this
will work well).

--
Peter Geoghegan

On 2014-10-02 20:08:33 -0400, Greg Smith wrote:
> I did a fair dive into double-checking the decision to just leave
> xloginsert_locks fixed at 8 for 9.4. My conclusion: good call, move along.
> Further improvements beyond what the 8-way split gives sure are possible.
> But my guess from chasing them a little is that additional places will pop
> up as things that must also be tweaked, before you'll see those gains turn
> significant.

Thanks for doing this.

> I'd like to see that box re-opened at one point. But if we do that, I'm
> comfortable that could end with a xloginsert_locks that tunes itself
> reasonably on large servers in the end, similar to wal_buffers. There's
> nothing about this that makes feel like it needs a GUC. I barely needed an
> exposed knob to do this evaluation.
>
> = Baseline =
>
> I rolled back a few commits to just before the GUC was removed and tested
> against that point in git time. Starting with the 4 client test case Heikki
> provided, the fastest runs on my 24 core server looked like this:
>
> tps = 56.691855 (including connections establishing)
>
> Repeat runs do need to drop the table and rebuild, because eventually AV
> kicks in on things in a big way, and then your test is toast until it's
> done. Attached is what I settled on for a test harness. Nothing here was so
> subtle I felt a more complicated harness was needed.
>
> Standard practice for me is to give pgbench more workers when worrying about
> any scalability tests. That gives a tiny improvement, to where this is
> typical with 4 clients and 4 workers:
>
> tps = 60.942537 (including connections establishing)
>
> Increasing to 24 clients plus 24 workers gives roughly the same numbers,
> suggesting that the bottleneck here is certainly not the client count, and
> that the suggestion of 4 was high enough:
>
> tps = 56.731581 (including connections establishing)
>
> Decreasing xloginsert_locks to 1, so back to the original problem, the rate
> normally looks like this instead:
>
> tps = 25.384708 (including connections establishing)
>
> So the big return you get just fine with the default tuning; great. I'm
> happy to see it ship like this as good enough for 9.4.
>
> = More locks =
>
> For the next phase, I stuck to 24 clients and 24 workers. If I then bump up
> xloginsert_locks to something much larger, there is an additional small gain
> to be had. With 24 locks, so basically ever client has their own, instead
> of 57-60 TPS, I managed to get as high as this:
>
> tps = 66.790968 (including connections establishing)
>
> However, the minute I get into this territory, there's an obvious bottleneck
> shift going on in there too. The rate of creating new checkpoint segments
> becomes troublesome as one example, with messages like this:
>
> LOG: checkpoints are occurring too frequently (1 second apart)
> HINT: Consider increasing the configuration parameter
> "checkpoint_segments".
>
> When 9.4 is already giving a more than 100% gain on this targeted test case,
> I can't see that chasing after maybe an extra 10% is worth having yet
> another GUC around. Especially when it will probably take multiple tuning
> steps before you're done anyway; we don't really know the rest of them yet;
> and when we do, we probably won't need a GUC to cope with them in the end
> anyway.

I've modified the test slightly, by having the different backends insert
into different relations. Even on my measly 5 year old workstation I
*do* see quite a bit more than 10%.

psql -f /tmp/prepare.sql && pgbench -n -f /tmp/fooinsert.sql -c 64 -j 64 -T 10
on a 2x E5520 server (2 sockets a 4 cores a 2 threads)
with the following configuration:
-c shared_buffers=2GB
-c wal_level=hot_standby
-c full_page_writes=off
-c checkpoint_segments=400
-c fsync=off (io system here is abysmally bad)
-c synchronous_commit=off

#define NUM_XLOGINSERT_LOCKS 1
tps = 52.711939 (including connections establishing)
#define NUM_XLOGINSERT_LOCKS 8
tps = 286.496054 (including connections establishing)
#define NUM_XLOGINSERT_LOCKS 16
tps = 346.113313 (including connections establishing)
#define NUM_XLOGINSERT_LOCKS 24
tps = 363.242111 (including connections establishing)

I'd not be surprised at all if you'd see bigger influence on a system
with 4 sockets.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 10/3/14, 8:26 AM, Andres Freund wrote:
> #define NUM_XLOGINSERT_LOCKS 1
> tps = 52.711939 (including connections establishing)
> #define NUM_XLOGINSERT_LOCKS 8
> tps = 286.496054 (including connections establishing)
> #define NUM_XLOGINSERT_LOCKS 16
> tps = 346.113313 (including connections establishing)
> #define NUM_XLOGINSERT_LOCKS 24
> tps = 363.242111 (including connections establishing)

Just to clarify: that 10% number I threw out was meant as a rough
estimate for a system with the default configuration, which is all that
I tested. It seemed like people would likely need to tune all the usual
things like checkpoint_segments, shared_buffers, etc. as well before
seeing much better. You did all that, and sure enough the gain went up;
thanks for confirming my guess.

I still don't think that means this needs a GUC for 9.4. Look at that
jump from 1 to 8. The low-hanging fruit here hasn't just been knocked
off. It's been blended into a frozen drink, poured into a glass, and
had a little paper umbrella put on top. I think that's enough for 9.4.
But, yes, let's see if we can add delivery to the side of the pool in
the next version too.

--
Greg Smith greg(dot)smith(at)crunchydatasolutions(dot)com
Chief PostgreSQL Evangelist - http://crunchydatasolutions.com/

On 2014-10-03 10:07:39 -0400, Gregory Smith wrote:
> On 10/3/14, 8:26 AM, Andres Freund wrote:
> >#define NUM_XLOGINSERT_LOCKS 1
> >tps = 52.711939 (including connections establishing)
> >#define NUM_XLOGINSERT_LOCKS 8
> >tps = 286.496054 (including connections establishing)
> >#define NUM_XLOGINSERT_LOCKS 16
> >tps = 346.113313 (including connections establishing)
> >#define NUM_XLOGINSERT_LOCKS 24
> >tps = 363.242111 (including connections establishing)
>
> Just to clarify: that 10% number I threw out was meant as a rough estimate
> for a system with the default configuration, which is all that I tested. It
> seemed like people would likely need to tune all the usual things like
> checkpoint_segments, shared_buffers, etc. as well before seeing much better.
> You did all that, and sure enough the gain went up; thanks for confirming my
> guess.
>
> I still don't think that means this needs a GUC for 9.4. Look at that jump
> from 1 to 8. The low-hanging fruit here hasn't just been knocked off. It's
> been blended into a frozen drink, poured into a glass, and had a little
> paper umbrella put on top. I think that's enough for 9.4. But, yes, let's
> see if we can add delivery to the side of the pool in the next version too.

So 25% performance on a relatively small machine improvements aren't
worth a GUC? That are likely to be larger on a bigger machine?

I utterly fail to see why that's a service to our users.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Fri, Oct 3, 2014 at 04:11:30PM +0200, Andres Freund wrote:
> On 2014-10-03 10:07:39 -0400, Gregory Smith wrote:
> > On 10/3/14, 8:26 AM, Andres Freund wrote:
> > >#define NUM_XLOGINSERT_LOCKS 1
> > >tps = 52.711939 (including connections establishing)
> > >#define NUM_XLOGINSERT_LOCKS 8
> > >tps = 286.496054 (including connections establishing)
> > >#define NUM_XLOGINSERT_LOCKS 16
> > >tps = 346.113313 (including connections establishing)
> > >#define NUM_XLOGINSERT_LOCKS 24
> > >tps = 363.242111 (including connections establishing)
> >
> > Just to clarify: that 10% number I threw out was meant as a rough estimate
> > for a system with the default configuration, which is all that I tested. It
> > seemed like people would likely need to tune all the usual things like
> > checkpoint_segments, shared_buffers, etc. as well before seeing much better.
> > You did all that, and sure enough the gain went up; thanks for confirming my
> > guess.
> >
> > I still don't think that means this needs a GUC for 9.4. Look at that jump
> > from 1 to 8. The low-hanging fruit here hasn't just been knocked off. It's
> > been blended into a frozen drink, poured into a glass, and had a little
> > paper umbrella put on top. I think that's enough for 9.4. But, yes, let's
> > see if we can add delivery to the side of the pool in the next version too.
>
> So 25% performance on a relatively small machine improvements aren't
> worth a GUC? That are likely to be larger on a bigger machine?
>
> I utterly fail to see why that's a service to our users.

Well, I think the issue is that having a GUC that can't reasonably be
tuned by 95% of our users is nearly useless. Few users are going to run
benchmarks to see what the optimal value is.

I remember Informix had a setting that had no description except "try
different values to see if it helps performance" --- we don't want to do
that.

What if we emit a server message if the setting is too low? That's how
we handle checkpoint_segments.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +

On Fri, Oct 3, 2014 at 1:40 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:

> On Fri, Oct 3, 2014 at 04:11:30PM +0200, Andres Freund wrote:
> > On 2014-10-03 10:07:39 -0400, Gregory Smith wrote:
> > > On 10/3/14, 8:26 AM, Andres Freund wrote:
> > > >#define NUM_XLOGINSERT_LOCKS 1
> > > >tps = 52.711939 (including connections establishing)
> > > >#define NUM_XLOGINSERT_LOCKS 8
> > > >tps = 286.496054 (including connections establishing)
> > > >#define NUM_XLOGINSERT_LOCKS 16
> > > >tps = 346.113313 (including connections establishing)
> > > >#define NUM_XLOGINSERT_LOCKS 24
> > > >tps = 363.242111 (including connections establishing)
> > >
> > > Just to clarify: that 10% number I threw out was meant as a rough
> estimate
> > > for a system with the default configuration, which is all that I
> tested. It
> > > seemed like people would likely need to tune all the usual things like
> > > checkpoint_segments, shared_buffers, etc. as well before seeing much
> better.
> > > You did all that, and sure enough the gain went up; thanks for
> confirming my
> > > guess.
> > >
> > > I still don't think that means this needs a GUC for 9.4. Look at that
> jump
> > > from 1 to 8. The low-hanging fruit here hasn't just been knocked
> off. It's
> > > been blended into a frozen drink, poured into a glass, and had a little
> > > paper umbrella put on top. I think that's enough for 9.4. But, yes,
> let's
> > > see if we can add delivery to the side of the pool in the next version
> too.
> >
> > So 25% performance on a relatively small machine improvements aren't
> > worth a GUC? That are likely to be larger on a bigger machine?
> >
> > I utterly fail to see why that's a service to our users.
>
> Well, I think the issue is that having a GUC that can't reasonably be
> tuned by 95% of our users is nearly useless. Few users are going to run
> benchmarks to see what the optimal value is.
>
> I remember Informix had a setting that had no description except "try
> different values to see if it helps performance" --- we don't want to do
> that.
>
> What if we emit a server message if the setting is too low? That's how
> we handle checkpoint_segments.
>
> --
> Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
> EnterpriseDB http://enterprisedb.com
>
> + Everyone has their own god. +
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>

Not all GUC need to be straight forward to tune.
If the gains are worthy I don't see any reason not to have it.

On Fri, Oct 3, 2014 at 03:00:56PM -0300, Arthur Silva wrote:
> I remember Informix had a setting that had no description except "try
> different values to see if it helps performance" --- we don't want to do
> that.
>
> What if we emit a server message if the setting is too low? That's how
> we handle checkpoint_segments.
>
> Not all GUC need to be straight forward to tune.
> If the gains are worthy I don't see any reason not to have it.

Every GUC add complexity to the system because people have to understand
it to know if they should tune it. No GUC is zero-cost.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +

On Fri, Oct 3, 2014 at 02:07:45PM -0400, Bruce Momjian wrote:
> On Fri, Oct 3, 2014 at 03:00:56PM -0300, Arthur Silva wrote:
> > I remember Informix had a setting that had no description except "try
> > different values to see if it helps performance" --- we don't want to do
> > that.
> >
> > What if we emit a server message if the setting is too low? That's how
> > we handle checkpoint_segments.
> >
> > Not all GUC need to be straight forward to tune.
> > If the gains are worthy I don't see any reason not to have it.
>
> Every GUC add complexity to the system because people have to understand
> it to know if they should tune it. No GUC is zero-cost.

Please see my blog post about the cost of adding GUCs:

http://momjian.us/main/blogs/pgblog/2009.html#January_10_2009

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +

On Fri, Oct 3, 2014 at 3:10 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:

> On Fri, Oct 3, 2014 at 02:07:45PM -0400, Bruce Momjian wrote:
> > On Fri, Oct 3, 2014 at 03:00:56PM -0300, Arthur Silva wrote:
> > > I remember Informix had a setting that had no description except
> "try
> > > different values to see if it helps performance" --- we don't want
> to do
> > > that.
> > >
> > > What if we emit a server message if the setting is too low?
> That's how
> > > we handle checkpoint_segments.
> > >
> > > Not all GUC need to be straight forward to tune.
> > > If the gains are worthy I don't see any reason not to have it.
> >
> > Every GUC add complexity to the system because people have to understand
> > it to know if they should tune it. No GUC is zero-cost.
>
> Please see my blog post about the cost of adding GUCs:
>
> http://momjian.us/main/blogs/pgblog/2009.html#January_10_2009
>
> --
> Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
> EnterpriseDB http://enterprisedb.com
>
> + Everyone has their own god. +
>

That's true Bruce (nice post, it was a good reading).
But how can we ignore 25%+ improvements (from 8 to 24)?
At very least we should delivery some pretty good defaults.

Bruce Momjian <bruce(at)momjian(dot)us> writes:
> On Fri, Oct 3, 2014 at 03:00:56PM -0300, Arthur Silva wrote:
>> Not all GUC need to be straight forward to tune.
>> If the gains are worthy I don't see any reason not to have it.

> Every GUC add complexity to the system because people have to understand
> it to know if they should tune it. No GUC is zero-cost.

In particular, the cost of putting this one back would be documenting
what it does and how to tune it. As mentioned upthread, we're not
following that Informix precedent ;-)

regards, tom lane

On Fri, Oct 3, 2014 at 03:30:35PM -0300, Arthur Silva wrote:
> > Every GUC add complexity to the system because people have to understand
> > it to know if they should tune it. No GUC is zero-cost.
>
> Please see my blog post about the cost of adding GUCs:
>
> http://momjian.us/main/blogs/pgblog/2009.html#January_10_2009
>
> That's true Bruce (nice post, it was a good reading).
> But how can we ignore 25%+ improvements (from 8 to 24)?
> At very least we should delivery some pretty good defaults.

Well, checkpoint_segments was a similar case where we couldn't give good
tuning advice so we went with a server log file warning if it needed to
be increased --- this might be a similar case.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +

On 10/03/2014 09:42 PM, Bruce Momjian wrote:
> On Fri, Oct 3, 2014 at 03:30:35PM -0300, Arthur Silva wrote:
>> > Every GUC add complexity to the system because people have to understand
>> > it to know if they should tune it. No GUC is zero-cost.
>>
>> Please see my blog post about the cost of adding GUCs:
>>
>> http://momjian.us/main/blogs/pgblog/2009.html#January_10_2009
>>
>> That's true Bruce (nice post, it was a good reading).
>> But how can we ignore 25%+ improvements (from 8 to 24)?
>> At very least we should delivery some pretty good defaults.
>
> Well, checkpoint_segments was a similar case where we couldn't give good
> tuning advice so we went with a server log file warning if it needed to
> be increased --- this might be a similar case.

I have no idea how to decide at runtime whether it should be increased
or not. If that was feasible, we probably could make it tune itself on
the fly - it's not like checkpoint_segments where you need more disk
space if you increase it.

I stand by my decision to make it a #define, at least until someone
voices their objection in the form of a documentation patch.

- Heikki

On Fri, Oct 3, 2014 at 2:49 PM, Heikki Linnakangas
<hlinnakangas(at)vmware(dot)com> wrote:
> I stand by my decision to make it a #define, at least until someone voices
> their objection in the form of a documentation patch.

I think that's exactly right. If we knew users should tune this, then
we might be able to make it self-tuning, which would be best of all.
At the least, we'd have useful information to provide to people who
want to change it. However, if we *can't* provide tuning advice, then
all making it a GUC does is give users a knob that says "turning this
might make things better! or worse!". Adding such knobs does little
harm individually, but in the aggregate it makes for a product that is
mysterious, hard to use, and hard to maintain and improve.

In practice, I suspect few people will get the amount of benefit that
Andres saw in his tests. You have to be xloginsert-bound, and a lot
of workloads aren't. And if they were before, they aren't now that we
have 8 xloginsert slots by default.

My suspicion, which I think is what Andres was getting at as well, is
that you'll want more slots if you have more CPUs. One way to measure
the effectiveness of a given value (and maybe even auto-tune) would be
to count how often you run out of slots. Of course, you'd need some
countervailing form of pressure based on the effort that you're
spending locking however many you have rather than some smaller
number. You can't just measure the upside without measuring the
downside.

The xlog slots use the kind of algorithm that I quite frankly hate
even when it works. It avoids one kind of loss but in a way that's
not scalable; as you crank up the number of slots you start to incur
cost elsewhere. Getting any more scalability beyond that point will
require doing something fundamentally different, or at least that's my
guess. So I feel like we've put our finger in the dike more than
anything, but that still beats letting the dike collapse.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Fri, Oct 3, 2014 at 2:20 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Fri, Oct 3, 2014 at 2:49 PM, Heikki Linnakangas
> <hlinnakangas(at)vmware(dot)com> wrote:
>> I stand by my decision to make it a #define, at least until someone voices
>> their objection in the form of a documentation patch.
>
> I think that's exactly right. If we knew users should tune this, then
> we might be able to make it self-tuning, which would be best of all.
> At the least, we'd have useful information to provide to people who
> want to change it. However, if we *can't* provide tuning advice, then
> all making it a GUC does is give users a knob that says "turning this
> might make things better! or worse!". Adding such knobs does little
> harm individually, but in the aggregate it makes for a product that is
> mysterious, hard to use, and hard to maintain and improve.

100% agree. It's a very simple standard: if you provide a performance
affecting GUC pleease provide guidelines to end user regarding the
tradeoffs or performance interactions being managed. Failure to do
this causes an interesting problem; let's take the case of shared
buffers for example which isn't explained very well. Failure to
properly document or explain the interactions causes the user to make
invalid assumptions about the setting (more memory = faster!) or take
obsolete advice (Tom Lane said in 1997 that 128mb is about right) as
gospel. This has been a long running gripe of mine.

merlin

On 2014-10-03 12:40:21 -0400, Bruce Momjian wrote:
> On Fri, Oct 3, 2014 at 04:11:30PM +0200, Andres Freund wrote:
> > On 2014-10-03 10:07:39 -0400, Gregory Smith wrote:
> > > On 10/3/14, 8:26 AM, Andres Freund wrote:
> > > >#define NUM_XLOGINSERT_LOCKS 1
> > > >tps = 52.711939 (including connections establishing)
> > > >#define NUM_XLOGINSERT_LOCKS 8
> > > >tps = 286.496054 (including connections establishing)
> > > >#define NUM_XLOGINSERT_LOCKS 16
> > > >tps = 346.113313 (including connections establishing)
> > > >#define NUM_XLOGINSERT_LOCKS 24
> > > >tps = 363.242111 (including connections establishing)
> > >
> > > Just to clarify: that 10% number I threw out was meant as a rough estimate
> > > for a system with the default configuration, which is all that I tested. It
> > > seemed like people would likely need to tune all the usual things like
> > > checkpoint_segments, shared_buffers, etc. as well before seeing much better.
> > > You did all that, and sure enough the gain went up; thanks for confirming my
> > > guess.
> > >
> > > I still don't think that means this needs a GUC for 9.4. Look at that jump
> > > from 1 to 8. The low-hanging fruit here hasn't just been knocked off. It's
> > > been blended into a frozen drink, poured into a glass, and had a little
> > > paper umbrella put on top. I think that's enough for 9.4. But, yes, let's
> > > see if we can add delivery to the side of the pool in the next version too.
> >
> > So 25% performance on a relatively small machine improvements aren't
> > worth a GUC? That are likely to be larger on a bigger machine?
> >
> > I utterly fail to see why that's a service to our users.
>
> Well, I think the issue is that having a GUC that can't reasonably be
> tuned by 95% of our users is nearly useless. Few users are going to run
> benchmarks to see what the optimal value is.

Sure. And the default loooks to be a good one. So it's not bad that they
don't tune it further.
But. How are we ever going to be able to tune it further without
feedback from actual production usage?

It's possible to convince customers to play with a performance
influencing parameter and see how the results are. Even in
production. It's near impossible to do so if that requires to download
source packages, change a define in some .c file, rebuild the packages,
distribute them to the respective servers. And then continue to do so
for every release thereafter.

Not only is that a *significant* amount of work. It often involves a
different set of people (sysadmin, not dba-ish people). And often
complicated procedures to allow patching the source code at all.

> I remember Informix had a setting that had no description except "try
> different values to see if it helps performance" --- we don't want to do
> that.

I'm pretty sure we can come up with a better description than that.

> What if we emit a server message if the setting is too low? That's how
> we handle checkpoint_segments.

I don't think it's realistically possible in this case. The
instrumentation we'd need to add would be too expensive for something
running as frequently as this.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> On 2014-10-03 12:40:21 -0400, Bruce Momjian wrote:
>> Well, I think the issue is that having a GUC that can't reasonably be
>> tuned by 95% of our users is nearly useless. Few users are going to run
>> benchmarks to see what the optimal value is.

> It's possible to convince customers to play with a performance
> influencing parameter and see how the results are. Even in
> production.

I'm a bit dubious that people will be willing to experiment in production
with a GUC that requires a database restart to change.

regards, tom lane

On 2014-10-03 17:55:19 -0400, Tom Lane wrote:
> Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> > On 2014-10-03 12:40:21 -0400, Bruce Momjian wrote:
> >> Well, I think the issue is that having a GUC that can't reasonably be
> >> tuned by 95% of our users is nearly useless. Few users are going to run
> >> benchmarks to see what the optimal value is.
>
> > It's possible to convince customers to play with a performance
> > influencing parameter and see how the results are. Even in
> > production.
>
> I'm a bit dubious that people will be willing to experiment in production
> with a GUC that requires a database restart to change.

I've convinced customers to restart databases with several different
shared_buffers settings... So I'm pretty sure it's possible, in *some*
cases, for xloginsert_slots.

And even if it's just test/pre-production machines - they're not going
to benchmark settings they can't reasonably set in production.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Fri, Oct 3, 2014 at 11:58:14PM +0200, Andres Freund wrote:
> On 2014-10-03 17:55:19 -0400, Tom Lane wrote:
> > Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> > > On 2014-10-03 12:40:21 -0400, Bruce Momjian wrote:
> > >> Well, I think the issue is that having a GUC that can't reasonably be
> > >> tuned by 95% of our users is nearly useless. Few users are going to run
> > >> benchmarks to see what the optimal value is.
> >
> > > It's possible to convince customers to play with a performance
> > > influencing parameter and see how the results are. Even in
> > > production.
> >
> > I'm a bit dubious that people will be willing to experiment in production
> > with a GUC that requires a database restart to change.
>
> I've convinced customers to restart databases with several different
> shared_buffers settings... So I'm pretty sure it's possible, in *some*
> cases, for xloginsert_slots.
>
> And even if it's just test/pre-production machines - they're not going
> to benchmark settings they can't reasonably set in production.

Do we really want to expose a setting a few of us _might_ ask customers
to change? I don't think so --- create a custom build if you want that.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +

On 2014-10-03 18:08:56 -0400, Bruce Momjian wrote:
> On Fri, Oct 3, 2014 at 11:58:14PM +0200, Andres Freund wrote:
> > On 2014-10-03 17:55:19 -0400, Tom Lane wrote:
> > > Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> > > > On 2014-10-03 12:40:21 -0400, Bruce Momjian wrote:
> > > >> Well, I think the issue is that having a GUC that can't reasonably be
> > > >> tuned by 95% of our users is nearly useless. Few users are going to run
> > > >> benchmarks to see what the optimal value is.
> > >
> > > > It's possible to convince customers to play with a performance
> > > > influencing parameter and see how the results are. Even in
> > > > production.
> > >
> > > I'm a bit dubious that people will be willing to experiment in production
> > > with a GUC that requires a database restart to change.
> >
> > I've convinced customers to restart databases with several different
> > shared_buffers settings... So I'm pretty sure it's possible, in *some*
> > cases, for xloginsert_slots.
> >
> > And even if it's just test/pre-production machines - they're not going
> > to benchmark settings they can't reasonably set in production.
>
> Do we really want to expose a setting a few of us _might_ ask customers
> to change?

They also will try that themselves. Our customers aren't a horde of dumb
people. Some of them are willing to try things if they hit scalability
problesm. And *lots* of people hit scalability problems with
postgres. In fact I've seen big users migrate away from postgres because
of them.

And it's not like this only affects absurd cases. Even a parallel
restore will benefit.

> I don't think so --- create a custom build if you want that.

Won't work in the majority of real world cases.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Sat, Oct 4, 2014 at 12:13:00AM +0200, Andres Freund wrote:
> > Do we really want to expose a setting a few of us _might_ ask customers
> > to change?
>
> They also will try that themselves. Our customers aren't a horde of dumb
> people. Some of them are willing to try things if they hit scalability
> problesm. And *lots* of people hit scalability problems with
> postgres. In fact I've seen big users migrate away from postgres because
> of them.
>
> And it's not like this only affects absurd cases. Even a parallel
> restore will benefit.

I disagree. I just don't see the value in having such undefined
variables.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +

On 2014-10-03 18:16:28 -0400, Bruce Momjian wrote:
> On Sat, Oct 4, 2014 at 12:13:00AM +0200, Andres Freund wrote:
> > > Do we really want to expose a setting a few of us _might_ ask customers
> > > to change?
> >
> > They also will try that themselves. Our customers aren't a horde of dumb
> > people. Some of them are willing to try things if they hit scalability
> > problesm. And *lots* of people hit scalability problems with
> > postgres. In fact I've seen big users migrate away from postgres because
> > of them.
> >
> > And it's not like this only affects absurd cases. Even a parallel
> > restore will benefit.
>
> I disagree. I just don't see the value in having such undefined
> variables.

"undefined variables"? I'm not arguing that we don't need documentation
for it. Obviously we'd need that. I'm arguing against taking away
significant scalability possibilities from our users. My bet is that
it's more than 50% on a bigger machine.

I don't think we can offer absolutely accurate tuning advice, but I'm
sure we can give some guidance. Let me try.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 04/10/14 11:21, Andres Freund wrote:
> On 2014-10-03 18:16:28 -0400, Bruce Momjian wrote:
>> On Sat, Oct 4, 2014 at 12:13:00AM +0200, Andres Freund wrote:
>>>> Do we really want to expose a setting a few of us _might_ ask customers
>>>> to change?
>>>
>>> They also will try that themselves. Our customers aren't a horde of dumb
>>> people. Some of them are willing to try things if they hit scalability
>>> problesm. And *lots* of people hit scalability problems with
>>> postgres. In fact I've seen big users migrate away from postgres because
>>> of them.
>>>
>>> And it's not like this only affects absurd cases. Even a parallel
>>> restore will benefit.
>>
>> I disagree. I just don't see the value in having such undefined
>> variables.
>
> "undefined variables"? I'm not arguing that we don't need documentation
> for it. Obviously we'd need that. I'm arguing against taking away
> significant scalability possibilities from our users. My bet is that
> it's more than 50% on a bigger machine.
>
> I don't think we can offer absolutely accurate tuning advice, but I'm
> sure we can give some guidance. Let me try.
>

+1

I think it is ok to document our reason for providing the new GUC -
along with that fact that it is a new one and we need more field testing
and benchmarks to provide comprehensive advice about how to set - and
recommend leaving it alone unless consult with experts/this list etc.

Regards

Mark

On Sat, Oct 4, 2014 at 12:00:36PM +1300, Mark Kirkwood wrote:
> >I don't think we can offer absolutely accurate tuning advice, but I'm
> >sure we can give some guidance. Let me try.
> >
>
> +1
>
> I think it is ok to document our reason for providing the new GUC -
> along with that fact that it is a new one and we need more field
> testing and benchmarks to provide comprehensive advice about how to
> set - and recommend leaving it alone unless consult with
> experts/this list etc.

I predict that such a setting will remain in postgresql.conf for years
with almost zero activity, as have other similar efforts.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +

On 10/3/14, 10:11 AM, Andres Freund wrote:
> So 25% performance on a relatively small machine improvements aren't
> worth a GUC? That are likely to be larger on a bigger machine? I
> utterly fail to see why that's a service to our users.

I didn't say that. I said I don't think they're worth a GUC today if it
can be quietly and automatically slipped into the next release--and that
seems quite feasible. I have introduced GUCs that almost no one can
tune properly into the system before. Can't say I was pleased with how
that played out.

Another thing I don't know yet, and this is going to take me a while to
characterize, is how that 25% gain on a benchmark that is specifically
designed to highlight the problem use case impacts the various mixes of
more average cases I try as well. Is it 0.1% for a typical pgbench
workload? Now that GUC isn't so exciting anymore either.

And there's one more big issue I'd prefer not to discover from
real-world complaints: is there any downside to making this number very
large on a system where it shouldn't be?

The history of settings like this says that providing an exposed knob
will result in some people tinkering it with and making the system
slower. The gain of going from 1 to 8 is so clear and simple that I'm
not worried too hard about performance regressions like that. But if we
make it a GUC and it can be set to 100 to extract maximum performance on
big machines...I'd take a bet that we'll find people setting it to 100
saying "more must be better" where it isn't. Every time someone wanders
into pgsql-performance with a complaint, each one of these obscure GUCs
they tweaked magnifies the troubleshooting mess a little.

I do not disagree with you fundamentally here: this *is* worth refining
further, for sure, and the gains might be even greater if that keeps
going. My guess is just that the Postgres community would not get a net
benefit chasing that as a GUC in 9.4, not by the time you try to account
for all the future overhead and risk that adds to the release. That was
Heikki's gut feel on this when he yanked it out already; I was mainly
trying to do sanity checking on that. You've made a good case that
wasn't the ideal answer. Even with that new data, I still don't think
it was a outright bad decision though--especially not in an October
where there's no new version out yet. This thread spun out of Open
Items, and cutting complexity should be the preferred direction for
everything left on there now.

--
Greg Smith greg(dot)smith(at)crunchydatasolutions(dot)com
Chief PostgreSQL Evangelist - http://crunchydatasolutions.com/

On Fri, Oct 3, 2014 at 07:39:25PM -0400, Greg Smith wrote:
> I do not disagree with you fundamentally here: this *is* worth
> refining further, for sure, and the gains might be even greater if
> that keeps going. My guess is just that the Postgres community
> would not get a net benefit chasing that as a GUC in 9.4, not by the
> time you try to account for all the future overhead and risk that
> adds to the release. That was Heikki's gut feel on this when he
> yanked it out already; I was mainly trying to do sanity checking on
> that. You've made a good case that wasn't the ideal answer. Even
> with that new data, I still don't think it was a outright bad
> decision though--especially not in an October where there's no new
> version out yet. This thread spun out of Open Items, and cutting
> complexity should be the preferred direction for everything left on
> there now.

Agreeed. Also, reality check --- we can't change postgresql.conf easily
without an initdb, and I think our last 9.4 initdb is going to be
9.4beta3, which is going to be packaged on Monday.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +

Bruce Momjian <bruce(at)momjian(dot)us> writes:
> Agreeed. Also, reality check --- we can't change postgresql.conf easily
> without an initdb, and I think our last 9.4 initdb is going to be
> 9.4beta3, which is going to be packaged on Monday.

Good point: independently of all else, it's a bit late to be adding new
features to 9.4.

regards, tom lane

On 2014-10-03 19:54:36 -0400, Tom Lane wrote:
> Bruce Momjian <bruce(at)momjian(dot)us> writes:
> > Agreeed. Also, reality check --- we can't change postgresql.conf easily
> > without an initdb, and I think our last 9.4 initdb is going to be
> > 9.4beta3, which is going to be packaged on Monday.
>
> Good point: independently of all else, it's a bit late to be adding new
> features to 9.4.

This is getting absurd. The feature was there three days ago.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Sat, Oct 4, 2014 at 01:57:01AM +0200, Andres Freund wrote:
> On 2014-10-03 19:54:36 -0400, Tom Lane wrote:
> > Bruce Momjian <bruce(at)momjian(dot)us> writes:
> > > Agreeed. Also, reality check --- we can't change postgresql.conf easily
> > > without an initdb, and I think our last 9.4 initdb is going to be
> > > 9.4beta3, which is going to be packaged on Monday.
> >
> > Good point: independently of all else, it's a bit late to be adding new
> > features to 9.4.
>
> This is getting absurd. The feature was there three days ago.

My point is we have to have in there/decide by Monday or it isn't going
to be in 9.4. I am unclear how we can resolve all the concerns about
its addition in that short of a time.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +

On 10/03/2014 05:04 PM, Bruce Momjian wrote:
> On Sat, Oct 4, 2014 at 01:57:01AM +0200, Andres Freund wrote:
>> On 2014-10-03 19:54:36 -0400, Tom Lane wrote:
>>> Bruce Momjian <bruce(at)momjian(dot)us> writes:
>>>> Agreeed. Also, reality check --- we can't change postgresql.conf easily
>>>> without an initdb, and I think our last 9.4 initdb is going to be
>>>> 9.4beta3, which is going to be packaged on Monday.
>>>
>>> Good point: independently of all else, it's a bit late to be adding new
>>> features to 9.4.
>>
>> This is getting absurd. The feature was there three days ago.
>
> My point is we have to have in there/decide by Monday or it isn't going
> to be in 9.4. I am unclear how we can resolve all the concerns about
> its addition in that short of a time.

Am I correct in that we still don't have any docs for this even if we
decided to accept the GUC? On that basis, I'd say accept the DEFINE and
punt the GUC to the next 9.5 Commitfest ... assuming someone wants to
write docs.

Heck, I might write them after some testing. But that testing won't
happen in time for 9.4 release. I certainly couldn't write the docs now.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> On 2014-10-03 19:54:36 -0400, Tom Lane wrote:
>> Good point: independently of all else, it's a bit late to be adding new
>> features to 9.4.

> This is getting absurd. The feature was there three days ago.

Well, an undocumented feature isn't a feature.

regards, tom lane

On 10/3/14, 5:23 PM, Andres Freund wrote:
> How are we ever going to be able to tune it further without feedback
> from actual production usage?

With improved targeted synthetic test cases that isolate the bottleneck
to where it's really obvious, way more obvious than it will ever be in
production? I just watched you do a very successful round of chewing
right through this problem that way. And haven't the synthetic tests
already been more useful to guide development than feedback from
production for some time?

An alternate way to state one of your questions along this angle is "how
can we tell if the fixed limit of 8 is still a bottleneck on a 9.4
server unless there's a GUC to increase past that"? In my first message
here I was trying to highlight that we have little idea what that world
looks like yet. This change is already so beneficial and large, the
hotspots on systems impacted by it are probably going to move to
somewhere *very* different than earlier versions.

And when that happens, it's typically not revisiting the thing you just
made way, way faster than is still the problem anymore. If it turns out
a large bottleneck in real-world 9.4 deployments is *still*
xloginsert_locks, and there's no GUC for tuning it beyond 8, then the
people in the support trenches are going to see removing that GUC as a
terrible error. That might happen. I'm not trying to criticize your
work here, but you haven't actually made the case yet this is
likely--you cheated a little with the I/O part to get around where the
bottleneck shifts once you have a decent number of slots (8). That
tweak was part of why you got a larger gain than I did.

So that's a bad path everyone might see in the future, and if we end up
there all of us in the support trenches will suffer for having done the
wrong thing. I get what you're saying there, and I'll owe you an
apology if it plays out that way. In every other potential future I can
imagine, and in every future I consider probable, eliminating the GUC
for now makes life easier. Everyone has already talked a bunch about
why extra GUCs have a support cost too. It's one less thing to
maintain, document, mess with, break in the field, talk about, and
suffer from unintended regressions.

Do you want to put the GUC right back again in the active branch to keep
progress on this easier to make, since it's really clear already there's
still gain to be chased here? I think you've already justified doing
that. I'm still running the commit with the GUC here. I will join you
among the annoyed that it's gone group the minute I do my next git pull.

--
Greg Smith greg(dot)smith(at)crunchydatasolutions(dot)com
Chief PostgreSQL Evangelist - http://crunchydatasolutions.com/

On 04/10/14 12:10, Bruce Momjian wrote:
> On Sat, Oct 4, 2014 at 12:00:36PM +1300, Mark Kirkwood wrote:
>>> I don't think we can offer absolutely accurate tuning advice, but I'm
>>> sure we can give some guidance. Let me try.
>>>
>>
>> +1
>>
>> I think it is ok to document our reason for providing the new GUC -
>> along with that fact that it is a new one and we need more field
>> testing and benchmarks to provide comprehensive advice about how to
>> set - and recommend leaving it alone unless consult with
>> experts/this list etc.
>
> I predict that such a setting will remain in postgresql.conf for years
> with almost zero activity, as have other similar efforts.
>

Sure that *may* happen. In fact in my experience the vast majority of
our current GUCs are never altered in the field - however when you run
into a situation where a certain GUC solves your performance issue, then
that seldom used GUC really gets some love.

So altho I get your point about endless proliferation of 'em not being
cost free, I'd like to plug the other side of the argument too - having
the flexibility to adjust your Postgres installation to work well with
<random platform with annoying quirks> is the corresponding benefit.

In addition with the increasing use of cloud platforms - the situation
above is likely to become *more* common (Postgres in Openstack using
Ceph for volume storage is a case in point).

Cheers

Mark

On Fri, Oct 3, 2014 at 08:04:08PM -0400, Bruce Momjian wrote:
> On Sat, Oct 4, 2014 at 01:57:01AM +0200, Andres Freund wrote:
> > On 2014-10-03 19:54:36 -0400, Tom Lane wrote:
> > > Bruce Momjian <bruce(at)momjian(dot)us> writes:
> > > > Agreeed. Also, reality check --- we can't change postgresql.conf easily
> > > > without an initdb, and I think our last 9.4 initdb is going to be
> > > > 9.4beta3, which is going to be packaged on Monday.
> > >
> > > Good point: independently of all else, it's a bit late to be adding new
> > > features to 9.4.
> >
> > This is getting absurd. The feature was there three days ago.
>
> My point is we have to have in there/decide by Monday or it isn't going
> to be in 9.4. I am unclear how we can resolve all the concerns about
> its addition in that short of a time.

Actually, another issue is that we were only able to remove
xloginsert_locks this week because we are forcing an initdb for
9.4beta3. If we had not, I am not sure how we would have removed this
undocumented setting, except to tell beta users to remove it from their
postgresql.conf file after we removed it from the server. This seems
sloppy --- we should have addressed this (documented it or removed it)
before beta1.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +

On 2014-10-04 11:07:13 -0400, Bruce Momjian wrote:
> On Fri, Oct 3, 2014 at 08:04:08PM -0400, Bruce Momjian wrote:
> > On Sat, Oct 4, 2014 at 01:57:01AM +0200, Andres Freund wrote:
> > > On 2014-10-03 19:54:36 -0400, Tom Lane wrote:
> > > > Bruce Momjian <bruce(at)momjian(dot)us> writes:
> > > > > Agreeed. Also, reality check --- we can't change postgresql.conf easily
> > > > > without an initdb, and I think our last 9.4 initdb is going to be
> > > > > 9.4beta3, which is going to be packaged on Monday.
> > > >
> > > > Good point: independently of all else, it's a bit late to be adding new
> > > > features to 9.4.
> > >
> > > This is getting absurd. The feature was there three days ago.
> >
> > My point is we have to have in there/decide by Monday or it isn't going
> > to be in 9.4. I am unclear how we can resolve all the concerns about
> > its addition in that short of a time.
>
> Actually, another issue is that we were only able to remove
> xloginsert_locks this week because we are forcing an initdb for
> 9.4beta3. If we had not, I am not sure how we would have removed this
> undocumented setting, except to tell beta users to remove it from their
> postgresql.conf file after we removed it from the server. This seems
> sloppy --- we should have addressed this (documented it or removed it)
> before beta1.

It was marked as GUC_NOT_IN_SAMPLE.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Sat, Oct 4, 2014 at 05:08:17PM +0200, Andres Freund wrote:
> On 2014-10-04 11:07:13 -0400, Bruce Momjian wrote:
> > On Fri, Oct 3, 2014 at 08:04:08PM -0400, Bruce Momjian wrote:
> > > On Sat, Oct 4, 2014 at 01:57:01AM +0200, Andres Freund wrote:
> > > > On 2014-10-03 19:54:36 -0400, Tom Lane wrote:
> > > > > Bruce Momjian <bruce(at)momjian(dot)us> writes:
> > > > > > Agreeed. Also, reality check --- we can't change postgresql.conf easily
> > > > > > without an initdb, and I think our last 9.4 initdb is going to be
> > > > > > 9.4beta3, which is going to be packaged on Monday.
> > > > >
> > > > > Good point: independently of all else, it's a bit late to be adding new
> > > > > features to 9.4.
> > > >
> > > > This is getting absurd. The feature was there three days ago.
> > >
> > > My point is we have to have in there/decide by Monday or it isn't going
> > > to be in 9.4. I am unclear how we can resolve all the concerns about
> > > its addition in that short of a time.
> >
> > Actually, another issue is that we were only able to remove
> > xloginsert_locks this week because we are forcing an initdb for
> > 9.4beta3. If we had not, I am not sure how we would have removed this
> > undocumented setting, except to tell beta users to remove it from their
> > postgresql.conf file after we removed it from the server. This seems
> > sloppy --- we should have addressed this (documented it or removed it)
> > before beta1.
>
> It was marked as GUC_NOT_IN_SAMPLE.

Oh, that's interesting. Thank you.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +