Re: further explain changes

Per recent discussion on pgsql-performance, and per discussion on
-hackers that it might not be too late for small patches after all,
here is a patch (as yet without documentation) which adds some
additional instrumentation to EXPLAIN for hashes: number of buckets,
number of batches, original number of batches, and peak memory
utilization. Thoughts?

I was also thinking about the possibility of adding a new option
called "output" and making that control whether the "Output" line gets
printed. It's kind of annoying to use EXPLAIN (ANALYZE, VERBOSE)
right now (and moreso with this patch) specifically because of that
line, which is quite... verbose. If we're going to change it ever we
should do it for 9.0, since we've made a lot of other changes that
people will be adjusting for anyhow.

Also, do we want to change the schema URL? The existing URL was
suggested by Peter but IIRC there was some thought that it might not
be the best choice.

http://www.postgresql.org/2009/explain

...Robert

On Sat, Jan 23, 2010 at 10:08 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>
> I was also thinking about the possibility of adding a new option
> called "output" and making that control whether the "Output" line gets
> printed.  It's kind of annoying to use EXPLAIN (ANALYZE, VERBOSE)

why not let it go in ANALYZE, just as the sort info

--
Atentamente,
Jaime Casanova
Soporte y capacitación de PostgreSQL
Asesoría y desarrollo de sistemas
Guayaquil - Ecuador
Cel. +59387171157

Le 24/01/2010 06:06, Jaime Casanova a écrit :
> On Sat, Jan 23, 2010 at 10:08 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>>
>> I was also thinking about the possibility of adding a new option
>> called "output" and making that control whether the "Output" line gets
>> printed. It's kind of annoying to use EXPLAIN (ANALYZE, VERBOSE)
>
> why not let it go in ANALYZE, just as the sort info
>

Yes, it would be more consistent. Other than that, this patch is quite
interesting.

--
Guillaume.
http://www.postgresqlfr.org
http://dalibo.com

isn't that line pretty much the main point of "verbose"? I would assume
anything new like this would get its own option which might default to on.

greg

On 24 Jan 2010 03:08, "Robert Haas" <robertmhaas(at)gmail(dot)com> wrote:

Per recent discussion on pgsql-performance, and per discussion on
-hackers that it might not be too late for small patches after all,
here is a patch (as yet without documentation) which adds some
additional instrumentation to EXPLAIN for hashes: number of buckets,
number of batches, original number of batches, and peak memory
utilization. Thoughts?

I was also thinking about the possibility of adding a new option
called "output" and making that control whether the "Output" line gets
printed. It's kind of annoying to use EXPLAIN (ANALYZE, VERBOSE)
right now (and moreso with this patch) specifically because of that
line, which is quite... verbose. If we're going to change it ever we
should do it for 9.0, since we've made a lot of other changes that
people will be adjusting for anyhow.

Also, do we want to change the schema URL? The existing URL was
suggested by Peter but IIRC there was some thought that it might not
be the best choice.

http://www.postgresql.org/2009/explain

...Robert

--
Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Sun, Jan 24, 2010 at 12:06 AM, Jaime Casanova
<jcasanov(at)systemguards(dot)com(dot)ec> wrote:
> On Sat, Jan 23, 2010 at 10:08 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>>
>> I was also thinking about the possibility of adding a new option
>> called "output" and making that control whether the "Output" line gets
>> printed.  It's kind of annoying to use EXPLAIN (ANALYZE, VERBOSE)
>
> why not let it go in ANALYZE, just as the sort info

It's kinda long-winded - it adds like 4 extra lines for each hash
join. I don't think I want to add that much clutter to regular E-A
output.

If people don't like making it controlled by VERBOSE, then I vote for
a new option, as Greg Stark suggested.

...Robert

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> On Sun, Jan 24, 2010 at 12:06 AM, Jaime Casanova
>> why not let it go in ANALYZE, just as the sort info

> It's kinda long-winded - it adds like 4 extra lines for each hash
> join. I don't think I want to add that much clutter to regular E-A
> output.

Well, that would only happen if you're deliberately obtuse about the
formatting. The sort code manages to fit all the extra on one line,
and I don't see why hash couldn't.

I'd vote for just adding it in the exact same cases that sort adds extra
info. -1 for either adding a new option or changing the meaning of the
ones that are there.

regards, tom lane

On Sun, Jan 24, 2010 at 12:30 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> On Sun, Jan 24, 2010 at 12:06 AM, Jaime Casanova
>>> why not let it go in ANALYZE, just as the sort info
>
>> It's kinda long-winded - it adds like 4 extra lines for each hash
>> join.  I don't think I want to add that much clutter to regular E-A
>> output.
>
> Well, that would only happen if you're deliberately obtuse about the
> formatting.  The sort code manages to fit all the extra on one line,
> and I don't see why hash couldn't.
>
> I'd vote for just adding it in the exact same cases that sort adds extra
> info.  -1 for either adding a new option or changing the meaning of the
> ones that are there.

Care to suggest a format?

...Robert

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> Care to suggest a format?

As much like the sort case as possible ...

regards, tom lane

On Sun, Jan 24, 2010 at 12:30 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> On Sun, Jan 24, 2010 at 12:06 AM, Jaime Casanova
>>> why not let it go in ANALYZE, just as the sort info
>
>> It's kinda long-winded - it adds like 4 extra lines for each hash
>> join.  I don't think I want to add that much clutter to regular E-A
>> output.
>
> Well, that would only happen if you're deliberately obtuse about the
> formatting.  The sort code manages to fit all the extra on one line,
> and I don't see why hash couldn't.

OK, here's a new version.

...Robert

On Thu, Jan 28, 2010 at 4:18 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Sun, Jan 24, 2010 at 12:30 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>>> On Sun, Jan 24, 2010 at 12:06 AM, Jaime Casanova
>>>> why not let it go in ANALYZE, just as the sort info
>>
>>> It's kinda long-winded - it adds like 4 extra lines for each hash
>>> join.  I don't think I want to add that much clutter to regular E-A
>>> output.
>>
>> Well, that would only happen if you're deliberately obtuse about the
>> formatting.  The sort code manages to fit all the extra on one line,
>> and I don't see why hash couldn't.
>
> OK, here's a new version.
>

OK, i have 3 hashes in a query and i got these 3 lines in an EXPLAIN
ANALYZE with your patch
"""
-> Hash (cost=3878.94..3878.94 rows=83594
width=34) (actual time=504.648..504.648 rows=83594 loops=1)
Buckets: 2048 Batches: 8 Memory Usage: 589kB
[...]
-> Hash (cost=14.49..14.49 rows=649 width=15)
(actual time=2.901..2.901 rows=649 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 32kB
[...]
-> Hash (cost=977.62..977.62 rows=6555 width=16)
(actual time=60.913..60.913 rows=6556 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 308kB
"""

I guess Memory Usage is per Batch, right? is this an average?
What is the unit measure for Bucket?
there are 14 temp files generated for this query and the only Sort
says it was on memory so these files should come from the hashes, can
we say that in the explain analyze? mmm... that memory usage, could be
Disk actually?

--
Atentamente,
Jaime Casanova
Soporte y capacitación de PostgreSQL
Asesoría y desarrollo de sistemas
Guayaquil - Ecuador
Cel. +59387171157

On Sat, Jan 30, 2010 at 12:26 PM, Jaime Casanova
<jcasanov(at)systemguards(dot)com(dot)ec> wrote:
> On Thu, Jan 28, 2010 at 4:18 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> On Sun, Jan 24, 2010 at 12:30 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>>>> On Sun, Jan 24, 2010 at 12:06 AM, Jaime Casanova
>>>>> why not let it go in ANALYZE, just as the sort info
>>>
>>>> It's kinda long-winded - it adds like 4 extra lines for each hash
>>>> join.  I don't think I want to add that much clutter to regular E-A
>>>> output.
>>>
>>> Well, that would only happen if you're deliberately obtuse about the
>>> formatting.  The sort code manages to fit all the extra on one line,
>>> and I don't see why hash couldn't.
>>
>> OK, here's a new version.
>>
>
> OK, i have 3 hashes in a query and i got these 3 lines in an EXPLAIN
> ANALYZE with your patch
> """
>                        ->  Hash  (cost=3878.94..3878.94 rows=83594
> width=34) (actual time=504.648..504.648 rows=83594 loops=1)
>                                 Buckets: 2048  Batches: 8  Memory Usage: 589kB
> [...]
>                     ->  Hash  (cost=14.49..14.49 rows=649 width=15)
> (actual time=2.901..2.901 rows=649 loops=1)
>                           Buckets: 1024  Batches: 1  Memory Usage: 32kB
> [...]
>               ->  Hash  (cost=977.62..977.62 rows=6555 width=16)
> (actual time=60.913..60.913 rows=6556 loops=1)
>                     Buckets: 1024  Batches: 1  Memory Usage: 308kB
> """
>
> I guess Memory Usage is per Batch, right? is this an average?

It's the maximum memory every used by that hash.

> What is the unit measure for Bucket?

There is no unit - it's just how many buckets.

> there are 14 temp files generated for this query and the only Sort
> says it was on memory so these files should come from the hashes, can
> we say that in the explain analyze?

I think maybe it's 2 temp files per batch in excess of 1, thus (8 - 1)
x 2 = 14 for the 8-batch join and none for the other two. Perhaps
you'd care to test that hypothesis?

> mmm... that memory usage, could be
> Disk actually?

No.

...Robert