Re: What do these terms mean in the SOURCE CODE?

I am going through the Executor code and come across the following terms
quite often. Can someone tell me what do they mean (in a few (may be a
couple of) sentences)?

1. Scan State
2. Plan State
3. Tuple Projection
4. EState
5. Qual
6. Expression

They sound quite ambiguous in the source code, specially when some of them
already have terms which have multiple meanings.

Thanks for your time.

-Vaibhav (*_*)

Is no one ready to help on this? :(

-Vaibhav
---------- Forwarded message ----------
From: Vaibhav Kaushal <vaibhavkaushal123(at)gmail(dot)com>
Date: Fri, Nov 19, 2010 at 9:11 PM
Subject: What do these terms mean in the SOURCE CODE?
To: pgsql-hackers(at)postgresql(dot)org

I am going through the Executor code and come across the following terms
quite often. Can someone tell me what do they mean (in a few (may be a
couple of) sentences)?

1. Scan State
2. Plan State
3. Tuple Projection
4. EState
5. Qual
6. Expression

They sound quite ambiguous in the source code, specially when some of them
already have terms which have multiple meanings.

Thanks for your time.

-Vaibhav (*_*)

On Fri, Nov 19, 2010 at 10:51 PM, Vaibhav Kaushal
<vaibhavkaushal123(at)gmail(dot)com> wrote:
> Is no one ready to help on this? :(

Apparently, no one is ready to drop what they are doing to immediately
answer your email before all the other ones they need to answer, but I
wouldn't infer from that that no one wants to help. It doesn't seem
very realistic to me to expect a 12-hour turnaround on free support,
but what do I know?

With regards to your question, for each type of plan node, there is an
associated "plan state" node. This is an important distinction
because, IIUC, plans can be reused, so plan state contains the
information that might need to be reset on each run. Scan state is
just a plan state node for a scan node. I believe a tuple projection
is what you get when you do something like SELECT
generate_series(1,10) FROM tbl - the set-returning function has to be
"projected" in multiple tuples. EState I could use some hints on
myself. A qual is a filter condition, e.g. in SELECT * FROM tbl WHERE
x = 1, the "x = 1" part is a qual. It's helpful to grep src/include
for the structures in question; the information they contain often
helps to understand their purpose.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Thanks a lot for the answer. I see a lot of people discussing so many things
so I thought my email would have been ignored by those with a lot coming in
already. Thanks for the enlightenment.

-Vaibhav (*_*)

On Sun, Nov 21, 2010 at 12:25 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:

> On Fri, Nov 19, 2010 at 10:51 PM, Vaibhav Kaushal
> <vaibhavkaushal123(at)gmail(dot)com> wrote:
> > Is no one ready to help on this? :(
>
> Apparently, no one is ready to drop what they are doing to immediately
> answer your email before all the other ones they need to answer, but I
> wouldn't infer from that that no one wants to help. It doesn't seem
> very realistic to me to expect a 12-hour turnaround on free support,
> but what do I know?
>
> With regards to your question, for each type of plan node, there is an
> associated "plan state" node. This is an important distinction
> because, IIUC, plans can be reused, so plan state contains the
> information that might need to be reset on each run. Scan state is
> just a plan state node for a scan node. I believe a tuple projection
> is what you get when you do something like SELECT
> generate_series(1,10) FROM tbl - the set-returning function has to be
> "projected" in multiple tuples. EState I could use some hints on
> myself. A qual is a filter condition, e.g. in SELECT * FROM tbl WHERE
> x = 1, the "x = 1" part is a qual. It's helpful to grep src/include
> for the structures in question; the information they contain often
> helps to understand their purpose.
>
> --
> Robert Haas
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>

On Sat, Nov 20, 2010 at 1:59 PM, Vaibhav Kaushal
<vaibhavkaushal123(at)gmail(dot)com> wrote:
> Thanks a lot for the answer. I see a lot of people discussing so many things
> so I thought my email would have been ignored by those with a lot coming in
> already. Thanks for the enlightenment.

Sure. Just FYI, with these kinds of things, I do try to keep track of
them because I like to try to make sure that everyone gets an answer
back. But, I don't always have time to do it immediately, and
sometimes I wait even if I do, if I think that someone more
knowledgeable on that particular topic might chime in.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

I sure do respect that. Appreciated. :)

-Vaibhav (*_*)

On Sun, Nov 21, 2010 at 12:33 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:

> On Sat, Nov 20, 2010 at 1:59 PM, Vaibhav Kaushal
> <vaibhavkaushal123(at)gmail(dot)com> wrote:
> > Thanks a lot for the answer. I see a lot of people discussing so many
> things
> > so I thought my email would have been ignored by those with a lot coming
> in
> > already. Thanks for the enlightenment.
>
> Sure. Just FYI, with these kinds of things, I do try to keep track of
> them because I like to try to make sure that everyone gets an answer
> back. But, I don't always have time to do it immediately, and
> sometimes I wait even if I do, if I think that someone more
> knowledgeable on that particular topic might chime in.
>
> --
> Robert Haas
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>

Some additional comments ...

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> With regards to your question, for each type of plan node, there is an
> associated "plan state" node. This is an important distinction
> because, IIUC, plans can be reused, so plan state contains the
> information that might need to be reset on each run.

Yeah. The plan tree is supposed to be read-only so far as the executor
is concerned, so it needs a parallel "plan state" tree to hold its
runtime state.

> I believe a tuple projection
> is what you get when you do something like SELECT
> generate_series(1,10) FROM tbl - the set-returning function has to be
> "projected" in multiple tuples.

I think "projection" is a term out of relational theory, referring to
any sort of tuple-by-tuple manipulation of the data. For instance,
if you have a row "a, b, c" and you compute "a, c+1" based on that,
that's a projection. ExecProject() does this in the general case.
The business with SRFs in a targetlist possibly producing multiple
rows is a PostQUEL-ism that doesn't have any standard technical name
that I know of.

> A qual is a filter condition, e.g. in SELECT * FROM tbl WHERE
> x = 1, the "x = 1" part is a qual.

qual = qualifier. We use that term with various shades of meaning;
notably an indexqual is a qualifier that is useful for searching an
index. Large parts of the planner also use the term "clause" to mean
about the same thing.

> It's helpful to grep src/include
> for the structures in question;

Here's my single biggest tip for newcomers to the Postgres source:
if you don't use ctags, glimpse, or some other tool that can quickly
show you all references to a given identifier, go out and get one.
It's one of the easiest ways to learn about things.

regards, tom lane

Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:
> Here's my single biggest tip for newcomers to the Postgres source:
> if you don't use ctags, glimpse, or some other tool that can quickly
> show you all references to a given identifier, go out and get one.
> It's one of the easiest ways to learn about things.

I've been using cscope (out of advice from Joshua Tolley), and even
better its integration into Emacs which is called xcscope.el --- I
wouldn't have been able to come up with the extension patch series
without that.

http://cscope.sourceforge.net/

And as a quick teaser, the keys I mostly use:

C-c s d cscope-find-global-definition
C-c s s cscope-find-this-symbol
C-c s f cscope-find-this-file
C-c s I cscope-index-files

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

Thanks for your concerns about my question. I am really happy to get the
answers.

@Mr. Tom Lane: Thanks for your explanation but I am already using Eclipse
which gets me to the definitions real easy. I do not post my questions on
the Hackers forum with any intention to disturb anyone. I just ask for help
when I am not able to understand it all. PG is my first attempt at OSS
development so it is particularly difficult. Of course PG has great support
for us because almost everything has got its comment, elaborating its use /
importance. However since I don't know everything yet about PG or even about
the OSS development, I come here asking. I have never been in so much of
code with so many Data Structures and typedefs. It gets confusing very
easily. This is the reason I came here asking.

I ask here after I have done my homework; and after I get messed up with the
code. Moreover the comments are those of developers. So when someone says
that "expression does not match qual" in the code, it is not so clear to me
as it might be to you people; because you know the terms, I don't. Thanks
for all the support I have been getting here to all of you. I will try to
ask as less as possible from now on.

@Mr. Dimitri Fontaine: Thanks for telling me the name of the tool. But I
like the GUI interfaces more and Eclipse has been serving me well for
browsing (only) the source code. I do not fear command line / shell but
prefer GUI more. Thanks for your answer.

-Vaibhav (*_*)

On Sun, Nov 21, 2010 at 2:49 AM, Dimitri Fontaine <dimitri(at)2ndquadrant(dot)fr>wrote:

> Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:
> > Here's my single biggest tip for newcomers to the Postgres source:
> > if you don't use ctags, glimpse, or some other tool that can quickly
> > show you all references to a given identifier, go out and get one.
> > It's one of the easiest ways to learn about things.
>
> I've been using cscope (out of advice from Joshua Tolley), and even
> better its integration into Emacs which is called xcscope.el --- I
> wouldn't have been able to come up with the extension patch series
> without that.
>
> http://cscope.sourceforge.net/
>
> And as a quick teaser, the keys I mostly use:
>
> C-c s d cscope-find-global-definition
> C-c s s cscope-find-this-symbol
> C-c s f cscope-find-this-file
> C-c s I cscope-index-files
>
> Regards,
> --
> Dimitri Fontaine
> http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
>

> Here's my single biggest tip for newcomers to the Postgres source:
> if you don't use ctags, glimpse, or some other tool that can quickly
> show you all references to a given identifier, go out and get one.
> It's one of the easiest ways to learn about things.

I guess Eclipse is the best tool for the "newcomer" as it gives
you all the references very easily and shows the functions/structs
declaration when you pass with the mouse over one...

http://wiki.postgresql.org/wiki/Working_with_Eclipse

Thanks for that informative article. :) Taught me a few new points about
Eclipse :)

On Sun, Nov 21, 2010 at 2:29 PM, Leonardo Francalanci <m_lists(at)yahoo(dot)it>wrote:

> > Here's my single biggest tip for newcomers to the Postgres source:
> > if you don't use ctags, glimpse, or some other tool that can quickly
> > show you all references to a given identifier, go out and get one.
> > It's one of the easiest ways to learn about things.
>
>
> I guess Eclipse is the best tool for the "newcomer" as it gives
> you all the references very easily and shows the functions/structs
> declaration when you pass with the mouse over one...
>
> http://wiki.postgresql.org/wiki/Working_with_Eclipse
>
>
>
>

Vaibhav Kaushal wrote:
> I am going through the Executor code and come across the following terms
> quite often. Can someone tell me what do they mean (in a few (may be a
> couple of) sentences)?
>
> 1. Scan State
> 2. Plan State
> 3. Tuple Projection
> 4. EState
> 5. Qual
> 6. Expression
>
> They sound quite ambiguous in the source code, specially when some of them
> already have terms which have multiple meanings.

Have you read src/backend/executor/README? It is hard to explain what
these mean without understanding the executor.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +