Re: FDW API / flow charts for the docs?

Hi,

I've been experimenting with the new reworked FDW API to get familiar
with it. The postgres_fdw is a great source of knowledge (huge thanks to
Shigeru Hanada, KaiGai Kohei and everyone else who made this happen),
but in the end I had to draw some flow charts in Dia, to understand how
exactly do the functions interact, pass private data etc. in various
scenarios.

Attached is the set of flow charts, showing the sequence of callbacks
for all the supported commands (i.e. SELECT, INSERT, UPDATE, DELETE,
ANALYZE). Wouldn't it be useful to put something like this into the
docs? I mean, the FDW API is not going to get any simpler, and for me
this was a significant help.

regards
Tomas

On Wed, Oct 16, 2013 at 8:35 PM, Tomas Vondra <tv(at)fuzzy(dot)cz> wrote:
>
> [...]
>
> Attached is the set of flow charts, showing the sequence of callbacks
> for all the supported commands (i.e. SELECT, INSERT, UPDATE, DELETE,
> ANALYZE).

Thank you very much... this flow charts will help many people, including me
;-)

> Wouldn't it be useful to put something like this into the
> docs? I mean, the FDW API is not going to get any simpler, and for me
> this was a significant help.
>

+1 to add into docs.

I think we can add this flow charts to [1].

Regards,

[1] http://www.postgresql.org/docs/9.3/interactive/fdwhandler.html

--
Fabrízio de Royes Mello
Consultoria/Coaching PostgreSQL
>> Timbira: http://www.timbira.com.br
>> Blog sobre TI: http://fabriziomello.blogspot.com
>> Perfil Linkedin: http://br.linkedin.com/in/fabriziomello
>> Twitter: http://twitter.com/fabriziomello

Tomas Vondra wrote:

> Attached is the set of flow charts, showing the sequence of callbacks
> for all the supported commands (i.e. SELECT, INSERT, UPDATE, DELETE,
> ANALYZE). Wouldn't it be useful to put something like this into the
> docs? I mean, the FDW API is not going to get any simpler, and for me
> this was a significant help.

Please see this thread
http://www.postgresql.org/message-id/4BB9E69F.9080203@usit.uio.no

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Alvaro,

* Alvaro Herrera (alvherre(at)2ndquadrant(dot)com) wrote:
> Tomas Vondra wrote:
> > Attached is the set of flow charts, showing the sequence of callbacks
> > for all the supported commands (i.e. SELECT, INSERT, UPDATE, DELETE,
> > ANALYZE). Wouldn't it be useful to put something like this into the
> > docs? I mean, the FDW API is not going to get any simpler, and for me
> > this was a significant help.
>
> Please see this thread
> http://www.postgresql.org/message-id/4BB9E69F.9080203@usit.uio.no

The conclusion of that thread appears to be "use dia", which was done
here.. Am I missing something there (full disclosure- I haven't looked
at the dia yet)?

Also, for my part, I'd suggest putting it on the wiki initially anyway,
as then it can be seen directly (load it as a png or what-have-you) and
it becomes immediately available to users. The .dia should also be on
the wiki, of course, and then included in the PG tree eventually if it's
added as part of the official docs.

Thanks,

Stephen

On 17 Říjen 2013, 5:32, Stephen Frost wrote:
> Alvaro,
>
> * Alvaro Herrera (alvherre(at)2ndquadrant(dot)com) wrote:
>> Tomas Vondra wrote:
>> > Attached is the set of flow charts, showing the sequence of callbacks
>> > for all the supported commands (i.e. SELECT, INSERT, UPDATE, DELETE,
>> > ANALYZE). Wouldn't it be useful to put something like this into the
>> > docs? I mean, the FDW API is not going to get any simpler, and for me
>> > this was a significant help.
>>
>> Please see this thread
>> http://www.postgresql.org/message-id/4BB9E69F.9080203@usit.uio.no
>
> The conclusion of that thread appears to be "use dia", which was done
> here.. Am I missing something there (full disclosure- I haven't looked
> at the dia yet)?

My impression from that thread was that one of the requirements is
reasonable versioning / diff support, and AFAIK that's not a good match
for any GUI-based product. So while I like dia and I used it for drawing
the charts I submitted today, I don't think it works with this (quite
reasonable) requirement.

The only tool that might be a good match is graphviz (also mentioned in
that thread). It's text-based, widely available and quite customizable.
I'm not a graphviz expert, but attached is a result of 5-minute work with
graphviz. Not perfect, but I'm pretty sure we could get much better /
nicer results in very short time.

It's easy to edit by hand, do versioning and /or diff on that, etc. I
doubt there's a better option available.

> Also, for my part, I'd suggest putting it on the wiki initially anyway,
> as then it can be seen directly (load it as a png or what-have-you) and
> it becomes immediately available to users. The .dia should also be on
> the wiki, of course, and then included in the PG tree eventually if it's
> added as part of the official docs.

No problem with that, but I'd like to know in advance if we're willing to
put that into the docs / under what requirements etc. Otherwise it might
result in a major effort just to get it from wiki into docs later.

Tomas

Tomas,

* Tomas Vondra (tv(at)fuzzy(dot)cz) wrote:
> My impression from that thread was that one of the requirements is
> reasonable versioning / diff support, and AFAIK that's not a good match
> for any GUI-based product. So while I like dia and I used it for drawing
> the charts I submitted today, I don't think it works with this (quite
> reasonable) requirement.

I'm not sure why you feel that way wrt dia..? As was pointed out in the
thread, if you decompress the dia, it's pretty reasonable XML and diffs,
etc, will work reasonably well with it.

> > Also, for my part, I'd suggest putting it on the wiki initially anyway,
> > as then it can be seen directly (load it as a png or what-have-you) and
> > it becomes immediately available to users. The .dia should also be on
> > the wiki, of course, and then included in the PG tree eventually if it's
> > added as part of the official docs.
>
> No problem with that, but I'd like to know in advance if we're willing to
> put that into the docs / under what requirements etc. Otherwise it might
> result in a major effort just to get it from wiki into docs later.

I can't see it being a major effort to get it from the wiki into the
docs, though perhaps I'm being a bit over-optomistic wrt that. Still,
I'd much rather have it somewhere than not have it at all...

Thanks,

Stephen

On 10/18/13 5:35 PM, Stephen Frost wrote:
> I can't see it being a major effort to get it from the wiki into the
> docs, though perhaps I'm being a bit over-optomistic wrt that.

Hah!

Consider that an image would have to work with the following
toolchains/output formats:

- standalone HTML output
- web site HTML output
- reasonable mobile/small-screen experience
- HTML XSLT
- jadetex
- pdfjadetex
- FOP
- Texinfo (at least not break it)
- man (at least not break it)
- EPUB

In my mind, it's not worth the effort unless we have, say, at least a
dozen really useful images to add. I don't want to go through this
entire pain for one image on a pretty minor topic.

On 2013-10-18 18:46:10 -0400, Stephen Frost wrote:
> As to your point about not wanting to do it for a single image- it seems
> we could potentially say that for every individual image proposed, but
> if we don't keep track of those images anywhere then we may not realize
> that 5 or 10 have actually been done and proposed but never integrated.

> [...] Images really can explain things in a much better
> way in many cases.

+many

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

* Peter Eisentraut (peter_e(at)gmx(dot)net) wrote:
> In my mind, it's not worth the effort unless we have, say, at least a
> dozen really useful images to add. I don't want to go through this
> entire pain for one image on a pretty minor topic.

These are certainly fair issues- my point was merely that once we have
support for such in the regular docs, the actual migration from the wiki
into the docs would hopefully not be too difficult.

As to your point about not wanting to do it for a single image- it seems
we could potentially say that for every individual image proposed, but
if we don't keep track of those images anywhere then we may not realize
that 5 or 10 have actually been done and proposed but never integrated.
If they're kept on the wiki then perhaps we would both keep track of the
ones proposed and realize when it's worthwhile to add support for them
to the doc build system.

As to Tom's point on the previous thread, that we would need to actually
maintain these images, that is helped by using dia, imv, since it's a
pretty simple tool to use and understand and is available on many
platforms. There's still some risk there, of course, but it could be
worth it in the end. Images really can explain things in a much better
way in many cases.

Thanks,

Stephen

On 18.10.2013 23:35, Stephen Frost wrote:
> Tomas,
>
> * Tomas Vondra (tv(at)fuzzy(dot)cz) wrote:
>> My impression from that thread was that one of the requirements is
>> reasonable versioning / diff support, and AFAIK that's not a good
>> match for any GUI-based product. So while I like dia and I used it
>> for drawing the charts I submitted today, I don't think it works
>> with this (quite reasonable) requirement.
>
> I'm not sure why you feel that way wrt dia..? As was pointed out in
> the thread, if you decompress the dia, it's pretty reasonable XML and
> diffs, etc, will work reasonably well with it.
>
>>> Also, for my part, I'd suggest putting it on the wiki initially
>>> anyway, as then it can be seen directly (load it as a png or
>>> what-have-you) and it becomes immediately available to users.
>>> The .dia should also be on the wiki, of course, and then included
>>> in the PG tree eventually if it's added as part of the official
>>> docs.
>>
>> No problem with that, but I'd like to know in advance if we're
>> willing to put that into the docs / under what requirements etc.
>> Otherwise it might result in a major effort just to get it from
>> wiki into docs later.
>
> I can't see it being a major effort to get it from the wiki into the
> docs, though perhaps I'm being a bit over-optomistic wrt that.
> Still, I'd much rather have it somewhere than not have it at all...

I meant something a bit different. Imagine I start with a simple chart
but spend a lot of time improving it over time (adding comments,
formattting) only to find out later I need to redo that from scratch in
a different tool. Wouldn't that be easier to start with the target tool
in the first place?

Tomas

On 18.10.2013 23:52, Peter Eisentraut wrote:
> On 10/18/13 5:35 PM, Stephen Frost wrote:
>> I can't see it being a major effort to get it from the wiki into
>> the docs, though perhaps I'm being a bit over-optomistic wrt that.
>
> Hah!
>
> Consider that an image would have to work with the following
> toolchains/output formats:
>
> - standalone HTML output
> - web site HTML output
> - reasonable mobile/small-screen experience
> - HTML XSLT
> - jadetex
> - pdfjadetex
> - FOP
> - Texinfo (at least not break it)
> - man (at least not break it)
> - EPUB
>
> In my mind, it's not worth the effort unless we have, say, at least
> a dozen really useful images to add. I don't want to go through this
> entire pain for one image on a pretty minor topic.

AFAIK graphviz can give you at least .ps .svg .fig .png .gif .dia
formats (and some other). I believe that covers most (if not all) of the
cases you've mentioned.

I would be more inclined to let GraphViz into the process than Dia; the
former fits *much* better into a Make-based process.

It is worth observing that there are schema diagramming systems (SchemaSpy
is mighty likable) that build diagrams using GraphViz. We have integrated
this into internal deployments at Afilias.

Rod Taylor's autodoc tool has some minimal Dia support, but it lacks the
ability to do layout; if you use Dia output, you'll have to manually drag
things around to have the tables *not* all atop one another. I'm a little
surprised Dia hasn't integrated in GraphViz to help with that; note that
the MacOS diagram tool OmniGraffle (proprietary, considered quite good,
competitive with Visio) uses GraphViz to help do automatic object layout.
That this is integrated inside an interactive GUI app is pretty wild/cool.

I'd think it a fine idea to add rules for .dot files (the usual GraphViz
language/format to describe graphs) to the PG Make rules.

On Fri, 2013-10-18 at 18:46 -0400, Stephen Frost wrote:
> As to your point about not wanting to do it for a single image- it seems
> we could potentially say that for every individual image proposed, but
> if we don't keep track of those images anywhere then we may not realize
> that 5 or 10 have actually been done and proposed but never integrated.
> If they're kept on the wiki then perhaps we would both keep track of the
> ones proposed and realize when it's worthwhile to add support for them
> to the doc build system.

Right.

> As to Tom's point on the previous thread, that we would need to actually
> maintain these images, that is helped by using dia, imv, since it's a
> pretty simple tool to use and understand and is available on many
> platforms.

I'd also take a look at ditaa, which might make this even simpler.

On Sat, 2013-10-19 at 00:52 +0200, Tomas Vondra wrote:
> AFAIK graphviz can give you at least .ps .svg .fig .png .gif .dia
> formats (and some other). I believe that covers most (if not all) of
> the cases you've mentioned.

I'm not concerned about that. It's the code that would need to be
written to implement those cases.

On 10/19/2013 02:22 AM, Christopher Browne wrote:
> I would be more inclined to let GraphViz into the process than Dia;
> the former fits *much* better into a Make-based process.

I also cast my vote for Graphviz.

--
Vik

On Sat, Oct 19, 2013 at 01:56:23PM -0400, Peter Eisentraut wrote:
> On Fri, 2013-10-18 at 18:46 -0400, Stephen Frost wrote:
> > As to your point about not wanting to do it for a single image- it seems
> > we could potentially say that for every individual image proposed, but
> > if we don't keep track of those images anywhere then we may not realize
> > that 5 or 10 have actually been done and proposed but never integrated.
> > If they're kept on the wiki then perhaps we would both keep track of the
> > ones proposed and realize when it's worthwhile to add support for them
> > to the doc build system.
>
> Right.
>
> > As to Tom's point on the previous thread, that we would need to actually
> > maintain these images, that is helped by using dia, imv, since it's a
> > pretty simple tool to use and understand and is available on many
> > platforms.
>
> I'd also take a look at ditaa, which might make this even simpler.

If we manage to keep Java out of our build toolchain, we will be much
the better for it.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

On 10/21/2013 06:24 AM, David Fetter wrote:
> On Sat, Oct 19, 2013 at 01:56:23PM -0400, Peter Eisentraut wrote:
>> I'd also take a look at ditaa, which might make this even simpler.
> If we manage to keep Java out of our build toolchain, we will be much
> the better for it.

Amen.

--
Vik