Re: Documentation and explanatory diagrams

Rafael Martinez wrote:
> Sorry for the delay, I got finally some extra time to work on this.
>
> I am sending you a proposal with 13 diagrams to include in the manual so
> we can get an idea of how it will be. If it gets approved I will spend
> more time creating and including other diagrams and improving the build
> process for including figures into the manual.

Great!

> I have done this:
>
> - - Create/convert 13 images with/to DIA (v.0.96.1) and generate a PNG
> version of them.
> - - Patch the sgml files that will include the images under doc/src/sgml/.
> - - Patch the Makefile under doc/src/sgml/ so the images get move to
> doc/src/sgml/html/ under the generation of the html manual.

I have built an HTML version using your patch:

http://momjian.us/expire/pgsql-docs/

Here is a sample doc image:

http://momjian.us/expire/pgsql-docs/log-shipping-alternative.html

and all images appear here:

http://momjian.us/expire/pgsql-docs/img/

> I could test only the generation of the HTML version of the manual. I
> have had problems with the generation of the PDF version and I do not
> know at the moment if we have to ajust some of the images for the PDF
> version.

I think PDF will be fine. I can't generate PDF either but once it
committed to CVS I will have someone check and make adjustments.

> If this proposal gets accepted we should work with:
>
> - - Automatic generation of PNG files from DIA source under the build
> process of the manual.

I don't think we want to require dia to build the docs, so we are going
to keep the dia and png files in CVS.

> - - Testing of the PDF build process with images.
> - - Update the text some places to reference the figures.
> - - Automatic generation of a Table of Figures
> - - Create/convert more figures to include them in the manual.

Great.

> If you want to generate the html manual with these figures yourself, you
> have to do this in a 9.0beta2 source tree.:
>
> - - Untar the attached file pg_manual_figures.tar.gz under doc/src/sgml/.
> This will create an img/ directory with the DIA and PNG files versions
> of the figures.
>
> - - Patch the 9.0beta2 source tree with the attached file
> 9.0beta2_pg_manual_figures.patch (patch -p0 -i
> 9.0beta2_pg_manual_figures.patch)
>
> - - Delete the file doc/src/sgml/html-stamp and run make in doc/src/sgml/
>
> Well, I will await for your feedback before spending more time on this
> just in case I am in the wrong path.

I did adjust the file paths sightly and modified the makefile; new
patch attached.

I did remove the vacuum_full image because the 9.0 vacuum full rewrites
the table, rather than modifying it in place.

Also, there are two images that need to be updated for every major
release; is that something we want to commit to doing?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ None of us is going to be here forever. +

Bruce Momjian <bruce(at)momjian(dot)us> writes:
> Rafael Martinez wrote:
>> I am sending you a proposal with 13 diagrams to include in the manual so
>> we can get an idea of how it will be. If it gets approved I will spend
>> more time creating and including other diagrams and improving the build
>> process for including figures into the manual.

> Great!

Where are the DIA sources for these? It's impossible to evaluate how
maintainable these will be without seeing the source code.

>> If this proposal gets accepted we should work with:
>> - - Automatic generation of PNG files from DIA source under the build
>> process of the manual.

> I don't think we want to require dia to build the docs, so we are going
> to keep the dia and png files in CVS.

Yeah, probably, at least for the first go-round.

> Also, there are two images that need to be updated for every major
> release; is that something we want to commit to doing?

-1 on that. The timeline thing is *certainly* not worth the maintenance
effort, if indeed it even belongs in the docs at all (it looks more like
marketing material). I think the lines-of-code diagram is not worth its
keep either.

One minor comment is that the page_layout diagram should say BLCKSZ not
8K, and the different-colored arrows in it don't really convey much ---
they need on-image labeling I think.

regards, tom lane

Tom Lane wrote:
> Bruce Momjian <bruce(at)momjian(dot)us> writes:
> > Rafael Martinez wrote:
> >> I am sending you a proposal with 13 diagrams to include in the manual so
> >> we can get an idea of how it will be. If it gets approved I will spend
> >> more time creating and including other diagrams and improving the build
> >> process for including figures into the manual.
>
> > Great!
>
> Where are the DIA sources for these? It's impossible to evaluate how
> maintainable these will be without seeing the source code.

OK, I put the dia sources online:

http://momjian.us/expire/pgsql-docs/dia/

They are XML files.

> > Also, there are two images that need to be updated for every major
> > release; is that something we want to commit to doing?
>
> -1 on that. The timeline thing is *certainly* not worth the maintenance
> effort, if indeed it even belongs in the docs at all (it looks more like
> marketing material). I think the lines-of-code diagram is not worth its
> keep either.

Sure, I can remove them, unless someone else wants to argue for their
inclusion.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ None of us is going to be here forever. +

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Bruce Momjian wrote:
> Rafael Martinez wrote:
[........]
>>
>> Well, I will await for your feedback before spending more time on this
>> just in case I am in the wrong path.
>
> I did adjust the file paths sightly and modified the makefile; new
> patch attached.
>

You use cp $(srcdir)/img/png/*.png html/img/ in the Makefile but in the
<imagedata fileref=""> tags the images are under html/. Am I missing
anything?

> I did remove the vacuum_full image because the 9.0 vacuum full rewrites
> the table, rather than modifying it in place.
>

Good, I found this out after I sent you all the images.

> Also, there are two images that need to be updated for every major
> release; is that something we want to commit to doing?
>

Ok, I agree with you and Tom. They are not difficult to maintain but
they are more marketing material than technical.

If we go ahead with having explanatory diagrams, how do you want me to
send you new/updated diagrams (src&png)? Last time I try to send a large
attachment to the list, the e-mail was not deliver to the list. Maybe I
can make them available via an URL.

- --
Rafael Martinez, <r(dot)m(dot)guerrero(at)usit(dot)uio(dot)no>
Center for Information Technology Services
University of Oslo, Norway

PGP Public Key: http://folk.uio.no/rafael/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkws6GIACgkQBhuKQurGihTzVACfa3VUeZJtJ9WXotGG8cTNWnDd
30IAnRwyOZVNthzi+DkbvvGbMaNVk4KQ
=0gQ8
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tom Lane wrote:

>
> One minor comment is that the page_layout diagram should say BLCKSZ not
> 8K, and the different-colored arrows in it don't really convey much ---
> they need on-image labeling I think.
>

ack. I will update this image with these changes.

- --
Rafael Martinez, <r(dot)m(dot)guerrero(at)usit(dot)uio(dot)no>
Center for Information Technology Services
University of Oslo, Norway

PGP Public Key: http://folk.uio.no/rafael/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkws6HEACgkQBhuKQurGihRGOwCgjdwP2VbDsMR/knX1eHLNMw+y
PNcAnRhhFaXjGVAS1nkJcOuaVUFHW9QE
=GutE
-----END PGP SIGNATURE-----

Excerpts from Rafael Martinez's message of jue jul 01 15:11:31 -0400 2010:

> If we go ahead with having explanatory diagrams, how do you want me to
> send you new/updated diagrams (src&png)? Last time I try to send a large
> attachment to the list, the e-mail was not deliver to the list. Maybe I
> can make them available via an URL.

I think we should have a Makefile rule to build the PNGs from the source
dia files; this seems absent from this patch. (This means you shouldn't
send PNG files around.)

I'm not sure if this means that we don't want to have the PNG files in
our SCM though, because it'd make the doc unbuildable for those without
access to dia.

Rafael Martinez wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Bruce Momjian wrote:
> > Rafael Martinez wrote:
> [........]
> >>
> >> Well, I will await for your feedback before spending more time on this
> >> just in case I am in the wrong path.
> >
> > I did adjust the file paths sightly and modified the makefile; new
> > patch attached.
> >
>
> You use cp $(srcdir)/img/png/*.png html/img/ in the Makefile but in the
> <imagedata fileref=""> tags the images are under html/. Am I missing
> anything?

I will need to adjust that before I apply the patch. I had not decided
on a directory structure yet. I am thinking of using simply /dia and
/png directories so we don't have to create an img/png directory in the
HTML, which would look odd.

> > Also, there are two images that need to be updated for every major
> > release; is that something we want to commit to doing?
> >
>
> Ok, I agree with you and Tom. They are not difficult to maintain but
> they are more marketing material than technical.
>
> If we go ahead with having explanatory diagrams, how do you want me to
> send you new/updated diagrams (src&png)? Last time I try to send a large
> attachment to the list, the e-mail was not deliver to the list. Maybe I
> can make them available via an URL.

Sure, or just email it to me and I will handle it.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ None of us is going to be here forever. +

Alvaro Herrera wrote:
> Excerpts from Rafael Martinez's message of jue jul 01 15:11:31 -0400 2010:
>
> > If we go ahead with having explanatory diagrams, how do you want me to
> > send you new/updated diagrams (src&png)? Last time I try to send a large
> > attachment to the list, the e-mail was not deliver to the list. Maybe I
> > can make them available via an URL.
>
> I think we should have a Makefile rule to build the PNGs from the source
> dia files; this seems absent from this patch. (This means you shouldn't
> send PNG files around.)
>
> I'm not sure if this means that we don't want to have the PNG files in
> our SCM though, because it'd make the doc unbuildable for those without
> access to dia.

Yeah, it would be nice to have a rule, but we also should have the PNG
files in our CVS.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ None of us is going to be here forever. +

On tor, 2010-07-01 at 19:10 -0400, Alvaro Herrera wrote:
> I'm not sure if this means that we don't want to have the PNG files in
> our SCM though, because it'd make the doc unbuildable for those
> without access to dia.

Is there something that makes installing dia more challenging than the
other documentation build tools?

Peter Eisentraut wrote:
> On tor, 2010-07-01 at 19:10 -0400, Alvaro Herrera wrote:
> > I'm not sure if this means that we don't want to have the PNG files in
> > our SCM though, because it'd make the doc unbuildable for those
> > without access to dia.
>
> Is there something that makes installing dia more challenging than the
> other documentation build tools?

No, but it is an additional requirement for diagrams that will rarely
change.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ None of us is going to be here forever. +

On tor, 2010-07-01 at 22:08 -0400, Bruce Momjian wrote:
> Peter Eisentraut wrote:
> > On tor, 2010-07-01 at 19:10 -0400, Alvaro Herrera wrote:
> > > I'm not sure if this means that we don't want to have the PNG files in
> > > our SCM though, because it'd make the doc unbuildable for those
> > > without access to dia.
> >
> > Is there something that makes installing dia more challenging than the
> > other documentation build tools?
>
> No, but it is an additional requirement for diagrams that will rarely
> change.

I dispute that the diagrams will/should rarely change, and also that
changing rarely is an acceptable criterion for breaking the build
system.

Excerpts from Peter Eisentraut's message of jue jul 01 21:52:00 -0400 2010:
> On tor, 2010-07-01 at 19:10 -0400, Alvaro Herrera wrote:
> > I'm not sure if this means that we don't want to have the PNG files in
> > our SCM though, because it'd make the doc unbuildable for those
> > without access to dia.
>
> Is there something that makes installing dia more challenging than the
> other documentation build tools?

Err, I dunno -- it's just an apt-get away for me, but what will Tom say
when it doesn't work on his ancient HP-UX 10.20 system?

Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
> Excerpts from Peter Eisentraut's message of jue jul 01 21:52:00 -0400 2010:
>> Is there something that makes installing dia more challenging than the
>> other documentation build tools?

> Err, I dunno -- it's just an apt-get away for me, but what will Tom say
> when it doesn't work on his ancient HP-UX 10.20 system?

I don't try to build the docs on that box anyway --- it does have
openjade but such an old version that they don't build. In practice
building the docs already takes much more modern infrastructure than
compiling the source code; and besides there are many fewer people
who care about doing it.

A more interesting question is whether Marc can install a working
version of dia on whatever he uses to wrap the tarballs.

regards, tom lane

On Fri, 2 Jul 2010, Tom Lane wrote:

> Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
>> Excerpts from Peter Eisentraut's message of jue jul 01 21:52:00 -0400 2010:
>>> Is there something that makes installing dia more challenging than the
>>> other documentation build tools?
>
>> Err, I dunno -- it's just an apt-get away for me, but what will Tom say
>> when it doesn't work on his ancient HP-UX 10.20 system?
>
> I don't try to build the docs on that box anyway --- it does have
> openjade but such an old version that they don't build. In practice
> building the docs already takes much more modern infrastructure than
> compiling the source code; and besides there are many fewer people
> who care about doing it.
>
> A more interesting question is whether Marc can install a working
> version of dia on whatever he uses to wrap the tarballs.

that shouldn't be an issue ... Peter runs an update every 3 hours on that
machine right now as it is ...

Marc G. Fournier Hub.Org Hosting Solutions S.A.
scrappy(at)hub(dot)org http://www.hub.org

Yahoo:yscrappy Skype: hub.org ICQ:7615664 MSN:scrappy(at)hub(dot)org

tgl(at)sss(dot)pgh(dot)pa(dot)us (Tom Lane) writes:
> Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
>> Excerpts from Peter Eisentraut's message of jue jul 01 21:52:00 -0400 2010:
>>> Is there something that makes installing dia more challenging than the
>>> other documentation build tools?
>
>> Err, I dunno -- it's just an apt-get away for me, but what will Tom say
>> when it doesn't work on his ancient HP-UX 10.20 system?
>
> I don't try to build the docs on that box anyway --- it does have
> openjade but such an old version that they don't build. In practice
> building the docs already takes much more modern infrastructure than
> compiling the source code; and besides there are many fewer people
> who care about doing it.
>
> A more interesting question is whether Marc can install a working
> version of dia on whatever he uses to wrap the tarballs.

Good news... It's not terribly difficult to use command line usage to
get dia to export .png files.

{wrox} dia --export=transport.png Transports.dia
Transports.dia --> transport.png
{wrox} file Transports.dia transport.png
Transports.dia: gzip compressed data, from Unix
transport.png: PNG image data, 753 x 774, 8-bit/color RGBA, non-interlaced

So the makefile rule is pretty much:
%.png : %.dia; dia --export=$@ $<

I'll observe that while dia claims to be able to export in JPEG form,
it doesn't necessarily work:

{wrox} dia --export=transport.jpg Transports.dia

** (dia:946): CRITICAL **: dia error: do not know how to export into transport.jpg

If we have a preference for JPEG, then that's presumably an
ImageMagick run away...
--
output = ("cbbrowne" "@" "gmail.com")
http://linuxdatabases.info/info/languages.html
HEADLINE: Suicidal twin kills sister by mistake!

Tom Lane wrote:
> Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
> > Excerpts from Peter Eisentraut's message of jue jul 01 21:52:00 -0400 2010:
> >> Is there something that makes installing dia more challenging than the
> >> other documentation build tools?
>
> > Err, I dunno -- it's just an apt-get away for me, but what will Tom say
> > when it doesn't work on his ancient HP-UX 10.20 system?
>
> I don't try to build the docs on that box anyway --- it does have
> openjade but such an old version that they don't build. In practice
> building the docs already takes much more modern infrastructure than
> compiling the source code; and besides there are many fewer people
> who care about doing it.
>
> A more interesting question is whether Marc can install a working
> version of dia on whatever he uses to wrap the tarballs.

One more issue is that the dia and png files are going to impact our
download sizes:

526k ./dia
483k ./png

If we generate the png from the dia files, we are looking at increasing
the source download by 526k and the binary downloads by 483k for all
existing images and, of course, as we add images, these sizes will
increase.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ None of us is going to be here forever. +

On Fri, 2 Jul 2010, Bruce Momjian wrote:

> Tom Lane wrote:
>> Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
>>> Excerpts from Peter Eisentraut's message of jue jul 01 21:52:00 -0400 2010:
>>>> Is there something that makes installing dia more challenging than the
>>>> other documentation build tools?
>>
>>> Err, I dunno -- it's just an apt-get away for me, but what will Tom say
>>> when it doesn't work on his ancient HP-UX 10.20 system?
>>
>> I don't try to build the docs on that box anyway --- it does have
>> openjade but such an old version that they don't build. In practice
>> building the docs already takes much more modern infrastructure than
>> compiling the source code; and besides there are many fewer people
>> who care about doing it.
>>
>> A more interesting question is whether Marc can install a working
>> version of dia on whatever he uses to wrap the tarballs.
>
> One more issue is that the dia and png files are going to impact our
> download sizes:
>
> 526k ./dia
> 483k ./png
>
> If we generate the png from the dia files, we are looking at increasing
> the source download by 526k and the binary downloads by 483k for all
> existing images and, of course, as we add images, these sizes will
> increase.

Which brings me to a point I brought up before ... do we want to start
looking at a dist file split similar to what Devrim does for packages?
lib vs client vs server vs docs vs ... ? where each could be invididually
built, or requires a prev set (ie. client would require lib first) sort of
thing ...

----
Marc G. Fournier Hub.Org Hosting Solutions S.A.
scrappy(at)hub(dot)org http://www.hub.org

Yahoo:yscrappy Skype: hub.org ICQ:7615664 MSN:scrappy(at)hub(dot)org

Marc G. Fournier wrote:
> On Fri, 2 Jul 2010, Tom Lane wrote:
>
> > Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
> >> Excerpts from Peter Eisentraut's message of jue jul 01 21:52:00 -0400 2010:
> >>> Is there something that makes installing dia more challenging than the
> >>> other documentation build tools?
> >
> >> Err, I dunno -- it's just an apt-get away for me, but what will Tom say
> >> when it doesn't work on his ancient HP-UX 10.20 system?
> >
> > I don't try to build the docs on that box anyway --- it does have
> > openjade but such an old version that they don't build. In practice
> > building the docs already takes much more modern infrastructure than
> > compiling the source code; and besides there are many fewer people
> > who care about doing it.
> >
> > A more interesting question is whether Marc can install a working
> > version of dia on whatever he uses to wrap the tarballs.
>
> that shouldn't be an issue ... Peter runs an update every 3 hours on that
> machine right now as it is ...

OK, everyone seems to like requiring dia. I wasn't sure how popular dia
was. One hack solution to allow builds without dia would be to create a
Makefile rule that creates empty PNG files to match the dia files.

Can someone provide the command-line to build the PNG files from the DIA
files?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ None of us is going to be here forever. +

On Fri, Jul 2, 2010 at 5:28 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> OK, everyone seems to like requiring dia.

I don't like it a bit. It's hard enough for people to build the docs as it is.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

On Fri, 2 Jul 2010, Robert Haas wrote:

> On Fri, Jul 2, 2010 at 5:28 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
>> OK, everyone seems to like requiring dia.
>
> I don't like it a bit. It's hard enough for people to build the docs as
> it is.

Why should anyone build the docs? Its part of the tarball process, so the
only ppl that should be doing it are those coming from CVS, no ... ?

----
Marc G. Fournier Hub.Org Hosting Solutions S.A.
scrappy(at)hub(dot)org http://www.hub.org

Yahoo:yscrappy Skype: hub.org ICQ:7615664 MSN:scrappy(at)hub(dot)org

Robert Haas wrote:
> On Fri, Jul 2, 2010 at 5:28 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> > OK, everyone seems to like requiring dia.
>
> I don't like it a bit. It's hard enough for people to build the docs as it is.

That was my reaction too, but I am only one person. Yeah, I know that
is hard to be belive. How can one person revert his own faulty patches
so quickly? LOL

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ None of us is going to be here forever. +

Marc G. Fournier wrote:
> On Fri, 2 Jul 2010, Robert Haas wrote:
>
> > On Fri, Jul 2, 2010 at 5:28 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> >> OK, everyone seems to like requiring dia.
> >
> > I don't like it a bit. It's hard enough for people to build the docs as
> > it is.
>
> Why should anyone build the docs? Its part of the tarball process, so the
> only ppl that should be doing it are those coming from CVS, no ... ?

People often built them to verify the SGML markup and to view the
content/markup before submitting a doc patch.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ None of us is going to be here forever. +

On Fri, 2 Jul 2010, Bruce Momjian wrote:

> Marc G. Fournier wrote:
>> On Fri, 2 Jul 2010, Robert Haas wrote:
>>
>>> On Fri, Jul 2, 2010 at 5:28 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
>>>> OK, everyone seems to like requiring dia.
>>>
>>> I don't like it a bit. It's hard enough for people to build the docs as
>>> it is.
>>
>> Why should anyone build the docs? Its part of the tarball process, so the
>> only ppl that should be doing it are those coming from CVS, no ... ?
>
> People often built them to verify the SGML markup and to view the
> content/markup before submitting a doc patch.

Good point ... so, part of the Makefile should be modified for a flag to
disable dia build? "make NO_DIA=yes" ? Or, maybe have it default to NO
and for the tarball build, I'd modify the script for building it?

----
Marc G. Fournier Hub.Org Hosting Solutions S.A.
scrappy(at)hub(dot)org http://www.hub.org

Yahoo:yscrappy Skype: hub.org ICQ:7615664 MSN:scrappy(at)hub(dot)org

On Fri, Jul 2, 2010 at 5:43 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> Marc G. Fournier wrote:
>> On Fri, 2 Jul 2010, Robert Haas wrote:
>>
>> > On Fri, Jul 2, 2010 at 5:28 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
>> >> OK, everyone seems to like requiring dia.
>> >
>> > I don't like it a bit.  It's hard enough for people to build the docs as
>> > it is.
>>
>> Why should anyone build the docs?  Its part of the tarball process, so the
>> only ppl that should be doing it are those coming from CVS, no ... ?
>
> People often built them to verify the SGML markup and to view the
> content/markup before submitting a doc patch.

Actually, they often DON'T, which is a problem, and adding more
requirements is only going to make it worse.

There is not much reason for an end-user to build the docs - most
end-users will install from RPMs or one-click installers or whatever.
But everyone who is a developer needs to be able to build them, many
can't already, and we have quite a lot of developers. Adding dia will
also create knock-on work for packagers, although that should be
mostly a one-time thing. I like the idea of being able to have
pictures in our documentation, but I'm a little nervous about the
ramifications.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

Marc G. Fournier wrote:
> On Fri, 2 Jul 2010, Bruce Momjian wrote:
>
> > Marc G. Fournier wrote:
> >> On Fri, 2 Jul 2010, Robert Haas wrote:
> >>
> >>> On Fri, Jul 2, 2010 at 5:28 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> >>>> OK, everyone seems to like requiring dia.
> >>>
> >>> I don't like it a bit. It's hard enough for people to build the docs as
> >>> it is.
> >>
> >> Why should anyone build the docs? Its part of the tarball process, so the
> >> only ppl that should be doing it are those coming from CVS, no ... ?
> >
> > People often built them to verify the SGML markup and to view the
> > content/markup before submitting a doc patch.
>
> Good point ... so, part of the Makefile should be modified for a flag to
> disable dia build? "make NO_DIA=yes" ? Or, maybe have it default to NO
> and for the tarball build, I'd modify the script for building it?

I am afraid if we don't enable dia by default then we will end up with
production documentation without images. I think we have to default dia
to on, and give a reasonable error when it doesn't exist, and mention
how to turn it off.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ None of us is going to be here forever. +

On Fri, 2 Jul 2010, Bruce Momjian wrote:

> I am afraid if we don't enable dia by default then we will end up with
> production documentation without images. I think we have to default dia
> to on, and give a reasonable error when it doesn't exist, and mention
> how to turn it off.

Okay, I have no problem with either default on or off ... but ... won't th
efailure to have prod documentation with images be my fault? or am I
missing something else here?

----
Marc G. Fournier Hub.Org Hosting Solutions S.A.
scrappy(at)hub(dot)org http://www.hub.org

Yahoo:yscrappy Skype: hub.org ICQ:7615664 MSN:scrappy(at)hub(dot)org

Marc G. Fournier wrote:
> On Fri, 2 Jul 2010, Bruce Momjian wrote:
>
> > I am afraid if we don't enable dia by default then we will end up with
> > production documentation without images. I think we have to default dia
> > to on, and give a reasonable error when it doesn't exist, and mention
> > how to turn it off.
>
> Okay, I have no problem with either default on or off ... but ... won't th
> efailure to have prod documentation with images be my fault? or am I
> missing something else here?

The point is you will have to explicitly turn off dia/images to build
without images. That should make anyone building production docs always
generate images, unless they override the default.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ None of us is going to be here forever. +

On Fri, Jul 2, 2010 at 6:07 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> Marc G. Fournier wrote:
>> On Fri, 2 Jul 2010, Bruce Momjian wrote:
>>
>> > I am afraid if we don't enable dia by default then we will end up with
>> > production documentation without images.  I think we have to default dia
>> > to on, and give a reasonable error when it doesn't exist, and mention
>> > how to turn it off.
>>
>> Okay, I have no problem with either default on or off ... but ... won't th
>> efailure to have prod documentation with images be my fault?  or am I
>> missing something else here?
>
> The point is you will have to explicitly turn off dia/images to build
> without images.  That should make anyone building production docs always
> generate images, unless they override the default.

Is it at all practical to ship the PNGs as well as the DIA files in
CVS, and make rebuilding the PNGs from the DIAs a separate step?

Also, I think we need to review these images carefully before adding
any of them into our docs. For example, I'd dispute the picture of the
world shown on the file_based_log_shipping.png image; that's certainly
not the only way to set it up (the archive needn't be on the primary
node, right?).

The image that is called hot_standby is really showing a combination
of SR + log shipping. It has nothing to do with Hot Standby.

The page_layout image is kinda confusing - I can tell what the purple
and brown arrows are supposed to represent, but only because I already
know what they're supposed to mean.

The pg_dir_layout image seems like it would work equally well (and
maybe better) as text.

The pg_standby image appears identical to the file_based_log_shipping image.

The pgclient_server image is a mishmash of concepts with no unifying theme.

The streaming_replication image is technically not correct. WAL
sender reads from disk - PostgreSQL doesn't duplicate the WAL as it's
generated. It also makes it look like WAL sender/reciever are not
part of PostgreSQL, which of course isn't the case.

The tablespaces image labels the toplevel entity "PGCluster" but
doesn't use the "PG" prefix on any of the other items. That seems
inconsistent.

On the vacuum image, I don't know that the terms "active" and "expire"
are really the best terminology.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

Bruce Momjian <bruce(at)momjian(dot)us> writes:
> One more issue is that the dia and png files are going to impact our
> download sizes:

> 526k ./dia
> 483k ./png

500k of source for just half a dozen simple images? There's something
seriously wrong there. I suspect the PNGs could be compressed a lot
more, but right now I'm wondering exactly how verbose the dia "source"
representation is.

regards, tom lane

On fre, 2010-07-02 at 17:59 -0400, Robert Haas wrote:
> Adding dia will
> also create knock-on work for packagers

The documentation is shipped prebuilt in the source tarball, so
packagers wouldn't need to do anything.

On fre, 2010-07-02 at 18:07 -0400, Bruce Momjian wrote:
> The point is you will have to explicitly turn off dia/images to build
> without images. That should make anyone building production docs
> always
> generate images, unless they override the default.

We have the "make draft" target. That could build without images, if
that's important.

On fre, 2010-07-02 at 19:02 -0400, Robert Haas wrote:
> Is it at all practical to ship the PNGs as well as the DIA files in
> CVS, and make rebuilding the PNGs from the DIAs a separate step?

Well, that's what we're arguing about here.

> Also, I think we need to review these images carefully before adding
> any of them into our docs.

That's another matter. I'm not convinced that all of the images are
useful.

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> On Fri, Jul 2, 2010 at 5:43 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
>> People often built them to verify the SGML markup and to view the
>> content/markup before submitting a doc patch.

> Actually, they often DON'T, which is a problem, and adding more
> requirements is only going to make it worse.

> There is not much reason for an end-user to build the docs - most
> end-users will install from RPMs or one-click installers or whatever.
> But everyone who is a developer needs to be able to build them,

I think that adding dia to the set of requirements isn't that big a
deal, assuming that it's a widely available package. It's just one
program and should be a lot easier to install and configure than our
other doc toolchain requirements.

What I'm more worried about at the moment is whether it's a reasonable
choice of tool. If the "source" for a diagram is larger than the PNG
image representation, there is something seriously wrong with the
language design.

regards, tom lane

Tom Lane wrote:
> Bruce Momjian <bruce(at)momjian(dot)us> writes:
> > One more issue is that the dia and png files are going to impact our
> > download sizes:
>
> > 526k ./dia
> > 483k ./png
>
> 500k of source for just half a dozen simple images? There's something
> seriously wrong there. I suspect the PNGs could be compressed a lot
> more, but right now I'm wondering exactly how verbose the dia "source"
> representation is.

Here is the PNG URL:

http://momjian.us/expire/pgsql-docs/img/

and the dia files:

http://momjian.us/expire/pgsql-docs/dia/

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ None of us is going to be here forever. +

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> Is it at all practical to ship the PNGs as well as the DIA files in
> CVS, and make rebuilding the PNGs from the DIAs a separate step?

Other than the bulky file sizes I see nothing wrong with that approach.
We have traditionally not required developers to have autoconf installed
unless they wanted to muck with the configure script, and this seems
more or less equivalent.

> Also, I think we need to review these images carefully before adding
> any of them into our docs.

Sure. These are proof of concept not ready-to-ship diagrams. What we
ought to be looking at is what the source representation is like and how
easy it is for people to adjust the diagrams.

regards, tom lane

Peter Eisentraut wrote:
> On fre, 2010-07-02 at 18:07 -0400, Bruce Momjian wrote:
> > The point is you will have to explicitly turn off dia/images to build
> > without images. That should make anyone building production docs
> > always
> > generate images, unless they override the default.
>
> We have the "make draft" target. That could build without images, if
> that's important.

Yes, that would be easy.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ None of us is going to be here forever. +

On fre, 2010-07-02 at 22:15 -0400, Bruce Momjian wrote:
> > 500k of source for just half a dozen simple images? There's
> something
> > seriously wrong there. I suspect the PNGs could be compressed a lot
> > more, but right now I'm wondering exactly how verbose the dia
> "source"
> > representation is.
>
> Here is the PNG URL:
>
> http://momjian.us/expire/pgsql-docs/img/
>
> and the dia files:
>
> http://momjian.us/expire/pgsql-docs/dia/

The dia stuff is XML that should compress well.

On Fri, Jul 2, 2010 at 10:15 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> Tom Lane wrote:
>> Bruce Momjian <bruce(at)momjian(dot)us> writes:
>> > One more issue is that the dia and png files are going to impact our
>> > download sizes:
>>
>> >     526k     ./dia
>> >     483k     ./png
>>
>> 500k of source for just half a dozen simple images?  There's something
>> seriously wrong there.  I suspect the PNGs could be compressed a lot
>> more, but right now I'm wondering exactly how verbose the dia "source"
>> representation is.
>
> Here is the PNG URL:
>
>        http://momjian.us/expire/pgsql-docs/img/
>
> and the dia files:
>
>        http://momjian.us/expire/pgsql-docs/dia/

Are these dia files generated by some other program, or did someone
really write all that XML by hand?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

Peter Eisentraut wrote:
> On fre, 2010-07-02 at 22:15 -0400, Bruce Momjian wrote:
> > > 500k of source for just half a dozen simple images? There's
> > something
> > > seriously wrong there. I suspect the PNGs could be compressed a lot
> > > more, but right now I'm wondering exactly how verbose the dia
> > "source"
> > > representation is.
> >
> > Here is the PNG URL:
> >
> > http://momjian.us/expire/pgsql-docs/img/
> >
> > and the dia files:
> >
> > http://momjian.us/expire/pgsql-docs/dia/
>
> The dia stuff is XML that should compress well.

Gzip takes the dia directory from 526k to 42k. And of course gzip does
little for the PNG files.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ None of us is going to be here forever. +

On 03/07/10 02:02, Robert Haas wrote:
> The page_layout image is kinda confusing - I can tell what the purple
> and brown arrows are supposed to represent, but only because I already
> know what they're supposed to mean.

I realize that these are just sample images, but I'd like to enact a
strict rule for any diagrams we include in the docs:

All diagrams must have a key.

Otherwise you don't know what all the boxes and arrows mean. Data flow,
control flow, dependency, inheritance, or what?

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On 3 July 2010 03:41, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> Peter Eisentraut wrote:
>> On fre, 2010-07-02 at 22:15 -0400, Bruce Momjian wrote:
>> > > 500k of source for just half a dozen simple images?  There's
>> > something
>> > > seriously wrong there.  I suspect the PNGs could be compressed a lot
>> > > more, but right now I'm wondering exactly how verbose the dia
>> > "source"
>> > > representation is.
>> >
>> > Here is the PNG URL:
>> >
>> >         http://momjian.us/expire/pgsql-docs/img/
>> >
>> > and the dia files:
>> >
>> >         http://momjian.us/expire/pgsql-docs/dia/
>>
>> The dia stuff is XML that should compress well.
>
> Gzip takes the dia directory from 526k to 42k.  And of course gzip does
> little for the PNG files.
>

How configurable is the generation process for these PNGs? It's just
that they don't appear to be optimised.

For example, pgclient_server.png is created at 75,089 bytes, but can
be reduced to 15,643 bytes like in the attached file
(pgclient_server2.png) by using a 16-colour indexed palette.

Thom

On 3 July 2010 13:59, Thom Brown <thombrown(at)gmail(dot)com> wrote:
> On 3 July 2010 03:41, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
>> Peter Eisentraut wrote:
>>> On fre, 2010-07-02 at 22:15 -0400, Bruce Momjian wrote:
>>> > > 500k of source for just half a dozen simple images?  There's
>>> > something
>>> > > seriously wrong there.  I suspect the PNGs could be compressed a lot
>>> > > more, but right now I'm wondering exactly how verbose the dia
>>> > "source"
>>> > > representation is.
>>> >
>>> > Here is the PNG URL:
>>> >
>>> >         http://momjian.us/expire/pgsql-docs/img/
>>> >
>>> > and the dia files:
>>> >
>>> >         http://momjian.us/expire/pgsql-docs/dia/
>>>
>>> The dia stuff is XML that should compress well.
>>
>> Gzip takes the dia directory from 526k to 42k.  And of course gzip does
>> little for the PNG files.
>>
>
> How configurable is the generation process for these PNGs?  It's just
> that they don't appear to be optimised.
>
> For example, pgclient_server.png is created at 75,089 bytes, but can
> be reduced to 15,643 bytes like in the attached file
> (pgclient_server2.png) by using a 16-colour indexed palette.
>
> Thom
>

Being a bit more generous, and using a 32-colour palette, it looks
like it yields a typical saving of 75%

Original images:

40290 file_based_log_shipping.png
54211 hot_standby.png
52400 page_layout.png
75089 pgclient_server.png
49117 pg_dir_layout.png
17692 pg_lines_code.png
41723 pg_standby.png
25135 pitr.png
10946 postgresql_hist.png
34156 streaming_replication.png
68679 tablespaces.png
18197 vacuum.png

TOTAL: 487635 bytes

32-colour palette images (compression %):

10700 file_based_log_shipping2.png (73.44%)
14245 hot_standby2.png (73.72%)
9896 page_layout2.png (81.11%)
13786 pg_dir_layout2.png (81.64%)
6347 pg_lines_code2.png (87.08%)
11211 pg_standby2.png (36.63%)
18173 pgclient_server2.png (56.44%)
7148 pitr2.png (71.56%)
3873 postgresql_hist2.png (64.62%)
8969 streaming_replication2.png (73.74%)
15338 tablespaces2.png (77.67%)
5846 vacuum2.png (67.87%)

TOTAL: 125532 bytes (74.26%)

..just an idea anyway.

Thom

On Sat, Jul 3, 2010 at 6:48 PM, Thom Brown <thombrown(at)gmail(dot)com> wrote:
> ..just an idea anyway.

Seems like a good one. But I'd still like to know the answer to the
question I asked upthread - how is anyone supposed to generate images
this way?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

On Sat, 3 Jul 2010, Robert Haas wrote:

> On Sat, Jul 3, 2010 at 6:48 PM, Thom Brown <thombrown(at)gmail(dot)com> wrote:
>> ..just an idea anyway.
>
> Seems like a good one. But I'd still like to know the answer to the
> question I asked upthread - how is anyone supposed to generate images
> this way?

One would think that configure could be used to detect dia and disable
generating images *if* it doesn't exist ... how will the docs look with
the images missing?

----
Marc G. Fournier Hub.Org Hosting Solutions S.A.
scrappy(at)hub(dot)org http://www.hub.org

Yahoo:yscrappy Skype: hub.org ICQ:7615664 MSN:scrappy(at)hub(dot)org

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> Seems like a good one. But I'd still like to know the answer to the
> question I asked upthread - how is anyone supposed to generate images
> this way?

Hmm, judging from
http://live.gnome.org/Dia

the answer is that .dia files aren't meant to be human readable but only
to be a storage format for WYSIWYG editing. I find this less than
thrilling: it essentially means you *can't* edit the images any other
way than using dia. (I'd bet a nickel that any small change results in
massive changes in the file contents, too, which will be un-fun for
keeping them in a VCS.)

regards, tom lane

On Sat, Jul 3, 2010 at 7:14 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> Seems like a good one.  But I'd still like to know the answer to the
>> question I asked upthread - how is anyone supposed to generate images
>> this way?
>
> Hmm, judging from
> http://live.gnome.org/Dia
>
> the answer is that .dia files aren't meant to be human readable but only
> to be a storage format for WYSIWYG editing.  I find this less than
> thrilling: it essentially means you *can't* edit the images any other
> way than using dia.  (I'd bet a nickel that any small change results in
> massive changes in the file contents, too, which will be un-fun for
> keeping them in a VCS.)

Anyone know of any other alternatives we could investigate?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> Anyone know of any other alternatives we could investigate?

I hesitate to suggest xfig, but at least it's got a well-specified file
format ...

regards, tom lane

On Jul 3, 2010, at 4:16 PM, Robert Haas wrote:

> On Sat, Jul 3, 2010 at 7:14 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>>> Seems like a good one. But I'd still like to know the answer to the
>>> question I asked upthread - how is anyone supposed to generate images
>>> this way?
>>
>> Hmm, judging from
>> http://live.gnome.org/Dia
>>
>> the answer is that .dia files aren't meant to be human readable but only
>> to be a storage format for WYSIWYG editing. I find this less than
>> thrilling: it essentially means you *can't* edit the images any other
>> way than using dia. (I'd bet a nickel that any small change results in
>> massive changes in the file contents, too, which will be un-fun for
>> keeping them in a VCS.)
>
> Anyone know of any other alternatives we could investigate?

SVG is one option.

Natively supported in all web browsers but one (and that has a
workaround), vector based, and there are a bunch of editors for it.

There are several rasterizers to png, I'm not sure which would be
least painful as part of a build chain.

( http://www.mediawiki.org/wiki/Manual:%24wgSVGConverters )

Cheers,
Steve

Tom Lane wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> > Anyone know of any other alternatives we could investigate?
>
> I hesitate to suggest xfig, but at least it's got a well-specified file
> format ...

xfig is what I use and I have edited those by hand in the past.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ None of us is going to be here forever. +

On lör, 2010-07-03 at 19:14 -0400, Tom Lane wrote:
> (I'd bet a nickel that any small change results in
> massive changes in the file contents, too, which will be un-fun for
> keeping them in a VCS.)

I asked about this in the original thread and was told that this
wouldn't be a problem.

On lör, 2010-07-03 at 19:17 -0400, Tom Lane wrote:
> I hesitate to suggest xfig, but at least it's got a well-specified
> file format ...

Looks a little old. It doesn't appear to support SVG output, for
example.

Peter Eisentraut wrote:
> On l?r, 2010-07-03 at 19:17 -0400, Tom Lane wrote:
> > I hesitate to suggest xfig, but at least it's got a well-specified
> > file format ...
>
> Looks a little old. It doesn't appear to support SVG output, for
> example.

It supports EPS output which I believe can be converted to SVG.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ None of us is going to be here forever. +

On 4 July 2010 04:49, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> Peter Eisentraut wrote:
>> On l?r, 2010-07-03 at 19:17 -0400, Tom Lane wrote:
>> > I hesitate to suggest xfig, but at least it's got a well-specified
>> > file format ...
>>
>> Looks a little old.  It doesn't appear to support SVG output, for
>> example.
>
> It supports EPS output which I believe can be converted to SVG.
>

It looks like it does support SVG output:

http://en.wikipedia.org/wiki/Xfig
http://epb.lbl.gov/xfig/printing.html#export
http://www.xfig.org/art11.html

Thom

Thom Brown wrote:
> On 4 July 2010 04:49, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> > Peter Eisentraut wrote:
> >> On l?r, 2010-07-03 at 19:17 -0400, Tom Lane wrote:
> >> > I hesitate to suggest xfig, but at least it's got a well-specified
> >> > file format ...
> >>
> >> Looks a little old. ?It doesn't appear to support SVG output, for
> >> example.
> >
> > It supports EPS output which I believe can be converted to SVG.
> >
>
> It looks like it does support SVG output:
>
> http://en.wikipedia.org/wiki/Xfig
> http://epb.lbl.gov/xfig/printing.html#export
> http://www.xfig.org/art11.html

Yes, I have xfig 3.2 and it has SVG export, though it is labeled as
beta.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ None of us is going to be here forever. +

Bruce Momjian <bruce(at)momjian(dot)us> writes:
> Thom Brown wrote:
>> It looks like it does support SVG output:
>>
>> http://en.wikipedia.org/wiki/Xfig
>> http://epb.lbl.gov/xfig/printing.html#export
>> http://www.xfig.org/art11.html

> Yes, I have xfig 3.2 and it has SVG export, though it is labeled as
> beta.

Why should we care about SVG export? AFAICT, SVG support is not
widespread enough or stable enough among web browsers that we'd want
to use SVG directly in the finished HTML docs.

(But having said that, I recall seeing at least one diagram editor
whose native format was SVG. That might be worth a look too.)

regards, tom lane

Robert Haas wrote:
> On Sat, Jul 3, 2010 at 7:14 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>>> Seems like a good one.  But I'd still like to know the answer to the
>>> question I asked upthread - how is anyone supposed to generate images
>>> this way?
>>
>> Hmm, judging from
>> http://live.gnome.org/Dia
>>
>> the answer is that .dia files aren't meant to be human readable but only
>> to be a storage format for WYSIWYG editing.  I find this less than
>> thrilling: it essentially means you *can't* edit the images any other
>> way than using dia.  (I'd bet a nickel that any small change results in
>> massive changes in the file contents, too, which will be un-fun for
>> keeping them in a VCS.)
>
> Anyone know of any other alternatives we could investigate?
>

All this was tested and sent to the list in april 2010. Check this post for more
information:

http://archives.postgresql.org/pgsql-docs/2010-04/msg00038.php

As you can see, any small change does not result in massive changes in the file
contents.

--
Rafael Martinez, <r(dot)m(dot)guerrero(at)usit(dot)uio(dot)no>
Center for Information Technology Services
University of Oslo, Norway

PGP Public Key: http://folk.uio.no/rafael/

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/03/2010 01:02 AM, Robert Haas wrote:

>
> Also, I think we need to review these images carefully before adding
> any of them into our docs.

I agree with this. I didn't want to use a lot time with the diagrams
until everything is decided and we agree to have diagrams in the
documentation.

They are a proof of concept, different diagram types for different parts
of the documentation. They all need to be reviewed if they are going to
be used.

If we go for having diagrams, we should:

- - Create a TODO list with the diagrams we would like to have and where.
- - Define a procedure and rules to send diagrams patches: how to send,
review, give feedback, back to send and final commit.
- - Define some guidelines about how to create a diagram, colors to use,
key information, etc.

We should also remember the users of our documentation:

1) DBAs
2) Application Developers
3) Newbies who want to learn postgres
4) People interested in postgres internals

I think, most of our users are 1) or 2) but certainly different users
need different types of diagrams.

> For example, I'd dispute the picture of the
> world shown on the file_based_log_shipping.png image; that's certainly
> not the only way to set it up (the archive needn't be on the primary
> node, right?).
>

It is not the only way of shipping files but it is certainly the safest.
Getting the files to ship from another source than the arch directory in
the primary is a disaster waiting to happen if something happens to the
process/mechanism that ships files, and this jobs is delayed or fails.

> The image that is called hot_standby is really showing a combination
> of SR + log shipping. It has nothing to do with Hot Standby.
>

Well, what is HS without SR + log shipping. Again, this is a typical
diagram for a DBA. What they need is an overview on how HS looks like
with all the components we suggest they should use when using HS.

A diagram about HS with internal information for a developer will be
completely different.

> The page_layout image is kinda confusing - I can tell what the purple
> and brown arrows are supposed to represent, but only because I already
> know what they're supposed to mean.
>

Typical example of what Heikki said in another message. All diagrams of
this type must have a key explaining the boxes/arrows.

>
> The pg_standby image appears identical to the file_based_log_shipping image.
>

They are not the same. One uses pg_standby and the other one the
internal mechanism in postgres to get the wal files to restore. Two very
different and incompatible ways of getting the same result, well
documented in the manual.

> The pgclient_server image is a mishmash of concepts with no unifying theme.
>

Typical diagram for a DBA. They need diagrams that give an overview of
the components/parts involved in the system they are administrating. The
top half of the diagram is just a representation of what is explained in
the section of the documentation where this diagram was suggested to be.
The bottom half is just a representation of the main parts involved in
the saving/getting data from disk/memory.

> The streaming_replication image is technically not correct. WAL
> sender reads from disk - PostgreSQL doesn't duplicate the WAL as it's
> generated. It also makes it look like WAL sender/reciever are not
> part of PostgreSQL, which of course isn't the case.
>

Again, typical diagram for a DBA with an overview of components and how
they interact with each other. They don't care where the wal sender gets
the WAL information, they care that the system has two processes talking
with each other and sending information between nodes. After testing SR,
I can say that the secondary node generates wal files under pg_xlog when
using SR.

regards,
- --
Rafael Martinez, <r(dot)m(dot)guerrero(at)usit(dot)uio(dot)no>
Center for Information Technology Services
University of Oslo, Norway

PGP Public Key: http://folk.uio.no/rafael/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEARECAAYFAkwxlm4ACgkQBhuKQurGihSQtwCfVG7tn0rmf+TlrwgShFFYXhsB
PYMAoKLrgHSdhiP7p1p1yHKO5Mny6BSw
=BPlU
-----END PGP SIGNATURE-----

On Mon, Jul 5, 2010 at 4:23 AM, Rafael Martinez
<r(dot)m(dot)guerrero(at)usit(dot)uio(dot)no> wrote:
>> For example, I'd dispute the picture of the
>> world shown on the file_based_log_shipping.png image; that's certainly
>> not the only way to set it up (the archive needn't be on the primary
>> node, right?).
>
> It is not the only way of shipping files but it is certainly the safest.
> Getting the files to ship from another source than the arch directory in
> the primary is a disaster waiting to happen if something happens to the
> process/mechanism that ships files, and this jobs is delayed or fails.

Well, if it's presented as a possible configuration rather than "the
only way to do it" then I'm fine with that. I agree many people will
set it up that way in practice.

>> The image that is called hot_standby is really showing a combination
>> of SR + log shipping.  It has nothing to do with Hot Standby.
>
> Well, what is HS without SR + log shipping. Again, this is a typical
> diagram for a DBA. What they need is an overview on how HS looks like
> with all the components we suggest they should use when using HS.

Hot Standby is the ability to run queries on the standby. That image
has nothing to do with running queries.

>> The streaming_replication image is technically not correct.  WAL
>> sender reads from disk - PostgreSQL doesn't duplicate the WAL as it's
>> generated.   It also makes it look like WAL sender/reciever are not
>> part of PostgreSQL, which of course isn't the case.
>
> Again, typical diagram for a DBA with an overview of components and how
> they interact with each other. They don't care where the wal sender gets
> the WAL information, they care that the system has two processes talking
> with each other and sending information between nodes. After testing SR,
> I can say that the secondary node generates wal files under pg_xlog when
> using SR.

I'm going to argue that diagrams should be made as technically correct
as possible even if we think that DBAs won't care.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

bruce(at)momjian(dot)us (Bruce Momjian) writes:
> Marc G. Fournier wrote:
>> On Fri, 2 Jul 2010, Tom Lane wrote:
>>
>> > Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
>> >> Excerpts from Peter Eisentraut's message of jue jul 01 21:52:00 -0400 2010:
>> >>> Is there something that makes installing dia more challenging than the
>> >>> other documentation build tools?
>> >
>> >> Err, I dunno -- it's just an apt-get away for me, but what will Tom say
>> >> when it doesn't work on his ancient HP-UX 10.20 system?
>> >
>> > I don't try to build the docs on that box anyway --- it does have
>> > openjade but such an old version that they don't build. In practice
>> > building the docs already takes much more modern infrastructure than
>> > compiling the source code; and besides there are many fewer people
>> > who care about doing it.
>> >
>> > A more interesting question is whether Marc can install a working
>> > version of dia on whatever he uses to wrap the tarballs.
>>
>> that shouldn't be an issue ... Peter runs an update every 3 hours on that
>> machine right now as it is ...
>
> OK, everyone seems to like requiring dia. I wasn't sure how popular dia
> was. One hack solution to allow builds without dia would be to create a
> Makefile rule that creates empty PNG files to match the dia files.
>
> Can someone provide the command-line to build the PNG files from the DIA
> files?

Here's a pretty suitable Makefile rule:
%.png : %.dia; dia --export=$@ $<
--
output = ("cbbrowne" "@" "gmail.com")
http://linuxdatabases.info/info/languages.html
HEADLINE: Suicidal twin kills sister by mistake!

Rafael Martinez wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 07/03/2010 01:02 AM, Robert Haas wrote:
>
> >
> > Also, I think we need to review these images carefully before adding
> > any of them into our docs.
>
> I agree with this. I didn't want to use a lot time with the diagrams
> until everything is decided and we agree to have diagrams in the
> documentation.
>
> They are a proof of concept, different diagram types for different parts
> of the documentation. They all need to be reviewed if they are going to
> be used.
>
> If we go for having diagrams, we should:
>
> - - Create a TODO list with the diagrams we would like to have and where.
> - - Define a procedure and rules to send diagrams patches: how to send,
> review, give feedback, back to send and final commit.
> - - Define some guidelines about how to create a diagram, colors to use,
> key information, etc.

Where are we on this? Application? dia? xfig? Number of colors? Once
we decide these items, we can start adding diagrams.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ None of us is going to be here forever. +

Excerpts from Bruce Momjian's message of vie jul 09 20:01:55 -0400 2010:

> Where are we on this? Application? dia? xfig? Number of colors? Once
> we decide these items, we can start adding diagrams.

I think the first step is defining the source format. The proposed
patch uses the dia XML source format, but this is said to have some
drawbacks (namely: the source is too verbose, the generated png is too
bloated, and only one tool to handle it). SVG was also proposed as an
alternative, but there wasn't much discussion about it. And finally we
have xfig.

So what is it?

Maybe we could have diagrams generated by different tools; e.g. we could
use GraphViz for some things.

Alvaro Herrera wrote:
> Excerpts from Bruce Momjian's message of vie jul 09 20:01:55 -0400 2010:
>
> > Where are we on this? Application? dia? xfig? Number of colors? Once
> > we decide these items, we can start adding diagrams.
>
> I think the first step is defining the source format. The proposed
> patch uses the dia XML source format, but this is said to have some
> drawbacks (namely: the source is too verbose, the generated png is too
> bloated, and only one tool to handle it). SVG was also proposed as an
> alternative, but there wasn't much discussion about it. And finally we
> have xfig.
>
> So what is it?
>
> Maybe we could have diagrams generated by different tools; e.g. we could
> use GraphViz for some things.

OK, all the images on my presentations page are from xfig, and the
representative file sizes are as below:

1092 query_ssl.fig
4370 query_ssl.png

PNG image attached.

What I like about xfig is that I can export EPS and embed those in PDF
files as _raster_ images. Not sure we can use those in our PDFs though.

I assume Dia has more image style options than xfig.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ None of us is going to be here forever. +

On Fri, Jul 9, 2010 at 10:46 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> PNG image attached.

Is it just me, or does that look pretty pixellated and awful?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

On Jul 9, 2010, at 8:35 PM, Robert Haas wrote:

> On Fri, Jul 9, 2010 at 10:46 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
>> PNG image attached.
>
> Is it just me, or does that look pretty pixellated and awful?

As a comparison I did an SVG version using OmniGraffle and a conversion to PNG. I suspect Inkscape (which is SVG native, cross-platform, open source and all that sort of thing) would give similar results, but I'm more familiar with OmniGraffle.

http://wordtothewise.com/queryssl/

Cheers,
Steve

On fre, 2010-07-09 at 22:29 -0400, Alvaro Herrera wrote:
> I think the first step is defining the source format. The proposed
> patch uses the dia XML source format, but this is said to have some
> drawbacks (namely: the source is too verbose, the generated png is too
> bloated, and only one tool to handle it).

The same is true of the DocBook toolchain, isn't it?

> SVG was also proposed as an alternative, but there wasn't much
> discussion about it.

SVG isn't a practical source format, in the same way that HTML isn't a
practical source format.

On fre, 2010-07-09 at 22:29 -0400, Alvaro Herrera wrote:
> Maybe we could have diagrams generated by different tools; e.g. we
> could use GraphViz for some things.

I have found that graphviz doesn't work very well for creating diagrams
of software systems or the like.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Bruce Momjian wrote:

>
> Where are we on this? Application? dia? xfig? Number of colors? Once
> we decide these items, we can start adding diagrams.
>

This is an attempt to summarize the current status and what have been
said/done about the subject.

After the first proposal of using DIA to create diagrams, some tests,
several diagrams and a patch of the documentation has been sent to the
list as a proof of concept. DIA was chosen because it initially met the
requirements proposed by list members.

* Requirements
- ---------------

1) We need a program to create and manipulate diagrams that is easy to
use and can be used on Unix, Windows, or Mac.

2) We need a format that is easy to maintain, that can check into VCS
and that will provide tools for automatically converting to a variety of
target formats.

* Main objections and feedback
- -------------------------------

1) DIA has a terrible UI.

2) dia-format is too verbose and it uses too much space when it is not
compressed.

3) You can't edit the dia-files in any other way than using DIA

4) PNG files generated by DIA are too large.

5) Two ways of getting the PNG files in the documentation: shipping them
vs. generating them in the build process.

6) xfig and SVG have been proposed as an alternative to dia files.

This is the status as per today. Below I will argue for and against
these objections and feedback and I hope this will help to take a final
decision:

1) DIA is not perfect and it could be better. That said, it is easy to
install, it does run on Unix, Windows, or Mac, it is easy to learn and
the end results are potentially professional. As far as I know it is the
only program that has all these features.

2) When compressed with gzip, we can get over 90% reduction in size.

3) I cannot see the problem with this. This will be the same with any
other format. Editing a xfig or svg file manually via a text editor is
as tiresome and counterproductive as doing it with a dia file.

4) True. By default PNG files generated with DIA have these attributtes:
Colorspace: RGB / Depth: 8-bit / Type: Truecolor. The 'Truecolor'
attributte is not necessary in explanatory diagrams and it is the reason
of the extra size.

PNG files generated with DIA could be converted with ImageMagick. We can
get between 50% and 75% reduction in size with ImageMagick if we want.
That said, I do not think the documentation will have more than maybe
30-40 diagrams. 40 diagrams using the same amount of space as the
examples sent to the list will use around 1.5MB. I really cannot see how
this can be a problem in 2010 and I would not even bother to reduce the
size if I had to decide this.

5) This is the big question. Personally, I think that generating the PNG
files in the build process is a more elegant solution. The
infrastructure needed for this is easy to install and you have RPM and
DEB packages to do this if you don't want to compile the source. The few
people who build the documentation themself should not have a problem
installing the extra program needed for this.

That said, I do not have a problem neither with shipping the files.

6) Do we have programs to work with these formats on Unix, Windows and
Mac? Are they easy to use and learn? What are the software requirements
needed to generate other formats?

Well, I hope this summary will help us to take the final decision about
the program/format/infrastructure to use, so we can begin working on
the diagrams we are going to include and the contents of them.

regards,
- --
Rafael Martinez, <r(dot)m(dot)guerrero(at)usit(dot)uio(dot)no>
Center for Information Technology Services
University of Oslo, Norway

PGP Public Key: http://folk.uio.no/rafael/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkw5c8EACgkQBhuKQurGihTjFACeLA77cj51zhoX57mRH6hsJHMP
jH0An2CKPmmqaK0CzdDa5ioGX5YJthTy
=vzmm
-----END PGP SIGNATURE-----

On Jul 11, 2010, at 12:33 AM, Rafael Martinez wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Bruce Momjian wrote:
>
>>
>> Where are we on this? Application? dia? xfig? Number of colors? Once
>> we decide these items, we can start adding diagrams.
>>
>
> This is an attempt to summarize the current status and what have been
> said/done about the subject.
>
> After the first proposal of using DIA to create diagrams, some tests,
> several diagrams and a patch of the documentation has been sent to the
> list as a proof of concept. DIA was chosen because it initially met the
> requirements proposed by list members.
>
>
> * Requirements
> - ---------------
>
> 1) We need a program to create and manipulate diagrams that is easy to
> use and can be used on Unix, Windows, or Mac.
>
> 2) We need a format that is easy to maintain, that can check into VCS
> and that will provide tools for automatically converting to a variety of
> target formats.
>
>
> * Main objections and feedback
> - -------------------------------
>
> 1) DIA has a terrible UI.
>
> 2) dia-format is too verbose and it uses too much space when it is not
> compressed.
>
> 3) You can't edit the dia-files in any other way than using DIA
>
> 4) PNG files generated by DIA are too large.
>
> 5) Two ways of getting the PNG files in the documentation: shipping them
> vs. generating them in the build process.
>
> 6) xfig and SVG have been proposed as an alternative to dia files.
>
>
> This is the status as per today. Below I will argue for and against
> these objections and feedback and I hope this will help to take a final
> decision:
>
> 1) DIA is not perfect and it could be better. That said, it is easy to
> install, it does run on Unix, Windows, or Mac, it is easy to learn and
> the end results are potentially professional. As far as I know it is the
> only program that has all these features.

Inkscape will run on all three platforms. It has an OS X .dmg, windows
installer (and is in the Ubuntu standard repositories, at least).

> 2) When compressed with gzip, we can get over 90% reduction in size.
>
> 3) I cannot see the problem with this. This will be the same with any
> other format. Editing a xfig or svg file manually via a text editor is
> as tiresome and counterproductive as doing it with a dia file.

You wouldn't edit any three of those file types in a text editor any more than
you'd edit a PNG with a hex editor (i.e. it's possible, and sometimes
useful, but not part of your workflow).

More importantly, perhaps, if you use a standard file format, rather than
something proprietary, you can edit it with whatever vector editor you like -
import SVG, edit, export SVG.

Choice of fonts to use, such that they'll be available on all the machines
that are used is probably important too. The fonts are not usually
packaged up with the vector format, rather relying on the fonts on the
machine rendering the image. SVG allows use of css-style descriptions
to choose the local font to use.

> 4) True. By default PNG files generated with DIA have these attributtes:
> Colorspace: RGB / Depth: 8-bit / Type: Truecolor. The 'Truecolor'
> attributte is not necessary in explanatory diagrams and it is the reason
> of the extra size.
>
> PNG files generated with DIA could be converted with ImageMagick. We can
> get between 50% and 75% reduction in size with ImageMagick if we want.
> That said, I do not think the documentation will have more than maybe
> 30-40 diagrams. 40 diagrams using the same amount of space as the
> examples sent to the list will use around 1.5MB. I really cannot see how
> this can be a problem in 2010 and I would not even bother to reduce the
> size if I had to decide this.

What is the purpose of the PNG files? For things like PDF, vector would
be the better source format.

For html, vector would be a better format to use than PNG (file
size, quality of image, ability to scale) where possible. My understanding
is that it's possible to render basic SVG in the browser natively on
everything but IE and with some javascript or flash assistance on IE.
I'm not sure that is the right way to go today (though it's likely to become
more so as future browser releases, including IE9, clean up their SVG
support).

> 5) This is the big question. Personally, I think that generating the PNG
> files in the build process is a more elegant solution. The
> infrastructure needed for this is easy to install and you have RPM and
> DEB packages to do this if you don't want to compile the source. The few
> people who build the documentation themself should not have a problem
> installing the extra program needed for this.
>
> That said, I do not have a problem neither with shipping the files.
>
> 6) Do we have programs to work with these formats on Unix, Windows and
> Mac? Are they easy to use and learn? What are the software requirements
> needed to generate other formats?

xfig, inkscape and dia are all available on all three platforms (if you count
the cygwin port of xfig, anyway), as are "Draw" from OpenOffice.org, glips
graffiti and probably others.

There are many more apps that are not available on all three platforms,
or aren't freely available, that can edit the same format. We don't require
that everyone use vi to edit text files, just anything that'll edit them in a
way that doesn't break their format for everyone else.

>
> Well, I hope this summary will help us to take the final decision about
> the program/format/infrastructure to use, so we can begin working on
> the diagrams we are going to include and the contents of them.

I have no dog in this fight, and I'd be overjoyed to see diagrams of
any sort. I do think that requiring use of a single, fairly clunky, graphics
package rather than allowing people who get the urge to create a
diagram to use whichever graphics program they're familiar with is
likely to lead to more, and better, documentation.

Cheers,
Steve

Steve Atkins <steve(at)blighty(dot)com> writes:
> On Jul 11, 2010, at 12:33 AM, Rafael Martinez wrote:
> [ assorted arguments for and against DIA ]

>> 2) dia-format is too verbose and it uses too much space when it is not
>> compressed. [vs]
>> 2) When compressed with gzip, we can get over 90% reduction in size.

>> 3) You can't edit the dia-files in any other way than using DIA [vs]
>> 3) I cannot see the problem with this. This will be the same with any
>> other format. Editing a xfig or svg file manually via a text editor is
>> as tiresome and counterproductive as doing it with a dia file.

> You wouldn't edit any three of those file types in a text editor any more than
> you'd edit a PNG with a hex editor (i.e. it's possible, and sometimes
> useful, but not part of your workflow).

More to the point here is that we want to be able to store the diagram
source files in an SCM (CVS, soon to be git), and one of the main
reasons for wanting to do that is so that people can see exactly what
was changed between revision A and revision B. If the data format isn't
amenable to diff'ing then that is a big strike against it. For this
reason, the proposal to gzip the files is Absolutely Right Out; and even
for text-based formats the readability of the files *is* a concern,
whether or not you'd be likely to try to edit them without a diagram
editor.

> I have no dog in this fight, and I'd be overjoyed to see diagrams of
> any sort.

I'm not backing any specific solution either. But Rafael's summary
ignores some requirements that I believe are significant.

regards, tom lane

On Jul 11, 2010, at 9:26 AM, Tom Lane wrote:

> Steve Atkins <steve(at)blighty(dot)com> writes:
>> On Jul 11, 2010, at 12:33 AM, Rafael Martinez wrote:
>> [ assorted arguments for and against DIA ]
>
>>> 2) dia-format is too verbose and it uses too much space when it is not
>>> compressed. [vs]
>>> 2) When compressed with gzip, we can get over 90% reduction in size.
>
>>> 3) You can't edit the dia-files in any other way than using DIA [vs]
>>> 3) I cannot see the problem with this. This will be the same with any
>>> other format. Editing a xfig or svg file manually via a text editor is
>>> as tiresome and counterproductive as doing it with a dia file.
>
>> You wouldn't edit any three of those file types in a text editor any more than
>> you'd edit a PNG with a hex editor (i.e. it's possible, and sometimes
>> useful, but not part of your workflow).
>
> More to the point here is that we want to be able to store the diagram
> source files in an SCM (CVS, soon to be git), and one of the main
> reasons for wanting to do that is so that people can see exactly what
> was changed between revision A and revision B. If the data format isn't
> amenable to diff'ing then that is a big strike against it. For this
> reason, the proposal to gzip the files is Absolutely Right Out; and even
> for text-based formats the readability of the files *is* a concern,
> whether or not you'd be likely to try to edit them without a diagram
> editor.

I suspect that any format that's intended to be edited in a GUI editor
is not likely to diff well. Some formats will be better than others, and
some GUI editors will be better than others, but we're never going
to be able to diff -c an image that has been edited as an image in
the same way we can a text file.

A clean, text or xml based format is likely to be the most diffable,
but it's all a bit speculative without trying it and seeing.

Maybe I'll grab a copy of dia and see how things compare this
afternoon.

Cheers,
Steve

On Sun, Jul 11, 2010 at 11:32 AM, Steve Atkins <steve(at)blighty(dot)com> wrote:
> I have no dog in this fight, and I'd be overjoyed to see diagrams of
> any sort. I do think that requiring use of a single, fairly clunky, graphics
> package rather than allowing people who get the urge to create a
> diagram to use whichever graphics program they're familiar with is
> likely to lead to more, and better, documentation.

Unless I'm misunderstanding what you're talking about here, this is a
really, really bad idea. If we let people use whatever editor they
want to generate diagrams, we'll be unable to easily modify those
diagrams when needed. If our version control system contains a
mixture of dia, xfig, Adobe illustrator, etc, etc source files it'll
be a nightmare.

I think it's important to keep in mind here that (a) whatever tool we
pick will become a requirement either for everyone who wants to build
the docs, or at a minimum for those people who want to work with the
images, (b) we will be stuck with it for a very long time, and (c) the
source files that the tool uses will need to be managed by a version
control system. The version control system and the patch files that
we email around are the lifeblood of this project. Any thought that
the quality or quantity of diagrams is more important than how well a
particular system plays with our version control system is, IMHO, 100%
backwards. I spent more time using git, cvs, and related tools than I
do any other utility on the system, with the possible exception of vi.
I suspect many other developers are in the same boat, modulo
s/vi/$EDITOR/.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

On Jul 16, 2010, at 7:20 PM, Robert Haas wrote:

> On Sun, Jul 11, 2010 at 11:32 AM, Steve Atkins <steve(at)blighty(dot)com> wrote:
>> I have no dog in this fight, and I'd be overjoyed to see diagrams of
>> any sort. I do think that requiring use of a single, fairly clunky, graphics
>> package rather than allowing people who get the urge to create a
>> diagram to use whichever graphics program they're familiar with is
>> likely to lead to more, and better, documentation.
>
> Unless I'm misunderstanding what you're talking about here, this is a
> really, really bad idea. If we let people use whatever editor they
> want to generate diagrams, we'll be unable to easily modify those
> diagrams when needed. If our version control system contains a
> mixture of dia, xfig, Adobe illustrator, etc, etc source files it'll
> be a nightmare.

You're misunderstanding.

My suggestion is to use a standard format, rather than a
proprietary format. A standard format can be edited by a
range of different tools, on different platforms. A proprietary
format can only be edited by a single tool (and single tools
that support multiple platforms tend to be mediocre).

Having our VCS containing, as one example, solely SVG
as a vector format, and allowing people to use any image
editor they want to to edit or create that format is
what I'm talking about.

(There are some issues - with diffs and suchlike - with
using multiple tools to edit a single format, but I suspect
they're fairly minor compared to the benefits of using a
standard, documented, open format.)

> I think it's important to keep in mind here that (a) whatever tool we
> pick will become a requirement either for everyone who wants to build
> the docs, or at a minimum for those people who want to work with the
> images,

So lets not pick a tool. Lets use a portable format.

> (b) we will be stuck with it for a very long time, and

So lets pick a simple, widely supported format, so we're not
stuck with one particular tool and it's format, and we can
migrate the content to a new, widely supported format easily
if the issue comes up.

> (c) the
> source files that the tool uses will need to be managed by a version
> control system.

And lets pick a format that's at least moderately text-based and
diff-friendly.

> The version control system and the patch files that
> we email around are the lifeblood of this project. Any thought that
> the quality or quantity of diagrams is more important than how well a
> particular system plays with our version control system is, IMHO, 100%
> backwards. I spent more time using git, cvs, and related tools than I
> do any other utility on the system, with the possible exception of vi.
> I suspect many other developers are in the same boat, modulo
> s/vi/$EDITOR/.

If the end result is crap, or the working files are not maintainable
then that's a bad thing for the project. That's true if the working files are
C source code and the end result is an executable or if the working
files are <some random vector format> and the end result is a
graphic in the docs.

My feeling is that SVG is a decent lowest common denominator. It's
not *great*, but it's very widely supported and can be generated or
edited by pretty much any tool, and can be rendered to PDF or PNG
by standard tools without too much effort (other than careful choice
of fonts in the original format).

I'm not gung-ho about this - if someone comes up with another
great way to generate diagrams, I'm all for it. But choosing a
proprietary format that's only supported by a single tool, and
that renders to horrible quality PNGs is something we'll regret,
I think.

Cheers,
Steve

Steve Atkins <steve(at)blighty(dot)com> writes:
> On Jul 16, 2010, at 7:20 PM, Robert Haas wrote:
>> Unless I'm misunderstanding what you're talking about here, this is a
>> really, really bad idea.

> You're misunderstanding.

Yeah, I think that in the main you guys are in violent agreement.

> My feeling is that SVG is a decent lowest common denominator.

I think this might be true in principle but not necessarily in
practice. For example, if two tools write out SVG with different
whitespace layout, that's going to kill diff-ing and VCS friendliness.
This could probably be worked around, eg by settling on some particular
tool as the one to run it through before committing (sort of a pgindent
for graphics). But we need to do some experimentation with specific
tools and see what the output really looks like and how it changes
given small changes in the diagram, before we choose anything.

regards, tom lane

Tom Lane wrote:
> Steve Atkins <steve(at)blighty(dot)com> writes:
> > On Jul 16, 2010, at 7:20 PM, Robert Haas wrote:
> >> Unless I'm misunderstanding what you're talking about here, this is a
> >> really, really bad idea.
>
> > You're misunderstanding.
>
> Yeah, I think that in the main you guys are in violent agreement.
>
> > My feeling is that SVG is a decent lowest common denominator.
>
> I think this might be true in principle but not necessarily in
> practice. For example, if two tools write out SVG with different
> whitespace layout, that's going to kill diff-ing and VCS friendliness.
> This could probably be worked around, eg by settling on some particular
> tool as the one to run it through before committing (sort of a pgindent
> for graphics). But we need to do some experimentation with specific
> tools and see what the output really looks like and how it changes
> given small changes in the diagram, before we choose anything.

Well, if we want _any_ images for the 9.0 docs, we had better decide
soon.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ None of us is going to be here forever. +

On sön, 2010-07-18 at 00:46 -0400, Bruce Momjian wrote:
> Well, if we want _any_ images for the 9.0 docs, we had better decide
> soon.

I'd be very much against putting any images into 9.0 at this time. I
was working under the assumption that this discussion would be targeting
9.1.

On 18 July 2010 09:36, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
> On sön, 2010-07-18 at 00:46 -0400, Bruce Momjian wrote:
>> Well, if we want _any_ images for the 9.0 docs, we had better decide
>> soon.
>
> I'd be very much against putting any images into 9.0 at this time.  I
> was working under the assumption that this discussion would be targeting
> 9.1.
>
>

+1

Considering that these will have to become a permanent fixture, I
don't expect this to be decided and working in time for 9.0. I
assumed this was for 9.1 too.

There are too many unresolved concerns, and one I think is important
is whichever format is selected, can small changes be made to a
diagram using an editor without massive changes to the file. Like Tom
said, we might have some editors which will behave and just alter a
piece of text we asked to be changed, and others decide it needs extra
markup and rearrange the order in which things are drawn etc. And
should there be a whitelist of editors known to behave which people
are to use?

Thom

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tom Lane wrote:

> But we need to do some experimentation with specific
> tools and see what the output really looks like and how it changes
> given small changes in the diagram, before we choose anything.
>

I did some tests sometime ago for my proposal (DIA) and the result were
sent to the list:
http://archives.postgresql.org/pgsql-docs/2010-04/msg00038.php

I have testet this again and here is the result:

* The XML file used by DIA has a block of xml definitions per object in
the diagram. This block contains all the attributtes of the object, from
color and position to lines style, witdh, etc. In addition to this,
there is a block at the begynning of the xml file with some general
information about the diagram, background, boundaries, etc.

* Changing one of the attributes of an object only change the value of
this attribute in the xml file. Tha same for the global information
attributtes.

* The objects in a diagram are saved always in the same order they were
created. e.g. if you create a diagram with 3 objetcs they will get
asigned an internal ID (00,01,02) and they will continue having this ID
(and position in the xml file) as long as they are not deleted. e.g. If
we delete the second object created, the IDs in the xml file will get
reasigned (00->00 and 02->01)

All this means that xml files generated by DIA er VCS friendliness, they
only change the attributes/objects changed between versions of the
diagram. It can be used without problems to diff the contents and create
patches that modify the last version of the diagram.

For all this to work this way, the xml files generated by DIA can not be
compressed (this is defined when you save a diagram). They will use more
diskspace in the VCS system but very little in the compressed gz/bz
source file distributted with every release. This can be a problem if a
few extra MB in the VCS system is a problem.

With this e-mail and my last summary
http://archives.postgresql.org/pgsql-docs/2010-07/msg00096.php, I think
I do not have more to say about my proposal.

regards,
- --
Rafael Martinez, <r(dot)m(dot)guerrero(at)usit(dot)uio(dot)no>
Center for Information Technology Services
University of Oslo, Norway

PGP Public Key: http://folk.uio.no/rafael/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkxC0d4ACgkQBhuKQurGihTIHQCghn8GRyOuqaZPMauGyCYjUR2j
zkwAoIiok1UITC3zanzbhZkpZKnTNOqr
=wXF5
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Steve Atkins wrote:
> But choosing a
> proprietary format that's only supported by a single tool, and
> that renders to horrible quality PNGs is something we'll regret,
> I think.

I think we should keep to the facts and you are not doing it here.

Can you explain how you got to the conclusion that dia renders to
horrible quality PNGs? Because this not what I and many have experienced
when generating PNG files with dia.

regards,
- --
Rafael Martinez, <r(dot)m(dot)guerrero(at)usit(dot)uio(dot)no>
Center for Information Technology Services
University of Oslo, Norway

PGP Public Key: http://folk.uio.no/rafael/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkxC05sACgkQBhuKQurGihScPQCeKynNzLwvJx4KIrbzJ/+ADDeJ
b7IAnA7e4MYA5gOXJ1ip5nTa0Yeou2PX
=Onxk
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Rafael Martinez wrote:

>
> All this means that xml files generated by DIA er VCS friendliness, they
> only change the attributes/objects changed between versions of the
> diagram. It can be used without problems to diff the contents and create
> patches that modify the last version of the diagram.
>

I should add to this that I don't know how easy will be to resolve a
'VCS conflict' if two different users try to commit new versions with
changes to the same portion of the file.

I suppose it won't be easy to fix this manually if you have many changes
to the same portion of the xml code.

regards,
- --
Rafael Martinez, <r(dot)m(dot)guerrero(at)usit(dot)uio(dot)no>
Center for Information Technology Services
University of Oslo, Norway

PGP Public Key: http://folk.uio.no/rafael/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkxC1hoACgkQBhuKQurGihRhwgCfesdb9DC5AzrlSSluw4MfbkeA
/WsAnjF1+EZB9EtEDcWIwgLtszd/UA0E
=PIBx
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Thom Brown wrote:

> Okay, I just edited an existing dia file: pgclient_server.dia. All I
> did was change the font size of the text "client application" at the
> top of the diagram, and changed the properties of a line so the
> arrowhead was no longer on the end. I used dia 0.97.1 to edit the
> file. The result when saving is attached in the diff file. That's a
> lot of file changes for what I did.
>
> Thom
>

Indeed.

The only explanation I can find to this is that I used dia.0.96.1 to
generate the file and you have used 0.97.1 to edit it.

Almost all changes are to the attributes:
<dia:rectangle val=....
<dia:point val=....

I suspect that these changes have been done because 'something' have
changed between the two main versions in the way they deal with these
attributes :(

regards,
- --
Rafael Martinez, <r(dot)m(dot)guerrero(at)usit(dot)uio(dot)no>
Center for Information Technology Services
University of Oslo, Norway

PGP Public Key: http://folk.uio.no/rafael/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkxC3lsACgkQBhuKQurGihQj4wCfYPdfK9xTNWKGCMruuD5zFtXp
YLYAnj0vlMYE0NbqH8txIcW2U4hpFgtR
=mGdx
-----END PGP SIGNATURE-----

On 18 July 2010 11:58, Rafael Martinez <r(dot)m(dot)guerrero(at)usit(dot)uio(dot)no> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Thom Brown wrote:
>
>> Okay, I just edited an existing dia file: pgclient_server.dia.  All I
>> did was change the font size of the text "client application" at the
>> top of the diagram, and changed the properties of a line so the
>> arrowhead was no longer on the end.  I used dia 0.97.1 to edit the
>> file. The result when saving is attached in the diff file.  That's a
>> lot of file changes for what I did.
>>
>> Thom
>>
>
> Indeed.
>
> The only explanation I can find to this is that I used dia.0.96.1 to
> generate the file and you have used 0.97.1 to edit it.
>
> Almost all changes are to the attributes:
>  <dia:rectangle val=....
>  <dia:point val=....
>
> I suspect that these changes have been done because 'something' have
> changed between the two main versions in the way they deal with these
> attributes :(

Has anyone yet come up with a reliable and sensible format we can use?

Thom

On 27 September 2010 15:29, Thom Brown <thombrown(at)gmail(dot)com> wrote:
> On 18 July 2010 11:58, Rafael Martinez <r(dot)m(dot)guerrero(at)usit(dot)uio(dot)no> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Thom Brown wrote:
>>
>>> Okay, I just edited an existing dia file: pgclient_server.dia.  All I
>>> did was change the font size of the text "client application" at the
>>> top of the diagram, and changed the properties of a line so the
>>> arrowhead was no longer on the end.  I used dia 0.97.1 to edit the
>>> file. The result when saving is attached in the diff file.  That's a
>>> lot of file changes for what I did.
>>>
>>> Thom
>>>
>>
>> Indeed.
>>
>> The only explanation I can find to this is that I used dia.0.96.1 to
>> generate the file and you have used 0.97.1 to edit it.
>>
>> Almost all changes are to the attributes:
>>  <dia:rectangle val=....
>>  <dia:point val=....
>>
>> I suspect that these changes have been done because 'something' have
>> changed between the two main versions in the way they deal with these
>> attributes :(
>
> Has anyone yet come up with a reliable and sensible format we can use?

Obviously nothing for this happened in time to make it for 9.1. So
are diagrams still something we plan to have?

Thom

Thom Brown wrote:
> On 27 September 2010 15:29, Thom Brown <thombrown(at)gmail(dot)com> wrote:
> > On 18 July 2010 11:58, Rafael Martinez <r(dot)m(dot)guerrero(at)usit(dot)uio(dot)no> wrote:
> >> -----BEGIN PGP SIGNED MESSAGE-----
> >> Hash: SHA1
> >>
> >> Thom Brown wrote:
> >>
> >>> Okay, I just edited an existing dia file: pgclient_server.dia. ?All I
> >>> did was change the font size of the text "client application" at the
> >>> top of the diagram, and changed the properties of a line so the
> >>> arrowhead was no longer on the end. ?I used dia 0.97.1 to edit the
> >>> file. The result when saving is attached in the diff file. ?That's a
> >>> lot of file changes for what I did.
> >>>
> >>> Thom
> >>>
> >>
> >> Indeed.
> >>
> >> The only explanation I can find to this is that I used dia.0.96.1 to
> >> generate the file and you have used 0.97.1 to edit it.
> >>
> >> Almost all changes are to the attributes:
> >> ?<dia:rectangle val=....
> >> ?<dia:point val=....
> >>
> >> I suspect that these changes have been done because 'something' have
> >> changed between the two main versions in the way they deal with these
> >> attributes :(
> >
> > Has anyone yet come up with a reliable and sensible format we can use?
>
> Obviously nothing for this happened in time to make it for 9.1. So
> are diagrams still something we plan to have?

Agreed, we need to move forward with something, and I am afraid we got
into bike-sheeting on this. Would someone please summarize the previous
discussion so we can make some decisions and move forward?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On Sun, 2011-06-12 at 17:44 -0400, Bruce Momjian wrote:
> Thom Brown wrote:
> >
> > Obviously nothing for this happened in time to make it for 9.1. So
> > are diagrams still something we plan to have?
>
> Agreed, we need to move forward with something, and I am afraid we got
> into bike-sheeting on this. Would someone please summarize the previous
> discussion so we can make some decisions and move forward?
>

Hei Bruce

I will try to summarize the previous discusion.

This discusion was about creating explanatory diagrams that could be
used to improve the PostgreSQL documentation.

These are some of the things we would like when creating diagrams:
------------------------------------------------------------------

* A program to create and manipulate diagrams that is easy to use and
can be used on Unix, Windows, or Mac.

* We need a format that is easy to maintain, that can check into VCS
and that will provide tools for automatically converting to a variety of
target formats.

* The requirements to create/convert to a format that can be used in the
documentation must be kept to a minimum.

* We do not want these diagrams to use to much disk space. Compression
is not an alternative.

The discusion started suggesting DIA [1]. This program uses its own XML
representation to define a diagram. More information about what was said
in the begynning about this:
http://archives.postgresql.org/pgsql-docs/2010-04/msg00038.php

The main objections and feedback about using DIA were:
------------------------------------------------------

1) DIA has a terrible UI.

2) dia-format is too verbose and it uses too much space when it is not
compressed.

3) You can't edit the dia-files in any other way than using DIA

4) PNG files generated by DIA are too large.

5) Not sure 100% that a change in a diagram will not affect code
everywhere in the XML representation of the diagram. This will make
difficult to diff between version in a VCS system.

I think the main problems in general were 3) and 5), specially 5)

XFIG and SVG were proposed as an alternative to DIA files. But no
further job was done to properly check these formats.

Good feedback about SVG was done here:
http://archives.postgresql.org/pgsql-docs/2010-07/msg00097.php

It refers among other things to Inkscape [2]. This program generates SVG
diagrams and is not so difficult to use. One of the good things with
Inkscape is that it works with a standard format so we are not bound to
a specific program or non standard format.

And the discussion stopped here.

The main problem I see is that all these formats that can be used to
generate quality diagrams are difficult, if not impossible, to
manipulate outside a WYSWYG program. So probably it will be difficult to
get good diffs between versions of a diagram.

[1] http://live.gnome.org/Dia/
[2] http://inkscape.org/

--
Rafael Martinez Guerrero
Center for Information Technology
University of Oslo, Norway

PGP Public Key: http://folk.uio.no/rafael/

On 06/13/2011 09:36 AM, Rafael Martinez wrote:
> It refers among other things to Inkscape [2]. This program generates SVG
> diagrams and is not so difficult to use. One of the good things with
> Inkscape is that it works with a standard format so we are not bound to
> a specific program or non standard format.
>

I've been staying out of this discussion so far because I truly hate
dia, and was waiting for others to decide against it too. Inkscape
saving to SVG is a much more reasonable choice. I just created a
drawing, saved it, then modified it a bit. The spurious diff from the
GUI was quite small: three lines of junk with the filename change and
some windowing metadata. And the new material added showed in a pretty
readable diff as I would hope too. Original file and diff attached as
samples.

The main issue I've seen with SVG is that it doesn't render the same way
in every program. The attached drawing1.svg shows up in Inkscape with a
text box and the word "stuff" in the middle. But if I open it in the
standard desktop viewer on my Linux system, "Eye of GNOME", or in
OpenOffice Draw, it shows a black box where the text is supposed to be.
(OpenOffice Draw can read some SVG files, but it can't save in that
format. Improving OOO support for SVG on import and export is a high
priority feature for the project though:
http://wiki.services.openoffice.org/wiki/SVG_Import_Filter ,
http://graphics.openoffice.org/svg/svg.htm )

I think that any SVG diagram might need to have QA that includes opening
it in more than one program, to confirm the file isn't using a fuzzy
feature in the standard that renders differently. Maybe we don't care,
and the fact that it renders correctly in whatever tool is used to
generate the docs is all that matters. But SVG is as good candidate for
a storage format as we're likely to find. I've been trying to find an
open replacement for the sort of diagrams I used to draw all the time in
Visio in the late 90's, and tools like Inkscape using SVG have finally
started to look good enough for me to consider using seriously in the
last two years. OpenOffice completely sorting out their issues with the
format is really the last blocker for my own purposes--it becomes much
easier for me to justify making these when I can easily pull them in and
out of presentation slides--but that may not be relevant to the
community documentation efforts the way it is to my personal work.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

On Mon, Jun 13, 2011 at 10:35:41AM -0400, Greg Smith wrote:
> On 06/13/2011 09:36 AM, Rafael Martinez wrote:
> >It refers among other things to Inkscape [2]. This program generates SVG
> >diagrams and is not so difficult to use. One of the good things with
> >Inkscape is that it works with a standard format so we are not bound to
> >a specific program or non standard format.
>
> I've been staying out of this discussion so far because I truly hate
> dia, and was waiting for others to decide against it too. Inkscape
> saving to SVG is a much more reasonable choice. I just created a
> drawing, saved it, then modified it a bit. The spurious diff from
> the GUI was quite small: three lines of junk with the filename
> change and some windowing metadata. And the new material added
> showed in a pretty readable diff as I would hope too. Original file
> and diff attached as samples.
>
> The main issue I've seen with SVG is that it doesn't render the same
> way in every program. The attached drawing1.svg shows up in
> Inkscape with a text box and the word "stuff" in the middle. But if
> I open it in the standard desktop viewer on my Linux system, "Eye of
> GNOME", or in OpenOffice Draw, it shows a black box where the text
> is supposed to be. (OpenOffice Draw can read some SVG files, but it
> can't save in that format. Improving OOO support for SVG on import
> and export is a high priority feature for the project though:
> http://wiki.services.openoffice.org/wiki/SVG_Import_Filter ,
> http://graphics.openoffice.org/svg/svg.htm )
>
> I think that any SVG diagram might need to have QA that includes
> opening it in more than one program, to confirm the file isn't using
> a fuzzy feature in the standard that renders differently. Maybe we
> don't care, and the fact that it renders correctly in whatever tool
> is used to generate the docs is all that matters. But SVG is as
> good candidate for a storage format as we're likely to find. I've
> been trying to find an open replacement for the sort of diagrams I
> used to draw all the time in Visio in the late 90's, and tools like
> Inkscape using SVG have finally started to look good enough for me
> to consider using seriously in the last two years. OpenOffice
> completely sorting out their issues with the format is really the
> last blocker for my own purposes--it becomes much easier for me to
> justify making these when I can easily pull them in and out of
> presentation slides--but that may not be relevant to the community
> documentation efforts the way it is to my personal work.

Just want to add a +1 to using SVG as an vector graphics
intermediate-exchange format that is amendable to manual
tweaking/cleanup via Inkscape. I routinely generate graphs from gnuplot
into svg, pull them into Inkscape to add notation and tweak labels, and
generate the PNG versions from there. Inkscape runs as a commandline
tool as well - in fact if I find need the same graph multiple times,
I'll workout the arcana to add the labels via gnuplot (or via a sed step
on the svg output) and completely automate the figure generation.

Ross
--
Ross Reedstrom, Ph.D. reedstrm(at)rice(dot)edu
Systems Engineer & Admin, Research Scientist phone: 713-348-6166
Connexions http://cnx.org fax: 713-348-3665
Rice University MS-375, Houston, TX 77005
GPG Key fingerprint = F023 82C8 9B0E 2CC6 0D8E F888 D3AE 810E 88F0 BEDE

On 13 June 2011 14:35, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:
> I just created a drawing, saved it, then
> modified it a bit.  The spurious diff from the GUI was quite small:  three
> lines of junk with the filename change and some windowing metadata.  And the
> new material added showed in a pretty readable diff as I would hope too.
>  Original file and diff attached as samples.

It appears you've saved it in the Inkscape SVG format as it uses
Inkscape markup. It's probably better to save it as a plain SVG since
other editors may remove the Inkscape-specific markup anyway.

I just did a quick test drawing a squiggly line in Inkscape, saved it
in both formats. The Inkscape SVG takes up 60 lines, the plain one is
35. I opened both in LibreOffice and exported them back to new SVG
files and the file lengths were further reduced, although both outputs
were identical to one another (the Inkscape format
LibreOffice-converted file identical to the plain format
LibreOffice-converted file). But this time it removed all line breaks
and became one continuous line of markup. So that would be disastrous
from a delta point of view.

--
Thom Brown
Twitter: @darkixion
IRC (freenode): dark_ixion
Registered Linux user: #516935

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 06/13/2011 07:35 AM, Greg Smith wrote:
> On 06/13/2011 09:36 AM, Rafael Martinez wrote:
>> It refers among other things to Inkscape [2]. This program generates SVG
>> diagrams and is not so difficult to use. One of the good things with
>> Inkscape is that it works with a standard format so we are not bound to
>> a specific program or non standard format.
>

As an FYI, Open/Libre/ApacheOffice can export to svg as wel.

JD

--
Command Prompt, Inc. - http://www.commandprompt.com/
PostgreSQL Support, Training, Professional Services and Development
The PostgreSQL Conference - http://www.postgresqlconference.org/
@cmdpromptinc - @postgresconf - 509-416-6579

On Mon, Jun 13, 2011 at 06:36:12PM +0000, Thom Brown wrote:
> On 13 June 2011 14:35, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:
> > I just created a drawing, saved it, then
> > modified it a bit.  The spurious diff from the GUI was quite small:  three
> > lines of junk with the filename change and some windowing metadata.  And the
> > new material added showed in a pretty readable diff as I would hope too.
> >  Original file and diff attached as samples.
>
> It appears you've saved it in the Inkscape SVG format as it uses
> Inkscape markup. It's probably better to save it as a plain SVG since
> other editors may remove the Inkscape-specific markup anyway.
>
> I just did a quick test drawing a squiggly line in Inkscape, saved it
> in both formats. The Inkscape SVG takes up 60 lines, the plain one is
> 35. I opened both in LibreOffice and exported them back to new SVG
> files and the file lengths were further reduced, although both outputs
> were identical to one another (the Inkscape format
> LibreOffice-converted file identical to the plain format
> LibreOffice-converted file). But this time it removed all line breaks
> and became one continuous line of markup. So that would be disastrous
> from a delta point of view.

It's fairly easy to put those back in w/ xmllint --format. Thom, what do
you get for:

xmllint --format for_test_here.svg | wc -l

However, my copy of OpenOffice.org (3.2.1.4) does evil things to a
pencil squiggle drawn in Inkscape. The original plain svg has a single
path (albeit w/ 451 points). After passing through ooodraw, that is
broken into 760 path elements. A quick google tells me that svg export
is one of the areas that LibreOffice has advanced in. That probably
explains the discrepancy.

So, just saying 'svg' isn't enough: we'd likely need to specify Inkscape
as the one true svg processor.

Ross
--
Ross Reedstrom, Ph.D. reedstrm(at)rice(dot)edu
Systems Engineer & Admin, Research Scientist phone: 713-348-6166
Connexions http://cnx.org fax: 713-348-3665
Rice University MS-375, Houston, TX 77005
GPG Key fingerprint = F023 82C8 9B0E 2CC6 0D8E F888 D3AE 810E 88F0 BEDE

On 06/13/2011 04:59 PM, Ross J. Reedstrom wrote:
> However, my copy of OpenOffice.org (3.2.1.4) does evil things to a
> pencil squiggle drawn in Inkscape.

That's why I referred to the lists of open issues in this area. It's
possible to get various levels of SVG support working in
OpenOffice/LibreOffice, and some tests may show success. But it's very
fragile and the quality of support you get quite variable based on your
version/distribution. OOO might be a viable SVG processor in the
future, which is good news for deciding to adopt that format. But for
now, it's better to avoid passing files through it.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

On 13 June 2011 20:59, Ross J. Reedstrom <reedstrm(at)rice(dot)edu> wrote:
> On Mon, Jun 13, 2011 at 06:36:12PM +0000, Thom Brown wrote:
>> On 13 June 2011 14:35, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:
>> > I just created a drawing, saved it, then
>> > modified it a bit.  The spurious diff from the GUI was quite small:  three
>> > lines of junk with the filename change and some windowing metadata.  And the
>> > new material added showed in a pretty readable diff as I would hope too.
>> >  Original file and diff attached as samples.
>>
>> It appears you've saved it in the Inkscape SVG format as it uses
>> Inkscape markup.  It's probably better to save it as a plain SVG since
>> other editors may remove the Inkscape-specific markup anyway.
>>
>> I just did a quick test drawing a squiggly line in Inkscape, saved it
>> in both formats.  The Inkscape SVG takes up 60 lines, the plain one is
>> 35.  I opened both in LibreOffice and exported them back to new SVG
>> files and the file lengths were further reduced, although both outputs
>> were identical to one another (the Inkscape format
>> LibreOffice-converted file identical to the plain format
>> LibreOffice-converted file).  But this time it removed all line breaks
>> and became one continuous line of markup.  So that would be disastrous
>> from a delta point of view.
>
> It's fairly easy to put those back in w/ xmllint --format. Thom, what do
> you get for:
>
> xmllint --format for_test_here.svg | wc -l

That restores some of the newlines, but inkscape seems to place tag
attributes on their own separate lines for readability, so the outputs
still differ, unless you pass both through that same process.
Comparing two processed outputs still reveals a difference, such as a
comment in the original saying "Created with Inkscape..." and some
attributes not preserved by LibreOffice, including namespaces Inkscape
uses that LibreOffice doesn't.

--
Thom Brown
Twitter: @darkixion
IRC (freenode): dark_ixion
Registered Linux user: #516935

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Mon, Jun 13, 2011 at 09:17:04PM +0000, Thom Brown wrote:
> >
> > xmllint --format for_test_here.svg | wc -l
>
> That restores some of the newlines, but inkscape seems to place tag
> attributes on their own separate lines for readability, so the outputs
> still differ, unless you pass both through that same process.
> Comparing two processed outputs still reveals a difference, such as a
> comment in the original saying "Created with Inkscape..." and some
> attributes not preserved by LibreOffice, including namespaces Inkscape
> uses that LibreOffice doesn't.
>

Those sort of things, if they don't dominate the diff, are
pretty easy to skim past, as long as larger scale changes (like altering
the paths, as described above) don't interfere. I think anything beyond
hand-crufted xfig is going to have some less-than-absolute minimal
changes. And there are only two of us are left who know how to hand
cruft xfig!

Ross "hyperbole for effect" Reedstrom
--
Ross Reedstrom, Ph.D. reedstrm(at)rice(dot)edu
Systems Engineer & Admin, Research Scientist phone: 713-348-6166
Connexions http://cnx.org fax: 713-348-3665
Rice University MS-375, Houston, TX 77005
GPG Key fingerprint = F023 82C8 9B0E 2CC6 0D8E F888 D3AE 810E 88F0 BEDE

Ross J. Reedstrom wrote:
> On Mon, Jun 13, 2011 at 09:17:04PM +0000, Thom Brown wrote:
> > >
> > > xmllint --format for_test_here.svg | wc -l
> >
> > That restores some of the newlines, but inkscape seems to place tag
> > attributes on their own separate lines for readability, so the outputs
> > still differ, unless you pass both through that same process.
> > Comparing two processed outputs still reveals a difference, such as a
> > comment in the original saying "Created with Inkscape..." and some
> > attributes not preserved by LibreOffice, including namespaces Inkscape
> > uses that LibreOffice doesn't.
> >
>
> Those sort of things, if they don't dominate the diff, are
> pretty easy to skim past, as long as larger scale changes (like altering
> the paths, as described above) don't interfere. I think anything beyond
> hand-crufted xfig is going to have some less-than-absolute minimal
> changes. And there are only two of us are left who know how to hand
> cruft xfig!

Agreed on the simplicity of manually modifying xfig files. :-)

I think we are agreed on using the SVG format to check images into git.
I think the unknown is which SVG editor XML file layouts we will
support.

I would like xfig's SVG export format to be supported as a way for me to
import my images into our docs, and it seems Greg Smith has similar
needs. I don't see that xfig can import SVG so I think it will be a
one-way operation.

I think the big question is whether we can afford to allow multiple SVG
editor file formats to be checked in, and hence support git diff churn
as we switch SVG editors for commits. This doesn't seem likely to
improve anytime soon so we should just decide and move forward.

Perhaps we should allow any SVG file to be imported, but only allow
modifications by a single SVG editor.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

Bruce Momjian <bruce(at)momjian(dot)us> writes:
> I think the big question is whether we can afford to allow multiple SVG
> editor file formats to be checked in, and hence support git diff churn
> as we switch SVG editors for commits. This doesn't seem likely to
> improve anytime soon so we should just decide and move forward.

> Perhaps we should allow any SVG file to be imported, but only allow
> modifications by a single SVG editor.

How is that sensible? The first change to the file will result in
exactly the mass of cosmetic diffs that we wish to avoid.

I think we probably have to specify the One True Graphics Editor, and
maybe even specific version(s) of it, if we want to avoid unreadable
diffs. The reason we have not managed to come to a consensus yet is
that people would prefer not to do that, but it's becoming obvious that
there just isn't any way around it.

regards, tom lane

Tom Lane wrote:
> Bruce Momjian <bruce(at)momjian(dot)us> writes:
> > I think the big question is whether we can afford to allow multiple SVG
> > editor file formats to be checked in, and hence support git diff churn
> > as we switch SVG editors for commits. This doesn't seem likely to
> > improve anytime soon so we should just decide and move forward.
>
> > Perhaps we should allow any SVG file to be imported, but only allow
> > modifications by a single SVG editor.
>
> How is that sensible? The first change to the file will result in
> exactly the mass of cosmetic diffs that we wish to avoid.

I suggested this so we could get images into the docs and then only have
a single diff change once the file is modified. If we don't do that
then all image additions have to be done by someone with the proper
editor.

> I think we probably have to specify the One True Graphics Editor, and
> maybe even specific version(s) of it, if we want to avoid unreadable
> diffs. The reason we have not managed to come to a consensus yet is
> that people would prefer not to do that, but it's becoming obvious that
> there just isn't any way around it.

Yes.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

Bruce Momjian <bruce(at)momjian(dot)us> writes:
> Tom Lane wrote:
>> Bruce Momjian <bruce(at)momjian(dot)us> writes:
>>> Perhaps we should allow any SVG file to be imported, but only allow
>>> modifications by a single SVG editor.

>> How is that sensible? The first change to the file will result in
>> exactly the mass of cosmetic diffs that we wish to avoid.

> I suggested this so we could get images into the docs and then only have
> a single diff change once the file is modified. If we don't do that
> then all image additions have to be done by someone with the proper
> editor.

Or at least, somebody passes it through the "proper editor" before
committing. As long as said editor can read SVG output from a
reasonable range of other tools, this doesn't seem that onerous.

regards, tom lane

On Mon, Jun 13, 2011 at 8:46 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Bruce Momjian <bruce(at)momjian(dot)us> writes:
>> Tom Lane wrote:
>>> Bruce Momjian <bruce(at)momjian(dot)us> writes:
>>>> Perhaps we should allow any SVG file to be imported, but only allow
>>>> modifications by a single SVG editor.
>
>>> How is that sensible?  The first change to the file will result in
>>> exactly the mass of cosmetic diffs that we wish to avoid.
>
>> I suggested this so we could get images into the docs and then only have
>> a single diff change once the file is modified.  If we don't do that
>> then all image additions have to be done by someone with the proper
>> editor.
>
> Or at least, somebody passes it through the "proper editor" before
> committing.  As long as said editor can read SVG output from a
> reasonable range of other tools, this doesn't seem that onerous.

Bad news, Bruce: you're going to have to learn a new tool. :-)

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas wrote:
> On Mon, Jun 13, 2011 at 8:46 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > Bruce Momjian <bruce(at)momjian(dot)us> writes:
> >> Tom Lane wrote:
> >>> Bruce Momjian <bruce(at)momjian(dot)us> writes:
> >>>> Perhaps we should allow any SVG file to be imported, but only allow
> >>>> modifications by a single SVG editor.
> >
> >>> How is that sensible? ?The first change to the file will result in
> >>> exactly the mass of cosmetic diffs that we wish to avoid.
> >
> >> I suggested this so we could get images into the docs and then only have
> >> a single diff change once the file is modified. ?If we don't do that
> >> then all image additions have to be done by someone with the proper
> >> editor.
> >
> > Or at least, somebody passes it through the "proper editor" before
> > committing. ?As long as said editor can read SVG output from a
> > reasonable range of other tools, this doesn't seem that onerous.
>
> Bad news, Bruce: you're going to have to learn a new tool. :-)

Yes, I am sensing that. ;-)

Do we have any agreement on which tool will be the definitive one?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On Tue, Jun 14, 2011 at 08:59:41AM -0400, Bruce Momjian wrote:
> Robert Haas wrote:
> > On Mon, Jun 13, 2011 at 8:46 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > >
> > > Or at least, somebody passes it through the "proper editor" before
> > > committing. ?As long as said editor can read SVG output from a
> > > reasonable range of other tools, this doesn't seem that onerous.

This scenario depends on the idea that the tool in question will serve
as an SVG normalizer. From what I can gather, none of the tools seem to
manage to do that. Or at least, do it in such a way as not to make
hamburger of the output SVG.

> > Bad news, Bruce: you're going to have to learn a new tool. :-)
>
> Yes, I am sensing that. ;-)
>
> Do we have any agreement on which tool will be the definitive one?

For my money, Inkscape seems to be the leader in this space (open source
generic SVG editor), and is portable across all three major platforms:
Unixes, Windows, and MacOS. I think it's even in the BSD ports collections,
Bruce. :-)

I would dearly love to have an SVG normalizer, but it seems no one has
done the deed.

Ross
--
Ross Reedstrom, Ph.D. reedstrm(at)rice(dot)edu
Systems Engineer & Admin, Research Scientist phone: 713-348-6166
Connexions http://cnx.org fax: 713-348-3665
Rice University MS-375, Houston, TX 77005
GPG Key fingerprint = F023 82C8 9B0E 2CC6 0D8E F888 D3AE 810E 88F0 BEDE

On Sun, Jun 12, 2011 at 10:08:05PM +0100, Thom Brown wrote:
> On 27 September 2010 15:29, Thom Brown <thombrown(at)gmail(dot)com> wrote:
> > On 18 July 2010 11:58, Rafael Martinez <r(dot)m(dot)guerrero(at)usit(dot)uio(dot)no> wrote:
> >> -----BEGIN PGP SIGNED MESSAGE-----
> >> Hash: SHA1
> >>
> >> Thom Brown wrote:
> >>
> >>> Okay, I just edited an existing dia file: pgclient_server.dia.  All I
> >>> did was change the font size of the text "client application" at the
> >>> top of the diagram, and changed the properties of a line so the
> >>> arrowhead was no longer on the end.  I used dia 0.97.1 to edit the
> >>> file. The result when saving is attached in the diff file.  That's a
> >>> lot of file changes for what I did.
> >>>
> >>> Thom
> >>>
> >>
> >> Indeed.
> >>
> >> The only explanation I can find to this is that I used dia.0.96.1 to
> >> generate the file and you have used 0.97.1 to edit it.
> >>
> >> Almost all changes are to the attributes:
> >>  <dia:rectangle val=....
> >>  <dia:point val=....
> >>
> >> I suspect that these changes have been done because 'something' have
> >> changed between the two main versions in the way they deal with these
> >> attributes :(
> >
> > Has anyone yet come up with a reliable and sensible format we can use?
>
> Obviously nothing for this happened in time to make it for 9.1. So
> are diagrams still something we plan to have?

One year later, we have made no progress on diagrams in our
documentation. Do we at least have an agreed-up approach?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +