Re: Can extension build own SGML document?

Hello,

The pgxs makefile (pgxs.ml) says:

# DOCS -- random files to install under $PREFIX/doc/$MODULEDIR

It is a bunch of files to be copied to document directory, however,
not looks like a variable to specify SGML source as PostgreSQL core
doing.

Do we have way to build SGML source of extension using makefile?

Thanks,
--
NEC Business Creation Division / PG-Strom Project
KaiGai Kohei <kaigai(at)ak(dot)jp(dot)nec(dot)com>

On 9/14/15 5:35 AM, Kouhei Kaigai wrote:
> Hello,
>
> The pgxs makefile (pgxs.ml) says:
>
> # DOCS -- random files to install under $PREFIX/doc/$MODULEDIR
>
> It is a bunch of files to be copied to document directory, however,
> not looks like a variable to specify SGML source as PostgreSQL core
> doing.
>
> Do we have way to build SGML source of extension using makefile?

No.

We could, but a project that's large enough to want this kind of thing
will likely have its own ideas and requirements, so it unclear how
difficult it would be to support that.

Peter Eisentraut wrote:
> On 9/14/15 5:35 AM, Kouhei Kaigai wrote:
> > Hello,
> >
> > The pgxs makefile (pgxs.ml) says:
> >
> > # DOCS -- random files to install under $PREFIX/doc/$MODULEDIR
> >
> > It is a bunch of files to be copied to document directory, however,
> > not looks like a variable to specify SGML source as PostgreSQL core
> > doing.
> >
> > Do we have way to build SGML source of extension using makefile?
>
> No.
>
> We could, but a project that's large enough to want this kind of thing
> will likely have its own ideas and requirements, so it unclear how
> difficult it would be to support that.

FWIW bdr has such a thing. Mostly, they replaced postgres.sgml with
bdr.sgml and stripped down and edited the Makefile to match. The
resulting doc is a standalone "book" (in docbook parlance). You can see
the source code here
http://git.postgresql.org/gitweb/?p=2ndquadrant_bdr.git;a=tree;f=doc;hb=HEAD
Note there's no support for anything other than HTML --- no PDF, no
EPUB, etc.

The resulting HTML is published as http://bdr-project.org/docs/stable/

I think the only way upstream Postgres could offer this is as a way to
build a separate "book", i.e. not a chapter/section within the main
book. I think it would require huge complications in doc/src/sgml's
Makefile. Not sure it's worthwhile.

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Tue, Sep 15, 2015 at 6:01 AM, Alvaro Herrera wrote:
> I think the only way upstream Postgres could offer this is as a way to
> build a separate "book", i.e. not a chapter/section within the main
> book. I think it would require huge complications in doc/src/sgml's
> Makefile. Not sure it's worthwhile.

I am not sure either, and as each project/developer have always
different requirements I am convinced that this is going to be
finished with enforced rules in Makefile rules for sure, so it is
really unclear what would be the potential benefit to have a
centralized facility. Take for example man pages, those should not be
installed in share/doc/extension/ which is the default path, but in
$(DESTDIR)$(mandir)/man1...
--
Michael

On 9/14/15 6:06 PM, Michael Paquier wrote:
> On Tue, Sep 15, 2015 at 6:01 AM, Alvaro Herrera wrote:
>> I think the only way upstream Postgres could offer this is as a way to
>> build a separate "book", i.e. not a chapter/section within the main
>> book. I think it would require huge complications in doc/src/sgml's
>> Makefile. Not sure it's worthwhile.

I'm not sure SGML is the way to go anymore anyways. Asciidoc offers a
lot of what our SGML does in a much easier to support toolchain. It's
also natively supported by github, which makes it nice for others to
view the output (see [1] as an exmaple). If asciidoc isn't powerful
enough for what you need you can switch to asciidoctor which is even
more powerful[2].

> I am not sure either, and as each project/developer have always
> different requirements I am convinced that this is going to be
> finished with enforced rules in Makefile rules for sure, so it is
> really unclear what would be the potential benefit to have a
> centralized facility. Take for example man pages, those should not be
> installed in share/doc/extension/ which is the default path, but in
> $(DESTDIR)$(mandir)/man1...

I do wish there was better infrastructure for creating extensions, but I
agree that the main project is not the way to handle that. There needs
to be more than just a Makefile you can include too. In particular, the
test framework is pretty ugly to deal with (unless you really like
wading through regress.c diff output...) Bumping extension versions,
creating distfiles and uploading to PGXN are also ripe for automation.

pgxn-utils makes an attempt at this by creating an extension template
for you to build from, but that's ultimately problematic because there's
no way to pull in improvements to the overall infrastructure (such as
how to create manpages). I think something you can pull in via a git
subtree might work better, at least for a makefile framework. I've been
meaning to create that but haven't found the time yet.

[1]
https://github.com/decibel/trunklet-format/blob/master/doc/trunklet-format.asc
[2] http://asciidoctor.org/docs/asciidoc-asciidoctor-diffs/
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com

Jim Nasby <Jim(dot)Nasby(at)BlueTreble(dot)com> writes:
> I'm not sure SGML is the way to go anymore anyways. Asciidoc offers a
> lot of what our SGML does in a much easier to support toolchain. It's
> also natively supported by github, which makes it nice for others to
> view the output (see [1] as an exmaple). If asciidoc isn't powerful
> enough for what you need you can switch to asciidoctor which is even
> more powerful[2].

AFAICT from a quick look at its documentation, asciidoc can produce
either html or docbook output; so as soon as you want something other
than html output (in particular, PDF), you're back to relying on the
exact same creaky docbook toolchain we use now. Only with one extra
dependency in front of it.

Personally I never look at anything but the HTML rendering, but I doubt
that dropping support for all other output formats would fly :-(

I do agree that the SGML toolchain is getting pretty long in the tooth
and we need to be looking for something else.

regards, tom lane

On 9/15/15 8:43 AM, Tom Lane wrote:
> Jim Nasby <Jim(dot)Nasby(at)BlueTreble(dot)com> writes:
>> I'm not sure SGML is the way to go anymore anyways. Asciidoc offers a
>> lot of what our SGML does in a much easier to support toolchain. It's
>> also natively supported by github, which makes it nice for others to
>> view the output (see [1] as an exmaple). If asciidoc isn't powerful
>> enough for what you need you can switch to asciidoctor which is even
>> more powerful[2].
>
> AFAICT from a quick look at its documentation, asciidoc can produce
> either html or docbook output; so as soon as you want something other
> than html output (in particular, PDF), you're back to relying on the
> exact same creaky docbook toolchain we use now. Only with one extra
> dependency in front of it.
>
> Personally I never look at anything but the HTML rendering, but I doubt
> that dropping support for all other output formats would fly :-(
>
> I do agree that the SGML toolchain is getting pretty long in the tooth
> and we need to be looking for something else.

I wasn't thinking of trying to replace the Postgres toolchain, but...

a2x (http://www.methods.co.nz/asciidoc/a2x.1.html) states that it can
generate "PDF, EPUB, DVI, PS, LaTeX, XHTML (single page or chunked), man
page, HTML Help or plain text formats using asciidoc(1) and other
applications (see REQUISITES section). SOURCE_FILE can also be a DocBook
file with an .xml extension."

What I expect would be a lot more effort is actually converting all the
SGML to asciidoc. A quick google search doesn't turn up anything promising.

If the only bad part of our current toolchain is PDF then perhaps we
should just rethink how that's being generated. Since a2x can take
docbook input maybe that's the way to go. But my understanding was that
we've modified docbook and that's where part of the pain is coming from?
In that case a full-out migration to asciidoc might be better.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com

Jim Nasby <Jim(dot)Nasby(at)bluetreble(dot)com> writes:
> On 9/15/15 8:43 AM, Tom Lane wrote:
>> AFAICT from a quick look at its documentation, asciidoc can produce
>> either html or docbook output; so as soon as you want something other
>> than html output (in particular, PDF), you're back to relying on the
>> exact same creaky docbook toolchain we use now. Only with one extra
>> dependency in front of it.

> a2x (http://www.methods.co.nz/asciidoc/a2x.1.html) states that it can
> generate "PDF, EPUB, DVI, PS, LaTeX, XHTML (single page or chunked), man
> page, HTML Help or plain text formats using asciidoc(1) and other
> applications (see REQUISITES section). SOURCE_FILE can also be a DocBook
> file with an .xml extension."

AFAICS, for all cases other than HTML output, the "other applications"
are basically the docbook toolchain.

> What I expect would be a lot more effort is actually converting all the
> SGML to asciidoc. A quick google search doesn't turn up anything promising.

Yeah, the cost of conversion means we're not likely to want to experiment
to see what's better :-(.

regards, tom lane

On Tue, Sep 15, 2015 at 9:43 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> AFAICT from a quick look at its documentation, asciidoc can produce
> either html or docbook output; so as soon as you want something other
> than html output (in particular, PDF), you're back to relying on the
> exact same creaky docbook toolchain we use now. Only with one extra
> dependency in front of it.
>
> Personally I never look at anything but the HTML rendering, but I doubt
> that dropping support for all other output formats would fly :-(

Just out of curiosity, really?

I mean, I can't see that building a PDF of the documentation really
has much value, and I don't know even what else we can build. Who in
2015 would use a PDF instead of HTML?

(If there is somebody, that is fine. But I am curious who it is and
why, because it seems to me like it would just be a nuisance.)

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, Sep 15, 2015 at 8:45 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> I mean, I can't see that building a PDF of the documentation really
> has much value, and I don't know even what else we can build. Who in
> 2015 would use a PDF instead of HTML?
>
> (If there is somebody, that is fine. But I am curious who it is and
> why, because it seems to me like it would just be a nuisance.)

Paper publication, say that for example:
http://www.network-theory.co.uk/postgresql9/vol1a/
I imagine that this applies as well to other languages as it is easier
to maintain a translation of the documentation by hacking the sgml
code.
--
Michael

Michael Paquier <michael(dot)paquier(at)gmail(dot)com> writes:
> On Tue, Sep 15, 2015 at 8:45 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> I mean, I can't see that building a PDF of the documentation really
>> has much value, and I don't know even what else we can build. Who in
>> 2015 would use a PDF instead of HTML?
>>
>> (If there is somebody, that is fine. But I am curious who it is and
>> why, because it seems to me like it would just be a nuisance.)

> Paper publication, say that for example:
> http://www.network-theory.co.uk/postgresql9/vol1a/

Peter's done work somewhat recently on the output of EPUB format,
so I assume he thinks there's still interest in that.

regards, tom lane

Tom Lane wrote:
> Michael Paquier <michael(dot)paquier(at)gmail(dot)com> writes:
> > On Tue, Sep 15, 2015 at 8:45 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> >> I mean, I can't see that building a PDF of the documentation really
> >> has much value, and I don't know even what else we can build. Who in
> >> 2015 would use a PDF instead of HTML?
> >>
> >> (If there is somebody, that is fine. But I am curious who it is and
> >> why, because it seems to me like it would just be a nuisance.)
>
> > Paper publication, say that for example:
> > http://www.network-theory.co.uk/postgresql9/vol1a/
>
> Peter's done work somewhat recently on the output of EPUB format,
> so I assume he thinks there's still interest in that.

Yeah, it's handy for e-readers. I don't think that's too nice for
reference material, but it is for longer prose chapters we have. You
can feed PDFs to e-readers but they don't work nearly as well, and if
you want to use HTML there I think the only way is to convert the HTML
to EPUB first, with worse results than creating the EPUB directly.

As I recall, we also support Info format. That could be useful to Emacs
fans, but I haven't heard of anyone actually using it.

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 9/15/15 11:45 AM, Robert Haas wrote:
> On Tue, Sep 15, 2015 at 9:43 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> AFAICT from a quick look at its documentation, asciidoc can produce
>> either html or docbook output; so as soon as you want something other
>> than html output (in particular, PDF), you're back to relying on the
>> exact same creaky docbook toolchain we use now. Only with one extra
>> dependency in front of it.
>>
>> Personally I never look at anything but the HTML rendering, but I doubt
>> that dropping support for all other output formats would fly :-(
>
> Just out of curiosity, really?

Man pages are in wide use, I think.

> I mean, I can't see that building a PDF of the documentation really
> has much value, and I don't know even what else we can build. Who in
> 2015 would use a PDF instead of HTML?

PDF is actually kind of neat for searches across the whole document.

On Wed, Sep 16, 2015 at 12:16 AM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
> On 9/15/15 11:45 AM, Robert Haas wrote:
>> On Tue, Sep 15, 2015 at 9:43 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> AFAICT from a quick look at its documentation, asciidoc can produce
>>> either html or docbook output; so as soon as you want something other
>>> than html output (in particular, PDF), you're back to relying on the
>>> exact same creaky docbook toolchain we use now. Only with one extra
>>> dependency in front of it.
>>>
>>> Personally I never look at anything but the HTML rendering, but I doubt
>>> that dropping support for all other output formats would fly :-(
>>
>> Just out of curiosity, really?
>
> Man pages are in wide use, I think.

Hmm, yeah.

>> I mean, I can't see that building a PDF of the documentation really
>> has much value, and I don't know even what else we can build. Who in
>> 2015 would use a PDF instead of HTML?
>
> PDF is actually kind of neat for searches across the whole document.

OK.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 9/15/15 10:13 AM, Tom Lane wrote:
> Jim Nasby <Jim(dot)Nasby(at)bluetreble(dot)com> writes:
>> On 9/15/15 8:43 AM, Tom Lane wrote:
>>> AFAICT from a quick look at its documentation, asciidoc can produce
>>> either html or docbook output; so as soon as you want something other
>>> than html output (in particular, PDF), you're back to relying on the
>>> exact same creaky docbook toolchain we use now. Only with one extra
>>> dependency in front of it.
>
>> a2x (http://www.methods.co.nz/asciidoc/a2x.1.html) states that it can
>> generate "PDF, EPUB, DVI, PS, LaTeX, XHTML (single page or chunked), man
>> page, HTML Help or plain text formats using asciidoc(1) and other
>> applications (see REQUISITES section). SOURCE_FILE can also be a DocBook
>> file with an .xml extension."
>
> AFAICS, for all cases other than HTML output, the "other applications"
> are basically the docbook toolchain.

I just started looking at , which seems to be the newer way to handle
asciidoc. Aside from being a lot faster than a2x/asciidoc, it can
produce docbook natively. However...

>> What I expect would be a lot more effort is actually converting all the
>> SGML to asciidoc. A quick google search doesn't turn up anything promising.
>
> Yeah, the cost of conversion means we're not likely to want to experiment
> to see what's better :-(.

If the only concern is handling docbook format (which is what our SGML
docs produce? Then https://github.com/asciidoctor/asciidoctor-fopub
might be an option. It's intended for use with asciidoctor, but the
README does state:

"Using the asciidoctor-fopub project, you can convert any DocBook file
into a nicely formatted PDF with nothing more than a Java runtime (JVM)
and development kit (JDK). All the open source software required to
perform the conversion is automatically fetched from the internet the
first time you run it."

So maybe it would allow removing some of more problematic parts of the
toolchain?

Also, if our SGML does produce docbook as an intermediate it might be
possible to convert that to asciidoc via
https://github.com/oreillymedia/docbook2asciidoc.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com

On 14 October 2015 at 13:04, Jim Nasby <Jim(dot)Nasby(at)bluetreble(dot)com> wrote:

> On 9/15/15 10:13 AM, Tom Lane wrote:
>
>> Jim Nasby <Jim(dot)Nasby(at)bluetreble(dot)com> writes:
>>
>>> On 9/15/15 8:43 AM, Tom Lane wrote:
>>>
>>>> AFAICT from a quick look at its documentation, asciidoc can produce
>>>> either html or docbook output; so as soon as you want something other
>>>> than html output (in particular, PDF), you're back to relying on the
>>>> exact same creaky docbook toolchain we use now. Only with one extra
>>>> dependency in front of it.
>>>>
>>>
>> a2x (http://www.methods.co.nz/asciidoc/a2x.1.html) states that it can
>>> generate "PDF, EPUB, DVI, PS, LaTeX, XHTML (single page or chunked), man
>>> page, HTML Help or plain text formats using asciidoc(1) and other
>>> applications (see REQUISITES section). SOURCE_FILE can also be a DocBook
>>> file with an .xml extension."
>>>
>>
>> AFAICS, for all cases other than HTML output, the "other applications"
>> are basically the docbook toolchain.
>>
>
> I just started looking at , which seems to be the newer way to handle
> asciidoc. Aside from being a lot faster than a2x/asciidoc, it can produce
> docbook natively. However...
>
> What I expect would be a lot more effort is actually converting all the
>>> SGML to asciidoc. A quick google search doesn't turn up anything
>>> promising.
>>>
>>
>> Yeah, the cost of conversion means we're not likely to want to experiment
>> to see what's better :-(.
>>
>
> If the only concern is handling docbook format (which is what our SGML
> docs produce? Then https://github.com/asciidoctor/asciidoctor-fopub might
> be an option. It's intended for use with asciidoctor, but the README does
> state:
>
> "Using the asciidoctor-fopub project, you can convert any DocBook file
> into a nicely formatted PDF with nothing more than a Java runtime (JVM) and
> development kit (JDK). All the open source software required to perform the
> conversion is automatically fetched from the internet the first time you
> run it."
>
> So maybe it would allow removing some of more problematic parts of the
> toolchain?
>
> Also, if our SGML does produce docbook as an intermediate it might be
> possible to convert that to asciidoc via
> https://github.com/oreillymedia/docbook2asciidoc.

There's a misconception there...

Our SGML *is* DocBook. Natively, no translation needed.

DocBook is a document type, and our documentation is already written using
that document type (DOCTYPE). Easily seen thus:

postgres(at)cbbrowne ~/p/d/s/sgml> grep DOCTYPE
postgres.sgml
master!?
<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook V4.2//EN" [

No "intermediate"; we're directly already using DocBook version 4.2.

We'd presumably lose something by a DocBook ---> asciidoc ---> DocBook
transition.

The trouble that we have is that what we have isn't a "DocBook file", but
rather a fairly large set of files representing a DocBook document.

I'm not sure what improvement we'd get out of using asciidoctor-fopub.

There would be some merit to some remapping to transform "creaky old
DocBook 4.2" (what we're using) to a newer version, perhaps biased towards
XML, and have our toolset merge the bits into a big XML (in DocBook 5,
presumably) file for processing using more modern DocBook tools.

I could probably build some DSSSL as helper (my HTML-to-DocBook DSSSL was
highly incomplete, and nonetheless surprisingly widely referenced for
years...), but we'd best be clear on what we think we're getting as
improvement. Switching to a less expressive format is unlikely to be a
win, however creaky the current DocBook/DSSSL tools are.
--
When confronted by a difficult problem, solve it by reducing it to the
question, "How would the Lone Ranger handle this?"

Christopher Browne wrote:

> There would be some merit to some remapping to transform "creaky old
> DocBook 4.2" (what we're using) to a newer version, perhaps biased towards
> XML, and have our toolset merge the bits into a big XML (in DocBook 5,
> presumably) file for processing using more modern DocBook tools.

As I recall, last time we discussed this, we found that the process for
XML docbook took 10 times longer to produce the output files. The XML
toolchain was just too young at the time. It would be nice to know
whether it has aged well, i.e. has the runtime to build our docs
improved?

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services