Re: PDF build issue with 9.0 Alpha5

I was trying to build PDF docs for 9.0 Alpha5, and I got this message:

! TeX capacity exceeded, sorry [number of strings=245830].

I checked texmf.cnf, and all values match the ones that are defined in
our docs. Which setting should I increase? Please note that Alpha4 PDF
was built successfully on the same machine (an up2date Fedora 12).

Regards,
--
Devrim GÜNDÜZ
PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer
PostgreSQL RPM Repository: http://yum.pgrpms.org
Community: devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org Twitter: http://twitter.com/devrimgunduz

Devrim =?ISO-8859-1?Q?G=DCND=DCZ?= <devrim(at)gunduz(dot)org> writes:
> I was trying to build PDF docs for 9.0 Alpha5, and I got this message:

> ! TeX capacity exceeded, sorry [number of strings=245830].

> I checked texmf.cnf, and all values match the ones that are defined in
> our docs. Which setting should I increase? Please note that Alpha4 PDF
> was built successfully on the same machine (an up2date Fedora 12).

This looks rather nasty. As best I can tell, there is a hard limit on
max_strings of 2^18; setting max_strings to more than that in texmf.cnf
does nothing. Changing this would require modifying internal data
representations inside TeX. That seems well beyond what anybody is
going to be willing to do to build the PG docs.

As a short-term expedient to get some alpha5 RPMs built, you could
disable generation of the index (remove "-i include-index" from
JADE.tex.call).

For an actual fix, it looks like we have got three alternatives:

* find out what's chewing up so many strings and get rid of it.

* break the manual into separate TeX documents (losing the ability to
cross-reference).

* abandon TeX-based document output path.

regards, tom lane

On Fri, April 2, 2010 08:20, Devrim GÜNDÜZ wrote:
>
> I was trying to build PDF docs for 9.0 Alpha5, and I got this message:
>
> ! TeX capacity exceeded, sorry [number of strings=245830].
>
> I checked texmf.cnf, and all values match the ones that are defined in
> our docs. Which setting should I increase? Please note that Alpha4 PDF
> was built successfully on the same machine (an up2date Fedora 12).

FWIW...

I can build 'make postgres-A4.pdf' against cvs without problem.

So I tried just now to build 'make postgres-A4.pdf' against alpha5, and it succeeded. I attach my
texmf.cnf, although I can't remember what I changed (if anything), but I do remember getting that
'number of strings' error as well (2 weeks or so ago).

hth

Erik Rijkers

I wrote:
> For an actual fix, it looks like we have got three alternatives:
> * find out what's chewing up so many strings and get rid of it.

Some tracing suggests that there may indeed be a single culprit that we
could fix or nuke. Practically all of the string pool is consumed by
strings like these:

x(at)116714
p(at)116714
x(at)116715
p(at)116715
x(at)116718
p(at)116718
x(at)116719
p(at)116719
x(at)116720
p(at)116720
x(at)116721
p(at)116721
x(at)116722
p(at)116722

These are presumably names of TeX control sequences, but I haven't found
what's defining them.

regards, tom lane

"Erik Rijkers" <er(at)xs4all(dot)nl> writes:
> I can build 'make postgres-A4.pdf' against cvs without problem.

Fails for me :-(. Whose distribution of jadetex are you using, and
what version exactly? I've pretty much convinced myself that the
string consumption is the fault of the FlowObject macros in jadetex.ltx.

regards, tom lane

On Fri, April 2, 2010 23:34, Tom Lane wrote:
> "Erik Rijkers" <er(at)xs4all(dot)nl> writes:
>> I can build 'make postgres-A4.pdf' against cvs without problem.
>
> Fails for me :-(. Whose distribution of jadetex are you using, and
> what version exactly? I've pretty much convinced myself that the
> string consumption is the fault of the FlowObject macros in jadetex.ltx.
>

This is CentOS 5.2,

yum list *jade*
Installed Packages
jadetex.noarch 3.12-13.1.1 installed
openjade.x86_64 1.3.2-27 installed

"Erik Rijkers" <er(at)xs4all(dot)nl> writes:
> This is CentOS 5.2,

> yum list *jade*
> Installed Packages
> jadetex.noarch 3.12-13.1.1 installed
> openjade.x86_64 1.3.2-27 installed

Hmm. Some rooting about in Red Hat's CVS says that that should be
practically indistinguishable from the 3.13 version shipping in current
Fedora. In particular there is no real difference in jadetex.dtx in the
two versions, so they should both eat about the same number of strings.

Could you try this experiment:

1. make postgres-A4.tex-pdf

2. Edit the resulting postgres-A4.tex-pdf to insert a line

\tracingstats=2

at the top.

3. make postgres-A4.pdf

4. In the resulting postgres-A4.log file, find the bit that looks like
this, and send it to the list:

Here is how much of TeX's memory you used:
245829 strings out of 245830
1743620 string characters out of 1810852
628334 words of memory out of 1500000
174597 multiletter control sequences out of 10000+200000
99198 words of font info for 149 fonts, out of 1200000 for 2000
645 hyphenation exceptions out of 8191
28i,13n,45p,1206b,3965s stack positions out of 5000i,500n,6000p,200000b,15000s

It should be near but not quite at the end of the log (there will be
some output from pdftex after it).

regards, tom lane

On Sat, April 3, 2010 00:04, Tom Lane wrote:
> "Erik Rijkers" <er(at)xs4all(dot)nl> writes:
>> This is CentOS 5.2,
>
>> yum list *jade*
>> Installed Packages
>> jadetex.noarch 3.12-13.1.1 installed
>> openjade.x86_64 1.3.2-27 installed
>
> Hmm. Some rooting about in Red Hat's CVS says that that should be
> practically indistinguishable from the 3.13 version shipping in current
> Fedora. In particular there is no real difference in jadetex.dtx in the
> two versions, so they should both eat about the same number of strings.
>
> Could you try this experiment:
>
> 1. make postgres-A4.tex-pdf
>
> 2. Edit the resulting postgres-A4.tex-pdf to insert a line
>
> \tracingstats=2
>
> at the top.
>
> 3. make postgres-A4.pdf
>
> 4. In the resulting postgres-A4.log file, find the bit that looks like
> this, and send it to the list:
>
> Here is how much of TeX's memory you used:
> 245829 strings out of 245830
> 1743620 string characters out of 1810852
> 628334 words of memory out of 1500000
> 174597 multiletter control sequences out of 10000+200000
> 99198 words of font info for 149 fonts, out of 1200000 for 2000
> 645 hyphenation exceptions out of 8191
> 28i,13n,45p,1206b,3965s stack positions out of 5000i,500n,6000p,200000b,15000s
>
Here is how much of TeX's memory you used:
246467 strings out of 246489
1749307 string characters out of 1824395
499695 words of memory out of 1000000
174744 multiletter control sequences out of 10000+200000
99198 words of font info for 149 fonts, out of 500000 for 2000
580 hyphenation exceptions out of 1000
28i,13n,45p,1206b,3963s stack positions out of 1500i,500n,5000p,200000b,15000s
PDF statistics:
113474 PDF objects out of 300000
84838 named destinations out of 131072
31417 words of extra memory for PDF output out of 65536

Some time ago I sucessfully built pdf, but today I got

pg-head(at)mira:~/cvs/HEAD/pgsql/doc/src/sgml$ make postgres-A4.pdf
openjade -D . -D . -c /usr/share/sgml/docbook/stylesheet/dsssl/modular/catalog -d ./stylesheet.dsl -t tex -V tex-backend -i output-print -i include-index -V texpdf-output -V '%paper-type%'=A4 -o postgres-A4.tex-pdf postgres.sgml
openjade:./stylesheet.dsl:621:2:E: flow object not accepted by port; only display flow objects accepted
make: *** [postgres-A4.tex-pdf] Segmentation fault (core dumped)
make: *** Deleting file `postgres-A4.tex-pdf'

This is Ubuntu 9.10
Linux mira 2.6.33-020633-generic #020633 SMP Thu Feb 25 10:10:03 UTC 2010 x86_64 GNU/Linux

Oleg

On Fri, 2 Apr 2010, Tom Lane wrote:

> "Erik Rijkers" <er(at)xs4all(dot)nl> writes:
>> I can build 'make postgres-A4.pdf' against cvs without problem.
>
> Fails for me :-(. Whose distribution of jadetex are you using, and
> what version exactly? I've pretty much convinced myself that the
> string consumption is the fault of the FlowObject macros in jadetex.ltx.
>
> regards, tom lane
>
>

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg(at)sai(dot)msu(dot)su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

"Erik Rijkers" <er(at)xs4all(dot)nl> writes:
> Here is how much of TeX's memory you used:
> 246467 strings out of 246489
> 1749307 string characters out of 1824395

Mph. So your installation is creating a slightly smaller number of
strings, but you're still on the hairy edge of failure (another half
dozen flow objects and it'd croak).

regards, tom lane

Devrim =?ISO-8859-1?Q?G=DCND=DCZ?= <devrim(at)gunduz(dot)org> writes:
> I was trying to build PDF docs for 9.0 Alpha5, and I got this message:
> ! TeX capacity exceeded, sorry [number of strings=245830].

I've found a possible solution for this. The bulk of the strings are
being created by jadetex.dtx: it makes two control sequences for each
flow object in the document. One of these is a page number and the
other seems to have no purpose except to prevent creating duplicate
hyperref anchors. However, diking out the latter doesn't create any
obvious ill effects --- either we have no occurrences in our docs of a
pattern that would result in a duplicate, or there isn't any real
adverse consequence of having a dup. (And in any case it's hard to
envision an adverse consequence that's worse than complete failure to
build the document.)

There turns out to be a very easy way to inject the code change, which
is to create a file "jadetex.cfg" in the doc/src/sgml directory,
containing the modified version of the relevant TeX macro.
(Note: in the vpath case we'd probably need to link it into the build
directory; I haven't experimented with that.) Put this in the file:

\def\FlowObjectSetup#1{%
\ifDoFOBSet
\ifLabelElements
\ifx\Label\(at)empty\let\Label\Element\fi
\fi
\ifx\Label\(at)empty\else
\bgroup
\ifNestedLink
\else
\hyper(at)anchorstart{\Label}\hyper@anchorend
\PageLabel{\Label}%
\fi
\egroup
\let\Label\(at)empty
\let\Element\(at)empty
\fi
\fi
}

For comparison, the original definition we're replacing is

\def\FlowObjectSetup#1{%
\ifDoFOBSet
\ifLabelElements
\ifx\Label\(at)empty\let\Label\Element\fi
\fi
\ifx\Label\(at)empty\else
\expandafter\ifx\csname x(at)\Label\endcsname\@madelink
\else
\bgroup
\ifNestedLink
\else
\hyper(at)anchorstart{\Label}\hyper@anchorend
\PageLabel{\Label}%
\expandafter\gdef\csname x(at)\Label\endcsname{ }%
\fi
\egroup
\let\Label\(at)empty
\let\Element\(at)empty
\fi
\fi
\fi
}

This should work with either jadetex 3.12 or 3.13, and probably older
versions but I don't have source code handy to look at. 3.12 dates
back to 2002 so it's probably old enough anyhow.

I haven't tried to build anything except PDF output with this
substituted code, so it's possible that it interferes with other
output formats.

Comments?

regards, tom lane

On Sat, 2010-04-03 at 15:53 -0400, Tom Lane wrote:
>
> There turns out to be a very easy way to inject the code change, which
> is to create a file "jadetex.cfg" in the doc/src/sgml directory,
> containing the modified version of the relevant TeX macro.
> (Note: in the vpath case we'd probably need to link it into the build
> directory; I haven't experimented with that.) Put this in the file:
<snip>

FYI: I built the PDFs with this.

Regards,
--
Devrim GÜNDÜZ
PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer
PostgreSQL RPM Repository: http://yum.pgrpms.org
Community: devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org Twitter: http://twitter.com/devrimgunduz

Tom Lane wrote:

> There turns out to be a very easy way to inject the code change, which
> is to create a file "jadetex.cfg" in the doc/src/sgml directory,
> containing the modified version of the relevant TeX macro.
> (Note: in the vpath case we'd probably need to link it into the build
> directory; I haven't experimented with that.)

FWIW I was able to build the PDF in a VPATH build by dropping this file
in the source dir, without any other change.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Alvaro Herrera wrote:
> Tom Lane wrote:
>
> > There turns out to be a very easy way to inject the code change, which
> > is to create a file "jadetex.cfg" in the doc/src/sgml directory,
> > containing the modified version of the relevant TeX macro.
> > (Note: in the vpath case we'd probably need to link it into the build
> > directory; I haven't experimented with that.)
>
> FWIW I was able to build the PDF in a VPATH build by dropping this file
> in the source dir, without any other change.

Huh, sorry, I rechecked by moving the file elsewhere and it still builds
:-( The TeX stats look like this:

Here is how much of TeX's memory you used:
246183 strings out of 246240
1743979 string characters out of 1818266
650918 words of memory out of 1585000
175202 multiletter control sequences out of 10000+262143
102154 words of font info for 154 fonts, out of 1200000 for 2000
54 hyphenation exceptions out of 8191
28i,13n,45p,1206b,3965s stack positions out of 1500i,500n,1500p,200000b,50000s

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

I wrote:
> Devrim =?ISO-8859-1?Q?G=DCND=DCZ?= <devrim(at)gunduz(dot)org> writes:
>> I was trying to build PDF docs for 9.0 Alpha5, and I got this message:
>> ! TeX capacity exceeded, sorry [number of strings=245830].

> I've found a possible solution for this. The bulk of the strings are
> being created by jadetex.dtx: it makes two control sequences for each
> flow object in the document. One of these is a page number and the
> other seems to have no purpose except to prevent creating duplicate
> hyperref anchors. However, diking out the latter doesn't create any
> obvious ill effects --- either we have no occurrences in our docs of a
> pattern that would result in a duplicate, or there isn't any real
> adverse consequence of having a dup.

It occurred to me that I could investigate the consequences of this hack
more closely by building the 8.4 docs with and without the jadetex.cfg
hack. I find that:

* The text content of the PDF is the same either way: the output of
pdftotext is bit-for-bit the same, and the output of pdf2ps seems to
be about the same (there are some differences that don't seem
consequential).

* I can't immediately find any particular problems in the PDF itself.
There are some broken intrapage links, but those are broken in the
un-hacked output as well.

* The TeX log shows a fair number of warnings like this:
TeX warning (ext4): destination with the same identifier (name{351})
has been already used, duplicate ignored
However, there are some of these even without the hack, so apparently
this isn't critical. Other than these warnings the .log files are
identical.

So I'm pretty well convinced that this is a usable workaround. It
certainly beats the heck out of not being able to build PDFs at all.

I also looked into the question of whether this would affect any other
output paths. The only other output type that jadetex is used for is
plain PostScript. I find that .ps output still builds on my machine
without using the hack. The output .ps file is different with the
hack in place, but not in significant ways as far as I can see
(it's a bit hard to tell though, since the output is pretty darn
unreadable). What is particularly interesting is that the un-hacked
.ps run is not anywhere near overflowing TeX's string capacity:

Here is how much of TeX's memory you used:
161458 strings out of 245673
1244334 string characters out of 1808311
832500 words of memory out of 1500000
175113 multiletter control sequences out of 10000+200000
69272 words of font info for 109 fonts, out of 1200000 for 2000
645 hyphenation exceptions out of 8191
29i,13n,44p,650b,3983s stack positions out of 5000i,500n,6000p,200000b,15000s

This surprised the heck out of me, because I thought that jadetex.dtx
was upstream of the specific output file format. It suggests that we
might look for an alternative solution by investigating just what's
different between the .ps and .pdf configurations. However, I don't
have time for that right now, and we need some usable fix so that
people can build docs for the beta.

Barring anybody having a better solution, I'll commit the
FlowObjectSetup hack tomorrow.

regards, tom lane