Re: Add %z support to elog/ereport?

Hi,

I'd like to add support for the length modifier %z. Linux' manpages
describes it as:
z A following integer conversion corresponds to a size_t or ssize_t argument.

Since gcc's printf format checks understand it, we can add support for
it similar to the way we added %m support.

Currently we just deal with wanting to print size_t/Size values by
casting them to uint32, uint64 or similar, but that's a) annoying
because 64bit numbers require the annoying UINT64_FORMAT b) more and
more likely to be problematic when casting to 32bit numbers.

Does anybody see prolbems with that?

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> I'd like to add support for the length modifier %z. Linux' manpages
> describes it as:
> z A following integer conversion corresponds to a size_t or ssize_t argument.

> Since gcc's printf format checks understand it, we can add support for
> it similar to the way we added %m support.

I think you'll find that %m is a totally different animal, because it
doesn't involve consuming an argument position. I'm less than sure that
every version of gcc will recognize %z, either ... and what about the
translation infrastructure?

regards, tom lane

On 2013-11-11 11:18:22 -0500, Tom Lane wrote:
> Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> > I'd like to add support for the length modifier %z. Linux' manpages
> > describes it as:
> > z A following integer conversion corresponds to a size_t or ssize_t argument.
>
> > Since gcc's printf format checks understand it, we can add support for
> > it similar to the way we added %m support.
>
> I think you'll find that %m is a totally different animal, because it
> doesn't involve consuming an argument position.

I was thinking of just replacing '%z' by '%l', '%ll' or '%' as needed
and not expand it inplace. That should deal with keeping the argument
position and such.
But that won't easily work if somebody specifies flags like padding :/

> I'm less than sure that every version of gcc will recognize %z, either
> ...

It's been in recognized in 2.95 afaics, so I think we're good.

> and what about the translation infrastructure?

That I have no clue about yet.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 2013-11-11 17:33:53 +0100, Andres Freund wrote:
> On 2013-11-11 11:18:22 -0500, Tom Lane wrote:
> > Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> > > I'd like to add support for the length modifier %z. Linux' manpages
> > > describes it as:
> > > z A following integer conversion corresponds to a size_t or ssize_t argument.

> > and what about the translation infrastructure?
>
> That I have no clue about yet.

gettext has support for it afaics, it's part of POSIX:
http://www.gnu.org/software/gettext/manual/gettext.html#c_002dformat
http://www.opengroup.org/onlinepubs/007904975/functions/fprintf.html

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> On 2013-11-11 11:18:22 -0500, Tom Lane wrote:
>> I think you'll find that %m is a totally different animal, because it
>> doesn't involve consuming an argument position.

> I was thinking of just replacing '%z' by '%l', '%ll' or '%' as needed
> and not expand it inplace. That should deal with keeping the argument
> position and such.
> But that won't easily work if somebody specifies flags like padding :/

[ reads manual ] Oh, I see that actually z *is* a flag, so you'd be
talking about replacing it with a different flag, ie %zu -> %llu or
similar. Yes, in principle that could work, but you'd have to be
prepared to cope with other flags, field width/precision, n$, etc,
appearing first. Sounds a bit messy, and this code is probably a
hot spot, remembering what we found out about the cost of fooling
with log_line_prefix.

>> I'm less than sure that every version of gcc will recognize %z, either
>> ...

> It's been in recognized in 2.95 afaics, so I think we're good.

[ fires up old HPUX box ] Nope:

$ cat test.c
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char**argv)
{
char buf[256];
size_t x = 0;

sprintf(buf, "%zu", (int)x);

return 0;
}
$ gcc -O2 -Wall test.c
test.c: In function `main':
test.c:9: warning: unknown conversion type character `z' in format
test.c:9: warning: too many arguments for format
$ gcc -v
Reading specs from /usr/local/lib/gcc-lib/hppa2.0-hp-hpux10.20/2.95.3/specs
gcc version 2.95.3 20010315 (release)

We might be willing to toss 2.95 overboard by now, but we'd need to be
sure of exactly what the new minimum usable version is.

>> and what about the translation infrastructure?

> That I have no clue about yet.

Me either. I do recall Peter saying that the infrastructure knows how to
verify conversion specs in translated strings, so it would have to be
aware of 'z' flags for this to work.

regards, tom lane

Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> gettext has support for it afaics, it's part of POSIX:

Really? [ pokes around at pubs.opengroup.org ] Hm, I don't see it
in Single Unix Spec v2 (1997), but it is there in POSIX issue 7 (2008).
Also, the POSIX page says it defers to the C standard, and I see it
in C99. Too bad not all our platforms are C99 yet :-(

Actually this raises an interesting testing question: how sure are we
that there aren't already some format strings in the code that depend
on C99 additions to the printf spec? If they're not in code paths
exercised by the regression tests, I'm not sure we'd find out from
the buildfarm.

regards, tom lane

On 2013-11-11 12:01:40 -0500, Tom Lane wrote:
> Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> >> I'm less than sure that every version of gcc will recognize %z, either
> >> ...
>
> > It's been in recognized in 2.95 afaics, so I think we're good.

Hm. Strange. Has to have been backpatched to the ancient debian I have
around. Unfortunately I can't easily "apt-get source" there...

The commit that added it to upstream is:
commit 44e9fa656d60bb19ab81d76698a61e47a4b0857c
Author: drepper <drepper(at)138bc75d-0d04-0410-961f-82ee72b054a4>
Date: Mon Jan 3 21:48:49 2000 +0000

(format_char_info): Update comment. (check_format_info): Recognize 'z'
modifier in the same way 'Z' was recognized. Emit warning for formats
new in ISO C99 only if flag_isoc9x is not set.

git-svn-id: svn+ssh://gcc.gnu.org/svn/gcc/trunk(at)31188 138bc75d-0d04-0410-961f-82ee72b054a4

That's 3.0. Verified it in the 3.0. tarball, although I didn't compile
test it.

> We might be willing to toss 2.95 overboard by now, but we'd need to be
> sure of exactly what the new minimum usable version is.

Well, we don't even need to toss it overboard, just live with useless
warnings there since we'd translate it ourselves.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Mon, Nov 11, 2013 at 10:50 AM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> Hi,
>
> I'd like to add support for the length modifier %z. Linux' manpages
> describes it as:
> z A following integer conversion corresponds to a size_t or ssize_t argument.
>
> Since gcc's printf format checks understand it, we can add support for
> it similar to the way we added %m support.

I seem to recall that our %m support involves rewriting the error
string twice, which I think is actually kind of expensive if, for
example, you've got a loop around a PL/pgsql EXCEPTION block. I'd
actually like to find a way to get rid of the existing %m support,
maybe by having a flag that says "oh, and by the way append the system
error to my format string"; or by changing %m to %s and having the
caller pass system_error_string() or similar for that format position.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2013-11-11 12:18:46 -0500, Tom Lane wrote:
> Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> > gettext has support for it afaics, it's part of POSIX:
>
> Really? [ pokes around at pubs.opengroup.org ] Hm, I don't see it
> in Single Unix Spec v2 (1997), but it is there in POSIX issue 7 (2008).
> Also, the POSIX page says it defers to the C standard, and I see it
> in C99. Too bad not all our platforms are C99 yet :-(

Seems to be 2001:
http://pubs.opengroup.org/onlinepubs/007904975/functions/fprintf.html

Even though the date says it's from 2004, it's IEEE Std 1003.1 + minor
errata.

> Actually this raises an interesting testing question: how sure are we
> that there aren't already some format strings in the code that depend
> on C99 additions to the printf spec? If they're not in code paths
> exercised by the regression tests, I'm not sure we'd find out from
> the buildfarm.

I agree, it's problematic. Especially as such code is likely to be in
error paths. I seem to recall you fixing some occurances you found
manually some time back so we clearly don't have an automated process
for it yet.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 2013-11-11 12:31:55 -0500, Robert Haas wrote:
> On Mon, Nov 11, 2013 at 10:50 AM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> > Hi,
> >
> > I'd like to add support for the length modifier %z. Linux' manpages
> > describes it as:
> > z A following integer conversion corresponds to a size_t or ssize_t argument.
> >
> > Since gcc's printf format checks understand it, we can add support for
> > it similar to the way we added %m support.
>
> I seem to recall that our %m support involves rewriting the error
> string twice, which I think is actually kind of expensive if, for
> example, you've got a loop around a PL/pgsql EXCEPTION block.

Yes, it does that. Is that actually where a significant amount of time
is spent? I have a somewhat hard time believing that.

> I'd
> actually like to find a way to get rid of the existing %m support,
> maybe by having a flag that says "oh, and by the way append the system
> error to my format string"; or by changing %m to %s and having the
> caller pass system_error_string() or similar for that format position.

I'd rather get our own printf implementation from somewhere before doing
that which would quite likely result in a bigger speedup besides the
significant portability improvments.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> On 2013-11-11 12:31:55 -0500, Robert Haas wrote:
>> I seem to recall that our %m support involves rewriting the error
>> string twice, which I think is actually kind of expensive if, for
>> example, you've got a loop around a PL/pgsql EXCEPTION block.

> Yes, it does that. Is that actually where a significant amount of time
> is spent? I have a somewhat hard time believing that.

I don't see any double copying. There is *one* copy made by
expand_fmt_string. Like Andres, I'd want to see proof that
expand_fmt_string is a significant time sink before we jump through
these kinds of hoops to get rid of it. It looks like a pretty cheap
loop to me. (It might be less cheap if we made it smart enough to
identify 'z' flags, though :-()

>> I'd
>> actually like to find a way to get rid of the existing %m support,
>> maybe by having a flag that says "oh, and by the way append the system
>> error to my format string"; or by changing %m to %s and having the
>> caller pass system_error_string() or similar for that format position.

The first of these doesn't work unless you require translations to
assemble the string the same way the English version does. The second
would work, I guess, but it'd sure be a pain to convert everything.

regards, tom lane

On 11/11/13, 12:01 PM, Tom Lane wrote:
> I do recall Peter saying that the infrastructure knows how to
> verify conversion specs in translated strings, so it would have to be
> aware of 'z' flags for this to work.

It just checks that the same conversion placeholders appear in the
translation. It doesn't interpret the actual placeholders. I don't
think this will be a problem.

On 2013-12-06 09:54:59 -0500, Peter Eisentraut wrote:
> On 11/11/13, 12:01 PM, Tom Lane wrote:
> > I do recall Peter saying that the infrastructure knows how to
> > verify conversion specs in translated strings, so it would have to be
> > aware of 'z' flags for this to work.
>
> It just checks that the same conversion placeholders appear in the
> translation. It doesn't interpret the actual placeholders. I don't
> think this will be a problem.

Ok, so here's a patch.

Patch 01 introduces infrastructure, and elog.c implementation.
Patch 02 converts some elog/ereport() callers to using the z modifier,
some were wrong at least on 64 bit windows.

In patch 01, I've modified configure to not define [U]INT64_FORMAT
directly, but rather just define INT64_LENGTH_MODIFIER as
appropriate. The formats are now defined in c.h.
INT64_LENGTH_MODIFIER is defined without quotes - I am not sure whether
that was the right choice, it requires using CppAsString2() in some
places. Maybe I should just have defined it with quotes instead. Happy
to rejigger it if somebody thinks the other way would be better.

Note that I have decided to only support the z modifier properly if no
further modifiers (precision, width) are present. That seems sufficient
for now, given the usecases, and more complete support seems to be
potentially expensive without much use.

I wonder if we should also introduce SIZE_T_FORMAT and SSIZE_T_FORMAT,
there's some places that have #ifdef _WIN64 guards and similar that look
like they could be simplified.

Btw, walsender.c/walreceiver.c upcast an int into an unsigned long for
printing? That's a bit strange.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Sun, 2013-12-08 at 01:32 +0100, Andres Freund wrote:
> Patch 02 converts some elog/ereport() callers to using the z modifier,
> some were wrong at least on 64 bit windows.

This is making the assumption that Size is the same as size_t. If we
are going to commit to that, we might as well get rid of Size.

On 2013-12-08 11:43:46 -0500, Peter Eisentraut wrote:
> On Sun, 2013-12-08 at 01:32 +0100, Andres Freund wrote:
> > Patch 02 converts some elog/ereport() callers to using the z modifier,
> > some were wrong at least on 64 bit windows.
>
> This is making the assumption that Size is the same as size_t. If we
> are going to commit to that, we might as well get rid of Size.

Well, we're unconditionally defining it as such in c.h and its usage is
pretty much what size_t is for. Given how widespread Size's use is, I am
not seing a realistic way of getting rid of it though.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 2013-12-08 01:32:45 +0100, Andres Freund wrote:
> On 2013-12-06 09:54:59 -0500, Peter Eisentraut wrote:
> > On 11/11/13, 12:01 PM, Tom Lane wrote:
> > > I do recall Peter saying that the infrastructure knows how to
> > > verify conversion specs in translated strings, so it would have to be
> > > aware of 'z' flags for this to work.
> >
> > It just checks that the same conversion placeholders appear in the
> > translation. It doesn't interpret the actual placeholders. I don't
> > think this will be a problem.
>
> Ok, so here's a patch.
>
> Patch 01 introduces infrastructure, and elog.c implementation.
> Patch 02 converts some elog/ereport() callers to using the z modifier,
> some were wrong at least on 64 bit windows.

There's just been another instance of printing size_t values being
annoying, reminding me of this patch. I'll add it to the current
commitfest given I've posted it over a month ago unless somebody
protests?

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> [ 0001-Add-support-for-printing-Size-arguments-to-elog-erep.patch ]

I think this approach is fundamentally broken, because it can't reasonably
cope with any case more complicated than "%zu" or "%zd". While it's
arguable that we can get along without the ability to specify field width,
since the [U]INT64_FORMAT macros never allowed that, it is surely going
to be the case that translators will need to insert "n$" flags in the
translated versions of messages.

Another objection is that this only fixes the problem in elog/ereport
format strings, not anyplace else that it might be useful to print a
size_t value. You suggest below that we could invent some additional
macros to support that; but since the "z" flag is in C99, there really
ought to be only a small minority of platforms where it doesn't work.
So I don't think we should be contorting our code with some nonstandard
notation for the case, if there's any way to avoid that.

I think a better solution approach is to teach our src/port/snprintf.c
about the z flag, then extend configure's checking to force use of our
snprintf if the platform's version doesn't handle z. While it might be
objected that this creates a need for a run-time check in configure,
we already have one to check if snprintf handles "n$", so this approach
doesn't really make life any worse for cross-compiles.

> In patch 01, I've modified configure to not define [U]INT64_FORMAT
> directly, but rather just define INT64_LENGTH_MODIFIER as
> appropriate. The formats are now defined in c.h.
> INT64_LENGTH_MODIFIER is defined without quotes - I am not sure whether
> that was the right choice, it requires using CppAsString2() in some
> places. Maybe I should just have defined it with quotes instead. Happy
> to rejigger it if somebody thinks the other way would be better.

Meh. This isn't needed if we do what I suggest above, but in any case
I don't approve of removing the existing [U]INT64_FORMAT macros.
That breaks code that doesn't need to get broken, probably including
third-party modules. It'd be more sensible to just add an additional
macro for the flag character(s). (And yeah, I'd put quotes in it.)

regards, tom lane

I wrote:
> Meh. This isn't needed if we do what I suggest above, but in any case
> I don't approve of removing the existing [U]INT64_FORMAT macros.
> That breaks code that doesn't need to get broken, probably including
> third-party modules.

After looking more closely I see you didn't actually *remove* those
macros, just define them in a different place/way. So the above objection
is just noise, sorry. (Though I think it'd be notationally cleaner to let
configure continue to define the macros; it doesn't need to do anything as
ugly as CppAsString2() to concatenate...)

regards, tom lane

Hi,

On 2014-01-17 13:50:08 -0500, Tom Lane wrote:
> I think a better solution approach is to teach our src/port/snprintf.c
> about the z flag, then extend configure's checking to force use of our
> snprintf if the platform's version doesn't handle z. While it might be
> objected that this creates a need for a run-time check in configure,
> we already have one to check if snprintf handles "n$", so this approach
> doesn't really make life any worse for cross-compiles.

Hm. I had thought about that, but dismissed it because I thought people
would argue about it being too invasive...
If we're going there, we should just eliminate expand_fmt_string() from
elog.c and test for it in configure too, right?

> You suggest below that we could invent some additional
> macros to support that; but since the "z" flag is in C99, there really
> ought to be only a small minority of platforms where it doesn't work.

Well, maybe just a minority numberwise, but one of them being windows
surely makes it count in number of installations...

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 2014-01-17 14:18:55 -0500, Tom Lane wrote:
> I wrote:
> > Meh. This isn't needed if we do what I suggest above, but in any case
> > I don't approve of removing the existing [U]INT64_FORMAT macros.
> > That breaks code that doesn't need to get broken, probably including
> > third-party modules.
>
> After looking more closely I see you didn't actually *remove* those
> macros, just define them in a different place/way. So the above objection
> is just noise, sorry. (Though I think it'd be notationally cleaner to let
> configure continue to define the macros; it doesn't need to do anything as
> ugly as CppAsString2() to concatenate...)

I prefer having configure just define the lenght modifier since that
allows to define further macros containing formats. But I think defining
them as strings instead row literals as I had might make it a bit less ugly...

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 2014-01-17 13:50:08 -0500, Tom Lane wrote:
> Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> > [ 0001-Add-support-for-printing-Size-arguments-to-elog-erep.patch ]
>
> I think this approach is fundamentally broken, because it can't reasonably
> cope with any case more complicated than "%zu" or "%zd". While it's
> arguable that we can get along without the ability to specify field width,
> since the [U]INT64_FORMAT macros never allowed that, it is surely going
> to be the case that translators will need to insert "n$" flags in the
> translated versions of messages.

Am I just too tired, or am I not getting how INT64_FORMAT currently
allows the arguments to be used posititional?

Admittedly most places currently just cast down the value avoiding the
need for INT64_FORMAT in many places.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> On 2014-01-17 13:50:08 -0500, Tom Lane wrote:
>> I think a better solution approach is to teach our src/port/snprintf.c
>> about the z flag, then extend configure's checking to force use of our
>> snprintf if the platform's version doesn't handle z.

> Hm. I had thought about that, but dismissed it because I thought people
> would argue about it being too invasive...

How so? It'd be a lot less invasive than what we'd have to do to use
'z' flags the other way.

> If we're going there, we should just eliminate expand_fmt_string() from
> elog.c and test for it in configure too, right?

If you mean "let's rely on glibc for %m", the answer is "not bloody
likely". See useful_strerror(), which is functionality we'd lose if
we short-circuit that.

>> You suggest below that we could invent some additional
>> macros to support that; but since the "z" flag is in C99, there really
>> ought to be only a small minority of platforms where it doesn't work.

> Well, maybe just a minority numberwise, but one of them being windows
> surely makes it count in number of installations...

Agreed, but I believe we're already using src/port/snprintf.c on Windows
because of lack of %n$ support (which is also required by C99).

regards, tom lane

Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> On 2014-01-17 13:50:08 -0500, Tom Lane wrote:
>> I think this approach is fundamentally broken, because it can't reasonably
>> cope with any case more complicated than "%zu" or "%zd".

> Am I just too tired, or am I not getting how INT64_FORMAT currently
> allows the arguments to be used posititional?

It doesn't, which is one of the reasons for not allowing it in
translatable strings (the other being lack of standardization of the
strings that would be subject to translation). Adding 'z' will only
fix this for cases where what we want to print is really a size_t.
That's a usefully large set, but not all the cases by any means.

regards, tom lane

I wrote:
> Andres Freund <andres(at)2ndquadrant(dot)com> writes:
>> On 2014-01-17 13:50:08 -0500, Tom Lane wrote:
>>> I think this approach is fundamentally broken, because it can't reasonably
>>> cope with any case more complicated than "%zu" or "%zd".

>> Am I just too tired, or am I not getting how INT64_FORMAT currently
>> allows the arguments to be used posititional?

> It doesn't, which is one of the reasons for not allowing it in
> translatable strings (the other being lack of standardization of the
> strings that would be subject to translation).

On second thought, that answer was too glib. There's no need for %n$
in the format strings *in the source code*, so INT64_FORMAT isn't getting
in the way from that perspective. However, expand_fmt_string() is
necessarily applied to formats *after* they've been through gettext(),
so it has to expect that it might see %n$ in the now-translated strings.

It's still the case that we need a policy against INT64_FORMAT in
translatable strings, though, because what's passed to the gettext
mechanism has to be a fixed string not something with platform
dependencies in it. Or so I would assume, anyway.

regards, tom lane

On Fri, Jan 17, 2014 at 8:36 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> I wrote:
>> Andres Freund <andres(at)2ndquadrant(dot)com> writes:
>>> On 2014-01-17 13:50:08 -0500, Tom Lane wrote:
>>>> I think this approach is fundamentally broken, because it can't reasonably
>>>> cope with any case more complicated than "%zu" or "%zd".
>
>>> Am I just too tired, or am I not getting how INT64_FORMAT currently
>>> allows the arguments to be used posititional?
>
>> It doesn't, which is one of the reasons for not allowing it in
>> translatable strings (the other being lack of standardization of the
>> strings that would be subject to translation).
>
> On second thought, that answer was too glib. There's no need for %n$
> in the format strings *in the source code*, so INT64_FORMAT isn't getting
> in the way from that perspective. However, expand_fmt_string() is
> necessarily applied to formats *after* they've been through gettext(),
> so it has to expect that it might see %n$ in the now-translated strings.
>
> It's still the case that we need a policy against INT64_FORMAT in
> translatable strings, though, because what's passed to the gettext
> mechanism has to be a fixed string not something with platform
> dependencies in it. Or so I would assume, anyway.

Well, that's kinda problematic, isn't it? Printing the variable into
a static buffer so that it can then be formatted with %s is pretty
pessimal for a message that we might not even be planning to emit.

Perhaps we should jettison entirely the idea of using the operating
system's built-in sprintf and use one of our own that has all of the
nice widgets we need, like a format code that's guaranteed to be right
for uint64 and one that's guaranteed to be right for Size. This could
turn out to be a bad idea if the best sprintf we can write is much
slower than the native sprintf on any common platforms ... and maybe
it wouldn't play nice with GCC's desire to check format strings. But
what we're doing now is a real nuisance, too.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas escribió:
> On Fri, Jan 17, 2014 at 8:36 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > I wrote:
> >> Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> >>> On 2014-01-17 13:50:08 -0500, Tom Lane wrote:

> >>> Am I just too tired, or am I not getting how INT64_FORMAT currently
> >>> allows the arguments to be used posititional?
> >
> >> It doesn't, which is one of the reasons for not allowing it in
> >> translatable strings (the other being lack of standardization of the
> >> strings that would be subject to translation).
> >
> > On second thought, that answer was too glib. There's no need for %n$
> > in the format strings *in the source code*, so INT64_FORMAT isn't getting
> > in the way from that perspective. However, expand_fmt_string() is
> > necessarily applied to formats *after* they've been through gettext(),
> > so it has to expect that it might see %n$ in the now-translated strings.

How difficult would it be to have expand_fmt_string deal with positional
modifiers? I don't think we need anything from it other than the %n$
notation, so perhaps it's not so problematic.

> Perhaps we should jettison entirely the idea of using the operating
> system's built-in sprintf and use one of our own that has all of the
> nice widgets we need, like a format code that's guaranteed to be right
> for uint64 and one that's guaranteed to be right for Size. This could
> turn out to be a bad idea if the best sprintf we can write is much
> slower than the native sprintf on any common platforms ... and maybe
> it wouldn't play nice with GCC's desire to check format strings. But
> what we're doing now is a real nuisance, too.

Maybe we can use our own implementation if the system's doesn't support
%z. It's present in glibc 2.1 at least, and it's part of in the 2004
edition of POSIX:2001.
http://pubs.opengroup.org/onlinepubs/009695399/functions/sprintf.html

(It is not present in SUSv2 (1997), and I wasn't able to find the
original POSIX:2001 version.)

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 2014-01-21 12:11:23 -0300, Alvaro Herrera wrote:
> How difficult would it be to have expand_fmt_string deal with positional
> modifiers? I don't think we need anything from it other than the %n$
> notation, so perhaps it's not so problematic.

I don't think there's much reason to go there. I didn't go for the
pg-supplied sprintf() because I thought it'd be considered to
invasive. Since that's apparently not the case...

> > Perhaps we should jettison entirely the idea of using the operating
> > system's built-in sprintf and use one of our own that has all of the
> > nice widgets we need, like a format code that's guaranteed to be right
> > for uint64 and one that's guaranteed to be right for Size. This could
> > turn out to be a bad idea if the best sprintf we can write is much
> > slower than the native sprintf on any common platforms ... and maybe
> > it wouldn't play nice with GCC's desire to check format strings. But
> > what we're doing now is a real nuisance, too.
>
> Maybe we can use our own implementation if the system's doesn't support
> %z. It's present in glibc 2.1 at least, and it's part of in the 2004
> edition of POSIX:2001.
> http://pubs.opengroup.org/onlinepubs/009695399/functions/sprintf.html

Yea, I have a patch for that, will send it as soon as some other stuff
is finished.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> On 2014-01-21 12:11:23 -0300, Alvaro Herrera wrote:
>> How difficult would it be to have expand_fmt_string deal with positional
>> modifiers? I don't think we need anything from it other than the %n$
>> notation, so perhaps it's not so problematic.

> I don't think there's much reason to go there. I didn't go for the
> pg-supplied sprintf() because I thought it'd be considered to
> invasive. Since that's apparently not the case...

Yeah, that would make expand_fmt_string considerably more complicated
and so presumably slower. We don't really need that when we can make
what I expect is a pretty trivial addition to snprintf.c. Also, fixing
snprintf.c will make it safe to use the z flag in contexts other than
ereport/elog.

regards, tom lane

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> On Fri, Jan 17, 2014 at 8:36 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> It's still the case that we need a policy against INT64_FORMAT in
>> translatable strings, though, because what's passed to the gettext
>> mechanism has to be a fixed string not something with platform
>> dependencies in it. Or so I would assume, anyway.

> Well, that's kinda problematic, isn't it? Printing the variable into
> a static buffer so that it can then be formatted with %s is pretty
> pessimal for a message that we might not even be planning to emit.

Well, it's a tad annoying, but a quick grep says there are maybe 40
cases of this in our source code, so I'm not sure it's rising to the
level of a must-fix problem.

> Perhaps we should jettison entirely the idea of using the operating
> system's built-in sprintf and use one of our own that has all of the
> nice widgets we need, like a format code that's guaranteed to be right
> for uint64 and one that's guaranteed to be right for Size. This could
> turn out to be a bad idea if the best sprintf we can write is much
> slower than the native sprintf on any common platforms ... and maybe
> it wouldn't play nice with GCC's desire to check format strings.

That last is a deal-breaker. It's not just whether "gcc desires" to check
this --- we *need* that checking, because people get it wrong without it.

I thought about proposing that we insist that snprintf support the "ll"
flag (substituting our own if not). But that doesn't really fix anything
unless we're willing to explicitly cast the corresponding values to
"long long", which is probably not workable from a portability standpoint.
I'm not really worried about platforms thinking long long is 128 bits ---
I'm worried about old compilers that don't have such a datatype.

regards, tom lane

On Jan21, 2014, at 18:56 , Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> Perhaps we should jettison entirely the idea of using the operating
>> system's built-in sprintf and use one of our own that has all of the
>> nice widgets we need, like a format code that's guaranteed to be right
>> for uint64 and one that's guaranteed to be right for Size. This could
>> turn out to be a bad idea if the best sprintf we can write is much
>> slower than the native sprintf on any common platforms ... and maybe
>> it wouldn't play nice with GCC's desire to check format strings.
>
> That last is a deal-breaker. It's not just whether "gcc desires" to check
> this --- we *need* that checking, because people get it wrong without it.

There's an attribute that enables this check for arbitrary functions AFAIR.

best regards,
Florian Pflug

Florian Pflug <fgp(at)phlo(dot)org> writes:
> On Jan21, 2014, at 18:56 , Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>>> it wouldn't play nice with GCC's desire to check format strings.

>> That last is a deal-breaker. It's not just whether "gcc desires" to check
>> this --- we *need* that checking, because people get it wrong without it.

> There's an attribute that enables this check for arbitrary functions AFAIR.

Yeah, we use it (to enable checking for ereport et al). The issue is that
the semantics of the format-string are pretty much hard-wired into gcc;
eg it knows that "%ld" should match an argument of type "long int".

IIRC it does know a couple of different styles corresponding to popular
libc implementations ... but it is not going to support some random
semantics that we dream up.

regards, tom lane

On 2014-01-21 11:33:40 -0500, Tom Lane wrote:
> Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> > On 2014-01-21 12:11:23 -0300, Alvaro Herrera wrote:
> >> How difficult would it be to have expand_fmt_string deal with positional
> >> modifiers? I don't think we need anything from it other than the %n$
> >> notation, so perhaps it's not so problematic.
>
> > I don't think there's much reason to go there. I didn't go for the
> > pg-supplied sprintf() because I thought it'd be considered to
> > invasive. Since that's apparently not the case...
>
> Yeah, that would make expand_fmt_string considerably more complicated
> and so presumably slower. We don't really need that when we can make
> what I expect is a pretty trivial addition to snprintf.c. Also, fixing
> snprintf.c will make it safe to use the z flag in contexts other than
> ereport/elog.

So, here's a patch adding %z support to port/snprintf.c including a
configure check to test whether we need to fall back. I am not too
happy about the runtime check as the test isn't all that meaningful, but
I couldn't think of anything better.

The second patch is a slightly updated version of a previously sent
version which is starting to use %z in some more places.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> So, here's a patch adding %z support to port/snprintf.c including a
> configure check to test whether we need to fall back.

OK, I'll take a look.

> I am not too
> happy about the runtime check as the test isn't all that meaningful, but
> I couldn't think of anything better.

Yeah, it's problematic for cross-compiles, but no more so than configure's
existing test for "%n$" support. In practice, since both these features
are required by C99, I think it wouldn't be such an issue for most people.

regards, tom lane

On 2014-01-23 11:14:05 -0500, Tom Lane wrote:
> OK, I'll take a look.

Thanks.

> > I am not too
> > happy about the runtime check as the test isn't all that meaningful, but
> > I couldn't think of anything better.
>
> Yeah, it's problematic for cross-compiles, but no more so than configure's
> existing test for "%n$" support. In practice, since both these features
> are required by C99, I think it wouldn't be such an issue for most people.

Currently we automatically fall back to our implementation if we're
cross compiling unless I am missing something, that's a bit odd, but it
should work ;)

I was wondering more about the nature of the runtime check than the fact
that it's a runtime check at all... E.g. snprintf.c simply skips over
unknown format characters and might not have been detected as faulty on
32bit platforms by that check. Which might be considered a good thing :)

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> I was wondering more about the nature of the runtime check than the fact
> that it's a runtime check at all... E.g. snprintf.c simply skips over
> unknown format characters and might not have been detected as faulty on
> 32bit platforms by that check. Which might be considered a good thing :)

Oh ... gotcha. Yeah, it's possible that snprintf would behave in a way
that masks the fact that it doesn't really recognize the "z" flag, but
that seems rather unlikely to me. More likely it would abandon processing
the %-sequence on grounds it's malformed.

I will try the patch on my old HPUX dinosaur, which I'm pretty sure
does not know "z", and verify this is the case.

Also, I'm guessing Windows' version of snprintf doesn't have "z" either.
Could someone try the patch's configure test program on Windows and see
what the result is?

regards, tom lane

On 2014-01-23 11:25:56 -0500, Tom Lane wrote:
> Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> > I was wondering more about the nature of the runtime check than the fact
> > that it's a runtime check at all... E.g. snprintf.c simply skips over
> > unknown format characters and might not have been detected as faulty on
> > 32bit platforms by that check. Which might be considered a good thing :)
>
> Oh ... gotcha. Yeah, it's possible that snprintf would behave in a way
> that masks the fact that it doesn't really recognize the "z" flag, but
> that seems rather unlikely to me. More likely it would abandon processing
> the %-sequence on grounds it's malformed.

Yea, hopefully.

> I will try the patch on my old HPUX dinosaur, which I'm pretty sure
> does not know "z", and verify this is the case.

I don't know how, but I've introduced a typo in the version I sent if
you haven't noticed yet, there's a " missing in
PGAC_FUNC_PRINTF_SIZE_T_SUPPORT. "%zu" instead of "%zu

> Also, I'm guessing Windows' version of snprintf doesn't have "z" either.
> Could someone try the patch's configure test program on Windows and see
> what the result is?

I've attached a version of that here, for $windowsperson's
convenience. I hope I got the llp stuff right...

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> snprintf(buf64, 100, UINT64_FORMAT, ~(size_t)0);

Actually, that coding isn't gonna work at all on platforms where size_t
isn't the same size as uint64. We could make it work by explicitly
casting the argument to whatever type we've decided to use as uint64
... but unless we want to include c.h here, that would require a lot of
extra cruft, and I'm really not sure it's gaining anything anyway.

I'm inclined to just print (size_t)0xFFFFFFFF and see if it produces
the expected result.

regards, tom lane

On 2014-01-23 12:54:22 -0500, Tom Lane wrote:
> Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> > snprintf(buf64, 100, UINT64_FORMAT, ~(size_t)0);
>
> Actually, that coding isn't gonna work at all on platforms where size_t
> isn't the same size as uint64. We could make it work by explicitly
> casting the argument to whatever type we've decided to use as uint64
> ... but unless we want to include c.h here, that would require a lot of
> extra cruft, and I'm really not sure it's gaining anything anyway.

Hm, yea, it should be casted. I think we should have the type ready in
PG_INT64_TYPE, confdefs.h should contain it at that point.

> I'm inclined to just print (size_t)0xFFFFFFFF and see if it produces
> the expected result.

Well, the reasoning, weak as it may be, was that that we want to see
whether we successfully recognize z as a 64bit modifier or not. And I
couldn't think of a better way to test that both for 32 and 64 bit
platforms...

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> On 2014-01-23 12:54:22 -0500, Tom Lane wrote:
>> Andres Freund <andres(at)2ndquadrant(dot)com> writes:
>>> snprintf(buf64, 100, UINT64_FORMAT, ~(size_t)0);

>> Actually, that coding isn't gonna work at all on platforms where size_t
>> isn't the same size as uint64. We could make it work by explicitly
>> casting the argument to whatever type we've decided to use as uint64
>> ... but unless we want to include c.h here, that would require a lot of
>> extra cruft, and I'm really not sure it's gaining anything anyway.

> Hm, yea, it should be casted. I think we should have the type ready in
> PG_INT64_TYPE, confdefs.h should contain it at that point.

Ah, I'd forgotten that configure defined any such symbol. Yeah, that
will work.

> Well, the reasoning, weak as it may be, was that that we want to see
> whether we successfully recognize z as a 64bit modifier or not.

I'm dubious that this is really adding much, but whatever.

I checked on my HPUX box and find that what it prints for "%zu" is
"zu", confirming my thought that it'd just abandon processing of the
%-sequence. (Interesting that it doesn't eat the "z" while doing
so, though.)

regards, tom lane

I wrote:
> I checked on my HPUX box and find that what it prints for "%zu" is
> "zu", confirming my thought that it'd just abandon processing of the
> %-sequence. (Interesting that it doesn't eat the "z" while doing
> so, though.)

Further testing on that box shows that its ancient gcc (2.95.3) doesn't
know "z" either, which means that the patch produces a boatload of
compile warnings like this:

mcxt.c: In function `MemoryContextAllocZero':
mcxt.c:605: warning: unknown conversion type character `z' in format
mcxt.c:605: warning: too many arguments for format

While I am not really expecting this gcc to compile PG cleanly anymore,
the idea that we might get many dozen such warnings on more-current
compilers is scarier, as that might well interfere with people's
ability to do development on, say, Windows. Could somebody check
whether MSVC for instance complains about format strings using "z"?
Or shall I just commit this and we'll see what the buildfarm says?

regards, tom lane

I wrote:
> the idea that we might get many dozen such warnings on more-current
> compilers is scarier, as that might well interfere with people's
> ability to do development on, say, Windows. Could somebody check
> whether MSVC for instance complains about format strings using "z"?
> Or shall I just commit this and we'll see what the buildfarm says?

Given the lack of response, I've pushed the patch, and will canvass
the buildfarm results later.

regards, tom lane

On 2014-01-23 17:21:11 -0500, Tom Lane wrote:
> I wrote:
> > the idea that we might get many dozen such warnings on more-current
> > compilers is scarier, as that might well interfere with people's
> > ability to do development on, say, Windows. Could somebody check
> > whether MSVC for instance complains about format strings using "z"?
> > Or shall I just commit this and we'll see what the buildfarm says?
>
> Given the lack of response, I've pushed the patch, and will canvass
> the buildfarm results later.

Thanks, I was afk, otherwise I'd have tried to pushing it to windows via
jenkins if that machine was running (it's running in Craig's flat...)…

Do we have a real policy against indenting nested preprocessor
statments? Just so I don't do those in future patches...

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> Do we have a real policy against indenting nested preprocessor
> statments? Just so I don't do those in future patches...

Indent 'em if you like, but pgindent will undo it. I ran the patch
through pgindent to see what it would do with those, and it left-justified
them, so I committed it that way.

regards, tom lane

I wrote:
>> the idea that we might get many dozen such warnings on more-current
>> compilers is scarier, as that might well interfere with people's
>> ability to do development on, say, Windows. Could somebody check
>> whether MSVC for instance complains about format strings using "z"?
>> Or shall I just commit this and we'll see what the buildfarm says?

> Given the lack of response, I've pushed the patch, and will canvass
> the buildfarm results later.

Just to follow up on that, I couldn't find any related warnings in the
buildfarm this morning (although there is one laggard machine, "anole",
which uses an unusual compiler and still hasn't reported in).

regards, tom lane