Re: Patch - Debug builds without optimization

Hello,

I'm sending following patch which disables optimization when
--enable-debug is passed. It was nasty (for me, at least) that debug
build required passing of CFLAGS with -O0 to get nice traceable code.

Regards,
Radek

On Thu, 16 Jun 2011 14:30:27 +0200, Radosław Smogura wrote:
> Hello,
>
> I'm sending following patch which disables optimization when
> --enable-debug is passed. It was nasty (for me, at least) that debug
> build required passing of CFLAGS with -O0 to get nice traceable code.
>
> Regards,
> Radek

Sorry for mess, this should be submited.

On Jun16, 2011, at 14:30 , Radosław Smogura wrote:
> I'm sending following patch which disables optimization when --enable-debug is passed. It was nasty (for me, at least) that debug build required passing of CFLAGS with -O0 to get nice traceable code.

Unfortunately, with some compilers (gcc, I'm looking at you) you get considerably fewer warnings with -O0 than with -O1, even if you specify -Wall. The reason seems to be that some of the warnings need information produces by some of the optimization passes.

I usually use -O1 for debug builds, these are usually still at least somewhat debuggable with gdb.

best regards,
Florian Pflug

--On 16. Juni 2011 14:30:27 +0200 Radosław Smogura <rsmogura(at)softperience(dot)eu>
wrote:

> Hello,
>
> I'm sending following patch which disables optimization when --enable-debug
> is passed. It was nasty (for me, at least) that debug build required passing
> of CFLAGS with -O0 to get nice traceable code.
>

-O0 hides bugs in your code (e.g. look at
<http://archives.postgresql.org/message-id/9714F5232AB2C4FCFCB392D5@amenophis>
and replies for an example to do it better). Doing this automatically on debug
builds would be a step backwards.

--
Thanks

Bernd

Florian Pflug <fgp(at)phlo(dot)org> writes:
> On Jun16, 2011, at 14:30 , Radosaw Smogura wrote:
>> I'm sending following patch which disables optimization when --enable-debug is passed. It was nasty (for me, at least) that debug build required passing of CFLAGS with -O0 to get nice traceable code.

> Unfortunately, with some compilers (gcc, I'm looking at you) you get
> considerably fewer warnings with -O0 than with -O1, even if you specify
> -Wall.

Yes. There is *zero* chance of this being accepted, because it would
break a lot of warnings that developers need to see.

> I usually use -O1 for debug builds, these are usually still at least
> somewhat debuggable with gdb.

I tend to do that too, but I still think that folding it into
--enable-debug would be a mistake. The normal assumption (at least when
using gcc) is that --enable-debug doesn't cost any performance. We
would annoy many people, especially packagers, if that stopped being
true.

I could see providing some other nonstandard configure switch that
changed the default -O level ... but realistically, would that do
anything that you couldn't already do by setting CFLAGS, ie

./configure CFLAGS="-O0 -g"

regards, tom lane

On Jun16, 2011, at 16:10 , Tom Lane wrote:
> Florian Pflug <fgp(at)phlo(dot)org> writes:
>> I usually use -O1 for debug builds, these are usually still at least
>> somewhat debuggable with gdb.
>
> I tend to do that too, but I still think that folding it into
> --enable-debug would be a mistake.

+1.

I didn't mean to suggest we fold -O1 into --enable-debug, I
was just handling out advice to the OP ;-)

best regards,
Florian Pflug

On 06/16/2011 10:10 AM, Tom Lane wrote:
> I could see providing some other nonstandard configure switch that
> changed the default -O level ... but realistically, would that do
> anything that you couldn't already do by setting CFLAGS, ie
>
> ./configure CFLAGS="-O0 -g"
>

I think a small discussion of the issue Radek ran into is appropriate to
put somewhere, with this example. The install procedure section of the
docs already includes a CFLAGS example:

./configure CC=/opt/bin/gcc CFLAGS='-O2 -pipe'

There is also a section talking about setting options like
--enable-cassert in the Developer's FAQ. Looking at all the info out
there about developer/debug builds, it's really kind of sketchy and
distributed though. No one place that pulls all the most common things
people need together into one resource.

What seems like the idea solution here is to add a new section to the
install procedure with brief coverage of this entire area. Here's a
prototype of text that might go there:

= Installation for development and debugging =

When modifying the PostgreSQL source code, or when trying to find the
source of a bug in the program, it may be helpful to build the program
in a way that makes this process easier. There are build-time only
changes that enable better error checking and debugging, including:

Pass --enable-cassert to configure. This can make bugs more visible,
because they cause operations to abort with a clear error. That makes
some types of debugging much easier. This is risky on a production
server, as described in the documentation for this parameter.

Pass --enable-debug to configure. This provides better information about
what the server is doing when looking at it using a debugger. It's less
risky to a production server than enabling assertions, and it normally
has less of a performance impact hgtoo. See its documentation for more
details.

Disable compiler optimization. When using a debugger to trace into the
source code of the server, steps may optimized away by the normal build
process. In some situations --enable-debug will disable such
optimization, but this is not always the case. Specifically disabling
optimization is possible with many compilers by setting the compiler
flags when configuration the source code build, such as:

./configure CFLAGS="-O0 -g"

This example for the gcc compiler disables optimizations, and tells the
compiler to provide extra debugging information most useful with the gdb
debugger.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

Excerpts from Radosław Smogura's message of jue jun 16 08:30:27 -0400 2011:
> Hello,
>
> I'm sending following patch which disables optimization when
> --enable-debug is passed. It was nasty (for me, at least) that debug
> build required passing of CFLAGS with -O0 to get nice traceable code.

I disagree with this change. Debug builds are very useful to have in
production, and you don't want to be running -O0 there. I have found
that you can use a src/Makefile.custom like this for those times when you
want to debug stuff in a particular set of files:

CFLAGS := $(patsubst -O2,-O0,$(CFLAGS))

Then you remove the .o files that you want to debug, and rerun make.
This places the burden on the developer wanting to mess with random code
changes. Of course, this means that production builds are not as
debuggable, but IME it's much less of a problem there.

--
Álvaro Herrera <alvherre(at)commandprompt(dot)com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Excerpts from Bernd Helmle's message of jue jun 16 09:37:24 -0400 2011:
>
>
> --On 16. Juni 2011 14:30:27 +0200 Radosław Smogura <rsmogura(at)softperience(dot)eu>
> wrote:
>
> > Hello,
> >
> > I'm sending following patch which disables optimization when --enable-debug
> > is passed. It was nasty (for me, at least) that debug build required passing
> > of CFLAGS with -O0 to get nice traceable code.
>
> -O0 hides bugs in your code (e.g. look at
> <http://archives.postgresql.org/message-id/9714F5232AB2C4FCFCB392D5@amenophis>
> and replies for an example to do it better). Doing this automatically on debug
> builds would be a step backwards.

Hah, seems I don't always do it the same way ;-)

--
Álvaro Herrera <alvherre(at)commandprompt(dot)com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
> I disagree with this change. Debug builds are very useful to have in
> production, and you don't want to be running -O0 there. I have found
> that you can use a src/Makefile.custom like this for those times when you
> want to debug stuff in a particular set of files:

> CFLAGS := $(patsubst -O2,-O0,$(CFLAGS))

> Then you remove the .o files that you want to debug, and rerun make.

FWIW, I only use Makefile.custom for more-or-less-permanent changes to
the build behavior of a particular machine. For one-shot things like
recompiling some particular file(s) at -O0, it's easier to do this:

rm foo.o
make PROFILE=-O0
reinstall postgres executable

The makefiles automatically add PROFILE at the end of CFLAGS, so you can
inject any compile flag this way --- I think the original intent was to
use it to add -pg for gprof-enabled builds. But it's handy for this.

BTW, if you're hacking Postgres code and don't already have a
"reinstall" script, you need one. Mine is basically

pg_ctl stop
cd $PGBLDROOT/src/backend
make install-bin
pg_ctl start

regards, tom lane

On Thu, 16 Jun 2011 16:00:21 -0400, Tom Lane wrote:
> Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
>> I disagree with this change. Debug builds are very useful to have
>> in
>> production, and you don't want to be running -O0 there. I have
>> found
>> that you can use a src/Makefile.custom like this for those times
>> when you
>> want to debug stuff in a particular set of files:
>
>> CFLAGS := $(patsubst -O2,-O0,$(CFLAGS))
>
>> Then you remove the .o files that you want to debug, and rerun make.
>
> FWIW, I only use Makefile.custom for more-or-less-permanent changes
> to
> the build behavior of a particular machine. For one-shot things like
> recompiling some particular file(s) at -O0, it's easier to do this:
>
> rm foo.o
> make PROFILE=-O0
> reinstall postgres executable
>
> The makefiles automatically add PROFILE at the end of CFLAGS, so you
> can
> inject any compile flag this way --- I think the original intent was
> to
> use it to add -pg for gprof-enabled builds. But it's handy for this.
>
> BTW, if you're hacking Postgres code and don't already have a
> "reinstall" script, you need one. Mine is basically
>
> pg_ctl stop
> cd $PGBLDROOT/src/backend
> make install-bin
> pg_ctl start
>
> regards, tom lane
Thanks,

Actually I do something like above, but good to know "install-bin"
target, I fired before "gmake -j5 install".

Regards,
Radek

On Thu, Jun 16, 2011 at 9:00 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> BTW, if you're hacking Postgres code and don't already have a
> "reinstall" script, you need one.  Mine is basically
>
>        pg_ctl stop
>        cd $PGBLDROOT/src/backend
>        make install-bin
>        pg_ctl start

I've always wondered what other people do to iterate quickly. It's a
bit of a pain that you can't just run the binary out of the build
tree. This looks a lot safer than some of the things I was considering
doing with symlinks.

--
greg

Greg Stark wrote:
> I've always wondered what other people do to iterate quickly.

I'd have bet money you had an elisp program for this by now!

The peg utility script I use makes a reinstall as simple as:

stop
peg build

The UI for peg is still is a little rough around switching to another
project when using git, and the PGDATA handling could be better. Being
able to give each patch I want to play with its own binary+data tree
with a couple of simple commands is the time consuming part to setup I
wanted to automate completely, and for that it works great:
https://github.com/gregs1104/peg

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

Greg Stark <stark(at)mit(dot)edu> Monday 20 of June 2011 03:39:12
> On Thu, Jun 16, 2011 at 9:00 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > BTW, if you're hacking Postgres code and don't already have a
> > "reinstall" script, you need one. Mine is basically
> >
> > pg_ctl stop
> > cd $PGBLDROOT/src/backend
> > make install-bin
> > pg_ctl start
>
> I've always wondered what other people do to iterate quickly. It's a
> bit of a pain that you can't just run the binary out of the build
> tree. This looks a lot safer than some of the things I was considering
> doing with symlinks.
I actually go to installation directory
and call in one line (simple because up arrow helps).

pg_ctl -D db stop; gmake -C ../postgresql -j5 install; pg_ctl -D db start

Regards,
Radek

Excerpts from Greg Smith's message of lun jun 20 00:25:08 -0400 2011:
> Greg Stark wrote:
> > I've always wondered what other people do to iterate quickly.
>
> I'd have bet money you had an elisp program for this by now!

Yeah :-)

> The peg utility script I use makes a reinstall as simple as:
>
> stop
> peg build

But you're building the entire server there, which was Tom's point --
you only need to build and reinstall the backend.

I have my own "runpg" utility which does a lot of these things too ...
The main difference (to Tom's approach) is that I don't use pg_ctl to
start/stop the server, because I always keep that running in a terminal,
which makes for easier debugging because the logs are always there and I
can ctrl-c it ... Well I guess it's pretty much the same thing, because
Tom probably has a script to stop the server.

--
Álvaro Herrera <alvherre(at)commandprompt(dot)com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
> Excerpts from Greg Smith's message of lun jun 20 00:25:08 -0400 2011:
>> The peg utility script I use makes a reinstall as simple as:
>>
>> stop
>> peg build

> But you're building the entire server there, which was Tom's point --
> you only need to build and reinstall the backend.

Right, I was trying to illustrate how to have minimal turnaround time
when testing a small code change. Rebuilding from scratch is slow
enough that you lose focus while waiting. (Or I do, anyway.)

Granted, stuff like ccache can help with that, but why not adopt a
process that's not slow in the first place?

regards, tom lane

On 06/20/2011 01:34 PM, Tom Lane wrote:
> I was trying to illustrate how to have minimal turnaround time
> when testing a small code change. Rebuilding from scratch is slow
> enough that you lose focus while waiting. (Or I do, anyway.)
>

I just keep upgrading to the fastest CPU I can possibly justify to avoid
losing focus; it goes fast with 8 cores. I was trying to demonstrate
that peg makes this very high level now, and I was more jousting at the
idea that everyone should bother to write their own individual reinstall
script.

The peg code makes it easy to assimilate whatever other neat
optimization ideas one might come across. I just pushed an update out
that absorbed this one, so now if you do:

stop
peg rebuild

It uses the install-bin trick you suggested. It even does a couple of
sanity checks so that it will probably fall back to a regular build if
it doesn't look like you have a good install and binary tree already.
Maybe I'll make a "reinstall" alias that does this combination next.

I don't expect to improve your workflow. But people who haven't already
invested a good chunk of work in automating things already will probably
take some time to catch up with where peg puts them on day one.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

I have applied the attached patch to help make suggestsions for server
developers. I didn't reproduce most of the text because it was already
listed with the options. Let me know if you want additional text.

---------------------------------------------------------------------------

Greg Smith wrote:
> On 06/16/2011 10:10 AM, Tom Lane wrote:
> > I could see providing some other nonstandard configure switch that
> > changed the default -O level ... but realistically, would that do
> > anything that you couldn't already do by setting CFLAGS, ie
> >
> > ./configure CFLAGS="-O0 -g"
> >
>
> I think a small discussion of the issue Radek ran into is appropriate to
> put somewhere, with this example. The install procedure section of the
> docs already includes a CFLAGS example:
>
> ./configure CC=/opt/bin/gcc CFLAGS='-O2 -pipe'
>
> There is also a section talking about setting options like
> --enable-cassert in the Developer's FAQ. Looking at all the info out
> there about developer/debug builds, it's really kind of sketchy and
> distributed though. No one place that pulls all the most common things
> people need together into one resource.
>
> What seems like the idea solution here is to add a new section to the
> install procedure with brief coverage of this entire area. Here's a
> prototype of text that might go there:
>
> = Installation for development and debugging =
>
> When modifying the PostgreSQL source code, or when trying to find the
> source of a bug in the program, it may be helpful to build the program
> in a way that makes this process easier. There are build-time only
> changes that enable better error checking and debugging, including:
>
> Pass --enable-cassert to configure. This can make bugs more visible,
> because they cause operations to abort with a clear error. That makes
> some types of debugging much easier. This is risky on a production
> server, as described in the documentation for this parameter.
>
> Pass --enable-debug to configure. This provides better information about
> what the server is doing when looking at it using a debugger. It's less
> risky to a production server than enabling assertions, and it normally
> has less of a performance impact hgtoo. See its documentation for more
> details.
>
> Disable compiler optimization. When using a debugger to trace into the
> source code of the server, steps may optimized away by the normal build
> process. In some situations --enable-debug will disable such
> optimization, but this is not always the case. Specifically disabling
> optimization is possible with many compilers by setting the compiler
> flags when configuration the source code build, such as:
>
> ./configure CFLAGS="-O0 -g"
>
> This example for the gcc compiler disables optimizations, and tells the
> compiler to provide extra debugging information most useful with the gdb
> debugger.
>
> --
> Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
> PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us
>
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On tis, 2011-11-29 at 16:32 -0500, Bruce Momjian wrote:
> I have applied the attached patch to help make suggestsions for server
> developers. I didn't reproduce most of the text because it was already
> listed with the options. Let me know if you want additional text.

Advising "server developers" to use CFLAGS="-O0 -g", without
qualification, is dangerous, in my mind, because that loses a lot of
compiler checks. The only reason to use -O0 is when you really need to
debug something in single steps and you can't make sense of it any other
way.

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> On tis, 2011-11-29 at 16:32 -0500, Bruce Momjian wrote:
>> I have applied the attached patch to help make suggestsions for server
>> developers. I didn't reproduce most of the text because it was already
>> listed with the options. Let me know if you want additional text.

> Advising "server developers" to use CFLAGS="-O0 -g", without
> qualification, is dangerous, in my mind, because that loses a lot of
> compiler checks. The only reason to use -O0 is when you really need to
> debug something in single steps and you can't make sense of it any other
> way.

Yes. -O0 is really a pretty horrid default choice, and we should NOT be
recommending it, especially not with no discussion of the disadvantages.

regards, tom lane

Tom Lane wrote:
> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> > On tis, 2011-11-29 at 16:32 -0500, Bruce Momjian wrote:
> >> I have applied the attached patch to help make suggestsions for server
> >> developers. I didn't reproduce most of the text because it was already
> >> listed with the options. Let me know if you want additional text.
>
> > Advising "server developers" to use CFLAGS="-O0 -g", without
> > qualification, is dangerous, in my mind, because that loses a lot of
> > compiler checks. The only reason to use -O0 is when you really need to
> > debug something in single steps and you can't make sense of it any other
> > way.
>
> Yes. -O0 is really a pretty horrid default choice, and we should NOT be
> recommending it, especially not with no discussion of the disadvantages.

I have applied the attached patch to mention the debugger. OK?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On Tue, Nov 29, 2011 at 7:13 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
>> Yes.  -O0 is really a pretty horrid default choice, and we should NOT be
>> recommending it, especially not with no discussion of the disadvantages.
>
> I have applied the attached patch to mention the debugger.  OK?

Not really. That's still too much encouragement. I think you should
just take that part out altogether.

Discussing changes before committing them might be a good idea, too.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas wrote:
> On Tue, Nov 29, 2011 at 7:13 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> >> Yes. ?-O0 is really a pretty horrid default choice, and we should NOT be
> >> recommending it, especially not with no discussion of the disadvantages.
> >
> > I have applied the attached patch to mention the debugger. ?OK?
>
> Not really. That's still too much encouragement. I think you should
> just take that part out altogether.
>
> Discussing changes before committing them might be a good idea, too.

Well, the original patch got no replies, so I figured it was OK.

I modified the docs to just mention that a debugger might need special
flags.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

Bruce Momjian <bruce(at)momjian(dot)us> writes:
> I have applied the attached patch to mention the debugger. OK?

> Server developers should consider using the configure options
> <option>--enable-cassert</> and <option>--enable-debug</> to enhance the
> ability to detect and debug server errors. They should also consider
> ! running configure with <literal>CFLAGS="-O0 -g"</> if using a debugger.

I still think this is basically useless. If we're going to mention the
topic at all, we should provide enough information to be helpful, which
this does not. Furthermore, it's concretely wrong in that it suggests
you need to say -g when --enable-debug already does that, and that it
fails to note that all this advice is gcc-specific.

I suggest wording along these lines:

When developing code inside the server, it's recommended to
use the configure options --enable-cassert, which turns on many
run-time error checks, and --enable-debug, which improves the
usefulness of debugging tools.

If you use gcc, it's best to build with an optimization level
of at least -O1, because using level -O0 disables some important
compiler warnings (such as use of an uninitialized variable).
However, nonzero optimization levels can complicate debugging
because stepping through the compiled code will usually not
match up one-to-one with source code lines. If you get confused
while trying to debug optimized code, recompile the specific
file(s) of interest with -O0. An easy way to do this with the
Unix makefiles is "make PROFILE=-O0 file.o".

regards, tom lane

Tom Lane wrote:
> Bruce Momjian <bruce(at)momjian(dot)us> writes:
> > I have applied the attached patch to mention the debugger. OK?
>
> > Server developers should consider using the configure options
> > <option>--enable-cassert</> and <option>--enable-debug</> to enhance the
> > ability to detect and debug server errors. They should also consider
> > ! running configure with <literal>CFLAGS="-O0 -g"</> if using a debugger.
>
> I still think this is basically useless. If we're going to mention the
> topic at all, we should provide enough information to be helpful, which
> this does not. Furthermore, it's concretely wrong in that it suggests
> you need to say -g when --enable-debug already does that, and that it
> fails to note that all this advice is gcc-specific.
>
> I suggest wording along these lines:
>
> When developing code inside the server, it's recommended to
> use the configure options --enable-cassert, which turns on many
> run-time error checks, and --enable-debug, which improves the
> usefulness of debugging tools.
>
> If you use gcc, it's best to build with an optimization level
> of at least -O1, because using level -O0 disables some important
> compiler warnings (such as use of an uninitialized variable).
> However, nonzero optimization levels can complicate debugging
> because stepping through the compiled code will usually not
> match up one-to-one with source code lines. If you get confused
> while trying to debug optimized code, recompile the specific
> file(s) of interest with -O0. An easy way to do this with the
> Unix makefiles is "make PROFILE=-O0 file.o".

OK, I make some slight modifications and applied the attached patch.

Ideally we could tell everyone to read the developer's FAQ, but that is
too large for people who are debugging problems in our shipped code ---
that is why I was excited to get something into our main docs.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +