Re: build environment: a different makefile

Currently trying to "enhance" the way we can make binaries that run
on Solaris. One thing I found was a scalability bottleneck in the
use of the ProcArrayLock. (this one has also been reported by a
couple of my colleagues).
One big user of this lock is GetSnapshotData. After it has taken
this lock it does its work and releases it again. While it is holding
the lock it is not doing any system calls and the lock holding
process is barely preempted.

The only way to make this code faster is making the code use less CPU
cycles to achieve its goal. One way is having the compiler do some
strong code in-lining.
The SunStudio compiler we are using fortunately has an option for
this. Unfortunately there are restrictions. One restriction I face is
its inability to deal with "ld -r"s. These are used in the build
environment to create all the SUBSYS.o object files.

I was hoping someone in the community already has a makefile that
"just" creates object files from C-sources directly that I can use to
try out the effect of in-lining to the performance of postgres.
Any other hints to achieve my goal are welcome too, of-course. Please
note that in-lining is done in both the compiler and the linker.

Thanks,
Paul

------------------------------------------------------------------------
---------------------
Paul van den Bogaard
Paul(dot)vandenBogaard(at)sun(dot)com
ISV-E -- ISV Engineering, Opensource Engineering group

Sun Microsystems, Inc phone: +31
334 515 918
Saturnus 1
extentsion: x (70)15918
3824 ME Amersfoort mobile: +31
651 913 354
The Netherlands
fax: +31 334 515 001

Paul van den Bogaard wrote:
> The SunStudio compiler we are using fortunately has an option for  
> this. Unfortunately there are restrictions. One restriction I face is  
> its inability to deal with "ld -r"s. These are used in the build  
> environment to create all the SUBSYS.o object files.
>
> I was hoping someone in the community already has a makefile that  
> "just" creates object files from C-sources directly that I can use to  
> try out the effect of in-lining to the performance of postgres.

I don't know if anyone has a makefile for it, but the following seems to work
for me:

pgsql/src/backend$ cc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Winline -Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing -g -L../../src/port -Wl,-rpath,'/home/peter/devel/pg83/pg-install/lib' -Wl,-E $(find -name "*.o" | grep -v SUBSYS | grep -v conversion_procs) ../../src/timezone/SUBSYS.o ../../src/port/libpgport_srv.a -lxslt -lxml2 -lpam -lssl -lcrypto -lgssapi_krb5 -lcrypt -ldl -lm -lldap -o postgres

If you find that the optimizations you are hoping for are useful, I'm sure
we could put an option of that sort somewhere in the makefiles.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> I don't know if anyone has a makefile for it, but the following seems to work
> for me:

> pgsql/src/backend$ cc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Winline -Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing -g -L../../src/port -Wl,-rpath,'/home/peter/devel/pg83/pg-install/lib' -Wl,-E $(find -name "*.o" | grep -v SUBSYS | grep -v conversion_procs) ../../src/timezone/SUBSYS.o ../../src/port/libpgport_srv.a -lxslt -lxml2 -lpam -lssl -lcrypto -lgssapi_krb5 -lcrypt -ldl -lm -lldap -o postgres

> If you find that the optimizations you are hoping for are useful, I'm sure
> we could put an option of that sort somewhere in the makefiles.

I've sometimes wondered whether the SUBSYS.o files really offer any
advantage compared to just linking all the individual .o files. They
certainly eat disk space, but perhaps they save some time ... or perhaps
not, especially in a one-off build.

I suppose that we might fall foul of command line length limits on
some platforms :-(. The output of your find command amounts to nearly
11000 characters in HEAD.

regards, tom lane

On Feb 6, 2008 11:12 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> I've sometimes wondered whether the SUBSYS.o files really offer any
> advantage compared to just linking all the individual .o files. They
> certainly eat disk space, but perhaps they save some time ... or perhaps
> not, especially in a one-off build.

Getting rid of them would certainly make building OSX universal binaries easier.

> I suppose that we might fall foul of command line length limits on
> some platforms :-(. The output of your find command amounts to nearly
> 11000 characters in HEAD.

What do other large build systems do?

/D

On Thu, Feb 07, 2008 at 08:09:24AM +0000, Dave Page wrote:
> On Feb 6, 2008 11:12 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
> > I've sometimes wondered whether the SUBSYS.o files really offer any
> > advantage compared to just linking all the individual .o files. They
> > certainly eat disk space, but perhaps they save some time ... or perhaps
> > not, especially in a one-off build.
>
> Getting rid of them would certainly make building OSX universal binaries easier.
>
> > I suppose that we might fall foul of command line length limits on
> > some platforms :-(. The output of your find command amounts to nearly
> > 11000 characters in HEAD.
>
> What do other large build systems do?

FWIW, the MSVC build ends up writing the list of object files to a temp
file and then having the linker read that list. (This is all done behind
the scenes though, it's not something we made up) IIRC the gcc linker can
also take the commandline from a file instead of the actual commandline,
which should be workable I think.

//Magnus

Am Donnerstag, 7. Februar 2008 schrieb Dave Page:
> What do other large build systems do?

I strongly suspect that they just fail unless run with GNU tools or some other
well-defined tool set.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut napsal(a):
> Paul van den Bogaard wrote:
>> The SunStudio compiler we are using fortunately has an option for
>> this. Unfortunately there are restrictions. One restriction I face is
>> its inability to deal with "ld -r"s. These are used in the build
>> environment to create all the SUBSYS.o object files.
>>
>> I was hoping someone in the community already has a makefile that
>> "just" creates object files from C-sources directly that I can use to
>> try out the effect of in-lining to the performance of postgres.
>
> I don't know if anyone has a makefile for it, but the following seems to work
> for me:
>
> pgsql/src/backend$ cc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Winline -Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing -g -L../../src/port -Wl,-rpath,'/home/peter/devel/pg83/pg-install/lib' -Wl,-E $(find -name "*.o" | grep -v SUBSYS | grep -v conversion_procs) ../../src/timezone/SUBSYS.o ../../src/port/libpgport_srv.a -lxslt -lxml2 -lpam -lssl -lcrypto -lgssapi_krb5 -lcrypt -ldl -lm -lldap -o postgres
>
> If you find that the optimizations you are hoping for are useful, I'm sure
> we could put an option of that sort somewhere in the makefiles.

Peter,
Suns studio performs inline optimization on -xO3 level. Optimization levels are
different from GCC. Maximal level is -xO5. I think Paul plays with xipo flag
which requires at least -xO4.

See
http://docs.sun.com/app/docs/doc/819-5265/bjapp?a=view

Zdenek

Zdenek is right.

Normal inlining using -O3 or higher means inlining within the source
file. I currently try to see the effect of inlining over all the
sources. The -xipo flag is specific to the Sun Studio (version 12)
suite. It creates large objects that now include meta data for the
final linking stage. During this linking stage all this stuff is
read, analyzed an the object files are changed. Later this can be
expanded by compiler profiling and/or other trickery. BTW the actual
binary does *not* include all that metadata and is just bigger due to
all that code being inlined in those objects.

This trick has been done for other applications and we find in
general significant approvement. Not a guarantee though ...

However my "suite" has a dependecy on dtrace so the initial idea I
saw from Peter is not complete. Currently creating a quick and dirty
build script. Just to get that "ipo"-ing done and tested.

I'll keep you all updated on the results. If we decide it is too good
to be excluded can we start thinking about an adaption of the build
environement. Hope this sound fair and acceptable to all.

Thanks so far for all that feedback

Cheers
Paul

BTW I build and test on a 16 SPARC @1350MHz V890. 64 bit mode,
currently 16GB of shared memory, and 7 disk arrays (RAID0). This to
exclude the IO part of the equation as much as possible, so we can
focus on CPU related matter: the need for faster (smarter) code
pathes and/or scalability issues like ProcArrayLock contention.

On 7-feb-2008, at 11:08, Zdenek Kotala wrote:

> Peter Eisentraut napsal(a):
>> Paul van den Bogaard wrote:
>>> The SunStudio compiler we are using fortunately has an option
>>> for this. Unfortunately there are restrictions. One restriction
>>> I face is its inability to deal with "ld -r"s. These are used in
>>> the build environment to create all the SUBSYS.o object files.
>>>
>>> I was hoping someone in the community already has a makefile
>>> that "just" creates object files from C-sources directly that I
>>> can use to try out the effect of in-lining to the performance of
>>> postgres.
>> I don't know if anyone has a makefile for it, but the following
>> seems to work
>> for me:
>> pgsql/src/backend$ cc -O2 -Wall -Wmissing-prototypes -Wpointer-
>> arith -Winline -Wdeclaration-after-statement -Wendif-labels -fno-
>> strict-aliasing -g -L../../src/port -Wl,-rpath,'/home/peter/devel/
>> pg83/pg-install/lib' -Wl,-E $(find -name "*.o" | grep -v SUBSYS |
>> grep -v conversion_procs) ../../src/timezone/SUBSYS.o ../../src/
>> port/libpgport_srv.a -lxslt -lxml2 -lpam -lssl -lcrypto -
>> lgssapi_krb5 -lcrypt -ldl -lm -lldap -o postgres
>> If you find that the optimizations you are hoping for are useful,
>> I'm sure
>> we could put an option of that sort somewhere in the makefiles.
>
> Peter,
> Suns studio performs inline optimization on -xO3 level.
> Optimization levels are different from GCC. Maximal level is -xO5.
> I think Paul plays with xipo flag which requires at least -xO4.
>
> See
> http://docs.sun.com/app/docs/doc/819-5265/bjapp?a=view
>
> Zdenek
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster

------------------------------------------------------------------------
---------------------
Paul van den Bogaard
Paul(dot)vandenBogaard(at)sun(dot)com
ISV-E -- ISV Engineering, Opensource Engineering group

Sun Microsystems, Inc phone: +31
334 515 918
Saturnus 1
extentsion: x (70)15918
3824 ME Amersfoort mobile: +31
651 913 354
The Netherlands
fax: +31 334 515 001

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> On Thu, Feb 07, 2008 at 08:09:24AM +0000, Dave Page wrote:
>> What do other large build systems do?

> FWIW, the MSVC build ends up writing the list of object files to a temp
> file and then having the linker read that list. (This is all done behind
> the scenes though, it's not something we made up) IIRC the gcc linker can
> also take the commandline from a file instead of the actual commandline,
> which should be workable I think.

Hmm. I'm not sure that's universal, but if it is then a simple
incremental improvement on what we're doing now would be to replace the
SUBSYS.o files with "subsys include files" that just list all the .o
files to be included.

regards, tom lane

Am Mittwoch, 6. Februar 2008 schrieb Paul van den Bogaard:
> I was hoping someone in the community already has a makefile that
> "just" creates object files from C-sources directly that I can use to
> try out the effect of in-lining to the performance of postgres.

This is now the default in 8.4devel. Let us know what you find out.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter,

finally I had a chance to check it out. One word: perfect!

Thanks
Paul

On 25-feb-2008, at 19:09, Peter Eisentraut wrote:

> Am Mittwoch, 6. Februar 2008 schrieb Paul van den Bogaard:
>> I was hoping someone in the community already has a makefile that
>> "just" creates object files from C-sources directly that I can use to
>> try out the effect of in-lining to the performance of postgres.
>
> This is now the default in 8.4devel. Let us know what you find out.
>
> --
> Peter Eisentraut
> http://developer.postgresql.org/~petere/
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 5: don't forget to increase your free space map settings

------------------------------------------------------------------------
---------------------
Paul van den Bogaard
Paul(dot)vandenBogaard(at)sun(dot)com
ISV-E -- ISV Engineering, Opensource Engineering group

Sun Microsystems, Inc phone: +31
334 515 918
Saturnus 1
extentsion: x (70)15918
3824 ME Amersfoort mobile: +31
651 913 354
The Netherlands
fax: +31 334 515 001