Compile errors when building 32-bit on 64-bit system

I was going to post this to pgsql-hackers but figured it might be more
suitable on this list first. I'm trying to compile a 32-bit PostgreSQL
client on a 64-bit system, however the SPI module is not respecting the
CFLAGS and LDFLAGS option "-m32."

My configure line is as follows:

CFLAGS="-m32" LD="ld -m32 -melf_i386" ./configure
--prefix=/opt/nmail32/pgsql --enable-thread-safety --with-openssl
--with-libxml --with-includes="..." --with-libraries="..."

All libraries listed in the --with-libraries are 32-bit. The "make"
command fails at the "contrib/spi" module with the error:

/usr/bin/ld: warning: i386 architecture of input file `refint.o' is
incompatible with i386:x86-64 output

The 2 commands before it are:

gcc -m32 -Wall -Wmissing-prototypes -Wpointer-arith -Winline
-Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing
-fwrapv -fpic -DREFINT_VERBOSE -I. -I../../src/include -D_GNU_SOURCE
-I/opt/nmail32/libxml2/include/libxml2 -I/opt/nmail32/zlib/include
-I/opt/nmail32/readline/include -I/opt/nmail32/openssl/include -c -o
refint.o refint.c
gcc -shared -o refint.so refint.o

The first command (the compile) has the "-m32" flag set, however the
linking command doesn't - causing the above message. If I run the above
2 commands manually, but add "-m32" in the second one (i.e. gcc -m32
-shared -o refint.so refint.o) it succeeds. The same failure also
occurs for the "autoinc.c" file in the same contrib module.

Thanks,
Andy