Re: static or dynamic libpgport

Recently I attempted to build an external package (pg_bulkload) against
the latest Fedora packages. Unfortunately this fails, as pgxs adds
"-lpgport" to any link line for an executable, and the corresponding
libpgport.a isn't there. And in fact, pg_bulkload does use some of the
functionality there (e.g. pg_strncasecmp), so just stripping "-lpgport"
out doesn't work either.

This happened because Fedora packaging guidelines
<http://fedoraproject.org/wiki/Packaging:Guidelines#Packaging_Static_Libraries>
are strongly against shipping static libraries, and so all the
PostgreSQL static libraries are excluded from the distribution (and I
believe there are similar restrictions for RHEL). Of these libraries, I
believe the only one that is *only* built as a static library is libpgport.

Is there any good reason why we shouldn't build and install a dynamic
libpgport.so?

(Of course, you could say "use the community RPMs", but that would be a
bit of a cop out. Some organizations have a perfectly reasonable policy
or requiring use of vendor packages wherever possible, since vendors are
naturally only going to support packages they provide. So either we
should be arguing to the Fedora/RedHat people that they should ship the
static library, or we should be providing them with a dynamic one, ISTM.)

cheers

andrew

On 9 December 2011 16:13, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
> Is there any good reason why we shouldn't build and install a dynamic
> libpgport.so?

+1 in favour of building and installing a dynamic libpgport.so. I
generally agree with your analysis.

I've seen this issue crop up a good few times now. I'm a Fedora user
myself, but about 2 years ago I got into a "he said she said"
situation with an OpenSUSE package maintainer over this, when I had to
build Slony on that platform. I'm a bit hazy on the details now, but
iirc he thought that it wasn't necessary to ship libpgport.a in
particular (though I don't think that they have a beef with static
libraries generally) - maybe they took a cue from Redhat there?

--
Peter Geoghegan       http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services

On 11-12-09 11:13 AM, Andrew Dunstan wrote:
> Recently I attempted to build an external package (pg_bulkload)
> against the latest Fedora packages. Unfortunately this fails, as pgxs
> adds "-lpgport" to any link line for an executable, and the
> corresponding libpgport.a isn't there. And in fact, pg_bulkload does
> use some of the functionality there (e.g. pg_strncasecmp), so just
> stripping "-lpgport" out doesn't work either.
>
> This happened because Fedora packaging guidelines
> <http://fedoraproject.org/wiki/Packaging:Guidelines#Packaging_Static_Libraries>
> are strongly against shipping static libraries, and so all the
> PostgreSQL static libraries are excluded from the distribution (and I
> believe there are similar restrictions for RHEL). Of these libraries,
> I believe the only one that is *only* built as a static library is
> libpgport.
>
> Is there any good reason why we shouldn't build and install a dynamic
> libpgport.so?

+1

We've struggled with slony and pgport because so many users have had
problems with pgport not being included in some distributions. It has
some useful functions, I think recent versions of slony use it on win32
but don't elsewhere. Wee have had at least one patch floating around
that makes conditionally includes certain small behaviours in slony
based on if pgport is available or not based on a configure check.

What package would a shared static pgport be installed with? Slony
requires a server + headers to build but slon and slonik only have a
runtime dependency on libpq (I don't know if anyone installs slon/slonik
on a machine without a postgresql server but you could)

Steve

> cheers
>
> andrew
>

On 12/09/2011 01:01 PM, Steve Singer wrote:
> On 11-12-09 11:13 AM, Andrew Dunstan wrote:
>> Recently I attempted to build an external package (pg_bulkload)
>> against the latest Fedora packages. Unfortunately this fails, as pgxs
>> adds "-lpgport" to any link line for an executable, and the
>> corresponding libpgport.a isn't there. And in fact, pg_bulkload does
>> use some of the functionality there (e.g. pg_strncasecmp), so just
>> stripping "-lpgport" out doesn't work either.
>>
>> This happened because Fedora packaging guidelines
>> <http://fedoraproject.org/wiki/Packaging:Guidelines#Packaging_Static_Libraries>
>> are strongly against shipping static libraries, and so all the
>> PostgreSQL static libraries are excluded from the distribution (and I
>> believe there are similar restrictions for RHEL). Of these libraries,
>> I believe the only one that is *only* built as a static library is
>> libpgport.
>>
>> Is there any good reason why we shouldn't build and install a dynamic
>> libpgport.so?
>
> +1
>
> We've struggled with slony and pgport because so many users have had
> problems with pgport not being included in some distributions. It has
> some useful functions, I think recent versions of slony use it on
> win32 but don't elsewhere. Wee have had at least one patch floating
> around that makes conditionally includes certain small behaviours in
> slony based on if pgport is available or not based on a configure check.
>
> What package would a shared static pgport be installed with? Slony
> requires a server + headers to build but slon and slonik only have a
> runtime dependency on libpq (I don't know if anyone installs
> slon/slonik on a machine without a postgresql server but you could)
>
>

In the Fedora world, a static lib would go in postgresql-devel, but a
dynamic lib would go in postgresql-libs, which is also where libpq is
shipped.

cheers

andrew

Andrew Dunstan wrote:
> >> Is there any good reason why we shouldn't build and install a dynamic
> >> libpgport.so?
> >
> > +1
> >
> > We've struggled with slony and pgport because so many users have had
> > problems with pgport not being included in some distributions. It has
> > some useful functions, I think recent versions of slony use it on
> > win32 but don't elsewhere. Wee have had at least one patch floating
> > around that makes conditionally includes certain small behaviours in
> > slony based on if pgport is available or not based on a configure check.
> >
> > What package would a shared static pgport be installed with? Slony
> > requires a server + headers to build but slon and slonik only have a
> > runtime dependency on libpq (I don't know if anyone installs
> > slon/slonik on a machine without a postgresql server but you could)
> >
> >
>
> In the Fedora world, a static lib would go in postgresql-devel, but a
> dynamic lib would go in postgresql-libs, which is also where libpq is
> shipped.

I am not against shipping a dynamic libpgport, but I will just point out
that this was never intended or anticipated. Are there any symbols in
there that might conflict with other software?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On 12/09/2011 06:27 PM, Bruce Momjian wrote:
>> In the Fedora world, a static lib would go in postgresql-devel, but a
>> dynamic lib would go in postgresql-libs, which is also where libpq is
>> shipped.
> I am not against shipping a dynamic libpgport, but I will just point out
> that this was never intended or anticipated. Are there any symbols in
> there that might conflict with other software?

Possibly. Below is a list of symbols from a recent build. The other
thing is we'd need to turn on flags that make the object suitable for a
dynamic library (e.g. -fpic). I'm not sure if that has any significant
impact - probably not or else people would avoid using them far more.

I think we should deal with this. Just Peter's and Steve's responses
upthread indicate a demand, and I came into this because a customer
encountered enormous difficulty in doing an out of tree build of a well
known piece of support software, so this is clearly a pain point. If
we're not going to do it, we should probably think about adjusting
people's expectations.

[andrew(at)diego port]$ nm libpgport.a | grep ' T '
00000000 T strlcat
00000000 T strlcpy
00000000 T getpeereid
00000000 T pg_get_encoding_from_locale
00000000 T pgfnames
000001a0 T pgfnames_cleanup
000001e0 T rmtree
00000240 T find_my_exec
000005f0 T find_other_exec
00000500 T pclose_check
00000760 T set_pglocale_pgservice
00000000 T inet_net_ntop
00000030 T pg_set_block
00000000 T pg_set_noblock
00000280 T canonicalize_path
00000110 T first_dir_separator
00000140 T first_path_var_separator
00000800 T get_doc_path
00000720 T get_etc_path
00000860 T get_home_path
00000820 T get_html_path
00000740 T get_include_path
00000780 T get_includeserver_path
000007a0 T get_lib_path
000007e0 T get_locale_path
00000840 T get_man_path
000008e0 T get_parent_directory
00000760 T get_pkginclude_path
000007c0 T get_pkglib_path
00000690 T get_progname
00000700 T get_share_path
000001b0 T join_path_components
00000170 T last_dir_separator
000001a0 T make_native_path
00000560 T path_contains_parent_reference
00000630 T path_is_prefix_of_path
00000610 T path_is_relative_and_below_cwd
00000000 T pg_check_dir
00000000 T pg_mkdir_p
00000000 T pg_usleep
00000330 T pg_ascii_tolower
00000310 T pg_ascii_toupper
00000000 T pg_strcasecmp
00000100 T pg_strncasecmp
00000290 T pg_tolower
00000210 T pg_toupper
000000d0 T pg_qsort
000000f0 T qsort_arg
00000000 T simple_prompt
00000010 T pqGetpwuid
00000000 T pqStrerror

cheers

andrew

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> On 12/09/2011 06:27 PM, Bruce Momjian wrote:
>> I am not against shipping a dynamic libpgport, but I will just point out
>> that this was never intended or anticipated. Are there any symbols in
>> there that might conflict with other software?

> Possibly. Below is a list of symbols from a recent build.

This doesn't seem like much of an issue to me, since anything wanting to
link against libpgport would be designed to work with whatever it
provides, no?

> The other
> thing is we'd need to turn on flags that make the object suitable for a
> dynamic library (e.g. -fpic).

Right now, libpq laboriously rebuilds all the .o files it needs from
src/port/ so as to get them with -fpic. It would be nice if we could
clean that up while we're doing this. It might be all right to always
build the client-side version of libpgport with -fpic, though I'd be sad
if that leaked into the server-side build.

regards, tom lane

On 12/10/2011 08:26 PM, Tom Lane wrote:
>
>> The other
>> thing is we'd need to turn on flags that make the object suitable for a
>> dynamic library (e.g. -fpic).
> Right now, libpq laboriously rebuilds all the .o files it needs from
> src/port/ so as to get them with -fpic. It would be nice if we could
> clean that up while we're doing this. It might be all right to always
> build the client-side version of libpgport with -fpic, though I'd be sad
> if that leaked into the server-side build.
>

Here's a small diff that seems to build things the right way. No leakage
of -fpic into the server side code. Still a deal of work to do, but it's
a start.

Would we want to link our own non-backend executables against the shared
lib? That would almost certainly break the buildfarm for Windows builds,
as it only currently copies the libpq DLL into the bin directory. That's
no reason on its own not to do it, of course, and there are only a
couple of owners other than me anyway, so it would be easy to fix.

How do you want to proceed for libpq (and the ecpg library cases that do
the same thing)? Just link in the object files directly?

cheers

andrew

*** Makefile 2011-12-03 17:21:59.944509111 -0500
--- GNUmakefile 2011-12-12 12:39:43.176260505 -0500
***************
*** 37,47 ****
# foo_srv.o and foo.o are both built from foo.c, but only foo.o has
-DFRONTEND
OBJS_SRV = $(OBJS:%.o=%_srv.o)

! all: libpgport.a libpgport_srv.a

! # libpgport is needed by some contrib
! install: all installdirs
! $(INSTALL_STLIB) libpgport.a '$(DESTDIR)$(libdir)/libpgport.a'

installdirs:
$(MKDIR_P) '$(DESTDIR)$(libdir)'
--- 37,52 ----
# foo_srv.o and foo.o are both built from foo.c, but only foo.o has
-DFRONTEND
OBJS_SRV = $(OBJS:%.o=%_srv.o)

! NAME = pgport
! SO_MAJOR_VERSION= 1
! SO_MINOR_VERSION= 1

! include $(top_srcdir)/src/Makefile.shlib
!
! all: all-lib libpgport_srv.a
!
! # libpgport is needed by any exe built with pgxs
! install: all installdirs install-lib

installdirs:
$(MKDIR_P) '$(DESTDIR)$(libdir)'
***************
*** 49,57 ****
uninstall:
rm -f '$(DESTDIR)$(libdir)/libpgport.a'

- libpgport.a: $(OBJS)
- $(AR) $(AROPT) $@ $^
-
# thread.o needs PTHREAD_CFLAGS (but thread_srv.o does not)
thread.o: thread.c
$(CC) $(CFLAGS) $(CPPFLAGS) $(PTHREAD_CFLAGS) -c $<
--- 54,59 ----
***************
*** 64,70 ****
$(AR) $(AROPT) $@ $^

%_srv.o: %.c
! $(CC) $(CFLAGS) $(subst -DFRONTEND,, $(CPPFLAGS)) -c $< -o $@

$(OBJS_SRV): | submake-errcodes

--- 66,72 ----
$(AR) $(AROPT) $@ $^

%_srv.o: %.c
! $(CC) $(subst $(CFLAGS_SL),,$(CFLAGS)) $(subst -DFRONTEND,,
$(CPPFLAGS)) -c $< -o $@

$(OBJS_SRV): | submake-errcodes

***************
*** 97,100 ****
echo "#define MANDIR \"$(mandir)\"" >>$@

clean distclean maintainer-clean:
! rm -f libpgport.a libpgport_srv.a $(OBJS) $(OBJS_SRV)
pg_config_paths.h
--- 99,102 ----
echo "#define MANDIR \"$(mandir)\"" >>$@

clean distclean maintainer-clean:
! rm -f libpgport.so* libpgport.a libpgport_srv.a $(OBJS)
$(OBJS_SRV) pg_config_paths.h

On lör, 2011-12-10 at 20:26 -0500, Tom Lane wrote:
> > The other
> > thing is we'd need to turn on flags that make the object suitable for a
> > dynamic library (e.g. -fpic).
>
> Right now, libpq laboriously rebuilds all the .o files it needs from
> src/port/ so as to get them with -fpic. It would be nice if we could
> clean that up while we're doing this. It might be all right to always
> build the client-side version of libpgport with -fpic, though I'd be sad
> if that leaked into the server-side build.

So would we continue to build the client binaries (psql, pg_dump, etc.)
against the static libpgport.a, thus keeping it "invisible" there, or
would we dynamically link them, thus creating a new dependency.

On fre, 2011-12-09 at 11:13 -0500, Andrew Dunstan wrote:
> Is there any good reason why we shouldn't build and install a dynamic
> libpgport.so?

Just note, if you do this, you need to carefully manage API, ABI,
soname, symbol list, and all that. Every time you tweak configure's
decision about when to include a replacement function, you need to
change the library version. Every time you remove a function, you need
to change the soname. Every backpatched portability fix has the
potential to escalate to a full shared library versioning dance.
Downstream packagers will be delighted, especially if this requires
changing the package name every three minor releases.

To see what this can lead to in the extreme, check the dependencies that
bind has on its internal libraries:

bind9 depends: libbind9-60, libdns69, libisc62, libisccc60, libisccfg62, liblwres60

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> On lr, 2011-12-10 at 20:26 -0500, Tom Lane wrote:
>> Right now, libpq laboriously rebuilds all the .o files it needs from
>> src/port/ so as to get them with -fpic. It would be nice if we could
>> clean that up while we're doing this. It might be all right to always
>> build the client-side version of libpgport with -fpic, though I'd be sad
>> if that leaked into the server-side build.

> So would we continue to build the client binaries (psql, pg_dump, etc.)
> against the static libpgport.a, thus keeping it "invisible" there, or
> would we dynamically link them, thus creating a new dependency.

I think that if possible we should avoid creating a new dependency for
either the client binaries or libpq.so itself; what I suggest above
is only a simplification of the build process for libpq. If we create
a new dependency we risk packagers breaking things by forgetting to
include it.

The Fedora/RHEL rule against static libraries is meant to prevent
situations where changes in a library would require rebuilding other
packages to get the fixes in place. If we had to make a quick security
fix in libpq, for example, it would suck if dozens of other packages had
to be rebuilt to propagate the change everywhere. However, I don't think
that concern applies to programs that are in the same source package as
the library --- they'd get rebuilt anyway. So I see nothing wrong with
continuing to statically link these .o files into files belonging to the
postgresql package. It's just that I can't export them in a .a file for
*other* source packages to use.

(Whether a security issue in libpgport is really likely to happen is not
a question that this policy concerns itself with ...)

regards, tom lane

On 12/12/2011 02:59 PM, Tom Lane wrote:
> Peter Eisentraut<peter_e(at)gmx(dot)net> writes:
>> On lör, 2011-12-10 at 20:26 -0500, Tom Lane wrote:
>>> Right now, libpq laboriously rebuilds all the .o files it needs from
>>> src/port/ so as to get them with -fpic. It would be nice if we could
>>> clean that up while we're doing this. It might be all right to always
>>> build the client-side version of libpgport with -fpic, though I'd be sad
>>> if that leaked into the server-side build.
>> So would we continue to build the client binaries (psql, pg_dump, etc.)
>> against the static libpgport.a, thus keeping it "invisible" there, or
>> would we dynamically link them, thus creating a new dependency.
> I think that if possible we should avoid creating a new dependency for
> either the client binaries or libpq.so itself; what I suggest above
> is only a simplification of the build process for libpq. If we create
> a new dependency we risk packagers breaking things by forgetting to
> include it.
>
>

OK, I'll work on this basis. The downside is that we'll be building it
but not using it, but I can see the advantages.

cheers

andrew

On 12/12/2011 02:59 PM, Tom Lane wrote:
> Peter Eisentraut<peter_e(at)gmx(dot)net> writes:
>> On lör, 2011-12-10 at 20:26 -0500, Tom Lane wrote:
>>> Right now, libpq laboriously rebuilds all the .o files it needs from
>>> src/port/ so as to get them with -fpic. It would be nice if we could
>>> clean that up while we're doing this. It might be all right to always
>>> build the client-side version of libpgport with -fpic, though I'd be sad
>>> if that leaked into the server-side build.
>> So would we continue to build the client binaries (psql, pg_dump, etc.)
>> against the static libpgport.a, thus keeping it "invisible" there, or
>> would we dynamically link them, thus creating a new dependency.
> I think that if possible we should avoid creating a new dependency for
> either the client binaries or libpq.so itself; what I suggest above
> is only a simplification of the build process for libpq. If we create
> a new dependency we risk packagers breaking things by forgetting to
> include it.
>
> The Fedora/RHEL rule against static libraries is meant to prevent
> situations where changes in a library would require rebuilding other
> packages to get the fixes in place. If we had to make a quick security
> fix in libpq, for example, it would suck if dozens of other packages had
> to be rebuilt to propagate the change everywhere. However, I don't think
> that concern applies to programs that are in the same source package as
> the library --- they'd get rebuilt anyway. So I see nothing wrong with
> continuing to statically link these .o files into files belonging to the
> postgresql package. It's just that I can't export them in a .a file for
> *other* source packages to use.
>
> (Whether a security issue in libpgport is really likely to happen is not
> a question that this policy concerns itself with ...)
>
>

OK, my possibly naïve approach is shown in the attached. Essentially it
builds libpgport-shared.so and then installs it as libpgport.so. That
ensures that the library is not used in building any postgres binaries
or libraries. Places such as libpq that formerly symlinked and
recompiled the sources in a way that is suitable for a shared library
now just link in the already built object files.

Is there a better way to do this?

cheers

andrew