Re: PG84 and SELinux

OS: CentOS-5.5

Earlier today I attempted to upgrade a production server from 8.1 to
8.4 using the pgdg-84-centos.repo. I say attempted because I could
never get it to support ssl connections and as that is a requirement
I had to roll back to 8.1.

Whatever was the cause of the ssl problem I also encountered a
surprising number of SELinux violations. The following details the
SELinux settings that I ultimately had to apply as a local module.
This took a considerable period of time as each had to be triggered
in turn in order that the error be identified.

#============= postgresql_t ==============
allow postgresql_t var_lib_t:dir rmdir;
allow postgresql_t var_lib_t:file { write getattr link read unlink
append };

Is this to be expected?

--
*** E-Mail is NOT a SECURE channel ***
James B. Byrne mailto:ByrneJB(at)Harte-Lyne(dot)ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3

"James B. Byrne" <byrnejb(at)harte-lyne(dot)ca> writes:
> Earlier today I attempted to upgrade a production server from 8.1 to
> 8.4 using the pgdg-84-centos.repo. I say attempted because I could
> never get it to support ssl connections and as that is a requirement
> I had to roll back to 8.1.

Can't comment on that without a lot more detail.

> Whatever was the cause of the ssl problem I also encountered a
> surprising number of SELinux violations. The following details the
> SELinux settings that I ultimately had to apply as a local module.
> This took a considerable period of time as each had to be triggered
> in turn in order that the error be identified.

> #============= postgresql_t ==============
> allow postgresql_t var_lib_t:dir rmdir;
> allow postgresql_t var_lib_t:file { write getattr link read unlink
> append };

> Is this to be expected?

AFAIK, the Red Hat RPMs work out-of-the-box with SELinux; I'm a bit
surprised to hear that the PGDG ones don't, because last I heard
they use the same file layout. What the above sounds like to me is that
the data directory tree wasn't correctly labeled as postgresql_db_t.
Maybe a restorecon would have helped?

regards, tom lane

On Wed, December 1, 2010 16:54, Tom Lane wrote:

>> Whatever was the cause of the ssl problem I also encountered a
>> surprising number of SELinux violations. The following details
>> the
>> SELinux settings that I ultimately had to apply as a local module.
>> This took a considerable period of time as each had to be
>> triggered
>> in turn in order that the error be identified.
>
>> #============= postgresql_t ==============
>> allow postgresql_t var_lib_t:dir rmdir;
>> allow postgresql_t var_lib_t:file { write getattr link read unlink
>> append };
>
>> Is this to be expected?
>
> AFAIK, the Red Hat RPMs work out-of-the-box with SELinux; I'm a bit
> surprised to hear that the PGDG ones don't, because last I heard
> they use the same file layout. What the above sounds like to me is
> that
> the data directory tree wasn't correctly labeled as postgresql_db_t.
> Maybe a restorecon would have helped?
>
> regards, tom lane
>

I tried a restorecon as suggested by sealert at the first error. It
had no effect insofar as I could determine.

--
*** E-Mail is NOT a SECURE channel ***
James B. Byrne mailto:ByrneJB(at)Harte-Lyne(dot)ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3

On Wed, December 1, 2010 16:54, Tom Lane wrote:
> "James B. Byrne" <byrnejb(at)harte-lyne(dot)ca> writes:
>> Earlier today I attempted to upgrade a production server from 8.1
>> to
>> 8.4 using the pgdg-84-centos.repo. I say attempted because I
>> could
>> never get it to support ssl connections and as that is a
>> requirement
>> I had to roll back to 8.1.
>
> Can't comment on that without a lot more detail.

Well, the only thing that I had to do was uncomment #ssl = off and
set it to ssl = on. Then on service restart I obtained an error
that looked similar to this:

. . . configuration file routines:STR_COPY:variable
has no value:conf_def.c:629:line 207

I cannot remember the exact error unfortunately and as I rolled back
the update I have no way to reproduce it at will.

--
*** E-Mail is NOT a SECURE channel ***
James B. Byrne mailto:ByrneJB(at)Harte-Lyne(dot)ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3

On Wed, 2010-12-01 at 16:54 -0500, Tom Lane wrote:
> AFAIK, the Red Hat RPMs work out-of-the-box with SELinux; I'm a bit
> surprised to hear that the PGDG ones don't, because last I heard
> they use the same file layout.

They should -- we are using the same routines for initdb'ing.

--
Devrim GÜNDÜZ
PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer
PostgreSQL RPM Repository: http://yum.pgrpms.org
Community: devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org Twitter: http://twitter.com/devrimgunduz

On Thu, December 2, 2010 15:23, Devrim GÜNDÜZ wrote:
> On Wed, 2010-12-01 at 16:54 -0500, Tom Lane wrote:
>> AFAIK, the Red Hat RPMs work out-of-the-box with SELinux;
>
> They should -- we are using the same routines for initdb'ing.
>

I will do a touch /.autorelabel and restart the server before I try
again. If there was just something odd about the SELinux contexts
on that particular host then that should clear it up. I will report
whichever way it goes thereafter.

--
*** E-Mail is NOT a SECURE channel ***
James B. Byrne mailto:ByrneJB(at)Harte-Lyne(dot)ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3

On Thu, December 2, 2010 15:32, James B. Byrne wrote:
>
> On Thu, December 2, 2010 15:23, Devrim GÜNDÜZ wrote:
>> On Wed, 2010-12-01 at 16:54 -0500, Tom Lane wrote:
>>> AFAIK, the Red Hat RPMs work out-of-the-box with SELinux;
>>
>> They should -- we are using the same routines for initdb'ing.
>>
>
> I will do a touch /.autorelabel and restart the server before I try
> again. If there was just something odd about the SELinux contexts
> on that particular host then that should clear it up. I will report
> whichever way it goes thereafter.

I restarted the server this morning, waited for the relabel to
finish ( a very long time ), and then upgraded to pg-8.4 without any
problems. SSL works fine as well. I can only infer that something
went seriously wrong with the SELinux context labels on that host.

Thank you for the help.

--
*** E-Mail is NOT a SECURE channel ***
James B. Byrne mailto:ByrneJB(at)Harte-Lyne(dot)ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3

On Wed, December 1, 2010 16:54, Tom Lane wrote:
> "James B. Byrne" <byrnejb(at)harte-lyne(dot)ca> writes:
>> Earlier today I attempted to upgrade a production server
>> from 8.1 to 8.4 using the pgdg-84-centos.repo. I say
>> attempted because I could never get it to support ssl
>> connections and as that is a requirement I had to roll
>> back to 8.1.
>
> Can't comment on that without a lot more detail.
>
On Fri, December 3, 2010 07:40, James B. Byrne wrote:
>
> I restarted the server this morning, waited for the relabel to
> finish ( a very long time ), and then upgraded to pg-8.4 without any
> problems. SSL works fine as well. . .

I wrote too soon. What I did was uncomment the ssl option. I
neglected to change the setting from off to on.

When I try to start the server with ssl=on it fails with this error:

Auto configuration failed
29006:error:0E065068:configuration file routines:STR_COPY:variable
has no value:conf_def.c:629:line 207

I have checked gpsql/data and the server.key and server.crt files
are both present:

-rw------- 1 postgres postgres 5213 Dec 12 2007 server.crt
-rw------- 1 postgres postgres 1675 Dec 12 2007 server.key

The only change made in the postgresql.conf file that triggered this
was changing 'off' to 'on' for ssl. Changing it back to 'off' makes
the problem disappear.

--
*** E-Mail is NOT a SECURE channel ***
James B. Byrne mailto:ByrneJB(at)Harte-Lyne(dot)ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3

"James B. Byrne" <byrnejb(at)harte-lyne(dot)ca> writes:
> I wrote too soon. What I did was uncomment the ssl option. I
> neglected to change the setting from off to on.

> When I try to start the server with ssl=on it fails with this error:

> Auto configuration failed
> 29006:error:0E065068:configuration file routines:STR_COPY:variable
> has no value:conf_def.c:629:line 207

AFAIK there is no place in the standard Postgres sources that could emit
an error message even vaguely like that. I'm guessing that you are
using some add-on code that tries to parse postgresql.conf, but I don't
know what that would be. Whose init script are you using?

regards, tom lane

On Sat, December 4, 2010 01:11, Tom Lane wrote:
> "James B. Byrne" <byrnejb(at)harte-lyne(dot)ca> writes:
>> I wrote too soon. What I did was uncomment the ssl option. I
>> neglected to change the setting from off to on.
>
>> When I try to start the server with ssl=on it fails with this
>> error:
>
>> Auto configuration failed
>> 29006:error:0E065068:configuration file routines:STR_COPY:variable
>> has no value:conf_def.c:629:line 207
>
> AFAIK there is no place in the standard Postgres sources that could
> emit an error message even vaguely like that. I'm guessing that
> you are using some add-on code that tries to parse postgresql.conf,
> but I don't know what that would be. Whose init script are you
> using?
>
> regards, tom lane
>

I see some familiar names. . .

# cat postgresql
#!/bin/sh
# postgresql This is the init script for starting up the PostgreSQL
# server
#
# chkconfig: - 64 36
# description: Starts and stops the PostgreSQL backend daemon that
handles \
# all database requests.
# processname: postmaster
# pidfile: /var/run/postmaster.pid

# Version 6.5.3-2 Lamar Owen
# Added code to determine if PGDATA exists, whether it is current
version
# or not, and initdb if no PGDATA (initdb will not overwrite a
database).

# Version 7.0 Lamar Owen
# Added logging code
# Changed PGDATA.

# Version 7.0.2 Trond Eivind Glomsrd <teg(at)redhat(dot)com>
# use functions, add conditional restart

# Version 7.0.3 Lamar Owen <lamar(at)postgresql(dot)org>
# Check for the existence of functions before blindly using them
# in particular -- check for success () and failure () before using.
# More Cross-distribution support -- PGVERSION variable, and docdir
checks.

# Version 7.1 Release Candidate Lamar Owen <lamar(at)postgresql(dot)org>
# initdb parameters have changed.

# Version 7.1.2 Trond Eivind Glomsrd <teg(at)redhat(dot)com>
# Specify shell for su
# Handle stop better - kill unwanted output, make it wait until the
database is ready
# Handle locales slightly differently - always using "C" isn't a
valid option
# Kill output from database initialization
# Mark messages for translation

# Version 7.1.2-2.PGDG Lamar Owen <lamar(dot)owen(at)wgcr(dot)org>
# sync up.
# Karl's fixes for some quoting issues.

# Version 7.2b2 Lamar Owen <lamar(dot)owen(at)wgcr(dot)org>
# version change.

# Version 7.2 final. Lamar Owen <lamar(dot)owen(at)wgcr(dot)org>
# reload from Peter E.
# Eliminate the pidof postmaster test in stop -- we're using pg_ctl
so we don't need pidof.
# Tested the $? return for the stop script -- it does in fact
propagate.
# TODO: multiple postmasters.

# Version 7.3 Lamar Owen <lamar(dot)owen(at)ramifordistat(dot)net>
# Multiple postmasters, courtesy Karl DeBisschop

# Version 7.4 Lamar Owen.

# Version 7.4.3 Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
# Support condstop for uninstall
# Minor other changes suggested by Fernando Nasser.

# Version 7.4.5 Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
# Rewrite to start postmaster directly, rather than via pg_ctl; this
avoids
# fooling the postmaster's stale-lockfile check by having too many
# postgres-owned processes laying about.

# Version 8.1 Devrim Gunduz <devrim(at)PostgreSQL(dot)org>
# Increased sleep time from 1 sec to 2 sec.

# Version 8.2 Devrim Gunduz <devrim(at)CommandPrompt(dot)com>
# Set initdb as a seperate option.

# Version 8.3 Devrim Gunduz <devrim(at)CommandPrompt(dot)com>

# Version 8.4 Devrim Gunduz <devrim(at)CommandPrompt(dot)com>
# Remove "sameuser" from initdb, to match the new hba conf file.
# Get rid of duplicate PGDATA assignment.
# Ensure pgstartup.log gets the right ownership/permissions during
initdb

# PGVERSION is the full package version, e.g., 8.4.0
# Note: the specfile ordinarily updates this during install
PGVERSION=8.4.4
# PGMAJORVERSION is major version, e.g., 8.4 (this should match
PG_VERSION)
PGMAJORVERSION=`echo "$PGVERSION" | sed 's/^\([0-9]*\.[0-9]*\).*$/\1/'`

--
*** E-Mail is NOT a SECURE channel ***
James B. Byrne mailto:ByrneJB(at)Harte-Lyne(dot)ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3

Tom Lane wrote:
> "James B. Byrne" <byrnejb(at)harte-lyne(dot)ca> writes:
>
>> I wrote too soon. What I did was uncomment the ssl option. I
>> neglected to change the setting from off to on.
>>
>
>
>> When I try to start the server with ssl=on it fails with this error:
>>
>
>
>> Auto configuration failed
>> 29006:error:0E065068:configuration file routines:STR_COPY:variable
>> has no value:conf_def.c:629:line 207
>>
>
> AFAIK there is no place in the standard Postgres sources that could emit
> an error message even vaguely like that.

That looks to be the str_copy routine from conf_def.c in the OpenSSL
code, i.e. line 624 of the version at:

http://code.google.com/p/commitmonitor/source/browse/trunk/common/openssl/crypto/conf/conf_def.c

So guessing something in the SSL autonegotiation is failing here in a
really unexpected way.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services and Support www.2ndQuadrant.us

On Mon, December 6, 2010 00:47, Greg Smith wrote:

>
> That looks to be the str_copy routine from conf_def.c in the OpenSSL
> code, i.e. line 624 of the version at:
>
> http://code.google.com/p/commitmonitor/source/browse/trunk/common/openssl/crypto/conf/conf_def.c
>
> So guessing something in the SSL autonegotiation is failing here in
> a really unexpected way.
>

The problem was an expired pki certificate. When we first used ssl
for pg we did not have our private CA set up. So we generated a
self-signed certificate. That certificate expired this past July
and I infer that while 8.1 did not care 8.4 evidently does.

In any case, we generated a new key and had a certificate signing
request signed by our CA. We installed both as server.key and
server.crt in the pgsql/data directory with chmod 600 and chown
postgres:postgres. Setting the postgresql.conf ssl option to on and
restarting the server no longer causes any error.

Than you all for the help.

--
*** E-Mail is NOT a SECURE channel ***
James B. Byrne mailto:ByrneJB(at)Harte-Lyne(dot)ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3

On Mon, December 6, 2010 13:29, James B. Byrne wrote:

>>
>
> The problem was an expired pki certificate. When we first used ssl
> for pg we did not have our private CA set up. So we generated a
> self-signed certificate. That certificate expired this past July
> and I infer that while 8.1 did not care 8.4 evidently does.

Wrong again. I misinterpreted what I was seeing. Even with the new
cert and key I get the same error.

Auto configuration failed
16276:error:0E065068:configuration file routines:STR_COPY:variable
has no value:conf_def.c:629:line 207

I will try to get help on the CentOS list.

--
*** E-Mail is NOT a SECURE channel ***
James B. Byrne mailto:ByrneJB(at)Harte-Lyne(dot)ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3

I received absolutely no reply to my question on the CentOS mailing
list so I have to turn to this venue again for help.

I note the following things:

postgresql-server.i386 8.4.4-2PGDG.el5
installed

openssl.i686 0.9.8e-12.el5_4.6
installed

Might there be a problem between the server being compiled for i386
and openssl for i686? I cannot for the life of me determine what
configuration problem causes this error.

On Fri, December 3, 2010 16:04, James B. Byrne wrote:
> When I try to start the server with ssl=on it fails with this error:
>
> Auto configuration failed
> 29006:error:0E065068:configuration file routines:STR_COPY:variable
> has no value:conf_def.c:629:line 207
>

--
*** E-Mail is NOT a SECURE channel ***
James B. Byrne mailto:ByrneJB(at)Harte-Lyne(dot)ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3

On Tue, 2010-12-07 at 16:54 -0500, James B. Byrne wrote:
> I received absolutely no reply to my question on the CentOS mailing
> list so I have to turn to this venue again for help.
>
> I note the following things:
>
> postgresql-server.i386 8.4.4-2PGDG.el5
> installed
>
> openssl.i686 0.9.8e-12.el5_4.6
> installed
>
>
> Might there be a problem between the server being compiled for i386
> and openssl for i686? I cannot for the life of me determine what
> configuration problem causes this error.

No those lib differences are both still 32bit. You would have a problem
if one was 64bit. So you should be fine there.

Joshua D. Drake

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579
Consulting, Training, Support, Custom Development, Engineering
http://twitter.com/cmdpromptinc | http://identi.ca/commandprompt

On Tue, December 7, 2010 16:56, Joshua D. Drake wrote:

>
> No those lib differences are both still 32bit. You would have a
> problem if one was 64bit. So you should be fine there.
>
> Joshua D. Drake
>

Ok. How do I get postgresql to cough up more processing detail on
startup? The message that I presently get makes no sense at all to
me.

--
*** E-Mail is NOT a SECURE channel ***
James B. Byrne mailto:ByrneJB(at)Harte-Lyne(dot)ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3

"James B. Byrne" <byrnejb(at)harte-lyne(dot)ca> writes:
> Ok. How do I get postgresql to cough up more processing detail on
> startup? The message that I presently get makes no sense at all to
> me.

The message isn't coming from postgres --- it's openssl that you're
wishing would be more verbose.

What I'd try next is strace'ing the postmaster so you can see what
happened right before the error report. With luck that will point you
at a specific configuration file that's (presumably) messed up.

regards, tom lane

On Tue, Dec 07, 2010 at 05:15:45PM -0500, James B. Byrne wrote:
>
> On Tue, December 7, 2010 16:56, Joshua D. Drake wrote:
>
> >
> > No those lib differences are both still 32bit. You would have a
> > problem if one was 64bit. So you should be fine there.
> >
> > Joshua D. Drake
> >
>
> Ok. How do I get postgresql to cough up more processing detail on
> startup? The message that I presently get makes no sense at all to
> me.

do we know that pg was compiled with ssl? maybe a bonehead question,
but low hanging fruit is my specialty.

I have now tracked down and resolved the problem. There were clues
to the solution in the error message but I lacked sufficient
experience with ssl to realize it. The error was an uncommented
line in /etc/pki/tls/openssl.cnf that depended upon an environment
variable (ALTNAME) being set (subjectAltName=$ENV::ALTNAME). This
was line 270 in that file. Note the error message:

> Auto configuration failed
> 29006:error:0E065068:configuration file routines:STR_COPY:variable
> has no value:conf_def.c:629:line 207

Given what I know now I infer that conf_def is the variable that
holds the actual file name of whatever configuration file is passed
to openssl. The error message would have been far more informative
had it provided the variable value rather than the variable name.
And, I have no idea why PG84 choked on this and PG81 did not.

Anyway, our upgraded PG84 service is now running with ssl enabled.
Many thanks for the hints and suggestions. They did in fact
eventually point me in the right direction.

--
*** E-Mail is NOT a SECURE channel ***
James B. Byrne mailto:ByrneJB(at)Harte-Lyne(dot)ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3