Re: Segfault related to pg_authid when running initdb from git master

Here's the output I see when $SUBJECT occurs, on a pg freshly built
from git master with --enable-debug and --enable-cassert:

[postgres(at)peter bin]$ uname -a
Linux peter.laptop 2.6.35.9-64.fc14.x86_64 #1 SMP Fri Dec 3 12:19:41
UTC 2010 x86_64 x86_64 x86_64 GNU/Linux
[postgres(at)peter bin]$ ./initdb -D /var/lib/pgsql/data
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale en_IE.utf8.
The default database encoding has accordingly been set to UTF8.
The default text search configuration will be set to "english".

fixing permissions on existing directory /var/lib/pgsql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 24MB
creating configuration files ... ok
creating template1 database in /var/lib/pgsql/data/base/1 ... ok
initializing pg_authid ... sh: line 1: 23515 Segmentation fault
(core dumped) "/usr/local/pgsql/bin/postgres" --single -F -O -c
search_path=pg_catalog -c exit_on_error=true template1 > /dev/null
child process exited with exit code 139
initdb: removing contents of data directory "/var/lib/pgsql/data"

I'm having difficulty producing a useful backtrace, because the
segfault seemingly doesn't actually occur within initdb - it occurs
within a postgres process. If someone could tell me the trick to
attaching to that process under these circumstances, I could look into
it further.

The trouble seems occur here, at line 1224 of initdb.c:

for (line = pg_authid_setup; *line != NULL; line++)
PG_CMD_PUTS(*line);

After I see the segmentation fault in stderr, gdb reports that initdb
has received SIGPIPE.

Hope that helps.

--
Regards,
Peter Geoghegan

On Tue, Dec 14, 2010 at 5:21 PM, Peter Geoghegan
<peter(dot)geoghegan86(at)gmail(dot)com> wrote:
> Here's the output I see when $SUBJECT occurs, on a pg freshly built
> from git master with --enable-debug and --enable-cassert:

I am suspicious of the fact that you are invoking initdb as ./initdb.
Is it possible you're invoking this from the build tree, and there's
an installed copy out there that doesn't match, but is getting used?
Like maybe in /usr/local/pgsql/bin?

> creating template1 database in /var/lib/pgsql/data/base/1 ... ok
> initializing pg_authid ... sh: line 1: 23515 Segmentation fault
> (core dumped) "/usr/local/pgsql/bin/postgres" --single -F -O -c
> search_path=pg_catalog -c exit_on_error=true template1 > /dev/null
> child process exited with exit code 139

Can you fire up gdb on this core dump, using "gdb
/usr/local/pgsql/bin/postgres /path/to/coredump"? Or, another
possibility is to run initdb with --noclean and then run the command,
without routing the output to /dev/null:

/usr/local/pgsql/bin/postgres" --single -F -O -c
search_path=pg_catalog -c exit_on_error=true template1

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 15 December 2010 01:35, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> I am suspicious of the fact that you are invoking initdb as ./initdb.
> Is it possible you're invoking this from the build tree, and there's
> an installed copy out there that doesn't match, but is getting used?
> Like maybe in /usr/local/pgsql/bin?

No, I'm not doing that. I'm running initdb from /usr/local/pgsql/bin
(nothing pg related can be found in my $PATH), but it's the only copy
on my system, which was installed from git master last night. It has
debugging symbols, and I've actually re-created this from initdb's
point of view within GDB with source level debugging.

> Can you fire up gdb on this core dump, using "gdb
> /usr/local/pgsql/bin/postgres /path/to/coredump"?  Or, another
> possibility is to run initdb with --noclean and then run the command,
> without routing the output to /dev/null:
>
> /usr/local/pgsql/bin/postgres" --single -F -O -c
> search_path=pg_catalog -c exit_on_error=true template1

I cannot find the coredump. Perhaps it's a permissions issue. What do you think?

Anyway, I have produced a useful backtrace by debugging postgres
directly after running initdb with --noclean as described:

[peter(at)peter bin]$ /usr/local/pgsql/bin/postgres --single -F -O -c
search_path=pg_catalog -c exit_on_error=true template1
Segmentation fault
[peter(at)peter bin]$ gdb postgres
GNU gdb (GDB) Fedora (7.2-26.fc14)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/pgsql/bin/postgres...done.
(gdb) set args --single -F -O -c search_path=pg_catalog -c
exit_on_error=true template1
(gdb) start
Temporary breakpoint 1 at 0x577360
Starting program: /usr/local/pgsql/bin/postgres --single -F -O -c
search_path=pg_catalog -c exit_on_error=true template1

Temporary breakpoint 1, 0x0000000000577360 in main ()
(gdb) c
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x000000000047615b in _bt_preprocess_keys ()
(gdb) bt
#0 0x000000000047615b in _bt_preprocess_keys ()
#1 0x0000000000475382 in _bt_first ()
#2 0x0000000000473d71 in btgettuple ()
#3 0x00000000006ba67c in FunctionCall2 ()
#4 0x000000000046e08a in index_getnext ()
#5 0x000000000046d556 in systable_getnext ()
#6 0x00000000006a92bf in LookupOpclassInfo ()
#7 0x00000000006a9a58 in RelationInitIndexAccessInfo ()
#8 0x00000000006aa9cb in RelationBuildDesc ()
#9 0x00000000006aabfd in load_critical_index ()
#10 0x00000000006ac12a in RelationCacheInitializePhase3 ()
#11 0x00000000006c19ca in InitPostgres ()
#12 0x000000000060058f in PostgresMain ()
#13 0x000000000057774d in main ()

For some reason, postgres has limited debugging symbols (no line
number information is available). Given that it is available from
initdb, that seems very odd:

Temporary breakpoint 1 at 0x577360
Starting program: /usr/local/pgsql/bin/postgres --single -F -O -c
search_path=pg_catalog -c exit_on_error=true template1

Temporary breakpoint 1, 0x0000000000577360 in main ()
(gdb) n
Single stepping until exit from function main,
which has no line number information.

Program received signal SIGSEGV, Segmentation fault.
0x000000000047615b in _bt_preprocess_keys ()

Hope that helps.

--
Regards,
Peter Geoghegan

On Wed, Dec 15, 2010 at 6:07 AM, Peter Geoghegan
<peter(dot)geoghegan86(at)gmail(dot)com> wrote:
> On 15 December 2010 01:35, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> I am suspicious of the fact that you are invoking initdb as ./initdb.
>> Is it possible you're invoking this from the build tree, and there's
>> an installed copy out there that doesn't match, but is getting used?
>> Like maybe in /usr/local/pgsql/bin?
>
> No, I'm not doing that. I'm running initdb from /usr/local/pgsql/bin
> (nothing pg related can be found in my $PATH), but it's the only copy
> on my system, which was installed from git master last night. It has
> debugging symbols, and I've actually re-created this from initdb's
> point of view within GDB with source level debugging.

Well, something's clearly funky here because your initdb has debugging
symbols but your postgres executable does not. I may be missing
something obvious, but I don't see how that can happen without mixing
up two different builds.

>> Can you fire up gdb on this core dump, using "gdb
>> /usr/local/pgsql/bin/postgres /path/to/coredump"?  Or, another
>> possibility is to run initdb with --noclean and then run the command,
>> without routing the output to /dev/null:
>>
>> /usr/local/pgsql/bin/postgres" --single -F -O -c
>> search_path=pg_catalog -c exit_on_error=true template1
>
> I cannot find the coredump. Perhaps it's a permissions issue. What do you think?

It would presumably get dumped into the data directory. So if
--noclean isn't used I expect it'll get nuked.

> Anyway, I have produced a useful backtrace by debugging postgres
> directly after running initdb with --noclean as described:
>
> [peter(at)peter bin]$ /usr/local/pgsql/bin/postgres --single -F -O -c
> search_path=pg_catalog -c exit_on_error=true template1
> Segmentation fault
> [peter(at)peter bin]$ gdb postgres
> GNU gdb (GDB) Fedora (7.2-26.fc14)
> Copyright (C) 2010 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /usr/local/pgsql/bin/postgres...done.
> (gdb) set args --single -F -O -c search_path=pg_catalog -c
> exit_on_error=true template1
> (gdb) start
> Temporary breakpoint 1 at 0x577360
> Starting program: /usr/local/pgsql/bin/postgres --single -F -O -c
> search_path=pg_catalog -c exit_on_error=true template1
>
> Temporary breakpoint 1, 0x0000000000577360 in main ()
> (gdb) c
> Continuing.
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x000000000047615b in _bt_preprocess_keys ()
> (gdb) bt
> #0  0x000000000047615b in _bt_preprocess_keys ()
> #1  0x0000000000475382 in _bt_first ()
> #2  0x0000000000473d71 in btgettuple ()
> #3  0x00000000006ba67c in FunctionCall2 ()
> #4  0x000000000046e08a in index_getnext ()
> #5  0x000000000046d556 in systable_getnext ()
> #6  0x00000000006a92bf in LookupOpclassInfo ()
> #7  0x00000000006a9a58 in RelationInitIndexAccessInfo ()
> #8  0x00000000006aa9cb in RelationBuildDesc ()
> #9  0x00000000006aabfd in load_critical_index ()
> #10 0x00000000006ac12a in RelationCacheInitializePhase3 ()
> #11 0x00000000006c19ca in InitPostgres ()
> #12 0x000000000060058f in PostgresMain ()
> #13 0x000000000057774d in main ()

Ugh. Maybe someone smarter can figure out what that means, but I have
no clue. _bt_preprocess_keys() is a pretty good-sized function;
there's no obvious way to know which pointer reference is blowing up
without line-number information.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 15 December 2010 16:26, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Wed, Dec 15, 2010 at 6:07 AM, Peter Geoghegan
> <peter(dot)geoghegan86(at)gmail(dot)com> wrote:
>> On 15 December 2010 01:35, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>>> I am suspicious of the fact that you are invoking initdb as ./initdb.
>>> Is it possible you're invoking this from the build tree, and there's
>>> an installed copy out there that doesn't match, but is getting used?
>>> Like maybe in /usr/local/pgsql/bin?
>>
>> No, I'm not doing that. I'm running initdb from /usr/local/pgsql/bin
>> (nothing pg related can be found in my $PATH), but it's the only copy
>> on my system, which was installed from git master last night. It has
>> debugging symbols, and I've actually re-created this from initdb's
>> point of view within GDB with source level debugging.
>
> Well, something's clearly funky here because your initdb has debugging
> symbols but your postgres executable does not.  I may be missing
> something obvious, but I don't see how that can happen without mixing
> up two different builds.

Just to make sure that I'm not going crazy, I did a git pull, rebuilt
pg passing --enable-debug and --enable-casssert to configure as
before, followed by make && make install. Then I tried this:

[peter(at)peter bin]$ pwd
/usr/local/pgsql/bin
[peter(at)peter bin]$ ls -l
total 7720
-rwxr-xr-x. 1 root root 53977 Dec 15 16:47 clusterdb
-rwxr-xr-x. 1 root root 55058 Dec 15 16:47 createdb
-rwxr-xr-x. 1 root root 58351 Dec 15 16:47 createlang
-rwxr-xr-x. 1 root root 58036 Dec 15 16:47 createuser
-rwxr-xr-x. 1 root root 53380 Dec 15 16:47 dropdb
-rwxr-xr-x. 1 root root 62052 Dec 15 16:47 droplang
-rwxr-xr-x. 1 root root 53382 Dec 15 16:47 dropuser
-rwxr-xr-x. 1 root root 707190 Dec 15 16:47 ecpg
-rwxr-xr-x. 1 root root 123447 Dec 15 16:47 initdb
-rwxr-xr-x. 1 root root 26435 Dec 15 16:47 pg_config
-rwxr-xr-x. 1 root root 25229 Dec 15 16:47 pg_controldata
-rwxr-xr-x. 1 root root 73784 Dec 15 16:47 pg_ctl
-rwxr-xr-x. 1 root root 301781 Dec 15 16:47 pg_dump
-rwxr-xr-x. 1 root root 75323 Dec 15 16:47 pg_dumpall
-rwxr-xr-x. 1 root root 32015 Dec 15 16:47 pg_resetxlog
-rwxr-xr-x. 1 root root 131867 Dec 15 16:47 pg_restore
-rwxr-xr-x. 1 root root 91006 Dec 6 11:34 pg_upgrade
-rwxr-xr-x. 1 root root 5380671 Dec 15 16:47 postgres
lrwxrwxrwx. 1 root root 8 Dec 15 16:47 postmaster -> postgres
-rwxr-xr-x. 1 root root 398677 Dec 15 16:47 psql
-rwxr-xr-x. 1 root root 55257 Dec 15 16:47 reindexdb
-rwxr-xr-x. 1 root root 32410 Dec 15 16:47 vacuumdb
[peter(at)peter bin]$ which postgres
/usr/bin/which: no postgres in
(/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/peter/bin)
[peter(at)peter bin]$ which initdb
/usr/bin/which: no initdb in
(/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/peter/bin)

Observe that the initdb and postgres timestamps are the same. This
laptop is less than 2 weeks old, and has never had any postgres
packages installed on it. I can once again reproduce the problem,
exactly as before. My postgres executable does have debugging symbols,
just less than initdb (I'm not sure what the exact term is, but it
just lacks line information while having some debugging symbols).

>> I cannot find the coredump. Perhaps it's a permissions issue. What do you think?
>
> It would presumably get dumped into the data directory.  So if
> --noclean isn't used I expect it'll get nuked.

It isn't there...it just looks like a virginal PGDATA directory.

> Ugh.  Maybe someone smarter can figure out what that means, but I have
> no clue.  _bt_preprocess_keys() is a pretty good-sized function;
> there's no obvious way to know which pointer reference is blowing up
> without line-number information.

That's a pity, because I don't have a clue how to get line number
information. I could always try printf() debugging, but I really
shouldn't have to.

--
Regards,
Peter Geoghegan

On Wed, Dec 15, 2010 at 12:39 PM, Peter Geoghegan
<peter(dot)geoghegan86(at)gmail(dot)com> wrote:
> Observe that the initdb and postgres timestamps are the same.

Hrm.

>>> I cannot find the coredump. Perhaps it's a permissions issue. What do you think?
>>
>> It would presumably get dumped into the data directory.  So if
>> --noclean isn't used I expect it'll get nuked.
>
> It isn't there...it just looks like a virginal PGDATA directory.

Double hrm.

I have no idea how you can be getting line number information for
initdb but not postgres. I think what you're getting from postgres is
normally what I'd expect to see without --enable-debug. It sounds
like you are doing it right, but I have no explanation for the
results.

What distro are you using? This can't be broken across the board,
given the lack of metoos. Can you use git bisect to figure out which
commit broke it?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> What distro are you using? This can't be broken across the board,
> given the lack of metoos. Can you use git bisect to figure out which
> commit broke it?

Before that, have you tried the old standby of "make distclean" and a
full rebuild/reinstall? The lack of buildfarm confirmation makes me
highly suspicious that there's any real problem.

regards, tom lane

> Before that, have you tried the old standby of "make distclean" and a
> full rebuild/reinstall?  The lack of buildfarm confirmation makes me
> highly suspicious that there's any real problem.

That's fixed both problems. I should have tried it much sooner. I
guess that even though the binaries built were new, they were somehow
linked with one or more older, "release" object files. Thanks.

--
Regards,
Peter Geoghegan

On Wed, Dec 15, 2010 at 2:40 PM, Peter Geoghegan
<peter(dot)geoghegan86(at)gmail(dot)com> wrote:
>> Before that, have you tried the old standby of "make distclean" and a
>> full rebuild/reinstall?  The lack of buildfarm confirmation makes me
>> highly suspicious that there's any real problem.
>
> That's fixed both problems. I should have tried it much sooner. I
> guess that even though the binaries built were new, they were somehow
> linked with one or more older, "release" object files. Thanks.

Gah. I assumed you had cleaned out your tree. Oh, well.

If you don't use --enable-depend, you can get this kind of issue.
Even if you do, it's worth trying a full clean out (I use git clean
-dfx) if you get something weird.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Excerpts from Peter Geoghegan's message of mié dic 15 16:40:41 -0300 2010:
> > Before that, have you tried the old standby of "make distclean" and a
> > full rebuild/reinstall?  The lack of buildfarm confirmation makes me
> > highly suspicious that there's any real problem.
>
> That's fixed both problems. I should have tried it much sooner. I
> guess that even though the binaries built were new, they were somehow
> linked with one or more older, "release" object files. Thanks.

This is probably caused by failure to use the --enable-depend configure
switch. I think we should try to make that the default on platforms
that support it. It seems silly not to use it.

--
Álvaro Herrera <alvherre(at)commandprompt(dot)com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

On 15 December 2010 19:43, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> Gah.  I assumed you had cleaned out your tree.  Oh, well.
>
> If you don't use --enable-depend, you can get this kind of issue.
> Even if you do, it's worth trying a full clean out (I use git clean
> -dfx) if you get something weird.

Thanks for the tip. I guess it simply didn't occur to me to "make
distclean" because I made the rather questionable assumption that it's
only necessary when there are weird linking issues.

--
Regards,
Peter Geoghegan