Re: pg_upgrade: out of memory

Hi,

I have attempted to upgrade my Postgres installation this morning from 9.0.1 to 9.2.0 and it failed with an out of memory problem using pg_dumpall to dump the first database.

So after backing out the change and restarting level 9.0.1, I've done some basic investigation into the failure of the 9.2.0 pg_upgrade by checking output in pg_upgrade_utility.log ...

command: "/opt/serviceMonitoring/postgres/bin/pg_dumpall" --port 50432 --username "postgres" --schema-only --binary-upgrade -f pg_upgrade_dump_all.sql >> "pg_upgrade_utility.log" 2>&1
pg_dump: out of memory
pg_dumpall: pg_dump failed on database "cahoot_monitoring", exiting

... rerunning pg_dump_all using existing version 9.0.1 ...

"/opt/serviceMonitoring/postgres/bin/pg_dumpall" --port 65432 --username "postgres" --schema-only --binary-upgrade -f pg_upgrade_dump_all.sql

... works fine.

Try with version 9.2.0 as per pg_upgrade ...

"/opt/serviceMonitoring/postgres_9.2/bin/pg_dumpall" --port 65432 --username "postgres" --schema-only --binary-upgrade -f pg_upgrade_dump_all.sql

pg_dump: out of memory
pg_dumpall: pg_dump failed on database "cahoot_monitoring", exiting

... tail -3 pg_upgrade_dump_all.sql ...

\connect cahoot_monitoring

... no help there.

Try with version 9.1.5 as per pg_upgrade ...

cd /tmp
"/opt/serviceMonitoring/postgres_9.1/bin/pg_dumpall" --port 65432 --username "postgres" --schema-only --binary-upgrade -f pg_upgrade_dump_all.sql

... works fine.

Has anyone else hit this problem ?

Matthew
Emails aren't always secure, and they may be intercepted or changed
after they've been sent. Produban doesn't accept liability if this
happens. If you think someone may have interfered with this email,
please get in touch with the sender another way. This message and any
documents attached to it do not create or change any contract unless
otherwise specifically stated. Any views or opinions contained in this
message are solely those of the author, and do not necessarily represent
those of Produban, unless otherwise specifically stated and the sender
is authorised to do so. Produban doesn't accept responsibility for
damage caused by any viruses contained in this email or its attachments.
Emails may be monitored. If you've received this email by mistake,
please let the sender know at once that it's gone to the wrong person
and then destroy it without copying, using, or telling anyone about its
contents. Produban Servicios Informaticos Generales, S.L. (UK Branch).
Registered office: Shenley Wood House, Chalkdell Drive, Shenley Wood,
Milton Keynes MK5 6LA. Branch registration number BR 008486.
Ref:[PDB#014]

"Carrington, Matthew (Produban)" <Matthew(dot)Carrington(at)Produban(dot)co(dot)uk> writes:
> I have attempted to upgrade my Postgres installation this morning from 9.0.1 to 9.2.0 and it failed with an out of memory problem using pg_dumpall to dump the first database.

Hm. I'm not aware of any reason for 9.2 pg_dump to take hugely more
memory than 9.0. How big is the database (how many objects)? When
you run 9.0 pg_dump against it, how big does the process get? (Watching
it in "top" is probably a close enough answer here.)

regards, tom lane

Tom,

As its AIX I don't have top but using topas and comparing it to other processes running a successful pg_dumpall doesn't get very large at all.

The database only has around 1000 tables and no more than anpther 500 view, triggers, functions, etc. so its not a big database. There are no BLOBs or anything even slightly non-mainstream.

The output file created by a successful pg_dumpall is 11MB for all the databases running on this postgres installation. The other databases are much larger than the first one which is where the failure occurred.

The machine is a large AIX box with 64GB of memory and the upgrade was run with nothing else running on the machine so I find it hard to believe that it is genuinely out of memory. The whole of the first database could fit in real memory as its only 28GB.

Matthew

-----Original Message-----
From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
Sent: 20 September 2012 15:48
To: Carrington, Matthew (Produban)
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: [GENERAL] pg_upgrade: out of memory

"Carrington, Matthew (Produban)" <Matthew(dot)Carrington(at)Produban(dot)co(dot)uk> writes:
> I have attempted to upgrade my Postgres installation this morning from 9.0.1 to 9.2.0 and it failed with an out of memory problem using pg_dumpall to dump the first database.

Hm. I'm not aware of any reason for 9.2 pg_dump to take hugely more
memory than 9.0. How big is the database (how many objects)? When
you run 9.0 pg_dump against it, how big does the process get? (Watching
it in "top" is probably a close enough answer here.)

regards, tom lane
Emails aren't always secure, and they may be intercepted or changed
after they've been sent. Produban doesn't accept liability if this
happens. If you think someone may have interfered with this email,
please get in touch with the sender another way. This message and any
documents attached to it do not create or change any contract unless
otherwise specifically stated. Any views or opinions contained in this
message are solely those of the author, and do not necessarily represent
those of Produban, unless otherwise specifically stated and the sender
is authorised to do so. Produban doesn't accept responsibility for
damage caused by any viruses contained in this email or its attachments.
Emails may be monitored. If you've received this email by mistake,
please let the sender know at once that it's gone to the wrong person
and then destroy it without copying, using, or telling anyone about its
contents. Produban Servicios Informaticos Generales, S.L. (UK Branch).
Registered office: Shenley Wood House, Chalkdell Drive, Shenley Wood,
Milton Keynes MK5 6LA. Branch registration number BR 008486.
Ref:[PDB#014]

"Carrington, Matthew (Produban)" <Matthew(dot)Carrington(at)Produban(dot)co(dot)uk> writes:
> As its AIX I don't have top but using topas and comparing it to other processes running a successful pg_dumpall doesn't get very large at all.

Hmm. Best guess at this point is that there's some specific DDL in your
database that confuses some recent change in pg_dump. Can you try to
narrow down what it is? Try taking a pg_dump -s (schema only) from
the cahoot_monitoring database, load parts of that into a scratch
database, see if 9.2 pg_dump fails on that. Alternatively, if there's
nothing terribly sensitive in your DDL, maybe you could send me the
pg_dump -s output off-list?

regards, tom lane

I've tried to explore the issue a bit more using 9.2.1. Here's a trace of the build and dump tests ...

tar zxf postgresql-9.2.1.tar.gz
cd postgresql-9.2.1

uname -a
AIX sx1190a 1 6 000A707AD400 powerpc AIX

export OBJECT_MODE=64
export CFLAGS="-maix64"
export LDFLAGS="-maix64 -Wl,-bbigtoc"
export AR="ar -X64"

./configure --prefix=/opt/serviceMonitoring/postgres_9.2.1 --disable-thread-safety

make
cd contrib
make
cd ..
make install
cd contrib
make install

Check to see if a dumpall will work ...

cd /tmp
"/opt/serviceMonitoring/postgres_9.2.1/bin/pg_dumpall" --port 65432 --username "postgres" --schema-only --binary-upgrade -f pg_upgrade_dump_all.sql

pg_dump: out of memory
pg_dumpall: pg_dump failed on database "cahoot_monitoring", exiting

... try something less ambitious ...

"/opt/serviceMonitoring/postgres_9.2.1/bin/pg_dump" --port 65432 --username "postgres" --verbose --schema-only --binary-upgrade -f dump.out template1
pg_dump: reading schemas
pg_dump: reading user-defined tables
pg_dump: reading extensions
pg_dump: reading user-defined functions
pg_dump: reading user-defined types
pg_dump: reading procedural languages
pg_dump: reading user-defined aggregate functions
pg_dump: out of memory

Has anyone else seen this problem on AIX or elsewhere ?

Matthew
Emails aren't always secure, and they may be intercepted or changed
after they've been sent. Produban doesn't accept liability if this
happens. If you think someone may have interfered with this email,
please get in touch with the sender another way. This message and any
documents attached to it do not create or change any contract unless
otherwise specifically stated. Any views or opinions contained in this
message are solely those of the author, and do not necessarily represent
those of Produban, unless otherwise specifically stated and the sender
is authorised to do so. Produban doesn't accept responsibility for
damage caused by any viruses contained in this email or its attachments.
Emails may be monitored. If you've received this email by mistake,
please let the sender know at once that it's gone to the wrong person
and then destroy it without copying, using, or telling anyone about its
contents. Produban Servicios Informaticos Generales, S.L. (UK Branch).
Registered office: Shenley Wood House, Chalkdell Drive, Shenley Wood,
Milton Keynes MK5 6LA. Branch registration number BR 008486.
Ref:[PDB#014]

"Carrington, Matthew (Produban)" <Matthew(dot)Carrington(at)Produban(dot)co(dot)uk> writes:
> I've tried to explore the issue a bit more using 9.2.1. Here's a trace of the build and dump tests ...

As long as you're building your own copy, how about building with
--enable-debug and getting a stack trace showing where the error is
being thrown? Setting a breakpoint at exit_horribly() ought to do
the trick. To be concrete:

gdb pg_dump
gdb> b exit_horribly
gdb> run --port 65432 ... rest of command line for pg_dump ...
... wait for breakpoint to be hit, then ...
gdb> bt
... copy and paste the printout here ...
gdb> quit

regards, tom lane

Tom,

OK, here you go ...

Rebuild postgres with debugging ...

export OBJECT_MODE=64
export CFLAGS="-maix64"
export LDFLAGS="-maix64 -Wl,-bbigtoc"
export AR="ar -X64"

./configure --enable-debug --prefix=/opt/serviceMonitoring/postgres_9.2.1 --disable-thread-safety

make
cd contrib
make
cd ..
make install
cd contrib
make install

... get the backtrace ...

cd /tmp
gdb "/opt/serviceMonitoring/postgres_9.2.1/bin/pg_dump"
GNU gdb (GDB) 7.5
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc-ibm-aix5.1.0.0".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /ukmetmon/data/dataCollection/postgres_9.2.1/bin/pg_dump...(no debugging symbols found)...done.
(gdb) b exit_horribly
Breakpoint 1 at 0x1002e354
(gdb) run --port 65432 --username "postgres" --verbose --schema-only --binary-upgrade -f dump.out template1
Starting program: /ukmetmon/data/dataCollection/postgres_9.2.1/bin/pg_dump --port 65432 --username "postgres" --verbose --schema-only --binary-upgrade -f dump.out template1
pg_dump: reading schemas
pg_dump: reading user-defined tables
pg_dump: reading extensions
pg_dump: reading user-defined functions
pg_dump: reading user-defined types
pg_dump: reading procedural languages
pg_dump: reading user-defined aggregate functions

Breakpoint 1, 0x000000010002e354 in exit_horribly ()
(gdb) bt
#0 0x000000010002e354 in exit_horribly ()
#1 0x000000010003243c in pg_malloc ()
#2 0x0000000100008f14 in getAggregates ()
#3 0x000000010002fcac in getSchemaData ()
#4 0x0000000100001330 in main ()
(gdb) quit
A debugging session is active.

Inferior 1 [process 241858] will be killed.

Quit anyway? (y or n) y

... hope that helps.

Matthew

-----Original Message-----
From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
Sent: 28 September 2012 15:16
To: Carrington, Matthew (Produban)
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: [GENERAL] pg_upgrade: out of memory

"Carrington, Matthew (Produban)" <Matthew(dot)Carrington(at)Produban(dot)co(dot)uk> writes:
> I've tried to explore the issue a bit more using 9.2.1. Here's a trace of the build and dump tests ...

As long as you're building your own copy, how about building with
--enable-debug and getting a stack trace showing where the error is
being thrown? Setting a breakpoint at exit_horribly() ought to do
the trick. To be concrete:

gdb pg_dump
gdb> b exit_horribly
gdb> run --port 65432 ... rest of command line for pg_dump ...
... wait for breakpoint to be hit, then ...
gdb> bt
... copy and paste the printout here ...
gdb> quit

regards, tom lane
Emails aren't always secure, and they may be intercepted or changed
after they've been sent. Produban doesn't accept liability if this
happens. If you think someone may have interfered with this email,
please get in touch with the sender another way. This message and any
documents attached to it do not create or change any contract unless
otherwise specifically stated. Any views or opinions contained in this
message are solely those of the author, and do not necessarily represent
those of Produban, unless otherwise specifically stated and the sender
is authorised to do so. Produban doesn't accept responsibility for
damage caused by any viruses contained in this email or its attachments.
Emails may be monitored. If you've received this email by mistake,
please let the sender know at once that it's gone to the wrong person
and then destroy it without copying, using, or telling anyone about its
contents. Produban Servicios Informaticos Generales, S.L. (UK Branch).
Registered office: Shenley Wood House, Chalkdell Drive, Shenley Wood,
Milton Keynes MK5 6LA. Branch registration number BR 008486.
Ref:[PDB#014]

"Carrington, Matthew (Produban)" <Matthew(dot)Carrington(at)produban(dot)co(dot)uk> writes:
> Reading symbols from /ukmetmon/data/dataCollection/postgres_9.2.1/bin/pg_dump...(no debugging symbols found)...done.

... hm, not sure why that didn't work, but anyway:

> (gdb) bt
> #0 0x000000010002e354 in exit_horribly ()
> #1 0x000000010003243c in pg_malloc ()
> #2 0x0000000100008f14 in getAggregates ()
> #3 0x000000010002fcac in getSchemaData ()
> #4 0x0000000100001330 in main ()

getAggregates() doesn't do that much. Can we see the results of the
query it would have been executing, namely

SELECT tableoid, oid, proname AS aggname,
pronamespace AS aggnamespace,
pronargs, proargtypes,
(SELECT rolname FROM pg_catalog.pg_roles WHERE oid = proowner) AS rolname,
proacl AS aggacl
FROM pg_proc p
WHERE proisagg AND (
pronamespace !=
(SELECT oid FROM pg_namespace WHERE nspname = 'pg_catalog')
OR EXISTS(SELECT 1 FROM pg_depend WHERE
classid = 'pg_proc'::regclass AND
objid = p.oid AND
refclassid = 'pg_extension'::regclass AND
deptype = 'e'));

regards, tom lane

Tom,

I traced through the problem with 'no symbol table' being created and managed to create a version of Postgres 9.2.1 with a symbol table. Here's the trace ...

Rebuild postgres with debugging ...

export OBJECT_MODE=64
export CFLAGS="-maix64 -g"
export LDFLAGS="-maix64 -Wl,-bbigtoc"
export AR="ar -X64"
export "CC=/opt/freeware/bin/gcc -maix64"

./configure --enable-debug --prefix=/opt/serviceMonitoring/postgres_9.2.1 --disable-thread-safety --enable-cassert

make
cd contrib
make
cd ..
make install
cd contrib
make install

... get the backtrace ...

gdb "/opt/serviceMonitoring/postgres_9.2.1/bin/pg_dump"
GNU gdb (GDB) 7.5
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc-ibm-aix5.1.0.0".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /ukmetmon/data/dataCollection/postgres_9.2.1/bin/pg_dump...done.
(gdb) b exit_horribly
Breakpoint 1 at 0x1002e3b4: file dumputils.c, line 1314.
(gdb) run --port 65432 --username "postgres" --verbose --schema-only --binary-upgrade -f dump.out template1
Starting program: /ukmetmon/data/dataCollection/postgres_9.2.1/bin/pg_dump --port 65432 --username "postgres" --verbose --schema-only --binary-upgrade -f dump.out template1
pg_dump: reading schemas
pg_dump: reading user-defined tables
pg_dump: reading extensions
pg_dump: reading user-defined functions
pg_dump: reading user-defined types
pg_dump: reading procedural languages
pg_dump: reading user-defined aggregate functions

Breakpoint 1, exit_horribly (modulename=0x0, fmt=0x10006a590 "out of memory\n")
at dumputils.c:1314
1314 dumputils.c: A file or directory in the path name does not exist..
(gdb) bt
#0 exit_horribly (modulename=0x0, fmt=0x10006a590 "out of memory\n") at dumputils.c:1314
#1 0x000000010003247c in pg_malloc (size=0) at dumpmem.c:47
#2 0x0000000100008f54 in getAggregates (fout=0x11000bad0, numAggs=0xffffffffffff73c)
at pg_dump.c:3614
#3 0x000000010002fcec in getSchemaData (fout=0x11000bad0, numTablesPtr=0xffffffffffff8a4)
at common.c:145
#4 0x0000000100001370 in main (argc=11, argv=0xffffffffffff950) at pg_dump.c:683
(gdb) quit
A debugging session is active.

Inferior 1 [process 483438] will be killed.

Quit anyway? (y or n) y

Hope that helps.

Matthew

-----Original Message-----
From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
Sent: 28 September 2012 16:27
To: Carrington, Matthew (Produban)
Subject: Re: [GENERAL] pg_upgrade: out of memory

"Carrington, Matthew (Produban)" <Matthew(dot)Carrington(at)produban(dot)co(dot)uk> writes:
> ... presumably pg_extension is a 9.2 thing and I only have 9.0.1 installed.

I thought you were doing this test with 9.2? Oh wait, this is 9.2
pg_dump against 9.0 server. In that case leave off the EXISTS check:

SELECT tableoid, oid, proname AS aggname,
pronamespace AS aggnamespace,
pronargs, proargtypes,
(SELECT rolname FROM pg_catalog.pg_roles WHERE oid = proowner) AS rolname,
proacl AS aggacl
FROM pg_proc p
WHERE proisagg AND (
pronamespace !=
(SELECT oid FROM pg_namespace WHERE nspname = 'pg_catalog')
);

regards, tom lane
Emails aren't always secure, and they may be intercepted or changed
after they've been sent. Produban doesn't accept liability if this
happens. If you think someone may have interfered with this email,
please get in touch with the sender another way. This message and any
documents attached to it do not create or change any contract unless
otherwise specifically stated. Any views or opinions contained in this
message are solely those of the author, and do not necessarily represent
those of Produban, unless otherwise specifically stated and the sender
is authorised to do so. Produban doesn't accept responsibility for
damage caused by any viruses contained in this email or its attachments.
Emails may be monitored. If you've received this email by mistake,
please let the sender know at once that it's gone to the wrong person
and then destroy it without copying, using, or telling anyone about its
contents. Produban Servicios Informaticos Generales, S.L. (UK Branch).
Registered office: Shenley Wood House, Chalkdell Drive, Shenley Wood,
Milton Keynes MK5 6LA. Branch registration number BR 008486.
Ref:[PDB#014]

Hi Tom/Matthew,

Just to chime in on this thread - I'm currently validating Postgres on
AIXv7.1 and confirm that I also see the same error. I can reproduce the
error with 9.2.1 and 9.2.0 but unlike Matthew I'm using a built from
source build using the IBM xcl compiler rather than gcc.

I don't believe this is data limit related as I see the error when
dumping database "template1" under user "postgres". Here's the output:-

[eg17ph01:ahastie] /ahastie $ pg_dump -v -U postgres template1 > test.psql
Password:
pg_dump: reading schemas
pg_dump: reading user-defined tables
pg_dump: reading extensions
pg_dump: reading user-defined functions
pg_dump: reading user-defined types
pg_dump: reading procedural languages
pg_dump: reading user-defined aggregate functions
pg_dump: out of memory
[eg17ph01:ahastie] /ahastie $ xlc -qversion
IBM XL C/C++ for AIX, V12.1 (5765-J02, 5725-C72)
Version: 12.01.0000.0001

I've tried the requested SQL query which returns zero rows. Is this as
expected ?
I will try the same with release 9.1.6 to see if we can pinpoint this as
a potential AIX only issue or a 9.2.n issue. Also to confirm what
Matthew has observed.

Regards,
Andrew

On 28/09/12 16:12, Tom Lane wrote:
> "Carrington, Matthew (Produban)" <Matthew(dot)Carrington(at)produban(dot)co(dot)uk> writes:
>> Reading symbols from /ukmetmon/data/dataCollection/postgres_9.2.1/bin/pg_dump...(no debugging symbols found)...done.
> ... hm, not sure why that didn't work, but anyway:
>
>> (gdb) bt
>> #0 0x000000010002e354 in exit_horribly ()
>> #1 0x000000010003243c in pg_malloc ()
>> #2 0x0000000100008f14 in getAggregates ()
>> #3 0x000000010002fcac in getSchemaData ()
>> #4 0x0000000100001330 in main ()
> getAggregates() doesn't do that much. Can we see the results of the
> query it would have been executing, namely
>
> SELECT tableoid, oid, proname AS aggname,
> pronamespace AS aggnamespace,
> pronargs, proargtypes,
> (SELECT rolname FROM pg_catalog.pg_roles WHERE oid = proowner) AS rolname,
> proacl AS aggacl
> FROM pg_proc p
> WHERE proisagg AND (
> pronamespace !=
> (SELECT oid FROM pg_namespace WHERE nspname = 'pg_catalog')
> OR EXISTS(SELECT 1 FROM pg_depend WHERE
> classid = 'pg_proc'::regclass AND
> objid = p.oid AND
> refclassid = 'pg_extension'::regclass AND
> deptype = 'e'));
>
>
> regards, tom lane
>
>

"Carrington, Matthew (Produban)" <Matthew(dot)Carrington(at)produban(dot)co(dot)uk> writes:
> pg_dump: reading user-defined aggregate functions

> Breakpoint 1, exit_horribly (modulename=0x0, fmt=0x10006a590 "out of memory\n")
> at dumputils.c:1314
> 1314 dumputils.c: A file or directory in the path name does not exist..
> (gdb) bt
> #0 exit_horribly (modulename=0x0, fmt=0x10006a590 "out of memory\n") at dumputils.c:1314
> #1 0x000000010003247c in pg_malloc (size=0) at dumpmem.c:47
> #2 0x0000000100008f54 in getAggregates (fout=0x11000bad0, numAggs=0xffffffffffff73c)
> at pg_dump.c:3614

Oh! Given your previous comment about there not being any user-defined
aggregates, I see what the problem is. AIX must be one of the platforms
where malloc(0) is defined to return NULL rather than a pointer to a
zero-size block. pg_malloc is not coping with that.

A quick fix would be

pg_malloc(size_t size)
{
void *tmp;

tmp = malloc(size);
- if (!tmp)
+ if (!tmp && size)
{
psql_error("out of memory\n");
exit(EXIT_FAILURE);
}

but I'm not sure if that's the best answer overall. Will take it up in
-hackers.

regards, tom lane

Tom,

Yes, that look right for AIX . Here's the relevant section from the malloc man page ...

Return Values

Upon successful completion, the malloc subroutine returns a pointer to space
suitably aligned for the storage of any type of object. If the size requested is
0, malloc returns NULL in normal circumstances. However, if the program was
compiled with the defined _LINUX_SOURCE_COMPAT macro, malloc returns a valid
pointer to a space of size 0.

If the request cannot be satisfied for any reason, the malloc subroutine returns

NULL.

Hope that helps.

Matthew

-----Original Message-----
From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
Sent: 01 October 2012 14:39
To: Carrington, Matthew (Produban)
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: [GENERAL] pg_upgrade: out of memory

"Carrington, Matthew (Produban)" <Matthew(dot)Carrington(at)produban(dot)co(dot)uk> writes:
> pg_dump: reading user-defined aggregate functions

> Breakpoint 1, exit_horribly (modulename=0x0, fmt=0x10006a590 "out of memory\n")
> at dumputils.c:1314
> 1314 dumputils.c: A file or directory in the path name does not exist..
> (gdb) bt
> #0 exit_horribly (modulename=0x0, fmt=0x10006a590 "out of memory\n") at dumputils.c:1314
> #1 0x000000010003247c in pg_malloc (size=0) at dumpmem.c:47
> #2 0x0000000100008f54 in getAggregates (fout=0x11000bad0, numAggs=0xffffffffffff73c)
> at pg_dump.c:3614

Oh! Given your previous comment about there not being any user-defined
aggregates, I see what the problem is. AIX must be one of the platforms
where malloc(0) is defined to return NULL rather than a pointer to a
zero-size block. pg_malloc is not coping with that.

A quick fix would be

pg_malloc(size_t size)
{
void *tmp;

tmp = malloc(size);
- if (!tmp)
+ if (!tmp && size)
{
psql_error("out of memory\n");
exit(EXIT_FAILURE);
}

but I'm not sure if that's the best answer overall. Will take it up in
-hackers.

regards, tom lane
Emails aren't always secure, and they may be intercepted or changed
after they've been sent. Produban doesn't accept liability if this
happens. If you think someone may have interfered with this email,
please get in touch with the sender another way. This message and any
documents attached to it do not create or change any contract unless
otherwise specifically stated. Any views or opinions contained in this
message are solely those of the author, and do not necessarily represent
those of Produban, unless otherwise specifically stated and the sender
is authorised to do so. Produban doesn't accept responsibility for
damage caused by any viruses contained in this email or its attachments.
Emails may be monitored. If you've received this email by mistake,
please let the sender know at once that it's gone to the wrong person
and then destroy it without copying, using, or telling anyone about its
contents. Produban Servicios Informaticos Generales, S.L. (UK Branch).
Registered office: Shenley Wood House, Chalkdell Drive, Shenley Wood,
Milton Keynes MK5 6LA. Branch registration number BR 008486.
Ref:[PDB#014]

Tom,

Following on from that man page extract I tried a build using the suggested compiler macro (_LINUX_SOURCE_COMPAT) ...

export "CC=/opt/freeware/bin/gcc -maix64"
export OBJECT_MODE=64
export CFLAGS="-D_LINUX_SOURCE_COMPAT -maix64 -g"
export LDFLAGS="-maix64 -Wl,-bbigtoc"
export AR="ar -X64"
export "CC=/opt/freeware/bin/gcc -maix64"

./configure --enable-debug --prefix=/opt/serviceMonitoring/postgres_9.2.1 --disable-thread-safety --enable-cassert

make
cd contrib
make
cd ..
make install
cd contrib
make install

... and tried it out ...

"/opt/serviceMonitoring/postgres_9.2.1/bin/pg_dump" --port 65432 --username "postgres" --verbose --schema-only --binary-upgrade -f dump.out template1

.. and the full dump as per pg_upgrade ...

"/opt/serviceMonitoring/postgres_9.2.1/bin/pg_dumpall" --port 65432 --username "postgres" --schema-only --binary-upgrade -f pg_upgrade_dump_all.sql

.. both of which worked without any problems.

Hope that helps.

Matthew

-----Original Message-----
From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
Sent: 01 October 2012 14:39
To: Carrington, Matthew (Produban)
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: [GENERAL] pg_upgrade: out of memory

"Carrington, Matthew (Produban)" <Matthew(dot)Carrington(at)produban(dot)co(dot)uk> writes:
> pg_dump: reading user-defined aggregate functions

> Breakpoint 1, exit_horribly (modulename=0x0, fmt=0x10006a590 "out of memory\n")
> at dumputils.c:1314
> 1314 dumputils.c: A file or directory in the path name does not exist..
> (gdb) bt
> #0 exit_horribly (modulename=0x0, fmt=0x10006a590 "out of memory\n") at dumputils.c:1314
> #1 0x000000010003247c in pg_malloc (size=0) at dumpmem.c:47
> #2 0x0000000100008f54 in getAggregates (fout=0x11000bad0, numAggs=0xffffffffffff73c)
> at pg_dump.c:3614

Oh! Given your previous comment about there not being any user-defined
aggregates, I see what the problem is. AIX must be one of the platforms
where malloc(0) is defined to return NULL rather than a pointer to a
zero-size block. pg_malloc is not coping with that.

A quick fix would be

pg_malloc(size_t size)
{
void *tmp;

tmp = malloc(size);
- if (!tmp)
+ if (!tmp && size)
{
psql_error("out of memory\n");
exit(EXIT_FAILURE);
}

but I'm not sure if that's the best answer overall. Will take it up in
-hackers.

regards, tom lane
Emails aren't always secure, and they may be intercepted or changed
after they've been sent. Produban doesn't accept liability if this
happens. If you think someone may have interfered with this email,
please get in touch with the sender another way. This message and any
documents attached to it do not create or change any contract unless
otherwise specifically stated. Any views or opinions contained in this
message are solely those of the author, and do not necessarily represent
those of Produban, unless otherwise specifically stated and the sender
is authorised to do so. Produban doesn't accept responsibility for
damage caused by any viruses contained in this email or its attachments.
Emails may be monitored. If you've received this email by mistake,
please let the sender know at once that it's gone to the wrong person
and then destroy it without copying, using, or telling anyone about its
contents. Produban Servicios Informaticos Generales, S.L. (UK Branch).
Registered office: Shenley Wood House, Chalkdell Drive, Shenley Wood,
Milton Keynes MK5 6LA. Branch registration number BR 008486.
Ref:[PDB#014]

Tom/Matthew,

I also tried the same macro with the xlc compiler with similar results
in that pg_dump now works as expected :-)

For info here's my build setup:-
./configure CC=xlc LIBS="-lssl -lcrypto -lz -lreadline -lcurses -lld
-lmass -lm" CFLAGS="-qlanglvl=extc89 -D_LINUX_SOURCE_COMPAT"
--with-template=aix --prefix=/home/ahastie/pgbuild
--with-includes=/opt/freeware/include
--with-libraries=/opt/freeware/lib
gmake
gmake check
gmake install

Tom: Is this something we should get added into the AIX Platform
specific notes?

Regards,
Andrew

On 01/10/12 15:50, Carrington, Matthew (Produban) wrote:
> Tom,
>
> Following on from that man page extract I tried a build using the suggested compiler macro (_LINUX_SOURCE_COMPAT) ...
>
> export "CC=/opt/freeware/bin/gcc -maix64"
> export OBJECT_MODE=64
> export CFLAGS="-D_LINUX_SOURCE_COMPAT -maix64 -g"
> export LDFLAGS="-maix64 -Wl,-bbigtoc"
> export AR="ar -X64"
> export "CC=/opt/freeware/bin/gcc -maix64"
>
> ./configure --enable-debug --prefix=/opt/serviceMonitoring/postgres_9.2.1 --disable-thread-safety --enable-cassert
>
> make
> cd contrib
> make
> cd ..
> make install
> cd contrib
> make install
>
> ... and tried it out ...
>
> "/opt/serviceMonitoring/postgres_9.2.1/bin/pg_dump" --port 65432 --username "postgres" --verbose --schema-only --binary-upgrade -f dump.out template1
>
> .. and the full dump as per pg_upgrade ...
>
> "/opt/serviceMonitoring/postgres_9.2.1/bin/pg_dumpall" --port 65432 --username "postgres" --schema-only --binary-upgrade -f pg_upgrade_dump_all.sql
>
> .. both of which worked without any problems.
>
> Hope that helps.
>
> Matthew
>
> -----Original Message-----
> From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
> Sent: 01 October 2012 14:39
> To: Carrington, Matthew (Produban)
> Cc: pgsql-general(at)postgresql(dot)org
> Subject: Re: [GENERAL] pg_upgrade: out of memory
>
> "Carrington, Matthew (Produban)" <Matthew(dot)Carrington(at)produban(dot)co(dot)uk> writes:
>> pg_dump: reading user-defined aggregate functions
>> Breakpoint 1, exit_horribly (modulename=0x0, fmt=0x10006a590 "out of memory\n")
>> at dumputils.c:1314
>> 1314 dumputils.c: A file or directory in the path name does not exist..
>> (gdb) bt
>> #0 exit_horribly (modulename=0x0, fmt=0x10006a590 "out of memory\n") at dumputils.c:1314
>> #1 0x000000010003247c in pg_malloc (size=0) at dumpmem.c:47
>> #2 0x0000000100008f54 in getAggregates (fout=0x11000bad0, numAggs=0xffffffffffff73c)
>> at pg_dump.c:3614
> Oh! Given your previous comment about there not being any user-defined
> aggregates, I see what the problem is. AIX must be one of the platforms
> where malloc(0) is defined to return NULL rather than a pointer to a
> zero-size block. pg_malloc is not coping with that.
>
> A quick fix would be
>
> pg_malloc(size_t size)
> {
> void *tmp;
>
> tmp = malloc(size);
> - if (!tmp)
> + if (!tmp && size)
> {
> psql_error("out of memory\n");
> exit(EXIT_FAILURE);
> }
>
> but I'm not sure if that's the best answer overall. Will take it up in
> -hackers.
>
> regards, tom lane
> Emails aren't always secure, and they may be intercepted or changed
> after they've been sent. Produban doesn't accept liability if this
> happens. If you think someone may have interfered with this email,
> please get in touch with the sender another way. This message and any
> documents attached to it do not create or change any contract unless
> otherwise specifically stated. Any views or opinions contained in this
> message are solely those of the author, and do not necessarily represent
> those of Produban, unless otherwise specifically stated and the sender
> is authorised to do so. Produban doesn't accept responsibility for
> damage caused by any viruses contained in this email or its attachments.
> Emails may be monitored. If you've received this email by mistake,
> please let the sender know at once that it's gone to the wrong person
> and then destroy it without copying, using, or telling anyone about its
> contents. Produban Servicios Informaticos Generales, S.L. (UK Branch).
> Registered office: Shenley Wood House, Chalkdell Drive, Shenley Wood,
> Milton Keynes MK5 6LA. Branch registration number BR 008486.
> Ref:[PDB#014]
>
>
>

Andrew Hastie <andrew(at)ahastie(dot)net> writes:
> Tom: Is this something we should get added into the AIX Platform
> specific notes?

No, it's something we need to fix. See hackers thread:
http://archives.postgresql.org/pgsql-hackers/2012-10/msg00029.php

That #define isn't a bad quick-workaround if you don't want to touch
the code, but it's not a useful solution from our standpoint because
it only fixes this on AIX.

regards, tom lane