Re: OSX doesn't accept identical source/target for strcpy() anymore

Hello,

I am trying to compile PostgreSQL 9.3.1 on the new OS X Mavericks, which uses Darwin Kernel Version 13.0.0.

The compile process finishes fine, but when I am trying to test the resulting binary “make check” fails with:

> ============== initializing database system ==============
>
> pg_regress: initdb failed
> Examine /<my path>/postgresql-9.3.1/src/test/regress/log/initdb.log for the reason.

The “initdb.log” file contains the following lines:

> creating directory /<my path>/postgresql-9.3.1/src/test/regress/./tmp_check/data ... ok
> creating subdirectories ... ok
> selecting default max_connections ... 100
> selecting default shared_buffers ... 128MB
> creating configuration files ... ok
> creating template1 database in /<my path>/postgresql-9.3.1/src/test/regress/./tmp_check/data/base/1 ... ok
> initializing pg_authid ... ok
> initializing dependencies ... ok
> creating system views ... ok
> loading system objects' descriptions ... ok
> creating collations ... ok
> creating conversions ... ok
> creating dictionaries ... ok
> setting privileges on built-in objects ... ok
> creating information schema ... sh: line 1: 54928 Abort trap: 6 "<my path>/postgresql-9.3.1/src/test/regress/tmp_check/install/<my path>/pgsql/bin/postgres" --single -F -O -c search_path=pg_catalog -c exit_on_error=true -j template1 > /dev/null
> child process exited with exit code 134

I was trying to compile with the default clang compiler as good as with the gcc-4.2 compiler. Both result in the same error message. I searched the mailing lists. Some fixed their "code 134” problems on other operating systems by deleting and re-compiling the PostgreSQL code directories. I did this a few times, but the result was always the same.
Any ideas?

Thank you for your support

Greetings from Luxembourg

Matthias Schmitt

Hi,

On 2013-10-28 15:21:46 +0100, Matthias Schmitt wrote:
> > creating information schema ... sh: line 1: 54928 Abort trap: 6 "<my path>/postgresql-9.3.1/src/test/regress/tmp_check/install/<my path>/pgsql/bin/postgres" --single -F -O -c search_path=pg_catalog -c exit_on_error=true -j template1 > /dev/null

Could you execute the following (replacing <my path> with your directory)
export PGDATA=<my path>/postgresql-9.3.1/src/test/regress/tmp_check/data/
<my path>/postgresql-9.3.1/src/test/regress/tmp_check/install/<my path>/pgsql/bin/postgres" --single -F -O -c search_path=pg_catalog -c exit_on_error=true -j template1

That might already fail. If so, the output would be interesting. If it
doesn't fail, could you execute the following in the shell the above
will open?

UPDATE information_schema.sql_implementation_info SET character_value = 'blub' WHERE implementation_info_name = 'DBMS VERSION';
COPY information_schema.sql_features (feature_id, feature_name, sub_feature_id, sub_feature_name, is_supported, comments) FROM E'<my path>/postgresql-9.3.1/src/test/regress/tmp_check/install/share/sql_features.txt'

Again, the error message (something about a failed assert) would be the
interesting part.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Hello Andres,

thank you for your extremely fast reaction.

> Could you execute the following (replacing <my path> with your directory)
> export PGDATA=<my path>/postgresql-9.3.1/src/test/regress/tmp_check/data/
> <my path>/postgresql-9.3.1/src/test/regress/tmp_check/install/<my path>/pgsql/bin/postgres" --single -F -O -c search_path=pg_catalog -c exit_on_error=true -j template1
>
> That might already fail.

It didn’t.

> If it doesn't fail, could you execute the following in the shell the above will open?
>
> UPDATE information_schema.sql_implementation_info SET character_value = 'blub' WHERE implementation_info_name = 'DBMS VERSION’;

No problem.

> COPY information_schema.sql_features (feature_id, feature_name, sub_feature_id, sub_feature_name, is_supported, comments) FROM E'<my path>/postgresql-9.3.1/src/test/regress/tmp_check/install/share/sql_features.txt’

I assume you meant:

COPY information_schema.sql_features (feature_id, feature_name, sub_feature_id, sub_feature_name, is_supported, comments) FROM E'<my path>/postgresql-9.3.1/src/test/regress/tmp_check/install/<my install path>/share/sql_features.txt'

I still have a working prompt. I cannot see any errors until here.

I first executed the COPY command using your path. This file does not exist on my drive. What astonishes me most: there was no error message using this non existing file path. Are the errors messages written somewhere else?

Matthias

On 2013-10-28 16:12:23 +0100, Matthias Schmitt wrote:
> > COPY information_schema.sql_features (feature_id, feature_name, sub_feature_id, sub_feature_name, is_supported, comments) FROM E'<my path>/postgresql-9.3.1/src/test/regress/tmp_check/install/share/sql_features.txt’
>
> I assume you meant:
>
> COPY information_schema.sql_features (feature_id, feature_name, sub_feature_id, sub_feature_name, is_supported, comments) FROM E'<my path>/postgresql-9.3.1/src/test/regress/tmp_check/install/<my install path>/share/sql_features.txt'

Hm, yes.

> I still have a working prompt. I cannot see any errors until here.
>
> I first executed the COPY command using your path. This file does not exist on my drive. What astonishes me most: there was no error message using this non existing file path. Are the errors messages written somewhere else?

In that case, could you enable coredumps and get a backtrace from that
coredump? I unfortunately have zero clue about OSX, so I can't really
help you with that.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Hello Andres,

On 28.10.2013, at 16:37, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:

> In that case, could you enable coredumps and get a backtrace from that
> coredump? I unfortunately have zero clue about OSX, so I can't really
> help you with that.

Yes, you are right. I totally forgot about the crash logs in OS X. Here it is:

Process: postgres [68949]
Path: /Users/*/postgres
Identifier: postgres
Version: 0
Code Type: X86-64 (Native)
Parent Process: sh [68948]
Responsible: Terminal [411]
User ID: 502

Date/Time: 2013-10-28 16:46:28.188 +0100
OS Version: Mac OS X 10.9 (13A603)
Report Version: 11
Anonymous UUID: 26964FA1-0D3B-C18F-E2E5-3A77BE0EC789

Crashed Thread: 0 Dispatch queue: com.apple.main-thread

Exception Type: EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000

Application Specific Information:
detected source and destination buffer overlap

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x00007fff93e00866 __pthread_kill + 10
1 libsystem_pthread.dylib 0x00007fff92c7335c pthread_kill + 92
2 libsystem_c.dylib 0x00007fff8c5a5bba abort + 125
3 libsystem_c.dylib 0x00007fff8c5a5d31 abort_report_np + 181
4 libsystem_c.dylib 0x00007fff8c5c98c5 __chk_fail + 48
5 libsystem_c.dylib 0x00007fff8c5c98d5 __chk_fail_overlap + 16
6 libsystem_c.dylib 0x00007fff8c5c9906 __chk_overlap + 49
7 libsystem_c.dylib 0x00007fff8c5c9a59 __strncpy_chk + 78
8 postgres 0x000000010b4c9045 namestrcpy + 86
9 postgres 0x000000010b1901f2 TupleDescInitEntry + 99
10 postgres 0x000000010b57f53e resolve_polymorphic_tupdesc + 933
11 postgres 0x000000010b57efc2 internal_get_result_type + 227
12 postgres 0x000000010b57ee03 get_expr_result_type + 72
13 postgres 0x000000010b2a205b expandRecordVariable + 1552
14 postgres 0x000000010b29734b ParseComplexProjection + 154
15 postgres 0x000000010b295025 ParseFuncOrColumn + 1055
16 postgres 0x000000010b2906f1 transformIndirection + 404
17 postgres 0x000000010b28fef1 transformExprRecurse + 617
18 postgres 0x000000010b28fc74 transformExpr + 52
19 postgres 0x000000010b29fa08 transformTargetEntry + 56
20 postgres 0x000000010b29fb6e transformTargetList + 288
21 postgres 0x000000010b2543a6 transformSelectStmt + 351
22 postgres 0x000000010b25318d transformStmt + 476
23 postgres 0x000000010b252faf transformTopLevelStmt + 211
24 postgres 0x000000010b45530a pg_analyze_and_rewrite_params + 96
25 postgres 0x000000010b24d1bc fmgr_sql_validator + 998
26 postgres 0x000000010b57d120 OidFunctionCall1Coll + 165
27 postgres 0x000000010b24ca77 ProcedureCreate + 7163
28 postgres 0x000000010b2dbc88 CreateFunction + 1843
29 postgres 0x000000010b45f422 ProcessUtilitySlow + 2625
30 postgres 0x000000010b45e9ac standard_ProcessUtility + 3692
31 postgres 0x000000010b45db3e ProcessUtility + 134
32 postgres 0x000000010b45ca5e PortalRunUtility + 287
33 postgres 0x000000010b45cc03 PortalRunMulti + 386
34 postgres 0x000000010b45c1f2 PortalRun + 854
35 postgres 0x000000010b4559e8 exec_simple_query + 1109
36 postgres 0x000000010b45a403 PostgresMain + 2383
37 postgres 0x000000010b36c49b main + 535
38 libdyld.dylib 0x00007fff976735fd start + 1

Thread 0 crashed with X86 Thread State (64-bit):
rax: 0x0000000000000000 rbx: 0x00007fff7ca45310 rcx: 0x00007fff54a77998 rdx: 0x0000000000000000
rdi: 0x0000000000000507 rsi: 0x0000000000000006 rbp: 0x00007fff54a779c0 rsp: 0x00007fff54a77998
r8: 0x00000000ffffffc0 r9: 0x00007fff54a77900 r10: 0x0000000008000000 r11: 0x0000000000000206
r12: 0x0000000042c80000 r13: 0x0000000000000000 r14: 0x0000000000000006 r15: 0x00007fb5948bf0cc
rip: 0x00007fff93e00866 rfl: 0x0000000000000206 cr2: 0x000000011488a000

Logical CPU: 0
Error Code: 0x02000148
Trap Number: 133

Binary Images:
0x10b186000 - 0x10b751fe7 +postgres (0) <A95DF512-D8FE-3C20-83FE-EF948757A323> /Users/*/postgres
0x7fff60a5a000 - 0x7fff60a8d817 dyld (239.3) <D1DFCF3F-0B0C-332A-BCC0-87A851B570FF> /usr/lib/dyld
0x7fff8b291000 - 0x7fff8b2acff7 libsystem_malloc.dylib (23.1.10) <FFE5C472-B23A-318A-85BF-77CDE61900D1> /usr/lib/system/libsystem_malloc.dylib
0x7fff8bca9000 - 0x7fff8bcafff7 libsystem_platform.dylib (24.1.4) <331BA4A5-55CE-3B95-99EB-44E0C89D7FB8> /usr/lib/system/libsystem_platform.dylib
0x7fff8c045000 - 0x7fff8c047ff7 libquarantine.dylib (71) <7A1A2BCB-C03D-3A25-BFA4-3E569B2D2C38> /usr/lib/system/libquarantine.dylib
0x7fff8c549000 - 0x7fff8c5d2ff7 libsystem_c.dylib (997.1.1) <61833FAA-7281-3FF9-937F-686B6F20427C> /usr/lib/system/libsystem_c.dylib
0x7fff8c64c000 - 0x7fff8c64eff3 libsystem_configuration.dylib (596.12) <C4F633D9-94C8-35D9-BB2D-84C5122533C7> /usr/lib/system/libsystem_configuration.dylib
0x7fff8c970000 - 0x7fff8c97afff libcommonCrypto.dylib (60049) <8C4F0CA0-389C-3EDC-B155-E62DD2187E1D> /usr/lib/system/libcommonCrypto.dylib
0x7fff8ca05000 - 0x7fff8ca09ff7 libcache.dylib (62) <BDC1E65B-72A1-3DA3-A57C-B23159CAAD0B> /usr/lib/system/libcache.dylib
0x7fff8ca47000 - 0x7fff8ca50ff3 libsystem_notify.dylib (121) <52571EC3-6894-37E4-946E-064B021ED44E> /usr/lib/system/libsystem_notify.dylib
0x7fff8cc09000 - 0x7fff8cc4bff7 libauto.dylib (185.5) <F45C36E8-B606-3886-B5B1-B6745E757CA8> /usr/lib/libauto.dylib
0x7fff8e01b000 - 0x7fff8e042ffb libsystem_info.dylib (449.1.3) <7D41A156-D285-3849-A2C3-C04ADE797D98> /usr/lib/system/libsystem_info.dylib
0x7fff8e04c000 - 0x7fff8e050fff libsystem_stats.dylib (93.1.26) <B9E26A9E-FBBC-3938-B8B7-6CF7CA8C99AD> /usr/lib/system/libsystem_stats.dylib
0x7fff8fab8000 - 0x7fff8fad2fff libdispatch.dylib (339.1.9) <46878A5B-4248-3057-962C-6D4A235EEF31> /usr/lib/system/libdispatch.dylib
0x7fff8fc36000 - 0x7fff8fc3dff7 liblaunch.dylib (842.1.4) <FCBF0A02-0B06-3F97-9248-5062A9DEB32C> /usr/lib/system/liblaunch.dylib
0x7fff8fc84000 - 0x7fff8fca8fff libxpc.dylib (300.1.17) <4554927A-9467-365C-91F1-5A116989DD7F> /usr/lib/system/libxpc.dylib
0x7fff90875000 - 0x7fff908c3fff libcorecrypto.dylib (161.1) <F3973C28-14B6-3006-BB2B-00DD7F09ABC7> /usr/lib/system/libcorecrypto.dylib
0x7fff9091d000 - 0x7fff90944ff7 libsystem_network.dylib (241.3) <8B1E1F1D-A5CC-3BAE-8B1E-ABC84337A364> /usr/lib/system/libsystem_network.dylib
0x7fff90e9a000 - 0x7fff90e9bffb libremovefile.dylib (33) <3543F917-928E-3DB2-A2F4-7AB73B4970EF> /usr/lib/system/libremovefile.dylib
0x7fff90e9c000 - 0x7fff90e9dfff libunc.dylib (28) <62682455-1862-36FE-8A04-7A6B91256438> /usr/lib/system/libunc.dylib
0x7fff91875000 - 0x7fff9187dfff libsystem_dnssd.dylib (522.1.11) <270DCF6C-502D-389A-AA9F-DE4624A36FF7> /usr/lib/system/libsystem_dnssd.dylib
0x7fff9199c000 - 0x7fff91b49f27 libobjc.A.dylib (551.1) <AD7FD984-271E-30F4-A361-6B20319EC73B> /usr/lib/libobjc.A.dylib
0x7fff92859000 - 0x7fff9285efff libmacho.dylib (845) <1D2910DF-C036-3A82-A3FD-44FF73B5FF9B> /usr/lib/system/libmacho.dylib
0x7fff92c71000 - 0x7fff92c78ff7 libsystem_pthread.dylib (53.1.4) <AB498556-B555-310E-9041-F67EC9E00E2C> /usr/lib/system/libsystem_pthread.dylib
0x7fff92da6000 - 0x7fff92dd5fd2 libsystem_m.dylib (3047.16) <B7F0E2E4-2777-33FC-A787-D6430B630D54> /usr/lib/system/libsystem_m.dylib
0x7fff92df7000 - 0x7fff92dfeff3 libcopyfile.dylib (103) <5A881779-D0D6-3029-B371-E3021C2DDA5E> /usr/lib/system/libcopyfile.dylib
0x7fff9371d000 - 0x7fff9371eff7 libDiagnosticMessagesClient.dylib (100) <4CDB0F7B-C0AF-3424-BC39-495696F0DB1E> /usr/lib/libDiagnosticMessagesClient.dylib
0x7fff9371f000 - 0x7fff93730ff7 libsystem_asl.dylib (217.1.4) <655FB343-52CF-3E2F-B14D-BEBF5AAEF94D> /usr/lib/system/libsystem_asl.dylib
0x7fff93deb000 - 0x7fff93e07ff7 libsystem_kernel.dylib (2422.1.72) <D14913DB-47F1-3591-8DAF-D4B4EF5F8818> /usr/lib/system/libsystem_kernel.dylib
0x7fff941f6000 - 0x7fff941fbff7 libunwind.dylib (35.3) <78DCC358-2FC1-302E-B395-0155B47CB547> /usr/lib/system/libunwind.dylib
0x7fff94323000 - 0x7fff94324ff7 libSystem.B.dylib (1197.1.1) <BFC0DC97-46C6-3BE0-9983-54A98734897A> /usr/lib/libSystem.B.dylib
0x7fff9485f000 - 0x7fff94888ff7 libc++abi.dylib (48) <8C16158F-CBF8-3BD7-BEF4-022704B2A326> /usr/lib/libc++abi.dylib
0x7fff95075000 - 0x7fff950c7fff libc++.1.dylib (120) <4F68DFC5-2077-39A8-A449-CAC5FDEE7BDE> /usr/lib/libc++.1.dylib
0x7fff974d1000 - 0x7fff974d8fff libcompiler_rt.dylib (35) <4CD916B2-1B17-362A-B403-EF24A1DAC141> /usr/lib/system/libcompiler_rt.dylib
0x7fff97670000 - 0x7fff97673ff7 libdyld.dylib (239.3) <62F4D752-4089-31A8-8B73-B95A68893B3C> /usr/lib/system/libdyld.dylib
0x7fff978b7000 - 0x7fff978b8ff7 libsystem_blocks.dylib (63) <FB856CD1-2AEA-3907-8E9B-1E54B6827F82> /usr/lib/system/libsystem_blocks.dylib
0x7fff981b7000 - 0x7fff981b8ff7 libsystem_sandbox.dylib (278.10) <A47E7E11-3C76-318E-B67D-98972B86F094> /usr/lib/system/libsystem_sandbox.dylib
0x7fff98246000 - 0x7fff98246ff7 libkeymgr.dylib (28) <3AA8D85D-CF00-3BD3-A5A0-E28E1A32A6D8> /usr/lib/system/libkeymgr.dylib

External Modification Summary:
Calls made by other processes targeting this process:
task_for_pid: 0
thread_create: 0
thread_set_state: 0
Calls made by this process:
task_for_pid: 0
thread_create: 0
thread_set_state: 0
Calls made by all processes on this machine:
task_for_pid: 2827
thread_create: 0
thread_set_state: 0

VM Region Summary:
ReadOnly portion of Libraries: Total=76.8M resident=29.3M(38%) swapped_out_or_unallocated=47.5M(62%)
Writable regions: Total=244.8M written=14.2M(6%) resident=14.3M(6%) swapped_out=0K(0%) unallocated=230.5M(94%)

REGION TYPE VIRTUAL
=========== =======
Kernel Alloc Once 4K
MALLOC 39.1M
MALLOC (admin) 16K
STACK GUARD 4K
Stack 64.0M
VM_ALLOCATE 140.9M
__DATA 1132K
__LINKEDIT 66.1M
__TEXT 10.8M
shared memory 4K
=========== =======
TOTAL 322.0M

I guess the developers can recognise some of their function calls in the thread listing.

Best regards,

Matthias Schmitt

Hi,

On -bugs it was reported that initdb of 9.3 failed with a
assertion.

On 2013-10-28 16:52:13 +0100, Matthias Schmitt wrote:
> > In that case, could you enable coredumps and get a backtrace from that
> > coredump? I unfortunately have zero clue about OSX, so I can't really
> > help you with that.
>
> Yes, you are right. I totally forgot about the crash logs in OS X. Here it is:

Ah, ok. I see the problem...

> Process: postgres [68949]
> Path: /Users/*/postgres
> Identifier: postgres
> Version: 0
> Code Type: X86-64 (Native)
> Parent Process: sh [68948]
> Responsible: Terminal [411]
> User ID: 502
>
> Date/Time: 2013-10-28 16:46:28.188 +0100
> OS Version: Mac OS X 10.9 (13A603)

> Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
> 0 libsystem_kernel.dylib 0x00007fff93e00866 __pthread_kill + 10
> 1 libsystem_pthread.dylib 0x00007fff92c7335c pthread_kill + 92
> 2 libsystem_c.dylib 0x00007fff8c5a5bba abort + 125
> 3 libsystem_c.dylib 0x00007fff8c5a5d31 abort_report_np + 181
> 4 libsystem_c.dylib 0x00007fff8c5c98c5 __chk_fail + 48
> 5 libsystem_c.dylib 0x00007fff8c5c98d5 __chk_fail_overlap + 16
> 6 libsystem_c.dylib 0x00007fff8c5c9906 __chk_overlap + 49
> 7 libsystem_c.dylib 0x00007fff8c5c9a59 __strncpy_chk + 78
> 8 postgres 0x000000010b4c9045 namestrcpy + 86
> 9 postgres 0x000000010b1901f2 TupleDescInitEntry + 99

It seems the newest version of OSX is more strict about strcpy than
previous ones. So the issue is likely the upgraded OS version. Which
means we're going to see this more frequently now.

There have been previous discussions about fixing strcpy calls with
identical source/destination (same for memcpy) but it was deemed not
worth the effort. I don't really see an alternative to fixing it now.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Matthias Schmitt <freak002(at)mmp(dot)lu> writes:
> Application Specific Information:
> detected source and destination buffer overlap

> Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
> 0 libsystem_kernel.dylib 0x00007fff93e00866 __pthread_kill + 10
> 1 libsystem_pthread.dylib 0x00007fff92c7335c pthread_kill + 92
> 2 libsystem_c.dylib 0x00007fff8c5a5bba abort + 125
> 3 libsystem_c.dylib 0x00007fff8c5a5d31 abort_report_np + 181
> 4 libsystem_c.dylib 0x00007fff8c5c98c5 __chk_fail + 48
> 5 libsystem_c.dylib 0x00007fff8c5c98d5 __chk_fail_overlap + 16
> 6 libsystem_c.dylib 0x00007fff8c5c9906 __chk_overlap + 49
> 7 libsystem_c.dylib 0x00007fff8c5c9a59 __strncpy_chk + 78
> 8 postgres 0x000000010b4c9045 namestrcpy + 86
> 9 postgres 0x000000010b1901f2 TupleDescInitEntry + 99
> 10 postgres 0x000000010b57f53e resolve_polymorphic_tupdesc + 933

Hmm. Well, it's pretty obvious what it's complaining about:
resolve_polymorphic_tupdesc is cheating a bit by telling
TupleDescInitEntry to re-initialize a tupledesc using the name value
that's already in the entry. What I find curious is that I can't
reproduce this problem on an OS X Mavericks machine here. You must
be using some nondefault compiler switches --- care to tell us what?

regards, tom lane

On 2013-10-28 12:22:56 -0400, Tom Lane wrote:
> Matthias Schmitt <freak002(at)mmp(dot)lu> writes:
> > Application Specific Information:
> > detected source and destination buffer overlap
>
> > Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
> > 0 libsystem_kernel.dylib 0x00007fff93e00866 __pthread_kill + 10
> > 1 libsystem_pthread.dylib 0x00007fff92c7335c pthread_kill + 92
> > 2 libsystem_c.dylib 0x00007fff8c5a5bba abort + 125
> > 3 libsystem_c.dylib 0x00007fff8c5a5d31 abort_report_np + 181
> > 4 libsystem_c.dylib 0x00007fff8c5c98c5 __chk_fail + 48
> > 5 libsystem_c.dylib 0x00007fff8c5c98d5 __chk_fail_overlap + 16
> > 6 libsystem_c.dylib 0x00007fff8c5c9906 __chk_overlap + 49
> > 7 libsystem_c.dylib 0x00007fff8c5c9a59 __strncpy_chk + 78
> > 8 postgres 0x000000010b4c9045 namestrcpy + 86
> > 9 postgres 0x000000010b1901f2 TupleDescInitEntry + 99
> > 10 postgres 0x000000010b57f53e resolve_polymorphic_tupdesc + 933
>
> Hmm. Well, it's pretty obvious what it's complaining about:
> resolve_polymorphic_tupdesc is cheating a bit by telling
> TupleDescInitEntry to re-initialize a tupledesc using the name value
> that's already in the entry. What I find curious is that I can't
> reproduce this problem on an OS X Mavericks machine here. You must
> be using some nondefault compiler switches --- care to tell us what?

Did you maybe compile using gcc or at least the gcc compatible frontend
instead of clang?

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> There have been previous discussions about fixing strcpy calls with
> identical source/destination (same for memcpy) but it was deemed not
> worth the effort. I don't really see an alternative to fixing it now.

I'm not seeing this with bare-bones

./configure --enable-debug --enable-cassert

on an OS X Mavericks machine with Xcode downloaded today. So there
is something unidentified as yet about Matthias's configuration.

regards, tom lane

On Mon, Oct 28, 2013 at 12:11 PM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> Hi,
>
> On -bugs it was reported that initdb of 9.3 failed with a
> assertion.
>
> On 2013-10-28 16:52:13 +0100, Matthias Schmitt wrote:
>> > In that case, could you enable coredumps and get a backtrace from that
>> > coredump? I unfortunately have zero clue about OSX, so I can't really
>> > help you with that.
>>
>> Yes, you are right. I totally forgot about the crash logs in OS X. Here it is:
>
> Ah, ok. I see the problem...
>
>> Process: postgres [68949]
>> Path: /Users/*/postgres
>> Identifier: postgres
>> Version: 0
>> Code Type: X86-64 (Native)
>> Parent Process: sh [68948]
>> Responsible: Terminal [411]
>> User ID: 502
>>
>> Date/Time: 2013-10-28 16:46:28.188 +0100
>> OS Version: Mac OS X 10.9 (13A603)
>
>> Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
>> 0 libsystem_kernel.dylib 0x00007fff93e00866 __pthread_kill + 10
>> 1 libsystem_pthread.dylib 0x00007fff92c7335c pthread_kill + 92
>> 2 libsystem_c.dylib 0x00007fff8c5a5bba abort + 125
>> 3 libsystem_c.dylib 0x00007fff8c5a5d31 abort_report_np + 181
>> 4 libsystem_c.dylib 0x00007fff8c5c98c5 __chk_fail + 48
>> 5 libsystem_c.dylib 0x00007fff8c5c98d5 __chk_fail_overlap + 16
>> 6 libsystem_c.dylib 0x00007fff8c5c9906 __chk_overlap + 49
>> 7 libsystem_c.dylib 0x00007fff8c5c9a59 __strncpy_chk + 78
>> 8 postgres 0x000000010b4c9045 namestrcpy + 86
>> 9 postgres 0x000000010b1901f2 TupleDescInitEntry + 99
>
> It seems the newest version of OSX is more strict about strcpy than
> previous ones. So the issue is likely the upgraded OS version. Which
> means we're going to see this more frequently now.
>
> There have been previous discussions about fixing strcpy calls with
> identical source/destination (same for memcpy) but it was deemed not
> worth the effort. I don't really see an alternative to fixing it now.

Ugh. Why in the world would Apple break this? We could try to go
through our code and fix this, but catching all of the instances is
likely to be hard, and everyone else who writes software of any
complexity is going to have the same darn problme.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> On 2013-10-28 12:22:56 -0400, Tom Lane wrote:
>> What I find curious is that I can't
>> reproduce this problem on an OS X Mavericks machine here. You must
>> be using some nondefault compiler switches --- care to tell us what?

> Did you maybe compile using gcc or at least the gcc compatible frontend
> instead of clang?

Well, I have

CC = gcc

which is what configure will pick by default, but I see

$ gcc -v
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/include/c++/4.2.1
Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn)
Target: x86_64-apple-darwin13.0.0
Thread model: posix

so it's some variant of LLVM/clang. In any case, Matthias claimed it
didn't make a difference which compiler he picked, and I'm pushing
back on that assertion.

regards, tom lane

Hello,

On 28.10.2013, at 17:22, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> Hmm. Well, it's pretty obvious what it's complaining about:
> resolve_polymorphic_tupdesc is cheating a bit by telling
> TupleDescInitEntry to re-initialize a tupledesc using the name value
> that's already in the entry. What I find curious is that I can't
> reproduce this problem on an OS X Mavericks machine here. You must
> be using some nondefault compiler switches --- care to tell us what?

I am not aware of using something special. Here is my configuration using Xcode 5.0.1 and the matching command line tools (october 2013):

Via environment variables:

export CFLAGS="-fPIC"
export CPPFLAGS="-fPIC"
export LDFLAGS="-fPIC"

I had trouble using the clang compiler in the past, so I am using the GCC compiler instead. I tried before also with clang, but I had the same problem, so I guess that it is no problem of GCC.

./configure CC='/usr/bin/gcc-4.2' --prefix=/<my path>/applications/pgsql --with-perl

Afterwards I am just doing ‘make’ and ‘make check’.

Just in case you need more information I added the output of the configuration to this mail.

checking build system type... x86_64-apple-darwin13.0.0
checking host system type... x86_64-apple-darwin13.0.0
checking which template to use... darwin
checking whether to build with 64-bit integer date/time support... yes
checking whether NLS is wanted... no
checking for default port number... 5432
checking for block size... 8kB
checking for segment size... 1GB
checking for WAL block size... 8kB
checking for WAL segment size... 16MB
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...

checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether /usr/bin/gcc-4.2 accepts -g... yes
checking for /usr/bin/gcc-4.2 option to accept ISO C89... none needed
checking whether /usr/bin/gcc-4.2 supports -Wdeclaration-after-statement... yes
checking whether /usr/bin/gcc-4.2 supports -Wendif-labels... yes
checking whether /usr/bin/gcc-4.2 supports -Wmissing-format-attribute... yes
checking whether /usr/bin/gcc-4.2 supports -Wformat-security... yes
checking whether /usr/bin/gcc-4.2 supports -fno-strict-aliasing... yes
checking whether /usr/bin/gcc-4.2 supports -fwrapv... yes
checking whether /usr/bin/gcc-4.2 supports -fexcess-precision=standard... no
checking whether /usr/bin/gcc-4.2 supports -funroll-loops... yes
checking whether /usr/bin/gcc-4.2 supports -ftree-vectorize... yes
checking whether the C compiler still works... yes
checking how to run the C preprocessor... /usr/bin/gcc-4.2 -E
checking allow thread-safe client libraries... yes
checking whether to build with Tcl... no
checking whether to build Perl modules... yes
checking whether to build Python modules... no
checking whether to build with GSSAPI support... no
checking whether to build with Kerberos 5 support... no
checking whether to build with PAM support... no
checking whether to build with LDAP support... no
checking whether to build with Bonjour support... no
checking whether to build with OpenSSL support... no
checking whether to build with SELinux support... no
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ld used by GCC... /usr/libexec/gcc/i686-apple-darwin11/4.2.1/ld
checking if the linker (/usr/libexec/gcc/i686-apple-darwin11/4.2.1/ld) is GNU ld... no
checking for ranlib... ranlib
checking for strip... strip
checking whether it is possible to strip libraries... yes
checking for ar... ar
checking for a BSD-compatible install... /usr/bin/install -c
checking for tar... /usr/bin/tar
checking whether ln -s works... yes
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking for a thread-safe mkdir -p... config/install-sh -c -d
checking for bison... /usr/bin/bison
configure: using bison (GNU Bison) 2.3
checking for flex... /usr/bin/flex
configure: using flex 2.5.35 Apple(flex-31)
checking for perl… /<my path>/applications/perl5/bin/perl
configure: using perl 5.16.3
checking for Perl archlibexp... /<my path>/applications/perl5/lib/5.16.3/darwin-2level
checking for Perl privlibexp... /<my path>/applications/perl5/lib/5.16.3
checking for Perl useshrplib... true
checking for flags to link embedded Perl... -fstack-protector -L/usr/local/lib -L/<my path>/applications/perl5/lib/5.16.3/darwin-2level/CORE -lperl -ldl -lm -lutil -lc
checking for main in -lm... yes
checking for library containing setproctitle... no
checking for library containing dlopen... none required
checking for library containing socket... none required
checking for library containing shl_load... no
checking for library containing getopt_long... none required
checking for library containing crypt... none required
checking for library containing fdatasync... none required
checking for library containing gethostbyname_r... no
checking for library containing shmget... none required
checking for library containing readline... -lreadline
checking for inflate in -lz... yes
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking crypt.h usability... no
checking crypt.h presence... no
checking for crypt.h... no
checking dld.h usability... no
checking dld.h presence... no
checking for dld.h... no
checking fp_class.h usability... no
checking fp_class.h presence... no
checking for fp_class.h... no
checking getopt.h usability... yes
checking getopt.h presence... yes
checking for getopt.h... yes
checking ieeefp.h usability... no
checking ieeefp.h presence... no
checking for ieeefp.h... no
checking ifaddrs.h usability... yes
checking ifaddrs.h presence... yes
checking for ifaddrs.h... yes
checking langinfo.h usability... yes
checking langinfo.h presence... yes
checking for langinfo.h... yes
checking poll.h usability... yes
checking poll.h presence... yes
checking for poll.h... yes
checking pwd.h usability... yes
checking pwd.h presence... yes
checking for pwd.h... yes
checking sys/ioctl.h usability... yes
checking sys/ioctl.h presence... yes
checking for sys/ioctl.h... yes
checking sys/ipc.h usability... yes
checking sys/ipc.h presence... yes
checking for sys/ipc.h... yes
checking sys/poll.h usability... yes
checking sys/poll.h presence... yes
checking for sys/poll.h... yes
checking sys/pstat.h usability... no
checking sys/pstat.h presence... no
checking for sys/pstat.h... no
checking sys/resource.h usability... yes
checking sys/resource.h presence... yes
checking for sys/resource.h... yes
checking sys/select.h usability... yes
checking sys/select.h presence... yes
checking for sys/select.h... yes
checking sys/sem.h usability... yes
checking sys/sem.h presence... yes
checking for sys/sem.h... yes
checking sys/shm.h usability... yes
checking sys/shm.h presence... yes
checking for sys/shm.h... yes
checking sys/socket.h usability... yes
checking sys/socket.h presence... yes
checking for sys/socket.h... yes
checking sys/sockio.h usability... yes
checking sys/sockio.h presence... yes
checking for sys/sockio.h... yes
checking sys/tas.h usability... no
checking sys/tas.h presence... no
checking for sys/tas.h... no
checking sys/time.h usability... yes
checking sys/time.h presence... yes
checking for sys/time.h... yes
checking sys/un.h usability... yes
checking sys/un.h presence... yes
checking for sys/un.h... yes
checking termios.h usability... yes
checking termios.h presence... yes
checking for termios.h... yes
checking ucred.h usability... no
checking ucred.h presence... no
checking for ucred.h... no
checking utime.h usability... yes
checking utime.h presence... yes
checking for utime.h... yes
checking wchar.h usability... yes
checking wchar.h presence... yes
checking for wchar.h... yes
checking wctype.h usability... yes
checking wctype.h presence... yes
checking for wctype.h... yes
checking for net/if.h... yes
checking for sys/ucred.h... yes
checking netinet/in.h usability... yes
checking netinet/in.h presence... yes
checking for netinet/in.h... yes
checking for netinet/tcp.h... yes
checking readline/readline.h usability... yes
checking readline/readline.h presence... yes
checking for readline/readline.h... yes
checking readline/history.h usability... yes
checking readline/history.h presence... yes
checking for readline/history.h... yes
checking zlib.h usability... yes
checking zlib.h presence... yes
checking for zlib.h... yes
checking whether byte ordering is bigendian... no
checking for an ANSI C-conforming const... yes
checking for inline... inline
checking for quiet inline (no complaint if unreferenced)... yes
checking for preprocessor stringizing operator... yes
checking for flexible array members... yes
checking for signed types... yes
checking for working volatile... yes
checking for __func__... yes
checking for _Static_assert... no
checking for __builtin_types_compatible_p... yes
checking for __builtin_constant_p... yes
checking for __builtin_unreachable... no
checking for __VA_ARGS__... yes
checking whether struct tm is in sys/time.h or time.h... time.h
checking for struct tm.tm_zone... yes
checking for tzname... yes
checking for union semun... yes
checking for struct sockaddr_un... yes
checking for struct sockaddr_storage... yes
checking for struct sockaddr_storage.ss_family... yes
checking for struct sockaddr_storage.__ss_family... no
checking for struct sockaddr_storage.ss_len... yes
checking for struct sockaddr_storage.__ss_len... no
checking for struct sockaddr.sa_len... yes
checking for struct addrinfo... yes
checking for intptr_t... yes
checking for uintptr_t... yes
checking for long long int... yes
checking for locale_t... yes (in xlocale.h)
checking for struct cmsgcred... no
checking for struct option... yes
checking for z_streamp... yes
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking size of off_t... 8
checking for int timezone... yes
checking types of arguments for accept()... int, int, struct sockaddr *, socklen_t *
checking whether gettimeofday takes only one argument... no
checking for cbrt... yes
checking for dlopen... yes
checking for fdatasync... yes
checking for getifaddrs... yes
checking for getpeerucred... no
checking for getrlimit... yes
checking for mbstowcs_l... yes
checking for memmove... yes
checking for poll... yes
checking for pstat... no
checking for readlink... yes
checking for setproctitle... no
checking for setsid... yes
checking for sigprocmask... yes
checking for symlink... yes
checking for sync_file_range... no
checking for towlower... yes
checking for utime... yes
checking for utimes... yes
checking for wcstombs... yes
checking for wcstombs_l... yes
checking for fseeko... yes
checking for _LARGEFILE_SOURCE value needed for large files... no
checking for posix_fadvise... no
checking whether posix_fadvise is declared... no
checking whether fdatasync is declared... no
checking whether strlcat is declared... yes
checking whether strlcpy is declared... yes
checking whether F_FULLFSYNC is declared... yes
checking for struct sockaddr_in6... yes
checking for PS_STRINGS... no
checking for snprintf... yes
checking for vsnprintf... yes
checking whether snprintf is declared... yes
checking whether vsnprintf is declared... yes
checking for isinf... yes
checking for crypt... yes
checking for fls... yes
checking for getopt... yes
checking for getrusage... yes
checking for inet_aton... yes
checking for random... yes
checking for rint... yes
checking for srandom... yes
checking for strerror... yes
checking for strlcat... yes
checking for strlcpy... yes
checking for unsetenv... yes
checking for getpeereid... yes
checking for getaddrinfo... yes
checking for getopt_long... yes
checking for sigsetjmp... yes
checking whether sys_siglist is declared... yes
checking for syslog... yes
checking syslog.h usability... yes
checking syslog.h presence... yes
checking for syslog.h... yes
checking for opterr... yes
checking for optreset... yes
checking for strtoll... yes
checking for strtoull... yes
checking for builtin locking functions... yes
checking for rl_completion_append_character... yes
checking for rl_completion_matches... yes
checking for rl_filename_completion_function... yes
checking for append_history... no
checking for history_truncate_file... yes
checking for the pthreads library -lpthreads... no
checking whether pthreads work without any flags... yes
checking whether pthreads work with -Kthread... yes
checking whether pthreads work with -kthread... yes
checking for the pthreads library -llthread... no
checking whether pthreads work with -pthread... yes
checking whether pthreads work with -pthreads... yes
checking whether pthreads work with -mthreads... no
checking for the pthreads library -lpthread... yes
checking whether pthreads work with --thread-safe... no
checking whether pthreads work with -mt... no
checking for the pthreads library -lpthreadGC2... no
checking pthread.h usability... yes
checking pthread.h presence... yes
checking for pthread.h... yes
checking for strerror_r... yes
checking for getpwuid_r... yes
checking for gethostbyname_r... no
checking whether getpwuid_r takes a fifth argument... yes
checking whether strerror_r returns int... yes
checking test program... ok
checking whether long int is 64 bits... yes
checking size of void *... 8
checking size of size_t... 8
checking size of long... 8
checking whether to build with float4 passed by value... yes
checking whether to build with float8 passed by value... yes
checking alignment of short... 2
checking alignment of int... 4
checking alignment of long... 8
checking alignment of double... 8
checking for int8... no
checking for uint8... no
checking for int64... no
checking for uint64... no
checking for sig_atomic_t... yes
checking for POSIX signal interface... yes
checking for working memcmp... yes
checking for perl.h... yes
checking for libperl... yes
checking for onsgmls... no
checking for nsgmls... no
checking for openjade... no
checking for jade... no
checking for DocBook V4.2... no
checking for DocBook stylesheets... no
checking for collateindex.pl... no
checking for xsltproc... xsltproc
checking for osx... no
checking for sgml2xml... no
checking for sx... no
checking thread safety of required library functions... yes
checking whether /usr/bin/gcc-4.2 supports -Wl,-dead_strip_dylibs... yes
configure: using compiler=i686-apple-darwin11-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5666) (dot 3)
configure: using CFLAGS=-fPIC -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv
configure: using CPPFLAGS=-fPIC
configure: using LDFLAGS=-fPIC -Wl,-dead_strip_dylibs
configure: creating ./config.status
config.status: creating GNUmakefile
config.status: creating src/Makefile.global
config.status: creating src/include/pg_config.h
config.status: creating src/include/pg_config_ext.h
config.status: creating src/interfaces/ecpg/include/ecpg_config.h
config.status: linking src/backend/port/tas/dummy.s to src/backend/port/tas.s
config.status: linking src/backend/port/dynloader/darwin.c to src/backend/port/dynloader.c
config.status: linking src/backend/port/sysv_sema.c to src/backend/port/pg_sema.c
config.status: linking src/backend/port/sysv_shmem.c to src/backend/port/pg_shmem.c
config.status: linking src/backend/port/unix_latch.c to src/backend/port/pg_latch.c
config.status: linking src/backend/port/dynloader/darwin.h to src/include/dynloader.h
config.status: linking src/include/port/darwin.h to src/include/pg_config_os.h
config.status: linking src/makefiles/Makefile.darwin to src/Makefile.port

Best regards

Matthias Schmitt

magic moving pixel s.a.
23, Avenue Grande-Duchesse Charlotte
L-3441 Dudelange
Luxembourg
Phone: +352 54 75 75 - 0
http://www.mmp.lu

On 28.10.2013, at 17:39, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> Well, I have
>
> CC = gcc
>
> which is what configure will pick by default, but I see
>
> $ gcc -v
> Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/include/c++/4.2.1
> Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn)
> Target: x86_64-apple-darwin13.0.0
> Thread model: posix
>
> so it's some variant of LLVM/clang. In any case, Matthias claimed it
> didn't make a difference which compiler he picked, and I'm pushing
> back on that assertion.

There is a difference between

gcc

which starts clang and

/usr/bin/gcc-4.2

which is the real GCC version 4.2.1.

Regards

Matthias

On 2013-10-28 12:39:03 -0400, Tom Lane wrote:
> Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> > On 2013-10-28 12:22:56 -0400, Tom Lane wrote:
> >> What I find curious is that I can't
> >> reproduce this problem on an OS X Mavericks machine here. You must
> >> be using some nondefault compiler switches --- care to tell us what?
>
> > Did you maybe compile using gcc or at least the gcc compatible frontend
> > instead of clang?
>
> Well, I have
>
> CC = gcc
>
> which is what configure will pick by default, but I see
>
> $ gcc -v
> Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/include/c++/4.2.1
> Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn)
> Target: x86_64-apple-darwin13.0.0
> Thread model: posix
>
> so it's some variant of LLVM/clang. In any case, Matthias claimed it
> didn't make a difference which compiler he picked, and I'm pushing
> back on that assertion.

A quick search reveals that we're not the only ones suffering from this:
https://issues.apache.org/bugzilla/show_bug.cgi?id=55696
http://lists.gnu.org/archive/html/bug-bash/2013-07/msg00011.html

There's a reference around to the code in question:
http://www.opensource.apple.com/source/Libc/Libc-997.1.1/secure/strcpy_chk.c
And it seems to be enabled by:

#if defined (__GNUC__) && _FORTIFY_SOURCE > 0 && !defined (__cplusplus)
/* Security checking functions. */
#include <secure/_string.h>
#endif
in
http://www.opensource.apple.com/source/Libc/Libc-997.1.1/include/string.h

So, if I understand it correctly, this only happens if _FORTIFY_SOURCE
is enabled. Not sure what does that on Matthias but not your machine.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Matthias Schmitt <freak002(at)mmp(dot)lu> writes:
> On 28.10.2013, at 17:22, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> What I find curious is that I can't
>> reproduce this problem on an OS X Mavericks machine here. You must
>> be using some nondefault compiler switches --- care to tell us what?

> ./configure CC='/usr/bin/gcc-4.2' --prefix=/<my path>/applications/pgsql --with-perl

Hm. There is no such file as /usr/bin/gcc-4.2 on my machine, in fact
there's no file following such a naming pattern anywhere on the
filesystem.

However, that doesn't seem to be relevant. Poking at this some more,
I see that the effect of the bit that Andres noticed in string.h is to
#define strncpy as __builtin___strncpy_chk, and *that replacement is
happening on my machine*. So why don't I see the trap? I am wondering
if there is some run-time environment test that controls whether the
trap occurs, and you've got an environment variable set that I don't.

regards, tom lane

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> On Mon, Oct 28, 2013 at 12:11 PM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
>> There have been previous discussions about fixing strcpy calls with
>> identical source/destination (same for memcpy) but it was deemed not
>> worth the effort. I don't really see an alternative to fixing it now.

> Ugh. Why in the world would Apple break this?

It's broken already; the C and POSIX standards say of strncpy:

If copying takes place between objects that overlap, the behavior is undefined.

Both gcc and glibc have been moving steadily in the direction of
aggressively exploiting "undefined behavior" cases for optimization
purposes. I don't know if there is yet a platform where strncpy with
src == dest behaves oddly, but we'd be foolish to imagine that it's
not going to happen eventually. If anything, Apple is probably doing
us a service by making it obvious where we're failing to adhere to spec.

However ... I still can't replicate this here, and as you say, there's
about zero chance of keeping our code clean of this problem unless we
can set up a buildfarm member that will catch it.

regards, tom lane

On 2013-10-28 14:11:12 -0400, Tom Lane wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> > On Mon, Oct 28, 2013 at 12:11 PM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> >> There have been previous discussions about fixing strcpy calls with
> >> identical source/destination (same for memcpy) but it was deemed not
> >> worth the effort. I don't really see an alternative to fixing it now.
>
> > Ugh. Why in the world would Apple break this?
>
> It's broken already; the C and POSIX standards say of strncpy:
>
> If copying takes place between objects that overlap, the behavior is undefined.
>
> Both gcc and glibc have been moving steadily in the direction of
> aggressively exploiting "undefined behavior" cases for optimization
> purposes. I don't know if there is yet a platform where strncpy with
> src == dest behaves oddly, but we'd be foolish to imagine that it's
> not going to happen eventually. If anything, Apple is probably doing
> us a service by making it obvious where we're failing to adhere to spec.
>
> However ... I still can't replicate this here, and as you say, there's
> about zero chance of keeping our code clean of this problem unless we
> can set up a buildfarm member that will catch it.

It'd be neat if we could get a buildfarm animal up that uses valgrind -
which would catch such and lots of other errors. That's where the topic
has come up in the past:
http://www.postgresql.org/message-id/20110312133224.GA7833%40tornado.gateway.2wire.net
http://www.postgresql.org/message-id/20130217142209.GA5073@awork2.anarazel.de

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Hi,

On Mon, Oct 28, 2013 at 7:11 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

>
> If copying takes place between objects that overlap, the behavior is
> undefined.
>
> Both gcc and glibc have been moving steadily in the direction of
> aggressively exploiting "undefined behavior" cases for optimization
> purposes. I don't know if there is yet a platform where strncpy with
> src == dest behaves oddly, but we'd be foolish to imagine that it's
> not going to happen eventually. If anything, Apple is probably doing
> us a service by making it obvious where we're failing to adhere to spec.
>
> However ... I still can't replicate this here, and as you say, there's
> about zero chance of keeping our code clean of this problem unless we
> can set up a buildfarm member that will catch it.
>
> regards, tom lane
>
>
I haven't a 10.9 box for double checking but there's a gcc command line
triggering the same assert for strcpy and gcc at
http://lists.gnu.org/archive/html/bug-bash/2013-07/msg00011.html.

Didier

On 10/28/2013 02:26 PM, Andres Freund wrote:
>
> It'd be neat if we could get a buildfarm animal up that uses valgrind -
> which would catch such and lots of other errors. That's where the topic
> has come up in the past:
> http://www.postgresql.org/message-id/20110312133224.GA7833%40tornado.gateway.2wire.net
> http://www.postgresql.org/message-id/20130217142209.GA5073@awork2.anarazel.de
>

How exactly is it going to do that?

Fundamentally, the buildfarm client is simply glue to run existing build
and test code, collect the results, and send them to the server. AFAICT
there are no configure or make targets for running under valgrind.

If someone provides the requisite support in the build system for this
I'll be happy to add buildfarm support for it.

cheers

andrew

Matthias Schmitt <freak002(at)mmp(dot)lu> writes:
> configure: using compiler=i686-apple-darwin11-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5666) (dot 3)

I realized that almost certainly, this compiler is leftover from a
Lion-era installation of Xcode. The build number is a little bit ahead
of the rather old Xcode on my own Lion machine, but the reference to
darwin11 seems rather conclusive.

So what is probably going on here is that the occurrence of the assert
trap is dependent on using this older compiler (and probably, older
system #include files along with it) along with the Mavericks version of
libc.dylib. I'm not sure exactly what the mechanism for that is, but
I see some differences in the predefined macros on my Lion box and on
my Mavericks box (notably __ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__)
so I'm guessing there's something about that that disables the trap.

If this is correct, then we can't really rely on the Apple build to flag
overlap violations, because it'll only happen on arguably-misconfigured
installations. So I'm thinking that Andres' idea of a buildfarm member
running with valgrind might be the most practical way to catch these.

regards, tom lane

On 2013-10-28 15:20:20 -0400, Andrew Dunstan wrote:
>
> On 10/28/2013 02:26 PM, Andres Freund wrote:
> >
> >It'd be neat if we could get a buildfarm animal up that uses valgrind -
> >which would catch such and lots of other errors. That's where the topic
> >has come up in the past:
> >http://www.postgresql.org/message-id/20110312133224.GA7833%40tornado.gateway.2wire.net
> >http://www.postgresql.org/message-id/20130217142209.GA5073@awork2.anarazel.de
> >
>
>
> How exactly is it going to do that?

The easiest method - somewhat slower than necessary - is to just run
"valgrind --suppressions=$srcdir/src/tools/valgrind.supp make check".

But the buildfarm supports running a postgres install before
installcheck, right? If we could run just that step using valgrind we'd
be very well of I think because we'd not run valgrind (slow!) if there
are plain regression failures around.

[.. looking for sources ...]
start_db in
https://github.com/PGBuildFarm/client-code/blob/master/run_build.pl is
where the server's run, right? Hm. That uses pg_ctl and not the server
itself and relies on pg_ctl -w returning when the server is
started... So it's not easy to make it use valgrind properly.

We could hack it by replacing bin/postgres with a wrapper around
valgrind that invokes postgres, but ick. Maybe we can make pg_ctl start
$PG_POSTGRES_BINARY instead of postgres if defined?

Better ideas?

> Fundamentally, the buildfarm client is simply glue to run existing build and
> test code, collect the results, and send them to the server. AFAICT there
> are no configure or make targets for running under valgrind.

> If someone provides the requisite support in the build system for this I'll
> be happy to add buildfarm support for it.

It'd be relatively easy to add support for make check (not installcheck)
wrapping postgres in valgrind via pg_regress, but I am not sure that's
the best way to go.

I think defining an additional CFLAG (USE_VALGRIND) shouldn't be a
problem?

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> It'd be relatively easy to add support for make check (not installcheck)
> wrapping postgres in valgrind via pg_regress, but I am not sure that's
> the best way to go.

> I think defining an additional CFLAG (USE_VALGRIND) shouldn't be a
> problem?

CFLAGS doesn't seem to have anything to do with this. I'd be more
inclined to add a "--wrapper=prog" argument to pg_regress and invoke
it with something like --wrapper="valgrind --trace-children=yes".

The larger problem though is what you'd do with the output. There's
enough false-positive noise from valgrind that I can't see having
the buildfarm run just fail if there are any messages. What to do
instead isn't very clear.

Anyway, to get back to the original problem, I've confirmed that
valgrind complains about the particular case at hand:

==9497== Source and destination overlap in strncpy(0xe1d5a3d, 0xe1d5a3d, 64)
==9497== at 0x4A081EF: strncpy (mc_replace_strmem.c:476)
==9497== by 0x6D2398: namestrcpy (name.c:221)
==9497== by 0x45F478: TupleDescInitEntry (tupdesc.c:507)
==9497== by 0x756FE1: internal_get_result_type (funcapi.c:557)
==9497== by 0x75727C: get_expr_result_type (funcapi.c:235)
==9497== by 0x534146: expandRecordVariable (parse_target.c:1524)
==9497== by 0x52C267: ParseFuncOrColumn (parse_func.c:1494)
==9497== by 0x5285CF: transformExprRecurse (parse_expr.c:463)
==9497== by 0x528DB1: transformExpr (parse_expr.c:117)
==9497== by 0x5350C5: transformTargetEntry (parse_target.c:94)
==9497== by 0x535AD4: transformTargetList (parse_target.c:167)
==9497== by 0x505FFF: transformStmt (analyze.c:929)

It seems to me the most reasonable fix for this is to make
TupleDescInitEntry notice that the passed "attributeName" points
at the tupdesc's name field and not call namestrcpy if so.
This would go with an API clarification stating that callers can
pass that if they want the name field to be unchanged. (Or we
could invent some other way to signal that, but note that NULL
is already in use for a different purpose.)

Another possibly-useful approach would be to hack namestrcpy itself
to handle name == str specially. However, that's legitimizing a
usage that's really a type cheat, so I don't like it as much, even
though it might fix more cases besides this one.

Any other thoughts about it?

regards, tom lane

On Mon, Oct 28, 2013 at 6:11 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Both gcc and glibc have been moving steadily in the direction of
> aggressively exploiting "undefined behavior" cases for optimization
> purposes. I don't know if there is yet a platform where strncpy with
> src == dest behaves oddly, but we'd be foolish to imagine that it's
> not going to happen eventually. If anything, Apple is probably doing
> us a service by making it obvious where we're failing to adhere to spec.

It's worth being aware of the fact that the upcoming GCC 4.9 release
is expected to ship with an "Undefined Behavior Sanitizer", as
described here:

http://gcc.gnu.org/gcc-4.9/changes.html

--
Peter Geoghegan

On 2013-10-28 16:02:36 -0400, Tom Lane wrote:
> Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> > It'd be relatively easy to add support for make check (not installcheck)
> > wrapping postgres in valgrind via pg_regress, but I am not sure that's
> > the best way to go.
>
> > I think defining an additional CFLAG (USE_VALGRIND) shouldn't be a
> > problem?
>
> CFLAGS doesn't seem to have anything to do with this. I'd be more
> inclined to add a "--wrapper=prog" argument to pg_regress and invoke
> it with something like --wrapper="valgrind --trace-children=yes".

Err. I am *obviously* not saying that it makes the backend run under
valgrind. But if we're going to have a buildfarm animal running
valgrind, it'd be useful to run to let it catch all errors it can
instead of hiding many of them via mcxt/aset.c? Right?

> The larger problem though is what you'd do with the output. There's
> enough false-positive noise from valgrind that I can't see having
> the buildfarm run just fail if there are any messages. What to do
> instead isn't very clear.

The false positives should be gone using the suppressions file we ship
these days (--suppressions=/path/to/pg/src/tools/valgrind.supp). We
might miss some more cases there, but it should be fairly easy to extend
it.

> It seems to me the most reasonable fix for this is to make
> TupleDescInitEntry notice that the passed "attributeName" points
> at the tupdesc's name field and not call namestrcpy if so.
> This would go with an API clarification stating that callers can
> pass that if they want the name field to be unchanged.

+1

> Another possibly-useful approach would be to hack namestrcpy itself
> to handle name == str specially. However, that's legitimizing a
> usage that's really a type cheat, so I don't like it as much, even
> though it might fix more cases besides this one.

-1

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 10/28/13, 4:11 PM, Peter Geoghegan wrote:
> On Mon, Oct 28, 2013 at 6:11 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Both gcc and glibc have been moving steadily in the direction of
>> aggressively exploiting "undefined behavior" cases for optimization
>> purposes. I don't know if there is yet a platform where strncpy with
>> src == dest behaves oddly, but we'd be foolish to imagine that it's
>> not going to happen eventually. If anything, Apple is probably doing
>> us a service by making it obvious where we're failing to adhere to spec.
>
> It's worth being aware of the fact that the upcoming GCC 4.9 release
> is expected to ship with an "Undefined Behavior Sanitizer", as
> described here:
>
> http://gcc.gnu.org/gcc-4.9/changes.html

Address Sanitizer, which is already in clang and gcc, does catch the
case were are talking about (as was previously discussed). But its only
reaction is crashing, and in my testing it's crashing already before it
gets to this place, so we'd have to put in some more work before it will
be useful.

Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> On 2013-10-28 16:02:36 -0400, Tom Lane wrote:
>> The larger problem though is what you'd do with the output. There's
>> enough false-positive noise from valgrind that I can't see having
>> the buildfarm run just fail if there are any messages. What to do
>> instead isn't very clear.

> The false positives should be gone using the suppressions file we ship
> these days (--suppressions=/path/to/pg/src/tools/valgrind.supp). We
> might miss some more cases there, but it should be fairly easy to extend
> it.

They're not all gone according to my testing; but there are far worse
problems:

1. The output goes to stderr which means it's mixed in with the backend's
normal log chatter.

2. valgrind causes autovacuum to dump core, at least on my box (RHEL6).
I'm prepared to believe that this is some relatively old bug that Red Hat
hasn't gotten round to including a patch for, but still it doesn't leave
me with any warm fuzzy feeling about the practicality of routine valgrind
testing.

regards, tom lane

On 2013-10-28 21:14:48 -0400, Tom Lane wrote:
> Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> > On 2013-10-28 16:02:36 -0400, Tom Lane wrote:
> >> The larger problem though is what you'd do with the output. There's
> >> enough false-positive noise from valgrind that I can't see having
> >> the buildfarm run just fail if there are any messages. What to do
> >> instead isn't very clear.
>
> > The false positives should be gone using the suppressions file we ship
> > these days (--suppressions=/path/to/pg/src/tools/valgrind.supp). We
> > might miss some more cases there, but it should be fairly easy to extend
> > it.
>
> They're not all gone according to my testing; but there are far worse
> problems:

Spurious or real bugs? Inside PG or libc?

> 1. The output goes to stderr which means it's mixed in with the backend's
> normal log chatter.

That's relatively easy to fix. We could just pass --log-file
redirecting the errors somewhere special and display them there.

What I've done so far is to tell valgrind to let child processes with
errors exit with a non-zero exitcode using the --error-exitcode
parameter and specify -q to reduce the chatter upon normal process
exit. That gives at least some correlation to the errors in the failed
tests.

> 2. valgrind causes autovacuum to dump core, at least on my box (RHEL6).
> I'm prepared to believe that this is some relatively old bug that Red Hat
> hasn't gotten round to including a patch for, but still it doesn't leave
> me with any warm fuzzy feeling about the practicality of routine valgrind
> testing.

Yea, I know which bug that is, I've pushed the valgrind guys into fixing
it... valgrind used to get confused about stack alignment in signal
handlers causing instructions that care about that (mostly xmm* register
using ones) to fail. elog() fails because we frequently pass many
parameters.
Since we fork processes from inside signal handlers, that situation
happens pretty often.

https://bugs.kde.org/show_bug.cgi?id=280114

3. valgrind gets floating point computations for
exp(larger_negative_double) wrong and returns the wrong error message:

regression=# SELECT exp(-808.3::float8);
ERROR: value out of range: overflow

exp sets errno=ERANGE and returns inf. That's not supposed to happen
according to my exp(3)...

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> On 2013-10-28 21:14:48 -0400, Tom Lane wrote:
>> They're not all gone according to my testing; but there are far worse
>> problems:

> Spurious or real bugs? Inside PG or libc?

I saw a bunch of uninitialized-value complaints in initdb, apparently from
places in BootstrapXLog that write out uninitialized pad bytes. I didn't
get far in testing the main regression tests because of the autovacuum
crash problem.

> 3. valgrind gets floating point computations for
> exp(larger_negative_double) wrong and returns the wrong error message:
> regression=# SELECT exp(-808.3::float8);
> ERROR: value out of range: overflow

Ugh ...

regards, tom lane

On Mon, Oct 28, 2013 at 09:14:48PM -0400, Tom Lane wrote:
> Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> > On 2013-10-28 16:02:36 -0400, Tom Lane wrote:
> >> The larger problem though is what you'd do with the output. There's
> >> enough false-positive noise from valgrind that I can't see having
> >> the buildfarm run just fail if there are any messages. What to do
> >> instead isn't very clear.
>
> > The false positives should be gone using the suppressions file we ship
> > these days (--suppressions=/path/to/pg/src/tools/valgrind.supp). We
> > might miss some more cases there, but it should be fairly easy to extend
> > it.
>
> They're not all gone according to my testing

True. Getting a clean Valgrind report is similar to getting the build
warning-free. Variations of compiler version, optimization level, host OS,
and CPU architecture can all affect the set of errors Valgrind reports, just
as they affect compiler warnings. Valgrind cleanliness has a lot of catching
up to do.

I never ran initdb under Valgrind, just "make installcheck", so that's novel
territory.

> 1. The output goes to stderr which means it's mixed in with the backend's
> normal log chatter.

As Andres explained, this is strictly a local configuration choice.

> 2. valgrind causes autovacuum to dump core, at least on my box (RHEL6).

Don't bother with versions older than Valgrind 3.8.1. Besides having a fix
for that bug, it runs PostgreSQL an order of magnitude faster, per the comment
in pg_config_manual.h.

nm

--
Noah Misch
EnterpriseDB http://www.enterprisedb.com

On Mon, Oct 28, 2013 at 04:02:36PM -0400, Tom Lane wrote:
> It seems to me the most reasonable fix for this is to make
> TupleDescInitEntry notice that the passed "attributeName" points
> at the tupdesc's name field and not call namestrcpy if so.

+1

--
Noah Misch
EnterpriseDB http://www.enterprisedb.com

Noah Misch <noah(at)leadboat(dot)com> writes:
> On Mon, Oct 28, 2013 at 09:14:48PM -0400, Tom Lane wrote:
>> 2. valgrind causes autovacuum to dump core, at least on my box (RHEL6).

> Don't bother with versions older than Valgrind 3.8.1.

$ rpm -qa | grep valgrind
valgrind-3.8.1-3.2.el6.x86_64

regards, tom lane

On 2013-10-28 22:20:02 -0400, Noah Misch wrote:
> > 2. valgrind causes autovacuum to dump core, at least on my box (RHEL6).
>
> Don't bother with versions older than Valgrind 3.8.1. Besides having a fix
> for that bug, it runs PostgreSQL an order of magnitude faster, per the comment
> in pg_config_manual.h.

I don't think that bugfix is in upstream's 3.8.1 given that was released
months earlier than the bugfix was committed... Might be backported in
your package though...

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Mon, Oct 28, 2013 at 10:30:10PM -0400, Tom Lane wrote:
> Noah Misch <noah(at)leadboat(dot)com> writes:
> > On Mon, Oct 28, 2013 at 09:14:48PM -0400, Tom Lane wrote:
> >> 2. valgrind causes autovacuum to dump core, at least on my box (RHEL6).
>
> > Don't bother with versions older than Valgrind 3.8.1.
>
> $ rpm -qa | grep valgrind
> valgrind-3.8.1-3.2.el6.x86_64

Apparently I only dreamt the bug was gone in Valgrind 3.8.1; my usual testing
configuration has autovacuum=off. Sorry for the noise.

--
Noah Misch
EnterpriseDB http://www.enterprisedb.com

Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> On 2013-10-28 21:14:48 -0400, Tom Lane wrote:
>> 2. valgrind causes autovacuum to dump core, at least on my box (RHEL6).

> Yea, I know which bug that is, I've pushed the valgrind guys into fixing
> it...
> https://bugs.kde.org/show_bug.cgi?id=280114

Thanks, I whined to Red Hat at
https://bugzilla.redhat.com/show_bug.cgi?id=1024162
and also updated our wiki page about Valgrind (where somebody overhastily
removed all mention of the issue ...)

regards, tom lane

On 2013-10-29 02:29:03 +0100, Andres Freund wrote:
> 3. valgrind gets floating point computations for
> exp(larger_negative_double) wrong and returns the wrong error message:
>
> regression=# SELECT exp(-808.3::float8);
> ERROR: value out of range: overflow
>
> exp sets errno=ERANGE and returns inf. That's not supposed to happen
> according to my exp(3)...

I've reported this as a valgrind bug... https://bugs.kde.org/show_bug.cgi?id=326821

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

i had the same issue. after looking into the macports version that works as
it should i figured out that setting ``CFLAGS=-pipe -Os -arch x86_64`` is
the key (where ``-Os`` decides between working or not).

tried that with apple-clang and apple-gcc42 from macports

--
View this message in context: http://postgresql.1045698.n5.nabble.com/Darwin-make-check-fails-with-child-process-exited-with-exit-code-134-tp5776081p5777592.html
Sent from the PostgreSQL - bugs mailing list archive at Nabble.com.