Lists: | pgsql-hackers-win32pgsql-patches |
---|
From: | "Cyril VELTER" <cyril(dot)velter(at)metadys(dot)com> |
---|---|
To: | <pgsql-hackers-win32(at)postgresql(dot)org> |
Subject: | Re : Win32 binaries test / pg_dump problem |
Date: | 2004-05-27 17:40:04 |
Message-ID: | 0a0401c44411$a4fd4110$dc01a8c0@lodgis.net |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-hackers-win32 pgsql-patches |
I reduced the pg_dump problem to a simple example. In fact, it seems to
be more a stdin / stdout problem
if you psql -f test.sql basetest from the windows shell to a windows or
linux database, you'll get a missing data error :
psql:test.sql:9: ERROR: missing data for column "b"
CONTEXT: COPY test, line 1: "a"
if you psql -f test.sql basetest from a linux shell to a windows or
linux database, you won't get an error
Another problem wich seems related to stdout occur while dumping a big
table (during a pg_dump, or manual COPY to STDOUT or COPY to FILE command).
At a point the backend crash :
LOG: server process (PID 3148) was terminated by signal 5
LOG: terminating any other active server processes
I haven't managed to reduce this problem to a simple case (the actual
table is 6GB).
Any idea where the problem can come from ?
BTW, I use the precompiled binaries from 23 may. Linux test are done with
7.4.1
cyril
> ----- Original Message -----
> From: Cyril VELTER
> To: pgsql-hackers-win32(at)postgresql(dot)org
> Sent: Monday, May 24, 2004 7:58 PM
> Subject: Win32 binaries test / pg_dump problem
>
>
>
> I just gave a try to the win32 binaries in my developpement
environment.
> I first tested it with a small database and everything appeared to work
OK.
>
> Great Works !!!!!!
>
> After that I tried to restore a bigger one (~16Go)
>
> restore from a 7.1.3 dump (in -d mode) : restore is VERY slow. For a
> very simple table 3 int8 columns, I get 32 records per second (Bi Xeon 2.4
> Ghz / ide 7200 disk / 2 Go RAM) while I get around 1000 per second on a
> linux box (the box is similar hardware except it is 10000 rpm scsi disk,
but
> I don't think that explain all that difference). During the import, CPU is
> idle and the disk is working very hard.
>
>
> While running pg_dump (the windows one) from a linux 7.4.1 database to
> windows, the speed is comparable to linux->linux, but I got the following
> message :
>
> ERROR : missing data for column "p32134"
> CONTEXT: COPY c2674, line ....
>
> After analyzing the dump, it seems to be caused by a ctrl-Z (1A)
> character embeded into a text column which seem to confuse the win32 psql
> ...
>
Attachment | Content-Type | Size |
---|---|---|
test.sql | application/octet-stream | 102 bytes |
From: | Andrew Dunstan <andrew(at)dunslane(dot)net> |
---|---|
To: | pgsql-hackers-win32(at)postgresql(dot)org |
Subject: | Re: Re : Win32 binaries test / pg_dump problem |
Date: | 2004-05-27 21:56:53 |
Message-ID: | 40B66425.1090102@dunslane.net |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-hackers-win32 pgsql-patches |
Cyril VELTER wrote:
> I reduced the pg_dump problem to a simple example. In fact, it seems to
> be more a stdin / stdout problem
>
> if you psql -f test.sql basetest from the windows shell to a windows or
> linux database, you'll get a missing data error :
>
> psql:test.sql:9: ERROR: missing data for column "b"
> CONTEXT: COPY test, line 1: "a"
>
>
> if you psql -f test.sql basetest from a linux shell to a windows or
> linux database, you won't get an error
>
>
>
Sounds like we should have psql open the file in binary mode on Windows.
Would that cause problems? I doubt it, but I wonder.
cheers
andrew
From: | "Cyril VELTER" <cyril(dot)velter(at)metadys(dot)com> |
---|---|
To: | "Andrew Dunstan" <andrew(at)dunslane(dot)net>, <pgsql-hackers-win32(at)postgresql(dot)org> |
Subject: | Re: Re : Win32 binaries test / pg_dump problem |
Date: | 2004-05-28 10:18:57 |
Message-ID: | 0b1201c4449d$2ff66e50$dc01a8c0@lodgis.net |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-hackers-win32 pgsql-patches |
From: "Andrew Dunstan" <andrew(at)dunslane(dot)net>
> Cyril VELTER wrote:
>
> > if you psql -f test.sql basetest from the windows shell to a windows
or
> > linux database, you'll get a missing data error :
> >
> > psql:test.sql:9: ERROR: missing data for column "b"
> > CONTEXT: COPY test, line 1: "a"
> >
> > if you psql -f test.sql basetest from a linux shell to a windows or
> > linux database, you won't get an error
>
> Sounds like we should have psql open the file in binary mode on Windows.
> Would that cause problems? I doubt it, but I wonder.
>
you might be right, I just found some information on msdn that fseek for
example is influenced by ctrl-z when the file is opened in text mode.
I'm not sure that this is the cause of the second problem (backend crash
on copy to) though.
do you known where this modification needs to be done ?
cyril
From: | "Andrew Dunstan" <andrew(at)dunslane(dot)net> |
---|---|
To: | <pgsql-hackers-win32(at)postgresql(dot)org> |
Subject: | Re: Re : Win32 binaries test / pg_dump problem |
Date: | 2004-05-28 11:04:05 |
Message-ID: | 1207.24.211.141.25.1085742245.squirrel@www.dunslane.net |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-hackers-win32 pgsql-patches |
Cyril VELTER said:
> From: "Andrew Dunstan" <andrew(at)dunslane(dot)net>
>
>> Cyril VELTER wrote:
>>
>> > if you psql -f test.sql basetest from the windows shell to a
>> > windows
> or
>> > linux database, you'll get a missing data error :
>> >
>> > psql:test.sql:9: ERROR: missing data for column "b"
>> > CONTEXT: COPY test, line 1: "a"
>> >
>> > if you psql -f test.sql basetest from a linux shell to a windows
>> > or
>> > linux database, you won't get an error
>>
>> Sounds like we should have psql open the file in binary mode on
>> Windows. Would that cause problems? I doubt it, but I wonder.
>>
>
> you might be right, I just found some information on msdn that fseek
> for
> example is influenced by ctrl-z when the file is opened in text mode.
>
> I'm not sure that this is the cause of the second problem (backend
> crash
> on copy to) though.
>
> do you known where this modification needs to be done ?
>
> cyril
>
probably in src/bin/psql/command.c::process_file()
instead of mode "r" we should probably use the predefined constant
PG_BINARY_R
cheers
andrew
From: | "Cyril VELTER" <cyril(dot)velter(at)metadys(dot)com> |
---|---|
To: | "Andrew Dunstan" <andrew(at)dunslane(dot)net>, <pgsql-hackers-win32(at)postgresql(dot)org> |
Subject: | Re: Re : Win32 binaries test / pg_dump problem |
Date: | 2004-05-28 17:09:41 |
Message-ID: | 0cd501c444d6$91598630$dc01a8c0@lodgis.net |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-hackers-win32 pgsql-patches |
----- Original Message -----
From: "Andrew Dunstan" <andrew(at)dunslane(dot)net>
> Cyril VELTER said:
> > From: "Andrew Dunstan" <andrew(at)dunslane(dot)net>
> >
> >> Cyril VELTER wrote:
> >>
> >> > if you psql -f test.sql basetest from the windows shell to a
> >> > windows
> > or
> >> > linux database, you'll get a missing data error :
> >> >
> >> > psql:test.sql:9: ERROR: missing data for column "b"
> >> > CONTEXT: COPY test, line 1: "a"
> >> >
> >> > if you psql -f test.sql basetest from a linux shell to a windows
> >> > or
> >> > linux database, you won't get an error
> >>
> >> Sounds like we should have psql open the file in binary mode on
> >> Windows. Would that cause problems? I doubt it, but I wonder.
> >>
>
> probably in src/bin/psql/command.c::process_file()
>
> instead of mode "r" we should probably use the predefined constant
> PG_BINARY_R
OK, that do the trick for that particular case. doing a more test.sql |
psql basetest lead to the same error but psql basetest < test.sql does works
so I think it is a windows shell or "more" problem (cat test.sql | psql
basetest under msys works).
Should I submit a patch, or somebody can commit this change ? (I just
replaced "r" by PG_BINARY_R in src/bin/psql/command.c::process_file().
BTW do you have any idea on the backend crash while doing a copy to
stdout ? (I reduced the case to copying a date column. Will try with a more
recent snapshot).
From: | Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> |
---|---|
To: | Andrew Dunstan <andrew(at)dunslane(dot)net> |
Cc: | pgsql-hackers-win32(at)postgresql(dot)org |
Subject: | Re: Re : Win32 binaries test / pg_dump problem |
Date: | 2004-05-28 18:34:51 |
Message-ID: | 200405281834.i4SIYpl01115@candle.pha.pa.us |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-hackers-win32 pgsql-patches |
Andrew Dunstan wrote:
> Cyril VELTER said:
> > From: "Andrew Dunstan" <andrew(at)dunslane(dot)net>
> >
> >> Cyril VELTER wrote:
> >>
> >> > if you psql -f test.sql basetest from the windows shell to a
> >> > windows
> > or
> >> > linux database, you'll get a missing data error :
> >> >
> >> > psql:test.sql:9: ERROR: missing data for column "b"
> >> > CONTEXT: COPY test, line 1: "a"
> >> >
> >> > if you psql -f test.sql basetest from a linux shell to a windows
> >> > or
> >> > linux database, you won't get an error
> >>
> >> Sounds like we should have psql open the file in binary mode on
> >> Windows. Would that cause problems? I doubt it, but I wonder.
> >>
> >
> > you might be right, I just found some information on msdn that fseek
> > for
> > example is influenced by ctrl-z when the file is opened in text mode.
> >
> > I'm not sure that this is the cause of the second problem (backend
> > crash
> > on copy to) though.
> >
> > do you known where this modification needs to be done ?
> >
> > cyril
> >
>
> probably in src/bin/psql/command.c::process_file()
>
> instead of mode "r" we should probably use the predefined constant
> PG_BINARY_R
Uh, but it isn't a binary file, it is SQL commands.
--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073
From: | Andrew Dunstan <andrew(at)dunslane(dot)net> |
---|---|
To: | pgsql-hackers-win32(at)postgresql(dot)org |
Subject: | Re: Re : Win32 binaries test / pg_dump problem |
Date: | 2004-05-28 18:52:04 |
Message-ID: | 40B78A54.50309@dunslane.net |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-hackers-win32 pgsql-patches |
Bruce Momjian wrote:
>Andrew Dunstan wrote:
>
>
>>Cyril VELTER said:
>>
>>
>>>From: "Andrew Dunstan" <andrew(at)dunslane(dot)net>
>>>
>>>
>>>
>>>>Cyril VELTER wrote:
>>>>
>>>>
>>>>
>>>>> if you psql -f test.sql basetest from the windows shell to a
>>>>> windows
>>>>>
>>>>>
>>>or
>>>
>>>
>>>>>linux database, you'll get a missing data error :
>>>>>
>>>>>psql:test.sql:9: ERROR: missing data for column "b"
>>>>>CONTEXT: COPY test, line 1: "a"
>>>>>
>>>>> if you psql -f test.sql basetest from a linux shell to a windows
>>>>> or
>>>>>linux database, you won't get an error
>>>>>
>>>>>
>>>>Sounds like we should have psql open the file in binary mode on
>>>>Windows. Would that cause problems? I doubt it, but I wonder.
>>>>
>>>>
>>>>
>>> you might be right, I just found some information on msdn that fseek
>>> for
>>>example is influenced by ctrl-z when the file is opened in text mode.
>>>
>>> I'm not sure that this is the cause of the second problem (backend
>>> crash
>>>on copy to) though.
>>>
>>> do you known where this modification needs to be done ?
>>>
>>> cyril
>>>
>>>
>>>
>>probably in src/bin/psql/command.c::process_file()
>>
>>instead of mode "r" we should probably use the predefined constant
>>PG_BINARY_R
>>
>>
>
>Uh, but it isn't a binary file, it is SQL commands.
>
>
>
If it can have an embedded ^Z it is not a legal Windows text file.
Adding the binary flag will have exactly 2 effects:
1. It will inhibit the behaviour of the driver in translating CRLF to
plain LF
2. It will not see ^Z as EOF.
We don't care about the first - we handle CRLF just fine. But we do care
about the second, and we don't want the library to interpret ^Z as EOF.
cheers
andrew
From: | Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> |
---|---|
To: | Andrew Dunstan <andrew(at)dunslane(dot)net> |
Cc: | pgsql-hackers-win32(at)postgresql(dot)org |
Subject: | Re: Re : Win32 binaries test / pg_dump problem |
Date: | 2004-06-10 02:42:43 |
Message-ID: | 200406100242.i5A2ghq27892@candle.pha.pa.us |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-hackers-win32 pgsql-patches |
Would someone verify that having psql open the file in binary mode fixes
the problem?
---------------------------------------------------------------------------
Andrew Dunstan wrote:
> Bruce Momjian wrote:
>
> >Andrew Dunstan wrote:
> >
> >
> >>Cyril VELTER said:
> >>
> >>
> >>>From: "Andrew Dunstan" <andrew(at)dunslane(dot)net>
> >>>
> >>>
> >>>
> >>>>Cyril VELTER wrote:
> >>>>
> >>>>
> >>>>
> >>>>> if you psql -f test.sql basetest from the windows shell to a
> >>>>> windows
> >>>>>
> >>>>>
> >>>or
> >>>
> >>>
> >>>>>linux database, you'll get a missing data error :
> >>>>>
> >>>>>psql:test.sql:9: ERROR: missing data for column "b"
> >>>>>CONTEXT: COPY test, line 1: "a"
> >>>>>
> >>>>> if you psql -f test.sql basetest from a linux shell to a windows
> >>>>> or
> >>>>>linux database, you won't get an error
> >>>>>
> >>>>>
> >>>>Sounds like we should have psql open the file in binary mode on
> >>>>Windows. Would that cause problems? I doubt it, but I wonder.
> >>>>
> >>>>
> >>>>
> >>> you might be right, I just found some information on msdn that fseek
> >>> for
> >>>example is influenced by ctrl-z when the file is opened in text mode.
> >>>
> >>> I'm not sure that this is the cause of the second problem (backend
> >>> crash
> >>>on copy to) though.
> >>>
> >>> do you known where this modification needs to be done ?
> >>>
> >>> cyril
> >>>
> >>>
> >>>
> >>probably in src/bin/psql/command.c::process_file()
> >>
> >>instead of mode "r" we should probably use the predefined constant
> >>PG_BINARY_R
> >>
> >>
> >
> >Uh, but it isn't a binary file, it is SQL commands.
> >
> >
> >
>
> If it can have an embedded ^Z it is not a legal Windows text file.
>
> Adding the binary flag will have exactly 2 effects:
> 1. It will inhibit the behaviour of the driver in translating CRLF to
> plain LF
> 2. It will not see ^Z as EOF.
>
> We don't care about the first - we handle CRLF just fine. But we do care
> about the second, and we don't want the library to interpret ^Z as EOF.
>
> cheers
>
> andrew
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo(at)postgresql(dot)org so that your
> message can get through to the mailing list cleanly
>
--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073
From: | Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> |
---|---|
To: | Andrew Dunstan <andrew(at)dunslane(dot)net> |
Cc: | pgsql-hackers-win32(at)postgresql(dot)org, PostgreSQL-patches <pgsql-patches(at)postgresql(dot)org> |
Subject: | Re: Re : Win32 binaries test / pg_dump problem |
Date: | 2004-07-11 00:58:00 |
Message-ID: | 200407110058.i6B0w0v25179@candle.pha.pa.us |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-hackers-win32 pgsql-patches |
OK, the attached applied patch opens text files in binary mode in psql.
As you said, it already handles CRLF OK, and we need this to read
control-Z in copy files.
I assume we can write control-Z in text mode just fine, right?
---------------------------------------------------------------------------
Andrew Dunstan wrote:
> Bruce Momjian wrote:
>
> >Andrew Dunstan wrote:
> >
> >
> >>Cyril VELTER said:
> >>
> >>
> >>>From: "Andrew Dunstan" <andrew(at)dunslane(dot)net>
> >>>
> >>>
> >>>
> >>>>Cyril VELTER wrote:
> >>>>
> >>>>
> >>>>
> >>>>> if you psql -f test.sql basetest from the windows shell to a
> >>>>> windows
> >>>>>
> >>>>>
> >>>or
> >>>
> >>>
> >>>>>linux database, you'll get a missing data error :
> >>>>>
> >>>>>psql:test.sql:9: ERROR: missing data for column "b"
> >>>>>CONTEXT: COPY test, line 1: "a"
> >>>>>
> >>>>> if you psql -f test.sql basetest from a linux shell to a windows
> >>>>> or
> >>>>>linux database, you won't get an error
> >>>>>
> >>>>>
> >>>>Sounds like we should have psql open the file in binary mode on
> >>>>Windows. Would that cause problems? I doubt it, but I wonder.
> >>>>
> >>>>
> >>>>
> >>> you might be right, I just found some information on msdn that fseek
> >>> for
> >>>example is influenced by ctrl-z when the file is opened in text mode.
> >>>
> >>> I'm not sure that this is the cause of the second problem (backend
> >>> crash
> >>>on copy to) though.
> >>>
> >>> do you known where this modification needs to be done ?
> >>>
> >>> cyril
> >>>
> >>>
> >>>
> >>probably in src/bin/psql/command.c::process_file()
> >>
> >>instead of mode "r" we should probably use the predefined constant
> >>PG_BINARY_R
> >>
> >>
> >
> >Uh, but it isn't a binary file, it is SQL commands.
> >
> >
> >
>
> If it can have an embedded ^Z it is not a legal Windows text file.
>
> Adding the binary flag will have exactly 2 effects:
> 1. It will inhibit the behaviour of the driver in translating CRLF to
> plain LF
> 2. It will not see ^Z as EOF.
>
> We don't care about the first - we handle CRLF just fine. But we do care
> about the second, and we don't want the library to interpret ^Z as EOF.
>
> cheers
>
> andrew
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo(at)postgresql(dot)org so that your
> message can get through to the mailing list cleanly
>
--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073
Attachment | Content-Type | Size |
---|---|---|
unknown_filename | text/plain | 3.0 KB |
From: | Peter Eisentraut <peter_e(at)gmx(dot)net> |
---|---|
To: | Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us>, Andrew Dunstan <andrew(at)dunslane(dot)net> |
Cc: | pgsql-hackers-win32(at)postgresql(dot)org, PostgreSQL-patches <pgsql-patches(at)postgresql(dot)org> |
Subject: | Re: [PATCHES] Re : Win32 binaries test / pg_dump problem |
Date: | 2004-07-11 08:10:34 |
Message-ID: | 200407111010.34146.peter_e@gmx.net |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-hackers-win32 pgsql-patches |
Bruce Momjian wrote:
> + #ifndef WIN32
> + #define R_TEXTFILE "r"
> + #else
> + #define R_TEXTFILE "rb"
> + #endif
This appears to be redundant with
#if defined(__CYGWIN__) || defined(WIN32)
#define PG_BINARY O_BINARY
#define PG_BINARY_R "rb"
#define PG_BINARY_W "wb"
#else
#define PG_BINARY 0
#define PG_BINARY_R "r"
#define PG_BINARY_W "w"
#endif
in c.h.
From: | Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> |
---|---|
To: | Peter Eisentraut <peter_e(at)gmx(dot)net> |
Cc: | Andrew Dunstan <andrew(at)dunslane(dot)net>, pgsql-hackers-win32(at)postgresql(dot)org, PostgreSQL-patches <pgsql-patches(at)postgresql(dot)org> |
Subject: | Re: [PATCHES] Re : Win32 binaries test / pg_dump |
Date: | 2004-07-11 13:19:54 |
Message-ID: | 200407111319.i6BDJs615002@candle.pha.pa.us |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-hackers-win32 pgsql-patches |
OK, thanks, adjusted, and I added a comment about control-z handling.
---------------------------------------------------------------------------
Peter Eisentraut wrote:
> Bruce Momjian wrote:
> > + #ifndef WIN32
> > + #define R_TEXTFILE "r"
> > + #else
> > + #define R_TEXTFILE "rb"
> > + #endif
>
> This appears to be redundant with
>
> #if defined(__CYGWIN__) || defined(WIN32)
> #define PG_BINARY O_BINARY
> #define PG_BINARY_R "rb"
> #define PG_BINARY_W "wb"
> #else
> #define PG_BINARY 0
> #define PG_BINARY_R "r"
> #define PG_BINARY_W "w"
> #endif
>
> in c.h.
>
--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073
From: | Andrew Dunstan <andrew(at)dunslane(dot)net> |
---|---|
To: | Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> |
Cc: | pgsql-hackers-win32(at)postgresql(dot)org, PostgreSQL-patches <pgsql-patches(at)postgresql(dot)org> |
Subject: | Re: Re : Win32 binaries test / pg_dump problem |
Date: | 2004-07-18 00:03:24 |
Message-ID: | 40F9BE4C.2020809@dunslane.net |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-hackers-win32 pgsql-patches |
Bruce Momjian wrote:
>OK, the attached applied patch opens text files in binary mode in psql.
>As you said, it already handles CRLF OK, and we need this to read
>control-Z in copy files.
>
>I assume we can write control-Z in text mode just fine, right?
>
>
>
I honestly don't know. Unless I see a knowledgeable response I will do
some experimentation and see.
cheers
andrew