Re: Ragged CSV import

From: "Kevin Grittner" <Kevin(dot)Grittner(at)wicourts(dot)gov>
To: "Josh Berkus" <josh(at)agliodbs(dot)com>, "Dann Corbit" <DCorbit(at)connx(dot)com>, "Andrew Dunstan" <andrew(at)dunslane(dot)net>
Cc: "PostgreSQL-development" <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Ragged CSV import
Date: 2009-09-09 21:03:26
Message-ID: 4AA7D1CE020000250002ABF4@gw.wicourts.gov
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

"Dann Corbit" <DCorbit(at)connx(dot)com> wrote:
> Kevin Grittner

>> It's been a while since I've had a need for something like this,
>> but of the copy features not currently available in PostgreSQL,
>> the two most useful are to read in only some of the defined
>> columns, and to output to a separate disk file any rows which
>> failed to match the expected format.
>> The latter would not cause the copy to fail unless the count of
>> such rows exceeded a user-specified threshold.

> Perhaps something like SQL Server's BCP format files could be
> used.
> http://support.microsoft.com/kb/67409
> http://technet.microsoft.com/en-us/library/ms178129.aspx
> http://www.nigelrivett.net/SQLTsql/BCP_quoted_CSV_Format_file.html
> http://doc.ddart.net/mssql/sql70/impt_bcp_16.htm

That is what I was thinking of when I wrote the above, although in
the Sybase incarnation, before Microsoft split off on their own. (I
see they haven't changed sybchar from the Sybase name yet....)

My reservation about referencing it is that it includes so many
bells and whistles that it's not as easy to use as it might be, even
with the "wizard" to generate the format description files. The
other problem with it is that it was far and away the *least* stable
part of the DBMS. You could count on it being broken in any version
until six months to a year into that version's life. We eventually
moved away from it because of the instability -- we could write code
from scratch each time with more confidence of correct behavior. I
think keeping it a little simpler might help with keeping it stable.

-Kevin

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Andrew Dunstan 2009-09-09 21:13:43 Re: Ragged CSV import
Previous Message Sam Mason 2009-09-09 21:00:03 Re: COALESCE and NULLIF semantics