Re: pg_restore [archiver] file offset in dump file is too

> > Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> >> Hmm. Windows reports an offset size of 4 bytes on a dump I
> just made
> >> ... is that relevant? What governs it?
> >
> > [ looks again... ] Hm, that is a 40Gb dump Kevin is
> complaining of,
> > isn't it. Do our Windows builds have LARGEFILE support?
> >
>
> I think from a cursory investigation the short answer is no,
> but they probably could. If so, that should definitely go on
> the TODO list. Windows gurus, any thoughts?

Windows certainly supports large files. I don't see why we wouldn't pick
this up in autoconf, perhaps the mingw libraries don't?
Definitly worth investigating, no time for 8.1, so putting it on TODO
seems very good :-)

//Magnus

Magnus Hagander wrote:

>>>Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
>>>
>>>
>>>>Hmm. Windows reports an offset size of 4 bytes on a dump I
>>>>
>>>>
>>just made
>>
>>
>>>>... is that relevant? What governs it?
>>>>
>>>>
>>>[ looks again... ] Hm, that is a 40Gb dump Kevin is
>>>
>>>
>>complaining of,
>>
>>
>>>isn't it. Do our Windows builds have LARGEFILE support?
>>>
>>>
>>>
>>I think from a cursory investigation the short answer is no,
>>but they probably could. If so, that should definitely go on
>>the TODO list. Windows gurus, any thoughts?
>>
>>
>
>Windows certainly supports large files. I don't see why we wouldn't pick
>this up in autoconf, perhaps the mingw libraries don't?
>Definitly worth investigating, no time for 8.1, so putting it on TODO
>seems very good :-)
>
>

There is no fseeko in the Windows libraries, nor any provision in the
mingw headers that I can see for a 64 bit off_t. So we would need to
roll our own to some extent - I think we need more than just a bit of
configure cleverness.

However, there is a Windows library routine to do a 64bit seek and
return the file position, so we could fairly easily implement fseeko and
ftello based on that. See
http://msdn.microsoft.com/library/en-us/vclib/html/_crt__lseek.2c_._lseeki64.asp

cheers

andrew

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> There is no fseeko in the Windows libraries, nor any provision in the
> mingw headers that I can see for a 64 bit off_t. So we would need to
> roll our own to some extent - I think we need more than just a bit of
> configure cleverness.

> However, there is a Windows library routine to do a 64bit seek and
> return the file position, so we could fairly easily implement fseeko and
> ftello based on that. See
> http://msdn.microsoft.com/library/en-us/vclib/html/_crt__lseek.2c_._lseeki64.asp

Is there any risk that the mingw libraries would fail when manipulating
a file whose current offset exceeds 32 bits? I'm wondering if we'd have
to roll our own stdio in toto, not just fseeko/ftello :-(

regards, tom lane

Tom Lane wrote:

>Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
>
>
>>There is no fseeko in the Windows libraries, nor any provision in the
>>mingw headers that I can see for a 64 bit off_t. So we would need to
>>roll our own to some extent - I think we need more than just a bit of
>>configure cleverness.
>>
>>
>
>
>
>>However, there is a Windows library routine to do a 64bit seek and
>>return the file position, so we could fairly easily implement fseeko and
>>ftello based on that. See
>>http://msdn.microsoft.com/library/en-us/vclib/html/_crt__lseek.2c_._lseeki64.asp
>>
>>
>
>Is there any risk that the mingw libraries would fail when manipulating
>a file whose current offset exceeds 32 bits? I'm wondering if we'd have
>to roll our own stdio in toto, not just fseeko/ftello :-(
>
>
>
>

AFAIK all this is coming straight from the base Windows libraries. I
guess we'll have to test it when we get around to it.

cheers

andrew

Andrew Dunstan wrote:
> There is no fseeko in the Windows libraries, nor any provision in the
> mingw headers that I can see for a 64 bit off_t. So we would need to
> roll our own to some extent - I think we need more than just a bit of
> configure cleverness.
>
> However, there is a Windows library routine to do a 64bit seek and
> return the file position, so we could fairly easily implement fseeko and
> ftello based on that. See
> http://msdn.microsoft.com/library/en-us/vclib/html/_crt__lseek.2c_._lseeki64.asp

See src/port/fseeko.c for a version built on fsetpos().

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:

>Andrew Dunstan wrote:
>
>
>>There is no fseeko in the Windows libraries, nor any provision in the
>>mingw headers that I can see for a 64 bit off_t. So we would need to
>>roll our own to some extent - I think we need more than just a bit of
>>configure cleverness.
>>
>>However, there is a Windows library routine to do a 64bit seek and
>>return the file position, so we could fairly easily implement fseeko and
>>ftello based on that. See
>>http://msdn.microsoft.com/library/en-us/vclib/html/_crt__lseek.2c_._lseeki64.asp
>>
>>
>
>See src/port/fseeko.c for a version built on fsetpos().
>
>
>

Yeah, I'm not made warm and fuzzy by the comments about fpos_t in
mingw's stdio.h, though:

/*
* An opaque data type used for storing file positions... The contents of
* this type are unknown, but we (the compiler) need to know the size
* because the programmer using fgetpos and fsetpos will be setting aside
* storage for fpos_t structres. Actually I tested using a byte array and
* it is fairly evident that the fpos_t type is a long (in CRTDLL.DLL).
* Perhaps an unsigned long? TODO? It's definitely a 64-bit number in
* MSVCRT however, and for now `long long' will do.
*/

But the example program on MSDN contains "pos = 14", which leads one to
assume that it really is some simple int underneath.

cheers

andrew

cheers

andrew

Added to TODO:

o Add long file support for binary pg_dump output

While Win32 supports 64-bit files, the MinGW API does not,
meaning we have to build an fseeko replacement on top of the
Win32 API, and we have to make sure MinGW handles it.

---------------------------------------------------------------------------

Andrew Dunstan wrote:
>
>
> Bruce Momjian wrote:
>
> >Andrew Dunstan wrote:
> >
> >
> >>There is no fseeko in the Windows libraries, nor any provision in the
> >>mingw headers that I can see for a 64 bit off_t. So we would need to
> >>roll our own to some extent - I think we need more than just a bit of
> >>configure cleverness.
> >>
> >>However, there is a Windows library routine to do a 64bit seek and
> >>return the file position, so we could fairly easily implement fseeko and
> >>ftello based on that. See
> >>http://msdn.microsoft.com/library/en-us/vclib/html/_crt__lseek.2c_._lseeki64.asp
> >>
> >>
> >
> >See src/port/fseeko.c for a version built on fsetpos().
> >
> >
> >
>
> Yeah, I'm not made warm and fuzzy by the comments about fpos_t in
> mingw's stdio.h, though:
>
> /*
> * An opaque data type used for storing file positions... The contents of
> * this type are unknown, but we (the compiler) need to know the size
> * because the programmer using fgetpos and fsetpos will be setting aside
> * storage for fpos_t structres. Actually I tested using a byte array and
> * it is fairly evident that the fpos_t type is a long (in CRTDLL.DLL).
> * Perhaps an unsigned long? TODO? It's definitely a 64-bit number in
> * MSVCRT however, and for now `long long' will do.
> */
>
>
> But the example program on MSDN contains "pos = 14", which leads one to
> assume that it really is some simple int underneath.
>
> cheers
>
> andrew
>
> cheers
>
> andrew
>

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> Added to TODO:

> o Add long file support for binary pg_dump output

> While Win32 supports 64-bit files, the MinGW API does not,
> meaning we have to build an fseeko replacement on top of the
> Win32 API, and we have to make sure MinGW handles it.

Wouldn't it be better to lobby the MinGW folk to fix their problem?
Or even help them with it? I can't see the rationale for implementing
a workaround that helps only us.

regards, tom lane

Tom Lane wrote:

>> While Win32 supports 64-bit files, the MinGW API does not,
>> meaning we have to build an fseeko replacement on top of the
>> Win32 API, and we have to make sure MinGW handles it.
>>
>>
>
>Wouldn't it be better to lobby the MinGW folk to fix their problem?
>Or even help them with it? I can't see the rationale for implementing
>a workaround that helps only us.
>
>
>
>

There is a library available from the gnuwin32 project that advertises
fseeko and fseeko64. So we probably have a choice of requiring this
library or doing it ourselves.

see http://gnuwin32.sourceforge.net/packages/libgw32c.htm

cheers

andrew

Andrew Dunstan wrote:
>
>
> Tom Lane wrote:
>
> >> While Win32 supports 64-bit files, the MinGW API does not,
> >> meaning we have to build an fseeko replacement on top of the
> >> Win32 API, and we have to make sure MinGW handles it.
> >>
> >>
> >
> >Wouldn't it be better to lobby the MinGW folk to fix their problem?
> >Or even help them with it? I can't see the rationale for implementing
> >a workaround that helps only us.
> >
> >
> >
> >
>
> There is a library available from the gnuwin32 project that advertises
> fseeko and fseeko64. So we probably have a choice of requiring this
> library or doing it ourselves.
>
> see http://gnuwin32.sourceforge.net/packages/libgw32c.htm

OK, updated:

o Add long file support for binary pg_dump output

While Win32 supports 64-bit files, the MinGW API does not,
meaning we have to build an fseeko replacement on top of the
Win32 API, and we have to make sure MinGW handles it. Another
option is to wait for the MinGW project to fix it, or use the
code from the LibGW32C project as a guide.

My bet is that the license for LibGW32C might require us to use it as a
guide rather than the code itself.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:

>Andrew Dunstan wrote:
>
>
>>Tom Lane wrote:
>>
>>
>>
>>>> While Win32 supports 64-bit files, the MinGW API does not,
>>>> meaning we have to build an fseeko replacement on top of the
>>>> Win32 API, and we have to make sure MinGW handles it.
>>>>
>>>>
>>>>
>>>>
>>>Wouldn't it be better to lobby the MinGW folk to fix their problem?
>>>Or even help them with it? I can't see the rationale for implementing
>>>a workaround that helps only us.
>>>
>>>
>>>
>>>
>>>
>>>
>>There is a library available from the gnuwin32 project that advertises
>>fseeko and fseeko64. So we probably have a choice of requiring this
>>library or doing it ourselves.
>>
>>see http://gnuwin32.sourceforge.net/packages/libgw32c.htm
>>
>>
>
>OK, updated:
>
> o Add long file support for binary pg_dump output
>
> While Win32 supports 64-bit files, the MinGW API does not,
> meaning we have to build an fseeko replacement on top of the
> Win32 API, and we have to make sure MinGW handles it. Another
> option is to wait for the MinGW project to fix it, or use the
> code from the LibGW32C project as a guide.
>
>My bet is that the license for LibGW32C might require us to use it as a
>guide rather than the code itself.
>
>
>

[after examining source]

It's LGPL. But basically it just uses fsetpos() just like ours does. So
maybe a little adjustment to our config so we can use ours in a 64 bit
manner would be the way to go.

cheers

andrew