Re: [DOCS] pg_total_relation_size() and CHECKPOINT

We are not alone ;-)

http://sourceforge.net/project/shownotes.php?group_id=129038&release_id=557649

Version 1.02
2007-01-25
* Fix the stat function (on Windows) to use GetFileAttributesEx insead of FindFirstFile

-----Original Message-----
From: Zubkovsky, Sergey
Sent: Friday, March 28, 2008 6:43 PM
To: Andrew Dunstan
Cc: Tom Lane; Alvaro Herrera; Gregory Stark; pgsql-hackers(at)postgresql(dot)org; Magnus Hagander
Subject: RE: [HACKERS] [DOCS] pg_total_relation_size() and CHECKPOINT

It seems I've found the cause and the workaround of the problem.
MSVC's stat() is implemented by using FindNextFile().
MSDN contains the following suspicious paragraph аbout FindNextFile():

"In rare cases, file attribute information on NTFS file systems may not be current at the time you call this function. To obtain the current NTFS file system file attributes, call GetFileInformationByHandle."

Since we generally cannot open an examined file, we need another way.

In the prepared custom build of PG 8.3.1 the native MSVC's stat() was rewrote by adding GetFileAttributesEx() to correct stat's st_size value.
I had seen that a result of MSVC's stat() and a result of GetFileAttributesEx() may be differ by the file size values at least.

The most important thing is the test in the original post
( http://archives.postgresql.org/pgsql-docs/2008-03/msg00041.php )
doesn't reproduce any inconsistence now.
All work fine.

This was tested on my WinXP SP2 platform but I suppose it will work on any NT-based OS.

Thanks,
Sergey Zubkovsky

-----Original Message-----
From: Andrew Dunstan [mailto:andrew(at)dunslane(dot)net]
Sent: Thursday, March 27, 2008 3:54 PM
To: Zubkovsky, Sergey
Cc: Tom Lane; Alvaro Herrera; Gregory Stark; pgsql-hackers(at)postgresql(dot)org; Magnus Hagander
Subject: Re: [HACKERS] [DOCS] pg_total_relation_size() and CHECKPOINT

Zubkovsky, Sergey wrote:
> Maybe this helps:
>
> "It is not an error to set a file pointer to a position beyond the end
> of the file. The size of the file does not increase until you call the
> SetEndOfFile, WriteFile, or WriteFileEx function. A write operation
> increases the size of the file to the file pointer position plus the
> size of the buffer written, which results in the intervening bytes
> uninitialized."
>
> http://msdn2.microsoft.com/en-us/library/aa365541(VS.85).aspx
>
> According to Windows' lseek implementation (attached) SetEndOfFile()
> isn't called for this case.
>
>
>

Yes, but we immediately follow the lseek bye a write(). See
src/backend/storage/smgr/md.c:mdextend() .

cheers

andrew

Zubkovsky, Sergey wrote:
>
> In the prepared custom build of PG 8.3.1 the native MSVC's stat() was rewrote by adding GetFileAttributesEx() to correct stat's st_size value.
> I had seen that a result of MSVC's stat() and a result of GetFileAttributesEx() may be differ by the file size values at least.
>
> The most important thing is the test in the original post
> ( http://archives.postgresql.org/pgsql-docs/2008-03/msg00041.php )
> doesn't reproduce any inconsistence now.
> All work fine.
>
> This was tested on my WinXP SP2 platform but I suppose it will work on any NT-based OS.
>
>
>

If you have a patch, please send it to the -patches list.

cheers

andrew

Hi,

Here is the patch.
Check it, please.

Locations of the added files:
src\include\port\win32_msvc\sys\stat.inl
src\port\stat_pg_fixed.c

Thank you.

-----Original Message-----
From: Andrew Dunstan [mailto:andrew(at)dunslane(dot)net]
Sent: Saturday, March 29, 2008 4:18 PM
To: Zubkovsky, Sergey
Cc: Tom Lane; Alvaro Herrera; Gregory Stark;
pgsql-hackers(at)postgresql(dot)org; Magnus Hagander
Subject: Re: [HACKERS] [DOCS] pg_total_relation_size() and CHECKPOINT

Zubkovsky, Sergey wrote:
>
> In the prepared custom build of PG 8.3.1 the native MSVC's stat() was
rewrote by adding GetFileAttributesEx() to correct stat's st_size value.
> I had seen that a result of MSVC's stat() and a result of
GetFileAttributesEx() may be differ by the file size values at least.
>
> The most important thing is the test in the original post
> ( http://archives.postgresql.org/pgsql-docs/2008-03/msg00041.php )
> doesn't reproduce any inconsistence now.
> All work fine.
>
> This was tested on my WinXP SP2 platform but I suppose it will work on
any NT-based OS.
>
>
>

If you have a patch, please send it to the -patches list.

cheers

andrew

I already applied a different patch for this quite some time ago. So
what's in HEAD (and current in 8.3 and 8.2) should be fixed already.

//Magnus

Zubkovsky, Sergey wrote:
> Hi,
>
> Here is the patch.
> Check it, please.
>
> Locations of the added files:
> src\include\port\win32_msvc\sys\stat.inl
> src\port\stat_pg_fixed.c
>
>
> Thank you.
>
>
> -----Original Message-----
> From: Andrew Dunstan [mailto:andrew(at)dunslane(dot)net]
> Sent: Saturday, March 29, 2008 4:18 PM
> To: Zubkovsky, Sergey
> Cc: Tom Lane; Alvaro Herrera; Gregory Stark;
> pgsql-hackers(at)postgresql(dot)org; Magnus Hagander
> Subject: Re: [HACKERS] [DOCS] pg_total_relation_size() and CHECKPOINT
>
>
>
>
> Zubkovsky, Sergey wrote:
>> In the prepared custom build of PG 8.3.1 the native MSVC's stat() was
> rewrote by adding GetFileAttributesEx() to correct stat's st_size value.
>> I had seen that a result of MSVC's stat() and a result of
> GetFileAttributesEx() may be differ by the file size values at least.
>> The most important thing is the test in the original post
>> ( http://archives.postgresql.org/pgsql-docs/2008-03/msg00041.php )
>> doesn't reproduce any inconsistence now.
>> All work fine.
>>
>> This was tested on my WinXP SP2 platform but I suppose it will work on
> any NT-based OS.
>>
>>
>
>
> If you have a patch, please send it to the -patches list.
>
> cheers
>
> andrew

Magnus,

Please, take a look at my implementation of stat().
It has at least two advantages: it's faster and doesn't have a bug with
local-to-UTC time conversion that native msvc's stat() has.
Maybe it will be useful.

Thank you.

-----Original Message-----
From: Magnus Hagander [mailto:magnus(at)hagander(dot)net]
Sent: Tuesday, April 29, 2008 4:54 PM
To: Zubkovsky, Sergey
Cc: Andrew Dunstan; Tom Lane; Alvaro Herrera; Gregory Stark;
pgsql-patches(at)postgresql(dot)org
Subject: Re: [HACKERS] [DOCS] pg_total_relation_size() and CHECKPOINT

I already applied a different patch for this quite some time ago. So
what's in HEAD (and current in 8.3 and 8.2) should be fixed already.

//Magnus

Zubkovsky, Sergey wrote:
> Hi,
>
> Here is the patch.
> Check it, please.
>
> Locations of the added files:
> src\include\port\win32_msvc\sys\stat.inl
> src\port\stat_pg_fixed.c
>
>
> Thank you.
>
>
> -----Original Message-----
> From: Andrew Dunstan [mailto:andrew(at)dunslane(dot)net]
> Sent: Saturday, March 29, 2008 4:18 PM
> To: Zubkovsky, Sergey
> Cc: Tom Lane; Alvaro Herrera; Gregory Stark;
> pgsql-hackers(at)postgresql(dot)org; Magnus Hagander
> Subject: Re: [HACKERS] [DOCS] pg_total_relation_size() and CHECKPOINT
>
>
>
>
> Zubkovsky, Sergey wrote:
>> In the prepared custom build of PG 8.3.1 the native MSVC's stat() was
> rewrote by adding GetFileAttributesEx() to correct stat's st_size
value.
>> I had seen that a result of MSVC's stat() and a result of
> GetFileAttributesEx() may be differ by the file size values at least.
>> The most important thing is the test in the original post
>> ( http://archives.postgresql.org/pgsql-docs/2008-03/msg00041.php )
>> doesn't reproduce any inconsistence now.
>> All work fine.
>>
>> This was tested on my WinXP SP2 platform but I suppose it will work
on
> any NT-based OS.
>>
>>
>
>
> If you have a patch, please send it to the -patches list.
>
> cheers
>
> andrew