Lists: | pgsql-hackers |
---|
From: | flyusa2010 fly <flyusa2010(at)gmail(dot)com> |
---|---|
To: | pgsql-hackers(at)postgresql(dot)org |
Subject: | Can postgres create a file with physically continuous blocks. |
Date: | 2010-12-19 07:10:56 |
Message-ID: | AANLkTinrYrVKN=RPG27m4wbcZfO02fiiUVb_HUzmJKWh@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-hackers |
folks,
Does postgres make an effort to create a file with physically continuous
blocks?
Thanks!
From: | Jim Nasby <jim(at)nasby(dot)net> |
---|---|
To: | flyusa2010 fly <flyusa2010(at)gmail(dot)com> |
Cc: | pgsql-hackers(at)postgresql(dot)org |
Subject: | Re: Can postgres create a file with physically continuous blocks. |
Date: | 2010-12-19 18:10:35 |
Message-ID: | 40FA905C-03B1-4341-A781-7F0250F8308D@nasby.net |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-hackers |
On Dec 19, 2010, at 1:10 AM, flyusa2010 fly wrote:
> Does postgres make an effort to create a file with physically continuous blocks?
AFAIK all files are expanded as needed. I don't think there's any flags you can pass to the filesystem to tell it "this file will eventually be 1GB in size". So, we're basically at the mercy of the FS to try and keep things contiguous.
BTW, it's worse with btree indexes; they end up non-sequential inside the file anyway, so even if the file is contiguous index access may not be.
--
Jim C. Nasby, Database Architect jim(at)nasby(dot)net
512.569.9461 (cell) http://jim.nasby.net
From: | Robert Haas <robertmhaas(at)gmail(dot)com> |
---|---|
To: | Jim Nasby <jim(at)nasby(dot)net> |
Cc: | flyusa2010 fly <flyusa2010(at)gmail(dot)com>, pgsql-hackers(at)postgresql(dot)org |
Subject: | Re: Can postgres create a file with physically continuous blocks. |
Date: | 2010-12-21 11:49:21 |
Message-ID: | AANLkTimecpfDdQEL86aBi7FnDEQ+wCC89yjXKjcJTRf6@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-hackers |
On Sun, Dec 19, 2010 at 1:10 PM, Jim Nasby <jim(at)nasby(dot)net> wrote:
> On Dec 19, 2010, at 1:10 AM, flyusa2010 fly wrote:
>> Does postgres make an effort to create a file with physically continuous blocks?
>
> AFAIK all files are expanded as needed. I don't think there's any flags you can pass to the filesystem to tell it "this file will eventually be 1GB in size". So, we're basically at the mercy of the FS to try and keep things contiguous.
There have been some reports that we would do better on some
filesystems if we extended the file more than a block at a time, as we
do today. However, AFAIK, no one is pursuing this ATM.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
From: | Rob Wultsch <wultsch(at)gmail(dot)com> |
---|---|
To: | Robert Haas <robertmhaas(at)gmail(dot)com> |
Cc: | Jim Nasby <jim(at)nasby(dot)net>, flyusa2010 fly <flyusa2010(at)gmail(dot)com>, pgsql-hackers(at)postgresql(dot)org |
Subject: | Re: Can postgres create a file with physically continuous blocks. |
Date: | 2010-12-22 01:45:55 |
Message-ID: | AANLkTim=-GLts=oAMD+paZJ_FW0wCQgs_HHFTSv1zggX@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-hackers |
On Tue, Dec 21, 2010 at 4:49 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Sun, Dec 19, 2010 at 1:10 PM, Jim Nasby <jim(at)nasby(dot)net> wrote:
>> On Dec 19, 2010, at 1:10 AM, flyusa2010 fly wrote:
>>> Does postgres make an effort to create a file with physically continuous blocks?
>>
>> AFAIK all files are expanded as needed. I don't think there's any flags you can pass to the filesystem to tell it "this file will eventually be 1GB in size". So, we're basically at the mercy of the FS to try and keep things contiguous.
>
> There have been some reports that we would do better on some
> filesystems if we extended the file more than a block at a time, as we
> do today. However, AFAIK, no one is pursuing this ATM.
>
The has been found to be the case in the MySQL world, particularly
when ext3 is in use:
http://forge.mysql.com/worklog/task.php?id=4925
http://www.facebook.com/note.php?note_id=194501560932
Also, InnoDB has an option for how much data should be allocated at
the end of a tablespace when it needs to grow:
http://dev.mysql.com/doc/refman/5.0/en/innodb-parameters.html#sysvar_innodb_data_file_path
--
Rob Wultsch
wultsch(at)gmail(dot)com
From: | Heikki Linnakangas <heikki(dot)linnakangas(at)enterprisedb(dot)com> |
---|---|
To: | Rob Wultsch <wultsch(at)gmail(dot)com> |
Cc: | Robert Haas <robertmhaas(at)gmail(dot)com>, Jim Nasby <jim(at)nasby(dot)net>, flyusa2010 fly <flyusa2010(at)gmail(dot)com>, pgsql-hackers(at)postgresql(dot)org |
Subject: | Re: Can postgres create a file with physically continuous blocks. |
Date: | 2010-12-22 07:15:17 |
Message-ID: | 4D11A585.2080706@enterprisedb.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-hackers |
On 22.12.2010 03:45, Rob Wultsch wrote:
> On Tue, Dec 21, 2010 at 4:49 AM, Robert Haas<robertmhaas(at)gmail(dot)com> wrote:
>> On Sun, Dec 19, 2010 at 1:10 PM, Jim Nasby<jim(at)nasby(dot)net> wrote:
>>> On Dec 19, 2010, at 1:10 AM, flyusa2010 fly wrote:
>>>> Does postgres make an effort to create a file with physically continuous blocks?
>>>
>>> AFAIK all files are expanded as needed. I don't think there's any flags you can pass to the filesystem to tell it "this file will eventually be 1GB in size". So, we're basically at the mercy of the FS to try and keep things contiguous.
>>
>> There have been some reports that we would do better on some
>> filesystems if we extended the file more than a block at a time, as we
>> do today. However, AFAIK, no one is pursuing this ATM.
>
> The has been found to be the case in the MySQL world, particularly
> when ext3 is in use:
> http://forge.mysql.com/worklog/task.php?id=4925
> http://www.facebook.com/note.php?note_id=194501560932
These seem to be about extending the transaction log, and we already
pre-allocate the WAL. The WAL is repeatedly fsync'd, so I can understand
that extending that in small chunks would hurt performance a lot, as the
filesystem needs to flush the metadata changes to disk at every commit.
However, that's not an issue with extending data files, they are only
fsync'd at checkpoints.
It might well be advantageous to extend data files in larger chunks too,
but it's probably nowhere near as important as with the WAL.
> Also, InnoDB has an option for how much data should be allocated at
> the end of a tablespace when it needs to grow:
> http://dev.mysql.com/doc/refman/5.0/en/innodb-parameters.html#sysvar_innodb_data_file_path
Hmm, innodb_autoextend_increment seems more like what we're discussing
here
(http://dev.mysql.com/doc/refman/5.0/en/innodb-parameters.html#sysvar_innodb_autoextend_increment).
If I'm reading that correctly, InnoDB defaults to extending files in 8MB
chunks.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
From: | Rob Wultsch <wultsch(at)gmail(dot)com> |
---|---|
To: | Heikki Linnakangas <heikki(dot)linnakangas(at)enterprisedb(dot)com> |
Cc: | Robert Haas <robertmhaas(at)gmail(dot)com>, Jim Nasby <jim(at)nasby(dot)net>, flyusa2010 fly <flyusa2010(at)gmail(dot)com>, pgsql-hackers(at)postgresql(dot)org |
Subject: | Re: Can postgres create a file with physically continuous blocks. |
Date: | 2010-12-22 07:25:40 |
Message-ID: | AANLkTim-TEzDdKCEo-OstKM3B7efkiyQ_rPiCpFVaggu@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-hackers |
On Wed, Dec 22, 2010 at 12:15 AM, Heikki Linnakangas
<heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
> On 22.12.2010 03:45, Rob Wultsch wrote:
>>
>> On Tue, Dec 21, 2010 at 4:49 AM, Robert Haas<robertmhaas(at)gmail(dot)com>
>> wrote:
>>>
>>> On Sun, Dec 19, 2010 at 1:10 PM, Jim Nasby<jim(at)nasby(dot)net> wrote:
>>>>
>>>> On Dec 19, 2010, at 1:10 AM, flyusa2010 fly wrote:
>>>>>
>>>>> Does postgres make an effort to create a file with physically
>>>>> continuous blocks?
>>>>
>>>> AFAIK all files are expanded as needed. I don't think there's any flags
>>>> you can pass to the filesystem to tell it "this file will eventually be 1GB
>>>> in size". So, we're basically at the mercy of the FS to try and keep things
>>>> contiguous.
>>>
>>> There have been some reports that we would do better on some
>>> filesystems if we extended the file more than a block at a time, as we
>>> do today. However, AFAIK, no one is pursuing this ATM.
>>
>> The has been found to be the case in the MySQL world, particularly
>> when ext3 is in use:
>> http://forge.mysql.com/worklog/task.php?id=4925
>> http://www.facebook.com/note.php?note_id=194501560932
>
> These seem to be about extending the transaction log, and we already
> pre-allocate the WAL. The WAL is repeatedly fsync'd, so I can understand
> that extending that in small chunks would hurt performance a lot, as the
> filesystem needs to flush the metadata changes to disk at every commit.
> However, that's not an issue with extending data files, they are only
> fsync'd at checkpoints.
>
> It might well be advantageous to extend data files in larger chunks too, but
> it's probably nowhere near as important as with the WAL.
Agree.
>> Also, InnoDB has an option for how much data should be allocated at
>> the end of a tablespace when it needs to grow:
>>
>> http://dev.mysql.com/doc/refman/5.0/en/innodb-parameters.html#sysvar_innodb_data_file_path
>
> Hmm, innodb_autoextend_increment seems more like what we're discussing here
> (http://dev.mysql.com/doc/refman/5.0/en/innodb-parameters.html#sysvar_innodb_autoextend_increment).
> If I'm reading that correctly, InnoDB defaults to extending files in 8MB
> chunks.
This is not pure apples to apples as InnoDB does direct io, however
doesn't the checkpoint completion target code call fsync repeatedly in
order to achieve the check point completion target? And for that
matter, haven't there been recent discussion on hackers about calling
fsync more often?
Sorry for the loopy email. I have not been getting anywhere near
enough sleep recently :(
--
Rob Wultsch
wultsch(at)gmail(dot)com
From: | Heikki Linnakangas <heikki(dot)linnakangas(at)enterprisedb(dot)com> |
---|---|
To: | Rob Wultsch <wultsch(at)gmail(dot)com> |
Cc: | Robert Haas <robertmhaas(at)gmail(dot)com>, Jim Nasby <jim(at)nasby(dot)net>, flyusa2010 fly <flyusa2010(at)gmail(dot)com>, pgsql-hackers(at)postgresql(dot)org |
Subject: | Re: Can postgres create a file with physically continuous blocks. |
Date: | 2010-12-22 07:29:01 |
Message-ID: | 4D11A8BD.7020204@enterprisedb.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-hackers |
On 22.12.2010 09:25, Rob Wultsch wrote:
> On Wed, Dec 22, 2010 at 12:15 AM, Heikki Linnakangas
> <heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
>> Hmm, innodb_autoextend_increment seems more like what we're discussing here
>> (http://dev.mysql.com/doc/refman/5.0/en/innodb-parameters.html#sysvar_innodb_autoextend_increment).
>> If I'm reading that correctly, InnoDB defaults to extending files in 8MB
>> chunks.
>
> This is not pure apples to apples as InnoDB does direct io, however
> doesn't the checkpoint completion target code call fsync repeatedly in
> order to achieve the check point completion target?
It only fsync's each file once. If there's a lot of files, it needs to
issue a lot of fsync's, but for different files.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com