Re: 2GB limit for temp_file_limit on 32bit platform

Hello

I did a backport of temp_file_limit feature to 9.1, but when we tested
this patch, we found very restristrictive limit to 2GB.

2GB is nonsense, because this is session limit of temp files, and
these files should be longer than 2GB.

Regards

Pavel

On 07/19/2012 01:04 PM, Pavel Stehule wrote:
> I did a backport of temp_file_limit feature to 9.1, but when we tested
> this patch, we found very restristrictive limit to 2GB.
>
> 2GB is nonsense, because this is session limit of temp files, and
> these files should be longer than 2GB.

I haven't read the patch but... don't all 32bit platforms have a 2GB
limit (by default)?

Sincerely,

jD

--
Command Prompt, Inc. - http://www.commandprompt.com/
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, Postgres-XC
@cmdpromptinc - 509-416-6579

On Thu, Jul 19, 2012 at 4:29 PM, Joshua D. Drake <jd(at)commandprompt(dot)com> wrote:
>
> On 07/19/2012 01:04 PM, Pavel Stehule wrote:
>>
>> I did a backport of temp_file_limit feature to 9.1, but when we tested
>> this patch, we found very restristrictive limit to 2GB.
>>
>> 2GB is nonsense, because this is session limit of temp files, and
>> these files should be longer than 2GB.
>
>
> I haven't read the patch but... don't all 32bit platforms have a 2GB limit
> (by default)?

I don't think so.

LFS got done in the mid-90s, which is long enough ago for people to
start forgetting about it. Are there any supported platforms that
didn't adopt LFS?

http://en.wikipedia.org/wiki/Large_file_support
--
When confronted by a difficult problem, solve it by reducing it to the
question, "How would the Lone Ranger handle this?"

On 07/19/2012 01:48 PM, Christopher Browne wrote:
>
> On Thu, Jul 19, 2012 at 4:29 PM, Joshua D. Drake <jd(at)commandprompt(dot)com> wrote:
>>
>> On 07/19/2012 01:04 PM, Pavel Stehule wrote:
>>>
>>> I did a backport of temp_file_limit feature to 9.1, but when we tested
>>> this patch, we found very restristrictive limit to 2GB.
>>>
>>> 2GB is nonsense, because this is session limit of temp files, and
>>> these files should be longer than 2GB.
>>
>>
>> I haven't read the patch but... don't all 32bit platforms have a 2GB limit
>> (by default)?
>
> I don't think so.
>
> LFS got done in the mid-90s, which is long enough ago for people to
> start forgetting about it. Are there any supported platforms that
> didn't adopt LFS?
>
> http://en.wikipedia.org/wiki/Large_file_support

Note: "by default" :). I know they could support LFS but as I recall you
had to compile specifically for it (at least on linux and old versions
of pg).

So I was curious if it was that specific limitation or a limitation
within the Pg code itself.

Sincerely,

Joshua D. Drake

--
Command Prompt, Inc. - http://www.commandprompt.com/
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, Postgres-XC
@cmdpromptinc - 509-416-6579

On 20/07/12 09:08, Joshua D. Drake wrote:
>
> On 07/19/2012 01:48 PM, Christopher Browne wrote:
>>
>> On Thu, Jul 19, 2012 at 4:29 PM, Joshua D. Drake
>> <jd(at)commandprompt(dot)com> wrote:
>>>
>>> On 07/19/2012 01:04 PM, Pavel Stehule wrote:
>>>>
>>>> I did a backport of temp_file_limit feature to 9.1, but when we tested
>>>> this patch, we found very restristrictive limit to 2GB.
>>>>
>>>> 2GB is nonsense, because this is session limit of temp files, and
>>>> these files should be longer than 2GB.
>>>
>>>
>>> I haven't read the patch but... don't all 32bit platforms have a 2GB
>>> limit
>>> (by default)?
>>
>> I don't think so.
>>
>> LFS got done in the mid-90s, which is long enough ago for people to
>> start forgetting about it. Are there any supported platforms that
>> didn't adopt LFS?
>>
>> http://en.wikipedia.org/wiki/Large_file_support
>
> Note: "by default" :). I know they could support LFS but as I recall
> you had to compile specifically for it (at least on linux and old
> versions of pg).
>
> So I was curious if it was that specific limitation or a limitation
> within the Pg code itself.
>
>

It is to do with the datatype of the GUC used for the setting - I
haven't got the patch in from of me to look at, but recall that going
larger meant using a float type which meant you couldn't get nice units
displayed (MB, GB etc).

I'll take a proper look later.

Cheers

Mark

On 20/07/12 09:58, Mark Kirkwood wrote:
> On 20/07/12 09:08, Joshua D. Drake wrote:
>>
>> On 07/19/2012 01:48 PM, Christopher Browne wrote:
>>>
>>> On Thu, Jul 19, 2012 at 4:29 PM, Joshua D. Drake
>>> <jd(at)commandprompt(dot)com> wrote:
>>>>
>>>> On 07/19/2012 01:04 PM, Pavel Stehule wrote:
>>>>>
>>>>> I did a backport of temp_file_limit feature to 9.1, but when we
>>>>> tested
>>>>> this patch, we found very restristrictive limit to 2GB.
>>>>>
>>>>> 2GB is nonsense, because this is session limit of temp files, and
>>>>> these files should be longer than 2GB.
>>>>
>>>>
>>>> I haven't read the patch but... don't all 32bit platforms have a
>>>> 2GB limit
>>>> (by default)?
>>>
>>> I don't think so.
>>>
>>> LFS got done in the mid-90s, which is long enough ago for people to
>>> start forgetting about it. Are there any supported platforms that
>>> didn't adopt LFS?
>>>
>>> http://en.wikipedia.org/wiki/Large_file_support
>>
>> Note: "by default" :). I know they could support LFS but as I recall
>> you had to compile specifically for it (at least on linux and old
>> versions of pg).
>>
>> So I was curious if it was that specific limitation or a limitation
>> within the Pg code itself.
>>
>>
>
> It is to do with the datatype of the GUC used for the setting - I
> haven't got the patch in from of me to look at, but recall that going
> larger meant using a float type which meant you couldn't get nice
> units displayed (MB, GB etc).
>
> I'll take a proper look later.
>
>

From src/backend/utils/misc/guc.c

{"temp_file_limit", PGC_SUSET, RESOURCES_DISK,
gettext_noop("Limits the total size of all temp files used
by each session."),
gettext_noop("-1 means no limit."),
GUC_UNIT_KB
},
&temp_file_limit,
-1, -1, INT_MAX,
NULL, NULL, NULL
},

Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com> writes:
> I did a backport of temp_file_limit feature to 9.1, but when we tested
> this patch, we found very restristrictive limit to 2GB.

> 2GB is nonsense, because this is session limit of temp files, and
> these files should be longer than 2GB.

This claim is nonsense. The variable's value is measured in KB, so the
effective limit is actually 2TB not 2GB.

regards, tom lane

On 20/07/12 12:02, Tom Lane wrote:
> Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com> writes:
>> I did a backport of temp_file_limit feature to 9.1, but when we tested
>> this patch, we found very restristrictive limit to 2GB.
>> 2GB is nonsense, because this is session limit of temp files, and
>> these files should be longer than 2GB.
> This claim is nonsense. The variable's value is measured in KB, so the
> effective limit is actually 2TB not 2GB.
>
>

Did you guys perchance pick up one of the earlier patches that had
MAX_KILOBYTES instead of INT_MAX as the limit? i.e:

{"temp_file_limit", PGC_SUSET, RESOURCES_DISK,
gettext_noop("Limits the total size of all temp files used
by each session."),
gettext_noop("-1 means no limit."),
GUC_UNIT_KB
},
&temp_file_limit,
-1, -1, **MAX_KILOBYTES**, <====
NULL, NULL, NULL
},

2012/7/20 Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>:
> Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com> writes:
>> I did a backport of temp_file_limit feature to 9.1, but when we tested
>> this patch, we found very restristrictive limit to 2GB.
>
>> 2GB is nonsense, because this is session limit of temp files, and
>> these files should be longer than 2GB.
>
> This claim is nonsense. The variable's value is measured in KB, so the
> effective limit is actually 2TB not 2GB.

you have true - it works on 9.2. The problem will be somewhere in
backport on 9.1

sorry for false alarm

Regards

Pavel Stěhule

>
> regards, tom lane

2012/7/20 Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>:
> 2012/7/20 Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>:
>> Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com> writes:
>>> I did a backport of temp_file_limit feature to 9.1, but when we tested
>>> this patch, we found very restristrictive limit to 2GB.
>>
>>> 2GB is nonsense, because this is session limit of temp files, and
>>> these files should be longer than 2GB.
>>
>> This claim is nonsense. The variable's value is measured in KB, so the
>> effective limit is actually 2TB not 2GB.
>
> you have true - it works on 9.2. The problem will be somewhere in
> backport on 9.1

it works well with 9.1.4, but not with 9.1.3

>
> sorry for false alarm
>
> Regards
>
> Pavel Stěhule
>
>>
>> regards, tom lane