Re: patch: preload dictionary new version

On Thu, Jul 8, 2010 at 7:03 AM, Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com> wrote:
> 2010/7/8 Robert Haas <robertmhaas(at)gmail(dot)com>:
>> On Wed, Jul 7, 2010 at 10:50 PM, Takahiro Itagaki
>> <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp> wrote:
>>> This patch allocates memory with non-file-based mmap() to preload text search
>>> dictionary files at the server start. Note that dist files are not mmap'ed
>>> directly in the patch; mmap() is used for reallocatable shared memory.
>>
>> I thought someone (Tom?) had proposed idea previously of writing a
>> dictionary precompiler that would produce a file which could then be
>> mmap()'d into the backend.  Has any thought been given to that
>> approach?
>
> The precompiler can save only some time related to parsing. But it
> isn't main issue. Without simple allocation the data from dictionary
> takes about 55 MB, with simple allocation about 10 MB. If you have a
> 100 max_session, then these data can be 100 x repeated in memory -
> about 1G (for Czech dictionary).  I think so memory can be used
> better.

A precompiler can give you all the same memory management benefits.

> Minimally you have to read these 10MB from disc - maybe from file
> cache - but it takes some time too - but it will be significantly
> better than now.

If you use mmap(), you don't need to anything of the sort. And the
EXEC_BACKEND case doesn't require as many gymnastics, either. And the
variable can be PGC_SIGHUP or even PGC_USERSET instead of
PGC_POSTMASTER.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company