Re: [Lsf-pc] Linux kernel impact on PostgreSQL performance

On Wed, Jan 15, 2014 at 08:03:28AM +1300, Gavin Flower wrote:
> On 14/01/14 14:09, Dave Chinner wrote:
> >On Mon, Jan 13, 2014 at 09:29:02PM +0000, Greg Stark wrote:
> >>On Mon, Jan 13, 2014 at 9:12 PM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> [...]
> >>The more ambitious and interesting direction is to let Postgres tell
> >>the kernel what it needs to know to manage everything. To do that we
> >>would need the ability to control when pages are flushed out. This is
> >>absolutely necessary to maintain consistency. Postgres would need to
> >>be able to mark pages as unflushable until some point in time in the
> >>future when the journal is flushed. We discussed various ways that
> >>interface could work but it would be tricky to keep it low enough
> >>overhead to be workable.
> >IMO, the concept of allowing userspace to pin dirty page cache
> >pages in memory is just asking for trouble. Apart from the obvious
> >memory reclaim and OOM issues, some filesystems won't be able to
> >move their journals forward until the data is flushed. i.e. ordered
> >mode data writeback on ext3 will have all sorts of deadlock issues
> >that result from pinning pages and then issuing fsync() on another
> >file which will block waiting for the pinned pages to be flushed.
> >
> >Indeed, what happens if you do pin_dirty_pages(fd); .... fsync(fd);?
> >If fsync() blocks because there are pinned pages, and there's no
> >other thread to unpin them, then that code just deadlocked. If
> >fsync() doesn't block and skips the pinned pages, then we haven't
> >done an fsync() at all, and so violated the expectation that users
> >have that after fsync() returns their data is safe on disk. And if
> >we return an error to fsync(), then what the hell does the user do
> >if it is some other application we don't know about that has pinned
> >the pages? And if the kernel unpins them after some time, then we
> >just violated the application's consistency guarantees....
> >
> [...]
>
> What if Postgres could tell the kernel how strongly that it wanted
> to hold on to the pages?

That doesn't get rid of the problems, it just makes it harder to
diagnose them when they occur. :/

Cheers,

Dave.
--
Dave Chinner
david(at)fromorbit(dot)com