Re: Fsync request queue

On 2018-04-25 09:19:52 +0300, Heikki Linnakangas wrote:
> On 24/04/18 21:00, Andres Freund wrote:
> > Right now if the queue is full and can't be compacted we end up
> > fsync()ing on every single write, rather than once per checkpoint
> > afaict. That's a fairly horrible.
> >
> > For the case that there's no space in the map, I'd suggest to just do
> > 10% or so of the fsync in the poor sod of a process that finds no
> > space. That's surely better than constantly fsyncing on every single
> > write.
>
> Clever idea. In principle, you could do that even with the current queue,
> without changing it to a hash table.

Right. I was thinking of this in the context of the fsync mess, which
seems to require us to keep FDs open across processes for reliable error
detection. Which then made me look at register_dirty_segment(). Which in
turn made me think that it's weird that we do all that work even if it's
likely that it's been done before...

> Is this a problem in practice, though? I don't remember seeing any reports
> of the fsync queue filling up, after we got the code to compact it. I don't
> know if anyone has been looking for that, so that might also explain the
> absence of reports, though.

It's probably hard to diagnose that as the origin of slow IO from the
outside. It's not exactly easy to diagnose that even if you know what's
going on.

Greetings,

Andres Freund