WAL Rate Limiting

From: Simon Riggs <simon(at)2ndQuadrant(dot)com>
To: PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: WAL Rate Limiting
Date: 2014-01-15 02:20:13
Message-ID: CA+U5nMLfxBgHQ1VLSeBHYEMjHXz_OHSkuFdU6_1quiGM0RNKEg@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

We've discussed previously the negative impact of large bulk
operations, especially wrt WAL writes. Patch here allows maintenance
operations to have their WAL generation slowed down as a replication
lag prevention feature.

I believe there was originally intended to be some work on I/O rate
limiting, but that hasn't happened and is in some ways orthogonal to
this patch and we will likely eventually want both.

Single new parameter works very similarly to vacuum_cost_delay

wal_rate_limit_delay = Xms

slows down CLUSTER, VACUUM FULL, ALTER TABLE (rewrite & set
tablespace), CREATE INDEX
so basically same things we optimise WAL for and the same places where
we honour maintenance_work_mem
(discuss: should we add COPY, CTAS etc also?)
(discuss: do we need another parameter to specify "cost"? Currently
patch uses "sleep every 64kB of WAL")

VACUUM is not included, since we already have controls for that -
honouring two controls would be complex and weird.

Uses GetCurrentTransactionWALVolume() patch, which is included within
the patch to make it easier to review as a whole.

Technically, we can't simply wait before/after WAL inserts because
these typically occur while holding buffer locks. So we need to put
the waits at a higher level, notably in safe places that currently do
CHECK_FOR_INTERRUPTS(). Doing that during query execution might make
locking of blocks for nested loops joins much worse.

--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Attachment Content-Type Size
wal_rate_limiting.v1.patch application/octet-stream 14.6 KB

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Dave Chinner 2014-01-15 02:25:56 Re: [Lsf-pc] Linux kernel impact on PostgreSQL performance
Previous Message Michael Paquier 2014-01-15 02:07:41 Re: CREATE TABLESPACE WITH