Re: Backup throttling

On 08/26/2013 12:50 PM, Antonin Houska wrote:
> On 08/22/2013 03:33 PM, Craig Ringer wrote:
>> On 08/22/2013 01:39 PM, PostgreSQL - Hans-Jürgen Schönig wrote:
>>
>>> what would be a reasonable scenario where limiting streaming would make sense? i cannot think of any to be honest.
>> I tend to agree. If anything we're likely to want the reverse - the
>> ability to throttle WAL *generation* on the master so streaming can keep up.
> (I assume you mean WAL *sending* rather than *generation*.)
I think he meant *generation*, that is making sure that no more
than X amount of WAL is generated in Y amount of time, thereby
making sure that you can stream less WAL without falling behind.
> I don't think we want to throttle WAL sending/receiving at all. The WAL
> senders should always keep up with the amount of WAL generated. If the
> rate of WAL sending is restricted, then the pg_basebackup (or another
> client application) might never catch up.
Yes, and this is exactly why restricting *generation* is needed.
> Also, IMO, pg_basebackup starts at the current WAL segment. Thus the
> unthrottled WAL transfer shouldn't cause significant additional load,
> such as reading many old WAL segments from the master's disk.
The possible extra load happens if the *network* not because of disk.
Think of replication over - possibly slow - WAN.

Cheers

--
Hannu Krosing
PostgreSQL Consultant
Performance, Scalability and High Availability
2ndQuadrant Nordic OÜ