Re: [ANNOUNCE] Bug-fix and new feature of pg_lesslog is released

It is an announcement that long-waited bug-fix of pg_lesslog is now
released. This includes the following.

1. Error in calculation of GiST-related WAL record length was fixed.
2. pg_compresslog now has an option to print WAL segment analysis.
3. New test script is added which analyzes what WAL record was
generated and if all the recovery was done successfully.

2 and 3 are added to ensure the tools run correctly.

Because 8.4 has additional WAL records, I made release material
separate for each PG version. You may visit the project page
http://pgfoundry.org/frs/?group_id=1000310
to download materials.

I acknowledge PG hackers who gave valuable comments/advices.
Especially Oleg Bartunov, Teodor Sigaev and Alvaro Herrera gave very
helpful information how to generate specific type of WAL records.

As Bruce suggested, this can be included in PostgreSQL test suits and
could be included in contrib module if it is ported to PostgreSQL 9.1.

Best Regards;
----------
Koichi Suzuki

# pg_lesslog is a project to compress large WAL segments when they're
stored and restore then for recovery.

Koichi Suzuki wrote:
> It is an announcement that long-waited bug-fix of pg_lesslog is now
> released. This includes the following.
>
> 1. Error in calculation of GiST-related WAL record length was fixed.
> 2. pg_compresslog now has an option to print WAL segment analysis.
> 3. New test script is added which analyzes what WAL record was
> generated and if all the recovery was done successfully.
>
> 2 and 3 are added to ensure the tools run correctly.
>
> Because 8.4 has additional WAL records, I made release material
> separate for each PG version. You may visit the project page
> http://pgfoundry.org/frs/?group_id=1000310
> to download materials.
>
> I acknowledge PG hackers who gave valuable comments/advices.
> Especially Oleg Bartunov, Teodor Sigaev and Alvaro Herrera gave very
> helpful information how to generate specific type of WAL records.
>
> As Bruce suggested, this can be included in PostgreSQL test suits and
> could be included in contrib module if it is ported to PostgreSQL 9.1.

[ email to hackers list only ]

Yes, I would love to get this into /contrib for PG 9.1!

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

Bruce Momjian <bruce(at)momjian(dot)us> writes:
> Yes, I would love to get this into /contrib for PG 9.1!

How much are people really going to care about pg_lesslog now that
we've got streaming replication? There might be some small use-case
still left, but it's hard to believe that it would be worth carrying
it in contrib.

regards, tom lane

On Wed, May 12, 2010 at 11:03 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Bruce Momjian <bruce(at)momjian(dot)us> writes:
>> Yes, I would love to get this into /contrib for PG 9.1!
>
> How much are people really going to care about pg_lesslog now that
> we've got streaming replication?  There might be some small use-case
> still left, but it's hard to believe that it would be worth carrying
> it in contrib.

Until we have a tool to use streaming replication to actually archive
backups, there's still a fairly large usecase for it...

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On Wed, 2010-05-12 at 23:09 +0200, Magnus Hagander wrote:
> On Wed, May 12, 2010 at 11:03 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > Bruce Momjian <bruce(at)momjian(dot)us> writes:
> >> Yes, I would love to get this into /contrib for PG 9.1!
> >
> > How much are people really going to care about pg_lesslog now that
> > we've got streaming replication? There might be some small use-case
> > still left, but it's hard to believe that it would be worth carrying
> > it in contrib.
>
> Until we have a tool to use streaming replication to actually archive
> backups, there's still a fairly large usecase for it...

Not to mention, WAN based standby, timed standby etc...

Joshua D. Drake

>
>
> --
> Magnus Hagander
> Me: http://www.hagander.net/
> Work: http://www.redpill-linpro.com/
>

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering

> Until we have a tool to use streaming replication to actually archive
> backups, there's still a fairly large usecase for it...

yes, pglesslog is for PITR. We have a client who keeps 3 days of log
files a time for forensic reasons, and I doubt they're the only one.

--
-- Josh Berkus
PostgreSQL Experts Inc.
http://www.pgexperts.com

Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> Bruce Momjian <bruce(at)momjian(dot)us> writes:
> > Yes, I would love to get this into /contrib for PG 9.1!
>
> How much are people really going to care about pg_lesslog now that
> we've got streaming replication? There might be some small use-case
> still left, but it's hard to believe that it would be worth carrying
> it in contrib.

I hope pg_lesslog would work as a WAL filter of streaming replication.
It might be hard-coded in WAL sender, or be an addon based on a new
common filtering infrastructure of WAL streaming.

Also, there is a long-standing issue in pg_lesslog; It slows down recovery
because we need to read data pages before write in recovery. We're avoiding
reading pages for full-page image in 8.3, but pg_lesslog will disable
the optimization. Recovery routine in core also needs to be adjusted to use
read-ahead, like posix_fadvise().

There was another idea, full-page image logs separated with WAL logging.
In theory, full-page images don't have to be written at commit, but only
by writing corresponding data pages, I'm not sure whether it is an actually
good idea or not, but if we go the direction, we won't need pg_lesslog.

Regards,
---
Takahiro Itagaki
NTT Open Source Software Center

Thanks a lot for the comment/advice. Yes, full page backup block
considerablly shortens the recovery time. As we discussed about two
years ago, I have a solution accelerate the recovery even without full
page image. I'd like to submit this solution to the community again.
When I evaluated this two years ago, recovery speed was as good as
those with full page image, depending upon application and tuning, of
course.

This is a separate tool and can be used in various scenes.

Regards;
----------
Koichi Suzuki

2010/5/13 Takahiro Itagaki <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>:
>
> Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
>> Bruce Momjian <bruce(at)momjian(dot)us> writes:
>> > Yes, I would love to get this into /contrib for PG 9.1!
>>
>> How much are people really going to care about pg_lesslog now that
>> we've got streaming replication?  There might be some small use-case
>> still left, but it's hard to believe that it would be worth carrying
>> it in contrib.
>
> I hope pg_lesslog would work as a WAL filter of streaming replication.
> It might be hard-coded in WAL sender, or be an addon based on a new
> common filtering infrastructure of WAL streaming.
>
> Also, there is a long-standing issue in pg_lesslog; It slows down recovery
> because we need to read data pages before write in recovery. We're avoiding
> reading pages for full-page image in 8.3, but pg_lesslog will disable
> the optimization. Recovery routine in core also needs to be adjusted to use
> read-ahead, like posix_fadvise().
>
> There was another idea, full-page image logs separated with WAL logging.
> In theory, full-page images don't have to be written at commit, but only
> by writing corresponding data pages, I'm not sure whether it is an actually
> good idea or not, but if we go the direction, we won't need pg_lesslog.
>
> Regards,
> ---
> Takahiro Itagaki
> NTT Open Source Software Center
>
>
>

WAL streming filter is an interesting idea. We can leave full page
backup for local recovery and decrease the amount of WAL to transfer.

If SR writes WAL in record by record basis, implementation will be
simple. If SR writes WAL in block by block basis, WAL records may be
divided from transfer to transfer and it is not simple to handle.

Regards;
----------
Koichi Suzuki

2010/5/13 Takahiro Itagaki <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>:
>
> Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
>> Bruce Momjian <bruce(at)momjian(dot)us> writes:
>> > Yes, I would love to get this into /contrib for PG 9.1!
>>
>> How much are people really going to care about pg_lesslog now that
>> we've got streaming replication?  There might be some small use-case
>> still left, but it's hard to believe that it would be worth carrying
>> it in contrib.
>
> I hope pg_lesslog would work as a WAL filter of streaming replication.
> It might be hard-coded in WAL sender, or be an addon based on a new
> common filtering infrastructure of WAL streaming.
>
> Also, there is a long-standing issue in pg_lesslog; It slows down recovery
> because we need to read data pages before write in recovery. We're avoiding
> reading pages for full-page image in 8.3, but pg_lesslog will disable
> the optimization. Recovery routine in core also needs to be adjusted to use
> read-ahead, like posix_fadvise().
>
> There was another idea, full-page image logs separated with WAL logging.
> In theory, full-page images don't have to be written at commit, but only
> by writing corresponding data pages, I'm not sure whether it is an actually
> good idea or not, but if we go the direction, we won't need pg_lesslog.
>
> Regards,
> ---
> Takahiro Itagaki
> NTT Open Source Software Center
>
>
>