BUG #6528: pglesslog still referenced in docs, but no 9.1 support

The following bug has been logged on the website:

Bug reference: 6528
Logged by: Stuart Bishop
Email address: stuart(dot)bishop(at)canonical(dot)com
PostgreSQL version: 9.1.3
Operating system: Ubuntu
Description:

In section 24.3.5.1 of the docs on setting up continuous archiving, there
are a few paragraphs and examples for using pglesslog (pg_compresslog and
pg_decompresslog). This tool did not get out of beta for 9.0 and the only
reports I find for 9.1 are for not being able to compile it. Suggest
removing it from the docs unless someone wants to pull this tool into the
core.

On Tue, Mar 13, 2012 at 5:00 AM, <stuart(dot)bishop(at)canonical(dot)com> wrote:
> The following bug has been logged on the website:
>
> Bug reference:      6528
> Logged by:          Stuart Bishop
> Email address:      stuart(dot)bishop(at)canonical(dot)com
> PostgreSQL version: 9.1.3
> Operating system:   Ubuntu
> Description:
>
> In section 24.3.5.1 of the docs on setting up continuous archiving, there
> are a few paragraphs and examples for using pglesslog (pg_compresslog and
> pg_decompresslog). This tool did not get out of beta for 9.0 and the only
> reports I find for 9.1 are for not being able to compile it. Suggest
> removing it from the docs unless someone wants to pull this tool into the
> core.

This complaint appears to be accurate. I think we should go ahead and
remove that mention.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Mon, Apr 09, 2012 at 10:10:34AM -0400, Robert Haas wrote:
> On Tue, Mar 13, 2012 at 5:00 AM, <stuart(dot)bishop(at)canonical(dot)com> wrote:
> > The following bug has been logged on the website:
> >
> > Bug reference:      6528
> > Logged by:          Stuart Bishop
> > Email address:      stuart(dot)bishop(at)canonical(dot)com
> > PostgreSQL version: 9.1.3
> > Operating system:   Ubuntu
> > Description:
> >
> > In section 24.3.5.1 of the docs on setting up continuous archiving, there
> > are a few paragraphs and examples for using pglesslog (pg_compresslog and
> > pg_decompresslog). This tool did not get out of beta for 9.0 and the only
> > reports I find for 9.1 are for not being able to compile it. Suggest
> > removing it from the docs unless someone wants to pull this tool into the
> > core.
>
> This complaint appears to be accurate. I think we should go ahead and
> remove that mention.

Agreed; removed with the attached patch. I didn't bother keeping the
gzip mention because I assume there is little value to using without
pglesslog.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> Robert Haas wrote:
>> <stuart(dot)bishop(at)canonical(dot)com> wrote:

>>> In section 24.3.5.1 of the docs on setting up continuous
>>> archiving, there are a few paragraphs and examples for using
>>> pglesslog (pg_compresslog and pg_decompresslog). This tool did
>>> not get out of beta for 9.0 and the only reports I find for 9.1
>>> are for not being able to compile it. Suggest removing it from
>>> the docs unless someone wants to pull this tool into the core.
>>
>> This complaint appears to be accurate. I think we should go
>> ahead and remove that mention.
>
> Agreed; removed with the attached patch. I didn't bother keeping
> the gzip mention because I assume there is little value to using
> without pglesslog.

We found that gzip without doing something about the unused portion
of the WAL file generally reduced file size by about 50%, so it
still might be worth mentioning that.

-Kevin

Excerpts from Bruce Momjian's message of lun abr 09 15:13:10 -0300 2012:
> On Mon, Apr 09, 2012 at 10:10:34AM -0400, Robert Haas wrote:

> > This complaint appears to be accurate. I think we should go ahead and
> > remove that mention.
>
> Agreed; removed with the attached patch. I didn't bother keeping the
> gzip mention because I assume there is little value to using without
> pglesslog.

I'm not sure that assumption holds. I think you're thinking of the hack
that zeroes out the "empty" holes in the middle of data pages; those
didn't compress well unless zeroed out before compression. This tool is
not about that, but rather about removing redundant info from WAL files.
It seems to me that WAL files would be as gzip-compressible regardless
of pglesslog being applied. (Another related tool is clearxlogtail
which zeroes areas from WAL files when they are empty because of an
early switch due to archive timeout).

The funny thing is, apparently pg_lesslog was intentionally broken by
changing XLR_BKP_REMOVABLE to XLP_BKP_REMOVABLE (different semantics?)
and the fixes to make it compile again look simple. It's a bit strange
that NTT stopped maintaining the tool .. Maybe it wasn't useful anymore?

--
Álvaro Herrera <alvherre(at)commandprompt(dot)com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

On Mon, Apr 09, 2012 at 03:37:09PM -0300, Alvaro Herrera wrote:
>
> Excerpts from Bruce Momjian's message of lun abr 09 15:13:10 -0300 2012:
> > On Mon, Apr 09, 2012 at 10:10:34AM -0400, Robert Haas wrote:
>
> > > This complaint appears to be accurate. I think we should go ahead and
> > > remove that mention.
> >
> > Agreed; removed with the attached patch. I didn't bother keeping the
> > gzip mention because I assume there is little value to using without
> > pglesslog.
>
> I'm not sure that assumption holds. I think you're thinking of the hack
> that zeroes out the "empty" holes in the middle of data pages; those
> didn't compress well unless zeroed out before compression. This tool is
> not about that, but rather about removing redundant info from WAL files.
> It seems to me that WAL files would be as gzip-compressible regardless
> of pglesslog being applied. (Another related tool is clearxlogtail

OK, docs updated with the attached patch.

> which zeroes areas from WAL files when they are empty because of an
> early switch due to archive timeout).

Should we document that?

> The funny thing is, apparently pg_lesslog was intentionally broken by
> changing XLR_BKP_REMOVABLE to XLP_BKP_REMOVABLE (different semantics?)
> and the fixes to make it compile again look simple. It's a bit strange
> that NTT stopped maintaining the tool .. Maybe it wasn't useful anymore?

No idea.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> On Mon, Apr 09, 2012 at 03:37:09PM -0300, Alvaro Herrera wrote:

>> (Another related tool is clearxlogtail which zeroes areas from
>> WAL files when they are empty because of an early switch due to
>> archive timeout).
>
> Should we document that?

Our shop has been using that since before pglesslog existed, and it
has continued to work across many major releases with no change to
source code because it doesn't get down to the level of looking at
the xlog records themselves, just the segment and page-level
structures. I should probably put packaging that up as a proper
extension and posting it to PGXN as another thing on the list of
things I'll do when I stumble over that pot of round tuits at the
end of the rainbow. Right now the source is on pgfoundry. If
anybody wants to do anything with it before I find time, feel free.

-Kevin

On Mon, Apr 9, 2012 at 02:07:43PM -0500, Kevin Grittner wrote:
> Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> > On Mon, Apr 09, 2012 at 03:37:09PM -0300, Alvaro Herrera wrote:
>
> >> (Another related tool is clearxlogtail which zeroes areas from
> >> WAL files when they are empty because of an early switch due to
> >> archive timeout).
> >
> > Should we document that?
>
> Our shop has been using that since before pglesslog existed, and it
> has continued to work across many major releases with no change to
> source code because it doesn't get down to the level of looking at
> the xlog records themselves, just the segment and page-level
> structures. I should probably put packaging that up as a proper
> extension and posting it to PGXN as another thing on the list of
> things I'll do when I stumble over that pot of round tuits at the
> end of the rainbow. Right now the source is on pgfoundry. If
> anybody wants to do anything with it before I find time, feel free.

I was going to add a mention of this to our docs, but it seems there are
no released files for the project:

http://pgfoundry.org/frs/?group_id=1000308

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> On Mon, Apr 9, 2012 at 02:07:43PM -0500, Kevin Grittner wrote:
>> Bruce Momjian <bruce(at)momjian(dot)us> wrote:
>>> On Mon, Apr 09, 2012 at 03:37:09PM -0300, Alvaro Herrera wrote:
>>>
>>>> (Another related tool is clearxlogtail which zeroes areas from
>>>> WAL files when they are empty because of an early switch due to
>>>> archive timeout).
>>>
>>> Should we document that?
>>
>> Our shop has been using that since before pglesslog existed, and
>> it has continued to work across many major releases with no
>> change to source code because it doesn't get down to the level of
>> looking at the xlog records themselves, just the segment and
>> page-level structures. I should probably put packaging that up
>> as a proper extension and posting it to PGXN as another thing on
>> the list of things I'll do when I stumble over that pot of round
>> tuits at the end of the rainbow. Right now the source is on
>> pgfoundry. If anybody wants to do anything with it before I find
>> time, feel free.
>
> I was going to add a mention of this to our docs, but it seems
> there are no released files for the project:

There is a single 175 line C source file to generate a stand-alone
executable filter. The source has not needed any changes since it
was finished over 4 years ago. Besides that C file, there is only a
make file. I wasn't sure what form of packaging would be easier
than just checking out the source from the SCM.

What do you think would be the right thing to do with it at this
point?

-Kevin

On Tue, Sep 4, 2012 at 12:11:53PM -0500, Kevin Grittner wrote:
> Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> > On Mon, Apr 9, 2012 at 02:07:43PM -0500, Kevin Grittner wrote:
> >> Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> >>> On Mon, Apr 09, 2012 at 03:37:09PM -0300, Alvaro Herrera wrote:
> >>>
> >>>> (Another related tool is clearxlogtail which zeroes areas from
> >>>> WAL files when they are empty because of an early switch due to
> >>>> archive timeout).
> >>>
> >>> Should we document that?
> >>
> >> Our shop has been using that since before pglesslog existed, and
> >> it has continued to work across many major releases with no
> >> change to source code because it doesn't get down to the level of
> >> looking at the xlog records themselves, just the segment and
> >> page-level structures. I should probably put packaging that up
> >> as a proper extension and posting it to PGXN as another thing on
> >> the list of things I'll do when I stumble over that pot of round
> >> tuits at the end of the rainbow. Right now the source is on
> >> pgfoundry. If anybody wants to do anything with it before I find
> >> time, feel free.
> >
> > I was going to add a mention of this to our docs, but it seems
> > there are no released files for the project:
>
> There is a single 175 line C source file to generate a stand-alone
> executable filter. The source has not needed any changes since it
> was finished over 4 years ago. Besides that C file, there is only a
> make file. I wasn't sure what form of packaging would be easier
> than just checking out the source from the SCM.
>
> What do you think would be the right thing to do with it at this
> point?

Well, there should probably be a tar files with a README. What about
adding this to /contrib?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> On Tue, Sep 4, 2012 at 12:11:53PM -0500, Kevin Grittner wrote:

>> What do you think would be the right thing to do with it at this
>> point?
>
> Well, there should probably be a tar files with a README. What
> about adding this to /contrib?

I would be happy to add it to contrib if people want it there.
There was some discussion of that when I wrote it, but then there
was a feeling that it would be uninteresting once people had
streaming replication. I'm skeptical that people will ever lose
interest in archiving WAL files, because of the not-uncommon need to
keep archival backups or restore to a "point in time" to recover
from a mangled table. As long as those needs are there, I suspect
that people will want to compress WAL files, including making
partially-filled files smaller.

-Kevin

On Tue, Sep 4, 2012 at 12:49:40PM -0500, Kevin Grittner wrote:
> Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> > On Tue, Sep 4, 2012 at 12:11:53PM -0500, Kevin Grittner wrote:
>
> >> What do you think would be the right thing to do with it at this
> >> point?
> >
> > Well, there should probably be a tar files with a README. What
> > about adding this to /contrib?
>
> I would be happy to add it to contrib if people want it there.
> There was some discussion of that when I wrote it, but then there
> was a feeling that it would be uninteresting once people had
> streaming replication. I'm skeptical that people will ever lose
> interest in archiving WAL files, because of the not-uncommon need to
> keep archival backups or restore to a "point in time" to recover
> from a mangled table. As long as those needs are there, I suspect
> that people will want to compress WAL files, including making
> partially-filled files smaller.

Agreed. Please move forward on the contrib idea.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On Tue, Sep 4, 2012 at 02:01:54PM -0400, Bruce Momjian wrote:
> On Tue, Sep 4, 2012 at 12:49:40PM -0500, Kevin Grittner wrote:
> > Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> > > On Tue, Sep 4, 2012 at 12:11:53PM -0500, Kevin Grittner wrote:
> >
> > >> What do you think would be the right thing to do with it at this
> > >> point?
> > >
> > > Well, there should probably be a tar files with a README. What
> > > about adding this to /contrib?
> >
> > I would be happy to add it to contrib if people want it there.
> > There was some discussion of that when I wrote it, but then there
> > was a feeling that it would be uninteresting once people had
> > streaming replication. I'm skeptical that people will ever lose
> > interest in archiving WAL files, because of the not-uncommon need to
> > keep archival backups or restore to a "point in time" to recover
> > from a mangled table. As long as those needs are there, I suspect
> > that people will want to compress WAL files, including making
> > partially-filled files smaller.
>
> Agreed. Please move forward on the contrib idea.

Kevin, did this get accomplished?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

Bruce Momjian escribió:
>
> On Tue, Sep 4, 2012 at 02:01:54PM -0400, Bruce Momjian wrote:
> > On Tue, Sep 4, 2012 at 12:49:40PM -0500, Kevin Grittner wrote:
> > > Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> > > > On Tue, Sep 4, 2012 at 12:11:53PM -0500, Kevin Grittner wrote:
> > >
> > > >> What do you think would be the right thing to do with it at this
> > > >> point?
> > > >
> > > > Well, there should probably be a tar files with a README. What
> > > > about adding this to /contrib?
> > >
> > > I would be happy to add it to contrib if people want it there.
> > > There was some discussion of that when I wrote it, but then there
> > > was a feeling that it would be uninteresting once people had
> > > streaming replication. I'm skeptical that people will ever lose
> > > interest in archiving WAL files, because of the not-uncommon need to
> > > keep archival backups or restore to a "point in time" to recover
> > > from a mangled table. As long as those needs are there, I suspect
> > > that people will want to compress WAL files, including making
> > > partially-filled files smaller.
> >
> > Agreed. Please move forward on the contrib idea.
>
> Kevin, did this get accomplished?

Just to be clear, what we're proposing adding to contrib is
clearxlogtail, not pglesslog, right?

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Thu, Jan 24, 2013 at 08:43:37PM -0300, Alvaro Herrera wrote:
> Bruce Momjian escribió:
> >
> > On Tue, Sep 4, 2012 at 02:01:54PM -0400, Bruce Momjian wrote:
> > > On Tue, Sep 4, 2012 at 12:49:40PM -0500, Kevin Grittner wrote:
> > > > Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> > > > > On Tue, Sep 4, 2012 at 12:11:53PM -0500, Kevin Grittner wrote:
> > > >
> > > > >> What do you think would be the right thing to do with it at this
> > > > >> point?
> > > > >
> > > > > Well, there should probably be a tar files with a README. What
> > > > > about adding this to /contrib?
> > > >
> > > > I would be happy to add it to contrib if people want it there.
> > > > There was some discussion of that when I wrote it, but then there
> > > > was a feeling that it would be uninteresting once people had
> > > > streaming replication. I'm skeptical that people will ever lose
> > > > interest in archiving WAL files, because of the not-uncommon need to
> > > > keep archival backups or restore to a "point in time" to recover
> > > > from a mangled table. As long as those needs are there, I suspect
> > > > that people will want to compress WAL files, including making
> > > > partially-filled files smaller.
> > >
> > > Agreed. Please move forward on the contrib idea.
> >
> > Kevin, did this get accomplished?
>
> Just to be clear, what we're proposing adding to contrib is
> clearxlogtail, not pglesslog, right?

I was not clear on exactly what Kevin was going to add. Kevin?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On 2012-04-09 15:37:09 -0300, Alvaro Herrera wrote:
>
> Excerpts from Bruce Momjian's message of lun abr 09 15:13:10 -0300 2012:
> > On Mon, Apr 09, 2012 at 10:10:34AM -0400, Robert Haas wrote:
>
> > > This complaint appears to be accurate. I think we should go ahead and
> > > remove that mention.
> >
> > Agreed; removed with the attached patch. I didn't bother keeping the
> > gzip mention because I assume there is little value to using without
> > pglesslog.
>
> I'm not sure that assumption holds. I think you're thinking of the hack
> that zeroes out the "empty" holes in the middle of data pages; those
> didn't compress well unless zeroed out before compression. This tool is
> not about that, but rather about removing redundant info from WAL files.
> It seems to me that WAL files would be as gzip-compressible regardless
> of pglesslog being applied. (Another related tool is clearxlogtail
> which zeroes areas from WAL files when they are empty because of an
> early switch due to archive timeout).
>
> The funny thing is, apparently pg_lesslog was intentionally broken by
> changing XLR_BKP_REMOVABLE to XLP_BKP_REMOVABLE (different semantics?)
> and the fixes to make it compile again look simple. It's a bit strange
> that NTT stopped maintaining the tool .. Maybe it wasn't useful anymore?

One problem is that it was hard to trust the tool, it had relatively bad
bugs several times and it wasn't maintained by pgdg...

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Thu, Jan 24, 2013 at 06:45:02PM -0500, Bruce Momjian wrote:
> On Thu, Jan 24, 2013 at 08:43:37PM -0300, Alvaro Herrera wrote:
> > Bruce Momjian escribió:
> > >
> > > On Tue, Sep 4, 2012 at 02:01:54PM -0400, Bruce Momjian wrote:
> > > > On Tue, Sep 4, 2012 at 12:49:40PM -0500, Kevin Grittner wrote:
> > > > > Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> > > > > > On Tue, Sep 4, 2012 at 12:11:53PM -0500, Kevin Grittner wrote:
> > > > >
> > > > > >> What do you think would be the right thing to do with it at this
> > > > > >> point?
> > > > > >
> > > > > > Well, there should probably be a tar files with a README. What
> > > > > > about adding this to /contrib?
> > > > >
> > > > > I would be happy to add it to contrib if people want it there.
> > > > > There was some discussion of that when I wrote it, but then there
> > > > > was a feeling that it would be uninteresting once people had
> > > > > streaming replication. I'm skeptical that people will ever lose
> > > > > interest in archiving WAL files, because of the not-uncommon need to
> > > > > keep archival backups or restore to a "point in time" to recover
> > > > > from a mangled table. As long as those needs are there, I suspect
> > > > > that people will want to compress WAL files, including making
> > > > > partially-filled files smaller.
> > > >
> > > > Agreed. Please move forward on the contrib idea.
> > >
> > > Kevin, did this get accomplished?
> >
> > Just to be clear, what we're proposing adding to contrib is
> > clearxlogtail, not pglesslog, right?
>
> I was not clear on exactly what Kevin was going to add. Kevin?

Any news on this?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +