Allow pg_archivecleanup to ignore extensions

One bit of feedback I keep getting from people who archive their WAL
files is that the fairly new pg_archivecleanup utility doesn't handle
the case where those archives are compressed. As the sort of users who
are concerned about compression are also often ones with giant archives
they struggle to cleanup, they would certainly appreciate having a
bundled utility to take care of that.

The attached patch provides an additional option to the utility to
provide this capability. It just strips a provided extension off any
matching file it considers before running the test for whether it should
be deleted or not. It includes updates to the usage message and some
docs about how this might be used. Code by Jaime Casanova and myself.

Here's an example of it working:

$ psql -c "show archive_command"
archive_command
----------------------------------------------------
cp -i %p archive/%f < /dev/null && gzip archive/%f
[Yes, I know that can be written more cleanly. I call external scripts
with more serious error handling than you can put into a single command
line for this sort of thing in production.]

$ psql -c "select pg_start_backup('test',true)"
$ psql -c "select pg_stop_backup()"
$ psql -c "checkpoint"
$ psql -c "select pg_switch_xlog()"

$ cd $PGDATA/archive
$ ls
000000010000000000000025.gz
000000010000000000000026.gz
000000010000000000000027.gz
000000010000000000000028.00000020.backup.gz
000000010000000000000028.gz
000000010000000000000029.gz

$ pg_archivecleanup -d -x .gz `pwd`
000000010000000000000028.00000020.backup
pg_archivecleanup: keep WAL file
"/home/gsmith/pgwork/data/archivecleanup/archive/000000010000000000000028"
and later
pg_archivecleanup: removing file
"/home/gsmith/pgwork/data/archivecleanup/archive/000000010000000000000025.gz"
pg_archivecleanup: removing file
"/home/gsmith/pgwork/data/archivecleanup/archive/000000010000000000000027.gz"
pg_archivecleanup: removing file
"/home/gsmith/pgwork/data/archivecleanup/archive/000000010000000000000026.gz"
$ ls
000000010000000000000028.00000020.backup.gz
000000010000000000000028.gz
000000010000000000000029.gz

We recenty got some on-list griping that pg_standby doesn't handle
archive files that are compressed, either. Given how the job I'm
working on this week is going, I'll probably have to add that feature
next. That's actually an easier source code hack than this one, because
of how the pg_standby code modularizes the concept of a restore command.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

Em 08-02-2011 04:57, Greg Smith escreveu:
> We recenty got some on-list griping that pg_standby doesn't handle
> archive files that are compressed, either. Given how the job I'm working
> on this week is going, I'll probably have to add that feature next.
> That's actually an easier source code hack than this one, because of how
> the pg_standby code modularizes the concept of a restore command.
>
This was already proposed a few years ago [1]. I have used a modified
pg_standby with this feature for a year or so.

[1]
http://archives.postgresql.org/message-id/e4ccc24e0810222010p12bae2f4xa3a11cb2bc51bd89%40mail.gmail.com

--
Euler Taveira de Oliveira
http://www.timbira.com/

On Tue, Feb 8, 2011 at 2:57 AM, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:
> One bit of feedback I keep getting from people who archive their WAL files
> is that the fairly new pg_archivecleanup utility doesn't handle the case
> where those archives are compressed.  As the sort of users who are concerned
> about compression are also often ones with giant archives they struggle to
> cleanup, they would certainly appreciate having a bundled utility to take
> care of that.

Please add this patch to the currently open CommitFest at

https://commitfest.postgresql.org/action/commitfest_view/open

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company