New option in pg_basebackup to exclude pg_log files during base backup

Hi all,

Following the discussion in message id - CAHGQGwFFMOr4EcugWHZpAaPYQbsEKDg66VmJ1rveJ6Z-EgaqAg(at)mail(dot)gmail(dot)com<mailto:CAHGQGwFFMOr4EcugWHZpAaPYQbsEKDg66VmJ1rveJ6Z-EgaqAg(at)mail(dot)gmail(dot)com> , I have developed the patch which gives option to user to exclude pg_log directory contents in pg_basebackup.

[Current situation]
During pg_basebackup, all files in pg_log directory will be copied to new backup directory.

[Design]
- Added new non-mandatory option "-S/--skip-log-dir" to pg_basebackup .
- If "skip-log-dir" is specified in pg_basebackup command, then in basebackup, exclude copying log files from standard "pg_log" directory and any other directory specified in Log_directory guc variable. (Still empty folder "pg_log"/$Log_directory will be created)
- In case, pg_log/$Log_directory is symbolic link, then an empty folder will be created

[Advantage]
It gives an option to user to avoid copying of large log files if they doesn't wish to and hence can save memory space.

Attached the patch.

Thanks & Regards,
Vaishnavi
Fujitsu Australia

On Wed, Apr 9, 2014 at 2:06 AM, Prabakaran, Vaishnavi <
vaishnavip(at)fast(dot)au(dot)fujitsu(dot)com> wrote:

> Hi all,
>
>
>
> Following the discussion in message id -
> CAHGQGwFFMOr4EcugWHZpAaPYQbsEKDg66VmJ1rveJ6Z-EgaqAg(at)mail(dot)gmail(dot)com , I
> have developed the patch which gives option to user to exclude pg_log
> directory contents in pg_basebackup.
>
>
>
> [Current situation]
>
> During pg_basebackup, all files in pg_log directory will be copied to new
> backup directory.
>
>
>
> [Design]
>
> - Added new non-mandatory option "-S/--skip-log-dir" to pg_basebackup .
>
> - If "skip-log-dir" is specified in pg_basebackup command, then in
> basebackup, exclude copying log files from standard "pg_log" directory and
> any other directory specified in Log_directory guc variable. (Still empty
> folder "pg_log"/$Log_directory will be created)
>
> - In case, pg_log/$Log_directory is symbolic link, then an empty folder
> will be created
>
>
>
> [Advantage]
>
> It gives an option to user to avoid copying of large log files if they
> doesn't wish to and hence can save memory space.
>
>
>
>
>
While pg_log is definitely the most common one being the default on many
platforms, we'll still be missing other ones. Should we really hardcode it,
or should we somehow derive it from the settings for log_directory instead?

As a more general discussion, is this something we might want to expose as
a more general facility rather than hardcode it to the log directory only?
And is it perhaps something we'd rather have configured at the server than
specified in pg_basebackup - like a guc saying which directories should
always be excluded from a basebackup? So you don't have to remember it
every time?

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

Magnus Hagander wrote:

> While pg_log is definitely the most common one being the default on many
> platforms, we'll still be missing other ones. Should we really hardcode it,
> or should we somehow derive it from the settings for log_directory instead?
>
> As a more general discussion, is this something we might want to expose as
> a more general facility rather than hardcode it to the log directory only?
> And is it perhaps something we'd rather have configured at the server than
> specified in pg_basebackup - like a guc saying which directories should
> always be excluded from a basebackup? So you don't have to remember it
> every time?

So it'd be an array, and by default you'd have something like:
basebackup_skip_path = $log_directory
?

Maybe use it to skip backup labels by default as well.
basebackup_skip_path = $log_directory, $backup_label_files

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Wed, Apr 9, 2014 at 4:55 PM, Alvaro Herrera <alvherre(at)2ndquadrant(dot)com>wrote:

> Magnus Hagander wrote:
>
> > While pg_log is definitely the most common one being the default on many
> > platforms, we'll still be missing other ones. Should we really hardcode
> it,
> > or should we somehow derive it from the settings for log_directory
> instead?
> >
> > As a more general discussion, is this something we might want to expose
> as
> > a more general facility rather than hardcode it to the log directory
> only?
> > And is it perhaps something we'd rather have configured at the server
> than
> > specified in pg_basebackup - like a guc saying which directories should
> > always be excluded from a basebackup? So you don't have to remember it
> > every time?
>
> So it'd be an array, and by default you'd have something like:
> basebackup_skip_path = $log_directory
> ?
>
> Maybe use it to skip backup labels by default as well.
> basebackup_skip_path = $log_directory, $backup_label_files
>

I hadn't considered any details, but yes, someting along that line. And
then you could also include arbitrary filenames or directories should you
want. E.g. if you use the data directory to store your torrents or
something.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

Magnus Hagander wrote:
> On Wed, Apr 9, 2014 at 4:55 PM, Alvaro Herrera <alvherre(at)2ndquadrant(dot)com>wrote:

> > So it'd be an array, and by default you'd have something like:
> > basebackup_skip_path = $log_directory
> > ?
> >
> > Maybe use it to skip backup labels by default as well.
> > basebackup_skip_path = $log_directory, $backup_label_files
> >
>
> I hadn't considered any details, but yes, someting along that line. And
> then you could also include arbitrary filenames or directories should you
> want. E.g. if you use the data directory to store your torrents or
> something.

Man, that's a great idea. Database servers have lots of diskspace in
that partition, so it should work really well. Thanks!

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Thursday, Apr 10,2014 at 1:15Am, Álvaro Herrera wrote:
>Magnus Hagander wrote:
>>On Wed, Apr 9, 2014 at 4:55 PM, Alvaro Herrera <alvherre(at)2ndquadrant(dot)com>wrote:

>> > So it'd be an array, and by default you'd have something like:
>> > basebackup_skip_path = $log_directory ?
>> >
>> > Maybe use it to skip backup labels by default as well.
>> > basebackup_skip_path = $log_directory, $backup_label_files
>> >
>>
>> I hadn't considered any details, but yes, someting along that line.
>> And then you could also include arbitrary filenames or directories
>> should you want. E.g. if you use the data directory to store your
>> torrents or something.

>Man, that's a great idea. Database servers have lots of diskspace in that partition, so it should work really well. Thanks!

Yes, It sounds like a good idea. I will look into this and start working in sometime.

Thanks & Regards,
Vaishnavi
Fujitsu Australia

On 4/9/14, 10:57 AM, Magnus Hagander wrote:
> So it'd be an array, and by default you'd have something like:
> basebackup_skip_path = $log_directory
> ?
>
> Maybe use it to skip backup labels by default as well.
> basebackup_skip_path = $log_directory, $backup_label_files
>
>
> I hadn't considered any details, but yes, someting along that line. And
> then you could also include arbitrary filenames or directories should
> you want.

What are the use cases for excluding anything else?

pg_basebackup ought to have some intelligence about what files are
appropriate to include or exclude, depending on what the user is trying
to do. It shouldn't become a general file copying tool.