Re: Looking for way to replicate master db to multiple mobile databases

Hi,
I'm looking for a way to replicate am master database to multiple (100+)
databases that are taken in to the field. Currently for each laptop we dump
and load the tables. However,there is only a small percentage of data that
changes on a frequent basis.

I've been looking around and come across pyerplica, londiste and bucardo -
the documentation on most of these is fairly sparse. It seems that Bucardo
may be the best bet - at least initially.

However, I thought I'd see if anyone is doing something similar and what
thoughts there might be out there as to a better way to accomplish this.

Thanks,
Bryan.

On Wed, Sep 23, 2009 at 11:11 AM, Bryan Montgomery <monty(at)english(dot)net> wrote:
> Hi,
> I'm looking for a way to replicate am master database to multiple (100+)
> databases that are taken in to the field. Currently for each laptop we dump
> and load the tables. However,there is only a small percentage of data that
> changes on a frequent basis.
>
> I've been looking around and come across pyerplica, londiste and bucardo -
> the documentation on most of these is fairly sparse. It seems that Bucardo
> may be the best bet - at least initially.
>
> However, I thought I'd see if anyone is doing something similar and what
> thoughts there might be out there as to a better way to accomplish this.

The problem domain you're working on is a bit different from regular
replication. Most replication solutions are made to keep two machines
that talk to each other all the time in sync. Disconnect one machine
and maybe replication will resume properly, and maybe it wont.

So, do you need the slave databases to be updatable? Do you need the
changes to go back into the master? Do you need conflict resolution?
Depending on the full fleshed out requirements, you may be stuck
writing your own solution, or re-writing one somebody already wrote.

Thanks for the reply. This is a one way push to the slaves. In theory, there
shouldn't be any conflicts .... although I wouldn't swear to that. If
there's a conflict, the master db should win. At the moment we just drop the
tables, recreate the schema and reload the tables. However, some of the
large tables literally take hours across the network, for maybe a few dozen
changes.

On Wed, Sep 23, 2009 at 4:58 PM, Scott Marlowe <scott(dot)marlowe(at)gmail(dot)com>wrote:

> On Wed, Sep 23, 2009 at 11:11 AM, Bryan Montgomery <monty(at)english(dot)net>
> wrote:
> > Hi,
> > I'm looking for a way to replicate am master database to multiple (100+)
> > databases that are taken in to the field. Currently for each laptop we
> dump
> > and load the tables. However,there is only a small percentage of data
> that
> > changes on a frequent basis.
> >
> > I've been looking around and come across pyerplica, londiste and bucardo
> -
> > the documentation on most of these is fairly sparse. It seems that
> Bucardo
> > may be the best bet - at least initially.
> >
> > However, I thought I'd see if anyone is doing something similar and what
> > thoughts there might be out there as to a better way to accomplish this.
>
> The problem domain you're working on is a bit different from regular
> replication. Most replication solutions are made to keep two machines
> that talk to each other all the time in sync. Disconnect one machine
> and maybe replication will resume properly, and maybe it wont.
>
> So, do you need the slave databases to be updatable? Do you need the
> changes to go back into the master? Do you need conflict resolution?
> Depending on the full fleshed out requirements, you may be stuck
> writing your own solution, or re-writing one somebody already wrote.
>

On Wed, Sep 23, 2009 at 06:00:03PM -0400, Bryan Montgomery wrote:
> Thanks for the reply. This is a one way push to the slaves. In theory, there
> shouldn't be any conflicts .... although I wouldn't swear to that. If
> there's a conflict, the master db should win. At the moment we just drop the
> tables, recreate the schema and reload the tables. However, some of the
> large tables literally take hours across the network, for maybe a few dozen
> changes.

Could you just replay WAL updates? I.e. have two copies of the database
on each device, one as the mirror of the "master" and one as the "live"
version. When you need to push the changes out, just push out the WAL
updates, dump the "live" version, copy the "master" into a new "live"
version and then replay the new WAL records.

Depends on how much disk space you have I guess.

--
Sam http://samason.me.uk/

Hi,

On Wed, Sep 23, 2009 at 8:11 PM, Bryan Montgomery <monty(at)english(dot)net> wrote:
> Hi,
> I'm looking for a way to replicate am master database to multiple (100+)
> databases that are taken in to the field. Currently for each laptop we dump
> and load the tables. However,there is only a small percentage of data that
> changes on a frequent basis.
>
> I've been looking around and come across pyerplica, londiste and bucardo -
> the documentation on most of these is fairly sparse. It seems that Bucardo
> may be the best bet - at least initially.

You might have a look at Mammoth Replicator (I'm a developer of it).
Although we haven't checked whether it works with hundreds of slaves,
in theory it should. The feature that can be useful to your setup is
'batched' updates, designed for slaves that are not constantly
connected to the master server. These slaves connect to the
replication server for some configurable period of time, get the new
data, and disconnect until the next attempt.

>
> However, I thought I'd see if anyone is doing something similar and what
> thoughts there might be out there as to a better way to accomplish this.
>
> Thanks,
> Bryan.

--
Alexey Klyukin wwww.commandprompt.com
The PostgreSQL Company - Command Prompt, Inc

Hi!

On Wed, Sep 23, 2009 at 10:11 AM, Bryan Montgomery <monty(at)english(dot)net> wrote:
> Hi,
> I'm looking for a way to replicate am master database to multiple (100+)
> databases that are taken in to the field. Currently for each laptop we dump
> and load the tables. However,there is only a small percentage of data that
> changes on a frequent basis.
>
> I've been looking around and come across pyerplica, londiste and bucardo -
> the documentation on most of these is fairly sparse. It seems that Bucardo
> may be the best bet - at least initially.

Bucardo is a good choice for this usage model because it was
originally designed to work over a lossy network connections.

You could issue 'kicks' for each laptop sync when you know for sure
that a laptop has got an active network connection to your master.
It's also pretty efficient with updates, only copying the current row
(that's changed) a single time, rather than multiple times if there
have been multiple changes to that row since the last time a sync
occurred.

-selena

--
http://chesnok.com/daily - me
http://endpoint.com - work

Le jeudi 24 septembre 2009, Selena Deckelmann a écrit :
> Hi!
>
> On Wed, Sep 23, 2009 at 10:11 AM, Bryan Montgomery <monty(at)english(dot)net>
wrote:
> > Hi,
> > I'm looking for a way to replicate am master database to multiple (100+)
> > databases that are taken in to the field. Currently for each laptop we
> > dump and load the tables. However,there is only a small percentage of
> > data that changes on a frequent basis.
> >
> > I've been looking around and come across pyerplica, londiste and bucardo
> > - the documentation on most of these is fairly sparse. It seems that
> > Bucardo may be the best bet - at least initially.
>
> Bucardo is a good choice for this usage model because it was
> originally designed to work over a lossy network connections.

yes, but isn't bucardo designed to 2 nodes only ?

>
> You could issue 'kicks' for each laptop sync when you know for sure
> that a laptop has got an active network connection to your master.
> It's also pretty efficient with updates, only copying the current row
> (that's changed) a single time, rather than multiple times if there
> have been multiple changes to that row since the last time a sync
> occurred.
>
> -selena
>

--
----
Cédric Villemain
Administrateur de Base de Données
Cel: +33 (0)6 74 15 56 53
http://dalibo.com - http://dalibo.org

On Thu, Sep 24, 2009 at 05:09:26PM +0200, Cédric Villemain wrote:
> > Bucardo is a good choice for this usage model because it was
> > originally designed to work over a lossy network connections.
>
> yes, but isn't bucardo designed to 2 nodes only ?

Bucardo's multi-master replication works only between two hosts, unless
perhaps you can assure that only certain primary keys will be updated on
certain hosts, or do some other trickery. Syncing from one master to multiple
slaves is straightforward, if all you need is master->slave.

--
Joshua Tolley / eggyknap
End Point Corporation
http://www.endpoint.com

On Thu, Sep 24, 2009 at 8:09 AM, Cédric Villemain
<cedric(dot)villemain(at)dalibo(dot)com> wrote:
> Le jeudi 24 septembre 2009, Selena Deckelmann a écrit :
>> Hi!
>>
>> On Wed, Sep 23, 2009 at 10:11 AM, Bryan Montgomery <monty(at)english(dot)net>
> wrote:
>> > Hi,
>> > I'm looking for a way to replicate am master database to multiple (100+)
>> > databases that are taken in to the field. Currently for each laptop we
>> > dump and load the tables. However,there is only a small percentage of
>> > data that changes on a frequent basis.
>> >
>> > I've been looking around and come across pyerplica, londiste and bucardo
>> > - the documentation on most of these is fairly sparse. It seems that
>> > Bucardo may be the best bet - at least initially.
>>
>> Bucardo is a good choice for this usage model because it was
>> originally designed to work over a lossy network connections.
>
> yes, but isn't bucardo designed to 2 nodes only ?

No, definitely not! You can replicate to any number of systems. And
you can group them in whatever groups you'd like. Multi-master (as
Joshua said) only works between two nodes, but master->slave can be
from a master, to any number of slaves.

--
http://chesnok.com/daily - me
http://endpoint.com - work

On 25/09/2009, at 12:50 PM, Selena Deckelmann wrote:

> On Thu, Sep 24, 2009 at 8:09 AM, Cédric Villemain
> <cedric(dot)villemain(at)dalibo(dot)com> wrote:
>> Le jeudi 24 septembre 2009, Selena Deckelmann a écrit :
>>> Hi!
>>>
>>> On Wed, Sep 23, 2009 at 10:11 AM, Bryan Montgomery <monty(at)english(dot)net
>>> >
>> wrote:
>>>> Hi,
>>>> I'm looking for a way to replicate am master database to multiple
>>>> (100+)
>>>> databases that are taken in to the field. Currently for each
>>>> laptop we
>>>> dump and load the tables. However,there is only a small
>>>> percentage of
>>>> data that changes on a frequent basis.
>>>>
>>>> I've been looking around and come across pyerplica, londiste and
>>>> bucardo
>>>> - the documentation on most of these is fairly sparse. It seems
>>>> that
>>>> Bucardo may be the best bet - at least initially.
>>>
>>> Bucardo is a good choice for this usage model because it was
>>> originally designed to work over a lossy network connections.
>>
>> yes, but isn't bucardo designed to 2 nodes only ?
>
> No, definitely not! You can replicate to any number of systems. And
> you can group them in whatever groups you'd like. Multi-master (as
> Joshua said) only works between two nodes, but master->slave can be
> from a master, to any number of slaves.
>
>
I use bucardo extensively across multiple sites and with complex
replication requirements. It does a great job.
regards
Grant

Le vendredi 25 septembre 2009, Selena Deckelmann a écrit :
> On Thu, Sep 24, 2009 at 8:09 AM, Cédric Villemain
>
> <cedric(dot)villemain(at)dalibo(dot)com> wrote:
> > Le jeudi 24 septembre 2009, Selena Deckelmann a écrit :
> >> Hi!
> >>
> >> On Wed, Sep 23, 2009 at 10:11 AM, Bryan Montgomery <monty(at)english(dot)net>
> >
> > wrote:
> >> > Hi,
> >> > I'm looking for a way to replicate am master database to multiple
> >> > (100+) databases that are taken in to the field. Currently for each
> >> > laptop we dump and load the tables. However,there is only a small
> >> > percentage of data that changes on a frequent basis.
> >> >
> >> > I've been looking around and come across pyerplica, londiste and
> >> > bucardo - the documentation on most of these is fairly sparse. It
> >> > seems that Bucardo may be the best bet - at least initially.
> >>
> >> Bucardo is a good choice for this usage model because it was
> >> originally designed to work over a lossy network connections.
> >
> > yes, but isn't bucardo designed to 2 nodes only ?
>
> No, definitely not! You can replicate to any number of systems. And
> you can group them in whatever groups you'd like. Multi-master (as
> Joshua said) only works between two nodes, but master->slave can be
> from a master, to any number of slaves.
>

Ah! thank you for clarifying that.

----
Cédric Villemain
Administrateur de Base de Données
Cel: +33 (0)6 74 15 56 53
http://dalibo.com - http://dalibo.org