Re: "snapshot too large" error when initializing logical replication (9.4)

I sometimes get the error "snapshot too large" from my logical
replication walsender process when in response to a CREATE_REPLICATION_SLOT.

This is in SnapBuildExportSnapshot in snapbuild.c

newxcnt is 212 at that point

I have max_connections = 200

procArray->maxProcs=212

Should we be testing
newxcnt > GetMaxSnapshotXidCount()

instead of
newxcnt >= GetMaxSnapshotXidCount()

Hi,

On 2014-10-25 18:09:36 -0400, Steve Singer wrote:
> I sometimes get the error "snapshot too large" from my logical replication
> walsender process when in response to a CREATE_REPLICATION_SLOT.

Yes. That's possible if 'too much' was going on until a consistent point
was reached. I think we can just use a much larger size for the array
if necessary.

I've attached patch for this. Could you try whether that helps? I don't
have a testcase handy that reproduces the problem.

> This is in SnapBuildExportSnapshot in snapbuild.c
>
> newxcnt is 212 at that point
>
> I have max_connections = 200
>
> procArray->maxProcs=212
>
> Should we be testing
> newxcnt > GetMaxSnapshotXidCount()
>
> instead of
> newxcnt >= GetMaxSnapshotXidCount()

It actually looks correct to me new - newxcnt is used as an offset into
an array of size GetMaxSnapshotXidCount().

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 10/28/2014 01:27 PM, Andres Freund wrote:
> Hi,
>
> On 2014-10-25 18:09:36 -0400, Steve Singer wrote:
>> I sometimes get the error "snapshot too large" from my logical replication
>> walsender process when in response to a CREATE_REPLICATION_SLOT.
> Yes. That's possible if 'too much' was going on until a consistent point
> was reached. I think we can just use a much larger size for the array
> if necessary.
>
> I've attached patch for this. Could you try whether that helps? I don't
> have a testcase handy that reproduces the problem.

This patch seems to fix things.
I've done numerous runs of the test with I was doing before with your
patch applied and don't seem to be having this issue anymore.

>> This is in SnapBuildExportSnapshot in snapbuild.c
>>
>> newxcnt is 212 at that point
>>
>> I have max_connections = 200
>>
>> procArray->maxProcs=212
>>
>> Should we be testing
>> newxcnt > GetMaxSnapshotXidCount()
>>
>> instead of
>> newxcnt >= GetMaxSnapshotXidCount()
> It actually looks correct to me new - newxcnt is used as an offset into
> an array of size GetMaxSnapshotXidCount().
>
> Greetings,
>
> Andres Freund
>
>
>

Andres Freund wrote:
> Hi,
>
> On 2014-10-25 18:09:36 -0400, Steve Singer wrote:
> > I sometimes get the error "snapshot too large" from my logical replication
> > walsender process when in response to a CREATE_REPLICATION_SLOT.
>
> Yes. That's possible if 'too much' was going on until a consistent point
> was reached. I think we can just use a much larger size for the array
> if necessary.
>
> I've attached patch for this. Could you try whether that helps? I don't
> have a testcase handy that reproduces the problem.

You haven't pushed this, have you?

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 2014-11-17 11:51:54 -0300, Alvaro Herrera wrote:
> Andres Freund wrote:
> > Hi,
> >
> > On 2014-10-25 18:09:36 -0400, Steve Singer wrote:
> > > I sometimes get the error "snapshot too large" from my logical replication
> > > walsender process when in response to a CREATE_REPLICATION_SLOT.
> >
> > Yes. That's possible if 'too much' was going on until a consistent point
> > was reached. I think we can just use a much larger size for the array
> > if necessary.
> >
> > I've attached patch for this. Could you try whether that helps? I don't
> > have a testcase handy that reproduces the problem.
>
> You haven't pushed this, have you?

No, but it's on my todo list.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Mon, Nov 17, 2014 at 03:52:38PM +0100, Andres Freund wrote:
> On 2014-11-17 11:51:54 -0300, Alvaro Herrera wrote:
> > Andres Freund wrote:
> > > Hi,
> > >
> > > On 2014-10-25 18:09:36 -0400, Steve Singer wrote:
> > > > I sometimes get the error "snapshot too large" from my logical replication
> > > > walsender process when in response to a CREATE_REPLICATION_SLOT.
> > >
> > > Yes. That's possible if 'too much' was going on until a consistent point
> > > was reached. I think we can just use a much larger size for the array
> > > if necessary.
> > >
> > > I've attached patch for this. Could you try whether that helps? I don't
> > > have a testcase handy that reproduces the problem.
> >
> > You haven't pushed this, have you?
>
> No, but it's on my todo list.

Uh, where are we on this?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +

On 2015-03-20 08:41:38 -0400, Bruce Momjian wrote:
> On Mon, Nov 17, 2014 at 03:52:38PM +0100, Andres Freund wrote:
> > On 2014-11-17 11:51:54 -0300, Alvaro Herrera wrote:
> > > Andres Freund wrote:
> > > > Hi,
> > > >
> > > > On 2014-10-25 18:09:36 -0400, Steve Singer wrote:
> > > > > I sometimes get the error "snapshot too large" from my logical replication
> > > > > walsender process when in response to a CREATE_REPLICATION_SLOT.
> > > >
> > > > Yes. That's possible if 'too much' was going on until a consistent point
> > > > was reached. I think we can just use a much larger size for the array
> > > > if necessary.
> > > >
> > > > I've attached patch for this. Could you try whether that helps? I don't
> > > > have a testcase handy that reproduces the problem.
> > >
> > > You haven't pushed this, have you?
> >
> > No, but it's on my todo list.
>
> Uh, where are we on this?

Nowhere, unfortunately. The fix I'd developed isn't actually
correct. The problem is that we can only import a transaction with fewer
than GetMaxSnapshotXidCount() transactions atm. The only reason it
worked in Steve's case is because he probably wasn't importin gthe
snapshot in the test case...

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services