Re: WIP: Access method extendability

On Tue, Oct 28, 2014 at 8:04 PM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:

> On 28 October 2014 14:22, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
>
> > Or put it another way, it will be easier to write new index AMs
> > because we'll be able to skip the WAL part until we know we want it.
>
> To be clear: I am suggesting you do *less* work, not more.
>
> By allowing AMs to avoid writing WAL we get
> * higher performance unlogged indexes
> * we get fewer bugs in early days of new AMs
> * writers of new AMs are OK to avoid majority of hard work and hard testing
>
> So overall, we get new AMs working faster because we can skip writing
> the WAL code until we are certain the new AM code is useful and bug
> free.
>
> For example, if GIN had avoided implementing WAL it would have been
> easier to change on-disk representation.

Major problem of changing on-disk representation is that we have to support
both binary formats because of pg_upgrade. This problem is even burdened
with WAL, because WAL record redo function also have to support both
formats. However, it's also quite independent of WAL.

Having access methods as extensions can significantly improves situations
here. Imagine, GIN was an extension. One day we decide to change its binary
format. Then we can issue new extension, GIN2 for instance. User can
upgrade from GIN to GIN2 in following steps:

1. CREATE EXTENSION gin2;
2. CREATE INDEX CONCURRENTLY [new_index] USING gin2 ([index_def]);
3. DROP INDEX CONCURRENTLY [old_index];
4. DROP EXTENSION gin;

No need to write and debug the code which reads both old and new binary
format. For sure, we need to support old GIN extension for some time. But,
we have to support old in-core versions too.

Unfortunately, I didn't mention this in the first post because I consider
this as a serious argument for extensible AMs.

Also, I'm not sure that many users have enough of courage to use unlogged
AMs. Absence of durability is only half of trouble, another half is lack of
streaming replication. I think if we have unlogged GIN then external
indexing engines would be used by majority of users instead of GIN.

------
With best regards,
Alexander Korotkov.