What's planned for 7.5?

Hi,

Is this all that's planned for 7.5? (based on current TODO list)

-Change factorial to return a numeric (Gavin)
-COMMENT ON [ CAST | CONVERSION | OPERATOR CLASS | LARGE OBJECT | LANGUAGE ]
(Christopher)
-Have psql \dn show only visible temp schemas using current_schemas()
-Have psql '\i ~/<tab><tab>' actually load files it displays from home dir
-Allow psql \du to show groups, and add \dg for groups
-Allow pg_dump to dump CREATE CONVERSION (Christopher)
-Use dependency information to dump data in proper order
-Use background process to write dirty shared buffers to disk

Thanks

__________________________________
Do you Yahoo!?
Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes
http://hotjobs.sweepstakes.yahoo.com/signingbonus

Native Win32 is planned for it (whether it makes it or not is another
question, but it is the goal) ...

On Mon, 12 Jan 2004, ow wrote:

> Hi,
>
> Is this all that's planned for 7.5? (based on current TODO list)
>
> -Change factorial to return a numeric (Gavin)
> -COMMENT ON [ CAST | CONVERSION | OPERATOR CLASS | LARGE OBJECT | LANGUAGE ]
> (Christopher)
> -Have psql \dn show only visible temp schemas using current_schemas()
> -Have psql '\i ~/<tab><tab>' actually load files it displays from home dir
> -Allow psql \du to show groups, and add \dg for groups
> -Allow pg_dump to dump CREATE CONVERSION (Christopher)
> -Use dependency information to dump data in proper order
> -Use background process to write dirty shared buffers to disk
>
> Thanks
>
>
>
>
>
>
> __________________________________
> Do you Yahoo!?
> Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes
> http://hotjobs.sweepstakes.yahoo.com/signingbonus
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
>
> http://archives.postgresql.org
>

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy(at)hub(dot)org Yahoo!: yscrappy ICQ: 7615664

Mensaje citado por "Marc G. Fournier" <scrappy(at)postgresql(dot)org>:

>
> Native Win32 is planned for it (whether it makes it or not is another
> question, but it is the goal) ...

Replication wasn't another BIG one?

--
select 'mmarques' || '@' || 'unl.edu.ar' AS email;
---------------------------------------------------------
Martín Marqués | Programador, DBA
Centro de Telemática | Administrador
Universidad Nacional
del Litoral
---------------------------------------------------------

On Mon, 12 Jan 2004, Martin Marques wrote:

> Mensaje citado por "Marc G. Fournier" <scrappy(at)postgresql(dot)org>:
>
> >
> > Native Win32 is planned for it (whether it makes it or not is another
> > question, but it is the goal) ...
>
> Replication wasn't another BIG one?

Actually, I think it was PITR (Point in Time Recovery). Of course, since
that works by replaying log files into the database, it's a pretty good
way of setting up replication, which would likely follow a release or so
after PITR was implemented.

OTOH, there are some replication solutions already available, just no as
part of the core.

Mensaje citado por ow <oneway_111(at)yahoo(dot)com>:

> Hi,
>
> Is this all that's planned for 7.5? (based on current TODO list)
>
> -Change factorial to return a numeric (Gavin)
> -COMMENT ON [ CAST | CONVERSION | OPERATOR CLASS | LARGE OBJECT | LANGUAGE ]
> (Christopher)
> -Have psql \dn show only visible temp schemas using current_schemas()
> -Have psql '\i ~/<tab><tab>' actually load files it displays from home dir
> -Allow psql \du to show groups, and add \dg for groups
> -Allow pg_dump to dump CREATE CONVERSION (Christopher)
> -Use dependency information to dump data in proper order
> -Use background process to write dirty shared buffers to disk

For what I have just seen in Robert Treats "PostgreSQL Weekly News" most of
these issues are already solved in the CVS, and I guess that in a bit less than
a year that's left for the release of 7.5 many new things will appear.

Anyway, from http://developer.postgresql.org/todo.php I see in the URGENT part
this:

# Add replication of distributed databases [replication]

* Automatic failover
* Load balancing
* Master/slave replication
* Multi-master replication
* Partition data across servers
* Queries across databases or servers (two-phase commit)
* Allow replication over unreliable or non-persistent links
* http://gborg.postgresql.org/project/pgreplication/projdisplay.php

# Point-in-time data recovery using backup and write-ahead log
# Create native Win32 port,
* http://momjian.postgresql.org/main/writings/pgsql/win32.html

The first isn't new, but as for what I know, it's a new aproach that Jan has for
adding replication to the server.

The other 2 didn't get in the 7.4 release.

--
select 'mmarques' || '@' || 'unl.edu.ar' AS email;
---------------------------------------------------------
Martín Marqués | Programador, DBA
Centro de Telemática | Administrador
Universidad Nacional
del Litoral
---------------------------------------------------------

On Mon, 12 Jan 2004, Martin Marques wrote:

> Mensaje citado por ow <oneway_111(at)yahoo(dot)com>:
>
> > Hi,
> >
> > Is this all that's planned for 7.5? (based on current TODO list)
> >
> > -Change factorial to return a numeric (Gavin)
> > -COMMENT ON [ CAST | CONVERSION | OPERATOR CLASS | LARGE OBJECT | LANGUAGE ]
> > (Christopher)
> > -Have psql \dn show only visible temp schemas using current_schemas()
> > -Have psql '\i ~/<tab><tab>' actually load files it displays from home dir
> > -Allow psql \du to show groups, and add \dg for groups
> > -Allow pg_dump to dump CREATE CONVERSION (Christopher)
> > -Use dependency information to dump data in proper order
> > -Use background process to write dirty shared buffers to disk
>
> For what I have just seen in Robert Treats "PostgreSQL Weekly News" most of
> these issues are already solved in the CVS, and I guess that in a bit less than
> a year that's left for the release of 7.5 many new things will appear.
>
> Anyway, from http://developer.postgresql.org/todo.php I see in the URGENT part
> this:
>
> # Add replication of distributed databases [replication]
>
> * Automatic failover
> * Load balancing
> * Master/slave replication
> * Multi-master replication
> * Partition data across servers
> * Queries across databases or servers (two-phase commit)
> * Allow replication over unreliable or non-persistent links
> * http://gborg.postgresql.org/project/pgreplication/projdisplay.php
>
> # Point-in-time data recovery using backup and write-ahead log
> # Create native Win32 port,
> * http://momjian.postgresql.org/main/writings/pgsql/win32.html
>
> The first isn't new, but as for what I know, it's a new aproach that Jan has for
> adding replication to the server.

Nope, pgreplication has nothing to do with Jan ... he's working on
Slonik-1 ...

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy(at)hub(dot)org Yahoo!: yscrappy ICQ: 7615664

martin(at)bugs(dot)unl(dot)edu(dot)ar (Martin Marques) writes:
> Mensaje citado por "Marc G. Fournier" <scrappy(at)postgresql(dot)org>:
>> Native Win32 is planned for it (whether it makes it or not is another
>> question, but it is the goal) ...
>
> Replication wasn't another BIG one?

Well, there are two "replication things" going on:

1. The eRserv Java server software is already available on gBorg.

It isn't likely to go into the software distribution any time
soon because of its dependence on Java. It really is an extra
add-on.

2. Jan Wieck's work on SLONY-1

Many are keen to see the code, but it's not out yet. And it is
not self-evident that it will necessarily be ready to release in
conjunction with 7.5 (Not to say it _can't_ happen, but just that
we'll see the code when we see it...)

It is planned to be implemented in C, so it would presumably be
more suitable for inclusion in the main code than eRserv. But it
stands as "hope," and not certainty.

And none of this is ...

3. PITR (Point In Time Recovery).

Where a characteristic "hope" would be to archive WAL files and
use them to bring a failed DB up to date.

Jan has plans to do something similar but _much_ less
obscure/opaque with SLONY-1. We'll see what happens...

Another fairly important item not mentioned: Nested Transactions.
--
let name="cbbrowne" and tld="libertyrms.info" in String.concat "@" [name;tld];;
<http://dev6.int.libertyrms.com/>
Christopher Browne
(416) 646 3304 x124 (land)

ow <oneway_111(at)yahoo(dot)com> writes:
> Is this all that's planned for 7.5? (based on current TODO list)

If you think the TODO list has *anything* to do with release planning,
you fundamentally misunderstand the way things are done around here.

The TODO list is a list of things that are widely agreed to be problems
(though sometimes it only means "Bruce thinks this is a problem") and
therefore will probably get worked on at some point in the future.

What actually gets done for 7.5 will depend on which itches bother which
developers enough to get scratched in the next few months. We do not
have any central planning.

It is a safe bet that 7.5 will include some things mentioned in the
present TODO, as well as many things not mentioned in it; and that
many items mentioned in the present TODO will still remain undone.

regards, tom lane

Any chance we'll see the VACUUM delay patch (throttle) get into 7.5? I only
had the chance to try the first patch by Tom Lane and it was very good
already. I was hoping it gets into 7.4.1 but it didn't. :-(

I really need the VACUUM delay patch because my servers are begging to die
every time VACUUM runs. Please let it be in 7.5.

Thanks.

Stephen

"Tom Lane" <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote in message
news:5658(dot)1073947557(at)sss(dot)pgh(dot)pa(dot)us(dot)(dot)(dot)
> ow <oneway_111(at)yahoo(dot)com> writes:
> > Is this all that's planned for 7.5? (based on current TODO list)
>
> If you think the TODO list has *anything* to do with release planning,
> you fundamentally misunderstand the way things are done around here.
>
> The TODO list is a list of things that are widely agreed to be problems
> (though sometimes it only means "Bruce thinks this is a problem") and
> therefore will probably get worked on at some point in the future.
>
> What actually gets done for 7.5 will depend on which itches bother which
> developers enough to get scratched in the next few months. We do not
> have any central planning.
>
> It is a safe bet that 7.5 will include some things mentioned in the
> present TODO, as well as many things not mentioned in it; and that
> many items mentioned in the present TODO will still remain undone.
>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
>
> http://archives.postgresql.org
>

"Stephen" <jleelim(at)xxxxxxx(dot)com> writes:
> Any chance we'll see the VACUUM delay patch (throttle) get into 7.5? I only
> had the chance to try the first patch by Tom Lane and it was very good
> already. I was hoping it gets into 7.4.1 but it didn't. :-(
>
> I really need the VACUUM delay patch because my servers are begging to die
> every time VACUUM runs. Please let it be in 7.5.

The hope, in 7.5, is to have ARC, which is the "super-duper-duper"
version, working.

The delay patch is simple enough to add in if you need it :-).
--
let name="cbbrowne" and tld="libertyrms.info" in String.concat "@" [name;tld];;
<http://dev6.int.libertyrms.com/>
Christopher Browne
(416) 646 3304 x124 (land)

Christopher Browne <cbbrowne(at)libertyrms(dot)info> writes:
> "Stephen" <jleelim(at)xxxxxxx(dot)com> writes:
>> Any chance we'll see the VACUUM delay patch (throttle) get into 7.5?

> The hope, in 7.5, is to have ARC, which is the "super-duper-duper"
> version, working.

Actually, I'm not sure that ARC should be considered to supersede the
usefulness of a per-page delay in VACUUM. ARC should prevent VACUUM
from trashing the contents of Postgres' shared buffer arena, but it
won't do much of anything to prevent VACUUM from trashing the kernel
buffer contents. And it definitely won't do anything to help if the
real problem is that you're short of disk bandwidth and VACUUM's extra
I/O demand pushes your total load over the knee of the response-time
curve. What you need then is a throttle.

The original patch I posted was incomplete for a number of reasons,
but I think it may still be worth working on. Jan, any comments?

regards, tom lane

Tom Lane wrote:
> Christopher Browne <cbbrowne(at)libertyrms(dot)info> writes:
>> "Stephen" <jleelim(at)xxxxxxx(dot)com> writes:
>>> Any chance we'll see the VACUUM delay patch (throttle) get into 7.5?
>
>> The hope, in 7.5, is to have ARC, which is the "super-duper-duper"
>> version, working.
>
> Actually, I'm not sure that ARC should be considered to supersede the
> usefulness of a per-page delay in VACUUM. ARC should prevent VACUUM
> from trashing the contents of Postgres' shared buffer arena, but it
> won't do much of anything to prevent VACUUM from trashing the kernel
> buffer contents. And it definitely won't do anything to help if the
> real problem is that you're short of disk bandwidth and VACUUM's extra
> I/O demand pushes your total load over the knee of the response-time
> curve. What you need then is a throttle.
>
> The original patch I posted was incomplete for a number of reasons,
> but I think it may still be worth working on. Jan, any comments?

I agree that there is considerable value in IO distribution. As such I
already have the basics of the Background Writer in there.

I left out the vacuum delay since I thought it was good enough to proove
that there is low hanging fruit, but that it was far from what I'd call
a solution. Ideally Vacuum would coordinate it's IO not only against
some GUC variables, but also against the BGWriter+BufStrategy combo.

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck(at)Yahoo(dot)com #

The vacuum delay patch is not the ideal solution but it worked like a charm
on my servers. I really need the vacuum delay patch or a better solution in
7.5. I'm getting millions of requests a month and running VACUUM without the
patch makes PostgreSQL useless for many consecutive hours. Not quite the
24/7 system I was hopping for. :-(

Unfortunately, it's rather difficult to patch so many machines as my entire
system runs on Redhat RPMs. I'm really hopping to see a solution to this
VACUUM problem in 7.5. I've been waiting for this fix for over 3 years and
now it's almost there.

Will this problem get addressed in the not so official TODO list?

Thanks and keep up the good work!

Stephen

"Jan Wieck" <JanWieck(at)Yahoo(dot)com> wrote in message
news:400544E3(dot)8030709(at)Yahoo(dot)com(dot)(dot)(dot)
> Tom Lane wrote:
> > Christopher Browne <cbbrowne(at)libertyrms(dot)info> writes:
> >> "Stephen" <jleelim(at)xxxxxxx(dot)com> writes:
> >>> Any chance we'll see the VACUUM delay patch (throttle) get into 7.5?
> >
> >> The hope, in 7.5, is to have ARC, which is the "super-duper-duper"
> >> version, working.
> >
> > Actually, I'm not sure that ARC should be considered to supersede the
> > usefulness of a per-page delay in VACUUM. ARC should prevent VACUUM
> > from trashing the contents of Postgres' shared buffer arena, but it
> > won't do much of anything to prevent VACUUM from trashing the kernel
> > buffer contents. And it definitely won't do anything to help if the
> > real problem is that you're short of disk bandwidth and VACUUM's extra
> > I/O demand pushes your total load over the knee of the response-time
> > curve. What you need then is a throttle.
> >
> > The original patch I posted was incomplete for a number of reasons,
> > but I think it may still be worth working on. Jan, any comments?
>
> I agree that there is considerable value in IO distribution. As such I
> already have the basics of the Background Writer in there.
>
> I left out the vacuum delay since I thought it was good enough to proove

> that there is low hanging fruit, but that it was far from what I'd call
> a solution. Ideally Vacuum would coordinate it's IO not only against
> some GUC variables, but also against the BGWriter+BufStrategy combo.
>
>
> Jan
>
> --
> #======================================================================#
> # It's easier to get forgiveness for being wrong than for being right. #
> # Let's break this rule - forgive me. #
> #================================================== JanWieck(at)Yahoo(dot)com #
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
>
> http://archives.postgresql.org
>

Christopher Browne wrote:

> 2. Jan Wieck's work on SLONY-1
>
> Many are keen to see the code, but it's not out yet. And it is
> not self-evident that it will necessarily be ready to release in
> conjunction with 7.5 (Not to say it _can't_ happen, but just that
> we'll see the code when we see it...)
>
> It is planned to be implemented in C, so it would presumably be
> more suitable for inclusion in the main code than eRserv. But it
> stands as "hope," and not certainty.

I've just made one major step backward on that. Discovered that my first
thread model was deadlock prone, so I better throw that away instead of
building a lot of code on top of it.

What I currently do is documenting the Slony-I ERD and the new thread
model I have in mind. This will be a document in work, but I plan to
have something readable by the end of this week, latest mid next week. I
will create a mailing list for Slony-I on gborg so we can start
discussing the implementation details.

About the inclusion of a replication solution into the core distributon.
The much I personally would be proud to see this one added, as a CORE
member I do not see any of the replication "things" fit. All of them,
and neither Slony-I nor Slony-II will be any different here, have pros
and cons, none is the "one size that fits all" magic solution. To select
one of the replication projects that high above all the others that it
will be added to the core distribution, it should be really outstanding
and general purpose. I think that Slony in the end will be outstanding,
but only for what it was designed for.

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck(at)Yahoo(dot)com #

Stephen wrote:

> The vacuum delay patch is not the ideal solution but it worked like a charm
> on my servers. I really need the vacuum delay patch or a better solution in
> 7.5. I'm getting millions of requests a month and running VACUUM without the
> patch makes PostgreSQL useless for many consecutive hours. Not quite the
> 24/7 system I was hopping for. :-(
>
> Unfortunately, it's rather difficult to patch so many machines as my entire
> system runs on Redhat RPMs. I'm really hopping to see a solution to this
> VACUUM problem in 7.5. I've been waiting for this fix for over 3 years and
> now it's almost there.
>
> Will this problem get addressed in the not so official TODO list?

Well, I had a few different "versions" of vacuum delay stuff out as
patches, together with ARC and the beginnings of the background writer.
Instead of getting some numbers on those, the whole discussion got stuck
in differences about how we actually let the background writer tell the
kernel "do something" ... the whole sync(), fsync(), fdatasync(),
fadvise() discussion.

I don't have the time to make enough different attempts to find the one
that pleases all. My argument still is that all this IO throttling and
IO optimizing is mainly needed for dedicated servers, because I think
that if you still run multiple services on one box you're not really in
trouble yet. So in the first round a configurable sync() approach would
do. So far nobody even agreed to that.

I currently have better to do. We do not have a big IO problem, we have
other problems, and I spend my time on solving them. If someone wants to
pick up the IO throttle problem, I am allways here to help, but I will
not waste my time with making patches nobody even gives a try.

>
> Thanks and keep up the good work!

Sorry for the venting, but I needed that out.

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck(at)Yahoo(dot)com #

On Mon, Jan 19, 2004 at 08:12:28AM -0500, Jan Wieck wrote:
> and cons, none is the "one size that fits all" magic solution. To select

Does anyone realy believe that there can be a one size fits all
solution? Heck, even Oracle and IBM offer a couple of different
systems, depending on what you need. (That also suggests that any
replication system need not always be shipped with the "basic"
distribution, but could instead be integrated into a larger,
postgresql_plus_enterprise_features.tgz or something like that.)

A

--
Andrew Sullivan | ajs(at)crankycanuck(dot)ca
I remember when computers were frustrating because they *did* exactly what
you told them to. That actually seems sort of quaint now.
--J.D. Baldwin

On Mon, 2004-01-19 at 08:37, Jan Wieck wrote:

> but I will not waste my time with making patches nobody even gives a try.

I downloaded and tested your patches. I just didn't get results get
results that were put together well enough to present to the group. I
hope this doesn't fall by the wayside, it is IMHO, on of the critical
problems that needs to be solved.

> Sorry for the venting, but I needed that out.

I understand. I'm sorry there wasn't more feedback as a result of your
work.

Andrew Sullivan wrote:
> On Mon, Jan 19, 2004 at 08:12:28AM -0500, Jan Wieck wrote:
>
>>and cons, none is the "one size that fits all" magic solution. To select
>
> Does anyone realy believe that there can be a one size fits all
> solution? Heck, even Oracle and IBM offer a couple of different
> systems, depending on what you need. (That also suggests that any
> replication system need not always be shipped with the "basic"
> distribution, but could instead be integrated into a larger,
> postgresql_plus_enterprise_features.tgz or something like that.)

I don't, but consider Linux which can be configured to run on devices as
small as a wristwatch and as large as the the big irons...

--
dave

ow wrote:
> Hi,
>
> Is this all that's planned for 7.5? (based on current TODO list)
>
> -Change factorial to return a numeric (Gavin)
> -COMMENT ON [ CAST | CONVERSION | OPERATOR CLASS | LARGE OBJECT | LANGUAGE ]
> (Christopher)
> -Have psql \dn show only visible temp schemas using current_schemas()
> -Have psql '\i ~/<tab><tab>' actually load files it displays from home dir
> -Allow psql \du to show groups, and add \dg for groups
> -Allow pg_dump to dump CREATE CONVERSION (Christopher)
> -Use dependency information to dump data in proper order
> -Use background process to write dirty shared buffers to disk

Those dashes mean the items are done and are already in CVS. As to what
additional items we will have in 7.5, no one knows.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Tom Lane wrote:
> ow <oneway_111(at)yahoo(dot)com> writes:
> > Is this all that's planned for 7.5? (based on current TODO list)
>
> If you think the TODO list has *anything* to do with release planning,
> you fundamentally misunderstand the way things are done around here.
>
> The TODO list is a list of things that are widely agreed to be problems
> (though sometimes it only means "Bruce thinks this is a problem") and
> therefore will probably get worked on at some point in the future.
>
> What actually gets done for 7.5 will depend on which itches bother which
> developers enough to get scratched in the next few months. We do not
> have any central planning.
>
> It is a safe bet that 7.5 will include some things mentioned in the
> present TODO, as well as many things not mentioned in it; and that
> many items mentioned in the present TODO will still remain undone.

Yes, the TODO is just a collection of items that could be done in 7.5.
Some will be, and some new items not on the TODO list will be in 7.5.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

>>-COMMENT ON [ CAST | CONVERSION | OPERATOR CLASS | LARGE OBJECT | LANGUAGE ]
>>(Christopher)

Hey Bruce,

You probably should add 'Dump LOB comments in custom dump format' to the
todo. That's the last part of that task above which I haven't done yet,
and for various reasons probably won't have time to try for a while.
Just so we don't forget it.

Chris

Christopher Kings-Lynne wrote:
>
> >>-COMMENT ON [ CAST | CONVERSION | OPERATOR CLASS | LARGE OBJECT | LANGUAGE ]
> >>(Christopher)
>
> Hey Bruce,
>
> You probably should add 'Dump LOB comments in custom dump format' to the
> todo. That's the last part of that task above which I haven't done yet,
> and for various reasons probably won't have time to try for a while.
> Just so we don't forget it.

Added:

* Dump large object comments in custom dump format

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073