Re: Packaging 7.1.1

I am running Postgresql 7.1 on a dedicated Redhat 7.0 box with 512meg ram
and an IDE hard drive.

All day long queries that usually seem to execute instantaneously have been
taking up to 10 second to run! I generally have about 6 postmasters
running, utilizing anywhere from 1% to 96% CPU utilization. Another server
running Apache and PHP is performing all the queries. I am not using
persistant connections. I vaccum daily, usually has little to no impact on
the server. The database is roughly 60 megs, there are no usually wide
tables... the is one table (user tracking) that has about 200,000 rows. It
is indexed.

We are running a proprietary e-commerce package. Right now I am getting
roughly one query per second.

Any input would be helpful! If you need additional info let me know.

BTW, Thanks to Lamar for some great tips today!

Here is the status from pg_ctl:
/usr/local/pgsql/bin/postmaster '-d2' '-N' '48' '-B' '10000' '-i' '-D'
'/usr/local/pgsql/data'

Here is a sample from the log:
DEBUG: ProcessQuery
DEBUG: CommitTransactionCommand
DEBUG: StartTransactionCommand
DEBUG: query: SELECT sale_price FROM ec_sale_prices WHERE sale_begins <=
CURRENT_TIMESTAMP AND sale_ends >= CURRENT_TIMESTAMP AND p
roduct_id = 137
DEBUG: ProcessQuery
DEBUG: CommitTransactionCommand
DEBUG: proc_exit(0)
DEBUG: shmem_exit(0)
DEBUG: exit(0)
/usr/local/pgsql/bin/postmaster: reaping dead processes...
/usr/local/pgsql/bin/postmaster: CleanupProc: pid 958 exited with status 0
/usr/local/pgsql/bin/postmaster: ServerLoop: handling reading 5
/usr/local/pgsql/bin/postmaster: ServerLoop: handling reading 5
/usr/local/pgsql/bin/postmaster: ServerLoop: handling writing 5
/usr/local/pgsql/bin/postmaster: BackendStartup: pid 959 user postgres db
pa_commerce socket 5
/usr/local/pgsql/bin/postmaster child[959]: starting with (postgres -d2
-v131072 -p pa_commerce )
FindExec: found "/usr/local/pgsql/bin/postgres" using argv[0]
DEBUG: connection: host=216.239.233.44 user=postgres database=pa_commerce
DEBUG: InitPostgres
DEBUG: StartTransactionCommand
DEBUG: query: SELECT web_user_id FROM pa_web_users WHERE session_id =
'34978ae91facc5fc9abb8e21db609b4c'
DEBUG: ProcessQuery
DEBUG: CommitTransactionCommand
DEBUG: StartTransactionCommand
DEBUG: query: SELECT web_user_id FROM pa_partner_user_map WHERE
web_user_id = 221256
DEBUG: ProcessQuery
DEBUG: CommitTransactionCommand
DEBUG: StartTransactionCommand
DEBUG: query: UPDATE pa_partner_user_map SET last_visited =
CURRENT_TIMESTAMP, partner_id = 'OmdxViUZtwA-*HFh50XeaHBc70n42b4iXA' WH
ERE web_user_id = 221256
DEBUG: ProcessQuery
DEBUG: CommitTransactionCommand
DEBUG: StartTransactionCommand
DEBUG: query: SELECT order_id FROM pa_orders WHERE web_user_id = 221256
AND order_state = 'in_basket'
DEBUG: ProcessQuery
DEBUG: CommitTransactionCommand
DEBUG: proc_exit(0)
DEBUG: shmem_exit(0)
DEBUG: exit(0)
/usr/local/pgsql/bin/postmaster: ServerLoop: handling reading 5
/usr/local/pgsql/bin/postmaster: ServerLoop: handling reading 5
/usr/local/pgsql/bin/postmaster: ServerLoop: handling writing 5
/usr/local/pgsql/bin/postmaster: BackendStartup: pid 960 user postgres db
pa_commerce socket 5
/usr/local/pgsql/bin/postmaster: reaping dead processes...
/usr/local/pgsql/bin/postmaster: CleanupProc: pid 959 exited with status 0
/usr/local/pgsql/bin/postmaster child[960]: starting with (postgres -d2
-v131072 -p pa_commerce )
FindExec: found "/usr/local/pgsql/bin/postgres" using argv[0]

!! I haven't ran VACUUM ANALYZE since last night. Just ran it -
performance has improved significantly. I think I am going to have to run
it hourly during this high traffic time. Postmasters are still utilizing
about 100% of the CPU. Is this normal? I am considering increasing the
shmmax again.

Thanks for the help Mitch!

-r

At 09:11 PM 5/3/01 -0400, Mitch Vincent wrote:

> > persistant connections. I vaccum daily, usually has little to no impact
>on
>
> You VACUUM ANALYZE too, don't you?
>
>-Mitch
>
>
>
>
>
>---
>Incoming mail is certified Virus Free.
>Checked by AVG anti-virus system (http://www.grisoft.com).
>Version: 6.0.251 / Virus Database: 124 - Release Date: 4/26/01

Here is some output from top...

9:20pm up 40 min, 1 user, load average: 3.77, 3.12, 3.74
41 processes: 36 sleeping, 5 running, 0 zombie, 0 stopped
CPU states: 99.2% user, 0.7% system, 0.0% nice, 0.0% idle
Mem: 515664K av, 303712K used, 211952K free, 37476K shrd, 39552K buff
Swap: 514068K av, 0K used, 514068K free 158980K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
1657 postgres 20 0 6820 6820 5712 R 33.4 1.3 0:12 postmaster
1671 postgres 20 0 6576 6576 5468 R 33.4 1.2 0:08 postmaster
1650 postgres 19 0 6952 6952 5848 R 32.6 1.3 0:16 postmaster
1444 postgres 0 0 1044 1044 844 R 0.5 0.2 0:02 top
1 root 0 0 540 540 476 S 0.0 0.1 0:06 init
2 root 0 0 0 0 0 SW 0.0 0.0 0:02 kflushd
3 root 0 0 0 0 0 SW 0.0 0.0 0:04 kupdate
4 root 0 0 0 0 0 SW 0.0 0.0 0:00 kpiod
5 root 0 0 0 0 0 SW 0.0 0.0 0:00 kswapd
6 root -20 -20 0 0 0 SW< 0.0 0.0 0:00 mdrecoveryd
373 root 0 0 836 836 700 S 0.0 0.1 0:00 syslogd
383 root 0 0 852 852 472 S 0.0 0.1 0:00 klogd
398 rpc 0 0 580 580 492 S 0.0 0.1 0:00 portmap
414 root 0 0 0 0 0 SW 0.0 0.0 0:00 lockd
415 root 0 0 0 0 0 SW 0.0 0.0 0:00 rpciod
425 rpcuser 0 0 832 832 720 S 0.0 0.1 0:00 rpc.statd
477 nobody 0 0 720 720 612 S 0.0 0.1 0:00 identd
484 nobody 0 0 720 720 612 S 0.0 0.1 0:00 identd
485 nobody 0 0 720 720 612 S 0.0 0.1 0:00 identd
488 nobody 0 0 720 720 612 S 0.0 0.1 0:00 identd
489 nobody 0 0 720 720 612 S 0.0 0.1 0:00 identd
496 daemon 0 0 580 580 504 S 0.0 0.1 0:00 atd
511 root 0 0 1040 1040 828 S 0.0 0.2 0:00 xinetd
520 root 0 0 1196 1196 1060 S 0.0 0.2 0:00 sshd
533 root 0 0 2008 2008 1652 R 0.0 0.3 0:00 sshd
542 lp 0 0 1112 1112 948 S 0.0 0.2 0:00 lpd
579 root 0 0 512 512 448 S 0.0 0.0 0:00 gpm
594 root 0 0 720 720 616 S 0.0 0.1 0:00 crond
679 xfs 0 0 4496 4496 808 S 0.0 0.8 0:00 xfs
704 ryan 0 0 1328 1328 1036 S 0.0 0.2 0:00 bash
752 postgres 0 0 2492 2492 2360 S 0.0 0.4 0:01 postmaster
778 root 0 0 444 444 380 S 0.0 0.0 0:00 mingetty
779 root 0 0 444 444 380 S 0.0 0.0 0:00 mingetty
780 root 0 0 444 444 380 S 0.0 0.0 0:00 mingetty
781 root 0 0 444 444 380 S 0.0 0.0 0:00 mingetty
782 root 0 0 444 444 380 S 0.0 0.0 0:00 mingetty
783 root 0 0 444 444 380 S 0.0 0.0 0:00 mingetty
1279 root 0 0 1060 1060 856 S 0.0 0.2 0:00 su
1280 root 0 0 1388 1388 1076 S 0.0 0.2 0:00 bash
1369 root 0 0 1020 1020 828 S 0.0 0.1 0:00 su
1370 postgres 0 0 1308 1308 1028 S 0.0 0.2 0:00 bash

oh btw, i completely forgot to mention the minor fixes to the linux init
scripts i mentioned earlier (about 2 weeks ago) for things that perhaps
should be in the 7.1.1 release. (someone sent out a mail that they were
branching 7.1.1)

Also i never got a response on who actually packages those linux init
scripts that appear in the RPM but not on the pgsql cvs tree. (i am also
curious on why it is different, and how the RPM is built).

-rchit

-----Original Message-----
From: Bruce Momjian [mailto:pgman(at)candle(dot)pha(dot)pa(dot)us]
Sent: Thursday, May 03, 2001 9:16 AM
To: PostgreSQL-development
Subject: [HACKERS] Packaging 7.1.1

I am starting to package 7.1.1, and I see I did not brand 7.1 properly.
I forgot the date in the HISTORY file, and didn't update register.txt.
I will do all those now for 7.1.1.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

Not sure on their status. Are they listed on the outstanding patches
page at the bottom of the developers page? Probably too late for 7.1.1
now.

[ Charset ISO-8859-1 unsupported, converting... ]
> oh btw, i completely forgot to mention the minor fixes to the linux init
> scripts i mentioned earlier (about 2 weeks ago) for things that perhaps
> should be in the 7.1.1 release. (someone sent out a mail that they were
> branching 7.1.1)
>
> Also i never got a response on who actually packages those linux init
> scripts that appear in the RPM but not on the pgsql cvs tree. (i am also
> curious on why it is different, and how the RPM is built).
>
> -rchit
>
> -----Original Message-----
> From: Bruce Momjian [mailto:pgman(at)candle(dot)pha(dot)pa(dot)us]
> Sent: Thursday, May 03, 2001 9:16 AM
> To: PostgreSQL-development
> Subject: [HACKERS] Packaging 7.1.1
>
>
> I am starting to package 7.1.1, and I see I did not brand 7.1 properly.
> I forgot the date in the HISTORY file, and didn't update register.txt.
> I will do all those now for 7.1.1.
>
> --
> Bruce Momjian | http://candle.pha.pa.us
> pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
> + If your life is a hard drive, | 830 Blythe Avenue
> + Christ can be your backup. | Drexel Hill, Pennsylvania 19026
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
>

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Rachit Siamwalla <rachit(at)ensim(dot)com> writes:

> Also i never got a response on who actually packages those linux init
> scripts that appear in the RPM but not on the pgsql cvs tree. (i am also
> curious on why it is different, and how the RPM is built).

Lamar Owen and I.

--
Trond Eivind Glomsrød
Red Hat, Inc.

Rachit Siamwalla wrote:
> oh btw, i completely forgot to mention the minor fixes to the linux init
> scripts i mentioned earlier (about 2 weeks ago) for things that perhaps
> should be in the 7.1.1 release. (someone sent out a mail that they were
> branching 7.1.1)

> Also i never got a response on who actually packages those linux init
> scripts that appear in the RPM but not on the pgsql cvs tree. (i am also
> curious on why it is different, and how the RPM is built).

That would be me. Before building and releasing 7.1.1 RPMs I will be
reviewing the various bugs and changes planned for the 7.1.1 RPM.

As to why the RPM init script is different from the one packaged in the
main source tree -- I can make assumptions in the RPM set that the
version in the source tree cannot.

As to how the RPMs are built -- to answer that question sanely requires
me to know how much experience you have with the whole RPM paradigm.
'How is the RPM built?' is a multifaceted question. The obvious simple
answer is that I maintain:
1.) A set of patches to make certain portions of the source
tree 'behave' in the different environment of the RPMset;
2.) The initscript;
3.) Any other ancilliary scripts and files;
4.) A README.rpm-dist document that tries to adequately document
both the differences between the RPM build and the WHY of the
differences, as well as useful RPM environment operations
(like, using syslog, upgrading, getting postmaster to
start at OS boot, etc);
5.) The spec file that throws it all together. This is not a
trivial undertaking in a package of this size.

I then download and build on as many different canonical distributions
as I can -- currently I am able to build on Red Hat 6.2, 7.0, and 7.1 on
my personal hardware. Occasionally I receive opportunity from certain
commercial enterprises such as Great Bridge and PostgreSQL Inc to build
on other distributions.

I test the build by installing the resulting packages and running the
regression tests. Once the build passes these tests, I upload to the
postgresql.org ftp server and make a release announcement. I am also
responsible for maintaining the RPM download area on the ftp site.

You'll notice I said 'canonical' distributions above. That simply means
that the machine is as stock 'out of the box' as practical -- that is,
everything (except select few programs) on these boxen are installed by
RPM; only official Red Hat released RPMs are used (except in unusual
circumstances involving software that will not alter the build -- for
example, installing a newer non-RedHat version of the Dia diagramming
package is OK -- installing Python 2.1 on the box that has Python 1.5.2
installed is not, as that alters the PostgreSQL build). The RPM as
uploaded is built to as close to out-of-the-box pristine as is
possible. Only the standard released 'official to that release'
compiler is used -- and only the standard official kernel is used as
well.

For a time I built on Mandrake for RedHat consumption -- no more.
Nonstandard RPM building systems are worse than useless. Which is not
to say that Mandrake is useless! By no means is Mandrake useless --
unless you are building Red Hat RPMs -- and Red Hat is useless if you're
trying to build Mandrake or SuSE RPMs, for that matter. But I would be
foolish to use 'Lamar Owen's Super Special RPM Blend Distro 0.1.2' to
build for public consumption! :-)

I _do_ attempt to make the _source_ RPM compatible with as many
distributions as possible -- however, since I have limited resources (as
a volunteer RPM maintainer) I am limited as to the amount of testing
said build will get on other distributions, architectures, or systems.

And, while I understand people's desire to immediately upgrade to the
newest version, realize that I do this as a side interest -- I have a
regular, full-time job as a broadcast
engineer/webmaster/sysadmin/Technical Director which occasionally
prevents me from making timely RPM releases. This happened during the
early part of the 7.1 beta cycle -- but I believe I was pretty much on
the ball for the Release Candidates and the final release.

I am working towards a more open RPM distribution -- I would dearly love
to more fully document the process and put everything into CVS -- once I
figure out how I want to represent things such as the spec file in a CVS
form. It makes no sense to maintain a changelog, for instance, in the
spec file in CVS when CVS does a better job of changelogs -- I will need
to write a tool to generate a real spec file from a CVS spec-source file
that would add version numbers, changelog entries, etc to the result
before building the RPM. IOW, I need to rethink the process -- and then
go through the motions of putting my long RPM history into CVS one
version at a time so that version history information isn't lost.

As to why all these files aren't part of the source tree, well, unless
there was a large cry for it to happen, I don't believe it should.
PostgreSQL is very platform-agnostic -- and I like that. Including the
RPM stuff as part of the Official Tarball (TM) would, IMHO, slant that
agnostic stance in a negative way. But maybe I'm too sensitive to
that. I'm not opposed to doing that if that is the consensus of the
core group -- and that would be a sneaky way to get the stuff into CVS
:-). But if the core group isn't thrilled with the idea (and my
instinct says they're not likely to be), I am opposed to the idea -- not
to keep the stuff to myself, but to not hinder the platform-neutral
stance. IMHO, of course.

Of course, there are many projects that DO include all the files
necessary to build RPMs from their Official Tarball (TM).

Bruce, should portions of that answer be part of the linux FAQ? I don't
want to have to write that too many times :-).
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Trond Eivind Glomsrød wrote:
>
> Rachit Siamwalla <rachit(at)ensim(dot)com> writes:
>
> > Also i never got a response on who actually packages those linux init
> > scripts that appear in the RPM but not on the pgsql cvs tree. (i am also
> > curious on why it is different, and how the RPM is built).
>
> Lamar Owen and I.

Egads! I forgot to mention Trond! My apologies! (I'm being serious...)

Trond, of Red Hat; Reinhard Max, of SuSE; and Thomas Lockhart, of
PostgreSQL Inc (:-)) have all been major contributors to the RPM
distribution. Karl DeBisschop, Mike Mascari, and many others have
provided fixes and ideas as well.

Sorry guys -- I got caught up in the process and forgot the people! :-(
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Thanks to input Bruce M., figured out my performance problems - had to do
with a few QUERIES FROM HELL! After running EXPLAIN a few times I fine
tuned some of the worst ones, mostly over use of sub queries. Still
combing through my query log.

Getting there...

-r

Ryan Mahoney wrote:
> Any input would be helpful! If you need additional info let me know.

> BTW, Thanks to Lamar for some great tips today!

You're more than welcome.

I forgot a basic tip, which leads to a question:
How often are you running VACUUM ANALYZE?

If this were PostgreSQL 7.0.3, we could ask Alfred about his lazy vacuum
patches, as they work as well for Red Hat 7 as they do for FreeBSD.

Personally, I look forward to the following note being placed into the
docs:
VACUUM: deprecated. And the feature that makes that note possible.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

I hope you have those postmasters listening on different ports.

> Here is some output from top...
>
> 9:20pm up 40 min, 1 user, load average: 3.77, 3.12, 3.74
> 41 processes: 36 sleeping, 5 running, 0 zombie, 0 stopped
> CPU states: 99.2% user, 0.7% system, 0.0% nice, 0.0% idle
> Mem: 515664K av, 303712K used, 211952K free, 37476K shrd, 39552K buff
> Swap: 514068K av, 0K used, 514068K free 158980K cached
>
> PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
> 1657 postgres 20 0 6820 6820 5712 R 33.4 1.3 0:12 postmaster
> 1671 postgres 20 0 6576 6576 5468 R 33.4 1.2 0:08 postmaster
> 1650 postgres 19 0 6952 6952 5848 R 32.6 1.3 0:16 postmaster
> 1444 postgres 0 0 1044 1044 844 R 0.5 0.2 0:02 top
> 1 root 0 0 540 540 476 S 0.0 0.1 0:06 init
> 2 root 0 0 0 0 0 SW 0.0 0.0 0:02 kflushd
> 3 root 0 0 0 0 0 SW 0.0 0.0 0:04 kupdate
> 4 root 0 0 0 0 0 SW 0.0 0.0 0:00 kpiod
> 5 root 0 0 0 0 0 SW 0.0 0.0 0:00 kswapd
> 6 root -20 -20 0 0 0 SW< 0.0 0.0 0:00 mdrecoveryd
> 373 root 0 0 836 836 700 S 0.0 0.1 0:00 syslogd
> 383 root 0 0 852 852 472 S 0.0 0.1 0:00 klogd
> 398 rpc 0 0 580 580 492 S 0.0 0.1 0:00 portmap
> 414 root 0 0 0 0 0 SW 0.0 0.0 0:00 lockd
> 415 root 0 0 0 0 0 SW 0.0 0.0 0:00 rpciod
> 425 rpcuser 0 0 832 832 720 S 0.0 0.1 0:00 rpc.statd
> 477 nobody 0 0 720 720 612 S 0.0 0.1 0:00 identd
> 484 nobody 0 0 720 720 612 S 0.0 0.1 0:00 identd
> 485 nobody 0 0 720 720 612 S 0.0 0.1 0:00 identd
> 488 nobody 0 0 720 720 612 S 0.0 0.1 0:00 identd
> 489 nobody 0 0 720 720 612 S 0.0 0.1 0:00 identd
> 496 daemon 0 0 580 580 504 S 0.0 0.1 0:00 atd
> 511 root 0 0 1040 1040 828 S 0.0 0.2 0:00 xinetd
> 520 root 0 0 1196 1196 1060 S 0.0 0.2 0:00 sshd
> 533 root 0 0 2008 2008 1652 R 0.0 0.3 0:00 sshd
> 542 lp 0 0 1112 1112 948 S 0.0 0.2 0:00 lpd
> 579 root 0 0 512 512 448 S 0.0 0.0 0:00 gpm
> 594 root 0 0 720 720 616 S 0.0 0.1 0:00 crond
> 679 xfs 0 0 4496 4496 808 S 0.0 0.8 0:00 xfs
> 704 ryan 0 0 1328 1328 1036 S 0.0 0.2 0:00 bash
> 752 postgres 0 0 2492 2492 2360 S 0.0 0.4 0:01 postmaster
> 778 root 0 0 444 444 380 S 0.0 0.0 0:00 mingetty
> 779 root 0 0 444 444 380 S 0.0 0.0 0:00 mingetty
> 780 root 0 0 444 444 380 S 0.0 0.0 0:00 mingetty
> 781 root 0 0 444 444 380 S 0.0 0.0 0:00 mingetty
> 782 root 0 0 444 444 380 S 0.0 0.0 0:00 mingetty
> 783 root 0 0 444 444 380 S 0.0 0.0 0:00 mingetty
> 1279 root 0 0 1060 1060 856 S 0.0 0.2 0:00 su
> 1280 root 0 0 1388 1388 1076 S 0.0 0.2 0:00 bash
> 1369 root 0 0 1020 1020 828 S 0.0 0.1 0:00 su
> 1370 postgres 0 0 1308 1308 1028 S 0.0 0.2 0:00 bash

>
> ---
> Outgoing mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.251 / Virus Database: 124 - Release Date: 4/26/01

>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/users-lounge/docs/faq.html

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Lamar Owen <lamar(dot)owen(at)wgcr(dot)org> writes:

> Ryan Mahoney wrote:
> > Any input would be helpful! If you need additional info let me know.
>
> > BTW, Thanks to Lamar for some great tips today!
>
> You're more than welcome.
>
> I forgot a basic tip, which leads to a question:
> How often are you running VACUUM ANALYZE?
>
> If this were PostgreSQL 7.0.3, we could ask Alfred about his lazy vacuum
> patches, as they work as well for Red Hat 7 as they do for FreeBSD.

Postgresql 7.0.3 from Red Hat Linux 7.1 should work just fine on Red
Hat Linux 7.

--
Trond Eivind Glomsrød
Red Hat, Inc.

Thus spake Ryan Mahoney
> !! I haven't ran VACUUM ANALYZE since last night. Just ran it -
> performance has improved significantly. I think I am going to have to run
> it hourly during this high traffic time. Postmasters are still utilizing
> about 100% of the CPU. Is this normal? I am considering increasing the
> shmmax again.

Although it isn't supposed to be necessary, I find that I have to dump and
reload once in a while to keep performance hight.

--
D'Arcy J.M. Cain <darcy(at){druid|vex}.net> | Democracy is three wolves
http://www.druid.net/darcy/ | and a sheep voting on
+1 416 425 1212 (DoD#0082) (eNTP) | what's for dinner.

Trond Eivind Glomsrød wrote:
>
> Rachit Siamwalla <rachit(at)ensim(dot)com> writes:
>
> > Also i never got a response on who actually packages those linux init
> > scripts that appear in the RPM but not on the pgsql cvs tree. (i am also
> > curious on why it is different, and how the RPM is built).
>
> Lamar Owen and I.

Is the current snapshot available? I have submitted fixes twice now for what I am fairly sure is a bug in the init script. At least one of the posts was the shortly after lamar posted the RC3 RPM. Yet the bug remained.

This is not a complaint -- you guys have put alot of effort into the RPMs and they are very solid IMHO. But I would like the chance to look at the RPM as it stands sometime before 7.1, as I have to customize the RPM yet again to distribute a working init script to our servers.

Have you thought about a CVS store some place for the RPM files?

--
Karl

Lamar Owen <lamar(dot)owen(at)wgcr(dot)org> writes:
> As to why all these files aren't part of the source tree, well, unless
> there was a large cry for it to happen, I don't believe it should.
> PostgreSQL is very platform-agnostic -- and I like that. Including the
> RPM stuff as part of the Official Tarball (TM) would, IMHO, slant that
> agnostic stance in a negative way.

Seems like that stuff should be in CVS somewhere ... if only so someone
else can pick up the ball if you get run over by a truck :-(.

If it's just a small amount of code, I don't see what the harm would be
in including it in the regular distro, though we should talk about just
where it should go. If it's a large amount of code then perhaps a
separate CVS project would be better, so that people who have no use for
it don't end up pulling/downloading it.

regards, tom lane

Tom Lane wrote:
> Seems like that stuff should be in CVS somewhere ... if only so someone
> else can pick up the ball if you get run over by a truck :-(.

My wife appreciates the sentiment :-). As it stands now, better
documentation distributed in the source RPM would help greatly.
Everything necessary to do the build and maintain the package is in the
source RPM as it stands now -- evidenced by the Linux distributors being
able to take our source RPM, massage it to fit their particular system,
and run with it. And I have a scad of history available in specfile
form....

> If it's just a small amount of code, I don't see what the harm would be
> in including it in the regular distro, though we should talk about just
> where it should go. If it's a large amount of code then perhaps a
> separate CVS project would be better, so that people who have no use for
> it don't end up pulling/downloading it.

Not counting the JDBC jars, it's a hundred K or so uncompressed. The
spec file is around 30k -- a small amount of code.

contrib/rpm-dist?
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Lamar Owen <lamar(dot)owen(at)wgcr(dot)org> writes:
> contrib/rpm-dist?

Contrib was my first thought also --- but on second thought, the RPM
packaging support is hardly contrib-grade material. For a large
proportion of our users it's a critical part of the distribution.
So, if we are going to have it in the CVS tree at all, I'd vote for
putting it in the main tree.

Perhaps src/rpm-tools/ or some such name.

regards, tom lane

Tom Lane wrote:
> Contrib was my first thought also --- but on second thought, the RPM
> packaging support is hardly contrib-grade material. For a large
> proportion of our users it's a critical part of the distribution.
> So, if we are going to have it in the CVS tree at all, I'd vote for
> putting it in the main tree.

> Perhaps src/rpm-tools/ or some such name.

Let's see where the rest of core and hackers weighs in....
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Lamar Owen wrote:
>
> Tom Lane wrote:
> > Seems like that stuff should be in CVS somewhere ... if only so someone
> > else can pick up the ball if you get run over by a truck :-(.
>
> My wife appreciates the sentiment :-). As it stands now, better
> documentation distributed in the source RPM would help greatly.
> Everything necessary to do the build and maintain the package is in the
> source RPM as it stands now -- evidenced by the Linux distributors being
> able to take our source RPM, massage it to fit their particular system,
> and run with it. And I have a scad of history available in specfile
> form....
>
> > If it's just a small amount of code, I don't see what the harm would be
> > in including it in the regular distro, though we should talk about just
> > where it should go. If it's a large amount of code then perhaps a
> > separate CVS project would be better, so that people who have no use for
> > it don't end up pulling/downloading it.
>
> Not counting the JDBC jars, it's a hundred K or so uncompressed. The
> spec file is around 30k -- a small amount of code.
>
> contrib/rpm-dist?

Seems to work. But I would prefer to look at how ither packaging schemes
work and come up with something that might be consistent and useful
across the board.

For starters, I'd make contrib/package/

Then make an rpm subdirectory. Also a pkg directory for system that use
pkgmk/pkginfo/pkgadd/pkgrm. If there's a way to may debain packages paly
the game, put them in as well. Then, if someaone is packages for a
variety of systems, there is alt least the possibility of some small
amount of consistency.

Extending things, you could have contrib/package/rpm/redhat for
redhat-specific stuff. contrib/package/rpm/mandrake for mandrafke stuff.
You get the idea.

At that point, I could even imagine contrib/mkpackage script that di som
OS detection, and built wahtever you wanted. That may be a little far
off, but I think there is an important nuggent in here. Tarballs are
great for developers, but they are not that great for system
administrators with large installed bases. PostgreSQL builds are great
for the portability. The next logical step might in fact be to extend
some of that consistency to the package creation arena.

--
Karl

Lamar Owen writes:

> contrib/rpm-dist?

A separate CVS module sounds like a better idea to me.

--
Peter Eisentraut peter_e(at)gmx(dot)net http://funkturm.homeip.net/~peter

> Lamar Owen <lamar(dot)owen(at)wgcr(dot)org> writes:
> > contrib/rpm-dist?
>
> Contrib was my first thought also --- but on second thought, the RPM
> packaging support is hardly contrib-grade material. For a large
> proportion of our users it's a critical part of the distribution.
> So, if we are going to have it in the CVS tree at all, I'd vote for
> putting it in the main tree.
>
> Perhaps src/rpm-tools/ or some such name.

It is platform-specific, which would seem to vote for /contrib.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
>> Perhaps src/rpm-tools/ or some such name.

> It is platform-specific, which would seem to vote for /contrib.

Huh? By that logic, all of src/makefiles/, src/template/, and
src/backend/port/, not to mention large chunks of the configure
mechanism, belong in contrib. Shall we rip out all BSD support
and move it to contrib?

contrib has never been about platform dependency in my mind; it's about
whether we consider something part of the project mainstream (in terms
of code quality and our willingness to support it). RPM support isn't
going away, and I'm willing to call it mainstream ...

regards, tom lane

For various definitions of "Platform". Linux runs on a NUMBER of
hardware platforms, and RPM is used by a LOT of LINUX distributions.

I'd vote for src/rpm-tools/ if I had a vote.

--
Larry Rosenman http://www.lerctr.org/~ler/
Phone: +1 972 414 9812 E-Mail: ler(at)lerctr(dot)org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749 US

>>>>>>>>>>>>>>>>>> Original Message <<<<<<<<<<<<<<<<<<

On 5/4/01, 11:57:05 AM, Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> wrote
regarding Re: [HACKERS] Packaging 7.1.1:

> > Lamar Owen <lamar(dot)owen(at)wgcr(dot)org> writes:
> > > contrib/rpm-dist?
> >
> > Contrib was my first thought also --- but on second thought, the RPM
> > packaging support is hardly contrib-grade material. For a large
> > proportion of our users it's a critical part of the distribution.
> > So, if we are going to have it in the CVS tree at all, I'd vote for
> > putting it in the main tree.
> >
> > Perhaps src/rpm-tools/ or some such name.

> It is platform-specific, which would seem to vote for /contrib.

> --
> Bruce Momjian | http://candle.pha.pa.us
> pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
> + If your life is a hard drive, | 830 Blythe Avenue
> + Christ can be your backup. | Drexel Hill, Pennsylvania 19026

> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?

> http://www.postgresql.org/search.mpl

Karl DeBisschop writes:

> PostgreSQL builds are great for the portability. The next logical step
> might in fact be to extend some of that consistency to the package
> creation arena.

This would have been cool in 1996. We would have evolved a large number
of different packages along with the build system. But it didn't happen
this way and now most packages are sufficiently contorted in a number of
ways because of vendor requirements, different ideas of how an operating
system is supposed to work, self-inflicted incompatibilities, and a number
of other reasons, including not least importantly the desire to have
control over what ships in your system. All valid reasons, of course.

If we can work at, and succeed at, resolving most of these oddities, then
tracking packages in the source tree might prove worthwhile. But as long
as we're still required to keep track what vendor has 'chkconfig' or what
version of what distribution has broken CFLAGS, to list some trivial
things, as long as the packages need to track anything but the development
of PostgreSQL itself, this undertaking is going to become a problem.

What would be worthwhile is setting up another cvs module so packages can
be developed and released at their own pace.

--
Peter Eisentraut peter_e(at)gmx(dot)net http://funkturm.homeip.net/~peter

Karl DeBisschop wrote:
>
> Trond Eivind Glomsrød wrote:
> >
> > Rachit Siamwalla <rachit(at)ensim(dot)com> writes:
> >
> > > Also i never got a response on who actually packages those linux init
> > > scripts that appear in the RPM but not on the pgsql cvs tree. (i am also
> > > curious on why it is different, and how the RPM is built).
> >
> > Lamar Owen and I.
>
> Is the current snapshot available?

The current snapshot is the 7.1-1 release as of this time.

>I have submitted fixes twice now for what I am fairly sure is a bug in the init script. At least one of the posts was the shortly after lamar posted the RC3 RPM. Yet the bug remained.

I thought I integrated that one, but I must not have. My apologies.

> This is not a complaint -- you guys have put alot of effort into the RPMs and they are very solid IMHO. But I would like the chance to look at the RPM as it stands sometime before 7.1, as I have to customize the RPM yet again to distribute a working init script to our servers.

Mail me the initscript as fixed. Put a [HACKERS] in the usbject so it
goes to the right folder. The extant 7.1-1 RPMset is the last build I
have made.

> Have you thought about a CVS store some place for the RPM files?

Yes. Discussion currently underway in HACKERS.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Peter Eisentraut wrote:
> What would be worthwhile is setting up another cvs module so packages can
> be developed and released at their own pace.

This is an _excellent_ point, and one I had thought of before but had
forgotten.

FWIW, I have a project set up at greatbridge.org -- I just have to get
myself in gear and get it done.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Peter Eisentraut wrote:
>
> Karl DeBisschop writes:
>
> > PostgreSQL builds are great for the portability. The next logical step
> > might in fact be to extend some of that consistency to the package
> > creation arena.
>
> This would have been cool in 1996. We would have evolved a large number
> of different packages along with the build system. But it didn't happen
> this way and now most packages are sufficiently contorted in a number of
> ways because of vendor requirements, different ideas of how an operating
> system is supposed to work, self-inflicted incompatibilities, and a number
> of other reasons, including not least importantly the desire to have
> control over what ships in your system. All valid reasons, of course.
>
> If we can work at, and succeed at, resolving most of these oddities, then
> tracking packages in the source tree might prove worthwhile. But as long
> as we're still required to keep track what vendor has 'chkconfig' or what
> version of what distribution has broken CFLAGS, to list some trivial
> things, as long as the packages need to track anything but the development
> of PostgreSQL itself, this undertaking is going to become a problem.
>
> What would be worthwhile is setting up another cvs module so packages can
> be developed and released at their own pace.

I think on the biggest point we agree. Working with packagers and making
that job easier and more consistent is a good thing, (so long as it does
not interfere with development on postgresql itself, of course).

On the projects I am involved with, however, my experience of what work
has been contary to the tactics you suggest for reaching tha goal. I
found it easiest to develop in close concert with packagers, and in my
case that meant hosting the various packaging scripts within the source
tree. Of course that was for smaller projects with much less legacy than
postgresql, so maybe it doesn't apply here.

I still think it would be cool to download just the tarball from the
site and have a little 'mkpackage' script that I run on solaris to get
the cannonical solaris packages, on Red Hat to get the cannonical Red
Hat rpms, on FreeBSD to get the cannonical port, etc. Maybe a ways off,
but an appealing end goal to me.

It would be even better if by unifying support of the packaging process,
the differences between install would be limited to the requirements of
each OS, and not be dictated by the personal whims of the packager. I
know Lamar and Oliver keep in close contact so their packages don't get
too idiosyncratic. I'm advocating any process that helps extend that
spirit across the board.

--
Karl

> Of course, there are many projects that DO include all the files
> necessary to build RPMs from their Official Tarball (TM).
>
> Bruce, should portions of that answer be part of the linux FAQ? I don't
> want to have to write that too many times :-).

I just had time to read that myself. Not sure about the Linux FAQ, but
it seems the file should be linked to from the Linux FAQ so people can
read this when needed.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

I have added this to the developer's FAQ.

---------------------------------------------------------------------------

> Rachit Siamwalla wrote:
> > oh btw, i completely forgot to mention the minor fixes to the linux init
> > scripts i mentioned earlier (about 2 weeks ago) for things that perhaps
> > should be in the 7.1.1 release. (someone sent out a mail that they were
> > branching 7.1.1)
>
> > Also i never got a response on who actually packages those linux init
> > scripts that appear in the RPM but not on the pgsql cvs tree. (i am also
> > curious on why it is different, and how the RPM is built).
>
> That would be me. Before building and releasing 7.1.1 RPMs I will be
> reviewing the various bugs and changes planned for the 7.1.1 RPM.
>
> As to why the RPM init script is different from the one packaged in the
> main source tree -- I can make assumptions in the RPM set that the
> version in the source tree cannot.
>
> As to how the RPMs are built -- to answer that question sanely requires
> me to know how much experience you have with the whole RPM paradigm.
> 'How is the RPM built?' is a multifaceted question. The obvious simple
> answer is that I maintain:
> 1.) A set of patches to make certain portions of the source
> tree 'behave' in the different environment of the RPMset;
> 2.) The initscript;
> 3.) Any other ancilliary scripts and files;
> 4.) A README.rpm-dist document that tries to adequately document
> both the differences between the RPM build and the WHY of the
> differences, as well as useful RPM environment operations
> (like, using syslog, upgrading, getting postmaster to
> start at OS boot, etc);
> 5.) The spec file that throws it all together. This is not a
> trivial undertaking in a package of this size.
>
> I then download and build on as many different canonical distributions
> as I can -- currently I am able to build on Red Hat 6.2, 7.0, and 7.1 on
> my personal hardware. Occasionally I receive opportunity from certain
> commercial enterprises such as Great Bridge and PostgreSQL Inc to build
> on other distributions.
>
> I test the build by installing the resulting packages and running the
> regression tests. Once the build passes these tests, I upload to the
> postgresql.org ftp server and make a release announcement. I am also
> responsible for maintaining the RPM download area on the ftp site.
>
> You'll notice I said 'canonical' distributions above. That simply means
> that the machine is as stock 'out of the box' as practical -- that is,
> everything (except select few programs) on these boxen are installed by
> RPM; only official Red Hat released RPMs are used (except in unusual
> circumstances involving software that will not alter the build -- for
> example, installing a newer non-RedHat version of the Dia diagramming
> package is OK -- installing Python 2.1 on the box that has Python 1.5.2
> installed is not, as that alters the PostgreSQL build). The RPM as
> uploaded is built to as close to out-of-the-box pristine as is
> possible. Only the standard released 'official to that release'
> compiler is used -- and only the standard official kernel is used as
> well.
>
> For a time I built on Mandrake for RedHat consumption -- no more.
> Nonstandard RPM building systems are worse than useless. Which is not
> to say that Mandrake is useless! By no means is Mandrake useless --
> unless you are building Red Hat RPMs -- and Red Hat is useless if you're
> trying to build Mandrake or SuSE RPMs, for that matter. But I would be
> foolish to use 'Lamar Owen's Super Special RPM Blend Distro 0.1.2' to
> build for public consumption! :-)
>
> I _do_ attempt to make the _source_ RPM compatible with as many
> distributions as possible -- however, since I have limited resources (as
> a volunteer RPM maintainer) I am limited as to the amount of testing
> said build will get on other distributions, architectures, or systems.
>
> And, while I understand people's desire to immediately upgrade to the
> newest version, realize that I do this as a side interest -- I have a
> regular, full-time job as a broadcast
> engineer/webmaster/sysadmin/Technical Director which occasionally
> prevents me from making timely RPM releases. This happened during the
> early part of the 7.1 beta cycle -- but I believe I was pretty much on
> the ball for the Release Candidates and the final release.
>
> I am working towards a more open RPM distribution -- I would dearly love
> to more fully document the process and put everything into CVS -- once I
> figure out how I want to represent things such as the spec file in a CVS
> form. It makes no sense to maintain a changelog, for instance, in the
> spec file in CVS when CVS does a better job of changelogs -- I will need
> to write a tool to generate a real spec file from a CVS spec-source file
> that would add version numbers, changelog entries, etc to the result
> before building the RPM. IOW, I need to rethink the process -- and then
> go through the motions of putting my long RPM history into CVS one
> version at a time so that version history information isn't lost.
>
> As to why all these files aren't part of the source tree, well, unless
> there was a large cry for it to happen, I don't believe it should.
> PostgreSQL is very platform-agnostic -- and I like that. Including the
> RPM stuff as part of the Official Tarball (TM) would, IMHO, slant that
> agnostic stance in a negative way. But maybe I'm too sensitive to
> that. I'm not opposed to doing that if that is the consensus of the
> core group -- and that would be a sneaky way to get the stuff into CVS
> :-). But if the core group isn't thrilled with the idea (and my
> instinct says they're not likely to be), I am opposed to the idea -- not
> to keep the stuff to myself, but to not hinder the platform-neutral
> stance. IMHO, of course.
>
> Of course, there are many projects that DO include all the files
> necessary to build RPMs from their Official Tarball (TM).
>
> Bruce, should portions of that answer be part of the linux FAQ? I don't
> want to have to write that too many times :-).
> --
> Lamar Owen
> WGCR Internet Radio
> 1 Peter 4:11
>

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

On Tuesday 27 November 2001 01:27 pm, Bruce Momjian wrote:
> I have added this to the developer's FAQ.

> > I then download and build on as many different canonical distributions
> > as I can -- currently I am able to build on Red Hat 6.2, 7.0, and 7.1 on
> > my personal hardware. Occasionally I receive opportunity from certain
> > commercial enterprises such as Great Bridge and PostgreSQL Inc to build
> > on other distributions.

Hmmm. Bruce, would it be possible to put a date on that entry, as that
answer has some fairly old information -- old as in last cycle. I currently
am only able to build and test on Red Hat 7.2 -- and that is subject to
change as time goes on. Maybe, in the FAQ, where you say 'Written by Lamar
Owen' you could expound that a little by adding 'on May 4 2001' (or whenever
I actually wrote it)....

BTW: I believe those new sections are very nice, even if I did write two of
them.... :-)
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

> On Tuesday 27 November 2001 01:27 pm, Bruce Momjian wrote:
> > I have added this to the developer's FAQ.
>
> > > I then download and build on as many different canonical distributions
> > > as I can -- currently I am able to build on Red Hat 6.2, 7.0, and 7.1 on
> > > my personal hardware. Occasionally I receive opportunity from certain
> > > commercial enterprises such as Great Bridge and PostgreSQL Inc to build
> > > on other distributions.
>
> Hmmm. Bruce, would it be possible to put a date on that entry, as that
> answer has some fairly old information -- old as in last cycle. I currently
> am only able to build and test on Red Hat 7.2 -- and that is subject to
> change as time goes on. Maybe, in the FAQ, where you say 'Written by Lamar
> Owen' you could expound that a little by adding 'on May 4 2001' (or whenever
> I actually wrote it)....
>
> BTW: I believe those new sections are very nice, even if I did write two of
> them.... :-)

OK, done. Dates are a good idea for these entries. I put the date in
standard Unix format at the top of each message.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026