Re: contrib promotion?

There has been action to clean up and remove some contrib modules, and
this is good. I would like to suggest that we should try to move one or
two the other way, namely right into the core proper, on the ground that
they have widespread applicability and should have maximum visibility.
I'm talking particularly about pgcrypto and tsearch2, and the main thing
lacking that I see with these is documentation.

Thoughts?

cheers

andrew

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> There has been action to clean up and remove some contrib modules, and
> this is good. I would like to suggest that we should try to move one or
> two the other way, namely right into the core proper, on the ground that
> they have widespread applicability and should have maximum visibility.
> I'm talking particularly about pgcrypto and tsearch2, and the main thing
> lacking that I see with these is documentation.

I don't see a strong need for moving pgcrypto into core, and there's at
least one argument against it: if someone needs a crypto-free version of
postgres for use someplace with benighted laws, they would be screwed.

tsearch2 is functionality that definitely should be in core eventually,
but even Oleg still says it's not done. Aside from the documentation
issue, it's not clear that we've got a stable API for it.

regards, tom lane

Tom Lane wrote:

>Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
>
>
>>There has been action to clean up and remove some contrib modules, and
>>this is good. I would like to suggest that we should try to move one or
>>two the other way, namely right into the core proper, on the ground that
>>they have widespread applicability and should have maximum visibility.
>>I'm talking particularly about pgcrypto and tsearch2, and the main thing
>>lacking that I see with these is documentation.
>>
>>
>
>I don't see a strong need for moving pgcrypto into core, and there's at
>least one argument against it: if someone needs a crypto-free version of
>postgres for use someplace with benighted laws, they would be screwed.
>
>

Could that be handled with a configure option?

>tsearch2 is functionality that definitely should be in core eventually,
>but even Oleg still says it's not done. Aside from the documentation
>issue, it's not clear that we've got a stable API for it.
>
>
>

Well, that's a pity. This will be the 4th release with it in contrib,
IIRC. I know it's advanced stuff, but surely it has to settle down
sometime.

Quite apart from anything else, it's important that we do get better
docco on these modules.

cheers

andrew

Tom Lane wrote:
> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
>> There has been action to clean up and remove some contrib modules, and
>> this is good. I would like to suggest that we should try to move one or
>> two the other way, namely right into the core proper, on the ground that
>> they have widespread applicability and should have maximum visibility.
>> I'm talking particularly about pgcrypto and tsearch2, and the main thing
>> lacking that I see with these is documentation.
>
> I don't see a strong need for moving pgcrypto into core, and there's at
> least one argument against it: if someone needs a crypto-free version of
> postgres for use someplace with benighted laws, they would be screwed.

Doesn't our inclusion of md5() pretty much blow that argument away?
(Just asking).

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

"Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> Tom Lane wrote:
>> I don't see a strong need for moving pgcrypto into core, and there's at
>> least one argument against it: if someone needs a crypto-free version of
>> postgres for use someplace with benighted laws, they would be screwed.

> Doesn't our inclusion of md5() pretty much blow that argument away?

No: md5 is hashing, not encryption. The difference is that you can't
retrieve the original plaintext from a hash. That is a very large
difference in the eyes of most munitions laws --- encryption is useful
for spies, hashing not so much.

regards, tom lane

On Jul 14, 2006, at 12:32 PM, Joshua D. Drake wrote:

> Doesn't our inclusion of md5() pretty much blow that argument away?
> (Just asking).

I don't think so because md5 is just a one way hash function. There
is no method to decrypt anything :).

John DeSoi, Ph.D.
http://pgedit.com/
Power Tools for PostgreSQL

Andrew Dunstan wrote:
> >I don't see a strong need for moving pgcrypto into core, and there's
> > at least one argument against it: if someone needs a crypto-free
> > version of postgres for use someplace with benighted laws, they
> > would be screwed.
>
> Could that be handled with a configure option?

In such a jurisdiction you would possibly have to distribute rebuilt
tarballs with all the source code removed. The contrib option makes
that much easier.

> Quite apart from anything else, it's important that we do get better
> docco on these modules.

I'm willing to help with DocBook options. What do you have in mind?

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

>> Doesn't our inclusion of md5() pretty much blow that argument away?
>> (Just asking).
>
> I don't think so because md5 is just a one way hash function. There
> is no method to decrypt anything :).

Actually, I've had to install pgcrypto on more than one occasion for
clients who needed to have sha1 instead of md5. I've had to install
pgcrypto for other functions as well, so +1 for me on coring it, but
at the least please consider adding in sha1.

- --
Greg Sabino Mullane greg(at)turnstep(dot)com
PGP Key: 0x14964AC8 200607141355
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8

-----BEGIN PGP SIGNATURE-----

iD8DBQFEt9sfvJuQZxSWSsgRAhrCAJ9VZneUN6+3pvq+vJf+Y6lhjamj/QCfUDYc
+rYM+ITWvhw2dvo2B1hta6g=
=h4aD
-----END PGP SIGNATURE-----

Greg Sabino Mullane wrote:
>
>>>> Doesn't our inclusion of md5() pretty much blow that argument away?
>>>> (Just asking).
>>> I don't think so because md5 is just a one way hash function. There
>>> is no method to decrypt anything :).
>
> Actually, I've had to install pgcrypto on more than one occasion for
> clients who needed to have sha1 instead of md5. I've had to install
> pgcrypto for other functions as well, so +1 for me on coring it, but
> at the least please consider adding in sha1.

I don't have a very strong opinion on that but sha1() is something I
need on a regular base too from pgcrypto.

Stefan

Peter Eisentraut wrote:

>>Quite apart from anything else, it's important that we do get better
>>docco on these modules.
>>
>>
>
>I'm willing to help with DocBook options. What do you have in mind?
>
>
>

Well, if we could make provision for sucking in a chapter per contrib
module if it exists that would be a good start.

(Speaking of DocBook, did we ever make progress on switching to XML?)

cheers

andrew

> tsearch2 is functionality that definitely should be in core eventually,
> but even Oleg still says it's not done. Aside from the documentation
> issue, it's not clear that we've got a stable API for it.

Issues/TODO to move tsearch2 into core (by fast look)
* memory management. Dictionaries and tsearch2 itself cache a lot
of data in memory allocated by malloc/palloc(TopContext) and
not all can be correctly freed by reset_tsearch() function.
* tsearch2 doesn't automatically reinit dictionary/configuration
after configuration changes
* Also, dictionary doesn't automatically recompile its files if they
were changed
* It will be good to store shared information (such as
dictionary's structure) in shared memory. For example,
ispell dict takes for compile its files about 1-5 seconds, sharing
should help
* I suppose, at least snowball stemmers should be compiled in
postgresql or compiled them into separate *.so libs. Sources
of all snowball's files take ~1Mb on disk. The goal is easier
configuration for end-user. Ispell dict's files take 1-3 Mb per
language.
* Now database can be created with another encoding then cluster was
initialized, but locale can't be changed. Tsearch2 depends on locale
because of lower/upper casing and isalpha(and somr another is*). So,
it's possible to get unworking tsearch2 configuration.

I'm going to vacation tomorrow for two week, so I havn't time to resolve
that issues before feature freeze, sorry.

We estimate 1 month for both us to complete these issues/todos and
1 month extra to write documentation, so we certainly are not ready
for 8.2, most probable 8.3. We're looking for sponsorship of our work.

Oleg

On Sat, 15 Jul 2006, Teodor Sigaev wrote:

>> tsearch2 is functionality that definitely should be in core eventually,
>> but even Oleg still says it's not done. Aside from the documentation
>> issue, it's not clear that we've got a stable API for it.
>
> Issues/TODO to move tsearch2 into core (by fast look)
> * memory management. Dictionaries and tsearch2 itself cache a lot
> of data in memory allocated by malloc/palloc(TopContext) and
> not all can be correctly freed by reset_tsearch() function.
> * tsearch2 doesn't automatically reinit dictionary/configuration
> after configuration changes
> * Also, dictionary doesn't automatically recompile its files if they
> were changed
> * It will be good to store shared information (such as
> dictionary's structure) in shared memory. For example,
> ispell dict takes for compile its files about 1-5 seconds, sharing
> should help
> * I suppose, at least snowball stemmers should be compiled in
> postgresql or compiled them into separate *.so libs. Sources
> of all snowball's files take ~1Mb on disk. The goal is easier
> configuration for end-user. Ispell dict's files take 1-3 Mb per
> language.
> * Now database can be created with another encoding then cluster was
> initialized, but locale can't be changed. Tsearch2 depends on locale
> because of lower/upper casing and isalpha(and somr another is*). So,
> it's possible to get unworking tsearch2 configuration.
>
>
> I'm going to vacation tomorrow for two week, so I havn't time to resolve that
> issues before feature freeze, sorry.
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faq
>

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg(at)sai(dot)msu(dot)su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

On 7/14/06, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> I don't see a strong need for moving pgcrypto into core, and there's at
> least one argument against it: if someone needs a crypto-free version of
> postgres for use someplace with benighted laws, they would be screwed.

Image of hypothetical evil government is not a thing to base decisions on :)

Although I've tried to develop pgcrypto to be easily mergable into core,
I don't want to push it myself, the push should come from users.

That said, there is one situation that is badly handled in current
setup - storing passwords in database. There is md5() function in
core and everything in /contrib in basically invisible in website
and official docs. So even PG core devs suggest using md5() for
this task. But this is inadequate - bruteforcing md5 hash can be
done pretty easily on todays desktop computers. PostgreSQL itself
can get away with it only because it regular users cant see the hash.
But that is not so for ordinary apps.

So I would like either some mention of the more useful/stable modules
in core docs or a way for contrib modules to become 'official' add-on
modules (like PL-s are).

Full merge into core would fix this also, but indeed there is not many
techical reasons for it. (And editing pg_proc.h is PITA - I'd consider
it technical reason against it ;)

--
marko

On 7/14/2006 12:01 PM, Tom Lane wrote:

> tsearch2 is functionality that definitely should be in core eventually,
> but even Oleg still says it's not done. Aside from the documentation
> issue, it's not clear that we've got a stable API for it.

Would moving it in its current state into core help it to get there and
what would the risk be if it is added and remains fragile for a release?
If the problems are wrong positives/negatives on search results, then it
is IMHO okay. If the problems are process crashes or the like, then it
is not.

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck(at)Yahoo(dot)com #

On Fri, Jul 14, 2006 at 08:08:11PM +0200, Stefan Kaltenbrunner wrote:
> Greg Sabino Mullane wrote:
> >
> >>>> Doesn't our inclusion of md5() pretty much blow that argument away?
> >>>> (Just asking).
> >>> I don't think so because md5 is just a one way hash function. There
> >>> is no method to decrypt anything :).
> >
> > Actually, I've had to install pgcrypto on more than one occasion for
> > clients who needed to have sha1 instead of md5. I've had to install
> > pgcrypto for other functions as well, so +1 for me on coring it, but
> > at the least please consider adding in sha1.
>
> I don't have a very strong opinion on that but sha1() is something I
> need on a regular base too from pgcrypto.

sha1 would be nice, as would actual datatypes for them (though the
datatypes are probably better left to pgFoundry).
--
Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461

On Tue, Jul 18, 2006 at 03:37:52PM +0300, Marko Kreen wrote:
> On 7/14/06, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> >I don't see a strong need for moving pgcrypto into core, and there's at
> >least one argument against it: if someone needs a crypto-free version of
> >postgres for use someplace with benighted laws, they would be screwed.
>
> Image of hypothetical evil government is not a thing to base decisions on :)
>
> Although I've tried to develop pgcrypto to be easily mergable into core,
> I don't want to push it myself, the push should come from users.
>
> That said, there is one situation that is badly handled in current
> setup - storing passwords in database. There is md5() function in
> core and everything in /contrib in basically invisible in website
> and official docs. So even PG core devs suggest using md5() for
> this task. But this is inadequate - bruteforcing md5 hash can be
> done pretty easily on todays desktop computers. PostgreSQL itself
> can get away with it only because it regular users cant see the hash.
> But that is not so for ordinary apps.
>
> So I would like either some mention of the more useful/stable modules
> in core docs or a way for contrib modules to become 'official' add-on
> modules (like PL-s are).

This is actually an issue that goes way beyond pgcrypto. I think the
manual should formally mention both /contrib and pgFoundry.org as
someplace to get add-on features. An even better long-term solution
would be something akin to CPAN, but I'm not holding my breath for
that...

> Full merge into core would fix this also, but indeed there is not many
> techical reasons for it. (And editing pg_proc.h is PITA - I'd consider
> it technical reason against it ;)
>
> --
> marko
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: explain analyze is your friend
>

--
Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461

>> So I would like either some mention of the more useful/stable modules
>> in core docs or a way for contrib modules to become 'official' add-on
>> modules (like PL-s are).
>
> This is actually an issue that goes way beyond pgcrypto. I think the
> manual should formally mention both /contrib and pgFoundry.org as
> someplace to get add-on features. An even better long-term solution
> would be something akin to CPAN, but I'm not holding my breath for
> that...

The manual does talk about pgFoundry, especially in 8.2 (that was my
patch ;)).

It talks about it in 8.1 as well but it is not as apparent.

Joshua D. Drake

>
>> Full merge into core would fix this also, but indeed there is not many
>> techical reasons for it. (And editing pg_proc.h is PITA - I'd consider
>> it technical reason against it ;)
>>
>> --
>> marko
>>
>> ---------------------------(end of broadcast)---------------------------
>> TIP 6: explain analyze is your friend
>>
>

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/