Re: Proposal: syntax of operation with tsearch's configuration

Hi!

Now we (Oleg and me) are working on moving tsearch into core.

Pls, review suggested syntax. Comments, suggestions, objections will be appreciated.

1) parser operation (pg_ts_parser table)
CREATE PARSER prsname (
START = funcname,
GETTOKEN = funcname,
END = funcname,
LEXTYPES = funcname
[ , HEADLINE = funcname ]
);

DROP PARSER [IF EXISTS] prsname [ CASCADE | RESTRICT ];
ALTER PARSER prsname RENAME TO newprsname;
COMMENT ON PARSER IS text;

2) dictionaries (pg_ts_dict)

CREATE DICTIONARY dictname (
INIT = funcname,
LEXIZE = funcname,
OPT = text,
);

--create new dictionary as already existed but with different
-- options for example
CREATE DICTIONARY dictname [(
[ INIT = funcname ]
[ , LEXIZE = funcname ]
[ , OPT = text ]
)] LIKE template_dictname;

DROP DICTINARY [IF EXISTS] dictname [ CASCADE | RESTRICT ];
ALTER DICTIONARY dictname RENAME TO newdictname;
ALTER DICTIONARY dictname SET OPT=text;
COMMENT ON DICTIONARY IS text;

3) configuration (pg_ts_cfg [,pg_ts_cfgmap])
CREATE TSEARCH CONFIGURATION cfgname (
PARSER = prsname
[, LOCALE = localename]
);

--create new configuration and optionally copies
--map of lexeme's type to dictionaries
CREATE TSEARCH CONFIGURATION cfgname [(
LOCALE = localename
)] LIKE template_cfg [WITH MAP];

DROP TSEARCH CONFIGURATION [IF EXISTS] cfgname [ CASCADE | RESTRICT ];
ALTER TSEARCH CONFIGURATION cfgname RENAME TO newcfgname;
ALTER TSEARCH CONFIGURATION cfgname SET LOCALE=localename;
ALTER TSEARCH CONFIGURATION cfgname SET PARSER=prsname;
COMMENT ON TSEARCH CONFIGURATION IS text;

4) operate mapping lexemes to list of dictionary
CREATE TSEARCH MAPPING ON cfgname FOR lexemetypename USE dictname1[, dictname2
[..] ];
DROP TSEARCH MAPPING [IF EXISTS] ON cfgname FOR lexemetypename;
ALTER TSEARCH MAPPING ON cfgname FOR lexemetypename USE dictname1[, dictname2
[..] ];

Next, tsearch's configuration will be readable by psql backslashed command (F
means fulltext):
\dF - list of configurations
\dF PATTERN - describe configuration with used parser and lexeme's mapping
\dFd - list of dictionaries
\dFd PATTERN - describe dictionary
\dFp - parser's list
\dFp PATETRN - describe parser

--
Teodor Sigaev E-mail: teodor(at)sigaev(dot)ru
WWW: http://www.sigaev.ru/

--
Teodor Sigaev E-mail: teodor(at)sigaev(dot)ru
WWW: http://www.sigaev.ru/

Teodor Sigaev <teodor(at)sigaev(dot)ru> writes:
> Now we (Oleg and me) are working on moving tsearch into core.
> Pls, review suggested syntax. Comments, suggestions, objections will be appreciated.

Is it really necessary to invent a batch of special-purpose commands?
Seems like this will add some thousands of lines of code and no actual
new functionality; not to mention loss of backwards compatibility for
existing tsearch2 users.

regards, tom lane

Hmm, IMHO, it's needed for consistent interface: nobody adds new column to table
by editing pg_class & pg_attribute, nobody looks for description of table by
selection values from system table.

Tom Lane wrote:
> Teodor Sigaev <teodor(at)sigaev(dot)ru> writes:
>> Now we (Oleg and me) are working on moving tsearch into core.
>> Pls, review suggested syntax. Comments, suggestions, objections will be appreciated.
>
> Is it really necessary to invent a batch of special-purpose commands?
> Seems like this will add some thousands of lines of code and no actual
> new functionality; not to mention loss of backwards compatibility for
> existing tsearch2 users.
>
> regards, tom lane

--
Teodor Sigaev E-mail: teodor(at)sigaev(dot)ru
WWW: http://www.sigaev.ru/

Teodor Sigaev wrote:
> Hmm, IMHO, it's needed for consistent interface: nobody adds new
> column to table by editing pg_class & pg_attribute, nobody looks for
> description of table by selection values from system table.
>
>
> Tom Lane wrote:
>> Teodor Sigaev <teodor(at)sigaev(dot)ru> writes:
>>> Now we (Oleg and me) are working on moving tsearch into core.
>>> Pls, review suggested syntax. Comments, suggestions, objections will
>>> be appreciated.
>>
>> Is it really necessary to invent a batch of special-purpose commands?
>> Seems like this will add some thousands of lines of code and no actual
>> new functionality; not to mention loss of backwards compatibility for
>> existing tsearch2 users.
>>
>>
>

Thousands of lines seems a high estimate, but maybe I'm wrong. I guess
an alternative would be to do this in some builtin functions, but that
seems a tad unclean.

I am also a bit concerned that the names of the proposed objects
(parser, dictionary) don't convey their purpose adequately. Maybe
TS_DICTIONARY and TS_PARSER might be better if we in fact need to name them.

cheers

andrew

On Fri, 17 Nov 2006, Andrew Dunstan wrote:

> Teodor Sigaev wrote:
>> Hmm, IMHO, it's needed for consistent interface: nobody adds new column to
>> table by editing pg_class & pg_attribute, nobody looks for description of
>> table by selection values from system table.
>>
>>
>> Tom Lane wrote:
>>> Teodor Sigaev <teodor(at)sigaev(dot)ru> writes:
>>>> Now we (Oleg and me) are working on moving tsearch into core.
>>>> Pls, review suggested syntax. Comments, suggestions, objections will be
>>>> appreciated.
>>>
>>> Is it really necessary to invent a batch of special-purpose commands?
>>> Seems like this will add some thousands of lines of code and no actual
>>> new functionality; not to mention loss of backwards compatibility for
>>> existing tsearch2 users.
>>>
>>>
>>
>
> Thousands of lines seems a high estimate, but maybe I'm wrong. I guess an
> alternative would be to do this in some builtin functions, but that seems a
> tad unclean.

As Teodor already wrote we want to be consistent with the current interface to
system catalog, as long as full text search is going to the pg core.
We don't invent anything new, we just extending current user's interface
to support full text search.

>
> I am also a bit concerned that the names of the proposed objects (parser,
> dictionary) don't convey their purpose adequately. Maybe TS_DICTIONARY and
> TS_PARSER might be better if we in fact need to name them.

this looks reasonable to me.

>
> cheers
>
> andrew
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo(at)postgresql(dot)org so that your
> message can get through to the mailing list cleanly
>

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg(at)sai(dot)msu(dot)su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

Oleg Bartunov wrote:
> On Fri, 17 Nov 2006, Andrew Dunstan wrote:

> >I am also a bit concerned that the names of the proposed objects (parser,
> >dictionary) don't convey their purpose adequately. Maybe TS_DICTIONARY and
> >TS_PARSER might be better if we in fact need to name them.
>
> this looks reasonable to me.

Huh, but we don't use keywords with ugly abbreviations and underscores.
How about "FULLTEXT DICTIONARY" and "FULLTEXT PARSER"? (Using
"FULLTEXT" instead of "FULL TEXT" means you don't created common
reserved words, and furthermore you don't collide with an existing type
name.)

I also think the "thousands of lines" is an exaggeration :-) The
grammar should take a couple dozen at most. The rest of the code would
go to their own files.

We should also take the opportunity to discuss new keywords for the XML
support -- will we use new grammar, or functions?

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Alvaro Herrera wrote:
> We should also take the opportunity to discuss new keywords for the
> XML support -- will we use new grammar, or functions?

The XML stuff is defined in the SQL standard and there are existing
implementations, so any nonstandard syntax is going to be significantly
less useful. (The other problem is that you can't implement most of
the stuff in functions anyway.)

I don't see any comparable arguments about this full-text search stuff.
In particular I don't see any arguments why a change would necessary at
all, including why moving to core would be necessary in the first
place.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Alvaro Herrera wrote:
> Oleg Bartunov wrote:
>
>> On Fri, 17 Nov 2006, Andrew Dunstan wrote:
>>
>
>
>>> I am also a bit concerned that the names of the proposed objects (parser,
>>> dictionary) don't convey their purpose adequately. Maybe TS_DICTIONARY and
>>> TS_PARSER might be better if we in fact need to name them.
>>>
>> this looks reasonable to me.
>>
>
> Huh, but we don't use keywords with ugly abbreviations and underscores.
> How about "FULLTEXT DICTIONARY" and "FULLTEXT PARSER"? (Using
> "FULLTEXT" instead of "FULL TEXT" means you don't created common
> reserved words, and furthermore you don't collide with an existing type
> name.)
>

good point. this works for me.

>
> We should also take the opportunity to discuss new keywords for the XML
> support -- will we use new grammar, or functions?
>
>

Well, it will have to be keywords if we want to be able to do anything
like the spec, IIRC.

cheers

andrew

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> I don't see any comparable arguments about this full-text search stuff.
> In particular I don't see any arguments why a change would necessary at
> all, including why moving to core would be necessary in the first
> place.

AFAIR the only argument in favor of that is basically a marketing one:
users perceive a feature as more real, or more supported, if it's in
core. I don't find this argument especially compelling myself.

regards, tom lane

On Fri, 17 Nov 2006, Tom Lane wrote:

> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> > I don't see any comparable arguments about this full-text search stuff.
> > In particular I don't see any arguments why a change would necessary at
> > all, including why moving to core would be necessary in the first
> > place.
>
> AFAIR the only argument in favor of that is basically a marketing one:
> users perceive a feature as more real, or more supported, if it's in
> core. I don't find this argument especially compelling myself.

I am currently in the position that my hosting provider is apprehensive
about installing modules in contrib because they believe they are less
secure. They cited (real or imagined) "security holes" as the reason they
would not install tsearch2, or any other contrib module. This leaves me
without any fulltext indexing option, as it requires a superuser to
install. I have currently worked around this by running my own postgres
instance from my home directory, as they provide shell access and allow
running background processes, but I was really happy when I heard that
tsearch2 was going to be integrated into core in 8.3.

I think I would settle for some sort of assurance somewhere by someone who
sounds authoritative that the contrib modules are not less secure than
postgres core, and are fully supported by the developers. I think if I
could point them at that, I may be able to convince them that it is safe.

>
> regards, tom lane
>

Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
> I also think the "thousands of lines" is an exaggeration :-)

I think a reasonable comparison point is the operator-class commands,
which are at least in the same general ballpark of complexity.
opclasscmds.c is currently 1075 lines, and that's not counting the
grammar additions, nor miscellaneous bits of support in places like
backend/nodes/, dependency.c if you expect to be able to DROP the
objects, namespace.c if they live in schemas, aclchk.c if they have
owners or permissions, comment.c, etc. Teodor is proposing to add not
one but four new kinds of system objects. In round numbers I would
bet that such a patch will add a lot closer to 10000 lines than 1000.

It may be worth doing anyway --- certainly CREATE OPERATOR CLASS was a
huge improvement over the previous ways of doing it --- but don't
underestimate the size of what we're talking about.

regards, tom lane

Tom Lane wrote:
> Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
> > I also think the "thousands of lines" is an exaggeration :-)
>
> I think a reasonable comparison point is the operator-class commands,
> which are at least in the same general ballpark of complexity.
> opclasscmds.c is currently 1075 lines, and that's not counting the
> grammar additions, nor miscellaneous bits of support in places like
> backend/nodes/, dependency.c if you expect to be able to DROP the
> objects, namespace.c if they live in schemas, aclchk.c if they have
> owners or permissions, comment.c, etc. Teodor is proposing to add not
> one but four new kinds of system objects. In round numbers I would
> bet that such a patch will add a lot closer to 10000 lines than 1000.
>
> It may be worth doing anyway --- certainly CREATE OPERATOR CLASS was a
> huge improvement over the previous ways of doing it --- but don't
> underestimate the size of what we're talking about.

Hmm, actually the tsearch2 directory contains 16500 lines of code
(generated using David A. Wheeler's 'SLOCCount'), so I didn't doubt that
it's a big piece of code as a whole -- but I thought what was being
discussed was the size of the grammar changes, which is why I mentioned
the "a couple dozen" figure.

Having the supporting code in core does not make much of a difference
otherwise from having it in contrib, does it?

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

On Fri, 17 Nov 2006, Tom Lane wrote:

> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
>> I don't see any comparable arguments about this full-text search stuff.
>> In particular I don't see any arguments why a change would necessary at
>> all, including why moving to core would be necessary in the first
>> place.
>
> AFAIR the only argument in favor of that is basically a marketing one:
> users perceive a feature as more real, or more supported, if it's in
> core. I don't find this argument especially compelling myself.

marketing is not always "swear-word" :) We live in real world and there are
many situations where marketing is the deciding vote. Not all are
Tom Lane, who could convince customer saying there is no difference
between contrib module and core feature, or that PostgreSQL is a mature
database with fts add-on, which could be installed separately (with
supersuser rights).

I think, this is a good question for the next poll on postgresql.org.

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg(at)sai(dot)msu(dot)su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
> Tom Lane wrote:
>> It may be worth doing anyway --- certainly CREATE OPERATOR CLASS was a
>> huge improvement over the previous ways of doing it --- but don't
>> underestimate the size of what we're talking about.

> Hmm, actually the tsearch2 directory contains 16500 lines of code
> (generated using David A. Wheeler's 'SLOCCount'), so I didn't doubt that
> it's a big piece of code as a whole -- but I thought what was being
> discussed was the size of the grammar changes, which is why I mentioned
> the "a couple dozen" figure.

No, what I was on about was the cost of inventing custom-SQL-statement
manipulation of the catalog entries that drive tsearch2. The analogy to
operator classes is fairly exact, because before 7.3 you had to
manipulate those using direct insertions of catalog entries. The
manipulation commands are just about independent of the actual use of
the catalog entries --- my count of "support" didn't include any of the
planner or index AM code that actually uses operator classes, and in the
same way the existing tsearch2 code doesn't have any particular
relationship to this new code that'd have to be written to support the
manipulation commands.

> Having the supporting code in core does not make much of a difference
> otherwise from having it in contrib, does it?

Given the nonextensibility of gram.y and keywords.c, it has to be in
core to even think about having special syntax :-(

regards, tom lane

Tom Lane wrote:
> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
>> I don't see any comparable arguments about this full-text search stuff.
>> In particular I don't see any arguments why a change would necessary at
>> all, including why moving to core would be necessary in the first
>> place.
>
> AFAIR the only argument in favor of that is basically a marketing one:
> users perceive a feature as more real, or more supported, if it's in
> core. I don't find this argument especially compelling myself.

On the flip side of that argument - the more non-SQL-standard pieces
are in core, the more "non-real" other pieces non-in-core appear.
People seem to have little doubts regarding the CPAN, or Ruby Gems.
I believe because to a large part that's because a lot of very
important and well supported functionality exists outside of their
core distributions. The less that's pre-baked into core, I think
the more people will be aware of the rich set of extensions postgresql
enables.

From a marketing point of view (should I have moved this to .advocacy),
it seems to me the biggest problem is the name "contrib". If it were
called "optional" or "advanced" or "extra" I think it'd be seen less
suspiciously by hosting companies (who seem to have the biggest problem
with contrib) and we wouldn't need as many discussions of which contribs
to move into core.

Ron M

On 11/17/06, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
> Alvaro Herrera wrote:
> > We should also take the opportunity to discuss new keywords for the
> > XML support -- will we use new grammar, or functions?
>
> The XML stuff is defined in the SQL standard and there are existing
> implementations, so any nonstandard syntax is going to be significantly
> less useful. (The other problem is that you can't implement most of
> the stuff in functions anyway.)

Yes, it's better not to mix XML syntax discussion and the Tsearch2
configuration syntax discussion in one place. Not only because these
are different things - here we have a discussion of syntax for catalog
manipulation commands, when XML stuff (at least that I was working on
during summer and am going to continue) is about functionality itself.
And in case of XML we have some things to stick to - the standard
papers and existent implementations...

However, Alvaro made me to recall my old thoughts - when I just
started to use Tsearch2 I was wondering why should I explicitly create
column for index - in other databases I shouldn't do this. Indeed,
this is the index and, ideally, all I have to do is to write "CREATE
INDEX ..." only, maybe with some custom (fulltext-special) additions
(and something like "fulltext" instead of "gist").

So, is it possible to let people to avoid explicit "ALTER TABLE .. ADD
COLUMN ... tsvector"? Maybe it would be a "syntax sugar" too, but I
suppose that (especially for postgres-novices) it would simplify the
overall use of Tsearch. For me such changes are more important than
syntax for manipulations with catalog (i.e., I would live with "insert
into ts_cfg ..." one or two years more :-) ). However, I'm sure that
Oleg and Teodor already considered this feature and there should be
some things that prevent from letting users write only "CREATE INDEX"
w/o ALTERing tables...

>
> I don't see any comparable arguments about this full-text search stuff.
> In particular I don't see any arguments why a change would necessary at
> all, including why moving to core would be necessary in the first
> place.

Many hosters with PostgreSQL support (e.g. goDaddy - one of the
biggest hosters) don't provide any contrib module - so people have to
live w/o fulltext search. Then, many sysadmins are afraid of the word
"contrib"... So, there is no doubt for me that adding to core is
really good thing :-)

--
Best regards,
Nikolay

On Fri, Nov 17, 2006 at 03:53:35PM -0500, Tom Lane wrote:
> > Having the supporting code in core does not make much of a difference
> > otherwise from having it in contrib, does it?
>
> Given the nonextensibility of gram.y and keywords.c, it has to be in
> core to even think about having special syntax :-(

Has anyone ever heard of extensible grammers? Just thinking wildly, you
could decree that commands beginning with @ are extensions and are parsed
by the module listed next. Then your command set becomes:

@tsearch CREATE PARSER ....

Then contrib modules can add their own parser. You'd have the overhead
of multiple lex/yacc parsers, but you wouldn't have to change the main
parser for every extension.

Has anyone ever heard of something like this?

Have a nice day,
--
Martijn van Oosterhout <kleptog(at)svana(dot)org> http://svana.org/kleptog/
> From each according to his ability. To each according to his ability to litigate.

Martijn van Oosterhout <kleptog(at)svana(dot)org> writes:
> On Fri, Nov 17, 2006 at 03:53:35PM -0500, Tom Lane wrote:
>> Given the nonextensibility of gram.y and keywords.c, it has to be in
>> core to even think about having special syntax :-(

> Has anyone ever heard of extensible grammers?

Yeah, I worked with systems that could do that at Hewlett-Packard, nigh
thirty years ago ... but they were much less pleasant to use than bison,
and if memory serves, slower and more limited in what they could parse
(something narrower than LALR(1), IIRC, which would make certain parts
of SQL even hairier to parse than they are now). I'm not in a big hurry
to go there, even though it would certainly take some of the steam out
of "I want this in core" arguments.

> ... decree that commands beginning with @ are extensions and are parsed
> by the module listed next. Then your command set becomes:

> @tsearch CREATE PARSER ....

This'd only work well for trivial standalone commands; as a counterexample
consider CREATE INDEX, which requires access to the core sub-grammars
for typename and expression. The SQL2003 XML additions couldn't be
handled this way either.

regards, tom lane

Jeremy Drake wrote:
> I am currently in the position that my hosting provider is
> apprehensive about installing modules in contrib because they believe
> they are less secure.

Using irrational and unfounded statements one can of course make
arguments for just about anything, but that won't help us.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Oleg Bartunov wrote:
> marketing is not always "swear-word" :) We live in real world and
> there are many situations where marketing is the deciding vote.

I don't know about you, but I market PostgreSQL partially using

1. sane design, not driven by random demands
2. extensibility

which would be completely contradicted by moving any module into core
for "marketing" reasons.

> Not
> all are Tom Lane, who could convince customer saying there is no
> difference between contrib module and core feature, or that
> PostgreSQL is a mature database with fts add-on, which could be
> installed separately (with supersuser rights).

It's not like PostgreSQL is the first software product in the world to
provide a module or plugin mechanism. (It is incidentally the first
DBMS to do so.) People who refuse to understand that are idiots, and
we don't design for idiots.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

On Sat, 2006-11-18 at 00:13 +0100, Martijn van Oosterhout wrote:
> On Fri, Nov 17, 2006 at 03:53:35PM -0500, Tom Lane wrote:
> > > Having the supporting code in core does not make much of a difference
> > > otherwise from having it in contrib, does it?
> >
> > Given the nonextensibility of gram.y and keywords.c, it has to be in
> > core to even think about having special syntax :-(
>
> Has anyone ever heard of extensible grammers?

(not specifically answering Martijn...)

The main thought for me on this thread is: Why do we need to invent
*any* grammar to make this work? Why not just use functions?

For PITR we have pg_start_backup() rather than BACKUP START. For
advisory locks we have pg_advisory_lock()

What's wrong with having pg_tsearch_ functions to do everything? There's
nothing wrong with additional catalog tables/columns that are
manipulated by function calls only. We have that already - look at
pg_stat_reset() - no grammar stuff there.

Anybody with an Oracle or SQLServer background is used to seeing system
functions available as function calls; as I've observed above, so are
we. We should keep the grammar clean to allow a very close adherence to
SQL standards, IMHO.

I would like to see Oleg and Teodor's good work come into core, but I
don't want to see bucketfuls of new grammar issues.

--
Simon Riggs
EnterpriseDB http://www.enterprisedb.com

On Sat, 18 Nov 2006, Simon Riggs wrote:

> On Sat, 2006-11-18 at 00:13 +0100, Martijn van Oosterhout wrote:
>> On Fri, Nov 17, 2006 at 03:53:35PM -0500, Tom Lane wrote:
>>>> Having the supporting code in core does not make much of a difference
>>>> otherwise from having it in contrib, does it?
>>>
>>> Given the nonextensibility of gram.y and keywords.c, it has to be in
>>> core to even think about having special syntax :-(
>>
>> Has anyone ever heard of extensible grammers?
>
> (not specifically answering Martijn...)
>
> The main thought for me on this thread is: Why do we need to invent
> *any* grammar to make this work? Why not just use functions?
>
> For PITR we have pg_start_backup() rather than BACKUP START. For
> advisory locks we have pg_advisory_lock()
>
> What's wrong with having pg_tsearch_ functions to do everything? There's
> nothing wrong with additional catalog tables/columns that are
> manipulated by function calls only. We have that already - look at
> pg_stat_reset() - no grammar stuff there.
>
> Anybody with an Oracle or SQLServer background is used to seeing system
> functions available as function calls; as I've observed above, so are
> we. We should keep the grammar clean to allow a very close adherence to
> SQL standards, IMHO.
>
> I would like to see Oleg and Teodor's good work come into core, but I
> don't want to see bucketfuls of new grammar issues.

Summarizing, we have two questions -

1. Will tsearch comes to the core
2. Do we need grammar changes

I hope, we have consensus about 1. - we need fts as a core feature.
Second question is not very principal, that's why we asked -hackers.
So, if we'll not touch grammar, are there any issues with tsearch2 in core ?

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg(at)sai(dot)msu(dot)su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

Oleg Bartunov wrote:
> So, if we'll not touch grammar, are there any issues with tsearch2 in
> core ?

Are there any issues with tsearch2 not in core?

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Hi,

Peter Eisentraut wrote:
> Are there any issues with tsearch2 not in core?

I have run into troubles when restoring a dump, especially across
different versions of PostgreSQL and tsearch2. Mainly because pg_ts_*
are not system tables and thus need to be restored or installed separately.

And there still is the packaging issue which needs to be addressed. It's
not complicated, but a PITA to compile stemmers and setup custom
dictionaries.

What's really needed IMO is a clever packaging, including stemmers and
dictionaries for as many languages as we can come up with. So on a
debian system, it should become as simple as:

apt-get install postgresql-contrib-8.3
apt-get install postgresql-language-pack-english-8.3
apt-get install postgresql-language-pack-german-8.3
apt-get install postgresql-language-russian-german-8.3

Inclusion into core surely does not help with that.

Relabeling contrib to modules or extras or something would probably give
some people a warm fuzzy feeling when installing. OTOH, these are
probably the very same people who get excited about tsearch2 in core, so
if we want to satisfy them, we better put it right into core... I dunno.

Regards

Markus

Peter Eisentraut wrote:
> Oleg Bartunov wrote:
>> So, if we'll not touch grammar, are there any issues with tsearch2 in
>> core ?
>
> Are there any issues with tsearch2 not in core?
>

Quite apart from anything else, it really needs documentation of the
standard we give other core features.

I think if a feature will be of sufficiently general use it should be a
candidate for inclusion, and text search certainly comes within that
category in my mind.

cheers

andrew

On Sat, 18 Nov 2006, Andrew Dunstan wrote:

> Peter Eisentraut wrote:
>> Oleg Bartunov wrote:
>>> So, if we'll not touch grammar, are there any issues with tsearch2 in
>>> core ?
>>
>> Are there any issues with tsearch2 not in core?
>>
>
>
> Quite apart from anything else, it really needs documentation of the
> standard we give other core features.

Sure. I just learned how to built (successfully) pg documentation and
researching on what's documentation standard. Should we need to write
separate full text search chapter and/or add description to relevant
chapters.

>
> I think if a feature will be of sufficiently general use it should be a
> candidate for inclusion, and text search certainly comes within that
> category in my mind.

It could helps us in Pg-MySQL discussions, at least, since we beat
mysql's fts :)

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg(at)sai(dot)msu(dot)su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

Andrew Dunstan wrote:
> Peter Eisentraut wrote:
>> Oleg Bartunov wrote:
>>> So, if we'll not touch grammar, are there any issues with tsearch2 in
>>> core ?
>> Are there any issues with tsearch2 not in core?
>>
>
>
> Quite apart from anything else, it really needs documentation of the
> standard we give other core features.
>
> I think if a feature will be of sufficiently general use it should be a
> candidate for inclusion, and text search certainly comes within that
> category in my mind.

I agree here - full text search is of general use (and a very often
requested) feature - including it in core will both help us in marketing
postgresql (which should notbe seen as a "bad" thing at all) and more to
the point it provides an in-core user and showcase for two very powerful
and innovative technologies - GIST and GIN.

Stefan

Alvaro Herrera wrote:
> Oleg Bartunov wrote:
>> On Fri, 17 Nov 2006, Andrew Dunstan wrote:
>
>>> I am also a bit concerned that the names of the proposed objects (parser,
>>> dictionary) don't convey their purpose adequately. Maybe TS_DICTIONARY and
>>> TS_PARSER might be better if we in fact need to name them.
>> this looks reasonable to me.
>
> Huh, but we don't use keywords with ugly abbreviations and underscores.
> How about "FULLTEXT DICTIONARY" and "FULLTEXT PARSER"? (Using
> "FULLTEXT" instead of "FULL TEXT" means you don't created common
> reserved words, and furthermore you don't collide with an existing type
> name.)

sounds fine

>
> I also think the "thousands of lines" is an exaggeration :-) The
> grammar should take a couple dozen at most. The rest of the code would
> go to their own files.
>
> We should also take the opportunity to discuss new keywords for the XML
> support -- will we use new grammar, or functions?
>

that is a good question and we should decide on a direction for that -
we already have a feature in shipping code that causes quite some
confusion in that regard(autovacuum).
What see I from supporting/consulting people is that there are more and
more people adapting autovacuum for there databases but those with
complex ones want to override them on a per table base.
We already provide a rather crude interface for that - namely manually
inserting some rows into a system table which is confusing the heck out
of people (they are either responding with "why is there now ALTER
AUTOVACUUM SET ..." or and equivalent pg_* function for that).
I'm not too sure what the most suitable interface for that would be but
finding a consistent solution for that might be good nevertheless.

Stefan

Peter Eisentraut wrote:
> Oleg Bartunov wrote:
> > So, if we'll not touch grammar, are there any issues with tsearch2 in
> > core ?
>
> Are there any issues with tsearch2 not in core?

No, but many think the idea of moving well-established code from
/contrib into the backend is true for tsearch2 once it works for
multi-byte encodings.

--
Bruce Momjian bruce(at)momjian(dot)us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +