Re: Filtering dictionaries support and unaccent dictionary

Hi there,

we'd like to introduce filtering dictionaries support for text search
and new contrib module unaccent, which provides useful example of
filtering dictionary. It finally solves the known problem of
incorrect generation of headlines of text with accents.

Also, this module provides unaccent() functions, which is a simple
wrapper on unaccent dictionary.

Regards,
Oleg

PS. I hope it's not late for July commitfest !

_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg(at)sai(dot)msu(dot)su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

Oleg Bartunov wrote:

Hi,

> we'd like to introduce filtering dictionaries support for text search
> and new contrib module unaccent, which provides useful example of
> filtering dictionary. It finally solves the known problem of incorrect
> generation of headlines of text with accents.

I'm curious about the pg_regress change ... is it really necessary?

AFAICS the changes to the core code are very small; I wonder if you
should commit it separately i.e. without the contrib module, and add the
that one in another commit.

As for the contrib module, I think it could use a lot more function
header comments! Also, it would be great if it could be used separately
from tsearch, i.e. that it provided a function unaccent(text) returns
text that unaccented arbitrary strings (I guess it would use the default
tsconfig).

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

> I'm curious about the pg_regress change ... is it really necessary?

To test unaccent dictionary it's needed to input accented characters, not all
encodings allow that. UTF8 allows that, but it doesn't compatible with a lot of
locales. So, --no-locale should be propagated to CREATE DATABASE command as it's
done for encoding.

> AFAICS the changes to the core code are very small; I wonder if you
> should commit it separately i.e. without the contrib module, and add the
> that one in another commit.
Split patch to two parts:
filter_dictionary-0.1.gz - core changes, including pg_regress changes
unaccent-0.5.gz - contrib module

Also, I added some comments into code and did cosmetic changes in docs.

> As for the contrib module, I think it could use a lot more function
> header comments! Also, it would be great if it could be used separately
> from tsearch, i.e. that it provided a function unaccent(text) returns
> text that unaccented arbitrary strings (I guess it would use the default
> tsconfig).
Umm? Module provides unaccent(text) and unaccent(regdictionary, text) functions.

--
Teodor Sigaev E-mail: teodor(at)sigaev(dot)ru
WWW: http://www.sigaev.ru/

On Tuesday 14 July 2009 22:12:28 Oleg Bartunov wrote:
> we'd like to introduce filtering dictionaries support for text search
> and new contrib module unaccent, which provides useful example of
> filtering dictionary. It finally solves the known problem of
> incorrect generation of headlines of text with accents.

What is the source of the unaccent rules, and how complete is the rule set?

On Wed, 29 Jul 2009, Peter Eisentraut wrote:

> On Tuesday 14 July 2009 22:12:28 Oleg Bartunov wrote:
>> we'd like to introduce filtering dictionaries support for text search
>> and new contrib module unaccent, which provides useful example of
>> filtering dictionary. It finally solves the known problem of
>> incorrect generation of headlines of text with accents.
>
> What is the source of the unaccent rules, and how complete is the rule set?

unicode tables from unicode.org. It'be nice if someone check the completeness.

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg(at)sai(dot)msu(dot)su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

Teodor Sigaev wrote:

> >As for the contrib module, I think it could use a lot more function
> >header comments! Also, it would be great if it could be used separately
> >from tsearch, i.e. that it provided a function unaccent(text) returns
> >text that unaccented arbitrary strings (I guess it would use the default
> >tsconfig).
> Umm? Module provides unaccent(text) and unaccent(regdictionary, text) functions.

Sorry, I failed to notice. Looks good.

Isn't that function leaking "res" pointer? Also, I'm curious why you're
allocating 2*sizeof(TSLexeme) in unaccent_lexize ...

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

On Sat, Aug 1, 2009 at 12:35 AM, Alvaro
Herrera<alvherre(at)commandprompt(dot)com> wrote:
> Teodor Sigaev wrote:
>
>> >As for the contrib module, I think it could use a lot more function
>> >header comments!  Also, it would be great if it could be used separately
>> >from tsearch, i.e. that it provided a function unaccent(text) returns
>> >text that unaccented arbitrary strings (I guess it would use the default
>> >tsconfig).
>> Umm? Module provides unaccent(text) and unaccent(regdictionary, text) functions.
>
> Sorry, I failed to notice.  Looks good.
>
> Isn't that function leaking "res" pointer?  Also, I'm curious why you're
> allocating 2*sizeof(TSLexeme) in unaccent_lexize ...

So are we waiting for an updated version of this patch?

...Robert

> Isn't that function leaking "res" pointer? Also, I'm curious why you're
fixed

> allocating 2*sizeof(TSLexeme) in unaccent_lexize ...
That's is a dictionary's interface part: lexize returns an array of TSLexeme and
last structure should have lexeme field NULL.

filter_dictionary file is not changed, it's attached only for consistency.
--
Teodor Sigaev E-mail: teodor(at)sigaev(dot)ru
WWW: http://www.sigaev.ru/

2009/8/6 Teodor Sigaev <teodor(at)sigaev(dot)ru>:
>> Isn't that function leaking "res" pointer?  Also, I'm curious why you're
>
> fixed
>
>> allocating 2*sizeof(TSLexeme) in unaccent_lexize ...
>
> That's is a dictionary's interface part: lexize returns an array of TSLexeme
> and last structure should have lexeme field NULL.
>
>
> filter_dictionary file is not changed, it's attached only for consistency.

I am not sure whether this has been formally reviewed by anyone yet;
do we think it's "Ready for Committer"?

Thanks,

...Robert

On Thu, Aug 6, 2009 at 10:46 PM, Robert Haas<robertmhaas(at)gmail(dot)com> wrote:
>
> I am not sure whether this has been formally reviewed by anyone yet;
> do we think it's "Ready for Committer"?
>

i was trying to make some review of this but besides that it compiles
fine and passes regression tests doesn't know how to test it

--
Atentamente,
Jaime Casanova
Soporte y capacitación de PostgreSQL
Asesoría y desarrollo de sistemas
Guayaquil - Ecuador
Cel. +59387171157

On Fri, Aug 7, 2009 at 10:44 AM, Jaime
Casanova<jcasanov(at)systemguards(dot)com(dot)ec> wrote:
> On Thu, Aug 6, 2009 at 10:46 PM, Robert Haas<robertmhaas(at)gmail(dot)com> wrote:
>>
>> I am not sure whether this has been formally reviewed by anyone yet;
>> do we think it's "Ready for Committer"?
>>
>
> i was trying to make some review of this but besides that it compiles
> fine and passes regression tests doesn't know how to test it
>

try to build the docs to see how to properly test this and seems like
you have to teach contrib.sgml and bookindex.sgml about
dict-unaccent... and when i did that i got this:

"""
openjade -wall -wno-unused-param -wno-empty -wfully-tagged -D . -c
/usr/share/sgml/docbook/stylesheet/dsssl/modular/catalog -d
stylesheet.dsl -t sgml -i output-html -V html-index postgres.sgml
openjade:dict-unaccent.sgml:48:1:E: non SGML character number 128
openjade:dict-unaccent.sgml:49:1:E: non SGML character number 129
openjade:dict-unaccent.sgml:50:1:E: non SGML character number 130
openjade:dict-unaccent.sgml:51:1:E: non SGML character number 131
openjade:dict-unaccent.sgml:52:1:E: non SGML character number 132
openjade:dict-unaccent.sgml:53:1:E: non SGML character number 133
openjade:dict-unaccent.sgml:54:1:E: non SGML character number 134
openjade:dict-unaccent.sgml:116:4:E: element "B" undefined
make: *** [HTML.index] Error 1
make: *** Se borra el archivo `HTML.index'
"""
--
Atentamente,
Jaime Casanova
Soporte y capacitación de PostgreSQL
Asesoría y desarrollo de sistemas
Guayaquil - Ecuador
Cel. +59387171157

On Tuesday 11 August 2009 08:28:24 Jaime Casanova wrote:
> try to build the docs to see how to properly test this and seems like
> you have to teach contrib.sgml and bookindex.sgml about
> dict-unaccent... and when i did that i got this:
>
> """
> openjade -wall -wno-unused-param -wno-empty -wfully-tagged -D . -c
> /usr/share/sgml/docbook/stylesheet/dsssl/modular/catalog -d
> stylesheet.dsl -t sgml -i output-html -V html-index postgres.sgml
> openjade:dict-unaccent.sgml:48:1:E: non SGML character number 128
> openjade:dict-unaccent.sgml:49:1:E: non SGML character number 129
> openjade:dict-unaccent.sgml:50:1:E: non SGML character number 130
> openjade:dict-unaccent.sgml:51:1:E: non SGML character number 131
> openjade:dict-unaccent.sgml:52:1:E: non SGML character number 132
> openjade:dict-unaccent.sgml:53:1:E: non SGML character number 133
> openjade:dict-unaccent.sgml:54:1:E: non SGML character number 134
> openjade:dict-unaccent.sgml:116:4:E: element "B" undefined
> make: *** [HTML.index] Error 1
> make: *** Se borra el archivo `HTML.index'
> """

You should escape the special characters as well as the <b> that appears as
part of the example output using character entitities (&amp; etc.).

On Tue, Aug 11, 2009 at 4:31 AM, Peter Eisentraut<peter_e(at)gmx(dot)net> wrote:
> On Tuesday 11 August 2009 08:28:24 Jaime Casanova wrote:
>> try to build the docs to see how to properly test this and seems like
>> you have to teach contrib.sgml and bookindex.sgml about
>> dict-unaccent... and when i did that i got this:
>>
>> """
>> openjade  -wall -wno-unused-param -wno-empty -wfully-tagged -D . -c
>> /usr/share/sgml/docbook/stylesheet/dsssl/modular/catalog -d
>> stylesheet.dsl -t sgml -i output-html -V html-index postgres.sgml
>> openjade:dict-unaccent.sgml:48:1:E: non SGML character number 128
>> openjade:dict-unaccent.sgml:49:1:E: non SGML character number 129
>> openjade:dict-unaccent.sgml:50:1:E: non SGML character number 130
>> openjade:dict-unaccent.sgml:51:1:E: non SGML character number 131
>> openjade:dict-unaccent.sgml:52:1:E: non SGML character number 132
>> openjade:dict-unaccent.sgml:53:1:E: non SGML character number 133
>> openjade:dict-unaccent.sgml:54:1:E: non SGML character number 134
>> openjade:dict-unaccent.sgml:116:4:E: element "B" undefined
>> make: *** [HTML.index] Error 1
>> make: *** Se borra el archivo `HTML.index'
>> """
>
> You should escape the special characters as well as the <b> that appears as
> part of the example output using character entitities (&amp; etc.).

Sounds like this patch needs a little bit of doc adjustment per the
above and is then ready for committer?

...Robert

Peter,

how to write accented characters in sgml ? Is't not allowed to write them
as is ?

Oleg
On Tue, 11 Aug 2009, Peter Eisentraut wrote:

> On Tuesday 11 August 2009 08:28:24 Jaime Casanova wrote:
>> try to build the docs to see how to properly test this and seems like
>> you have to teach contrib.sgml and bookindex.sgml about
>> dict-unaccent... and when i did that i got this:
>>
>> """
>> openjade -wall -wno-unused-param -wno-empty -wfully-tagged -D . -c
>> /usr/share/sgml/docbook/stylesheet/dsssl/modular/catalog -d
>> stylesheet.dsl -t sgml -i output-html -V html-index postgres.sgml
>> openjade:dict-unaccent.sgml:48:1:E: non SGML character number 128
>> openjade:dict-unaccent.sgml:49:1:E: non SGML character number 129
>> openjade:dict-unaccent.sgml:50:1:E: non SGML character number 130
>> openjade:dict-unaccent.sgml:51:1:E: non SGML character number 131
>> openjade:dict-unaccent.sgml:52:1:E: non SGML character number 132
>> openjade:dict-unaccent.sgml:53:1:E: non SGML character number 133
>> openjade:dict-unaccent.sgml:54:1:E: non SGML character number 134
>> openjade:dict-unaccent.sgml:116:4:E: element "B" undefined
>> make: *** [HTML.index] Error 1
>> make: *** Se borra el archivo `HTML.index'
>> """
>
> You should escape the special characters as well as the <b> that appears as
> part of the example output using character entitities (&amp; etc.).
>

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg(at)sai(dot)msu(dot)su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

Oleg Bartunov wrote:
> Peter,
>
> how to write accented characters in sgml ? Is't not allowed to write
> them as is ?

&aacute; for á, etc. You can't use characters that aren't in Latin-1 I think.
Writing them literally is not allowed.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

On Thursday 13 August 2009 18:07:51 Alvaro Herrera wrote:
> Oleg Bartunov wrote:
> > Peter,
> >
> > how to write accented characters in sgml ? Is't not allowed to write
> > them as is ?
>
> &aacute; for á, etc. You can't use characters that aren't in Latin-1 I
> think. Writing them literally is not allowed.

It's somehow possible, but it's not as straightforward as say with XML. And
you might get into a Latin-1 vs UTF-8 mixup. At least that's what I noticed
in my limited testing the other day.

Peter Eisentraut wrote:
> On Thursday 13 August 2009 18:07:51 Alvaro Herrera wrote:
> > Oleg Bartunov wrote:
> > > Peter,
> > >
> > > how to write accented characters in sgml ? Is't not allowed to write
> > > them as is ?
> >
> > &aacute; for ?, etc. You can't use characters that aren't in Latin-1 I
> > think. Writing them literally is not allowed.
>
> It's somehow possible, but it's not as straightforward as say with XML. And
> you might get into a Latin-1 vs UTF-8 mixup. At least that's what I noticed
> in my limited testing the other day.

The top of release.sgml has instructions on that because that is often
something we need to do for names in release notes:

non-ASCII characters convert to HTML4 entity (&) escapes

official: http://www.w3.org/TR/html4/sgml/entities.html
one page: http://www.zipcon.net/~swhite/docs/computers/browsers/entities_page.html
other lists: http://www.zipcon.net/~swhite/docs/computers/browsers/entities.html
http://www.zipcon.net/~swhite/docs/computers/browsers/entities_page.html
http://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_entity_references

we cannot use UTF8 because SGML Docbook
does not support it
http://www.pemberley.com/janeinfo/latin1.html#latexta

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Thanks, Bruce !

Oleg
On Thu, 13 Aug 2009, Bruce Momjian wrote:

> Peter Eisentraut wrote:
>> On Thursday 13 August 2009 18:07:51 Alvaro Herrera wrote:
>>> Oleg Bartunov wrote:
>>>> Peter,
>>>>
>>>> how to write accented characters in sgml ? Is't not allowed to write
>>>> them as is ?
>>>
>>> &aacute; for ?, etc. You can't use characters that aren't in Latin-1 I
>>> think. Writing them literally is not allowed.
>>
>> It's somehow possible, but it's not as straightforward as say with XML. And
>> you might get into a Latin-1 vs UTF-8 mixup. At least that's what I noticed
>> in my limited testing the other day.
>
> The top of release.sgml has instructions on that because that is often
> something we need to do for names in release notes:
>
> non-ASCII characters convert to HTML4 entity (&) escapes
>
> official: http://www.w3.org/TR/html4/sgml/entities.html
> one page: http://www.zipcon.net/~swhite/docs/computers/browsers/entities_page.html
> other lists: http://www.zipcon.net/~swhite/docs/computers/browsers/entities.html
> http://www.zipcon.net/~swhite/docs/computers/browsers/entities_page.html
> http://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_entity_references
>
> we cannot use UTF8 because SGML Docbook
> does not support it
> http://www.pemberley.com/janeinfo/latin1.html#latexta
>
>

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg(at)sai(dot)msu(dot)su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83