Re: Modulus operator returns negative values / numeric division rounds up sometimes

From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Paul Tillotson <pntil(at)shentel(dot)net>
Cc: Postgres General <pgsql-general(at)postgresql(dot)org>
Subject: Re: Modulus operator returns negative values / numeric division rounds up sometimes
Date: 2005-05-26 05:18:00
Message-ID: 23606.1117084680@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

Paul Tillotson <pntil(at)shentel(dot)net> writes:
> I don't think anyone wants to defend the negative modulus as such, but to fix it, we have to do one of these:

> (1) Keep rounding division, but rewrite the numeric modulus operator to use a form of division that always rounds towards zero.

> or

> (2) Give up rounding division in favor of truncating towards zero.

or (3) increase the calculation precision (rscale), as suggested by
Alvaro's message.

Possibly that cannot work, but I haven't seen a proof.

> It looks like the "bug" can be easily fixed by changing the end of div_var where it says
> round_var(result, rscale);
> to
> trunc_var(result, scale);

I cannot believe that that won't create problems at least as bad as it
solves. Have you even tried the regression tests on this?

regards, tom lane

In response to

Responses

Browse pgsql-general by date

  From Date Subject
Next Message Dave E Martin 2005-05-26 06:33:20 enable_sort optimization problem
Previous Message Paul Tillotson 2005-05-26 04:43:12 Modulus operator returns negative values / numeric division rounds up sometimes