From: | Oliver Jowett <oliver(at)opencloud(dot)com> |
---|---|
To: | pg(at)fastcrypt(dot)com |
Cc: | "pgsql-jdbc(at)postgresql(dot)org" <pgsql-jdbc(at)postgresql(dot)org> |
Subject: | Re: patch for getXXX methods |
Date: | 2004-07-11 23:37:56 |
Message-ID: | 40F1CF54.1040606@opencloud.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-jdbc |
Dave Cramer wrote:
> Oliver,
>
> I don't believe you will lose precision if the number is below MAX_LONG
> ? When I tested it on my system, I was able to retrieve a double that
> was equal to MAX_LONG without losing precision.
The attached testcase says otherwise. It produces this output:
> Mismatch: 9223372036854775806 => 9.223372036854776E18 => 9223372036854775807
> Mismatch: 9223372036854775805 => 9.223372036854776E18 => 9223372036854775807
> Mismatch: 9223372036854775804 => 9.223372036854776E18 => 9223372036854775807
> Mismatch: 9223372036854775803 => 9.223372036854776E18 => 9223372036854775807
> Mismatch: 9223372036854775802 => 9.223372036854776E18 => 9223372036854775807
> Mismatch: 9223372036854775801 => 9.223372036854776E18 => 9223372036854775807
> Mismatch: 9223372036854775800 => 9.223372036854776E18 => 9223372036854775807
> Mismatch: 9223372036854775799 => 9.223372036854776E18 => 9223372036854775807
[...]
> Mismatch: 9223372036854775296 => 9.223372036854776E18 => 9223372036854775807
> Mismatch: 9223372036854775295 => 9.2233720368547748E18 => 9223372036854774784
> Mismatch: 9223372036854775294 => 9.2233720368547748E18 => 9223372036854774784
and so on.
The problem is that near MAX_LONG you need almost 64 bits of mantissa to
exactly represent the value -- but a double is only a 64-bit value
including space for the exponent. I can't remember the exact split but
from the above it looks like there is around 10 bits of exponent so you
only have ~54 bits for the mantissa -- so you only get a precision of
about +/- 512 when you're dealing with numbers of a magnitude around
MAX_LONG.
-O
Attachment | Content-Type | Size |
---|---|---|
TestDoublePrecision.java | text/x-java | 296 bytes |
From | Date | Subject | |
---|---|---|---|
Next Message | Oliver Jowett | 2004-07-11 23:49:52 | Re: Timestamp Question |
Previous Message | Greg Markham | 2004-07-11 20:54:27 | Re: Timestamp Question |