Re: [WebDNA] Strange MATH result

This WebDNA talk-list message is from

2010


It keeps the original formatting.
numero = 104389
interpreted = N
texte = --00504502d56759fe8d047cf99342 Content-Type: text/plain; charset=ISO-8859-1 On Tue, Jan 12, 2010 at 5:33 AM, wrote: > Temporary solution is to specify how many decimal numbers you want, using > [format] > > > [format 6.2f][math]36.70+0.00+0.00+0.00-6.95-29.75[/math][/format] > > - chris > > On Jan 12, 2010, at 2:17, Stuart Tremain wrote: > > > Why does [math]36.70+0.00+0.00+0.00-6.95-29.75[/math] = > 3.5527136788005e-015 ????? > The issue you have run in to is common with computer programming. I would categorize it as a computer "bug". http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems It is similar to other issues that you may run in to such as: 9223372036854775807 + 1 = -9223372036854775807 or 2147483647 + 1 = -2147483647 (2^63-1) + 1 = -(2&63) Computers have certain amounts of precision, and sometimes subtracting a number from itself may result in a number that is very nearly zero, but not quite. The solution is to use the format context, as Christophe stated. --00504502d56759fe8d047cf99342 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
On Tue, Jan 12, 2010 at 5:33 AM, <christophe.billi= ottet@webdna.us> wrote:
Temporary solution is to specify how many decimal numbers you want, using [= format]


[format 6.2f][math]36.70+0.00+0.00+0.00-6.95-29.75[/math][/format]

- chris

On Jan 12, 2010, at 2:17, Stuart Tremain wrote:

> Why does [math]36.70+0.00+0.00= +0.00-6.95-29.75[/math] =3D 3.5527136788005e-015 ?????
The issue you have run in to is common with computer programmi= ng.=A0 I would categorize it as a computer "bug".

h= ttp://en.wikipedia.org/wiki/Floating_point#Accuracy_problems

It is similar to other issues that you may run in to such as:
9223372036854775807 + 1 =3D -9223372036854775807
or
2147483647 + 1 =3D -2147483647

(2^63-1) + 1 =3D -(2&63)

Computers have certain amounts of precision, and sometimes subtracting a number from itself may result in a number that is very nearly zero, but not quite.


The solution is to use the format context, as Christophe stated.


--00504502d56759fe8d047cf99342-- Associated Messages, from the most recent to the oldest:

    
  1. Re: [WebDNA] Strange MATH result (Stuart Tremain 2010)
  2. Re: [WebDNA] Strange MATH result (JJ WebDNA 2010)
  3. Re: [WebDNA] Strange MATH result - BUG (Govinda 2010)
  4. Re: [WebDNA] Strange MATH result (christophe.billiottet@webdna.us 2010)
  5. Re: [WebDNA] Strange MATH result (christophe.billiottet@webdna.us 2010)
  6. Re: [WebDNA] Strange MATH result - BUG (Stuart Tremain 2010)
  7. Re: [WebDNA] Strange MATH result - BUG (Govinda 2010)
  8. Re: [WebDNA] Strange MATH result (Stuart Tremain 2010)
  9. Re: [WebDNA] Strange MATH result (Govinda 2010)
  10. Re: [WebDNA] Strange MATH result - BUG (Stuart Tremain 2010)
  11. [WebDNA] Strange MATH result (Stuart Tremain 2010)
--00504502d56759fe8d047cf99342 Content-Type: text/plain; charset=ISO-8859-1 On Tue, Jan 12, 2010 at 5:33 AM, wrote: > Temporary solution is to specify how many decimal numbers you want, using > [format] > > > [format 6.2f][math]36.70+0.00+0.00+0.00-6.95-29.75[/math][/format] > > - chris > > On Jan 12, 2010, at 2:17, Stuart Tremain wrote: > > > Why does [math]36.70+0.00+0.00+0.00-6.95-29.75[/math] = > 3.5527136788005e-015 ????? > The issue you have run in to is common with computer programming. I would categorize it as a computer "bug". http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems It is similar to other issues that you may run in to such as: 9223372036854775807 + 1 = -9223372036854775807 or 2147483647 + 1 = -2147483647 (2^63-1) + 1 = -(2&63) Computers have certain amounts of precision, and sometimes subtracting a number from itself may result in a number that is very nearly zero, but not quite. The solution is to use the format context, as Christophe stated. --00504502d56759fe8d047cf99342 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
On Tue, Jan 12, 2010 at 5:33 AM, <christophe.billi= ottet@webdna.us> wrote:
Temporary solution is to specify how many decimal numbers you want, using [= format]


[format 6.2f][math]36.70+0.00+0.00+0.00-6.95-29.75[/math][/format]

- chris

On Jan 12, 2010, at 2:17, Stuart Tremain wrote:

> Why does [math]36.70+0.00+0.00= +0.00-6.95-29.75[/math] =3D 3.5527136788005e-015 ?????
The issue you have run in to is common with computer programmi= ng.=A0 I would categorize it as a computer "bug".

h= ttp://en.wikipedia.org/wiki/Floating_point#Accuracy_problems

It is similar to other issues that you may run in to such as:
9223372036854775807 + 1 =3D -9223372036854775807
or
2147483647 + 1 =3D -2147483647

(2^63-1) + 1 =3D -(2&63)

Computers have certain amounts of precision, and sometimes subtracting a number from itself may result in a number that is very nearly zero, but not quite.


The solution is to use the format context, as Christophe stated.


--00504502d56759fe8d047cf99342-- JJ WebDNA

DOWNLOAD WEBDNA NOW!

Top Articles:

Talk List

The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...

Related Readings:

Problems with Price field (1997) japanese characters (1997) Speed Admin? (Was FlushDatabases) (1997) Summing fields (1997) Requiring that certain fields be completed (1997) Redirect frame targets (1998) Migrating to NT (1997) Database Locking?!? (2000) ShowNext (1997) Credit card processing - UK (1997) [WebDNA] Need some PHP help (2009) Two stores, one server (1998) quantity minimum problem (1997) I'm tired of all this! (2000) Test (2003) WebCatalog - Mac vs Linux (2000) Server slowing down. (1997) oops private message leaked into talk list (1997) passing search criteria (1997) Exists? (1997)