Re: [WebDNA] Strange MATH result

This WebDNA talk-list message is from

2010


It keeps the original formatting.
numero = 104389
interpreted = N
texte = --00504502d56759fe8d047cf99342 Content-Type: text/plain; charset=ISO-8859-1 On Tue, Jan 12, 2010 at 5:33 AM, wrote: > Temporary solution is to specify how many decimal numbers you want, using > [format] > > > [format 6.2f][math]36.70+0.00+0.00+0.00-6.95-29.75[/math][/format] > > - chris > > On Jan 12, 2010, at 2:17, Stuart Tremain wrote: > > > Why does [math]36.70+0.00+0.00+0.00-6.95-29.75[/math] = > 3.5527136788005e-015 ????? > The issue you have run in to is common with computer programming. I would categorize it as a computer "bug". http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems It is similar to other issues that you may run in to such as: 9223372036854775807 + 1 = -9223372036854775807 or 2147483647 + 1 = -2147483647 (2^63-1) + 1 = -(2&63) Computers have certain amounts of precision, and sometimes subtracting a number from itself may result in a number that is very nearly zero, but not quite. The solution is to use the format context, as Christophe stated. --00504502d56759fe8d047cf99342 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
On Tue, Jan 12, 2010 at 5:33 AM, <christophe.billi= ottet@webdna.us> wrote:
Temporary solution is to specify how many decimal numbers you want, using [= format]


[format 6.2f][math]36.70+0.00+0.00+0.00-6.95-29.75[/math][/format]

- chris

On Jan 12, 2010, at 2:17, Stuart Tremain wrote:

> Why does [math]36.70+0.00+0.00= +0.00-6.95-29.75[/math] =3D 3.5527136788005e-015 ?????
The issue you have run in to is common with computer programmi= ng.=A0 I would categorize it as a computer "bug".

h= ttp://en.wikipedia.org/wiki/Floating_point#Accuracy_problems

It is similar to other issues that you may run in to such as:
9223372036854775807 + 1 =3D -9223372036854775807
or
2147483647 + 1 =3D -2147483647

(2^63-1) + 1 =3D -(2&63)

Computers have certain amounts of precision, and sometimes subtracting a number from itself may result in a number that is very nearly zero, but not quite.


The solution is to use the format context, as Christophe stated.


--00504502d56759fe8d047cf99342-- Associated Messages, from the most recent to the oldest:

    
  1. Re: [WebDNA] Strange MATH result (Stuart Tremain 2010)
  2. Re: [WebDNA] Strange MATH result (JJ WebDNA 2010)
  3. Re: [WebDNA] Strange MATH result - BUG (Govinda 2010)
  4. Re: [WebDNA] Strange MATH result (christophe.billiottet@webdna.us 2010)
  5. Re: [WebDNA] Strange MATH result (christophe.billiottet@webdna.us 2010)
  6. Re: [WebDNA] Strange MATH result - BUG (Stuart Tremain 2010)
  7. Re: [WebDNA] Strange MATH result - BUG (Govinda 2010)
  8. Re: [WebDNA] Strange MATH result (Stuart Tremain 2010)
  9. Re: [WebDNA] Strange MATH result (Govinda 2010)
  10. Re: [WebDNA] Strange MATH result - BUG (Stuart Tremain 2010)
  11. [WebDNA] Strange MATH result (Stuart Tremain 2010)
--00504502d56759fe8d047cf99342 Content-Type: text/plain; charset=ISO-8859-1 On Tue, Jan 12, 2010 at 5:33 AM, wrote: > Temporary solution is to specify how many decimal numbers you want, using > [format] > > > [format 6.2f][math]36.70+0.00+0.00+0.00-6.95-29.75[/math][/format] > > - chris > > On Jan 12, 2010, at 2:17, Stuart Tremain wrote: > > > Why does [math]36.70+0.00+0.00+0.00-6.95-29.75[/math] = > 3.5527136788005e-015 ????? > The issue you have run in to is common with computer programming. I would categorize it as a computer "bug". http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems It is similar to other issues that you may run in to such as: 9223372036854775807 + 1 = -9223372036854775807 or 2147483647 + 1 = -2147483647 (2^63-1) + 1 = -(2&63) Computers have certain amounts of precision, and sometimes subtracting a number from itself may result in a number that is very nearly zero, but not quite. The solution is to use the format context, as Christophe stated. --00504502d56759fe8d047cf99342 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
On Tue, Jan 12, 2010 at 5:33 AM, <christophe.billi= ottet@webdna.us> wrote:
Temporary solution is to specify how many decimal numbers you want, using [= format]


[format 6.2f][math]36.70+0.00+0.00+0.00-6.95-29.75[/math][/format]

- chris

On Jan 12, 2010, at 2:17, Stuart Tremain wrote:

> Why does [math]36.70+0.00+0.00= +0.00-6.95-29.75[/math] =3D 3.5527136788005e-015 ?????
The issue you have run in to is common with computer programmi= ng.=A0 I would categorize it as a computer "bug".

h= ttp://en.wikipedia.org/wiki/Floating_point#Accuracy_problems

It is similar to other issues that you may run in to such as:
9223372036854775807 + 1 =3D -9223372036854775807
or
2147483647 + 1 =3D -2147483647

(2^63-1) + 1 =3D -(2&63)

Computers have certain amounts of precision, and sometimes subtracting a number from itself may result in a number that is very nearly zero, but not quite.


The solution is to use the format context, as Christophe stated.


--00504502d56759fe8d047cf99342-- JJ WebDNA

DOWNLOAD WEBDNA NOW!

Top Articles:

Talk List

The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...

Related Readings:

Calculating multiple shipping... (1997) Help name our technology! (1997) Problems appending to database (1997) 2nd WebCatalog2 Feature Request (1996) Email Tracking (2004) Listfiles on network with NT (2000) Request Time Out (1997) FAX orders (1996) Macauth: Dates and No Scripting... (1997) Re:WebCatalog 3 for Macintosh (1998) WebCat2 beta 11 - new prefs ... (1997) referrer (2000) Re:Searching for ALL / empty form field (1997) Updating emailer (server) (2000) Any problem putting ALL dbs in Globals folder? (2000) WebCat2b13MacPlugIn - [include] doesn't allow creator (1997) Hiding HTML and breaking the page (1997) Can I invoke an ssi plugin from within a webcat page (1997) Reversing [ipaddress] normalization (1998) Hierarchy of form/text/math variables (2000)