View topic - decimal precision



All times are UTC + 1 hour


Post new topic Reply to topic  [ 6 posts ] 
Author Message
 Post subject: decimal precision
PostPosted: 03 Sep 2011, 15:13 
Content Developer
User avatar

Joined: 13 Jan 2005, 00:46
Location: ModalitÃ
val = val/60 -- convert to minutes
val = val*10 -- shift the decimal place
val = (math.floor(val))/10 -- truncate and reshift

So then I output the results
40.166667938232
401.66668701172
40.099998474121<-wtf

well I investigated it. The math floor operation does work but when I divide by ten it adds back all the decimal places...

does lua have a shift that I can use or a precision since the division is flawed?


Top
 Offline Profile  
 
 Post subject: Re: decimal precision
PostPosted: 03 Sep 2011, 15:21 
Moderator
User avatar

Joined: 26 Oct 2007, 15:21
Loss of precision is induced by there being no exact binary representation of your number in a 32bit float. Thus you get something near it.


Top
 Offline Profile  
 
 Post subject: Re: decimal precision
PostPosted: 03 Sep 2011, 19:18 
Spring Developer
User avatar

Joined: 28 Jun 2007, 06:30
Code:
local function round(num, idp)
  return format("%." .. (idp or 0) .. "f"):format(num)
end


Top
 Online Profile  
 
 Post subject: Re: decimal precision
PostPosted: 04 Sep 2011, 11:06 
Moderator

Joined: 19 May 2009, 20:10
http://citeseerx.ist.psu.edu/viewdoc/su ... .1.22.6768


Top
 Offline Profile  
 
 Post subject: Re: decimal precision
PostPosted: 04 Sep 2011, 15:37 
Content Developer
User avatar

Joined: 13 Jan 2005, 00:46
Location: ModalitÃ
SirMaverick wrote:
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.22.6768

http://www.freebsd.org/doc/en/articles/ ... /x218.html


Top
 Offline Profile  
 
 Post subject: Re: decimal precision
PostPosted: 04 Sep 2011, 21:08 
Kernel Panic Co-Developer
User avatar

Joined: 16 Nov 2004, 13:08
I use:
Code:
function FormatNbr(x,digits)
    local _,fractional = math.modf(x)
    if fractional==0 then
        return x
    elseif fractional<0.01 then
        return math.floor(x)
    elseif fractional>0.99 then
        return math.ceil(x)
    else
        local ret=string.format("%."..(digits or 0).."f",x)
        if digits and digits>0 then
            while true do
                local last = string.sub(ret,string.len(ret))
                if last=="0" or last=="." then
                    ret = string.sub(ret,1,string.len(ret)-1)
                end
                if last~="0" then
                    break
                end
            end
        end
        return ret
    end
end
To turn my decimal number into nice strings.

I'm sure it could be made much faster, but at least this work, while other exemples from the net do not.


Top
 Offline Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 6 posts ] 

All times are UTC + 1 hour


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group

Site layout created by Roflcopter et al.