Spring RTS Engine

Posted: **03 Sep 2011, 16:13**

val = val/60 -- convert to minutes
val = val*10 -- shift the decimal place
val = (math.floor(val))/10 -- truncate and reshift

So then I output the results
40.166667938232
401.66668701172
40.099998474121<-wtf

well I investigated it. The math floor operation does work but when I divide by ten it adds back all the decimal places...

does lua have a shift that I can use or a precision since the division is flawed?

Posted: **03 Sep 2011, 16:21**

Loss of precision is induced by there being no exact binary representation of your number in a 32bit float. Thus you get something near it.

Posted: **03 Sep 2011, 20:18**

Code: Select all

local function round(num, idp)
  return format("%." .. (idp or 0) .. "f"):format(num)
end

Posted: **04 Sep 2011, 12:06**

http://citeseerx.ist.psu.edu/viewdoc/su ... .1.22.6768

Posted: **04 Sep 2011, 16:37**

SirMaverick wrote:http://citeseerx.ist.psu.edu/viewdoc/su ... .1.22.6768

http://www.freebsd.org/doc/en/articles/ ... /x218.html

Posted: **04 Sep 2011, 22:08**

I use:

Code: Select all

function FormatNbr(x,digits)
    local _,fractional = math.modf(x)
    if fractional==0 then
        return x
    elseif fractional<0.01 then
        return math.floor(x)
    elseif fractional>0.99 then
        return math.ceil(x)
    else
        local ret=string.format("%."..(digits or 0).."f",x)
        if digits and digits>0 then
            while true do
                local last = string.sub(ret,string.len(ret))
                if last=="0" or last=="." then
                    ret = string.sub(ret,1,string.len(ret)-1)
                end
                if last~="0" then
                    break
                end
            end
        end
        return ret
    end
end

To turn my decimal number into nice strings.

I'm sure it could be made much faster, but at least this work, while other exemples from the net do not.

Spring RTS Engine

decimal precision

decimal precision

Re: decimal precision

Re: decimal precision

Re: decimal precision

Re: decimal precision

Re: decimal precision