decimal precision

[smoth]

val = val/60 -- convert to minutes
val = val*10 -- shift the decimal place
val = (math.floor(val))/10 -- truncate and reshift

So then I output the results
40.166667938232
401.66668701172
40.099998474121<-wtf

well I investigated it. The math floor operation does work but when I divide by ten it adds back all the decimal places...

does lua have a shift that I can use or a precision since the division is flawed?

[Beherith]

Loss of precision is induced by there being no exact binary representation of your number in a 32bit float. Thus you get something near it.

[jK]

Code: Select all

local function round(num, idp)
  return format("%." .. (idp or 0) .. "f"):format(num)
end

[SirMaverick]

http://citeseerx.ist.psu.edu/viewdoc/su ... .1.22.6768

[smoth]

http://citeseerx.ist.psu.edu/viewdoc/su ... .1.22.6768

http://www.freebsd.org/doc/en/articles/ ... /x218.html

[zwzsg]

I use:

Code: Select all

function FormatNbr(x,digits)
    local _,fractional = math.modf(x)
    if fractional==0 then
        return x
    elseif fractional<0.01 then
        return math.floor(x)
    elseif fractional>0.99 then
        return math.ceil(x)
    else
        local ret=string.format("%."..(digits or 0).."f",x)
        if digits and digits>0 then
            while true do
                local last = string.sub(ret,string.len(ret))
                if last=="0" or last=="." then
                    ret = string.sub(ret,1,string.len(ret)-1)
                end
                if last~="0" then
                    break
                end
            end
        end
        return ret
    end
end

To turn my decimal number into nice strings.

I'm sure it could be made much faster, but at least this work, while other exemples from the net do not.