View topic - decimal precision

 All times are UTC + 1 hour

 Page 1 of 1 [ 6 posts ]
 Print view Previous topic | Next topic
Author Message
 Post subject: decimal precisionPosted: 03 Sep 2011, 15:13
 Content Developer

Joined: 13 Jan 2005, 00:46
Location: ModalitÃ
 val = val/60 -- convert to minutes val = val*10 -- shift the decimal place val = (math.floor(val))/10 -- truncate and reshiftSo then I output the results40.166667938232401.6666870117240.099998474121<-wtfwell I investigated it. The math floor operation does work but when I divide by ten it adds back all the decimal places...does lua have a shift that I can use or a precision since the division is flawed?

 Post subject: Re: decimal precisionPosted: 03 Sep 2011, 15:21
 Moderator

Joined: 26 Oct 2007, 15:21
 Loss of precision is induced by there being no exact binary representation of your number in a 32bit float. Thus you get something near it.

 Post subject: Re: decimal precisionPosted: 03 Sep 2011, 19:18
 Spring Developer

Joined: 28 Jun 2007, 06:30
 Code:local function round(num, idp)  return format("%." .. (idp or 0) .. "f"):format(num)end

 Post subject: Re: decimal precisionPosted: 04 Sep 2011, 11:06
 Moderator

Joined: 19 May 2009, 20:10

 Post subject: Re: decimal precisionPosted: 04 Sep 2011, 15:37
 Content Developer

Joined: 13 Jan 2005, 00:46
Location: ModalitÃ
 SirMaverick wrote:http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.22.6768http://www.freebsd.org/doc/en/articles/ ... /x218.html

 Post subject: Re: decimal precisionPosted: 04 Sep 2011, 21:08
 Kernel Panic Co-Developer

Joined: 16 Nov 2004, 13:08
 I use:Code:function FormatNbr(x,digits)    local _,fractional = math.modf(x)    if fractional==0 then        return x    elseif fractional<0.01 then        return math.floor(x)    elseif fractional>0.99 then        return math.ceil(x)    else        local ret=string.format("%."..(digits or 0).."f",x)        if digits and digits>0 then            while true do                local last = string.sub(ret,string.len(ret))                if last=="0" or last=="." then                    ret = string.sub(ret,1,string.len(ret)-1)                end                if last~="0" then                    break                end            end        end        return ret    endendTo turn my decimal number into nice strings.I'm sure it could be made much faster, but at least this work, while other exemples from the net do not.

 Display posts from previous: All posts1 day7 days2 weeks1 month3 months6 months1 year Sort by AuthorPost timeSubject AscendingDescending
 Page 1 of 1 [ 6 posts ]

 All times are UTC + 1 hour

#### Who is online

Users browsing this forum: No registered users and 1 guest

 You cannot post new topics in this forumYou cannot reply to topics in this forumYou cannot edit your posts in this forumYou cannot delete your posts in this forumYou cannot post attachments in this forum

Search for:
 Jump to:  Select a forum ------------------ Spring    Community News       News Submissions    General Discussion    Help & Bugs    Game Lobby       Spring Clans       Lobby Clients    Tournaments    Linux    Mac OS X    Off Topic Discussion Content Creation    Game Development       Game Development Tutorials & Resources    Map Creation       Map Tutorials & Resources    Art & Modelling       Modelling & Texturing Tutorials & Resources    Project Subforums       Kernel Panic       Zero-K       Balanced Annihilation       Spring: 1944       Star Wars: Imperial Winter       XTA       Supreme Annihilation       Caydr's Projects       Evolution RTS       Argh's Projects       The Cursed       NOTA       Damned       Conflict Terra       Engines Of War       Spring Tanks       Journeywar       BattleTech: Legacy       Dune: Evolution       Progressive Annihilation       TA: Factions    Sound Effects & Music Development    News    Development       Dedicated Developer Discussion       New Developer Discussion       Infrastructure Development       Meeting Minutes    AI    Feature Requests    Lua Scripts       The Werewolf Game Site content    Site content       Banners       Welcome images       Videos       Videos - contributions       Screenshots       Screenshots - contributions