I’ve written econmerce systems and to avoid floating point errors I’d just use ints and divide by 100 on the front end.
Comment on IEEE 754
Sadbutdru@sopuli.xyz 2 weeks agoIs that done in some environments? I think Maxima gives answers as fractions by default, but didn’t know how that worked. And my Casio calculator can stick to fractions and surds up to a point, but I think it’s just using float under the surface and converting back ok until too much rounding error creeps in…
Semi_Hemi_Demigod@lemmy.world 2 weeks ago
bamboo@lemmy.blahaj.zone 2 weeks ago
It’s likely based on the theory of continued fractions and the calculator has a limit to the maximum denominator allowed to return an approximation of the fraction.