think Maxima gives answers as fractions by default, but didn’t know how that worked.
It’s likely based on the theory of continued fractions and the calculator has a limit to the maximum denominator allowed to return an approximation of the fraction.
Comment on IEEE 754
Sadbutdru@sopuli.xyz 8 hours agoIs that done in some environments? I think Maxima gives answers as fractions by default, but didn’t know how that worked. And my Casio calculator can stick to fractions and surds up to a point, but I think it’s just using float under the surface and converting back ok until too much rounding error creeps in…
bamboo@lemmy.blahaj.zone 8 hours ago
Semi_Hemi_Demigod@lemmy.world 58 minutes ago
I’ve written econmerce systems and to avoid floating point errors I’d just use ints and divide by 100 on the front end.