Is that done in some environments? I think Maxima gives answers as fractions by default, but didn’t know how that worked. And my Casio calculator can stick to fractions and surds up to a point, but I think it’s just using float under the surface and converting back ok until too much rounding error creeps in…
think Maxima gives answers as fractions by default, but didn’t know how that worked.
It’s likely based on the theory of continued fractions and the calculator has a limit to the maximum denominator allowed to return an approximation of the fraction.
Sadbutdru@sopuli.xyz 8 hours ago
Is that done in some environments? I think Maxima gives answers as fractions by default, but didn’t know how that worked. And my Casio calculator can stick to fractions and surds up to a point, but I think it’s just using float under the surface and converting back ok until too much rounding error creeps in…
Semi_Hemi_Demigod@lemmy.world 1 hour ago
I’ve written econmerce systems and to avoid floating point errors I’d just use ints and divide by 100 on the front end.
bamboo@lemmy.blahaj.zone 8 hours ago
It’s likely based on the theory of continued fractions and the calculator has a limit to the maximum denominator allowed to return an approximation of the fraction.