IEEE 754

Comments

• Semi_Hemi_Demigod@lemmy.world ⁨2⁩ ⁨weeks⁩ ago

  Just store the fraction as a varchar, duh

  source

  • Sadbutdru@sopuli.xyz ⁨2⁩ ⁨weeks⁩ ago

    Is that done in some environments? I think Maxima gives answers as fractions by default, but didn’t know how that worked. And my Casio calculator can stick to fractions and surds up to a point, but I think it’s just using float under the surface and converting back ok until too much rounding error creeps in…

    source

    • bamboo@lemmy.blahaj.zone ⁨2⁩ ⁨weeks⁩ ago

      think Maxima gives answers as fractions by default, but didn’t know how that worked.

      It’s likely based on the theory of continued fractions and the calculator has a limit to the maximum denominator allowed to return an approximation of the fraction.

      source
    • Semi_Hemi_Demigod@lemmy.world ⁨2⁩ ⁨weeks⁩ ago

      I’ve written econmerce systems and to avoid floating point errors I’d just use ints and divide by 100 on the front end.

      source
• iocase@lemmy.zip ⁨2⁩ ⁨weeks⁩ ago

  The coolest part is you can do log math since you have the mantissa and exponent already for extremely fast multiplication and division (+ or - log values)

  source

  • far_university1990@reddthat.com ⁨2⁩ ⁨weeks⁩ ago

    Fast inverse square root go brrrr.

    source