OpenEvidence Sounds Promising, but is it Reliable?

Comments

• liv@beehaw.org ⁨3⁩ ⁨weeks⁩ ago

  Article about an AI that aims to give treatment suggestions to doctors.

  source
• jarfil@beehaw.org ⁨3⁩ ⁨weeks⁩ ago

  When OpenEvidence took the US Medical Licensing Exam recently, it was wrong 9% of the time

  The passing score for USMLE is ~200 out of 300… how many “wrong times” is that?

  www.kaptest.com/study/usmle/passing-scores/

  source

  • liv@lemmy.nz ⁨3⁩ ⁨weeks⁩ ago

    When we look at passing scores, is there any way to quantitatively grade them for magnitude?

    Not all bad advice is created equal.

    source

    • jarfil@beehaw.org ⁨3⁩ ⁨weeks⁩ ago

      The grading is a mess. It goes about qualitative, quantitative… and statistical corrections “to make it fair”.

      Anyway, there is ~30% margin on the scores for passing, so chances are that 9% is better than the worst doctor who still “passed”.

      source

      • -> View More Comments
• technocrit@lemmy.dbzer0.com ⁨3⁩ ⁨weeks⁩ ago

  It’s not reliable. The name itself is misleading. The “evidence” is apparently already open. Is the statistical model actually open? Probably not.

  More accurate name: ClosedSummary

  source