I think your instinct is right and invertebrates might be more affected by caffeine, because I did a lab on how fruit fly larvae respond to caffeine in which about a third of the class inadvertently killed their larvae.

I also suspect humans are less likely to continue trying fine motor tasks if they’re close to dangerously overdosing on caffeine, because it’s unpleasant as hell and most people would probably seek out medical attention instead of knitting through it.