All those people that treated you as “weird” this do it themselves as well. If you were gaining special attention for this in particular such that multiple people noticed and weren’t just messing with you, then it’s probably not the fact of your blank staring, but the length of time. Normal steady gaze at nothing in particular for periods is normal and people only perceive it as staring when it’s reached some threshold length of time.

Why you’d be doing this past that threshold I couldn’t tell you. Many are suggesting ADHD or Autism, could well be but otherwise maybe you’re just finding yourself easily distracted or prefer longer trips into your own head. If you worry about it, best suggestion is just to randomize your gaze periodically in response to stimuli in the room, but the thing is if you’re not already doing that automatically it’s probably because you’re too engrossed in thought for that stimuli to get through.