Just say “Sorry, I have some urgent work to do. We’ll talk later”. People don’t care, no one will think you’re an asshole. You can say it right after ‘hello’ or after quick exchange. I mostly avoid talking to people at work. I have my ANC headphones, sit at my desk and do my job/browse lemmy most of the time. But when people need help with something I do it fast and without needless bureaucracy so a lot of people know me as a reliable guy. After all it’s work. Be polite, be professional, do your job and you’ll be fine. And if you meet someone who’s fun to talk to have lunch with them from time to time.