Their tone or inflection as well as context and social ques are important qualities to distinguish the intent.

Additionally, there are many idioms that you’ll just need to know to properly get it. (You can think of idioms like kanji or memes if that helps.)