Good answers here, but ignoring probably the most realistic and practical truth of the matter in my opinion.

You won't immediately be sent to the stocks for saying "I don't want to answer", the worst case scenario is that some officer of the court informs you that you must answer the question even if you don't want to. And even that is only going to happen if the attorney answering the question insists. And I struggle to imagine a situation where a competent attorney would do so.

Being hostile towards your prospective jurors, making them feel exposed and uncomfortable, is not a way to march to victory in a trial. They want to ensure you aren't prejudiced against their client/case. Making you dislike them personally IS prejudice. Causing prejudice is a bad way to eliminate prejudice.

They will ask questions, mostly yes/no ones, that you need to answer honestly. They may ask for clarification. If you don't want to answer and say so, it's unlikely anyone will press you because that unnwillingness to answer is just as clear an indication of who you are as anything else.