Does his talking exhaust you or is it the need for you to come up with responses? Or is it that you just want to be alone?

Depending on the answer,

“I love being with you but I’m too tired for talking. How about you tell me stuff you’re learning/a story while we do Lego and I’ll just keep quiet.”

“I love you but I’m feeling peopled out, I need to be alone today.”

“I would love to spend some time with you but my ears have had a hard day and I’m going to put some cotton in and rest them. Do you think you want to just be quiet together, or would you rather go where you can be noisy?”