It does say "check the results manually". Not that this changes anything. For the record, always double check anything any AI tells you unless you can verify the response off the top of your head. Also for the record, double check anything anybody else tells you. If you haven't seen it from more than one source, you don't know if it's true.

Hell, if the thing people learn from AI summaries is to never believe anything the see on the Internet without double checking it we'll be better off than we were before.

Also, every negative impact you assign to AI is also applicable to traditional search. I was hearing communication scholars warn people of the issues with algorithmic selection and personalized search back in the 90s. They were correct.

I am endlessly fascinated by the billions of boiling frogs that hadn't realized their perception of the world was owned by Google until Google made a noticeably change to their advertising engine. I am increasingly glad that AI is as unreliable as it is at this point.