With sparse attention, very interesting. It seems GQA is a thing of the past.

GLM 4.6 is reportedly about to drop too.