To my best understanding:

Mog = outdo another person in aesthetics

Hence a mogbattle would be a battle of looks, in this case determined by popular vote facilitated through the Discord reaction system.