The "balance to the force" was misread by the Jedi council as meaning a complete Jedi victory over the Sith. Even in the first and second movies, Yoda and Windu both express doubts about the accuracy of the prophecy.

What a more pragmatic view would be is that a the force wanted a new type of force wielder philosophy where emotions don't rule over you(what the Sith ended as) but also one where emotion and connection are allowed to exist(what the Jedi outlawed.)

To that end, pre-Disney Luke and his academy were that new force wielder academy. One could argue that prequels Anakin struggled to meet that balance and his rebuff by the Jedi masters led to him being manipulated by Palpatine.