And what might be the most important part cannot be elided over: market capitalism is HIGHLY efficient at solving optimization problems, but it only responds to incentives.

So if you can create the right incentives to reward the result you want and punish results you don't want, a market solution is going to do a marvelous job. It's great at, say, price discovery. But if the incentives do not align with the desired result, it's going to grind you under heel.

The incentives the insurance companies are responding to, frankly, are the ones you have outlined and essentially no others. Collect more premiums, make fewer payouts. There's no "breaking point" here because they have an absolutely vast customer base that has no choice to opt out of the system for a variety of reasons (ranging from the ACA individual mandate to the fact that it is not possible for an individual to make fully-informed financial decisions about their health even WITH advanced knowledge and training that nearly no one has).

Health insurance is pretty much a textbook example of a service that the free market is ill-suited to provide.

So over time, market capitalism is going to make them collect endlessly-increasing premiums and pay out less and less. It is going to continue to get worse. Period. It will eventually get nationalized. Period. All the argument in the meantime is just over how long we want to continue to let people be sick and broke before we apply the only fix.