The concerning part imo is that it will occasionally bend its interpretation of the system prompt to justify clearly unacceptable behavior. Just dismissing the number of recent security incidents as the responsibility of AI companies advertising their product or even state sponsored hacking would be far more comforting to me. Instead I really do believe we don’t have control over the core technology with the current ethos of move fast break things. And as the technology improves and becomes more capable we should be concerned by this.