Comment on Someone got Gab's AI chatbot to show its instructions
teawrecks@sopuli.xyz 8 months agoYeah, as soon as you feed the user input into the 2nd one, you’ve created the potential to jailbreak it as well. You could possibly even convince the 2nd one to jailbreak the first one for you, or If it has also seen the instructions to the first one, you just need to jailbreak the first.
This is all so hypothetical, and probabilistic, and hyper-applicable to today’s LLMs that I’d just want to try it. But I do think it’s possible, given the paper mentioned up at the top of this thread.
sweng@programming.dev 8 months ago
Only true if the second LLM follows instructions in the user’s input. There is no reason to train it to do so.
teawrecks@sopuli.xyz 8 months ago
Any input to the 2nd LLM is a prompt, so if it sees the user input, then it affects the probabilities of the output.
There’s no such thing as “training an AI to follow instructions”. The output is just a probibalistic function of the input. This is why a jailbreak is always possible, the probability of getting it to output something that was given as input is never 0.
sweng@programming.dev 8 months ago
You are wrong. arxiv.org/abs/2402.18243
teawrecks@sopuli.xyz 8 months ago
Ah, TIL about instruction fine-tuning. Thanks, interesting thread.
Still, as I understand it, if the model has seen an input, then it always has a non-zero chance of reproducing it in the output.