You would have references from your current job, even if you’re cartoonishly unlikeable. Keep light contact with people you get on with even (probably especially) after they part ways with the organization you work for. If/when you need a job, ask those people if they know any leads you might follow.