I see absolutely no problem, the process of interviewing for a company goes both ways: for them to see if you would be a good fit for their open position, and for you to evaluate if what they offer is right for you.

No contract have been signed, you are free to revoke any appointment. Be courteous, let them know you have made a decision not to pursue this venture and that will be enough.