Depends where you are. In America, yes. The term is associated with freak shows and traveling circuses. Nowadays, both ‘little person’ and ‘dwarf’ are acceptable terms, although they might not necessarily fit within your story if you want to avoid modern-sounding language/terms that already apply to another species in your story.