Your replicator is probably too small to replicate larger components, which would be a major inconvenience at best or a showstopper at worst. And industrial replicators are even harder to come by than starships.

Then there’s getting access to the replicator patterns for sensitive or dangerous components. Dilithium chambers, weapons, Mercassium composite for shield generators, etc. are classified by Starfleet.

Then there are substances that can’t be replicated, such as verterium cortenide for the warp coils. I don’t think it’s explicitly stated that VC can’t be replicated, but we know that Voyager had to find some to refit their warp coils, they couldn’t just replicate it. Also dilithium.

And finally, there’s antimatter. Building a starship won’t do you much good if you don’t have gas for the tank. Antimatter does not occur in large quantities in nature, and probably can’t be replicated (or at least not safely.) So you’d need some sort of industrial base to produce it, further complicating your plans.