I believe it could and should be made harder, but it is already a high barrier to purchase an investment property. For a business loan on residential housing, an investor needs 25-30% down payment for the property. Also I think the longest terms are 15 years and not 30, but I could be wrong.

All the small time landlords acquired their homes through primary residence loans which allows for PMI and smaller down payments that only exist because they are subsidized by the government. A primary residence loans either requires an owner to lie to the government and bank which puts them at serious liability in the sense they could make the loan due immediately if found out, or the owners have lived in that home for at least one year.