I use scrapy. It has a steeper learning curve than other libraries, but it’s totally worth it.