Comment on Chad scraper

madcaesar@lemmy.world ⁨2⁩ ⁨years⁩ ago

• Sotuanduso@lemm.ee ⁨2⁩ ⁨years⁩ ago

  By getting someone to hire you to do it.

  source

  • anteaters@feddit.de ⁨2⁩ ⁨years⁩ ago

    Mind blowing stuff

    source
  • madcaesar@lemmy.world ⁨2⁩ ⁨years⁩ ago

    No I mean more what is the use case where it would be worth scrapping on a massive scale?

    source

    • oatscoop@midwest.social ⁨2⁩ ⁨years⁩ ago

      When the data is on multiple sites or sources.

      API licenses can be expensive, and some sources might not even have an API.

      source

      • madcaesar@lemmy.world ⁨2⁩ ⁨years⁩ ago

        I get the concept but a concrete example. What company could possibly want to pay for scraping a site?

        Some dude as a hobby I get it, but what, like Amazon will pay some guy to scrape competition prices or something?

        source

        • -> View More Comments
• Hawk@lemmynsfw.com ⁨2⁩ ⁨years⁩ ago

  Imagine an investment firm looking at a property market. They need data like price trends in the surrounding area.

  Real estate API is expensive, scraping is free. By hiring an employee the can save money.

  source
• Theharpyeagle@lemmy.world ⁨2⁩ ⁨years⁩ ago

  There’s a ton of money to be made from scraping, consolidating, and organizing publicly accessible data. A company I worked for did it with health insurance policy data because every insurance company has a different website with a different data format and data that updates every day. People will pay da big bux for someone to wrap all that messiness into a neat, consistent package.

  Right now, gathering machine learning data is hot, cause you need a lot of it to train a model. Companies may specialize in getting, say, social media posts from all kinds of sites and putting them together in a consistent format.

  source