WEB SCRAPING ISN'T COOL

I remember when I first heard about web scraping, I started dreaming of it like it was some sort of magical being. Scraping the web felt cool in all reality scraping the web is like automating a robbery. Imagine how quick you’ll rack in the cash. After years of being on the fence about web scraping, I tried it for myself. This was of course done for educational purposes only, however, to be honest, no one should be scared of web scraping and the legal implications just ask google right?

Before I begin telling you my experience I do want to make a couple of points regarding safety around web scraping. Most likely you can do it without any legal consequences but please be mindful of the website’s servers, if your scraper Is making thousands of requests a minute, just remember someone is paying for the server and you are effectively extracting value from the website without aiding the site owner or site mission. So be mindful about the number of requests you are making.

When I first started learning how to make android applications I started learning java, after a couple of months, I got bored by just learning language features and baseless tutorials. I decided to create my first project. It was a news app, I needed news articles and I wasn’t a journalist. I had some prior skills in python, so I decided to use existing news websites and scrape their news and display them on the app.

I came to the justification I was providing the news websites traffic since I was linking to their websites under every article I scraped and displayed on the app. If you are interested in downloading and having a look at the App here’s the link https://play.google.com/store/apps/details?id=org.nativescript.hellonsvue

Web scraping is a simple process, download the website, extract the data needed and process it. For example, yahoo.com it has a news section I would grab the website from python code I would then use a library called beautiful soup to extract the type of data I needed and store that data in a database. The end. Seriously it was that easy.

To conclude web scraping is basic, there’s not much to it. There are hundreds of libraries and code available to make an awesome scraper, however, beginners please learn your respective language before diving headfirst into web scraping having a strong foundation in the language you are going to use will result in a more efficient and faster programming experience. If you have any questions, then please contact me using LinkedIn @ https://www.linkedin.com/in/tayyab-javed-45a552146/

You can visit my Git Hub type Microsomes.