Web Scraping Like Your Time is Valuable

Unless you’re a not-so-ethical hourly worker, or just a sadist, you probably don’t want to spend crazy amounts of time on mundane work. I’m going to give you a quick, easy recipe to increase productivity and give your web scraping workflow a pick-me-up. I’ll keep this short and sweet.

Ingredients:

• Burp Suite (Community Edition is fine)
• Ruby

DISCLAIMER: I don’t recommend this for really complex automation (simply from a code bulk point of view), but it will save you a ton of time busting out all of those side gigs you got on Upwork last week. Don’t freak out, you might actually have time to finish them.

Why did I accept 58 projects? Who does that?

Burp Suite: Sounds gross, what is it?

Burp Suite is your new best friend. An old buddy to web app pentesters around the world, it is an extremely useful tool when it comes to analyzing traffic to/from your browser. Head over to PortSwigger and grab a (free) copy. I’ll wait.

Ok, so you’ve successfully downloaded and installed Burp. Good to know. Now, go to the ‘Proxy’ tab and click on the subsection ‘Options’. You’ll need to set up listeners for two types of traffic: HTTP/HTTPS (of course).

Do this:

• Add a listener for HTTP