Crawling the web with Python 3.x by Doug Mahugh.
From the post:
These days, most everyone is familiar with the concept of crawling the web: a piece of software that systematically reads web pages and the pages they link to, traversing the world-wide web. It’s what Google does, and countless tech firms crawl web pages to accomplish tasks ranging from searches to archiving content to statistical analyses and so on. Web crawling is a task that has been automated by developers in every programming language around, many times — for example, a search for web crawling source code yields well over a million hits.
So when I recently came across a need to crawl some web pages for a project I’ve been working on, I figured I could just go find some source code online and hack it into what I need. (Quick aside: the project is a Python library for managing EXIF metadata on digital photos. More on that in a future blog post.)
But I spent a couple of hours searching and playing with the samples I found, and didn’t get anywhere. Mostly because I’m working in Python version 3, and the most popular Python web crawling code is Scrapy, which is only available for Python 2. I found a few Python 3 samples, but they all seemed to be either too trivial (not avoiding re-scanning the same page, for example) or too needlessly complex. So I decided to write my own Python 3.x web crawler, as a fun little learning exercise and also because I need one.
In this blog post I’ll go over how I approached it and explain some of the code, which I posted on GitHub so that others can use it as well.
…
Doug has been writing publicly about his hip replacement surgery so I don’t think this has any privacy issues. 😉
I am interested to see what he writes once he is fully recovered.
My contacts at the American Medical Association disavow any knowledge of hip replacement surgery driving patients to write in Python and/or to write web crawlers.
I suppose there could be liability implications, especially for C/C++ programmers who lose their programming skills except for Python following such surgery.
Still, glad to hear Doug has been making great progress and hope that it continues!