Popular python modules for web scraping
mechanize is a popular python module for it allows creation of a browser instance. It also maintains sessions which aids as a toolkit to obtain tasks like login, signup automation, etc.
BeautifulSoup is another beautiful python module which aids scraping the data required from html/xmls via tags. With beautiful you can scrape almost everything because it aids different methods like searching via tags, finding all links, etc.
Although selenium is a well known module for automated browser testing, it can be used as a web scraping tool as well. I promise you, it pays pretty well. With methods to find element via ids, name, class, etc, selenium would allow you to have anything on the website.
lxml is another wonderful library for parsing xml/htmls, however I would say beautifulsoup beats it in terms of usability. You could option to use any of the modules among lxml and BeautifulSoup since they pretty much do the same stuff.
I have used all of the above modules extensively in my projects and they allowed me to move faster. I was able to do some cool things with these modules. For example: automating conversation between two cleverbots(AI featured bots), getting paid courses at udemy, finding the most popular facebook fan page among my friends,etc. Therefore I totally recommend them. Below is a link to some of the most interesting things I’ve done with these modules.
Tell me how you felt the article was in the comments section below and/or you can add some of your favorite modules too. There’s always thanks for reading