Web Scraping Libraries

Written by Art, June 19, 2017

Use PIP to install all packages.

Pip is a package management system used to install and manage software packages written in Python. Many packages can be found in the Python Package Index (PyPI). Python 2.7.9 and later (on the python2 series), and Python 3.4 and later include pip (pip3 for Python 3) by default.

For more info and installation:

Pip and virtualenv on Mac
Pip and virtualenv on Windows

Fetching URLs

Urllib module for python

Urllib is a Python module for fetching URLs. You do not have to install it. Urllib module comes with Python package. For python 3.6 use:

https://docs.python.org/3.6/howto/urllib2.html

For python 2.7 use:

https://docs.python.org/2.7/howto/urllib2.html

Requests library

Requests is HTTP library for Python, official documentation is here:

http://docs.python-requests.org/en/master/

Installation:

pip install requests

WGET library

Python download utility WGET, official documentation is here:

https://pypi.python.org/pypi/wget

Installation:

pip install wget

Scraping

Beautiful Soup

Beautiful Soup is a Python library for pulling data out of HTML and XML files. Official documentation is here:

https://www.crummy.com/software/BeautifulSoup/bs4/doc/#calling-a-tag-is-like-calling-find-all

Installation:

pip install beautifulsoup4

PDFminer3k

PDFminer3k PDF parser and analyzer, official documentation is here:

https://pypi.python.org/pypi/pdfminer3k

Installation:

pip install pdfminer3k


Share with:

About author


Art

Art is a FinTech enthusiast who has a great passion for coding and teaching. He earned a M.Sc. from Adelphi University, Garden City, New York. Currently, he develops software for the financial services industry and leads classes and workshops in Python at PracticalProgramming.co

Python 101

This class aims to help beginners to feel justifiably confident to start using Python programming language

Learn more

Python for Data Science

Learn how to apply Python analytics libraries Pandas, NumPy and Matplotlib for Data Science

Learn more

Python Immersive

Become a proficient Python programmer, master programming skills by working on real life projects

Learn more

Web Scraping

We go through the whole process of gathering, storing and analyzing data from the Web

Learn more

Build Web App

This is practical course for those who want to create a Web Application from scratch using Django

Learn more