Web Scraping Libraries

Written by Art, June 19, 2017

Use PIP to install all packages.

Pip is a package management system used to install and manage software packages written in Python. Many packages can be found in the Python Package Index (PyPI). Python 2.7.9 and later (on the python2 series), and Python 3.4 and later include pip (pip3 for Python 3) by default.

For more info and installation:

Pip and virtualenv on Mac
Pip and virtualenv on Windows

Fetching URLs

Urllib module for python

Urllib is a Python module for fetching URLs. You do not have to install it. Urllib module comes with Python package. For python 3.6 use:


For python 2.7 use:


Requests library

Requests is HTTP library for Python, official documentation is here:



pip install requests

WGET library

Python download utility WGET, official documentation is here:



pip install wget


Beautiful Soup

Beautiful Soup is a Python library for pulling data out of HTML and XML files. Official documentation is here:



pip install beautifulsoup4


PDFminer3k PDF parser and analyzer, official documentation is here:



pip install pdfminer3k

Share with:

About author


Art is a FinTech enthusiast who has a great passion for coding and teaching. He earned a M.Sc. from Adelphi University, Garden City, New York. Currently, he develops software for the financial services industry and leads classes and workshops in Python at PracticalProgramming.co

Web Development Front-End Immersive

Learn how to use HTML5 and CSS, JavaScript and React to Develop a modern single-page web application.

Learn more

Python Immersive

Become a proficient Python programmer, master programming skills by working on real life projects

Learn more

Python 101

This class aims to help beginners to feel justifiably confident to start using Python programming language

Learn more

Python for Data Science

Acquire Crucial Skills for the 21st century, Weekends , Master your analytical skills by working on real life projects

Learn more

Blockchain 101 NYC

Get introduced to Blockchain from the ground up and Build your own blockchain with Solidity

Learn more