Web Scraping Libraries

Written by Art, June 19, 2017

Use PIP to install all packages.

Pip is a package management system used to install and manage software packages written in Python. Many packages can be found in the Python Package Index (PyPI). Python 2.7.9 and later (on the python2 series), and Python 3.4 and later include pip (pip3 for Python 3) by default.

For more info and installation:

Pip and virtualenv on Mac
Pip and virtualenv on Windows

Fetching URLs

Urllib module for python

Urllib is a Python module for fetching URLs. You do not have to install it. Urllib module comes with Python package. For python 3.6 use:


For python 2.7 use:


Requests library

Requests is HTTP library for Python, official documentation is here:



pip install requests

WGET library

Python download utility WGET, official documentation is here:



pip install wget


Beautiful Soup

Beautiful Soup is a Python library for pulling data out of HTML and XML files. Official documentation is here:



pip install beautifulsoup4


PDFminer3k PDF parser and analyzer, official documentation is here:



pip install pdfminer3k

Share with:

About author


Art is a FinTech enthusiast who has a great passion for coding and teaching. He earned a M.Sc. from Adelphi University, Garden City, New York. Currently, he develops software for the financial services industry and leads classes and workshops in Python at PracticalProgramming.co

Intro to Crypto & Blockchain

This course explains how to analyze, trade and build your own Blockchain tokens

Learn more

Python Immersive

Become a proficient Python programmer, master programming skills by working on real life projects

Learn more

Machine Learning Immersive

Today’s modern machine learning libraries have thoughtful APIs that allow anyone who is comfortable with code to dive into ML

Learn more

Javascript fundamentals

We speak to beginner coders on HOW TO GET STARTED as a developer as well as practice typing Javascript live coding

Learn more

Python 101

This class aims to help beginners to feel justifiably confident to start using Python programming language

Learn more