The Ultimate Web Scraping With Python Bootcamp 2024

The Ultimate Web Scraping With Python Bootcamp 2024, available at $64.99, has an average rating of 4.32, with 161 lectures, based on 276 reviews, and has 2578 subscribers.

You will learn about Understand the fundamentals of web scraping in python from absolute scratch Scrape information from static and dynamic websites and extract it to a variety of formats Intercept and emulate hidden APIs to identify highly productive alternatives to getting your data Master the requests library for working with HTTP Parse and extract content from HTML using beautifulsoup, selectolax, and Microsoft Playwright Master complex CSS selectors including descendant, child, sibling combinators Understand how the web works, including HTTP, HTML, CSS, and JavaScript Create scrapy crawlers and practice items, itemloaders and custom pipelines Integrate scrapy with playwright for highly performant, fine-tuned dynamic website crawling Practice processing and extracting data to a variety of formats including csv, json, xml, and SQL This course is ideal for individuals who are Anyone who wants to learn how to collect data from the web programmatically or Students with or without web scraping experience looking to level up or Complete beginners with no experience It is particularly useful for Anyone who wants to learn how to collect data from the web programmatically or Students with or without web scraping experience looking to level up or Complete beginners with no experience.

Enroll now: The Ultimate Web Scraping With Python Bootcamp 2024

Summary

Title: The Ultimate Web Scraping With Python Bootcamp 2024

Price: $64.99

Average Rating: 4.32

Number of Lectures: 161

Number of Published Lectures: 161

Number of Curriculum Items: 161

Number of Published Curriculum Objects: 161

Original Price: $199.99

Quality Status: approved

Status: Live

What You Will Learn

Understand the fundamentals of web scraping in python from absolute scratch

Scrape information from static and dynamic websites and extract it to a variety of formats

Intercept and emulate hidden APIs to identify highly productive alternatives to getting your data

Master the requests library for working with HTTP

Parse and extract content from HTML using beautifulsoup, selectolax, and Microsoft Playwright

Master complex CSS selectors including descendant, child, sibling combinators

Understand how the web works, including HTTP, HTML, CSS, and JavaScript

Create scrapy crawlers and practice items, itemloaders and custom pipelines

Integrate scrapy with playwright for highly performant, fine-tuned dynamic website crawling

Practice processing and extracting data to a variety of formats including csv, json, xml, and SQL

Who Should Attend

Anyone who wants to learn how to collect data from the web programmatically

Students with or without web scraping experience looking to level up

Complete beginners with no experience

Target Audiences

Anyone who wants to learn how to collect data from the web programmatically

Students with or without web scraping experience looking to level up

Complete beginners with no experience

Welcome to the Ultimate Web Scraping With Python Bootcamp, the only course you need to go from a complete beginner in python to a very competent web scraper.

Web scraping is the process of programmatically extracting data from the web. Scraping agents visit a web resource, extract content from it, and then process the resulting data in order to parse some specific information of interest.

Scraping is the kind of programming skill that offers immediate feedback, and can be used to automate a wide variety of data collection and processing tasks.

Over the next 17+ hours, we will methodically cover everything you need to know to write web scraping agents in python.

This bootcamp is organized in three parts of increasing difficulty designed to help you progressively build your skill.

Part I – Begin

We’ll start by understanding how the web works by taking a closer look at HTTP, the key application layer communication protocol of the modern web. Next, we’ll explore HTML, CSS, and JavaScript from first principles to get a deeper understanding of how website are built. Finally, we’ll learn how to use python to send HTTP requests and parse the resulting HTML, CSS, and JavaScript to extract the data we need. Our goal in the first part of the course is to build a solid foundation in both web scraping and python, and put those skills to practice by building functional web scrapers from scratch. Selected topics include:

a detailed overview the request-response cycle

understanding user-agents, HTTP verbs, headers and statuses

understanding why custom headers can often be used to bypass paywalls

mastering the requests library to work with HTTP in python

what stateless means and how cookies work

exploring the role of proxies in modern web architectures

mastering beautifulsoup for parsing and data extraction

Part II – Refine

In the second part of the course, we’ll build on the foundation we’ve already laid to explore more advanced topics in web scraping. We’ll learn how to scrape dynamic websites that use JavaScript to render their content, by setting up Microsoft Playwright as a headless browser to automate this process. We’ll also learn how to identify and emulate API calls to scrape data from websites that don’t have formally public APIs. Our projects in this section will include an image scraper that can download a set number of high-resolution images given some keyword, as well as another scraping agent that extracts price and content of discounted video games from a dynamically rendered website. Topics include:

identifying and using hidden APIs and understanding the benefits they offer

emulating headers, cookies, and body content with ease

automatically generating python code from intercepted API requests using postman and httpie

working with the highly performant selectolax parsing library

mastering CSS selectors

introducing Microsoft Playwright for headless browsing and dynamic rendering

Part III – Master

In the final part of the course, we’ll introduce scrapy. This will give us an excellent, time-tested framework for building more complex and robust web scrapers. We’ll learn how to set up scrapy within a virtual environment and how to create spiders and pipelines to extract data from websites in a variety of formats. Having learned how to use scrapy, we’ll then explore how to integrate it with Playwright so that we tackle the challenge of scraping dynamic websites from right within scrapy. We’ll conclude this section by building a scraping agent that executes custom JavaScript code before returning the resulting HTML to scrapy. Some topics from this section:

learning how to set up scrapy and explore its command line interface (“the scrapy tool“)

dynamically explore response objects using scrapy shell

understand and define item schemas and load data using itemloaders and input/output processors

integrate Playwright into scrapy to tackle dynamically rendered JavaScript sites

write PageMethods to specify highly specific instructions to the headless browser from right within scrapy

define custom pipelines for saving into SQL databases and highly customized output formats

In this bootcamp, I will take you step-by-step through engaging video lectures and teach you everything you need to know to get started with web scraping in python.

By the end of this course, you will have a complete toolset to conceptualize and implement scraping agents for any website you can imagine.

See you inside!

Course Curriculum

Chapter 1: Introduction

Lecture 1: Prerequisites

Lecture 2: A Useful Mental Model

Lecture 3: All Code Resources

Chapter 2: The HTTP Protocol

Lecture 1: What Is HTTP?

Lecture 2: The Request-Response Cycle

Lecture 3: Extra: But, This Website Remembers Me

Lecture 4: User-Agents

Lecture 5: HTTP Verbs

Lecture 6: Status Codes

Lecture 7: Headers

Lecture 8: Extra: Headers Do Lie

Lecture 9: Proxies

Chapter 3: HTML, CSS, And JavaScript

Lecture 1: The Ingredients

Lecture 2: Markup

Lecture 3: Attributes

Lecture 4: Presentation

Lecture 5: Some More Rules

Lecture 6: Behaviour

Lecture 7: More JavaScript

Lecture 8: JavaScript In Web Scraping

Lecture 9: Comments

Lecture 10: Embedded

Chapter 4: Web Requests In Python

Lecture 1: Urllib

Lecture 2: Requests

Lecture 3: Setting Headers

Lecture 4: Query Parameters

Lecture 5: Authentication And Authorization

Lecture 6: Aside From GET

Lecture 7: POSTing Data

Chapter 5: Parsing And Extraction

Lecture 1: BeautifulSoup

Lecture 2: Tags

Lecture 3: Parents, Children, And Descendants

Lecture 4: Siblings

Lecture 5: Extracting Text

Lecture 6: All Strings

Lecture 7: Search

Lecture 8: Challenge

Lecture 9: Solution

Lecture 10: Solution Refinement

Lecture 11: An Extra: pandas

Lecture 12: Functional Search Patterns

Lecture 13: Text Search

Lecture 14: Searching By CSS

Lecture 15: Just One Tag

Chapter 6: Project 1 – Portfolio Valuation With Google Finance

Lecture 1: Scope Statement

Lecture 2: An Extra: Some Finance Concepts

Lecture 3: Parsing Price

Lecture 4: Non-USD Prices

Lecture 5: Adding Structure With Dataclasses

Lecture 6: Position And Portfolio

Lecture 7: Tabular Display

Chapter 7: APIs: The Hidden Gems

Lecture 1: Befriend The Network Tab

Lecture 2: Case Study: Coffee Shop Locations

Lecture 3: The Advantages Of APIs

Lecture 4: Full Header Emulation

Lecture 5: An Extra: Postman

Lecture 6: Code Generation

Lecture 7: Challenge

Lecture 8: Solution: Interacting With The API

Lecture 9: Solution: Processing The Data

Lecture 10: Solution: Adding Geocode

Chapter 8: Selectolax And Advanced CSS Selectors

Lecture 1: Introduction

Lecture 2: What Is selectolax?

Lecture 3: CSS Combinators

Lecture 4: Sibling Combinators

Lecture 5: Selector Types

Chapter 9: Project 2 – Image Scraper

Lecture 1: Scope Statement

Lecture 2: Prospecting

Lecture 3: NOTE: Quick Correction To CSS Selector

Lecture 4: Scraping HTML

Lecture 5: Filtering Relevant URLs

Lecture 6: Extracting High-Res Image URLs

Lecture 7: Saving The Images

Lecture 8: Stepping It Up With Logging

Lecture 9: Back To The API

Lecture 10: Filtered Canonical URLs

Lecture 11: Pagination Prospecting

Lecture 12: Wrapping Up

Chapter 10: Tackling JavaScript With Microsoft PlayWright

Lecture 1: What You See vs. What You Get

Lecture 2: Rendering JavaScript

Lecture 3: PlayWright Over Selenium

Lecture 4: Case Study: Show Me The Money

Chapter 11: Project 3 – Building A Configurable Scraping Pipeline

Lecture 1: Scope Statement

Lecture 2: Initial Setup

Lecture 3: Fully Loaded Site

Lecture 4: Selecting Game Containers

Lecture 5: More Robust Render Thresholds

Lecture 6: Extracting Title And Thumbnail

Lecture 7: Game Category Tags

Instructors

Andy Bek
Software Consultant

Rating Distribution

1 stars: 7 votes

2 stars: 5 votes

3 stars: 21 votes

4 stars: 57 votes

5 stars: 186 votes

Frequently Asked Questions

How long do I have access to the course materials?

You can view and review the lecture materials indefinitely, like an on-demand channel.

Can I take my courses with me wherever I go?

Definitely! If you have an internet connection, courses on Udemy are available on any device at any time. If you don’t have an internet connection, some instructors also let their students download course lectures. That’s up to the instructor though, so make sure you get on their good side!