Web Scraping in C#: C# Scraping Libraries vs Selenium, Playwright, and .NET Alternatives

Use plain C# HTTP scraping first, and bring in Playwright only when the page refuses to give you data. That is the simple rule. Selenium is still useful, but it is rarely the first tool I would pick for a new C# scraping job.

TLDR: For simple pages, use HttpClient plus HtmlAgilityPack or AngleSharp. For JavaScript heavy pages, use Playwright for .NET, because it is faster and cleaner than Selenium in many scraping tasks. Example: a product price scraper may fetch 1,000 static product pages in under 3 minutes with HttpClient, while the same work in a real browser may take 20 to 40 minutes. If only 15% of pages need clicks or scrolling, scrape 85% with libraries and send the tricky 15% to Playwright.

Start with the boring tool

Web scraping in C# can feel like choosing a vehicle. A bicycle, a car, and a tank can all move forward. You just do not take the tank to buy milk.

For many sites, the best scraper is boring. Send a request. Read the HTML. Pick the data. Save it. Done.

That usually means this stack:

• HttpClient for downloading pages.
• HtmlAgilityPack for messy HTML.
• AngleSharp for modern CSS selectors.
• System.Text.Json for JSON APIs hiding in plain sight.

This setup is fast. It is light. It does not open a browser. It does not need a screen. It does not panic because a cookie popup blinked at it.

Honestly, it feels like cheating when you find the same data inside a clean JSON response. You skip the whole page and grab the payload directly.

Image not found in postmeta

C# scraping libraries: the fast little robots

HtmlAgilityPack is the old reliable wrench. It handles broken HTML well. That matters, because the web is full of cursed markup. Missing tags. Weird nesting. Surprise comments. All the classics.

AngleSharp feels more modern. It supports CSS selectors, so you can grab items like this idea:

Find every product card. Read the title. Read the price. Read the link.

That is easy to think about. It is also easy to maintain.

CsQuery used to be popular for jQuery style scraping. It still exists in older projects, but I would not start there now. AngleSharp is usually the safer pick.

RestSharp and Flurl can help when scraping starts to look more like calling APIs. They make headers, query strings, and retries nicer to manage. Still, plain HttpClient is enough for many teams.

When libraries win

Use C# scraping libraries when the data is already in the HTML or in a network response. This covers more sites than people expect.

Good cases include:

• News articles.
• Blog pages.
• Directory listings.
• Product pages with server rendered HTML.
• Tables, specs, and public records.
• Simple pagination links.

The speed gap is huge. A library based scraper may finish one page in 100 to 500 milliseconds. A browser based scraper may need 2 to 8 seconds for the same page. That gap gets painful at scale.

Expect to waste time on browser automation if you use it for every tiny page. Waiting for load states. Closing popups. Fighting ads. Watching a headless browser eat memory like it pays rent.

Selenium: the old giant

Selenium has been around forever. It controls real browsers. It can click buttons, type into forms, scroll pages, and wait for elements.

That sounds perfect for scraping. Sometimes it is. But Selenium was built for testing web apps, not mass data collection. It can feel heavy.

Use Selenium when:

• Your team already knows it well.
• You maintain browser tests and scrapers together.
• You need support across many browser setups.
• You are stuck with legacy code.

It drives me crazy that a Selenium script can fail because an element moved by three pixels or an animation took one second longer than usual. Yes, you can fix it with better waits. No, it is still not fun.

Selenium is powerful. It is also chatty. You write many lines to do simple things. For fresh C# scraping work, Playwright often feels cleaner.

Playwright for .NET: the cool browser driver

Playwright is a newer browser automation tool from Microsoft. It supports Chromium, Firefox, and WebKit. The .NET version is very good.

Playwright shines when sites need real user actions. Click a tab. Wait for results. Scroll a list. Open a menu. Download a file. Grab data after JavaScript has finished.

Image not found in postmeta

It also has nice waiting behavior. That means fewer ugly sleeps like wait five seconds and pray. Playwright can wait for a selector, a network call, or a page state.

Use Playwright when:

• The page renders data with JavaScript.
• You need login flows.
• You must click filters or buttons.
• Infinite scroll hides the content.
• The page blocks plain HTTP requests.

Playwright is still slower than simple HTTP scraping. Of course it is. It runs a browser. But when you need a browser, it is a strong pick.

PuppeteerSharp and other .NET options

PuppeteerSharp is the C# port of Puppeteer. It controls Chromium. It is useful, but Playwright has wider browser support and a smoother .NET story now.

WebView2 can also load pages in a .NET app. It is handy for desktop tools, but it is not my favorite scraper engine. It is better when a human uses the app and scraping is only one feature.

ScrapySharp exists too. It wraps web requests and parsing helpers. It can work, but many projects now use HtmlAgilityPack or AngleSharp directly.

There is also the best alternative of all: do not scrape the page. Check for an API. Check the network tab. Check feeds. Check sitemap files. Check downloadable CSV files. The cleanest scraper is often not a scraper.

A simple decision guide

Pick your tool like this:

• Static HTML? Use HttpClient plus AngleSharp or HtmlAgilityPack.
• Messy old HTML? Use HtmlAgilityPack.
• Need CSS selectors? Use AngleSharp.
• Hidden JSON request? Use HttpClient and parse JSON.
• Need clicks, login, or scroll? Use Playwright.
• Existing Selenium code? Keep Selenium unless it hurts too much.
• Desktop app with embedded browser? Consider WebView2.

This mixed approach is the sweet spot. Let the small tools do the simple work. Let the browser handle the weird parts.

Watch out for rules and manners

Scraping is not a free pass to hammer a site. Read the terms. Respect robots.txt where it applies. Add delays. Use caching. Identify your client when suitable. Do not collect private data. Do not bypass paywalls or access controls.

Also, build for failure. Pages change. Class names vanish. A price moves from one span to another. Your scraper should log errors, retry gently, and alert you when output drops.

The practical C# scraper stack

For most teams, I would start with this setup:

• HttpClientFactory for stable HTTP usage.
• Polly for retries and backoff.
• AngleSharp for selectors.
• System.Text.Json for JSON.
• Playwright for the stubborn pages.
• SQLite, PostgreSQL, or files for storage.

This gives you speed, control, and a backup plan. You avoid opening browsers for easy pages. You still have browser power when the site fights back.

The final rule is simple: scrape with the lightest tool that works. Start with C# libraries. Upgrade to Playwright when the page demands a real browser. Keep Selenium for legacy needs or test heavy teams. Your CPU, your schedule, and your future self will thank you.