User-Agent

A User-Agent is an HTTP header sent with every browser request that identifies the software making the request - including the browser name, version, rendering engine, and operating system. Servers read User-Agents to customize responses or restrict access based on the client type.

Web scrapers often set their User-Agent to mimic a regular browser, since many websites serve different content to - or outright block - requests from User-Agents that identify themselves as bots. Using a realistic browser User-Agent is a basic evasion technique in scraping.

User-Agent detection is just one layer of anti-bot defense. Sophisticated systems also check browser fingerprints, JavaScript execution patterns, and request timing alongside the User-Agent header. APIs authenticate via API keys and do not involve User-Agent management.