Jump to content

User-Agent header

Edit links
Page protected with pending changes
From Wikipedia, the free encyclopedia

This is an old revision of this page, as edited by Crashdoom (talk | contribs) at 19:24, 29 December 2021 (Restored revision 1062655103 by Beauty School Dropout (talk): Fix revert). The present address (URL) is a permanent link to this revision, which may differ significantly from the current revision.

An accepted version of this page, accepted on 29 December 2021, was based on this revision.

In computing, a user agent is any software, acting on behalf of a user, which "retrieves, renders and facilitates end-user interaction with Web content".[1] A user agent is therefore a special kind of software agent.

Some prominent examples of user agents are web browsers and email readers. Often, a user agent acts as the client in a client–server system. In some contexts, such as within the Session Initiation Protocol (SIP), the term user agent refers to both end points of a communications session.[2]

User agent identification

When a software agent operates in a network protocol, it often identifies itself, its application type, operating system, device model, software vendor, or software revision, by submitting a characteristic identification string to its operating peer. In HTTP,[3] SIP,[2] and NNTP[4] protocols, this identification is transmitted in a header field User-Agent. Bots, such as Web crawlers, often also include a URL and/or e-mail address so that the Webmaster can contact the operator of the bot.

Use in HTTP

In HTTP, the User-Agent string is often used for content negotiation, where the origin server selects suitable content or operating parameters for the response. For example, the User-Agent string might be used by a web server to choose variants based on the known capabilities of a particular version of client software. The concept of content tailoring is built into the HTTP standard in RFC 1945 "for the sake of tailoring responses to avoid particular user agent limitations".

The User-Agent string is one of the criteria by which Web crawlers may be excluded from accessing certain parts of a website using the Robots Exclusion Standard (robots.txt file).

As with many other HTTP request headers, the information in the "User-Agent" string contributes to the information that the client sends to the server, since the string can vary considerably from user to user.[5]

Format for human-operated web browsers

The User-Agent string format is currently specified by section 5.5.3 of HTTP/1.1 Semantics and Content. The format of the User-Agent string in HTTP is a list of product tokens (keywords) with optional comments. For example, if a user's product were called WikiBrowser, their user agent string might be WikiBrowser/1.0 Gecko/1.0. The "most important" product component is listed first.

The parts of this string are as follows:

  • product name and version (WikiBrowser/1.0)
  • layout engine and version (Gecko/1.0)

During the first browser war, many web servers were configured to send web pages that required advanced features, including frames, to clients that were identified as some version of Mozilla only.[6] Other browsers were considered to be older products such as Mosaic, Cello, or Samba, and would be sent a bare bones HTML document.

For this reason, most Web browsers use a User-Agent string value as follows:

Mozilla/[version] ([system and browser information]) [platform] ([platform details]) [extensions]

For example, Safari on the iPad has used the following: 

Mozilla/5.0 (iPad; U; CPU OS 3_2_1 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Mobile/7B405

The components of this string are as follows:

  • Mozilla/5.0: Previously used to indicate compatibility with the Mozilla rendering engine.
  • (iPad; U; CPU OS 3_2_1 like Mac OS X; en-us): Details of the system in which the browser is running.
  • AppleWebKit/531.21.10: The platform the browser uses.
  • (KHTML, like Gecko): Browser platform details.
  • Mobile/7B405: This is used by the browser to indicate specific enhancements that are available directly in the browser or through third parties. An example of this is Microsoft Live Meeting which registers an extension so that the Live Meeting service knows if the software is already installed, which means it can provide a streamlined experience to joining meetings.

Before migrating to the Chromium code base, Opera was the most widely used web browser that did not have the User-Agent string with "Mozilla" (instead beginning it with "Opera"). Since July 15, 2013,[7] Opera's User-Agent string begins with "Mozilla/5.0" and, to avoid encountering legacy server rules, no longer includes the word "Opera" (instead using the string "OPR" to denote the Opera version).

Format for automated agents (bots)

Automated web crawling tools can use a simplified form, where an important field is contact information in case of problems. By convention the word "bot" is included in the name of the agent.[8] For example: 

Googlebot/2.1 (+http://www.google.com/bot.html)

Automated agents are expected to follow rules in a special file called "robots.txt".

User agent spoofing

The popularity of various Web browser products has varied throughout the Web's history, and this has influenced the design of websites in such a way that websites are sometimes designed to work well only with particular browsers, rather than according to uniform standards by the World Wide Web Consortium (W3C) or the Internet Engineering Task Force (IETF). Websites often include code to detect browser version to adjust the page design sent according to the user agent string received. This may mean that less-popular browsers are not sent complex content (even though they might be able to deal with it correctly) or, in extreme cases, refused all content.[9] Thus, various browsers have a feature to cloak or spoof their identification to force certain server-side content. For example, the Android browser identifies itself as Safari (among other things) in order to aid compatibility.[10][11]

Other HTTP client programs, like download managers and offline browsers, often have the ability to change the user agent string.

Spam bots and Web scrapers often use fake user agents.

A result of user agent spoofing may be that collected statistics of Web browser usage are inaccurate.

User agent sniffing

Main article: Browser sniffing

User agent sniffing is the practice of websites showing different or adjusted content when viewed with certain user agents. An example of this is Microsoft Exchange Server 2003's Outlook Web Access feature. When viewed with Internet Explorer 6 or newer, more functionality is displayed compared to the same page in any other browsers. User agent sniffing is considered poor practice, since it encourages browser-specific design and penalizes new browsers with unrecognized user agent identifications. Instead, the W3C recommends creating standard HTML markup,[12] allowing correct rendering in as many browsers as possible, and to test for specific browser features rather than particular browser versions or brands.[13]

Websites intended for display by mobile phones often rely on user agent sniffing, since mobile browsers often differ greatly from each other.

Encryption strength notations

Web browsers created in the United States, such as Netscape Navigator and Internet Explorer, previously used the letters U, I, and N to specify the encryption strength in the user agent string. Until 1996, when the United States government allowed encryption with keys longer than 40 bits to be exported, vendors shipped various browser versions with different encryption strengths. "U" stands for "USA" (for the version with 128-bit encryption), "I" stands for "International" – the browser has 40-bit encryption and can be used anywhere in the world – and "N" stands (de facto) for "None" (no encryption).[14] Following the lifting of export restrictions, most vendors supported 256-bit encryption.

Deprecation of User-Agent header

In 2020, Google announced that they would be phasing out support for the User-Agent header in their Google Chrome browser. They stated that other major web browser vendors were supportive of the move, but that they did not know when other vendors would follow suit.[15] Google stated that a new feature called Client Hints would replace the functionality of the User-Agent string.[16]

See also

References

  1. ^ "W3C Definition of User Agent". www.w3.org. 16 June 2011. Retrieved 2018-10-20.
  2. ^ a b RFC 3261, SIP: Session Initiation Protocol, IETF, The Internet Society (2002)
  3. ^ RFC 7231, Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content, IETF, The Internet Society (June 2014)
  4. ^ Netnews Article Format. IETF. November 2009. sec. 3.2.13. doi:10.17487/RFC5536. RFC 5536.
  5. ^ Eckersley, Peter (27 January 2010). "Browser Versions Carry 10.5 Bits of Identifying Information on Average". Electronic Frontier Foundation. Retrieved 25 August 2011.
  6. ^ History of the browser user-agent string. WebAIM.
  7. ^ "Opera User Agent Strings: Opera 15 and Beyond". dev.opera.com. 15 July 2013. Retrieved 2014-05-05.
  8. ^ "https://www.rankwatch.com/blog/handle-bot-herding-and-spider-wrangling-for-rankings/"
  9. ^ Burstein complaining "... I've been rejected until I come back with Netscape"
  10. ^ "Android Browser Reports Itself as Apple Safari". Archived from the original on August 6, 2011. Retrieved August 9, 2011.
  11. ^ "User Agent String explained: Android Webkit Browser". UserAgentString.com. Retrieved 29 July 2012. Mozilla/5.0 (Linux; U; Android 2.2; en-sa; HTC_DesireHD_A9191 Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1
  12. ^ Pemberton, Stephen. "W3C Markup Validation Service". W3C. Retrieved 2011-10-18.
  13. ^ Clary, Bob (10 February 2003). "Browser Detection and Cross Browser Support". Mozilla Developer Center. Mozilla. Retrieved 2009-05-30.
  14. ^ Zawinski, Jamie (28 March 1998). "user-agent strings (obsolete)". mozilla.org. Retrieved 2010-01-08.
  15. ^ "Chrome Phasing out Support for User Agent". InfoQ. Retrieved 2020-03-25.
  16. ^ Cimpanu, Catalin. "Google to phase out user-agent strings in Chrome". ZDNet. Retrieved 2020-03-25.
Retrieved from "https://en.wikipedia.org/w/index.php?title=User-Agent_header&oldid=1062655755"