Difference between revisions of "HTTP"

Revision as of 20:25, 13 June 2022

HyperText Transfer Protocol (commonly abbreviated as HTTP) is the underlying protocol that drives the majority of communications on the World Wide Web.

Standards

• Universal Resource Identifiers (URI) in WWW: http://rfc.askapache.com/rfc1630/rfc1630.txt
• Uniform Resource Locators (URL): http://rfc.askapache.com/rfc1738/rfc1738.txt
• Memento protocol: http://www.mementoweb.org/guide/quick-intro/ (archived/versioned HTTP calls)

HTTP 1.x

• Hypertext Transfer Protocol -- HTTP/1.1: http://tools.ietf.org/html/rfc2616
• Hypertext Transfer Protocol -- HTTP/1.1 -- Header Field Definitions: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html

HTTP 2.x

• IETF -- Hypertext Transfer Protocol Version 2 (HTTP/2): https://tools.ietf.org/html/rfc7540
• Usage statistics of HTTP/2 for websites: https://w3techs.com/technologies/details/ce-http2
• Preload directive: https://w3c.github.io/preload/
• Akamai's HTTP 1.x .vs. HTTP 2.x demo: https://http2.akamai.com/demo

^{[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16]}

HTTP 3.x

• HTTP/3 with QUIC protocol: https://http3.net
• IETF -- HTTP/3 QUIC spec (draft): https://datatracker.ietf.org/doc/draft-ietf-quic-http/?include_text=1
• HTTP/2 (SPDY) and HTTP/3 (QUIC): https://medium.com/faun/http-2-spdy-and-http-3-quic-bae7d9a3d484

^{[17] [18] [19] [20] [21]}

HTTP Header

• wikipedia: List of HTTP header fields
• MozDev -- HTTP Headers: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers

^{[22] [23] [24]}

User Agent

User Agent strings are snippets of information that provide some basic identifying data about the Device, Operating System (OS) and/or Browser type and versions. This information is used by Browsers (or Native App webviews) when connecting from their client to respective Web Servers when making requests for content; many advanced websites/webapps then use this information to dynamically adjust features based on the capabilities of the particular Device/OS/Browser combination, which is known as "Browser Detection".

• List of (known/public) User Agents: http://www.user-agents.org/
• Masquerading Your Browser: http://www.ericgiguere.com/articles/masquerading-your-browser.html
• WURFL Device Detector Tool: https://tools.scientiamobile.com

^{[25] [26]} (RDF-based spec for describing "Mobile Device capabilities") ^{[27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37] [38] [39] [40] [41] [42] [43] [44] [45] [46] [47] [48] [49] [50] [51] [52] [53]}

Client-Hints

"Client-Hints" is a proposal from Google that at this point is quite likely to be rolled out to actual Device/OS/Browser combinations on Android (for Mobile Devices/Tablets in 2023), Chromebooks (also (2023), and even sooner within their Chrome web browser (as early as 2022), for the purpose of limiting identifiable/fingerprint-able data about a given client and its user(s). It allows clients to advertise information about itself through these fields so the server can determine which resources should be included in its response. As an abstract example, the server might indicate in an Accept-CH response header of an initial request that it can accept device pixel ratio information from the client. The client will then make future requests using a header field containing these details, and the server will respond with images appropriate for that device pixel ratio.

Before the "Client-Hints" change is made, a standard Google/Android device's User-Agent string would look like the following:

User-Agent: Mozilla/5.0 (Linux; Android 12; Pixel 6 Pro) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.74 Mobile Safari/537.36

After the "Client-Hints" change is made, all Google/Android devices will now give a generic User-Agent header like:

User-Agent: Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.0.0 Mobile Safari/537.36

Along with additional HTTP Headers such as:

Sec-CH-UA-Mobile: ?1
Sec-CH-UA-Platform: "Android"
Sec-CH-UA: "Google Chrome";v="95", "Chromium";v="95", ";Not A Brand";v="99"

It would be one thing if they were just moving this information around; however, they are not. By default they will not send any "Sec-CH" headers. To request the client send back a 2nd small request indicating the Device/OS/Browser type:

Accept-CH: Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version

Or, to force the page to stop rendering and refresh while forcing :

Critical-CH:  Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version

• User-Agent Client Hints spec: https://wicg.github.io/ua-client-hints/

^{[54] [55]}

HTTP Status Codes

• wikipedia: List of HTTP status codes

Servers

Apache

Apache Web Server

lighttpd

• lighttpd: http://www.lighttpd.net/

mongoose

• Mongoose (formerly tiny-http)- an easy to use web server: http://code.google.com/p/mongoose/

micro_httpd

• micro_httpd - really small HTTP server: http://www.acme.com/software/micro_httpd/

Tools

• JS Fiddle -- JS UA Detection: https://jsfiddle.net/bcmoney/aV39w/show (my own take on UA detection, with extras like Java/Flash/OS/Screen-size detection attempts)
• JS Fiddle -- CSS Device Detection: https://jsfiddle.net/bcmoney/unhpb3hd/show (a summary of some useful CSS-hack based Device Detection patterns)
• StatCounter -- User-Agent Detection tool: http://gs.statcounter.com/detect
• HTTP Bin - Web Service for HTTP info about your client: http://httpbin.org | EXAMPLE - IP (to find your IP Address Remotely)
• Curl command-linecall builder: https://curlbuilder.com/
• See CURL
• wget: https://www.gnu.org/software/wget/^[56][57]
• HTTPie: https://httpie.org (Python-based command line HTTP client with an intuitive UI, JSON support, syntax highlighting, wget-like downloads, plugins, and more)

Netstat

Use the following command to list all running network connections:

 netstat -a 

• Netstat (WINDOWS): http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/netstat.mspx?mfr=true
• The netstat Command (LINUX): http://www.faqs.org/docs/linux_network/x-087-2-iface.netstat.html
• From cmd console in Windows, list all connections (HTTP or otherwise; useful for finding hung Tomcat server instances/ports): http://mail-archives.apache.org/mod_mbox/tomcat-users/

Resources

• List of HTTP methods (verbs): http://annevankesteren.nl/2007/10/http-methods
• HTTP status codes: http://www.w3.org/Protocols/HTTP/HTRESP.html
• HTTP Status Messages: http://www.w3schools.com/tags/ref_httpmessages.asp
• Y! Slow -- .htaccess HTTP server settings to improve speed score: https://davidwalsh.name/yslow-htaccess^[58]
• Apache HTTP Server Version 1.3 -- Module mod_expires: http://httpd.apache.org/docs/1.3/mod/mod_expires.html
• RosettaCode - Basic HTTP Request code (in many programming languages): http://rosettacode.org/wiki/HTTP_Request
• RosettaCode - Basic HTTPS request: http://rosettacode.org/wiki/HTTPS_Request
• RosettaCode - Basic HTTPS request with Authentication: http://rosettacode.org/wiki/HTTPS_request_with_authentication
• HTTP status code - additional info: http://gif.phpnet.org/frederic/programs/http_status_codes/
• Using Fiddler to debug HTTP: http://www.techrepublic.com/blog/programming-and-development/using-fiddler-to-debug-http/4074
• User-Agent List: https://udger.com/resources/ua-list | OS list
• FireFox -- User-Agent Switcher plugin (UA list): http://techpatterns.com/forums/about304.html | DOWNLOAD

Tutorials

• HTTP Response and Request Header Manipulation using Apache .htaccess: http://www.askapache.com/htaccess/using-http-headers-with-htaccess.html
• How Web Caches Work: http://www.web-caching.com/mnot_tutorial/how.html
• HTTP ETag Explained: http://www.xpertdeveloper.com/2011/03/http-etag-explained/
• YSlow Review, extension for the Firebug add-on for FireFox (guide to the rules): http://www.seifi.org/performance/yslow-review-the-firebug-addon.html
• .htaccess Tutorial for Apache Webmasters (April 10th, 2007): http://www.askapache.com/htaccess/htaccess-for-webmasters.html
• Advanced Htaccess Demo/Example using Cookies, Headers, RewritesMarch 31st, 2009: http://www.askapache.com/htaccess/pdf-cookies-headers-rewrites.html
• PHP and .htaccess tutorial and examples for Apache (April 10th, 2007): http://www.askapache.com/htaccess/php-htaccess-tips-and-tricks.html
• The HTTP protocol: http://en.kioskea.net/contents/internet/http.php3
• Curl with Cookies and Headers: http://joelpm.com/curl/tools/2010/06/17/curl-with-cookies-and-headers.html
• How to Build a simple HTTP server in C: http://stackoverflow.com/questions/176409/how-to-build-a-simple-http-server-in-c
• Microsoft httpserver.c: http://research.microsoft.com/en-us/um/redmond/projects/invisible/src/web/http/httpserver.c.htm
• Setting Up 301 Redirects: http://www.gottaquirk.com/2007/11/27/setting-up-301-redirects/
• Apache .htaccess - Setting Up 301 Redirects for Dynamic URLs: http://www.seoverflow.com/seo/setting-up-301-redirects-for-dynamic-urls/
• HTTP response codes explained with cats: http://www.flickr.com/photos/girliemac/sets/72157628409467125/show/ (by Tomomi Imura)
• Choosing an HTTP Status Code — Stop Making It Hard: https://www.codetinkerer.com/2015/12/04/choosing-an-http-status-code.html
• A Complete Guide and List of HTTP Status Codes: https://kinsta.com/blog/http-status-codes/
• Read UNIX / Linux System IP Address In a Shell Script: http://www.cyberciti.biz/tips/read-unixlinux-system-ip-address-in-a-shell-script.html

• How to Change Your (Desktop) Browser’s User Agent Without Installing Any Extensions: https://www.howtogeek.com/113439/how-to-change-your-browsers-user-agent-without-installing-any-extensions/
• Pick up the Android version in the browser by Javascript: https://stackoverflow.com/questions/7184573/pick-up-the-android-version-in-the-browser-by-javascript
• Change the User Agent on an Android Phone: https://lifehacker.com/5547037/change-the-user-agent-on-an-android-phone
• Is there a web browser for iOS that will allow me to change the user agent?: https://apple.stackexchange.com/questions/40647/is-there-a-web-browser-for-ios-that-will-allow-me-to-change-the-user-agent
• iOS -- Change the User Agent in Safari Without Jailbreak: https://www.technipages.com/ios-change-the-user-agent-in-safari
• How to Change Firefox User Agents on an iPad: http://smallbusiness.chron.com/change-firefox-user-agents-ipad-45665.html
• Make Your iPhone & iPad Surf the Web Like It’s Internet Explorer or Firefox: http://www.macsinlaw.com/make-your-iphone-ipad-surf-like-internet-explorer-or-firefox/
• Change the Browser User Agent in Chrome, Safari & Firefox: http://osxdaily.com/2013/01/16/change-user-agent-chrome-safari-firefox/

• HttpClient 4 Tutorial: http://www.baeldung.com/httpclient-guide
• HttpClient 4 Cookbook: http://www.baeldung.com/httpclient4
• Java 11 HTTP/2 API Tutorial: https://blog.codefx.org/java/http-2-api-tutorial/
• Does Java have a complete enum for HTTP response codes?: https://stackoverflow.com/questions/730283/does-java-have-a-complete-enum-for-http-response-codes

External Links

• wikipedia: HTTP
• wikipedia:List of HTTP headers
• HTTP Headers: http://www.edginet.org/techie/website/http.html
• HTTP Charset: http://www.w3.org/International/O-HTTP-charset
• HTTP codes as Valentine’s Day comics: https://medium.com/@hanilim/http-codes-as-valentines-day-comics-8c03c805faa0
• Best Practices for Speeding Up Your Web Site: http://developer.yahoo.com/performance/rules.html
• Scaling PHP applications to get more out of your Server with Varnish: http://www.ibm.com/developerworks/library/os-php-varnish/index.html
• How to use HTTP connection instead of a stream socket : http://www.java-tips.org/java-me-tips/midp/how-to-use-http-connection-instead-of-a-stream-socket.html
• Java applications using URL objects: http://hemswell.lincoln.ac.uk/~slawson/napier/CO42022/labs/lab01.html
• HTTP/FTP client Libraries: http://curl.haxx.se/libcurl/competitors.html
• HOWTO - Useful ASP page to return configurable HTTP Status codes: http://blogs.msdn.com/b/david.wang/archive/2006/05/08/howto-useful-asp-page-to-return-configurable-http-status-codes.aspx
• C/C++ Web Server Library?: http://stackoverflow.com/questions/175507/c-c-web-server-library
• Life beyond HTTP 1.1 - Google’s SPDY: http://www.igvita.com/2011/04/07/life-beyond-http-11-googles-spdy
• HTTP/2 is Done: https://www.mnot.net/blog/2015/02/18/http2
• How HTTP/2 will speed up your web browsing: http://www.zdnet.com/article/how-http2-will-speed-up-your-web-browsing/
• Trends in HTTP Requests: http://httparchive.org/trends.php
• How HTTP/2 is Changing Web Performance Best Practices: https://blog.newrelic.com/2016/02/09/http2-best-practices-web-performance/
• HTTP methods - Idempotency & Safety: https://www.mscharhag.com/api-design/http-idempotent-safe
• The Internet changes - HTTP/3 will not use TCP anymore: https://medium.com/drill/the-internet-changes-http-3-will-not-use-tcp-anymore-427e82eeadc0
• Chrome and Firefox Go to 100, Potentially Breaking Websites (due to User-Agent strings with triple-digit version numbers): https://www.howtogeek.com/786940/chrome-and-firefox-go-to-100-potentially-breaking-websites/
• A New Definition of HTTP: https://www.mnot.net/blog/2022/06/06/http-core

References

1. ↑ HTTP2 official website: https://http2.github.io/
2. ↑ Hypertext Transfer Protocol version 2: http://tools.ietf.org/html/draft-ietf-httpbis-http2-17
3. ↑ HTTP/2 – A Real-World Performance Test and Analysis: https://css-tricks.com/http2-real-world-performance-test-analysis/
4. ↑ HTTP/2 -- Frequently Asked Questions: https://http2.github.io/faq/
5. ↑ Getting Ready For HTTP2 -- A Guide For Web Designers And Developers: https://www.smashingmagazine.com/2016/02/getting-ready-for-http2/
6. ↑ What is HTTP/2?: https://yoast.com/what-is-http2/
7. ↑ How to enable HTTP/2 support in Apache: https://http2.pro/doc/Apache
8. ↑ Setting up HTTP/2 on your web server: https://www.tunetheweb.com/performance/http2/
9. ↑ How to enable http2 on centos7: https://stackoverflow.com/questions/40720703/how-to-enable-http2-on-centos7
10. ↑ Install Apache with mod_ssl & mod_http2 on CentOS 8 / RHEL 8: https://computingforgeeks.com/install-apache-with-ssl-http2-on-rhel-centos/
11. ↑ Why Everyone Should Be Moving To HTTP/2: https://searchengineland.com/everyone-moving-http2-236716
12. ↑ HTTP/2 Statistics (2015) - KeyCDN Report on HTTP/2 Distribution: https://www.keycdn.com/blog/http2-statistics (North America focused so skews heavier on HTTP/2 usage than actual worldwide stats at the time)
13. ↑ InternetArchive -- Web Almanac (2019) Part IV, Chapter 20 - HTTP/2: https://almanac.httparchive.org/en/2019/http2 (biggest stat was 95% of all users on the web can potentially make use of HTTP/2)
14. ↑ InternetArchive -- Web Almanac (2020) Part IV, Chapter 22 - HTTP/2: https://almanac.httparchive.org/en/2020/http2 (biggest stat was 64% of all websites' web servers were supporting HTTP/2
15. ↑ April 2021 Web Server Survey: https://news.netcraft.com/archives/2021/04/30/april-2021-web-server-survey.html
16. ↑ HTTP/2 .vs. HTTP/3: https://w3techs.com/technologies/comparison/ce-http2,ce-http3
17. ↑ What is HTTP/3?: https://react-etc.net/entry/what-is-http-3
18. ↑ The Internet changes - HTTP/3 will not use TCP anymore: https://medium.com/drill/the-internet-changes-http-3-will-not-use-tcp-anymore-427e82eeadc0
19. ↑ HTTP/3 - the past, the present, and the future: https://blog.cloudflare.com/http3-the-past-present-and-future/
20. ↑ HTTP 3 is Out and About!: https://laptrinhx.com/http-3-is-out-and-about-1549222830/
21. ↑ HTTP/3 is Fast (and almost certainly PROD ready in 2022): https://requestmetrics.com/web-performance/http3-is-fast
22. ↑ The (HTTP) headers we don't want: https://www.fastly.com/blog/headers-we-dont-want
23. ↑ yCombinator HackerNews -- HTTP Headers we don't want discussion: https://news.ycombinator.com/item?id=17074721
24. ↑ Preventing Malicious Request Loops: https://blog.cloudflare.com/preventing-malicious-request-loops/
25. ↑ wikipedia: User agent
26. ↑ wikipedia: UAProf
27. ↑ List of User Agent strings: https://deviceatlas.com/blog/list-of-user-agent-strings
28. ↑ User Agent listing: https://user-agents.net/devices/mobiles
29. ↑ List of some of the most popular User Agent strings 2021: https://deviceatlas.com/blog/list-user-agent-strings-2021
30. ↑ DeviceAtlas -- THE COMPLETE GUIDE TO USER-AGENT STRINGS: https://deviceatlas.com/sites/deviceatlas.com/files/images/DeviceAtlas_Complete_Guide_To_User_Agents 2021.pdf
31. ↑ WhatIsMyBrowser -- Explore User Agents by Operating Platform: https://developers.whatismybrowser.com/useragents/explore/operating_platform_string/
32. ↑ User Agents listing: https://user-agents.net/
33. ↑ List-of-user-agents: https://github.com/tamimibrahim17/List-of-user-agents
34. ↑ Google Dev docs -- Chrome browser - User-Agent strings: https://developer.chrome.com/docs/multidevice/user-agent/
35. ↑ Samsung Dev docs -- Samsung Internet Browser - User-Agent string format: https://developer.samsung.com/internet/user-agent-string-format.html
36. ↑ Amazon Dev docs -- Fire TV - User Agent strings: https://developer.amazon.com/docs/fire-tv/user-agent-strings.html
37. ↑ MozDev -- Firefox - user agent string reference: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent/Firefox
38. ↑ UserAgentString -- Safari - most common (legacy) UAs: http://useragentstring.com/pages/useragentstring.php?name=safari | ALL UAs
39. ↑ Complete List of iOS User-Agent Strings: https://enterpriseios.com/wiki/Complete_List_of_iOS_User_Agent_Strings
40. ↑ Detect Apple iPhone Model Numbers and User-Agents: https://51degrees.com/blog/detect-apple-iphone-model-numbers-and-user-agents
41. ↑ Tech Stuff - Mobile Browser ID (User-Agent) Strings: http://www.zytrax.com/tech/web/mobile_ids.html
42. ↑ List of mobile user-agents: http://www.codereye.com/2014/12/list-of-mobile-user-agents.html
43. ↑ Complete List of iOS User-Agent Strings: http://www.enterpriseios.com/wiki/Complete_List_of_iOS_User_Agent_Strings
44. ↑ wikipedia: List of iOS and iPadOS devices
45. ↑ iOS device reference: https://iosref.com/ios (release date, first OS, max upgradable OS)
46. ↑ Which version of iOS for which phone? Compatibility list: https://story.backmarket.com/which-version-of-ios-for-which-iphone-compatibility-list/
47. ↑ User Agent Strings of iOS: www.webapps-online.com/online-tools/user-agent-strings/dv/operatingsystem51849/ios
48. ↑ MyIP.ms -- Safari 10 User Agent variations: https://myip.ms/view/comp_browsers/1983/Safari_10.html
49. ↑ Detecting iOS Version Number from User Agent using Regular Expressions: https://stackoverflow.com/questions/11171895/detecting-ios-version-number-from-user-agent-using-regular-expressions
50. ↑ How to detect iOS 6 and all minor versions by user agent?: https://stackoverflow.com/questions/12567260/how-to-detect-ios-6-and-all-minor-versions-by-user-agent/12580745#12580745
51. ↑ What is the iOS 5.0 user agent string?: https://stackoverflow.com/questions/7825873/what-is-the-ios-5-0-user-agent-string
52. ↑ The latest user agents for Safari: https://www.whatismybrowser.com/guides/the-latest-user-agent/safari
53. ↑ List of OTT user agents: https://deviceatlas.com/blog/list-ott-user-agents
54. ↑ wikipedia: HTTP Client Hints
55. ↑ User-Agent, Client-Hints and the Future of Device Detection: https://www.scientiamobile.com/user-agent-client-hints/
56. ↑ WGET for Windows: https://eternallybored.org/misc/wget/
57. ↑ Legacy WGET for Windows32 (SVN repo): http://gnuwin32.sourceforge.net/packages/wget.htm
58. ↑ How To Improve Your Website’s Yahoo Yslow Score: http://www.elegantthemes.com/blog/tips-tricks/how-to-improve-your-websites-yahoo-yslow-score

See Also

TCP/IP | FTP | SSH | Internet | Web | Web Server | Browser Detect | Caching | Computer | Mobile Device | Mobile Phone | Tablet | Android | iOS