In this chapter, we'll look inside the messages exchanged in an HTTP transaction. We'll learn about message types, HTTP headers, and status codes. Understanding what is inside an HTTP message is vitally important for developers who work on the web. Not only will you build better applications by responding with the right types of messages, but you'll also be able to spot problems and debug issues when web applications aren't working.

Requests and Responses

Imagine walking up to a stranger in an airport and asking, "Do you know what time it is?" In order for the stranger to respond with the correct time, a few things have to be in place. First, the stranger has to understand your question, because if he or she does not know English, he or she might not be able to make any response. Secondly, the stranger will need access to a watch or some other time-keeping device.

This airport analogy is similar to how HTTP works. You, the client, need a resource from some other party (the resource being information about the time of day). So, you make a request to the other party using a language and vocabulary you hope the other party will understand. If the other party understands your request and has the resource available, it can reply. If it understands the request but doesn't have the resource, it can still respond and tell you it doesn't know. If the other party doesn't understand what you are saying, you might not get any response.

HTTP is a request and response protocol. A client sends an HTTP request to a server using a carefully formatted message that the server will understand. A server responds by sending an HTTP response that the client will understand. The request and the response are two different message types that are exchanged in a single HTTP transaction. The HTTP standards define what goes into these request and response messages so that everyone who speaks "HTTP" will understand each other and be able to exchange resources (or when a resource doesn't exist, a server can still reply and let you know).

A Raw Request and Response

A web browser knows how to send an HTTP request by opening a network connection to a server machine and sending an HTTP message as text. There is nothing magical about the request-it's just a command in plain ASCII text and formatted according to the HTTP specification. Any application that can send data over a network can make an HTTP request. You can even make a manual request using an application like Telnet from the command line. A normal Telnet session connects over port 23, but as we learned in the first chapter, the default network port for HTTP is port 80.

The following figure is a screenshot of a Telnet session that connects to odetocode.com on port 80, makes an HTTP request, and receives an HTTP response.

The Telnet session starts by typing:

telnet www.odetocode.com 80

Please note that the Telnet client is not installed by default on Windows 7, Windows Server 2008 R2, Windows Vista, or Windows Server 2008. You can install the client by following the procedure listed at http://technet.microsoft.com/en-us/library/cc771275(v=ws.10).aspx.

This command tells the operating system to launch the Telnet application, and tells the Telnet application to connect to www.odetocode.com on port 80.

Once Telnet connects, we can type out an HTTP request message. The first line is created by typing the following text then pressing Enter:

GET / HTTP/1.1

This information will tell the server we want to retrieve the resource located at "/" (i.e. the root resource or the home page), and we will be using HTTP 1.1 features. The next line we type is:

host:www.odetocode.com

This host information is a required piece of information in an HTTP 1.1 request message. The technical reason to do this is to help servers that support multiple websites, i.e. both www.odetocode.com and www.odetofood.com could be hosted on the same server, and the host information in the message will help the web server direct the request to the proper web application.

After typing the previous two lines we can press Enter twice to send the message to the server. What you see next in the Telnet window is the HTTP response from the web server. We'll go into more details later, but the response says that the resource we want (the default home page of www.odetocode.com), has moved. It has moved to the location odetocode.com. It's up to the client now to parse this response message and send a request to odetocode.com instead of www.odetocode.com if it wants to retrieve the home page. Any web browser will go to the new location automatically.

These types of "redirects" are common, and in this scenario the reason is to make sure all the requests for resources from OdeToCode go through odetocode.com and not www.odetocode.com (this is a search engine optimization known as URL canonicalization).

Now that we've seen a raw HTTP request and response, let's dig into specific pieces.

HTTP Request Methods

The GET word typed into the Telnet session is one of the primary HTTP methods. Every request message must include one of the HTTP methods, and the method tells the server what the request wants to do. An HTTP GET wants to get, fetch, and retrieve a resource. You could GET an image (GET /logo.png), or GET a PDF file, (GET /documents/report.pdf), or any other retrievable resource the server might hold. A list of common HTTP operators is shown in the following table.

Of these five methods, just two are the primary workhorses of the web: GET and POST. A web browser issues a GET request when it wants to retrieve a resource, like a page, an image, a video, or a document. GET requests are the most common type of request.

A web browser sends a POST request when it has data to send to the server. For example, clicking "Add to Cart" on a site like amazon.com will POST information to Amazon about what we want to purchase. POST requests are typically generated by a <form> on a webpage, like the form you fill out with <input> elements for address and credit card information.

GET and Safety

There is a part of the HTTP specification that talks about the "safe" HTTP methods. Safe methods, as the name implies, don't do anything "unsafe" like destroy a resource, submit a credit card transaction, or cancel an account. The GET method is one of the safe methods since it should only retrieve a resource and not alter the state of the resource. Sending a GET request for a JPG image doesn't change the image, it only fetches the image for display. In short, there should never be a side-effect to a GET request.

An HTTP POST is not a safe method. A POST typically changes something on the server-it updates an account, submits an order, or does some other special operation. Web browsers typically treat GET and POST differently since GET is safe and POST is unsafe. It's OK to refresh a webpage retrieved by a GET request-the web browser will just reissue the last GET request and render whatever the server sends back. However, if the page we are looking at in a browser is the response of an HTTP POST request, the browser will warn us if we try to refresh the page. Perhaps you've seen these types of warnings in your web browser.

Because of warnings like this, many web applications always try to leave the client viewing the result of a GET request. After a user clicks a button to POST information to a server (like submitting an order), the server will process the information and respond with an HTTP redirect (like the redirect we saw in the Telnet window) telling the browser to GET some other resource. The browser will issue the GET request, the server will respond with a "thank you for the order" resource, and then the user can refresh or print the page safely as many times as he or she would like. This is a common web design pattern known as the POST/Redirect/GET pattern.

Now that we know a bit more about POST and GET, let's talk about some common scenarios and see when to use the different methods.

Common Scenario-GET

Let's say you have a page and want the user to click a link to view the first article in this series. In this case a simple hyperlink is all you need.

<a href="http://odetocode.com/Articles/741.aspx">Part I</a>

When a user clicks on the hyperlink in a browser, the browser issues a GET request to the URL specified in the href attribute of the anchor tag. The request would look like this:

GET http://odetocode.com/Articles/741.aspx HTTP/1.1
Host: odetocode.com

Scenario-POST

Now imagine you have a page where the user has to fill out information to create an account. Filling out information requires <input> tags, and we nest these inputs inside a <form> tag and tell the browser where to submit the information.

<form action="/account/create" method="POST">
 
    <label for="firstName">First name</label>
    <input id="firstName" name="firstName" type="text" />
   
    <label for="lastName">Last name</label>
    <input id="lastName" name="lastName" type="text" />
   
    <input type="submit" value="Sign up!"/>
 
</form>

When the user clicks the submit button, the browser realizes the button is inside a form. The form tells the browser that the HTTP method to use is POST, and the path to POST is /account/create. The actual HTTP request the browser makes will look something like this.

POST http://localhost:1060/account/create HTTP/1.1
Host: server.com

firstName=Scott&lastName=Allen

Notice the form inputs are included in the HTTP message. This is very similar to how parameters appear in a URL, as we saw in the previous article. It's up to the web application that receives this request to parse those values and create the user account. The application can then respond in any number of ways, but there are three common responses:

1. Respond with HTML telling the user that the account has been created. Doing so will leave the user viewing the result of a POST request, which could lead to issues if he or she refreshes the page-it might try to sign them up a second time!
2. Respond with a redirect instruction like we saw earlier to have the browser issue a safe GET request for a page that tells the user the account has been created.
3. Respond with an error, or redirect to an error page. We'll take a look at error scenarios a little later in the book.

Forms and GET Requests

A third scenario is a search scenario. In a search scenario you need an <input> for the user to enter a search term. It might look like the following.

<form action="/search" method="GET">
 
     <label for="term">Search:</label>
     <input id="term" name="term" type="text" />
     <input type="submit" value="Sign up!"/>
 
</form>

Notice the method on this form is GET, not POST. That's because a search is a safe retrieval operation, unlike creating an account or booking a flight to Belgium. The browser will collect the inputs in the form and issue a GET request to the server:

GET http://localhost:1060/search?term=love HTTP/1.1
Host: searchengine.com

Notice instead of putting the input values into the body of the message, the inputs go into the query string portion of the URL. The browser is sending a GET request for /search?term=love. Since the search term is in the URL, the user can bookmark the URL or copy the link and send it in an email. The user could also refresh the page as many times as he or she would like, again because the GET operation for the search results is a safe operation that won't destroy or change data.

A Word on Methods and Resources

We've talked quite a bit about resources as physical resources on the file system of a server. Quite often, resources like PDF files, video files, image files, and script files do exist as physical files on the server. However, the URLs pointing inside of many modern web applications don't truly point to files. Technologies like ASP.NET and Ruby on Rails will intercept the request for a resource and respond however they see fit. They might read a file from a database and return the contents in the HTTP response to make it appear as if the resource really existed on the server itself.

A good example is the POST example we used earlier that resulted in a request to /account/create. Chances are there is no real file named "create" in an "account" directory. Instead, something on the web server picks up this request, reads and validates the user information, and creates a record in the database. The /account/create resource is virtual and doesn't exist. However, the more you can think of a virtual resource as a real resource, the better your application architecture and design will adhere to the strengths of HTTP.

HTTP Request Headers

So far we've seen a raw HTTP request and talked about the two popular HTTP methods-GET and POST. But as the Telnet output demonstrated, there is more to an HTTP request message than just the HTTP method. A full HTTP request message consists of the following parts:

[method] [URL] [version]
[headers]
 
[body]

The message is always in ASCII text, and the start line always contains the method, the URL, and the HTTP version (most commonly 1.1, which has been around since 1999). The last section, the body section, can contain data like the account sign-in parameters we saw earlier. When uploading a file, the body section can be quite large.

The middle section, the section where we saw Host: odetocode.com, contains one or more HTTP headers (remember, in HTTP 1.1 host is a required header). Headers contain useful information that can help a server process a request. For example, in the last article we talked about resource representations and how the client and server can negotiate on the best representation of a resource (content negotiation). If the client wants to see a resource in French, for example, it can include a header entry (the Accept-Language header) requesting French content.

GET http://odetocode.com/Articles/741.aspx HTTP/1.1
Host: odetocode.com
Accept-Language: fr-FR

There are numerous headers defined by the HTTP specification. Some of the headers are general headers that can appear in either a request or a response message. An example is the Date header. The client or server can include a Date header indicating when it created the message.

GET http://odetocode.com/Articles/741.aspx HTTP/1.1
Host: odetocode.com
Accept-Language: fr-FR
Date: Fri, 9 Aug 2002 21:12:00 GMT

Everything but the host header is optional, but when a header does appear it must obey the standards. For example, the HTTP specification says the value of the date header has to be in RFC822 format for dates.

Some of the more popular request headers appear in the following table.

A full HTTP request might look like the following.

GET http://odetocode.com/ HTTP/1.1
Host: odetocode.com
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) Chrome/16.0.912.75 Safari/535.7
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Referer: http://www.google.com/url?&q=odetocode
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3

As you can see, some headers contain multiple values, like the Accept header. The Accept header is listing the MIME types it likes to see, including HTML, XHTML, XML, and finally */* (meaning I like HTML the best, but you can send me anything (*/*) and I'll try to figure it out).

Also notice the appearance of "q" in some of the headers. The q value is always a number from 0 to 1 and represents the quality value or "relative degree of preference" for a particular value. The default is 1.0, and higher numbers indicate a higher preference.

The Response

An HTTP response has a similar structure to an HTTP request. The sections of a response are:

[version] [status] [reason]
[headers]
 
[body]

The full HTTP response to the last full request we listed might look like this (with most of the HTML omitted for brevity).

HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/7.0
X-AspNet-Version: 2.0.50727
X-Powered-By: ASP.NET
Date: Sat, 14 Jan 2012 04:00:08 GMT
Connection: close
Content-Length: 17151
 
<html>
<head>
    <title>.NET-related Articles, Code and Resources</title>
</head>
<body>
 ... content ...
</body>
</html>

The opening line of a request starts off with the HTTP version, and then the all-important status code and reason.

Response Status Codes

The status code is a number defined by the HTTP specification and all the numbers fall into one of five categories.

Although we won't detail all of the possible HTTP status codes, the following table will detail the most common codes.

Response status codes are an incredibly important part of the HTTP message because they tell the client what happened (or in the case of redirects, where to go next).

HTTP Status Codes Versus Your Application

Remember that the HTTP status code is a code to indicate what is happening at the HTTP level. It doesn't necessarily reflect what happened inside your application. For example, imagine a user submits a sign-in form to the server, but didn't fill out the Last Name field. If your application requires a last name it will fail to create an account for the user. This doesn't mean you have to return an HTTP error code indicating failure. You probably want quite the opposite to happen-you want to successfully return some content to the client with a 200 (OK) status code. The content will tell the user a last name was not provided. From an application perspective the request was a failure, but from an HTTP perspective the request was successfully processed. This is normal in web applications.

Response Headers

A response includes header information that gives a client metadata it can use to process the response. For example, the content type will be specified as a MIME type, as we talked about in the last article. In the following response we can see the content type is HTML, and the character set used to encode the type is UTF-8. The headers can also contain information about the server, like the name of the software and the version.

HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/7.0
X-AspNet-Version: 2.0.50727
X-Powered-By: ASP.NET
Date: Sat, 14 Jan 2012 04:00:08 GMT
Connection: close
Content-Length: 17151
 
<html>
<head>
    <title>.NET-related Articles, Code and Resources</title>
</head>
<body>
 ... content ...
</body>
</html>

The response headers that appear will often depend on the type of response. For example, a redirection response needs to include a Location header that tells the client where to go next.

There are a number of headers devoted to caching and performance optimizations. ETag, Expires, and Last-Modified all provide information about the cacheability of a response. An ETag is an identifier that will change when the underlying resource changes, so comparing ETags is an efficient way to know if something needs to be refreshed. An Expires header tells a client how long to cache a particular resource. We'll return and look at caching in more detail later.

Where Are We?

In this chapter we've learned that HTTP messages always come in pairs. First there is the request, and then there is the response. The information in these messages is all in readable text, and there are lots of tools you can use to inspect HTTP requests being made on your machine. Fiddler is one such tool if you are running Windows (http://fiddler2.com). It's easy to use, and you can see the raw HTTP requests being made, including all of the headers.

Messages are all about making sure both parties in a transaction understand what they are receiving. The first line of an HTTP message is always explicit about its intent. In a request message, the URL and HTTP method appear first to identify what should happen to a particular resource. In a response the status code will indicate how the request was processed. We also have headers moving in both directions that provide even more information about the request and response. In the next chapter we'll learn a little more about how these messages travel across the network.