One of the important features of the HTTP specification is that GET, the operation that is used for almost all attempts to retrieve a Web page, has some very carefully crafted semantics. In particular, GET is inappropriate for any request that, directly or as a side effect, updates the state of a server. What’s an update? Well, taking money out of your bank account, confirming a plane reservation, or, in my opinion anyway, using up one of your 20 free New York Times page accesses. The way the HTTP specification puts this is:

In particular, the convention has been established that the GET and HEAD methods SHOULD NOT have the significance of taking an action other than retrieval. These methods ought to be considered “safe”. This allows user agents to represent other methods, such as POST, PUT and DELETE, in a special way, so that the user is made aware of the fact that a possibly unsafe action is being requested.

Exactly. I really want to be warned before I access a page that’s coming out of my monthly quota, and that doesn’t happen today. Furthermore, it’s perfectly legitimate for software to follow links for me without me even asking. Consider an e-mail reader that might offer to download for me local copies of any page linked from any e-mails I might have received, perhaps so that I can read my correspondence while I’m traveling. Such an agent is unlikely to be aware that such accesses can quickly deplete my NYT allowance, and perhaps for pages I never intended to read anyway.

Does this actually bother anyone in practice? Yes. In fact, I think this is the technical explanation for one of the most frustrating aspects of the paywall. There are all sorts of situations in which one winds up unintentionally clicking on a link to an NYT article, only to discover after the fact that yet another “free” access has been accounted. This happens when links are sent using URL shorteners, when the link text or image does not show the URL, or when the user neglects to read the target URI carefully before following a link.

So, what should the NYT do if they want to continue to offer free articles, and also follow Web architecture? One acceptable answer would be: the response to a GET request for an article that counts against the “free” limit should redirect to a simple Web form that says:

“You are about to use one of your 20 free accesses to the New York Times for this month. If you wish to go ahead, click the button below”.

The button would, of course, do an HTTP POST,  to properly account the access before returning the article.

To minimize the need to go through such prompts repeatedly, links from NYT pages to other “free” NYT content could use Javascript onClick tricks to directly retrieve using POST, without first going through the prompt page. Ideally, some less disruptive indication, such as a popup tooltip, could warn in advance if an access would count further against a user’s quota. With suitable use of cookies, it would also be possible for users to set preferences to disable such warnings.

As it stands, the paywall misuses HTTP, and the consequences are indeed frustrating to users. Furthermore, if too many sites start misusing GET, then it will become more difficult for us all to explore the Web without worrying about the consequences for each link we follow, and it may also be more difficult for engines like the Google crawler to retrieve Web content.

By the way, the W3C Technical Architecture Group, which I chair, has written about safe use of HTTP in its finding:  URIs, Addressability, and the use of HTTP GET and POST.

Note that the first 2/3 of the talk is a very interesting exploration of the security characteristics of Bitcoin, also showing how the Bitcoin database can be used as a peristent shared store. The latter third of the talk introduces Dan’s tools for detecting artificial delays introduced by ISPs.

Background: the #! fuss

I’ve been thinking about all this for quite some time, but the recent fuss over so-called “hash-bang” (#!) URIs raises the stakes considerably. As previously noted, Jeni Tennison has posted a terrific overview, with links to contributions from many others. If you’re not up on all that, please read those postings first.

Three examples

Consider three Web applications, each of which makes extensive use of Javascript.

1. The simulator is an interactive environment that simulates, we’ll say, driving a race car. You launch the application by navigating to its URI, and then Javascript takes over, providing an interactive simulation of the car driving around a race course, while giving you control over the car’s acceleration and direction. Just to make the example more clearly Ajax, we can assume that the application periodically goes out to various Web sites to retrieve data to be integrated into the simulation (weather information, etc.)
2. The résumé browser is similar in style to AJAX implementations of GMail or Yahoo mail, but its purpose is to let the user browse through collections of job application résumés. The initial screen provides a list with one line per applicant, but the user can request to see individual résumés as well.  Under the covers, all the usual Ajax tricks are being used to pre-cache résumés, to provide fluid interactivity for the user, etc. Users require the ability to email and post links to the individual résumés.
3. The final example is the full (as opposed to mobile) implementation of Google Maps. Like the others, this is an Ajax application. Users can retrieve maps for most any part of the world, zoom, pan, and annotate the maps with points of interest or driving directions. Crucially, an interface is provided that gives a URI (with no # in the syntax!) for the current location, zoom level, annotations, etc. As with the résumé application, these URIs can be emailed, linked from other Web pages, etc., and indeed this is a very valuable capability that is often exploited by users.

Resource identification

With respect to URIs and resource identification, the résumé and simulator applications are quite different in character. Indeed, if we ignore implementation concerns, the résumé browser closely resembles a traditional, non-Ajax, document-based Web site. Users have every reason to want each résumé to be integrated into the Web as a first class resource with its own stable URI, per the Architecture of the World Wide Web‘s mandate to Identify with URIs. It’s very important that people be able to email links to these résumés, put links to them on other Web pages, etc. Ideally, non-Javascript user agents should show the same résumés as Javascript-enabled browsers (albeit with less fluid navigation). Furthermore, it’s very useful if crawlers and other server-side agents can find and retrieve the documents, and for the FWD/BACK buttons in the browser to work as they would in a non-Ajax implementation.

Even though the implementation is very similar in its use of Javascript, the simulator is very different in character. There is no sense in which the user is navigating through a succession of pages.  Maybe or maybe not the application would find some use for FWD/BACK, or could provide links to intermediate simulation states that might usefully be emailed, but in any case there seems little need that such links be crawlable, or that non-Javascript user agents do much with them.

My main concern here is that too few Ajax applications take the trouble to meet the goals set out for the résumé browser, even when they could. With some hand waving about how the Web has moved past its old static past, and making some quite valid points regarding what happens when you update the address bar in typical browsers (you tend to kick off a page retrieval if anything other than the fragment identifier is touched), the designers of these applications often punt on providing useful URIs for the individual documents managed by their applications. Furthermore, if such identifiers are provided, they are often distinguished only in the fragment identifier, resulting in all the disadvantages noted by Jeni, Tim Bray, and others. For example, the resources tend not to be accessible from non-Javascript  agents, rather ugly kludges are required if crawling is to be enabled at all, etc.

There are two main points I’m trying to make in this note. The first is:

Use of AJAX implementation technology is not a sufficient excuse for failing to provide first class URI identification for documents on the Web

If you’re building a simulator, this advice may not apply; if you’re building the résumé browser, then provide URIs for each résumé, and probably for each sorted collection of résumés, etc. Ensure that those URIs work, at least to display the referenced document, when accessed with non-Javascript enabled user agents.

Documents when possible

Now let’s consider the Google Maps example. What’s interesting about it is that the Google engineers could have easily made the claim that the maps application was like the simulator, provided a single URI for the entire application, and stopped there. They might, to go a bit further, have done some local kludging to enable FWD/BACK to work, while still not exposing URIs for the different views of the map (or if you prefer, for the different maps.)

In fact, Google Maps is valuable in part because Google did take the trouble to model their application as a collection of document-like Web resources, and in fact to create URIs that work equally well in Javascript- and non-Javascript enabled user agents. To do this, they created URIs with no fragments (I.e. no #) for each view of the map; the compromise, at least for now, is that these URIs do not show up automatically in the address bar, but are available from a separate link button supported by the Javascript.  These URIs have some interesting characteristics:

• They are understood by both server and client: when a request is received from a non-Javascript enabled user agent (often from a mobile device), the server interprets the entire URI and sends a relatively static image of the map as a response; when the same URI is received from a Javascript agent, the Ajax implementation is used. Interpretation of query parameters is (potentially) distributed between the server and the client, and that division of labor can change without affecting the URIs.
• Because the same URI is used in all cases, a link can be created on a Javascript-enabled client, emailed to a non-Javascript client, and the right thing happens. Contrast that with http://twitter.com/#!/noahmendelsohn vs http://twitter.com/noahmendelsohn,  which are provided by Twitter for Javascript and non-Javascript clients respectively. The former does nothing useful if emailed to a non-Javascript client.  Pointing again to the advice given in Web Arch  Avoid URI aliases: “The problem with aliases is that if half of the neighborhood points to one URI for a given resource, and the other half points to a second, different URI for that same resource, the neighborhood is divided” The #! forms also break the Self-describing Web.

Google Maps shows the great value of using the document abstraction when you reasonably can. It wasn’t the easiest choice for that application, but the benefits are very significant. In spite of being very fluid and interactive, each state of the Google Map is modeled as a document with its own URI. You can email links to particular maps, to driving directions for your house, or to maps annotated with places of interest. Each such document is available, using the same URI, from the non-Javascript implementations, and in principle, server agents can crawl links to the different maps (e.g. to find maps that are popular).

So, using Google Maps as my poster child:

Where practical, model your application as a collection of documents, each with its own URI

Syntax: # or ? or neither

Note: this section was updated on 10 March 2010 based on the important observation from Jeni that the query component isn’t necessarily more important than the hierarchical path as an alternative to # fragment syntax.

I continue to have a general feeling that fragment syntax (#) has been over-emphasized in all this, and that identification using other parts of the URI, such as the path for hierarchical identification and/or the query string (with ?) for non-hierarchical may be preferable. Yes, for that to happen, we’ll have to get changes like pushState() and changeState() widely deployed, or maybe we’ll need other changes too. Still, from an architectural perspective, # seems somehow more fragile: until we ran unto address-bar update problems, fragments were used with documents to identify, well, fragments. Yes, with new media types we can do what we want, but in this case we don’t have a new media type, and the normative specifications make clear that for text/html, fragments refer to anchors within the document.

To be clear, I don’t think there’s any outright architectural barrier to server-side interpretation of fragment ids in cases where agents like crawlers happen to get hold of them, but of course, typical clients don’t send them to the server at all. Maybe we can find a way to use # that meets all the goals, and update the normative specs to support going that way. My guess is that, in the long term, avoiding fragments by using path and/or query is a better solution. Either way, we need to align the specs. with common practice.

Anyway, my main point is to model things as documents where possible, and to give each document a unique, crawlable URI that works consistently at client and server.

Finally, I hope it’s clear that I am in no way discouraging Ajax implementations, whether for résumés, simulators, or maps. I don’t want to let Ajax be an excuse for building a Web full of documents that can’t be properly linked.

Documents in applications by Noah Mendelsohn is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

By the way, Alan Kay used to have some wonderful online lists of recommended readings, and I haven’t been able to find them lately.  Anyone know whether they’re still posted?

(somewhat rearranging Norm’s text):

In short, if all you need are bundles of atomic values and especially if you expect to exchange data with JavaScript, JSON is the obvious choice. I don’t lose any sleep over that.  [...] XML wasn’t designed to solve the problem of transmitting structured bundles of atomic values. XML was designed to solve the problem of unstructured data. In a word or two: mixed content. [...]   I’ve seen attempts to represent mixed content in JSON and simple they aren’t. 

XML deals remarkably well with the full richness of unstructured data. I’m not worried about the future of XML at all even if its death is gleefully celebrated by a cadre of web API designers. [...] I look forward to seeing what the JSON folks do when they are asked to develop richer APIs. When they want to exchange less well strucured [sic] data, will they shoehorn it into JSON?

That is indeed the tradeoff.  If you want to send along a list of job applicants and their recent salaries, JSON does fine;  if you want to send their resumes, well JSON isn’t quite as helpful.   A surprising amount of the world’s important information is in just such semi-structured documents.  Think insurance policies, shop manuals, and even Web pages themselves.  XML is designed to provide a standard means for encoding and interchanging such information, with good enough pure data facilities that if you then want a unified framework to also handle the applicant list, you can.  When you prefer a simpler, more Javascript-compatible means of exchanging simple data, by all means use JSON.

Little of this is new.  Doug has said these things before, and he does acknowledge that, security aside, the new HTML5 features will be valuable.  Nonetheless, his conclusion is:

HTML5 has a lot of momentum and appears to be doomed to succeed. I think the wiser course is to get it right first. We have learned the hard way that once an error gets into a web standard, it is really hard to get it out.

Striking the right balance will be very difficult in practice.  There’s a huge investment in HTML5 at this point, and slowing down to revisit security will be difficult.  I think Doug’s right that it’s an option that deserves very serious and sober consideration.

By the way, I picked up on this article from a posting by Dan Connolly, who adds some interesting musings and a bit of history.

“…each [Emacs] keystroke generates around 1◦ C thermal spike…”

Who knew?  In their tests, the startup phase of Emacs raised the processor temperature in the area being measured (simulated) by 15◦ C.  Most of what’s driving modern trends toward multicore architectures is the tendency for higher speed single core machines to run too hot.  Indeed, I saw one presentation at IBM pointing out that heat densities were passing those found on the surface of a clothes iron!  Anyway, this is an interesting paper if you’re interest in modern processor design.

Dan is a good friend, and he has contributed more deeply and more selflessly to the success of the Web than most of the public will ever understand.  Unfortunately, I will not be able to join the dinner group in Cambridge, but with luck I might manage a little musical acknowledgement on Wed evening.

UPDATE: pictures from Cambridge #danfest