There’s a nice Web page up with a brief history of the ACS, and a video to a Computer History Museum video of a Feb. 2010 ACS reunion meeting. ACS pioneered many important features including instruction pre-fetch and dynamic out-of-order execution. The Web page and video are worth checking out if you’re interested in the history of computer architecture.

(Thanks to Lynn Wheeler for the link to the ACS page.)

If you are into audio and recording you probably know why this matters. If not, here’s a quick tutorial:  compression is a technique that’s used to reduce the difference between loud and soft passages in live or recorded music. Building a good sounding compressor is tricky, but the basic idea is to use a circuit or processing function that gently boosts the volume in quieter passages, but not in loud ones. (BTW: this is totally different from the file size compression that computer folks do — we’re talking here about audio compression).  There are some good reasons for using compression, and some less good reasons.

The most obvious benefit of using compression is that it makes it somewhat easier to hear music or speech in noisy places such as in cars. AM radio, and the single records that were for years targeted at AM radio, were often compressed quite a bit. Even the quieter passages of these recordings could be heard well over the noise in a car. Gentle compression is also often used on the microphones of singers on stage; again, it helps the quieter passages to cut through, and it also helps compensate for bad microphone technique. Unfortunately, any processing of audio risks losing the impact of the original, and too much compression can kill any sense of space and dynamics.

Unfortunately, there’s another “benefit” of compression that has proven appealing for modern recordings: it can let you record at higher average levels. Any recording medium has some limit on the peak waveform it can convey. For digital, you get hard clipping if you exceed that level; for things like vinyl records, you get needles jumping out of grooves, distortion, etc. So, if you don’t use compression, and there’s even one loud drum hit or cymbal crash in your piece, you’ve got to turn down all the rest so that one bit doesn’t clip. With compression, you can bring up the rest, and that’s where the loudness wars come in. It’s not just that you can minimize the difference between soft and loud, you can actually make the recording much louder on average. If someone’s playing 3 songs in a row, and just one is heavily compressed and recorded “hot”, that one will tend to come blasting out louder than the others. In the interest of maximizing that impact, recording engineers have been using more and more compression in recent years. So, we’ve been getting a lot of crummy sounding recordings that are way over-compressed, recorded too loud, and that sound harsh and artificial. All the space and dynamics of the music is lost.

To fight back, Ian Davis is sponsoring the annual Dynamic Range Day, to encourage listeners to look for material that’s well recorded, and to encourage recording engineers to put more emphasis on the quality of the sound, and less on making it loud. Ian’s also written an Open Letter to the Music Industry.

Happy Dynamic Range Day.

I had the opportunity to see the reconstruction of Colossus in operation, breaking actual codes, when I visited the British National Museum of Computing last year. “Lynetter” has posted to Youtube a video introduction to the reconstructed machine and also video of the late Tony Sale showing the reconstruction in operation. Unfortunately, when I visited, Tony had just recently died.

One twist on all this struck me as interesting: when Colossus was (partially) declassified in the mid 1970′s, books like The Ultra Secret claimed that it was used to break the German enigma code. Recently, more information has been declassified, and it’s now clear that’s wrong. The British did break the enigma at Bletchley park using a machine known as the Bombe. The Bombe (not bomb!) was not an electronic digital computer and it did not use valves (what we Americans call vacuum tubes). The Bombe was built of technology similar to old phone company rotary stepping switches and relays. Patch panels encoded information relating to the keys to be broken, and the switches would rotate through all possible positions until patterns suggesting a possible key match were detected. The switches would then stop in position, and from the final positions keys could be determined.

The enigma was used to encipher field and naval communications. In fact, we now know that Colossus, which was an electronic computer using large numbers of “valves” (along with relays, high speed paper tape, etc.) was built specifically to crack the much more difficult Lorenz cipher, which was used by the German high command. The British were thus able to decrypt traffic going to and from Adolf Hitler himself.

One of the important features of the HTTP specification is that GET, the operation that is used for almost all attempts to retrieve a Web page, has some very carefully crafted semantics. In particular, GET is inappropriate for any request that, directly or as a side effect, updates the state of a server. What’s an update? Well, taking money out of your bank account, confirming a plane reservation, or, in my opinion anyway, using up one of your 20 free New York Times page accesses. The way the HTTP specification puts this is:

In particular, the convention has been established that the GET and HEAD methods SHOULD NOT have the significance of taking an action other than retrieval. These methods ought to be considered “safe”. This allows user agents to represent other methods, such as POST, PUT and DELETE, in a special way, so that the user is made aware of the fact that a possibly unsafe action is being requested.

Exactly. I really want to be warned before I access a page that’s coming out of my monthly quota, and that doesn’t happen today. Furthermore, it’s perfectly legitimate for software to follow links for me without me even asking. Consider an e-mail reader that might offer to download for me local copies of any page linked from any e-mails I might have received, perhaps so that I can read my correspondence while I’m traveling. Such an agent is unlikely to be aware that such accesses can quickly deplete my NYT allowance, and perhaps for pages I never intended to read anyway.

Does this actually bother anyone in practice? Yes. In fact, I think this is the technical explanation for one of the most frustrating aspects of the paywall. There are all sorts of situations in which one winds up unintentionally clicking on a link to an NYT article, only to discover after the fact that yet another “free” access has been accounted. This happens when links are sent using URL shorteners, when the link text or image does not show the URL, or when the user neglects to read the target URI carefully before following a link.

So, what should the NYT do if they want to continue to offer free articles, and also follow Web architecture? One acceptable answer would be: the response to a GET request for an article that counts against the “free” limit should redirect to a simple Web form that says:

“You are about to use one of your 20 free accesses to the New York Times for this month. If you wish to go ahead, click the button below”.

The button would, of course, do an HTTP POST,  to properly account the access before returning the article.

To minimize the need to go through such prompts repeatedly, links from NYT pages to other “free” NYT content could use Javascript onClick tricks to directly retrieve using POST, without first going through the prompt page. Ideally, some less disruptive indication, such as a popup tooltip, could warn in advance if an access would count further against a user’s quota. With suitable use of cookies, it would also be possible for users to set preferences to disable such warnings.

As it stands, the paywall misuses HTTP, and the consequences are indeed frustrating to users. Furthermore, if too many sites start misusing GET, then it will become more difficult for us all to explore the Web without worrying about the consequences for each link we follow, and it may also be more difficult for engines like the Google crawler to retrieve Web content.

By the way, the W3C Technical Architecture Group, which I chair, has written about safe use of HTTP in its finding:  URIs, Addressability, and the use of HTTP GET and POST.

The first war of physics : the secret history of the atom bomb, 1939-1949 [ISBN: 1605981974] by Jim Baggott is an excellent history of the physics, politics, wartime events and espionage that all contributed to the remarkable history of the bomb. This is a moderately long book, but very readable, even gripping (well, I like this stuff). Very highly recommended for anyone with an interest in 20th century history, military history, or the history of technology. The physics is explained in readable terms, in the few places where it’s important to the story, but no technical background at all is required to appreciate this important book. Again, very highly recommended.

Note that the first 2/3 of the talk is a very interesting exploration of the security characteristics of Bitcoin, also showing how the Bitcoin database can be used as a peristent shared store. The latter third of the talk introduces Dan’s tools for detecting artificial delays introduced by ISPs.

May 7 (this Sat): Princeton Station, in Chelmsford, MA (on Route 4, just east of Route 3)
May 13 & 14: The Haluwa Lounge, in Nashua, NH (in the Nashua Mall, exit 6 off Route 3)
May 28: Back at Princeton Station one more time

All shows start at 9 PM, and there’s no cover at either Princeton or the Haluwa. I expect to be playing bass all four nights.

If you’d like to be on the mailing list to get announcements of all our gigs, please e-mail me at “blogmaster AT arcanedomain DOT com”, making the obvious substitutions for AT and DOT.

Noah

One of the things that I love about both the NYT and public radio is that they are such important resources for educating everyone. When I was growing up we certainly weren’t rich, but a copy of the Times was quite affordable, and the Times was the main way I learned about the world. The day Kennedy was shot, it cost ten cents to read the Times’ coverage, so everyone could. It was and is an extraordinary resource: in those days, you read the Times to get, for example, the full text of presidential speeches, or to see the Pentagon papers unfold, or just to read multiple articles each day with differing perspectives on the Vietnam war. The archives of the Times were freely available on microfilm at my local public library which, of course, paid for those copies with tax dollars — a daily history of the whole 20th century, effectively for free. Every school child was expected to be able to do current events research using the Times, and that often involved scanning way more than 20 articles/month. And by the way, if you did pay your dime for a copy, that was for the whole family, and often for friends too. The Times paywall is a system which, due to its high cost and very limited exceptions, limits this wonderful resource to those who can afford a very significant payment. In fact, even the act of mailing someone a link to a Times article becomes disruptive, now, because following that link might  cause the recipient to exceed his or her free limits — you probably didn’t notice that the Kennedy link above goes to an NYT page until you clicked it (that page is probably free, but you certainly don’t know that unless you check the link very carefully before clicking it, and maybe not even then).

I understand that the Times needs a new business model. In fact, I would be very happy if the Times were like public radio, I.e. a nonprofit that accepts donations from those who care to pay, to support universal access. Whether that would be practical I have no clue, but if so I would contribute generously.

Even if none of the above were a concern, the Times has made a huge mistake by building a system that is so easily defeated and for which the rules are so confusing. Links from Twitter are said not to count, but the Times hasn’t told us exactly how that works. Is it only links followed from the Twitter Web interface? Do links followed from Tweetdeck count? If I find a link in a Tweet and email it to a friend, and they follow it from the email, does that count? Does it matter if URL shorteners are used? How do we know until it’s too late whether following a link has counted toward our limits? I may be missing something, but systems like this don’t make for happy users. Simple instructions for avoiding the limits are already everywhere on the Web; I presume at least some of them work. So, in addition to all the other frustrations, honest users face the temptation to cheat, just to see whether that 21st article is interesting or not.

In a nutshell, the Times now has a system that, if used according to the rules, limits quality access to mainly the wealthy. As an added twist, the rules are confusing, brittle, supposedly easy to circumvent, and poorly documented. This is nothing at all like public radio.

Background: the #! fuss

I’ve been thinking about all this for quite some time, but the recent fuss over so-called “hash-bang” (#!) URIs raises the stakes considerably. As previously noted, Jeni Tennison has posted a terrific overview, with links to contributions from many others. If you’re not up on all that, please read those postings first.

Three examples

Consider three Web applications, each of which makes extensive use of Javascript.

1. The simulator is an interactive environment that simulates, we’ll say, driving a race car. You launch the application by navigating to its URI, and then Javascript takes over, providing an interactive simulation of the car driving around a race course, while giving you control over the car’s acceleration and direction. Just to make the example more clearly Ajax, we can assume that the application periodically goes out to various Web sites to retrieve data to be integrated into the simulation (weather information, etc.)
2. The résumé browser is similar in style to AJAX implementations of GMail or Yahoo mail, but its purpose is to let the user browse through collections of job application résumés. The initial screen provides a list with one line per applicant, but the user can request to see individual résumés as well.  Under the covers, all the usual Ajax tricks are being used to pre-cache résumés, to provide fluid interactivity for the user, etc. Users require the ability to email and post links to the individual résumés.
3. The final example is the full (as opposed to mobile) implementation of Google Maps. Like the others, this is an Ajax application. Users can retrieve maps for most any part of the world, zoom, pan, and annotate the maps with points of interest or driving directions. Crucially, an interface is provided that gives a URI (with no # in the syntax!) for the current location, zoom level, annotations, etc. As with the résumé application, these URIs can be emailed, linked from other Web pages, etc., and indeed this is a very valuable capability that is often exploited by users.

Resource identification

With respect to URIs and resource identification, the résumé and simulator applications are quite different in character. Indeed, if we ignore implementation concerns, the résumé browser closely resembles a traditional, non-Ajax, document-based Web site. Users have every reason to want each résumé to be integrated into the Web as a first class resource with its own stable URI, per the Architecture of the World Wide Web‘s mandate to Identify with URIs. It’s very important that people be able to email links to these résumés, put links to them on other Web pages, etc. Ideally, non-Javascript user agents should show the same résumés as Javascript-enabled browsers (albeit with less fluid navigation). Furthermore, it’s very useful if crawlers and other server-side agents can find and retrieve the documents, and for the FWD/BACK buttons in the browser to work as they would in a non-Ajax implementation.

Even though the implementation is very similar in its use of Javascript, the simulator is very different in character. There is no sense in which the user is navigating through a succession of pages.  Maybe or maybe not the application would find some use for FWD/BACK, or could provide links to intermediate simulation states that might usefully be emailed, but in any case there seems little need that such links be crawlable, or that non-Javascript user agents do much with them.

My main concern here is that too few Ajax applications take the trouble to meet the goals set out for the résumé browser, even when they could. With some hand waving about how the Web has moved past its old static past, and making some quite valid points regarding what happens when you update the address bar in typical browsers (you tend to kick off a page retrieval if anything other than the fragment identifier is touched), the designers of these applications often punt on providing useful URIs for the individual documents managed by their applications. Furthermore, if such identifiers are provided, they are often distinguished only in the fragment identifier, resulting in all the disadvantages noted by Jeni, Tim Bray, and others. For example, the resources tend not to be accessible from non-Javascript  agents, rather ugly kludges are required if crawling is to be enabled at all, etc.

There are two main points I’m trying to make in this note. The first is:

Use of AJAX implementation technology is not a sufficient excuse for failing to provide first class URI identification for documents on the Web

If you’re building a simulator, this advice may not apply; if you’re building the résumé browser, then provide URIs for each résumé, and probably for each sorted collection of résumés, etc. Ensure that those URIs work, at least to display the referenced document, when accessed with non-Javascript enabled user agents.

Documents when possible

Now let’s consider the Google Maps example. What’s interesting about it is that the Google engineers could have easily made the claim that the maps application was like the simulator, provided a single URI for the entire application, and stopped there. They might, to go a bit further, have done some local kludging to enable FWD/BACK to work, while still not exposing URIs for the different views of the map (or if you prefer, for the different maps.)

In fact, Google Maps is valuable in part because Google did take the trouble to model their application as a collection of document-like Web resources, and in fact to create URIs that work equally well in Javascript- and non-Javascript enabled user agents. To do this, they created URIs with no fragments (I.e. no #) for each view of the map; the compromise, at least for now, is that these URIs do not show up automatically in the address bar, but are available from a separate link button supported by the Javascript.  These URIs have some interesting characteristics:

• They are understood by both server and client: when a request is received from a non-Javascript enabled user agent (often from a mobile device), the server interprets the entire URI and sends a relatively static image of the map as a response; when the same URI is received from a Javascript agent, the Ajax implementation is used. Interpretation of query parameters is (potentially) distributed between the server and the client, and that division of labor can change without affecting the URIs.
• Because the same URI is used in all cases, a link can be created on a Javascript-enabled client, emailed to a non-Javascript client, and the right thing happens. Contrast that with http://twitter.com/#!/noahmendelsohn vs http://twitter.com/noahmendelsohn,  which are provided by Twitter for Javascript and non-Javascript clients respectively. The former does nothing useful if emailed to a non-Javascript client.  Pointing again to the advice given in Web Arch  Avoid URI aliases: “The problem with aliases is that if half of the neighborhood points to one URI for a given resource, and the other half points to a second, different URI for that same resource, the neighborhood is divided” The #! forms also break the Self-describing Web.

Google Maps shows the great value of using the document abstraction when you reasonably can. It wasn’t the easiest choice for that application, but the benefits are very significant. In spite of being very fluid and interactive, each state of the Google Map is modeled as a document with its own URI. You can email links to particular maps, to driving directions for your house, or to maps annotated with places of interest. Each such document is available, using the same URI, from the non-Javascript implementations, and in principle, server agents can crawl links to the different maps (e.g. to find maps that are popular).

So, using Google Maps as my poster child:

Where practical, model your application as a collection of documents, each with its own URI

Syntax: # or ? or neither

Note: this section was updated on 10 March 2010 based on the important observation from Jeni that the query component isn’t necessarily more important than the hierarchical path as an alternative to # fragment syntax.

I continue to have a general feeling that fragment syntax (#) has been over-emphasized in all this, and that identification using other parts of the URI, such as the path for hierarchical identification and/or the query string (with ?) for non-hierarchical may be preferable. Yes, for that to happen, we’ll have to get changes like pushState() and changeState() widely deployed, or maybe we’ll need other changes too. Still, from an architectural perspective, # seems somehow more fragile: until we ran unto address-bar update problems, fragments were used with documents to identify, well, fragments. Yes, with new media types we can do what we want, but in this case we don’t have a new media type, and the normative specifications make clear that for text/html, fragments refer to anchors within the document.

To be clear, I don’t think there’s any outright architectural barrier to server-side interpretation of fragment ids in cases where agents like crawlers happen to get hold of them, but of course, typical clients don’t send them to the server at all. Maybe we can find a way to use # that meets all the goals, and update the normative specs to support going that way. My guess is that, in the long term, avoiding fragments by using path and/or query is a better solution. Either way, we need to align the specs. with common practice.

Anyway, my main point is to model things as documents where possible, and to give each document a unique, crawlable URI that works consistently at client and server.

Finally, I hope it’s clear that I am in no way discouraging Ajax implementations, whether for résumés, simulators, or maps. I don’t want to let Ajax be an excuse for building a Web full of documents that can’t be properly linked.

Documents in applications by Noah Mendelsohn is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

• Saturday, Feb. 5th, at Princeton Station in Chelmsford, MA.  If you live in northern Boston suburbs like Arlington or Lexington, this is about 20 minutes up Route 3.
• Friday and Saturday, Feb. 11th and 12th, in the lounge at the Haluwa Restaurant in the Nashua Mall. That’s at Exit 6 off Route 3 in Nashua, NH, a few miles north of the Mass. border
• Friday Feb. 18th, we’ll be appearing in the student center at Bentley University.

Come on out and join us! The Princeton and Haluwa shows start at 9PM and go until about  1AM. I think at Bentley we play two sets, starting at 9:40 PM and ending at 12:30 AM. We play covers ranging from the 60′s (Blues Brothers, Sam & Dave, Beatles) to the present (Lady Gaga, P!nk, Black Eyed Peas, Kings of Leon). If you need directions or more information, let me know (contact information can be found on the About this blog page).

By the way, Alan Kay used to have some wonderful online lists of recommended readings, and I haven’t been able to find them lately.  Anyone know whether they’re still posted?