Most people are aware they should check for an SSL connection when logging into a banking web site or other sensitive web application. The little yellow padlock and "https://" prefix are supposed to indicate the web site is "secure" and no one can compromise your sensitive data.

SSL is the only choice to protect web data in transit and to verify the identity of the server. However, SSL is often used when the only data that must be protected are authentication credentials: usernames and passwords. Public discussion forums, public web mail and blogs are examples:  the content is already public so encrypting that data with SSL is pointless. All that is really needed for those types of applications is protection against interception of usernames and passwords.

The encryption routines of SSL seriously tax a web server when under load. Yahoo Mail, for example, avoids this overhead by using SSL only to encrypt the login page then switches to unencrypted HTTP after that.

There are many times when just the login credentials need to be protected. Acquiring a costly SSL server certificate from a certificate authority then writing the web app to ensure it only uses it for authentication is overkill. Even with SSL in place, many web applications stores usernames and passwords in a database in clear text. In such a situation, an attacker compromising the server has access to all user authentication credentials.

HTTP digest authentication is a little-known standard built into all major web servers and web browsers. With it, login credentials are never transmitted across the network, nor are they stored on the server in plain text. There is no need to purchase an expensive SSL host certificate each year, nor worry about CPU load due to cryptographic processing.  Also, the web server itself performs authentication, removing the need to develop that code in the web application itself. The application only has to verify that authentication has taken place.

Digest authentication is not the same as it’s older, dumber brother Basic authentication. Web servers and browsers have always supported Basic authentication, where username and password are transmitted in easily-decoded Base64 and must be stored on the server in plain text. Digest authentication looks similar from the user’s perspective… the browser pops up a dialog box asking for username and password. However, under the hood the protocol uses hashes instead of plain text.

Likewise, Digest authentication is not the same as it’s evil cousin NTLM authentication. Microsoft’s proprietary LAN Manager authentication (now dubbed "Integrated Windows Authentication") doesn’t transmit credentials in plain text, but requires a continuous TCP connection between server and client, violating the stateless nature of the HTTP protocol. NTLM has a hard time traversing most web proxies and load balancers (ironically, the only reliable way to get NTLM to do that is encapsulate it in SSL). Although it’s now semi-supported now in non-Microsoft browsers such as Firefox, NTLM is proprietary and not an Internet standard.

How Digest authentication works

Digest authentication is a challenge-response mechanism:

• The browser sends an HTTP request (e.g. a GET) to a web server
• The server sees the URL being accessed has been configured to require Digest authentication and replies with a 401 "Authentication Required" status plus a "nonce": a unique hash of several data items, one of which is a secret key known only to the server.
• The browser pops up a dialog box requesting username and password. Once the user enters their information, an MD5 hash of the username, password, nonce and URL are computed and the browser re-sends the original request along with the hash.
• The web server compares that hash with it’s own computation of the same values. If they match, the original HTTP request is allowed to complete.

For a complete explanation of Digest authentication see RFC 2617, but the idea is that since hashes are used instead of the actual data, an eavesdropper intercepting the communications never sees the username or password.

Replay attacks where the interceptor sends the exact same data at a later time are prevented in several ways. In the Apache web server, a unique nonce is generated for each 401 message and incorporates time and other information to prevent replays.

How to use it

Digest authentication is applied to specific URLs or physical directories on your web site. One convenient aspect is that protecting one directory usually results in Digest authentication being automatically applied to all lower directories as well. The specifics of implementing Digest authentication depend on the particular web server:

• Apache Digest authentication
• IIS 6.0 Digest authentication

For Apache, a text file is used to store the username in plain text plus an MD5 hash of the password. Credentials can also be stored in a database using Apache modules such as mod_authn_dbi from the 2.1 Authentication Project.

Rolling your own

Many web developers try to create their own authentication mechanism rather than use standard mechanism, sometimes for no other reason than they dislike appearance of the browser’s password pop-up dialog box. Many prominent web sites don’t even bother to try preventing sending username and password in the clear.

Some developers are more vigilant and try to roll their own hash mechanism to perform authentication. However, this involves writing either Javascript, a Java applet, ActiveX or similar on the client side to compute the hashes, plus a matching component in the server app. Cryptography is tricky. Even when it’s only using a cryptographic hash to verify credentials it takes care, time and effort to get it right. For example, we show what’s required to to use HMAC digesting to protect web forms in our paper Preventing HTML form tampering.

It’s much easier and more reliable to use the Digest mechanism that already exists in popular web browsers and servers. The dialog presented to the user is not pretty, but Digest authentication it can be implemented in nearly zero development time, with the added benefit that the implementation is more likely to be free of serious errors.

One recommendation: when writing a web app that relies on the web server to perform authentication, it’s a very good idea for the application to verify authentication has taken place. The default state for web servers is to allow access to a URL… only a single configuration directive causes the server to prompt for a username.

In Apache when a user has been authenticated the environment variable "REMOTE_USER" is set with the username. The web app should test for that and display an error if it’s not present. It’s too easy for a web server configuration mistake to disable the authentication prompt and allow wide open access to data: an additional check that authentication has occurred should exist in the web application itself.

Other caveats and gotchas

Digest authentication is appropriate where you need to protect only the username and password from compromise. When the actual content must be protected (i.e. form data sent from the browser) or when the user needs strong assurance that they are connecting to the right server (e.g. a bank), SSL is required.

Also, Digest authentication is of course single factor and thus subject to all the weaknesses of traditional username / password authentication. If strong authentication of the client is needed, look into implementing SSL client certificates or some other two-factor mechanism.

Note that the MD5 has algorithm has been shown to be cryptographically weaker than previously thought. A "collision" (an identical hash from different data) can be found with less effort than once thought. MD5 hashes are still considered strong, but the crypto community accepts that it is time to start moving to other mechanisms like SHA-256. However, because of the way MD5 hashes are used (e.g. combining them with a short-lived nonce), the issue doesn’t really affect Digest authentication.

Protecting the login credentials is often all that many web applications need. In those situations, Digest authentication provides an good lightweight authentication mechanism. It is a published standard supported by all major web servers and recent browsers, provides good protection against interception, and removes both the need to code your own login mechanism and to store usernames and passwords in plain text on the server.