asp.net - IIS application using application pool identity loses primary token?

Question

(This is a question about a vague problem. I try to present all relevant data, in the hope that someone has helpful information; apologies for the long description.)

Our web app

We have a .NET 4 web application running in IIS 7.5 accessing Active Directory and a SQL Server database.

This web application is running under a virtual 'app pool identity', by setting the Identity of the application's application pool to ApplicationPoolIdentity. A concise description of virtual identities can be found in a StackOverflow answer, and the blog post to which it refers: an app pool identity is just an additional group which is added to the web application's worker processes which is running as 'network service'. However, one source vaguely suggests that "Network Service and ApplicationPoolIdentity do have differences that IIS.net site documents do not publish." So a virtual identity might be more than just an additional group.

We chose to use ApplicationPoolIdentity, as opposed to NetworkService, because it became the default in IIS 7.5 (see, e.g., here), and per Microsoft's recommendation: "This identity allows administrators to specify permissions that pertain only to the identity under which the application pool is running, thereby increasing server security." (from processModel Element for add for applicationPools [IIS 7 Settings Schema]) "Application Pool Identities are a powerful new isolation feature" which "make running IIS applications even more secure and reliable. " (from IIS.net article "Application Pool Identities")

The application uses Integrated Windows Authentication, but with <identity impersonate="false"/>, so that not the end user's identity but the virtual app pool identity is used to run our code.

This application queries Active Directory using the System.DirectoryServices classes, i.e., the ADSI API. In most places this is done without specifying an additional username/password or other credentials.

This application also connects to a SQL Server database using Integrated Security=true in the connection string. If the database is local, then we see that IIS APPPOOLOurAppPoolName is used to connect to the database; if the database is remote, then the machine account OURDOMAINourwebserver$ is used.

Our problems

We regularly have issues where a working installation starts to fail in one of the following ways.

• When the database is on a remote system, then the database connection starts to fail: "Login failed for user 'NT AUTHORITYANONYMOUS LOGON'. Reason: Token-based server access validation failed with an infrastructure error. Check for previous errors." The previous error is "Error: 18456, Severity: 14, State: 11." So it seems that now OURDOMAINourwebserver$ is not used anymore, but instead anonymous access is attempted. (We have anecdotal evidence that this problem occurred when UAC was switched off, and that it went away after switching on UAC. But note that changing UAC requires a reboot...) A similar problem is reported in IIS.net thread "use ApplicationPoolIdentity to connect to SQL", specifically in one reply.

• Active Directory operations through ADSI (System.DirectoryServices) start to fail with error 0x8000500C ("Unknown Error"), 0x80072020 ("An operations error occurred."), or 0x200B ("The specified directory service attribute or value does not exist").

• Signing in to the application from Internet Explorer starts to fail, with HTTP 401 errors. But if in IIS we then put NTLM before Negotiate then it works again. (Note that access to AD is needed for Kerberos but not for NTLM.) A similar problem is reported in IIS.net thread "Window Authentication Failing with AppPool Identity".

Our hypothesis and workaround

At least the AD and sign-in problems always seem to go away when switching the application pool from ApplicationPoolIdentity to NetworkService. (We found one report confirming this.)

Page "Troubleshooting Authentication Problems on ASP Pages" has some suggestions related to primary vs. secondary tokens, and what I find encouraging is that it links the first two of our errors: it mentions NT AUTHORITYANONYMOUS LOGON access, and AD errors 0x8000500C and "The specified directory service attribute or value does not exist".

(The same page also mentions ADSI schema cache problems, but everything we can find on that topic is old. For now we consider this to be unrelated.)

Based on the above, our current working hypothesis is that, only when running under a virtual app pool identity, our web application (IIS? worker process?) suddenly loses its primary token, so that IIS only has a secondary token, so that all access to Active Directory and SQL Server is done anonymously, leading to all of the above errors.

For now we intend to switch from ApplicationPoolIdentity to NetworkService. Hopefully this makes all of the above problems go away. But we are not sure; and we would like to switch back if possible.

Our question

Is the above hypothesis correct, and if so, is this a bug in IIS/Windows/.NET? Under which circumstances does this primary token loss occur?

Answer 1

Through Microsoft Support I found out that we ran into the issue described in Microsoft Knowledge Base article KB2545850. This only occurs when ApplicationPoolIdentity is used. It occurs very easily, namely, after the machine account password is changed (which by default happens automatically every 30 days), and then IIS is restarted (e.g., through iisreset). Note that the problem goes away after a reboot, according to Microsoft and our observations.

According to Microsoft it is not possible to check if your Windows/IIS has gotten into this state.

Microsoft has a hotfix attached to this KB article. There is no indication when that hotfix will be rolled into an official delivery, and the hotfix is already 10 months old. In our specific case, we decided to switch to NetworkService instead.