In recent months we have had various RDS servers experiencing recurring issues with becoming unresponsive, in the sense that no users are able to log on to the RDS servers until the affected server(s) are power cycled.
Overview: When the servers experience the issues RDP connection to the affected RDS server(s) will immediately display the "Please Wait for the Group Policy Client" message, and will hang at this message indefinitely until the server is power cycled. When attempting to connect to the vCenter console of the affected server(s) a logon screen is displayed. However, when attempting to log on to the console using either a Domain Admin or the server's local Administrator account it will indefiinitely hang at the "Please wait for the User Profile Service..." screen.
When this issue occurs we are still able to remotely connect to the Event Viewer and Services console of the affected server(s), although we are unable to log on to the server. The chain of events leading up to the issue in the event log always appears to start with the following unresponsive service error in the System log.
"A timeout (30000 milliseconds) was reached while waiting for a transaction response from the ShellHWDetection service."
While this is always the first error event that seems to occur, there are also various other services displaying the same errors subsequently. These events will continue recurring in the log, until the server is power cycled.
A timeout (30000 milliseconds) was reached while waiting for a transaction response from the ShellHWDetection service. A timeout (30000 milliseconds) was reached while waiting for a transaction response from the wuauserv service. A timeout (30000 milliseconds) was reached while waiting for a transaction response from the BITS service. A timeout (30000 milliseconds) was reached while waiting for a transaction response from the CertPropSvc service. A timeout (30000 milliseconds) was reached while waiting for a transaction response from the Schedule service. A timeout (30000 milliseconds) was reached while waiting for a transaction response from the seclogon service. A timeout (30000 milliseconds) was reached while waiting for a transaction response from the SessionEnv service.
Additional Details:
- When this issue occurs the CPU and RAM resource utilization on the server will be normal, and there will be a normal amount of user sessions on the server(s).
- This issue has happened repeatedly for some of the same servers, but has happened for various RDS servers in multiple different 2008R2 RDS environments.
- The resource utilization on the Domain Controllers is normal, and there are never any error or otherwise suspicious events in any of the logs on the Domain Controllers when this occurs.
- When domain users log on to the RDS servers some redirected folders are in use at a DFS path. We have tried restarting the DFS services when the issue occurs to no avail, although DFS was originally not a suspect since the issue only seems to happen to one server at a time.
- The problem seems to only happen to about one server at a time in an environment. The remaining RDS servers in the farm(s) will allow users to log on to them normally.
- It has been reported that when the issue happens existing sessions for users on the affected server(s) will also become frozen and unresponsive, although we have not had a test account logged on at the time that the issue starts to confirm.
- We tried disconnecting the NIC on the VM in vCenter so that it would not be able to attempt to contact a domain controller, and then logging on with the local Administrator account, but still get stuck at the same "Please wait for the User Profile Service..." screen.
- We have tried disabling all GPOs that are linked to the OUs that the servers are in while the problem is happening, but it does not change anything.
- Aside from the unresponsive service errors, no other obvious error events appear to be present when the problem starts.
- The services that state show as being unresponsive in the event logs actually appear to be in a running state when the problem happens.
- There are no individual users who have been logged on to the server each time when the problem happens.
- The problem seems to have started within around the last 1 - 2 months (the first reports that we are currently aware of began coming in in Mid-October 2017.
- Aside from regular Windows patching and application maintenance we are not aware of any notable changes in the environments in the recent past.
We identified older articles which stated that issues with very similar, if not identical, issues had in the past been addressed with certain hotfixes. However, the hotfixes referenced in the older articles no longer appear to be applicable, as they are outdated, and we have not been able to locate any newer hotfixes for these issues.
For reference, the old hotfixes that both say that they are no longer applicable when attempting to install them are:
https://social.technet.microsoft.com/Forums/lync/en-US/2c4b8121-da1c-4c11-b11d-2dff099ba245/windows-server-2008-r2-sp1-rds-hang-and-stop-responding-new-rdp-session-already-connected-session?forum=winserverTS
https://support.microsoft.com/en-us/help/2622802/the-group-policy-client-service-crashes-on-a-terminal-server-that-is-r
https://support.microsoft.com/en-us/help/2653810/security-group-filter-in-group-policy-preferences-does-not-remove-a-us
Has anyone recently seen any issues such as these and been able to identify fixes for them?