Hi,
I've been having an ongoing issue with Public Folder performance, which didn't exist prior to upgrading to Modern Public Folders. I've been through 2 rounds of support from Microsoft (one from support Europe and one from support US) without any resolution. I'm looking for idea about tracking down the cause or fixing this issue.
I have 2 DAG's at 2 separate sites (Exchange 2013 CU12 - Windows 2012 R2 - 4 Mailbox servers and 2 CAS per site - 1000 users per DAG). The users at Site B are heavy public folder users although 100 or so users from Site A access the folders on Site B. The primary hierarchy mailbox lives at Site A. People accessing these public folders (about 1000 total folders) experience major client issues where Outlook freezes and hangs (sometimes in non-responsive state) for minutes while browsing folders or messages within the folders.
During these times, we see MAPI backoffs in the MAPI client logs to the order of 10,000+ backoffs per day. This is not from a throttling policy. We tried implementing the RPC backoff reg hacks to no avail. After that, we implemented MAPI-over-HTTPS, which is when the backoffs shifted from RPC logs to the MAPI logs. Accessing these folders through OWA is much better, so the issue seems to specifically affect MAPI/RPC clients.
Main actions taken and checked:
- Server resources (CPU, memory, disk, etc all performing with no issues)
- Network latency (Network does not appear to be heavily burdened)
- Throttling (not triggered by policy and even created a policy with huge limits to no effect)
- RPC backoffs (reg hack did not work - implemented MAPI over HTTPS - this moved backoffs from RPC logs to MAPI logs)
- Extrace on mailstores during issues (Nothing found by MS support team)
- Prevented data-serving mailboxes from serving hierarchy and created new empty PF mailboxes assigning them to users as their default PF mailbox
- Split out top level folders to own mailbox (eg /Folder/Sub1 and /Folder/Sub2 are the main issues. Moved /Folder/Sub1 to its own mailbox and /Folder/Sub2 to its own mailbox - This allowed for some relief, but not much)
- Changed MailboxLockMaxmimumWaitCount in Microsoft.Exchange.Store.Worker.exe.config file to 20 and then again 50.
- Created DWORD with value of 120 seconds for HKLM\Software\Policies\Microsoft\Windows NT\RPC\MinimumConnectionTimeout
- Added HttpProxy.ConcurrencyGuards.TargetBackendLimit with value of 2000 to web.config files on FrontEnd and BackEnd /mapi directories in IIS.
Those are the greatest hits. This the backoff error in the MAPI logs:
RpcEndPoint: [ServerTooBusyException] Client is being backed off -> [ClientBackoffException] Backend reported that it is too busy -> [StorageTransientException] The process failed to get the correct properties. -> [MapiExceptionRpcServerTooBusy] Unable to get properties on object. (hr=0x80004005, ec=2419)
Any input is appreciated. I'm getting ready to create a bunch more mailboxes and shift folders to them, but that's a lot of work and hard to back out of. Thanks.