Sunday, January 3, 2016

Non-Organizer Cancels Recurring Meeting

A while back we had an interesting issue where a non-organizer cancelled a recurring meeting for an entire group of people. And, life being what it is, it was of course for a group of executives in a large organization.

The message text was something like this:

From: Microsoft Outlook On Behalf Of Susan Smith
Sent: Monday, November 30, 2015 7:00 PM
To: John Roberts
Subject: Canceled: Management Meeting
When: Occurs every 1week(s) on Tuesday effective 1/1/2015 from 9:00 AM to 11:00 AM (UTC-06:00) Central Time (US & Canada).
Where: Boardroom

*~*~*~*~*~*~*~*~*~*
This meeting has been canceled by the organizer.

This meeting has been canceled by the organizer. To see the original contents of the meeting, view the item in your calendar.

  _____  
Sent by Microsoft Exchange Server 2010

This type of message was send to all meeting participants, not just John Roberts. The meeting organizer was not Susan Smith. And the meeting organizer indicated that she didn't cancel the meeting.

Reviewing the message tracking logs indicated that a cancellation message for each participate was sent by Susan Smith. The message cancellation came from a Blackberry Z30 which uses ActiveSync.

It took a while to verify this, and was complicated by the fact that Susan Smith was not in the office to verify any information. We had also recently turned on the Calendar Repair Assistant and reviewed those logs to verify that it had not done this. We verified that the Calendar Repair Assistant had not processed Susan Smiths mailbox yet. We also verified that the repair performed for the meeting organizer didn't adjust this meeting.

Eventually, we found a Microsoft KB article that indicates some ActiveSync implementations have a flaw that allow attendees to become organizer of a meeting if they modify a meeting instance. The organizer isn't updated for other users, just the user with the ActiveSync device. The relevant text is this:

Issue 1.2 - An attendee becomes the meeting organizer

When a user synchronize their iOS or Android device by using Exchange ActiveSync, they may unexpectedly become the organizer for a meeting to which they were invited. This does not change the meeting for all attendees.

This issue can occur if you change the reminder for a single occurrence of a recurring meeting on an iOS or Android device. There may be other changes to meeting items that cause the same problem.

Current issues (Exchange Online, Exchange 2016, Exchange 2013, Exchange 2010 and Exchange 2007)
Recent issues that have similar symptoms have been reported to Microsoft. These issues can affect users on all versions of Exchange Server. Currently, Microsoft cannot reduce the effect of these new issues. We encourage users to work with their device vendors to find a solution to the problem.
At this point, my best explanation is that the Z30 allowed Susan to become organizer. Then when Susan deleted a meeting which she would not be attending, the meeting was cancelled for all users. This fits our facts:
  • Susan was the source of the cancellation
  • Susan was using an ActiveSync device
  • The email message indicated that the meeting was cancelled by the organizer

Reference:

Saturday, December 12, 2015

Netvanta UC Poor Quality Audio with NIC Teaming

I have one customer that uses Netvanta UC for their voicemail system. It runs on a virtual machine hosted on a Hyper-V cluster. That's a bit of an issue in itself because the software fails if the VM is migrated between hosts. To migrate it between hosts you need to reactivate it. Not cool.

For the past while, they've been having audio quality issues with this system and have investigated all points in the system, but none seemed to show any signs of an issue.

I'll let the customer describe it for you:


We have been plagued with garbled voicemail for a while now. Finally, we found some Adtran techs who took a hard look and found that when voicemail was hitting our NetvantaUC server sometimes (fairly often actually) up to 35% of the packets were out of order, causing a robotic and sometime impossible to understand recording. They also checked our PRI gateway and noted that there were no issues from the PRI (MTS) or leaving the gateway. Of course this left one switch in question, which we had already ruled out (no issues on the switch we could find).

This sort of left the VM host itself. I removed one nic from the team on Host 03 (where our NetvantaUC vm is hosted) and dedicated it to the NetvantaUC VM. For the past three days we have had no garbled voicemail and touch wood no phone issues at all (we had other phone issues daily, dropped calls, on hold and transfer occasionally not working etc).
So, it appears that there is some odd interaction between NIC teaming on Windows Server 2012 R2 and the Netvanta UC. At this point we're not sure if this is a general issue with VoIP software and NIC teaming or something specific to Netvanta UC.

Friday, November 27, 2015

The Total Data Received From the Remote Client Exceeded Allowed Maximum

Working with a large number of mailboxes is usually about the same a working with a small number of mailboxes, except that you need to include -ResultSize Unlimited in your Get-* cmdlets. However, I recently ran into the following error when getting a large list of mailboxes (approx 38K) with Get-Mailbox and piping them to Set-Mailbox:
Sending data to a remote command failed with the following error message: The total data received from the remote client exceeded allowed maximum. Allowed maximum is 524288000.
In the end this seemed to be a limitation of piping from Get-Mailbox to Set-Mailbox. For example, this would generate the error:
Get-Mailbox -ResultSize Unlimited | Set-Mailbox -SingleItemRecoveryEnabled $true
One option to work around this is to chunk up the work and do it by database instead:
Get-Mailbox -Database XXX -ResultSize Unlimited | Set-Mailbox -SingleItemRecoveryEnabled $true
However, my preferred workaround was this:
$mbx=Get-Mailbox -ResultSize Unlimited
foreach($m in $mbx) {Set-Mailbox $m -SingleItemRecoveryEnabled $true}
Using a foreach loop instead of piping seems to avoid the error when dealing with large data sets. The powershell.exe process for this task consumed about 2.5GB of memory.

Thursday, November 26, 2015

List Disabled User Accounts with Mailboxes

I'm currently working with a large organization where they've lost track of mailbox removals when accounts are disabled. They keep disabled accounts for an extended period of time after users leave. So, some disabled accounts are years old and still have a mailbox.

We're going through and cleaning up right now. So, we wanted a list of disabled user accounts with an Exchange Mailbox. However, since resource mailboxes have a disabled user account, we need to exclude those. It's a nice easy one line of PowerShell:
Get-Mailbox -Filter {ExchangeUserAccountControl -eq "AccountDisabled" -and IsResource -eq $false} -ResultSize unlimited | export-csv C:\temp\DisabledWMailbox.csv

Saturday, November 21, 2015

View Service Status for MTS

If you are having data connectivity issues on the MTS network this web site will show you the status of their services. Useful to figure out whether it's systemic or just you (or your customer).

Tuesday, November 10, 2015

Exchange 2010 Addressbook Errors with NetScaler 11.0

I'm working with a client that is upgrading the load balancing infrastructure that services their Exchange 2010 organization. They are implementing NetScaler SDX appliances that run NetScaler VPX virtual load balancers. Being as this is new infrastructure, version 11.0 (first release June 30, 2015, latest release Oct 8, 2015) is being implemented.

To test the new load balancing, we modified the hosts file on few computers to direct the load balanced names (outlook.domain.com/webmail.domain.com) to the IP address of the new load balancer. Initially, this seemed to be working fine, but we didn't do any specific testing.

After a bit, we noticed that we often got the following error in Outlook when accessing the address book:
The connection to Microsoft Exchange is unavailable. Outlook must be online or connected to complete this action.

This error was reproducible after not accessing the address book for a few minutes. In addition, when looking at the connection status, we would see failures talking to the directory. Connectivity to mail generally seemed fine, but also had occasional failures.

Connection Status with directory access failures

When you research this problem, you will find lots of references to incorrect network timeouts closing connections before the application is ready. This is exactly what we seemed to be having. However, we adjusted every possible setting on the NetScaler to allow connections to last up to two hours. No matter what we adjusted, the same problem remained.

Other network components were the same as before. So, it seemed the problem had to be the new NetScaler. Clients still using the existing load balancer did not have any issues.

After adjusting everything we could think of, we called Citrix support. In conjunction with support, the network team did some packet traces and confirmed that resets were being performed on the RPC connections for the address book.

The fix ended up being using version 10.5 (latest release Sept 2015) instead of version 11.0 for the VPX instance. As soon as version 10.5 was implemented the mysterious reset issue was resolved and the response time for the directory lookups dropped significantly.

Connection Status after implementing version 10.5
Citrix is continuing to evaluate the issue and I'll update when I have more information.

Update (Nov 10/15): Apparently NetScaler 11.0 has issues with Outlook Anywhere also: http://discussions.citrix.com/topic/368750-netscaler-11-6210-outlook-anywhere-broken/page-2

Update (Nov 13/15): Running version 10.5 we've still had no issues. It has been submitted as a bug with Citrix and we're waiting for a update. Hopefully the next revision will fix it, but they are definitely aware it's an issue.

Update (Nov 27/15): Citrix did some testing with the network folks again today with Version 11.0 (I'm assuming a new build) and still not working properly. We are back to Version 10.5.

Resources:

Monday, November 2, 2015

Query Mailbox by LegacyExchangeDN

In a large organization with mailbox quotas, you will see an ongoing number of events indicating that users are at the limit of their quotas. In the event log you will see:
Event Source: MSExchangeIS
Event ID: 8528
The mailbox for /o=ExchangeOrg/ou=AdministrativeGroup/cn=Recipients/cn=UserIdentifier has exceeded the maximum mailbox size. This mailbox cannot send or receive messages. Incoming messages to this mailbox are returned to sender.  The mailbox owner should be notified about the condition of the mailbox as soon as possible.
The overall identifier for the mailbox is the LegacyExchangeDN attribute for the user mailbox. Most of the time, you can look at this value and quickly figure out which user is having the issue. If it's a current user, you can wait or contact them. In a large environment, you may not personally know the users, and you can investigate whether this is a disabled account that needs to be removed.

Today I was looking at one of these events and saw that the final UserIdentifier portion of the LegacyExchangeDN attribute was a long GUID number. To find the actual user account I needed to query the mailbox with a filter for LegacyExchangeDN.
Get-Mailbox -Filter {LegacyExchangeDN -eq " /o=ExchangeOrg/ou=AdministrativeGroup/cn=Recipients/cn=UserIdentifier"}
Please note that using the -Filter parameter is much more efficient that doing Get-Mailbox and piping it to Where-Object. Using the -Filter parameter is almost instant whereas piping Get-Mailbox to Where-Object can take several minutes.