90% of Teams Performance Issues are *NOT* Microsoft Related – Get Network Monitoring

With the recent announcement of 145 million daily active users for Microsoft Teams it’s not surprising to hear call quality troubleshooting, and Teams performance issues are top of mind for everyone.  When Microsoft has an outage it makes the news and it’s a big deal, but not all issues are Microsoft related and even that it often takes Microsoft 30 minutes to scope the issue and do their own root cause analysis.  With the recent push for flexible work schedules, increased working from home, from literally anywhere the emphasis and need for IT to quickly address quality issues has never been more important, but also never been more complex.  According to Call Quality Data millions of signals from Teams users, “90% of Teams performance issues are NOT Microsoft related.”  In fact, most happen with the client machine including CPU, Memory, and at home or with the local internet service provider. 

When troubleshooting Teams support issues, don’t jump to conclusions.  It’s important to break down the data and look closely into the 4Ws… 

When Every Minute Counts:  (Who, What, When, Where, Why, Which and How?) 

When tickets start to come in, you need to quickly qualify the issue to determine the best course of action.  

Who are the users impacted?

What workloads are affected?

When did it start?

Where are the bottlenecks?

Why did this outage occur?

Which team is responsible?

How are the lines of business affected?

It typically takes between 30 minutes to an hour for Microsoft to send a notification to its users indicating that there is an outage occurring. So, what do you do during this time? Will your entire company wait for a confirmation tweet from Microsoft before taking any action? 

Martello’s DEM Solutions for Microsoft 365 Can Help 

Martello’s Digital Experience Monitoring (DEM) solution for Microsoft 365 helps organizations to qualify any service issues they have in minutes, allowing them to identify where the problem comes from, what is affected, where, and who is really experiencing its effects.

Martello gathers data from your existing monitoring tool and correlates them into a Microsoft service delivery map to quickly pinpoint the root cause of the issue.  

These are key Metrics Microsoft Teams Metrics:

From a general point of view, to reach these targets you should make sure that:

  • The latency one-way is stable during busy and non-busy hours
  • The latency RTT is stable and stays close to twice the latency one way
  • Packet loss are not increasing too much during busy hours (it is normal if it rises)
  • Jitter does not increase too much as well
  • Edge connection tests stay better than client connections ones

Being able to quickly determine if home LAN or egress including local network components, your VPN, as well as ISPs are to blame is beyond what most IT organizations have access to, otherwise they are blind in a situation like this waiting for complete Call Quality Dashboard reports that often miss individual call quality data impact on the client itself.  

Lay the Foundation  

Martello DEM has just helped you to considerably reduce the impact of the Microsoft outage by allowing you to identify, qualify, understand and take decisive actions before any official announcement of the problem.

As the problem is solved, Martello shows you in real-time that the performance is improving, and you know again beforehand that the service is about to be restored.

Martello provides the following:

  • Martello tests the service on a 24/7 basis, alerting you as soon as performance degradation is experienced with its robots which are running in each of your critical locations.
  • Martello often gives you advance notice of detected outages prior to the Microsoft notification being issued. Every second counts.
  • Martello tells you which workloads and which features of the service are unavailable or degraded and in which location.
  • Martello gathers data from your existing monitoring tool and correlates them into a Microsoft service delivery map to quickly pinpoint the root cause of the issue.  
  • Martello lets you go further and check your infrastructure to identify if there are any additional issues.
  • With Martello, you can quickly identify which users are being affected by the outage in each of the locations you have selected.  

Don’t wait for the next outage to start laying the foundation for an exceptional Microsoft user experience. The time to act is now.  

Want to improve your monitoring? Start with looking at this Microsoft Teams performance guide that describes the 8 metrics you need to be measuring. It also outlines Microsoft’s best practices and more “actionable nuggets” from Martello dashboards so you can improve your Teams user experience. Martello helps you identifying where the problem comes from, what is affected, where, and who is really experiencing its effects.

>> Download the Teams Performance Guide for Troubleshooting & Optimization <<

Microsoft 365 Virtual Marathon 2021 Insights

It’s hard to put to words what Microsoft 365 Virtual Marathon as an experience was for me and many others. I’m sure everyone had a different experience, but what started not this year but last year at the beginning of the pandemic brought a great need to find a way to help the community find connection through technology. Last year it was pushing Microsoft Teams Live events to host an event that brought so many of us together. This year, a year later it was the evolution of Teams Meetings leveraging the community tenant, using Presenter mode, videos on Together Mode, attendee, Microsoft Teams.

I’m happy to share a number of statistics in the spirit of transparency. Only just a few short weeks ago there was a plea for less virtual events. I know and realize many are exhausted of virtual events. So many of these events and there’s already fatigue. How and why would we possibly run a 60 hour event? We had 5150 wonderful attendees register for event with over 20,000 express interest. When we started this was to meet a need and we’ve evolved the event to focus on the purpose of “By the Community For the Community.” Last year we were able to help out sponsors who were looking for help with the transition to virtual from a number of failed events. This year we decided we wouldn’t focus on money and instead focus on any sponsor money would go toward marketing the event and getting the word out. None of the Executive Producers pocketed a dime from this event.

My proudest moment of the event was seeing just how far the reach went. As a traveler I was super excited to see in our attendee reports that we had over 115 countries represented across our attendees. Japan really took on the vision of the event as the #2 position with Mexico, Canada, Germany and India all with over 200 attendees. It was the fact we had participants joining from Myanmar, Iraq, Afghanistan, Mozambique, Ghana, Nigeria, Ivory Coast, Maldives and Faroe Islands helped us realize we truly are a global community.

LESSONS LEARNED

We learned a number of things running an event of this size… here are a few of my key takeaways:

  1. Teams “bombing” comment spam is real – We had a spammer who started the comments somewhat benign saying things like “I have a question,” but before long the meeting was spammed by hundreds of comment spam adult themes including ascii art. Blocking the user didn’t stop the comments as they’d pop in with another account within seconds. Turning off comments in the meeting was the only way to stop it, which we did for the duration of that meeting. We ended up turning comments off across all rooms for 1 hour. The spammer didn’t come back for the duration of the event, and we were able to turn comments back on in all rooms.
  2. Even though a room may seem persistent they only last 24 hours – We wanted the rooms in our event to last 60 hours which ultimately we were able to do, but at least once every 24 hours we needed to fully end and close down a room and start it back up (which only the organizer can do). The impact of a room reaching 24 hours any guest presenters will be kicked out of the room as the room is reset.
  3. Recording requires the Moderator or Presenter to be Logged in the Tenant – We delivered the event from the Microsoft Community Tenant. We had 75 moderators, but only a few of them were using their community accounts so many were unable to start and stop the recordings. With recordings now going to OneDrive, it can be a bit more complex to manage the more than 200 recordings vs. Live events where all the recordings are saved and managed in a single place. Live Meetings allowed us to automatically record all sessions with no babysitting. Meetings required someone in the tenant start and stop recording for every session.
  4. You CAN moderate Teams Meetings and present from Teams Mobile App – It was really with meeting options to configure the tiered roles in Teams meetings, and not only from the desktop but from mobile… it was very nice being able to elevate users from attendee to presenter. We setup a structure where the executive producers (organizers) were able to elevate moderators, and elevate speakers making it very simple with a single link for all users and presenters then only elevate those we want. This was a big improvement over mobile for Teams Live events which had an attendee only experience on mobile. Desktop allows you to be connected to 4 meetings simultaneously, but only one on mobile. We were able to see chat and activity across all the meetings in the tenant.
  5. Attendee getting started video and “How to attend” – You’d be surprised how many virtual events are not clear for attendees on how to attend. Making sure you have a good plan for handling support tickets and helping people who are attending. Communication can be a real challenge even for Teams Meetings. We ultimately had 145 support tickets over the 60 hours even though we had videos, and a big button on how to attend.
  6. Have a Communication Plan – We also were running the event when a global Teams outage occurred. Lesson learned, we will be integrating a website based alert for communicating issues out of our control where the platform fails to perform. By the way I’m sure it’s very rare, and 90% of network and infrastructure issues are NOT related to Microsoft, but it’s important to have a good backup plan or a good plan to communicate in the case where you need to communicate to all speakers, moderators and attendees. Email is a half way acceptable plan, but how much better if you have a big banner in the schedule…
  7. Start Recruiting Early for Moderators and Volunteers – If you’re running a large event you want to get really good volunteers who are willing to make themselves available for blocks of time. We had very few who took the day off, those were our super heroes! It was easier to find people willing to handle a single track for 1 or 2 hours, but this requires even more coordination. We had over 100 volunteers with 75 specifically named Moderators. When coordinating this with 250+ speakers who also making changes really encouraged us to have room checks being done by the exec producers to ensure the moderators were in the rooms really early. We were saying at least 25 min early with speakers at least 15 min early. A solid schedule that’s communicated and is a living document. Sessionize makes it easy to organize speakers, but gathering volunteers and moderators is a lot harder. We started with MS Forms used multi user spreadsheets. It’s easy for speakers and moderators both to be frustrated if they get scheduled for a time that doesn’t fit their schedule.
  8. Time Zones are your Friend or Your Enemy – We doubled down on dividing up the world into 3 areas. Americas, APAC (Asia Pacific & Australia and Greater Oceania), and EMEA (Europe, Middle East and Africa). Most speakers chose a preferred Area of the 8 hour zone, but many chose 2 or even all 3. This allowed us to simply put a given speaker into an area zone and not have to worry about individual time zones. This single preferred area reduced the amount of changes. When we first published the schedule there was some confusion, and sessionize does NOT make it easy to know who made what changes and if they were committed. All it takes is for 2 exec producers to be making changes and it will overwrite any changes that were not committed. If two have the schedule builder open it will only commit the last save. If both are making changes and click save, the last one wins and it won’t refresh the other person’s screen. We are sure this bit us a couple of times. There’s no multi user editing in schedule builder. In addition, users who view the schedule see it in their own time zone. We had an amazing schedule and finder interface built by Yugi, but most attendees and even speakers don’t know how to change to their own time zones in UTC or GMT (which is even more confusing with Daylight Savings Time), so relying on the sessionize schedule was our best bet despite the lack of search finder features. Anything that might mean an inconsistency in timing was very concerning for the speakers.
  9. Teams Meetings and Scale – One of the organizers of a previous event was worried about our strategy of elevating speakers from attendee. Despite the concern of 300 attendees interactive, we never ran into any debilitating issues even when we scaled into close to the 400+ range. We are looking forward to the Teams webinar and registration features for next year including the announced 1000+ attendees with interactivity. We were a little annoyed we couldn’t disable video on attendees when we were playing video, but more of these features have already been announced. We successfully were able to use Teams Meetings at scale for every session. No need to fall back on Teams Live events for this event.

One of the more interesting things about an event like this where you have hundreds of MVPs and community leaders providing sessions is the variety of sessions. Getting over 100 power user, end user, business level sessions as the largest topic is very powerful. Often the Microsoft focused events focus exclusively on Admins or Developers. Finding sessions focused on the users and power users are harder to find.

I was also pleased to see from an attendee perspective that the technologies they were most interested in were all of them. Right at around half of all attendees checked all the boxes when given a choice between individual technology focus. There’s a lot of interest in Teams right now and that’s totally understandable. I was also excited to see the panels and diversity and inclusion topics have gained a lot of momentum in the past few months. We have encouraged more panels and think those can reduce the online exhaustion. Our AltSpaceVR community sessions still have a long way to go, but one session on mixed reality was conducted in VR and while we had a dozen or so attendees, it was a great start at doing more community events in mixed reality with an even more immersive experience.

The session recordings that were allowed will be posted to the Virtual Marathon Youtube account at http://bit.ly/M365VMVideos the best way to get info on the recording availability is to subscribe.

Thanks so much for attending. We couldn’t have done this without you! We hope you will plan to join us next year… Will there be a next one? Yes, this event is not like other virtual events that are simply virtual due to COVID. We plan to keep up the tradition of getting the global community together once a year and we hope the interest continues… see you next year on May 4, 2022!

.

Troubleshoot Call Quality in Teams Fast

Microsoft Teams is Down

We Love Microsoft Teams, But We Hate Call Quality Troubleshooting!

Every IT admin has been in this fretful situation at least once. The solution or platform they support is broken and failing. They want to research what is causing the issue so they can get it fixed quickly.  But they need the detailed information to do the troubleshooting and find the root cause. There in lies the challenge when it comes to troubleshooting call quality issues for Teams. Getting to the root cause fast often means getting to the data otherwise you’ll end up pointing fingers saying… It’s not me.

Get the End to End data and analytics you need fast to troubleshoot Microsoft 365 Service and Call Quality Issues

Microsoft Teams has quickly surpassed all other enterprise tools to become the heartbeat of the enterprise.  Combining communication and collaboration in one tool puts all our energy and focus on a single tool to get work done. But what happens when calls start dropping for Executives, or meetings are garbled with audio that sounds like a robot voice?  Call quality is essential, and no one wants to wait around to hear excuses from IT. 

When Teams call quality gets jittery, fingers start getting pointed and you aren’t sure if it’s Microsoft, or the Network, the ISP, the client device?  You need answers FAST!

Microsoft Teams Blame

The folks at panagenda know this story well.  They have been helping customers quickly isolate and identify the issues with IT collaboration platforms for years.  And there are a lot of moving parts involved in the end-to-end voice traffic for Teams. Having full visibility into the complete experience from the user perspective is the key.  Any given hop could contain answers as to why you’re having call quality issues and having the accurate data to quickly pinpoints the smoking gun will help remediate the issue fast!

Some examples of issues that cause call quality problems …

  1. Background Processes Running on Computers (chewing up CPU) – Long running processes can eat up valuable CPU on a client machine. Antivirus running on a machine can ultimately lock it up and without realizing it you’re on a call wondering why your boss sounds like a robot. It’s not like a cell phone where the CPU is more compartmentalized.
  2. Too many Apps Open (Word, Excel, PPT) and web pages (chewing up available memory) – Many users often open too many programs and windows simply allows the users to reclaim the memory. Outlook, Word, Photoshop, it really doesn’t take much to get an older machine that may have other things installed and finding that the memory has run out and disk is paging. Client performance can quickly impact call quality in a big way. How many browser windows do you have open? Do you really need to close the browser? Don’t be surprised if you’ve got 100 tabs why the call may seem a bit scratchy and unclear.
  3. Slow home networks with low upload speeds – Users have an expectation that their calls will be problem free… and they should. Unfortunately for the user at home, the streaming video at home, the large video downloading on your desktop, and that cloud backup that’s running are all running over your little home connection. Network performance often is impacted by ISP outages as well. Poorly configured gateway or bad routes causing unnecessary hops to get to the Microsoft 365 Front Door can absolutely impact performance in negative ways. Even drivers can prevent optimum usage. A recent update to cable modems such as Docsis 3.1 has increased speeds with updated drivers. Mismatched or outdated drivers can greatly impact download speeds.
  4. Slow ISP’s with long peering distance to Microsoft Cloud – Internet service providers have their own DNS and routes. Often these routes are neglected or not necessarily optimized for the heavy usage of Microsoft 365 Online Services. In a corporates network setting it’s possible to manage routes and optimize hops, but this year has led to more work from home and network upgrades have not kept up. Bandwidth constraints, dropped packets increased jitter from so much more traffic has made important need for performance analysis and transparency. While some users may not have much to say about their ISP, others may find alternatives or monitoring solutions that can lead to optimization on home networks and gateways or equipment that can be updated or configured for optimization. The Microsoft Front Door can be optimized for the fastest route with the least hops and reduced line noise. We’ll show you how that’s possible.
Dive into the data:
Network Performance
Troubleshooting Client Performance
Performance Dashboard

I’m excited to announce an upcoming webinar hosted by panagenda where we will deep dive into troubleshooting Microsoft Teams Voice Call Quality.  We’ll also get a close look at their OfficeExpert End Point Monitoring solution and see how it makes troubleshooting fast and easy for IT support teams.

WHEN?  MAY 6TH, 2021 at 1PM (EDT) | 12PM (CDT) | 10AM (PDT)

>>Register now!

IT administration and support has shifted their focus over the past few years – USER experience is CRUCIAL. Enterprise organizations are looking for ways to improve usage and adoption for technologies at the heart of digital adoption, especially Microsoft Teams for calls and meetings. This has prompted IT support groups to find better methods to track and identify quality issues, especially for users working remotely.

Traditional network monitoring tools are optimized for data centers and centralized office locations where employees would access SaaS cloud applications from the corporate infrastructure. These tools no longer work in today’s enterprise landscape with many users working from home. Instead, enterprises need a holistic monitoring tool that provides end-to-end visibility into the employee experience, regardless of where they are working.

In this webinar, you will receive an introduction to OfficeExpert Endpoint Performance Monitoring (EPM). This new module provides data analytics for Microsoft 365 performance and availability from the end-user perspective. If you want to know the truth about your Teams call quality experience, then please join us to see how actionable insights can speed troubleshooting and help remediate issues before they become escalations from an executive.  

During the webinar demonstration you will explore topics that include: 

  • Proactive monitoring for home office network performance
  • Identify slow ISP’s causing poor call quality for remote users
  • Detailed analytics for end-point computers (CPU, memory usage, patch levels, etc.)
  • Fast troubleshooting for call quality issue resolution
  • And more…  

Don’t miss the opportunity to take advantage of this timely information.

>> Register Now!<<

Attendees will be entered into a raffle for a chance to win a free Xbox One!

Not Just Another Virtual Event? For THE Community BY The Community… Microsoft 365 Virtual Marathon 2021 is on!

It’s been a year since we came together as a community to run the Microsoft 365 Virtual Marathon in 2020. That event was a first-of-its-kind community event focused on including attendees and speakers from all parts of the planet in a truly global inclusive event designed to help people make new connections and be more aware of much of the rollout that was happening to support remote work in the early days of COVID while also supporting deep technical knowledge in 15 simultaneous tracks in deep technical content as well as end user and business content delivered by the community. 

Those that brought you M365 Virtual Marathon have come back by popular demand April 26-28, 2021 with a continued focus on community engagement, including featured speakers and keynotes from Microsoft and Community Leaders.  The M365 Virtual Marathon also features sessions covering new technologies, such as Microsoft Viva, providing a deeper dive, building on the product announcements at Microsoft Ignite. Keynotes include Jeff Teper, Corporate Vice President Teams, SharePoint, OneDrive (and Viva), Karuana Director Microsoft Teams, Heather Newman, Power Platform Principal PM Power Platform Engineering Community, and Laurie Pottmeyer, Microsoft Teams Community

>> Read more about the event at http://m365virtualmarathon.com for Event details

The M365 Virtual Marathon features experts panels on topics from Microsoft Teams, Power Platform in multiple regions around the globe. Over 280 sessions are being delivered by over 100 Microsoft MVPs, MCTs & RDs, in addition to 150 experts and leaders from the community.  You’ve heard many announcements from Microsoft from Ignite, the community experts will expand and deep dive on product announcements, as well as provide industry verticals including education and healthcare.   We hope you find the event offerings allow you to engage on your terms and attend the session type that interests you most, or catch up and connect with other attendees.

Check out the schedule and line up of speakers on our event page or on SPSEvents.org

The M365 Virtual Marathon begins April 26, 2021, at 09:00 PDT and is 60-hours, ending on April 28, 2021, at 09:00 PDT.  We sorted through over 450 sessions to pick the top 280 sessions from over 250 speakers the Americas, Asia-Pacific, Europe, the Middle East, and Africa.  The event has sessions presented in seven languages, including French, German, Spanish, Portuguese, Japanese, Arabic, and English.  This event has sessions running for more than 2 days straight, so you’re sure to find a time to check out the incredible line up of sessions.

This event is provided for free. We hope to record the sessions and provide them, but please give us time to provide to process and upload what will be 280 hours or 16,800 minutes of sessions. The volunteers on the community spent over 100 hours splicing, downloading then uploading the content last year on our M365VM youtube channel. Thanks for liking and subscribing!

We’re excited to join you in a few weeks for another great event, BY THE COMMUNITY FOR THE COMMUNITY!

For the Community by the Community

Register using the links below and if you would like to volunteer to be a session moderator, please sign up using the Moderator Sign Up form below. The schedule will go LIVE with join links in the session details on SPSevents.org and on m365virtualmarathon.com on 4/26/21 at 9am Pacific, but we encourage you to register so we can be better prepared to serve you.

>>Click Here to Register for M365 Virtual Marathon 2021 and Join the Global Community for this Event<<

>>Click Here to Sign Up to be a Volunteer Session Moderator

Teams Governance: 6 Keys to What Really Matters

Microsoft Teams Governance

What really matters in a Microsoft Teams Deployment? Can you just turn it on and see what happens? What if you were to figure it out along the way? What is Teams Sprawl and why do they refer to your Teams environment as the Wild West? I like to call it herding cats. There’s no reason to herd cats if you can keep your arms around your environment. In fact you can accomplish both good governance and user satisfaction at the same time.

  1. Focused Teams Adoption and Champions Plan – End user confusion is not the thing to start with, but most Teams deployments will likely not be greenfield. There will be some previous environment even if you’re replacing Slack, Skype for Business, or moving data from SharePoint to be used in Teams. Start with a clear migration path, and a very strong community of champions in a pilot environment, DO NOT add confusion by staying in a persistent “islands mode” deployment where users are sent back and forth from Skype and Teams. Adoption is one of the most important keys to a successful environment with Teams. Users need a clear path on what to use when on both Collaboration, File sharing, Voice, and Apps.
  2. Streamlined Provisioning & Lifecycle – Teams provisioning can be simple or a nightmare. You want a good balance of easy to obtain with light approval or to borrow from Nintex a “lazy” approval process where Teams can be auto or quickly approved. You absolutely need to avoid duplication and sprawl. On the back end when someone is done with a Team how does it get archived? What if it is needed later? You need to have a good Teams lifecycle strategy.
  3. Classification & Unified Labeling Strategy with Hubs and Information Architecture – A trusted, structured and easy to use environment starts with good labels such as all external access teams being put into a Hub marked for external. Likewise internal HR Teams should be “Hubified” together into HR to allow search, design, and consistency. From both a management and compliance perspective keeping “like” things together will allow for improved auditing and compliance. It helps both IT and the Users in maintaining a clean, well managed and easy to use environment. Labeling for compliance will be much much easier when setup properly with AIP, DLP, policies, templates, retention and sensitivity labels. Information policies and further security policies can be enforced through good A good structure will support a “Unified Labeling” across Microsoft 365 driven by good governance and Information architecture in Teams.
  4. Manage Sharing for External Files & Guest Access – A well managed environment makes it easy to track external file sharing in the right places. As well managing external guests either through white labeled domains or a process for managing and expiring both links and users. Ensure you have good processes and procedures for either automation or operations and reporting.
  5. Quality Assurance for User Performance with Analytics and User Insights – You need to look beyond out of the box reports for ensuring Network and Call Quality. Early in the deployment you will see optimization that can be done. Don’t be quick to blame Microsoft as 90% of outages are often from Telco’s and network infrastructure that can be optimized. Even user device configuration can be optimized to ensure good performance. In the beginning this means focusing on the bottom 10% and then the bottom 1%, but do NOT ignore the user experience.
  6. Teams as a Platform Application Management including full Power Platform Environments for Dev/Test/Staging (preproduction) and Production – While many would look at the policies in Teams and say it’s about either enabling or disabling apps for governance. I say Teams as a Platform is the goal. It is your job. If you’ve done governance and deployment well, you will find that a solid Teams deployment is one that embraces the Teams as a Platform mantra and looks to see how it can be matured to support “Environments.” Mature Teams environments that support business unit application development in pre production is a healthy way to support the growth of Teams inside your organization. Once your Teams becomes the hub of the company you’ll see how this is way more than deployment, but an investment in your company and where work gets done.
Microsoft Teams Governance
6 Keys to Successful Teams Governance

Come join me for 60 minutes on Teams Governance with Q&A. I’ll be diving into each of these 6 Keys to Successful Governance.

When: Wed, Apr 14, 2021 10:00 AM – 11:00 AM PDT / 1PM EDT

Don’t miss the chance to attend the Microsoft Teams governance workshop by Joel Oleson, MVP
Attend this webinar to understand the key elements for streamlining governance using Teams and other Microsoft 365 workloads without compromising on end-user autonomy and IT control.

Please join me, Joel Oleson, MVP, Regional Director, and former Microsoft IT Architect in this free 1 hour webinar on Teams Governance hosted by TeamsHub by Cyclotron for insights and solutions to increase collaboration, productivity, innovation, and adoption for your Microsoft 365 investments.

>> Register now! <<