Large Scale Microsoft Teams Outage – Due to Expired Authentication Certificate…

nbeam published 2 weeks ago in Cloud Services, Microsoft, News, Office365. Tags: , ,

0

Got into the office this morning and immediately started the scramble because of reports from several users that Microsoft Office365 TEAMS was not working (a key communication app for us and many other businesses).

Microsoft officially had no outages reported when I looked ~8:50 AM EST. So I think this is very fresh. Teams is currently only working on mobile devices for us. If you look at DownDetector.com (here: https://downdetector.com/status/teams/) the chart is telling. 0 reports of issues until around 8:20 AM they start trickling in, 7k+ reported issues by 9:30 AM.

Looks like Microsoft isn’t having a great start to the week. Back to phone calls and emails for now… If you can swing TEAMS on your mobile device though, thankfully that still works.

Side note, the WEB Application is unfortunately ALSO not working in our testing.

Confirmed from another news source… Microsoft IS having issues this morning:
https://www.onmsft.com/news/microsoft-teams-is-down-this-morning-the-company-is-investigating

Microsoft’s Twitter Feed for Office365 status can be seen here:
https://twitter.com/msft365status

If you have access to your office365 Admin portal, you can also see active issues here:
https://portal.office.com/adminportal/home#/servicehealth

Currently they show a TEAMS Service Degradation – “Can’t access Microsoft Teams” – reported/logged at 9:11 AM EST… the issue actually started around 8:20 AM EST based on reports in Down Detector.

Here is the explanation for the issue per what O365 Admins can see:

Current status: We’ve determined that an authentication certificate has expired causing users who have logged out and those that are still logged in to have issue using the service. We’re developing a fix to apply a new authentication certificate to the service which will remediate impact.

Scope of impact: This issue may potentially affect any of your users attempting to access Microsoft Teams.”


Auth certificate expiration… seriously 🙁

UPDATE/Correction:

The actual ticket states that the issue started at 8:15 AM EST. The ticket was LOGGED around 9:11 AM EST.

Create Cloudflare ACL on Azure App Service

nbeam published 1 month ago in Azure, Cloud Security, Cloud Services, Firewalls, Information Security, Microsoft, Network Security, Powershell, Web Administration, Web Hosting. Tags: , , ,

2

FIRST – I am stealing code here and re-sharing (with very little modification). All credit goes the fine gentleman that wrote these two articles, I would urge you to read them:

Bulk Add IP Access Restrictions to Azure App Service Using AZ Powershell

Bulk Add Cloudflares IPs to Azure App Service Access Restrictions Using AZ Powershell

I made a few minor modifications the provided code. First, I like to just run a lot of my Azure Powershell stuff from an ISE session and don’t like encapsulating everything in new commands. Partly because I am not all that familiar with working that way even though it is probably a MUCH better way of doing things.

Before we get to the code though, what is this for exactly?

If you use cloudflare as a protection and CDN layer for a website it works by acting as a reverse proxy for your site. I.E. client connects to your site by a cloudflare hosted DNS record… instead of connecting directly to your server, their connection terminates at Cloudflare, they do things, then the pass the connection along to your actual service. ‘Nuff said, google if you need more info.

In the case of an Azure Web App (or any other web server I supposed), the app is hosted/available on some public IP and/or azure domain name that azure provides when you create the app…

What this means is that someone can easily bypass your cloudflare layer (and the associated performance enhancements and protections like web application firewalls) if they know your source systems IP address and in the case of azure, your azure provided domain name for your app.

So what that means is that you need to setup an ACL (Access Control List) on your source system to say “Allow traffic from all Cloudflare IP ranges and block everyone else”.

Cloudflare has like 20 IP ranges… And setting up that ACL by hand on a web app in Azure is arduous at best. But that is why we have scripting… to make things that are generally a pain in the rear… NOT a pain in the rear. (more…)

VSSAdmin Delete Shadows FAIL – DiskShadow to the Rescue

nbeam published 2 months ago in Microsoft, Server 2008 R2, Server 2012, Server 2012R2, Storage, Windows Administration. Tags: , , ,

2

I won’t go into great detail here but in short, if VSSAdmin is not letting you delete shadow copies and is throwing this message:

Error: Snapshots were found, but they were outside of your allowed context.  Try removing them with the backup application which created them.

There is another (less safe…) program called DiskShadow which will.

It’s built into windows server… and using it to get rid of pesky shadow copies that won’t otherwise goes away looks like this:

C:\DiskShadow
DISKSHADOW> Delete Shadows All

and away they go…

One of the best blog articles I came across that details both VSSADMIN and DiskShadow tool and why VSSAdmin sometimes falls short is here:
https://danblee.com/diskshadow-vssadmins-best-friend/

Cheers!

Remove Dead RDS Session Host Server – Otherwise You Can’t Manage Remote Desktop Services

nbeam published 1 year ago in Microsoft, Remote Desktop Services, Server 2012R2, Windows Administration. Tags: , , , , ,

6

After working with RDS (Remote Desktop Services, previously known as “Terminal Services”, also referred to “The biggest pain in the rear and the only way to get more than two remote desktop sessions on a server because Microsoft either hates admins, hates this product, or both”) I have come to the conclusion that Microsoft really needs to make something which should be simple, simple. Alas it isn’t simple. And today’s topic…

You have an RDS licensing and management server… you have several Session Host servers as part of your deployment. At some point one of those servers died and was henceforth removed from the domain. It’s gone, never to return. You login to manage Remote Desktop Services licensing… and are told that a bunch of servers are missing from the management pool.

Something along these lines:

The following servers in this deployment are not part of the server pool:
servername.domain.local
The servers must be added to the server pool.

Instead of MS just letting you manage RDS, you have to fix this first. Fine… you go to add them back in but wait…. you have that server which was a session host which died a horrible death and therefore cannot be added back in… What now?
(more…)

Missing your Windows 7 OEM Install Media? – Get it from here!

nbeam published 1 year ago in Consumer Tech, Microsoft, Windows 7, Windows Administration. Tags: , , , ,

0

Several years ago Microsoft provided Windows 7 operating system ISO file downloads easily through a site called Digital River. Legitimate, clean copies of all versions of Windows 7.

Then they stopped.

Why is this a problem?

I refurbish a lot of older desktops and laptops; often to give them away. Those systems usually have a legal OEM (original equipment manufacturer) license. OEM licenses are unique in that they are lower cost licenses provided to system vendors (Dell, HP, etc.) that are tied to the hardware (i.e. they can only be used on the box that they came pre-installed on).

OEM Licenses then are a real blessing, low cost Windows OS for the masses… However if you lose your system disk (or if the OS on your manufacturers media is pre-service pack 1 making it a pain in the rear once installed) and need to get a copy to reinstall on your computer things are no longer as easy as they used to be. Microsoft shut down Digital River and if you want to get install media you now have to enter a license key and have it checked before you can download. Fine… but not fine… if you enter an OEM key you get a really helpful message to contact the manufacturer… Great, if you call Dell, HP, Gateway, etc. they will often tell you that you are out of luck or, at best, charge you some ridiculous fee to ship you an install disk.

Let me reiterate that if you have an OEM key and are using it on the original system with which the key came then you have a legal right to use the operating system. You just need to find install media which is now a royal pain.

Thankfully a kind soul grabbed all of the original Digital River and Technet (another now deprecated option for getting install media) Windows 7 ISO files and provides them via Bittorrent. (more…)