Got into the office this morning and immediately started the scramble because of reports from several users that Microsoft Office365 TEAMS was not working (a key communication app for us and many other businesses).

Microsoft officially had no outages reported when I looked ~8:50 AM EST. So I think this is very fresh. Teams is currently only working on mobile devices for us. If you look at DownDetector.com (here: https://downdetector.com/status/teams/) the chart is telling. 0 reports of issues until around 8:20 AM they start trickling in, 7k+ reported issues by 9:30 AM.

Looks like Microsoft isn’t having a great start to the week. Back to phone calls and emails for now… If you can swing TEAMS on your mobile device though, thankfully that still works.

Side note, the WEB Application is unfortunately ALSO not working in our testing.

Confirmed from another news source… Microsoft IS having issues this morning:
https://www.onmsft.com/news/microsoft-teams-is-down-this-morning-the-company-is-investigating

Microsoft’s Twitter Feed for Office365 status can be seen here:
https://twitter.com/msft365status

If you have access to your office365 Admin portal, you can also see active issues here:
https://portal.office.com/adminportal/home#/servicehealth

Currently they show a TEAMS Service Degradation – “Can’t access Microsoft Teams” – reported/logged at 9:11 AM EST… the issue actually started around 8:20 AM EST based on reports in Down Detector.

Here is the explanation for the issue per what O365 Admins can see:

Current status: We’ve determined that an authentication certificate has expired causing users who have logged out and those that are still logged in to have issue using the service. We’re developing a fix to apply a new authentication certificate to the service which will remediate impact.

Scope of impact: This issue may potentially affect any of your users attempting to access Microsoft Teams.”


Auth certificate expiration… seriously 🙁

UPDATE/Correction:

The actual ticket states that the issue started at 8:15 AM EST. The ticket was LOGGED around 9:11 AM EST.

FIRST – I am stealing code here and re-sharing (with very little modification). All credit goes the fine gentleman that wrote these two articles, I would urge you to read them:

Bulk Add IP Access Restrictions to Azure App Service Using AZ Powershell

Bulk Add Cloudflares IPs to Azure App Service Access Restrictions Using AZ Powershell

I made a few minor modifications the provided code. First, I like to just run a lot of my Azure Powershell stuff from an ISE session and don’t like encapsulating everything in new commands. Partly because I am not all that familiar with working that way even though it is probably a MUCH better way of doing things.

Before we get to the code though, what is this for exactly?

If you use cloudflare as a protection and CDN layer for a website it works by acting as a reverse proxy for your site. I.E. client connects to your site by a cloudflare hosted DNS record… instead of connecting directly to your server, their connection terminates at Cloudflare, they do things, then the pass the connection along to your actual service. ‘Nuff said, google if you need more info.

In the case of an Azure Web App (or any other web server I supposed), the app is hosted/available on some public IP and/or azure domain name that azure provides when you create the app…

What this means is that someone can easily bypass your cloudflare layer (and the associated performance enhancements and protections like web application firewalls) if they know your source systems IP address and in the case of azure, your azure provided domain name for your app.

So what that means is that you need to setup an ACL (Access Control List) on your source system to say “Allow traffic from all Cloudflare IP ranges and block everyone else”.

Cloudflare has like 20 IP ranges… And setting up that ACL by hand on a web app in Azure is arduous at best. But that is why we have scripting… to make things that are generally a pain in the rear… NOT a pain in the rear. (more…)

I had a recent requirement from one of our clients that took a little bit of tinkering to figure out… we will call our client Contoso LLC. and our project that we host for them we will call the “Cool Widget Project.”

We built a really neat widget of an application for Contoso to use and we are hosting it under a sub-domain of a domain we control. We needed to keep hosting it under this domain. However, our client, Contoso, wanted to hand out a link for their users to the new widget we built using an existing sub-domain from a domain they control. This was of course under their main domain, constosollc.com, and they already had existing users that came to the old version of the widget (built by another vendor) at widget.contosollc.com.

Our company was hosting the new widget app at widgetapp.appworks.net.

To further complicate things… our company appreciates security, likes fast DNS updates, and the app really benefits from using a CDN… so we are using Cloudflare to manage DNS for the appworks.net domain. Better yet, we also like the cloud, and this new widgetapp is actually an Azure Web App.

So there’s the situation…

We essentially need this to happen:

User visits widget.contosollc.com --> widgetapp.appworks.net.

Oh but this is Azure… so actually widgetapp.appworks.net is already a CNAME record and it actually points to widgetapp.azurewebsites.net. So it is this:

User visits widget.contosollc.com --> widgetapp.appworks.net --> widgetapp.azurewebsites.net.

To elaborate the above just a little bit more:

widget.contosollc.com (DNS from random provider) --> widgetapp.appworks.net (Cloudflare DNS, CNAME) --> widgetapp.azurewebsites.net (the DNS name provided by Azure for the application)

Simple right? Just get our client to create a CNAME record that points to widgetapp.appworks.net and move on with life… wrong…
(more…)

Apparently a handful of customers using Cloudflare for DNS, and specifically CNAME records experienced a brief outage of name resolution services on New Year’s. I found the reason why to be rather interesting. Devs at cloudflare assumed time can’t move backwards… An understandable assumption but actually faulty because of leap seconds… Anyhow, if you do programming you might find the root cause analysis for this hiccup to be interesting and informative:

https://blog.cloudflare.com/how-and-why-the-leap-second-affected-cloudflare-dns/

Well worth a quick read. No, unfortunately it didn’t have anything to do with Dark Matter and/or what happens were a black hole and a Delorian traveling at 88 mph suddenly to meet while Superman flies around the planet at light speed. But, it is still curious enough all the same. Happy New Year… Sanitize your outputs…

Currently I am looking into a couple of different cloud platforms for new infrastructure projects. Microsoft Azure is creeping up rather highly on the list.

A few years ago the concepts of “security” and “cloud hosting” were diametrically opposed in many people’s minds. Security is an ironic field of IT in that technology, vulnerabilities and exploits, defense and remediation strategy, etc. all evolve very rapidly (like other areas of IT) but due to being tied in tightly with things like regulatory compliance the ideology and actual implementation of change in this area moves at a snail’s pace.

However IT is largely shifting towards cloud technologies and regulation must shift with it. The major players in the cloud hosting space have recognized a need to address security concerns and have made a concerted effort to do so.
(more…)