Why Would a Processor Slow Down Under Load?

A little over a month ago, when Steve and I did the live show, he noticed that the video coming from me was out of sync with my voice. During that show, it only got worse the longer we went on. Eventually my video started doing what we affectionately call the Max Headroom effect. That name comes from a TV show in 1984 in which the main character was a digitally sampled video head that jerked around and the voice was weird too. I put a link in the shownotes to a short video in case you can’t picture what we mean.

This failure of my Mac to be able to stream usable video has turned into a technical rat hole, unlike anything I’ve ever “enjoyed” before. If you’ve noticed my contributions to the show as being on the lighter side lately, it’s because this problem has consumed my life.

I could jump to the punch line and simply state what’s wrong with my Mac, but the interesting part is the journey. Steve and I are both engineers with master’s degrees so we’re well trained in the concept of doing controlled experiments to isolate problems, and yet it took us well over a month to finally figure this out.

In order to explain the problem, I need to explain some parts of how we create the live show. If I leave out too much, you’ll not realize how complex of a problem this was to solve, but if I tell you too much your brains will explode. Wish me luck on finding the happy medium!

Live Show Hardware

The hardware I use to create the live show is high end but not particuarly complicated. I have a 2016 15″ MacBook Pro hooked up to a CalDigit TS3+ Thunderbolt dock. My Mac gets power, USB, Ethernet, video and audio all from this dock.

I have an LG 5K display connected via Thunderbolt to the dock, along with a Logitech c920 webcam and a set of headphones. My microphone is a Heil PR40, which is an XLR mic. The XLR connector is that round one with three giant pins you may have seen before. The important thing for this discussion is that you can’t plug an XLR mic directly into a Mac, you have to use some sort of interface. I use a Shure Mvi interface that takes the XLR input and converts the audio with a preamp and sends the audio out over USB. I plug the Shure Mvi into my CalDigit dock and that audio input is also sent over the single Thunderbolt 3 cable to my Mac.

Ok, so we’ve got audio in, audio out, video in, video out and power all going to my Mac via the CalDigit Thunderbolt 3 dock. If you’re confused, I did include a visual diagram in the shownotes.

Live Show Software

I wish the software was as easy to explain as the hardware. I have a giant, annoying three page document that explains in text and visually all of the software along with the hardware but I can’t even get Steve to study it so I don’t expect any of you to look at it. I put it together mostly so that I can refer to it if anything gets messed up.

But let’s see if we can talk through just a couple of the pieces that made this entire adventure so fascinating.

Audio

I use Hindenburg as my multi-track audio recording software. In order for the live audience to hear me talking and hear playback of any pre-recorded segments from Hindenburg, I need a way to route both of those sources.

Loopback and Audio Hijack, both from Rogue Amoeba do this job. With these two tools, I can create virtual sources and route my audio wherever I want. they do more than this but for this discussion, that’s enough detail. I create a virtual source I call YouTube Live Input that includes me and Hindenburg playback.

Video

Video is more complicated. A few years ago we started using mimoLive from Boinx. Oliver Breidenbach is the CEO and a stellar guy and worked with us a lot on what happened.

We switched to mimoLive for two reasons. In the previous incarnations of the live show, I used my laptop for 100% of the tools to create the video and audio streams. Not only was my laptop overly taxed, but it was also a lot of balls for me to juggle – creating the audio content you hear in the real podcast, but also juggling all of this extra video and audio production stuff.

mimoLive is a professional video production application that we now run on Steve’s iMac, which allows him to do video switching between sources, control the audio sources, and create the stream to YouTube Live. With him running mimoLive, that frees me up to simply record the podcast and stop from time to time to chat with the live audience.

Even with him running mimoLive, I need to send my video and audio to him. mimoLive can create what’s called a mimoCall. This is a URL that I go to in Chrome (doesn’t work in Safari) which has controls much like any web-conferencing tool where I choose my camera and microphone input.

Steve likes to show my recording software Hindenburg while I’m actually recording (he covers up his own face video). In order to send that to mimoLive I use a free piece of software called NDI Scan Converter from Newtek. NDI stands for Network Device Interface, and it’s a royalty-free software standard from NewTek for sending live video streams. It’s super light – I just launch the app, point it at Hindenburg and now Steve can receive that stream over our internal network as an input to his mimoLive session.

In the Chrome window for mimoCall, I can see everything Steve’s sending to the YouTube Live stream.

Fault Isolation

Whew – In around 1000 words, we’ve got the hardware and software defined for the live show and a brief explanation of the problem. To recap the problem to be solved, suddenly my video started stuttering, getting delayed by multiple seconds and having serious sync issues. This is what we call the Max Headroom effect.

And this is where we actually start trying to isolate the root cause.

Tools to Measure the Problem — Intel Power Gadget + iStat Menus

We needed a way to measure what was happening and we looked first at my CPU utilization using iStat Menus from Bjango. I could see graphs that showed my CPU was completely hammered. Chrome was the biggest hog, taking up more than 350% where my max is 400% since I have four cores in my system. I’m recording audio, streaming webcam video, and desktop video, right?

In Energy Saver in System Preferences, the default is to have automatic graphics switching turned on, which means under some conditions your Mac will use the lower-powered integrated graphics instead of the more power-hungry discrete graphics processor. I made sure to turn that off, and then started watching the graphics card graph in iStat Menus and found it also was nearly hitting 100%. Good data point, but there’s not much I can do about it.

I had installed a free tool called Intel Power Gadget that allows me to see a running graph of the performance of my Intel processor. It also puts the frequency (speed) of the processor and core temperature inside iStat Menus which is awesome. Intel Power Gadget wasn’t going to solve the problem but it was the best way for me to test changes to the entire system to see what was causing all of these problems.

Now let’s have a quick lesson on thermal management of processors in computers. They are designed for self-preservation. If you ask a lot of your processor, it will heat up and could destroy itself. Processors are designed to avoid this, so if they get too hot, they actually slow themselves down. The processor in my Mac is a 2.67 GHz that can turbo boost when needed up to 3.6 GHz. But it can also slow itself down if it detects the processor is getting too hot.

With the Intel Power Gadget running, I can see the frequency (speed) of the processor and the temperature at the same time. If my processor is working properly, the more load I throw at it, the faster it should go, up until it starts to overheat at which time it should slow itself down.

Now here’s what’s fascinating about my problem. As soon as I start demanding a lot of my processor, it slows down instead of speeding up, all while the temperature is still low at around 60 deg C. It’s not that the processor speed degrades over a long time under high load, it slows down almost immediately, well before the processor can even get a chance to heat up.

I can watch Intel Power Gadget idling around 2.5-3 GHz, then launch Chrome with my video, and the processor speed will plummet in less than a minute to as low as 1.00 GHz. At anything much below 2 GHz, my video starts to stutter and I get that huge lag. Max Headroom is in the house.

Note that the MacBook Pro is sitting on a riser with plenty of airflow under it and sitting on a heat dissipation gel pad. It doesn’t have time to get hot but just in case I give it every chance to stay cool.

mimoLive – nope

Our first thought was that Boinx changed something in mimoLive as it sends the mimoCall from Chrome, but we worked with Oliver and his brother/developer Achim to roll back to previous versions of mimoLive that were working for us a couple months ago and the problem was still there.

I also tested using a Google Hangout (no mimoCall involved) to grab my video and Max Headroom effect was in full force. So it’s not mimoLive.

Chrome – maybe

Perhaps something changed in Chrome. The frustrating thing here is that on a Mac you can’t roll back the version of Chrome you’re running (you can on a PC) and you can’t use any other browser to run mimoCall. The Internets suggested a misbehaving plugin. I don’t have many running, but I disabled them all and yet the problem continued.

I uninstalled Chrome and reinstalled from scratch; same problem.

Hindenburg – nope

I had really high hopes that maybe my recording software Hindenburg was the root cause. Not because I don’t like it, but because the developers are so very responsive. I knew that they’d give me a rolled back version in a heartbeat to test and if it was the root cause they’d fix it. I swapped in Amadeus Pro, the application I used to use for audio recording, but sadly that didn’t fix the problem.

macOS

Would you believe I even reinstalled macOS to try to fix this? Now I didn’t do a clean install but rather used the Recovery Partition to reinstall over the existing OS and that didn’t fix it either.

Maybe Hardware?

With software eliminated, we got to thinking maybe there was a hardware problem. I removed my CalDigit TS3+ Thunderbolt 3 dock entirely from the equation and the problem persisted. I replaced the Thunderbolt 3 cable from my 5K display to my Mac, same problem. I should mention that the fun part was that throughout these tests, something would appear to change the behavior but when I tried to repeat the same tests they would be inconclusive.

I thought maybe it was my Logitech c920 webcam causing the problem. I have an internal iSight camera on my Mac and a camera in my LG display but using both of those with the rest of the setup brought Max Headroom back to life.

Steve tore down his setup to let me test using his 27″ Apple Cinema Display instead of the 5K display. Same problem. Then we got really drastic. I dusted off my 2013 15″ MacBook Pro and installed all of my live show apps and configurations, and used that with Steve’s Apple Cinema Display … and the problem persisted.

You can see why we went back to thinking this was a software problem.

mimoLive instead of Chrome

I explained up front that Steve uses mimoLive to receive and assemble the video and audio on his end and pipe it to YouTube live. But Oliver from Boinx explained that we could put mimoLive on my Mac, and use that to send my video to Steve. That would actually eliminate Chrome from the whole plot.

I installed mimoLive and configured it to send my video to Steve and things got better, but then Max Headroom came back to play. My processor slowed down and even flatlined to 1.0 GHz.

I then went back to the 2013 MacBook Pro and using mimoLive everything works perfectly. I can actually run the live show on the trusty six-year-old machine!

One more thing I didn’t check…

Late this week, Oliver asked if we’d do a Skype session so I could show him the problem. I’m not sure why he was so willing to help since we’d clearly proven that mimoLive was not to blame. I think it’s a combination of the fact that he’s one of the nicest people you’d ever hope to meet combined with his simple fascination with the problem.

We illustrated what was happening and he was just as confused as we were why a processor would slow down when more load was thrown at it. While I had my 2016 MacBook Pro hooked up to the dock and all that and running the tests to show Oliver the problem, I opened my 2013 MacBook Pro. I had no hardware hooked up to it, and none of the original software.

I watched the Intel Power Gadget while I launched Chrome with mimoCall as a video input and watched as the processor in this old Mac climbed faster and faster. I launched Hindenburg and it took another jump up in speed just as it should! I put screenshots of both Macs in the shownotes so you can see how the new Mac with my full setup does not respond properly to high system load.

As we were talking to Oliver, I said, “the only thing I haven’t changed is my microphone! And then I thought, “I haven’t changed my microphone!” I thought back to one experiment I didn’t mention early on. At one point I unplugged every single bit of hardware from my Mac and ran a test … and Max headroom was nowhere to be found.

I’ve been withholding a detail from you. At some time in the not too distant past, I was listening to Dave Hamilton on the Mac Geek Gab telling John F Braun that you never want to plug a microphone into a USB hub. I had been having some problems while recording video tutorials for ScreenCastsOnline where I was getting little clicks. I thought maybe it’s because I’ve got my mic interface plugged into my dock. Now a dock is not a hub so it shouldn’t have caused a problem but I moved my Shure MVi to be directly connected to my Mac using a USB dongle.

So now with all of this info and all of these graphs and all of these tools, I unplugged my mic and its Shure Mvi interface and I had zero stutter and my processor stayed at a reasonable speed! This is only true if I do not run Chrome, but use mimoLive to broadcast video.

The 15″ MacBook Pro has four Thunderbolt 3 ports, two on each side. I had put my mic interface into the left side via USB-C, along with the Thunderbolt cable coming from the dock for everything else. I moved the mic over to the right side instead and my CPU speed stayed up high even with video streaming.

I’ve been running experiments on this problem for quite some time and so many times I’d get a conclusive result, but I’d been unable to replicate the results. Not to be fooled again, I swapped my mic back and forth from left to right and back and 100% of the time if it was on the left, the CPU would drop, and on the right it would go back up. So maybe it’s not the mic after all, maybe it’s the Thunderbolt 3 controller on the left that’s failing.

I sent Oliver a note declaring victory and went to sleep.

And then today I reran the tests … and now I cannot replicate my success of yesterday. It doesn’t matter which side I plug my mic interface into, the CPU speed drops like a stone if I’m sending video. I am so very tired of this problem!

Bottom Line

I skipped a couple of important details that I’ll tell you about now. This Mac has a failing battery, and I have a ticket opened with Apple on the issue and plan on taking it in for replacement a week from Monday. This Mac also has the butterfly keyboard double-letter failure.

Now these things may not seem related, but from talking to Bart I learned that the CPU in a laptop gets its power directly from the battery rather than the power supply (because it only needs 5 Watts) so maybe the processor is slowing down because it’s not getting enough power from the failing battery.

The good news is that I bought AppleCare and it doesn’t run out until November. It sounds to me like there’s a really good chance that the entire bottom case of my Mac will be replaced since just about everything from the keys down is falling apart as I speak.

The other good news is that I didn’t sell my 2013 MacBook Pro so the show will go on!