VTDecoderXPCService CPU usage
  • The continuous and motion capture frame rate settings in Security Spy probably should be exactly the same as the frame rates set in the cameras. Otherwise, there has to be decompression and recompression to change the camera frame rate to the recording frame rate. Has that also been done?

    As I recall, the web browser interface works best if the output stream to the web browser is motion jpeg. That reduces the CPU load for compressing the outgoing video.
  • OK. They are. Connection issues remain in spite of making that correction.
  • By any chance are you recording to a network attached drive instead of a locally attached drive? I've also see some 20-30 second periods of Mac unresponsiveness when a network attached drive was busy. Switched to Thunderbolt connected RAID and no more spinning wait cursors. Just another thought to complete the picture.
  • Thunderbolt drive.
  • Hmmnnn. I might have to see what my Mac Mini does instead of my iMac. The iMac is doing great handling 14 cameras.
  • Hi - I have the same problem with resource use (or rather the VTDecoder battering my CPU). This was never a problem on earlier versions, and it has persisted through the last couple of betas and the current app.

    These are my cameras:
    https://www.dropbox.com/s/a5dumffxkdwt7zf/cams.png?dl=0

    The spec of the machine (which is only used to run SSpy, with a GPU-enable dongle):
    https://www.dropbox.com/s/pbqzec6j4vsykoe/spec.png?dl=0

    The load / temp of the CPU, showing a couple of SSpy restarts:
    https://www.dropbox.com/s/phdju8kunenxa7f/temps.png?dl=0

    And finally the resource usage:
    https://www.dropbox.com/s/fbidtu972uml0sl/VTdecoder.png?dl=0

    Any way ahead on this would be great. This machine was an appropriate spec for months (I have logs of load and temps to prove) but the off-loading of H.264 decode to the Apple process has seen a dramatic increase in duty cycle / thermal load which is really bad for the hardware.

    Previously a fan setting of 3k RPM constant was enough to keep everything cold. Occasionally it would spike. Now the main fan is pegged and the CPU bounces off 100 degrees!

    Help!

    PS > I am using a SSpy title / timestamp, which I know forces a recompression. I guess you'll suggest turning this off, but for me it's a critical feature.
    PPS > I have matched the frame rates with those set in the camera GUIs.
    PPPS > I am also using an external fan pack to force air over the Mini case to try and improve thermal scavenging. Probably helps a little, but obviously not enough.
  • After installing the latest Security Spy update (with cameras set for continuous recording), the system is now 90% idle and I’m not seeing VCD consuming any CPU, unless I open a window to view cameras! I see that the release notes mentioned something about connection issues with ONVIF cameras being resolved.

    I’ll experiment with turning off continuous and turning on motion detection and see what happens to the CPU load. Some people like the continuous recording and others like the motion detection (easier to figure out when an event was), so I’ll try both and then discuss which to keep active.
  • Interesting.

    My system has been sitting at 90% idle (continuous capture, all motion detection turned off, all cameras set to 10 FPS), with no VTD threads active (21 threads 0% CPU).

    Open a window to view one camera, CPU idle drops to 80%. Open a window to view all 5 cameras, CPU idle drops to 25% and fans kick in.

    Next test will be to turn motion detection on and see what happens.
  • I always have the "All Cameras" window open showing 6 cameras (usually, sometimes 7) and my idle remains at 90% (across all CPU cores) on average with VTDecoder using about 14-18% (of one core) on average with 11 Threads. 2 camera's on continuous 24/7, other camera's sometimes continuous recording on schedule, 2 always motion & action set, others on schedules.

    camguyfbr: do your cameras not have an option to embed the time directly on the camera? (4 of my 6 do)
  • Hi - my idle is around 30%. This looks like a runaway process spawning threads in VTDecoder to me. If I kill the process (even from terminal) it takes several hours to creep back up to maxing out the CPU / thermal throttling with max fans. Brian, I can run an OSD with date/time/name on my cams, but they are Samsung firmware and the labels are like something from a 1980s VCR. Plus the other cams I use (Hikvision) use a different format of OSD. Previously I was fine with the overhead of re-compressing because CPU usage was reasonable for H.264. Now I'm not so sure. I might try disabling all of the recompression and SSpy OSDs to see what difference it makes....
  • I’ve had a window open (separate screen on my desktop), showing one camera, all day and the CPU idle has slowly gone down from 80% to 55% with 27 threads.

    Shut down that window and the CPU has gone back to 70% idle. Not the 90% idle it was before I opened the window. The thread count hasn’t dropped back to the baseline 21 either.
  • May need to look into some other elements of the H.264 streams like Bit Rate which is an option on some/many cameras. Will try to remember to check what my cameras are set to tonight and report back. Other elements of streams include bit depth like 8 or 10 - I suspect this isn't an option on most cameras. Audio stream format - I've seen these elements cause issues with playback in different players on multiple platforms - one of them could be causing an issue with VTDecoder (if there is no packet loss that could also explain issues)
  • I've been experiencing occasional high VTDecoderXPCService CPU usage that lasts until I restart SecuritySpy (4.1.6), and it's very easy to reproduce consistently.

    The problem can be triggered simply by viewing captured files via the web interface or the SecuritySpy or Spyglass iOS apps.

    Normally, VTDecoderXPCService on my iMac sits at 10% CPU usage or less while SecuritySpy runs. Viewing captured files once causes it to increase to ~110% and stay there indefinitely. Viewing captured files a second time causes another permanent ~100% increase in CPU usage, and so on.

    Viewing captured files appears to trigger some kind of runaway decoding bug. Previous posts by the developer have claimed that since VTDecoderXPCService is an Apple service the bug must not be in SecuritySpy, but this actually seems to be a case of a SecuritySpy bug triggering excessive unnecessary video decoding, which manifests as high VTDecoderXPCService CPU usage.

    The bug is almost certainly in SecuritySpy, not VTDecoderXPCService.
  • Hi @yaypie - it is interesting that you are reporting that this is related to viewing captured video via the web server. As far as I'm aware, no one else has reported this.

    Are you viewing the "HQ" files, or the "LQ" files via the web server? Do you see the problem with either, or both?

    It certainly could be a SecuritySpy bug, but at this point it just doesn't seem that this is the case. We have checked SecuritySpy's code thoroughly for any problems related to this but can't find any. Furthermore, we can't reproduce the problem. It's only a handful of users who are reporting this, indicating it's specific to certain hardware/macOS combinations. The common factor here seems to be macOS 10.12, which indicates this is likely a bug in VideoToolbox (the system library responsible for decoding/encoding video) in this particular system version.

    However I am still open to all possibilities.
  • @Ben VTDecoderXPCService CPU usage increases as soon as I visit the "Captured Files" page by clicking either the "View As List" or "View As Grid" button in the web interface, and then remains high until I restart SecuritySpy. It's not necessary to actually view a video in order to trigger the bug.

    The issue also occurs when viewing the list of captured videos via the SecuritySpy or Spyglass iOS apps, which was reported previously in this thread by @donhoffman: http://www.bensoftware.com/forum/discussion/comment/6004#Comment_6004

    After trying various things to see what might have an effect on my ability to reproduce the issue, I discovered that it occurs consistently when the first video in the list is a particular file generated by my Hikvision DS-2CD2032-I. When I recorded a new video from that camera and it became the first file in the list or when I viewed a list in which the first video was from another camera, the bug stopped occurring. When I deleted the new video, returning the previous one to the first position, it began occurring again.

    The video file that causes the issue has a section at the beginning with dropped frames, but is otherwise unremarkable. This camera frequently (but not always) generates files with sections of dropped frames like this, so it may have been a coincidence that I've been able to consistently reproduce it.

    Here's the file in question, in case it's useful to you: https://www.dropbox.com/s/bi3g5alns9p2req/07-11-2017 20-37-54 M Front door.m4v?dl=0
  • I hear what you're saying Ben but I think I agree with @yaypie on this. Still looks like someone's bug causing a runaway process. Here are some more system usage screenshots from tonight.

    This is what the runaway VTDecoder looks like on my Mac Mini:
    https://www.dropbox.com/s/vhvf5k1kudoihq3/Processes.jpg?dl=0

    And this is the thread from Activity Monitor:
    https://www.dropbox.com/s/580th6yo8chibpu/thread.jpg?dl=0

    On this screenshot from iStat (remote monitoring) you can see the usage creeping up. I was connected to the app on my phone for some points of the evening, although I haven't tested exhaustively. Idle was 0:
    https://www.dropbox.com/s/2khekojzwrt19wj/CPU Load.jpg?dl=0

    Finally the effect: maxed out fans and a cooking CPU die:
    https://www.dropbox.com/s/ryphhodyv86o52n/CPU Temp.jpg?dl=0

    I've turned off all recompression, killed audio streams and carefully matched frame rates to no avail. If anyone has any ideas, I'd be interested. I'm half tempted to write a daily security spy restarter cron job.

    Note that killing VTDecoder works, even if you don't restart SSpy - but it's annoying to have to run a top -U to get the PID from a terminal session if you're administering without using the GUI.

    HELP!
  • ...And this is what a VTDecoder kill / restart does:
    https://www.dropbox.com/s/w6h38jadmpqjejj/killed.jpg?dl=0

    For good measure, here's the CPU die temp for the last 3mo. The number of cams and the hardware has been unchanged, apart from some tweaking in the last fortnight to try and protect my hardware. There is a clear correlation with the beta and subsequent new release of Security Spy:
    https://www.dropbox.com/s/nzl2xeinncfw01b/CPU Die 3mo.jpg?dl=0

    @Ben is there anything I can do to help fact-finding?
  • Anything? Anything at all?
    Another day of my CCTV machine cooking itself. Not good.
    If we have a lot longer like this I will need to stop using the app.
    @Ben - please can you provide an update?
    There's really nothing unusual about my setup, so I am mystified by this.
  • VTDecoder is a core OS feature - Apple is responsible.

    as for getting PID - you can do a grep after top - this shows just any VTDecoder processes
    top -l1 | grep 'VTDecoder'

    Download an earlier version if you don't need anything added to the newest releases of SecuritySpy and verify that behaviour goes back to normal (other possible causes - macOS has had updates in the same timeframe with 10.12.5 being released in May, and 10.12.4 a month or so before that)
  • Hi Brian. I'll try a version roll-back rather than halting use of the sw.

    Is there a version of 4.x that does not rely on VTDecoder? I'm not keen on going all the way back to 3.x. Thank you.
  • When you view a list of captured files via the web server, SecuritySpy delivers the small thumbnail images that you can see next to each file. This uses the encoder/decoder service, so this could be related somehow (especially if you click the button to load all thumbnails, which may result in thousands of these images having to be generated). We'll investigate whether there is some problem with this mechanism that could explain this issue.

    To answer your question @camguyfbr - VideoToolbox (which spawns the VTDecoder process) is really the only option for macOS apps doing video encoding/decoding. It is fast and powerful, and maintained well by Apple. It seems unlikely that it would have a bug that would cause this problem, however from all the testing we have done and the information we have about this, a bug in SecuritySpy also seems unlikely at this point. But the bug must reside either in VideoToolbox or SecuritySpy and we're keeping an open mind about this.
  • Thanks for the reply.

    I'm not sure if spindumps are helpful at this stage, but last night at about 0440 something happened and the machine went into toaster mode again. I've pulled the spindump from console.... Let me know if the whole log would help at all.

    I'm at a bit of a loss really - although there are bug-squashing efforts underway with several other video-decoding apps (like Chrome) which point to something going on in VTDecoder in some circumstances.

    Event:
    https://www.dropbox.com/s/guaq0j00s616lwf/0440_cooking.jpg?dl=0

    Spindump:
    https://www.dropbox.com/s/ngdo5bt74sp4ms3/spindump.jpg?dl=0

    I will be very peeved if the life of the machine is curtailed by this constant thermal throttling - it feels like it's bouncing off the redline.
  • sudo killall VTDecoderXPCService

    ...and repeat :-|
  • has this problem been resolved yet? I started having it also since I upgraded to version 4: after a few days CPU use goes to 100% and all I can do then is to close securityspy and open it again or, in some cases, I have to restart my Mac. I just upgraded to version 4.2.
  • As far as I have seen this VTDecoder using increasing levels of cpu usage over the course of a day is the same for all systems apart from my 2.3Ghz Mac mini 2012, i7 quad. It shows no usage of the VTD for the last few months, and I am running everything the same as it was when the VTD was using 300%+
    I still see the issue on all systems I install using 2017 iMacs though. So something weird is going on, not only with this VTD increasing issue, but why does the Mac mini show nothing.
    I set my SS to restart every day using automator calendar event, running 7 cameras 2 x 4mp / 5 x 2mp 24 hr recording + motion to USB drive, all running at 10fps.
    I've just completed another project using a brand new Mac 2017 i5, 3Ghz quad, running 13 cameras, all 2mp, 11 @ 5fps, 2 @ 10fps, and it shows VTD at 50% from the start of SS, and then by 24hrs later, cpu is @ 100%, and VTD @ 300+%. That restarts at 10am every day and sends the cpu usage back to 30-40%, and so it begins again!
  • Ok, I may be going mad here, but has anyone updated to High Sierra and noticed the loss of VTDecoder usage?
    I have just updated the new iMac 2017 model I installed on the last project and the cpu usage dropped from 50% rising to 300+% over a day down to 4% rising to only 17% over 12hrs!!
    WTF is that about?

    Have Apple made a big difference in the way video is sandboxed?
  • I had been considering upgrading my Mini to handle more cams because I was always at 100%, since High Sierra I’m consistently around 40%, and I’ve never seen 100% since upgrading. It’s great.
  • Having recently purchased SS and been playing with it on various different machines, I can report having the same issue on one of my iMacs.

    As part of my setup, I have two identical iMacs: 2009 21.5in models, both running the same ram and HDD. On one, in view only mode, there is no increase in CPU usage as time goes by. On another, with the registered version of SS (4 cam licence) with the same setup as the other usage starts at about 40%, then hits 150% within a couple of hours. Whereupon I have to restart SS.

    I wonder if there are any initial ideas to fix the issue?
  • Oh, this might be interesting. Reading all the above, this comment stuck out...

    "The problem can be triggered simply by viewing captured files via the web interface".

    The only difference between the two machines mentioned about is one can be viewed remotely, this is the machine with the registered version of the app, and I did initially think the CPU loading issues were being instigated only after I'd view captured files remotely. I ruled it out, for no reason that it seemed unlikely, but now reading the above comment, I'm wondering if there is anything in this?
  • Weirdly, after a reboot everything is now running perfectly. So no idea what caused the problem, but for now, it's away.

  • Sadly this issue is back, on the two machines. I'll post a new thread about it,
  • Found this VideoToolbox API talk from Apple WDC, Most of it is over my head. But on topic.
    https://developer.apple.com/videos/play/wwdc2014/513/
  • As an update the issue is back with a vengeance! Both machines are hitting max CPU load within n hour or two. I've tried everything I can think of... Rebooting, turning off http & https web feature (only possible on the one machine as the other is in demo mode), not logging into the web interface, closing the camera view when not in use, closing all other apps, and so on. I'm at a loss.

    On one of the machines before going to bed last night I rebooted and only opened SS, this morning VTDecoderXPCService CPU load was over 700%. Nothing was done with the machine other than SS in the interim.

    My only final thoughts to try is changing from 24hr motion detect segments to hourly. Back when I trailed SS on my main work machine I'm pretty sure this is how I ran it, and on that machine I never saw this issue.

    I'm hoping though someone can offer some help on my other thread with some sort of auto kill script for VTDecoderXPCService.

    I don't think though this is a SS issue as such, but something else that isn't playing ball. I just hope a workaround can be found.
  • Installed latest update und experienced this the first time VTDDec on 280%. Restarting mac hoping this goes away.
  • Reboot did not help. VTD goes up 300% Never got stuff like this from Axis Cameras which they are:

    Error from camera "Kurzwarten rechts", it will be closed. (Failed to decompress incoming video frame 22185,71042 Decompression failed)

    Error from camera "Kurzwarten", it will be closed. (Failed to decompress incoming video frame 22185,71042 Decompression failed)
  • "Error from camera "Kurzwarten rechts", it will be closed. (Failed to decompress incoming video frame 22185,71042 Decompression failed)"

    Yes, that's the error I get, afterwards, VTD goes mental.

    In terminal if you kill VTD, SS restarts it within a few secs, then everything seems to work fine again until that error message appears. It can go for hours sometimes, but eventually, it happens.

    I'm trying to work with SS support to somehow kill VTD if/when it's load gets too high, but I'm not getting very far: They're reluctant to help me kill a process that causes SS to stop recording. I do understand this, but at the moment it seems like the lesser of all evils, because if I don't kill VTD then it sooner or later it kills my system!

    Rock? Hard place? Humm!
  • What about downgrading SS this happened after the last update for me first time
  • Ben can we have older V4 Versions to download please?
  • This is a weird one because this is an issue that I have never seen - I run SS 24/7 for months at a time, 4 cameras, SS CPU usage is currently round 10% and VTD is 0%, all cameras are being recorded at 1080p/720p at 30fps.

    I am on a 2013 iMac, High Sierra, with the latest official SS.
  • Its possible to limit cpu usage for a specific app. This will only works with OS X 10.10 or higher. Installs to /usr/local without using sudo. Kinda Safer :)
    VTD is causing issue with other apps since the Sierra update. (Dropdox,Photo Agent,etc).
    This has been a solution for others with the same runaway VTD issue. Not sure about SS. Hack at your own risk! if your not comfortable with Command Line Tools. You will need to bone up...or possibly brick your OS or wait for Apple to patch it. This does not seem to be a issue with El Capitan what i'm running. Im going to wait on the upgrade HS till this gets hashed out.
    Homebrew package manager- https://brew.sh/ CPU Throttle- http://www.willnolan.com/cputhrottle/cputhrottle.html
  • Conversely, Meatsuit, the two machines I'm running SS on, both identical machines (licence 4cam on on, demo on the other), were both on Ec and both had the issue.

    I say were as yesterday I updated the licenced machine to HS, and so far, 24hrs on, I'm yet to see the VTD problem!

    As they say.... Go figure!

    I do agree with you though that this is not an SS problem per se.
  • We've had a few reports now that upgrading to High Sierra fixes this issue - can anyone else confirm this?
  • Since updating one machine to HS I've not seen this issue again. Will be updating the other tomorrow. Hopefully, I've seen the last of it :-)
  • So is this issue fixed? There was a lot of chatter before this last update. I was thinking about upgrading to High Sierra. Wanted to wait till this issue was put to bed.
    Thanks
  • I am using High Sierra and the upgrade has definitely NOT solved the problem. The problem may not be due to SS but we are all having the problem when using SS.
  • Here's where we're at with this:

    VideoToolbox (VT) is the Apple API that SecuritySpy uses to decode incoming compressed data from IP cameras. For each application using VT for video decompression, VT spawns one separate process called VTDecoderXPCService, which does the actual processing.

    The main reason for using a separate process in this way is so that if the decoder crashes, it doesn't bring down the process that is using the decoder (in this case SecuritySpy). Video data can come from a variety of sources and may be of variable quality and integrity, so there is always a chance that a chunk of bad data can crash the decoder. When this happens, VTDecoderXPCService is designed to silently restart and continue processing.

    Unfortunately, something appears to be going wrong with VTDecoderXPCService in certain unknown circumstances, whereby with the same throughput of frames being decoded, its CPU usage goes up uncontrollably over time. Fortunately it is not affecting many users, but for those affected, it can be a significant problem.

    We haven't been able to reproduce the problem ourselves, which is really hampering our ability to investigate this, but we have investigated the problem on a customer's machine, and we have confirmed that the number of frames that SecuritySpy is sending to VT to be decoded remains constant while the CPU usage of VTDecoderXPCService steadily increases. If SecuritySpy subsequently stops sending frames to VT, CPU usage remains high. Periodically reinitialising each camera's decompression sequence with VT also does not prevent the problem.

    The problem seems internal to VT, and unfortunately nothing we have tried thus far has been effective. It doesn't help that this problem is so difficult for us to reproduce, and reports we have received about which Mac hardware and macOS versions are affected are inconsistent. So we have not been able to put together a demonstration to send to Apple to ask them to fix it.

    At this point, it seems that the only effective workaround is for SecuritySpy to detect this situation and completely restart the whole VTDecoderXPCService process. This is not ideal, as there will be a temporary gap in decoding services, and some frames will be lost, but it shouldn't be too disruptive, and it shouldn't be required very often.

    This has been implemented in the 4.2.4b3 beta version of SecuritySpy. Please can you all try this and report back about how this beta performs. Please make sure you are specifically using version 4.2.4b3, as we may subsequently remove this workaround if we find that it has undesirable side effects.
  • Ben, I have three identical in every way iMacs: Mid 2009 etc (can get you the exact model if required).

    These are all setup the same in terms of hardware, and almost the same in terms of software. I have three of these as they are dirt cheap to pick up 2nd hand (c.£150-200) and are still plenty powerful enough to run most modern stuff, and they're still supported with OS updates (most likely HS is the last though).

    Anyway, I tested SS on all three, and on all three the VT problem arose. On my newer iMacs it didn't.

    I updated one to HS and it solved the problem. On another machine I did try the suggestion you made on another thread about adding some software to kill/reboot VTDecoder, instead of updating the OS, but I couldn't get it to work, so just updated it. Again, problem solved. So I updated the third and also problem solved.

    I mention this because if you want to pick up a machine on which it seems likely you'll experience this issue, this model might be a contender. If you want the model number let me know and I'll dig it out (Can't get it now as the machines are not local to me, but I'll be there at some point this week).

    :-)
  • I have now upgraded 6 Macs to HS with SS setups and all have been fixed, and the VT use has vanished and cpu idle is down very low and all, the newer Macs 2017 i5, 3.1Ghz quads run 10+ cameras all 24 hr recording at a ridiculously low 4-6% cpu usage.
  • Ben, I found this tread as I too am experiencing high CPU. I wanted to try 4.2.4b3 to try and see if killing the process and restarting it would help but the link above takes me to beta version is 4.2.4b5 which i am not sure has that feature. Do you have a link for the correct beta? Thanks.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!