V5 Slow Web Server
Hello,
I looked in the discussions and saw no other reports of this, but I assume most people don't hit the web interface as much as I do.
This request can take up to 5 seconds sometimes.
time curl "http://127.0.0.1:8000/++image?cameraNum=1&width=375&height=281&quality=1" > /dev/null
The quality, height and width options don't seem to change the response time much. Sometimes it's very fast at around half a second, but usually it's over a second.
CPU running around 40% - most of that decoding cameras to do motion detection.
If I close SS5 and open SS4 and repeat the request a few dozen times I might get 1 or 2 that take over a second. Is this fixable?
Thanks!
-captain
PS. This just gave me the idea to stick response times into a time series database and graph them.
I looked in the discussions and saw no other reports of this, but I assume most people don't hit the web interface as much as I do.
This request can take up to 5 seconds sometimes.
time curl "http://127.0.0.1:8000/++image?cameraNum=1&width=375&height=281&quality=1" > /dev/null
The quality, height and width options don't seem to change the response time much. Sometimes it's very fast at around half a second, but usually it's over a second.
CPU running around 40% - most of that decoding cameras to do motion detection.
If I close SS5 and open SS4 and repeat the request a few dozen times I might get 1 or 2 that take over a second. Is this fixable?
Thanks!
-captain
PS. This just gave me the idea to stick response times into a time series database and graph them.
Comments
Two possible causes:
1. If the incoming frame rate of the camera itself is low, then this will increase the time that SecuritySpy has to wait for the next frame before it can be delivered via the web server. What is the frame rate at which the camera is sending video to SecuritySpy?
2. If you have checked the "Decompress incoming video frames only when required" option in the General preferences, this can significantly increase response times, as then SecuritySpy has to wait for the next I-frame in the stream before it can satisfy the web request, which may take several seconds. Do you have this option checked?
2. It was checked. Unchecking that box seems to have made it worse. Definitely didn't get any better.
I'm going to play around with fewer cameras enabled. My GPU (MSI GeForce GT 710 2GB) runs about 50% GPU usage and 75% memory usage. The CPU is 50-60% idle on average. I have 9 cameras enabled - one of them is a Logitech webcam, the other 8 are Dahua IP cameras 4-6MP.
Additional detail: when I receive an image using the web interface as shown before (except redirect to a file instead of null) any motion in the image is almost always distorted. The same distortions are not present in the captured/saved video files. Examples (I have thousands more....):
https://share.getcloudapp.com/mXum9KEp
https://share.getcloudapp.com/5zuJgNDd
https://share.getcloudapp.com/L1uknrzJ
https://share.getcloudapp.com/KouexJLB
This same distortion is now happening in SS4 too, so I believe the macOS upgrade is part of the culprit here. Thanks for the help!
It's a IPC-HFW4631F-ZSA which turns out to be a firmware hacked (and locked) chinese-version camera with firmware from 2017. It supports H264H and an incompatible H265. I suppose I'll be replacing it shortly.
EDIT:
I'm still getting video distortions.
...and, I just unchecked "Enable this camera" in the camera settings.
Try this: click on the SecuritySpy menu, and from the Debug submenu, check the option "Disable Hardware Accelerated Video Decoding". The quit and reopen SecuritySpy. How does this affect CPU usage, GPU usage, web response time, and the distortion issue?
https://share.getcloudapp.com/X6uDWGeJ
The video:
Would you be willing to allow us access to your Mac (e.g. via TeamViewer) so that I can connect and run some tests? I think this is going to be the best way forward. If this is OK, please email support@bensoftware.com with the TeamViewer details and let me know when would be a good time to connect.
I found an old POS system in my closet, slapped a couple drives in it and booted up unRAID. I then moved unifi controller, unifi-poller, nginx, grafana, telegraf, influxdb and the time machine server out of my security spy system and into the new unraid system as docker images.
The security spy system now runs: indigo, motifini, messages, little snitch, glances and backblaze. I'm soon going to remove the backblaze app too, and setup a daily cron job to move captured security spy files to the backup server on my network.
I'm still in a weird situation with the limited authentication abilities within the securityspy's web server. I have to run little snitch to create an inbound firewall to limit who/what can access the unauthenticated security spy APIs. Indigo is in the same boat, with very archaic and limited web auth configurations. Once these apps provide better authentication and authorization inputs I can remove little snitch.
Motifini will always be on my system, but I will probably write a pushover (or telegram) integration to remove and replace iMessages.app as my notifications platform.
I use glances to put system metrics (disk, cpu, ram, CS, intr, net) into influxdb (on a remote host). I'll probably put telegraf back because glances doesn't give me everything I need/want.
In the end, I believe InfluxDB and the unifi controller were the culprits. Grafana's image renderer plugin probably wasn't helping either. All of these things create many quick CPU spikes and lots of context switching and interrupts. While my CPU wasn't maxed out, I think the OS was.
I'm now running with 8 IP cameras enabled, motion detection enabled on 5 of them at about 60% idle CPU. The GPU runs at 40-30% idle.
I have no more distortions and the API always returns in under 1 second.
Thanks!
In terms of the authentication issues you mention, could you please outline the problem and maybe I can suggest a better solution than the use of Little Snitch?
01/01/2020 01:26:36: Error running neural network. 5.2,2060,15802 Vehicle classifier failed. CoreML prediction error.
01/01/2020 01:56:16: Error running neural network. 5.2,2060,15802 Vehicle classifier failed. CoreML prediction error.
Disabling human and vehicle detection has completely solved my problem and returned my system to normal behavior. Hope this tip helps someone else! Good luck!
If you go to the Debug submenu via the SecuritySpy menu, and select the "Run Classifier Test", do you see correct predictions, or errors reported there?
As for the classifier test: looks like it throws errors. See screenshot: https://share.getcloudapp.com/5zuJNKmm
I am running Jira, Confluence, SSpy and a mess of node on the same box, and had to jump it up. ( also many file handles for NAS storage and DB support )
Then installed SecuritySpy, added 2 cameras, and immediately saw the same video distortions on my Dahua speed dome. It's running at 40fps 1080p. I'm convinced there is nothing I can do to fix this, and the problem lies in the interactions between SecuritySpy and the newer MacOS. This system is literally doing nothing except running SS, and the video preview window watching a car drive by looks like soup.
As for 40fps, that's not even enough. I want to run it at 60 fps, but SS pretty much chokes up on anything north of 30. I've since set the camera to 30fps/4MP. It's better, but I'm still regularly seeing distortions on jpeg snapshots and in the camera display window(s) directly on the iMac. This is affecting several cameras, but doesn't ever seem like it gets them all at the same time. Perhaps 1 or 2 of them use the video card at a time, and it's part of the problem? We may find out..
This iMac runs at roughly 70% idle CPU and the video card hovers around 75% idle.