5.2.3 no more AI classify events
Earlier this week I upgraded (presumably from 5.2.2, I usually keep things fairly up to date) to 5.2.3. I believe this is the version where the AI trigger was separated (i.e. the thresholds for motion capture and actions used to be one pair of sliders, now it's two). Every since the update, I have not captured one motion capture video on any of the cameras.
I have also updated from 5.2.3 to the 5.2.4b7 hoping this might clear it up, but it's still behaving the same way.
I thought at first it was just something with the arming or overrides, but this does not appear to be the case. I have 6 cameras and they are configured as 12 cameras; the high quality streams have motion capture/actions which are triggered from the 6 low resolution streams which do the actual motion detection and trigger their respective high res camera). I have a wget session running which just watches the event stream, and I'm watching camera 9 (the low res motion detect stream for my "garage front" camera). Usually I'll see MOTION events followed by CLASSIFY events. I see the MOTION events now, but there aren't ever any CLASSIFY events. The AI classification settings are set to 75% for human and 85% for vehicle, which were the settings for a long time (and were working reasonably well).
The full motion capture settings for the MD camera are as follows:
Video Motion Detection: checked
Trigger time: 1.0s
Mask: (the same mask I've always had)
AI MC trigger: Human 75% Vehicle 85%
AI Action trigger: Human 75 Vehicle 85%
Audio Level: not checked / 0 (these cameras do not have audio)
Camera Input Port 0: not checked
Camera Input Port 1: not checked
I'm happy to email the output of the ++systemInfo if it will help.
edit: if I use the API to trigger an MD event (++triggermd&cameraNum=X) of the high-res camera, I get the actions I configured (email with pictures). If I trigger an MD event on the low-res camera, I still get the correct response from the high-res camera, so the "chaining" is working as expected. It definitely looks like it's something in the "detecting motion" part.
I have also updated from 5.2.3 to the 5.2.4b7 hoping this might clear it up, but it's still behaving the same way.
I thought at first it was just something with the arming or overrides, but this does not appear to be the case. I have 6 cameras and they are configured as 12 cameras; the high quality streams have motion capture/actions which are triggered from the 6 low resolution streams which do the actual motion detection and trigger their respective high res camera). I have a wget session running which just watches the event stream, and I'm watching camera 9 (the low res motion detect stream for my "garage front" camera). Usually I'll see MOTION events followed by CLASSIFY events. I see the MOTION events now, but there aren't ever any CLASSIFY events. The AI classification settings are set to 75% for human and 85% for vehicle, which were the settings for a long time (and were working reasonably well).
The full motion capture settings for the MD camera are as follows:
Video Motion Detection: checked
Trigger time: 1.0s
Mask: (the same mask I've always had)
AI MC trigger: Human 75% Vehicle 85%
AI Action trigger: Human 75 Vehicle 85%
Audio Level: not checked / 0 (these cameras do not have audio)
Camera Input Port 0: not checked
Camera Input Port 1: not checked
I'm happy to email the output of the ++systemInfo if it will help.
edit: if I use the API to trigger an MD event (++triggermd&cameraNum=X) of the high-res camera, I get the actions I configured (email with pictures). If I trigger an MD event on the low-res camera, I still get the correct response from the high-res camera, so the "chaining" is working as expected. It definitely looks like it's something in the "detecting motion" part.
Comments
20200628190639 3827 14 MOTION 216 0 88 62
20200628190639 3828 14 MOTION 216 0 88 62
20200628190639 3829 14 MOTION 216 0 88 62
20200628190639 3830 14 MOTION 0 0 0 0
20200628190640 3831 14 MOTION 116 0 234 68
20200628190640 3832 14 MOTION 128 0 224 78
20200628190640 3833 14 CLASSIFY HUMAN 30VEHICLE 32
another time, *did* see me as human, *says* it did trigger (again, on the wrong camera) and did not create any motion capture video:
20200628190325 3752 14 MOTION 10 0 88 48
20200628190325 3753 14 MOTION 14 0 88 48
20200628190325 3754 14 MOTION 14 0 88 50
20200628190325 3755 14 MOTION 20 0 156 260
20200628190326 3756 14 MOTION 14 0 162 258
20200628190326 3757 14 CLASSIFY HUMAN 90VEHICLE 1
20200628190326 3758 14 TRIGGER_M 128
20200628190326 3759 14 TRIGGER_A 128
20200628190326 3760 14 MOTION 48 0 144 282
- The thresholds for Motion Capture vs. Actions have always been separate; there is no change here that could cause this problem.
- The "MOTION 0 0 0 0" event in the stream is perplexing. Basically it shouldn't be happening, we are looking into this (though this shouldn't prevent recording).
- For Camera A to trigger recording on Camera B, the Actions mode of Camera A must be armed, and the Motion Capture mode of Camera B must be armed. Is this the case?
- Under your setup, the AI must be turned OFF for the high-res cameras (because you aren't doing motion detection on them). Is this the case?
- Too look under the hood at the AI predictions, an "AI Predictions" folder must be created in the ~/SecuritySpy folder, as described in this blog post: Optimising SecuritySpy’s AI Object Detection (the location used to be the Desktop folder, but was changed to the SecuritySpy folder, so apologies for any confusion).
- The event streams look reasonable (apart from the "MOTION 0 0 0 0" thing). You should not expect a 1:1 ratio of CLASSIFY:MOTION events, because the AI invocation is throttled to maximum 1 classification per camera per second, due to its high resource usage.
- Your other forum post mentioned that the low-res camera instances are being decoded in software. Does this meant that the high-res instances are being decoded in hardware? If so, then I would question the need to duplicate the cameras and implement the complicated setup with half the cameras triggering the other half. You may actually get better performance by removing all the low-res streams, and simply using the high-res streams for everything.
If the above doesn't help, then perhaps we can set up a TeamViewer session so that we can log onto your system and run some tests? Your setup is rather unusual, and this may result in you seeing problem that other users aren't seeing, and that we are unable to reproduce here, so directly testing with your system may yield some valuable results.
I'll get a bunch of screen shots, grab the debug logs and send an email over to support@; if you need more info I'll happily set up a Teams or Skype or whatever works best for you.
I'll report back shortly on the "MOTION 0 0 0 0" bugfix. Gotta get some more caffeine in my bloodstream first. :-)
I can also confirm now (and will send three videos shortly) that something broke in 5.2.3 and continues to be broken in 5.2.4b8. I ran 5.2.2b5 (the most recent version until I downloaded 5.2.3) and triggers and motion work perfectly. Quit, run 5.2.3, no triggers. Quit, run 5.2.4b8, no triggers. Go back to 5.2.2b5 and it's working great again. No configuration changes between the application version runs.
Meanwhile, I had an idea:
In order to do motion detection, SecuritySpy doesn't look at all pixels in the image, as this would be inefficient - instead, it scales down the image before processing, so it only has to process a limited number of pixels. SecuritySpy 5.2.3 uses a larger scaled-down image for greater detection accuracy, but a side effect of this is that there is a now a larger minimum size of image that can be processed. I suspect that your low-res feeds (the ones that aren't working) might now be falling below this minimum size. The minimum size used to be 192x96, but is now 288x288.
So, what is the size of your low-res feeds? Show the "Video Size" column in the Camera Info window to inspect this.
In the new 5.2.4b9 beta version, we've made some modifications that allowed us to reduce the minimum size to 96x96, while keeping the higher accuracy motion detection implemented in 5.2.3. Does this resolve the problem?
If so, I would advise that the size of these feeds is so low that motion detection accuracy will suffer, and you will absolutely not get good results from the AI (because the AI works on only the subset of the image where motion is detected, which in your case will be truly tiny - in fact we may disable the AI altogether for such low-res feeds in the future, as the bad results obtained from such would lead to a bad user experience and therefore reflect badly on the software).
I would suggest that it would be far better to scrap the small feeds, and work from the full-res feeds only. If this is consuming too much CPU initially, reduce the frame rates of the cameras until this is under control. Note that the i7-950 CPU in your Mac has 8 virtual cores, and the CPU usage shown in Activity Monitor is per-virtual-core. So this would have to go up to 600-700% (as the sum of all processes) to start to be a problem. I would expect your Mac to easily be able to handle your 6x 2 MP feeds at a decent frame rate.
I'll give the 5.2.4b9 a shot but I suspect you've nailed the problem. I also completely understand what you're saying about the reduced resolution streams now becoming practically useless for motion detection (and in particular, AI MD). I'll go back to decoding the larger streams. I know the i7-950 can handle it (CPU load is pegged at just over 100% which you're right, is just a single core) -- it was just rubbing me the wrong way and has no logical reason behind it. I have a half a mind to take one of my larger FPGA dev boards and see if I can implement an h.264 decoder core and maybe collaborate on an AI engine. :-)
Thank you again for all of your assistance. This is by far one of the best-supported software applications I've had the pleasure of working with!
SecuritySpy's configuration has a framerate text field in the main settings (IP/auth/etc.) for each camera, and another framerate text field under the motion capture video settings; Do I set the main one down to 20/15/10fps to try to reduce the motion detection CPU load, and will I be able to set the motion capture video one back up to 30fps to get a higher fps video?
For most IP cameras, you must reduce their frame rate via their own settings pages (connect to them using a web browser and locate the video settings).
The green screens are concerning, and indicate a decoding issue. This could be overloading or it could be something else. Does reducing the frame rates help with this? What make/models are these cameras (apologies if you've told me this before).
I like your idea of implementing a hardware H.264 decoder board - that would be a stiff technical challenge! You mentioned that you're running a Hackintosh - any chance instead of upgrading the CPU to a newer Intel Core model that supports H.264 hardware decoding?
Backing down again to 15fps helps a little more, but the green screens still persist. CPU load is 165% for SecuritySpy and 176% for VTDecoderXPCService, and the Camera Info window shows between 20-40% per camera.
The cameras are generic Hikivision cameras bought off of Aliexpress (https://www.aliexpress.com/item/x/32766575257.html); pretty standard PoE 2MP cameras with crappy IE-only ActiveX configuration web interfaces. An example configuration for the cameras is here: https://imgur.com/a/IxPDEHN. The only thing I don't understand with these cameras is the "static configuration of" selection. I have baseline, main, and high as options, but none of them seems to make much of a difference, either in bitrate nor quality.
Last minute edit: I forgot I had the ~/SecuritySpy/AI Predictions folder; the SecuritySpy process' CPU use has dropped down to ~20-30%. I'll let this run for a while and see if I can bump things back up to 20fps. :-)
I'll keep an eye on this over the next little while but it'll be awesome if this did resolve the issue!