Latest release - tons of false positives

ckurowic

EDIT: FYI - opened a ticket following the advice Ben provided another user.

Hey all. Been running SS for years. I love the new human / vehicle detection feature which seemed to work great in the last version. Now, with the latest update, I'm getting rather constant false positives. We're talking stuff like a car is identified a human, a shadow, my dog, things like that. I upped the threshold to 90+ on my cameras, and still getting the same deal. Restarts seem to have no effect. I've read about tuning things for detection, and inspected the files that it claims a human was detected..and even for a basic AI there's no way those items could even remotely be considered human-like.

Ben

I see that we're communicating via email about this, and I think that's the best place to help you with this issue. I think this can be improved by some settings tweaks. I'll follow up by email when we have some images from your "AI Predictions" folder.

ckurowic

Hi Ben - yes I've been in contact with support about this. I think it perhaps comes down to the accuracy of the AI system. As I shared with support and echoing here for the benefit of others is that 1) I've never had a false negative, and 2) given the raw amount of data (frames) that are sent to SS, if the accuracy of the AI is 98% for arguments sake then you're going to see 2 false positives out of every 100 events. That's just the way it is. The false positives I'm seeing with my dog are quite humorous. He'll stand on his hind legs and use his front legs to attempt to open the door appearing "human-like". Also, the other night we had a wind storm and unknown to us the gate was blown open. I got a human alert at the front door and there is doggo waiting patiently. So, all things considered I'll take those false positives and live with it. A developer I am not, but having dabbled in Swift and other programming languages I'm incredibly impressed with the package you and your team have put together.

Ben

Thanks for your efforts here and providing helpful information by email. It is indeed hard for the AI to distinguish between dogs and humans sometimes, especially apparently your dog, which is a large short-haired variety (who looks adorable by the way!). After all, dogs have many of the same features - eyes, nose, mouth, limbs.

This is exacerbated by situations that deviate significantly from the average. The AI is trained on hundreds of thousands of real CCTV images, but in these, dogs/animals are present in only a small minority (around 1%). Therefore, when training and optimising the AI, it doesn't have to distinguish accurately between humans and dogs in order to get a very high overall score (e.g. even if it classifies all dogs as humans, because the number of dog samples is so small, this doesn't hurt the overall accuracy too much and you can still end up with 95% accuracy overall).

There isn't really a great solution for this, as the dataset used for training should be representative of the frequency of images that the AI will encounter in production, which it currently is, on average. By including more dog samples in the dataset used to train the model, this may make it more accurate for you, but would make it less accurate for someone else who doesn't own a dog.

The ultimate solution is to do on-device training so that each user could further train the AI for their own purposes, but this isn't easy to implement, and would probably require a significant amount of user input to achieve. We are however taking a look at this possibility.

ostermann

Very interesting discussion, and exactly in line with my limited work with neural nets and other AI. Here's another training datapoint for you, though. The AI picks up the raccoon visitors on my back porch as "Human" most of the time. That's exactly the way I like it, though, because one of the reasons I have a camera back there is to let me know what they're up to in the middle of the night (because they can cause damage). It ignores birds and squirrels, though, which is fine with me. Apparently, your training dataset included some shots of Cruella Deville! Keep up the good work, the AI is extremely useful to folks like me!

ckurowic

No disagreement from me, ostermann. Though it depends on my anxiety levels with the area...I've had some bad experiences in this city and sometimes when the neighbor cat is detected as human and an alert is triggered gives me a bit of a fright. My problem, I know..but I'm hoping to see AI improvement as time goes on. What would be really neat is if we could eventually get alerts that say "animal detected" versus "human" or "vehicle".