"Failed to record video frame 1556,98002" error occurs every few days
I have an odd error I get every few days that appears to be related to the writing of the continuous capture file to the external hard disk. The error reads:
"Error performing continuous capture for the camera "", continuous capture has been disarmed. (Failed to record video from 1556,98002 The disk is too slow and cannot keep up with wiring data - you may need to reformat or replace the disk)"
When this error occurs, it is always the *exact same frame number*, and usually lists about 5 or 6 of the 13 cameras having this error at the same time. But this error only occurs every few days...
I already tried doing a deep health scan of the disk which came back as OK. Disk is an external USB3-connected 2-day RAID0 setup by Western Digital.
Thanks in advance for any help! Please let me know what I can do to help diagnose...
"Error performing continuous capture for the camera "", continuous capture has been disarmed. (Failed to record video from 1556,98002 The disk is too slow and cannot keep up with wiring data - you may need to reformat or replace the disk)"
When this error occurs, it is always the *exact same frame number*, and usually lists about 5 or 6 of the 13 cameras having this error at the same time. But this error only occurs every few days...
I already tried doing a deep health scan of the disk which came back as OK. Disk is an external USB3-connected 2-day RAID0 setup by Western Digital.
Thanks in advance for any help! Please let me know what I can do to help diagnose...
Comments
We see this problem mainly with RAID setups, probably because of the extra layer of complexity involved in RAID vs standard single-drive setups. In terms of the error code itself, it just identifies this specific problem, it doesn't refer to anything like a frame number.
After this happens, SecuritySpy will attempt to re-arm the cameras and continue recording, so hopefully the loss of footage isn't too significant.
In terms of a solution, I would suggest that instead of a RAID0 setup, it's preferable to reconfigure this as two separate drives (each in their own enclosure, each connected to the Mac via USB 3), and split the cameras between these two drives. As well as better performance, this method will give you better resilience (if one drive fails you only lose data for half the cameras, whereas in the RAID0 setup, if one drive fails, you lose all the data for all cameras).
When this error does happen, it is always for the continuous recording, and it is always for many cameras at the same time.
For continuous recording, I have it set to record / save in 24-hour increments that always end at midnight. Could it be that all 13 cameras (and their 15 GB 24-hour video files each) all attempt to get saved to the USB drive at once, resulting in this issue? I don’t see any obvious way to stagger saves to 12:01 / 12:02 / 12:03 per camera for example to try and help minimize this.
Or does SecuritySpy already sequentially try and stagger the writes one by one? I’m curious what is actually happening when midnight hits and it attempts to save these 13 massive files at all at once, and if there is a way the software could handle that temporary condition better to resolve the problem and be more resilient to hard drive slow downs? Thanks.
Are you seeing this error happen at midnight only? If this is the case, then it could be some other disk-related task that happens at midnight (e.g. a backup task?).
Continuous recording is the mode that puts most demands on the disk, and all cameras that write to the same disk share the same write queue, so it's not surprising that this problem is simultaneously affecting the cameras that are capturing continuously.
To make SecuritySpy more resilient to this type of drive slowdown, I have increased the file writing queue size in the latest beta version of SecuritySpy (currently 5.2.1b4) so please install this version and let me know how you get on with it. This uses a larger memory buffer in order to reduce the sensitivity to temporary disk issues. There is only so far that we want to push this though - if the memory buffer were too large, this could potentially cause memory problems when the disk slows down.
So, I would still recommend reconfiguring your drive setup as I outlined above.
But one thing I noticed relatively quickly is this latest 5.2.1b4 app seems to lock up / crash now when I disable a camera, apply preferences, and then go back to re-enable it and a apply preferences later. As soon as I attempt to apply the re-enable preferences, I just get a spinning ball and the server goes down. This is the first build where I have seen that issue as I often disable / enable cameras. Would you mind double checking if you can reproduce that issue? Thanks.
- Reproduce the problem
- Open Activity Monitor
- Select SecuritySpy in the list (it should say “not responding”)
- Click the gear icon at the top and select “Sample Process”
- Save the resulting log and email it to us.
This log should tell us exactly what is happening and hopefully how to fix it.
Great to hear that the new drive arrangement is working well!
So your originally theory was correct: there was some issue with the 2-drive RAID0 enclosure storage causing I/O slowdown occasionally. Fingers crossed that things stay solid as I assume the better test will be when the hard drive is at full capacity and SS is constantly swapping out oldest data for newest data.