Skip to content

"Failed to record video frame 1556,98002" error occurs every few days

edited December 2019 in SecuritySpy
I have an odd error I get every few days that appears to be related to the writing of the continuous capture file to the external hard disk. The error reads:

"Error performing continuous capture for the camera "", continuous capture has been disarmed. (Failed to record video from 1556,98002 The disk is too slow and cannot keep up with wiring data - you may need to reformat or replace the disk)"

When this error occurs, it is always the *exact same frame number*, and usually lists about 5 or 6 of the 13 cameras having this error at the same time. But this error only occurs every few days...

I already tried doing a deep health scan of the disk which came back as OK. Disk is an external USB3-connected 2-day RAID0 setup by Western Digital.

Thanks in advance for any help! Please let me know what I can do to help diagnose...

Comments

  • From your description, it sounds like the drive is experiencing significant transient slowdowns every few days. When this happens, data backs up in SecuritySpy's RAM buffers, but this can only be tolerated so much before SecuritySpy will stop recording and throw this error. The error represents a real problem with the drive; it is not a problem with SecuritySpy or an erroneous message.

    We see this problem mainly with RAID setups, probably because of the extra layer of complexity involved in RAID vs standard single-drive setups. In terms of the error code itself, it just identifies this specific problem, it doesn't refer to anything like a frame number.

    After this happens, SecuritySpy will attempt to re-arm the cameras and continue recording, so hopefully the loss of footage isn't too significant.

    In terms of a solution, I would suggest that instead of a RAID0 setup, it's preferable to reconfigure this as two separate drives (each in their own enclosure, each connected to the Mac via USB 3), and split the cameras between these two drives. As well as better performance, this method will give you better resilience (if one drive fails you only lose data for half the cameras, whereas in the RAID0 setup, if one drive fails, you lose all the data for all cameras).
  • Thanks Ben for the detailed response, makes sense.

    When this error does happen, it is always for the continuous recording, and it is always for many cameras at the same time.

    For continuous recording, I have it set to record / save in 24-hour increments that always end at midnight. Could it be that all 13 cameras (and their 15 GB 24-hour video files each) all attempt to get saved to the USB drive at once, resulting in this issue? I don’t see any obvious way to stagger saves to 12:01 / 12:02 / 12:03 per camera for example to try and help minimize this.

    Or does SecuritySpy already sequentially try and stagger the writes one by one? I’m curious what is actually happening when midnight hits and it attempts to save these 13 massive files at all at once, and if there is a way the software could handle that temporary condition better to resolve the problem and be more resilient to hard drive slow downs? Thanks.
  • The video and audio data is saved to disk continuously, as it comes in from the cameras. As this is happening, the metadata (frame/packet sizes and timing information) is kept in memory. When the file is finished, this metadata is then written to the end of the file as a "movie resource". So there is a bit of extra data that needs to be written at the end, but not much, and this shouldn't be the thing that is causing this problem.

    Are you seeing this error happen at midnight only? If this is the case, then it could be some other disk-related task that happens at midnight (e.g. a backup task?).

    Continuous recording is the mode that puts most demands on the disk, and all cameras that write to the same disk share the same write queue, so it's not surprising that this problem is simultaneously affecting the cameras that are capturing continuously.

    To make SecuritySpy more resilient to this type of drive slowdown, I have increased the file writing queue size in the latest beta version of SecuritySpy (currently 5.2.1b4) so please install this version and let me know how you get on with it. This uses a larger memory buffer in order to reduce the sensitivity to temporary disk issues. There is only so far that we want to push this though - if the memory buffer were too large, this could potentially cause memory problems when the disk slows down.

    So, I would still recommend reconfiguring your drive setup as I outlined above.
  • Thanks! I've installed it and will report back on how the write errors change.

    But one thing I noticed relatively quickly is this latest 5.2.1b4 app seems to lock up / crash now when I disable a camera, apply preferences, and then go back to re-enable it and a apply preferences later. As soon as I attempt to apply the re-enable preferences, I just get a spinning ball and the server goes down. This is the first build where I have seen that issue as I often disable / enable cameras. Would you mind double checking if you can reproduce that issue? Thanks.
  • I'm afraid I cannot reproduce this issue. Please do the follow:

    - Reproduce the problem
    - Open Activity Monitor
    - Select SecuritySpy in the list (it should say “not responding”)
    - Click the gear icon at the top and select “Sample Process”
    - Save the resulting log and email it to us.

    This log should tell us exactly what is happening and hopefully how to fix it.
  • Thanks Ben, just sent you a log from the non-responding Security Spy. By the way, so far after 4 days of testing I haven't seen one of the "failed to record video frame" errors. Fingers crossed that your adjustment has added enough headroom to keep it solid!
  • Thanks for sending the log, this problem should now be fixed in the latest beta version of SecuritySpy (currently 5.2.1b7). Please confirm.

    Great to hear that the new drive arrangement is working well!
  • b7 fixed the hang! You guys are good. :) thanks!
  • Great to hear that!
  • Hi Ben, just to follow up on this issue: while your changes improved the situation and reduced the error occurrence, I was still receiving the write failure occasionally. So I bit the bullet and replaced the 2-drive USB3 external storage with a newer 1-drive version (of even larger capacity). I’ve been running that newer 1-drive setup alone for 2-3 weeks now without a single error.

    So your originally theory was correct: there was some issue with the 2-drive RAID0 enclosure storage causing I/O slowdown occasionally. Fingers crossed that things stay solid as I assume the better test will be when the hard drive is at full capacity and SS is constantly swapping out oldest data for newest data.
  • Many thanks for reporting back, and great to hear that the hardware change has resolved the problem, I'm sure this will be helpful for other users experiencing the same thing. Yes, when the drive is at its limit and SecuritySpy has to start deleting files it will be under more load, but this extra load will be minor and shouldn't cause any problems for a well-functioning drive over a fast USB 3 connection.
  • Hi Ben, just FYI, I received this same error again with the new USB3 hard drive today for the first time. Not sure what the cause is... SecuritySpy only uses about 17% of the CPU on this Mac.
  • I’ve just started seeing this error today, My drives are also set up in a RAID configuration, but they are internal drives in my Mac Pro. It got so bad in version 5.2.1 that it completely crashed the system. Once I got the program to close I could never reopen it. I had to do a full system restart before I could open it again. I went back to version 5.1.0 and I’m seeing the same message. But without the full system crash. Today has been the only time I’ve had this issue come up.
  • Hi @nckfrtg28 - the version of SecuritySpy won't help here - it's not a bug in SecuritySpy. It's a problem with the drive. When the can't cope with the data rate that SecuritySpy is attempting to write to it, SecuritySpy has no option other than to throw this error and (temporarily) stop recording. And if the drive is suffering severe problems, this can also cause system instability. It sounds to me that one of the drives in your RAID config has gone bad. Your RAID software should provide a healthcheck utility, and hopefully this will identify the bad drive, which you can replace.
  • Hi, Ben, I'm also receiving this error, but using a single WD Purple on a USB3 connection. I'm only saving two cameras and CPU usage is very low. The HDD is not empty, but it is at its limit in SecuritySpy.
  • Hi @john999, if you are getting this error with your external drive I would recommend swapping this out for a new drive. Hopefully your USB3 enclosure is easy to open to replace the drive.
  • Thanks Ben. What kind of write speeds should I be expecting on a WD Purple USB3? What do I need per camera to avoid issues? I just reformatted the disk and I'm seeing about 165 MB/s read & write.
  • For two cameras, your disk should easily be fast enough. That's why this error message indicates a more fundamental problem with the disk (e.g. bad sectors or some other degradation) rather than merely a general lack of speed. See how you go with the reformat - hopefully this will stop the problem, at least for a while. But keep an eye on it and if you see any further disk-related issues then replacing it would be the way to go.
Sign In or Register to comment.