Error - Disks too slow reported but not the case...??
  • Hi Ben / knowledgeable operators

    In the last couple of releases of the software, I have started to get a glitch occur. I get the error window up and a massive list of these pouring in (in batches). Sometimes nothing for a few days, other times, non stop for a day. The thing is, the hardware hasn't changed and running a speedtest on the 6Gb SAS 12TB formatted RAID6 array (at the same time as the 5 cameras are recording to it) I am getting a consistent throughput of 570MB write and 630MB read. This was over many tests using BlackMagicDesign Disk Speed Test. All drives 'smart' report ok, array integrity is 100% and all other services on the server are working normally. Manually copying files onto the array and back is very fast. There is about 6% free space on the Array (as managed by SecuritySpy). The System Drive is a 6Gb SAS SSD with 60% Free, and 24GB RAM with plenty free. There are 2 x Quad Core Xeon CPU's humming along with about 80% idle for the most. Everything is CAT6 gig linked (some cameras are only 100Mb though) all routed through a Cisco 3750G-48. The Xserve is running 10.11.6 with Svr5.1.7 and has been for a couple of years now (10.9 for a few years prior to that then 10.6 prior to that). This issue only started about 3-4months ago. I an running SS 4.2.9. I have run SS for the last 6 or so years on the same core hardware, upgrading components and OS over time. My hardware/network has not changed in the last 3+ years bar the router/firewall about a year ago.

    The Error is:
    Error performing continuous capture for the camera "xx" continuous capture mode has been disarmed. Failed to record video frame 1556,98002. The disk is too slow and cannot keep up with writing data - you may meed to reformat or replace the disk.

    The odd thing is it is the same frame# and will list all the different cameras in the 'xx' section.

    Any thoughts on this? Would it be helpful to send over any files/logs?

    Many thanks
    Oli
  • Hi Oli,

    This message does indicate a real problem. SecuritySpy has a large memory buffer of data that is used to buffer writes to the drive and smooth out temporary drive slow-downs, but when this gets full because the drive can't keep up with the amount of data that is being attempted to be written to it, there is nothing SecuritySpy can do but to stop recording and throw this error. This happens for all cameras at the same time, presumably because they are writing to the same drive.

    You could try our own file writing speed test app that simulates the typical kind of file writing that SecuritySpy performs - it would be interesting to get the results from this test.

    One possibility is that you have one drive that is slowly going bad, and it performs well most of the time until it gets to one of its bad sectors, and then slows down dramatically.

    Also, check for things like Time Machine backups and Spotlight indexing, both of which can significantly slow down drives. You should add the drive to the Exclude list in the Time Machine system preference as well as the Privacy section of the Spotlight system preference.

    In any case, I am very confident that this is not a bug in SecuritySpy, but does actually represent a real issue.
  • I am getting this same exact issue. Same exact error and frame number as the original post.
    Is there any insight to this?
    It is not a write speed issue since my write speeds are 3500mb
  • Hi @CVH, as far as we know, this error does always indicate a write speed issue; there is no known bug in SecuritySpy related to this. Could you please try our File Writing Test app and let me know what it reports for your write speed?
  • Thanks for the response. I tested with the File Writing Test app and here are the results:
    File Count: 16
    Test complete.
    Time taken: 16.54 seconds
    Amount of data written: 5130 MB
    Average data rate: 310.21 MB/s

    The disk is an SSD Evo 2tb connected through usb3.

    I have 9 HD cameras, I have limited the bitrate and turned down the frame rate to 12fps from 15fps with the same result.

    I am still getting this error on a daily basis to all 9 cameras within the same second, although there is no issue with the records.

    This is the error:
    03/02/2019 18:50:28: Error performing continuous capture for the camera "XXX", continuous capture mode has been disarmed. 4.2.9,1556,98002 Failed to record video frame. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.
  • I too get that error occasionally, by which I mean it may happen multiple times a night, then nothing for a few days.
    SS writes to an external drive connected over USB 3.0 and I have found that it happens when my Mac is doing something pretty processor intensive, be that decompressing multiple large files, or transferring across my other external drives. My interpretation of it is even though System info isn't showing full processor usage across all the cores the iMac is still having problems doing everything at the same time and this manifests itself as SS not being able to keep up with writing the video files - there is enough network bandwidth for it to do so but the processor cannot do everything it is being asked to do so prioritises somethings over others.

    My interpretation is probably wrong but I have never had this issue when my Mac isn't 'busy', and can almost force it to happen. I am not sure what the Mac is doing sometimes, particularly when it happens in the night.
  • My system is at 10-15% consistently. It’s only purpose is SS. New Mac mini.
    I reformatted the drive and have set to drive to save 70gb before deleting files. Maybe it was an issue where it was trying to write and delete old files simultaneously.
    I will report back when the drive fills up in a little over a week.
  • I've been plagued by the 'disk is too slow and cannot keeps up with writing data" for a long time now. Late-2013 Mac Pro, 31 cameras, Drobo 5D for storage (Thunderbolt 2).

    Ran the File Writing Test app for 31 cameras and only get 36MB/s. The Drobo is full of 7200 RPM drives, configured for dual-drive failure survivability. I should be getting at least double that, shouldn't I?

    Liking Drobos less and less the more I use them. I think allocating some space on my Synology NAS (RS1912+) might be faster, even over ethernet.
  • I have six days on the drive, another six before it is full. No errors so far.
    Seems that when the files are being overwrites or deleted is when the error occurs.
    Since the speeds of the drive, this should not be an issue. I’m stumped.
  • It is certainly possible that when the auto-delete task runs, this is slowing down the drive sufficiently to cause write problems. We have attempted to mitigate this effect by throttling the delete process, but perhaps this isn't working well enough.

    As a test, could you all try turning off the auto-delete options and see if this prevents the errors? You may like to first clear sufficient space from the drive so as not to run out of space.
  • Thanks for the response Ben.
    So far no errors. I will have to wait another 5 days of record before the auto delete process takes place.
    I have a 2tb SSD. I had the delete process to take place with 40gb space left.
    I have several cameras that record daily files of 60gb.
    Maybe that is the issue. I increased the delete process to start when 80gb space left now.
  • This is useful information. So it could be either the simple fact of low disk space, or it could be the activity of the automatic file deletion. Please let me know what happens when you hit the 80 GB threshold and the auto-deletion task kicks in again. If it turns out that this is the problem, we can throttle it back some more. 80 GB should be enough free space to avoid significant slowdowns purely from the fullness of the drive.
  • I am getting the errors once the deleting process starts. When the drive has enough space to record, there are no errors.
    There are actually no interruptions in the records, the only reason I notice the errors are from the daily reports.
  • Thanks for the feedback. I have made some changes to make the file deletion routines more efficient and less stressful on the disk, and have posted a new beta version of SecuritySpy with these improvements (version number 4.2.10b13). Please install this one and let me know how you get on.
  • Anyone currently testing this, please grab this updated beta version of SecuritySpy (4.2.10b14) as it has some further improvements in this regard.
  • Thanks Ben, although this version seems to be ignoring the deletion process and is running the drive down to 0mb. I have the preferences set for 60gb.
  • Just noticed that I was running b13. Just installed b14 and it corrected the issue.
  • OK great. Any sign of the "disk too slow" error?
  • Yes same error.
    I really don’t think the disk is too slow, I have results of +300MB/s when I use the speedtest software you recommended.
    The error is for all the 9 cameras, all for the same second, yet I don’t see a frameloss in the records.
    There is usually 1 error for each camera per day.
  • Does the error occur at roughly the same time of day, or is it random?

    Have you excluded this disk for Spotlight indexing? (System Preferences -> Spotlight -> Privacy).

    It’s strange that the error is so sporadic - if it were due to SecuritySpy’s auto-deletion routines it should happen more often, as this runs every few minutes.

    300 MB/s is definitely fast, but there must be something that is causing a temporary short-term slowdown that is reducing this speed drastically, resulting in this error.

    When the error happens, the cameras will be temporarily disarmed, so there will be some loss of recording for the short time that the cameras are unarmed. They will then automatically re-arm based on the current schedules set for them. So while there is some loss, it should be very short.
  • It is random. I’ll do some investigating over the next several days. I did not have spotlight indexing excluded, I do now.
  • b14 hasn't fixed the issue for me either. :-(
  • @jlbrown - I suspect the cause of your problem is simply a low disk, as you are only getting 36MB/s from the file writing test software. Your setup should be much faster than this in theory, so there must be something wrong somewhere. Perhaps one of the drives is going bad? With a drive this slow, this can only be fixed by improving the drive speeds or reducing your recording rates; it won't be fixed with a SecuritySpy update.
  • I updated my macmini to 10.14.5 from High Sieraa. I also have the latest b7 installed. I have not had an error in 7 days.
    Jlbrown, are you on High Sierra?
  • I have started to see this issue as well - just went from 5.0 to 5.01. 13 Cameras going to an 8Tb external disk attached via USB3 - all are motion triggered. I have been slowly replacing old NTSC cameras with 3-5Mp IP cameras, so data rates are going up.

    I will check on spotlight and time machine. All cameras are on a 7/24 schedule - once marked offline, they don't seem to come back. Was there a timestamp in the alert box that pops up? I don't recall seeing one.
  • Hi @finkej try our File Writing Test with SecuritySpy closed and let me know what you get.

    As for the errors, the log file contains time stamps (File menu -> Open Log).

    You can also check disk pressure with the new Dashboard feature (available from the Window menu) - 100% disk pressure means that all disk writing buffers are full and this error will be generated. In normal usage with a fast disk, the pressure should remain well under 10%. What does your disk pressure graph look like?
  • @CVH - running on Mojave 10.14.6.
    Ben - didn't know about the Dashboard and Disk Pressure. Mine is usually 0%, occasionally goes as high as 5%. When I was the 'disk too slow' errors yesterday, disk pressure was 1%.

    File Writing Test gives 52.96 MB/s.
  • 53 MB/s is a moderate speed, but should be adequate for up to a dozen cameras or so. Check the Dashboard again but this time turn off smoothing - this allows you to see the peaks better. Do you see it spike up to a high value?
  • Hey Ben, I was just cruising through recent posts and saw this one. I never got around to e-mailing support but I'll add my "me too" to this post here. I get this error every day around midnight each day and all cameras temporarily disarm. Disk pressure is almost always at 0% and the File Writing Speed Test was 96.18 MB/s @ 16 files. The storage drive for the cameras is a 5-drive Synology NAS and the only thing that happens nightly on the NAS is an off-site backup. I just manually ran this same backup task and nothing showed in the SS log at all. Not sure what's up, but it's just an irritating error window I deal with right now.

    Also, would you mind explaining smoothing? It drastically adjusts the visualized data. For example, if I view Disk Pressure at the time of the logged error with smoothing off I see a blip for a minute or so at 40% but if I move smoothing up, that same blip then reads 1%.

    Unrelated, in Dashboard when viewing packet loss, if I check my crappy wifi camera that constantly drops packets, it reports 650% packet loss. What on earth can that possibly mean? :-)

    I'm happy to troubleshoot if you've got any ideas. Please let me know if I should just make this a support case instead. Thanks for everything.
  • I will also add a "me too". For me, it is sporadic. dashboard can show 0 (or close to it) disk back pressure for hours before the problematic spike, and then there is a spike to 90% which corresponds to the error log. Recording data rate (smoothed) is under 664 KB/s, peaking to about 830 KB/s (2 hours earlier). CPU usage stays below 36% total, which is the peak at about the same timeframe (otherwise seems to stay below 26%). If I look at Activity Monitor, I do see a rather large spike in writes at possibly the same timeframe, so perhaps it is related to other processes simultaneously accessing the disk at that moment. I do have Backblaze backups running, and bztransmit is shown as the 3rd largest "Bytes written", but I presumed most of those were network writes incorrectly showing up under the Disk tab (similarly, launchd is shown as the highest writer at 2.47 TB, and SecuritySpy at 409GB, thus I presumed there were network writes incorrectly categorized). bztransmit does do a lot of disk reads however, which would definitely slow down writes (bztransmit shows the highest bytes read at 288GB).

    I don't seem to know how to post a snapshot of the dashboard to the forum...
  • There is a common theme here, which is a disk that is generally fast enough to keep up with capturing, but with short periods where the speed is dropping drastically, causing SecuritySpy's file writing buffers to fill up and this error to be generated. The short-term slow performance could be caused by many things (e.g. an automatic backup, another disk-intensive application, or the system clearing purgeable space).

    So what I've done is to double the size of the disk writing buffer, in the latest beta version of SecuritySpy (currently 5.0.2b8). Please can you all test this and let me know if this reduces the incidence of this error or not.

    However, I'm wary about increasing the disk buffer too much, as this could potentially use a lot of RAM.

    In general, for best performance, the disk should be:

    - An SSD or fast HDD (NOT a fusion drive)
    - Not the system drive
    - Connected via a fast connection (Internal, Thunderbolt, USB 3)
    - Not used for other purposes
  • @billie - Backblaze can write a lot of data to disk in the form of temporary compressed files that are queued for upload. You can control where it creates these files using the "Temporary data drive" setting in the Backblaze settings - make sure this is not set to the drive that SecuritySpy is using for capture.

    @cstout - smoothing applies a moving average filter to the graph for its display, which has the effect of smoothing out extreme peaks and troughs. A higher smoothing setting is more useful for seeing the overall trend of the data over time, whereas a low smoothing setting is required to see the actual values of these short-term spikes.

    As for the 650% packet loss - this should only happen if you are summing the packet loss from multiple cameras, is this the case, or are you viewing the packet loss from just one camera? If the latter then this is definitely a bug! Please could you email us a screenshot of this to demonstrate. Thanks.
  • @ben - yes, temporary data drive is set to Macintosh HD, which is a 2 TB internal SSD. Backblaze is constantly running, as I don't think it is possible to catch up due to my relatively low upload bandwidth (only 20Mbps). So I think the first option is to simply disable Backblaze from backing up my video capture volume (a Drobo 8D connected via Thunderbolt 3). I had previously disabled uploading the *.unf files, but should have disabled the entire volume once I added 4 x 4K cameras (Lorex LNB9232S). I also have the TimeMachine volume on the same Drobo. The capture rate seems to be about 100GB/day/camera, and I have 4 more cameras on back order (Lorex LNE8964AB). Ideally, the continuous and/or motion captures could go directly to an SSD, and then once closed, be moved (versus copied) to the larger capacity drive for long term storage (30 days or so). Storage would need about 24TB for 30 days, which is what I allocated to the volume. I do see "Upload to" options for each camera, and saw that local transfers were permitted (according to manual), but it is unclear if I can still use SecuritySpy to browse/view such files that have been moved (particularly if I've deleted them from the original destination, perhaps by setting the "Delete old files by age" to 1 day). Perhaps the real solution is to find some Mac OS software to automatically handle a tiered storage arrangement, with new files written to the SSD, and those files migrated automatically behind the scenes to disk, but still presenting a full view of the files to user space...
  • @ben it is also curious that everybody reports the same frame number (1556,98002 ) in their error log. is there an incorrect format specifier? I've disabled Backblaze from backing up the capture volume and we'll see how it goes. I also am running an "fs_usage | gzip > fs_usage.gz" in the background to a different device (the SSD). If I get the error again, perhaps I will be able to correlate it with specific process activity. I suspect it is backblaze or spotlight. I've also added the capture volume to the spotlight privacy tab as a pre-emptive strike...
  • @billie yeah, all of my errors report 1556,98002 in the log.

    @ben this is all I have for the last two days of log entries. As far as the packet loss graph goes, yeah, it was only for the one camera. I'll send over a screenshot.

    09/02/2019 00:12:54: Error performing continuous capture for the camera "SW Driveway", continuous capture mode has been disarmed. 5.0.1,1556,98002 Failed to record video frame. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.

    09/02/2019 00:12:54: Error performing continuous capture for the camera "Back Yard Side", continuous capture mode has been disarmed. 5.0.1,1556,98002 Failed to record video frame. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.

    09/02/2019 00:12:54: Error performing continuous capture for the camera "Garage", continuous capture mode has been disarmed. 5.0.1,1556,98002 Failed to record video frame. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.

    09/02/2019 00:15:03: Error performing continuous capture for the camera "Back Yard", continuous capture mode has been disarmed. 5.0.1,1556,98002 Failed to record video frame. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.

    09/02/2019 00:15:03: There was an error while recording audio data for camera "Front Door". 5.0.1,7460,98002 Failed to record audio chunk. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.

    09/02/2019 00:15:06: Error performing continuous capture for the camera "Front Door", continuous capture mode has been disarmed. 5.0.1,1556,98002 Failed to record video frame. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.

    09/03/2019 00:13:54: Error performing continuous capture for the camera "SW Driveway", continuous capture mode has been disarmed. 5.0.1,1556,98002 Failed to record video frame. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.

    09/03/2019 00:13:54: There was an error while recording audio data for camera "Garage". 5.0.1,7460,98002 Failed to record audio chunk. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.

    09/03/2019 00:13:54: Error performing continuous capture for the camera "Garage", continuous capture mode has been disarmed. 5.0.1,1556,98002 Failed to record video frame. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.

    09/03/2019 00:13:54: Error performing continuous capture for the camera "Back Yard Side", continuous capture mode has been disarmed. 5.0.1,1556,98002 Failed to record video frame. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.

    09/03/2019 00:13:54: There was an error while recording audio data for camera "Back Yard". 5.0.1,7460,98002 Failed to record audio chunk. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.

    09/03/2019 00:13:55: Error performing continuous capture for the camera "Front Door", continuous capture mode has been disarmed. 5.0.1,1556,98002 Failed to record video frame. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.

    09/03/2019 00:17:01: Error communicating with the network device "SW Driveway". 5.0.1,10,835 Excessive packet loss from network device, the network may be too slow or defective, or this computer may be overloaded. Check the network and/or reduce this camera's frame rate.
  • With both BackBlaze and Spotlight disabled on the capture volume, the problem has not recurred for me. Incoming data rate (sum) peaks at 10490 KB/s; Recording data rate (sum) peaks at 11520 KB/s; disk pressure (sum) 0; memory pressure 0; cpu usage (total) peaks at 32%; still have 4 more 4K cameras to add (still on backorder...)
  • Well it happened again. But this time, I did capture some additional data via fs_usage. I'm not sure how to interpret it though. I'll try to send some info via email.
  • Please both make sure you are using the latest beta version of SecuritySpy (currently 5.0.2b9) as this has an enlarged disk write buffer. @cstout your log entries indicate that you are still using v5.0.1.
  • :-) I looked right over that comment in your post. Just loaded up the beta. I'll let you know if I get my nightly messages.
  • For my own setup, which uses 8 x 16TB Seagate Ironwolf Pro NAS drives within a Drobo 8D (direct connect via TB3), I discovered that disk I/O to the Drobo would simply stop (independent of SecuritySpy). For example, music playing off a volume on the Drobo would simply pause for several seconds. The pause correlates to the error messages sent by SecuritySpy. Drobo support indicates that some of my drives have an unexpectedly high "busy" time, i.e. disk operations are taking longer than they should. I've moved the capture streams off the Drobo and onto a faster (but much much smaller) SSD drive while I investigate the Seagate performance issue. The Drobo itself was showing write throughput over 100MB/s at times (while running some other applications), so I know it is capable of the sub 20MB/s rate that SecuritySpy would be sending.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!