Skip to content

Error - Disks too slow reported but not the case...??

edited February 2019 in SecuritySpy
Hi Ben / knowledgeable operators

In the last couple of releases of the software, I have started to get a glitch occur. I get the error window up and a massive list of these pouring in (in batches). Sometimes nothing for a few days, other times, non stop for a day. The thing is, the hardware hasn't changed and running a speedtest on the 6Gb SAS 12TB formatted RAID6 array (at the same time as the 5 cameras are recording to it) I am getting a consistent throughput of 570MB write and 630MB read. This was over many tests using BlackMagicDesign Disk Speed Test. All drives 'smart' report ok, array integrity is 100% and all other services on the server are working normally. Manually copying files onto the array and back is very fast. There is about 6% free space on the Array (as managed by SecuritySpy). The System Drive is a 6Gb SAS SSD with 60% Free, and 24GB RAM with plenty free. There are 2 x Quad Core Xeon CPU's humming along with about 80% idle for the most. Everything is CAT6 gig linked (some cameras are only 100Mb though) all routed through a Cisco 3750G-48. The Xserve is running 10.11.6 with Svr5.1.7 and has been for a couple of years now (10.9 for a few years prior to that then 10.6 prior to that). This issue only started about 3-4months ago. I an running SS 4.2.9. I have run SS for the last 6 or so years on the same core hardware, upgrading components and OS over time. My hardware/network has not changed in the last 3+ years bar the router/firewall about a year ago.

The Error is:
Error performing continuous capture for the camera "xx" continuous capture mode has been disarmed. Failed to record video frame 1556,98002. The disk is too slow and cannot keep up with writing data - you may meed to reformat or replace the disk.

The odd thing is it is the same frame# and will list all the different cameras in the 'xx' section.

Any thoughts on this? Would it be helpful to send over any files/logs?

Many thanks
Oli
«1

Comments

  • BenBen
    edited February 2019
    Hi Oli,

    This message does indicate a real problem. SecuritySpy has a large memory buffer of data that is used to buffer writes to the drive and smooth out temporary drive slow-downs, but when this gets full because the drive can't keep up with the amount of data that is being attempted to be written to it, there is nothing SecuritySpy can do but to stop recording and throw this error. This happens for all cameras at the same time, presumably because they are writing to the same drive.

    You could try our own file writing speed test app that simulates the typical kind of file writing that SecuritySpy performs - it would be interesting to get the results from this test.

    One possibility is that you have one drive that is slowly going bad, and it performs well most of the time until it gets to one of its bad sectors, and then slows down dramatically.

    Also, check for things like Time Machine backups and Spotlight indexing, both of which can significantly slow down drives. You should add the drive to the Exclude list in the Time Machine system preference as well as the Privacy section of the Spotlight system preference.

    In any case, I am very confident that this is not a bug in SecuritySpy, but does actually represent a real issue.
  • CVHCVH
    edited March 2019
    I am getting this same exact issue. Same exact error and frame number as the original post.
    Is there any insight to this?
    It is not a write speed issue since my write speeds are 3500mb
  • Hi @CVH, as far as we know, this error does always indicate a write speed issue; there is no known bug in SecuritySpy related to this. Could you please try our File Writing Test app and let me know what it reports for your write speed?
  • CVHCVH
    edited March 2019
    Thanks for the response. I tested with the File Writing Test app and here are the results:
    File Count: 16
    Test complete.
    Time taken: 16.54 seconds
    Amount of data written: 5130 MB
    Average data rate: 310.21 MB/s

    The disk is an SSD Evo 2tb connected through usb3.

    I have 9 HD cameras, I have limited the bitrate and turned down the frame rate to 12fps from 15fps with the same result.

    I am still getting this error on a daily basis to all 9 cameras within the same second, although there is no issue with the records.

    This is the error:
    03/02/2019 18:50:28: Error performing continuous capture for the camera "XXX", continuous capture mode has been disarmed. 4.2.9,1556,98002 Failed to record video frame. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.
  • I too get that error occasionally, by which I mean it may happen multiple times a night, then nothing for a few days.
    SS writes to an external drive connected over USB 3.0 and I have found that it happens when my Mac is doing something pretty processor intensive, be that decompressing multiple large files, or transferring across my other external drives. My interpretation of it is even though System info isn't showing full processor usage across all the cores the iMac is still having problems doing everything at the same time and this manifests itself as SS not being able to keep up with writing the video files - there is enough network bandwidth for it to do so but the processor cannot do everything it is being asked to do so prioritises somethings over others.

    My interpretation is probably wrong but I have never had this issue when my Mac isn't 'busy', and can almost force it to happen. I am not sure what the Mac is doing sometimes, particularly when it happens in the night.
  • CVHCVH
    edited March 2019
    My system is at 10-15% consistently. It’s only purpose is SS. New Mac mini.
    I reformatted the drive and have set to drive to save 70gb before deleting files. Maybe it was an issue where it was trying to write and delete old files simultaneously.
    I will report back when the drive fills up in a little over a week.
  • I've been plagued by the 'disk is too slow and cannot keeps up with writing data" for a long time now. Late-2013 Mac Pro, 31 cameras, Drobo 5D for storage (Thunderbolt 2).

    Ran the File Writing Test app for 31 cameras and only get 36MB/s. The Drobo is full of 7200 RPM drives, configured for dual-drive failure survivability. I should be getting at least double that, shouldn't I?

    Liking Drobos less and less the more I use them. I think allocating some space on my Synology NAS (RS1912+) might be faster, even over ethernet.
  • I have six days on the drive, another six before it is full. No errors so far.
    Seems that when the files are being overwrites or deleted is when the error occurs.
    Since the speeds of the drive, this should not be an issue. I’m stumped.
  • It is certainly possible that when the auto-delete task runs, this is slowing down the drive sufficiently to cause write problems. We have attempted to mitigate this effect by throttling the delete process, but perhaps this isn't working well enough.

    As a test, could you all try turning off the auto-delete options and see if this prevents the errors? You may like to first clear sufficient space from the drive so as not to run out of space.
  • Thanks for the response Ben.
    So far no errors. I will have to wait another 5 days of record before the auto delete process takes place.
    I have a 2tb SSD. I had the delete process to take place with 40gb space left.
    I have several cameras that record daily files of 60gb.
    Maybe that is the issue. I increased the delete process to start when 80gb space left now.
  • BenBen
    edited March 2019
    This is useful information. So it could be either the simple fact of low disk space, or it could be the activity of the automatic file deletion. Please let me know what happens when you hit the 80 GB threshold and the auto-deletion task kicks in again. If it turns out that this is the problem, we can throttle it back some more. 80 GB should be enough free space to avoid significant slowdowns purely from the fullness of the drive.
  • CVHCVH
    edited April 2019
    I am getting the errors once the deleting process starts. When the drive has enough space to record, there are no errors.
    There are actually no interruptions in the records, the only reason I notice the errors are from the daily reports.
  • Thanks for the feedback. I have made some changes to make the file deletion routines more efficient and less stressful on the disk, and have posted a new beta version of SecuritySpy with these improvements (version number 4.2.10b13). Please install this one and let me know how you get on.
  • Anyone currently testing this, please grab this updated beta version of SecuritySpy (4.2.10b14) as it has some further improvements in this regard.
  • Thanks Ben, although this version seems to be ignoring the deletion process and is running the drive down to 0mb. I have the preferences set for 60gb.
  • Just noticed that I was running b13. Just installed b14 and it corrected the issue.
  • OK great. Any sign of the "disk too slow" error?
  • CVHCVH
    edited April 2019
    Yes same error.
    I really don’t think the disk is too slow, I have results of +300MB/s when I use the speedtest software you recommended.
    The error is for all the 9 cameras, all for the same second, yet I don’t see a frameloss in the records.
    There is usually 1 error for each camera per day.
  • BenBen
    edited April 2019
    Does the error occur at roughly the same time of day, or is it random?

    Have you excluded this disk for Spotlight indexing? (System Preferences -> Spotlight -> Privacy).

    It’s strange that the error is so sporadic - if it were due to SecuritySpy’s auto-deletion routines it should happen more often, as this runs every few minutes.

    300 MB/s is definitely fast, but there must be something that is causing a temporary short-term slowdown that is reducing this speed drastically, resulting in this error.

    When the error happens, the cameras will be temporarily disarmed, so there will be some loss of recording for the short time that the cameras are unarmed. They will then automatically re-arm based on the current schedules set for them. So while there is some loss, it should be very short.
  • CVHCVH
    edited April 2019
    It is random. I’ll do some investigating over the next several days. I did not have spotlight indexing excluded, I do now.
  • b14 hasn't fixed the issue for me either. :-(
  • @jlbrown - I suspect the cause of your problem is simply a low disk, as you are only getting 36MB/s from the file writing test software. Your setup should be much faster than this in theory, so there must be something wrong somewhere. Perhaps one of the drives is going bad? With a drive this slow, this can only be fixed by improving the drive speeds or reducing your recording rates; it won't be fixed with a SecuritySpy update.
  • I updated my macmini to 10.14.5 from High Sieraa. I also have the latest b7 installed. I have not had an error in 7 days.
    Jlbrown, are you on High Sierra?
  • I have started to see this issue as well - just went from 5.0 to 5.01. 13 Cameras going to an 8Tb external disk attached via USB3 - all are motion triggered. I have been slowly replacing old NTSC cameras with 3-5Mp IP cameras, so data rates are going up.

    I will check on spotlight and time machine. All cameras are on a 7/24 schedule - once marked offline, they don't seem to come back. Was there a timestamp in the alert box that pops up? I don't recall seeing one.
  • Hi @finkej try our File Writing Test with SecuritySpy closed and let me know what you get.

    As for the errors, the log file contains time stamps (File menu -> Open Log).

    You can also check disk pressure with the new Dashboard feature (available from the Window menu) - 100% disk pressure means that all disk writing buffers are full and this error will be generated. In normal usage with a fast disk, the pressure should remain well under 10%. What does your disk pressure graph look like?
  • @CVH - running on Mojave 10.14.6.
    Ben - didn't know about the Dashboard and Disk Pressure. Mine is usually 0%, occasionally goes as high as 5%. When I was the 'disk too slow' errors yesterday, disk pressure was 1%.

    File Writing Test gives 52.96 MB/s.
  • 53 MB/s is a moderate speed, but should be adequate for up to a dozen cameras or so. Check the Dashboard again but this time turn off smoothing - this allows you to see the peaks better. Do you see it spike up to a high value?
  • Hey Ben, I was just cruising through recent posts and saw this one. I never got around to e-mailing support but I'll add my "me too" to this post here. I get this error every day around midnight each day and all cameras temporarily disarm. Disk pressure is almost always at 0% and the File Writing Speed Test was 96.18 MB/s @ 16 files. The storage drive for the cameras is a 5-drive Synology NAS and the only thing that happens nightly on the NAS is an off-site backup. I just manually ran this same backup task and nothing showed in the SS log at all. Not sure what's up, but it's just an irritating error window I deal with right now.

    Also, would you mind explaining smoothing? It drastically adjusts the visualized data. For example, if I view Disk Pressure at the time of the logged error with smoothing off I see a blip for a minute or so at 40% but if I move smoothing up, that same blip then reads 1%.

    Unrelated, in Dashboard when viewing packet loss, if I check my crappy wifi camera that constantly drops packets, it reports 650% packet loss. What on earth can that possibly mean? :-)

    I'm happy to troubleshoot if you've got any ideas. Please let me know if I should just make this a support case instead. Thanks for everything.
  • I will also add a "me too". For me, it is sporadic. dashboard can show 0 (or close to it) disk back pressure for hours before the problematic spike, and then there is a spike to 90% which corresponds to the error log. Recording data rate (smoothed) is under 664 KB/s, peaking to about 830 KB/s (2 hours earlier). CPU usage stays below 36% total, which is the peak at about the same timeframe (otherwise seems to stay below 26%). If I look at Activity Monitor, I do see a rather large spike in writes at possibly the same timeframe, so perhaps it is related to other processes simultaneously accessing the disk at that moment. I do have Backblaze backups running, and bztransmit is shown as the 3rd largest "Bytes written", but I presumed most of those were network writes incorrectly showing up under the Disk tab (similarly, launchd is shown as the highest writer at 2.47 TB, and SecuritySpy at 409GB, thus I presumed there were network writes incorrectly categorized). bztransmit does do a lot of disk reads however, which would definitely slow down writes (bztransmit shows the highest bytes read at 288GB).

    I don't seem to know how to post a snapshot of the dashboard to the forum...
  • There is a common theme here, which is a disk that is generally fast enough to keep up with capturing, but with short periods where the speed is dropping drastically, causing SecuritySpy's file writing buffers to fill up and this error to be generated. The short-term slow performance could be caused by many things (e.g. an automatic backup, another disk-intensive application, or the system clearing purgeable space).

    So what I've done is to double the size of the disk writing buffer, in the latest beta version of SecuritySpy (currently 5.0.2b8). Please can you all test this and let me know if this reduces the incidence of this error or not.

    However, I'm wary about increasing the disk buffer too much, as this could potentially use a lot of RAM.

    In general, for best performance, the disk should be:

    - An SSD or fast HDD (NOT a fusion drive)
    - Not the system drive
    - Connected via a fast connection (Internal, Thunderbolt, USB 3)
    - Not used for other purposes
Sign In or Register to comment.