Need help understanding error messages

jms703
edited January 2014 in SecuritySpy
I'm getting the following error messages on an almost daily basis:

* Error communicating with the network device "Network camera 1" 3.2.1,10,835 Excessive packet loss from network device, the network may be too slow or defective, or this computer may be overloaded. Check the network and/or reduce this camera's frame rate.


Lately, I've been seeing these messages (separately, not in the same email):

* Error communicating with the network device "Network camera 2" 3.2.1,89900,54 Connection reset by peer (ECONNRESET)

* Error communicating with the network device "Network camera 2" 3.2.1,551122,801 Data from network device not as expected


I'm not sure what the issue is. I do notice that when I was reviewing my video settings, the small window that shows what the camera "sees" tends to turn to blue and read "timeout error". I'm wondering if I have some connectivity issues? I'm not sure.


My setup:

# Computer
Mac Mini
2.5 GHz Intel Core i5
8GB 1067 MHz DDR3
OSX 10.9.1 (Mavericks)
Security Spy 3.2.1
Video is written to external Seagate 2TB USB2 drive.

# Network
Network is a Netgear GS110TP (8 port gigabit PoE) running version 5.4.2.9 (latest)
Ethernet cables are the CAT5e that shipped with cameras (very long, and excess is coiled up)

# Cameras
Network camera 1: SWNHD-820CAM running firmware V5.0.2 130805
Network camera 2: SWNHD-820CAM running firmware V5.0.2 130805
Camera 3: Logitech C920 USB

# SecuritySpy video configuration for camera 1 and camera 2:
Device type: Hikvision
Format: H.264 RTSP
Input number: 1
No recompression of video (and audio) boxes checked

# SecuritySpy camera configuration
Camera enabled
Motion detection enabled
Capture movie when motion is detected is checked
Frame rate 10
Pre capture 1 second
Post capture 10 seconds

Comments

  • Hi,

    The 835 message typically occurs because the network or the computer is too slow to keep up with the processing of the video stream, resulting in some data packets being dropped. This is surprising with the setup you describe, which should be easily fast enough to cope with the video streams (unless the computer is also being used for something else that is taking significant CPU time).

    The 54 and 800 (timeout) errors are general network connectivity problems. If you get these once in a while it's fine; if you get them several times per day then that's unusual and points to a problem with the network.

    The 801 error is not often seen - it means that the camera sent some data that couldn't be understood by SecuritySpy. How often do you get this, and does is happen in conjunction with the network errors? It could be the camera's fault or the network's fault. Do you also get these errors with Network camera 1?

    Is the computer, as well as the cameras, plugged directly into the Netgear switch, or is there any other network hardware involved?
  • All of the devices are plugged into the same Netgear GS110TP switch and all settings are default. The cameras negotiate at 100MB and the Mac Mini negotiates at 1000MB.
  • I almost posted a similar question.

    My log file is filled with these messages :
    06-02-2014 1528-10: Error communicating with the network device "Camera1" 3.2.1,10,835 Excessive packet loss from network device, the network may be too slow or defective, or this computer may be overloaded. Check the network and/or reduce this camera's frame rate.

    My Mac is even more powerfull, it's an iMac i7 with 12Gb ram, recording to the internal drive and at the moment this is the only task it performs. The iMac was a clean install (10.9.1).

    I have a similar network setup, and my cameras are also Hikvision (I also have a dlink which never reports an error).

    These are 3mp cameras, but I have set the to 1920x1080, at 4fps, and 1024 bitrate.
    I can't believe this is still too much.

    Are there any tools available to troubleshoot these network packets, or to investigate further. I'd love to get rid off these error messages.

    Not only is an error generated, but also an alarm.
    If I look at the captured movie, I can see that the video freezes for a few seconds and than catches back up, generating too much motion for SecuritySpy, which then generates the alarm. Since I also use Remote Patrol to receive notifications, we don't get much sleep at night :-)
  • It's strange that you both should be getting these errors on fast computers connected by fast networks. In the RTP stream, each packet is numbered, and SecuritySpy checks the numbers of the incoming packets to reliably detect packet loss - this error is only generated when 4% of packets were lost over the past 40 seconds. This is a significant loss that will severely affect the stream (as you have seen Pete) and which you would generally want to do something about, which is why SecuritySpy reports it as an error.

    We very rarely see this issue, which is why it seems unlikely that it's a bug in SecuritySpy itself; otherwise we would get hundreds of users reporting it. More likely it's a problem with the cameras themselves - further evidenced by the fact that the D-Link camera never reports this problem.

    The network tool that would help is Wireshark - but it's big and complicated and will take some time to do so I'm not suggesting you attempt this! But if you felt inclined you could use its RTP analyser to look at the RTP packets. If, when you get this error in SecuritySpy, you see perfectly sequential packet sequence numbers in Wireshark, you can point to SecuritySpy as the problem. If however you see discontinuous packet numbers in Wireshark then this points to the network/camera.

    If you look at the timings of this error, does each camera produce this error at the same time, or are the timings sporadic and not correlated between the cameras?
  • Hi Ben, I'm pretty sure it's a problem with the camera's and not SecuritySpy.

    In the webinterface of the camera there is an option called SVC, and this is what they say in the manual :
    SVC:
    Scalable Video Coding is an extension of the H.264/AVC standard. The technology encodes the video signal with layers;
    the basic layer and several enhanced layers and it is adaptive to the network condition to transfer different video streams.
    For example, when the bandwidth is limited, only the basic layer data is encoded and transferred. You can enable the function when you want o see the video with several terminals , such as the mobile phone with 3G network, or the personal computer with IP network.

    So it sounds like Hikvision do some funky stuff with their streams. However I have disabled this on my camera, but I don't see any difference at all. Not in video quality and also the errors keep popping up.
    Hikvision also has something called Region of interest, where you can mark a region in the video that gets better encoding, where everything outside that region will be more heavily compressed.
    I suppose it's all non-standard stuff they do.
    I even tried to open the rtsp stream in vlc, and the main stream is garbled over the half of the frame, and the frame freezes, so even VLC is not able to display the stream correctly.
    The main stream is always h264, but in the substream I am able to choose mjpeg, but sadly the resolution is so bad that this might not be a good option.

    Also selecting mjpeg in SecuritySpy solves all the errors, but then I get only 1 frame/sec, which is a bit too slow.
    The funny thing is that I have 4 hikvision cameras, 3 are on a powerline, and the one that is connected directly to the router is giving me all the grief.

    I am playing with wireshark and see a lot of stuff happening, but when trying to run the rtp analizer, it tells me I need to select an RTP packet, but strangely, I cannot find an RTP packet. They are all TCP...

    It's too late for me to return these cameras, and apart from the errors popping up, I am quite satisfied with the quality and features.
    Is there anything else I can try to solve this ?
  • Ben, while I can't quite figure out how to use wireshark, I do have an old license of evocam (bought it long before I tried SecuritySpy).
    I have just set up this problem-cam with evocam, and set both SecuritySpy and Evocam to record on different macs.
    After the packet loss occurred (looked in the SecuritySpy log), I looked at both videos, and sure enough the glitch (a freeze frame lasting 4 seconds long) occurred in both recordings.
    I guess this prooves that the video stream coming out of the Hikvision cameras is a bit wonky ?
    Is there a possibility for SecuritySpy to grab more than 1fps when this camera is configured for jpg over http ?
  • Hi Pete,

    Thanks for the information, as you say it does appear that this camera is doing something a bit weird with the stream. There are two things I can suggest you try.

    Under the RTSP standard there are two ways to stream the actual media: TCP and UDP. By default, SecuritySpy will use TCP as it's more reliable. However SecuritySpy does also support UDP, because certain cameras cannot do TCP - in this case SecuritySpy will automatically negotiate with the camera the use of UDP. So I have prepared a special version of SecuritySpy for you to test, in which you can explicitly select between TCP and UDP to see if this has any impact on the errors (I'll send you a link to the download by email - for anyone else wanting this feature it'll be in the next update of SecuritySpy).

    To test this, select the Manual profile in the Video Device Settings window in SecuritySpy, select the "RTSP UDP" format option, and enter Streaming/channels/1 as the Request.

    The second thing to try is to get a good JPEG stream going. In the camera, is it possible to select JPEG as the format for either the first or the second stream? If so, do this and again with Manual configuration in SecuritySpy use the RTSP TCP option and Streaming/channels/1 or Streaming/channels/2 as the Request, for the first and second stream respectively.

    I've never actually used Wireshark to analyse RTP packets so I'm not sure if I can help with this unfortunately, but I can tell you what's going on protocol-wise so that you know what to look for. The session is set up with the RTSP protocol, which is always communicated over TCP. The client and server negotiate the streaming parameters and how to actually send the media streams: this can either be on the same TCP connection as the RTSP conversation, or it can be via separate UDP streams. Whether the media is transmitted via TCP or UDP though, actual protocol that carries the media data packets is always RTP.

    Finally, your sentence "the one that is connected directly to the router is giving me all the grief" concerns me - if you are using a router as the hub between the camera and the computer, this could be what is causing the problem. I'd always recommend using a high-quality switch to run your network. See my blog post on the topic.

    Let me know how you get on.
  • Hello Ben,
    Thanks for the beta version. I have been trying all the combinations, but I'm afraid it doesn't change a thing.
    Even with the new UDP option in the manual configuration, I keep getting these bandwidth errors.
    I have also gone through all the other options (RTSP over TCP, RTSP over HTTP...) but they all generate the same error.

    Also, with the regular Hikvision profile, I could choose HTTP (video only) and get a 1fps feed, but in the manual configuration I can't get this to work, I only see a blue screen. Is this perhaps a different Request I need to fill in ?

    About the requests :
    Streaming/channels/1 is the Main stream for Hikvision, and is only available in H.264 encoding
    Streaming/channels/2 is the Sub stream, and here you can choose H.264 as well as MJPEG, but the max resolution is only 704x576, and the compression in the substream is really awfull, even when set to a high bitrate.

    I also mistakingly said I was using a router, but I meant a switch. I'm using a netgear switch.

    I think it's maybe time for me to try and contact Hikvision about this. The problem is clearly on their end.

    Anyway thanks for your effort and help.


  • Ben
    Ben
    edited February 2014
    Hi Pete, it's a pity that UDP streaming had no effect, I was hopeful about that.

    It's also annoying (and totally unnecessary) that the JPEG stream should be limited in resolution and quality. They should have taken care to properly implement this.

    As for the manual configuration for HTTP JPEG, you will need to use Streaming/channels/1/picture?snapShotImageType=JPEG as the Request, but you could also try Streaming/channels/1/picture in case that makes a difference to the frame rate.
  • 10-02-2014 1712-47: Error communicating with the network device "Hik_Drive" 3.2.1,89900,54 Connection reset by peer (ECONNRESET)

    10-02-2014 1721-27: Error communicating with the network device "Hik_Drive" 3.2.1,10,835 Excessive packet loss from network device, the network may be too slow or defective, or this computer may be overloaded. Check the network and/or reduce this camera's frame rate.

    I'm seeing the same hence my request for other options rather than buy more Hikvision.
  • Hi Ben,
    (sorry for my english)
    I have same problem, I have 6x cameras Tenvis IP-391W HD (newest firmware 1.3.3.3).
    Main stream settings: 1280x720p, 15fps, VBR, 2MBit/s, h.264.
    Mac server mini i7 2GHz, 8GB RAM, GB switch.

    From every camera i have randomly this error message:

    Error communicating with the network device "Vchod" 3.3,10,835 Excessive packet loss from network device, the network may be too slow or defective, or this computer may be overloaded. Check the network and/or reduce this camera's frame rate.

    I think, I have tried everything. I tried manual camera settings and use UDP stream with parameters /0/av0 (camera main stream) but problem is not solved. I have used too TENVIS IPROBOT3, but same error. (IP391W HD and IPROBOT3 = same hardware inside).

    When I define motion capture, in every captured video missing 5 seconds in the middle.
    When I define continuous recording, I see that some video frames missing, but not whole 5 seconds.

    Do you have any idea what to setup to use spysoftware with tenvis cameras?
    I can give you access to spysoftware web interface, if this help.
  • Ben,

    I stopped capturing video to external USB drive and switched to capturing video on an AFP share. These errors went away.