The situation:
We have about 60 weather stations accessed by Raven XT cell modems; they are in remote areas such as farm pastures and such. We have been having what appears to be modem issues at many of our stations that have IP addresses. This started in July of 2015 and we have been working with service providers (Verizon and AT&T), Sierra Wireless, and Campbell Scientific to try and find a solution.
The problem: The modems fail to communicate every few weeks, cannot be accessed through LoggerNet, and have to be manually reset (through Acemanager) for them to start working again for a while. Then, they go through another cycle and fail once again.
After consulting with Campbell Sci, we have disabled the PING option on our modems and changed the port numbers that we have been using (from 3001 and 12345 numbers). This has not been successful so far.
Hopefully some others out there have had similar troubles and have found solutions. I am posting with the hope that we can avoid duplicating your effort.
Thanks!
Jared,
Are your modems setup for serial or PPP mode. If via PPP you can do it using commands from the datalogger to the modem. There is an example on how to do this posted on the Campbell Scientific Blog at https://www.campbellsci.com/blog/access-control-raven-modem .
If setup in serial server mode, you can do it using a controlled relay or via the SW12 instruction (if the modem is connected to the loggers SW12 port) using TimeIsBetween and TimeIntoInterval instructions https://www.campbellsci.com/blog/do-more-than-store-data .
We also recommend the latest datalogger operating system be run in the logger. There have been a lot of fixes and upgrades done in the OS to address issues simular to what you are reporting.
Hi Gary,
Thank you for your response. We have our modems set up in serial mode. Although it may be helpful, we aren't considering automating the modem resets; prior to July 2015 we did not have to reset at all for the past 10 yrs or more. I'm trying to figure out why we have multiple modems with IP addresses failing now. As far as we know, nothing has changed from our end that could have triggered this.
FYI, we contact our stations hourly and all of our modems are connected to the 12V, not SW12V. We are running LoggerNet Admin on a virtual polling server. About 2 to 5 stations a day become unreachable through loggernet, but are reachable by Acemanager. Any more ideas?
Jared,
What version of the operating system is in the dataloggers? There have been many bug fixes and updates made that might resolve the problem you are having.
I had that issue with several Raven modems in the past. I called it "stupid mode" as the lights would indicate all was well yet I couldn't connect. I didn't take the time to try to figure out why, I just connected them to SW12 or other relay and turn them off once per hour for 5 minutes (my scan rate in most cases). The negative is it cost me time to visit all the sites and rewire the power, the positive is I have not had the problem since.
1. If AceManager can reach a "non-responsive" modem, confirm the serial port settings in the modem are still correct, including port number. Obvious, I know, but can be done in a few minutes.
2. More interestingly, we know the modem is receiving packets because it answers AceManager. But what does that modem do with packets from LoggerNet? You could connect a laptop running a terminal emulator to the modem and watch while someone at the office tries a manual collection with LoggerNet.
All,
Our network is having similar issue to the ones mentioned here. However, the difference is when the modems become non-responsive, we can no longer access them through the AceManager interface either. Currently, we do not have these wired into a datalogger, but rather run them seperately through a deep cycle marine battery.
My question is, is there any way to have the modem automatically reboot through the AceManager interface (on a defined schedule)? Or is the most efficient way to have the modem connected through the datalogger and program loggernet to automatically reboot it at a given interval?
Spheuser (and all),
If you are unable to achieve what you want through AceManager, you could use one of the plug-in power cycle adapters we built for one of our customers with a similar network drop issue.Their issue was not with a cellular modem but a different communications interface; the data logger was effectively inaccessible and SW12 was already used for another purpose. This plug-in unit power cycles the communications interface once a day (for 2 seconds) to provide a regular power-off reset and did not require any change in data logger code. It should be useful to you in this situation if you cannot find a software solution. http://www.pordis.com/products.html#160A
I can confirm that you are not alone in this issue. It has been getting worse. We have been having (mostly) Verizon modems (Raven and LS300) stop responding left and right, I'd say currently we have at least 10-15 that stop talking once a week.
We don't like running the cell modems off the datalogger 12V/SW12 ports because we've had issues in the past when we have alot of sensors on a station running off 12V (we noticed it happened with 6 flowmeters) the 12v port on the datalogger would drop below 12v during a scan/measurement. We run the power directly off a PS150 (sometimes adding a second PS150) or whatever source we are using to supply power to the datalogger.
While SW12 is good we've begun installing relays that are are normally open. We've used the SW12 or a Control Port on our CR800, CR1000, and CR6 to toggle these relays with a 3 second delay. This power cycles the modems and they come back online however we have sensitive applications including control and power cycling LS300's with 3-5 minute provisioning can create issues in itself.
In the past I've been told by Campbell it's Verizon's network and Verizon points the finger right back at the hardware leaving us without an explanation for the issue.
To further troubleshoot, we use both RS-232 (Raven) and Ethernet (to NL120/121) connections and both stop working. I've used the default AceManager firmware the modems come with and I've upgraded AceManager to the latest version. If I had to make a really uneducated guess, I'd say there is something not operating correctly with Dynamic DNS (if Verizon is stating nothing has changed with their network and Campbell is saying nothing is wrong with their hardware). That is the only piece not Verizon or Campbell based.
Most stations are Central/Eastern Washington.
Whether or not any of this helps, I don't know, but at least you know you are not alone.
Thanks everyone for your help.
Gary, the station operating systems are mostly updated and still have this problem. Updated templates as well.
BO 2, we verified the port numbers, thanks.
PS2User, thanks for your post. Yes, its nice to know we're not alone in this problem. We are now considering switching to the SW12.
I had similar problem two weeks ago. At that time, I could reach my modem by using web version Acemanager; but could not reach my datalogger. After I post my problem and got help from DAM:
https://www.campbellsci.com/forum?forum=1&l=thread&tid=2977
I embedded codes:
...
SlowSequence
Scan(30,min,1,0)
...
RealTime( rTime()) 'get datalogger's time
...
If (rTime(4) = 4) Then 'if Hour = 4:00 AM
SetStatus("FullMemReset",-98765) 'force a "watchdog reset"
EndIf
NextScan
EndSequence
Started from that time, the connection from my LoggerNet to datalogger has never been interrupted.
The difference is that I used high scan rate in my main scan, from 20 Hz to 50 Hz. That is why I reset "watchdog" everyday. In your case, you can reset "watchdog" every week or every ten (10) days depends on how long time the connection interrupted. Best wishes.
Jared I'd suggest using a seperate relay, we use the Crydom DC60S3B and just use the SW12 or C1-C8 ports to toggle it high for 3 seconds using a delay at a specified interval (it's a backwards relay so it's on when the signal to it is low).
Bluegrass - Thanks for the info, though I must say the text below scares me.
"You can programatically force a resart by using:
SetStatus("FullMemReset",-98765)
Make sure to include the -.
This will force a "watchdog reset".
This will save you a trip to the site, but is clearly not the desired behavior."
It'd be nice to have an explanation of why these things are happening instead of just using workarounds.
PS2User,
It scared me as well before I used that command. And I hesitated to use it at the beginning. After I used it for two weeks, everything is ok. Nothing lost. One thing for sure - I do not need to drive 160 miles to manually reset my datalogger by using this command.
Thank DAM from CSI!
I'm having this issue as well, glad I'm not alone. Clearly affecting a lot of users. I don't understand why Campbell cant get to the bottom of this issue. It is occurring for us on a brand new Raven XTV with the latest OS, so obviously the os fixes did not solve the problem. Verizon denies any responsibility. Having to reset via SW12 is irritating, especially since the current limit on this port is 900ma and we have other things on the SW12 ports (ventilator on a CNR4 radiation sensor). We also have one of our modems on a CR5000 which, while it has two SW12 ports they can only be (stupidly) controlled together, unlike on the CR3000. I also don't understand how the 'FullMemReset' works for people- per the CR3000 manual it deletes everything on the loggers CPU, which is unacceptable. Any update, Campbell?
The RavenXT and RV50 modems have a setting on them for "Periodic Reset Timer". If the issue is resolved by power cycling the modem itself, you could use that setting on the modem and not need to switch power to it. The setting is found in the Administration>Advanced section.
Thanks for the info!
We had a similar problem that occurred occasionally with a CR800/1000 and modem combination using GSM. The modems were always power-cycled but the only way we could re-establish communications was by going to site and physically turning the logger off and then on again. I found someone else on the forum who had a similar problem using a GPRS link. The solution, similar to that offered by Bluegrass, was code in the logger, forcing it to reboot -
'Declare variables - software reboot
Public rTime(9) 'declare as public and dimension rTime to 9
Alias rTime(1) = Year 'assign the alias Year to rTime(1)
Alias rTime(2) = Month 'assign the alias Month to rTime(2)
Alias rTime(3) = DOM 'assign the alias Day to rTime(3)
Alias rTime(4) = Hour 'assign the alias Hour to rTime(4)
Alias rTime(5) = Minute 'assign the alias Minute to rTime(5)
Alias rTime(6) = Second 'assign the alias Second to rTime(6)
Alias rTime(7) = uSecond 'assign the alias uSecond to rTime(7)
Alias rTime(8) = WeekDay 'assign the alias WeekDay to rTime(8)
Alias rTime(9) = Day_of_Year 'assign the alias Day_of_Year to rTime(9)
Public ThisFN As String * 40
Then in the scan -
'Reboot Data Logger at 10:15 daily
RealTime(rTime) 'get time
If rTime(4)=10 AND rTime(5)=15 Then
ThisFN=Status.ProgName
FileManage(ThisFN,6)
EndIf
I am no expert in this area but if it helps somone - you are welcome. It certainly solved the problem for us.
Thanks for the reply. We've been able to solve the modem issue by manually cycling power to the modem. Resetting the logger is not ideal for us as we are collecting 10hz eddy covariance data. I'm going to try getting the automatic periodic reset to work in AceManager for the modem.
Important
After review with our engineering department can I change the advice given in the discussion above.
Using the instruction: SetStatus("FullMemReset",-98765) will reset the datalogger but can possibly lead to loss of data, especially if data is being written to files on the datalogger. This is because it forces the logger to literally stop dead, which then triggers a watchdog after some seconds (this function was introduced to test the watchdog). However, because the datalogger often writes data in background process to memory or files, if the setstatus command is called with these parameters when background processes are running which are storing data there is a risk some may be lost, files may be left in an open state or even corrupted.
The cleaner way to restart a program is to use the FileManage command (as shown above) which is called infrequently or on some event, e.g. lost of internet connectivity. Running Filemanage is akin to reloading the program from a PC where the program is first stopped and all data is flushed to memory or cards and files closed properly.
Even more simply the Restart command can be found in the CR300 datalogger and will be introduced in future operating systems.
We were chasing our tails with these Raven XT's, trying their keepalive feature, etc. without success. The modems would need a remote reset to get the port the datalogger used to work again. At the suggestion of a tech at Campbell, we now have our modems configured as follows: no keepalives, no ping response (was set to "ALEOS responds") and 24 hour resets. We have not experienced any issues for several months now. There were two modems (older firmware, maybe?) that didn't have the option to self-reset. I'm keeping my fingers crossed.
Replies from JohnK and APS are very helpful - thanks. I have had internet communications fail repeatedly with various CR1000 loggers with Campbell and third-party GPRS modems; fixed IP in each case. Various frequencies of communucation failure. OS 31.08 and earlier.
CR1000 OS 32 now gives us the Restart function while OS 32.02 gives us a bug fix for PPPClose(). I am hopeful that the latter will resolve my issues, and will be very glad if it does. But the former would also be worth using now, as per JohnK's post above, as a fall-back embedded into programs. Could anyone kindly suggest some example code which would force a Reset if (a) internet connectivity lost for > 24 hours and (b) battery voltage above a safe threshold such as 12 V. This would make a useful update to this discussion thread.
Many thanks.