Monday Blues

Arrived at work on Monday hoping to find a nice 3-day log file collected over the weekend. However, apparently for some reason the server rebooted itself early Sunday morning (about 3 am, possible due to automatic OS updates), causing all log file collection to halt (and in turn causing the WiFi board & Nios app to hang). So we only ended up with a 2-day data file. Still, it's better than nothing, and is probably still worth running through processing, at least Gordon's initial translation, to see if the frequency of serial glitches has gone down. The data file is "node0.uart - Copy (13)cut.trnscr" in the Server Code folder. (I had to manually cut out some short sections left over from earlier runs from last week's debugging.)

Yep, I just checked, and the system was set up to automatically install updates every Sunday at 3 am, which involves automatic reboot. To prevent this from happening again, I turned off automatic installation of updates; the system should now just automatically download updates, and notify me when installation is needed.

Some other things to do today (or soon):

• Check out that glitch with the PPSCNTR value skipping a beat that I found during data analysis.
  
  - I examined the raw data file, and, sure enough, there was an extra spurious PPSCNTR event, approximately 1/3rd of way between two valid PPSCNTR events. Possibly due to noise on the PPSCNTR input, or some kind of very rare bug or glitch in the input-capture circuitry, or the Nios interrupt mechanism. We'll have to be careful to watch for these events.
  
  
• Talk to Gordon about what analysis he needs to do in Scilab. He will be here between 1:30 and 2:30.
  
  
• Thinking about reconfiguring a few LEDs on the DE3 board to show the instantaneous status of the various data fields from the GPS NMEA messages that will tell us right away when we're not getting a good time signal. It's easier than staring at data records.
  
  
• Maybe also soon start doing the server-side processing of the incoming GPSAPP data. After the last round of analysis I have a clearer idea of how to handle this. For example, below is the relative OCXO frequency variation (at two different window sizes) during the 3.5-hour GPS time outage in the Copy(11) run. We can see that at the start of the outage, the frequency quickly wanders outside the range of both the usual +/- 0.1 Hz variations (seen for the 60s window size) and +/- 0.01 Hz variations (seen for the 600s window size). When this is seen to happen, or when the phase wanders unexpectedly far from where we'd expect it to be based on earlier data, we can "lock in" the present OCXO frequency based on (say) the average of the last 10 minutes (or last hour) before the frequency started to wander, and ignore the GPS times at least until TRAIM comes back online. Anyway, there are lots of options for detecting and patching over outages.
  

Finally, before leaving for lunch & errands, I wrote up some instructions on the data analysis in SciLab for Gordon, and emailed them to him.