Input separators

Did a little more work on tikiterm.py (my hand-rolled terminal widget in Python) at home this evening. Now I've got it to the point where there are designated "output" and "input" areas in each terminal window, and a graphical separator in between. Output to the terminal automatically appears above the separator. However, I haven't yet set up the event handlers needed to confine the insertion point to the input area, so the user can still, for example, select & replace text in the output area, or even the separator graphic itself. Fix this on Wednesday.

Actually, I went ahead and did this tonight, before bed. There are still some "gotchas" - like if the user selects stuff before (or overlapping) the start of the insert area, he can still use BackSpace to delete it. Haven't figured out an easy way to prevent that yet (unless I disable selections altogether). But, at least it now prevents the user from placing the insertion point outside of the input area, or backspacing past the start of the input area.

Plans for this week

OK, looks like my data-collection run that I started on Friday is still running, so that's good. I should have a full weekend's worth of data. The file is 111 MB. Some next steps:

• Contact Gordon to see what his status is on the code. I emailed him just now, asked him to please finish it in the next few days, or hand it over to me.
  
  
• Alternatively, as a contingency plan, we could always slurp the raw data directly into Matlab and do all the processing there. However, although this probably wouldn't be difficult, it doesn't really make sense to replicate the work Gordon has already done. It would probably be easier to just fix the Java code, even if it turns out that Gordon is no longer available to finish it up.
  
  
• Aha! Gordon is here now. He modified his code to count the number of bad checksums. He found 70-something glitches in the 74MB data file from Jan. 26th (just before the new level shifter and gender bender were put in place), that's an average rate of about 1 glitch per MB. Not too bad, but of course we'd prefer to see zero. I stopped my data-collection run that was going over the weekend at 112 MB - Gordon can run his code again on that.
  
  
• If there still a lot of serial glitches, and we decide we need to reduce them further, there are two more things I can try: (1) Lowering the baud rate on all 3 boards (GPS, DE3, WiFi), and (2) Seeing if I can turn on CTS/RTS flow control for the DE3->WiFi link, at least - this may be possible.
  
  
• Independently of all this offline data-analysis, I could [ ] add some features to the Python server to allow us to graph the incoming time data (in particular, the OCXO frequency drift, the self-reported GPS accuracy, the # of satellites, etc.) in real time. Not sure if this is worth the effort though. Meanwhile, another needed change to the server is: [ ] Allow us to type input in the TikiTerm windows. That is probably easier, so maybe I should do that first.
  
  
• Also need to clean off my desk, do some filing, fill in my timesheet.

After Gordon left, I spent a while re-studying all my old TikiTerm code, adding some comments, and getting ready to add the user input capability to it. I've been wanting to keep the input line below the output. I guess I need to figure out if I can intercept cursor-placement events and enforce that they can only happen below all output. Then I have to somehow receive the text that has been typed, and place it in an input buffer that client threads can read from. Tricky! Need to spend some time studying the TkInter docs to do this...

I think I can do it. I can create a new mark to designate the start of the input region. New output can always be inserted before (to the left of) this point. I can add a new event handler to the mouse-click and key-press events, while keeping the existing ones. I can use mark_set() to force-move the insertion point back to the start of the input region, in case the user moves it out of the input region. And when the user hits enter, I can grab the text in between the start and end of the input region, change its color, append it to the input buffer, and adjust the start-input marker to the new spot. Meanwhile, other threads can asynchronously consume data from the input buffer, using an appropriate lock. When no data is available, they can block on a condition flag that we can raise whenever we put new info into the input buffer.

You Tubin'

This afternoon I made a YouTube video of how to start up the electronics making up the core of the COSMICi Central Timing Unit (CTU).

Gender Bender Activated

OK, today I put in place the new gender-bender and null-modem adapter that I got from Radio Shack the other day. Here's what the setup looks like currently:


(The "yo" on the display is an acknowledgment that a line of data was received from the GPS.) And here is the new logic-level serial waveform output by the level shifter:


The rise time is noticeably shorter, it is now measured to be about 1.3 us on average, whereas previously I eyeballed it to be about 4 us or so. Hopefully this will cut down on serial glitches! Unless they're due to buffer overflows. Facebook friend Kragen suggested I also try cutting down the baud rate. It's currently 115,200 baud. I will do that if necessary, but first I want to see if this change helps. I need to collect a new data set, and perhaps while I'm at it, I will also instrument the firmware to look for checksum errors in the lines received from the GPS.

OK, I successfully modified the firmware to check the NMEA checksums in the lines received from the GPS before passing them on to the PC over the wireless connection. If there are errors, they are reported to STDOUT as well as to the server via Wi-Fi. I also made a video on my digital camera to demonstrate how to start up the whole system; I will post it on YouTube later. I think I will leave the current run going over the weekend; we can collect a whole 3-day dataset for Gordon to process later with his Java code. I will email him now to give him a heads-up about that.

A few more photos, just for fun:

1. Scope trace showing about a half-second worth of stuff. Purple: PPS pulse. Cyan: Level-shifted (logic-level) serial data from GPS. White: CTS from Wi-Fi and TX to Wi-Fi.
   
   
2. Server display showing (a) Wi-Fi diagnostic STDOUT-->COM terminal window used during startup (top left), (b) Main window (middle), (c) Message stream window (upper right), (d) Auxilliary I/O window (lower left), (e) Application data stream window (lower right), (f) Data stream transcript file viewer window (back).
   
   
3. FPGA board diagnostics. (a) Nios 2 terminal window (STDOUT stream via JTAG UART via USB, right), (b) Terminal transcript file viewer window (left).
   
   
4. Firmware code in Nios 2 IDE. Top of main routine.
   

Cheerio!

New Level Shifter

Today I am going to try putting in place the serial level shifter I received from SparkFun, to replace my old hand-wired one. This will require soldering a few wires from the shifter PCB to a serial connector. After this I will observe whether the quality of the bit waveforms in the shifted serial data stream is improved, and whether the number of serial errors is reduced. I emailed Gordon to ask him to instrument his Java data-file reformatting program to count the number of serial glitches; I will collect "before" and "after" data runs which will allow us to compare the quality of the serial data streams.

First, for the record, here is the new serial level shifter, ready for soldering:


And here is my old spaghetti-wired version of the same circuit:

And here is a scope plot showing a slow rise time on the level-shifted output (light blue trace), perhaps due to parasitic capacitance, or maybe the BJT we used just had inadequate specs:

We have to make the following connections. To get straight the serial terminology, the GPS is acting in the role of DCE (data communications equipment or modem), so is using DB9 pin 2 (straight-through cable) to transmit, while the FPGA board is acting as DTE (data terminal equipment or PC), so is using DB9 pin 2 to receive. Meanwhile, pin 3 of the DB9 connectors at both ends of the cable are used to send data in the opposite direction - from DTE (FPGA board) to DCE (GPS module).

• VCC (JP2 hole 1) = shifted high level -- bare yellow wire (to alligator-clip to 3.3V supply)
• GND (JP2 hole 2) = RS-232 ground (hi-V DB9 pin 5) -- green wire -- lo-V DB9 pin 5 (top row, right).
• TX-O (JP2 hole 3) = lo-V, inverted version of hi-V DB9 pin 2 (DCE->DTE) --> red wire --> lo-V DB9 pin 2 (top row, 2nd from left)
• RX-I (JP2 hole 4) = lo-V input, inverted & raised to hi-V DB9 pin 3 (DTE->DCE) <-- white wire <-- lo-V DB9 pin 3

Here's the completed solder job:


Now we'll power off the 3.3V supply (which turns off both the OCXO and the existing level shifter, thereby pausing all data collection) and close the server, halting the data run. The data file (about two days' worth, Tuesday to today) is about 74MB in size.

Disconnected serial cables from old level shifter, connected them to new level shifter. Copied node 0 UART transcript file, restarted server (to start new transcript file) and Wi-Fi board (to re-establish server connections). Here's the new shifter rig, with cables and lo-V output probe:


The 7-segment display says "oh" acknowledging that the FW received some startup messages from the Wi-Fi board, although it didn't understand them. I should change the FW to display an error code in that case. OK, did that. Restarting FW from within Nios II IDE. OK, it is ready to go...


Turning on 3.3V supply... This will start the run (hopefully!)

OK, whoops, discovered that the level shifter is expecting the hi-V input to come in on pin 3, not pin 2, so had to switch the GPS to DTE mode, to use the opposite pin. Here are the corrections:

• VCC (JP2 hole 1) = shifted high level -- bare yellow wire (to alligator-clip to 3.3V supply)
• GND (JP2 hole 2) = RS-232 ground (hi-V DB9 pin 5) -- green wire -- lo-V DB9 pin 5 (top row, right).
• TX-O (JP2 hole 3) = lo-V, inverted version of hi-V DB9 pin 3 --> red wire --> lo-V DB9 pin 2 (top row, 2nd from left)
• RX-I (JP2 hole 4) = lo-V, inverted & raised to hi-V DB9 pin 2 <-- white wire <-- lo-V DB9 pin 3

Hm, conundrum: With this long serial cable connected to the output, the rise time seems about as bad as before,


and, worse, I am not receiving any serial data! I think maybe the output voltage is too low. It is only 2.97V, whereas it should go almost all the way to the 3.3V rail. Aha, just realized that I forgot to ground the level shifter to the power supply. OK, with that connection added (via a convenient alligator clip) now the high output level is 3.28V, as expected. However, the firmware is still not seeing any data! Perhaps the gelware UART peripheral got hosed while I was fiddling around. Let me re-compile the design the Quartus (to pull in the latest FW changes) and re-program it onto the board. Aha, that fixed it. We are getting serial data again from the GPS. Here's the UART bridge TikiTerm window on the server:

Still, since the rise time doesn't look noticeably better, I don't know if all this is going to help with the serial data glitches at all. However, one thing I noticed while testing is that if the long serial cable from the level shifter to the FPGA board wasn't present, the rise time on the low-V level shifter output was much shorter. So this suggests that perhaps the serial glitches would be eliminated if I simply used a shorter serial cable. I should run by Fouraker's and/or Radio Shack and see if I can find a shorter cable. Or, I could even plug the level shifter directly into the FPGA board, except that the hand-wired header connectors on both ends are female, so I would need a M-M gender bender. Wait, maybe we have one? No, but maybe I could rig one up using some bare DB9 connectors... Here we go:


The one connector's pair of solder flange rows is just wedged between the other connector's pair of solder flange rows, and the whole thing is bound up with electrical tape. Not the greatest connection prolly, but at least I didn't have to do more soldering!

Argh, no, that won't do it... The pin numbering is inconsistent now between the two sides. Let me just go buy a real gender bender (hopefully that will cross over the wires to keep the pin numbering consistent)! Off to the store...

UPDATE @7:20 pm: Fouraker's was closing as I arrived (@ 5:30), but Radio Shack had what I needed. I got a null modem adapter to crossover pins 2 & 3, which is to sit between the GPS and the level-shifter, so that I can just leave the GPS unit in DCE mode all the time (since it needs to be in that mode when controlling it from the PC). And I got a male-to-male DE9 gender-bender adapter, which will allow me to interconnect the hand-wired female connectors that are dangling off the FPGA board & the level shifter, and hopefully improve the rise-time on the 3.3V serial input & hopefully reduce the number of serial comm glitches.

HOWEVER, I should remark with the caveat that the serial glitches *could* conceivably be due to buffer overflows, say in a UART device somewhere, due to our near-complete lack of flow control, and not to any electrical problems with my serial connection per se. Really, I need to instrument the Nios firmware so that it verifies the checksums on the NMEA messages from the GPS before echoing them out over the Wi-Fi connection, and report an error on the diagnostic output if there is a bad or missing checksum. Then, in the Nios command shell, we can pipe the nios2-terminal output to a "tee" command to log a transcript file from the app, and then if we detect serial glitches, we can go back and refer to the transcript see whether they were already present in the NMEA lines when the FW first received them from the GPS. This will narrow down whether the glitches are occurring in the GPS->DE3 serial link (perhaps due to lack of flow control there), or in the rest of the communication path. They could also be occurring in the DE3->WiFi serial link, which has pins that could be used to implement CTS/RTS flow control, but I need to look back at the EZURiO manuals to see if/how I can configure it to use these signals. At the FPGA end, the flow control might need to be implemented manually, because if I recall, the Avalon UART core ignores the CTS/RTS signals even when configured to pass them through. Next, the characters could be getting dropped due to internal buffer overflows in the WiFi board's bridging function, perhaps due to limitations of its RTOS; if so, then this could conceivably be addressed by using the more efficient "flyover" mode instead. Next, conceivably the characters could be getting lost during network transmission, although normally I'd expect that the various PHY, data-link layer and session-level network protocols (e.g., TCP) would do enough error-checking to prevent this. I don't think the losses could possibly be happening in the server itself, since the Python language is pretty tight with respect to that sort of problem, and I don't think my Communicator classes had any arbitrary buffer-size limitations.

Power supply testing

• Gordon emailed over the weekend that his code is still having a problem parsing the "*" splits. He shared his Dropbox code folder with me. Installing NetBeans so I can start working with it.
  
  
• Going to work today on figuring out if our existing ATX-type power supply for the FPGA board can also power the GPS and the EZURiO board. First need to see what voltages are available on the extra cable. Downloaded datasheet, it says that 3.3 and 5V are available, but it is not very informative regarding the cable pinouts. Going to use multimeter to test.
  
  
• Found these voltages on primary connector: 2.7, 5.1. Found these voltages on secondary connector: .44, 1.7. Not sure why I'm not finding these other documented voltages: 3.3, +12, -12. Wonder if what's supplied is conditional on how it's connected. Or, maybe I'm just not using the right grounds to measure against. The different grounds seem to be internally connected though. Perhaps the best thing is just to buy another ATX supply.
  
  
• Our power supply needs are as follows:
  - EZURiO Wi-Fi board: 5V, max current unknown (need to experiment) but must be within the limits of USB.
  - GPS module: 5V USB -or- 6.5V AC/DC adapter block -or- 12-to-6.5V car adapter. Max current unknown but adapters are limited to 600 mA
  - OCXO: 3.3V, max current about 850 mA.
  
  
• Juan came by and Mike & Juan worked together to debug latest firmware changes (command processing). STOP, GO, RESET, RESTART commands have all been verified. There was a bug where GO wasn't turning the sync pulser back on, but that's fixed now.
  
  
• Now Juan is working to help debug Gordon's parser code. The docs for Pattern confirm that "\\*" is indeed the correct literal-escape syntax. Aha, it turns out that there was a line with a missing "*" character. Modified code to skip these lines and it now processes the data file with no errors. However, records that have bad checksums aren't getting handled properly in the output. Suggested to Gordon that we need to include an extra field in the output which indicates whether there were errors.
  
  
• Another thought: If we turn down the baud rate of the communication stream to/from the GPS (which has to go through our crufty level-shifter), this might reduce the serial errors. Alternatively, we could replace the hand-wired level shifter with the small PCB we bought from SparkFun. That might help a lot too. Day's almost over, so [ ] do this tomorrow. Also, starting a data-collection run before heading out - just for fun.
  

Serial I/O Improvements To Do

• Spoke to Gordon on the phone, asked him to try to give us an ETA on finishing the Java code. His problem at the moment is that he has no Internet at home and no laptop. He may be able to do the work using machines at CS, but isn't sure yet. He also has very little overlap in his schedule with Mike. Alternate Fridays look like the only option.
  
  
• Facebook friend Kragen pointed out that I really should be using memmove() instead of memcpy() in my line-buffered input routine, for extra safety in the face of possible future reimplementations of the Nios version of the newlib API. He is right, and this is an easy fix, so do this ASAP [ ].
  
  
• Kragen also suggested I handle EOF explicitly. Right now, with the default UART driver, there is no way to detect an EOF-like condition. But, I could add gelware to route the RS-232 DTR pin to an Avalon PIO peripheral (which could also generate an IRQ if I want). This would then allow me to check explicitly for a remote device power-down or physical disconnect in my I/O routines. (The more "elegant" way would probably be to write my own custom device driver, but that may be too much of a pain.) However, the problem with even trying to handle this contingency with maximum gracefulness is that it requires significant extra software complexity. The best we could do, if the Wi-Fi board goes down, is to buffer up the real-time data until the board comes back up, then stream it to the server as fast as possible until we catch up. We could actually buffer up quite a large amount of data - maybe even an entire run's worth - if we use the DRAM SIMM card, and/or an SD card. If the Wi-Fi goes down and no one notices for a while, this could save our @$$ if we are trying to do a long data run. So it's probably worth doing eventually, but still not a top priority - since our main goal right now is to get the demonstration system working before the grant runs out, which (worst case) is the end of this summer.
  

Re-entrant output routines

• Juan is here and I am showing him my current issue w.r.t. trying to import or re-code re-entrant version of the STDIO output routines fprintf() and/or fputs(), to allow the remote command routines to produce output w/o interfering with the output from the PPS ISR. Juan is researching what it would take to import the latest version of newlib into our environment, while Mike studies what it would take to re-implement.
  
• Ray came by and we talked about plans/strategy/schedule. He wants to be ready to put demo in CLC by end of year. I'm not sure we can get that far along in time.
• Juan found the newlib source used in our system. It appears that fputs() may already be re-entrant. So we are just going to try using it. Ditto for fprintf().
• Mike thinks he found a way to burn the design into Flash! 1st try didn't work but we should try again later.
  
• Code compiles now but is hanging upon startup. Juan and Mike went to lunch to mull things over. At lunch, Mike realized he probably just should have initialized his re-entrancy structures (still used in _printf_r()).
  
• After lunch, Mike looked at the reent.h header file, and sure enough, there is a macro _REINT_INIT() that should have been used in the structure initializer. With that done, the code works perfectly now. Well, no... There's no STDOUT output! Maybe I should have initialized those fields of the re-entrancy structure explicitly. OK, that's fixed now.
• Mike wrote a command processor for GO/STOP/RESET/RESTART commands, and added support for them to the Wi-Fi autorun script (to pass them through from the server), but none of this is tested yet. First, the autorun script needs to be recompiled & reloaded onto the Wi-Fi module. But it's almost 6:30 now, and Mike needs to head home. Finish this next week.
  

GPS app command input

• My main goal for today is to try to get input from both serial ports working. I decided to start by putting the read ports in non-blocking mode and polling - that seems easier than a multi-threaded approach, which might require a whole RTOS or something.
• Gordon is here and is working on the version of his code that computes & checks checksums on all lines, since my data files now have those even on the new custom $PPSCNTR lines.
• At the end of the day, Gordon had almost finished fixing up his code, and I (Mike) was almost finished with the GPS app changes. I got non-blocking fgets() calls activated on the serial ports, but now I am having a problem where the re-entrant versions of the fprintf() and fputs() routines that I use for serial output don't seem to be available. I might have to rewrite the ouput code to use sprintf()+fwrite() or some such.
  

Cleaning up code.

• Today I'm working on cleaning up the code (now that it is working again) to remove unnecessary diagnostic code & flushes. However, since I already have the 7-segment output facility, I'm going to use that instead to display the clock discrepancies & display genuine error codes.
  
• That is basically done now; at present the system alternates between displaying the discrepancy after receiving the PPS pulse, and displaying "yo" after receiving NMEA text data from the GPS to retransmit.
  
• Emailed Gordon to remind him to process the data file from last month, which is available now. I also have another file now that I collected over the last couple of days, which might be worth looking at.
• Next up: [ ] Rejigger the UART from the Wi-Fi board to see if I can have it generate an IRQ; if so, then we can do a remote command processor. I have to think carefully however about how to manage the concurrency - we need to make sure that certain operations like serial output are atomic. Perhaps I should use an RTOS like MicroC/OS-II, which provides concurrent programming abstractions like threads, semaphores, etc. Or, I could maybe just roll my own minimal facility for serializing serial output. Hmm... Let's not get ahead of ourselves here; at this point I'm not even sure how to enable a serial receive interrupt in the context of HAL and the default UART driver. I might have to write my own driver! It might be simpler/preferable to just set up a polling loop to just constantly do nonblocking reads on both UARTs.
  

Settling back in.

• Gordon tried to come in this morning, but we missed each other. We exchanged some schedule information and he will try again on Wednesday. Our plan for this month is to process the big dataset I collected last month and prepare some nice graphs in Matlab to use in papers.
  
• Haven't heard anything yet on status of time entry for the last pay period. Hope I get paid!
• Plan for today: Get back into gelware/firmware, work on fixing serial comm problem.
• Serial crashing problem is still there, & still as bizarre & intractable as ever. It could conceivably be a stack problem, since I only have ~1800 bytes available for stack+heap. Or maybe a buffer overflow. But it could also be just a loose connection or crossed wires. Thinking about getting some IDE cables from Best Buy, I think they would fit the GPIO headers. Also maybe will get some individual pin sockets from Fouraker; they would make for much neater connections than my wire-wrapped/soldered BS. Also, need to look at the compiler warnings from Quartus.
• Whadda know; I just doubled the RAM size from 64K to 128K, and that seems to have completely solved the problem! I guess before I must have just been running out of memory, and the stack was stomping over the heap, or vice-versa. Hooray!
  

New blog for work log.

After more than a year's worth of struggling to maintain a daily work log on Google Sites, it occurred to me that I really should just create a blog instead. Thus, this blog. This is for logging my work for Ray O'Neal at the FAMU Physics Department, on the COSMICi Project as well as other related projects.