Tue., Apr. 3rd (AM)

Woke up early this morning and couldn't get back to sleep, so I got up and finished the coding work to automatically start the CTU as soon as the CTU and FEDM hosts are both ready and we are getting accurate time values from the GPS.  The new code still needs to be tested, of course.  Once it's working, the server should automatically start the run shortly after all of the boards are turned on and ready.

However, the Timekeeper and DataCollector modules still have not been implemented, so we still are not really doing anything meaningful with the data, such as processing & visualizing it.  To recap, the functions of these modules will be:

• (in timekeeper.py) timekeeper.Timekeeper - Worker thread class.  The Timekeeper gets created just before the CTU is started up.  Its function is to take in all of the data from the CTU (including both PPS  time-references and GPS data records) and do several things with it:
• (1) Store it in memory, at least for a while (although older records may be rotated out of memory eventually), and also store it persistently in a database file that can be queried offline from other processes;
• (2) Based on the stored data, provide a service of reconstructing accurate absolute times (ideally to an uncertainty of only a few ns) given relatively "raw" time data for an event.  The CTU currently marks time in 4/3 ns intervals from the start of the run - so the core function would be to take some number of "ticks" of this clock (>0) and return a data structure representing the corresponding absolute time in some more standard format (day, hour, minute, second) with few-ns accuracy.  The function from ticks to absolute time should be monotonically increasing, but is not necessarily perfectly linear, since there may be some gradual drift of the OCXO clock frequency over time which will need to be measured & accommodated when shaping the translation function.  Also, the function is not necessarily unchanging, since as we receive more data, we may continually refine the mapping.  Ideally the output structure would also provide some idea of the uncertainty of the time value, based on the current data.
• (3) Display some kind of real-time graphical visualizations of important data having to do with the timekeeping function.  This could include, e.g.:
• (a) Graphing the history of key information reported by the GPS, including the number of satellites we're currently receiving data from, and the number that are "good" vs. "bad" in terms of whether their time data has been eliminated from the timing solution.
• (b) Graphing the time-deltas between PPS pulses with a vertical scale marked according to the deviation in Hz between the implied OCXO frequency and the nominal frequency (10 MHz).  Superimpose a horizontal line showing the average frequency (over some appropriately-sized time window).
• (c) Graphing the cumulative phase wander of the GPS PPS signal vs. OCXO since the start of the run; this provides a nice visual indication of any excursions which may result from e.g. satellites going out of range, etc.  Superimposed on this curve can be another curve representing the reconstructed mapping from OCXO-based times to absolute times; the latter curve ignoring excursions due to a lost signal, and applying appropriate averaging to improve the accuracy of the inferred times beyond the accuracy of individual PPS edges.  A gray region around the latter curve indicating the uncertainty of the inferred times can also be shown.  The user should be able to zoom in & out on this graph using the mouse to inspect portions of it in detail, to examine the quality of the time reconstruction at particular points.
• (d) Graphing the Allan deviation of the two time references against each other?  Not sure if it's feasible, CPU-wise, to update this one in real time, but we can have a separate background thread that updates it periodically.
• (in datacollector.py) datacollector.DataCollector (worker thread class) - The DataCollector gets created just before the CTU is started up.  It takes in time-tagged event data of various types from the FEDM and does several things with it:
• (1) Store it (at least recent data) in memory, and also store it persistently in a database that we can separately query. (Like with the TimeKeeper.)  Data will be organized into tables based on what kind of data it is - candidate coincidence pulses, non-coincidence pulse counts, or error events.
• (2) Do real-time processing of the candidate-coincidence data via a number of steps:
• (a) From the raw data points, reconstruct a model pulse shape in some appropriate way.  Ideally this would involve some Bayesian inference of the ideal shape that best explains the data.  But in practice we will probably do some simpler least-squares fit of the model parameters to the data points.  When there are not enough data points to e.g. calculate the leading-edge to trailing-edge width ratio, we can use an average of previously inferred ratios as a placeholder.
• (b) Once inferred peak times for pulses have been inferred, cluster them into the most-likely shower sets based on how close-together in time the peaks are on the 3 detectors.
• (c) For each candidate shower event, infer the source heading (angle of incidence).  Store the so-processed data for shower events in another database.  A separate astronomy module should perhaps take care of mapping ground-relative headings and times to locations on the celestial sphere.
• (3) Do real-time visualization of various data:
• (a) Graph the number of error events (FIFO full, lost pulses) seen at each detector over time.
• (b) Graph the total number of pulse events (coincidence & non-coincidence) seen at each detector over time.  (A histogram of pulses of each height would also be nice.)
• (c) 2D polar scatter plot of the inferred ground-relative headings of all accumulated shower events since the start of the run.  (Since this is ground-relative, a celestial point source would revolve around the North star on a diurnal basis.)
• (d) Similar plot in inferred galactic coordinates on an elliptic projection - although this should perhaps be taken care of by a separate astronomy module.

Mon., Apr. 2nd

Juan is here and working on finishing up the refactoring of the FEDM gelware (separating the high-speed components into one entity).

I am getting ready to test the wireless communication through my Verizon hotspot, which is presently configured with WPA2 Personal security.  My laptop is successfully able to connect to the Internet through it.

The script seems to be hanging.  Turning on debug output to UART and trying again...  Went through a few rounds of turning on more and more options to display more output, finally narrowed things down to the attachment failing after the new security steps...  Reordered the steps (the main change being to set the key before turning on security), it works now.  Turned debugging modes back off.  OK, all the server connections open again, now over my access point & with WPA2-Personal security.

After I finished that test, the ME guys (George and Brian) showed up with the main board for the electronics box and tested the mount points for the spacers to make sure the two major PCBs (DE3 and FEDM) would fit on them.  With some fiddling, that worked.  They also identified where to mount Samad's board and the clamp that will hold the OCXO board.  They brought their hand-made SMA cables; I can rig up a test to look for signal reflections where I connect the DE3 to the scope through those cables and look at the waveform quality.  Still to do on the ME side:  Assemble bracket to hold the cooling system (I told them, make height adjustable!), drill mounting holes in Samad's power board (1/8" hole diameter, 1/8" from edge to hole center), drill holes in main board to hold Samad's board and the OCXO board clamp.  They also wanted to do another coat of paint & clear-coat, so took the board with them to do that.  Also, we still need a permanent solution for the various signal & power wires between boards (to get adequate lengths), but we can perhaps rig up something temporary for that.  Maybe I'll stop by Fouraker's and/or Radio Shack on my way home this evening and look for a solution.

Also, Samad stopped by earlier and we discussed the need to solder through-hole pins to both sides of his board since the holes are not plated.  I gave him a pack of breakaway headers with longer pins to make that easier.  He will see if he can get the mounting holes drilled at ECE.

After the kids left, I spent some time rebuilding the serial "cables" (really jumper wire bundles) to both Wi-Fi boards to standardize the wire color-codes and go through a double-sided M-M header for easier connect/disconnect and longer length.  This is still just a temporary solution, however, and something that makes a more solid/reliable connection is still needed for later.  I hooked node 0 back to the CTU (whose serial port on GPIO 0 I also re-wired) and verified that that connection still works.

Powering the Wi-Fi board through the DE3 still doesn't work reliably, so we still need Samad's new power board to supply +5V to the Wi-Fi board (or else just power it through USB in the meantime).

Next up (tomorrow):  Do the same thing (extend the cables) with the serial connection to the GPS and the other miscellaneous connections to the GPS, and the connections to the OCXO board; get whole current demo working with new longer cables.

Fri., Mar. 30th

David and Darryl were here today, and were thinking and talking with Dr. O'Neal about what to do next on the paper wrt updating the figures.

David went through part of a tutorial on PADS and learned about a lot of nice features.  Perhaps before the Senior Design students disappear, he can sit down with Samad and they can work together on designing the new OCXO board - this is an extremely simple board which will be good practice for both of them.

George stopped by with some SMA cables that he had picked up at Radio Shack, but they didn't have the right connectors to interface with our boards.  Brian thinks he can make his own cable, but I told George that they really need to order some off-the-shelf ones online as a backup, and also because making a good quality coax cable with no impedance nonuniformities (which can cause signal reflections) is in general quite difficult; making a cable that really offers a reliable high-quality connection suitable for GHz-bandwidth signals can be a challenge.  Brian supposedly has experience making coax cables, but we can't risk any more delays if they can't build a working cable.  We found 12" cables at Cables to Go (hopefully these will be long enough; 16" or 18" might also be suitable, if we can find them) and George says he will order them and they should arrive on Tuesday.  But hopefully he & Brian can make a cable before then that will work well enough to at least wire up & test the main electronics box assembly.

The main thing I worked on today was trying to configure the Wi-Fi connection using my Verizon MiFi 4G/LTE mobile hotspot.  We tried WEP 64-bit, but the modules wouldn't connect, perhaps because the MiFi would only let me enter a 10 hex digit (40-bit) key?  The Norton might have been blocking the connection, but I switched off the relevant features and this didn't fix the problem.  If I continue having problems, I'll uninstall Norton; it only has 11 days before the demo license expires anyway.  I went ahead and configured rules in Windows Firewall to open the needed ports.

In the meantime, I changed the Wi-Fi script to turn on info-level output, so I can see exactly where it is having problems.  I'm also trying to configure it for WPA2 Personal since that is the most secure level supported by the MiFi - as long as we're trying different things, we might as well try the most secure mode first.  The EZURiO does support WPA2 Personal, according to the docs; however I haven't tested this mode on it before.  I'm working on augmenting the script to support this mode as an option selectable in the site configuration.

* * *

Working while on a trip this evening, I finished the script changes to support WPA2-Personal security, and got the script to compile.  (As usual when adding code, this involved moving more strings out to the strings.txt file.)  The changes still need to be tested on a real Wi-Fi board next time I am in the lab (or at home).

Thu., Mar. 29th

It occurred to me this morning that we'll need a wireless access point at the Senior Design Fair. An easy solution would be my Verizon mobile hotspot. I can rename it "MikeRoCosm" (for Mike's Roaming COSMOS, or Micro-COSMOS) and configure it with minimal or no security and the modify the boards' Wi-Fi script to connect to it. Also, the server can run on my (or someone else's) laptop which can also be set up to connect to the AP. I can test this in lab on Friday. We need to arrive real early on Thursday to start setting up, duct-tape cables to the floor, find a good spot for the GPS antenna and the detectors, run electrical power, etc. I need to ask Donte about the power arrangements. It might be possible to write a real quick-and-dirty visualization over the next few days, maybe give that a shot. It will not be a permanent solution though.

Wed., Mar. 28th

Juan is here and working on timing optimization.  He found this nice tool called Timing Optimization Advisor that made suggestions of ways to improve the timing performance.  It had a couple of suggestions which looked promising that we are trying.

To help out, I am going to make a list of all the settings I have fiddled with:

• Compilation Process Settings:
• Use smart compilation - I don't know if this affects performance at all; it supposedly can improve compilation speed, so I've been leaving it turned on.
• Incremental Compilation:
• Incremental compilation = Full incremental compilation.  I think this is necessary for partitions & LogicLock to be helpful.
• Physical Synthesis Optimizations:
• Optimize for performance (physical synthesis):
• Perform physical synthesis for combinational logic = ON
• Perform register retiming = ON
• Effort Level = Extra
• Fitter netlist optimizations:
• Perform register duplication = ON
• Analysis & Synthesis Settings:
• Optimization Technique = Speed
• Timing-Driven Synthesis = ON
• Fitter Settings:
• Fitter effort = Standard fit (highest effort)
• More Settings...:
• Placement effort multiplier = 4.0
• Router Timing Optimization Level = Maximum

At least, these are all the settings that I think were helping.

I'm also looking at possibly changing the state encoding for the pulse_cap FSM to a one-hot encoding (this was one of the things that the Timing Optimization Advisor suggested).  Not sure yet if this will help at all because that logic was, I think, pretty well-optimized already.

589.28 MHz - Baseline speed for high-speed datapath.

455.37 MHz - Speed after switching to the one-hot encoding.  Nope, it didn't help!

Continued working on server coding.  When I left off at the end of the day, I was about to write the code for the GPS_Manager that would cause it to watch for the condition where the TRAIM is not eliminating all satellites from the solution and is reporting a non-null accuracy value (and maybe also verify that the current time as reported by the GPS is consistent with the system clock), and in this case, it should inform the application's main entity (the CosmicIServer instance) that the GPS time is good, by calling its yo_GPS_time_is_good() method, which will then relay this information to the RunManager instance, causing the RunManager to proceed to the next step in its startup sequence, which has not been implemented yet (empty method bodies for now);  that step will be to start up the Timekeeper and then the DataCollector (both of which have also not been written yet) and then (next step) start the CTU - that last step should be really easy to implement though, since all the infrastructure for it is already there.  (We can go ahead soon and test this & the rest of the startup sequence, leaving empty method bodies in the routines to start up the Timekeeper and DataCollector temporarily.)

Tue., Mar. 27th

My plan for today:  Work on the startup sequence.  Maybe create a new class/worker thread called RunManager that is responsible for overall management of the startup sequence and maintenance of the data-collection run?

Michael Dean is supposed to come in today; he can continue working on the refactoring of the FEDM gelware into high-speed vs. low-speed modules.

Darryl (if he's feeling better) & David are also supposed to be here; I can maybe work with them to collect some higher-quality data for inclusion in the paper.

Sometime, David might also work on the PADS layout modifications.  The FlexNet license server seems to be in a good mood today and David was able to run PADS on the Acer with no complaints.  I told David he also needs to install the OrCAD demo version on the Acer sometime so that he can inspect the schematics.  To modify the schematics, though, we'll have to use the OrCAD at the College of Engineering since the demo version we're using here won't let us save changes.  Sometime I need to go over with David in more depth how to actually modify layouts in PADS.

More thoughts about RunManager:

Here's what it does:

1. It waits for the CTU node to be created, and for its host to become ready.
2. It waits for the FEDM (ShowerDetector) node to be created, and for its host to become ready.
3. It waits for the GPS_Manager to report at least one valid TRAIM reading with finite accuracy and w/o all satellites eliminated from the solution.
4. It tells the CTU host to start running (generating PPSCNTR time-reference data and timing sync pulses).  (At the same time, the TimeKeeper worker thread is started, which is responsible for archiving/visualizing the time data.  Also, the DataCollector worker thread is started, which is responsible for archiving/visualizing the FEDM data.)  After this point, subsequent data output by the FEDM will be tagged with meaningful absolute time-reference data (previous data will have all "0" values for the time-reference fields).

Started writing the file, runmgr.py.

We managed to get some nice screenshots of PMT pulses crossing multiple thresholds for the paper:

Oscilloscope screenshot of a PMT pulse that crosses 4 out of 5 voltage thresholds,
spaced linearly at -200 mV intervals, together with the corresponding digital outputs from the threshold comparators.

We also have the data from this trace in spreadsheet form, so we can re-plot the traces for the figure in the paper if we want.  Darryl can work on incorporating this into the paper.

* * *

At home this evening, I spent a while making changes to the server code to parse FEDM messages.

Mon., Mar. 26th

David texted me earlier asking if he should come today or tomorrow - I told him tomorrow so he can work with Darryl on the paper, perhaps.

Ray stopped by to touch base on the project status.  We're hoping the ME students will get the wall/ceiling mounting brackets installed at CLC before the end of next week, and also get the electronics boards mounted to the chassis before the Senior Design Fair.

Juan is here, and I suggested he work on the Acer instead of the VirtualBox since it is faster, and loaned him my 8GB USB flash drive so he could transfer the files.  He is doing that.

Aarmondas is here, maybe I can start taking him through some things that need to be done on the Python code.  I re-shared the Server Code folder with him and it is downloading onto his laptop.  He's also installing Python v.3.1.4.

I saw some glitches in the 300 MHz pulseform-capture system when I tested it the other day; I wanted to work on that today (or soon).

Some things to do (coming up) on the server code:

• Put FEDM messages into data structures; publish them using the Publisher interface.
• Finish the startup sequence coding (automatically start CTU after FEDM is ready)
• Write some simple early visualization modules - e.g. graphical display of CTU time data

Back to testing/debugging.  I re-soldered the jury-rigged SMA connector onto the OCXO output (it had come detached last time I was working with the CTU - I'll be really glad when Samad and I get the new OCXO board made).

Aha, forgot to add support for the LOST_PULSES message.  Did that (skeleton at least).

OK, we're still getting the glitches in the 300 MHz data.  I had an idea the other day to try to fix this, which was to add a synchronizer chain on the handshake return in pulse_cap.  Let me do that...

OK, I added a 4-stage synchronizer chain to the hs_cons inputs to both se_pulse_cap_56.vhd and se_pulse_cap_tsedge_56.vhd.  Hope 4 stages is enough!  We'll see.  Note: Adding this synchronizer increases slightly the minimum "dead time" between separate pulses we can detect on a single input channel.  At 500 MHz, the increase will be by 4*(2 ns) = 8 ns.  However, it is already quite a bit larger than this, as cs_combine already takes a few 20 ns cycles to return its handshake.

Speed of new design:  587.89 MHz (high-speed components only); 353.23 MHz (whole design).

The change seems to have done the trick WRT eliminating the glitches.  The data we're collecting at 300 MHz looks clean now.

Aarmondas is looking at a couple of possible solutions for the database:  SQLlib (in Python), vs. MySQL.  I told him he should consider which approaches are most flexible in terms of allowing us to actively query the database from another process while the main server is still writing data to the database.  A separate database server process might be needed to do this (if we don't want to integrate a SQL server into the main server app).