Fri., Mar. 23rd

I'm not sure if anyone else is coming in today.  Darryl is out sick.  David isn't coming today.  Aarmondas usually comes in from 4-5 on Fridays, but I have to leave early (4:30) today.  If Aarmondas shows up before I leave we will go over the FEDM rewrite together and he can pick up where M.D. left off yesterday.  I think one of the ME guys might drop by at some point?  (They didn't.)

Meanwhile, I think today I will just continue working on the startup sequence, and the server-side code for parsing the FEDM data and reading it into data structures.

Samad came in and we are working to get started on the new OCXO board design.  I put the files for the old OCXO board design on Dropbox in COSMICi_devel/OCXO_board/Old_CTU and shared OCXO_board with Samad; the new board design will be created in another subfolder of it.

We first tried to see if he could use PADS remotely on the ACER through Windows' built-in Remote Terminal support, but apparently it doesn't support having multiple people logged in simultaneously.  Guess you'd need some Server version of Windows for that.

So next, Samad is installing PADS on the ACER, and we will see if it can use the license hosted by the FlexNet server on COSMICi, at port 27000 (lmgrd process).

I spent some time with Samad going through various details of the old OCXO board design in PADS, and showing him how to use various features of the PADS software.

Samad hasn't yet received the new instance of the power distribution board from Donte (the old one had some broken rings around the vias); when he gets the new one, he can solder the various headers onto it and then we can test it in practice.

Thu., Mar. 22nd

Came in briefly from ~1:45 - 2:15 so that Michael Dean could get in to continue work on the restructured FEDM gelware.  Check Michael Dean's blog post for details on what he did.  Later in the day I left a comment on his blog post.  We probably need to go over the new design carefully next week and check it for correctness before we try compiling.

Stopped back by at ~6:15 pm to make sure door was locked.  It was.  Came in for a few minutes anyway to check email and take notes.

Wed., Mar. 21st

Juan is here, and working on finishing up Michael Dean's changes to put all of the high-speed components into a single entity under the top-level schematic.

I am going to install my current Q:\ compile onto the FEDM.  The PLL frequency is 300 MHz.  The Fmax from this particular compile is 340.02 MHz.

Spoke to Ray about forthcoming goals:

Hardware goal:  Mounting hardware/brackets installed in CLC by April 7th, also electronics mounted in case, cooling system installed but not glued to the chip (thermal paste OK if easily removed).  We may need to wait a little longer than this to actually move the electronics box to CLC, in case we are still doing development work on the system.

Today I will continue working on the server-side code to support the FEDM.  First trying to see if my changes to recognize the FEDM messages are working.

[ ] Still need to fix the GPS Manager so that it automatically turns POSHOLD mode back on after a manual reset.

The GPS is acting a little odd today - it keeps eliminating all satellites from the timing solution.

Having an issue that we're not seeing the HOST_STARTING message from the server b/c it's sent before the Wi-Fi is ready?  Let me try again.

For some reason, the UART bridge connection from the FEDM board keeps closing itself shortly after it opens.  Not sure how.  Maybe a buffer overflow in the bridge implementation in the Wi-Fi firmware, caused by too much data being streamed to the Wi-Fi board before the bridge is fully established?

Having trouble even getting that Wi-Fi board (node #1) to establish a connection now.  Wonder if the script got erased?  That happens sometimes.

OK, the disconnects were a server bug.  Fixed that.  Now I'm having another problem: Flaky connection from the OCXO, which has come unsoldered again, for like the 3rd time.  We   However, by holding it in place manually I managed to get some pulses.  Here are a couple of adjacent output lines from the period during which the OCXO output seems to have been connected:

NC_PULSES,26595,23828622948,23828752409,118,78,203
NC_PULSES,28763,24055954150,24056162251,135,84,201

Let's just look at the two time-reference data points (first two fields of each line).  The difference in sync pulse counts is 28,763 - 26,595 = 2,168.  Divided by the new nominal sync pulse frequency of 2,861.022,949,218,75 kHz gives a time interval of 0.757,770,922,67 seconds.  Meanwhile, the difference in PLL clock cycle counts was 24,055,954,150 - 23,828,622,948 = 227,331,202.  Dividing that by the current PLL clock frequency of 300 MHz also gives 0.757,770,673 seconds.  The two clocks (OCXO vs. FEDM's TCXO) are thus out of calibration with each other by only about 0.33 ppm; not too shabby.

So, this verifies that the PLL cycle counter on the FEDM really is working at 300 MHz in the current compile, so that's good.  However, there were a few outputs from the pulseform-capture datapath that seem to have got corrupted somehow, e.g.:

Wed Mar 21 17:19:32 2012 + 697 ms: < CON_PULSE,0,0,1,1,1026250529,3,(0,(1,(2,6),7),7)

Wed Mar 21 17:19:32 2012 + 698 ms: < CON_PULSE,0,0,3,1,1026250530,4,(0,(0,(1,(3,5),8),6),11)

Wed Mar 21 17:19:32 2012 + 699 ms: < CON_PULSE,0,0,2,1,1026250531,3,(0,(1,(1,11),6),3)

Wed Mar 21 17:19:32 2012 + 701 ms: < CON_PULSE,0,0,2,2,1026250531,2,(0,(443296,7),4)

What happening here?  First, in the first 3 lines, we see a nice-looking shower event where all three detectors cross the 1st threshold (-200 mV) within 2 PLL clock cycles (i.e., 6.7 ns) of each other, and all of them cross 3 or 4 thresholds with roughly similar-looking patterns.  Then, we get a spurious 2nd pulse from channel #2 that supposedly starts in the very same clock cycle as the previous one (how is that even possible?), and only crosses 2 thresholds, with an anomalous time delta (1.48 ms) between crossing the first and second threshold (this could very easily be the time delta between two completely separate pulses).  Something is clearly screwy there!  And this same kind of glitch seems to happen during most of the other shower events.  I didn't notice these kinds of glitches when we were running at 200 MHz, so possibly there is some kind of timing issue at work.

Hm, I wonder if I should add a synchronizer chain when feeding the low-speed data consumer's (cs_combine's) handshake-acknowledge signal back into the high-speed front-end pulse-capture module?  Without that, there are possible metastability issues that could come into play, which could conceivably destabilize the updating of the pulse-cap module's high-speed state machine temporarily, possibly causing the spurious extra outputs.  This might be worth a try.  However, the day's about over so we'll worry about that tomorrow.

The ME students came by at one point today and took some more measurements for the chassis construction.

Tue., Mar. 20th

David is here.  Dr. O'Neal brought in his Windows 7 CD so David is using it to install Windows on the new hard drive for the Acer.  Unfortunately it wouldn't accept the license key (the CD is Professional, the license key is for Home), so we might have to tweak the installation later to downgrade it from Professional to Home, or something.  David is downloading a new ISO of W7 Home Premium (non-SP1) which he will burn onto a DVD-R.  We seem to be having a problem where this download keeps getting interrupted right at the end.

Darryl should be here later, and we can all talk a little about the paper if needed.  We were going to collect some new data to include in the figures in the paper, but it might be best to wait until we finish boosting the FEDM speed to 500 MHz so that the data we collect can come from the actual system (as opposed to a simplified mock-up).  An exception might be the figure showing an input PMT pulse next to the comparator outputs, since we don't actually need the high-speed clocks for that one.

Michael Dean is here.  He & the other Senior Design computer engineering students can work on making a single top-level entity for all of the high-speed components, which I think is the next logical step, since the lack of such an entity is the only reason I can think of why the speed of the high-speed stuff still isn't getting preserved when we add the other stuff - it's possible that, even though the individual high-speed instances are logic-locked, the routes between them aren't getting logic-locked currently.

Meanwhile, I could work on the FEDM model/proxy code, which has barely been started so far.  Basically I first just need to add the handlers for the various FEDM-specific messages.

What to load onto the FEDM for testing?  Looks like all my compiles from yesterday had the PLL speed set to 250 MHz.  I should be able to turn it up to at least 300 MHz based on the last Fmax from yesterday.  What if I compile with PLL=500 MHz, does that affect Fmax at all?  Let's try it...

Fast stuff only:    552.49 MHz.
Add slow stuff:   307.22 MHz.  Boo.

On the FEDM model coding:  Currently, we need to add support for these 3 message types:

• NC_PULSES - 6 arguments
• FIFO_FULL - 4 arguments
• CON_PULSE - 7 arguments (last one has nested commas in parens)

Modified fedm.py to dispatch these 3 methods to (currently still empty) handlers, so that at least we won't get error messages for them.  David tested COSMICi_server.py to make sure it runs on Windows 7 - it does (at least up to the starting screen).

Meanwhile, Trying another compile on laptop.  That one had the pipeline registers.  Tried merging partitions after LogicLocking; that caused the multi-hier partition to get a LogicLock icon.  Maybe that will help.

Fast only:  564.33 MHz.

Add slow:  335.23 MHz.  (I think this is the best yet with the full system in there?)

The placement effort level on the laptop wasn't turned up all the way.  Fixed that; redoing.  Meanwhile, another compile on PC, with pipeline registers and entire merged partition in LogicLock:

Fast only:   582.07 MHz.
Add slow:  266.24 MHz

What did I do wrong?  Failed to recompile Top from source?  No, that only got us to 266.81 MHz.  Hm.  Not sure why results are so different between PC & laptop compiles right now.  Some minor settings difference?

Laptop:

Fast only:   596.66 MHz  (Is this about the best so far for just the high-speed components?)
Add slow:  345.9 MHz  (Is this the best so far with the full system?)

Now on laptop got 346.48 MHz for the whole system.  Can we hit 350 MHz?  We're so close!  No, blah, this time I only got 321.96 MHz!  Yuck!

Trying compile on desktop with less-aggressive settings for the top partition.  Maybe this will help now that the multi-hier is logic-locked?  No, blah.  Made desktop settings same as laptop.  Now getting 338.98 MHz.  At least it's over 300.

Mon., Mar. 19th

David is here.  The new hard drive came in and David installed it.  Now we just need to find the Windows 7 boot CD.  Ray has one at home - asked him to bring it in.

Juan is going to be here Wednesday instead of today due to a conflict with his RA job.

Aarmondas & Samad came in.  Samad had his PCB fabbed by Donte, but there were some fabrication problems so Donte is going to redo it.  Aarmondas is experimenting with compiles.  David is looking through Altera docs trying to find new things to try.

I heard the guys did some more work last Wednesday, but did not yet get the FEDM input-capture working at 500 MHz.  David is checking the group blog.  Juan says he tried changing the fitter placement effort but it only helped a little.  He said he didn't take out the extra pulse-cap instance yet.

The last test compile in Q:\ got 334.9 MHz.  Or no, that was with the high-speed components only?  Not sure; need to recompile.

Let's try:

• Take out 6th pulse-cap from the pulseform-cap module.
• Turn off "optimize multi-corner timing" (we only care about hot corner (worst-case) for now).
• Change fitter placement effort multiplier from 2.0 to 4.0.
• Change fitter routing effort multiplier from 2.0 to 1.0.  (These are the settings that worked best for the CTU app.)
• Router timing optimization level:  Leave at Maximum.

OK, first I merged all the high-speed parts, and am compiling them from source with Top set to Empty.

Then (if that meets the speed target), plan is to unmerge them, logic-lock them, set them to compile from post-fit netlist (strict with placement/routing preservation), and set top to Source and recompile.

OK, we got 577.03 MHz for the high-speed stuff.  However, I'm not sure if this approach will really work - splitting the merged partition looks like it might be destroying the post-fit results.  If this doesn't work, I'll try it again and next time I won't split it; will instead add all the instances to LogicLock manually.

Another thing to try is turning down the aggressiveness of the compile for the non-time-critical parts.  It's possible that when it's aggressively optimizing those, it encroaches on the time-critical parts.

Interrupting the current compile b/c it seems to be stuck in the middle of analysis & synthesis - it's been at about 50% for 20 minutes.

OK, compiling just the high-speed stuff as a merged partition again.  Then I'll apply the LogicLock, manually using wildcards if I have to.  Then add in the low-speed stuff with relaxed compile settings.

I'm learning how to use wildcards more cleverly (I think) so that there are fewer separate entries that need to be added in the LogicLock list.

For the slow-speed logic, I relaxed the overall fitter setting from "Standard" to "Auto," reduced placement effort multiplier to 1.0, and turned router timing optimization level to Standard.

I could also try turning off physical synthesis optimization for speed in the low-speed compiles.

Bleh, 295.95 MHz.  Even worse than before!

Speed is currently 565.29 MHz for just the high-speed stuff.  Let's add the slow-speed stuff in without turning down the optimization settings at all.  Still got my high-speed stuff in LogicLock.  Compiling now...

Argh, 295.95 MHz again!!!

Trying one more thing:  Creating a reserved rectangular LogicLock region for the high-speed stuff.  This way it won't even be able to try optimizing other stuff by changing stuff in the LogicLock region.  So, hoping that might help.  However, I'm not sure the design will even still fit if a rectangular region is reserved - there might not be enough cells left, it was a really tight fit before - or that the high-speed part will still be fast enough.

OK, that last worry was unfounded - speed in rectangular region is 567.21 MHz.  It reserved a big region on the left side of the chip.  Now let's try adding the slow-speed stuff, outside that reserved region.

Nope, we are 160 RAM cells short.  I could try taking out the pipeline registers, there are 4x2x56 = 448 bits worth of those, so those might be enough to allow us to fit again.  However, I'm not sure whether it will run fast enough without them.  Worth a try though.

Another thing to try:  The compile without the merge, move all the instances individually into LogicLock, then turn down optimization for compiling the rest.

For some reason, analysis & synthesis always seems to hang on this computer when all the high-speed instances are in separate partitions.  (Something to do with parallelizing the compile on multiple CPUs, perhaps?)  So anyway, I'll skip that idea for now.  (It might work on another computer though.)

In the meantime, I'll try again taking out the pipeline registers at the front of the datapaths.  First, from source with no logic-lock.  Got 572.74 MHz.  So those front pipeline registers were never really necessary!

Now, from source w. HS elements in root logic-lock region.  Same speed.

Now, from source w. HS elements in their own logic-lock region (Region_0).  569.8 MHz.  No, wait, that logic-lock region came out the wrong size for some reason.  Redo so it's the right size.  Now 582.41 MHz!  Not sure what changed to get that.

Now, turn on Reserved, turn on post-fit (strict) for HS partition, switch Top partition from Empty to Source, and recompile...

Nope; still 160 RAM cells short!  I'm surprised that taking out the pipeline registers didn't seem to help at all.

Let's go back to putting things in the Root logic-lock region, non-reserved, and see what we get (haven't tried that since taking the pipeline registers out).  582.41 MHz still.  Adding low-speed parts & recompiling...

Another thing to try:  Put all the high-speed logic in a single entity.  It's possible that the instances in the logic-lock list are being preserved, but the routing between them isn't.  Putting them all into a single entity that is itself logic-locked might help.  This requires substantial reorganization of code though.  Still, it's a lot easier than revamping our whole data representation.  Probably makes sense as the next thing to try, if my current compile doesn't work.

304.23 MHz for the whole thing.  Still no dice.

I think the next logical step is to try putting all the high-speed stuff into a single entity under the top-level schematic, which can be in its own partition, and logic-locking just that entity.  It will have some humongous number of output ports, but no matter.  This is what worked for me when I was doing the other app (CTU GPS app on the DE3).

That's a big job, so this is a good stopping point for today.  Let the students work on it...

Tue., Mar. 13th

To do today:

• [ ] Do a test run with new FW burned into FEDM. - Last night's Quartus compile stalled.  Finished it up.  Burned it onto board.  Got a high rate of garbage output - looks like PLL was set at 500 MHz.  Turned it down to 250 MHz (slightly above hot-corner speed of ~249 MHz) to see if that works; recompiling now.  Quartus keeps crashing!  I think my incremental compilation settings are confusing it.  More below.
• [/] Ray should drop by after his class to sign timesheets.  Get it signed & turned in to Sonja. - We signed 'em and David ran 'em over.
• [/] Reserve rental car for SEALER trip.  - Done (Avis), printed confirmation.  Hope they don't need a credit card - maybe I should call them later.  OK, the terms on the reservation say they do take debit cards at this location.  Printed out terms to bring with me in case there's any question.
• [/] Installing Quartus on laptop. - Got full 91 base; adding SP2 now.

Other notes:

* Juan/Aarmondas's "high speed" partitions had some extra stuff in them.  Mike Dean is fixing that.  He also stubbed out the 6th pulse_prep in each pulseform-capture datapath to make more space & is putting the pipeline registers back in.

* I discovered that you can merge partitions.

* Seems to be necessary to use LogicLock also (not just partitions) to preserve performance - I tried using partitions without LogicLock, and Fmax slowed down when I added the less performance-critical logic back in.

* I tried dragging the merged partition into the LogicLock region, but it seemed to only add one instance.  I went through and added all the high-speed instances to the LogicLock region using wildcards.  Hope I did it right.

* After adding the slow-speed logic in Top to the merged partition, Fmax slowed down again to 300-something.  Guess I could go ahead and burn this version since I'm only asking for a PLL speed of 250 at the moment.

* Quartus has been acting really odd today.  It keeps hanging at specific places.  Maybe I need to reboot.

* Rebooted desktop; also copied Q:\ contents to laptop (under my desktop in q91sp2\FEDM\).  Amazingly, Quartus seems to run significantly faster on my new laptop than it does on my desktop!  Maybe it's the combination of the i3 processor and the SSD.

* Michael Dean left Quartus compiling under VirtualBox - this is a test of the logic-locked high-speed logic, in a post-fit (strict) partition preserving placement & routing, with the slower-speed stuff added back in around it.  Unfortunately, the Fmax at the hot corner only came out to 311.92 MHz.  So, something is still not right.  Did we forget to include one of the modules clocked by the PLL clock in the stuff that is included in the partitions & the LogicLock?  It looks to me like he got everything, although someone else should probably double-check.

One thing though:  The pulse_cap modules for the 6th threshold could be removed from the partitions and the LogicLock, since we are not using them anyway.  This might help - since it reduces the number of instances of that module from 18 to 15, and makes the fitting easier.  (Those modules aren't getting automatically eliminated, since they are locked in.)  I actually did this in the copy I'm working with on my laptop -- we'll see how that compile comes out.  - That one yielded 324.04 MHz; a little faster but not much, and still far from our 500 MHz target.  We'll have to play with it more another day.

Another thing to try:  Turn the fitter placement effort from the default 1.0 up to 4.0.  This helped (and seemed to be necessary) in my DE3 GPS app.  It's worth trying in the FEDM code as well.  Of course, this will make the fitter run even slower than it does now, but hey...  You gotta do what you gotta do.

Currently, placement and routing effort are set at 2.0.  I found in the other project that it actually did worse at 2.0 than it did at 1.0; but then it got better again at 4.0.  The same might be true here.

Mon., Mar. 12th

Things to maybe work on today/tomorrow:

• [ ] Get timesheet signed (turn in tomorrow). - Forgot to give it to Ray today before he left but can do it tomorrow.
• [.] More testing of GPS initialization code. - Some incremental improvements.
• [.] Maybe start developing the (very important) Timekeeper module. - Wrote file header.
• [ ] Maybe work some more on the automated startup-sequence code.  As things stand right now, we get through GPS initialization; but then we also need to wait for the FEDM to start up, and then start generating timing-sync pulses and sending them to it.
• [ ] Write up some notes for the students on things they can work on later in the week while I am away.
• [ ] Consider maybe starting a time data collection run before I leave town, so it can be running while I'm away.  One issue:  Will it keep running even if I'm logged out?

Juan came in and I showed him how to use Design Partitions.  He's going to try that for the FEDM design.

Aarmondas is here.

Tomorrow David & Darryl should be here, and we can all talk about the paper, maybe go over my latest round of markups which I sent to them over the weekend.

Remark:  Sometimes the GPS module eliminates all (or all-but-one) of the satellites from the solution.  This seems silly, since what then is it basing the "solution" on, and ?  In such cases, it might be good to do a hot restart.  Not sure yet; need more experience with this case.

Currently, if you invoke a restart manually, the server does not automatically turn on TRAIM and POSHOLD modes again, because it thinks they are already turned on.  Just fixed this in the case of TRAIM; haven't yet done it for POSHOLD - first need to subscribe to POSHOLD messages.

I wrote a header with a description for timekeeper.py, but haven't written any of the code for it first.  It will take some time to develop, b/c I have to research topics like how to do database interfaces in Python, how to draw graphs in TkInter, etc.

Before we start trying to archive real data in databases, it probably makes sense to work a little more on the startup sequence so that we can start runs in a more automated fashion, and also because the timekeeper will need to be informed when the start of the run actually happens - getting the startup sequence automated will help with that.

Let's get set up to do a full test run, so we can see where we are in terms of the startup sequence.  The GPS is already on and (last we checked) acquiring satellites.  So now, just:

1. Plug in PMTs.
2. Put cooling block on FEDM.
3. Power on CTU Wi-Fi.
4. Power on DE3.
5. Power on FEDM + fan.

At this point, the FEDM goes thru its startup (HOST_STARTING, DAC_LEVELS, HOST_READY) starts spitting out various operating messages:  NC_PULSES, FIFO_FULL, and real PULSE messages.  (There also are a couple of ACK and ERR scattered in there, in response to the WIFI_READY message from the EZURiO module.)  Initially, all the pulse messages start with 0,0 meaning that no timing-sync data has been received yet, until one sends the "HOST START" command to the CTU.

Now, really, as soon as the server gets HOST_READY from the FEDM, it is OK to go ahead and send the "HOST START" command to the CTU.  Now, where to implement that behavior?

By the way, should NC_PULSES also include the most recent timing-sync data?  Probably so.  OK, did that (loaded new FEDM FW via JTAG; not yet compiled into Quartus programming files in Q:\).

Should we also be putting timestamps on our FIFO_FULL messages?  It might be a good idea.  OK, that's done too.

I'm renaming PULSE message to CON_PULSE (given that coincidence filtering is turned on), since this makes this message the same length as FIFO_FULL and NC_PULSES, which makes for a prettier display.

OK, this is looking pretty good now.  Here's some FEDM output right after starting the CTU:


NC_PULSES,1618,3352941653,3352948689,115,68,203
FIFO_FULL,2392,3407048110,3,1
CON_PULSE,3186,3462552667,3,1,3462608072,2,(0,(2,2),5)
CON_PULSE,3186,3462552667,2,3,3462608072,2,(0,(1,4),2)
NC_PULSES,3286,3469543165,3469557613,93,70,200
FIFO_FULL,4738,3571045202,3,1
NC_PULSES,4848,3578734750,3578768009,94,58,206
CON_PULSE,6044,3662341109,3,2,3662381026,2,(0,(0,8),3)
CON_PULSE,6044,3662341109,2,4,3662381026,2,(0,(1,5),2)
NC_PULSES,6559,3698342174,3698393175,85,65,200
CON_PULSE,7476,3762445043,2,5,3762466512,1,(0,4)
CON_PULSE,7476,3762445043,1,3,3762466515,1,(0,2)
NC_PULSES,8491,3833398601,3833429056,112,78,200
NC_PULSES,10654,3984603078,3984613854,118,71,201
NC_PULSES,12294,4099247249,4099252435,105,65,204

Let's sanity-check the relative speeds at which the sync-pulse counter and PLL clock cycle counter are increasing.  

Currently, with the 375 MHz PLL clock in the CTU, its dual-edged counter is incrementing at 750 Mcps.  The low 18 bits will take 2^18 = 262,144 counts to roll over, which is 349.5253 us (i.e., we now have a 2,861.023 kHz frequency for the timing-sync pulses).

Meanwhile, my version of the FEDM is still clocked using a 200 MHz version of the PLL (students still are working on the speedup task), so the PLL counter increments every 5 ns.

OK, so let's look at the time interval between the first two NC_PULSES message above (each pair of NC_PULSES should have about the same time interval between them, if there's a steady rate of background pulses):

NC_PULSES,1618,3352941653,3352948689,115,68,203
NC_PULSES,3286,3469543165,3469557613,93,70,200

So, in terms of the timing-sync pulses, we have 3,286 - 1,618 = 1,668 of those.  Multiplying by 349.5253 us gives 583.0082 ms between the last pulses reported in each of these two reports.  (This implies that the approximate pulse rates on the three input channels was 160 Hz, 120 Hz, and 343 Hz respectively.)

In terms of the PLL cycles, we have 3,469,543,165 - 3,352,941,653 = 116,601,512 of those.  Multiplying this by 5 ns gives 583.0076 ms.

In other words, the figures are only different by only roughly 1.03 ppm, which is in line with the specified frequency calibration of the FEDM board clock, which IIRC was about 1 ppm.  The additional excess could come from other sources, such as the OCXO clock, whose frequency calibration is usually off by about 1.3 ppm.  (Actually, this result suggests that the FEDM board clock and the OCXO clock have their frequencies off in the same direction, so that the relative discrepancy is less than that for the OCXO by itself.)

Anyway, this result is a good validation test, and is evidence that the FEDM is still not missing any timing-sync pulses (if it even missed one, then the two times would be off by hundreds of ppm).

It would probably be a good idea at some point to do a diagnostic graphical display showing the rates of non-coincidence pulses on the three detectors as a function of time.

Perhaps I'll start an overnight data-collection run as a test?  Actually, let me wait and not do it right now - first I want to actually burn the new FEDM code onto the board.  Starting a Quartus compile in Q:\ to prepare the new programming files.

Of course, at this point, all of the FEDM messages are still unrecognized by the server, so before we can do anything with the FEDM data (like graphing it) we have to remedy that situation.  More server programming...

Juan & Aarmondas have been working on trying to get the design partitions working in the FEDM.  They ran into some fitting issues which Aarmondas is working on.