It's Moanday... Ugh.

Today started with being woken up WAY too early by the stupid cat, followed by not being able to get back to sleep, and feeling rather unwell for the next several hours. Finally was able to get myself together & out the door. Then followed a >$500 car repair payment (following an extended wait in line at the overworked mechanic's shop to pay for it & pick up my car). Finally, on the way to work from the shop my car started overheating, and so I had to stop & check the oil (which was not low), and then I had to turn off the A/C for the rest of the trip (which seemed to work around the problem, for now). Stereotypical Monday!

When I got in, I checked out the status of the XP machine, which Ray said was still stuck on update 26, which is where I think it was when I left on Friday. Apparently IE8 had a dialog open and was insisting on getting a definite answer to a stupid "want to participate in sending extra data to Microsoft" question before it finished installing itself. Got past that and the remainder of the 90-something updates are now installing. Hey Microsoft, how about making your software engineers use sound coding practices, so they don't create so many holes in the first place? Then maybe you wouldn't need quite as many "critical security updates."

As you can tell, I am in a rather foul mood today. Not unusual for the day after the kids go home, I suppose. On these days particularly, I despise my sociopathic ex-wife for manipulating me into marrying her, and then taking my only child away. It seems worse than usual today though. I suppose it's the stupid cat's fault - I'm grumpy because I didn't get enough sleep. After waking me up early for no reason, she jumped up onto my (bare) shoulders while I was at my desk checking my email, and I really wasn't in the mood for it, so I just shoved her off of me and let her fall to the floor - since I was already tired & grouchy and I REALLY didn't want to deal with her probably raking her claws into my unprotected skin, which is what usually happens when she is trying to get down safely from her perch on my shoulders. Maybe she'll think twice before the next time she jumps up there.

Anyway... When I left Monday, I was about to try running the DRAM demo under the Nios II IDE debugger. OK, it is running now (getting past where it was stuck on Friday), but the memory test failed!! Each time I tried it! First time at i=0, subsequent times at i=47.... Let's try rerunning it in RUN mode instead of DEBUG mode... Now it fails at i=94... Then i=47... repeatedly. OK, one more thing to try: Recompile in Quartus to pull latest compiled firmware into the FPGA bitstream, then monitor the output from nios2-terminal. Again, the test fails, this time with i=47 first, and then i=0 each time (but with different read vs. expected values).

Ah, OK, RTFM! When I started the demo by running the Demo Batch file (in "C:\LOCAL\DE3_demonstrations\150\DE3_DDR2\demo batch\de3_ddr2.bat"), instead of manually, it worked. I guess that means I need to figure out what's different about the way the demo is being run from within the batch file... Hm... Looking at it, there seems to be nothing unusual. Let's dissect it piece by piece:

First line:

%QUARTUS_ROOTDIR%\\bin\\quartus_pgm.exe -z -m JTAG -c USB-Blaster[USB-0] -o "p;DE3_DDR2_Q8.sof"

Running quartus_pgm.exe with the --help=z, etc. flag says:

"-z | --haltcc: Halts the on-chip auto-configuration controller of the device to allow programming via the JTAG interface."

"-m ... specify which programming mode to use."

"-c ..." (obviously)

"-o ..." and "p;" means "Program"

Nothing unexpected there. Then, the .BAT file just sets up some path variables and runs Cygwin bash on the de3_dd2_bashrc script. That just runs the standard "nios_bash" script for sopc_builder (which I think the normal nios shell must do also, right?), sets up some aliases, then does:

nios2-download ddr2_TEST.elf -c USB-Blaster[USB-0] -r -g
nios2-terminal -c USB-Blaster[USB-0]

The --help for nios2-download says: -c specifies the cable (clearly), -r says to reset the target SOPC system before use, and -g says run processor from entry point after downloading. Meanwhile, the -c argument to nios2-terminal just sets the cable.

Who knows, maybe if I run it the usual way, it doesn't start from the right entry point, or something... I guess it doesn't matter because in a pinch I can always load/run my design using the command-line tools instead of the GUI, and just imitate what the batch file is doing...

Still, it would be nice to be able to use the interactive debugger... Right now it gives some kind of warning about a version mismatch... Forcing a recompile (i.e., like make clean; make) of all the firmware, to see if that helps... Nope. Now trying: Regenerate the SOPC system in SOPC Builder, recompile Quartus design, and recompile firmware... Argh, still same problem.

DRAM'ed if you do, DRAM'ed if you don't

I'm getting ready to expand my project to be able to use the 1 GB DDR2 667 MHz SDRAM (SO-DIMM card) that came with the DE3 board. ("DDR2 SDRAM" just means the 2nd version of the standards specification for interfacing to double data rate synchronous dynamic random access memory.)

First, looking at instructions in the DE3 user manual. It says that due to a limitation of SOPC Builder, only 256MB of the DRAM can be accessed if SOPC Builder is used. The demo is limited to 256 MB. So, if I want to use the full 1 GB DRAM, I might have to first learn how to build the project without SOPC Builder (using command-line tools, I assume). However, it might make more sense to first just limit the size of my memory buffer to 256 MB. That will take about 27 seconds to fill at 10 Msps, at 1B/sample. Actually, it might be wise to time the OCXO's falling edges as well; so then that's 20 Msps, and the memory will fill up in about 13 seconds (assuming we can write it that fast).

First, I'm copying the entire DE3_Demonstrations\150 folder from C:\Users\Public\Documents\AlteraDE3boardCD to C:\LOCAL\, so that I can access it a little more conveniently.

Opening the DE3_DDR2_Q8.qpf Quartus project in Quartus 9.1. (Sometime I should perhaps upgrade to Quartus 10, but for now - if it ain't broke, don't fix it. :) Looking at the SOPC Builder design. It looks like it uses a couple of PIOs to communicate with the DDR2 controller. There is also an "altmemddr" device. That dialog is pretty complicated! There are also some Tcl scripts you have to run to set up the controller IP, the source code of which is not viewable.

Also viewing the Nios 2 code. Looks like I modified it slightly earlier (just to print a message, to make sure I could modify it).

Recompiling the Quartus project now, just to make sure I can (in the new location).

OK, my modified software is loaded and running. However, the memory test seems to be taking a while... Several minutes just to test 256MB? It's running at half speed, but that doesn't fully explain it... Oh, maybe it is the slow clock speed of the Nios 2 that's the problem...

Meanwhile, regenerating the SOPC system, to make sure I can do that... Yes.

The run still seems to be stuck, recompiling the software now, to test that and so I can run in debugger...

Time to go to Gainesville, but we will continue this next Monday...

The Timekeeper Speaketh

(LOL, I'm imagining I'm a Marvel-style Cosmic Abstract called "The Timekeeper," whose job it is to measure time on all scales, from Planck scale to cosmic.)

My thinking as to next steps, at the moment is this:

1) Put the ring oscillator and carry-save counter on the DE3 board, measure ring oscillator frequency. Expecting around a GHz.

2) Try using the dual edge-triggered flip-flop to double the frequency again.

3) Reconstruct our input-capture circuit at this faster timescale, for registering 10MHz OCXO edges against the ring oscillator cycle counter. (We don't want to use the 50MHz on-board clock since it is less stable.) At 2 GHz, we could count ring oscillator cycles for >292 years without rolling over a 64-bit counter (although, since the cycle counts will always be in the general neighborhood of 200, give or take 100, there would also be no information lost by just using a 8-bit counter, and calculating the deltas by just subtracting modulo 256). If there are any metastability problems due to the short cycle time, fix them by adding some latch stages.

3) Learn how to use the DRAM in the DE3 SIMM card slot, and use it to store 1 GB of time delta data. Using 8-bit data values (adequate to cover a pretty range of deltas, from say 72 to 327), this gives us 1 billion (well, 2^30) samples. At the 10MHz OCXO frequency, that gigabyte of data will take only 100 seconds to collect. However, transferring it to the server over 56Kbaud serial would take about 2 days. Instead of doing that, I should either (a) collect less data, or (b) copy it to a file on an SD card, which I can then transfer to the server physically.

4) Do the Allan deviation computation for this dataset in C (instead of Scilab, which is too slow). In C, each pass through the GB of data (once it's loaded into RAM) should only take a few seconds, at most. Generating a thousand data points for a high-res data plot would then take around an hour, which is quite tolerable. Again, however, the range of timescales covered by the plot will only be up to 100 seconds. The minimum of the Allan deviation curve may well be below that point, though. (If not, we can set up the Ethernet on the DE3 board, and collect more data by streaming it at 10 MB/s to the server in real time while the data collection is running.)

5) Find the minimum of the Allan deviation curve for the ring oscillator vs. the OCXO. This will tell us the optimum window size T (hopefully <100 sec.) for measuring the ring oscillator frequency most precisely.

6) Implement a circular storage buffer of that size (10 MB/s)*T, and use it to store the last T amount of time worth of deltas, and also keep around a register large enough to store the running sum of all the deltas currently in the window. Add in the new values, and subtract out the old ones. This then becomes a proportional representation of the average frequency of the ring oscillator (relative to the OCXO) over the last T amount of time.

7) This technology then, I think, becomes sufficient to measure sensor event times with resolution about equal to the half-period of the ring oscillator (about 0.5 ns). We can measure threshold-crossing times by using an HSMC daughter card to do DAC conversion, to generate the threshold levels to send into the FPGA's LVDS input comparators. Then, when a pulse comes in from the PMT (or APD), we can capture that input transition, and store the threshold-crossing time, measured in ring-oscillator half-cycles since the last OCXO clock edge, as well as OCXO cycles since the start of the run. The data sent to the server can include the sum of the ring-oscillator cycle-count deltas from the most recent size-T window, which tells the server (quite accurately) the present mean frequency of the ring oscillator. Furthermore, we could even keep a running tally of the 1-cycle Allan variance over this window, which would give us an up-to-the-minute measure of the time deviation inherent in extrapolating times using the ring oscillator over the 100 ns OCXO cycle. (We could refer, instead, to OCXO half-cycles, but the problem with that is that the duty cycle of the OCXO is only specified to +/-5%, so we would have to first characterize the variability of that quantity as well.)

I think that's as far as I'm going to look ahead as to my plans for this line of development, for now. Let me get some feedback from Ray on this first.

He says he'd like to proceed with Sachin's board anyway, for now at least, but I might still spend a little time tinkering with the DE3.

On tap for Monday: Open Science Grid meeting, and installing LabView on the XP partition of the Acer (now that Juan has restored access to the Windows 7 partition and we know the state of that machine is stable). Right now, I am installing critical Windows Updates, to get the machine ready for the LabView install.

Running Rings around the Oscillators

Working from home today because my car battery is dead. :(

Today, SUCCESS at building a fast ring oscillator on the DE2 board (in VHDL). The speed of course depends on a lot of things (placement, routing, temperature), and so differs quite a bit between runs, but the fastest speed I have seen so far is 666 MHz.

I measured the frequency by building a fast counter with a carry-save architecture, which can keep up with the ring oscillator. Bit 6 of the counter goes 128x slower than the underlying clock and was measured at up to ~5.2 MHz using my cheapo home oscilloscope (which in theory can measure signals up to 25 MHz but which in practice has trouble getting above 10 or so, due I think to crappy probes.)

To get this working, I had to learn how to set the KEEP attribute in VHDL, to keep it from compiling away my ring oscillators and clock fanout buffers.

The Stratix II should be able to go even faster, I'm guessing pretty close to 1 GHz at least.

If I use dual-edge-triggered flip-flops in my counter (I should try that soon), we could count half-cycles, and so effectively measure time at 2 GHz (every 0.5 nanosecond).

So what I'm thinking now is that we could just directly sample the comparator outputs at the frequency of one ring oscillator to get nanosecond time resolution, without even having to worry about fine-tuning two different ring oscillators to be close together in frequency, and then counting cycles until the phase crossover to get sub-cycle resolution, like Sachin has been trying to do. This seems like it would be a lot easier and more straightforward.

The ring oscillator frequency can be easily calibrated against the time sync signal by simply latching the counter value when the time sync goes high, just like how we are registering the OCXO against the GPS PPS signal now.

So basically, what I'm thinking now is, we should completely redo Sachin's Quartus design with a totally different architecture. Maybe even just use the serial port for communication, like we were planning to do originally (and then we can use Wi-Fi instead of Wireless USB, and get better range). However, this might mean we have to redo all the LabView interface stuff (assuming we still want to use LabView).

If we can, also get a Nios II core on there, so that we can manage the I/O and high-level functionality in C, instead of having to do custom gelware for that part.

We do still need to interface with the DACs to set the comparator thresholds, but that shouldn't be too difficult.

Here's a video of today's test:

Weekend Thoughts

I started thinking about the project today, and had an idea.

Presently one of the major sources of error we are dealing with in our time measurements is the quantization error due to the fact that we are only sampling the PPS signal from the GPS at 100ns intervals. The standard deviation of the resulting phase error is 100/sqrt(12) = 28.9 ns.

Meanwhile, the GPS module itself uses a 16.368 MHz clock, which means the phase noise due to the quantization error in that clock is less, probably 61.1/sqrt(12) = 17.6 ns.

We could reduce our contribution to the phase error in half if we use dual-edge flip-flops to sample the signal every 50 ns; then the RMS phase error will only be 50/sqrt(12) = 14.4 ns.

However, we could do better yet if we implement our own TDC directly on our FPGA board. Spent some time today trying to do this with the DE2 board I have at home...

Now what?

What to work on today?

* Emailed Brian Stearns @ DeLorme to let him know that the Allan deviation problem is resolved.

* Maybe I should update my "Recent Results" page with the new graphs... OK, did that.

* Updated my latest graph to clean up the colors and titles & add back in some of the curves that accidentally got deleted earlier:

Untitled Blog Post

I couldn't think of a clever title today. My brain is kinda fried from being busy lately. Anyway, what to do today?

* Ray called me at home this morning to ask about Xmath status. Checked status of Xmath requisition when I got in. It is indeed marked as "PO dispatched". Should we ask Purchasing to email us a PDF so we can get it to LeCroy more quickly? Also, should we check Sonya's office to see if the license arrived in a physical package? Ray says he will call LeCroy.

* Guess I should spend most of today continuing to work on material for the journal article.

* I was thinking on the way to work about whether/how we could obtain more accurate GPS times by averaging the relative OCXO/GPS frequency over longer periods than one second and interpolating a best-fit time trajectory. According to the OCXO datasheet, the 1-sec. Allan deviation of the OCXO is nominally 5e-11. So, we'd expect the OCXO's absolute time deviation over n seconds to be 5e-11*sqrt(n/3), I think... So, suppose we averaged the trajectories over a 10,000-second (2.78 hour) period. That gives a time deviation for the OCXO of 2.89 ns. Meanwhile, the Allan deviation for the OCXO vs. GPS for frequencies averaged over that period is (from our 10-day dataset) 2.20e-9. So together, I think this means we should have no worse than about 2.9 ns inaccuracy for absolute times inferred by averaging over that window size.

We can maybe do a little bit better than this if we optimize the window size further. The below graph plots the raw (red) and quantization-error-corrected (blue) values of the Allan deviation, and the expected time deviation due to inherent OCXO instability, as a function of the size of the averaging window. The max of the blue and yellow curves is minimized at a window size of about 6,918 secs. (1.92 hrs.), at which the adjusted relative Allan deviation is 2.41e-9, and the expected OCXO time deviation is 2.40e-9 secs.

So, what is this saying exactly?

First, the corrected Allan deviation says that, supposing that we measured the OCXO vs. GPS average relative frequency over a ~2 hr. period, we would expect the frequency so measured, even in the absence of any quantization error, to be offset from the actual mean relative frequency over the period consisting of the current 2 hr. period adjoined with the following 2 hr. period by an amount that is distributed with a standard deviation of +/- 2.41e-9, expressed as a multiple of the actual relative frequency. Given a nominal OCXO frequency of 10 MHz, this means that the expected frequency deviation of the OCXO vs. GPS is about 0.0241 Hz. But, the OCXO is really much more stable than that. So the measured relative frequency deviation is actually due almost entirely to variations in the frequency of the PPS signal from the GPS. The GPS PPS signal is supposed to be at 1 Hz, so this means that its average frequency over this size time period is offset from what it ought to be by a standard deviation of 2.41e-9 nHz. Translating frequency deviation to time deviation, this means that (if it were free-running) the GPS's unit's idea of the total time elapsed since the start of this ~2hr period could be off by as much as ~29 microseconds (std. dev.)! This makes sense, particularly with respect to the ~50us size of the phase excursions that we (not infrequently) do see, due to time-lock being lost as a result of the limited sky visibility through our window. However, except at times when time-lock is actually lost and a major (typically many-microsecond) phase excursion is happening, the GPS unit's clock is not really free-running, but is phase-locked with the actual GPS signal from the satellites. So, during such "good" periods, we can expect the actual time deviation of individual PPS pulses could be much less than the 116 nanosecond value. Typically, TRAIM reports that it is; although at present we are not yet confident in these reports. (Since they seem inconsistent with the Allan deviation values we're seeing, which indicate a larger time uncertainty.)

Let's think a little more carefully about how to model all this. First, there is "real" (UTC/USNO) time, flowing (more or less) equably, which the GPS satellites know, and which our GPS module keeps in sync with, albeit imprecisely. The GPS unit includes an internal clock which experiences random frequency fluctuations on short timescales. Our Allan deviation measurements are evidence of this.

However, it occurs to me that I don't really know if our Allan deviation measurements would still be accurate during intervals where time lock isn't lost. I need to redo the Allan deviation calculation for a known "good" interval. The first 40,000 or so seconds of our 10-day run looks pretty good; I should probably redo the Allan deviation calculation just for that portion. Working on that now.

Actually, let's take the 68,000-sec. (~18.9-hr.) interval from 665,000 to 733,000 seconds into the run, which is pretty smooth:

Indeed, when we plot the Allan deviation just for this interval, we find it is much smaller than the overall plot for the entire run:

In fact, at least for small window sizes, this deviation (blue line) seems to be dominated by the quantization error:
The yellow line shows the quantization error's contribution to the total Allan deviation, and the green line shows the adjusted version of the blue line if the variance for the quantization error is subtracted from the total Allan variance.

Let's throw in another curve, this one from the first 40k seconds:

That looks pretty similar to the one from the other "flat" period.

But all this, now, totally throws into question a bunch of my recent conclusions, including the notion that the Allan deviation ought to scale down with 1/sqrt(t) - although it may still do so in some regions of these curves - as well as the impression that the GPS unit's clock had an inherent frequency instability on the order of 1.67e-7 (1-sec). Instead, in the new curves it is as low as 3.14e-8 (@1 sec), or 31.4 ppb. This makes more sense, from the perspective that the GPS clock's phase is being adjusted using the satellites. In fact, it is in line with the typical ~30ns accuracies that we get from the TRAIM algorithm. Actually, just did a measurement of the average nonzero TRAIM accuracy over the 10-day run, and it was 18.1 ns. From quantization error alone, the standard deviation of time error would be 28.9 ns. And the DeLorme doc specifies 62 ns. However, using the 1/sqrt(12) rule gives 17.6 ns, which is pretty close to what TRAIM reports. Actually I should calculate the RMS TRAIM accuracy, OK, that is 21.1 ns, also still close. Probably, however, I should make a worst-case assumption and put it back up at around 60 ns, which is about the documented accuracy from DeLorme as well as the "corrected" accuracy from the 48ks run segment.

Now, this is nice: If we average over 151-s intervals, and use the purple line from the 1st 40ks, we can get theoretically an accuracy of 372 ps. Really meaning, the frequency deviations at that timescale are only 3.72e-10, so if take a frequency measurement based on the last 151s, it should be only 372 ppt from the "real" mean frequency for that timescale... And meanwhile, extrapolating OCXO times over that timescale should only result in time errors of that magnitude. So combining these we might be able to get absolute time accuracy less than a nanosecond! Need to think some more about this... Here's the intersect graph...