Wednesday, September 28, 2011

Spatter Analysis

Let's see, yesterday I was not in, b/c I had presentations over at COE all afternoon.  The COSMICi group's presentation is scheduled for this afternoon at 4:30 pm in room A223, and I will leave here to go to it at about 4:00 pm.  Actually, I need to leave a little bit earlier, to get the rubrics ready.

Monday, Darryl and David came in and we talked a little about the bad DAC problem, and probed its output pin (pin 7) directly with the multimeter to verify that the problem was not in the pin-pad connection (if it had been, I don't know if that would have made it any easier to fix, but I thought we should at least check).

Probably, what we need to do at this point is just to shift DACs #3-6 up to positions 2-5, replace the comparator outputs from DAC #2 with Vcc, and modify the firmware to skip DAC #2 when configuring the voltage ladder.  We will only have 5 levels, but we rarely cross all 6 as it is.  We can space the levels out to cover the same range.  It is less information than we had originally, sure, but not too bad.  I just hope we don't lose another one!

In case the problem was caused by water drips from the cooling system, we will hereby institute a policy of only running the fan (not the thermoelectric plate) to eliminate condensation.  If it turns out that the plate absolutely has to be used, we will minimize the chance of drips by only running for a few minutes at a time for short tests.  No more long overnight runs with aggressive cooling until we get a better cooling system.

 We also peered at the board with the magnifying glass, and there is clear residue, probably from evaporated water dripped down from the cooling plate.  Dr. Ordonez suggested during his visit last week that we turn the system upside-down, but we will still need some mechanical support to do that.  Also, that is not a permanent solution b/c we don't really want water dripping onto the carpet & people in the classroom in CLC.  If a cold plate is still used in the new design, then either the cold side has to be sealed from contact with the air, or else there has to be some kind of tray to catch the drips.

In other news:  I am giving a graduate seminar in the ECE department at 4:15 pm next Tuesday (in A337) to survey issues in hardware-software codesign (controlling custom hardware from C) that are exemplified by the work that was done in this project on the FEDM board.  All COSMICi computer engineering students (and all Senior Design students in general) should attend.

Wednesday, September 21, 2011

Dead DAC Diagnostic

Today, I want to see if I can narrow down why the dead DAC (#2) is not working.  I want to probe its pins directly, to see if maybe the problem is with the connection between the IC and its pads.

First step: Find the DAC datasheet.  To do that, I really need to clean up my desk papers.

A little later, the EE student (Samad) is going to come by, and we will start working on gathering information on the voltage requirement / current draw of various system components.   Cleaning my desk papers will be helpful for that as well.

He's here, we're starting to go through some of the power-related stuff stuff.  Some notes:

  • The PMT base is rated at 250 mW @ 12Vdc = 20.8 mA (quiescent power).  Ray said he could pick up some more barrel connectors at Fouraker's, so that we could do a direct measurement with the paddles connected.  We'll wait to try that measurement until he can do that.

  • We directly measured the power consumption of the Wi-Fi board by powering it from the Keithley sourcemeter through J8 pin 3.  It typically consumes at most about 300 mA at high data rates.  It would be a good idea though to inflate this figure to (say) 500 mA in case the board uses more power when located farther from the base station (currently it is less than 3 feet away).  A limit of 500 is appropriate also because the on-board voltage regulator U2 (which generates the +3.3V logic level) can handle at most 500 mA anyway.
  • We spent a while looking for information on the voltages and currents drawn by the DE3 board (for the GPS app), but the provided docs are frustratingly ambiguous on this point.  We think it is probably using at most just the +12V, +5V, and +3.3V lines from the external power supply, but we don't know the actual current draw on each.  We could supply these three using the Agilent and Keithley supplies and measure them that way.  However, we decided that rather than doing an experiment today to measure this, we would postpone it to Monday afternoon, when we will have more time to set it up.

Meet & Greet

Today (Tue.) at 3:00 pm in 206, we will (finally) have our first meeting with the entire (we hope) COSMICi team, including all ECE & ME students, and all 3 advisors (myself, Dr. O'Neal & Dr. Ordonez).

Dr. O'Neal has to leave the meeting before 3:30 pm to give a Physics dept. seminar on the COSMICi project, but the rest of us can continue.

Future advising meetings (the ME ones, at least) can be over in the College of Engineering, for more convenience to Dr. Ordonez.

We had the meeting; it started a little late b/c we were waiting for more of the students to show up.  Everyone did arrive eventually.  I went over some administrative issues, then gave a brief overview of the project on the whiteboard, and then I showed the lab and the equipment to the ME group members who hadn't seen it yet (George Chakhtoura & Dr. Ordonez).  The meeting ended before 4:30.

The EE student (Samad) is going to come back around 3:00 pm tomorrow (Wed.) to discuss requirements for the power supplies in more detail - we will start doing some measurements of current drawn by different system components whose total power draw is not yet known.  I also discussed with him about the need for us to investigate what's in the ceiling at CLC in terms of power, and determine whether battery power is feasible for the stage 1 sensor nodes, at least.  If we draw AC power from the ceiling, then a professional electrician should probably be contracted to install any needed ceiling outlets.

This reminds me, I haven't yet heard from our contact at the Challenger center; Ray and I should probably go down there sometime and look him up or find out who else to talk to.

Monday, September 19, 2011

His Toe-Gram

A thought for today: Plot a histogram of the distribution of coincidence delays.  This is a good sanity check to see if it's consistent with the known angular distribution of showers.  We should expect to see more from the zenith (near the E-W-zenith plane) than from near the N-S axis.  Ray says he might show the graph in his astronomy class.

Also: Run a hodoscope configuration with perpendicular paddles to reduce slewing.  Picture of that setup (jury-rigged, as usual):


This will (we think) give us a picture of coincidences due to (mostly) single-particle events passing (more or less) vertically through the far ends of both paddles, giving us (perhaps) a clearer picture of the difference in gain of the two paddles.  At this depth inside the building (away from window, under the 5th floor and the roof), the vertical cosmics should be mostly muons.

Whoops, for some reason, threshold 2 seems to be messed up today.  It's reading 70 mV when it is supposed to be 2.000V.  Perhaps a water drip caused a voltage surge that zapped the DAC #2 chip?  Not sure what to do here.  I've tried re-loading a couple different versions of the gelware; it makes no difference.  Could try cleaning the pads on that IC, in case there is some residue causing a short.  If all else fails, we can route around it, just use the other 5 DACs.  It would be nice to first know why it's failing though.

Also need to set up Neutralino accounts for the students.  Need to remember to take pictures of them for the site next time they're here.

Spent a little while inspecting the board.  There is a fair amount of residue on the board, perhaps a result of drips from the cooling system.  We should probably look into washing off the board, in some appropriate way.  Ray says Radio Shack might have an electronics wash spray which we should try.

Anyway, here is a plot of the histogram of absolute time deltas from the run a couple of weeks ago.  The number of events for deltas other than 0 is halved to compensate for the fact that only half of the data comes from positive vs. negative deltas.  (We haven't yet explicitly separated out the data to check for any North/South asymmetry.)

From run of Sep. 6th-7th:  This shows how the number of coincidence events falls off with an increase in the (binned) absolute time difference between arrivals at the two detectors.  (Counts other than at delta=0 have been halved to compensate for the fact that data from positive and negative values of delta are thrown together.)

That's all for tonight.

Wednesday, September 14, 2011

Reading my Hodoscope

My Scilab run finished overnight.  In the stacked (hodoscope) configuration from the overnight run earlier in the week, we got 16,325 "concidences."  False-coincidences (the range for these is still set at 30 to 50 ns) were only 2, so I think we can be confident that almost all of the 16,325 coincidences were "real" (i.e., the result of a particle or particle-shower passing through both detectors).

As a percentage of total pulse count:  The coincidences accounted for 16,325/110,912 = 14.7% of events on PMT1, and 16,325/280,036 = 5.83% of events on PMT2.

The run turns out to have had some interesting variation in the total pulse rate over the course of the run:
Pulse rates from two PMTs hodoscope configuration, from 5:15 pm on Mon. 9/12 to 11:06 am on Tue. 9/13.  Horizontal axis:  Minute in run.  Vertical axis:  Pulses in that minute.  Lower curve: PMT1; Middle curve: PMT2; Upper curve: Sum.
It could possibly be related to ambient light (since this was an overnight run), but I'm unsure whether the shape of the variation really fits with that hypothesis.  Too bad we don't have a light meter that could record the room light levels independently.  Interestingly, the detector with the higher pulse rate also had more (candidate) apparent diurnal variation.  Possibly if we just sealed the gun case better (or threw a blanket over it), the pulse rate would be more in line with the other detector.  I should probably do a test where we see if switching the lights on & off affects it.

Tuesday, September 13, 2011

Garbage In, Garbage Out

Arrived this morning to find lots of garbage output and multiple automatic restarts in the UART log.  Water drips?  Overheating?  This just goes to show our need for a better cooling solution!

The first crash was at 12:18 pm (shortly after noon), after 280,036 pulses on PMT2, and 110,912 pulses on PMT1.  However, there was a good bit of garbage output even before then.  The last clean data was at 11:06 am at pulse 279,719 (PMT2) and 110,787 (PMT1).  The run started at 5:15 pm the night before.  Thus the run length was 17 hours 49 minutes = 1,069 minutes; the average pulse rate was 261.96 ppm on PMT1 and 103.75 ppm on PMT2.

Now doing a data-analysis run (anal-pulses.sce) to check the coincidence count.  This experiment was performed with the two paddles stacked on top of each other, so we are expecting a high count.  Indeed, it is looking like the count will be high.

Emailed the Senior Design team to get them in here.  Darryl and David are re-hired to help train the new students.  Also added Dr. Ordonez to the Blackboard group and emailed him asking for a meeting time.

Monday, September 12, 2011

Staying Grounded

Still having trouble with the 2nd input-capture datapath (channel #1) today.  I'm routing the comparator outputs to the B2 bus on the scope, so we can see if they're working right, at least.

In our full data run from last week, we got 431 coincidences within 50 ns, and only 12 of them were identifiable "false coincidences" of 30 ns time difference or larger.  The distribution of time differences among the false coincidences was:

30 ns:  ####
35 ns:  #####
40 ns:  ##
45 ns:
50 ns:  #

Most of them were clustered at the low end of the scale, so these are unlikely to be purely accidental coincidences.  Some theories:
  1. They represent individual particle pairs in occasional thicker-than-normal EAS pancakes, at least 10-20 ns thick, or perhaps outliers from a normal-thickness pancake.
  2. Maybe there are multiple showers occurring close together in time?
  3. There is some unexpected occasional source of up to +/- 20 ns inaccuracy in our time measurements.
By the way, here is the graph of the average pulse rates in that data set (extended a few hours from last time):

Aha, looking at the comparator outputs showed me the problem:  I never properly grounded the AC input node on PMT2; it was just floating.  It was supposed to be tied to the +2.5V supply so that the negative excursions of the pulses would take it past the DAC levels.  We were just lucky that it worked at all before!  I placed a jumper bridge/sheath across J69, and this fixes the problem.

Anyway, let's look at the coincidences with anomolously large time differences in a little more detail.  Here's the first one:

Coincidence #9 with delta = 30 ns:

    3.596D+11    3545.    2.    1.    0.    6.    0.    0.    0.    0.    0.    0.    0.    0.    0.    0.    0. 
    3.596D+11    26549.    1.    1.    0.    1.    0.    0.    0.    0.    0.    0.    0.    0.    0.    0.    0. 

Here are the raw data lines:

Tue Sep 06 17:34:01 2011 + 528 ms: < PULSE,2,3545,359642909738,1,(0,6)
Tue Sep 06 17:34:01 2011 + 527 ms: < PULSE,1,26549,359642909744,1,(0,1)

Not too noteworthy.

Coincidence #10 with delta = 30 ns:

    3.840D+11    3784.    2.    2.    0.    1.    2.    4.    0.    0.    0.    0.    0.    0.    0.    0.    0. 
    3.840D+11    28326.    1.    2.    0.    3.    1.    6.    0.    0.    0.    0.    0.    0.    0.    0.    0. 

These at least are bigger (2-threshold) pulses.

Friday, September 9, 2011

Stacking the Deck

My plan for my schedule for my work on the COSMICi project this semester is:  Be here at FAMU on MTWF (except for alternate Fridays) from about 1:30 pm till about 6:30 pm (5 hours). For the Fridays I miss, I'll try to put in some extra time here and there to make up for it.

Sadly, the run that was going on Tuesday through Wednesday ended Wednesday night at about 9:00 pm because the server PC rebooted itself for some reason.  I don't yet know whether it was a power outage, or Windows automatically rebooting itself like an idiot after installing automatic updates (although I thought I turned that "feature" off), or something else (maybe a cosmic ray!).  I should probably check the system logs for clues.  Although, unless it happens again frequently, I probably won't bother with that for now.

I started a new run with the two scintillator paddles stacked on top of each other, to see how this affects the rate of coincidences.  We expect that this should increase the rate of coincidences substantially, because now, even isolated particles that pass vertically through the stack will often intersect both paddles and trigger both PMTs.  Indeed, I can visually perceive a high number of coincidences happening on the scope - a significant fraction of over-threshold pulses on either PMT are accompanied by a pulse (although not always an over-threshold one) on the other PMT.

Meanwhile, while that is running, I'm starting a re-analysis of the run that ended Wednesday night, just to see if there were any interesting variations in the pulse rate between 3:00 pm (the end of the part I looked at the other day) and 9:00 pm.

Run start time:   Tue Sep 06 17:04:03 2011
Run stop time:  Wed Sep 07 21:04:34 2011

DAC settings / comparator threshold levels (mV):  -350,-500,-650,-800,-950,-1100.

Length of run:  28 hours, 0 minutes, 31 seconds = 1,680.5 minutes.
  • PMT1 (near window, in "Case #1"):  
    • Total # of (over-threshold) pulses = 1,369,385; 
    • avg. pulse rate = 815 pulses/min. = 13.6 pps.
  • PMT2 (near hallway, in "Case #2"):  
    • Total # of (over-threshold) pulses =  190,183; 
    • avg. pulse rate = 113 pulses/min. = 1.89 pps.  
    • We concluded the other day that this 2nd gun case just has a "bad paddle" in it - i.e., there is something awry with the scintillator itself, or the PMT tube, or the interface between them that causes the paddle's sensitivity to be reduced.  The base itself is OK.

Also:  I added some new code to anal-pulses.sce on Wednesday to check for "false" coincidences, that is ones with time differences that are just outside the range of time differences that could possibly be due to real EAS's passing through.  Time differences from 0 to +/- ~25 ns we generally assume to be potentially "real" coincidences (due to actual showers).  By counting the number of coincidences in the range from +/- ~30 ns through ~50 ns (which cannot actually be due to shower fronts propagating at c, due to the maximum distance between detectors in our lab of ~25 ft.), we can get a rough picture of how many of the assumed-real coincidences might actually be false positives, caused by simple chance or by other mechanisms not involving lightspeed particle fronts (e.g., power surges?).  Due to my earlier analysis of the coincidence rate based on the Poisson distribution, I have concluded that generally the number of false positives will be very small, but it is good to get independent confirmation of this from the data.

This seems to have worked as expected; the number of false-coincidences seen in this run was only 2, compared to ___ (assumed) "real" coincidences.  Let's now take a look at the time deltas of the false coincidences - if they were right on the edge (30 ns, say), then they could potentially be real coincidences whose time deltas simply got mischaracterized due to imprecision in our time measurements.

I have to run over to ECE now but I'll be back for the group meeting after 4 pm.

Wednesday, September 7, 2011

Not Just a Coincidence

Analyzing a 91 KB data file from overnight run, started 9/6 (2011) at 5:04:03 pm, snapshotted for data analysis on 9/7 at at 3:18:13 pm; that's about a 22-hour run (22 hours, 14 minutes, a.k.a. 1,334 minutes).

SciLab is slow as a dead dog; I am looking forward to the day when we will be processing the data in real time in our Python server.  Maybe the students can work on that.

According to Scilab analysis script (anal-pulses.sce), over this period, there were 337 coincidences, for an average rate of one every 4 minutes.

We might conceivably see a still higher rate if we turned up the sensitivity, and/or figured out why the one paddle is responding so much more weakly than the other, and fixed that.  Perhaps its scintillator is less polished, or its optical glue is more bubbly.

Let's take a look at some of the pulses, to make sure everything is copacetic:

Coincidence #1:

Coincidence #1 with delta = 0 ns:

    3.679D+10    351.    2.    1.    0.    5.    0.    0.    0.    0.    0.    0.    0.    0.    0.    0.    0. 
    3.679D+10    2730.    1.    1.    0.    8.    0.    0.    0.    0.    0.    0.    0.    0.    0.    0.    0. 

Raw data:

Tue Sep 06 17:07:07 2011 + 237 ms: < PULSE,2,351,36791996524,1,(0,5)
Tue Sep 06 17:07:07 2011 + 237 ms: < PULSE,1,2730,36791996524,1,(0,8)

Both pulses crossed just 1 threshold, but the PMT1 (window) pulse was ~15 ns longer.  The time difference was zero, so the pulse came from close to the east-west-zenith plane.

Coincidence #2:



Coincidence #2 with delta = 10 ns:

    1.398D+11    1410.    2.    2.    0.    2.    1.    4.    0.    0.    0.    0.    0.    0.    0.    0.    0. 
    1.398D+11    10397.    1.    1.    0.    6.    0.    0.    0.    0.    0.    0.    0.    0.    0.    0.    0. 

Raw data:

Tue Sep 06 17:15:42 2011 + 163 ms: < PULSE,2,1410,139780145512,2,(0,(2,1),4)
Tue Sep 06 17:15:42 2011 + 162 ms: < PULSE,1,10397,139780145514,1,(0,6)

This shower hit the North (PMT2) detector ~10 ns before the South one.  Currently, distance between them is about 22', so this shower came from about acos(10/22) = 63 deg away from the N-S axis.

The pulse from the North detector crosses 2 thresholds (-0.5V) and is total width 7 time steps (~35 ns):
   *******
     *


And the one from the South is width 6 time steps (~30 ns), but only crosses 1 threshold (-0.35V).

Let's skip ahead to one of the larger pulses:

Coincidence #24:

Coincidence #24 with delta = 5 ns:

    8.166D+11    59690.    1.    4.    0.    1.    1.    1.    6.    3.    2.    8.    0.    0.    0.    0.    0. 
    8.166D+11    8182.    2.    2.    0.    1.    9.    3.    0.    0.    0.    0.    0.    0.    0.    0.    0. 

Tue Sep 06 18:12:05 2011 + 940 ms: < PULSE,1,59690,816571367255,4,(0,(1,(1,(1,6),3),2),8)
Tue Sep 06 18:12:05 2011 + 941 ms: < PULSE,2,8182,816571367256,2,(0,(1,9),3)

This one hit the South paddle first (slightly).  The pulse profile of the larger (PMT1) pulse was:

**********************
 *************
  **********
   ******

With the present DAC settings, that one is over 800 mV amplitude!  The PMT2 pulse was smaller:

*************
 *********

This is consistent with our observation that the pulse rate on PMT2 is lower.  In general the gain on that paddle is lower, as we determined yesterday.

Ray says the rate of showers we are seeing now (1 every 4 minutes) is reasonable.  We're going to keep the current run going one more night (to look for diurnal variation).  However, in the data so far, there doesn't seem to be any diurnal variation per se, but just a slow settling down of the PMT1 pulse rate near the start of the run:

Pulse rates for paddle #1 (middle), #2 (bottom), and their sum (top).  Horizontal axis is minutes in run, vertical axis is pulses in each minute-wide bin.  Note there is a slow settling down of the pulse rate for PMT#2, which was first put into service shortly before the start of the run.  The bias voltage for both PMTs is 1.2 kV, and the detection threshold for these (negative) pulses was -350 mV.
That's a good stopping point for today.

Tuesday, September 6, 2011

Paddling Upstream

When I left on Thursday I did a Quartus compile to pull in the latest firmware; loading that image.  It sets the DACs at -300 mV.

Hooked up the 2nd paddle (at PMT1); it worked, but I wasn't getting any pulses from the 1st paddle (at PMT2)!  Left it sitting a while, and it eventually started working.  Weird.  Perhaps the FPGA had to cool down first in order for the PMT2 datapath to start working reliably?  That is the one that uses logic registers instead of RAM blocks in its FIFO; perhaps it's a little slower for that reason...

Interestingly, the pulse rate is higher for PMT1 (the paddle near the hallway).  I would have expected a higher pulse rate for from one closer to the window!  The supply voltage is slightly (like 10 mV) higher for the one near the hallway, but that does not seem like enough to account for the difference.

I think I'll try swapping the PMT bases and see if that changes things.  Stopping the current run.  PMT1 (with base #1) had 46392 pulses, PMT2 (with base #2) had 13220, for a ratio of 3.5:1.

OK, swapped the bases; now retrying.  PMT1 input (ICDP channel #0) still has the higher pulse rate - stopped it after running for a while and the ratio is 7,513:1,991 = 3.77:1 (even worse!).

Next, I'll try swapping the inputs, in case there is some difference in the way the two different pulse-capture datapaths are responding.  No, after swapping the inputs, input PMT2 now has the higher pulse rate.  Let's stop the run for a moment to check the ratio - 2,463:556 = 4.43:1 (even worse!).

Let's now try swapping the paddle locations (without changing which cable is going to which side of the room).  That restores input PMT1 to the higher pulse rate; ratio is 1,885:581 = 3.24:1 (back down below what we had originally).  This means it's the scintillator/PMT assembly that's causing (the bulk of) the difference in response that we're seeing.  The paddle assembly that's near the window now (input PMT1) is the one that's responding more strongly.  We just need to remember this, and maybe turn down the base voltage on that one somewhat to compensate (or turn up the base voltage on the other one).

OK, I'm starting a new overnight run to check for coincidences, so we can figure out what the shower pulses will look like with the paddles.  If they are very large compared to the present voltage ladder, then we may want to increase the spacing between the voltage steps somewhat.  It might make sense to also increase the base voltage of the ladder to cut down on the piddly little noise pulses.

Using the ladder:  Base -350, step -150.

Ray suggests tomorrow also trying a test with the paddles on top of each other.  This should give us a much higher rate of coincidences.

Saturday, September 3, 2011

Second Paddle

Ray got the 2nd paddle ready on Thursday, unfortuntely there wasn't time to test it out.  We will hook it up on Tuesday and try it out.  We're expecting that, although the isolated particle detection events are smaller than before (due to the thinner scintillator), when a dense shower comes through, the pulses should be larger, since the volume of the paddles is (I think) larger than our cylindrical test PMTs.  Also, the number of coincidences should be larger, since the larger area of the paddles means that a lower shower particle density is required to hit both detectors.  We've already seen a larger rate of single-particle events with the one paddle (although the pulses are smaller).