Monday, October 31, 2011

Mon., Oct. 31st

Need to change my FAMMail password again soon.  Remember this time to also change it in Gmail!

Ray called Mentor about the status of the donation and they asked him to forward the licensing agreement to someone who has legal signing authority on behalf of the university.  Ray is sending it to the General Counsel's office.

Ray said I can go ahead and start testing coincidence detection.  His pulse height peak in the current configuration (90 degree overlapping orthogonal paddle ends, 30" apart, 3 ns arrival time filter) was 241 mV for the paddle with the higher peak, and 236 mV for the lower peak.  (Bias voltage still 1,200 V.)  He suggests we set the first threshold at 200 mV.

I should perhaps change the firmware to count and periodically report the number of non-coincidence peaks, so we can see how many of them we are skipping.

I could perhaps integrate the (56-bit) changes by Juan and Michael Dean into the Quartus design before I test my code, but perhaps I'll wait; it might be better to just test one thing at a time.

Started "Nios II 9.1 Software Build Tools for Eclipse;" switched it back to my main workspace for the FEDM project ("Q:\eclipse_workspaces\mpf_workspace").

Opened pulsebuf.h and uncommented "#define FILTER_COINS"; this enables compilation of the code for coincidence filtering.

Edited pulsebuf.h and pulsebuf.c to define macros for normal & error return codes from pb_get().

Modifying server_stream_pulses() in pulsebuf.c to just count up the non-coincidence pulses, and output the counts periodically.

The guys helped me hook up the paddles on opposite sides of the room and tested.  That is working now, as far as it goes, except that for some reason I am seeing isolated pulses reported as coincidences - this should never happen, right!  Anyway, debug later.

Got David & Darryl started on calculating the standard deviation of the triangle distribution representing the analytical distribution over actual pulse widths for a given measured pulse width.

Found the bug in my coincidence detection code - forgot to update last_pulse_time[] when pulling pulses from the queue.  That's fixed now.  Started an overnight run.

Wednesday, October 26, 2011

Wed., Oct. 26th

Ray reminded me to respond to Sachin's email.  Looks like I didn't receive it because Gmail hasn't known my FAMmail password since June!  Fixed that, and scanning the email subjects to see if I missed anything else important...  Aha, replies from Reed Lambdin, and some emails about Zemax.  Anyway, replied to Sachin just now to see if he can continue to give us advice as we modify the board design.

Today I'm going to try out the Wi-Fi board with the new 40-pin header for the hand-wired serial port.

Regarding DSR, apparently I did indeed have it wrong in the original rat's nest - pin 10 (not 11) *is* DSR.

On the new Wi-Fi board (labeling it "#1"), I need to hook up pins 1+2 of J10 temporarily to ground the /EN input of U3, i.e., to enable the level-shifter on the main serial input so I can configure it from the PC.  Using one of my new Adafruit jumpers (a purple one) since I am short on jumper bridges.

Starting UwTerminal on COM1.  Hooking up serial cable (#1) from PC to main serial input.  Hooking up USB power.  Switching board on.  Get the usual "\00\00" as the serial port initializes (sending null bytes).  Hit enter and get the usual "00" prompt.  Enter "at" and get it again.  Enter at+dir and it looks like all the files are there.  However, I'm not sure if this board really has the most recent version of the script, so I'm going to re-burn it.

Entering "at & f *".  Get "01 E03C MEDIA_CORRUPT" as expected.  Power-cycle.  Board now back to just factory.def.pc in the filesystem.  Entering at+set 42="128", 40="1000", 41="1000" (three separate at+set commands).  I hope this board has the right version of the firmware!  Now, Download->Data->Data File+ on nodeid.txt and strings.txt in the "My Dropbox\FAMU\COSMICi\Wi-Fi Script" folder.  Then do Cross-Compile+Load on "autorun.uws" in that same folder.  Now loading.  I think the firmware version is right, or the cross-compiler wouldn't have worked.

OK, got everything in there now.  Now, start the server (C:\SHARED\Server Code\COSMICi_server.py).  Server is up and running.  Now, let's test the autorun feature.  First, we'll test with the control coming from the PC.  With DTR checked in UwTerminal, we'll power-cycle the board.  No autorun!

Now, uncheck DTR in UwTerminal, and power-cycle the board again.  That worked; we got the 3 TikiTerm terminals popping up.

Let's now disconnect the serial cable from UwTerminal, and try just manipulating pin 10 of J2 (DSR, new yellow jumper).  First, typing "exit" command over the wireless, so script exits to 00 prompt.  Next, re-routing pin 2 of J10 (purple jumper wire) over to pin 27 of J2, which is 3Vout from the module.  Now, we'll power-cycle again.  That worked, and the autorun activated.  Again, this could be due to a pullup resistor on DSR.

To test this, let's try wiring the DSR input to ground.  Pin 38 of J2 is a ground pin.  Plugging the other end of the yellow wire into that.  Now, power-cycle again.  This time, the autorun did NOT activate!  This confirms that setting DSR to *high* is what activates autorun (matches the documentation).  Also, it tells us that when a box is "checked" in UwTerminal, the corresponding logic signal gets set LOW, not high.  So the checkbox in UwTerminal is telling us the line level, not the logic level.  And pin 10 of J2 is indeed the right pin.

With all this confirmed, we are ready to wire up our new logic-level serial "cable" (really just a wire bundle) to the FEDM board.  The pins are nicely labeled in the male socket on the board.  So we can just plug directly into that, as follows:
  • Black     - Common GND (Pin 5)
  • Red        - WiFi M_TX --> RxD on FPGA (Pin 2)
  • Blue       - WiFi M_RX <-- TxD on FPGA (Pin 3)
  • Orange   - WiFi M_RTS --> CTS on FPGA (Pin 8)
  • Green     - WiFi M_CTS <-- RTS on FPGA (Pin 7)
  • Yellow   - WiFi M_DSR <-- DTR on FPGA (Pin 4)
Now, looking at the OrCAD schematic, pin 4 (DTR) on the FPGA side can be tied to ground through J50, but really we want to tie it to Vcc to make sure autorun is enabled.  So instead, we can use J46 to tie it to pin T7 of the FPGA, and then set it in firmware - maybe!  But will it be set soon enough?  It takes some time for the design to load from EEPROM.  Really, we want it to be high as soon as the boards power up.  So let's identify a +3V line we can connect to instead.  Node +3.3VDOWN is available on pin 1 (top pin) of JP12.  So, connect a yellow wire (representing a constant ~3V signal) from there to pin 1 (right pin) of J46.  Now we can hook up the pins above directly to the connector.

A thought: Should we go ahead and try supplying the +5V power through RI (pin 9) as planned?  If we want to avoid actually clipping the metal loop jumper on pins 1 & 2 of J8, and soldering a new bridge between pins 2 & 3 of J8, however, we cannot actually supply it there, i.e., on PC_RI (which is really on the DE9 connector J7, anyway).  The +5V can also be supplied on the tip of the barrel plug (J6), and on pin 3 of JP3, which is a bare thru-hole at present (not populated).  It does NOT come in on J2 at all!  (So my old orange RI/+5V input wire on my old Wi-Fi board #3 rat's nest was totally wrong-headed.)

I'm thinking the easiest thing might just be to solder a new male pin header onto pin 3 of JP3, and connect to that - that way I don't even need to use the barrel plug.

Clipped a new male pin header off of the extra-long, double-ended ones I got from Adafruit, and soldered it on there.  Connecting a white (for "white-hot" +5V) F-F jumper wire to that; will add it to the serial "cable" (bundle of jumper wires).  So now we have:

  • Black     - Common GND (Pin 5)
  • Red        - Wi-Fi M_TX --> RxD on FPGA (Pin 2)
  • Blue       - Wi-Fi M_RX <-- TxD on FPGA (Pin 3)
  • Orange   - Wi-Fi M_RTS --> CTS on FPGA (Pin 8)
  • Green     - Wi-Fi M_CTS <-- RTS on FPGA (Pin 7)
  • Yellow   - Wi-Fi M_DSR <-- DTR on FPGA (Pin 4)
  • White    -  Wi-Fi DC_IN <-- "RI" (really +5VCC) from FPGA (pin 9)
So, I hooked up all these connections to the FEDM board just now, and tested it by switching on the Wi-Fi (and then switching on the +5V power to the FEDM), and it worked perfectly!  The Wi-Fi board ran the autorun script, popped open its windows, sent the "WIFI_READY" message to the FEDM, and the FEDM sent the acknowledgement back via the Wi-Fi board to the UART bridge window.

The new, simplified, neat-and-pretty version of the detector electronics.
At left: FEDM board.  At right: EZURiO Wi-Fi dev. board.  Look ma, no wires!
At some point, we need to change the FEDM firmware (as well as that of the GPS app) so that it waits to receive the "WIFI_READY" message from the Wi-Fi board before it starts transmitting data - this is to make sure that its early output doesn't get lost in the serial input buffers of the Wi-Fi board before the Wi-Fi autorun script has finished configuring the bridge connection to the server.

Juan C. and Michael D. came by and delivered their code for the 56-bit version of the datapaths.  We started a compile and fixed a few bugs till it got past synthesis.  However, it later stopped in the fitter because the device was not set.  However, the code is now I think ready to import into the main folder.  The changes in the top-level file need to be merged into my latest top-level file.

Tuesday, October 25, 2011

Tue., Oct. 25th

David had to study for his exam today.

Darryl is here working on the new architecture diagram for the paper.

Used my solder gun to solder two 20-pin breakaway male pin header strips to the 40-pin header footprint (J2) on one of the currently unused Wi-Fi boards.

Tried the electronics cleaning spray I picked up from Radio Shack to rinse the solder flux & oxidation residue off of the back of the board; then rinsed the area briefly in industrial water and dried with a paper towel.  Leaving it to dry overnight, will test tomorrow.  I went ahead and wired up the connections.  Below are the old (left) vs. new (right) versions of the hand-wired logic-level serial port, for comparison.

Old hand-wired logic-level serial port
New hand-wired logic-level serial port












The color code on the wires in the new version is:
  • Black     - Common GND
  • Red        - M_TX = 3.3V TxD from Wi-Fi board through wire (to FPGA board)
  • Blue       - M_RX = 3.3V RxD into Wi-Fi board from FPGA board
  • Orange   - M_RTS = 3.3V RTS (req. to send) HW handshaking (flow control) from Wi-Fi thru wire to FPGA
  • Green     - M_CTS = 3.3V CTS (clr. to send) HW handshaking (flow control) into Wi-Fi board from FPGA
  • Yellow   - M_DSR = 3.3V DSR (data set ready) status input into Wi-Fi board from FPGA - Used to enable autorun (when high).
The old version had all 9 pins of the extra DE9 socket wired up (not just 6), but the others were basically unused.  Also, the old color code was not as informative, because the wire-wrap wire I used in the original hookup had only 3 colors (red=output, blue=input, white=ground) plus one extra orange wire for a +5V power input on the RI line (which was anyway unused).
    The new version has no DE9 socket on it, but what I'm thinking is to just do away with the serial cable and connect the other end of the F-F jumper wires directly to the pins in the GPIO1 header or the DE9 connector on the other end (on the FPGA board).

    Looking at the photo of the new connector from home just now, I think that pins 10 & 11 might be swapped with each other.  Need to double-check against the Wi-Fi board schematic when I get to work. Actually, looking at the module datasheet, it looks like it is the old version of the connection that is wrong, connecting DSR to a GND pin!  Ah, but DSR was disconnected at the other end, I think (I hope!).  So how did the Wi-Fi board do autorun?  Maybe the M_DSR input has a pullup resistor.  Anyway, we should hook this up properly!  It would be a good idea to run an experiment to make sure that whether the FPGA does autorun really depends on whether DSR is asserted on the FPGA side.

    Monday, October 24, 2011

    Mon., Oct. 24th

    I'm tired of trying to come up with creative blog post titles.  No more!

    I went by Fouraker's earlier and got some breakaway headers, some extra breadboards, & some electronics cleaner spray.  And a couple of D-cell batteries for my pencil sharpener at home.

    Ray is still collecting data on the scope - this time with "Minimum" instead of "Amplitude" measurement; this could explain the discrepancies

    We have a new volunteer from ME, Arianna.  Note to self: Add her to neutralino in a few days.

    Pascal is here, he took some board measurements and I emailed him some mechanical drawings.  He says the team is meeting Reed Lambdin at CLC tomorrow at 3-something.  I probably can't join them but I reminded him some things to discuss.

    Friday, October 21, 2011

    The Coincidence Detective

    Plan for today is to test the on-board coincidence detection code, as soon as Ray is done with his data collection on the Lecroy.

    I keep forgetting, I've been meaning to stop by Fouraker's and pick up a couple of 40-pin headers and/or sockets that I can mount on the Wi-Fi board.

    Ray is having the students work on the paper.  I recommended to Darryl that he make a new up-to-date figure of the FPGA design based on ideas from the architecture diagram from my Grad Seminar talk (5th slide), and the old draft slides in the Dropbox file "FEDM_design\design notes\Input Capture.odp", which are more up-to-date that the figure that Darryl was using originally.

    Didn't get to the coincidence detection today.  However, in his scope tests on the LeCroy, Ray found that the pulse height peak in the histogram totally depends on where the threshold level is placed - meaning the scope is basically totally unreliable for collecting any real scientific data.

    Wednesday, October 19, 2011

    Cool Stuph

    Some sweet, mostly inexpensive embedded components to play with sometime, for Cosmic Cube or other projects:

    Serial Cable Redux

    Today I soldered up a 9-pin male D-sub (DE9) connector to the appropriate pins of the 40-pin ribbon cable connected to GPIO1 on the DE3 board.  The pin connections are:

    GPIO1/Ribbon      FPGA       Wi-Fi     DE9 connector
    Cable pin #     function    function     pin #
    ------------    --------    --------     -----
         4             Rx          Tx          2
         5             Tx          Rx          3
         6           DTR out     DTR in        4
         8             RTS         CTS         7
         9             CTS         RTS         8
        12            (common ground)          5

    Then Samad arrived, and our next task was to get the GPS app running again.  Despite the new, more robust GPIO1 serial connection with the ribbon cable, we again experienced intermittent problems getting the board programmed and running.  These are troubling, but with some retries & fiddling, we got past them.  

    One problem that we did manage to diagnose was why the serial I/O to the Wi-Fi board was still not working - the Wi-Fi script's baud rate had been changed to the maximum (115,200 baud) when I was trying to maximize the pulse rate - but the GPS app SOPC system still had its default rate set to half that (57,600), which I had done before to try to reduce serial errors.  So, no wonder the communication was not working!  After fixing this, everything was working normally again (except that during this run, the GPS did not seem to have proper time lock).

    Note: I am using Rev_A of the GPS App, because I think that Rev_B may have just been leftover from when I was starting to experiment with using the DRAM SIMM to store GPS data for analysis. 

    To measure power, we manually wired up the 12-pin power connector to external supplies, using the nice jumper wires from Adafruit that I just got at home & brought in.  

    At the moment, the +3.3V OCXO is still being powered through the DE3 board.  Here are the sources we used.
    • The Keithley Sourcemeter for the +3.3V.  With the demo running, this is maxing out at 1.05A.
    • The Agilent +6V supply for the +5V.  With the demo running, this is using a steady 0.32A.
    • The Agilent +12V supply for the +12V.  With the demo running, this is drawing zero current.  Inspecting the manual reveals that the +12V is used to run an optional cooling fan (not needed in this app).
    We next separated the +3.3V supply for the OCXO and the main board, powering the OCXO from the <25V supply on the Agilent (no longer needed for the +12V), so we could distinguish the two parts of the total power draw.
    • The OCXO starts at 0.86A and after a while settles down around 0.31 A.
    • The DE3 (running the demo) draws about 0.51 A on the +3.3V.  Note: In this test, the serial level-shifter to the GPS serial cable is also being powered through the DE3.
    With the real GPS app running on the DE3 board, and communicating successfully with both the GPS kit and the Wi-Fi, here are the figures:
    • +3.3V to DE3 board draws 0.346 A.
    • +5V to DE3 board draws 0.336 A.
    Samad took notes on all this, and is going to write a "Power Analysis" appendix to include in the Project  Plan report.

    Juan (Calderin) and Michael (Dean) showed up about the time that Samad was leaving, and we spent a little time going over their changes (so far) to the Quartus code, and the content of their draft report, and talking about what needed to be in the report.

    Ray spoke to LeCroy about his triggering issues but didn't get a clear response from them.  He wants to run some text for the paper by me.

    It might be a good idea to replace the wire-wraps on GPIO0 with a ribbon cable soon as well, or at least with my nice new Adafruit jumper wires.  Pins currently used are:

    • GPIO0 pin 13 - GPIO0_D[8] - TxD for FEDM's uart_2
    • GPIO0 pin 16 - GPIO0_D[11] - RxD for FEDM's uart_2
    • GPIO0 pin 40 - GPIO0_D[31] - PPS_IN from GPS
    • GPIO0 pin 29 - +3.3V out - Can be used to supply uart_2's level shifter
    • GPIO0 pin 30 - GND - For uart_2 / to level shifter

    Tuesday, October 18, 2011

    GPS App Power

    Today Samad and I are planning to do power measurements for the GPS application board (DE3) when running.  But first I have to get the app up and running again - it's been a while.

    I am going to temporarily detach the DE3 power supply from my earlier jury-rigged setup that tapped out the +5V supply.  After I verify that the DE3 app is still running, we'll patch in some external supplies to the connector.

    I'll also plug the Wi-Fi board back into the USB supply for now.

    Hooked up the OCXO via the 6" SMA cable, with a jury-rigged connection between the SMA connector on the OCXO side and the test point for the clock output.  We need +3.3V supply for the OCXO; can we get this from the DE3 board?  Pin 29 on each GPIO header is the +3.3V supply, opposite pin 30 GND.  We can rig wires from there to the OCXO board.

    At some point, I should maybe break out a ribbon cable from GPIO1 to make a proper connector.

    Inserted serial cable 3 between my wire-wrapped DE9 connector on GPIO1, and the plug-wired connector on the Wi-Fi board.

    Plugged in the power connector and the USB-blaster cable to the DE3 board.

    Before adding the GPS module into the mix, let's verify Wi-Fi communications.

    Starting up the server.  Powering on the Wi-Fi board.  TikiTerm windows pop up.

    Now, let's power on the DE3 board.  I think it will just start up with the demo?  No, nothing...

    Let's start UwTerminal first, so we can see any I/O on Wi-Fi stdout...

    Weird, the Wi-Fi board restarted by itself for some reason...

    With UwTerminal running (at 57,600 baud), this time when powering up the board the demo started up - but the WiFi board reset itself again!  Weird...

    Let's load the GPS app design.  First, have to locate the latest version... 

    The version in C:\f\DE3\S3\SB+SOPC\GPS_FPGA_app\Quartus_II_Project\DE3_GPSapp was last touched circa 5/17-5/24.  Whereas, the one on Dropbox (C:\Users\Mike\Documents\My Dropbox\FAMU\COSMICi\GPS_FPGA_app\Quartus_II_Project\DE3_GPSapp) was last touched 1/19.  So the one in \f is newer.

    Let's look back at my notes from 5/17-5/24...  Nothing really helpful there.  Probably whatever I touched then was not essential.

    The other day, I copied the GPSapp on \f to C:\LOCAL\GPS_FPGA_app, to make it easier to navigate to.  Later I'll put it in C:\SHARED to share it with the students.

    Opening the project, C:\LOCAL\GPS_FPGA_app\Quartus_II_Project\DE3_GPSapp\DE3_GPSapp.qpf, I'm finding there are two revisions, Rev_A and Rev_B.  I don't know what the difference was.  We're on Rev_A currently.  Rev_B was created May 16th.  Let's look back at the blog...  Still can't figure it out.

    Let's try Rev_B first, since that is the newest revision.  Wait, no programming file??  Recompiling...  OK, now we have a Rev_B.sof file.  Selecting USB-Blaster.

    Programming... No stdout yet, but I see the "Hi" LED message.  OK, haven't hooked up the GPS...

    Let's open the firmware, it was in the legacy Nios II IDE...  Running code in debugger...  Great, now the IDE seems to have hung...  Before even starting the Nios terminal...  Killing it & restarting.

    The hanging problem might be related to moving the project folder.  Reopened original workspace (C:\f\DE3\S3\SB+SOPC\GPS_FPGA_app\Nios2IDE_Workspace) and ran from there and it works, sort of - still not seeing any output on the UART bridge.  However, after some fiddling, I did get the diagnostic output re: PPS and NMEA data from the GPS to appear on stdout.  It is flaky though - the DE3 board keeps resetting itself, and the Wi-Fi board resets itself whenever the DE3 board powers up.  At one point, I couldn't even get the .sof file to load, but fiddling with the wire-wrap spaghetti on GPIO1 fixed that problem, so there must have been a sporadic short somewhere in that mess o' wires.

    After Samad got here, we also noticed some actual disconnected wires in the GPIO1 wire-wrap rat's nest, including a broken Tx/Rx wire, which would account for the lack of Wi-Fi data.  We decided to replace the wire-wrap garbage with a proper (40-pin) ribbon cable so it is less prone to breakage.  I brought in a cable assembly earlier, left over from some old IDE drive.  We removed the wire-wrap from GPIO1, cut the ribbon cable, and identified, separated, stripped, and color-coded the appropriate wires of the cable, and identified which pins on the back of a male DE9 connector they need to be soldered to.

    Samad is coming back at 2 pm tomorrow and we will actually do the soldering, then connect the cable to the Wi-Fi board, and hopefully see data output from the GPS app again.

    Monday, October 17, 2011

    3, 2, 1, Contact...

    Pascal sent me a Google Sketchup sketch of the arrangement of the detectors in the room space, and I sent him some feedback on it.

    Still working on re-establishing contact with CLC.  Ray spoke to the CLC director, Michelle Personette, the other day, and confirmed that Reed Lambdin is still our contact person.  I sent Reed another email, and CC'd the students on it, as well as Michelle.  Also Ray found Reed's phone number on CLC's website; I posted that in the group blog, and also emailed it to the students.  Hopefully things will get moving soon.

    Ray got some histogram results on pulse heights from Gun Case #2 (new PMT #1), and there was a fairly sharp peak around 370 mV.  This was also where the peak of the coincidence pulses was - of course, the detectors are still overlapping at the moment.  Ray started another measurement run with the other PMT to see where the peak is on that one.  Perhaps later we should also do one with a logic trigger when the detectors are across the room from each other (to filter for the real coincidences, i.e., shower events).

    At some point soon, I need to test the new coincidence detection firmware - soon after Ray finishes his present data-collection run on the scope.

    Ray submitted a disclosure on the Cosmic Cube concept to the FAMU tech licensing office, they sent it off to some lawyers in South Florida, who sent us some feedback, which was somewhat off the mark.  At least they responded quickly.

    Samad came by my office earlier today to discuss batteries for the Project Plan report; he is coming by the lab tomorrow and we are planning to do power measurements on the GPS application.  Before he gets here, I need to get that system up & running again.  However, in this test, we're going to try driving it from external supplies, to provide the +3.3V, +5V, and +12V inputs, so we can measure the currents.  I hope the board still works without the -12V input!

    Wednesday, October 12, 2011

    Filter On Board

    Plan for today: Continue implementing optional on-board coincidence filtering.

    Ray and I couldn't visit CLC this morning, and he's busy tomorrow and I'm busy Friday, but he's going to go over there by himself on Friday.

    Samad came by and we talked a little about the battery life analysis.  He is going to come by again next Tuesday so we can investigate the power draw for the DE3 board when running the GPS app.

    Juan came by and recommended SVN for version control in Quartus (which I had asked him to investigate).  It costs money to get a private hosted server, but we could set up our own server on the xserve.  I asked Juan to investigate what ports are needed and ask Antony how hard it would be to get the ports open.

    Juan is going to see if he can run Quartus under VirtualBox on the center Mac, and access the Quartus license server through wireless via our router.  However, when he tried accessing the Internet through the wireless connection, it was only working sporadically for some reason.  We added the DNS hosts, but it still had problems.

    Juan found that SourceForge supports private SVN repositories, so we are probably just going to use that.

    I finished my changes to the pulse-buffer module in the firmware (pulsebuf.{h,c}) to support on-board coincidence filtering.  Compilation of this new version of the code is enabled by defining the FILTER_COINS preprocessor symbol in pulsebuf.h, and we can revert to the old version of the code easily by commenting out that line (leaving FILTER_COINS undefined).  The new version adds only about 1K to the code size.  It still fits!

    Can't do a test right now, because I think Ray may still be in the middle of collecting coincidence data.  Speaking of that, the SciLab run I started yesterday to count coincidences in that million-line dataset is still running!  It's at 23,657 coincidences so far, but it is just CRAWLING!!!  This just goes to show how doing the coincidence filtering in real-time on the board will be much better...

    Anyway, I'll test my new code next time I'm here...  Probably next Monday PM.

    Tuesday, October 11, 2011

    Sigh Lab

    Ray texted me that one of the scintillators is producing pulses 1,000x wider than the other, but I don't understand how that's possible?!?!?  I didn't see that behavior.  Ask him to show me what he is seeing when he gets in.  Aha, turned out one of the scope inputs was set to the wrong impedance (1 Mohm instead of 50 ohm).

    Sigh, stupid SciLab ran out of memory again even, when processing only 1/4th of the data file, a mere 2 million lines.  Let's try cutting the file in half again...

    Cutting at Wed Oct 05 22:41:00 2011.

    The new data file "node0.uart-cut.1st-egth.trnscr" has 931,499 lines.  Changing Scilab script (anal-pulses.sce) to preallocate space for 1,000,000 data records.

    In case this is going to work this time, let's generate some summary data.

    The length of this (1/8th) run is 3,518,106,163,688 time steps (5 ns each), which is 17,590.53081844 seconds or 293.17551364067 minutes or 4.886258560678 hours.

    Start time:  Wed Oct 05 17:47:49 2011 + 629 ms
    End time:    Wed Oct 05 22:40:59 2011 + 978 ms

    Length of run (according to when data was logged on the server) was thus 5 hours, -7 minutes, 9 seconds, 349 ms, or in other words 293 minutes, 9.349 seconds, i.e., 293.1558167 minutes, or 17,589.349 seconds, so in other words, the server's idea of the length of the run differed from the board's idea of the length of the run by only 1.182 seconds out of ~17,590, or in other words, by only 0.0067% or 67 parts per million.  Not too shabby... Sometime, I might want to figure out what accounts for most of the discrepancy (board clock drift or communication delays), but it's not a priority at the moment.

    PMT #1 had 718,080 pulses over the 17,590 secs., for an average pulse rate of 40.82 pps.
    PMT #2 had 213,413 pulses over the 17,590 secs., for an average pulse rate of 12.13 pps.

    Ray is using the LeCroy scope to collect histogram data for the pulse height in the coincidences.

    We are thinking maybe we really should start doing coincidence detection on the board, so that we can lower the thresholds without exhausting our data rate.  I may start working on this code.

    Went back and reviewed my notes on this from an earlier blog post.  Started making the changes, but in the context of an "#ifdef FILTER_COINS" preprocessor option, so that we can easily revert to the old version of we have trouble fitting the new code.

    When I left, I had just modified pb_add() in pulsebuf.c (in Q:\software_v4\FEDM_ctrl_fw\), and was about to modify pb_get(), which is where most of the work of the new algorithm will be.

    The SciLab code is still running; over 11,000 coincidences so far on this dataset...

    Monday, October 10, 2011

    Memory Lapse

    Ray and I are planning to go down to CLC Wednesday morning to re-establish that connection.  Hope someone there still remembers us!

    Used 'wc -l' in Cygwin to count lines in the data file ("C:\SHARED\Server Code\logs 2011-10-07\node0.uart-cut.trnscr").  It is 8,619,504.  Increased array size to 8,620,000.  Now Scilab complains it needs 146,540,153 memory, and we only set stacksize to 100 million previously.  Let's then increase stacksize to 150,000,000.  Now SciLab complains it can't allocate that much memory!  OK, let's go back to 100 million, and split the data file in half.

    Splitting into data before and after Thu Oct 06 15:56:00 2011, which is roughly in the middle of the data file.  Now have two files:
    • node0.uart-cut.1st-half.trnscr - 4,143,039 lines
    •  node0.uart-cut.2nd-half.trnscr - 4,476,465 lines
    I'm processing the first half now.  I'm worried, though, that I might still run out of memory partway through the script.   Yes, indeed it did.  Let's see if we can increase the stack size to 125,000,000.  Nope.

    Let's now extract just the 1st 1/4th of the data, before Thu Oct 06 04:20:33 2011:
    • node0.uart-cut.1st-qrtr.trnscr- 1,984,492 lines
    The analysis script anal-pulses.sce is running now, and hasn't run out of memory yet.  But it hasn't finished yet.  Check on it tomorrow...

    Friday, October 7, 2011

    Fried Day

    My brain is fried... Long day.

    The run we started on Wednesday was still running.  Guess the crashing problems were caused by the heartbeat function!  I wonder if it just needs its own reentrancy structure - try that change sometime.

    Anyway, I stopped the run and cropped the data file for analysis.

    One of the Senior Design students (Michael Dean) came by and I spent a while with David taking him through the Quartus design.  We also went over the Scilab script.  I don't think the script will actually complete on the latest dataset, because it is too big.  Maybe just extract a limited-time-period excerpt from the data file?  Worry about that next week.

    Wednesday, October 5, 2011

    Up the Creek...

    Paddle assembly #1 (in gun case #2), which was responding more weakly than the other one, has been removed and replaced by the other paddle assembly (not yet yested).  Call this paddle assembly #3.  I put masking-tape labels on paddles #2+3 to tell them apart.

    Let's now begin a new data-collection run, to see if the new paddle is performing any better than the old one.  Let these notes also serve as a guide for students as to how to start a new run.

    First, I moved the previous set of server log & transcript files (COSMICi.server.log, COSMICi.node0.log, node0.auxio.trnscr, node0.uart.trnscr) to "C:\SHARED\Server Code\logs 2011-10-05" (labeling them with today's date, as is my convention).

    Now, restarting "C:\SHARED\Server Code\COSMICi_server.py".  The Python interpreter's Windows console ("C:\Python31\python.exe") and the main TikiTerm window "COSMICi Server Console [Main Window]" open as expected, and the server starts generating heartbeats.

    Now switching on the Wi-Fi board.  The TikiTerm windows for the 3 automatic connections from the board open up:
    1. MAIN connection window ("Main Server Connection #0 from 192.168.0.6:49646") for log messages and other commands from the Wi-Fi board to the server.
    2. AUXIO connection window ("Node #0 AUXIO Bridge #0") for auxilliary text output from the Wi-Fi script for diagnostic messages, user command prompt, etc.
    3. UART connection window ("Node #0 UART Bridge #0") for data bridged directly from the FEDM UART.
    Now, power up the cooling fan, currently powered at ~4.5 V.  The thermoelectric plate is outputting ~ 2-3 mA (wonder why?).

    Next, power up the FEDM at 6 V (5 V is also OK).  Current is 1.86 A.  An old version of the firmware is programmed in.  Before going on, I want to update it.  The Quartus license server is still running.  Starting Quartus.  Selecting Q:\COSMICi_FEDM.qpf.  Starting Programmer.  Selecting mode "Active Serial" to reprogram EPROM.  Switching JTAG connector from U7 to J1.  Selecting "New_with_Nios_trim.pof" programming file ("New_with_Nios_trim" being the current project revision).  Selecting "Program/Configure" and "Verify" options, and clicking "Start".  We get the expected startup message from the latest firmware:

    FEDM_STARTING,v0.6
    DAC_LEVELS,-0.300,-2.500,-0.400,-0.500,-0.600,-0.700
    FEDM_READY
    BADUMP,1
    BADUMP,2

    Now Juan is here, and we plugged in the detectors.  The new paddle assembly (#3), in gun case #2, plugged into SMA connector #1, is now responding more strongly than the others!

    Juan is going to set up the Macs so that VirtualBox runs only in one dedicated account which we all share.

    I shared the FEDM_design Dropbox folder with Juan.  He is going to look into how to do version management in Quartus development.

    Samad came by and we verified that the current draw of the detector is about 20 mA, which is what we had expected based on the quiescent power dissipation figure from the manual for the bases.

    We also probed the voltages on the DE3 power supply.  Pins 2 (black) and 3 (green) have to be connected to cause the supply to turn on.  We determined the voltages supplied on the other pins: Altogether, the levels available are +3.3V, +5V, +12V, and -12V.  Samad and I discussed that in stage 1 of the design we might be able to just use this supply to power all our electronics from that supply.

    After Samad left, I hooked up both the Wi-Fi board and the FEDM to the +5V output from the DE3 supply.  That is working fine.

    Meanwhile, since the pulse rate on the new paddle is high, we are getting hanging problems again.  I ran the firmware under the debugger, and during a hang I suspended execution and observed that it was in the output code under the heartbeat callback.  The "reentrant" stdio routines aren't really re-entrant!  Perhaps because uC/OS-II isn't running.  Anyway, I commented out the heartbeat setup and am running again.  If that is the only problem, maybe it won't crash now!

    Tuesday, October 4, 2011

    Board, Interrupted

    Came in at 3:00 pm on Tuesday to see how run was going, only to find it had stopped at some point.

    Looking back at the transcript file (C:\SHARED\Server Code\node0.uart.trnscr), the run started at 5:14 pm, with a 250 mV threshold and 100 mV ladder steps.  (Note that DAC level 2 is unset.)

    Mon Oct 03 17:14:32 2011 + 209 ms: < DAC_LEVELS,-0.250,-2.500,-0.350,-0.450,-0.550,-0.650

    The end of the run, when the firmware hung, was 6:31 pm - around the time I left?

    Mon Oct 03 18:31:19 2011 + 668 ms: < PULSE,1,30100,804013730552,1,(0,5)
    Mon Oct 03 18:31:19 2011 + 669 ms: < PULSE,2,56923,804017705582,1,(0,4)
    I'm wondering if something I did when I was leaving caused the run to halt.  Maybe I closed Eclipse while it was being used for STDOUT?

    Anyway, let's try restarting.  I started a Quartus compile before I left, and it succeeded, so we can just reload everything from there.

    BTW, I hooked the Peltier cooler to the ammeter function on the multimeter, and am getting readings like -0.450 A (-45 mA).  Actually, that was when the FPGA was in the crashed state; now that we're running again, the reading is decreasing (in its absolute value).  At this moment, it's -41.2 mA.

    It makes sense that the reading is negative, because the device is passively responding to an outwards heat flow, so the bottom side of the plate is warmer than the top side.  To actively cool the system, we have to push current in the opposite direction from the way it flows naturally, so that the bottom side of the plate then becomes *cooler* than the top side.

    Pulse rates for that 1-hour run were:

    PMT#1:  30,100 pulses / 77 minutes = 390.1 ppm =   6.51 pps
    PMT#2:  56,923 pulses / 77 minutes = 739.3 ppm = 12.32 pps

    Stopped the current run, because Ray wants to replace the paddle assembly (scintillator + PMT) for SMA channel #1 (which is currently the one in Gun Case #2) with the third one to see if that improves things.

    Monday, October 3, 2011

    Work-Around

    Today David & Darryl came in and we tested the modifications David made on Friday to stub out the broken DAC #2.  All works fine.

    We were seeing too many events at a 250 mV (1st) threshold, causing the system to hang, so we upped the 1st threshold to 300 mV.

    There is a weird problem where the third comparator output isn't transitioning on the scope display, but it seems to be working fine in the actual code, so we are not worried about it too much at the moment.  (The 2nd one also isn't transitioning, but that is expected due to the bad DAC.)

    We manually checked all the DAC levels to be sure they are still good, and they are all fine except for DAC #2 which is (still) stuck at about +70 mV.  But that doesn't matter now that we are skipping it.

    Started a new data-collection run in the perpendicular hodoscope configuration (finally!).  We are running with the cooling fan only (thermoelectric plate disconnected) to prevent condensation/drips.  TO DO:  Hook up plate terminals to a resistor, measure voltage across resistor; this then gives a measurement of heat flow through the plate.  :)