Table of contents

Known problems and troubleshooting

Troubleshooting a few common problems one may encounter... a work in progress!

Remote alert script

To try and avoid missing important problems (such as a finished or crashed acquisition) or some other event, there is a rudimentary Python script you can run on your local computer to monitor things. This will give you a visible and, on a Mac, spoken alert of important events or problems. The advantage of this is that since it runs locally you will get an alert if the VPN connection drops out, etc.

You can get the Python script here. You will need to have password-less SSH access set up between your computer and both the VAMPIRES computer and scexao2 (see e.g. here if you're not sure how to do this). The script simply monitors the arrival time of new files in the VAMPIRES data directory and alerts you if new ones don't show up within some time. It will also optionally monitor the SCExAO comments log and read new entries, so you'll know if something important happens (e.g. PyWFS loop change).

There are some options at the top of the script - most are self explanatory and can be left as they are, but the ones you might care about are:

  • dateString - Current date in format YYYYMMDD, i.e. which corresponds to both the VAMPIRES data and SCExAO logging directory names you want to monitor.
  • vampiresTimeout - Alert if no new data file appears in this time (s)
  • waitTime - Seconds to wait between ssh attempts - 10 s or so is fine
  • warnVampires - Set to 'True' or 'False' to enable VAMPIRES warnings
  • warnScexao - Set to 'True' or 'False' to enable warnings when SCExAO log is changed

Camera Initialization Error during vControl startup

This can happen if the cameras were not previously shut down (or are currently being used in another session). It can also occasionally happen if an acquisition crashes, as the cameras will wait indefinitely for a new trigger command. The solution is simple - power-cycle the cameras via the NPS. A handy shell script will do this for you, on scexao2 just do:

scexao@scexao2:~$ restartVampCameras

Other initalization errors during vControl startup

This is usually because one of the many USB/serial devices is not responding. If so, Matlab will give an error telling you which device is not responding. Check if it is really there or not in /dev and/or /dev/serial/by-id/. If it is missing or otherwise unresponsive, it usually means someone knocked a cable out. Check what device this corresponds to (TODO: include a table of USB devices here) which for now can be done by looking at lines 236 - 384 of vControl.m and then investigating it up at the Nasmyth.

vControl hangs while acquiring (doesn't finish cube)

This is an ongoing problem and the subject of an ongoing investigation. The symptom is that the cameras do not take the number of acquisitions instructed, and then simply hang indefinitely so the cube acquisition doesn't finish. A watchdog timer in vControl should detect this and restart the acquisition (with __RESTARTED__ appended to the filenames).

If it doesn't restart automatically, try to abort it by pressing the Abort! button and waiting. This will usually work. If not, you may have to hit Exit in vControl and then restart it. If this doesn't work then you may have to kill -9 Matlab itself. In either case, if these violent methods are used to escape from a hanged camera then you probably need to restart the cameras as described above. On scexao2 just do

scexao@scexao2:~$ restartVampCameras