Merge pull request #110 from KeckObservatory/dev

v3.0.5

Author: joshwalawender

README.md

@@ -45,11 +45,9 @@ The schedule is filled based on your remote observing request, but this is a man
 - Generate ssh public/private key pair **(do not set a passphrase)**
     ```
     cd ~/.ssh
-    ssh-keygen -t rsa -b 4096
+    ssh-keygen -t ed25519
     ```
 
-- Make sure that the resulting key is an RSA key.  The **private** key should have a first line which looks like `-----BEGIN RSA PRIVATE KEY-----` (it should not be an OPENSSH key).  If you do get an OPENSSH key (we've seen this on macOS and Ubuntu Linux), try generating the key with the `-m PEM` option: `ssh-keygen -t rsa -b 4096 -m PEM`
-
 - Upload your **public** key file at your [Observer Login Page](https://www2.keck.hawaii.edu/inst/PILogin/login.php). Click on "Manage Your Remote Observing SSH Key" and follow the instructions.
 
 - After you have uploaded the key, note the "API key".  This will be a long string of letters and numbers.  You will need this key to connect (see the section below titled "Configure Keck Remote Observing Software").
@@ -112,7 +110,10 @@ Note: The examples below assume sudo/root installation for all users and were or
     - **For macOS**: Install a VNC viewer application if needed.
         - [Tiger VNC](https://tigervnc.org) has the advantage of supporting automatic window positioning, but does not support scaling, and can not enter and exit view only mode interactively.
         - Real VNC's [VNC Viewer](https://www.realvnc.com/en/connect/download/viewer/) does not support automatic window positioning, but allows window scaling for use on small or high resolution monitors.  In addition, it supports changing modes both interactively and on the command line (e.g. scaling and the toggling of view only mode).  Note: this is the free software, you do not need VNC Viewer Plus.
-        - It is also possible to use the built in VNC viewer on macOS, but we have seen a few instances where the screen freezes and the client needs to be closed and reopened to get an up to date screen.
+        - Both Tiger VNC and Real VNC have the viewers available for install via the macOS [Homebrew package manager](https://brew.sh). This provides the same software as is available from the download links above, but may be an easier install for users who already use homebrew.
+            - To install Real VNC Viewer: `brew install vnc-viewer`
+            - To install Tiger VNC Viewer: `brew install tigervnc-viewer`
+        - It is also possible to use the built in VNC viewer on macOS.  The example configuration file has an example setup for this in the "VNC Viewer Command" section for users who want to try it.  Be aware, however, that we have seen a few instances where the screen freezes and the client needs to be closed and reopened to get an up to date screen, so if you see that problem, please try another VNC Viewer client.
 
 **--> Important! <--** If you are using TigerVNC on either OS, in the `~/.vnc` directory, create a file named `default.tigervnc` with these two lines:
 ```
@@ -286,6 +287,14 @@ A more extreme version would be to only keep one or two sessions open at a time
 
 VNC does not carry sounds, so we have a separate system for playing instrument sounds such as "exposure complete" indicators on the remote machine.  This system has several moving parts, so troubleshooting can be challenging.  The vast majority of sound problems however are local to the users machine.  To play a test sound, type the `p` command.  This will play a local sound file (it will need to be downloaded on the first instance of this).  If you can't hear this test sound (the quality is poor and scratchy, but it sounds like a doorbell), then check your local machine's volume settings and speaker configuration.  You may also not have configured your local `aplay` instance properly.
 
+## No Sounds on macOS
+
+The Remote Observing software triggers sounds using one of several `soundplay` executables packaged with the software (in the `soundplayer/` subdirectory).  This has not been compiled on macOS for ARM (for "Apple Silicon") and relies on Apple's Rosetta software to convert an `x86_64` executable to something which can execute on modern Apple hardware.
+
+Normally the need to run the executable through Rosetta is detected automatically, but since we are calling the executable within a python program, it appears this is not happening.  Thus, if you are not getting sounds and everything else appears to be working, what may be happening is that the executable is not getting routed through Rosetta to be translated for Apple Silicon.
+
+The workaround for this is to manually call the executable once from the command line.  This attaches the proper metadata to the executable and it will work properly from within python thereafter.  This means one should navigate to the `soundplayer/` directory and call `./soundplay.darwin.x86_64` once.  Ignore the errors, and `control-c` out immediately.  After that, future connections to sounds via the Remote Observing software should run normally.
+
 # Upgrading the Software
 
 The software does a simple check to see if it is the latest released version.  You can see a log line with this information on startup, or you can get the saem result using the `v` command.

keck_vnc_launcher.py

@@ -25,7 +25,7 @@
 
 
 ## Module vars
-__version__ = '3.0.4'
+__version__ = '3.0.5'
 supportEmail = 'remote-observing@keck.hawaii.edu'
 KRO_API = 'https://www3.keck.hawaii.edu/api/kroApi/'
 SESSION_NAMES = ('control0', 'control1', 'control2',
@@ -91,8 +91,8 @@ def create_parser():
     ## add options
     parser.add_argument("-c", "--config", dest="config", type=str,
         help="Path to local configuration file.")
-#     parser.add_argument("--vncserver", type=str,
-#         help="Name of VNC server to connect to.  Takes precedence over all.")
+    parser.add_argument("--vncserver", type=str,
+        help="Name of VNC server to connect to.  Takes precedence over all.")
     parser.add_argument( '--vncports', nargs='+', type=str,
         help="Numerical list of VNC ports to connect to.  Takes precedence over all.")
 
@@ -136,11 +136,12 @@ def create_logger(args):
         return
 
     #create log file and log dir if not exist
+    log_path = Path(__file__).parent / 'logs'
     try:
-        Path('logs/').mkdir(parents=True, exist_ok=True)
+        log_path.mkdir(parents=True, exist_ok=True)
     except PermissionError as error:
         print(str(error))
-        print(f"ERROR: Unable to create logger at logs/")
+        print(f"ERROR: Unable to create logger at {log_path}")
         print("Make sure you have write access to this directory.\n")
         log.info("EXITING APP\n")
         sys.exit(1)
@@ -168,7 +169,7 @@ def create_logger(args):
     except:
         # Works with older pyhon versions (>=3.9)
         ymd = datetime.datetime.utcnow().strftime('%Y%m%d_%H%M%S')
-    logFile = Path(f'logs/keck-remote-log-utc-{ymd}.txt')
+    logFile = log_path / f'keck-remote-log-utc-{ymd}.txt'
     logFileHandler = logging.FileHandler(logFile)
     logFileHandler.setLevel(logging.DEBUG)
     logFileHandler.setFormatter(logFormat_with_time)
@@ -234,6 +235,7 @@ def __init__(self, server, username, ssh_pkey, remote_port, local_port,
                  ssh_additional_kex=None,
                  ssh_additional_hostkeyalgo=None,
                  ssh_additional_keytypes=None,
+                 ssh_command='ssh',
                  proxy_jump=None):
         self.log = logging.getLogger('KRO')
         self.server = server
@@ -251,10 +253,6 @@ def __init__(self, server, username, ssh_pkey, remote_port, local_port,
         self.log.info(f"Opening SSH tunnel for {address_and_port} "
                  f"on local port {local_port}.")
 
-        if re.match(r'svncserver\d.keck.hawaii.edu', server) is not None:
-            self.log.debug('Extending timeout for svncserver connections')
-            timeout = 60
-
         # We now know everything we need to know in order to establish the
         # tunnel. Build the command line options and start the child process.
         # The -N and -T options below are somewhat exotic: they request that
@@ -263,9 +261,9 @@ def __init__(self, server, username, ssh_pkey, remote_port, local_port,
 
         forwarding = f"{local_port}:localhost:{remote_port}"
         if proxy_jump is None:
-            cmd = ['ssh', server, '-l', username, '-L', forwarding, '-N', '-T', '-x']
+            cmd = [ssh_command, server, '-l', username, '-L', forwarding, '-N', '-T', '-x']
         else:
-            cmd = ['ssh', '-J', f"{username}@{proxy_jump}", f"{username}@{server}", '-L', forwarding, '-N', '-T', '-x']
+            cmd = [ssh_command, '-J', f"{username}@{proxy_jump}", f"{username}@{server}", '-L', forwarding, '-N', '-T', '-x']
         cmd.append('-oStrictHostKeyChecking=no')
         cmd.append('-oCompression=yes')
 
@@ -335,6 +333,7 @@ def __init__(self, server, username, ssh_pkey, local_port,
                  ssh_additional_kex=None,
                  ssh_additional_hostkeyalgo=None,
                  ssh_additional_keytypes=None,
+                 ssh_command='ssh',
                  ):
         self.log = logging.getLogger('KRO')
         self.server = server
@@ -353,7 +352,7 @@ def __init__(self, server, username, ssh_pkey, local_port,
         # the login process not execute any commands and that the server does
         # not allocate a pseudo-terminal for the established connection.
 
-        cmd = ['ssh', server, '-l', username, '-N', '-T', '-x', '-D', f"{local_port}"]
+        cmd = [ssh_command, server, '-l', username, '-N', '-T', '-x', '-D', f"{local_port}"]
         cmd.append('-oStrictHostKeyChecking=no')
         cmd.append('-oCompression=yes')
 
@@ -421,6 +420,7 @@ def __init__(self, args):
         #init vars we need to shutdown app properly
         self.config = None
         self.log = None
+        self.ssh_command = 'ssh'
         self.sound = None
         self.ssh_tunnels = dict()
         self.vnc_threads = list()
@@ -429,9 +429,9 @@ def __init__(self, args):
         self.instrument = None
         self.vncserver = None
         self.ssh_key_valid = False
-        self.ssh_additional_kex = '+diffie-hellman-group1-sha1'
-        self.ssh_additional_hostkeyalgo = '+ssh-dss,ssh-rsa'
-        self.ssh_additional_keytypes = '+ssh-dss,ssh-rsa'
+        self.ssh_additional_kex = None
+        self.ssh_additional_hostkeyalgo = None
+        self.ssh_additional_keytypes = None
         self.exit = False
         self.geometry = list()
         self.tigervnc = None
@@ -466,6 +466,13 @@ def start(self):
         self.log.debug("\n***** PROGRAM STARTED *****")
         self.log.debug(f"Command: {' '.join(sys.argv)}")
 
+        ##---------------------------------------------------------------------
+        ## Read configuration
+        self.get_config()
+        self.check_config()
+        if self.args.authonly is False:
+            self.get_vncviewer_properties()
+
         ##---------------------------------------------------------------------
         ## Log basic system info
         self.log_system_info()
@@ -474,13 +481,6 @@ def start(self):
         if self.args.authonly is False:
             self.get_display_info()
 
-        ##---------------------------------------------------------------------
-        ## Read configuration
-        self.get_config()
-        self.check_config()
-        if self.args.authonly is False:
-            self.get_vncviewer_properties()
-
         ##---------------------------------------------------------------------
         # Verify Tiger VNC Config
         if self.args.authonly is False:
@@ -594,9 +594,9 @@ def log_system_info(self):
             self.log.debug(trace)
 
         try:
-            whereisssh = subprocess.check_output(['which', 'ssh'])
+            whereisssh = subprocess.check_output(['which', self.ssh_command])
             self.log.debug(f'SSH command is {whereisssh.decode().strip()}')
-            sshversion = subprocess.check_output(['ssh', '-V'],
+            sshversion = subprocess.check_output([self.ssh_command, '-V'],
                                     stderr=subprocess.STDOUT)
             self.log.debug(f'SSH version is {sshversion.decode().strip()}')
         except:
@@ -744,7 +744,7 @@ def get_config(self):
         # open file a second time to properly read config
         config = yaml.load(open(file), Loader=yaml.FullLoader)
 
-        for key in ['ssh_pkey', 'vncviewer', 'soundplayer', 'aplay']:
+        for key in ['ssh_path', 'ssh_pkey', 'vncviewer', 'soundplayer', 'aplay']:
             if key in config.keys():
                 config[key] = os.path.expanduser(config[key])
                 config[key] = os.path.expandvars(config[key])
@@ -757,6 +757,7 @@ def get_config(self):
         self.config = config
 
         # Load some values
+        self.ssh_command = self.config.get('ssh_path', 'ssh')
         self.ssh_pkey = self.config.get('ssh_pkey', None)
         lps = self.config.get('local_port_start', None)
         self.local_port = self.LOCAL_PORT_START if lps is None else lps
@@ -993,7 +994,7 @@ def do_ssh_cmd(self, cmd, server, account):
             self.log.debug('Extending timeout for svncserver connections')
             timeout = 60
 
-        command = ['ssh', server, '-l', account, '-T', '-x']
+        command = [self.ssh_command, server, '-l', account, '-T', '-x']
         if self.args.verbose is True:
             command.append('-v')
             command.append('-v')
@@ -1080,9 +1081,9 @@ def get_vnc_server(self, account, instrument):
                 self.log.error(f'Could not determine VNC server from API')
 
         #cmd line option
-#         if self.args.vncserver is not None:
-#             self.log.info("Using VNC server defined on command line")
-#             vncserver = self.args.vncserver
+        if self.args.vncserver is not None:
+            self.log.info("Using VNC server defined on command line")
+            vncserver = self.args.vncserver
 
         if vncserver:
             self.log.info(f"Got VNC server: '{vncserver}'")
@@ -1234,6 +1235,7 @@ def open_ssh_tunnel(self, server, username, ssh_pkey, remote_port,
         if server in ['vm-k1obs.keck.hawaii.edu', 'vm-k2obs.keck.hawaii.edu']:
             self.log.debug('Using proxy jump to open SSH tunnel')
             t = SSHTunnel(server, username, ssh_pkey, remote_port, local_port,
+                          ssh_command=self.ssh_command,
                           session_name=session_name,
                           timeout=self.config.get('ssh_timeout', 10),
                           ssh_additional_kex=self.ssh_additional_kex,
@@ -1242,6 +1244,7 @@ def open_ssh_tunnel(self, server, username, ssh_pkey, remote_port,
                           proxy_jump='mosfire.keck.hawaii.edu')
         else:
             t = SSHTunnel(server, username, ssh_pkey, remote_port, local_port,
+                          ssh_command=self.ssh_command,
                           session_name=session_name,
                           timeout=self.config.get('ssh_timeout', 10),
                           ssh_additional_kex=self.ssh_additional_kex,
@@ -1268,6 +1271,7 @@ def open_ssh_for_proxy(self):
         t = SSHProxy(proxy_server,
                      self.kvnc_account, self.ssh_pkey,
                      local_port,
+                     ssh_command=self.ssh_command,
                      session_name='proxy',
                      timeout=self.config.get('ssh_timeout', 10),
                      ssh_additional_kex=self.ssh_additional_kex,
@@ -1955,21 +1959,6 @@ def test_ssh_key_format(self):
         with open(self.ssh_pkey, 'r') as f:
             contents = f.read()
 
-        # Check if this is an RSA key
-#         foundrsa = re.search('BEGIN RSA PRIVATE KEY', contents)
-#         if not foundrsa:
-#             self.log.error(f"Your private key does not appear to be an RSA key")
-#             failcount += 1
-
-        # Check if this is an OPENSSH key
-        foundopenssh = re.search('BEGIN OPENSSH PRIVATE KEY', contents)
-        if foundopenssh:
-            self.log.warning(f"Your SSH key may or may not be formatted correctly.")
-            self.log.warning(f"If no other tests fail and you can connect to the Keck VNCs,")
-            self.log.warning(f"then you can ignore this message.  If you can not connect,")
-            self.log.warning(f"then try regenerating and uploading your SSH key and make")
-            self.log.warning(f"sure you use the `-m PEM` option when generating the key.")
-
         # Check that there is no passphrase
         foundencrypt = re.search(r'Proc-Type: \d,ENCRYPTED', contents)
         if foundencrypt:

start_keck_viewers

@@ -1,34 +1,35 @@
-#!/bin/bash
+#!/bin/bash -i
 
 # If we're using conda python
-if [ "$CONDA_PREFIX" != "" ]; then
-    # activate conda environment
-    # NOTE: The KRO environment is created with:
-    #       conda env create -f environment.yaml
-    source $CONDA_PREFIX/etc/profile.d/conda.sh
-    conda deactivate
-    conda activate KRO
+CONDAEXE=$(which conda)
+echo "CONDAEXE:   $CONDAEXE"
+
+if [ "$CONDAEXE" != "" ]; then
+    CONDA_BASE=$(conda info --base)
+    echo "CONDA_BASE: $CONDA_BASE"
+    source $CONDA_BASE/etc/profile.d/conda.sh
+
+    KROLINE=$(conda info --envs | grep KRO)
+    echo "KROLINE:    $KROLINE"
 
-    #change to script dir (so we don't need full path to keck_vnc_launcher.py)
     DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
-    cd $DIR
+    echo "DIR:        $DIR"
 
-    # Launch the script
-    if [ -e $CONDA_PREFIX/envs/KRO/bin/python3 ]; then
-        # Launch using conda KRO environment
-        $CONDA_PREFIX/envs/KRO/bin/python3 keck_vnc_launcher.py $@
+    if [ "$KROLINE" != "" ]; then
+        echo "Launching using conda KRO environment"
+        conda activate KRO
     else
-        # Try launching via conda base environment
-        $CONDA_PREFIX/bin/python3 keck_vnc_launcher.py $@
+        echo "Launching using conda current environment"
     fi
+    python3 $DIR/keck_vnc_launcher.py -c $DIR/local_config.yaml $@
 else
     echo "We are unable to determine the correct python version to run the"
     echo "Remote Observing software.  We will now try a generic python3 call,"
     echo "if this fails, simply execute the keck_vnc_launcher.py file using the"
     echo "correct python version for your system and use the same arguments you"
     echo "would use with the start_keck_viewers script.  For example:"
     echo "  /path/to/python3 keck_vnc_launcher.py numbered_account"
-    echo "Of coure, you should use the proper path to your python executable"
+    echo "Of course, you should use the proper path to your python executable"
     echo "and the correct numbered account."
 
     # just try whatever is in the path and hope it works