Merge pull request #243 from DataBiosphere/dev

PR for 0.4.7 release

Author: johntbates

README.md

@@ -17,24 +17,17 @@ and Azure Batch.
 
 ## Getting started
 
-You can install `dsub` from [PyPI](https://pypi.org/project/dsub/), or you can clone and
-install from [github](https://github.com/DataBiosphere/dsub).
+`dsub` is written in Python and requires Python 3.6 or higher.
 
-### Sunsetting Python 2 support
-
-Python 2 support ended in January 2020.
-See Python's official [Sunsetting Python 2 announcement](https://www.python.org/doc/sunset-python-2/) for details.
-
-Automated `dsub` tests running on Python 2 have been disabled.
-[Release 0.3.10](https://github.com/DataBiosphere/dsub/releases/tag/v0.3.10) is
-the last version of `dsub` that supports Python 2.
-
-Use Python 3.6 or greater. For earlier versions of Python 3, use `dsub` 0.4.1.
+* For earlier versions of Python 3, use `dsub` [0.4.1](https://github.com/DataBiosphere/dsub/releases/tag/v0.4.11).
+* For Python 2, use `dsub`[0.3.10](https://github.com/DataBiosphere/dsub/releases/tag/v0.3.10).
 
 ### Pre-installation steps
 
+#### Create a Python virtual environment
+
 This is optional, but whether installing from PyPI or from github,
-you are encouraged to use a
+you are strongly encouraged to use a
 [Python virtual environment](https://docs.python.org/3/library/venv.html).
 
 You can do this in a directory of your choosing.
@@ -56,9 +49,27 @@ virutalenv before calling `dsub`, `dstat`, and `ddel`. They are in the
 use these scripts if you don't want to activate the virtualenv explicitly in
 your shell.
 
+#### Install the Google Cloud SDK
+
+While not used directly by `dsub` for the `google-v2` or `google-cls-v2` providers, you are likely to want to install the command line tools found in the [Google
+Cloud SDK](https://cloud.google.com/sdk/).
+
+If you will be using the `local` provider for faster job development,
+you *will* need to install the Google Cloud SDK, which uses `gsutil` to ensure
+file operation semantics consistent with the Google `dsub` providers.
+
+1. [Install the Google Cloud SDK](https://cloud.google.com/sdk/)
+2. Run
+
+        gcloud init
+
+
+    `gcloud` will prompt you to set your default project and to grant
+    credentials to the Google Cloud SDK.
+
 ### Install `dsub`
 
-Choose one of the following:
+Choose **one** of the following:
 
 #### Install from PyPI
 
@@ -167,12 +178,7 @@ The steps for getting started differ slightly as indicated in the steps below:
 
      [Enable the Cloud Life Sciences, Storage, and Compute APIs](https://console.cloud.google.com/flows/enableapi?apiid=lifesciences.googleapis.com,storage_component,compute_component&redirect=https://console.cloud.google.com)
 
-1.  [Install the Google Cloud SDK](https://cloud.google.com/sdk/) and run
-
-        gcloud init
-
-    This will set up your default project and grant credentials to the Google
-    Cloud SDK. Now provide [credentials](https://developers.google.com/identity/protocols/application-default-credentials)
+1. Provide [credentials](https://developers.google.com/identity/protocols/application-default-credentials)
     so `dsub` can call Google APIs:
 
         gcloud auth application-default login
@@ -423,57 +429,88 @@ specified and they can be specified in any order.
 
 #### Mounting "resource data"
 
-If you have one of the following:
+While explicitly specifying inputs improves tracking provenance of your data,
+there are cases where you might not want to expliclty localize all inputs
+from Cloud Storage to your job VM.
+
+For example, if you have:
+
+- a large set of resource files
+- your code only reads a subset of those files
+- runtime decisions of which files to read
+
+OR
+
+- a large input file over which your code makes a single read pass
+
+OR
+
+- a large input file that your code does not read in its entirety
 
-1. A large set of resource files, your code only reads a subset of those files,
-and the decision of which files to read is determined at runtime, or
-2. A large input file over which your code makes a single read pass or only
-needs to read a small range of bytes,
+then you may find it more efficient or convenient to access this data by
+mounting read-only:
 
-then you may find it more efficient at runtime to access this resource data via
-mounting a Google Cloud Storage bucket read-only or mounting a persistent disk
-created from a
-[Compute Engine Image](https://cloud.google.com/compute/docs/images) read-only.
+- a Google Cloud Storage bucket
+- a persistent disk that you pre-create and populate
+- a persistent disk that gets created from a
+[Compute Engine Image](https://cloud.google.com/compute/docs/images) that you
+pre-create.
 
-The `google-v2` and `google-cls-v2` providers support these two methods of providing access to
-resource data. The `local` provider supports mounting a local directory in a
-similar fashion to support your local development.
+The `google-v2` and `google-cls-v2` providers support these methods of
+providing access to resource data.
+
+The `local` provider supports mounting a
+local directory in a similar fashion to support your local development.
+
+##### Mounting a Google Cloud Storage bucket
 
 To have the `google-v2` or `google-cls-v2` provider mount a Cloud Storage bucket using
 Cloud Storage FUSE, use the `--mount` command line flag:
 
-    --mount MYBUCKET=gs://mybucket
+    --mount RESOURCES=gs://mybucket
 
 The bucket will be mounted into the Docker container running your `--script`
 or `--command` and the location made available via the environment variable
-`${MYBUCKET}`. Inside your script, you can reference the mounted path using the
+`${RESOURCES}`. Inside your script, you can reference the mounted path using the
 environment variable. Please read
 [Key differences from a POSIX file system](https://cloud.google.com/storage/docs/gcs-fuse#notes)
 and [Semantics](https://github.com/GoogleCloudPlatform/gcsfuse/blob/master/docs/semantics.md)
 before using Cloud Storage FUSE.
 
+##### Mounting an existing peristent disk
+
+To have the `google-v2` or `google-cls-v2` provider mount a persistent disk that
+you have pre-created and populated, use the `--mount` command line flag and the
+url of the source disk:
+
+    --mount RESOURCES="https://www.googleapis.com/compute/v1/projects/your-project/global/images/your-image 50"
+
+##### Mounting a persistent disk, created from an image
+
 To have the `google-v2` or `google-cls-v2` provider mount a persistent disk created from an image,
 use the `--mount` command line flag and the url of the source image and the size
 (in GB) of the disk:
 
-    --mount MYDISK="https://www.googleapis.com/compute/v1/projects/your-project/global/images/your-image 50"
+    --mount RESOURCES="https://www.googleapis.com/compute/v1/projects/your-project/global/images/your-image 50"
 
 The image will be used to create a new persistent disk, which will be attached
 to a Compute Engine VM. The disk will mounted into the Docker container running
 your `--script` or `--command` and the location made available by the
-environment variable `${MYDISK}`. Inside your script, you can reference the
+environment variable `${RESOURCES}`. Inside your script, you can reference the
 mounted path using the environment variable.
 
 To create an image, see [Creating a custom image](https://cloud.google.com/compute/docs/images/create-delete-deprecate-private-images).
 
+##### Mounting a local directory (`local` provider)
+
 To have the `local` provider mount a directory read-only, use the `--mount`
 command line flag and a `file://` prefix:
 
-    --mount LOCAL_MOUNT=file://path/to/my/dir
+    --mount RESOURCES=file://path/to/my/dir
 
 The local directory will be mounted into the Docker container running your
 `--script`or `--command` and the location made available via the environment
-variable `${LOCAL_MOUNT}`. Inside your script, you can reference the mounted
+variable `${RESOURCES}`. Inside your script, you can reference the mounted
 path using the environment variable.
 
 ### Setting resource requirements

dsub/_dsub_version.py

@@ -26,4 +26,4 @@
   0.1.3.dev0 -> 0.1.3 -> 0.1.4.dev0 -> ...
 """
 
-DSUB_VERSION = '0.4.6'
+DSUB_VERSION = '0.4.7'

dsub/commands/ddel.py

@@ -1,4 +1,3 @@
-# Lint as: python3
 # Copyright 2016 Google Inc. All Rights Reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");

dsub/commands/dstat.py

@@ -1,4 +1,3 @@
-# Lint as: python3
 # Copyright 2016 Google Inc. All Rights Reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");

dsub/commands/dsub.py

@@ -1,4 +1,3 @@
-# Lint as: python3
 # Copyright 2016 Google Inc. All Rights Reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");

dsub/lib/dsub_errors.py

@@ -1,4 +1,3 @@
-# Lint as: python3
 # Copyright 2017 Google Inc. All Rights Reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");

dsub/lib/dsub_util.py

@@ -1,4 +1,3 @@
-# Lint as: python3
 # Copyright 2016 Google Inc. All Rights Reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");

dsub/lib/job_model.py

@@ -1,4 +1,3 @@
-# Lint as: python3
 # Copyright 2017 Google Inc. All Rights Reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -399,6 +398,14 @@ def __new__(cls, name, value, docker_path, disk_size, disk_type):
         cls, name, value, docker_path, disk_size=disk_size, disk_type=disk_type)
 
 
+class ExistingDiskMountParam(MountParam):
+  """A MountParam representing an existing Google Persistent Disk."""
+
+  def __new__(cls, name, value, docker_path):
+    return super(ExistingDiskMountParam, cls).__new__(cls, name, value,
+                                                      docker_path)
+
+
 class LocalMountParam(MountParam):
   """A MountParam representing a path on the local machine."""
 

dsub/lib/output_formatter.py

@@ -1,4 +1,3 @@
-# Lint as: python3
 # Copyright 2019 Verily Life Sciences Inc. All Rights Reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");

dsub/lib/param_util.py

@@ -1,4 +1,3 @@
-# Lint as: python3
 # Copyright 2016 Google Inc. All Rights Reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -237,8 +236,32 @@ class MountParamUtil(object):
   def __init__(self, docker_path):
     self._relative_path = docker_path
 
-  def _parse_image_uri(self, raw_uri):
-    """Return a valid docker_path from a Google Persistent Disk url."""
+  def _is_gce_disk_uri(self, raw_uri):
+    """Returns true if we can parse the URI as a GCE disk path."""
+
+    # Full disk URI should look something like:
+    # https://www.googleapis.com/compute/<api_version>/projects/<project>/regions/<region>/disks/<disk>
+    # https://www.googleapis.com/compute/<api_version>/projects/<project>/zones/<zone>/disks/<disk>
+    #
+    # This function only returns True if we were able to recognize the path
+    # as clearly for a GCE disk. This is different than the Image path parsing
+    # in "make_param" below, which was made very forgiving.
+
+    if raw_uri.startswith('https://www.googleapis.com/compute'):
+      parts = raw_uri.split('/')
+
+      # Parts will look something like
+      # ['https:', '', 'www.googleapis.com', 'compute', '<version>', 'projects',
+      #  '<project>', '[regions/zones]', '<region/zone>', 'disks', '<disk>']
+      #
+      return ((parts[0] == 'https:') and (not parts[1]) and
+              (parts[2] == 'www.googleapis.com') and (parts[3] == 'compute') and
+              (parts[5] == 'projects') and (parts[9] == 'disks'))
+
+    return False
+
+  def _gce_uri_to_docker_uri(self, raw_uri):
+    """Return a valid docker_path from a GCE disk or image url."""
     # The string replace is so we don't have colons and double slashes in the
     # mount path. The idea is the resulting mount path would look like:
     # /mnt/data/mount/http/www.googleapis.com/compute/v1/projects/...
@@ -263,13 +286,22 @@ def _parse_gcs_uri(self, raw_uri):
     return docker_uri
 
   def make_param(self, name, raw_uri, disk_size):
-    """Return a MountParam given a GCS bucket, disk image or local path."""
-    if raw_uri.startswith('https://www.googleapis.com/compute'):
+    """Return a MountParam given a GCS bucket, disk uri, image uri or local path."""
+
+    if self._is_gce_disk_uri(raw_uri):
+      docker_path = self._gce_uri_to_docker_uri(raw_uri)
+      return job_model.ExistingDiskMountParam(name, raw_uri, docker_path)
+    elif raw_uri.startswith('https://www.googleapis.com/compute'):
+      # In retrospect, this function should have been more precise to only
+      # treat a raw_uri as being for an "Image" if the path followed a known
+      # format. Just checking for the googleapis.com/compute prefix is too
+      # forgiving.
+
       # Full Image URI should look something like:
       # https://www.googleapis.com/compute/v1/projects/<project>/global/images/
       # But don't validate further, should the form of a valid image URI
       # change (v1->v2, for example)
-      docker_path = self._parse_image_uri(raw_uri)
+      docker_path = self._gce_uri_to_docker_uri(raw_uri)
       return job_model.PersistentDiskMountParam(
           name, raw_uri, docker_path, disk_size, disk_type=None)
     elif raw_uri.startswith('file://'):
@@ -374,6 +406,11 @@ def get_persistent_disk_mounts(mounts):
   return _get_filtered_mounts(mounts, job_model.PersistentDiskMountParam)
 
 
+def get_existing_disk_mounts(mounts):
+  """Returns the existing disk mounts from mounts."""
+  return _get_filtered_mounts(mounts, job_model.ExistingDiskMountParam)
+
+
 def get_local_mounts(mounts):
   """Returns the local mounts from mounts."""
   return _get_filtered_mounts(mounts, job_model.LocalMountParam)