diff --git a/README.md b/README.md index 9002fd21..8aa3cb5d 100644 --- a/README.md +++ b/README.md @@ -6,9 +6,9 @@ A scheduler which aims to distribute OpenShift clusters among a pool of vCenters number vCenters, datacenter, and clusters, ensuring that an OpenShift cluster is being installed in to a environment with sufficient capactiy. -![overview](/doc/vSphere%20Resource%20Manager.png) +User-focused diagrams and walkthroughs: [doc/README.md](doc/README.md) (see [How it works](doc/how-it-works.md)). -## Teminology +## Terminology ### Pools @@ -35,7 +35,7 @@ metadata: labels: boskos-lease-id: "test-id" spec: - requiredPool: + required-pool: vcpus: 24 memory: 96 networks: 1 @@ -175,9 +175,9 @@ A pool can be excluded from consideration unless a lease specifically requests i unique environment, or configuration, which warrants intentional scheduling to the pool. To exclude a pool from scheduling, set `spec.exclude` to true. -To request a specific pool, a Lease must set `spec.requiredPool` to the name of the pool. +To request a specific pool, a Lease must set `spec.required-pool` to the **metadata name** of the pool. -TO-DO: implement a poolSelector paradigm +To restrict scheduling by **labels**, use `spec.poolSelector` (key/value map on pool labels). To restrict by **dedicated hardware or queues**, use **taints** on the Pool and **tolerations** on the Lease. See [doc/scheduling.md](doc/scheduling.md). ## Networks diff --git a/doc/README.md b/doc/README.md new file mode 100644 index 00000000..0a8196ac --- /dev/null +++ b/doc/README.md @@ -0,0 +1,22 @@ +# vSphere Capacity Manager — user documentation + +The vSphere Capacity Manager is a Kubernetes operator that tracks **capacity** (vCPU, memory, networks) per vSphere failure domain and **fulfills Leases** by choosing a **Pool** and **Network** that satisfy each request. + +## Contents + +| Document | Audience | +|----------|----------| +| [Concepts](concepts.md) | What Pool, Lease, and Network mean | +| [How it works](how-it-works.md) | Reconciliation flow and diagrams | +| [Scheduling](scheduling.md) | `poolSelector`, taints, tolerations, exclude / noSchedule | +| [Purpose-built networks](networks-purpose-built.md) | Adding a Network CR and wiring it to a Pool | +| [CLI](cli.md) | `oc` / `kubectl` and the optional `oc-vcm` plugin | +| [Pools and networks inventory](inventory-pools-networks.md) | Snapshot of CRs in one environment (refresh manually) | +| [openshift/release and vsphere-elastic](ci-openshift-release.md) | Boskos, ci-operator `cluster_profile`, step-registry `-vcm` chains | +| [CI / Prow / vsphere-elastic](doc.md) | Job env vars, `SHARED_DIR` files, Vault, step pairs | + +Developer build and test commands remain in the [repository README](../README.md). + +## API group + +All custom resources use API version `vspherecapacitymanager.splat.io/v1`. They are **namespaced**; examples in this repo often use `vsphere-infra-helpers` — use the namespace where your operator runs. diff --git a/doc/ci-openshift-release.md b/doc/ci-openshift-release.md new file mode 100644 index 00000000..e6d113ca --- /dev/null +++ b/doc/ci-openshift-release.md @@ -0,0 +1,85 @@ +# vsphere-elastic, ci-operator, and openshift/release + +This page maps how **OpenShift CI** (Prow, **ci-operator**, and the **step-registry**) reaches the vSphere Capacity Manager. Source of truth for paths below is the **[openshift/release](https://github.com/openshift/release)** repository (for example a local clone at `~/Development/release`). + +## End-to-end flow + +1. **Boskos** hands out an abstract quota slice (names like `vsphere-elastic-0`, `vsphere-elastic-1`, …). Types and resources are defined in [`core-services/prow/02_config/_boskos.yaml`](https://github.com/openshift/release/blob/master/core-services/prow/02_config/_boskos.yaml). +2. **ci-operator** turns a test that declares **`cluster_profile: vsphere-elastic`** in [`ci-operator/config`](https://github.com/openshift/release/tree/master/ci-operator/config) into a ProwJob annotated with **`ci-operator.openshift.io/cloud-cluster-profile: vsphere-elastic`**. Pods for that job see **`CLUSTER_PROFILE_NAME=vsphere-elastic`**. +3. **`ipi-conf-vsphere-check-vcm`** runs only when `CLUSTER_PROFILE_NAME` **is** `vsphere-elastic` (otherwise it exits immediately). It creates **`Lease`** resources (`apiVersion: vspherecapacitymanager.splat.io/v1`) in **`vsphere-infra-helpers`** using **`oc`** and **`SA_KUBECONFIG`** (default in the script: `/var/run/vault/vsphere-ibmcloud-ci/vsphere-capacity-manager-kubeconfig`). It waits until **`status.phase=Fulfilled`**, then writes install metadata under **`${SHARED_DIR}`** (`vsphere_context.sh`, `govc.sh`, `platform.yaml`, `subnets.json`, `LEASE_*.json`, `NETWORK_*.json`, etc.). +4. Other **`*-vcm`** steps read those files. **Legacy** steps (no `-vcm` suffix) do the opposite: they exit early when the profile **is** `vsphere-elastic`, so one workflow can serve both modes. + +```mermaid +sequenceDiagram + participant Prow as Prow_Boskos + participant Step as check_vcm_step + participant API as Lease_API + participant Op as VCM_operator + participant Shared as SHARED_DIR + Prow->>Step: LEASED_RESOURCE_and_secrets + Step->>API: create_Leases + Op->>API: fulfill_Leases + Step->>API: wait_Fulfilled + Step->>Shared: write_platform_and_context +``` + +Scripts: [`ipi-conf-vsphere-check-vcm-commands.sh`](https://github.com/openshift/release/blob/master/ci-operator/step-registry/ipi/conf/vsphere/check/vcm/ipi-conf-vsphere-check-vcm-commands.sh), legacy sibling [`ipi-conf-vsphere-check-commands.sh`](https://github.com/openshift/release/blob/master/ci-operator/step-registry/ipi/conf/vsphere/check/ipi-conf-vsphere-check-commands.sh). + +## Step-registry chains + +Chains intentionally list **both** legacy and **`-vcm`** steps. Only the branch that matches **`CLUSTER_PROFILE_NAME`** does real work. + +**Standard IPI configure chain** — [`ipi/conf/vsphere/ipi-conf-vsphere-chain.yaml`](https://github.com/openshift/release/blob/master/ci-operator/step-registry/ipi/conf/vsphere/ipi-conf-vsphere-chain.yaml): + +- `ipi-conf-vsphere-check` then `ipi-conf-vsphere-check-vcm` +- `ipi-conf-vsphere-vips` then `ipi-conf-vsphere-vips-vcm` +- `ipi-conf-vsphere-dns`, `ipi-conf`, `ipi-conf-telemetry`, `ipi-conf-vsphere`, `ipi-conf-vsphere-vcm`, … + +**Multi–vCenter IPI configure chain** — [`ipi/conf/vsphere/multi-vcenter/ipi-conf-vsphere-multi-vcenter-chain.yaml`](https://github.com/openshift/release/blob/master/ci-operator/step-registry/ipi/conf/vsphere/multi-vcenter/ipi-conf-vsphere-multi-vcenter-chain.yaml): same check/vips split, then `ipi-conf-vsphere-multi-vcenter` and `ipi-conf-vsphere-vcm`. + +## Job configuration (ci-operator) + +Set the cluster profile on the test (exact YAML shape depends on repo and file): + +```yaml +tests: +- as: example-e2e + steps: + cluster_profile: vsphere-elastic + env: + POOLS: "" # optional: space-separated pool metadata names + POOL_COUNT: "1" # pools to request when not using POOLS + POOL_SELECTOR: "region=us-east" # optional: comma-separated key=value → Lease poolSelector + NETWORK_TYPE: single-tenant # or multi-tenant, nested-multi-tenant, … + OPENSHIFT_REQUIRED_CORES: "24" + OPENSHIFT_REQUIRED_MEMORY: "96" + workflow: openshift-e2e-vsphere-… +``` + +Authoritative env list and defaults for the check step: [`ipi-conf-vsphere-check-vcm-ref.yaml`](https://github.com/openshift/release/blob/master/ci-operator/step-registry/ipi/conf/vsphere/check/vcm/ipi-conf-vsphere-check-vcm-ref.yaml). Additional behavior (multi-NIC, multi-network failure domains, Vault-driven defaults) is described in comments at the top of `ipi-conf-vsphere-check-vcm-commands.sh`. + +## Selectors and tolerations from CI + +| Mechanism | In CI today | +|-----------|-------------| +| **`POOL_SELECTOR`** | Implemented: comma-separated `key=value` pairs are turned into **`spec.poolSelector`** on each created **Lease**. | +| **Tolerations** | Supported on the **Lease** API; see [scheduling.md](scheduling.md). The **check-vcm** script does **not** set tolerations from an environment variable. To use pool taints from Prow, you would extend the step or apply a custom manifest. | + +## Other `-vcm` steps (step-registry) + +All under **`ci-operator/step-registry/`**: + +| Step | Role (high level) | +|------|---------------------| +| [`ipi/conf/vsphere/check/vcm`](https://github.com/openshift/release/tree/master/ci-operator/step-registry/ipi/conf/vsphere/check/vcm) | Create/wait on **Leases**; populate **`SHARED_DIR`**. | +| [`ipi/conf/vsphere/vips/vcm`](https://github.com/openshift/release/tree/master/ci-operator/step-registry/ipi/conf/vsphere/vips/vcm) | VIP handling for VCM-derived config. | +| [`ipi/conf/vsphere/vcm`](https://github.com/openshift/release/tree/master/ci-operator/step-registry/ipi/conf/vsphere/vcm) | Build **`install-config.yaml`** from VCM context. | +| [`upi/conf/vsphere/vcm`](https://github.com/openshift/release/tree/master/ci-operator/step-registry/upi/conf/vsphere/vcm) | UPI configure path when profile is `vsphere-elastic`. | +| [`upi/conf/vsphere/ova/vcm`](https://github.com/openshift/release/tree/master/ci-operator/step-registry/upi/conf/vsphere/ova/vcm) | OVA step sibling for elastic profile. | +| [`ipi/deprovision/vsphere/diags/vcm`](https://github.com/openshift/release/tree/master/ci-operator/step-registry/ipi/deprovision/vsphere/diags/vcm) | Diagnostics deprovision for VCM workflows. | + +## See also + +- [doc.md](doc.md) — tables of **`SHARED_DIR`** files, VCM vs legacy step pairs, multi-tenant Vault lists, and job examples. +- [scheduling.md](scheduling.md) — `poolSelector`, taints, tolerations on the CRs themselves. +- [concepts.md](concepts.md) — Pool, Lease, Network. diff --git a/doc/cli.md b/doc/cli.md new file mode 100644 index 00000000..bd41eff6 --- /dev/null +++ b/doc/cli.md @@ -0,0 +1,33 @@ +# CLI reference + +Replace the namespace if yours differs from `vsphere-infra-helpers`. + +## List and inspect CRs + +```sh +NS=vsphere-infra-helpers + +oc get pools.vspherecapacitymanager.splat.io -n "$NS" -o wide +oc get leases.vspherecapacitymanager.splat.io -n "$NS" -o wide +oc get networks.vspherecapacitymanager.splat.io -n "$NS" +``` + +Describe a single object: + +```sh +oc describe pool.vspherecapacitymanager.splat.io/ -n "$NS" +``` + +## Optional `oc-vcm` plugin + +The repo ships a helper script — see [repository README](../README.md#oc-plugin-installation). After installing: + +```sh +oc vcm +``` + +Subcommands include `status`, `networks`, pool cordon/uncordon, exclude/include, VLAN helpers, etc. + +## Inventory snapshot + +To regenerate the tables in [inventory-pools-networks.md](inventory-pools-networks.md), use the refresh section at the bottom of that file. diff --git a/doc/concepts.md b/doc/concepts.md new file mode 100644 index 00000000..60cd5006 --- /dev/null +++ b/doc/concepts.md @@ -0,0 +1,39 @@ +# Concepts + +## Pool + +A **Pool** is one schedulable slice of vSphere capacity: vCenter connection, datacenter / cluster / datastore topology, total vCPU and memory, and the list of **port group paths** that may be used for installs. + +- **Status** fields (`vcpus-available`, `memory-available`, `network-available`, `lease-count`) reflect what the operator thinks is still free after fulfilled leases. +- **exclude**: pool is skipped by default scheduling; a lease can still target it with `spec.required-pool` (or match via labels/tolerations as documented in [scheduling](scheduling.md)). +- **noSchedule**: like cordoning a node — existing leases stay; **new** leases are not placed here. + +## Lease + +A **Lease** is a request for resources: vCPU, memory, number of networks (today **`spec.networks` is 1**), optional storage, and optional **network type** (single-tenant, multi-tenant, etc.). + +When the operator can place the lease, **status.phase** becomes **Fulfilled** and status carries failure-domain and env information consumers use to build install configs. + +For **multiple failure domains** (e.g. multiple vSphere clusters), create **one Lease per domain**. + +## Network + +A **Network** CR describes one vSphere **port group** at a given **pod** / **datacenter**: VLAN, machine CIDR, gateways, etc. Only networks that are both **listed on a Pool** and **not already owned by another lease** can be assigned. + +See [Purpose-built networks](networks-purpose-built.md) for how to add one. + +## How they connect + +```mermaid +flowchart LR + subgraph resources [Custom resources] + L[Lease] + P[Pool] + N[Network] + end + L -->|scheduled onto| P + P -->|topology lists port groups| N + L -->|claims| N +``` + +The operator’s job is to pick a **Pool** with enough free capacity, then a **Network** that matches the lease’s **network-type** and is tied to that pool. diff --git a/doc/doc.md b/doc/doc.md index 34232f16..d9d3346a 100644 --- a/doc/doc.md +++ b/doc/doc.md @@ -1,5 +1,9 @@ # Overview +Operator concepts, scheduling, and CLI: see [doc/README.md](README.md). + +Where this fits in **openshift/release** (Boskos, chains, `ipi-conf-vsphere-check-vcm`): [ci-openshift-release.md](ci-openshift-release.md). + vSphere CI requires the use of mutliple environments in order to support the number of jobs and various required configurations. Historically, this has been handled by the creation of purpose targeted lease pools. While this has worked, some environments are overutilized while some environments are idle. The VCM handles scheduling jobs to the most appropriate environment based on the requirements of the job and the utilization of environments. # Job Configuration diff --git a/doc/how-it-works.md b/doc/how-it-works.md new file mode 100644 index 00000000..a5928156 --- /dev/null +++ b/doc/how-it-works.md @@ -0,0 +1,31 @@ +# How it works + +## High-level flow + +1. A **Lease** is created (or updated) with CPU, memory, network count, and optional scheduling constraints. +2. The operator finds **Pool**(s) that fit capacity and policy ([scheduling](scheduling.md)). +3. For each pool, it looks for a free **Network** compatible with `spec.network-type`. +4. When successful, it updates **Lease status** (phase **Fulfilled**, pool info, env snippets) and records ownership so the network is not double-booked. + +```mermaid +stateDiagram-v2 + [*] --> Pending + Pending --> Partial: partial allocation + Pending --> Fulfilled: all requirements met + Partial --> Fulfilled: remaining work done + Pending --> Failed: unrecoverable error + Fulfilled --> [*]: lease released + Failed --> [*]: lease released +``` + +Phases are defined in the API (for example `Pending`, `Partial`, `Fulfilled`, `Failed`). Conditions on the Lease give more detail while work is in progress. + +## Related leases and networks + +When several leases share the same **boskos-lease-id** label and the **same vCenter**, the operator tries to give them a **consistent network** story so multi–failure-domain jobs can coordinate. (See [repository README](../README.md) for the short bullet list.) + +## Where to go next + +- [Scheduling](scheduling.md) — labels, taints, `required-pool` +- [CLI](cli.md) — inspect Pools, Leases, Networks +- [CI-focused detail](doc.md) — Prow, `vsphere-elastic`, files under `SHARED_DIR` diff --git a/doc/inventory-pools-networks.md b/doc/inventory-pools-networks.md new file mode 100644 index 00000000..e5ac3356 --- /dev/null +++ b/doc/inventory-pools-networks.md @@ -0,0 +1,162 @@ +# Pools and networks inventory + +This file is a **point-in-time snapshot** of custom resources in the management cluster. It is not regenerated automatically when the cluster changes. + +| Field | Value | +|--------|--------| +| Captured (UTC) | 2026-03-24 12:50 | +| `kubectl` / `oc` context | `default/api-build02-vmc-ci-openshift-org:6443/jcallen` | +| Namespace | `vsphere-infra-helpers` | + +## Pools + +| Name | vCenter (`spec.server`) | Port groups in spec | Excluded | NoSchedule | Taints | vCPUs avail | Memory (GB) avail | Networks avail | Leases | +|------|---------------------------|---------------------|----------|------------|--------|-------------|------------------|----------------|--------| +| `vcenter-1.ci.ibmc.devcluster.openshift.com-cidatacenter-2-cicluster-3` | vcenter-1.ci.ibmc.devcluster.openshift.com | 69 | false | false | 0 | 416 | 3200 | 34 | 10 | +| `vcenter-1.devqe.ibmc.devcluster.openshift.com-devqedatacenter-1-devqecluster-1` | vcenter-1.devqe.ibmc.devcluster.openshift.com | 18 | true | false | 0 | 256 | 2047 | 18 | 0 | +| `vcenter-110.ci.ibmc.devcluster.openshift.com-vcenter-110-dc01-vcenter-110-cl01` | vcenter-110.ci.ibmc.devcluster.openshift.com | 73 | false | false | 0 | 240 | 3520 | 38 | 6 | +| `vcenter-120.ci.ibmc.devcluster.openshift.com-wldn-120-dc-wldn-120-cl01` | vcenter-120.ci.ibmc.devcluster.openshift.com | 61 | false | false | 0 | 235 | 3808 | 29 | 3 | +| `vcenter-130.ci.ibmc.devcluster.openshift.com-wldn-130-dc-wldn-130-cl01` | vcenter-130.ci.ibmc.devcluster.openshift.com | 61 | true | false | 0 | 422 | 4096 | 29 | 0 | +| `vcenter-7-nested-dal10.pod03` | 10.93.60.135 | 4 | true | false | 0 | 96 | 384 | 3 | 0 | +| `vcenter.ci.ibmc.devcluster.openshift.com-cidatacenter-1-cicluster-1` | vcenter.ci.ibmc.devcluster.openshift.com | 69 | false | false | 0 | 180 | 719 | 34 | 5 | +| `vcenter.ci.ibmc.devcluster.openshift.com-cidatacenter-1-cicluster-2` | vcenter.ci.ibmc.devcluster.openshift.com | 70 | false | false | 0 | 176 | 703 | 35 | 4 | +| `vcenter.ci.ibmc.devcluster.openshift.com-cidatacenter-cicluster` | vcenter.ci.ibmc.devcluster.openshift.com | 70 | false | false | 0 | 372 | 3024 | 35 | 12 | +| `vcenter.devqe.ibmc.devcluster.openshift.com-devqedatacenter-devqecluster` | vcenter.devqe.ibmc.devcluster.openshift.com | 18 | true | true | 0 | 192 | 3070 | 18 | 0 | + +## Networks + +| Name | Port group | VLAN | Pod | Datacenter | Machine CIDR | `network-type` label | +|------|------------|------|-----|--------------|--------------|----------------------| +| `ci-vlan-1108-1-dal10-dal10.pod03` | ci-vlan-1108 | 1108 | dal10.pod03 | dal10 | 10.221.127.0/25 | nested-multi-tenant | +| `ci-vlan-1108-1-dal10-dal10.pod03-1` | ci-vlan-1108-1 | 1108 | dal10.pod03 | dal10 | 10.221.127.0/25 | multi-tenant | +| `ci-vlan-1108-1-dal10-dal10.pod03-2` | ci-vlan-1108-2 | 1108 | dal10.pod03 | dal10 | 10.221.127.0/25 | multi-tenant | +| `ci-vlan-1108-1-dal10-dal10.pod03-3` | ci-vlan-1108-3 | 1108 | dal10.pod03 | dal10 | 10.221.127.0/25 | multi-tenant | +| `ci-vlan-1108-1-dal10-dal10.pod03-4` | ci-vlan-1108-4 | 1108 | dal10.pod03 | dal10 | 10.221.127.0/25 | multi-tenant | +| `ci-vlan-1140-dal12-dal12.pod01` | ci-vlan-1140 | 1140 | dal12.pod01 | dal12 | 10.241.55.128/25 | single-tenant | +| `ci-vlan-1146-dal12-dal12.pod01` | ci-vlan-1146 | 1146 | dal12.pod01 | dal12 | 10.241.82.0/25 | single-tenant | +| `ci-vlan-1146-disconneted-dal12-dal12.pod01` | ci-vlan-1146-disconneted | 1146 | dal12.pod01 | dal12 | 10.241.82.0/25 | single-tenant | +| `ci-vlan-1148-dal10-dal10.pod03` | ci-vlan-1148 | 1148 | dal10.pod03 | dal10 | 10.93.43.128/25 | nested-multi-tenant | +| `ci-vlan-1148-dal10-dal10.pod03-2` | ci-vlan-1148-1 | 1148 | dal10.pod03 | dal10 | 10.93.43.128/25 | multi-tenant | +| `ci-vlan-1148-dal10-dal10.pod03-3` | ci-vlan-1148-2 | 1148 | dal10.pod03 | dal10 | 10.93.43.128/25 | multi-tenant | +| `ci-vlan-1148-dal10-dal10.pod03-4` | ci-vlan-1148-3 | 1148 | dal10.pod03 | dal10 | 10.93.43.128/25 | multi-tenant | +| `ci-vlan-1148-dal10-dal10.pod03-5` | ci-vlan-1148-4 | 1148 | dal10.pod03 | dal10 | 10.93.43.128/25 | multi-tenant | +| `ci-vlan-1148-dal12-dal12.pod01` | ci-vlan-1148 | 1148 | dal12.pod01 | dal12 | 10.241.103.0/25 | single-tenant | +| `ci-vlan-1154-dal12-dal12.pod01` | ci-vlan-1154 | 1154 | dal12.pod01 | dal12 | 10.241.96.128/25 | single-tenant | +| `ci-vlan-1158-dal12-dal12.pod01` | ci-vlan-1158 | 1158 | dal12.pod01 | dal12 | 10.184.20.128/25 | single-tenant | +| `ci-vlan-1161-dal10-dal10.pod03` | ci-vlan-1161 | 1161 | dal10.pod03 | dal10 | 10.93.63.128/25 | single-tenant | +| `ci-vlan-1164-dal12-dal12.pod01` | ci-vlan-1164 | 1164 | dal12.pod01 | dal12 | 10.241.111.128/25 | single-tenant | +| `ci-vlan-1164-disconneted-dal12-dal12.pod01` | ci-vlan-1164-disconneted | 1164 | dal12.pod01 | dal12 | 10.241.111.128/25 | single-tenant | +| `ci-vlan-1165-dal12-dal12.pod01` | ci-vlan-1165 | 1165 | dal12.pod01 | dal12 | 10.241.95.128/25 | single-tenant | +| `ci-vlan-1166-dal12-dal12.pod01` | ci-vlan-1166 | 1166 | dal12.pod01 | dal12 | 10.241.153.128/25 | single-tenant | +| `ci-vlan-1166-disconneted-dal12-dal12.pod01` | ci-vlan-1166-disconneted | 1166 | dal12.pod01 | dal12 | 10.241.153.128/25 | single-tenant | +| `ci-vlan-1169-dal12-dal12.pod01` | ci-vlan-1169 | 1169 | dal12.pod01 | dal12 | 10.241.157.0/25 | single-tenant | +| `ci-vlan-1169-disconneted-dal12-dal12.pod01` | ci-vlan-1169-disconneted | 1169 | dal12.pod01 | dal12 | 10.241.157.0/25 | single-tenant | +| `ci-vlan-1190-dal10-dal10.pod03` | ci-vlan-1190 | 1190 | dal10.pod03 | dal10 | 10.93.254.128/25 | single-tenant | +| `ci-vlan-1197-dal10-dal10.pod03` | ci-vlan-1197 | 1197 | dal10.pod03 | dal10 | 10.38.134.0/25 | single-tenant | +| `ci-vlan-1207-dal10-dal10.pod03` | ci-vlan-1207 | 1207 | dal10.pod03 | dal10 | 10.94.123.128/25 | single-tenant | +| `ci-vlan-1216-dal12-dal12.pod01` | ci-vlan-1216 | 1216 | dal12.pod01 | dal12 | 10.184.36.0/25 | single-tenant | +| `ci-vlan-1220-dal12-dal12.pod01` | ci-vlan-1220 | 1220 | dal12.pod01 | dal12 | 10.184.103.128/25 | single-tenant | +| `ci-vlan-1221-dal12-dal12.pod01` | ci-vlan-1221 | 1221 | dal12.pod01 | dal12 | 10.184.6.128/25 | single-tenant | +| `ci-vlan-1225-dal10-dal10.pod03` | ci-vlan-1225 | 1225 | dal10.pod03 | dal10 | 10.38.83.128/25 | single-tenant | +| `ci-vlan-1227-dal10-dal10.pod03` | ci-vlan-1227 | 1227 | dal10.pod03 | dal10 | 10.38.252.0/25 | single-tenant | +| `ci-vlan-1229-dal10-dal10.pod03` | ci-vlan-1229 | 1229 | dal10.pod03 | dal10 | 10.94.182.0/25 | single-tenant | +| `ci-vlan-1232-dal10-dal10.pod03` | ci-vlan-1232 | 1232 | dal10.pod03 | dal10 | 10.38.192.128/25 | single-tenant | +| `ci-vlan-1233-dal10-dal10.pod03` | ci-vlan-1233 | 1233 | dal10.pod03 | dal10 | 10.38.121.0/25 | single-tenant | +| `ci-vlan-1234-dal10-dal10.pod03` | ci-vlan-1234 | 1234 | dal10.pod03 | dal10 | 10.94.146.128/25 | single-tenant | +| `ci-vlan-1235-dal10-dal10.pod03` | ci-vlan-1235 | 1235 | dal10.pod03 | dal10 | 10.38.221.128/25 | single-tenant | +| `ci-vlan-1237-dal10-dal10.pod03` | ci-vlan-1237 | 1237 | dal10.pod03 | dal10 | 10.38.247.0/25 | single-tenant | +| `ci-vlan-1238-dal10-dal10.pod03` | ci-vlan-1238 | 1238 | dal10.pod03 | dal10 | 10.38.114.128/25 | single-tenant | +| `ci-vlan-1240-dal10-dal10.pod03` | ci-vlan-1240 | 1240 | dal10.pod03 | dal10 | 10.38.202.0/25 | single-tenant | +| `ci-vlan-1243-dal10-dal10.pod03` | ci-vlan-1243 | 1243 | dal10.pod03 | dal10 | 10.38.204.128/25 | single-tenant | +| `ci-vlan-1246-dal10-dal10.pod03` | ci-vlan-1246 | 1246 | dal10.pod03 | dal10 | 10.94.72.128/25 | single-tenant | +| `ci-vlan-1249-dal10-dal10.pod03` | ci-vlan-1249 | 1249 | dal10.pod03 | dal10 | 10.38.110.0/25 | single-tenant | +| `ci-vlan-1254-dal10-dal10.pod03` | ci-vlan-1254 | 1254 | dal10.pod03 | dal10 | 10.5.183.0/25 | single-tenant | +| `ci-vlan-1255-dal10-dal10.pod03` | ci-vlan-1255 | 1255 | dal10.pod03 | dal10 | 10.38.220.0/25 | single-tenant | +| `ci-vlan-1259-dal10-dal10.pod03` | ci-vlan-1259 | 1259 | dal10.pod03 | dal10 | 10.23.6.0/24 | single-tenant | +| `ci-vlan-1260-dal10-dal10.pod03` | ci-vlan-1260 | 1260 | dal10.pod03 | dal10 | 10.93.99.128/25 | single-tenant | +| `ci-vlan-1271-dal10-dal10.pod03` | ci-vlan-1271 | 1271 | dal10.pod03 | dal10 | 10.94.100.0/25 | single-tenant | +| `ci-vlan-1272-dal10-dal10.pod03` | ci-vlan-1272 | 1272 | dal10.pod03 | dal10 | 10.94.173.0/25 | single-tenant | +| `ci-vlan-1274-dal10-dal10.pod03` | ci-vlan-1274 | 1274 | dal10.pod03 | dal10 | 10.93.165.0/25 | single-tenant | +| `ci-vlan-1279-dal10-dal10.pod03` | ci-vlan-1279 | 1279 | dal10.pod03 | dal10 | 10.94.31.128/25 | single-tenant | +| `ci-vlan-1284-dal10-dal10.pod03` | ci-vlan-1284 | 1284 | dal10.pod03 | dal10 | 10.38.84.0/25 | nested-multi-tenant | +| `ci-vlan-1284-dal10-dal10.pod03-1` | ci-vlan-1284-1 | 1284 | dal10.pod03 | dal10 | 10.38.84.0/25 | multi-tenant | +| `ci-vlan-1284-dal10-dal10.pod03-2` | ci-vlan-1284-2 | 1284 | dal10.pod03 | dal10 | 10.38.84.0/25 | multi-tenant | +| `ci-vlan-1284-dal10-dal10.pod03-3` | ci-vlan-1284-3 | 1284 | dal10.pod03 | dal10 | 10.38.84.0/25 | multi-tenant | +| `ci-vlan-1284-dal10-dal10.pod03-4` | ci-vlan-1284-4 | 1284 | dal10.pod03 | dal10 | 10.38.84.0/25 | multi-tenant | +| `ci-vlan-1287-dal10-dal10.pod03` | ci-vlan-1287 | 1287 | dal10.pod03 | dal10 | 10.38.201.128/25 | single-tenant | +| `ci-vlan-1289-dal10-dal10.pod03` | ci-vlan-1289 | 1289 | dal10.pod03 | dal10 | 10.38.153.0/25 | single-tenant | +| `ci-vlan-1296-dal10-dal10.pod03` | ci-vlan-1296 | 1296 | dal10.pod03 | dal10 | 10.94.169.0/25 | single-tenant | +| `ci-vlan-1298-dal10-dal10.pod03` | ci-vlan-1298 | 1298 | dal10.pod03 | dal10 | 10.5.172.0/25 | single-tenant | +| `ci-vlan-1300-dal10-dal10.pod03` | ci-vlan-1300 | 1300 | dal10.pod03 | dal10 | 10.94.27.0/25 | single-tenant | +| `ci-vlan-1302-dal10-dal10.pod03` | ci-vlan-1302 | 1302 | dal10.pod03 | dal10 | 10.94.196.0/25 | single-tenant | +| `ci-vlan-832-dal10-dal10.pod03` | ci-vlan-832 | 832 | dal10.pod03 | dal10 | 10.93.68.0/25 | single-tenant | +| `ci-vlan-847-1-dal10-dal10.pod03` | ci-vlan-847 | 847 | dal10.pod03 | dal10 | 10.93.122.0/25 | single-tenant | +| `ci-vlan-852-1-dal10-dal10.pod03` | ci-vlan-852 | 852 | dal10.pod03 | dal10 | 10.95.248.0/25 | single-tenant | +| `ci-vlan-871-dal12-dal12.pod01` | ci-vlan-871 | 871 | dal12.pod01 | dal12 | 10.241.31.128/25 | single-tenant | +| `ci-vlan-879-1-dal10-dal10.pod03` | ci-vlan-879 | 879 | dal10.pod03 | dal10 | 10.95.242.0/25 | single-tenant | +| `ci-vlan-893-2-dal10-dal10.pod03` | ci-vlan-893 | 893 | dal10.pod03 | dal10 | 10.95.108.0/25 | single-tenant | +| `ci-vlan-894-1-dal10-dal10.pod03` | ci-vlan-894 | 894 | dal10.pod03 | dal10 | 10.95.168.128/25 | single-tenant | +| `ci-vlan-896-1-dal10-dal10.pod03` | ci-vlan-896 | 896 | dal10.pod03 | dal10 | 10.93.93.128/25 | nested-multi-tenant | +| `ci-vlan-910-1-dal10-dal10.pod03` | ci-vlan-910 | 910 | dal10.pod03 | dal10 | 10.95.160.0/25 | mutli-tenant | +| `ci-vlan-910-1-dal10-dal10.pod03-multi-1` | ci-vlan-910-1 | 910 | dal10.pod03 | dal10 | 10.95.160.0/25 | multi-tenant | +| `ci-vlan-910-1-dal10-dal10.pod03-multi-2` | ci-vlan-910-2 | 910 | dal10.pod03 | dal10 | 10.95.160.0/25 | multi-tenant | +| `ci-vlan-910-1-dal10-dal10.pod03-multi-3` | ci-vlan-910-3 | 910 | dal10.pod03 | dal10 | 10.95.160.0/25 | multi-tenant | +| `ci-vlan-910-1-dal10-dal10.pod03-multi-4` | ci-vlan-910-4 | 910 | dal10.pod03 | dal10 | 10.95.160.0/25 | multi-tenant | +| `ci-vlan-910-1-dal10-dal10.pod03-multi-5` | ci-vlan-910-5 | 910 | dal10.pod03 | dal10 | 10.95.160.0/25 | multi-tenant | +| `ci-vlan-910-1-dal10-dal10.pod03-multi-6` | ci-vlan-910-6 | 910 | dal10.pod03 | dal10 | 10.95.160.0/25 | multi-tenant | +| `ci-vlan-910-1-dal10-dal10.pod03-multi-7` | ci-vlan-910-7 | 910 | dal10.pod03 | dal10 | 10.95.160.0/25 | multi-tenant | +| `ci-vlan-918-2-dal10-dal10.pod03` | ci-vlan-918 | 918 | dal10.pod03 | dal10 | 10.93.157.0/25 | mutli-tenant | +| `ci-vlan-918-2-dal10-dal10.pod03-multi-1` | ci-vlan-918-1 | 918 | dal10.pod03 | dal10 | 10.93.157.0/25 | multi-tenant | +| `ci-vlan-918-2-dal10-dal10.pod03-multi-2` | ci-vlan-918-2 | 918 | dal10.pod03 | dal10 | 10.93.157.0/25 | multi-tenant | +| `ci-vlan-918-2-dal10-dal10.pod03-multi-3` | ci-vlan-918-3 | 918 | dal10.pod03 | dal10 | 10.93.157.0/25 | multi-tenant | +| `ci-vlan-918-2-dal10-dal10.pod03-multi-4` | ci-vlan-918-4 | 918 | dal10.pod03 | dal10 | 10.93.157.0/25 | multi-tenant | +| `ci-vlan-918-2-dal10-dal10.pod03-multi-5` | ci-vlan-918-5 | 918 | dal10.pod03 | dal10 | 10.93.157.0/25 | multi-tenant | +| `ci-vlan-918-2-dal10-dal10.pod03-multi-6` | ci-vlan-918-6 | 918 | dal10.pod03 | dal10 | 10.93.157.0/25 | multi-tenant | +| `ci-vlan-918-2-dal10-dal10.pod03-multi-7` | ci-vlan-918-7 | 918 | dal10.pod03 | dal10 | 10.93.157.0/25 | multi-tenant | +| `ci-vlan-922-dal12-dal12.pod01` | ci-vlan-922 | 922 | dal12.pod01 | dal12 | 10.184.15.128/25 | single-tenant | +| `ci-vlan-923-1-dal10-dal10.pod03` | ci-vlan-923 | 923 | dal10.pod03 | dal10 | 10.38.167.128/25 | single-tenant | +| `ci-vlan-937-dal12-dal12.pod01` | ci-vlan-937 | 937 | dal12.pod01 | dal12 | 10.241.72.0/24 | public-ipv6 | +| `ci-vlan-938-1-dal10-dal10.pod03` | ci-vlan-938 | 938 | dal10.pod03 | dal10 | 10.93.251.0/25 | mutli-tenant | +| `ci-vlan-938-1-dal10-dal10.pod03-multi-1` | ci-vlan-938-1 | 938 | dal10.pod03 | dal10 | 10.93.251.0/25 | multi-tenant | +| `ci-vlan-938-1-dal10-dal10.pod03-multi-2` | ci-vlan-938-2 | 938 | dal10.pod03 | dal10 | 10.93.251.0/25 | multi-tenant | +| `ci-vlan-938-1-dal10-dal10.pod03-multi-3` | ci-vlan-938-3 | 938 | dal10.pod03 | dal10 | 10.93.251.0/25 | multi-tenant | +| `ci-vlan-938-1-dal10-dal10.pod03-multi-4` | ci-vlan-938-4 | 938 | dal10.pod03 | dal10 | 10.93.251.0/25 | multi-tenant | +| `ci-vlan-938-1-dal10-dal10.pod03-multi-5` | ci-vlan-938-5 | 938 | dal10.pod03 | dal10 | 10.93.251.0/25 | multi-tenant | +| `ci-vlan-938-1-dal10-dal10.pod03-multi-6` | ci-vlan-938-6 | 938 | dal10.pod03 | dal10 | 10.93.251.0/25 | multi-tenant | +| `ci-vlan-938-1-dal10-dal10.pod03-multi-7` | ci-vlan-938-7 | 938 | dal10.pod03 | dal10 | 10.93.251.0/25 | multi-tenant | +| `ci-vlan-946-1-dal10-dal10.pod03` | ci-vlan-946 | 946 | dal10.pod03 | dal10 | 10.38.133.128/25 | single-tenant | +| `ci-vlan-956-dal10-dal10.pod03` | ci-vlan-956 | 956 | dal10.pod03 | dal10 | 10.93.134.0/25 | single-tenant | +| `ci-vlan-958-1-dal10-dal10.pod03` | ci-vlan-958 | 958 | dal10.pod03 | dal10 | 10.93.152.0/25 | mutli-tenant | +| `ci-vlan-958-1-dal10-dal10.pod03-multi-1` | ci-vlan-958-1 | 958 | dal10.pod03 | dal10 | 10.93.152.0/25 | multi-tenant | +| `ci-vlan-958-1-dal10-dal10.pod03-multi-2` | ci-vlan-958-2 | 958 | dal10.pod03 | dal10 | 10.93.152.0/25 | multi-tenant | +| `ci-vlan-958-1-dal10-dal10.pod03-multi-3` | ci-vlan-958-3 | 958 | dal10.pod03 | dal10 | 10.93.152.0/25 | multi-tenant | +| `ci-vlan-958-1-dal10-dal10.pod03-multi-4` | ci-vlan-958-4 | 958 | dal10.pod03 | dal10 | 10.93.152.0/25 | multi-tenant | +| `ci-vlan-958-1-dal10-dal10.pod03-multi-5` | ci-vlan-958-5 | 958 | dal10.pod03 | dal10 | 10.93.152.0/25 | multi-tenant | +| `ci-vlan-958-1-dal10-dal10.pod03-multi-6` | ci-vlan-958-6 | 958 | dal10.pod03 | dal10 | 10.93.152.0/25 | multi-tenant | +| `ci-vlan-958-1-dal10-dal10.pod03-multi-7` | ci-vlan-958-7 | 958 | dal10.pod03 | dal10 | 10.93.152.0/25 | multi-tenant | +| `ci-vlan-976-1-dal10-dal10.pod03` | ci-vlan-976 | 976 | dal10.pod03 | dal10 | 10.93.117.128/25 | single-tenant | +| `ci-vlan-981-1-dal10-dal10.pod03` | ci-vlan-981 | 981 | dal10.pod03 | dal10 | 10.93.60.128/25 | nested-multi-tenant | +| `ci-vlan-981-1-dal10-dal10.pod03-2` | ci-vlan-981 | 981 | dal10.pod03 | dal10 | 10.93.60.128/25 | multi-tenant | +| `ci-vlan-981-1-dal10-dal10.pod03-3` | ci-vlan-981 | 981 | dal10.pod03 | dal10 | 10.93.60.128/25 | multi-tenant | +| `ci-vlan-981-1-dal10-dal10.pod03-4` | ci-vlan-981 | 981 | dal10.pod03 | dal10 | 10.93.60.128/25 | multi-tenant | +| `ci-vlan-988-dal12-dal12.pod01` | ci-vlan-988 | 988 | dal12.pod01 | dal12 | 10.184.115.0/25 | single-tenant | +| `ci-vlan-990-dal12-dal12.pod01` | ci-vlan-990 | 990 | dal12.pod01 | dal12 | 10.241.112.0/25 | single-tenant | +| `ci-vlan-990-disconneted-dal12-dal12.pod01` | ci-vlan-990-disconneted | 990 | dal12.pod01 | dal12 | 10.241.112.0/25 | single-tenant | +| `ci-vlan-991-dal12-dal12.pod01` | ci-vlan-991 | 991 | dal12.pod01 | dal12 | 10.241.99.128/25 | single-tenant | +| `nested-vcenter-7-vmnetwork` | ci-vlan-981-testing-nested7 | 981 | dal10.pod03 | dal10 | 10.93.60.128/25 | multi-tenant | +| `nested-vcenter-7-vmnetwork-1` | ci-vlan-981-testing-nested7-1 | 981 | dal10.pod03 | dal10 | 10.93.60.128/25 | multi-tenant | +| `nested-vcenter-7-vmnetwork-2` | ci-vlan-981-testing-nested7-2 | 981 | dal10.pod03 | dal10 | 10.93.60.128/25 | multi-tenant | + +## Refreshing this document + +From a machine with cluster access: + +```sh +NS=vsphere-infra-helpers +oc config current-context +oc get pools.vspherecapacitymanager.splat.io -n "$NS" -o wide +oc get networks.vspherecapacitymanager.splat.io -n "$NS" +``` + +To rebuild the tables, re-run the `jq` snippets against `oc get ... -o json` and update the markdown manually. Omit sensitive annotations (for example Vault paths) if you paste full objects. + diff --git a/doc/networks-purpose-built.md b/doc/networks-purpose-built.md new file mode 100644 index 00000000..7fe5f6b9 --- /dev/null +++ b/doc/networks-purpose-built.md @@ -0,0 +1,37 @@ +# Purpose-built networks + +Use this when vSphere already has a port group (VLAN, IP plan) and you want the capacity manager to **allocate it** like any other network. + +## 1. vSphere and IPAM + +- Create or reuse a **distributed port group** (name is what you will put in the CR as `spec.portGroupName`). +- Know **pod**, **datacenter**, VLAN id, gateway, machine CIDR, and any IPv6 fields your installs need. + +## 2. Create the `Network` CR + +- **`metadata.name`**: unique; convention is often `{portgroup}-{datacenter}-{pod}` with suffixes if you shard one VLAN into multiple logical networks. +- Fill **`spec`** to match your subnet (see examples in the [repository README](../README.md) or [doc.md](doc.md)). +- **Optional — lease matching:** set label **`vsphere-capacity-manager.splat-team.io/network-type`** to one of the values allowed on a Lease’s **`spec.network-type`** (`single-tenant`, `multi-tenant`, `nested-multi-tenant`, `public-ipv6`, …). + If the label is **missing**, the operator treats the network as **`single-tenant`**. + +## 3. Attach the network to a `Pool` + +The controller links pools to networks in `getNetworksForPool` (`pkg/controller/leases.go`): + +1. For each path in **`pool.spec.topology.networks`**, take the **basename** (last segment of the path). That string must equal **`network.spec.portGroupName`**. +2. **`network.spec.podName`** must equal **`pool.spec.ibmPoolSpec.pod`**. + +Add the full vSphere inventory path of the port group to **`spec.topology.networks`** on the Pool (same style as existing pools in your cluster). + +## 4. Verify + +```sh +oc get network.vspherecapacitymanager.splat.io -n vsphere-infra-helpers +oc get pool.vspherecapacitymanager.splat.io -n vsphere-infra-helpers -o yaml +``` + +Create a test **Lease** with the right **`network-type`** and ensure it reaches **Fulfilled** using the new network. + +## CI jobs + +For Prow jobs using **`vsphere-elastic`**, network type is often driven by **`NETWORK_TYPE`** and related env — see [doc.md](doc.md). diff --git a/doc/scheduling.md b/doc/scheduling.md new file mode 100644 index 00000000..a35bb538 --- /dev/null +++ b/doc/scheduling.md @@ -0,0 +1,69 @@ +# Scheduling: poolSelector, taints, and tolerations + +This page describes how a **Lease** is matched to **Pool** instances beyond raw capacity. Detailed logic lives in `pkg/utils/pools.go` (`GetFittingPools`, `poolMatchesSelector`, `leaseToleratesPoolTaints`). + +## poolSelector (on the Lease) + +Field: **`spec.poolSelector`** (map of string → string). + +Semantics match **Kubernetes `nodeSelector`**: **every** key in the map must exist on the Pool’s **`metadata.labels`** with the **exact same value**. There is no `In` / `NotIn` / set-based selector. + +- Empty or omitted → no label constraint. +- Example: only pools labeled `region=us-east`: + +```yaml +spec: + poolSelector: + region: us-east +``` + +Ensure the Pool objects carry those labels; otherwise the lease will not schedule. + +## Taints and tolerations + +**Pools** may define **`spec.taints`** (key, optional value, effect `NoSchedule` or `PreferNoSchedule`). + +**Leases** may define **`spec.tolerations`** (operator `Equal` or `Exists`, optional effect). + +Rules: + +- If a pool has **no** taints, any lease may use it (subject to other rules). +- If a pool has taints, the lease must **tolerate every taint** on that pool. One missing toleration disqualifies the pool. +- **`Exists`** with a key can match that taint key regardless of value; empty key with `Exists` is a broad match (see unit tests in `pkg/utils/pools_test.go`). + +Example pool taint: + +```yaml +spec: + taints: + - key: workload + value: gpu + effect: NoSchedule +``` + +Matching lease: + +```yaml +spec: + tolerations: + - key: workload + operator: Equal + value: gpu + effect: NoSchedule +``` + +## Interaction with other pool controls + +| Mechanism | Effect | +|-----------|--------| +| **`spec.exclude`** on Pool | Pool is skipped **unless** the lease names it with **`spec.required-pool`** (exact Pool metadata name). | +| **`spec.noSchedule`** on Pool | Pool cannot take **new** leases; existing ones remain. | +| **`spec.required-pool`** on Lease | Lease may **only** use that pool name if it passes capacity and taint/selector checks. | +| **`poolSelector`** | Pool must match **all** listed labels. | +| **Taints / tolerations** | Every pool taint must be tolerated. | + +Capacity (vCPU, memory, networks), excluded pools, and network availability are still evaluated after these gates. + +## Network type + +Independent of pool selection, the lease’s **`spec.network-type`** (e.g. `single-tenant`, `multi-tenant`) filters which **Network** CRs are eligible; see [Purpose-built networks](networks-purpose-built.md).