Add doc describing provisioning restrictions

A common problem users have is that they can't directly edit the NGINX resources that we provision, and we direct them to our docs on how to set the fields they want.

I'm adding this document here to describe why this restriction is in place. This can help developers on the team understand it, to help explain it if needed, and can also be used as a reference to any users who may be curious.

Author: sjberman

docs/architecture/provisioning.md

@@ -0,0 +1,11 @@
+A common question that comes up from a lot of our users is why they can't edit the NGINX Deployment or Service that our control plane provisions when a Gateway is created. Any time they make changes, they immediately get reverted. This document explains why this is happening.
+
+When a user creates a Gateway resource, the NGINX Gateway Fabric control plane, which is the Deployment that is initially installed in the `nginx-gateway` namespace, provisions an NGINX instance for that Gateway (this is the data plane). This includes a Deployment, Service, ConfigMaps, and other resources required for the system to run. By doing this, the control plane "owns" these resources. Since it's deploying them, it needs to know the configuration for them that a user wants, such as number of replicas, service type, resource limits, and more. A user tells us up front what they want, and when the NGINX resources are provisioned, they are provisioned based on that information. That information is provided to the control plane through the NginxProxy API, contained in a CRD.
+
+A benefit of providing the control plane with this information up front, is that a user doesn't have to go in and manually change the NGINX Deployment or Service as soon as it's deployed to fit their needs. For example, if our control plane simply installed with just default settings -- which means a Service of type LoadBalancer -- but the user always wanted a Service of type NodePort, they're initially going to get a LoadBalancer provisioned, before they can go in and edit the Service themselves to be of type NodePort. Wouldn't it be nice if it was just provisioned as a NodePort? This is why we provide this API for a user to tell us what they want.
+
+Here comes the problem, and why we added the restriction in place to where a user can't ever edit the resource directly (with the exception of annotations, which we've since allowed). The fact that we've introduced a new API (NginxProxy CRD) for configuring Kubernetes resources, means there are now two APIs to do this. Ours, and the Kubernetes API. That's two sources of truth. As a controller, this makes it very difficult to reconcile, in a deterministic way, what the user really wants. If a user omits configuration fields in the NginxProxy CRD, how do we interpret that? Does it mean that we use Kubernetes defaults? Or does it mean that we defer to what the user set directly on the resource through the Kubernetes API? These require different actions from the control plane. Because of this, we consider our own API to be the source of truth.
+
+Now, we considered just allowing users to edit those resources directly and leave those changes in place. But the next time there is a config reconciliation, either because of an edit to the NginxProxy CRD, or due to a controller restart, it's going to revert those changes back to what it thinks the configuration should be, based on what's defined in our API. It's a ticking time bomb. A user may make a ton of changes directly to the NGINX Deployment or Service config, and then time passes. The controller restarts due to an unexpected reason (maybe the Node went down), and now the new controller checks the state of the world, and resets the Deployment or Service to what it expects it should be. That could throw everything into chaos if a user wasn't expecting their Deployment to suddenly change.
+
+Because of these reasons, we decided to restrict the ability for users to edit the NGINX resources directly. It allows us to have a single API as the source of truth, and it prevents unexpected reversions when the control plane reconciles the state of the world.