NM-341-v6: AMB Multi-tenancy Management

[VishalDalwadi]

Describe your changes

Provide Issue ticket number if applicable/not in title

Provide testing steps

Checklist before requesting a review

• My changes affect only 10 files or less.
• I have performed a self-review of my code and tested it.
• If it is a new feature, I have added thorough tests, my code is <= 1450 lines.
• If it is a bugfix, my code is <= 200 lines.
• My functions are <= 80 lines.
• I have had my code reviewed by a peer.
• My unit tests pass locally.
• Netmaker is awesome.

[abhishek9686]

@tenki-reviewer

[tenki-reviewer]

Review complete. 13 potential issues to review.

Files Reviewed: 221
Findings: 13

By Severity:

• 🔴 Critical: 1
• 🟠 High: 7
• 🟡 Medium: 5

---

This PR introduces a multi-tenant architecture with critical authorization bypasses, cross-tenant data leakage, and broken multi-tenancy isolation. The org-owner creation endpoint is unauthenticated, the license enforcement can be silently disabled, and several routes bypass tenant-scoped queries — any of these could allow tenants to access or delete other tenants' data.

Files Reviewed (221 files)

auth/host_session.go
cli/cmd/user/roles.go
clickhouse/clickhouse.go
clickhouse/initdb.d/02_create_flows_table.sql
controllers/acls.go
controllers/controller.go
controllers/device.go
controllers/dns.go
controllers/dns_test.go
controllers/egress.go
controllers/enrollmentkeys.go
controllers/ext_client.go
controllers/gateway.go
controllers/hosts.go
controllers/inet_gws.go
controllers/middleware.go
controllers/network.go
controllers/network_test.go
controllers/node.go
controllers/node_test.go
controllers/org.go
controllers/server.go
controllers/user.go
grpc/auditlogs/auditlogs.pb.go
grpc/auditlogs/auditlogs.proto
grpc/flow/flow.pb.go
grpc/flow/flow.proto
grpc/siem/client.go
grpc/siem/siem.pb.go
grpc/siem/siem.proto
grpc/siem/siem_grpc.pb.go
logic/acls.go
logic/acls_selected_ips_test.go
logic/auth.go
logic/clients.go
logic/device.go
logic/device_auth.go
logic/dns.go
logic/egress.go
logic/enrollmentkey.go
logic/exit_node.go
logic/extpeers.go
logic/gateway.go
logic/hooks.go
logic/host_test.go
logic/hosts.go
logic/jit.go
logic/jwts.go
logic/metrics.go
logic/networks.go
logic/nodes.go
logic/peers.go
logic/pro/netcache/netcache.go
logic/relay.go
logic/scope.go
logic/security.go
logic/server.go
logic/serverconf.go
logic/settings.go
logic/status.go
logic/telemetry.go
logic/usage.go
logic/user_mgmt.go
logic/users.go
logic/util.go
logic/zombie.go
main.go
middleware/scope.go
migrate/migrate.go
migrate/migrate_internet_egress.go
migrate/migrate_multitenancy.go
migrate/migrate_v1_5_1.go
migrate/migrate_v1_7_0.go
migrate/types/types_v1_7_0.go
migrate/utils.go
models/api_host.go
models/api_node.go
models/enrollment_key.go
models/events.go
models/node.go
models/structs.go
models/user_mgmt.go
mq/device_hooks.go
mq/handlers.go
mq/mq.go
mq/publishers.go
mq/serversync.go
orchestrator/extensions/factory.go
orchestrator/extensions/node.go
orchestrator/extensions/user.go
orchestrator/network.go
orchestrator/node.go
orchestrator/node_test.go
orchestrator/options.go
orchestrator/repository.go
orchestrator/tenants.go
orchestrator/user.go
pro/auth/auth.go
pro/auth/azure-ad.go
pro/auth/error.go
pro/auth/github.go
pro/auth/google.go
pro/auth/headless_callback.go
pro/auth/oidc.go
pro/auth/okta.go
pro/auth/provider.go
pro/auth/register_callback.go
pro/auth/registry.go
pro/auth/sync.go
pro/auth/validate_email.go
pro/controllers/auto_relay.go
pro/controllers/flows.go
pro/controllers/integrations.go
pro/controllers/jit.go
pro/controllers/metrics.go
pro/controllers/middleware.go
pro/controllers/networks.go
pro/controllers/posture_check.go
pro/controllers/server.go
pro/controllers/tags.go
pro/controllers/users.go
pro/email/email.go
pro/email/jit_request.go
pro/email/smtp.go
pro/email/validate.go
pro/idp/azure/azure.go
pro/idp/google/google.go
pro/idp/okta/okta.go
pro/initialize.go
pro/integration/edr/active.go
pro/integration/edr/lookup.go
pro/integration/edr/sentinelone/sentinelone.go
pro/integration/edr/sync.go
pro/integration/edr/sync_host.go
pro/integration/edr/sync_host_id.go
pro/integration/edr/sync_test.go
pro/integration/mdm/active.go
pro/integration/mdm/config.go
pro/integration/mdm/intune/intune.go
pro/integration/mdm/iru/iru.go
pro/integration/mdm/jamf/compliance.go
pro/integration/mdm/jumpcloud/jumpcloud.go
pro/integration/mdm/lookup.go
pro/integration/mdm/sync.go
pro/integration/mdm/sync_host.go
pro/integration/mdm/sync_serial.go
pro/integration/providers.go
pro/license/cache.go
pro/license/license.go
pro/license/types.go
pro/license/util.go
pro/license/utils.go
pro/logic/acls.go
pro/logic/auto_relay.go
pro/logic/device.go
pro/logic/dns.go
pro/logic/egress.go
pro/logic/egress_test.go
pro/logic/events.go
pro/logic/flows.go
pro/logic/jit.go
pro/logic/metrics.go
pro/logic/migrate.go
pro/logic/nodes.go
pro/logic/posture_check.go
pro/logic/security.go
pro/logic/server.go
pro/logic/siem.go
pro/logic/status.go
pro/logic/tags.go
pro/logic/user_mgmt.go
pro/logic/user_mgmt_role_downgrade_test.go
pro/orchestrator/extensions/factory.go
pro/orchestrator/extensions/node.go
pro/orchestrator/extensions/user.go
pro/orchestrator/node_test.go
pro/remote_access_client.go
schema/acl_record.go
schema/cache_record.go
schema/device_edr_state.go
schema/dns.go
schema/dns_record.go
schema/egress.go
schema/enrollment_keys.go
schema/event.go
schema/extclient_record.go
schema/hosts.go
schema/integrations.go
schema/internal.go
schema/jit_grant.go
schema/jit_request.go
schema/mdm_device_state.go
schema/metrics_record.go
schema/models.go
schema/networks.go
schema/nodes.go
schema/org_memberships.go
schema/org_settings.go
schema/organizations.go
schema/pending_hosts.go
schema/pending_users.go
schema/posture_check.go
schema/posture_check_violations.go
schema/sso_state_record.go
schema/tag_record.go
schema/tenant_memberships.go
schema/tenant_settings.go
schema/tenants.go
schema/user_access_token.go
schema/user_groups.go
schema/user_invites.go
schema/user_roles.go
schema/users.go
schema/utils.go
scope/scope.go
servercfg/serverconf.go
test/utils/host.go
test/utils/network.go
test/utils/node.go
test/utils/org.go
test/utils/tag.go

[tenki-reviewer]

Review complete. 13 potential issues to review.

---

Critical Security Gaps in Multi-Tenant Implementation

This PR (221 files, ~9.5k insertions, ~5k deletions) introduces a multi-tenant rearchitecture across Netmaker. The review uncovered 6 critical/high-severity security issues that undermine tenant isolation:

Authentication & Authorization Bypasses

• Org owner creation is completely unauthenticated in controllers/org.go:28 — any unauthenticated request can create a super-admin across tenants, enabling full multi-tenant takeover.
• User enumeration is possible via the unauthenticated /api/v1/auth/ endpoint in controllers/user.go:44, leaking tenant information.

Cross-Tenant Data Leakage

• Network cleanup in controllers/network.go:254 uses context.TODO() inside a goroutine, losing all tenant/scope context and risking cross-tenant data deletion.
• License enforcement silently disabled in pro/license/utils.go:35 when the cache is unavailable due to a swallowed error from getCachedResp.
• TOCTOU race condition in pro/license/utils.go:45 allows concurrent requests to exceed licensed resource counts.
• MetricsRecord upsert in schema/metrics_record.go:54 does not include tenant_id in the OnConflict update clause, rendering records invisible to scoped queries.

Code Correctness

• Compilation failures exist: logic/device_test.go:219 uses old function signatures without ctx parameters, and pro/logic/security_test.go:36 references a removed FullAccess struct field.

Remediation Priority

1. Add authentication to the org-owner creation endpoint immediately
2. Ensure all goroutines propagate tenant context (not context.TODO())
3. Fix the license enforcement error swallowing path
4. Complete the MetricsRecord upsert to update tenant_id on conflict
5. Fix compilation errors in test files

[tenki-reviewer]

Review complete. 39 potential issues to review.

---

Critical Issues (9 findings)

GORM Chain Ordering Bug (finding-001 through finding-007): In 8 Get() methods across schema/ and models/, the pattern db.First(&record).Where("id = ?", id) is used. GORM executes First() immediately, returning the first row in the table before the Where() filter is applied. This means the id parameter is silently ignored and the wrong record is returned from every affected table. Affected files: schema/egress.go, schema/dns.go, schema/pending_hosts.go, schema/posture_check.go, schema/user_access_token.go, schema/event.go, schema/tenant_settings.go, schema/sso_state_record.go, schema/cache_record.go, and models/accessToken.go.

EMQX ACL (finding-008): mq/emqx_on_prem.go:260 creates a default ACL rule granting unrestricted pubsub access on # (all topics) to all authenticated users. This allows any authenticated MQTT client to publish/subscribe to any tenant's topics, bypassing multi-tenant isolation.

Bcrypt Cost Factor (finding-017): logic/auth.go:180 uses bcrypt cost=5, which modern hardware can crack at millions of attempts per second. Industry minimum is 10-12.

Shared OAuth Secret (finding-028): logic/auth.go:68 uses a single shared secret for all SSO/OAuth users, enabling an authenticated user to forge tokens for any other user.

Encryption Chunk Delimiter (finding-012): netclient/ncutils/encryption.go:84 embeds chunk delimiter bytes in the encrypted data stream. If these bytes appear naturally in ciphertext, the decrypted output will be truncated or corrupted.

HTTP Response Handling (finding-011): controllers/enrollmentkeys.go:621 handleHostRegister returns without writing an HTTP response after EMQX user creation failure, causing a client hang.

High Severity Issues (10 findings)

CLI Argument Swaps (finding-013): cli/cmd/gateway/create.go:40 swaps NETWORK NAME and NODE ID in the gateway create command arguments.

Index Panic (finding-014): cli/cmd/host/refresh_keys.go:15 panics with index-out-of-range when called without arguments.

Migration Issues (findings-021, 039): Multi-tenant enrollment key collision in migration 007, and non-atomic check-then-create in migration init.

Performance (findings-022, 026): N+1 queries in migration v1_7 and peer host lookups.

Other high issues: Race in timer hook replacement (finding-015), IPv6 validation gap (finding-016), license cache storing nil (finding-024), orphaned database node (finding-025), missing context.Context in test (finding-018).

Medium Severity Issues (17 findings)

Config YAML decode errors silently ignored, HTTP response body leaks in multiple places, API error responses leaking internal provider data, defense-risk defaulting unrecognized scores to "no risk", and world-readable config directory permissions (0777).

Advisory

This is the initial repository commit with no prior review baseline. The GORM chain-ordering bug alone (8 files) indicates systemic data integrity risk. The EMQX ACL and shared OAuth secret are security-critical architectural flaws. Recommend addressing all critical and high findings before production deployment.