Support graceful script isolation fallback when filesystem does not support locking

[rhysparry]

Script isolation lock directory fallback for filesystems without full locking support

Motivation

Calamari's script isolation mechanism uses filesystem file locks to serialise or permit concurrent script execution. The lock file has historically been written into the Tentacle working directory (TentacleHome).

On some deployment targets the working directory resides on a filesystem that does not fully support file locking semantics — NFS mounts, certain FUSE-based filesystems, and some SMB shares being common examples. On these filesystems, lock acquisition either silently succeeds when it should block, or throws errors that are not meaningful to the caller. The result is that script isolation is silently broken: scripts that should be mutually exclusive may run concurrently, or scripts that should run in parallel are incorrectly serialised (or errored).

This PR introduces runtime filesystem lock capability detection and an automatic fallback to a temporary directory that does support locking, so that script isolation remains as strong as the host OS allows — without requiring any manual configuration from operators.

---

What changed

New abstractions

File	Purpose
ScriptIsolation/LockCapability.cs	Enum: Unsupported, ExclusiveOnly, Supported; unknown capability represented as null (LockCapability?)
ScriptIsolation/LockFile.cs	Sealed record pairing a LockDirectory with a specific FileInfo; exposes Open, Supports, Exists, and Delete, and owns the nested LockHandle implementation
ScriptIsolation/LockDirectory.cs	Sealed record capturing a DirectoryInfo + its detected LockCapability; exposes GetLockFile and Supports(LockType)
ScriptIsolation/LockDirectoryFactory.cs + ILockDirectoryFactory.cs	Injectable factory that implements the fallback algorithm (Create) and the live 4-probe DetectLockSupport logic; injected with IMountedDrivesProvider, IFileLockService, and ITemporaryDirectoryFallbackProvider
ScriptIsolation/CachedDriveInfo.cs	Snapshot of a drive/mount's format, type, and detected lock capability; LockSupport returns a statically-known LockCapability? based on format/drive-type heuristics, or null when the drive is unrecognised (triggering live detection)
ScriptIsolation/MountedDrives.cs + ICachedDriveInfoProvider.cs	Maps a path to its best-matching drive via longest-prefix matching with symlink resolution; implements ICachedDriveInfoProvider
ScriptIsolation/SystemMountedDrivesProvider.cs + IMountedDrivesProvider.cs	Injectable provider that calls DriveInfo.GetDrives() and constructs a MountedDrives instance — separates OS enumeration from path-matching logic
ScriptIsolation/TemporaryDirectoryFallbackProvider.cs + ITemporaryDirectoryFallbackProvider.cs	Platform-aware enumeration of candidate temp directories to fall back to
ScriptIsolation/RequestedLockOptions.cs + RequestedLockOptionsFactory.cs	Record capturing the validated, parsed lock request (type, mutex name, timeout, preferred directory); factory creates it from CommonOptions.ScriptIsolationOptions
ScriptIsolation/LockOptionsFactory.cs	Creates the final LockOptions by resolving the lock directory via ILockDirectoryFactory and applying the degradation policy (UseExclusiveIfSharedIsNotSupported) — always promotes a shared lock to exclusive when shared locking is unavailable
ScriptIsolation/ScriptIsolationEnforcer.cs + IScriptIsolationEnforcer.cs	Injectable class (replacing the static Isolation class) that orchestrates lock acquisition via RequestedLockOptionsFactory and LockOptionsFactory
ScriptIsolation/ScriptIsolationModule.cs	Autofac Module that registers all script-isolation services (ScriptIsolationEnforcer, LockDirectoryFactory, RequestedLockOptionsFactory, LockOptionsFactory, providers, etc.)
ScriptIsolation/LockAcquisitionResiliencePipelineBuilder.cs	Builds the Polly retry/timeout pipeline; extracted from LockOptions to keep the record type focused
ScriptIsolation/IFileLockService.cs + FileLockService.cs	Injectable abstraction over CreateDirectory and AcquireLock — enables hermetic unit tests
ScriptIsolation/IPathResolutionService.cs + DefaultPathResolutionService.cs + PathResolutionServiceExtensions.cs	Injectable abstraction and extension over Path.GetFullPath and symlink resolution — necessary to correctly match paths on macOS where /tmp → /private/tmp

Modified files

File	Change
ScriptIsolation/LockOptions.cs	Simplified to a pure data record; IsFullySupported/IsSupported renamed to BothSharedAndExclusiveAreSupported/RequestedLockTypeIsSupported; FromScriptIsolationOptionsOrNull and BuildLockAcquisitionPipeline removed (moved to dedicated factories/builder)
ScriptIsolation/FileLock.cs	Minor: adapts to LockFile record shape
Plumbing/Commands/CommonOptions.cs	scriptIsolationSharedFallback CLI option and PromoteToExclusiveLockWhenSharedLockUnavailable property removed — lock promotion is now unconditional

Removed files

File	Reason
ScriptIsolation/Isolation.cs	Replaced by ScriptIsolationEnforcer (injectable) + IScriptIsolationEnforcer

---

Decision trees

Phase 1 — Lock directory selection (LockDirectoryFactory.Create)

flowchart TD
    Start([Candidate path]) --> MapDrive[Map candidate to its drive]
    MapDrive --> CheckHeuristic{drive?.LockSupport?}
    CheckHeuristic -- Supported --> ReturnCandidate([Return candidate\nLockCapability.Supported])
    CheckHeuristic -- null no drive or unknown format --> ProbeCandidateDrive
    CheckHeuristic -- ExclusiveOnly or Unsupported --> IterateFallbacks

    ProbeCandidateDrive[DetectLockSupport on candidate path]
    ProbeCandidateDrive --> CandidateProbeResult{Candidate\ndetected capability?}
    CandidateProbeResult -- Supported --> ReturnCandidate2([Return candidate\nLockCapability.Supported])
    CandidateProbeResult -- ExclusiveOnly or Unsupported or null --> IterateFallbacks

    IterateFallbacks[Iterate temp fallback directories]
    IterateFallbacks --> HasTemp{More temp\ncandidates?}
    HasTemp -- No --> AnyExclusiveTemp
    HasTemp -- Yes --> MapTempDrive[Map temp dir to its drive]
    MapTempDrive --> CheckTempHeuristic{tempDrive?.LockSupport?}
    CheckTempHeuristic -- Supported --> ReturnTemp([Return temp dir\nLockCapability.Supported])
    CheckTempHeuristic -- null no drive or unknown format --> ProbeTemp[DetectLockSupport on temp path]
    CheckTempHeuristic -- ExclusiveOnly --> RecordFirstExclusive[Record as best-so-far\nif none recorded]
    ProbeTemp --> TempResult{Temp capability?}
    TempResult -- Supported --> ReturnTemp
    TempResult -- ExclusiveOnly --> RecordFirstExclusive
    TempResult -- Unsupported or null --> HasTemp
    RecordFirstExclusive --> HasTemp

    AnyExclusiveTemp{Any ExclusiveOnly\ntemp found?}
    AnyExclusiveTemp -- Yes, and candidate is ExclusiveOnly --> ReturnCandidate3([Return candidate\nLockCapability.ExclusiveOnly])
    AnyExclusiveTemp -- Yes, and candidate is Unsupported/null --> ReturnBestTemp([Return best temp dir\nLockCapability.ExclusiveOnly])
    AnyExclusiveTemp -- No --> ReturnUnsupported([Return candidate\nLockCapability.Unsupported])

Loading

Key design choice: CachedDriveInfo.LockSupport acts as a fast heuristic — known-good filesystem formats (ntfs, ext4, apfs, tmpfs, etc.) return Supported immediately without probing. When LockSupport returns null (unknown format, network drive with no override, or no matching drive at all), LockDirectoryFactory.DetectLockSupport is called to empirically verify lock semantics. When both the preferred directory and the best temp are ExclusiveOnly, the preferred directory is still returned — using a temp directory offers no additional isolation benefit and introduces unnecessary path indirection.

---

Phase 2 — Lock options resolution (LockOptionsFactory.UseExclusiveIfSharedIsNotSupported)

flowchart TD
    Start([LockOptions\nType + LockDirectory capability]) --> FullySupported{BothSharedAndExclusiveAreSupported?\nShared + Exclusive work\ncorrectly}

    FullySupported -- Yes --> ReturnOriginal([Use requested lock\nNo warning])
    FullySupported -- No --> IsSupported{RequestedLockTypeIsSupported?\nRequested LockType\nis supported}

    IsSupported -- Yes, Exclusive requested --> ReturnExclusive([Use Exclusive lock\nNo warning])

    IsSupported -- No, Shared requested\non ExclusiveOnly dir --> ExclusiveAvail{Supports\nExclusive?}
    ExclusiveAvail -- Yes --> PromoteToExclusive([Promote to Exclusive lock\nWarn: shared unavailable])

    ExclusiveAvail -- No --> NoLockUnsupported([No lock acquired\nWarn: no isolation available])

Loading

Shared locks are always promoted to exclusive when shared locking is unavailable. The scriptIsolationSharedFallback CLI option and the PromoteToExclusiveLockWhenSharedLockUnavailable flag have been removed — there is no longer a user-configurable "skip" behaviour.

---

Areas especially relevant for review

LockDirectoryFactory.DetectLockSupport — live filesystem probing

Four sequential probe operations are run against a temporary .tmp file in the lock directory to empirically verify lock semantics. The probes use TimeSpan.Zero timeout so they fail fast if the filesystem rejects the operation. The test file is always cleaned up in a finally block. Any exception during probing is treated conservatively as Unsupported. Reviewers should check whether the probe-file cleanup is sufficient and whether the TimeSpan.Zero strategy is reliable across all target filesystems.

PathResolutionServiceExtensions.ResolvePath — ancestor-walk symlink resolution

The ancestor-walk is necessary because a lock directory might not exist yet at the time of drive matching (e.g. first ever run). The algorithm peels off non-existent path segments from the end, resolves the nearest existing ancestor's symlink, then re-attaches the original tail. This is especially important on macOS where Path.GetFullPath("/tmp/x") returns /tmp/x but the drive is mounted at /private/tmp. All five GetFullPath exception types are caught and cause the input to be returned unchanged.

MountedDrives.GetAssociatedDrive — longest-prefix mount matching

The match is done on the symlink-resolved path, with platform-appropriate case sensitivity (OrdinalIgnoreCase on Windows/macOS, Ordinal on Linux). Reviewers should verify the StartsWith prefix check correctly handles the case where the drive root IS the path (no trailing separator needed since RootDirectory.FullName ends with a separator on all .NET platforms).

TemporaryDirectoryFallbackProvider.GetFallbackCandidates — platform-specific candidate order

On Linux, /dev/shm is included as a candidate after /tmp. tmpfs-backed /dev/shm is one of the most reliably lock-capable paths on systems where the working directory is NFS-mounted. The candidates are namespace-scoped using the last segment of the preferred directory path to prevent cross-deployment lock file collisions.

LockOptionsFactory.UseExclusiveIfSharedIsNotSupported — unconditional promotion policy

When a shared lock is requested but only exclusive locking is supported, the lock is always promoted to exclusive and a warning is logged. There is no longer a user-configurable flag to skip this promotion. If no locking at all is supported, null is returned and a warning is logged.

---

Test coverage

LockDirectoryFixture — 1,084 lines, all groups tagged RunOnceOnWindowsAndLinux

All tests in this fixture use injected fakes (FakeLockService, FakePathResolutionService, FakeMountedDrivesProvider, FakeTemporaryDirectoryFallbackProvider) and run entirely in-memory — no real filesystem access.

Group	What is tested
A — LockDirectory.Supports	All LockCapability × LockType combinations (5 cases)
B — CachedDriveInfo.LockSupport static	Known-good formats (apfs, ntfs, ext4, etc.) → Supported; network drives → null; DetectedLockSupport overrides (13 cases)
C — LockDirectoryFactory.DetectLockSupport	All four live-probe outcomes: Unsupported, three variants of ExclusiveOnly (broken shared, broken exclusive-blocks-shared, broken shared-blocks-exclusive), and Supported (5 cases)
D — LockDirectoryFactory.Create	10 scenario tests: preferred already Supported; preferred null → detected Supported before temps inspected (preferred over a Supported temp); preferred Unsupported → Supported temp used; preferred null → detected Supported before ExclusiveOnly temps inspected; both preferred and temp are ExclusiveOnly → stay on preferred; temp ExclusiveOnly + preferred Unsupported → use temp; nothing works → Unsupported on preferred; empty drive table → live detection runs (succeeds with fully-supported lock service); temp with no associated drive skipped; first Supported temp wins when multiple candidates
E — MountedDrives.GetAssociatedDrive	7 tests: symlink resolution before matching; case-sensitive rejection; case-insensitive match; normalisation; longest-prefix wins (POSIX); DirectoryNotFoundException when no match; ancestor symlink resolved when child doesn't exist
F — DefaultPathResolutionService	3 real-filesystem integration tests (2 Unix-only): existing symlink, non-existent child under symlink ancestor, fully non-existent path
G — PathResolutionServiceExtensions.ResolvePath	10 tests: non-existent path, existing non-symlink, symlink followed, ancestor symlink + tail re-attached, multiple tail segments re-attached, path normalisation via GetFullPath, all GetFullPath exception types caught (5 parameterised)

IsolationFixture — integration tests via Autofac container

The fixture builds a real IContainer using ScriptIsolationModule and exercises IScriptIsolationEnforcer end-to-end against the local filesystem.

Tests	What is tested
10 integration tests	Enforce/EnforceAsync with valid options acquires a lock file; releases on dispose; throws LockRejectedException on contention; allows concurrent shared locks; waits for lock release; enforces timeout and throws after it elapses

LockOptionsFixture — unit tests for factory classes

Tests	What is tested
RequestedLockOptionsFactory tests	Returns null for missing/whitespace Level, MutexName, or TentacleHome; maps FullIsolation→Exclusive and NoIsolation→Shared; case-insensitive level parsing; defaults timeout to Infinite on invalid or missing input; throws on invalid mutex name chars
LockOptionsFactory tests	CreateLockOptionsOrNull_BuildsCorrectLockFileName asserts on LockFile.File.Name; UseExclusiveIfSharedIsNotSupported — 5 cases: Supported→original returned; ExclusiveOnly+Exclusive→original returned, no warning; ExclusiveOnly+Shared→promoted to Exclusive + warning; Unsupported+Exclusive→null + warning; Unsupported+Shared→null + warning
Pipeline timeout test	Fake-time test verifying the Polly pipeline honours a timeout > 1 day

What is not covered by unit tests

• The end-to-end path through LockDirectoryFactory.Create against a real NFS or FUSE mount. The live probing logic is verified against a real (local) filesystem via IsolationFixture and Group F in LockDirectoryFixture, but the specific scenario of a network filesystem detecting as Unsupported and falling back to /tmp is exercised only through the fakes in Group D.
• Windows-specific temp directory candidates (%LOCALAPPDATA%\Calamari, %TEMP%) are not covered by integration tests.

[gb-8]

I've had a scan of the PR and while the approach seems good, and the test coverage seems good, I did find the code hard to follow.
I had to really work to follow the logic through, and see exactly where the determination of locking support was performed.
The logic is all hidden inside things that don't make it clear what they are doing.
Once you are happy that the functionality is all correct, then I'd like to pair on seeing if we can refactor to make it more readable and understandable.

[gb-8]

I think it is good practice to set the default value explicitly to 0.

[rhysparry]

Done (Unsupported is now the default). I was toying with the idea of using [Flags], but I didn't see much benefit to it.

[gb-8]

Notes from our discussion:

---

How we are actually solving the isolation:

Possible support levels:

• full (exclusive and shared)
• partial (exclusive only)
• none

Possible requested isolation levels:

• isolated (needs exclusive lock)
• not-isolated (needs shared lock - to help out other uses)

For phase two, we have a known support level, and a requested isolation level.

Algorithm:

• If support level is full, return requested isolation level.
• If support level is partial:
  • always return isolated
  • warn if this is different from request.
• If support level is none:
  • we cannot do any isolation, so we should warn.
  • lets add alerting so we can see how common this is (might require service message).

---

Expected implementation

Get the requested locking data:
- requested isolation level
- preferred lockfile folder
- mutex / lockfile name
- timeouts

1. Find the best lock location (drive or folder)

2. Determine actual isolation level to use (as we may need to escalate from requested level)

3. Actually acquire the lock

Thoughts on refactoring:

• Breaking down the whole process in a hierarchy where sibling actions are at a similar level of abstraction.
  • Avoiding nesting high-level concerns inside other things.
• Move IO into types which make it clear to expect IO inside them, and separate I/O from other logic.
• I personally find putting logic in static methods on data-holding types to be unexpected, if that logic is not simple in-memory dat munging.
  • Static factory methods are good, when they don't have any real logic (i.e. stuff you'd be happy to have in a constructor).
  • Anything more complex is better in a separate builder type or factory.

[gb-8]

This factory has a bunch of logic and intelligence in it, so maybe should be renamed to communicate that it is doing more than simple constructions.

E.g. LockOptionResolver or something.

[rhysparry]

Renaming to LockOptionsResolver

[gb-8]

This can also "downgrade" rather than "escalate" the lock you actually get, so a better name would be:

DetermineActualKindOfLockWeWillUseBasedOnSupport (or something better).

[rhysparry]

Renaming to DetermineActualLockTypeToUseBasedOnSupport and refactoring to make things clearer.

[gb-8]

Better (but not great) name:
DetermineWhatLockToUseBasedOnOptions

The key thing to communicate is that we are making decisions inside here.

[rhysparry]

I ended up with DeterminLockOptionsToEnforceBasedOnIsolationOptions

[gb-8]

I think we have settled on:
CreateFromIsolationOptions()

[rhysparry]

Will rename to CreateFromIsolationOptions

[gb-8]

✅ Pre-approval, with various renaming suggested, and some a suggested refactor of LockOptionsFactory.UseExclusiveIfSharedIsNotSupported.

We will likely want to write out service messages each time we escalate/deescalate level of locking do we can write warnings in server logs that we can monitor or alert on. That can go into this PR too or a follow up.

[rhysparry]

I just want to break out the service message details first because I'd like to enrich the message with details of what we tried.

[rhysparry]

Renaming to LockOptionsResolver

[rhysparry]

Renaming to DetermineActualLockTypeToUseBasedOnSupport and refactoring to make things clearer.

[rhysparry]

Will rename to CreateFromIsolationOptions

[rhysparry]

I ended up with DeterminLockOptionsToEnforceBasedOnIsolationOptions