Skip to content

Commit a6c2976

Browse files
committed
Add asset history table and functionality (backend/CLI/web/data migration/documentation), fix role overflow and NAG supressions.
1 parent 9af1bed commit a6c2976

49 files changed

Lines changed: 2558 additions & 53 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.kiro/steering/BACKEND_CDK_DEVELOPMENT_WORKFLOW.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ infra/
5252

5353
One folder per domain. The current domains:
5454

55-
- `assets/` — Asset handlers (`assetService.py` is the GOLD STANDARD; `assetVersions.py` covers version CRUD + archive/unarchive + update)
55+
- `assets/` — Asset handlers (`assetService.py` is the GOLD STANDARD; `assetVersions.py` covers version CRUD + archive/unarchive + update; `assetHistory.py` serves the paged asset lifecycle history lookup)
5656
- `auth/` — Auth handlers (authorizer, constraints, cognito, preTokenGen, apiKeyService)
5757
- `authz/` — Casbin ABAC/RBAC enforcer (`CasbinEnforcer` proxy)
5858
- `assetLinks/` — Asset relationship management

.kiro/steering/CDK_DEVELOPMENT_WORKFLOW.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -112,6 +112,7 @@ interface storageResources {
112112
assetVersionsStorageTable;
113113
assetFileVersionsStorageTable;
114114
assetFileVersionHistoryStorageTable;
115+
assetHistoryStorageTable;
115116
assetFileMetadataVersionsStorageTable;
116117
authEntitiesStorageTable;
117118
commentStorageTable;

CHANGELOG.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,9 +38,14 @@ All notable changes to this project will be documented in this file. See [standa
3838
- New Coordinate Transform pipeline — Reprojects point cloud files between coordinate reference systems (CRS). Supports E57, LAS, LAZ, and PLY inputs and outputs LAZ, LAS, E57, or PLY, running as an AWS Batch Fargate container with PDAL- and pyproj-based transformation. Source/target CRS accept EPSG codes, PROJ strings, WKT, or custom named local grids, and the position-dependent local scale factor of well-defined projections (e.g., OSGB36 `EPSG:27700`) is handled automatically. Parameters can be set as pipeline defaults or overridden per asset via VAMS metadata keys (`sourceCrs`, `targetCrs`, `outputFormats`, scale factors, and more). Enable via `app.pipelines.useConversionCoordinateTransform` in `infra/config/config.json`; the container image builds via AWS CodeBuild + Amazon ECR. See the [Coordinate Transform pipeline documentation](documentation/docusaurus-site/docs/pipelines/coordinate-transform.md) for details.
3939
- NVIDIA Isaac Lab Training pipeline now supports a configuration building its container image via AWS CodeBuild + ECR instead of a local Docker build, matching the existing NVIDIA Cosmos and Gr00t pipelines.
4040
- The Application Load Balancer (ALB) web access logs bucket is now removed (with its contents) on stack teardown instead of being retained. Because this bucket carries a fixed name derived from the configured domain host under ALB deployments, a retained bucket previously orphaned on `cdk destroy` and blocked a subsequent redeploy with the same configuration. It now matches the web app bucket's `autoDeleteObjects` + `DESTROY` behavior.
41-
- **CDK** DynamoDB table names, non-asset S3 bucket names, and audit CloudWatch log group names are now resolved from AWS Systems Manager Parameter Store instead of being injected as Lambda environment variables. A new `ResourceNamesBuilder` nested stack publishes SSM String parameters under the deployment prefix (28 DynamoDB tables, 2 S3 buckets, 9 audit log groups). Non-pipeline backend Lambda functions receive the SSM parameter prefix (`VAMS_RESOURCE_PARAM_PREFIX`) and ssm:GetParameter/GetParameters/GetParametersByPath IAM grants; handlers call `get_table_name(ResourceKeys.*)` / `get_bucket_name(ResourceKeys.*)` / `get_log_group_name(ResourceKeys.*)` from `backend.common.resourceNames` at module level, which caches the full parameter set for 60 minutes and falls back to legacy environment variables for testing and local utilities. The resource-name constants are defined in `infra/common/resourceParamKeys.ts` (TypeScript) and mirrored in `backend/backend/common/resourceNames.py` (Python `ResourceKeys` class). Deprecated tables retained for data migration are also published (under `dynamoTables/legacy/`), along with the reindexer Lambda function name (`lambdaFunctions/crOsReindexer`), for consumption by data-migration tooling. The core stack exposes the base prefix as the CloudFormation output `ResourceNamesSSMParamPrefixOutput`; the v2.5-to-v2.6 data migration script resolves the reindexer function name from SSM via the new shared utility `infra/deploymentDataMigration/tools/ssm_resource_lookup.py`, so operators only fill in the base prefix and region (explicit per-resource config values remain supported as optional overrides). Pipeline Lambda functions (`backendPipelines/`) continue to use legacy environment variables and are excluded from SSM resolution. VPC deployments with `useForAllLambdas` enabled and `addVpcEndpoints` disabled now require an operator-managed SSM Systems Manager interface VPC endpoint — all Lambda functions resolve resource names at cold start and will fail without SSM API access.
41+
- **CDK** DynamoDB table names, non-asset S3 bucket names, and audit CloudWatch log group names are now resolved from AWS Systems Manager Parameter Store instead of being injected as Lambda environment variables. A new `ResourceNamesBuilder` nested stack publishes SSM String parameters under the deployment prefix (29 DynamoDB tables, 2 S3 buckets, 9 audit log groups). Non-pipeline backend Lambda functions receive the SSM parameter prefix (`VAMS_RESOURCE_PARAM_PREFIX`) and ssm:GetParameter/GetParameters/GetParametersByPath IAM grants; handlers call `get_table_name(ResourceKeys.*)` / `get_bucket_name(ResourceKeys.*)` / `get_log_group_name(ResourceKeys.*)` from `backend.common.resourceNames` at module level, which caches the full parameter set for 60 minutes and falls back to legacy environment variables for testing and local utilities. The resource-name constants are defined in `infra/common/resourceParamKeys.ts` (TypeScript) and mirrored in `backend/backend/common/resourceNames.py` (Python `ResourceKeys` class). Deprecated tables retained for data migration are also published (under `dynamoTables/legacy/`), along with the reindexer Lambda function name (`lambdaFunctions/crOsReindexer`), for consumption by data-migration tooling. The core stack exposes the base prefix as the CloudFormation output `ResourceNamesSSMParamPrefixOutput`; the v2.5-to-v2.6 data migration script resolves the reindexer function name from SSM via the new shared utility `infra/deploymentDataMigration/tools/ssm_resource_lookup.py`, so operators only fill in the base prefix and region (explicit per-resource config values remain supported as optional overrides). Pipeline Lambda functions (`backendPipelines/`) continue to use legacy environment variables and are excluded from SSM resolution. VPC deployments with `useForAllLambdas` enabled and `addVpcEndpoints` disabled now require an operator-managed SSM Systems Manager interface VPC endpoint — all Lambda functions resolve resource names at cold start and will fail without SSM API access.
4242
- **CDK** Advanced IAM role customization for restricted environments — Two optional, opt-in mechanisms let deployers who cannot create IAM roles map pre-created roles instead. A new `app.iamRoleConfig` config section toggles `useCustomBootstrapRoles` (replace the CDK bootstrap roles via a custom stack synthesizer, or use `CliCredentialsStackSynthesizer` for no bootstrap roles at all) and `useCustomVamsStackRoles`. The actual mappings (role ARNs, construct-path-to-role-name maps) live in a separate `infra/config/policy/iamRoleConfig.json` file, keeping the verbose values out of the main config. Both options default to disabled, preserving the existing behavior where VAMS manages all IAM roles. See the [configuration reference](documentation/docusaurus-site/docs/deployment/configuration-reference.md) for the full workflow.
4343
- Asset/File search APIs now include additional fields when passing general search filter queries (asset id, database id, s3 bucket id/name, s3 bucket prefix)
44+
- Asset lifecycle history — VAMS now keeps a permanent per-asset audit history of lifecycle operations (create, edit, archive, unarchive, permanent delete) in a new `AssetHistoryStorageTable` Amazon DynamoDB table. Each record captures the operation, the acting user, the change origin (API vs. S3 bucket-sync ingestion, recorded as `create`/`createDirect`, `unarchive`/`unarchiveDirect`, etc.), and an open-schema snapshot of the asset fields after the operation (name, description, distributable flag, tags, bucket, location key, archive/unarchive reasons). History records survive asset permanent deletion; recreating an asset with the same asset ID continues the same history trail.
45+
- New paged API endpoint `GET /database/{databaseId}/assets/{assetId}/assetHistory` returns records newest first with `pageSize`/`startingToken` pagination and two-tier authorization against the asset (history of a permanently deleted, non-recreated asset ID returns 404).
46+
- **CLI** New `vamscli assets history` command with the standard pagination options (`--page-size`, `--starting-token`, `--auto-paginate`, `--max-items`) and `--json-output`.
47+
- **Web** New Asset History modal on the Asset View File Manager details panel — a `(History)` link on the asset root node's Type row opens a server-side paged history table with per-record snapshot details.
48+
- **CDK** The v2.5-to-v2.6 data migration script gains an asset history backfill phase that infers `create` records from each asset's v0 version record and `archive`/`unarchive` records from the asset's archive fields; backfilled records are flagged `migratedRecord: true` and are idempotent on re-run.
4449
- Asset file change history — VAMS now tracks per-version change provenance (how a file version was created and by whom) for uploads, workflow executions, copies, moves, renames, archives, and direct S3 changes. Provenance is stamped as `vams-change*` S3 object metadata when a version is created and recorded into a new `assetFileVersionHistoryStorageTable` Amazon DynamoDB table on ingest. File list/detail responses surface the current version's change source and modifying user, and file version history surfaces the full per-version provenance (including workflow and source-location details). Exposed through the file GET APIs, the `vamscli file` commands, and the web file manager (details panel and version history view). Versions created before this release report blank provenance.
4550
- File listing responses now include each file's S3 `etag`. The `listFiles` API returns it for every file in basic and full mode, matching the `fileInfo` API which already returns it for the requested file. Surfaced by the `vamscli file list` and `vamscli file info` commands.
4651
- API route listing endpoints — New `GET /auth/routes/api` returns the full list of VAMS API routes (paths, methods, categories) from a new master route definition file, and `GET /auth/routes/api/allowed` returns the routes and methods the requesting user is authorized to call (evaluated through Casbin). Exposed through new `vamscli auth routes list` and `vamscli auth routes allowed` CLI commands.

CLAUDE.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -257,7 +257,7 @@ ASSET_STORAGE_TABLE_NAME = "vams-asset-storage" # VIOLATION
257257

258258
```typescript
259259
// ✅ CORRECT - CDK publishes resource names to SSM and injects prefix
260-
// ResourceNamesBuilder nested stack publishes 39 SSM parameters
260+
// ResourceNamesBuilder nested stack publishes 40 SSM parameters
261261
import { RESOURCE_PARAM_KEYS } from "../../common/resourceParamKeys";
262262
new ssm.StringParameter(this, "AssetStorageTableParam", {
263263
parameterName: `${prefix}/${RESOURCE_PARAM_KEYS.dynamoTables.assetStorage}`,

backend/CLAUDE.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,8 @@ backend/
4747
│ │ ├── validators.py # Input validation regex patterns and validate() dispatcher
4848
│ │ ├── s3.py # S3 file validation (extension + MIME type checks)
4949
│ │ ├── s3MetadataKeys.py # Canonical S3 object user-metadata keys (assetid, vams-*)
50+
│ │ ├── assetHistory.py # Asset lifecycle history writer (change source constants,
51+
│ │ │ # build_asset_snapshot, write_asset_history_record; best-effort)
5052
│ │ ├── s3PathPatterns.py # Reserved S3 prefix folders, .previewFile. pattern,
5153
│ │ │ # allowed preview extensions (mirrored in
5254
│ │ │ # web/src/common/constants/fileFormats.ts)
@@ -64,6 +66,7 @@ backend/
6466
│ ├── handlers/ # Lambda handlers (one folder per domain)
6567
│ │ ├── assets/assetService.py # GOLD STANDARD handler -- follow this pattern
6668
│ │ ├── assets/assetVersions.py # Asset version CRUD + archive/unarchive + update (versionAlias)
69+
│ │ ├── assets/assetHistory.py # Asset lifecycle history lookup (paged GET, newest first)
6770
│ │ ├── auth/ # Auth handlers (authorizer, constraints, cognito, preTokenGen, apiKeyService)
6871
│ │ │ ├── apiGatewayAuthorizerRest.py # REST REQUEST authorizer entry point (returns IAM policy;
6972
│ │ │ │ # delegates JWT/API-key/IP validation to common/auth/authorizerCore.py)
@@ -101,6 +104,7 @@ backend/
101104
│ │ # USER_API_KEY_MAX_EXPIRATION_DAYS = 365)
102105
│ ├── pipelines.py # Pipeline models (PipelineExecutionType enum, SQS/EventBridge fields)
103106
│ ├── workflows.py # Workflow models (Step Functions ASL generation)
107+
│ ├── assetHistory.py # Asset history request/record/response models (open-schema snapshot)
104108
│ ├── common.py # Response helpers, error functions, APIGatewayProxyResponseV2
105109
│ └── [domain].py # Domain-specific models
106110
├── lambdaLayers/ # Lambda layer definitions

backend/backend/common/apiRoutes.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -126,6 +126,9 @@ def matches(self, path: str) -> bool:
126126
"assets",
127127
)
128128
API_ASSET_EXPORT = ApiRoute("/database/{databaseId}/assets/{assetId}/export", (POST,), "assets")
129+
API_GET_ASSET_HISTORY = ApiRoute(
130+
"/database/{databaseId}/assets/{assetId}/assetHistory", (GET,), "assetHistory"
131+
)
129132

130133
ASSET_ROUTES: Tuple[ApiRoute, ...] = (
131134
API_ASSETS,
@@ -139,6 +142,7 @@ def matches(self, path: str) -> bool:
139142
API_DOWNLOAD_ASSET_STREAM,
140143
API_AUXILIARY_PREVIEW_ASSETS_STREAM,
141144
API_ASSET_EXPORT,
145+
API_GET_ASSET_HISTORY,
142146
)
143147

144148
# ---------------------------------------------------------------------------
Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
# Copyright 2026 Amazon.com, Inc. or its affiliates. All Rights Reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
import boto3
5+
import uuid
6+
from datetime import datetime
7+
from botocore.config import Config
8+
from customLogging.logger import safeLogger
9+
from common.resourceNames import ResourceKeys, get_table_name
10+
11+
retry_config = Config(retries={'max_attempts': 5, 'mode': 'adaptive'})
12+
dynamodb = boto3.resource('dynamodb', config=retry_config)
13+
logger = safeLogger(service_name="AssetHistory")
14+
15+
# Asset lifecycle history change sources. The *Direct variants mark records
16+
# originated from S3 bucket-sync ingestion rather than a VAMS API call.
17+
CHANGE_SOURCE_CREATE = "create"
18+
CHANGE_SOURCE_CREATE_DIRECT = "createDirect"
19+
CHANGE_SOURCE_EDIT = "edit"
20+
CHANGE_SOURCE_ARCHIVE = "archive"
21+
CHANGE_SOURCE_UNARCHIVE = "unarchive"
22+
CHANGE_SOURCE_UNARCHIVE_DIRECT = "unarchiveDirect"
23+
CHANGE_SOURCE_PERMANENT_DELETE = "permanentDelete"
24+
25+
try:
26+
_asset_history_table_name = get_table_name(ResourceKeys.ASSET_HISTORY_STORAGE_TABLE)
27+
except Exception:
28+
_asset_history_table_name = None
29+
30+
asset_history_table = dynamodb.Table(_asset_history_table_name) if _asset_history_table_name else None
31+
32+
33+
def build_asset_snapshot(asset_record, archived_reason=None, unarchived_reason=None):
34+
"""Build the open-schema assetSnapshot map from an asset record's fields
35+
as they stand after the operation being recorded."""
36+
snapshot = {
37+
'assetName': asset_record.get('assetName', ''),
38+
'description': asset_record.get('description', ''),
39+
'isDistributable': asset_record.get('isDistributable', False),
40+
'tags': asset_record.get('tags', []),
41+
'bucketId': asset_record.get('bucketId', ''),
42+
}
43+
asset_location = asset_record.get('assetLocation') or {}
44+
if asset_location.get('Key'):
45+
snapshot['assetLocationKey'] = asset_location['Key']
46+
if archived_reason:
47+
snapshot['archivedReason'] = archived_reason
48+
if unarchived_reason:
49+
snapshot['unarchivedReason'] = unarchived_reason
50+
return snapshot
51+
52+
53+
def write_asset_history_record(database_id, asset_id, change_source, change_user_id, asset_snapshot):
54+
"""Write one asset lifecycle history record. Best-effort: failures are
55+
logged and never raised into the calling operation. Records persist across
56+
asset permanent deletes."""
57+
try:
58+
if not asset_history_table:
59+
logger.warning("Asset history table not configured; skipping history record")
60+
return
61+
record_date = datetime.utcnow().isoformat() + "Z"
62+
item = {
63+
'databaseId:assetId': f"{database_id}:{asset_id}",
64+
'historyRecordId': f"{record_date}#{uuid.uuid4().hex[:8]}",
65+
'databaseId': database_id,
66+
'assetId': asset_id,
67+
'recordDate': record_date,
68+
'changeSource': change_source,
69+
'changeUserId': change_user_id or 'SYSTEM_USER',
70+
'assetSnapshot': asset_snapshot or {},
71+
}
72+
asset_history_table.put_item(Item=item)
73+
except Exception as e:
74+
logger.warning(f"Failed writing asset history record for {asset_id}: {e}")

backend/backend/common/resourceNames.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ class ResourceKeys:
3535
ASSET_FILE_VERSION_HISTORY_STORAGE_TABLE = ResourceParamKey("dynamoTables/assetFileVersionHistoryStorage", ("ASSET_FILE_VERSION_HISTORY_STORAGE_TABLE_NAME",))
3636
ASSET_FILE_METADATA_VERSIONS_STORAGE_TABLE = ResourceParamKey("dynamoTables/assetFileMetadataVersionsStorage", ("ASSET_FILE_METADATA_VERSIONS_STORAGE_TABLE_NAME",))
3737
ASSET_FILE_METADATA_STORAGE_TABLE = ResourceParamKey("dynamoTables/assetFileMetadataStorage", ("ASSET_FILE_METADATA_STORAGE_TABLE_NAME",))
38+
ASSET_HISTORY_STORAGE_TABLE = ResourceParamKey("dynamoTables/assetHistoryStorage", ("ASSET_HISTORY_STORAGE_TABLE_NAME",))
3839
AUTH_ENTITIES_STORAGE_TABLE = ResourceParamKey("dynamoTables/authEntitiesStorage", ("AUTH_TABLE_NAME", "AUTH_ENTITIES_TABLE"))
3940
COMMENT_STORAGE_TABLE = ResourceParamKey("dynamoTables/commentStorage", ("COMMENT_STORAGE_TABLE_NAME",))
4041
CONSTRAINTS_STORAGE_TABLE = ResourceParamKey("dynamoTables/constraintsStorage", ("CONSTRAINTS_TABLE_NAME",))

0 commit comments

Comments
 (0)