The following instance properties relate to handling the state of Sleeper tables.
| Property Name | Description | Default Value | Run CDK Deploy When Changed |
|---|---|---|---|
| sleeper.default.lambda.table.state.memory.mb | Default value for amount of memory in MB for each lambda that holds the state of Sleeper tables in memory. These use a state store provider which caches a number of tables at once, set in sleeper.statestore.provider.cache.size. Not all lambdas are covered by this, e.g. see sleeper.batch.table.lambdas.memory.mb. |
4096 | true |
| sleeper.batch.table.lambdas.memory.mb | The amount of memory in MB for lambdas that create batches of tables to run some operation against, eg. create compaction jobs, run garbage collection, perform partition splitting. | 1024 | true |
| sleeper.batch.table.lambdas.timeout.seconds | The timeout in seconds for lambdas that create batches of tables to run some operation against, eg. create compaction jobs, run garbage collection, perform partition splitting. | 60 | true |
| sleeper.cache.table.properties.provider.timeout.minutes | The timeout in minutes for when the table properties provider cache should be cleared, forcing table properties to be reloaded from S3. | 60 | false |
| sleeper.statestore.provider.cache.size | The maximum size of state store providers. If a state store is needed and the cache is full, the oldest state store in the cache will be removed to make space. This can be disabled by setting a negative value, e.g. -1. | 10 | false |
| sleeper.statestore.provider.min.heap.target.amount | The minimum amount of heap space that the state store provider will try to keep available when sizing the cache to the available memory. This will usually only be used in cases where the state store cache is the main use of memory, currently only in a state store committer on the EC2 platform. This affects how many state stores can be cached in memory. | 100M | false |
| sleeper.statestore.dynamo.pointintimerecovery | This specifies whether point in time recovery is enabled for the DynamoDB state store. This is set on the DynamoDB tables. | false | true |
| sleeper.statestore.s3.dynamo.pointintimerecovery | This specifies whether point in time recovery is enabled for the S3 state store. This is set on the revision DynamoDB table. | false | true |
| sleeper.statestore.snapshot.creation.batch.size | The number of tables to create transaction log snapshots for in a single invocation. This will be the batch size for a lambda as an SQS FIFO event source. This can be a maximum of 10. | 1 | true |
| sleeper.statestore.snapshot.creation.lambda.period.seconds | The frequency in seconds with which the transaction log snapshot creation lambda is run. | 60 | true |
| sleeper.statestore.snapshot.creation.lambda.timeout.seconds | The timeout in seconds after which to terminate the transaction log snapshot creation lambda. | 900 | true |
| sleeper.statestore.snapshot.creation.memory.mb | The amount of memory in MB for the transaction log snapshot creation lambda. | true | |
| sleeper.statestore.snapshot.creation.concurrency.reserved | The reserved concurrency for the snapshot creation lambda. See reserved concurrency overview at: https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html |
true | |
| sleeper.statestore.snapshot.creation.concurrency.max | The maximum given concurrency allowed for the snapshot creation lambda. See maximum concurrency overview at: https://aws.amazon.com/blogs/compute/introducing-maximum-concurrency-of-aws-lambda-functions-when-using-amazon-sqs-as-an-event-source/ |
true | |
| sleeper.statestore.snapshot.deletion.batch.size | The number of tables to delete old transaction log snapshots for in a single invocation. This will be the batch size for a lambda as an SQS FIFO event source. This can be a maximum of 10. | 1 | true |
| sleeper.statestore.snapshot.deletion.lambda.period.minutes | The frequency in minutes with which the transaction log snapshot deletion lambda is run. | 60 | true |
| sleeper.statestore.snapshot.deletion.concurrency.reserved | The reserved concurrency for the snapshot deletion lambda. See reserved concurrency overview at: https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html |
true | |
| sleeper.statestore.snapshot.deletion.concurrency.max | The maximum given concurrency allowed for the snapshot deletion lambda. See maximum concurrency overview at: https://aws.amazon.com/blogs/compute/introducing-maximum-concurrency-of-aws-lambda-functions-when-using-amazon-sqs-as-an-event-source/ |
true | |
| sleeper.statestore.transaction.deletion.batch.size | The number of tables to delete old transaction log transactions for in a single invocation. This will be the batch size for a lambda as an SQS FIFO event source. This can be a maximum of 10. | 1 | true |
| sleeper.statestore.transaction.deletion.lambda.period.minutes | The frequency in minutes with which the transaction log transaction deletion lambda is run. | 60 | true |
| sleeper.statestore.transaction.deletion.concurrency.reserved | The reserved concurrency for the transaction deletion lambda. See reserved concurrency overview at: https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html |
true | |
| sleeper.statestore.transaction.deletion.concurrency.max | The maximum given concurrency allowed for the transaction deletion lambda. See maximum concurrency overview at: https://aws.amazon.com/blogs/compute/introducing-maximum-concurrency-of-aws-lambda-functions-when-using-amazon-sqs-as-an-event-source/ |
true | |
| sleeper.statestore.transaction.deletion.lambda.timeout.seconds | The maximum timeout for the transaction deletion lambda in seconds. | 900 | true |
| sleeper.statestore.transaction.follower.concurrency.reserved | The reserved concurrency for the lambda that follows the state store transaction log to trigger updates. See reserved concurrency overview at: https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html |
true | |
| sleeper.statestore.transaction.follower.concurrency.max | The maximum given concurrency allowed for the lambda that follows the state store transaction log to trigger updates. See maximum concurrency overview at: https://aws.amazon.com/blogs/compute/introducing-maximum-concurrency-of-aws-lambda-functions-when-using-amazon-sqs-as-an-event-source/ |
false | |
| sleeper.statestore.transaction.follower.lambda.timeout.seconds | The maximum timeout in seconds for the lambda that follows the state store transaction log to trigger updates. | 900 | true |
| sleeper.statestore.transaction.follower.memory.mb | The amount of memory in MB for the lambda that follows the state store transaction log to trigger updates. | true | |
| sleeper.tables.index.dynamo.pointintimerecovery | This specifies whether point in time recovery is enabled for the Sleeper table index. This is set on the DynamoDB tables. | false | true |
| sleeper.tables.index.dynamo.consistent.reads | This specifies whether queries and scans against the table index DynamoDB tables are strongly consistent. | true | false |
| sleeper.statestore.committer.platform | The platform that the state store committer will be deployed to for execution. Valid values are: [lambda, ec2] NB: The EC2 platform is currently considered experimental. |
LAMBDA | true |
| sleeper.statestore.committer.lambda.memory.mb | The amount of memory in MB for the lambda that commits state store updates. | true | |
| sleeper.statestore.committer.lambda.timeout.seconds | The timeout for the lambda that commits state store updates in seconds. | 900 | true |
| sleeper.statestore.committer.batch.size | The number of state store updates to be sent to the state store committer lambda in one invocation. This will be the batch size for a lambda as an SQS FIFO event source. This can be a maximum of 10. | 10 | true |
| sleeper.statestore.committer.concurrency.reserved | The reserved concurrency for the state store committer lambda. Presently this value defaults to 10 to align with expectations around table efficiency. This is to ensure that state store operations can still be applied to at least 10 tables, even when concurrency is used up in the account. See reserved concurrency overview at: https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html |
10 | true |
| sleeper.statestore.committer.concurrency.max | The maximum given concurrency allowed for the state store committer lambda. See maximum concurrency overview at: https://aws.amazon.com/blogs/compute/introducing-maximum-concurrency-of-aws-lambda-functions-when-using-amazon-sqs-as-an-event-source/ |
true | |
| sleeper.statestore.committer.ec2.type | The EC2 instance type that the multi-threaded state store committer should be deployed onto. | m8g.xlarge | true |