DRILL-8504: Add Schema Caching to Splunk Plugin

[cgivre]

DRILL-8504: Add Schema Caching to Splunk Plugin

Description

Whenever Drill executes a Splunk query, it must retrieve a list of indexes from Splunk. This step can add a considerable amount of time to the planning phase. This PR introduces a simple in-memory cache for the Splunk plugin which caches the list of indexes to avoid having to query Splunk repeatedly to obtain this information.

This PR also makes a few unrelated minor improvements:

• Updates the test container to Splunk version 9.3 which at the time of writing is the most current version. I had to update some unit tests as a result.
• Adds a new config option for the maximum columns returned in Splunk
• Adds the actual SPL sent to Splunk to the query plan. This can be useful for debugging.

Documentation

(Added to README)
For every query that you send to Splunk from Drill, Drill will have to pull schema information from Splunk. If you have a lot of indexes, this process can cause slow planning time. To improve planning time, you can configure Drill to cache the index names so that it does not need to make additional calls to Splunk.

There are two configuration parameters for the schema caching: maxCacheSize and cacheExpiration. The maxCacheSize defaults to 10k bytes and the cacheExpiration defaults to 1024 minutes. To disable schema caching simply set the cacheExpiration parameter to a value less than zero.

Testing

Ran all unit tests and tested manually.

[jnturton]

Can we achieve the same thing using Guava's caching? The reason I ask is that we already have this insanely big dependency tree and Guava is already in it...

https://www.baeldung.com/guava-cache

[jnturton]

But so is caffeine now that I look! So I guess we can ignore this suggestion.

[cgivre]

@jnturton Is that a +1? I somehow broke the versioning when I rebased on the current master, but I'll fix before merging.

[jnturton]

Sorry, I got pulled away before I could continue but will complete the review today.

[jnturton]

Please lift the caffeine.version property from metastore/iceberg-metastore/pom.xml to the root pom and either

• add caffeine with caffeine.version to dependencyManagement in the root pom and remove version numbers here and in the Iceberg metastore, or
• make both this pom and Iceberg metastore pom specify <version>${caeffine.version}</version>.

So that we standardise the version of Caffeine that gets pulled in.

[cgivre]

@jnturton It looks like the GitHub CI is failing on the Hadoop 2 tests with Hive.

[jnturton]

Please lift the caffeine.version property from metastore/iceberg-metastore/pom.xml to the root pom and either

• add caffeine with caffeine.version to dependencyManagement in the root pom and remove version numbers here and in the Iceberg metastore, or
• make both this pom and Iceberg metastore pom specify <version>${caeffine.version}</version>.

So that we standardise the version of Caffeine that gets pulled in.

[jnturton]

It feels like it would be more natural (and efficient) for the cache to hold one entry per Splunk index, rather than a single entry containing the list of all indexes. Is there a reason it isn't built that way?

[cgivre]

@jnturton I think that's exactly what it does. The invalidate is just the delete method, so the code there removes any cache entries with that entry. Also as an FYI, the cache adds the username to the index name so that if user translation is enabled, users will not see other users' cache.

[jnturton]

@cgivre is the cache really storing one Splunk index per key? Here it looks to me like there's a single cache key, derived from the queryUserName and the plugin name, that holds a list of indexes. Or am I just being confused by the Caffeine API?

[cgivre]

@jnturton ,
I'm sorry, I misspoke. The way I intended the cache to work was that we combine the plugin name + user name to create a key, and the value is a list of indexes. Every time a user adds or drops an index, we have to recreate the cache entry for that plugin/username.

So:

splunk1-cgivre: [index1, index2, index2]
splunk2-cgivre: [index5, index6, index7]
splunk1-jnturton: [index1, index2]

That's what the cache should look like. (In theory)

[jnturton]

Okay, in that case, can I propose

• a Map of caches with one cache per plugin name + query user and one cache key per Splunk index or
• a unified cache with one cache key per plugin name + query user + Splunk index name?

[cgivre]

@jnturton I'm remembering why I did it this way. When the plugin is accessed, Drill calls the registerTable method which loads the list of indexes in to memory. From my recollection, all similar storage plugins, do something similar in that they load the schemata into memory when the plugin is first accessed. Here the change I made was to first check the memory cache and if that's not populated, then call the index list from Splunk.

I'm happy to do this as you suggest but I'm a little confused as to what exactly you're asking for.

[jnturton]

Let's go with this implementation. Fine tuning can come later if it's needed.

[cgivre]

@jnturton
Thanks for the advice about the swap file. We got a clean CI run. Are we good to merge?