datastax - Enable one time Cassandra Authentication and Authorization check and cache it forever

Question

I use the authentication and authorization in my single node Cassandra setup, But I frequently get the following error in Cassandra server logs,

ERROR [SharedPool-Worker-71] 2018-06-01 10:40:36,661 ErrorMessage.java:338 - Unexpected exception during request
java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 1 responses.
        at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:489) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:269) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.auth.RolesCache.getRoles(RolesCache.java:66) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:51) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:76) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:68) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.service.ClientState.authorize(ClientState.java:412) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:345) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:322) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:309) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:293) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:198) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:203) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:487) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:464) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:130) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) [apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) [apache-cassandra-3.0.8.jar:3.0.8]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final]
        at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final]
        at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
        at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) [apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.0.8.jar:3.0.8]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 1 responses.
        at org.apache.cassandra.service.ReadCallback.awaitResults(ReadCallback.java:132) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:137) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:145) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.awaitResultsAndRetryOnDigestMismatch(StorageProxy.java:1715) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1664) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1605) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1524) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:954) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:263) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:497) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:485) ~[apache-cassandra-3.0.8.jar:3.0.8]
        ... 27 common frames omitted

So considering this I try to enable one time Cassandra Authentication and Authorization check and cache it forever based on the following setting observed in the URL,

https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/security/secAuthCacheSettings.html

authenticator: PasswordAuthenticator
authorizer: CassandraAuthorizer
role_manager: CassandraRoleManager
roles_validity_in_ms: 0
permissions_validity_in_ms: 0

But still I see the above errors frequently in the server logs, Is it necessary to add this configuration also : credentials_validity_in_ms: 0 Or Am I missing something?

Answer 1

This message is really a signal of something wrong with your setup - machines are overloaded, or something like.

Instead of disabling these settings completely (changing password or chaning role will require restarting of the nodes), I would suggest to do following instead:

• Set roles_validity_in_ms, permissions_validity_in_ms & credentials_validity_in_ms to some quite high value, something like month;
• Configure roles_update_interval_in_ms, credentials_update_interval_in_ms & permissions_update_interval_in_ms to some value, like a minute

It also makes sense to tune permissions_cache_max_entries if you have big number of users & tables.