wso2 am 1.10.0 API Store: "Error occurred while executing the action generateApplicationKey" with " Invalid credentials provided."

Question

Product: WSO2 AM 1.10.0 DB: MSSQL Security: SAML2 integrated with PingIdentity OS: Linux

When I tried to create or generate keys from "/store/site/pages/subscriptions.jag" page, UI gave me:

Error occurred while executing the action generateApplicationKey

Here is what's in the log:

Summary of the errors:

Error while obtaining the application access token for the application:safsa org.wso2.carbon.apimgt.api.APIManagementException: Error occurred while Creating Keys. at org.wso2.carbon.apimgt.impl.utils.APIUtil.handleException(APIUtil.java:1195) ...

Caused by: org.wso2.carbon.apimgt.api.APIManagementException: Error occurred while executing SubscriberKeyMgtClient. at org.wso2.carbon.apimgt.impl.utils.APIUtil.handleException(APIUtil.java:1195) ...

Caused by: org.wso2.carbon.apimgt.api.APIManagementException: Can not create OAuth application : safsa_SANDBOX at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.handleException(AMDefaultKeyManagerImpl.java:592) ...

Caused by: org.apache.axis2.AxisFault: Access Denied. Authentication failed - Invalid credentials provided. at org.apache.axis2.util.Utils.getInboundFaultFromMessageContext(Utils.java:531)

Here are the complete log messages: /repository/logs/wso2-apigw-errors.log <==

2016-07-07 21:04:14,427 [-] [http-nio-9443-exec-2] ERROR APIStoreHostObject Error while obtaining the application access token for the application:DefaultApplication
org.wso2.carbon.apimgt.api.APIManagementException: Error occurred while Creating Keys.
        at org.wso2.carbon.apimgt.impl.utils.APIUtil.handleException(APIUtil.java:1195)
        at org.wso2.carbon.apimgt.impl.APIConsumerImpl.completeApplicationRegistration(APIConsumerImpl.java:2751)
        at org.wso2.carbon.apimgt.impl.UserAwareAPIConsumer.completeApplicationRegistration(UserAwareAPIConsumer.java:36)
        at org.wso2.carbon.apimgt.hostobjects.APIStoreHostObject.jsFunction_createApplicationKeys(APIStoreHostObject.java:2348)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
        at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:386)
        at org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)
        at org.jaggeryjs.rhino.store.modules.subscription.c2._c_anonymous_2(/store/modules/subscription/key.jag:36)
        at org.jaggeryjs.rhino.store.modules.subscription.c2.call(/store/modules/subscription/key.jag)
        at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430)
        at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269)
        at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97)
        at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42)
        at org.jaggeryjs.rhino.store.modules.subscription.c0._c_anonymous_9(/store/modules/subscription/module.jag:32)
        at org.jaggeryjs.rhino.store.modules.subscription.c0.call(/store/modules/subscription/module.jag)
        at org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)
        at org.jaggeryjs.rhino.store.site.blocks.subscription.subscription_add.ajax.c0._c_anonymous_1(/store/site/blocks/subscription/subscription-add/ajax/subscription-add.jag:240)
        at org.jaggeryjs.rhino.store.site.blocks.subscription.subscription_add.ajax.c0.call(/store/site/blocks/subscription/subscription-add/ajax/subscription-add.jag)
        at org.mozilla.javascript.optimizer.OptRuntime.call0(OptRuntime.java:23)
        at org.jaggeryjs.rhino.store.site.blocks.subscription.subscription_add.ajax.c0._c_script_0(/store/site/blocks/subscription/subscription-add/ajax/subscription-add.jag:3)
        at org.jaggeryjs.rhino.store.site.blocks.subscription.subscription_add.ajax.c0.call(/store/site/blocks/subscription/subscription-add/ajax/subscription-add.jag)
        at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:394)
        at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3091)
        at org.jaggeryjs.rhino.store.site.blocks.subscription.subscription_add.ajax.c0.call(/store/site/blocks/subscription/subscription-add/ajax/subscription-add.jag)
        at org.jaggeryjs.rhino.store.site.blocks.subscription.subscription_add.ajax.c0.exec(/store/site/blocks/subscription/subscription-add/ajax/subscription-add.jag)
        at org.jaggeryjs.scriptengine.engine.RhinoEngine.execScript(RhinoEngine.java:567)
        at org.jaggeryjs.scriptengine.engine.RhinoEngine.exec(RhinoEngine.java:273)
        at org.jaggeryjs.jaggery.core.manager.WebAppManager.exec(WebAppManager.java:587)
        at org.jaggeryjs.jaggery.core.manager.WebAppManager.execute(WebAppManager.java:507)
        at org.jaggeryjs.jaggery.core.JaggeryServlet.doPost(JaggeryServlet.java:29)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:748)
        at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:486)
        at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:378)
        at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:338)
        at org.jaggeryjs.jaggery.core.JaggeryFilter.doFilter(JaggeryFilter.java:21)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
        at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
        at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:57)
        at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
        at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
        at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:159)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
        at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1739)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1698)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.wso2.carbon.apimgt.api.APIManagementException: Error occurred while executing SubscriberKeyMgtClient.
        at org.wso2.carbon.apimgt.impl.utils.APIUtil.handleException(APIUtil.java:1195)
        at org.wso2.carbon.apimgt.impl.workflow.AbstractApplicationRegistrationWorkflowExecutor.dogenerateKeysForApplication(AbstractApplicationRegistrationWorkflowExecutor.java:186)
        at org.wso2.carbon.apimgt.impl.APIConsumerImpl.completeApplicationRegistration(APIConsumerImpl.java:2735)
        ... 66 more
Caused by: org.wso2.carbon.apimgt.api.APIManagementException: Can not create OAuth application  : DefaultApplication_SANDBOX
        at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.handleException(AMDefaultKeyManagerImpl.java:592)
        at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.createApplication(AMDefaultKeyManagerImpl.java:106)
        at org.wso2.carbon.apimgt.impl.workflow.AbstractApplicationRegistrationWorkflowExecutor.dogenerateKeysForApplication(AbstractApplicationRegistrationWorkflowExecutor.java:150)
        ... 67 more
Caused by: org.apache.axis2.AxisFault: Access Denied. Authentication failed - Invalid credentials provided.
        at org.apache.axis2.util.Utils.getInboundFaultFromMessageContext(Utils.java:531)
        at org.apache.axis2.description.OutInAxisOperationClient.handleResponse(OutInAxisOperation.java:370)
        at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:445)
        at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
        at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
        at org.wso2.carbon.apimgt.keymgt.stub.subscriber.APIKeyMgtSubscriberServiceStub.createOAuthApplicationByApplicationInfo(APIKeyMgtSubscriberServiceStub.java:2099)
        at org.wso2.carbon.apimgt.keymgt.client.SubscriberKeyMgtClient.createOAuthApplicationbyApplicationInfo(SubscriberKeyMgtClient.java:62)
        at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.createApplication(AMDefaultKeyManagerImpl.java:104)
        ... 68 more

Answer 1

What I did to fix the issue was to 1) add admin user inside ApiKeyValidaor in api-manager.xml also into admin user via management console and into user-mgt.xml; 2) Inside api-manager.xml:

Change the following:

https://${carbon.local.ip}:${mgt.transport.https.port}${carbon.context}/services/

to: https://[FQDN_OF_HOST}:${mgt.transport.https.port}${carbon.context}/services/

Reason is my server certificate only recorded the domain name, not ip address.