Encountered a problem with Agile PLM where the database user credentials were invalid, but the error returned referenced the WebLogic Server (WLS) superadmin credentials.
This caused us (me!) to look in the wrong area for troubleshooting for several hours, and needless downtime of a systemthat, while not production, was critical for training efforts for a new process to go live several days later.
Hopefully by putting this data here it might show up in google it will save someone else some grief.
Startup uses WLS encrypted boot.properties for the WLS credentials. While this is not how the system is setup by Agile install binary, changing this around is a) trivial and b) common sense.
Parking login credentials in a bash script variable is asking for trouble.
I do not believe this has anything to do with the problem. But certain people (coughOraclecough) were confused on this point.
Symptoms
On startup of WLS Node Manager fails with this error message.
<Critical> <Security> <BEA-090402> <Authentication denied:
Boot identity not valid; The user name and/or password from the boot identity file (boot.properties)
is not valid. The boot identity may have been changed since the boot identity file was created.
Please edit and update the boot identity file with the proper values of username and password.
The first time the updated boot identity file is used to start the server, these new values are encrypted.>
This is, needless to say, alarming.
In an effort to simplify things, do some fault isolation, one might change the startup scripts to restore the WLS superadmin credentials and is rewarded with this error message.
<Critical>
<WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException:
Authentication for user superadmin denied
weblogic.security.SecurityInitializationException: Authentication for user superadmin denied
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(Unknown Source)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(Unknown Source)
at weblogic.security.service.SecurityServiceManager.initialize(Unknown Source)
at weblogic.security.SecurityService.start(SecurityService.java:141)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
Truncated. see log file for complete stacktrace>
<Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
One can even remove the shell variables from the startup script, and will be prompted to manually input the credentials. Which will fail with the above error message.
The problem was NOT with the WLS superadmin credentials. I know this because we can look in the database for those. While they are encrypted, after a while one learns that the encrypted value AE43938383AKKH23566 eq 'tartan' [1].
What Happened
We have different credentials for the Agile database user in our five different environments.
The day prior, we had refreshed our Training environment with a copy of the Production database. To do this we used the cloning facility provided by NetApp.
The database came up 'ok', but when the application supplied database credentials on startup ... it failed.
Which one could expect. The passwords are different.
What one doesn't expect is that the error returned does not reference an actual database account, but WLS superadmin credentials, which are still valid: authentication for user superadmin denied.
As soon as the DBA applied the Training environment credentials to the database, I was able to start the Node Manager and Managed Hosts.
Sha-zam!
So What?
Based on my experience, by comments from the Oracle SE, I believe that any Agile PLM version 9.3 [2] database account that is locked out, has invalid credentials, will experience the same problem: that is they will have an error that indicates the WLS superadmin credentials are wrong, when they are not.
This seems to not be common knowledge with the Agile support team at Oracle, or if it is it's not something the support engineer assigned to the case knew about. And when he closed the case he was real quick to not make a KB out of the problem because it was, he said, specific to the way we implemented Agile PLM.
Which seems wrong but whatever. [3]
If this happens to you, hopefully you will find this document, not have to waste hours and hours of time fixing your problem.
[1] Not the real password.
[2] At least this version. The way to bet is all versions that use WLS.
[3] Ref.
This caused us (me!) to look in the wrong area for troubleshooting for several hours, and needless downtime of a systemthat, while not production, was critical for training efforts for a new process to go live several days later.
Hopefully by putting this data here it might show up in google it will save someone else some grief.
- Agile PLM 9301
- WebLogic Server 10.3
- Solaris 10
Startup uses WLS encrypted boot.properties for the WLS credentials. While this is not how the system is setup by Agile install binary, changing this around is a) trivial and b) common sense.
Parking login credentials in a bash script variable is asking for trouble.
I do not believe this has anything to do with the problem. But certain people (coughOraclecough) were confused on this point.
Symptoms
On startup of WLS Node Manager fails with this error message.
<Critical> <Security> <BEA-090402> <Authentication denied:
Boot identity not valid; The user name and/or password from the boot identity file (boot.properties)
is not valid. The boot identity may have been changed since the boot identity file was created.
Please edit and update the boot identity file with the proper values of username and password.
The first time the updated boot identity file is used to start the server, these new values are encrypted.>
This is, needless to say, alarming.
In an effort to simplify things, do some fault isolation, one might change the startup scripts to restore the WLS superadmin credentials and is rewarded with this error message.
<Critical>
<WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException:
Authentication for user superadmin denied
weblogic.security.SecurityInitializationException: Authentication for user superadmin denied
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(Unknown Source)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(Unknown Source)
at weblogic.security.service.SecurityServiceManager.initialize(Unknown Source)
at weblogic.security.SecurityService.start(SecurityService.java:141)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
Truncated. see log file for complete stacktrace>
<Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
One can even remove the shell variables from the startup script, and will be prompted to manually input the credentials. Which will fail with the above error message.
The problem was NOT with the WLS superadmin credentials. I know this because we can look in the database for those. While they are encrypted, after a while one learns that the encrypted value AE43938383AKKH23566 eq 'tartan' [1].
What Happened
We have different credentials for the Agile database user in our five different environments.
The day prior, we had refreshed our Training environment with a copy of the Production database. To do this we used the cloning facility provided by NetApp.
The database came up 'ok', but when the application supplied database credentials on startup ... it failed.
Which one could expect. The passwords are different.
What one doesn't expect is that the error returned does not reference an actual database account, but WLS superadmin credentials, which are still valid: authentication for user superadmin denied.
As soon as the DBA applied the Training environment credentials to the database, I was able to start the Node Manager and Managed Hosts.
Sha-zam!
So What?
Based on my experience, by comments from the Oracle SE, I believe that any Agile PLM version 9.3 [2] database account that is locked out, has invalid credentials, will experience the same problem: that is they will have an error that indicates the WLS superadmin credentials are wrong, when they are not.
This seems to not be common knowledge with the Agile support team at Oracle, or if it is it's not something the support engineer assigned to the case knew about. And when he closed the case he was real quick to not make a KB out of the problem because it was, he said, specific to the way we implemented Agile PLM.
Which seems wrong but whatever. [3]
If this happens to you, hopefully you will find this document, not have to waste hours and hours of time fixing your problem.
[1] Not the real password.
[2] At least this version. The way to bet is all versions that use WLS.
[3] Ref.
