Fixing Oracle EM Agent SSL Handshake Errors Due to Weak Certificate Key

In a recent my environment, while working on securing Oracle Enterprise Manager (OEM) Agent communication during a data center migration, we encountered an issue where the OEM Agent failed to start due to TLS handshake failures. The root cause was related to a weak SSL certificate (1024-bit key), which modern TLS clients reject due to security standards.

Environment

• OEM Version: Oracle Enterprise Manager 13c Release 5
• OS: Oracle Linux
• Agent Port: 3872
• Certificate Protocol: TLS 1.2 with self-signed keys

Problem Summary

After the agent was deployed and configured, we noticed:

• The agent wouldn’t start properly.
• SSL handshake using the openssl s_client command returned errors such as: verify error:num=66:EE certificate key too weak
  verify error:num=67:CA certificate key too weak
• Enterprise Manager logs (from emctl secdiag) showed the following: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated

Example openssl output from the environment:

CONNECTED(00000003)
depth=0 CN = <agent-host>
verify error:num=66:EE certificate key too weak
verify return:1
depth=1 CN = <agent-host>
verify error:num=19:self signed certificate in certificate chain
verify error:num=67:CA certificate key too weak
verify return:1
---
Server public key is 1024 bit
Verify return code: 67 (CA certificate key too weak)

Testing the URL via wget:

wget --no-check-certificate https://<agent-host>:3872/emd/main/
WARNING: The certificate of '<agent-host>' is not trusted.
WARNING: The certificate hasn't got a known issuer.
WARNING: The certificate was signed using an insecure algorithm.
HTTP request sent, awaiting response... 200 OK

Using emctl secdiag:

emctl secdiag openurl -url https://<agent-host>:3872/emd/main/
Following exception occurred when running OpenPage
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated

Root Cause

The default SSL certificate generated during agent deployment used only a 1024-bit key. Current TLS policies enforce a minimum of 2048 bits for certificate keys. Additionally, the certificate was self-signed and used a weak signing algorithm.

Solution

1. Stop the agent: emctl stop agent
2. Edit the emd.properties file: /u01/app/oracle/agent_inst/sysman/config/emd.properties
3. Modify the value: Change SSLCertKeySize-1024 to SSLCertKeySize=2048
4. Regenerate the secure agent certificate: emctl secure agent Accept the prompt to regenerate the certificate.
5. Start the agent again: emctl start agent
6. Re-test using openssl s_client and wget: The certificate chain now shows a 2048-bit key, and the error regarding weak certificate key is no longer observed.

Validation

After the fix, the following validations were successful:

• openssl s_client confirmed certificate key size upgraded to 2048-bit.
• wget –no-check-certificate returned HTTP 200 without certificate errors blocking the call.
• emctl upload agent worked without handshake issues.
• Agent successfully communicated with the OMS.

Final Thoughts

This issue highlights the importance of aligning agent certificates with modern TLS standards. For secure environments or cloud-integrated monitoring, using at least 2048-bit key length is essential to prevent SSL handshake failures and ensure reliable agent registration with the Oracle Management Server.