After upgrading to JAMS V6.X or higher, you may see the Invalid Key Format error. 

Exception while sending job to Agent:
Invalid key format.

So why does this happen?

Rebex (a security tool) has a required minimum key length of 512. Keys generated in older versions of JAMS may not meet this improved security requirement.

To confirm your key length, run the following query on the JAMS Database:

select Datalength(encrypted_privatekey)
from UserSecurity where username = 'PROBLEMUSER'

If the length listed is not 512 or higher, implement a new security key to resolve your error. A new key must be generated or loaded. If you receive keys from an outside source, a new compliant key must be loaded into JAMS.

Generate a new key

First, clear out the Problem Key 

• Select the trouble user from the management section, and click on Key Management.
• From the dialog box that appears, clear the key.

Then, generate the new Key

• Select the trouble user from the management section, and click on Key Management.
• From the dialogue box that appears, Generate a new Key.

You can also generate a key programmatically:

• Generate Key Method

Load a new key

First, obtain a new key from your responsible party.

Then, use the method above to clear the existing problem key.

Finally, load the new key into JAMS.