For encrypting the database password use this command:
dm_encrypt_password -location C:\Documentum\dba\secure\aek.key -docbase test -rdbms -encrypt ****
For encrypting other passwords like the dm_bof_registry password use this command:
java -cp dfc.jar com.documentum.fc.tools.RegistryPasswordUtils ****
In above java is in the path and you are located in the DFC Shared folder.