If your application must handle passwords or non-public keys (such as session keys, private keys, or secret keys), overwrite them immediately after using them so they have minimal exposure. For example, in Java, don't use the type String to store a password because Strings are immutable (they will not be overwritten until garbage-collected and reused, possibly a far time in the future). Instead, in Java use char[] to store a password, so it can be immediately overwritten.
Also, if your program handles such secret values, be sure to disable creating core dumps (via ulimit). Otherwise, an attacker may be able to halt the program and find the secret value in the data dump. Also, beware - normally processes can monitor other processes through the calls for debuggers (e.g., via ptrace(2) and the /proc pseudo-filesystem) [Venema 1996] Kernels usually protect against these monitoring routines if the process is setuid or setgid (on the few ancient ones that don't, there really isn't a way to defend yourself other than upgrading). Thus, if your process manages secret values, you probably should make it setgid or setuid (to a different unprivileged group or user) to forceably inhibit this kind of monitoring.