There are often situations in which a program must ensure that it has exclusive rights to something. On POSIX systems this is traditionally done by creating a file to indicate a lock, because this is portable to many systems.
However, there are several traps to avoid. First, a program with root privileges can open a file, even if it sets the O_EXCL mode (which normally fails if the file already exists). If that can happen, don't use open(2), use link(2) to create the file. If you just want to be sure that your server doesn't execute more than once on a given machine, consider creating a lockfile as /var/log/NAME.pid with the pid as its contents. This has the disadvantage of hanging around if the program prematurely halts, but it's common practice and is easily handled by other system tools.
Second, if the lock file may be on an NFS-mounted filesystem, then you have the problem that NFS doesn't completely support normal file semantics. The manual for open(2) explains how to handle things in this case (which also handles the case of root programs):
... programs which rely on [the O_CREAT and O_EXCL flags of open(2)] for performing locking tasks will contain a race condition. The solution for performing atomic file locking using a lockfile is to create a unique file on the same filesystem (e.g., incorporating hostname and pid), use link(2) to make a link to the lockfile and use stat(2) on the unique file to check if its link count has increased to 2. Do not use the return value of the link(2) call.
Where possible, don't write code to handle passwords. In particular, if the application is local, try to depend on the normal login authentication by a user. If the application is a CGI script, depend on the web server to provide the protection. If the application is over a network, avoid sending the password as cleartext (where possible) since it can be easily captured by network sniffers and reused later. For networks, consider at least using digest passwords (which are vulnerable to active attack threats but protect against passive network sniffers).
If your application must handle passwords, overwrite them immediately after use so they have minimal exposure. In Java, don't use the type String to store a password because Strings are immutable (they will not be overwritten until garbage-collected and reused, possibly a far time in the future). Instead, in Java use char[] to store a password, so it can be immediately overwritten.
If your application permits users to set their passwords, check the passwords and permit only ``good'' passwords (e.g., not in a dictionary, having certain minimal length, etc.). You may want to look at information such as http://consult.cern.ch/writeup/security/security_3.html on how to choose a good password.
The Linux kernel (since 1.3.30) includes a random number generator. The random number generator gathers environmental noise from device drivers and other sources into an entropy pool. When accessed as /dev/random, random bytes are only returned within the estimated number of bits of noise in the entropy pool (when the entropy pool is empty, the call blocks until additional environmental noise is gathered). When accessed as /dev/urandom, as many bytes as are requested are returned even when the entropy pool is exhausted. If you are using the random values for cryptographic purposes (e.g., to generate a key), use /dev/random. More information is available in the system documentation random(4).
Often cryptographic algorithms and protocols are necessary to keep a system secure, particularly when communicating through an untrusted network such as the Internet. Where possible, use session encryption to foil session hijacking and to hide authentication information, as well as to support privacy.
Cryptographic algorithms and protocols are difficult to get right, so do not create your own. Instead, use existing standard-conforming protocols such as SSL, SSH, IPSec, GnuPG/PGP, and Kerberos. Use only encryption algorithms that have been openly published and withstood years of attack (examples include triple DES, which is also not encumbered by patents). In particular, do not create your own encryption algorithms unless you are an expert in cryptology and know what you're doing; creating such algorithms is a task for experts only.
In a related note, if you must create your own communication protocol, examine the problems of what's gone on before. Classics such as Bellovin [1989]'s review of security problems in the TCP/IP protocol suite might help you, as well as Bruce Schneier [1998] and Mudge's breaking of Microsoft's PPTP implementation and their follow-on work. Of course, be sure to give any new protocol widespread review, and reuse what you can.
Some security-relevant programs on Linux may be implemented using the Java language and/or the Java Virtual Machine (JVM). Developing secure programs on Java is discussed in detail in material such as Gong [1999]. The following are a few key points extracted from Gong [1999]:
Most Linux distributions include PAM (Pluggable Authentication Modules), a flexible mechanism for authenticating users. This includes Red Hat Linux, Caldera, Debian as of version 2.2; note that FreeBSD also supports PAM as of version 3.1. By using PAM, your program can be independent of the authentication scheme (passwords, SmartCards, etc.). Basically, your program calls PAM, which at run-time determines which ``authentication modules'' are required by checking the configuration set by the local system administrator. If you're writing a program that requires authentication (e.g., entering a password), you should include support for PAM. You can find out more about the Linux-PAM project at http://www.kernel.org/pub/linux/libs/pam/index.html.
Have your program check at least some of its assumptions before it uses them (e.g., at the beginning of the program). For example, if you depend on the ``sticky'' bit being set on a given directory, test it; such tests take little time and could prevent a serious problem. If you worry about the execution time of some tests on each call, at least perform the test at installation time.
Write audit logs for program startup, session startup, and for suspicious activity. Possible information of value includes date, time, uid, euid, gid, egid, terminal information, process id, and command line values. You may find the function syslog(3) helpful for implementing audit logs.
Have installation scripts install a program as safely as possible. By default, install all files as owned by root or some other system user and make them unwriteable by others; this prevents non-root users from installing viruses. Allow non-root installation where possible as well, so that users without root permission and administrators who do not fully trust the installer to still use the program.
If possible, don't create setuid or setgid root programs; make the user log in as root instead.
Sign your code. That way, others can check to see if what's available was what was sent.
Consider statically linking secure programs. This counters attacks on the dynamic link library mechanism by making sure that the secure programs don't use it.
When reading over code, consider all the cases where a match is not made. For example, if there is a switch statement, what happens when none of the cases match? If there is an ``if'' statement, what happens when the condition is false?
Ensure the program works with compile-time and run-time checks turned on, and leave them on where practical. Perl programs should turn on the warning flag (-w), which warns of potentially dangerous or obsolete statements, and possibly the taint flag (-T), which prevents the direct use of untrusted inputs without performing some sort of filtering. Security-relevant programs should compile cleanly with all warnings turned on. For C or C++ compilations using gcc, use at least the following as compilation flags (which turn on a host of warning messages) and try to eliminate all warnings:
gcc -Wall -Wpointer-arith -Wstrict-prototypes