DQE with sendmail and procmail -- HOWTO

Background

(Note: parts of this document are out-of-date; you will have to do some experimenting to get everything to work. Please email us if you get stuck.)

This document describes how to set up the client-side of DQE.

The basic idea of the system is this: every e-mail sender is given a quota of stamps. When a receiver's e-mail server receives e-mail, the server cancels the stamp by storing a record of the stamp in the enforcer. With this approach, a sender can use each stamp only once, which will prevent any given entity from sending more than a certain number of e-mails. If a sender reuses a stamp, the receiver will see that the enforcer already has a copy of the stamp.

A bit more detail: e-mail senders attach to each e-mail a stamp that consists of two pieces:

  1. A certificate signed by an allocator whose public key is trusted by everyone; the certificate declares the e-mail sender's quota and authorizes the e-mail sender's public key;
  2. A number between 1 and N (where N is the quota declared by the quota allocator).

When an e-mail recipient gets an e-mail, the recipient looks at the stamp and checks that it has a certificate signed by a private key whose corresponding public key is trusted by the e-mail recipient. If the certificate is valid, then the e-mail recipient trusts the public key that the certificate is authorizing. Then, the e-mail recipient checks that the number in the stamp is between 1 and N. Then, the e-mail recipient creates a hash of the stamp and first checks that this hash has not appeared in the enforcer. If the hash has appeared, then the stamp is probably being reused, which could indicate that the e-mail is spam. If the hash has not appeared, the recipient stores the hash in the enforcer.

In the rest of this document, we describe how to set up e-mail servers so that the sending direction affixes stamps and the receiving direction checks and cancels stamps. We describe the following steps in turn:

  1. Setting up the outgoing mail server to affix stamps to all outgoing messages
  2. Setting up the incoming mail server to check incoming mail for stamps, either site-wide or per-user (both are described here).

Please Note: These instructions assume that separate servers are used for outgoing and incoming mail.

I. Setting up an Outgoing Mail Server

Setting up the outgoing mail server requires several steps, which we describe below. Briefly, we need to:

  1. Set up a pool of stamps to be used for outgoing mail;
  2. Make sendmail "filter" all outgoing mail with procmail;
  3. With this "filtering", make procmail affix a new stamp to all mail before sending the mail to its intended recipients.

1. Setting up the stamp pool

Recall from the above background that under DQE, each outgoing e-mail gets a new stamp. Each stamp has two pieces:
  1. A certificate signed by a quota authority (right now, the quota authority is the grad student working on DQE) expressing to the world that a given entity (as identified by a public key) is allowed to manufacture a certain number, N, of distinct stamps;
  2. A number between 1 and N, signed by the entity's public key.
Here are the steps to set up a stamp pool:

  1. On the outgoing e-mail server, unpack the DQE tarball, e.g.: 

    
	> mv dqe.tar.gz /tmp/
	> cd /tmp ; tar -zxvf dqe.tar.gz 
	> mv dqe /var/local/dqe
    We'll call the DQE installation location $DQM_PATH. In the example above, this location is /var/local/dqe.

  2. Now you need to tell python where to find the DQE files. Set the environment variable $PYTHONPATH to $DQM_PATH (expand $DQM_PATH yourself; it's just a placeholder in this document).

  3. Output a public key like this:
    > cd $DQM_PATH/crypto ; ./gen_key -p -b dqm_identity
    (This results in two files, dqm_identity and dqm_identity.pub, being output.)

  4. Now you need to get your public key authorized. For now, just create another key using gen_key, and then sign that key with the new private key by calling gen_cert.

  5. Now you need to output a pool of stamps (a pool of stamps is just a list of integers, in a random permutation; the randomness is for privacy so recipients won't know how much e-mail a given sender sends). Output a pool of stamps like this: 

    

    > cd $DQM_PATH/stamp_pool
    > ./gen_dbpool <quota> $DQM_PATH/res/disp.pool.1/pool_db

    where <quota> is the quota that you gave yourself in the quota allocation step.

  6. Edit a client-side config file as follows (when we write $DQM_PATH, we mean for the reader to do the expansion; the config file doesn't understand variables so you need to type out the full path):

    Note that there is one more value to be edited in this file (set_queue), which we describe below.

  7. Now test your set-up. Execute:
    > cd $DQM_PATH/user ; ./stamp-printer.py -w dqm.config
    You should see a bunch of funny characters, the first of line of which looks like this:
    0.9#THE_QUOTA_DUDE:AAAAgJx8yQlqe5134UgSEScZ6zTCq1BOea2t2dC8l6mkX72fZZf
    If that test worked, then your computer can print stamps. If not, then please e-mail dqm-dev2 at the domain nms.csail.mit.edu.

  8. The final step is set things up so that the stamp pool renews itself every week. This step is necessary because the quota mentioned above is really a weekly quota and stamps are printed with the current week to prevent reuse in future weeks. Add the lines below to the crontable of root on the outgoing mailserver, and make sure you replace /var/local/dqm with the DQE installation directory (but you need to write out the whole path in the variable assignment because cron doesn't support variable expansion in an assignment). You can get a sample crontab file containing these lines in $DQM_PATH/res/sample-outgoing.crontab. 
    PYTHONPATH     = /var/local/dqe
DQM_CONFIG     = /var/local/dqe/dqm.config

DB_GEN         = /var/local/dqe/stamp_pool/gen_dbpool
DB_GEN_WRAPPER = /var/local/dqe/scripts/wrap_gen_dbpool.sh
# put whatever quota you have been using here!
QUOTA          = 20000

# replace the supply of stamps each week. should be done just as
# the ISO week begins, which is Monday after midnight.
5 0 * * mon     /bin/bash $DB_GEN_WRAPPER $DQM_CONFIG $DB_GEN $QUOTA &> /dev/null
    The above lines will create a new stamp pool each week. If you created the stamp pool as $DQM_PATH/res/disp.pool.1/db_pool, then in the following weeks, the pool will be $DQM_PATH/res/disp.pool.2/db_pool, $DQM_PATH/res/disp.pool.3/db_pool, etc.

2. Making sendmail filter outgoing mail with procmail

This step requires some additions to the sendmail.mc file.

  1. Backup the mail server's sendmail.cf and sendmail.mc files to retore the original setup in case the new setup does not work.

  2. Modify your sendmail.mc by appending the following lines: 

    LOCAL_CONFIG
# add .PROCMAIL to the pseudo-domain list
CP.PROCMAIL
LOCAL_RULE_0

# match all mail going to pseudo-domain .procmail 
# and forward it using the esmtp mailer (you may need
# to use smtp, depending on your configuration)
R$+ < @ $+ .procmail . >
	$#esmtp $@ $2 $: $1<@$2>
# match all other mail and send it to procmail script 
# stamp-adder, rewriting the destination by adding .procmail
R$+ < @ $+ . >
	$#procmail $@/etc/procmailrcs/stamp-adder $:$1<@$2.procmail>
    The white space in the above lines in front of $#esmtp and $#procmail is tabs, \t, ASCII character #9, etc. They must be tabs in order for your setup to work.

    Please Note: The assumption that the outgoing mail server doesn't handle any incoming mail plays a key role in this step: if you plan on receiving mail on this server, you need to make sure that local mail does not get to these two lines (you must add a rule that delivers local mail before sendmail tries to smtp it back to itself), and then you must handle the special case of a local user sending mail to another local user, and decide whether to affix a stamp in that case.

    (If you find a simpler way to handle incoming and outgoing mail on a single server, please let us know! (email dqm-dev2 at the domain nms.csail.mit.edu).

  3. Recompile your sendmail.mc file and restart sendmail.

3. Making procmail affix stamps to all mail

  1. Copy the script $DQM_PATH/scripts/stamp-adder to /etc/procmailrcs/stamp-adder, or whereever else is appropriate for your system, making sure to modify the path in your sendmail.mc to match.

  2. Make sure stamp-adder is readable by the mailer daemon. The mailer daemon (sendmail, in this case) usually runs as root. If you have a different configuration, make sure whichever user sendmail runs as has access to this file.

  3. Make sure that in stamp-adder the variable $DQM_DIR is set to the location where DQE was installed (we have been calling this location $DQM_PATH in this document).

Now do an end-to-end test: send an e-mail using the e-mail server as the outgoing relay. The e-mail should have a header line X-dqm-stamp with a string of characters after it.

II. Setting up the Incoming Mail Server

Depening on whether you prefer a setup that applies to all users of your incoming mail server or else a per-user setup, these instructions will differ. However, the general idea is the same.

The incoming mail server part of DQE has two components. The first is a script, stamp-verifier.py, that is invoked by a procmail rule. stamp-verifier.py tests the stamps of incoming messages and affixes a header, X-dqm-checked, to each message. The value of this header depends on the results of the tests. Possible values of this header are SUCCESS, STAMPLESS_EMAIL, and several others. SUCCESS and STAMPLESS_EMAIL are self-explanatory. The other error codes (WRONG_FORMAT, INVALID_CERT_SIG, EXPIRED_CERT, INVALID_DISPOSABLE_SIG, BAD_WEEKSTAMP, INVALID_CERT_SIGNER, OVER_QUOTA) should be treated the same -- as indicating that the original e-mail carried an invalid stamp. After affixing the additional header, stamp-verifier.py deposits the stamp to be canceled in a queue on disk.

The second component is a backgrounded stamp canceller that reads this on-disk queue and cancels the stamps in it soon after they are deposited. This cancellation occurs as the backgrounded stamp canceller communicates with the enforcer. The reason for this separation is to decrease the amount of time procmail needs to spend per message, e.g., to decrease the complexity of the incoming-mail message path.

The procmail script: checking stamps

Please Note: If you are deploying on a per-user basis and your users are clueful, the simplest deployment strategy is to have your users run the add-stamp-checking-to-procmailrc.sh script. The script is a bash shell script, so invoke it like this:
> cd $DQM_PATH/scripts ; bash add-stamp-checking-to-procmailrc.sh

In general, however, this process consists of adding the following lines to your procmailrc, whether it is your site-wide procmailrc (found by default at /etc/procmailrc) or your user-specific procmailrcs (found by default at $HOME/.procmailrc): 

:0 c:
$HOME/DQM-MAIL-COPY

# name of dummy user
DQM_DUMMY         = dqm-dummy
DQM_DUMMY_ID      = /var/local/dqe/${DQM_DUMMY}_id_rsa

# location on host.with.scripts of DQM installation
DQM_HOME          = /var/local/dqe
PYTHONPATH        = $DQM_HOME
# where the DQM magic happens
DQM_RESOURCES_DIR = $DQM_HOME
DQM_SCRIPTS_DIR   = $DQM_HOME/scripts
DQM_CONFIG        = $DQM_RESOURCES_DIR/dqm.config
STAMP_VERIFIER    = $DQM_SCRIPTS_DIR/stamp-verifier.py
DQM_LOG           = /tmp/dqm.log

:0 fhw:
| /usr/bin/ssh $DQM_DUMMY@host.with.scripts -o StrictHostKeyChecking=no -o BatchMode=yes -i $DQM_DUMMY_ID 
        "export PYTHONPATH=$PYTHONPATH ; $STAMP_VERIFIER $DQM_CONFIG $DQM_LOG"
This seems complex, but we have tried to make it as general as possible, assuming that the verification script lives on a machine distinct from the incoming mail server.

Make sure the above procmail code reflects your site configuration. In particular, you may not need the DQM_DUMMY_ID variable if stamp-verifier.py is running on the same host as the one that is receiving the e-mail.

When you have completed these steps, send yourself e-mail and see whether the e-mail has the header X-dqm-checked.

The cronjob: canceling stamps

Stamps are cancelled in the background by a program called stamp-canceller.py. A program called check-canceller.sh keeps stamp-canceller.py running. Expand root's crontable, so it incorporates the lines below. You can get a sample crontab file containing these lines in $DQM_PATH/res/sample-incoming.crontab. Make sure you replace /var/local/dqm with the DQE installation directory (but you need to write out the whole path in the variable assignment because cron doesn't support variable expansion in an assignment): 
PYTHONPATH     = /var/local/dqe
DQM_CONFIG     = /var/local/dqe/dqm.config

CHECK_CANCELLER = /var/local/dqe/sh-scripts/check-canceller.sh
CANCELLER       = /var/local/dqe/user/stamp-canceller.py
CANCELLER_LOG   = /tmp/canceller.log

# keep the stamp-canceller going.
*/1 * * * *     bash $CHECK_CANCELLER $DQM_CONFIG $CANCELLER $CANCELLER_LOG
You need to tell the enforcer client where to queue stamps up for cancellation. In the file $DQM_PATH/res/dqm.config, set the value for set_queue to $DQM_PATH/res/set.queue/queue_db (of course, don't write $DQM_PATH; rather, expand the path yourself).

End-to-end Test

Send yourself an e-mail using the outgoing and incoming mail setups you established above. Check your e-mail. The X-dqm-stamp: field should be set, and there should be a header that says
X-dqm-checked: SUCCESS.

If that worked, you're done! Please mail dqe at the domain nms.csail.mit.edu to let us know that you're using DQE. If that didn't work, please send us mail in that case too so we can help debug.