Email
Creating Lists in Sympa
Creating a list in Sympa involves a whole host of files interacting with each other, and all sorts of configuration snippets are used to create the filename of a resource needed to build the list. This attempts to describe what is taken from where, when a list is created from the command-line with a “sympa.pl --create_list” command.
The command (run as root) is
sympa.pl --create_list --input_file=<xml-file>
<xml-file> is a copy of /etc/sympa/sotontemplate.xml with tokens substituted in to set values which are inserted into the template config file. The <xml-file> can be deleted immediately after the list is created, it is not needed.
The <xml-file> looks like this (note the *1* and so on for footnotes):
Notes:
The XML file is meshed into the template config file in /etc/sympa/create_list_templates/<type>/config.tt2 and then interpreted as a TT2 template (see www.tt2.org), and the output goes into the list's own individual config file which is at
/var/lib/sympa/list_data/all.soton.ac.uk/<listname>/config
That file is compiled by Sympa automatically when needed. It can be edited manually at any time, and Sympa will automatically recompile it as and when needed, without you needing to tell it about the change.
Note that the syntax of the template config file config.tt2 is very strict. It is constructed of a series of "paragraphs" which are a bunch of lines, with a blank line terminating the paragraph. It does not support comments, blank lines are *very* important, and leading white space on a line breaks everything. You have been warned!
"Editors" is what Sympa calls list moderators. When they are used depends on the setting of the "send" parameter in the list config file. See the section on "Authorisation Scenarios".
The editors of a list are not statically defined, but are pulled from a SQL database query as needed. The data is cached by Sympa for speed.
If the list has any editors, the editors are defined in the list's config file by a "paragraph" like this:
Notes:
The data source file looks like this:
Note this file *does* handle leading white space on lines.
Most of this is obvious, and it can handle any <db_type> that is available in the Perl DBI modules.
The "source_parameters" from the list's config file are received as the "param" array, and the data source file is again a TT2 file (see www.tt2.org).
The default editor address for every list is “postmaster@ecs.soton.ac.uk".
"Subscribers" are what Sympa calls list members. Only list subscribers receive a copy of postings to the list. So people can be able to post to a list without being able to receive a copy of their own posting. Posters who are subscribers always receive a copy of their own posting.
The subscriber list for a list is generated by an SQL query defined directly in the list's config file, it does not use a .incl file like the editors. The paragraphs defining the subscribers (taken from the template config file, not a list's real config file) look like this:
Notes:
The list of subscribers for each list is cached by Sympa for a length of time defined by the /etc/sympa/sympa.conf setting "default_ttl" and is set to 3600 seconds (1 hour) at the time of writing. So all changes to the table of list subscribers in the "all_lists" database will take up to 1 hour to propagate to the lists. There is no way for force an update of a list's subscribers.
For the purposes of my project, virtually everything is closed and blocked, as users don't need to be able to use the capabilities of the email-driven or web-driven interface to manage lists, as they are all done automatically anyway. The only time authorisation scenari are used is to control who can post to a list.
The people that can post to a list are defined in the list's config file by the paragraph:
/etc/sympa/scenari/send.<scenario-name>
There is also a zero-length file
/etc/sympa/scenari/send.<scenario-name>:ignore
which tells Sympa not to tell anyone about the name or details of the scenario in the web management interface (which we're not using anyway).
The scenario file looks like this:
No comments are allowed. The syntax is strict. Each rule contains a condition, a list of methods, "->" and an action. The "smtp" method is the one we're interested in, it takes the address from the "From:" line of the email message. Rule conditions are matched in order from top to bottom, and the first matching rule is used, at which point rule processing stops.
This returns true or false. If true and the method used is listed, the action happened. You can include single-quoted strings like 'uos-staff', aliases for the name of the list, address of the sender and so on, function calls, all sorts of stuff. The '[sender]' can also be a regexp like '/\@soton\.ac\.uk$/' to cause regexp matches instead of simple string comparison. You can also calls Perl functions, SQL queries, LDAP queries, all sorts of stuff. In a Perl condition, the name of the file is taken from "CustomCondition::<custom-filename>" and the file is located in
/etc/sympa/custom_conditions/<custom-filename>.pm
In this example the list of "Received:" headers is passed to it as a parameter. If it returns 1, the action is executed. If it returns 0, the action is not executed. If it returns undef, there was an error. Changes to the <custom-filename>.pm file only take effect if Sympa is restarted with "service sympa restart".
I have created a send.<scenario-name> file for each type of list we use. Sympa does not notice changes to the scenari unless the datestamp on the list's config file is updated, at which point it updates the scenari for that one list. If you change a scenario file, you have to update the datestamp of the config files for all the lists that use it, forcing on-the-fly recompilation of the lists' config files.
You can change every bit of text ever output by Sympa, and the whole thing is multilingual too. I have stuck with en_US.
All the text output by email to users is contained in the TT2 files in
/etc/sympa/mail_tt2
Changes to files in this directory are automatically noticed by Sympa and processed immediately. Note this does not apply to the file "sympa.po" below.
For example, the "<reason_code>" above, used as a parameter to the "reject()" action, is defined in
/etc/sympa/mail_tt2/authorization_reject.tt2
The best way of finding which file contains the string you are trying to change is to search for a small bit of it in all the mail_tt2/*.tt2 files. Beware the actual text output will not be in these files, but something very close to it will be if you are working in English. Continue reading to find out why...
Let's take the example of the <reason_code> being "send_building". The "reject()" output messages are all defined in the file mentioned above, which is basically a huge "if...then...elsif...elsif...else" statement with hundreds of conditional expressions defined to test the "reason" parameter it is passed by Sympa.
Look through that file and find the "send_building" "ELSIF" condition. That will produce this bit of code:
The actual text output to the user is taken by internationalising the string "send_building", which is passed to the "loc()" function which does the translation. The "loc()" function can have parameters passed to it which are used to insert various bits of text into the translated text. Most of the original text to be translated is *almost* exactly the same as the output of the English translation, but not quite. More fun that way round. A few greps will find the text you're looking to change.
The translation (into better English) is done using the file
/usr/share/locale/en_US/LC_MESSAGES/sympa.mo
This file is a compiled version of the file
/etc/sympa/sympa.po
You can compile it by running
cd /etc/sympa && msgfmt -c -o /usr/share/locale/en_US/LC_MESSAGES/sympa.mo sympa.po
In sympa.po there are multiple definitions which look like this:
service sympa restart
Changes will not take effect until Sympa restarts.
Many mailing lists email all their subscribers automatically at the start of each month telling them they are subscribed to that list, and how to unsubscribe and post successfully. The frequency of such actions are given in the list's config file by the paragraphs:
/etc/sympa/list_task_models/<action>.<task_model>.task
so "expire_task never" is described in
/etc/sympa/list_task_models/expire.never.task
In my project expiry and reminders are not needed nor wanted. So to stop them working, their frequencies have been set to 1000 years. The reason for this is fairly obvious if you look at the structure and syntax of one of the task model files.
The command (run as root) is
sympa.pl --create_list --input_file=<xml-file>
<xml-file> is a copy of /etc/sympa/sotontemplate.xml with tokens substituted in to set values which are inserted into the template config file. The <xml-file> can be deleted immediately after the list is created, it is not needed.
The <xml-file> looks like this (note the *1* and so on for footnotes):
<?xml version="1.0" ?>
<list>
<listname>comp3020</listname> *1*
<type>soton</type> *2*
<listtitle>comp3020</listtitle> *3*
<subject>Students enrolled on COMP3020</subject>
<editor>false</editor>
<replypolicy>other_email</replypolicy>
<replyaddress>god@ecs.soton.ac.uk</replyaddress>
<tag>true</tag>
<ismodule>true</ismodule>
<isbuilding>false</isbuilding>
<isuos>false</isuos>
<owner multiple="1"><email>jkf@ecs.soton.ac.uk</email></owner> *4*
</list>
Notes:
- Sets the name of the list. Must be set in the XML file, not the template config file.
- Sets the value of "type" which is used to determine the location of the template config file used to construct the list's own config file. Must be set in the XML file. Location is /etc/sympa/create_list_templates/<type>/config.tt2
- These all set lower-case-named variables which are referenced in the template config file just mentioned.
- This sets the owner(s) of the list. Must be set in the XML file, not the template config file.
- Any characters such as “&”, "<" and ">" must be encoded as XML entities “&", “<” and “>” or it errors out.
The XML file is meshed into the template config file in /etc/sympa/create_list_templates/<type>/config.tt2 and then interpreted as a TT2 template (see www.tt2.org), and the output goes into the list's own individual config file which is at
/var/lib/sympa/list_data/all.soton.ac.uk/<listname>/config
That file is compiled by Sympa automatically when needed. It can be edited manually at any time, and Sympa will automatically recompile it as and when needed, without you needing to tell it about the change.
Note that the syntax of the template config file config.tt2 is very strict. It is constructed of a series of "paragraphs" which are a bunch of lines, with a blank line terminating the paragraph. It does not support comments, blank lines are *very* important, and leading white space on a line breaks everything. You have been warned!
Editors / Moderators
"Editors" is what Sympa calls list moderators. When they are used depends on the setting of the "send" parameter in the list config file. See the section on "Authorisation Scenarios".
The editors of a list are not statically defined, but are pulled from a SQL database query as needed. The data is cached by Sympa for speed.
If the list has any editors, the editors are defined in the list's config file by a "paragraph" like this:
editor_include
source editors *1*
reception mail
visibility conceal
source_parameters fpas-ecs-sys *2*
Notes:
- The <source> name is used to tell Sympa where to get the SQL query that will produce a list of the email addresses of the editors. The SQL query is defined in a data source file at
/etc/sympa/data_sources/<source>.incl - The comma-separated list of source_parameters are passed into the SQL data source file as param.0, param.1 and so on.
The data source file looks like this:
include_sql_query
db_type mysql
host localhost
db_name all_lists
user secretusername!
passwd secretpassword!
sql_query SELECT email FROM list_moderators WHERE list_address='[% param.0 %]'
Note this file *does* handle leading white space on lines.
Most of this is obvious, and it can handle any <db_type> that is available in the Perl DBI modules.
The "source_parameters" from the list's config file are received as the "param" array, and the data source file is again a TT2 file (see www.tt2.org).
The default editor address for every list is “postmaster@ecs.soton.ac.uk".
Subscribers / Members
"Subscribers" are what Sympa calls list members. Only list subscribers receive a copy of postings to the list. So people can be able to post to a list without being able to receive a copy of their own posting. Posters who are subscribers always receive a copy of their own posting.
The subscriber list for a list is generated by an SQL query defined directly in the list's config file, it does not use a .incl file like the editors. The paragraphs defining the subscribers (taken from the template config file, not a list's real config file) look like this:
user_data_source include2 *1*
include_sql_queryname subscribers *2*
db_type mysql *3*
host localhost
db_name all_lists
user secretusername!
passwd secretpassword!
sql_query SELECT email FROM list_members WHERE list_address='[% listname %]' *4*
Notes:
- This tells Sympa what sort of data source it is. Only "include2" exists in Sympa now.
- Note this is for information only, and has no apparent effect.
- The database type can be anything supported by the Perl DBI modules.
- The [% listname %] is replaced with the name of the list when the list is created, it cannot be written indirectly like that in the list's own config file.
The list of subscribers for each list is cached by Sympa for a length of time defined by the /etc/sympa/sympa.conf setting "default_ttl" and is set to 3600 seconds (1 hour) at the time of writing. So all changes to the table of list subscribers in the "all_lists" database will take up to 1 hour to propagate to the lists. There is no way for force an update of a list's subscribers.
Authorisation Scenari (Who Can Post?)
For the purposes of my project, virtually everything is closed and blocked, as users don't need to be able to use the capabilities of the email-driven or web-driven interface to manage lists, as they are all done automatically anyway. The only time authorisation scenari are used is to control who can post to a list.
The people that can post to a list are defined in the list's config file by the paragraph:
send <scenario-name>
where <scenario-name> tells it to read the rules of the scenario from/etc/sympa/scenari/send.<scenario-name>
There is also a zero-length file
/etc/sympa/scenari/send.<scenario-name>:ignore
which tells Sympa not to tell anyone about the name or details of the scenario in the web management interface (which we're not using anyway).
The scenario file looks like this:
title.gettext Building list, Private, moderated for University staff
CustomCondition::not_from_campus([msg_header->Received]) smtp,dkim,smime,md5 -> reject(reason='from_offcampus')
is_subscriber([listname],[sender]) smtp,dkim,md5,smime -> do_it
is_editor([listname],[sender]) smtp,dkim,md5,smime -> do_it
is_subscriber('uos-staff',[sender]) smtp,dkim,md5,smime -> editorkey
true() smtp,dkim,smime,md5 -> reject(reason='send_building')
No comments are allowed. The syntax is strict. Each rule contains a condition, a list of methods, "->" and an action. The "smtp" method is the one we're interested in, it takes the address from the "From:" line of the email message. Rule conditions are matched in order from top to bottom, and the first matching rule is used, at which point rule processing stops.
Condition
This returns true or false. If true and the method used is listed, the action happened. You can include single-quoted strings like 'uos-staff', aliases for the name of the list, address of the sender and so on, function calls, all sorts of stuff. The '[sender]' can also be a regexp like '/\@soton\.ac\.uk$/' to cause regexp matches instead of simple string comparison. You can also calls Perl functions, SQL queries, LDAP queries, all sorts of stuff. In a Perl condition, the name of the file is taken from "CustomCondition::<custom-filename>" and the file is located in
/etc/sympa/custom_conditions/<custom-filename>.pm
In this example the list of "Received:" headers is passed to it as a parameter. If it returns 1, the action is executed. If it returns 0, the action is not executed. If it returns undef, there was an error. Changes to the <custom-filename>.pm file only take effect if Sympa is restarted with "service sympa restart".
Method
- "smtp" is the one we're interested in. This takes the [sender] from the "From:" line of the incoming email message.
Action
- "do_it" says the operation should be done (in this case, the message should be posted).
- "reject(reason='<reason_code>')" says the message should be rejected and a message sent to the sender explaining why. See the "Responses and Messages" section below for information about the <reason_code>.
- "editorkey" says the message should be saved locally and sent to all the editors of the list with an MD5 checksum. If the editor then sends Sympa the MD5 checksum in an email message telling it to proceed, the message will be posted. If the editor tells Sympa the message should be rejected, it is rejected (don't know what the sender is sent in this case). There are many other actions.
I have created a send.<scenario-name> file for each type of list we use. Sympa does not notice changes to the scenari unless the datestamp on the list's config file is updated, at which point it updates the scenari for that one list. If you change a scenario file, you have to update the datestamp of the config files for all the lists that use it, forcing on-the-fly recompilation of the lists' config files.
Responses and Messages
You can change every bit of text ever output by Sympa, and the whole thing is multilingual too. I have stuck with en_US.
All the text output by email to users is contained in the TT2 files in
/etc/sympa/mail_tt2
Changes to files in this directory are automatically noticed by Sympa and processed immediately. Note this does not apply to the file "sympa.po" below.
For example, the "<reason_code>" above, used as a parameter to the "reject()" action, is defined in
/etc/sympa/mail_tt2/authorization_reject.tt2
The best way of finding which file contains the string you are trying to change is to search for a small bit of it in all the mail_tt2/*.tt2 files. Beware the actual text output will not be in these files, but something very close to it will be if you are working in English. Continue reading to find out why...
Let's take the example of the <reason_code> being "send_building". The "reject()" output messages are all defined in the file mentioned above, which is basically a huge "if...then...elsif...elsif...else" statement with hundreds of conditional expressions defined to test the "reason" parameter it is passed by Sympa.
Look through that file and find the "send_building" "ELSIF" condition. That will produce this bit of code:
[% ELSIF reason == 'send_building' -%]
[%|loc()%]send_building[%END%]
The actual text output to the user is taken by internationalising the string "send_building", which is passed to the "loc()" function which does the translation. The "loc()" function can have parameters passed to it which are used to insert various bits of text into the translated text. Most of the original text to be translated is *almost* exactly the same as the output of the English translation, but not quite. More fun that way round. A few greps will find the text you're looking to change.
The translation (into better English) is done using the file
/usr/share/locale/en_US/LC_MESSAGES/sympa.mo
This file is a compiled version of the file
/etc/sympa/sympa.po
You can compile it by running
cd /etc/sympa && msgfmt -c -o /usr/share/locale/en_US/LC_MESSAGES/sympa.mo sympa.po
In sympa.po there are multiple definitions which look like this:
msgid "send_building"
msgstr "University Building lists are restricted to occupants of the building, and Estates and Facilities staff.\n\n"
"If you think you should be able to post to this list, use http://all.soton.ac.uk/search to search for yourself by staff/student number, and be sure you are sending from the correct address."
- msgid -- This is the text value passed to loc() in the file authorization_reject.tt2 described above.
- msgstr -- This line, and every line starting '"' immediately after it, define the actual text that is sent to the user.
service sympa restart
Changes will not take effect until Sympa restarts.
Regular Tasks
Many mailing lists email all their subscribers automatically at the start of each month telling them they are subscribed to that list, and how to unsubscribe and post successfully. The frequency of such actions are given in the list's config file by the paragraphs:
expire_task never
remind_task never
The frequency in the "<action>_task <task_model>" statement is described in the list task model file/etc/sympa/list_task_models/<action>.<task_model>.task
so "expire_task never" is described in
/etc/sympa/list_task_models/expire.never.task
In my project expiry and reminders are not needed nor wanted. So to stop them working, their frequencies have been set to 1000 years. The reason for this is fairly obvious if you look at the structure and syntax of one of the task model files.
Comments
Dropbox
13/05/10 22:21 Filed in: Linux
Update 24 May 2010: I have added some more features to Dropbox and I have released my new “Dropoff” over at www.dropoff.me. Get all the latest and greatest there in future!
I have found a great solution to the common problem of sending and receiving files from other sites and research partners, and generally sending large files around the web where email won’t do the trick.
It’s called “Dropbox” and was originally written by the University of Delaware.
I am launching it as a service at work called “Dropoff” in order that people don’t think you are talking about the service provided by www.dropbox.com which is a totally different thing.
The idea is that you don’t even need to login to send a file to a user within your site/company/University/institution, so external people can use it to send files to people in your institution. It can handle arbitrarily large files, there are no fixed limits. If you log in to the Dropbox website, you can send files to people outside your institution. People who cannot log in can only send files to people within your institution. This stops the rest of the world using it to send people to other people who aren’t members of your institution.
I have added various extra features to it:
I have customised it quite a bit just for our site, so if you want a copy of my patched version, along with a guide as to what changes I have made, then please contact me.
I have found a great solution to the common problem of sending and receiving files from other sites and research partners, and generally sending large files around the web where email won’t do the trick.
It’s called “Dropbox” and was originally written by the University of Delaware.
I am launching it as a service at work called “Dropoff” in order that people don’t think you are talking about the service provided by www.dropbox.com which is a totally different thing.
The idea is that you don’t even need to login to send a file to a user within your site/company/University/institution, so external people can use it to send files to people in your institution. It can handle arbitrarily large files, there are no fixed limits. If you log in to the Dropbox website, you can send files to people outside your institution. People who cannot log in can only send files to people within your institution. This stops the rest of the world using it to send people to other people who aren’t members of your institution.
I have added various extra features to it:
- Active Directory AD authentication (to multiple AD sites at once if needed)
- Virus scanning of uploaded files, using ClamAV
I have customised it quite a bit just for our site, so if you want a copy of my patched version, along with a guide as to what changes I have made, then please contact me.
Anti-Phishing and Spear-Phishing Version 2
Update 11 October 2009: This has now been moved to www.ScamNailer.com. Please check there for all future information and updates to this package.
Update 20 September 2009: The Google-hosted data file has been moved to SourceForge, so I have updated the URL it downloads it from. You need to update your script to the new version 2.05.
Update 16 June 2009: I have changed the rule structures to make them considerably faster than the old ones. Download the updated script from the link below.
I have acquired a new reliable feed of email addresses used in phishing attacks. These addresses have all been checked by real people, and they come from a very reliable and well-known source.
The new data file is provided by means of DNS and an Anycast network, which makes it pretty resilient to attack. The previous spear-phishing data is gathered from the project hosted by Google in the traditional way, that hasn’t changed.
I have updated my script so that it fetches both sets of data. It makes use of a temporary directory under /var/cache, which is configurable at the start of the script, and which needs to be writable by the user the scripts runs as (normally just ‘root’ so this doesn’t present any problem at all to most people).
You can download version 2.05 of the script.
If you are not using MailScanner with this script, you will need to comment out or delete the line that mentions “service MailScanner reload” about 1/3 of the way down the script (search and ye shall find!).
For more explanation of this whole problem and the way this script works, please refer back to my earlier article.
Update 20 September 2009: The Google-hosted data file has been moved to SourceForge, so I have updated the URL it downloads it from. You need to update your script to the new version 2.05.
Update 16 June 2009: I have changed the rule structures to make them considerably faster than the old ones. Download the updated script from the link below.
I have acquired a new reliable feed of email addresses used in phishing attacks. These addresses have all been checked by real people, and they come from a very reliable and well-known source.
The new data file is provided by means of DNS and an Anycast network, which makes it pretty resilient to attack. The previous spear-phishing data is gathered from the project hosted by Google in the traditional way, that hasn’t changed.
I have updated my script so that it fetches both sets of data. It makes use of a temporary directory under /var/cache, which is configurable at the start of the script, and which needs to be writable by the user the scripts runs as (normally just ‘root’ so this doesn’t present any problem at all to most people).
You can download version 2.05 of the script.
If you are not using MailScanner with this script, you will need to comment out or delete the line that mentions “service MailScanner reload” about 1/3 of the way down the script (search and ye shall find!).
For more explanation of this whole problem and the way this script works, please refer back to my earlier article.
Anti Spear Phishing
Update 2009-October-11: This package is now hosted at www.scamnailer.com. Please check there for all future information and updates.
Update 2009-June-15: There is now a brand new additional data feed of known phishing email addresses, which I have added to my script.
Spear phishing is a technique used by spammers and scammers to try to get your email username and password. They send you an email claiming to be from your email provider, in which they say that your account will be deleted unless you supply them with your username and password “for authentication” or some other similar ruse.
If they get your username and password, they then use your email account and email provider to send out millions of spam messages. Because the spam comes from a genuine email system (yours!) it will be accepted by most sites and will automatically pass many spam checks.
I have written a script which takes a file of addresses commonly used in these attacks. It also allows an additional list of addressed you can add to. From these, it generates a set of SpamAssassin rules that detect the presence of these addresses, which can be used in MailScanner to stop the spear-phishing attacks completely.
Download the script here. Note that the script is gzipped to ensure your browser doesn’t do anything silly when fetching it, so you’ll need to “gunzip” it before doing anything with it. To start with, just copy it into your “/etc/cron.hourly” directory, and run the command “chmod a+rx /etc/cron.hourly/Spear.Phishing.Rules” to make it run every hour.
It is pretty much a finished script, and is directly usable by you guys without you having to do much to it except read the settings at the top and tweak the filenames if you want to change where it puts things.
I have taken a lot of care to ensure that this won't match any false alarms, I don't just dumbly look for the strings in any surrounding text, which certain commercial AV vendors have been caught doing in the past!
I make a suggestion in the comments at the top of the script about how I use the rule within MailScanner, you probably want to do something similar, and not just delete anything that matches, just in case you do get any false alarms.
It also looks for numbers at the end of the username bit of the address, and assumes that these are numbers which the scammers may change; so if it finds them, it replaces them with a pattern that will match any number instead. There's starting to be a lot of this about, as it's the easiest way for the scammers to try to defeat simple address lists targeted against them, while still being able to remember what addresses they have to check for replies from your dumb users. I thought I would make it a tiny bit harder for them...
You can also add addresses of your own (which can include "*" as a wildcard character to mean "any series of valid characters" in the email address), one address per line, in an optional extra file. Again, read the top of the script and you'll see it mentioned there. That file is optional, it doesn't matter if it doesn't exist. As a starter, you might want to put
m i c h a e l l o u c a s * @ g m a i l . c o m
(without the extra spaces) in that file, as it will nicely catch a lot of "Job opportunity" spams.
It looks for any of these addresses appearing **anywhere** in the message, not just in the headers. So if you start talking to people about these addresses, don't be surprised when the messages get caught by the trap.
It does a "wget", so make sure you have that binary installed, or else change the script to fetch the file by some other means.
The very end of the script does a "service MailScanner restart", so if you need some other command to restart MailScanner or your SpamAssassin setup, then edit it for your system. It needs to be a "restart" and not a "reload" as I have to force it to re-build the database of SpamAssassin rules. If you don’t use MailScanner, but do use “spamd” in some setup or other, then a simple “service spamd restart” would do at the end of the script.
My aim was that, on a RedHat system running MailScanner, you could just copy the script into /etc/cron.hourly and make it executable, and it will just get on with the job for you. I do advise you read the bit in the script about "SpamAssassin Rule Actions" though.
Please do let me know how you would like me to improve it, and tell me what you think of it in general. (be polite, now!)
Update 13th January 2009:
A colleague on the MailScanner mailing list has made this simpler to use. You don’t have the flexibility of adding your own addresses to the list, but you can get the latest list along with all your regular SpamAssassin updates with the “sa-update” command.
Here are his instructions:
wget http://www.bastionmail.co.uk/spear.txt
sa-update --import spear.txt
Add “spear.bastionmail.com” to the list of channels that you update from (either add “--channel spear.bastionmail.com” to your sa-update command, or add “spear.bastionmail.com” to the file pointed to by the sa-update “--channelfile” command-line option).
Add the key “06EF70A3” to the trusted keys (either add “--gpgkey 06EF70A3” to your sa-update command, or add “06EF70A3” to the file pointed to by the sa-update “--gpgkeyfile” command-line option).
Then these SpamAssassin rules will be automatically updated every time your system runs the “sa-update” command, which is daily on a standard MailScanner system.
Update 2009-June-15: There is now a brand new additional data feed of known phishing email addresses, which I have added to my script.
Spear phishing is a technique used by spammers and scammers to try to get your email username and password. They send you an email claiming to be from your email provider, in which they say that your account will be deleted unless you supply them with your username and password “for authentication” or some other similar ruse.
If they get your username and password, they then use your email account and email provider to send out millions of spam messages. Because the spam comes from a genuine email system (yours!) it will be accepted by most sites and will automatically pass many spam checks.
I have written a script which takes a file of addresses commonly used in these attacks. It also allows an additional list of addressed you can add to. From these, it generates a set of SpamAssassin rules that detect the presence of these addresses, which can be used in MailScanner to stop the spear-phishing attacks completely.
Download the script here. Note that the script is gzipped to ensure your browser doesn’t do anything silly when fetching it, so you’ll need to “gunzip” it before doing anything with it. To start with, just copy it into your “/etc/cron.hourly” directory, and run the command “chmod a+rx /etc/cron.hourly/Spear.Phishing.Rules” to make it run every hour.
It is pretty much a finished script, and is directly usable by you guys without you having to do much to it except read the settings at the top and tweak the filenames if you want to change where it puts things.
I have taken a lot of care to ensure that this won't match any false alarms, I don't just dumbly look for the strings in any surrounding text, which certain commercial AV vendors have been caught doing in the past!
I make a suggestion in the comments at the top of the script about how I use the rule within MailScanner, you probably want to do something similar, and not just delete anything that matches, just in case you do get any false alarms.
It also looks for numbers at the end of the username bit of the address, and assumes that these are numbers which the scammers may change; so if it finds them, it replaces them with a pattern that will match any number instead. There's starting to be a lot of this about, as it's the easiest way for the scammers to try to defeat simple address lists targeted against them, while still being able to remember what addresses they have to check for replies from your dumb users.
I thought I would make it a tiny bit harder for them... You can also add addresses of your own (which can include "*" as a wildcard character to mean "any series of valid characters" in the email address), one address per line, in an optional extra file. Again, read the top of the script and you'll see it mentioned there. That file is optional, it doesn't matter if it doesn't exist. As a starter, you might want to put
m i c h a e l l o u c a s * @ g m a i l . c o m
(without the extra spaces) in that file, as it will nicely catch a lot of "Job opportunity" spams.
It looks for any of these addresses appearing **anywhere** in the message, not just in the headers. So if you start talking to people about these addresses, don't be surprised when the messages get caught by the trap.
It does a "wget", so make sure you have that binary installed, or else change the script to fetch the file by some other means.
The very end of the script does a "service MailScanner restart", so if you need some other command to restart MailScanner or your SpamAssassin setup, then edit it for your system. It needs to be a "restart" and not a "reload" as I have to force it to re-build the database of SpamAssassin rules. If you don’t use MailScanner, but do use “spamd” in some setup or other, then a simple “service spamd restart” would do at the end of the script.
My aim was that, on a RedHat system running MailScanner, you could just copy the script into /etc/cron.hourly and make it executable, and it will just get on with the job for you. I do advise you read the bit in the script about "SpamAssassin Rule Actions" though.
Please do let me know how you would like me to improve it, and tell me what you think of it in general. (be polite, now!)
Update 13th January 2009:
A colleague on the MailScanner mailing list has made this simpler to use. You don’t have the flexibility of adding your own addresses to the list, but you can get the latest list along with all your regular SpamAssassin updates with the “sa-update” command.
Here are his instructions:
wget http://www.bastionmail.co.uk/spear.txt
sa-update --import spear.txt
Add “spear.bastionmail.com” to the list of channels that you update from (either add “--channel spear.bastionmail.com” to your sa-update command, or add “spear.bastionmail.com” to the file pointed to by the sa-update “--channelfile” command-line option).
Add the key “06EF70A3” to the trusted keys (either add “--gpgkey 06EF70A3” to your sa-update command, or add “06EF70A3” to the file pointed to by the sa-update “--gpgkeyfile” command-line option).
Then these SpamAssassin rules will be automatically updated every time your system runs the “sa-update” command, which is daily on a standard MailScanner system.
Talking from sendmail to Exchange over SMTP auth
05/06/08 09:42 Filed in: Exchange Server 2007 | Email
There are various things you can do in Exchange, such as control who can address distribution lists, that can be restricted to authenticated senders only. So how do you make your sendmail box an authenticated sender?
Start at http://www.sendmail.org/~ca/email/auth.html -- about half way down it starts talking about “Using sendmail as a client with AUTH”. That tells you how to setup your sendmail box (which is the client) so that it talks SMTP auth to Exchange (which is the server).
Start at http://www.sendmail.org/~ca/email/auth.html -- about half way down it starts talking about “Using sendmail as a client with AUTH”. That tells you how to setup your sendmail box (which is the client) so that it talks SMTP auth to Exchange (which is the server).
Building an MX the Easy Way
Boot off RHEL5 disk 1.
linux rescue
Activate network interface eth0 (first interface).
Give IP and so on.
Use fdisk to create
/dev/sda1 /boot Linux 100Mb
/dev/sda2 Linux swap 2048Mb
/dev/sda3 / Linux all the rest
mkfs.ext3 /dev/sda1
mkfs.ext3 /dev/sda3
mkswap /dev/sda2
mkdir /mnt2
mount /dev/sda1 /mnt2
ssh crow ‘dump 0f - /dev/sda3’ | ( cd /mnt2 && restore -rf - )
umount /mnt2
mount /dev/sda3 /mnt2
ssh crow.ecs.soton.ac.uk ‘dump 0f - /dev/mapper/VolGroup00-LogVol00’ | ( cd /mnt2 && restore -rf - )
or else
ssh crow.ecs.soton.ac.uk ‘cd / && tar clf - .’ | ( cd /mnt2 && tar xvBpf - )
Fix /mnt2/etc/fstab so it points to all the right partitions.
If you really want to use partition labels, use the “e2label” command to set the label of each partition so that your shiny new /etc/fstab can find them. Syntax is obvious: /sbin/e2label device [ new-label ]
umount /mnt2
Installing Grub
Cloning an RHEL4 system using the RHEL5 rescue disc? You must use Grub from RHEL4 to setup a boot record for an RHEL4 system. So mount the (newly copied) root filesystem in /mnt2 and copy /mnt2/sbin/grub to /sbin/grub. Then follow the instructions below.
mkdir /boot
mount /dev/sda1 /boot
rm -rf /boot/boot
grub
grub> root (hd0,0)
grub> setup (hd0)
Edit /boot/grub/grub.conf and change the root command to (hd0,0) and the kernel root-filesystem argument to /dev/sda3.
Repeat that edit for all the other kernels available.
Unplug network interface
Reboot and it should boot from hard disk
cd /var/spool/mqueue.in
rm -f *
cd /var/spool/mqueue
rm -f *
cd /var/spool/MailScanner/quarantine
rm -rf *
cd ../incoming
rm -rf *
cd ../archive
rm -rf *
cd /var/log
Remove all old logs
service syslog restart
Fix ethernet and IP address in /etc/sysconfig/network-scripts/ifcfg-eth0 and /etc/sysconfig/network
Fix /etc/hosts
Repair ownership and permissions of /home/* and /usr/local/share/clamav (and subdirectories).
Fix extra ClamAV databases so that “MailScanner --lint” runs correctly.
Reboot with network interface connected.
Re-register with RedHat network for yum updates, get the info from the Systems KB.
yum update
linux rescue
Activate network interface eth0 (first interface).
Give IP and so on.
Use fdisk to create
/dev/sda1 /boot Linux 100Mb
/dev/sda2 Linux swap 2048Mb
/dev/sda3 / Linux all the rest
mkfs.ext3 /dev/sda1
mkfs.ext3 /dev/sda3
mkswap /dev/sda2
mkdir /mnt2
mount /dev/sda1 /mnt2
ssh crow ‘dump 0f - /dev/sda3’ | ( cd /mnt2 && restore -rf - )
umount /mnt2
mount /dev/sda3 /mnt2
ssh crow.ecs.soton.ac.uk ‘dump 0f - /dev/mapper/VolGroup00-LogVol00’ | ( cd /mnt2 && restore -rf - )
or else
ssh crow.ecs.soton.ac.uk ‘cd / && tar clf - .’ | ( cd /mnt2 && tar xvBpf - )
Fix /mnt2/etc/fstab so it points to all the right partitions.
If you really want to use partition labels, use the “e2label” command to set the label of each partition so that your shiny new /etc/fstab can find them. Syntax is obvious: /sbin/e2label device [ new-label ]
umount /mnt2
Installing Grub
Cloning an RHEL4 system using the RHEL5 rescue disc? You must use Grub from RHEL4 to setup a boot record for an RHEL4 system. So mount the (newly copied) root filesystem in /mnt2 and copy /mnt2/sbin/grub to /sbin/grub. Then follow the instructions below.
mkdir /boot
mount /dev/sda1 /boot
rm -rf /boot/boot
grub
grub> root (hd0,0)
grub> setup (hd0)
Edit /boot/grub/grub.conf and change the root command to (hd0,0) and the kernel root-filesystem argument to /dev/sda3.
Repeat that edit for all the other kernels available.
Unplug network interface
Reboot and it should boot from hard disk
cd /var/spool/mqueue.in
rm -f *
cd /var/spool/mqueue
rm -f *
cd /var/spool/MailScanner/quarantine
rm -rf *
cd ../incoming
rm -rf *
cd ../archive
rm -rf *
cd /var/log
Remove all old logs
service syslog restart
Fix ethernet and IP address in /etc/sysconfig/network-scripts/ifcfg-eth0 and /etc/sysconfig/network
Fix /etc/hosts
Repair ownership and permissions of /home/* and /usr/local/share/clamav (and subdirectories).
Fix extra ClamAV databases so that “MailScanner --lint” runs correctly.
Reboot with network interface connected.
Re-register with RedHat network for yum updates, get the info from the Systems KB.
yum update