{"id":475,"date":"2012-07-09T01:43:32","date_gmt":"2012-07-08T23:43:32","guid":{"rendered":"https:\/\/www.npcglib.org\/~stathis\/blog\/?p=475"},"modified":"2013-09-18T04:58:04","modified_gmt":"2013-09-18T02:58:04","slug":"linux-task-sorting-mail-with-imapfilter","status":"publish","type":"post","link":"https:\/\/www.npcglib.org\/~stathis\/blog\/2012\/07\/09\/linux-task-sorting-mail-with-imapfilter\/","title":{"rendered":"linux task: sorting mail with imapfilter"},"content":{"rendered":"

Marking the 10th birthday of my e-mail server I decided it is time to share with the world my simple solution to sorting the massive amounts of incoming mail, including spam. Over the years I have accumulated a good number of different email accounts that are difficult to track, thus I have setup a central IMAP server where I fetch all mail into and read exclusively what I choose to. I use several methods to fetch and sort mail, but maybe the most useful utility for sorting mail is imapfilter<\/a>. imapfilter is written in Lua<\/a> and relies on configuration files written in Lua to process emails. I’m not a Lua expert, but I managed to setup a simple and scalable method for filtering my mail.<\/p>\n

I use three types of filter tables that I can add filters to as I see it fit:<\/p>\n

    \n
  1. pre_filters<\/strong>: filters applied before any other sorting rules. These rules are great for getting rid of abusive mail, spam, or ban any hosts from filling your mailbox.<\/li>\n
  2. filters<\/strong>: filters that do the bulk of message processing and sorting.<\/li>\n
  3. post_filters<\/strong>: filters applied after all message processing has been completed. This is useful if for instance you decide to get rid of any messages that are marked as spam, but they were not sorted due to any of the previous rules (so they are most likely spam). This also allows you to place messages that matched no rules to special folders, etc.<\/li>\n<\/ol>\n

    Filter tables are simple Lua tables, of which every filter is another table that contains the details of that particular filter:<\/p>\n

    \r\nfilter_table_name = {\r\n  { filter1 },\r\n  { filter2 },\r\n}\r\n<\/pre>\n
    Individual filters are tables with index keys (header<\/em>, p<\/em> and moveto<\/em>):<\/p>\n
    \r\nfilter_table_name = {\r\n  { header<\/em> = \"Header_String\" , p<\/em> = \"Pattern_String\", moveto<\/em> = account['folder'] },\r\n  { header<\/em> = \"Other_Header_String\" , p<\/em> = \"Other_Pattern_String\", moveto<\/em> = account['folder'] },\r\n}\r\n<\/pre>\n
    In the order they appear, filters are applied on a table of email messages, e.g. those messages present in the INBOX of an IMAP account. To apply these filters, I wrote a very simple Lua function (see below parseRules<\/em>) that takes a table of email messages and a table of filters to apply to them. Bear in mind, that to avoid losing any email, I typically just sort unwanted email in the Trash or a Junk folder that I clean up later, either manually or at certain intervals with imapfilter functionality (not shown here).<\/p>\n
    Putting altogether in an imapfilter configuration file (e.g. ~\/.imapfilter\/config.lua) it should look similar to the following:<\/p>\n
    \r\n----------------------\r\n--  Global Options  --\r\n----------------------\r\n\r\noptions.timeout = 120\r\noptions.subscribe = true\r\n\r\n----------------\r\n--  Accounts  --\r\n----------------\r\n-- For every account create an account entry and check() its status\r\n\r\nacc1 = IMAP {\r\n        server = 'collector.server.com',\r\n        username = 'stathis-user',\r\n        password = 'secret',\r\n        ssl = 'ssl3',\r\n}\r\n\r\nacc1.INBOX:check_status()\r\n\r\n----------------\r\n\r\nacc2 = IMAP {\r\n        server = 'imap.gmail.com',\r\n        username = 'stathis-user',\r\n        password = 'secret',\r\n        ssl = 'ssl3',\r\n}\r\n\r\nacc2.INBOX:check_status()\r\n\r\n----------------\r\n\r\nacc3 = IMAP {\r\n        server = 'imap.work.com',\r\n        username = 'stathis-user',\r\n        password = 'secret',\r\n        ssl = 'ssl3',\r\n}\r\n\r\nacc3.INBOX:check_status()\r\n\r\n----------------\r\n\r\n-- select all messages from all INBOXes and construct a single table of messages\r\nresults = acc1.INBOX:select_all() + acc2.INBOX:select_all() + acc3.INBOX:select_all()\r\n\r\n-----------------\r\n--  Functions  --\r\n-----------------\r\n-- parseRules is used to filter message results using a table of rules (see below)\r\n\r\n-- @param res         the table of messages to filter\r\n-- @param ruleTable   the table of rules to match messages against\r\nparseRules = function ( res, ruleTable )\r\nlocal subresults = {}\r\n  for _,entry in pairs(ruleTable) do\r\n    -- don't use match_field, it downloads part of the msg (slow)\r\n    subresults = res:contain_field(entry[\"header\"], entry[\"p\"])\r\n    if subresults:move_messages( entry[\"moveto\"] ) == false then\r\n      print(\"Cannot move messages!\")\r\n    end\r\n  end\r\nend\r\n\r\n---------------\r\n--  Filters  --\r\n---------------\r\n\r\npre_filters = {\r\n  -- Banned IPs typically sending me spam\r\n  { header = \"Received\" , p = \"85.31.186.26\", moveto = acc1['Trash'] },\r\n  { header = \"Received\" , p = \"176.251.35.146\", moveto = acc1['Trash'] },\r\n}\r\n\r\nfilters = {\r\n  \r\n  -- Google Alerts\r\n  { header = \"From\", p = \"googlealerts@google.com\", moveto = acc1['Misc\/Google_Alerts'] },\r\n  \r\n  -- Intel\r\n  { header = \"From\", p = \"isd@intel-dispatch.com\", moveto = acc1['Misc\/Intel'] },\r\n  { header = \"From\", p = \"@softwaredispatch.intel.com\", moveto = acc1['Misc\/Intel'] },\r\n  { header = \"From\", p = \"@software-dispatch.intel.com\", moveto = acc1['Misc\/Intel'] },\r\n  \r\n  -- Boost\r\n  { header = \"Sender\", p = \"@lists.boost.org\", moveto = acc1['Mailing_Lists\/Boost'] },\r\n  { header = \"To\", p = \"users@lists.boost.org\", moveto = acc1['Mailing_Lists\/Boost'] },\r\n  \r\n  -- OpenCV\r\n  { header = \"Subject\", p = \"[OpenCV]\", moveto = acc1['Mailing_Lists\/OpenCV'] },\r\n  { header = \"Subject\", p = \"[OpenCV - Bug #\", moveto = acc1['Misc\/Bugs\/OpenCV'] },\r\n  \r\n}\r\n\r\npost_filters = {\r\n  \r\n  -- Any remaining mail having in the subject SPAM? goes in Junk\r\n  { header = \"Subject\" , p = \"[SPAM?] \", moveto = acc1['Junk'] },\r\n  \r\n}\r\n\r\n---------------------\r\n--  Parse messages --\r\n---------------------\r\n\r\n-- pre filter messages\r\nparseRules(results, pre_filters)\r\n\r\n-- sort messages\r\nparseRules(results, filters)\r\n\r\n-- post filter messages\r\nparseRules(results, post_filters)\r\n\r\n----------------\r\n<\/pre>\n\n
    Software used<\/h3>\n