{"id":6555,"date":"2016-06-10T20:08:12","date_gmt":"2016-06-10T18:08:12","guid":{"rendered":"http:\/\/www.b.shuttle.de\/hayek\/hayek\/jochen\/wp\/blog-en\/?p=6555"},"modified":"2019-05-29T16:16:30","modified_gmt":"2019-05-29T14:16:30","slug":"how-spf-resp-srs-broke-my-mail-filtering-and-cost-me-a-lot-of-sleep-recently","status":"publish","type":"post","link":"https:\/\/wp.jochen.hayek.name\/blog-en\/2016\/06\/10\/how-spf-resp-srs-broke-my-mail-filtering-and-cost-me-a-lot-of-sleep-recently\/","title":{"rendered":"how SPF resp. SRS broke my mail filtering and cost me a lot of sleep recently"},"content":{"rendered":"\n<ul class=\"wp-block-list\"><li><a href=\"https:\/\/en.wikipedia.org\/wiki\/Sender_Policy_Framework\">https:\/\/en.wikipedia.org\/wiki\/Sender_Policy_Framework<\/a>\u00a0= SPF<\/li><li><a href=\"https:\/\/en.wikipedia.org\/wiki\/Sender_Rewriting_Scheme\">https:\/\/en.wikipedia.org\/wiki\/Sender_Rewriting_Scheme<\/a>\u00a0= SRS<\/li><li><a data-wplink-edit=\"true\" href=\"_wp_link_placeholder\">https:\/\/www.getmailbird.com\/what-spf-resources-are-available-now-that-openspf-org-is-gone\/<\/a><\/li><li><a href=\"http:\/\/www.openspf.org\/SRS\">http:\/\/www.openspf.org\/SRS<\/a> (unfunctional starting 2019-02)<\/li><li><a href=\"http:\/\/www.heise.de\/newsticker\/meldung\/United-Internet-verschaerft-Spam-Bekaempfung-3225281.html\">http:\/\/www.heise.de\/newsticker\/meldung\/United-Internet-verschaerft-Spam-Bekaempfung-3225281.html<\/a>\u00a0(in German) \u2013 the action that triggered my trouble<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">I am using e-mail addresses on my &#8220;personal domains&#8221; (like <a href=\"http:\/\/Hayek.name\">Hayek.name<\/a>), which implies some sort of forwarding to an IMAP server.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The company, that hosts my &#8220;personal domains&#8221;, (UDAG) also forwards massive amount of e-mail messages to some companies that dominate the German market (like GMX and Web.de). The latter ones recently introduced SRS (the obligatory variant!), which in turned caused a lot of e-mail bouncing to UDAG. Apparently UDAG was a little surprised by that, and they in turn also implemented SRS w\/o announcements noticeable by their customers.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">How did that lightning hit me? (Almost) all of my incoming e-mail messages did not get matched appropriately by my mail filtering any more, and because SPAM Assassin classified them as more or less bad SPAM, all these messages went to SPAM folders.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">What kind of e-mail was concerned?<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>recruiting offers<\/li><li>banking events<\/li><li>telephone calls (resp. their notifications) to my home phone numbers<\/li><li>various mailing lists<\/li><li>\u2026<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">I can tell you, it was a big, big mess. A lot of messages in the wrong IMAP folders. That&#8217;s a true plague.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">My preferred approach to fix the trouble was to get SRS rewriting switched off (for me). UDAG frankly denied that. (I guess, I am just not important at all.) (Further down I will tell you, how I sort of continued that path anyway.)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Why did it hit me in the 1st place? Because I query &#8220;<em>Return-Path<\/em>&#8221; (which is the main focus of SRS) and not &#8220;<em>From<\/em>&#8220;. In 20 years of procmail rule writing I learned, that querying <em>Return-Path<\/em> is far more reliable than querying <em>From<\/em>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I talked to a couple of support guys, and what did they suggest?<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>I should query <em>From<\/em>, apparently that&#8217;s their general approach of mail filtering. I will not do that, <em>From<\/em> is not reliable.<\/li><li>I should rewrite my filtering rules obeying to SRS. You don&#8217;t rewrite your procmail regular expressions in an SRS way. Not if you have more than a hundred lines of code \u2013 or 20.000 LOC as I do.<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">I also got told, that procmail would be replaced by Sieve sooner or later in my IMAP server&#8217;s environment, so my traditional procmail filtering would not work any longer (then) anyway. WOW! More bad news please!<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">What to do?<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>rewrite the procmail rules by using a very simple DSL-like approach<\/li><li>create the original procmail rules from that DSL<\/li><li>and also create procmail rules with regular expressions considering the SRS-mangled e-mail addresses \u2013 but only SRS Level 0 \u2013 you might not be aware of this, but SRS address mangling can come in multiple levels<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">I focused on the rules, that <strong>only<\/strong> query <em>Return-Path<\/em>. I assume, I am covering more than 80% of my code that way.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Another 5% of my code queries <em>List-ID<\/em> and <em>List-Unsubscribe<\/em>. They are really, really reliable and also very stable. I will create a DSL feature, that expresses their use independent of procmail. Later! Not now! But I guess rather, rather soon \u2013 because it looks so intriguing.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I had started writing procmail rules 20 years ago, and they ran on my notebook then for a couple of years. But once I started created duplicates (with slight changes) that would run on &#8220;my IMAP server out there&#8221;. That was smart but also silly. <em>Duplicates with changes<\/em> are always a PITA. I should have created that DSL approach from the beginning \u2013 and I actually did \u2013 but in a way, that was just not simple enough and too painful to carry through. I got stuck with that 15 years ago. I always knew, that procmail would die in &#8220;public environments&#8221; one day, and I would either have to go with Sieve or do some own mail filtering around IMAP. I started the latter one, and I mentioned that in conversations with ESR in the context of <em>fetchmail<\/em>, and he made that public, and (shame on me!) I never delivered. And you can still find &#8220;the shame&#8221; on some websites.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Rewriting my procmail rules in that DSL way allows me to merge those duplicates again \u2013 and I rather enjoyed that, but it was a lot of work \u2013 a true lot of work. With procmail you have to write the addresses you want to match as regular expressions. Not a big thing, not difficult at all. If somebody has a couple of addresses, and you want to match them all through the same procmail rule, you can also do that using a smarter regular expression. Not a big thing, too. With my DSL I added a feature, that allows listing plain addresses as plain addresses. So for quite some rules I rewrote the regular expressions to a simple list of plain addresses. These rules look far more readable now.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The current code generator of my DSL I am just targeting procmail. But I was keeping in mind from the very beginning, that targeting Sieve or something different should be quite easy to achieve.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Another solution path, that I envisaged after talking to various support staff, is to get the incoming message rewritten back to a state, where Return-Path looks, as it has always looked. I implemented a 20-lines Perl script and a test set-up with a couple of sample messages, and that looked very promising. The communication with the resp. support \/ sysadmin staff took dragged on (???) for quite a couple of days, finally they told me<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>they can&#8217;t do that for me<\/li><li>but I can actually do it myself.<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Really? Yes, procmail has a feature, that allows it to rewrite its input and process the rewritten input than. Phantastic!&nbsp; \ud83d\ude06 Looks like this voids the DSL work I had done before, but actually it does not. Once the incoming messages will look again, as they always have, I can remove the generated rules coping with SRS Level 0 (un)mangling. My new DSL code looks&nbsp;far, far better than my previous code.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I spent like 30 extra hours during the last ten days on this effort, and I think, the work accomplished was rather worth it.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>https:\/\/en.wikipedia.org\/wiki\/Sender_Policy_Framework\u00a0= SPF https:\/\/en.wikipedia.org\/wiki\/Sender_Rewriting_Scheme\u00a0= SRS https:\/\/www.getmailbird.com\/what-spf-resources-are-available-now-that-openspf-org-is-gone\/ http:\/\/www.openspf.org\/SRS (unfunctional starting 2019-02) http:\/\/www.heise.de\/newsticker\/meldung\/United-Internet-verschaerft-Spam-Bekaempfung-3225281.html\u00a0(in German) \u2013 the action that triggered my trouble I am using e-mail addresses on my &#8220;personal domains&#8221; (like Hayek.name), which implies some sort of forwarding to an IMAP server. The company, that hosts my &#8220;personal domains&#8221;, (UDAG) also forwards massive amount of e-mail messages to [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_crdt_document":"","jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"_share_on_mastodon":"0"},"categories":[465],"tags":[1297],"class_list":["post-6555","post","type-post","status-publish","format-standard","hentry","category-procmail","tag-sieve"],"share_on_mastodon":{"url":"","error":""},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/paO0kP-1HJ","jetpack_likes_enabled":true,"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/posts\/6555","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/comments?post=6555"}],"version-history":[{"count":2,"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/posts\/6555\/revisions"}],"predecessor-version":[{"id":10616,"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/posts\/6555\/revisions\/10616"}],"wp:attachment":[{"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/media?parent=6555"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/categories?post=6555"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/tags?post=6555"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}