SPF test with ISP DHCP ranges - ORF Forums

SPF test with ISP DHCP ranges RSS Back to forum

1

Hello.
i recently had some spam from IPs relayed from DHCP ranges of ISPs:

-- EVENT SUMMARY --
Time: 24.02.2014 08:18:06 GMT+0100 W. Europe Standard Time
Sender Email:
Recipient Email: [email protected]***
Related IP: 188.79.210.239
Action: (not available)
Email Subject: (not available)

The IP originates from jazztel.es DHCP range and is not yet blacklisted. It also has a forged:

-- EVENT MESSAGE --
Error checking the SPF policy of domain "jazztel.es": The requested A/MX record was not found for "_spf.jazztel.es".

-- EVENT DETAILS --
Filtering Point: Before Arrival
Event Class: System Message
Severity: Warning
Server: ****
Event Source: MSEXCHANGE
HELO Domain: 170.247.21.95.dynamic.jazztel.es
Message ID: (not available)
Log Mode: Verbose
ORF Version: 5.1 REGISTERED

also, the SPF trom this domain dot explicitly allow this IP:
; > DiG 9.9.4-P2-RedHat-9.9.4-11.P2.fc20 > _spf.jazztel.es any
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37328
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;_spf.jazztel.es. IN ANY

;; ANSWER SECTION:
_spf.jazztel.es. 18866 IN TXT "v=spf1 mx ip4:62.14.3.170/30 ip4:62.14.3.174 ip4:62.14.3.191 ip4:62.14.3.192 ip4:62.14.3.193 ip4:62.14.3.194 ip4:62.14.3.195 ip4:62.14.3.196 ip4:62.14.3.197 ip4:62.14.3.198 ip4:212.106.192.69 ip4:212.106.192.72 ?all"

;; Query time: 31 msec
;; SERVER: 172.16.21.1#53(172.16.21.1)
;; WHEN: Mo Feb 24 12:45:38 CET 2014
;; MSG SIZE rcvd: 272


In the end, I would like to mark all messages with this warning to be SPAM, since I have yet to encounter a valid source with SPF SOFTFAIL.

Is this possible?
Thanks!
Daniel

by daniel.helgenberger 5 years ago
2

@daniel.helgenberger: Hello,

The 188.79.210.239 IP address is listed on both Spamcop and CBL -- you may want to give a try to these DNSBLs.

As for the SPF policy, I'm afraid jazztel.es has a flawed policy and that's why the SPF check fails. The SPF policy for jazztel.es is:

v=spf1 redirect=_spf.jazztel.es

This instructs the SPF client in ORF to evaluate the policy of _spf.jazztel.es instead. It looks like this:

v=spf1 mx ip4:62.14.3.170/30 ip4:62.14.3.174 ip4:62.14.3.191 ip4:62.14.3.192 ip4:62.14.3.193 ip4:62.14.3.194 ip4:62.14.3.195 ip4:62.14.3.196 ip4:62.14.3.197 ip4:62.14.3.198 ip4:212.106.192.69 ip4:212.106.192.72 ?all

It starts with an "MX" mechanism right away. When "MX" stands alone with no arguments, it means the mechanism will match if the MX record of the _SPF domain being tested_ contains the IP address being verified.

The issue is with the "SPF domain being tested" part here, because the "redirect" modifier from the original policy changes that to _spf.jazztel.es. Due to this, ORF will attempt to look up the MX record of _spf.jazztel.es, which does not exist and in turn this causes the SPF evaluation to fail.

by Péter Karsai (Vamsoft) 5 years ago
(in reply to this post)

3

@Péter Karsai (Vamsoft): Hello Péter,

thanks for your quick reply! I already have Spamcop enabled, but now also enabled CBL. True, they are now listed. But these spammers use other IPs (dynamic range) now so the listings are obsolete again.

I now it is not ORF's fault jazztel.es is using a flawd policy. I was asking if there is a way to blacklist those flawed polices.

SPF is really great, but IMHO things are a little bit unclear or can be improved in ORF. Please correct me if I am wrong:
Have the test BEFORE ARRIVAL and ON ARRIVAL behave differently (i know this might be hard to do by design?). Specifically I can imagine:
- Before Arrival: Blacklist HARDFAIL
- On Arrival: Blacklist failed policies (like the one above), SOFTFAIL

As I said, if you have a failing policy on domains which really handle valid traffic this policy will be sorted out soon. Like in the case above, it most likely never uses the redirect and therefore does only get applied in cases like spam.

by daniel.helgenberger 5 years ago
(in reply to this post)

4

@daniel.helgenberger: Hi Daniel,

Some of the DNSBLs (for instance, Spamhaus PBL, part of Spamhaus ZEN) blacklist dynamic IP addresses if they were specifically declared by their owners (i.e. the ISP) as unauthorized for direct email delivery. Unfortunately, not all ISPs make such declaration, out of ignorance or sometimes with malicious intent -- there're are tons of spammer-friendly ISPs out there. I've seen some of our clients experimenting with PTR blacklists for dynamic-looking reverse names, but I suppose with varying luck (this be something like enabling the Sender IP Reverse Name validation under Reverse DNS in ORF, then setting up a name pattern regex that matches anything that looks like an IP address, followed by the word "dynamic", "dyna", etc.). I think the link is too weak here and paying attention to other tests in ORF (specifically, the content-based tests like SURBLs) may result in better gains with less false positives.

I am not sure if I understand your proposal about SPF -- can you please elaborate? In any case, you should know that an error during the SPF evaluation will not cause the email to be whitelisted or blacklisted. All ORF does is to give up the SPF test in particular and proceed with the next test. In this sense, a flawed SPF policy like that of jazztel.es is neither a good thing, nor a good one -- ORF wastes a little time trying to evaluate the policy, but that's all.

Also, the SPF standard is very clear about how errors must be treated and with a few exceptions, the SPF evaluation must stop with an error (TempError or PermError). This is a sensible approach, because the SPF client (ORF) cannot make any reliable decision in this scenario. Judging from the amount of flawed SPF policies we see on a daily basis I'd suspect there are SPF clients are more allowing in this regard, but I don't think they are right about that: any attempt to second-guessing the publisher's intentions will inevitably fail due to lack of information.

by Péter Karsai (Vamsoft) 5 years ago
(in reply to this post)

5

@Péter Karsai (Vamsoft): Hello Peter,

thanks for clarifying how the dynamic ISP ranges are blacklisted - I always wonded.

I hope you don't get me wrong, ORF is a very good product and since using it we barely get any spam. I think you really took care to implement every feature like the SPF test by the book - RFC in this case. (Maybe it works too good, I moved most of the tests before arrival, see below).

And I am happy to elaborate more; maybe I did get something wrong in the end. I can have a test before and on arrival for most tests. I can enable it also on both.
But one question: If a test is positive before arrival and the mail gets rejected, how and when is this test applied on arrival? Why can I enable it on both? Can you supply an example where this makes sense?

I propose to give the user more granularity on the test and let him choose where to apply a specific feature. This means having effectively two sets of configurations for each test.
Two examples, one is the SPF test, the other the reverse DNS test you mentioned:
- I do most of the testing which cleary is spam before arrival. This more effective for the system; as we have about 60%-75% Spam rate and I do not my the users clicking through phishing mails in their spam;
- For SPF, if the test is positive the mail is Spam, nearly 99.9%. But, I like to turn on the SOFTFAIL feature to. I did this in December, when there is the most spam (out of experience). But for this feature I would like to move the flag and move the mails effectively applying the test on arrival.
How can I achive this?

The same is true for the Reverse DNS test. I want to apply the reverse A/MX before arrival but the regex match on arrival.

Thanks for taking the time,
Daniel

by daniel.helgenberger 5 years ago
(in reply to this post)

6

@daniel.helgenberger: Hello Daniel,

No problem, I was just trying to give some insight into why ORF does things in a particular way :)

"If a test is positive before arrival and the mail gets rejected, how and when is this test applied on arrival?"

Once the email is rejected at Before Arrival, On Arrival tests will not run on that email, because the delivery stops right at Before Arrival. Although there are exceptions to this (see http://vamsoft.com/support/docs/orf-help/5.2/fltrpointconcept to see how that is possible), it has limited relevance. Otherwise, the two filtering points are entirely independent from each other.

"Why can I enable it on both? Can you supply an example where this makes sense?"

Sure, they come handy in a number of situations, but the most typical is when you have a secondary MX. In this scenario, some email is received directly by your email server (running ORF), but some emails will be routed through the secondary MX. This setup requires adding the secondary MX IP to the Intermediate Host List of ORF [1]. However, emails from Intermediate Hosts get whitelisted automatically at Before Arrival [2]. If you assign all tests to Before Arrival, emails arriving via your secondary will bypass all checks. If you assign all tests to On Arrival, that's fine, but then some things like Recipient Validation, etc. are better done at Before Arrival. When assigning tests to both filtering points, you can give some email the Before Arrival treatment and the rest the On Arrival one.

[1] Many ORF tests work with the original source IP of the email. For instance, the SPF test evaluates the original source IP against the SPF policy of the sender. Running the SPF test against your secondary MX IP would cause a false positive, but the Intermediate Host List tells ORF not to do that.

[2] Some of those IP-based tests mentioned in [1] are whitelists (e.g. the IP Whitelist). ORF rather opts to whitelist the email at Before Arrival just to make sure it can enforce your IP Whitelist at On Arrival.

***

From what I have gathered, you would need more granularity with actions, is that correct? For instance, you would trust some tests more and let them reject the email. Less trusted tests would get to e.g. tag the email only. In a sense, Before Arrival and On Arrival definitely enables that, but you could achieve the same by switching most/all tests to On Arrival and using a more granular action control than currently available. If that is the case, please vote the feature requests http://vamsoft.com/support/feature-requests/per-test-actions and http://vamsoft.com/support/feature-requests/per-item-actions -- it's the best way to add your voice.

by Péter Karsai (Vamsoft) 5 years ago
(in reply to this post)

7

Hello Péter,

I think the per-test-action feature is exactly what I need. Thanks pointing out the feature request section to me!
But one question to my original problem:
Would it make sense to extend the SPF check by a feature like "Unresolvable SPF Policy". Right now, I only have this with spam (see above) and my options are limited to HARD and SOFT fail. As I pointed out before, nearly all of the sender domains I encounter employ a correct policy if they have one.Right now, if it does not check out (syntactically incorrect for instance) the SPF check will be skipped altogether with a warning. I would like to flag it as spam for my part.

Also, the configuration with both on arrival / before arrival makes more sense to me now. But, I think such a setup would be less desirable; in your best practices you already point out to deploy ORF on all smart hosts (this way I have it deployed here).

Thank you for your time,
Daniel

by daniel.helgenberger 5 years ago
8

@daniel.helgenberger: "Would it make sense to extend the SPF check by a feature like "Unresolvable SPF Policy"."

No: as Peter pointed out, blacklisting emails whenever an error occurs (e.g., due to faulty SPF records) is a very bad idea and would definitely cause a high number of false hits. Having a syntactically incorrect SPF policy published under no circumstances means that the sender host is a spammer and should be treated so.

by Krisztián Fekete (Vamsoft) 5 years ago
(in reply to this post)

9

Hello Krisztián,

I think this is turning in circles. I know - this is not RFC; but you also implemented such an option for 'SPF neutral'. IMHO it should be up to the user to decide what to do with it, as I can decide on SOFTFAIL and Neutral.
At least in my particular environment I only have syntax / unresolvable errors from spam, witch otherwise passed the tests. Is seems to occur more often for spammers witch use misconfigured DHCP networks form ISPs. After sending out spam they just reset their connection to get a new IP; making it impossible for SBL's to catch up.

by daniel.helgenberger 5 years ago
10

@daniel.helgenberger: "I know - this is not RFC; but you also implemented such an option for 'SPF neutral'. IMHO it should be up to the user to decide what to do with it, as I can decide on SOFTFAIL and Neutral."

Neutral is different, as a domain may report SPF Neutral on possible forgery attempts. SPF Neutral states that the policy publisher neither permits, nor denies the sender as legitimate for the domain. Yet, you can expect SPF Pass result from legitimate senders. By enabling that option you can still blacklist on SPF Neutral on a per domain basis.

But if the SPF record is syntactically incorrect, it is impossible to tell whether the sender is authorized to send in the name of the domain or not.

by Krisztián Fekete (Vamsoft) 5 years ago
(in reply to this post)

11

As I said, circles. It might be impossible to tell by RFC but it is quite undeniable that I have the above case where I want them to be blacklisted.

If its is ok I try my luck with a feature request. I will link to this thread.

Thanks!

by daniel.helgenberger 5 years ago

New comment

Fill in the form below to add a new comment. All fields are required. If you are a registered user on our site, please sign in first.

It will not be published.
hnp1 | hnp2