Regex filter all emails with links except to specific tlds - ORF Forums

@CBGraham: Hello CBGraham,

If you remove the whitespace requirement (i.e. \s) after the non-capturing group, then I think your regex should work. It should be noted though, that your regex pattern is not restricted to match URLs inside the HTML <a> anchor tags but anywhere in the document - which may or may not be an issue for you.

As for the configuration in ORF, you have two options here, either should work:

EDIT: The 'User-Defined URL Domain Blacklist' (option 2) should not be used for this. The SURBL test extracts the domains from the values of the 'src' and 'href' attributes found anywhere in the HTML source.

1) Add your regex to the Keyword Blacklist of ORF with an "Email body + Body raw HTML source" search scope (Keyword Blacklist > Keyword Filter Properties dialog > Filter Properties tab). Make sure you select the "Regular expression (Perl-compatible)" expression type on the 'Filter Expression' tab as well.

2) Create a new regex without the http(s) prefix to match the unwanted TLDs in the domains that ORF has already extracted from the mail body, then add it to the 'User-Defined URL Domain Blacklist' of ORF (Blacklists > SURBL Test > User-Defined URL Domain Blacklist - Configure button). See: User-Defined URL Domain Blacklist: https://vamsoft.com/support/docs/orf-help/5.5.1/adm-urlblacklist#manubl

@CBGraham: The URLs that ORF extracts from an email is distilled to a "pure" domain list, meaning the resulting list contains only strings in a 'sub.domain.tld' format. http(s), ftp, www etc. prefixes and other suffixes are stripped away. Each domain is then checked against the enabled SURBLs and user-defined wildcard/regex expressions.

Although your regex pattern is a bit overcomplicated**, it should work properly without false positives - as far as I can tell. Are you sure that it was your TLD filter that triggered the blacklist event (if you add a comment to the filter expression, that will be logged on each hit)? What message was logged for the event?

** I would recommend a simpler solution: (?!.*\.(com|org|net|gov|edu|us))

@CBGraham: I don't quite see how this is a bad match - unless the domain "01D424CA.AB974BC0" was not actually harvested from an URL. Do you have the email that contained this domain by any chance? If so, could you save the email in an .eml or .msg format and send it to us () for analysis, please?

@Daniel Novak (Vamsoft): Hello CBGraham,

I have asked the devs whether the SURBL Test harvests URLs from anything other than actual HTML hyperinks ( i.e. from anchor elements) and unfortunately it does. It looks for the 'src' and 'href' attributes in the raw HTML source and extracts the domain from those. Thus, I must withdraw my previous suggestion that you should use the 'User-Defined URL Domain Blacklist' for your filter idea.

Instead, add the following regex to the Keyword Blacklist of ORF with the following settings:

1. Start the ORF Administration Tool, and connect to the local or a remote instance
2. Navigate to 'Blacklists > Keyword Blacklist' page
3. Click 'New'
4. In the 'Keyword Filter Properties' dialog, set the search scope to 'Email body' and mark the 'Body raw HTML source' checkbox enabled
5. Add a 'Comment' text (e.g., "URL TLD filter")
6. On the 'Filter Expression' tab, add the following expression:

.*&lta[^>]*href=['"](?![^>]*\.(com|org|net|gov|edu|us))[^>]*>.*

7. Set the expression type to 'Regular expression (Perl-compatible)'
8. Click 'OK'
9. Save your settings to apply the changes by pressing Ctrl + S.

Please let me know if this has helped.

@CBGraham: I am afraid that will not solve the issue in the long term. It may not cause false-positives in case of URLs such as, "cid:[email protected]", but the URLs will still be harvested from every HTML element that has an 'src' or 'href' attribute.

Of course, this is not an issue if you want to filter the top-level domains everywhere in the email - even in image URLs. In that case, consider the problem solved :)