#Find spamsieve how to
#Find spamsieve pro
#Find spamsieve software
#Find spamsieve code

#Find spamsieve how to

Google knows how to scrape web pages better than anyone.

Scraping HTML may sound kludgy, but it works.

Or so I thought.īut I’ve now implemented technique #3 by outsourcing the whole thing to Google Custom Search and the initial results are spectacular. #3 seemed like a particularly poor solution because (a) you lose track of the differences between titles and tags, for example, and (b) it’s kludgy. Solr uses technique #2, and that’s clearly better for all sorts of reasons.

#Find spamsieve code

My fancy search code used method #1 and the resulting code generated some of the longest, most confusing and slowest SQL queries I’ve ever seen.

Provide users advanced-search options such as boolean logic and restricting the search to a subset of objects such as only searching programs or only searching feeds.

Develop an ranking algorithm to display the most-relevant results first.

Make your search smart by ignoring noise words, being tolerant of (or correct) spelling mistakes, understand synonyms, etc.

There are three ways of doing this: (1) read the database tables in which these strings are stored in real time (2) in background/batch, build a separate table of the integrated text from the separate tables, then search this integrated table in real time or (3) build the integrated table by scraping/crawling the site’s HTML pages then, as in #2, search that table in real time.

You’ve got to search the text of titles, descriptions, keywords, tags and comments, which in our case are stored in separate database tables.

Searching for audio and video programs from a database that will hit 250,000 in the next few hours comes down to a few architectural issues: But the more I thought about it, the more I realized that would be only marginally better. I then considered implementing something like Solr, based on Lucene. Bottom line: It just didn’t do a decent job of finding stuff. There are all sorts of reasons why, but it sucked. Six weeks and a few thousand lines of code later, I had a new system that…well, sucked. In mid-March I started writing a fancy new full-text search module that worked across database tables and allowed all sorts of customization and advanced-search features. calls itself a site for “finding and sharing audio and video spoken-word recordings.” Sounds great, but our “finding” capabilities (search, in particular) have been pretty bad. It was downright spooky to see the messages moving without a clue as to why, but as soon as I realized my laptop was also running email, it became instantly clear.

#Find spamsieve software

It was my laptop, running this other instance of my spam filtering software that was moving messages around on the email server and hence on my desktop client. And because I’m using IMAP4, this change was sent to the server and then to the email client running on the desktop. The copy of SpamSieve on that computer decided some of them were spam and would move them to the spam folder. Messages would come into Google and, in some cases, my laptop would grab them.

#Find spamsieve pro

So here’s what was happening: My MacBook Pro had been on and running it’s own instances of Mail and SpamSieve. Because I have three different email clients (if you count the iPhone) I use IMAP4 instead of POP3 to communicate between those clients and the Google server and keep things in sync. Yes, I use their spam filtering, too - it’s much better than SpamSieve - but that wasn’t it. I use Google as my inbound and outbound email server. I even caught the nasty gremlin in the act. (Any of you email geeks starting to get a clue here?)įor a totally separate reason I pulled out my MacBook Pro, and that’s when it hit me. With absolutely no spam filtering turned on, stuff was still being flagged and moved.

Next, all the usual steps: rebooting, re-initializing this and that. So then I shut it down altogether: Whoa! I was *still* getting messages sent to the spam folder. But recently I’ve been noticing that the spam detection has been hyperactive: way too any false positives. I use OS X’s Mail app along with SpamSieve for spam filtering. So I’ve been having this realy strange problem.