Handling Incoming Email With Your Web Application

This morning I was looking for a way to handle incoming email in a web application (similar to the way Highrise and Evernote let you email things to a special email address and have them put into their system). There are a number of ways to do this via procmail, or by using something to connect to your mail server using POP or IMAP and reading emails, but I was looking for a way to do this without having to host my own email infrastructure. Ideally, I want something like Twilio, that will receive the email and then do an HTTP POST to the endpoint of my choosing.

Here’s what I found.

CloudMailIn

Still in beta (and free while it is), this looks robust. It’s also available as a Heroku addon, if that’s how you roll.

(A tip of the hat to @peterc for pointing me to this one)

APInbox

Looks similar to CloudMailIn, though not in beta. There’s a free plan for up to 100 emails a day, and then it goes up from there. Their site was down when I first went to it this morning, which makes me a tad nervous, but that may well be an isolated thing.

SendGrid

SendGrid is a heavy hitter in the email space, mostly doing outbound delivery. They do however have a Parse API that seems to perform the same function as the other two services. I’m not sure on the pricing here, their basic plan is \$9.95 per month for 10,000 emails, but I’m not sure if that includes incoming or not. UPDATE: I heard from SendGrid. Their plans cover both incoming and outgoing, so for the in the case of the \$9.95 plan, it could be a mix of both, up to 10,000

(thanks to Twilio’s @johnsheehan for the pointer to SendGrid)

I haven’t used any of these yet, so I can’t make an endorsement of one over the other, but I thought I’d post it here in case anyone else is looking for this kind of provider. If you have experience with any of these, please comment with your opinion.

The Programmable Government

We are headed toward a time where the workings of government are much more visible to the American public.

Through things like the Freedom of Information Act, this information has technically been available for some time - but not in a form that is easily consumed. This is starting to change.

The emergence of open APIs that provide access to information about how the government is operating is a massive step in the right direction. It will, I hope, bring forth a new wave of websites that mine the data that these web services provide, and expose it to the world. Voting records, government expenditures, bids, and bill details all need to be made available for anyone to consume.

Here are a few of the APIs that I have come across. Some are available from the government themselves, others are from third parties.

The New York Times Congress API

This API, part of a growing set of APIs from the Times, let’s you program Congress. Well, not exactly:

The initial release exposes four types of data: a list of members for a given Congress and chamber, details of a specific roll-call vote, biographical and role information about a specific member of Congress, and a member’s most recent positions on roll-call votes.

Our database contains House votes since 1991 and Senate votes since 1989. House members are from 1983 and Senate members date back to 1947

Follow the Money

This API provides information about campaign contributions for state-level campaigns.

Sunlight Labs

The Sunlight Labs API provides methods for obtaining basic information on Members of Congress, legislator IDs used by various websites, and lookups between places and the politicians that represent them. The primary purpose of the API is to facilitate mashups involving politicians and the various other APIs that are out there.

Capital Words

This site, another project of the Sunlight Foundation tracks word frequency from the congressional record. Most frequently used word: Proposed.

USASpending.gov

One of the few official government APIs, this allows you to find out where your money goes:

Have you ever wanted to find more information on government spending? Have you ever wondered where federal contracting dollars and grant awards go? Or perhaps you would just like to know, as a citizen, what the government is really doing with your money. The Federal Funding Accountability and Transparency Act of 2006 (Transparency Act) requires a single searchable website, accessible by the public for free that includes for each Federal award:
1. The name of the entity receiving the award;
2. The amount of the award;
3. Information on the award including transaction type, funding agency, etc;
4. The location of the entity receiving the award;
5. A unique identifier of the entity receiving the award.

The GOP API

The Republicans are ahead of the Democrats on this one, but I would doubt we’ll have to wait to long to see them follow suit. This site is similar in concept to the Times’ Congress API mentioned above (though obviously only for the Republican members), but provides some additional information that the Times does not.

Conclusion

This is only a sampling of the APIs that are available, and hopefully this is only the beginning. I’m optimistic that both the government and third parties will provide more of these, and that groups will make use of this information to expose the inner workings of the government to the people who elect them.

Are their any APIs that I missed? Are there any sites that are using this information in interesting ways? What do you want the government to make easily consumable that it isn’t today?

The Two Kinds of Programmers


Photo courtesy of Roby72

In my time as a developer, and now managing a team of developers, I have come to realize that there are two kinds of programmers: the Journeyman and the Craftsman. These terms aren’t mine - I’ve seen them used other places - but they describe the developers I’ve worked with pretty well.

The Journeyman

…knows one programming language.

…knows one operating system.

…can’t be bothered to learn something on their own.

…doesn’t know anything about the operating system or hardware their applications run on: “Someone else takes care of that”.

…never masters his tools. “I know my way around my IDE, that’s good enough”

…doesn’t refactor: “It’s ugly, but it works. Leave it alone!”

…only learns about the part of the system they are working on. No need to learn the rest of the system: “That’s not my job”.

…doesn’t want to take on an unfamiliar technology: “I haven’t had any training on x”.

The Craftsman

…knows a handful of programming languages, and is always on the lookout for the next one he should learn. He knows that learning any new language will stretch his mind and make him a better programmer in the language he uses day to day.

…devotes time to learning about new technologies, and helps to make others aware of them.

…understands the platform and operating system his applications run on, because he knows that’s the only way to diagnose many problems.

…masters his tools. He can perform magic in his chosen editor, and is always looking for ways to make himself more efficient.

…rarely passes up an opportunity to broaden his knowledge of the system he is working on.

…is always willing to take on something he’s unfamiliar with. He can pick up most things pretty easily, and enjoys the challenge of learning something new.

One craftsman is worth three or four journeymen. Easily.

It’s the journeymen whose jobs often end up moving overseas (and rightfully so, they add little, if any, value).

The longer I manage development projects, the more I value the craftsmen I have around me.

Wil Shipley on Unit Testing (and Why He's Wrong)

Wil Shipley is the author of several well known Mac applications, notably OmniGraffle and Delicious Library , and has a well-read blog. In a post here he asserts that “Unit testing is teh suck”.

Although Wil is a bright guy and very likely a better programmer than I will ever be, this is completely out of touch with reality. I suspect anyone that has done TDD, or even used a unit testing tool would agree with me.

Wil starts off with this statement:

I've NEVER, EVER seen a structured test program that (a) didn't take like 100 man-hours of setup time, (b) didn't suck down a ton of engineering resources, and © actually found any particularly relevant bugs.

Well, I think Wil may be exaggerating a bit with the 100 hours nonsense, but let’s assume he’s not for just a second. I’ve never written a single line of ObjectiveC code, but if OCUnit is anything like the other unit testing tools out there (NUnit, JUnit, Test::Unit), this is a ridiculous statement. As for not finding relevant bugs, I find it unlikely as well.

Next up is this statement:

It's actually provably impossible to test your program with every conceivable type of input programmatically, but if you test by hand you can change the input in ways that you, the programmer, know might be prone to error.
Sorry, wrong answer. Anything you can do manually can (and should) be automated. Two reasons: - Manual tests aren't reproducible. Duh. - Manual tests take a **lot** of time to do. Because they're manual. Double-duh. Furthermore, things that are time consuming are often skipped when people get tired, lazy, or just need to get the thing out the door. And unless you wrote down all of those manual tests, if the person who wrote the code (and was doing the testing) leaves the company, you're in trouble. Summary: **Unless you have a good set of unit tests, real regression testing is nearly impossible.** Next, Wil describes his testing methodology:
When you modify your program, test it yourself. Your goal should be to break it, NOT to verify your code. That is, you should turn your huge intellect to "if I hated this code, how could I break it" as SOON as you get a working build, and you should document all the ways you break it. \[Sure, maybe you don't want to bother fixing the bug where if you enter 20,000 lines of text into the "item description" your program gets slow. But you should test it, document that there is a problem, and then move on.\] You KNOW that if you hated someone and it was your job to break their program, you could find some way to do it. Do it to every change you make.

There amount of time it would take to do this on a program of any size is ridiculous. Although Wil may have the self-discipline to do this, the rest of us likely don’t. And again, it’s not a repeatable process.

Here’s the bottom line: Unit tests are important. Extremely important. And they don’t add time to a project, in fact in most cases they will reduce it. At first blush this doesn’t make much sense, but here’s the rationale:

  • You typically write this code anyway, but end up throwing it away. How often have you written some chunk of code, then written a piece of throwaway code to test it. Unit tests are all about taking that code and making it automatic.
  • You’re trading test-coding time for testing and debugging time. Yes, it takes time to write unit tests. But it also takes a lot of time (arguably more) to run a bunch of manual tests. It also takes time to debug problems found later on that could have been caught earlier.

Wil has forgotten one of the basic rules of software development: Catch Bugs Early. If you catch a bug when you’re just getting started, it costs nearly nothing to fix. Find that same bug after you have released it to the world, and it costs significantly more - particularly if you’re not talking about a web application.

On top of all of this, there are a couple of other points worth making about Unit Tests:

  • Refactoring, or any type of major code change, is scary without good unit tests. Often, people avoid changing the code altogether and just live with it’s warts. I can’t find the post now, but one of the 37Signals guys blogged about how having unit tests made the architectural changes that were made to Basecamp a lot easier.
  • When you have to integrate with other developers, unit tests are invaluable. This is particularly true when coupled with Continuous Integration.
  • Unit tests (if done right) will improve the architecture of your code. You simply can’t have a highly-coupled application if you are unit testing everything independently.
  • Unit tests serve as documentation for how the code actually works. Forget that dusty API reference that hasn’t been updated for three versions. A set of passing unit tests tells you what the code does right now. Paired with a code coverage tool, it can even tell you what’s not documented.

Summary: Unit tests == Awesome, and represent all that is good in the world. I certainly don’t think that they’re a panacea, but they’re certainly not “teh suck”.

I didn’t expect this to turn into such a rant, but when people who are respected and admired in the community make harmful and incorrect statements, they need to be corrected.