Archives For July 2011

As you might know, we’ve been hiring a steady trickle of engineers, and we’re always eager for more. But we find ourselves without a true sysadmin, and honestly, it’s starting to show.

A long time ago Witold and I joked about scalability being “a problem we’d love to have”. When you are struggling to get anybody to use your product, it just seems like the last thing on your mind. Well, as much as I love the fact that there are hundreds of thousands of people using Expensify — it’s starting to add up to a real problem. Servers are getting overloaded. Databases are getting slow. I think the final straw was when I realized it was going to take over a week — sending mail as fast as we possibly could, 24/7 — just to send out a newsletter to all our users. So I love it, but it’s still a problem, and one that we’re not well equipped to solve.

Sure, we can make due — the great thing about being a programmer is there’s no problem we can’t solve. But our solutions are sometimes, shall we say, suboptimal. We can build anything from scratch, but sometimes it’s better to just install and configure off the shelf.

So this is my request: can you help us hire a sysadmin? Someone ideally familiar with Amazon Web Services (S3, EC2, SES), Ubuntu, lighttpd, PHP, postfix, ssh, and all the other myriad tools of a modern web infrastructure. But also someone with a deep intuitive understanding of disk and memory caching on virtual machines (where every virtual hard drive is actually layered on top of a *physical* hard drive — so there are two layers to manage), is well versed in the consequences of WAN versus LAN replication (high latency, low reliability), and who brings a deep reservoir of enthusiasm and expertise tearing apart a programmer system and rebuilding it the Right Way.

We’ll always need engineers. But engineering isn’t everything. Help us find a sysadmin and you’ll have my eternal gratitude. To apply, please ask them to read up on us on our jobs page and send answers to the following questions to

  1. When did you get started with computers and why?
  2. Contrast the pros and cons of virtual, dedicated, or colocated servers for a company like Expensify?
  3. What’s the largest, fastest, most reliable, most complicated environment you’ve ever managed?
  4. What’s a Linux command or tool that you think is more awesome than most people realize, and why?
  5. Based on what you can determine from outside the network, what would you suggest we change and why?
  6. When you receive a text message, what’s your first thought?

Thanks, I really appreciate your help!

This is the third in a six-part series elaborating on a presentation I gave at the AlwaysOn OnMobile 2011 conference titled “Disrupting the Enterprise”. I suggest reading the first (or watching the video) and then continuing on below.

Condition #3: A Cascade of App Stores

This third condition is done; it was a one-time wave that will likely never be repeated. I’m referring to the overnight rise of a series of web and app stores, and the cascading effect of new stores inhereting the rank of the old, creating a sort of “meta rank” where apps on top of one tend to dominate the rest.

It all started with the iPhone. (You might argue it started with the BlackBerry or even Sidekick, but those stores never really caught on like it did for iPhone.) It seemed like overnight a million apps appeared, vying to appear at the very top of a seemingly infinite list of applications in each category. Furthermore, because your rank was largely a function of your install rate (and your install rate was largely a function of your rank), this created a feedback loop where the people at the top tended to stay at the top and increase their lead over time: it created a strategic high ground with incredible staying power.

Quick on the heels of iPhone was Android. To a degree, the apps at the top of the iPhone App Store were the first ported to Android, and re-established their dominance in the Android Marketplace. (The difficulty of porting to a new platform allowed a bit of discrepancy to appear, but the iPhone ports came in with an advantage over original Android apps.) And then after Android came the Amazon, Verizon, and Windows Mobile app stores — each reinforcing this hierarchy to a greater or lesser degree, and each opening a brief window of opportunity that has since irreparably closed.

(As an aside, the conventional wisdom — so far as I can tell — is to just buy your way up to the top of the app store using TapJoy and its brethren, each time hoping that some of your rank sticks. This does seem to work a bit. But it also tends to flood your data with low-quality leads, such that the more you employ this strategy, the less actual awareness you have about whether any of the signups are worth having. Furthermore, it seems the quality of the installs “at the top” aren’t as good as the quality somewhere in the middle: people who scroll down a couple pages to find you seem more loyal and genuine than those that simply install whatever app appears at the top of the list. But that’s a different topic.)

The upshot is in rapid succession, the mobile app stores established a murky order that tends to slosh around a bit, but generally settle back where it started. I speak here from experience: we missed this wave, and thus our app has a very low rank on each store — and there’s very little we can do about it except fight hard each and every step. But as luck would have it, we were given a second chance.

Right after we launched, Salesforce held a fortuitous contest to create the best expense reporting app, listing the result in the Salesforce AppExchange. We did really well, and earned our place amongst the illustrious “Force 40” — the top 40 Salesforce apps. Right around then, Intuit saw the meteoric rise of the iPhone and Android app stores and thought “I wonder if we can do the same with QuickBooks?” So they came out with the Intuit App Center. And naturally, they approached us to be early adopters, primarily due to our high rank in the AppExchange. This good fortunue repeated with the launch of the Google Apps Marketplace, which obviously couldn’t launch without the top Salesforce AppExchange and Intuit App Center apps (including us). This process repeated a couple times more, and now Expensify appears at the top of all the major web app stores, much to our delight.

But all that is done. Sure, new app stores open up every day — for both mobile and web. But all the major players have already made their moves, and their not going to make them again. That wave has washed ashore, and though we can scramble along the beach for a new position, those that were washed far in have a large, enduring lead over those who weren’t.

Why you should care: If you have an app with high rank, celebrate. If you don’t, prepare for a very long, hard slog to the top. It’s not impossible — and there are snakeoil salesmen touting every dirty trick to accelerate it — but ultimately the deck is stacked against you for the forseeable future.

Tune in next week to continue the series and read about how these three conditions have created a new opportunity for selling into the enterprise in a faster, lower-cost, and higher-quality way than ever before. See you then!

Previous: Word Of Mouth is Winner Takes All (2/6)
Next: The Bottom Up Adoption Curve (4/6)

Hello everyone out in Expensify Land! Today we write with two mission-critical questions:

1. What reporting requirements does your company have that our application has not yet addressed?
2. What features could we add that aren’t required but would simply make your life easier?

Sure we’d like to know more – like what adoption barriers Expensify faces at your company – but let’s face it, in the Internet age our attention span is about as short as a hamster’s memory capacity is small. So go ahead. Now’s your chance. The comment gate is wide open!

We’d like to hear from you on these topics because you – that’s right YOU – are the maverick employees helping us lead the charge against arcane norms in the expense reporting world. You – whether you know it or not – are out on the front lines fighting with us against awkward “solutions” that weren’t intended to make expense reporting easy, like Excel or the manila folder.

The Battle of Excel

The Battle Against Excel (Dramatic Re-enactment)

Now if you’d read this far…then congrats to you! You’re a rare breed; most people don’t make it here. We’d love to hear interesting success stories that your entire company has had using Expensify. In fact, WE’RE HAPPY TO FEATURE YOU ON THIS BLOG!

If you’re shy, then no problem. Contact us at or Share your feedback and success stories with us!

Up next week: We’ll respond to what everyone had to say and keep the conversation going. Until then, don’t be shy! Contribute in your own way. We want to make our compelling product even more so. Together, we can make something that’s doesn’t suck!

As I mentioned last week, I recently gave a brief talk at the AlwaysOn OnMobile 2011 conference titled “Disrupting the Enterprise” (video). I was asked to give a bit more detail, so I’m doing that now, in six parts. This is the second one; read last week’s and then continue on below.

Condition #2: Word of Mouth is now Winner Takes All

A second, no less important condition is how blogs and social networks now cause conversations to converge swiftly and globally. This means that word of mouth — something that might have previously shown strong regional or vertical tendencies — has instead become a universal phenomenon with “winner takes all” properties.

Now I’m no social scientist, and I think this trend has been growing for a long time. But there are more options than ever to directly broadcast your ideas to world in a completely unfiltered fashion. Prior to the Internet, what were your options to broadcast to the world without the permission of some large publisher? Ham radio? BBS’s helped, but were mostly regional (due to long-distance fees, and Fidonet was a pain). Usenet was limited to very techy spheres. IM was instant and global, but not broadcast. Email mailing lists and web forums are starting to get truly global reach amongst average folk. Comments on blog posts are even better — it combines quality editorial content with feedback from the unwashed masses. (Oddly, Facebook is a step back as you only talk to your friends.)

But Twitter takes the cake. Now anybody can broadcast from essentially any device, using essentially any form of media, to essentially any number of people — at essentially no cost. The “follower” model allows it to take advantage of quality (*ahem*) editorial content, while realtime searching, hashtags, trending topics, and retweets allow even the unpopular to get heard. It’s as close to an unfiltered meritocracy of ideas as we’ve ever seen. All this creates any number of interesting consequences. But of particular relevance to this discussion is that it helps the whole world converge on a consistent story with exceptional speed.

I say this based on the theory that the “distance” an idea travels is some function of the “strength” of the idea and the “resistance” of the medium, and that given two equally strong but conflicting ideas you’re more likely to believe with whichever one you’ve heard longest and most frequently. This means the more resistant the medium, the more it allows competing ideas to survive simultaneously in different regions or “verticals”. Then, by the time the ideas “collide”, each has built up a following that has heard it so often for so long that they’re not nearly as receptive to the idea as had they heard them both first.

But all these advances in broadcast communication have reduced the “resistance” of the medium to such a degree that every new idea is heard by almost everyone, all at once, everywhere in the world. When TechCrunch covers a new product, it’s a matter of hours, minutes, or seconds (but certainly not days or weeks) before everyone who might conceivably care knows. This triggers an immediate global discussion to evaluate the impact of that idea, with the winning option being broadcast further and faster than the loser. The result is on any given topic, there’s one major idea that dominates (the winner), maybe one alternative (the runner up), and that’s about it. Because as information flow goes up, attention span goes down (Twitter’s 140-character limit illustrating this nicely), meaning it’s simply not possible to sustain more than a couple competing ideas on a given topic for long.

When the conversation dies down and everybody moves to a different topic, the winning idea gets the lion’s share of the attention. And because everybody is biased in favor of what they’ve heard the most in the past, the winning idea of the last conversation has an advantage in the next conversation, causing it to strengthen its position over time. So in a field of largely equivalent ideas, once one gets a slight edge, that edge just keeps increasing forever, until all the other ideas are forgotten — even if they might have been nearly as good in the first place.

Why you should care: If you find yourself in a field of equals, with none of you getting any attention, beware: once somebody gets a little attention, they’ll get all the attention. Make sure it’s you.

Expensify example: Even though Expensify is without question the leading small-business expense reporting system out there today, we didn’t start out that way. In fact, a few other very similar options came onto the scene right before or around us (and new ones are springing up all the time). And though I wouldn’t say we were all equivalent (because we actually all had very different approaches), I would say that none had a clear advantage over the other. But somehow we got an edge. I’m not sure exactly what happened: we had good early coverage, knew some good people, were emphatically interactive with and supportive of our early adopters, and just had a really good product. Maybe even the best product. But once we had a slight edge, it wasn’t so much that people suddenly started talking about us a whole lot, but they simply stopped talking about the alternatives. The discussion shifted from “There are several options including X, Y, and Z” (where we might not be listed) to “Concur is for big companies, Expensify is for small companies.” And once you hear that a few times — especially when you try us and realize that we’re actually quite good — it’s just not worth your time to investigate alternatives. After all, if nobody discusses them, how good can they really be? Granted, I’m hesitant to say any of this without knocking on wood — I’m extremely conscious of how fast the tides can turn.  But I’m willing to take a chance as I think we now have a very comfortable lead in terms of functionality and usability over any competitor. But back then we didn’t, though you wouldn’t have known it by listening to the global consensus.


Tune in next week to continue the series and read about “A Cascade of App Stores”.  (Follow us on Twitter at @expensify to be notified when it comes out.)  See you then!

Previous: How To Disrupt Your Industry (1/6)
Next: A Cascade of App Stores (3/6)