At the Expensify World Headquarters, you’ll often hear some employees talking about curling and it’s not always referring to the bash command.  Sometimes, we’re talking about the ice sport.  And to clarify, when I say “some employees” I mean me.  I started curling about a year ago and I’ve been annoying my coworkers about it ever since.  I recently took some Expensifiers curling to show them how awesome it is.

Old school curling Our story begins in the 1500s, in a faraway land called Scotland, where the same quarry has been used to create the 42 lb stones.

Nowadays we play indoors (mostly) on pebbled ice, so it’s not the ice skating ice you’re used to. To give you an idea of what pebbled ice is like, imagine the once smooth ice has the topographic profile of an orange rind.  This helps to reduce friction between the ice and rock, making it easier for the rock (and player), to slide along the ice. This also makes it surprisingly easier to move on the ice.

The heroes of our story, Mike and Matt, had to first learn how to throw the stone, then sweep, and lastly compete in a curling match to the death (not really).

They couldn’t play alone – a curling team requires four people.  Each teammate throws two stones in a row, alternating each stone with the opposing team.

Our brave skip, Mike, is the team’s mastermind behind this “end” – an end is curling’s version of an inning.  He decides where Matt should aim the rock and how hard it should be thrown.
He also tells the sweepers when to sweep to ensure that it reaches its destination.

delivery

The play is up to Mike.  Will he call it so the rock glides into the house? Does he want to put up a guard stone to protect the stones that are in play? Or will the rock be thrown with such power that it eliminates an opponent’s rock from play?

Mike sets his broom on the ice, giving Matt a target at which to aim.  Matt fearlessly eases into the hack, ensuring his shoulders are square to the broom.  He slides out, lunging forwards on the ice, staring intently at his target. The moment before he releases the rock he applies a gentle turn to the handle, imparting the signature “curl”.

The sweepers walk alongside the rock, judging the speed of the rock and calling out to the skip whether they think it looks fast or slow. Mike drops to one knee, eyeing the rock’s trajectory, ready to urge the sweepers into action if needed. “SWEEP, SWEEP, SWEEEEEP” Mike yells.  The sweepers begin vigorously sweeping.  The sweeping generates heat, reducing the friction and preventing the rock from losing its momentum.

They pick up their speed to keep up with rock.  While sweeping they must ensure that they don’t hit the stone with their broom. The sweepers also need to avoid colliding with any of the other stones that are scattered on the ice.

sweeping

“Whoa, Whoa, OFF, Whoa” Mike yells, the stone’s current trajectory is exactly what he wants. The sweepers look up and take their brooms off the ice.  They continue to walk besides the stone, ready to start sweeping again if Mike calls for it.

The stone slows down, coming to its resting place in the the blue circle in the house (the concentric circles on the ice).

The end is over and there are four stones in the house.  Three yellow ones, and one blue one.

Only one team can score an end.  So how did our heroes do?

They rocked it! (See what I did there?)

The team with the rock closest to the button (the center of the house) gets a point for each stone that’s in the house and closer to the button than their opponent’s nearest stone. Mike and Matt’s yellow stones are lying two.  This means that the two yellow stones are between the button and the closest opponent’s stone, the blue stone.  The third yellow stone lies beyond the blue stone and does not count.

Our heroes score two points this end.  They maintain this lead for the other 7 ends in the match and leave the ice champions.

Like after any proper curling match, they walk off the ice and into the bar.  Winning the game means they owe their opponents a round of drinks.

Here at Expensify, our team is as diverse as they come. One thing we have in common is that we absolutely love what we do. So much so that we oftentimes find ourselves practicing our passion for building even when not using our powers to make users happy.

You know that phenomenon- the spillover of creative energies that is manifest in the form of side projects: those small unexpected products that arise from weekends of pleasurable hacking and challenge seeking.

The benefits of having such side projects are well established. The plethora of blog posts out there usually have these oft-repeated, but sound points:

  1. Staying on Top of New Ideas in the Industry, a.k.a Employability
  2. Refreshing old skills
  3. Participating in a community – Open Source, Non-profits
  4. ???
  5. Profit!

Indeed, given the rate of ideas coming forth in our fast paced industry, we should not find ourselves taking sledgehammers to our helpless workstations, like these Luddites:

Hammertime!

Let’s switch the perspective a bit: Sure, personal, professional development is great, but what does all this mean for a team?

With those previous benefits in mind, I contend that the collective health and culture of a group can be accurately judged by the side projects of its members.

From the perspective of a prospective candidate, these projects provide windows through which they can evaluate the nature of the team that they are seeking to join, for they usually have the following questions:

  1. Do members of the team take the initiative to explore new ideas and better themselves?
  2. Do they play with and familiarize themselves to new ideas so that they are never disrupted by new technologies?
  3. Do they provide opportunities for one to learn from the team, and for one to be introduce to new ideas and networks?
  4. Are they fun? Do they express creativity and their personality through their work outside of work?
  5. Are they individuals who are passionate in what they do, and seek to solve problems outside of their main focus?

These are crucial questions that are answered in the affirmative by the team’s side projects. Evaluating those works help a candidate decide if the team is a cultural and professional fit. A group, with a vibrant culture that could hardly contain itself, shows its character through such avenues.

Below you’ll find a selection of side projects from our own awesome team.  Some coworkers are a bit shy, and many of us have unfinished projects hanging around, but the collection reveals a candid cross-section. Without further ado, here are a few things we’ve done:

Whenopedia

http://www.whenopedia.org/ by Philip Sharp

This well-polished tool allows you to pull up snapshots of Wikipedia articles for a specific date and time. Now, you can read up on your shows without fear of spoilers!

Whenopedia

Samples

http://appstore.com/giorgiocalderolla/samples by Giorgio Calderolla

An audio sampler for your iOS device…

Add sound effects, loops, mix samples into your music, and much, much more!

appstore3

BPM Lover

http://bpmlover.com/ by Lois Di Qual

Tempo extraction technology right in your browser. The future is here!

bpmlover

CJK Typesetting

http://type.fernjager.net/ by Robert J Chen

Here’s a one-weekend experiment with Go and SQlite for a pastebin-type editor for East Asian characters.

CJK Typesetter

Fun with R

David Barrett, our CEO, has been tinkering with R with data from our massive dataset.
RViz

From this limited selection, one can see that we’re quite the motley crew of hackers, with diverse interests in music, media, and foreign languages! If you liked what you’ve seen here, come join our eclectic team: http://we.are.expensify.com/why-work-here

Happy Valentine’s Day!

Our romantic offering to all of our users this year in a video of our CEO on FOX Business news discussing Bitcoin.

David was asked to appear on MONEY with Melissa Francis on FOX Business News to explain why the recent issues and regulation talks surrounding Bitcoin is ultimately a good thing for the crypto-currency.

 

UPDATE: All systems are back online. Thank you for your patience!

WHEN: February 11th, 8pm – 10pm PST

WHAT: Full site downtime, all services down.  Mobile apps will continue to work as if they were offline.

As part of our commitment to upholding the highest security standards, we rotate the encryption keys for our server infrastructure annually.  Our next window for updating these keys will be from 8pm – 10pm PST on February 11th, during which users will be unable to access the main Expensify website, including through partner portals such as Salesforce.

We apologize for the inconvenience and appreciate your understanding.

Imagine you ran a coffeeshop with great coffee, a cool vibe, comfortable seating… and no customers.  Not an uncommon problem, and the classic solution promoted by social media experts would be something like:

“Create a Twitter handle and offer discounts to customers who follow. Broadcast weekly events and promotions via Twitter and engage with your audience to encourage them to come and bring their friends.  More discounts to people who get their friends to follow.  Create a mailing list; Tumblr; Facebook page; etc, etc.”

It’s an alluring prospect, that no matter where you are, using online social media tools you can tap into a global audience, create a network of influencers, and drive people to your coffeshop in droves.  The only issue?  I frankly doubt you’d get even a single customer out of it.  It’s all reasonable, common advice.  I just don’t believe it would work.  My advice?

“Stand outside your coffeeshop, up the block a bit.  Watch the people who come by.  When you see someone who looks like a potential customer, approach them and ask in a friendly, non-confrontational way: Why didn’t you come in?”

I wager in the first hour you’ll get at least one real customer, dozens of potential customers who are much more likely to stop by in the future, and a hundred tangible ideas on what you can do right now to attract more business:

  • Did they not like the vibe?  Redecorate!
  • Did they doubt the quality of your coffee?  Emphasize the roaster’s brand to demonstrate quality!
  • Did they just not notice you?  Put an A-board out front with an arrow pointing at your store!
  • Did they not want to go to an empty place?  Promote free coffee whenever you’re empty to anyone who comes in and sits for an hour!

The problem with “social media” is that you actually don’t care about a global audience: whatever you are, you want a very, very localized audience.  And the most important audience of all are the people who stopped by, checked you out, and walked away.  Just ask them why, and let that guide you.

Granted, I’ve chosen a physical example of a coffeeshop to illustrate this idea.  But as I imagine is obvious: the principle is exactly the same for your website.  The fact that “the internet” has billions of people is irrelevant to your service — the handful of people who actually visit your site are the ones that matter the most.

Again, everybody would agree with this.  But what would the classic social media advice be?  Probably to use the same sort of bland strategy the fictional coffeeshop owner would follow — and probably with the same (lack of) results.  Instead, my advice to you would be:

“Do everything you can to talk with your visitors.  Put a giant signup button on the homepage requiring nothing but an email address.  If that doesn’t work, just have it open directly to a chat window with you logged in ready to talk.  Ask them why they came, what they’re looking for.  You’ll be amazed what you hear.”

It sounds so obvious.  It’s so easy to do.  And yet, nearly nobody does this.  Luckily, we did, and I count as one of the most important techniques we ever used to get real world customer feedback.  I literally believe that had we not done this, Expensify simply wouldn’t be here today.  Our initial assumptions about what people cared about were so far off, in so many directions, had we not a tool like this we simply wouldn’t have survived the first year, and never become the leading brand we are today.

But despite this being such a hilariously obvious technique, I’m almost embarrassed to say we only stumbled into the technique by accident:

  1. Our first homepage had a big signup button on it that required only an email address.  (This is common now, but at the time it was really controversial: What, I don’t need to create a password? *mindblown*)
  2. Immediately after signup, you’d get a fancy stylized HTML email with a bunch of information nobody would ever read, and a validation link that most people would never click.
  3. However, we had a system that every hour, would email everybody who signed up in the past hour.
  4. This system ran on whatever increment in the hour that we started it, so if we started it at 1:17, it would run at 2:17, 3:17, etc.
  5. Furthermore, this system only sent plain-text emails, and only using my actual email account (“David Barrett <dbarrett@expensify.com>”)
  6. And the message, contrary to all marketing best practices, was extremely open ended with no actionable link to click:
From: David Barrett <dbarrett@expensify.com>
Subject: Welcome to Expensify!
-------------------------------------------------------
Hi there!  I see you just signed up for Expensify, welcome!  Can you tell me about yourself?  Namely, what sort of work do you do, how big is your company, how do you currently do expense reports, how did you learn about Expensify, how do you hope to use it -- that sort of thing.  Thanks!

-david
Founder and CEO of Expensify
Follow us here: http://twitter.com/expensify

A good response rate to an email like this is 2-3%.  A great response rate is 5%.  But this email got a 12% response rate.  And not just any responses — pages and pages of thoughtful, priceless feedback, from people who genuinely wanted us to succeed.  We were floored.  Not only was the feedback inspirational and motivational, but it created long term relationships with our champions that have stayed with us for the many years since. As for why it performed so well, I have a few theories:

  • It came a random duration after you signed up, on a random minute in the hour, so it wasn’t obviously an automated message.
  • This was reinforced by the fact that it was a simple text email.  Let’s be honest: nobody “real” writes well stylized emails with a formal greeting and logo — all it does is mentally flag you as spam.
  • It was written by me, and sent from my actual email address.  Hitting reply went straight to me, not some obvious mailbox.  (Many responses would start with “You can’t have actually just emailed me for real, so I doubt you’ll actually read this and I’d be shocked if you actually replied, but since you asked here are my thoughts…”)

And most important of all:

  • It came on average 30 minutes after signing up.  This meant that you received it right after your first interaction with the product, while it was still on your mind and your initial impressions were on the tip of your tongue, just waiting for someone to ask.

I can’t overstate how helpful this was in shaping the company, and building us into the responsive, user-focused company we are today.  Clearly, this gets harder and harder to maintain as you scale — when I send out our newletter to my millions of closest friends, I get a lot of responses.  It takes a long time to go through them all.  But I do, because “social media strategy” isn’t about the media, and it’s only loosely about strategy.  It’s really just about being social, talking with the people who like you (and those who don’t), and being responsive to those who matter the most to your business: customers, users, and those who might be some day.

At Expensify, everything we do is a balance. As a startup we can’t build every feature we and our users want, or install as many servers as we can imagine. Sometimes, though, we see a change we can make that won’t cost much (in time or money) and will benefit for our users. Here’s the story of one of those times that didn’t work out as well as we hoped.

Most of our website is written in PHP. While there is some healthy debate among our engineering staff, most of us like PHP for its rapid development and ease of deployment. Our web servers use the Alternative PHP Cache (APC) to cache compiled code and speed up requests for our users. A few months ago we updated our web server configuration to use less memory for each PHP process.

The original configuration looked like this:

fastcgi.server = (
    ".php" => ((
        "bin-path" => "/usr/bin/php5-cgi",
        "socket" => "/tmp/php.socket",
        "max-procs" => 96,
        "bin-environment" => (
            "PHP_FCGI_MAX_REQUESTS" => "500"
         )
    ))
)

It was a very standard configuration, which creates 96 PHP parent processes per server, each of which creates one child “worker” process. However, APC stores the cache in memory and creates one cache per parent process. That means we were caching the same data 96 times!

The best practice given in documentation for both PHP and Lighttpd is to start 1 parent process, and let it create all of the child processes. The child processes will share the parent’s APC cache.

The new configuration looked like this:

fastcgi.server = (
    ".php" => ((
        "bin-path" => "/usr/bin/php5-cgi",
        "socket" => "/tmp/php.socket",
        "max-procs" => 1,
        "bin-environment" => (
            "PHP_FCGI_CHILDREN" => "96",
            "PHP_FCGI_MAX_REQUESTS" => "500"
        )
    ))
)

This saved us the memory space of 95 extra caches, which we could use to support more users on each server.

Until, a few weeks later…

One Monday morning we found the site was running very slowly. We we checked our server logs, we found that the web server was being overloaded, and that the PHP processes could not handle the number of simultaneous requests that they did previously. We reverted to the old configuration, and the site started responding better.

We started researching further to determine what the problem was. With the help of Apache Bench we tested several possible configurations and found a surprising pattern. Each server could handle a maximum of PARENTS * CHILDREN + 129 * PARENTS connections. This meant that our original configuration could handle 12,480 connections per server, but our new configuration could only handle 225!

More research showed us that we were hitting the default maximum socket connections (128), that allow a backlog for each PHP process. Because we had fewer PHP parent processes, even though we had the same number of child processes, we had created a bottleneck that our traffic couldn’t fit through during Monday morning. (Which is, we know, the best time to do your expense reports.)

The lesson: best practices may be a good idea but they can cause other problems, and free ideas can have costs.

A modern interview sessionBased on the sudden spike in applications we’ve seen, there are a lot of people out there with New Year’s resolutions to make the jump.  Before you do, I’d recommend asking your employer-to-be the following questions:

  1. How many people do you hire in a typical month?  Great people are just hard to come by.  I don’t know of any startup that can reliably hire more than one truly fantastic person per month, so if they’re hiring people in droves, they’re not getting the cream of the crop.  Hiring is easy if you don’t care who you get: make sure they care.
  2. Who was the last person to quit, and why?  People change.  Companies change.  Even amazing people and amazing companies.  A great fit one day might gradually become less great over time, and the best people don’t tolerate that for long.  It’s an awkward topic, but see how they handle it.  Are they defensive?  Introspective?  Do they throw their former teammate under the bus, or provide a thoughtful explanation?  Is the company better off without that person, or what did they learn and change in response?  Nobody likes to be dumped, but if it doesn’t happen sometimes, you aren’t aiming high enough.
  3. When do you intend to raise money, and why?  The worst answer is “we just raised so we’re not even thinking about it” — raising is something you should always think about.  A better answer is “we have X months of runway before we need to raise again” — this means they’re at least paying attention.  But the best answer is “once X happens in the business, then we’ll raise”.  It’s subtle, but this question is really determining if a company is burning investor dollars to survive (and without it, they die) or to grow (and without it, they just grow slower).  Is the company primarily financed by selling product, or selling itself?  To be fair, most startups linger one failed round from certain death, especially in the early days.  But be aware that until you are growing under your own steam, you are wholly dependent on investors for your survival — and investors don’t always share your vision.
  4. Can I see where I’ll sit?  This actually wraps up a bunch of questions into one.  Do they even know?  Do they have assigned seating, and if so, can you influence it?  Did you meet the people you’ll be sitting next to in the interview process?  Is it clean and is there natural light?  Is it quiet, or is there music, or just generally noisy?  And most important: is it a place you really want to sit 2000+ hours a year?  Your workspace is your home away from home, and there is an enormous range between “we just have an awesome office full of diverse regions and you can sit anywhere” and “here’s your awesome standing desk with big monitor”.  There are no right answers, but put some real thought into what’s important to you and make sure to factor that in.
  5. And last but not least: What do I want to do accomplish in my life, and how does this help me down that path?  This is one you should ask yourself — every day, about everything you do.  Our industry is unique in that anybody reasonably good can make money, and with a bit of luck you can make a lot.  But all the ways you can spend it get boring much faster than you’d expect.  Hard problems and cool technology will keep you occupied for a while, but there are always more problems and more technology, and eventually even those get old.  If you’re great — or if you want to be great — your motivations likely run deeper.  You have some kind of secret, quiet ambition that is so grand, so audacious, that you are embarrassed to say it out loud.  So big that even if you devote the rest of your life to it, you still couldn’t finish it.  But so important to you, that you genuinely want to try anyway.  Knowing what you want out of life dramatically increases the odds you’ll get it, so figure out what that is, figure out what’s preventing you from doing it right now, and then evaluate whether this startup is going to give you the tools to take the next step.  Maybe you lack money, or maybe it’s skill, but more likely you just lack the confidence that comes from being part of a successful team.  Make sure your next job is the launchpad for the rest of your life, and not just a landing pad with an extended layover.

I picked five to make the subject link click-baity, but really you should ask a lot of questions.  If you’re great, you can probably work anywhere you want.  Interview them as much or more than they’re interviewing you.  If they’re great too, they’ll recognize you’re worth it and thank you for the effort.

[Edit] Lots of great discussion about interviewing advice over at HackerNews.