Here at Expensify, our team is as diverse as they come. One thing we have in common is that we absolutely love what we do. So much so that we oftentimes find ourselves practicing our passion for building even when not using our powers to make users happy.

You know that phenomenon- the spillover of creative energies that is manifest in the form of side projects: those small unexpected products that arise from weekends of pleasurable hacking and challenge seeking.

The benefits of having such side projects are well established. The plethora of blog posts out there usually have these oft-repeated, but sound points:

  1. Staying on Top of New Ideas in the Industry, a.k.a Employability
  2. Refreshing old skills
  3. Participating in a community – Open Source, Non-profits
  4. ???
  5. Profit!

Indeed, given the rate of ideas coming forth in our fast paced industry, we should not find ourselves taking sledgehammers to our helpless workstations, like these Luddites:

Hammertime!

Let’s switch the perspective a bit: Sure, personal, professional development is great, but what does all this mean for a team?

With those previous benefits in mind, I contend that the collective health and culture of a group can be accurately judged by the side projects of its members.

From the perspective of a prospective candidate, these projects provide windows through which they can evaluate the nature of the team that they are seeking to join, for they usually have the following questions:

  1. Do members of the team take the initiative to explore new ideas and better themselves?
  2. Do they play with and familiarize themselves to new ideas so that they are never disrupted by new technologies?
  3. Do they provide opportunities for one to learn from the team, and for one to be introduce to new ideas and networks?
  4. Are they fun? Do they express creativity and their personality through their work outside of work?
  5. Are they individuals who are passionate in what they do, and seek to solve problems outside of their main focus?

These are crucial questions that are answered in the affirmative by the team’s side projects. Evaluating those works help a candidate decide if the team is a cultural and professional fit. A group, with a vibrant culture that could hardly contain itself, shows its character through such avenues.

Below you’ll find a selection of side projects from our own awesome team.  Some coworkers are a bit shy, and many of us have unfinished projects hanging around, but the collection reveals a candid cross-section. Without further ado, here are a few things we’ve done:

Whenopedia

http://www.whenopedia.org/ by Philip Sharp

This well-polished tool allows you to pull up snapshots of Wikipedia articles for a specific date and time. Now, you can read up on your shows without fear of spoilers!

Whenopedia

Samples

http://appstore.com/giorgiocalderolla/samples by Giorgio Calderolla

An audio sampler for your iOS device…

Add sound effects, loops, mix samples into your music, and much, much more!

appstore3

BPM Lover

http://bpmlover.com/ by Lois Di Qual

Tempo extraction technology right in your browser. The future is here!

bpmlover

CJK Typesetting

http://type.fernjager.net/ by Robert J Chen

Here’s a one-weekend experiment with Go and SQlite for a pastebin-type editor for East Asian characters.

CJK Typesetter

Fun with R

David Barrett, our CEO, has been tinkering with R with data from our massive dataset.
RViz

From this limited selection, one can see that we’re quite the motley crew of hackers, with diverse interests in music, media, and foreign languages! If you liked what you’ve seen here, come join our eclectic team: http://we.are.expensify.com/why-work-here

Happy Valentine’s Day!

Our romantic offering to all of our users this year in a video of our CEO on FOX Business news discussing Bitcoin.

David was asked to appear on MONEY with Melissa Francis on FOX Business News to explain why the recent issues and regulation talks surrounding Bitcoin is ultimately a good thing for the crypto-currency.

 

UPDATE: All systems are back online. Thank you for your patience!

WHEN: February 11th, 8pm – 10pm PST

WHAT: Full site downtime, all services down.  Mobile apps will continue to work as if they were offline.

As part of our commitment to upholding the highest security standards, we rotate the encryption keys for our server infrastructure annually.  Our next window for updating these keys will be from 8pm – 10pm PST on February 11th, during which users will be unable to access the main Expensify website, including through partner portals such as Salesforce.

We apologize for the inconvenience and appreciate your understanding.

Imagine you ran a coffeeshop with great coffee, a cool vibe, comfortable seating… and no customers.  Not an uncommon problem, and the classic solution promoted by social media experts would be something like:

“Create a Twitter handle and offer discounts to customers who follow. Broadcast weekly events and promotions via Twitter and engage with your audience to encourage them to come and bring their friends.  More discounts to people who get their friends to follow.  Create a mailing list; Tumblr; Facebook page; etc, etc.”

It’s an alluring prospect, that no matter where you are, using online social media tools you can tap into a global audience, create a network of influencers, and drive people to your coffeshop in droves.  The only issue?  I frankly doubt you’d get even a single customer out of it.  It’s all reasonable, common advice.  I just don’t believe it would work.  My advice?

“Stand outside your coffeeshop, up the block a bit.  Watch the people who come by.  When you see someone who looks like a potential customer, approach them and ask in a friendly, non-confrontational way: Why didn’t you come in?”

I wager in the first hour you’ll get at least one real customer, dozens of potential customers who are much more likely to stop by in the future, and a hundred tangible ideas on what you can do right now to attract more business:

  • Did they not like the vibe?  Redecorate!
  • Did they doubt the quality of your coffee?  Emphasize the roaster’s brand to demonstrate quality!
  • Did they just not notice you?  Put an A-board out front with an arrow pointing at your store!
  • Did they not want to go to an empty place?  Promote free coffee whenever you’re empty to anyone who comes in and sits for an hour!

The problem with “social media” is that you actually don’t care about a global audience: whatever you are, you want a very, very localized audience.  And the most important audience of all are the people who stopped by, checked you out, and walked away.  Just ask them why, and let that guide you.

Granted, I’ve chosen a physical example of a coffeeshop to illustrate this idea.  But as I imagine is obvious: the principle is exactly the same for your website.  The fact that “the internet” has billions of people is irrelevant to your service — the handful of people who actually visit your site are the ones that matter the most.

Again, everybody would agree with this.  But what would the classic social media advice be?  Probably to use the same sort of bland strategy the fictional coffeeshop owner would follow — and probably with the same (lack of) results.  Instead, my advice to you would be:

“Do everything you can to talk with your visitors.  Put a giant signup button on the homepage requiring nothing but an email address.  If that doesn’t work, just have it open directly to a chat window with you logged in ready to talk.  Ask them why they came, what they’re looking for.  You’ll be amazed what you hear.”

It sounds so obvious.  It’s so easy to do.  And yet, nearly nobody does this.  Luckily, we did, and I count as one of the most important techniques we ever used to get real world customer feedback.  I literally believe that had we not done this, Expensify simply wouldn’t be here today.  Our initial assumptions about what people cared about were so far off, in so many directions, had we not a tool like this we simply wouldn’t have survived the first year, and never become the leading brand we are today.

But despite this being such a hilariously obvious technique, I’m almost embarrassed to say we only stumbled into the technique by accident:

  1. Our first homepage had a big signup button on it that required only an email address.  (This is common now, but at the time it was really controversial: What, I don’t need to create a password? *mindblown*)
  2. Immediately after signup, you’d get a fancy stylized HTML email with a bunch of information nobody would ever read, and a validation link that most people would never click.
  3. However, we had a system that every hour, would email everybody who signed up in the past hour.
  4. This system ran on whatever increment in the hour that we started it, so if we started it at 1:17, it would run at 2:17, 3:17, etc.
  5. Furthermore, this system only sent plain-text emails, and only using my actual email account (“David Barrett <dbarrett@expensify.com>”)
  6. And the message, contrary to all marketing best practices, was extremely open ended with no actionable link to click:
From: David Barrett <dbarrett@expensify.com>
Subject: Welcome to Expensify!
-------------------------------------------------------
Hi there!  I see you just signed up for Expensify, welcome!  Can you tell me about yourself?  Namely, what sort of work do you do, how big is your company, how do you currently do expense reports, how did you learn about Expensify, how do you hope to use it -- that sort of thing.  Thanks!

-david
Founder and CEO of Expensify
Follow us here: http://twitter.com/expensify

A good response rate to an email like this is 2-3%.  A great response rate is 5%.  But this email got a 12% response rate.  And not just any responses — pages and pages of thoughtful, priceless feedback, from people who genuinely wanted us to succeed.  We were floored.  Not only was the feedback inspirational and motivational, but it created long term relationships with our champions that have stayed with us for the many years since. As for why it performed so well, I have a few theories:

  • It came a random duration after you signed up, on a random minute in the hour, so it wasn’t obviously an automated message.
  • This was reinforced by the fact that it was a simple text email.  Let’s be honest: nobody “real” writes well stylized emails with a formal greeting and logo — all it does is mentally flag you as spam.
  • It was written by me, and sent from my actual email address.  Hitting reply went straight to me, not some obvious mailbox.  (Many responses would start with “You can’t have actually just emailed me for real, so I doubt you’ll actually read this and I’d be shocked if you actually replied, but since you asked here are my thoughts…”)

And most important of all:

  • It came on average 30 minutes after signing up.  This meant that you received it right after your first interaction with the product, while it was still on your mind and your initial impressions were on the tip of your tongue, just waiting for someone to ask.

I can’t overstate how helpful this was in shaping the company, and building us into the responsive, user-focused company we are today.  Clearly, this gets harder and harder to maintain as you scale — when I send out our newletter to my millions of closest friends, I get a lot of responses.  It takes a long time to go through them all.  But I do, because “social media strategy” isn’t about the media, and it’s only loosely about strategy.  It’s really just about being social, talking with the people who like you (and those who don’t), and being responsive to those who matter the most to your business: customers, users, and those who might be some day.

At Expensify, everything we do is a balance. As a startup we can’t build every feature we and our users want, or install as many servers as we can imagine. Sometimes, though, we see a change we can make that won’t cost much (in time or money) and will benefit for our users. Here’s the story of one of those times that didn’t work out as well as we hoped.

Most of our website is written in PHP. While there is some healthy debate among our engineering staff, most of us like PHP for its rapid development and ease of deployment. Our web servers use the Alternative PHP Cache (APC) to cache compiled code and speed up requests for our users. A few months ago we updated our web server configuration to use less memory for each PHP process.

The original configuration looked like this:

fastcgi.server = (
    ".php" => ((
        "bin-path" => "/usr/bin/php5-cgi",
        "socket" => "/tmp/php.socket",
        "max-procs" => 96,
        "bin-environment" => (
            "PHP_FCGI_MAX_REQUESTS" => "500"
         )
    ))
)

It was a very standard configuration, which creates 96 PHP parent processes per server, each of which creates one child “worker” process. However, APC stores the cache in memory and creates one cache per parent process. That means we were caching the same data 96 times!

The best practice given in documentation for both PHP and Lighttpd is to start 1 parent process, and let it create all of the child processes. The child processes will share the parent’s APC cache.

The new configuration looked like this:

fastcgi.server = (
    ".php" => ((
        "bin-path" => "/usr/bin/php5-cgi",
        "socket" => "/tmp/php.socket",
        "max-procs" => 1,
        "bin-environment" => (
            "PHP_FCGI_CHILDREN" => "96",
            "PHP_FCGI_MAX_REQUESTS" => "500"
        )
    ))
)

This saved us the memory space of 95 extra caches, which we could use to support more users on each server.

Until, a few weeks later…

One Monday morning we found the site was running very slowly. We we checked our server logs, we found that the web server was being overloaded, and that the PHP processes could not handle the number of simultaneous requests that they did previously. We reverted to the old configuration, and the site started responding better.

We started researching further to determine what the problem was. With the help of Apache Bench we tested several possible configurations and found a surprising pattern. Each server could handle a maximum of PARENTS * CHILDREN + 129 * PARENTS connections. This meant that our original configuration could handle 12,480 connections per server, but our new configuration could only handle 225!

More research showed us that we were hitting the default maximum socket connections (128), that allow a backlog for each PHP process. Because we had fewer PHP parent processes, even though we had the same number of child processes, we had created a bottleneck that our traffic couldn’t fit through during Monday morning. (Which is, we know, the best time to do your expense reports.)

The lesson: best practices may be a good idea but they can cause other problems, and free ideas can have costs.

A modern interview sessionBased on the sudden spike in applications we’ve seen, there are a lot of people out there with New Year’s resolutions to make the jump.  Before you do, I’d recommend asking your employer-to-be the following questions:

  1. How many people do you hire in a typical month?  Great people are just hard to come by.  I don’t know of any startup that can reliably hire more than one truly fantastic person per month, so if they’re hiring people in droves, they’re not getting the cream of the crop.  Hiring is easy if you don’t care who you get: make sure they care.
  2. Who was the last person to quit, and why?  People change.  Companies change.  Even amazing people and amazing companies.  A great fit one day might gradually become less great over time, and the best people don’t tolerate that for long.  It’s an awkward topic, but see how they handle it.  Are they defensive?  Introspective?  Do they throw their former teammate under the bus, or provide a thoughtful explanation?  Is the company better off without that person, or what did they learn and change in response?  Nobody likes to be dumped, but if it doesn’t happen sometimes, you aren’t aiming high enough.
  3. When do you intend to raise money, and why?  The worst answer is “we just raised so we’re not even thinking about it” — raising is something you should always think about.  A better answer is “we have X months of runway before we need to raise again” — this means they’re at least paying attention.  But the best answer is “once X happens in the business, then we’ll raise”.  It’s subtle, but this question is really determining if a company is burning investor dollars to survive (and without it, they die) or to grow (and without it, they just grow slower).  Is the company primarily financed by selling product, or selling itself?  To be fair, most startups linger one failed round from certain death, especially in the early days.  But be aware that until you are growing under your own steam, you are wholly dependent on investors for your survival — and investors don’t always share your vision.
  4. Can I see where I’ll sit?  This actually wraps up a bunch of questions into one.  Do they even know?  Do they have assigned seating, and if so, can you influence it?  Did you meet the people you’ll be sitting next to in the interview process?  Is it clean and is there natural light?  Is it quiet, or is there music, or just generally noisy?  And most important: is it a place you really want to sit 2000+ hours a year?  Your workspace is your home away from home, and there is an enormous range between “we just have an awesome office full of diverse regions and you can sit anywhere” and “here’s your awesome standing desk with big monitor”.  There are no right answers, but put some real thought into what’s important to you and make sure to factor that in.
  5. And last but not least: What do I want to do accomplish in my life, and how does this help me down that path?  This is one you should ask yourself — every day, about everything you do.  Our industry is unique in that anybody reasonably good can make money, and with a bit of luck you can make a lot.  But all the ways you can spend it get boring much faster than you’d expect.  Hard problems and cool technology will keep you occupied for a while, but there are always more problems and more technology, and eventually even those get old.  If you’re great — or if you want to be great — your motivations likely run deeper.  You have some kind of secret, quiet ambition that is so grand, so audacious, that you are embarrassed to say it out loud.  So big that even if you devote the rest of your life to it, you still couldn’t finish it.  But so important to you, that you genuinely want to try anyway.  Knowing what you want out of life dramatically increases the odds you’ll get it, so figure out what that is, figure out what’s preventing you from doing it right now, and then evaluate whether this startup is going to give you the tools to take the next step.  Maybe you lack money, or maybe it’s skill, but more likely you just lack the confidence that comes from being part of a successful team.  Make sure your next job is the launchpad for the rest of your life, and not just a landing pad with an extended layover.

I picked five to make the subject link click-baity, but really you should ask a lot of questions.  If you’re great, you can probably work anywhere you want.  Interview them as much or more than they’re interviewing you.  If they’re great too, they’ll recognize you’re worth it and thank you for the effort.

[Edit] Lots of great discussion about interviewing advice over at HackerNews.

Nomming on pizza in Zagreb, Croatia.  Photo by Robert.

Nomming on pizza in Zagreb, Croatia. Photo by Robert.

When I moved to San Francisco in 2011 for Expensify, I was moving across the country to a city where I knew a whopping two people amongst a city of 825,000. Those two really adopted me at first and helped introduce me to a number of great people. However, being passionate about what I was doing had me working really late hours and so our schedules just never lined up.

And with that excuse, I started to worry that I wasn’t going to make friends in the city. I found myself constantly talking to my coworkers about this: over lunch, after work at the bars, during spontaneous weekend adventures, and even during the trips overseas for our annual offshore where we’d spend a month+ working and traveling together.

And then I realized that I had actually made a ton of really close friends in the city – and over the course of my years at Expensify, I’d been a part of hiring and relocating them to San Francisco as well.

The thing is, these aren’t just close friends now. I work with family. Expensify is family. After spending 50+ hours a week in the office with my coworkers getting sh*! done, you might be surprised at how many hours we spend out of the office together. Outside of the apartments that Expensifiers live in together (my roommate included), almost every day people are meeting up outside the office and spending time together. It doesn’t matter what team you’re on, what your role is in the company, or even if you don’t really drink alcohol – we work hard, play hard, together.

Over half of the Expensify team in San Francisco relocated to the Bay Area from a different country or time zone – almost all of them not knowing any more people here when they first arrived than I did. But I would say that every single one of them now has a group of people they can rely on both in the office and outside.

When I describe Expensify to outsiders, I use the phrase “family style” – we’ve no private offices, cubicles, or even dedicated desks. What we do have are three mega tables that we all just plop our laptops down on when we show up in the morning and work from there. Or if you’re needing to relax a bit more, we’ve mini living rooms and couches scattered throughout the office that you can take a call in, have a quick meeting, or even a quick nap. We leverage that flexibility to sit where it makes the most sense for whatever you’re working on. If you’re an engineer and needing to build something for a sales guy – just pick up your laptop and go sit next to him. This works for every team and every person in the office. Even our CEO David has his legs kicked up at these tables throughout the day. And almost no one ever gets lunch without a posse of the team with them.

Carlos and the Pumpkin

Hailing from Venezuela, Carlos had never carved a pumpkin before starting at Expensify, so we fixed that. Photo by Thomas.

But that family style extends way after we leave the office. When we found out the majority of our international team hadn’t celebrated a proper Halloween before, we got a van and picked pumpkins for carving. We celebrate birthdays together. We meet parents when they visit the city. I’ve spent almost every single one of the last three years’ holidays with this family and we have had a great time.

We push each other in and out of the office to grow as professionals and in life. Our relationship is borderline unhealthy but we leverage that to get better context in how people are doing and how to continue making Expensify awesome together.

This work/life balance isn’t for everyone, but it works for us. And to me that’s what matters the most. There are so many different types of workplaces out there that can meet different needs. But we’ve really tried to figure out what works best for the people we love to work with, and have spent years building a culture of community and family that I’m personally so proud to be a part of.  Expensify: Expense reports that don’t suck and a work environment that doesn’t either.