As you might know, we’ve been hiring a steady trickle of engineers, and we’re always eager for more. But we find ourselves without a true sysadmin, and honestly, it’s starting to show.
A long time ago Witold and I joked about scalability being “a problem we’d love to have”. When you are struggling to get anybody to use your product, it just seems like the last thing on your mind. Well, as much as I love the fact that there are hundreds of thousands of people using Expensify — it’s starting to add up to a real problem. Servers are getting overloaded. Databases are getting slow. I think the final straw was when I realized it was going to take over a week — sending mail as fast as we possibly could, 24/7 — just to send out a newsletter to all our users. So I love it, but it’s still a problem, and one that we’re not well equipped to solve.
Sure, we can make due — the great thing about being a programmer is there’s no problem we can’t solve. But our solutions are sometimes, shall we say, suboptimal. We can build anything from scratch, but sometimes it’s better to just install and configure off the shelf.
So this is my request: can you help us hire a sysadmin? Someone ideally familiar with Amazon Web Services (S3, EC2, SES), Ubuntu, lighttpd, PHP, postfix, ssh, and all the other myriad tools of a modern web infrastructure. But also someone with a deep intuitive understanding of disk and memory caching on virtual machines (where every virtual hard drive is actually layered on top of a *physical* hard drive — so there are two layers to manage), is well versed in the consequences of WAN versus LAN replication (high latency, low reliability), and who brings a deep reservoir of enthusiasm and expertise tearing apart a programmer system and rebuilding it the Right Way.
We’ll always need engineers. But engineering isn’t everything. Help us find a sysadmin and you’ll have my eternal gratitude. To apply, please ask them to read up on us on our jobs page and send answers to the following questions to jobs@expensify.com:
- When did you get started with computers and why?
- Contrast the pros and cons of virtual, dedicated, or colocated servers for a company like Expensify?
- What’s the largest, fastest, most reliable, most complicated environment you’ve ever managed?
- What’s a Linux command or tool that you think is more awesome than most people realize, and why?
- Based on what you can determine from outside the network, what would you suggest we change and why?
- When you receive a text message, what’s your first thought?
Thanks, I really appreciate your help!
Expensify Blog 