Expensify Blog

Wow! April was a big month for Expensify in the media. Our Bitcoin Reimbursement announcement drew a large amount of attention. Our CEO, David Barrett was asked to speak on 3 different news programs: FoxBusiness, SkyNews, and CNBC! In addition to television appearances, we had a multitude of tech blogs write up on why our Bitcoin adoption was a big deal. Here are some highlights:

• Wall Street Journal - Receipt Wranglers: Expensify Virtual Accountant
• Bloomberg Businessweek - Expensify brings Bitcoin to Main Street
• TechCrunch - Expensify adds Bitcoin support as PayPal Alternative
• PCWorld - Bitcoin Surges past $100, gains Expensify support
• International Business Times - Bitcoins Accepted by Expensify, Surpasses 20 National Currencies
• VentureBeat - Expensify Adds Bitcoin Support
• The Next Web - Expensify Adds Bitcoin to Offer Instantaneous International Reimbursement
• CITE World - Bitcoin Takes Big Step into Mainstream with Expensify Support
• TechFaster - Combining Two Things that Don’t Suck, Expensify Now Supports Bitcoin
• Falkvinge - Why Expensify Endorsing Bitcoin is a Really Big Deal

Another exciting development this past month was our Office Dog, Lemony Cricket, winning the Muse Ultimate Office Mate competition! She faced fierce competition from other adorable dogs from the offices of Global Giving, AOL, Uber, & Zoosk. In the end, Lemony emerged victorious as the Ultimate Office Mate! Thank you to everyone who supported our beloved pup. We’ve included some adorable GIFs of Lemony as thanks!

                             

                           

We’ve added a ton of cool features to Expensify so far this year and we’re just getting started. We have a HUGE announcement coming up this month, so stay tuned. Expensify is only getting better! Thanks for your support and have a great weekend!

For today’s outrageously informative blog post, we’ll walk you through how to integrate Expensify with Sage 50 (PeachTree), a leading cloud-based accounting package.  Sage 50 integration is available on the Corporate plan, and you can also reference this post if you’d like to integrate Expensify with other cloud-based accounting software, such as Intacct.

Prerequisites

These basic company setup steps will need to be taken care of before setting up the Sage 50 integration:

1. Upgrade to the Corporate plan.
2. Create a company expense policy.
3. Ensure that GL codes are asscoiated with Categories.
4. Input Job ID as a report field.

Configuring Your Export File

After you’ve chosen the appropriate format for your company, you will create an export format that any admin can use to export expense data into Sage 50. To do so, visit the Export format section of your policy settings (Settings > Policies > [Policy Name] > Export Formats). Once the export format has been created, any policy admin can choose a group of reports and export them to the custom spreadsheet file.

While accounting setups are different from company to company, below is an example of a flow that has proven to be successful when integrating Expensify with Sage 50. If this does not seem to be the case for your company please write us at help@expensify.com and we will be sure to get you set up.

Example Export Format:

And you’re done! At the end of your next accounting period, you can choose which reports you’d like to export, and the information will be formatted for direct upload into Sage 50. We also have some great materials to help user import the file from Expensify into Sage 50, which you can find here. Let us know if you have any questions!

 

We are very happy to announce that our Reconciliation and Central Administration feature is now available for American Express corporate cards. Because the feature was such a hit with commercial card holders, we decided to develop for Corporate cards as well.

What is Reconciliation?

Reconciliation is an amazing feature that allows you to reconcile all incurred Corporate card expenses against reported expenses in the Expensify system. Reconciliation reports are great for seeing any outstanding Corporate card expenses that still need submitting as well as those that have already been submitted.

What is Central Administration?

Central Administration allows you to import your American Express Corporate card feed and assign individual cards to specific employees. Simply put, it allows you to populate your employees’ Expensify accounts with expenses from their individual Corporate card without any work from them.

How do I get it?

To have access to Reconciliation reports, you will need to upgrade to the Corporate plan and enable Domain Control. From the admin tab, you will then be able to centrally assign your corporate cards to users.

How do I create a Reconciliation report?

To run a reconciliation report, go to the Reconciliation section of the Admin tab. From there you will be able to choose the date range and criteria over which you want to run reconciliation. After the process is done, you will be able to download a spreadsheet file of your reconciliation report.

To get a more detailed run-through, please visit our Reconciliation help page.

How do I use Central Administration?

First, go to the Company Cards section of the Admin tab. Here click the ‘Import…’ button, enter your Master Account login credentials, and click ‘Sign in’.

Once your card has connected successfully, simply type in the user’s email address you’d like to add the card to, enter the last 4 digits of the corresponding card and click the ‘assign a commercial card’ button. For more information on Central Administration, please check out our Company Cards help page.

If you would like to see Reconciliation and Central Administration made available for another credit card provider, email us at help@expensify.com .

Happy Monday!

Name: Giorgio Calderolla

Alter ego/Nickname: Gio (Joe?), Giorgio Calderoni, Giorgio Maccheroni, Giorgio Lasagne, Giorgio (insert italian food or car make here)

Hometown: Conegliano, Italy (sometimes referred to as “Coney Island” by the locals)

Role(s) at Expensify: Frontend developer, other things developer, prettifier, speedifier, streamlinizer. Being confused and hesitating for a second when someone tries to high five me, and then saying ‘ouch’.

Expensify Start Date: I don’t know! It’s all a blur. Somewhere between August ’11 and February ’12.

What is/has been your favorite project to participate on at Expensify?

I enjoy working on the little UI things. And the big UI things.

Life outside of Expensify?

Mostly app-making. Occasionally, city-exploring.

How much wood could a woodchuck chuck if a woodchuck could chuck wood?

Oh shut up.

Favorite Beverage

I think I’m a machine that transforms coffee into code. Not sure how it works. But I can tell you for sure that when the coffee drinking stops, the code writing and other associated activities stop as well. It’s weird. So to answer your question, my favorite beverage is coffee.

If a tree falls in a forest and no one is around to hear it, does it make a sound?

Yes. A ‘craaaack thump’ sound.

Favorite flower?

Spaghetti Trees

What’s your present state of mind?

Zen-like.

We’ve bolstered our current SalesForce integration (available on our Team plan) by adding support for FinancialForce. The FinancialForce integration is available on our Corporate plan. Once you’ve setup these integrations up in your account you can:

• Centrally manage users
• Tag expenses with Salesforce Accounts and Opportunities
• Tag expenses with FinancialForce Projects and Milestones
• Submit expense reports directly to FinancialForce

Connect Expensify and Salesforce + FinancialForce

Visit our Salesforce page for instructions on how to connect Expensify and Salesforce. Once you have connected to Salesforce, you will see the option to have reports submitted to FinancialForce PSA as well. Make sure that you have created an Expensify account that has been upgraded to the Corporate plan prior to connecting to Salesforce.

Once you’ve checked the ‘Submit reports to FinancialForce PSA’ checkbox, you’ll need to click ‘Sync’ in order to initiate the connection. Whenever you need to update your FinancialForce connection, simply navigate to this page and click ‘Sync’ again.

Using the Salesforce + FinancialForce Integration

Step 1: Tag Expenses

If tags are synced, all Accounts, Opportunities, Projects, and Milestones will be pulled into Expensify from Salesforce/FinancialForce and available for users as tags.

Step 2: Submit to FinancialForce

Our submit to FinancialForce feature works just like submitting any other report in Expensify. Just click submit and you’re done!

As always, if you have any questions at all about connecting to SalesForce or FinancialForce, please reach out to our support team at help@expensify.com. Welcome to the Force!

Expensify has three geo-redundant, realtime-replicated datacenters — each of which holds more than enough hardware to power the full Expensify site, and all three combined should be massive overkill.  So why has Expensify been so slow these past few days?  A few reasons, actually:

Massive Traffic Spike.

In short, this month is off with a bang.  Nearly every day has set a new all-time traffic record, blowing away all historical averages.  Being featured in USA Today, NBC News, and the Wall Street Journal definitely contributes.  And traffic is always up at the start of the week, and the start of the month, so that’s a double-whammy.  But I think the biggest contributing factor is our friend the IRS: the April 15th deadline is looming large among our millions of users, and they are starting to take it very seriously.  This is bringing out a new user behavior we don’t have a ton of experience with: people uploading receipts in bulk by the hundreds (or even *thousands*), putting new stresses on the system.   All told, it’s causing the site to break in new and exciting ways we’ve never experienced before.  Most of those ways only affect small numbers of users, but today was different.

The Short Story.

A workaround is in place and a permanent fix is expected next week.

The Long Story.

Remember those three realtime-replicated datacenters?  ”Replicated” is a key word.  Our proprietary WAN database replication technology (which we’re hoping to open source) has been dutifully and reliably powering the site for years.  In a very general way, every command is processed like this:

1. Webserver receives a request from the web browser
2. Webserver sends the command to one of our 5 database servers
3. If it’s a “read only” command, the database processes the command and immediately responds
4. If it’s a “read/write” command, then it “escalates” the command to the “master” database — the server that’s in charge of coordinating all writes.
5. The master processes the command using a transaction
6. The master sends the SQL of the command to the child databases, who re-execute the transaction locally
7. The children all respond “looks good!” as soon as they finish their transaction
8. Once half the children respond with the all-clear, the master commits the transaction, notifies the children to commit, and notifies the original database that its command has been processed.
9. The original database receives notification from the master that the command is done, and then in turn responds to the websever with the results.

All this normally happens in around 100ms, with the replication stage (steps 6-8) taking about 40ms.  Today, it started taking on the order of 14,000ms, with replication taking about 200ms.  Furthermore, all our databases were operating at 100% CPU — even though performance had slowed to a crawl, not to mention those servers are rated for far more traffic than they were receiving.  All three of these seemed impossible on the surface, and even more impossible given that we didn’t change any hardware or software related to any of these systems.  What could make this so slow?  Three things:

• A cache
• A list
• A socket

The first seemed really obvious: let’s prioritize the important commands over the less important commands.  In order to make the Report page load quick, we precalculate a variety of aspects about the report and store them in a cache.  But updating the cache isn’t as important as actually responding to a user request, so we deprioritize it.  This is normally fine — the cache update is put on the end of the master’s command list, and when there is a lull we update them all, without delaying any realtime user actions.  What could possibly go wrong with this great optimization?

The problem is if there is no lull.  Recall that we’re seeing unprecedented levels of traffic, meaning activity levels that were previously brief spikes have become the new norm.  This means for long periods every day, we get so many high-priority commands that the master never has a chance to process the low-priority cache-update commands (known as “starvation”).  This should normally be fine — the low-priority commands would wait until the spike finished, and then everything should update fine.  No problem.

However, a byproduct of that “no problem” is an extremely large queue of escalated commands — the children sent a huge number of cache commands to the master.  This should normally be fine, except for when I wrote that code (in 2009) I assumed the number of escalated commands from any child database would always be very small.  So I used a list to hold it, meaning whenever any database got a response to an escalated command, it would need to iterate over that list to find a match.  (And there were several other occasions that it iterated over that list.)  Normally this would be fine.  But if that list gets long — like, really, really long — then iterating over that list at extremely high frequencies gets very expensive.  My bad.

And this exacerbated the third issue.  Recall that when the master goes to commit a transaction, it waits for approval from two children.  Normally that happens incredibly fast.  However, when the children started seeing their CPU eaten up by the list iteration, that caused our load to increase substantially — even though it wasn’t doing anything useful.  This had the unfortunate side effect of making the child databases just generally slow down,  meaning it was slow to process messages from peers.  In particular, it meant that the child servers started processing replication commands slower than it would otherwise.

The sum of all three created a vicious cycle: as the child server’s CPU increased, the replication speed decreased, causing the backlog of low-priority command to grow, repeat.

The Workaround

Once we understood this, the first thing we did was disable that cache-update command, then restart all the child databases, and then restart the master.  This cleared the backlog, caused CPU to drop, and caused replication speeds to increase back up to normal.  Problem solved… sorta.

The Quick Solution

However, replication speeds were still too slow.  Replication is highly influenced by network latency.  The latency between our Los Angeles and Santa Clara datacenters is around 10ms.  But between either of those and Las Vegas is 40ms.  It hadn’t always been this way: we’d known that there was some problem that slowed down that link, but it hadn’t been a major problem… until our traffic started going up dramatically in March.  This means that replication has slowed to a point where at peak traffic it can’t keep up.  This causes a backlog of write commands that delays the processing of commands.  (The commands individually are going at full speed, but delayed.)  To mitigate this, we moved the master from Las Vegas to Santa Clara, meaning that it could replicate quickly down to LA (without waiting for the slower Las Vegas link), speeding up replication speeds by about 4x and making all right in the world.

The Long Term Solution

But that’s still not a great solution: our report-caching command is still disabled, and replication speeds are still not where we’d like them.  So we still have work to do:

1. Stop using a list of escalated commands, and replace with a “map” — allowing fast lookup without the CPU problem.
2. Re-enable the report caching command, which will speed up the Reports page.
3. Reduce latency between our datacenters.  This can be done with a dedicated fiber connection, which we’re investigating now.  This will generally increase performance across the board, as well as provide a bigger buffer against temporary replication speed problems.

Conclusion

The final conclusion is that we’re terribly sorry about the delays.  We agonize over every problem encountered by our users, and are always disappointed when something slips through the cracks.  We’ve invested a tremendous amount of time, people, and hard cash into the best possible solution, and we do everything we can to anticipate and prevent any conceivable problem.  But this is complex stuff, and sometimes our best efforts come up short.  Thank you for your patience and I appreciate your understanding as we continue to grow this company by leaps and bounds.

One of the things we pride ourselves on at Expensify is flexibility. Our users have a very diverse set of needs and our goal is to find a simple solution that meets all of them. While the QuickBooks integration that we have built is great for most users, we realize that some people like an extra level of customization. This is why we are taking the time to let you know about this custom QuickBooks integration option.

Why use the Custom Method

The custom QuickBooks integration involves using the third party software Transaction Pro Importer made by Baystate Consulting. This software is available for a one-time cost of $199.95. This method creates a true 2-way connection in which accounts that exist in Expensify can be created in QuickBooks. It will allow full customization of the way information captured in Expensify will appear in QuickBooks and is available for both QuickBooks Online and QuickBooks Desktop.

Some common reasons for wanting to use the custom QuickBooks method:

• You want to export as something other than a check, bill, credit card charge, or general journal entry.
• You need to map the merchant name of expenses in Expensify to vendors in QuickBooks.
• You need to capture class and customer/job at the expense line-item level. This will also require enabling multi-level tagging.
• You want to export as items instead of expenses.
• Any other customization that Expensify’s traditional export method doesn’t handle.

How it Works

You will first create a CSV export format based on the information that you would like to import into QuickBooks. Transaction Pro Importer will let you know the needed information for your given import type (check, bill, etc.). You will then export expense reports to that export format.

You will then import that CSV file to QuickBooks using Transaction Pro Importer.

That’s it! You have now fully customized your QuickBooks integration. For full instructions please visit our custom QuickBooks help page. As always, if you have questions, please don’t hesitate to email us at help@expensify.com.