Expensify Blog

Giorgio with his idol, Ronald McDonald, in Thailand

Name: Giorgio Calderolla

Alter ego/Nickname: Gio (Joe?), Giorgio Calderoni, Giorgio Maccheroni, Giorgio Lasagne, Giorgio (insert italian food or car make here)

Hometown: Conegliano, Italy (sometimes referred to as “Coney Island” by the locals)

Role(s) at Expensify: Frontend developer, other things developer, prettifier, speedifier, streamlinizer. Being confused and hesitating for a second when someone tries to high five me, and then saying ‘ouch’.

Expensify Start Date: I don’t know! It’s all a blur. Somewhere between August ’11 and February ’12.

What is/has been your favorite project to participate on at Expensify?

I enjoy working on the little UI things. And the big UI things.

Life outside of Expensify?

Mostly app-making. Occasionally, city-exploring.

How much wood could a woodchuck chuck if a woodchuck could chuck wood?

Oh shut up.

Favorite Beverage

I think I’m a machine that transforms coffee into code. Not sure how it works. But I can tell you for sure that when the coffee drinking stops, the code writing and other associated activities stop as well. It’s weird. So to answer your question, my favorite beverage is coffee.

If a tree falls in a forest and no one is around to hear it, does it make a sound?

Yes. A ‘craaaack thump’ sound.

Favorite flower?

Spaghetti Trees

What’s your present state of mind?

Zen-like.

We’ve bolstered our current SalesForce integration (available on our Team plan) by adding support for FinancialForce. The FinancialForce integration is available on our Corporate plan. Once you’ve setup these integrations up in your account you can:

• Centrally manage users
• Tag expenses with Salesforce Accounts and Opportunities
• Tag expenses with FinancialForce Projects and Milestones
• Submit expense reports directly to FinancialForce

Connect Expensify and Salesforce + FinancialForce

Visit our Salesforce page for instructions on how to connect Expensify and Salesforce. Once you have connected to Salesforce, you will see the option to have reports submitted to FinancialForce PSA as well. Make sure that you have created an Expensify account that has been upgraded to the Corporate plan prior to connecting to Salesforce.

Once you’ve checked the ‘Submit reports to FinancialForce PSA’ checkbox, you’ll need to click ‘Sync’ in order to initiate the connection. Whenever you need to update your FinancialForce connection, simply navigate to this page and click ‘Sync’ again.

Using the Salesforce + FinancialForce Integration

Step 1: Tag Expenses

If tags are synced, all Accounts, Opportunities, Projects, and Milestones will be pulled into Expensify from Salesforce/FinancialForce and available for users as tags.

Step 2: Submit to FinancialForce

Our submit to FinancialForce feature works just like submitting any other report in Expensify. Just click submit and you’re done!

As always, if you have any questions at all about connecting to SalesForce or FinancialForce, please reach out to our support team at help@expensify.com. Welcome to the Force!

Expensify has three geo-redundant, realtime-replicated datacenters — each of which holds more than enough hardware to power the full Expensify site, and all three combined should be massive overkill.  So why has Expensify been so slow these past few days?  A few reasons, actually:

Massive Traffic Spike.

In short, this month is off with a bang.  Nearly every day has set a new all-time traffic record, blowing away all historical averages.  Being featured in USA Today, NBC News, and the Wall Street Journal definitely contributes.  And traffic is always up at the start of the week, and the start of the month, so that’s a double-whammy.  But I think the biggest contributing factor is our friend the IRS: the April 15th deadline is looming large among our millions of users, and they are starting to take it very seriously.  This is bringing out a new user behavior we don’t have a ton of experience with: people uploading receipts in bulk by the hundreds (or even *thousands*), putting new stresses on the system.   All told, it’s causing the site to break in new and exciting ways we’ve never experienced before.  Most of those ways only affect small numbers of users, but today was different.

The Short Story.

A workaround is in place and a permanent fix is expected next week.

The Long Story.

Remember those three realtime-replicated datacenters?  ”Replicated” is a key word.  Our proprietary WAN database replication technology (which we’re hoping to open source) has been dutifully and reliably powering the site for years.  In a very general way, every command is processed like this:

1. Webserver receives a request from the web browser
2. Webserver sends the command to one of our 5 database servers
3. If it’s a “read only” command, the database processes the command and immediately responds
4. If it’s a “read/write” command, then it “escalates” the command to the “master” database — the server that’s in charge of coordinating all writes.
5. The master processes the command using a transaction
6. The master sends the SQL of the command to the child databases, who re-execute the transaction locally
7. The children all respond “looks good!” as soon as they finish their transaction
8. Once half the children respond with the all-clear, the master commits the transaction, notifies the children to commit, and notifies the original database that its command has been processed.
9. The original database receives notification from the master that the command is done, and then in turn responds to the websever with the results.

All this normally happens in around 100ms, with the replication stage (steps 6-8) taking about 40ms.  Today, it started taking on the order of 14,000ms, with replication taking about 200ms.  Furthermore, all our databases were operating at 100% CPU — even though performance had slowed to a crawl, not to mention those servers are rated for far more traffic than they were receiving.  All three of these seemed impossible on the surface, and even more impossible given that we didn’t change any hardware or software related to any of these systems.  What could make this so slow?  Three things:

• A cache
• A list
• A socket

The first seemed really obvious: let’s prioritize the important commands over the less important commands.  In order to make the Report page load quick, we precalculate a variety of aspects about the report and store them in a cache.  But updating the cache isn’t as important as actually responding to a user request, so we deprioritize it.  This is normally fine — the cache update is put on the end of the master’s command list, and when there is a lull we update them all, without delaying any realtime user actions.  What could possibly go wrong with this great optimization?

The problem is if there is no lull.  Recall that we’re seeing unprecedented levels of traffic, meaning activity levels that were previously brief spikes have become the new norm.  This means for long periods every day, we get so many high-priority commands that the master never has a chance to process the low-priority cache-update commands (known as “starvation”).  This should normally be fine — the low-priority commands would wait until the spike finished, and then everything should update fine.  No problem.

However, a byproduct of that “no problem” is an extremely large queue of escalated commands — the children sent a huge number of cache commands to the master.  This should normally be fine, except for when I wrote that code (in 2009) I assumed the number of escalated commands from any child database would always be very small.  So I used a list to hold it, meaning whenever any database got a response to an escalated command, it would need to iterate over that list to find a match.  (And there were several other occasions that it iterated over that list.)  Normally this would be fine.  But if that list gets long — like, really, really long — then iterating over that list at extremely high frequencies gets very expensive.  My bad.

And this exacerbated the third issue.  Recall that when the master goes to commit a transaction, it waits for approval from two children.  Normally that happens incredibly fast.  However, when the children started seeing their CPU eaten up by the list iteration, that caused our load to increase substantially — even though it wasn’t doing anything useful.  This had the unfortunate side effect of making the child databases just generally slow down,  meaning it was slow to process messages from peers.  In particular, it meant that the child servers started processing replication commands slower than it would otherwise.

The sum of all three created a vicious cycle: as the child server’s CPU increased, the replication speed decreased, causing the backlog of low-priority command to grow, repeat.

The Workaround

Once we understood this, the first thing we did was disable that cache-update command, then restart all the child databases, and then restart the master.  This cleared the backlog, caused CPU to drop, and caused replication speeds to increase back up to normal.  Problem solved… sorta.

The Quick Solution

However, replication speeds were still too slow.  Replication is highly influenced by network latency.  The latency between our Los Angeles and Santa Clara datacenters is around 10ms.  But between either of those and Las Vegas is 40ms.  It hadn’t always been this way: we’d known that there was some problem that slowed down that link, but it hadn’t been a major problem… until our traffic started going up dramatically in March.  This means that replication has slowed to a point where at peak traffic it can’t keep up.  This causes a backlog of write commands that delays the processing of commands.  (The commands individually are going at full speed, but delayed.)  To mitigate this, we moved the master from Las Vegas to Santa Clara, meaning that it could replicate quickly down to LA (without waiting for the slower Las Vegas link), speeding up replication speeds by about 4x and making all right in the world.

The Long Term Solution

But that’s still not a great solution: our report-caching command is still disabled, and replication speeds are still not where we’d like them.  So we still have work to do:

1. Stop using a list of escalated commands, and replace with a “map” — allowing fast lookup without the CPU problem.
2. Re-enable the report caching command, which will speed up the Reports page.
3. Reduce latency between our datacenters.  This can be done with a dedicated fiber connection, which we’re investigating now.  This will generally increase performance across the board, as well as provide a bigger buffer against temporary replication speed problems.

Conclusion

The final conclusion is that we’re terribly sorry about the delays.  We agonize over every problem encountered by our users, and are always disappointed when something slips through the cracks.  We’ve invested a tremendous amount of time, people, and hard cash into the best possible solution, and we do everything we can to anticipate and prevent any conceivable problem.  But this is complex stuff, and sometimes our best efforts come up short.  Thank you for your patience and I appreciate your understanding as we continue to grow this company by leaps and bounds.

One of the things we pride ourselves on at Expensify is flexibility. Our users have a very diverse set of needs and our goal is to find a simple solution that meets all of them. While the QuickBooks integration that we have built is great for most users, we realize that some people like an extra level of customization. This is why we are taking the time to let you know about this custom QuickBooks integration option.

Why use the Custom Method

The custom QuickBooks integration involves using the third party software Transaction Pro Importer made by Baystate Consulting. This software is available for a one-time cost of $199.95. This method creates a true 2-way connection in which accounts that exist in Expensify can be created in QuickBooks. It will allow full customization of the way information captured in Expensify will appear in QuickBooks and is available for both QuickBooks Online and QuickBooks Desktop.

Some common reasons for wanting to use the custom QuickBooks method:

• You want to export as something other than a check, bill, credit card charge, or general journal entry.
• You need to map the merchant name of expenses in Expensify to vendors in QuickBooks.
• You need to capture class and customer/job at the expense line-item level. This will also require enabling multi-level tagging.
• You want to export as items instead of expenses.
• Any other customization that Expensify’s traditional export method doesn’t handle.

How it Works

You will first create a CSV export format based on the information that you would like to import into QuickBooks. Transaction Pro Importer will let you know the needed information for your given import type (check, bill, etc.). You will then export expense reports to that export format.

You will then import that CSV file to QuickBooks using Transaction Pro Importer.

That’s it! You have now fully customized your QuickBooks integration. For full instructions please visit our custom QuickBooks help page. As always, if you have questions, please don’t hesitate to email us at help@expensify.com.

Expensify now integrates with Sage 50 (formerly Sage Peachtree) using our customized CSV export format. Now your employees can easily create and submit expense reports in compliance with Sage 50 data requirements!

Prerequisites

These basic company setup steps will need to be taken care of before setting up your Sage 50 integration:

1. Upgrade to the Corporate plan.
2. Create a company expense policy.
3. Ensure that GL codes are asscoiated with Categories.
4. Add Vendor and Job ID as Report Fields.

Configuring Your Export File

After you’ve chosen the appropriate format for your company, you will create an export format that any admin can use to export expense data into Sage 50. To do so, visit the Export format section of your policy (Settings > Policies > [Policy Name] > Export Formats). Once the export format has been created, any policy admin can choose a group of reports and export them to the custom spreadsheet file.

If you have any clarifying questions about your Sage 50 export format, please contact us at help@expensify.com.

Approved: past participle, past tense of ap·prove (Verb) - Officially agree to or accept as satisfactory: “Your expense report has been approved and you will now be reimbursed.”

Approver: (Noun) One who has the authority to approve: “I have submitted my expense report to an approver.”

As an Approver in Expensify you are not restricted to merely “Approving” reports (yay!) or “Rejecting” reports (boo!) based on minor mistakes. Approvers have some pretty substantial abilities when it comes to reports submitted to them. You can change anything in the report EXCEPT merchant name, date, amount of expense or currency.  That means you  DO have the ability to change:

• Expense categories
• Expense tags (including multi-level tagging!)
• Report titles
• Report fields
• Even the Report Policy itself!

Let’s say for example you have an employee, Sally, who keeps on forgetting that she must categorize her expenses within the correct policy before submitting her report.  You don’t want to reject her report or expenses, send it back to her, and risk the potential for hurting her feelings :’(  But you certainly can’t approve it with those violations either…

In come the Approver Super Powers!  You can change the Categories or Tags of her expenses until the report is ready to be approved.

Now doesn’t that report look better? Don’t you feel powerful?  And, no feelings were hurt in the process! If you have any questions about other unknown super powers in Expensify, please contact us at help@expensify.com.

*P.S If you’re concerned about extraneous changes being made to reports by all-powerful approvers, you can enable digital signatures. With this enabled, a PDF of the report will be saved with every report action.