On December 26th, OwnerRez experienced a database corruption issue with our cloud service provider.
We saw the issue right away and started working on it, but there was a domino effect that grew significantly beyond our initial expectations. Our engineering staff has been recalled from vacation, connecting in from overseas, etc. to diagnose and resolve the problems as quickly as possible.
At present, there are no indications of any hostile activity or attacks, and no reason to believe that any previously gathered data has been lost. We are currently operating on a restored system with up-to-date database backups that occurred at the same time the issue started to develop.
We will be posting information and analysis (here in this same blog post), and holding a public webinar, as soon as we have a more thorough understanding of what went wrong and future preventive actions.
In the meantime, here is a list of common questions that have been coming in. This list will grow over time as we add more.
- Why did it happen? What caused it?
- Were others affected, is this a widespread issue?
- Was any data lost? Doesn't restoring a backup mean some recent data was lost?
- Was any sensitive data (credit card, phone number, PII) lost or taken?
- Don't you have redundancies in place to prevent this from happening?
- Does Airbnb automatically reactivate property listings, or must that be done manually?
- During the outage, I disconnected Airbnb. Now what?
- I did a full resync for Airbnb, but nothing changed.
- Where are my Booking.com bookings? They aren't showing up on my calendar.
- OwnerRez says my Airbnb properties are fully synced and online, but, I still can't see them on Airbnb.
Why did it happen? What caused it?
We are still investigating all the details, and there are puzzling issues we still don't fully understand, but what we can say is this...
This past Saturday, as part of a routine update, we upgraded some database servers with a new configuration that we had used extensively for many months. The configuration was a very routine thing and, again, something that our engineers have used in their own environments (and elsewhere in testing/staging) for a while. There was no reason to believe the update or new configuration would cause anything out of the ordinary.
From Saturday until Monday, the new configuration started to corrupt data. We're not sure exactly when or why (this is the part still being investigated) but several database areas began to be unresponsive. The entire platform started to go down on Monday, and our remediation/failover processes started to kick in, but there was a domino effect, and the remediation/failover processes were affected similarly as the system bounced from up to down and back again.
The investigation into the bad database configuration is ongoing. By all accounts, it should not have happened. Our cloud provider has service guarantees with us, so we are working with them on the investigation.
Were others affected, is this a widespread issue?
No, not that we know of. This is isolated to something we did (updating a database configuration) to our database servers with our cloud provider. But the obvious question is - what about other businesses that used the same database configuration?
Yes, indeed - we are wondering the same thing. We purposefully watch to see if software updates by third parties are stable before upgrading our server configurations. In fact, we purposefully waited on this database configuration (from back in March) because there were some known issues with it affecting servers. We skipped over it and used a different configuration that was reported to be stable and in use by others, and we used that configuration in test/stage environments for several months before moving forward. As mentioned above, we are very puzzled by why it happened and by all accounts, it should not have.
(And no, the Southwest Airlines debacle is not related to this. At least, we highly doubt it.)
Was any data lost? Doesn't restoring a backup mean some recent data was lost?
The short answer is: no lost data, but we are still investigating and there may be discrepancies with third parties.
The reason we say "no lost data" is because we have data redundancies in place that save data to multiple places at the same time. When our primary systems go down, we still have access to other systems. When we rebuilt and restored the underlying configurations, we took data from secondary systems that were up to date as of the time the issue happened.
After the recovery period finished, we went back and compared the new data with the corrupted data. There were no discrepancies in terms of missing records. In other words, no records were showing in the old corrupted version that didn't exist in the new recovered version.
That being said, there are many channels and 3rd parties that we connect with that also send data back and forth and could have "sent" or "received" some data that never got through during the outage. So a channel or 3rd party might show something different than what OwnerRez does, which means the data is incorrect on one side or another (or both). Messaging could be another example of this. Gmail and other messaging services watch volume and throttle accordingly. When the final recovery effort fully kicked in, a large firehose of queued messaging went out (eg. from you to guests, or from OR to you). Gmail may have throttled or dropped some of that due to volume concerns on their side. We have not seen signs of that, but it is possible.
Also, there were intermittent "up" periods, early in the recovering period, when you (or a third party) may have changed data that ultimately did not get saved when we did the final recovery effort. So there too, data may be missing.
Was any sensitive data (credit card, phone number, PII) lost or taken?
No. This was not the result of a security breach nor any outside influence. Nothing was hacked or stolen. This was a database configuration issue where a seemingly-insignificant update led to data corruption and then domino'd elsewhere as our fail-over processes started responding.
Don't you have redundancies in place to prevent this from happening?
Yes, many. Our engineering team spends a lot of time (and on an ongoing regular basis) managing and analyzing our infrastructure. We have lots of tooling and logging in place that gives us visibility into errors, slow responses, spikes, partner issues, background queues, and much more. OwnerRez is large and full of many moving parts, and in each of those moving parts, we store data where it can be replicated and archived stably as it grows.
Yesterday, we saw the issue immediately and our fail-over process kicked into gear, but the database configuration affected every remediation effort that swung into place. We were forced to pause everything completely, get to the root of the problem, and rebuild configurations before turning the firehose back on.
The bad configuration was, itself, something we tested and used extensively for months beforehand, so we are still puzzled by (and investigating) why it happened. But we made the decision to stop the failover/automation and roll back to a configuration that we thought would make a bigger impact. We knew it would take a couple of hours, but we believed (and still do) that it was a better trade-off for a return to stability rather than fight corrupt data in rolling cycles for what could be many days.
But we clearly have work to do. We do not want to shift blame or throw up our hands here. We are committed to being your elite PMS and channel manager and that comes with certain expectations on your part and responsibilities on our part. We are taking steps to learn from this and incorporate new processes for additional redundancies and better communication. Some stark problems became very clear over the past 36 hours.
Does Airbnb automatically reactivate property listings, or must that be done manually?
During the outage, you may have received automatic emails from Airbnb stating that your listings were disabled or hidden because your software (ie. OwnerRez) was offline. Airbnb does this as a safety precaution so that guests are not double-booking your property since Airbnb has no way of confirming with you (ie. via OwnerRez) that everything is okay.
Post-outage, Airbnb should have reactivated any hidden listings automatically, but you should check your listings and not assume this is the case. When our system started communicating with Airbnb again, we confirmed that they were turning listings back on again by checking with specific listings and seeing that they were visible. However, do not trust that this is the case across the board. Check your listings so that you know for sure. Our helpdesk has reported that some users are still seeing hidden listings. That could be for specific reasons that don't apply to you, but everyone should still check that their listings are showing and the calendars are open and bookable. We apologize for the inconvenience this places on you.
If you did not receive any "hidden listings" email from Airbnb, you should be fine. Airbnb only hides listings if they can't send bookings through, not for other reasons. However, if you're worried about it, we recommend that you still verify your listings on Airbnb to make sure they are showing and bookable. It could be that it happened to you, but you didn't see the email or an email was never sent.
During the outage, I disconnected Airbnb. Now what?
During the outage, some users were recommending to each other to go into Airbnb and manually disconnect OwnerRez from the Airbnb side so that they could return to "platform mode" and deal with guests and bookings directly on Airbnb. If you did this, there is no way to undo it from the Airbnb side. You need to go into OwnerRez > Settings > API > Airbnb and reconnect the account from the OwnerRez side. This is safe to do, and your previous connection will reactivate and merge in changes safely.
I did a full resync for Airbnb, but nothing changed.
Triggering a "full resync" for Airbnb does not happen immediately in OwnerRez, and it never has. The request for the resync is put into the queue, along with many other sync actions, and is pulled out as the queue processes every few seconds. Typically, in normal conditions, the queue is processed so quickly it can appear to be immediate, but it always follows the queue.
After the outage, the queues were filled with many waiting requests, so everything took time to process. If you were clicking the "trigger full resync" option on the channel, it was put into the queue behind other things and probably took a while to process.
Post outage, we have been monitoring all of our queues to watch many types of requests process - bookings, channel syncs, messaging, and so on. If you are waiting for a message to be delivered or channel sync to go through, make sure to give it some time to catch up. If it's been a while (ie. an hour) reach out to helpdesk, and we'll take a look.
Where are my Booking.com bookings? They aren't showing up on my calendar.
Unlike some of our other channel partners, Booking.com does not provide an automatic method to restore lost or missed data, including booking records. Our team does have access to logs and is manually working through that information to find affected accounts and properly update calendars. This should be completed by the end of today, but, note that not all booking information can be restored. However, the calendar data will be correctly entered, preventing any double bookings.
OwnerRez says my Airbnb properties are fully synced and online, but, I still can't see them on Airbnb.
We are seeing a relative handful of this situation - as in, a few dozen properties out of tens of thousands. It appears to be individual properties affected - that is, you might have 10 Airbnb properties in one API-connected account, 9 are working fine, but 1 refuses to come back online no matter what you do. We've tried several things to get those properties working again and haven't had success either - so, we've filed a Critical bug with Airbnb partner support. If you have properties in this condition and have not already reported it to us, either via the Helpdesk or in this thread, please do so - we'll add your information to the Airbnb bug report and track them all together.
**UPDATE** - All but a couple of the offline Airbnb properties have been restored, and those are expected to be fixed by the end of the day.
