Last updated: 2022-06-16
Runn’s platform and data host, Heroku / Salesforce, was compromised on April 7th 2022.
All information Heroku has communicated thus far and Runn’s internal investigation have not turned up any evidence that any Runn system or Runn customers' data was accessed by the attacker.
Runn has been informed that we some secrets were exfiltrated from Heroku's system. There is one Runn secret of significance that was exfiltrated, an API key for one of our mailing systems. We have not seen evidence of this key being used by unauthorised person and the key was changed on May 9th 2022.
At this stage, Runn customers do not need to take any action.
According to Heroku, the attacker gained access to Heroku systems on April 7th and had access to their systems until April 14th. During this time they stole Heroku customers' usernames, hashed and salted passwords, and encrypted environment variables (secrets).
According to Heroku's update on May 7th “The threat actor did not access the encryption key necessary to decrypt config var secrets”. This would mean that the secrets stolen are not accessible to the attacker and could not be used against Runn. Our security team has assessed the risk of these stolen credentials as low.
However, a subsequent update on May 18th from Heroku revealed that some secrets from an area called "Pipeline Services" was exfiltrated, along with their encryption key.
On May 7th, within 4 hours of the Runn security team becoming aware of this incident, we had started a process to change our credentials (including passwords, environmental vars, and secrets), contacted our customers notifying them of the incident, and started reviewing our logs for any unauthorized access to Runn's systems.
Runn is treating this as a potential very high severity incident and is proactively ensuring our customers' data is protected in the event that the original compromise is worse than Heroku has stated.
We have no evidence or reason to believe that any Runn customer data was accessed or any Runn internal system compromised by unauthorized persons.
This incident has now been closed by Runn and Heroku.
Notification:
Heroku notified all their customers that a “Subset of Github Private repositories” had been accessed via a Heroku Github Access token. This incident was reported to them on April 13th, 2022 by Github, and the source code was downloaded on the April 9th, 2022.
Action:
Runn proceeded to disconnect any Heroku authorization to our Github account, verified the authenticity of all other access tokens, reviewed the codebase to ensure no secrets were stored in the code (as is Runn policy), and viewed Runn’s codebase access logs going back to April 7th, 2022. Runn did not find any suspicious activity, or any issues that concerned us.
Notification:
Heroku continued to update the incident status. No notice was given of any change in the incident or that any detail beyond the Github Access token was compromised.
Notification:
Heroku/Salesforce sent out a new notification stating that the initial attack took place on April 7th, and that during the attack, the attacker gained “access to a database and exfiltrated the hashed and salted passwords for customers’ user accounts”.
No information about how this occurred or if any other data had been compromised was provided.
Action:
Immediately when Runn’s security team became aware of this update, our security protocols were activated.
Within four hours of Runn being aware of incident, we had undertaken the following mitigation strategies:
Actions:
Runn continues to monitor the situation closely and in the process of updating all low risk credentials stored on Heroku (such as analytics API keys) and continues our review of logs looking for any unauthorized access.
Notifications:
Heroku provided an update confirming their encryption keys were not stolen. However encrypted environment variables, including Runn's, were exfiltrated from Heroku's systems.
"... we confirmed that the threat actor had access to encrypted Heroku customer secrets stored in config var, the secrets are encrypted at rest and the threat actor did not access the encryption key necessary to decrypt config var secrets."
Actions:
Runn had preemptively rotated our secrets and environment variables before this announcement.
Runn's security team has accessed the risk of our encrypted secrets having been exfiltrated and believe to our system to be low. We will also enlist the expertise of an independent security consultant to independently assess the risk posed by this data having been exfiltrated.
Runn is continuing our investigation to assess if any unauthorised access to any Runn system or customer data occurred. To date we have no found evidence of unauthorised access.
Runn sent out another communication to our customers updating them on the latest information regarding the situation with our host and its affects on Runn's customer.
Actions:
We have spoken with our external and independent security advisors regarding this incident, and they have agreed the risk is our customer is low given the current information available. This includes the exfiltrated of our encrypted environmental variables and secrets from Heroku's hosting platform.
We have continued our audit of system access, and have not found any unauthorised access or unexpected access attempts.
At this stage, we continue to believe no Runn data or services were compromised.
We will continue our updates as more information from Heroku becomes available.
Actions:
We have completed our audit of internal systems and have found no evidence of unauthorised access, or any attempt at unauthorised access.
The information provided to us from our hosting provide, Heroku / Salesforce has given us no reason to believe our systems or our customers' data could have been compromised.
We have therefore concluded our investigation. We will continue to monitor Heroku's security updates, and should any further action or announcement be required we re-open our investigation.
Notifications:
Heroku informed us, via direct email that "On that same day [April 7th], the threat actor downloaded data from another database that stores pipeline-level config vars for Review Apps and Heroku CI." After contacting their security team for clarification we were informed that "Heroku customer secrets stored in pipeline-level config vars are encrypted at rest and the threat actor was able to access the encryption key necessary to decrypt pipeline config var secrets." Confirming that any secrets stored here were readable to the attacker.
Actions:
We confirmed which secrets we had stored in Heroku's Pipeline Services and found there to be a single secret of concern, an API key for one of our email services. We verified that we had rotated this key on May 9th as part of our initial incident response.
Upon investigating, we found that this API could be used to view password reset emails and account invite emails sent in the previous 5 days. This opens up the possibility of a targeted attack vector against Runn users where the password reset process could be deliberately started by an attacker, and then the reset link read and used by the attacker to change a password and login to an account. This action would also automatically log out the existing user and their password would no longer work.
This attack vector applies to the majority of Runn accounts, except those who have the "SSO Only" account setting turned on. In this case a password reset would not allow access to the user account.
Given this new information, we went through our system logs looking for any unusual activity related to password resets and account invites. We also verified that no Runn staff member has any unusual activity on their accounts. We found no evidence that such an attack took place, however it not possible to rule it out completely.
We also spoke with our customer success team, and reviewed customer messages to see if anyone has reported any unusual activity such as passwords no longer working or unexpected password resets. We did not find any evidence of this.
Our team assess the risk from this stole credentials as low due to the fact that it would require a very targeted attack of Runn's systems, would allow access to only a single account at a time, and the secret was leaked as part of a much wider leaker of tens of thousands of Heroku / Salesforce accounts.
However, out of an abundance we decided to revoke all outstanding account invites between April 1st and 10th May. Our password resets link automatically expire after 6 hours and did not require any action.
Finally, we have sent out an email to all account admins informing them of the latest information, and asking to inform us about any suspicious activity related to passwords or logins.
Notifications:
Heroku has closed their incident with no new information or exposure found.
Actions:
We continue to keep the incident open until a final report from Heroku is released
Notifications:
Heroku has released their incident report.
Actions:
We have reviewed the incident report, and have found no need to further investigation.
We have concluded that no data on any of Runn's internal systems were compromised or exposed.
This incident is now closed.