Heroku Security Incident - April / May 2022

Last updated: 2022-05-07

Key Points

  • We have found no reason to believe any of our systems or customer data have been compromised.
  • We continue to monitor for updates from our hosting provider regarding the incident on their systems.
  • One of our credentials (API key for a email system) was exposed in the attack, these keys were changed on May 9th, and we have not found evidence of unauthorised use.
  • No action is required by Runn customers
  • No Runn customer data is known to be affected by this incident
  • Heroku / Saleforce who are Runn’s platform and data host were compromised on April 7th, 2022
  • Runn was alerted to the severity of the issue on May 5th, 2022 and on the same day changed necessary credentials and alerted customers to the incident
  • Runn received an update on May 8th that confirmed the attack on Heroku's system did not access Heroku's encryption keys
  • Runn received an email on May 18th that stated the attack on Heroku's system did gain access to some credentials and secrets stored on their 'Pipeline Services'. For Runn this means that an API key for our email system was exposed.

Overview

Runn’s platform and data host, Heroku / Salesforce, was compromised on April 7th 2022. 

All information Heroku has communicated thus far and Runn’s internal investigation have not turned up any evidence that any Runn system or Runn customers' data was accessed by the attacker.

Runn has been informed that we some secrets were exfiltrated from Heroku's system. There is one Runn secret of significance that was exfiltrated, an API key for one of our mailing systems. We have not seen evidence of this key being used by unauthorised person and the key was changed on May 9th 2022.

At this stage, Runn customers do not need to take any action.

According to Heroku, the attacker gained access to Heroku systems on April 7th and had access to their systems until April 14th. During this time they stole Heroku customers' usernames, hashed and salted passwords, and encrypted environment variables (secrets).

According to Heroku's update on May 7th “The threat actor did not access the encryption key necessary to decrypt config var secrets”. This would mean that the secrets stolen are not accessible to the attacker and could not be used against Runn. Our security team has assessed the risk of these stolen credentials as low.

However, a subsequent update on May 18th from Heroku revealed that some secrets from an area called "Pipeline Services" was exfiltrated, along with their encryption key.

On May 7th, within 4 hours of the Runn security team becoming aware of this incident, we had started a process to change our credentials (including passwords, environmental vars, and secrets), contacted our customers notifying them of the incident, and started reviewing our logs for any unauthorized access to Runn's systems.

Runn is treating this as a potential very high severity incident and is proactively ensuring our customers' data is protected in the event that the original compromise is worse than Heroku has stated.

We have no evidence or reason to believe that any Runn customer data was accessed or any Runn internal system compromised by unauthorized persons.

We will continue to keep our customers informed of any changes to the situation. You may also view updates from Heroku's incident team.

Timeline

2022-04-14

Notification:
Heroku notified all their customers that a “Subset of Github Private repositories” had been accessed via a Heroku Github Access token. This incident was reported to them on April 13th, 2022 by Github, and the source code was downloaded on the April 9th, 2022.

Action:
Runn proceeded to disconnect any Heroku authorization to our Github account, verified the authenticity of all other access tokens, reviewed the codebase to ensure no secrets were stored in the code (as is Runn policy), and viewed Runn’s codebase access logs going back to April 7th, 2022. Runn did not find any suspicious activity, or any issues that concerned us.

2022-04-15 to 2022-05-04

Notification:
Heroku continued to update the incident status. No notice was given of any change in the incident or that any detail beyond the Github Access token was compromised.

2022-05-05

Notification:
Heroku/Salesforce sent out a new notification stating that the initial attack took place on April 7th, and that during the attack, the attacker gained “access to a database and exfiltrated the hashed and salted passwords for customers’ user accounts”.

No information about how this occurred or if any other data had been compromised was provided.

Action:
Immediately when Runn’s security team became aware of this update, our security protocols were activated.

Within four hours of Runn being aware of incident, we had undertaken the following mitigation strategies:

  • All users with Heroku access (4 in total) changed their passwords and revoked any API tokens they had with Heroku.
  • Our list of credentials stored on Heroku was reviewed and all credentials that have access to customer data had been changed (this includes: Database, Redis and OAuth).
  • All secrets stored on Heroku were reviewed and changed (this includes our secrets for session information which was the cause of all users being logged out)
  • We audited our codebase to ensure that none of the secrets stored on Heroku could have compromised our customers' emails, passwords or any other confidential information.
  • We contacted our Board of Directors to inform them of the incident.
  • We contacted all admins of our customer accounts to inform them of the incident.
  • We posted an update to our status.runn.io page regarding the incident.
  • We made a copy of our logs (which are usually kept for 30 days) and extended our log retention to 90 days.
  • We contacted our Heroku account manager to get additional details and additional logs stored by Heroku.
  • We started the process of reviewing our logs looking for any unauthorized access.

2022-05-06

Actions:
Runn continues to monitor the situation closely and in the process of updating all low risk credentials stored on Heroku (such as analytics API keys) and continues our review of logs looking for any unauthorized access.

2022-05-07

Notifications:
Heroku provided an update confirming their encryption keys were not stolen. However encrypted environment variables, including Runn's, were exfiltrated from Heroku's systems.

"... we confirmed that the threat actor had access to encrypted Heroku customer secrets stored in config var, the secrets are encrypted at rest and the threat actor did not access the encryption key necessary to decrypt config var secrets."

Actions:
Runn had preemptively rotated our secrets and environment variables before this announcement.

Runn's security team has accessed the risk of our encrypted secrets having been exfiltrated and believe to our system to be low. We will also enlist the expertise of an independent security consultant to independently assess the risk posed by this data having been exfiltrated.

Runn is continuing our investigation to assess if any unauthorised access to any Runn system or customer data occurred. To date we have no found evidence of unauthorised access.

Runn sent out another communication to our customers updating them on the latest information regarding the situation with our host and its affects on Runn's customer.

2022-05-11

Actions:
We have spoken with our external and independent security advisors regarding this incident, and they have agreed the risk is our customer is low given the current information available. This includes the exfiltrated of our encrypted environmental variables and secrets from Heroku's hosting platform.

We have continued our audit of system access, and have not found any unauthorised access or unexpected access attempts.

At this stage, we continue to believe no Runn data or services were compromised.

We will continue our updates as more information from Heroku becomes available.

2022-05-16

Actions:

We have completed our audit of internal systems and have found no evidence of unauthorised access, or any attempt at unauthorised access.

The information provided to us from our hosting provide, Heroku / Salesforce has given us no reason to believe our systems or our customers' data could have been compromised.

We have therefore concluded our investigation. We will continue to monitor Heroku's security updates, and should any further action or announcement be required we re-open our investigation.

2022-05-19

Notifications:

Heroku informed us, via direct email that "On that same day [April 7th], the threat actor downloaded data from another database that stores pipeline-level config vars for Review Apps and Heroku CI." After contacting their security team for clarification we were informed that "Heroku customer secrets stored in pipeline-level config vars are encrypted at rest and the threat actor was able to access the encryption key necessary to decrypt pipeline config var secrets." Confirming that any secrets stored here were readable to the attacker.

Actions:

We confirmed which secrets we had stored in Heroku's Pipeline Services and found there to be a single secret of concern, an API key for one of our email services. We verified that we had rotated this key on May 9th as part of our initial incident response.

Upon investigating, we found that this API could be used to view password reset emails and account invite emails sent in the previous 5 days. This opens up the possibility of a targeted attack vector against Runn users where the password reset process could be deliberately started by an attacker, and then the reset link read and used by the attacker to change a password and login to an account. This action would also automatically log out the existing user and their password would no longer work.

This attack vector applies to the majority of Runn accounts, except those who have the "SSO Only" account setting turned on. In this case a password reset would not allow access to the user account.

Given this new information, we went through our system logs looking for any unusual activity related to password resets and account invites. We also verified that no Runn staff member has any unusual activity on their accounts. We found no evidence that such an attack took place, however it not possible to rule it out completely.

We also spoke with our customer success team, and reviewed customer messages to see if anyone has reported any unusual activity such as passwords no longer working or unexpected password resets. We did not find any evidence of this.

Our team assess the risk from this stole credentials as low due to the fact that it would require a very targeted attack of Runn's systems, would allow access to only a single account at a time, and the secret was leaked as part of a much wider leaker of tens of thousands of Heroku / Salesforce accounts.

However, out of an abundance we decided to revoke all outstanding account invites between April 1st and 10th May. Our password resets link automatically expire after 6 hours and did not require any action.

Finally, we have sent out an email to all account admins informing them of the latest information, and asking to inform us about any suspicious activity related to passwords or logins.

Smart resource and capacity planning