SupportDedicated support services for our customers.

Request a Demo
Service Incident Notification & Root Cause Analysis (RCA): RDUK-7211
Changelog
  • Updated 'What Happened' section as more investigation carried out and information gathered - specifically the handling of recipient purging, and SMTP session handling.
  • Update 'Root Cause(s)' to include details on BDAT chunk processing, SMTP session reuse, and fault conditions.
  • Status Update: Changed from 'Remediation Underway' to 'Incident Response Active'
  • Status Update: Added issue mitigation date
  • Changed timestamps from GMT to UTC

ARTICLE DATE: 22.01.25

STATUS: Closed

INTRODUCTION

REPORTED EVENT: A software bug was discovered in the Censornet EMS module, whereby a subset of emails were inadvertently delivered to unintended recipients

This issue has been investigated by the Technical Support team, deemed critical and quickly bumped to P1 status for escalation to the Censornet developer team.

WHAT HAPPENED: An update was rolled out to the production environment in order to address an issue whereby messages including bare line feeds were not routed correctly. Microsoft stopped automatically removing bare line feeds from Email messages in order to better support security features such as DKIM. A Microsoft support article was published in January 2024 in response to numerous community issues reporting non-delivery reports relating to bare line feeds: https://learn.microsoft.com/en-us/exchange/troubleshoot/email-delivery/ndr/fixerror- code-550-5-6-11-in-exchange-online

Based on increasing customer requests, on 22/11/24 a development ticket was created to address email delivery issues whereby messages containing bare line feeds could not be delivered to recipients because the destination server did not support the required protocol. To address this compatibility issue, a code update was created to support bare line feed characters. The code passed QA and Software Acceptance Testing and entered the release pipeline for the next cycle (January 2025).

This release included enhancements to mail transport services for SMTP messages by using ‘chunking’ as a way to deliver messages in blocks using the SMTP BDAT command (RFC3030) as a replacement for the DATA command which does not handle bare line feeds. The fix had unintended effects on Email messages delivered to servers using BDAT whereby transmission chunks (message bodies) were not cleared properly between emails in the session, causing email body content from the first email to carry over and be incorrectly associated with the next email’s transmission.

The release was pushed to production on US infrastructure 15/01/2025, 11:30 UTC.  This issue was brought to our attention via a single 1st line support ticket on 16/01/25 at 19:19 UTC; the agent attempted to reproduce the symptoms working with the partner to collect log data and message traces.  Several additional tickets were raised on 20/01/25 after the update rolled to EU infrastructure on 19/01/2025, 00:40 UTC. These were then immediately escalated to engineering, with the fault identified at 13:43 UTC and resolved as of 20/01/25 14:12 UTC.

OUR RESPONSE

ACTION TAKEN: At 14:12 UTC 20/01/25 a rollback was completed, disabling the BDAT functionality and no subsequent cases were reported. Work tasks were then run to identify the impacted customers, and individual message GUIDs.

ROOT CAUSE(S):

  1. A code update to EMS to add support for SMTP BDAT command and message chunking was released to production environment with unintended results after message processing.
  2. When being processed by the Censornet EMS infrastructure, messages sent to servers using the BDAT protocol caused chunk accumulation where the body content of one email was not cleared and ended up being attached to the next email, even though the recipient information was correctly reset.
  3. This fault affected both emails from the same sender and different senders within the same session.  If multiple emails were sent from different senders in the same session (which is common for SMTP transmission), the body content from the first email was carried over and mistakenly included with the next email.
  4. The fault could be reproduced only in a case when the BDAT command is supported by the receiving server, this command is chosen for email transmission, and multiple emails were sent in the same SMTP session.

SOLUTION(S): A complete code rollback has been applied limiting the exposure of this issue and BDAT functionality disabled for SMTP delivery. Mis-delivered messages are being identified, and partners/customers will be notified without undue delay the exact Emails that were affected in order to inform any unintended recipients.

MITIGATION AND CONTINUOUS IMPROVEMENT:

  1. Whilst the standard code-release process was followed and deployment executed in a staged manner to single regions at a time, followed by a waiting period, it has been deemed appropriate to extend this window for releases relating to Email Security in order to allow the flow/processing of a large sample of messages before pushing to other regions.

A post-mortem is underway with the executive leadership team, including an engineering process review. QA and Acceptance Testing will be assessed in detail and further simulations added to the SDLC.

As with all customer-impacting platform issues we take this extremely seriously and will be putting plans in place to ensure future mitigation.

We would like to extend our apologies for this incident and are committed to maintaining transparency and keeping you informed.

If you have any questions, please do not hesitate to contact us directly via your account manager, or our Support Team.

As a Censornet Email Security user, how do I know if I was affected?

We have put all our efforts into analysing the Email transmission logs for the time period the issue was present and identified all messages impacted by the software bug, separating them out into individual exports. Our support team will communicate directly with you via the notification email address registered at time of account provisioning and provide additional details. In some instances your service provider or reseller may contact you on our behalf if we do not hold the relevant contact details directly.  

0.6% of outbound Emails processed through the EU/UK infrastructure were affected by the bug, and 1.6% of Emails in the US infrastructure, however if you haven’t received a notification but suspect an issue, please contact us at  support@censornet.com for assistance.

Was any sensitive data contained in the affected Emails?

We do not have access to the content of customer emails, only the message headers and metadata.  The unintended inclusion of recipients in BCC may have resulted in privacy concerns, depending on the content of the email. If you are among the customers identified by our investigation, we will contact you and provide the relevant logs to help locate the messages in your environment and cooperate fully to assist with any data protection documentation required for incident response and compliance.

What types of message were affected?
The fault could only be reproduced where the BDAT command was supported and selected by the server, and multiple emails were sent in the same SMTP session, it is not limited to any specific email type (i.e. marketing communications).

Is the recipient list (To, CC, BCC) being carried over from the previous message?
No, the recipient list was correctly reset between each email in the session.  Some BDAT chunks of the message body were however carried over – essentially accumulating multiple messages’ MIME content as the session progressed.

Were unintended recipients known to the sender?
Recipients were unrelated.  The message chunks carried over the body of the first email into the next, meaning the content was sent to the subsequent message’s recipients, however in batch processing the recipients could be known to the sender.

How are multiple messages processed in a single SMTP session?
This is based entirely on how the sending SMTP server is configured. This means the decision to process multiple emails in one session is determined by the sending server’s settings and processes, such as batch processing or session reuse. As a result, it was most observed when emails were sent from the same company or when using systems like Office 365, which handle many customers and often reuse SMTP sessions for efficiency and to reduce connection overhead.  Each time a new email is sent the sender and recipient lists are reset, this is standard behaviour.

What steps can I take to mitigate any risks?

At this time, unless you have received a notification from Censornet there is no required action from your side. If you have concerns about specific email communications, we recommend reviewing your email logs and contacting our support team if you need further assistance.

🍪 This website uses cookies to improve your web experience.

If you wish to chat to someone about our products or services please contact our UK office on the number below:

0845 230 9590

Request a demo

Fill out the form below, and we will be in touch shortly.

What would you like to demo?*