Falcon Content material Replace Remediation and Steering Hub

Up to date 2024-07-24 0335 UTC

Preliminary Put up Incident Overview (PIR): Content material Configuration Replace Impacting the Falcon Sensor and the Home windows Working System (BSOD)

That is CrowdStrike’s preliminary Put up Incident Overview (PIR). We shall be detailing our full investigation within the forthcoming Root Trigger Evaluation that shall be launched publicly. All through this PIR, we have now used generalized terminology to explain the Falcon platform for improved readability. Terminology in different documentation could also be extra particular and technical.

What Occurred?
On Friday, July 19, 2024 at 04:09 UTC, as a part of common operations, CrowdStrike launched a content material configuration replace for the Home windows sensor to assemble telemetry on potential novel risk strategies.

These updates are an everyday a part of the dynamic safety mechanisms of the Falcon platform. The problematic Speedy Response Content material configuration replace resulted in a Home windows system crash.

Techniques in scope embody Home windows hosts operating sensor model 7.11 and above that had been on-line between Friday, July 19, 2024 04:09 UTC and Friday, July 19, 2024 05:27 UTC and obtained the replace. Mac and Linux hosts weren’t impacted.

The defect within the content material replace was reverted on Friday, July 19, 2024 at 05:27 UTC. Techniques coming on-line after this time, or that didn’t join throughout the window, weren’t impacted.

What Went Improper and Why?
CrowdStrike delivers safety content material configuration updates to our sensors in two methods: Sensor Content material that’s shipped with our sensor straight, and Speedy Response Content material that’s designed to reply to the altering risk panorama at operational velocity.

The problem on Friday concerned a Speedy Response Content material replace with an undetected error.

Sensor Content material
Sensor Content material supplies a variety of capabilities to help in adversary response. It’s at all times a part of a sensor launch and never dynamically up to date from the cloud. Sensor Content material contains on-sensor AI and machine studying fashions, and contains code written expressly to ship longer-term, reusable capabilities for CrowdStrike’s risk detection engineers.

These capabilities embody Template Sorts, which have pre-defined fields for risk detection engineers to leverage in Speedy Response Content material. Template Sorts are expressed in code. All Sensor Content material, together with Template Sorts, undergo an intensive QA course of, which incorporates automated testing, handbook testing, validation and rollout steps.

The sensor launch course of begins with automated testing, each previous to and after merging into our code base. This contains unit testing, integration testing, efficiency testing and stress testing. This culminates in a staged sensor rollout course of that begins with dogfooding internally at CrowdStrike, adopted by early adopters. It’s then made usually obtainable to prospects. Prospects then have the choice of choosing which elements of their fleet ought to set up the most recent sensor launch (‘N’), or one model older (‘N-1’) or two variations older (‘N-2’) by way of Sensor Replace Insurance policies.

The occasion of Friday, July 19, 2024 was not triggered by Sensor Content material, which is barely delivered with the discharge of an up to date Falcon sensor. Prospects have full management over the deployment of the sensor — which incorporates Sensor Content material and Template Sorts.

Speedy Response Content material
Speedy Response Content material is used to carry out quite a lot of behavioral pattern-matching operations on the sensor utilizing a extremely optimized engine. Speedy Response Content material is a illustration of fields and values, with related filtering. This Speedy Response Content material is saved in a proprietary binary file that accommodates configuration knowledge. It’s not code or a kernel driver.

Speedy Response Content material is delivered as “Template Situations,” that are instantiations of a given Template Kind. Every Template Occasion maps to particular behaviors for the sensor to look at, detect or forestall. Template Situations have a set of fields that may be configured to match the specified conduct.

In different phrases, Template Sorts signify a sensor functionality that allows new telemetry and detection, and their runtime conduct is configured dynamically by the Template Occasion (i.e., Speedy Response Content material).

Speedy Response Content material supplies visibility and detections on the sensor with out requiring sensor code adjustments. This functionality is utilized by risk detection engineers to assemble telemetry, establish indicators of adversary conduct and carry out detections and preventions. Speedy Response Content material is behavioral heuristics, separate and distinct from CrowdStrike’s on-sensor AI prevention and detection capabilities.

Speedy Response Content material Testing and Deployment
Speedy Response Content material is delivered as content material configuration updates to the Falcon sensor. There are three main programs: the Content material Configuration System, the Content material Interpreter and the Sensor Detection Engine.

The Content material Configuration System is a part of the Falcon platform within the cloud, whereas the Content material Interpreter and Sensor Detection Engine are elements of the Falcon sensor. The Content material Configuration System is used to create Template Situations, that are validated and deployed to the sensor by way of a mechanism referred to as Channel Information. The sensor shops and updates its content material configuration knowledge by way of Channel Information, that are written to disk on the host.

The Content material Interpreter on the sensor reads the Channel File and interprets the Speedy Response Content material, enabling the Sensor Detection Engine to look at, detect or forestall malicious exercise, relying on the client’s coverage configuration. The Content material Interpreter is designed to gracefully deal with exceptions from doubtlessly problematic content material.

Newly launched Template Sorts are stress examined throughout many elements, corresponding to useful resource utilization, system efficiency impression and occasion quantity. For every Template Kind, a particular Template Occasion is used to emphasize take a look at the Template Kind by matching towards any potential worth of the related knowledge fields to establish opposed system interactions.

Template Situations are created and configured by way of using the Content material Configuration System, which incorporates the Content material Validator that performs validation checks on the content material earlier than it’s printed.

Timeline of Occasions: Testing and Rollout of the InterProcessCommunication (IPC) Template Kind
Sensor Content material Launch: On February 28, 2024, sensor 7.11 was made usually obtainable to prospects, introducing a brand new IPC Template Kind to detect novel assault strategies that abuse Named Pipes. This launch adopted all Sensor Content material testing procedures outlined above within the Sensor Content material part.

Template Kind Stress Testing: On March 05, 2024, a stress take a look at of the IPC Template Kind was executed in our staging surroundings, which consists of quite a lot of working programs and workloads. The IPC Template Kind handed the stress take a look at and was validated to be used.

Template Occasion Launch through Channel File 291: On March 05, 2024, following the profitable stress take a look at, an IPC Template Occasion was launched to manufacturing as a part of a content material configuration replace. Subsequently, three further IPC Template Situations had been deployed between April 8, 2024 and April 24, 2024. These Template Situations carried out as anticipated in manufacturing.

What Occurred on July 19, 2024?
On July 19, 2024, two further IPC Template Situations had been deployed. Because of a bug within the Content material Validator, one of many two Template Situations handed validation regardless of containing problematic content material knowledge.

Based mostly on the testing carried out earlier than the preliminary deployment of the Template Kind (on March 05, 2024), belief within the checks carried out within the Content material Validator, and former profitable IPC Template Occasion deployments, these situations had been deployed into manufacturing.

When obtained by the sensor and loaded into the Content material Interpreter, problematic content material in Channel File 291 resulted in an out-of-bounds reminiscence learn triggering an exception. This surprising exception couldn’t be gracefully dealt with, leading to a Home windows working system crash (BSOD).

How Do We Stop This From Taking place Once more?

Software program Resiliency and Testing

• Enhance Speedy Response Content material testing by utilizing testing varieties corresponding to:
  • Native developer testing
  • Content material replace and rollback testing
  • Stress testing, fuzzing and fault injection
  • Stability testing
  • Content material interface testing
• Add further validation checks to the Content material Validator for Speedy Response Content material. A brand new verify is in course of to protect towards this kind of problematic content material from being deployed sooner or later.
• Improve present error dealing with within the Content material Interpreter.

Speedy Response Content material Deployment

• Implement a staggered deployment technique for Speedy Response Content material wherein updates are progressively deployed to bigger parts of the sensor base, beginning with a canary deployment.
• Enhance monitoring for each sensor and system efficiency, gathering suggestions throughout Speedy Response Content material deployment to information a phased rollout.
• Present prospects with higher management over the supply of Speedy Response Content material updates by permitting granular collection of when and the place these updates are deployed.
• Present content material replace particulars through launch notes, which prospects can subscribe to.

Along with this preliminary Put up Incident Overview, CrowdStrike is dedicated to publicly releasing the complete Root Trigger Evaluation as soon as the investigation is full.