UPDATE: We have pushed the fix to the known impacted endpoints, and are continuing to finish the rest of the rollout of the update. We are going to resolve this issue since impacted endpoints have received the new update.
Posted Feb 09, 2024 - 15:26 UTC
Update
We are continuing to monitor for any further issues.
Posted Feb 08, 2024 - 16:23 UTC
Monitoring
UPDATE: We have pushed the fix to the known impacted endpoints. We will continue to roll out the update to all remaining endpoints and monitor the progress. No action is needed on your end.
Posted Feb 08, 2024 - 00:07 UTC
Identified
UPDATE: We have identified the root cause of the outage impacting Huntress Managed EDR and have begun pushing out an update to address the problem. No action is needed on your end.
We are pushing the fix to known impacted endpoints first and will continue updating all endpoints on a rolling basis afterward. Please continue to follow this page for more updates.
The issue was caused due to the Huntress Rio service (which gathers telemetry for our EDR process insights feature) being stuck in a “stop pending” state. The update being pushed will ensure that any hung service is killed and restarted on the new version.
Posted Feb 07, 2024 - 20:04 UTC
Investigating
Huntress is currently experiencing an outage that is impacting Huntress Managed EDR.
Currently, a portion of our agent population is experiencing an outage where the Huntress SOC cannot receive some EDR data. The issue is due to the “Huntress Rio” service being stuck in a “stop pending” state. As a result, the agent is not collecting “process insights” data and the SOC cannot analyze related activity. Other detection capabilities are still functioning but this data source is an important one and we are taking this outage extremely seriously.
The issue occurs on shutdown or restart of the service on systems with a lot of event log activity, primarily servers. We pushed out an agent version last night that attempted to resolve the problem, but was unsuccessful. We are currently preparing another version that will correct that, which we expect to send to the specific systems impacted by the end of the day today.
We will post updates here as additional information is available.
Posted Feb 07, 2024 - 17:46 UTC
This incident affected: Agent Deployment and Communication.