Incident Post-Mortem: Facebook Comment Processing Issues affecting other channels due to the queue overflow

Summary

We identified several performance and reliability issues affecting Facebook comment processing, particularly for posts containing Reels and conversations operating in THREAD_ON_COMMENT mode.

The issues resulted in unnecessary processing delays and, in some cases, comments not being processed successfully.

Root Causes

1. Repeated Reel Downloading in Post Cover Messages

When a Post Cover Message referenced a Reel, the system downloaded the video to our infrastructure before sending it to the customer. As a result, the same video could be downloaded multiple times for different conversations or requests.

For larger video files, this introduced significant and unnecessary processing delays.

Resolution:
Reels are no longer downloaded and re-uploaded. The Post Cover Message now contains a small preview icon together with a direct link to the original Facebook post.

Impact:
This significantly reduces processing time, bandwidth usage, and dependency on Facebook video download performance.

2. Inefficient Root Comment Lookup

In THREAD_ON_COMMENT mode, the system needs to identify the root comment associated with a conversation.

The existing lookup algorithm performed more operations than necessary, increasing processing time, particularly for posts with a large number of comments.

Resolution:
The root-comment lookup algorithm was substantially simplified and optimized.

Impact:
Root comments can now be identified faster, reducing overall message-processing latency.

3. Root Comment Lookup Failure for Large Requests

We identified an error in the root-comment lookup logic when requesting up to 1,000 comments from Facebook in a single request.

In some cases, Facebook’s API failed to return such a large result set and responded with an error. Our request then entered the retry mechanism and continued retrying until the retry limit was exhausted.

Once all retries were exhausted, the root comment could not be identified and the corresponding comment was effectively lost from the processing flow.

Resolution:
The lookup logic has been corrected so that it no longer depends on successfully retrieving such a large batch of comments in a single Facebook API request.

Impact:
This prevents comments from being lost due to Facebook API failures when processing posts with large comment volumes.

Outcome

The implemented changes address both performance and reliability:

• Reel processing no longer requires repeated video downloads.
• Root-comment lookup in THREAD_ON_COMMENT mode is significantly faster.
• Large Facebook API requests can no longer cause comments to be lost after retry exhaustion.