Full Pipeline Tour

From Request to Distribution: understand how a tweet makes its way into a user's For You Feed.

Stage 1

Query Hydration

When a user opens the app, the system first gathers context:

  • User Features: Language, region, device type.
  • Action Sequence: The user's last 200 interactions (who they liked, what they clicked).
  • Social Graph: Follower lists, mutes, and blocks.
File: user_action_seq_query_hydrator.rs
Stage 2

Candidate Sourcing

The system retrieves around 1,500 candidate tweets from millions:

  • Thunder (In-Network): Recent and top tweets from accounts you follow.
  • Phoenix (Out-of-Network): Uses Embedding-based vector search to find relevant content from authors you don't know yet.
Files: thunder_source.rs, phoenix_source.rs
Stage 3

Filtering

Before scoring, obviously unsuitable content is discarded:

  • DropDuplicates: Removes duplicate tweet IDs.
  • AgeFilter: Drops stale tweets (e.g., older than 72 hours).
  • MutedKeyword: Filters out tweets containing user-muted words.
Files: filters/*.rs
Stage 4

Scoring

The core ranking stage. The Phoenix Transformer predicts probabilities:

Score = Σ (Weight_i * P(Action_i))

It predicts P(Like), P(Reply), P(Repost), etc., and combines them into a weighted total.

Files: phoenix_scorer.rs, weighted_scorer.rs
Stage 5

Selection & VF

Final processing and delivery:

  • Top-K Selection: Selects the top ~100 highest-scoring tweets.
  • Visibility Filtering (VF): Final check for safety violations and appropriateness.
  • Author Diversity: Attenuates scores for repetitive authors to keep the feed fresh.