Relevance Beats Reach

YouTube is one of the most cited domains in AI search, yet there is no rigorous account of why particular videos get picked. Most prior work compares metadata such as duration, engagement, and channel size across cited and uncited videos. Those descriptive comparisons cannot tell you what the AI system is actually responding to.

We took a different approach. Instead of describing cited videos, we tried to replicate the ranking decision behind Google AI Overviews source selection: given a prompt, pick the cited video out of a lineup of relevant alternatives that were not cited. Inspecting the model that does this well tells us which signals matter.

Gumshoe's citation data includes 1,105,198 YouTube citations across 395,775 unique videos, an average of 2.78 citations per video. This study focuses on Google AI Overviews.

The goal is to identify which features of a YouTube video are associated with its inclusion in AI search results. To do that, we compare videos cited by AI Overviews against topically similar videos that were not cited for the same prompt. Two design choices shape that comparison.

Before training anything, we compared the two groups on the characteristics most often cited in prior YouTube analyses: duration, popularity, engagement, and channel size.

With samples this large, almost any difference is statistically significant. To judge practical importance, we measured effect size with Cliff's delta and the probability that a randomly chosen cited video scores higher than a randomly chosen alternative.

Each ranking task contains one cited video and every deduplicated alternative returned by its searches, about 54 videos per task. The model's job is to rank the cited video above the rest. We split the 5,000 tasks 80/10/10 into training, validation, and test sets, keeping each task intact within one split. The test set was held back until all development was finished.

We used a CatBoost ranker, a gradient-boosting model that handles categorical features and grouped ranking objectives. The aim was not to find the best possible architecture but to test whether the features carry real signal, and to see how the model uses them.

Using only video and channel metadata, the ranker put the cited video first in 7.6% of validation tasks and in the top 10 in 48.6%, against a random baseline of 2.0% and 19.7%. The median cited video landed at rank 11.

The model favors videos longer than about 10 minutes, longer descriptions, fewer comments, and channels with more uploads and subscribers. These relationships are not causal. Making a video longer or padding its description would not necessarily earn a citation. More likely, these features act as proxies for video types that tend to be relevant to the prompt, like explainers and reviews from established channels.

If metadata is mostly a stand-in for relevance, a model that measures relevance directly should do at least as well.

For each prompt and video, we computed four similarity scores: word overlap, phrase overlap, TF-IDF similarity, and cosine similarity between OpenAI text-embedding-3-large embeddings of the prompt and the video's title and description.

The semantic-only ranker beat the metadata-only ranker on every validation metric! It placed the cited video in the top 10 for 59.6% of tasks, versus 48.6%, and moved the median cited rank from 11 to 8. Embedding similarity did most of the work.

Finally, we combined metadata and semantic features in a single ranker trained on the same splits. It substantially outperformed either feature family alone, ranking the cited video first in 20.4% of validation tasks and in the top 10 in 75.4%, with a median rank of 5.

Embedding similarity is the strongest feature by roughly four times. Duration, channel subscribers, and description length still add signal beyond relevance, which suggests that metadata and semantic fit capture different parts of how AI Overviews chooses what to cite.

With development complete, we scored the three frozen models once on the 500 held-out test tasks, which played no role in feature selection, training, or model comparison.

Chasing views and engagement is a weak strategy for AI visibility on YouTube. Among videos that could plausibly answer a prompt, the ones AI Overviews cites are the ones whose titles and descriptions most closely match what the buyer asked. Reach adds a little on top of that.

In practice, that points to a few priorities:

• Build videos around the specific questions your buyers ask AI, and use that language in the title and description.
• Write full descriptions that explain what the video covers, rather than a line and a list of links.
• Favor substantive explainers and reviews over short clips. Cited videos run about 50% longer at the median.
• Publish from an established brand channel where possible, since channel size adds signal beyond relevance.

These findings are correlational. The model shows what AI Overviews tends to favor, not what will cause a citation. The next step is to test the ranker directly by publishing videos that score highly on our own channel and tracking whether they get cited.