flypost.aiflypost.ai
Glossary

Semantic Deduplication

Definition

Semantic deduplication compares the meaning of posts, not their words. flypost.ai's Originality Engine embeds each candidate angle and drops any within 0.85 cosine similarity of your full posting history, then diversity-samples across clusters.

The naive way to avoid repeating yourself is to compare new posts against old ones by text. That works until the model rewords. "5 study habits" becomes "Five strategies for studying" becomes "What top scorers actually do." Same idea, three surface forms, and a string match waves all three through.

Semantic deduplication compares meaning instead. We embed every angle into a high-dimensional vector, where reworded duplicates land close together even when they share almost no words. The check is on the idea, not the phrasing.

In flypost.ai's Originality Engine, the strategist generates eight to twelve candidate angles, each is embedded and compared by cosine similarity against your full posting history, anything within 0.85 is dropped, and the survivors are clustered so we sample across them. You stay distinct, at scale.

Related

Originality Engine

Content Velocity

The Originality Engine

FAQ

Semantic Deduplication, answered.

Semantic deduplication compares the meaning of content, not its exact words. It embeds each post as a vector so reworded duplicates, which share little text but the same idea, get caught and dropped.
It generates eight to twelve candidate angles, embeds each, compares them by cosine similarity to your full history, drops anything within 0.85, then diversity-samples across clusters so each post is distinct.
Because models reword. The same idea in three different phrasings fools a string match but not an embedding comparison, which measures meaning and catches near-duplicates that share almost no words.