Requirement
The fund was evaluating retail sentiment as an input to a long/short equity book. They had been pitched sentiment data before and rejected it twice, which made them a useful counterparty: they arrived with the objections already formed and specific.
Their position was that most sentiment datasets fail for reasons that have nothing to do with whether retail sentiment is predictive.
| # | Failure mode | Why it invalidates a backtest |
|---|---|---|
| T1 | History restated rather than recorded | Backtest sees a past no trader could have seen |
| T2 | Deleted messages absent | Survivorship makes the crowd look more prescient than it was |
| T3 | Labels inferred by a model | Adds a second model's error, correlated with the hardest language |
| T4 | Backtest run by the vendor | Universe, construction, and period chosen by an interested party |
The evaluation was these four tests, run against the panel.
T1 Point-in-time integrity
If historical sentiment has been smoothed, gap-filled, or rebuilt from a later snapshot, a backtest sees a view of the past that was not available at the time. The strategy that results is not optimistic, it is fiction, and the failure is silent because the resulting equity curve looks like a good one.
| Property | Panel |
|---|---|
| Construction | Recorded as it happened |
| Restatement | None |
| Continuous since | 2009 |
| Standard | Any analysis sees exactly what was knowable at each timestamp |
T2 Deleted-message survivorship
A panel assembled from a current scrape is a panel of messages that survived. Deletion is not random: retail posts that aged badly are deleted at a higher rate than posts that aged well.
The resulting bias points in the most dangerous available direction. It makes the crowd look most prescient on exactly the names where it was most wrong.
Because the panel is recorded continuously rather than reconstructed, deleted messages remain in the historical record as they were when posted. The fund tested this directly rather than accepting the description, which was the correct response.
T3 Label provenance
Most sentiment feeds run a model over text and call the output sentiment. That is a second model's opinion about a first author's intent, and its errors concentrate in the language hardest to parse: sarcasm, in-group jargon, ticker-specific idiom.
| Property | This panel | Typical NLP feed |
|---|---|---|
| Label source | Applied by the author | Inferred by a classifier |
| Error surface | Author's own stated intent | Classifier error, correlated with hard language |
| Auditability | Label is a stated fact | Label is a model output |
A bullish tag here is bullish because the person posting said so. We also ran the direct comparison against NLP-derived sentiment from a general social platform and published it, so the difference is inspectable rather than asserted.
T4 Whose backtest
Our own construction showed 18.1% CAGR on a Nasdaq 100 long/short. We told them to disregard it.
Not because it is wrong, but because a vendor-run backtest is structurally unpersuasive. The vendor chose the universe, the construction, the rebalance, and the period, and had every incentive to keep choosing until something worked.
| Party | Role |
|---|---|
| Gerra | Supplies point-in-time data, documented delivery shape, stated baselines, no restatement |
| Fund | Chooses universe, construction, rebalance, period, and runs the test |
The only backtest that should move a fund is the one the fund runs. They ran theirs.
Deployment
| Property | Value |
|---|---|
| Primary feed | Firehose, roughly 200ms latency |
| Secondary feed | Symbol-level engagement stream |
| Fields in use | Sentiment tags, message velocity, watchlist adds, pageviews |
| Modeling note | Attention and direction treated as separate variables |
Message velocity, watchlist adds, and pageviews carried information distinct from the sentiment tags. That is a common result and worth designing for: attention and direction are different variables, and collapsing them loses signal.
Reusable output
The general checklist from this evaluation is published on the research index as a buyer guide. It applies to any sentiment vendor, including us.