Polimath

Suppose we want to recommend podcast episodes to users. Instead of having users rate each episode, we want to infer from their listening/skipping behavior how much they liked each episode we offer to them.  What we want is some kind of rating value we can infer from their behavior that captures how positive an experience they had with our app.  In turn this helps us offer more of the kinds of things they enjoy.

How much is it worth for a user to listen to an episode? Clearly listening to more of an episode is better than listening to less of an episode (Assumption 1). Almost as clear is the idea that listening to all of a longer episode shows more engagement than listening to all of a shorter episode (Assumption 2).

Suppose we have two podcast episodes: A is 5 minutes long and B is 30 minutes long.

Alice listens to all of both A and B.  We could give A a value of 5 and B a value of 30, the listening time in minutes (supporting Assumption 2).  However, basing recommendations on these values will reward too much longer episodes over shorter ones.  In this case, B is valued six times what A is.  We could give both A and B ratings of 1 (in line with Assumption 1) indicating that Alice listened to 100% of both episodes.  This violates Assumption 2 that B should get a higher rating than A.

Bob listens to all five minutes of A and only five minutes of B.  We could give both of them ratings of 5, the listening time.  However, finishing A shows more satisfaction than skipping out of B 17% of the way into it, which goes against the spirit of Assumption 1.  We could give A a value of 1 and B a value of 5/30 = .17.  This follows Assumption 1 but emphasizes episode A too much by giving it six times the rating of B when they both captured the same amount of the user’s time.

What we need is a rating that is somewhere between listening time and percent completed.

Percent time completed is calculated as

We can “calculate” listening time as

To get something in between, we need to find the exponent e on duration so that 0 < e < 1.

Here’s the trick:  we need to choose two pairs of (listening time, duration) that we want to have the same value.  This gives us an equation

with solution

Suppose you want the same Time Scaled Value for listening to 90% of a two minute story and half of a thirty minute podcast.  Listening time 1 is 1.8 minutes and listening time 2 is 15 minutes.  Solving, this gives us e = .783.  Using this, consider the TSV for listening 1.8 minutes to episodes of varying length. We get

This makes sense: as the duration increases, the fraction of the episode heard decreases, so the value decreases.  Yet, it’s not as fast a decrease as if we just used the percent time completed.  You can see this by looking at, say, the ratio of the TSVs for 5 minutes and 30 minutes: .511 / .126 = 4.07.  If we used listening time, the ratio would be 1; if we used percent time completed the ratio would be 6.

Does this make sense?  Listening to 1.8 minutes of a 30 minute episode shows that you heard the intro and the very beginning of the story, but passed on the rest.  It shows interest, but not much commitment.  Listening to 1.8 minutes of a two minute story means you heard most of the story, but 1.8 minutes isn’t very long; you might have just toughed it out rather than reach for your phone to skip the story.

Empirically, we’ve found that .912 works well for the app we’re developing.  The ranking of TSVs mirrors other measures of success like the rate at which stories are shared on social media.

If you’re looking for something that gives more weight to longer content but not so much that it swamps the percent of the content, Time Scaled Values are worth a look.