From Feed to Discovery
Why recommendation algorithms are broken and how to fix them
The recommendation algorithm is broken. It is a result of first-order thinking. The algorithm works by recommending the-most-similar-thing™ immediately after you consume some content. It is an infinite ingest of the same. It feeds you and feeds you and feeds you. This is wrong. This isn’t how humans like to consume, and this is causing social problems.
First, how do humans like to consume? Does the recommendation algorithm accurately model what people like? After you eat a baked potato, do you want another baked potato… and then another?
Let’s look at the Beatles’ album Sgt. Pepper’s Lonely Hearts Club Band as an example of how humans consume content globally. What are its credentials for doing so? The Beatles are the best-selling music act in history. Still. Still no one in all history has sold more records than the Beatles. Not Beyonce, not Taylor Swift. More humans spent their money on Beatles records than any other act today, and there are more than double the people on the planet now than when the Beatles broke up in 1970. Sgt. Peppers is their best selling album–32 million records sold.
Planet Earth decided, pre-algorithm, to consume Sgt. Pepper’s as their next album to buy. Why? Was it the most similar thing to the last album that all record player-owning humans consumed? No. The Beatles sold 32 million records because there is literally no musical creation anything like Sgt. Pepper’s. What are the myriad of ways this album differed from anything before it?
It is one of the first concept albums. The album in its entirety has a theme that runs across it and synthesizes the record into a cohesive statement. Until then, albums had merely been a collection of songs.
The album helped turn the album cover into album art. The cover of the album itself became a work of art that could be inspected to incredible detail with the myriad of iconic faces, flowers, costumes, objects. The cover is a radical departure from what before was basically a straightforward advertisement of the contents of the album.
Every song matters. The Beatles didn’t treat the album as a collection of hits and filler. There is no ‘content’. Every song is an individual work of art.
Every song varies from the last. Every song is still completely different from the preceding one and from each other across the entire album. Yet each song perfectly switches the mood. Rocker? Check. Classical Harp? Check. Granny Jazz? Check? Far out psychedelia? Check. Indian Raga? Check. Avant Garde hodge-podge that makes you question the very nature of songwriting and life itself? Check. Who in music today does this? Look at current Spotify numbers. Can you believe that A Day in the Life is actually the 3rd most listened to song on the album? Really??
As the most successful album by the most successful musical act, Sgt. Pepper’s is proof that what people desire from their culture is expansion and growth, dynamism and eclecticism. The Beatles’ audience surely learned to appreciate a new music genre they didn’t know before. “Within You, Without You–what is that instrument? A sitar? What’s a raga? Who is Ravi Shankar?” And now millions of people get turned on to Indian Classical Music who wouldn’t have otherwise.
What the algorithm today provides is the walls closing in. You wonder why trap music is in decline? How is any song by a given artist different from any other? Different is refreshing. Different is discovery. Different is progress. Same is bland. Same is limitation. Same is stagnation. And you hear it in today's music. You see it in your aunt’s social media posts. You see it in today’s politics. Social media is a solitary hall of mirrors and we cannot live off of mirrored glass.
So how do social media, movie streaming, music streaming, news and others go from feed to discovery? How do we program our computers to give people what they want, while expanding their horizons? After all, for all their experimentation and innovation, what The Beatles wrote still boils down to pop tunes. How can we program variation and discovery into the architecture of an algorithm? I think we can go to the 80/20 rule for guidance in creating natural ratios of content delivery.
Discovery:
|--------------------80% things I like and similar ------------—----| |-----20% I don’t know or dislike—--|
Let’s break it down from here.
Within the 80% of things I like and know, you can recommend
|--------------------80% my favorite things ------------—---------| |-----20% Related things —--|
My favorite things being the algorithm as it stands now. I like The Beatles, the algorithm should recommend it to me.
Related things could be The Rolling Stones. I’ve heard of them, but I don’t know their music too well. I should check it out!
Within the 20% of things I don’t know or like, you can recommend
|--------------------80% Things I don’t know —------------—----| |-----20% Things I don’t like —--|
Things I don’t know could be how to sew a cross stitch. How to play polo. What are the most recent fish discovered. Who is the new artist in town.
Things I don’t like. Opposing POVs. Not what is most infurating, but the steel man argument for the other side: Why Rush is better than The Beatles. It’s wrong, but I can understand why someone might think that.
This discovery portal now gives you overall:
64% my favorite things
16% things related to my favorite things, but I haven’t searched for yet
16% things I don’t know.
4% things I don’t like
Algorithms are different in every platform and this discovery formula doesn’t have to be the real rule, but it at least gives people a chance to see something different. Give people what they want, but also confront them with something new and even opposing ideas. I never knew I could enjoy Indian Classical Music until I was confronted with it in Sgt. Pepper’s Lonely Hearts Club Band and it is a beautiful counterpoint to their other creations.
As the world becomes more reliant on software, software will shape our culture. We need to create software systems that model and open humans up to the best of what’s out there. The walls aren’t closing in. There’s a whole world outside your window, waiting for you to discover.
