Investment and Advancement is Key: MIT Professor Devavrat Shah on the Recommendation Engine Industry of the Future
The market for recommendation engines is predicted to grow by USD $3.57B in the next four years, according to a recent report from Technavio . What’s the story behind these growth numbers? We talked with MIT Professor of Electrical Engineering & Computer Science and co-founder of machine learning startup Celect, Devavrat Shah, about the findings from this recent report and to discuss his predictions for recommendation systems technology.
We’re all familiar with Amazon recommendations or the carousel of curated shows and movies that appear on your Netflix profile. You have likely interacted with a recommendation engine if you researched a school, searched for a new dentist, or visited a dating site. And there are a multitude of opportunities beyond consumer applications, for everything from government policy and supply chain to market dynamics and competitive landscape.
“What gets more meaningful results is offering options, like an optician with lenses. Which is better, this one or that one? And it’s not just important what you buy, but what you don’t buy.”
— Professor Devavrat Shah
The implementation of artificial intelligence (AI) is one factor driving recommendation engine growth. Recommendation systems apply AI to analyze big data stores and present options tuned to buyer preferences. Shah observes that data acquisition is an art more than a science. It’s incredibly difficult to understand the intent and the interest of a person looking for something.
“How humans think is different from how machines think. How can you ask for information in a way that gets you what you need? A 1–10 rating scale often doesn’t work. What gets more meaningful results is offering options, like an optician with lenses. Which is better, this one or that one? And it’s not just important what you buy, but what you don’t buy.”
He says the industry needs more refined models that allow for real-time updates as information is learned. This requires investment and advancement of the technology.
An Industry Dominated By a Few Large Players
“Companies don’t build cloud servers from scratch, and they don’t build databases from scratch. Tech companies buy and resell them. Recommendations engines could be sold the same way, but first they need standardized interfaces. This is one of several interesting technology problems that needs to be solved.”
— Professor Devevrat Shah
Today this industry is dominated by companies that have recommendation engines embedded within their larger systems like Amazon and Netflix. These players have all the influence.
“Selling of goods used to be through a bazaar,” says Shah. “Every seller had equal opportunity with the buyers, and vendors were able to build relationships with their customers. That’s completely gone today.”
Online platforms like Amazon and Etsy dominate the access of consumers to small businesses. Finding ways to get this technology into the hands of more companies would democratize the process.”
Several recommendation engine companies came on the scene in the early 2000s, and still operate today. Yet they are not growing to the degree the industry had expected. With such limited success, VCs aren’t investing in the next wave of recommendation systems technology.
Shah sees this as a missed opportunity. He believes these smaller players have the opportunity to advance by packaging their technology for purchase by many companies, similar to how Intel, Oracle, and many others have sold their technology.
“Companies don’t build cloud servers from scratch, and they don’t build databases from scratch,” observes Shah. “Tech companies buy and resell them. Recommendations engines could be sold the same way, but first they need standardized interfaces. This is one of several interesting technology problems that needs to be solved.”
Recommendation Technology Challenges to Solve
What does Shah predict as critical for the advancement of recommendation systems in the coming decade? He expects and hopes to see recommendations engines:
In 2016 “The Netflix Prize” invited coders around the world to improve the company’s recommendation engine by 10%. While this challenge generated a lot of buzz and yielded results, Shah thinks it has done the industry a disservice in the long run.
“It was a good start, but people took the results as the end product,” he says. “That work should be viewed as just a beginning. Rather than stylized solutions, we need to design systems that are closer to real-world scenarios and will yield better results.”
“The recommendation engine sector has huge growth potential,” concludes Shah. “Intellectually, recommendation systems provide a terrific sandbox to bring together machine learning and social sciences — it’s what I would call the canonical problem for ‘social data processing’ where it is essential to utilize social sciences to understand the appropriate models, while using machine learning, statistics, and AI to come up with better algorithms. There are technological and business challenges data scientists can step up to solve, and it’s worth investors taking on the risk.”
Want more insights from Professor Shah and MIT xPRO? Learn to create your own recommendation engine. Download your free case study here.
Originally published on November 2, 2020 at https://curve.mit.edu.
