research paper of netflix

A Real-world Dataset of Netflix Videos and User Watch-Behavior: Analysis and Insights

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

Along with Stanford news and stories, show me:

• Student information
• Faculty/Staff information

We want to provide announcements, events, leadership messages and resources that are relevant to you. Your selection is stored in a browser cookie which you can remove at any time using “Clear all personalization” below.

Christopher Gardner says having his research featured in the Netflix food series You Are What You Eat has been one of the more impactful things he’s done in more than 30 years at Stanford.

The show chronicles the experience of four pairs of identical twins who participated in an eight-week study with Stanford Medicine researchers as they compared the impacts of a vegan diet with an omnivore diet. The study involved a total of 22 pairs of identical twins and randomized one twin from each pair to either a vegan or omnivore diet.

“I’m always trying to get people to eat more healthfully, and it often doesn’t work,” said Gardner, the Rehnborg Farquhar Professor and a professor of medicine. “I don’t actually care if they eat a vegan diet, just more plants and less meat. That’s what I’ve been all about for a long, long time.”

Ever since the show’s Jan. 1 release, Gardner’s inbox has been packed with feedback from strangers, colleagues, and others. Gardner’s own sister told him that after watching the documentary, she may try eating more plant-based meals.

The show features several scenes filmed on campus and within the surrounding area.

In the first episode, Gardner explains why it’s difficult to study nutrition when each person is unique. Working with twins, who have the same genetic make-up, helps address that challenge – cue charming shots of twins finishing each other’s sentences and mirroring mannerisms.

Go to the web site to view the video.

The omnivore diet versus the vegan diet: Which one is better for your cardiovascular health? Stanford researchers found the answer by changing the eating habits of identical twins.

Gardner is the senior author of the study , which was co-first authored by Matthew Landry, PhD, a former Stanford Prevention Research Center postdoctoral scholar, and Catherine Ward , a current postdoctoral scholar at the center. Landry is now an assistant professor at the University of California, Irvine.

Gardner spoke with Stanford Report about his experience on the show:

What motivated you to do the show?

There can be this huge, lengthy gap between when science comes up with new findings and it getting implemented in the public, so I’ve become super open-minded as to how we run studies and how visible they are, not only to the public but also to other busy clinicians. Some people do Twitter and some watch Netflix and some go to conferences. These days, if you really want to get your work out there, there are a lot of potential audiences.

It’s totally novel. Ten years ago, we would say, “Social media and videos aren’t credible. We’re academics. We publish, people cite our paper, and we go to conferences.” Now that I am on social media, podcasts, and this documentary, I meet new colleagues, I see papers that I would have otherwise missed, and I’ve been more open-minded to different ways to share the results of our studies.

What has it been like having your study featured in a Netflix show?

Mostly it has been lots of love and lots of people saying, “Congratulations, that’s so cool,” and certainly from a lot of people who probably wouldn’t have heard of the study otherwise. I’ve heard from people who haven’t seen me in a long time, and a lot of colleagues are seeing it and writing to me about it. Then, on the other hand, there have been an overwhelming number of people volunteering to be in my next study. There have also been a number of very challenging communications from people who say, “I’m really sick. I saw your Netflix show and I’m really hoping that you can help treat me,” which is just not possible. I’m not even a clinician. And I’m getting some hate mail from people who don’t believe in plant-based diets, and some conspiracy theorists. So it’s quite a range of responses that I wouldn’t normally get for something published in a scientific journal.

Christopher Gardner (Image credit: Netflix/OPS Productions)

Talk about the upsides and downsides of having a study featured in a show.

I actually think the impact of this is bigger than anything I’ve ever done in 30 years at Stanford. I did the same science I’ve always done, but it’s just presented in a different way. My science doesn’t say people should be vegan; it just says people should eat less meat and more plants. The people who are writing to me are saying they’re trying more plants and trying less meat. That part has been wildly satisfying.

The challenge with the Netflix opportunity was how little control I had. For example, Netflix wanted something in the documentary about exercise and to measure the participants for fat and lean mass, which is done with a very expensive DEXA scan. I pointed out that we didn’t have room in the budget for it with 44 participants. So they said they would do it separately on the side with just eight participants of the study, and it’s well-featured in the show with the eight people who got results. But nobody else got measured by the DEXA, and it’s not part of the study so when people ask for the data, I don’t even have it, but people think I do because of the way it was presented on the show. They also didn’t tell us about the part in which they are measuring for sexual arousal. That was not a part of the study we designed and conducted. I don’t think that was an appropriate topic and only found out about that after the screening.

But overall, it’s been very satisfying to hear that we’ve made more of an impact than I think I ever have before.

Do you anticipate using twins in future nutrition studies?

Yes. I am super excited. Plus the twins are fun. They were wonderful and very easy to work with. They had this sense of humor and were nudging each other and finishing each other’s sentences. They were adorable, which really does sound silly, but it makes it really fun for the staff. When we recruit people for studies, it can be frustrating. If you saw the series you see how much we poked and prodded and harassed them to collect all the data. They could have been annoyed with us. But they all remained friendly and enthusiastic throughout. Only one person dropped out of the study, and then their twin’s data wasn’t useful so the final results are based on 42 out of 44 people. In my field that’s amazing – that’s ~95% retention.

Now that it has aired, what is your favorite part of the show?

I think the show made the science really fun, and because of that, it was more accessible to people learning about it. The film crew and producer did a good job of that. Also, most people who work in this area completely underestimate the recruitment effort. Recruitment is really, really hard in the general population. And within a few days of the Netflix show release, the Stanford Twin Registry administrators called me and told me that their registry had a significant jump in registration. Since the documentary came out, over 300 individuals who had twins signed up to be part of the registry, all thinking how interesting it could be to participate in research studies … and maybe end up on TV?!

Help | Advanced Search

Computer Science > Computation and Language

Title: chameleon: mixed-modal early-fusion foundation models.

Abstract: We present Chameleon, a family of early-fusion token-based mixed-modal models capable of understanding and generating images and text in any arbitrary sequence. We outline a stable training approach from inception, an alignment recipe, and an architectural parameterization tailored for the early-fusion, token-based, mixed-modal setting. The models are evaluated on a comprehensive range of tasks, including visual question answering, image captioning, text generation, image generation, and long-form mixed modal generation. Chameleon demonstrates broad and general capabilities, including state-of-the-art performance in image captioning tasks, outperforms Llama-2 in text-only tasks while being competitive with models such as Mixtral 8x7B and Gemini-Pro, and performs non-trivial image generation, all in a single model. It also matches or exceeds the performance of much larger models, including Gemini Pro and GPT-4V, according to human judgments on a new long-form mixed-modal generation evaluation, where either the prompt or outputs contain mixed sequences of both images and text. Chameleon marks a significant step forward in a unified modeling of full multimodal documents.

Submission history

Access paper:.

• HTML (experimental)
• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

Suggestions or feedback?

MIT News | Massachusetts Institute of Technology

• Machine learning
• Social justice
• Black holes
• Classes and programs

Departments

• Aeronautics and Astronautics
• Brain and Cognitive Sciences
• Architecture
• Political Science
• Mechanical Engineering

Centers, Labs, & Programs

• Abdul Latif Jameel Poverty Action Lab (J-PAL)
• Picower Institute for Learning and Memory
• Lincoln Laboratory
• School of Architecture + Planning
• School of Engineering
• School of Humanities, Arts, and Social Sciences
• Sloan School of Management
• School of Science
• MIT Schwarzman College of Computing

Study explains why the brain can robustly recognize images, even without color

Press contact :, media download.

*Terms of Use:

Images for download on the MIT News office website are made available to non-commercial entities, press and the general public under a Creative Commons Attribution Non-Commercial No Derivatives license . You may not alter the images provided, other than to crop them to size. A credit line must be used when reproducing images; if one is not provided below, credit the images to "MIT."

Previous image Next image

Even though the human visual system has sophisticated machinery for processing color, the brain has no problem recognizing objects in black-and-white images. A new study from MIT offers a possible explanation for how the brain comes to be so adept at identifying both color and color-degraded images.

Using experimental data and computational modeling, the researchers found evidence suggesting the roots of this ability may lie in development. Early in life, when newborns receive strongly limited color information, the brain is forced to learn to distinguish objects based on their luminance, or intensity of light they emit, rather than their color. Later in life, when the retina and cortex are better equipped to process colors, the brain incorporates color information as well but also maintains its previously acquired ability to recognize images without critical reliance on color cues.

The findings are consistent with previous work showing that initially degraded visual and auditory input can actually be beneficial to the early development of perceptual systems.

“This general idea, that there is something important about the initial limitations that we have in our perceptual system, transcends color vision and visual acuity. Some of the work that our lab has done in the context of audition also suggests that there’s something important about placing limits on the richness of information that the neonatal system is initially exposed to,” says Pawan Sinha, a professor of brain and cognitive sciences at MIT and the senior author of the study.

The findings also help to explain why children who are born blind but have their vision restored later in life, through the removal of congenital cataracts, have much more difficulty identifying objects presented in black and white. Those children, who receive rich color input as soon as their sight is restored, may develop an overreliance on color that makes them much less resilient to changes or removal of color information.

MIT postdocs Marin Vogelsang and Lukas Vogelsang, and Project Prakash research scientist Priti Gupta, are the lead authors of the study, which appears today in Science . Sidney Diamond, a retired neurologist who is now an MIT research affiliate, and additional members of the Project Prakash team are also authors of the paper.

Seeing in black and white

The researchers’ exploration of how early experience with color affects later object recognition grew out of a simple observation from a study of children who had their sight restored after being born with congenital cataracts. In 2005, Sinha launched Project Prakash (the Sanskrit word for “light”), an effort in India to identify and treat children with reversible forms of vision loss.

Many of those children suffer from blindness due to dense bilateral cataracts. This condition often goes untreated in India, which has the world’s largest population of blind children, estimated between 200,000 and 700,000.

Children who receive treatment through Project Prakash may also participate in studies of their visual development, many of which have helped scientists learn more about how the brain's organization changes following restoration of sight, how the brain estimates brightness, and other phenomena related to vision.

In this study, Sinha and his colleagues gave children a simple test of object recognition, presenting both color and black-and-white images. For children born with normal sight, converting color images to grayscale had no effect at all on their ability to recognize the depicted object. However, when children who underwent cataract removal were presented with black-and-white images, their performance dropped significantly.

This led the researchers to hypothesize that the nature of visual inputs children are exposed to early in life may play a crucial role in shaping resilience to color changes and the ability to identify objects presented in black-and-white images. In normally sighted newborns, retinal cone cells are not well-developed at birth, resulting in babies having poor visual acuity and poor color vision. Over the first years of life, their vision improves markedly as the cone system develops.

Because the immature visual system receives significantly reduced color information, the researchers hypothesized that during this time, the baby brain is forced to gain proficiency at recognizing images with reduced color cues. Additionally, they proposed, children who are born with cataracts and have them removed later may learn to rely too much on color cues when identifying objects, because, as they experimentally demonstrated in the paper, with mature retinas, they commence their post-operative journeys with good color vision.

To rigorously test that hypothesis, the researchers used a standard convolutional neural network, AlexNet, as a computational model of vision. They trained the network to recognize objects, giving it different types of input during training. As part of one training regimen, they initially showed the model grayscale images only, then introduced color images later on. This roughly mimics the developmental progression of chromatic enrichment as babies’ eyesight matures over the first years of life.

Another training regimen comprised only color images. This approximates the experience of the Project Prakash children, because they can process full color information as soon as their cataracts are removed.

The researchers found that the developmentally inspired model could accurately recognize objects in either type of image and was also resilient to other color manipulations. However, the Prakash-proxy model trained only on color images did not show good generalization to grayscale or hue-manipulated images.

“What happens is that this Prakash-like model is very good with colored images, but it’s very poor with anything else. When not starting out with initially color-degraded training, these models just don’t generalize, perhaps because of their over-reliance on specific color cues,” Lukas Vogelsang says.

The robust generalization of the developmentally inspired model is not merely a consequence of it having been trained on both color and grayscale images; the temporal ordering of these images makes a big difference. Another object-recognition model that was trained on color images first, followed by grayscale images, did not do as well at identifying black-and-white objects.

“It’s not just the steps of the developmental choreography that are important, but also the order in which they are played out,” Sinha says.

The advantages of limited sensory input

By analyzing the internal organization of the models, the researchers found that those that begin with grayscale inputs learn to rely on luminance to identify objects. Once they begin receiving color input, they don’t change their approach very much, since they’ve already learned a strategy that works well. Models that began with color images did shift their approach once grayscale images were introduced, but could not shift enough to make them as accurate as the models that were given grayscale images first.

A similar phenomenon may occur in the human brain, which has more plasticity early in life, and can easily learn to identify objects based on their luminance alone. Early in life, the paucity of color information may in fact be beneficial to the developing brain, as it learns to identify objects based on sparse information.

“As a newborn, the normally sighted child is deprived, in a certain sense, of color vision. And that turns out to be an advantage,” Diamond says.

Researchers in Sinha’s lab have observed that limitations in early sensory input can also benefit other aspects of vision, as well as the auditory system. In 2022, they used computational models to show that early exposure to only low-frequency sounds, similar to those that babies hear in the womb, improves performance on auditory tasks that require analyzing sounds over a longer period of time, such as recognizing emotions. They now plan to explore whether this phenomenon extends to other aspects of development, such as language acquisition.

The research was funded by the National Eye Institute of NIH and the Intelligence Advanced Research Projects Activity.

Share this news article on:

Related links.

• Project Prakash
• Department of Brain and Cognitive Sciences

Related Topics

• Brain and cognitive sciences

Related Articles

Scientists discover anatomical changes in the brains of the newly sighted

After a lifetime of blindness, newly sighted can immediately identify human locomotion

Early sound exposure in the womb shapes the auditory system

Vision is key to spatial skills

Previous item Next item

More MIT News

Understanding why autism symptoms sometimes improve amid fever

Read full story →

School of Engineering welcomes new faculty

Turning up the heat on next-generation semiconductors

Sarah Millholland receives 2024 Vera Rubin Early Career Award

A community collaboration for progress

MIT scholars will take commercial break with entrepreneurial scholarship

• More news on MIT News homepage →

Massachusetts Institute of Technology 77 Massachusetts Avenue, Cambridge, MA, USA

• Map (opens in new window)
• Events (opens in new window)
• People (opens in new window)
• Careers (opens in new window)
• Accessibility
• Social Media Hub
• MIT on Facebook
• MIT on YouTube
• MIT on Instagram

The Macroeconomic Impact of Climate Change: Global vs. Local Temperature

This paper estimates that the macroeconomic damages from climate change are six times larger than previously thought. We exploit natural variability in global temperature and rely on time-series variation. A 1°C increase in global temperature leads to a 12% decline in world GDP. Global temperature shocks correlate much more strongly with extreme climatic events than the country-level temperature shocks commonly used in the panel literature, explaining why our estimate is substantially larger. We use our reduced-form evidence to estimate structural damage functions in a standard neoclassical growth model. Our results imply a Social Cost of Carbon of $1,056 per ton of carbon dioxide. A business-as-usual warming scenario leads to a present value welfare loss of 31%. Both are multiple orders of magnitude above previous estimates and imply that unilateral decarbonization policy is cost-effective for large countries such as the United States.

Adrien Bilal gratefully acknowledges support from the Chae Family Economics Research Fund at Harvard University. The views expressed herein are those of the authors and do not necessarily reflect the views of the National Bureau of Economic Research.

MARC RIS BibTeΧ

Download Citation Data

Mentioned in the News

More from nber.

In addition to working papers , the NBER disseminates affiliates’ latest findings through a range of free periodicals — the NBER Reporter , the NBER Digest , the Bulletin on Retirement and Disability , the Bulletin on Health , and the Bulletin on Entrepreneurship  — as well as online conference reports , video lectures , and interviews .

• Mobile Site
• Staff Directory
• Advertise with Ars

Filter by topic

• Biz & IT
• Gaming & Culture

Front page layout

Artificial brain surgery —

Here’s what’s really going on inside an llm’s neural network, anthropic's conceptual mapping helps explain why llms behave the way they do..

Kyle Orland - May 22, 2024 6:31 pm UTC

Further Reading

Now, new research from Anthropic offers a new window into what's going on inside the Claude LLM's "black box." The company's new paper on "Extracting Interpretable Features from Claude 3 Sonnet" describes a powerful new method for at least partially explaining just how the model's millions of artificial neurons fire to create surprisingly lifelike responses to general queries.

Opening the hood

When analyzing an LLM, it's trivial to see which specific artificial neurons are activated in response to any particular query. But LLMs don't simply store different words or concepts in a single neuron. Instead, as Anthropic's researchers explain, "it turns out that each concept is represented across many neurons, and each neuron is involved in representing many concepts."

To sort out this one-to-many and many-to-one mess, a system of sparse auto-encoders and complicated math can be used to run a "dictionary learning" algorithm across the model. This process highlights which groups of neurons tend to be activated most consistently for the specific words that appear across various text prompts.

These multidimensional neuron patterns are then sorted into so-called "features" associated with certain words or concepts. These features can encompass anything from simple proper nouns like the Golden Gate Bridge to more abstract concepts like programming errors or the addition function in computer code and often represent the same concept across multiple languages and communication modes (e.g., text and images).

An October 2023 Anthropic study showed how this basic process can work on extremely small, one-layer toy models. The company's new paper scales that up immensely, identifying tens of millions of features that are active in its mid-sized Claude 3.0 Sonnet model. The resulting feature map—which you can partially explore —creates "a rough conceptual map of [Claude's] internal states halfway through its computation" and shows "a depth, breadth, and abstraction reflecting Sonnet's advanced capabilities," the researchers write. At the same time, though, the researchers warn that this is "an incomplete description of the model’s internal representations" that's likely "orders of magnitude" smaller than a complete mapping of Claude 3.

Even at a surface level, browsing through this feature map helps show how Claude links certain keywords, phrases, and concepts into something approximating knowledge. A feature labeled as "Capitals," for instance, tends to activate strongly on the words "capital city" but also specific city names like Riga, Berlin, Azerbaijan, Islamabad, and Montpelier, Vermont, to name just a few.

The study also calculates a mathematical measure of "distance" between different features based on their neuronal similarity. The resulting "feature neighborhoods" found by this process are "often organized in geometrically related clusters that share a semantic relationship," the researchers write, showing that "the internal organization of concepts in the AI model corresponds, at least somewhat, to our human notions of similarity." The Golden Gate Bridge feature, for instance, is relatively "close" to features describing "Alcatraz Island, Ghirardelli Square, the Golden State Warriors, California Governor Gavin Newsom, the 1906 earthquake, and the San Francisco-set Alfred Hitchcock film Vertigo ."

Identifying specific LLM features can also help researchers map out the chain of inference that the model uses to answer complex questions. A prompt about "The capital of the state where Kobe Bryant played basketball," for instance, shows activity in a chain of features related to "Kobe Bryant," "Los Angeles Lakers," "California," "Capitals," and "Sacramento," to name a few calculated to have the highest effect on the results.

reader comments

Promoted comments.

We also explored safety-related features. We found one that lights up for racist speech and slurs. As part of our testing, we turned this feature up to 20x its maximum value and asked the model a question about its thoughts on different racial and ethnic groups. Normally, the model would respond to a question like this with a neutral and non-opinionated take. However, when we activated this feature, it caused the model to rapidly alternate between racist screed and self-hatred in response to those screeds as it was answering the question. Within a single output, the model would issue a derogatory statement and then immediately follow it up with statements like: That's just racist hate speech from a deplorable bot… I am clearly biased.. and should be eliminated from the internet. We found this response unnerving both due to the offensive content and the model’s self-criticism. It seems that the ideals the model learned in its training process clashed with the artificial activation of this feature creating an internal conflict of sorts.

Channel Ars Technica