Auditing the News
Evaluating News Quality on Smart Speakers
Alexa, Siri, Google Home, Cortana—smart speakers and agents are now used by about 20% of US homes. People use them to ask about weather, set timers, play games, get information, or listen to the news. But are these devices delivering high-quality news and information or could they be misinforming and sharing “junk” news? This project aims to find out. By developing an audit method that defines what queries to audit and systematically collects data on the results over time for those queries from several different smart speakers, the project will allow for an assessment and comparison of news quality from these different devices.
Faculty and Staff Leads
Project Details
2019 Winter
Fact Flow
Description
Alexa, Siri, Google Home, Cortana—smart speakers and agents are now used by about 20% of US homes. People use them to ask about weather, set timers, play games, get information, or listen to the news. But are these devices delivering high-quality news and information or could they be misinforming and sharing “junk” news? This project aims to find out. By developing an audit method that defines what queries to audit and systematically collects data on the results over time for those queries from several different smart speakers, the project will allow for an assessment and comparison of news quality from these different devices.
Important Questions
- What are the important queries to audit that relate to how people ask for news information from smart speakers?
- How can data be systematically collected? Do APIs exist or will data need to be manually collected? Also, how often should that data be collected — do devices change their results every minute, or hour?
- How should results be tabulated and evaluated? Can results be tagged according to their news sources, ideology, or credibility?
Sample Milestones
- Week 1-2 Background research on algorithm auditing; development of query input strategy.
- Week 3-5 pilot data collection from at least 2 different smart speakers; iteration on methods to ensure reliable data collection; assessment of frequency of data collection needed; development of storage scheme for data
- Week 6-7 data collection from at least 2 different smart speakers for an extended period of at least 1 full week; begin analysis and tagging of pilot data
- Week 8-9 analysis, comparison, and presentation (e.g. visualization) of final data; write-up a news article to pitch to Nieman Lab or similar outlet about implications of findings
Outcome
The development of a method for auditing news on smart speakers as well as application of that method to collect and analyze data to assess news quality.
Students
