#515 – AI and Research, Step 4: Evaluating Information

#515 — AI and Research, Step 4: Evaluating Information

AVID Open Access
15 min

Transcript

The following transcript was automatically generated from the podcast audio by generative artificial intelligence.  Because of the automated nature of the process, this transcript may include unintended transcription and mechanical errors.

Paul Beckermann 0:01
Welcome to Tech Talk for Teachers. I’m your host, Paul Beckermann.

Transition Music with Rena’s Children 0:05
Check it out. Check it out. Check it out. Check it out. What’s in the toolkit? What is in the toolkit? What’s in the toolkit? Check it out.

Paul Beckermann 0:16
The topic of today’s episode is AI and research, Step 4: Evaluating Information. This is part four of a seven-part series exploring the possible integration of AI into the research process. So far, we’ve checked off three boxes in that research process: we’ve defined our task, we’ve developed a preliminary plan by gaining some general background knowledge, and we’ve identified the types of sources that may best match our research goals.

Step four builds off these first three steps, and in many ways, it may be one of the most important skills our students can develop in this age of hyper-available information. In this fourth step, students need to actively find credible content that will help them support their research question or proposed thesis statement. Now that they’ve identified the types of sources to target, they need to dig into each type of resource and find relevant articles, entries, titles, and content.

The process of using a database or conducting an effective web search are important skills in themselves, and each research tool requires users to know how to navigate and search that platform. Because these tools and platforms are continually changing, your library media specialist is probably your best resource in this area. Rather than trying to keep up on all of these yourself, it can be more effective to bring those experts into your classroom to teach or co-teach lessons on using the specific tools.

Therefore, I’m going to bypass the details of each of those specific tools and instead focus on the broader skill sets that apply to any search tool your students may be using. For those specific database skills, again, I’d suggest leaning on your library media specialist for support and skill-building. As for those broader skills, we’ll examine transferable skills students will need to apply as they research. This includes the continual evaluation of the information they find for relevance, credibility, quality, and completeness. Let’s break down each of those four important criteria in more detail, one at a time, and explore how AI may be effectively woven into that process.

Transition Music with Rena’s Children 2:30
Let’s count it. Let’s count it. Let’s count it down.

Paul Beckermann 2:33
Number one: relevance. A source can be accurate, credible, and well-written, and still not be useful for the specific research task at hand. It must also be relevant to the research goals. As students focus on relevance, they need to ask questions like, “Does this information actually help me answer my research question?” and “Does it connect to one of the areas in my preliminary outline?”

This is where the planning students did earlier becomes especially valuable. Specifically, the preliminary outline that students created in step two can now function like a research topic checklist. Students need to find information to support each section of information identified in that outline. As students encounter information about their broader topic area, they should ask if and where it fits into their plan. If they can’t identify a clear connection, then the source is likely something they do not need. In other words, it’s not relevant.

Students need to resist the urge to simply gather the first batch of resources that are loosely connected to their topic. They should narrow their lens and examine each item they find to determine if it’s actually relevant to their narrowed purpose. Having their preliminary plan readily accessible while they research can be very helpful.

As for AI, it can be helpful here as a thinking partner and to help students perform an initial relevance scan. For instance, students might provide an AI tool with their research question and preliminary outline, along with the text from a source or perhaps initial notes from that source, and ask questions like, “Is this content relevant to my search question? If so, in what ways?” or “Which sections of my outline could this source potentially support?”

One tool that can be especially useful while doing this is Google’s Gemini Notebook, which was previously named NotebookLM. Students can upload multiple sources into their notebook and then ask specific questions about individual resources or the resources as a group. This is a great tool for conducting this initial vetting for relevance, and it can be a big time-saver. Resources that are not relevant can be removed from the notebook, while relevant content can remain for deeper exploration later in the process.

Of course, you certainly don’t have to allow students to use AI during this phase of the research process. That’s your choice, but if you do allow it, it’s important to stress that the students themselves must make the final decisions about content relevance. The AI can help to save time by scanning long documents or by identifying alignment with a preliminary outline, but students must not relinquish their decision-making role in the process.

If, in this initial scan, students think the content will be relevant, they should remember that they might still change their mind as they dig deeper and find additional research. The process is fluid, and opinions about relevance can change along the way. As they begin reading, listening, or viewing the content, students may find that their initial decision was inaccurate—that’s perfectly acceptable and common.

Number two: credibility. This is probably the area most people immediately think about when we talk about evaluating information, and it is incredibly important. When we say credibility, we’re essentially saying trustworthiness and accuracy. Can I trust this information? Is it accurate and reliable? To determine credibility, students should ask targeted questions like: Who created this information? What expertise or experience do they have? Who published it? What evidence is provided? Can important claims be verified somewhere else, and why is this information being shared?

Another important check is to see what happens when we leave the source itself and look at what other sources say about it. This process of reading across multiple resources for consistency and fact-checking is called lateral reading, and it’s an important skill for students to develop. While students can stay on a website and look for an “About” page to find out more about the author or organization, it’s equally important to open additional tabs and investigate what other sources are saying about the author, organization, publication, or claims. This can actually be more insightful and trustworthy than what the source says about itself.

AI can support this process, but it’s also an area where students need to be especially careful about outsourcing judgment. A student might ask AI, “What questions should I investigate to determine whether this organization is a credible source on the topic?” That’s very different than asking, “Is this source credible?” The first prompt gives students an investigation plan and helps them build their research skill set. The second hands the judgment over to the AI. We want students doing the judging.

Additionally, AI-generated information itself should be evaluated using those same habits. If an AI tool gives students a claim, statistic, quotation, or explanation, that output doesn’t get a free pass simply because it sounds confident. Students should trace important information back to reliable sources whenever possible. Students should ask, “Where did AI pull this information from, and did it do so accurately?”

This is true both when interacting with an AI chatbot as well as when students review AI-generated web search summaries. Students may be tempted to blindly accept the accuracy of those summaries because they sound authoritative and reasonable, but they should be aware that they may, and sometimes do, contain errors and inaccuracies.

Number three: quality. Credibility and quality overlap, but there’s an important distinction. A source may come from a credible organization and still not be particularly useful or strong. Students must look at the quality of the actual information itself, not just the quality of the source. Even the most respected experts can drift off-topic or share an idea that has not yet been fully developed, vetted, or supported.

Some key questions a student can ask are: Does this author provide evidence? Are claims explained and supported? Is the information specific rather than vague? Does it acknowledge complexity or competing perspectives? Are sources cited, and does the information appear to be primarily informative, persuasive, promotional, or entertaining? Students might also compare several sources covering the same idea or claim in order to ensure there’s agreement on the issue.

That comparison can be incredibly powerful, and students may encounter topics where there are conflicting opinions. It’s important that they understand this and are able to distinguish these nuances in the information they find. Even when two sources agree on the core ideas, students can still ask themselves which of these sources provides the strongest evidence for this part of my research and why.

AI can again serve as a thinking partner in this process. Students might paste text or notes from two or three sources into an AI tool and ask it to identify similarities, differences, missing evidence, or claims that deserve further investigation. They might ask, “What important differences do you notice in how these three sources explain this issue?” or “What claims appear in one source but not the others that I should investigate further?” or perhaps even more importantly, “Are there claims in one source that are refuted or contradicted in another?”

With this approach, AI is helping students notice things, and each of those things requires a student to investigate further and make the final decision about the quality of the content. Again, Gemini Notebook can be an effective tool for this.

Number four: completeness. This is an evaluation skill that sometimes gets overlooked. Students may find several credible, high-quality sources and assume their research is finished. However, the question they need to be asking is, “Do I have enough information to understand this topic well?”

This is where students should return again to their preliminary outline and ask questions like: “Do I have enough information to support each section of my outline? Do I need to revise my outline or find more information? Am I relying too heavily on one source? Do all my sources represent essentially the same perspective? Are there important voices or viewpoints missing? Are there questions I still cannot answer?”

While the preliminary outline is very helpful and should be used as a guide, it’s not infallible either. After all, students created this at the beginning of the process before they dug more deeply into the content. As they learn more, they may realize they missed the mark when they created their initial plan. It’s okay, and even expected, that they will revise this plan as they progress through the research process and learn more about their topic.

This is another place where AI can be useful. Students can provide their research question, outline, and notes, and ask, “What gaps did you notice in the information I’ve collected so far?” or “What additional perspectives might help me develop a more complete understanding of this topic?”

If students are using Gemini Notebook, they can check off all their resources and then paste in a copy of their preliminary plan along with their question. This neatly limits the AI response to those related resources. During this review, AI might identify something the student overlooked. If there are gaps, the student knows to continue researching, or they might determine that the section is not necessary and can be removed from the plan. Once again, the AI response is a suggestion to investigate, not an answer to automatically accept.

Across all four of these criteria, there’s a larger skill we’re trying to develop. We want students to become thoughtful and critical consumers of information. As the information landscape continues to expand, these skills matter more now than ever before. Students can now use online tools to generate an explanation in seconds. Search engines can summarize results before students ever visit a source. Social media algorithms continually place information in front of them, and increasingly, students may not even know whether the words, images, audio, or video they encounter were created by a person, generated by AI, or produced through some combination of the two.

That makes evaluation less of a step students complete during a research process and more of an ongoing habit of mind whenever they encounter information. As students complete this step of the process, they should develop the habit of asking four key questions every time they’re presented with new information: Is this relevant? Is this credible? Is it high-quality? And do I have enough information to understand the issue well?

AI can help students ask those questions, compare information, identify gaps, and surface things worth investigating. But this is one part of the research process where we should be particularly careful not to outsource the important thinking to AI. In fact, you could have students complete the step without any AI use at all—that’s an option. However, we do know that students are increasingly turning to AI as a thought partner, whether we explicitly allow it or not.

Because of that, it might be prudent to guide them through what responsible AI use can look like in the research process to set them up for success researching when we’re not there to guide them.

To learn more about today’s topic and explore other free resources, visit avidopenaccess.org. Specifically, I encourage you to check out the article collection, “AI in the K-12 Classroom,” and of course, be sure to join us next Wednesday for our full-length podcast, Unpacking Education, where we’re joined by exceptional guests and explore education topics that are important to you. Thanks for listening. Take care, and thanks for all you do. You make a difference.