How to Tell If the Answer AI Just Gave You Is Any Good

Last month I asked AI to research a funder and it told me not to apply. I trusted that answer only because I went and checked the eligibility page myself.

Regular readers saw that story last week, so I will keep the recap short. The tool found a geography mismatch and a two year lockout after a decline, both of them sitting in documents I didn’t read carefully myself. It was right. But the reason I could act on what it told me is that I spent about four minutes verifying the two facts that actually drove the decision.

That verification step is what this piece is about, and it is the step I see skipped often.

This is part one of four. Over the next month I am writing the guide I wish someone had handed me when I started, and I am beginning here rather than with tools or prompts, because the rest of it is hard to use without this part.

The trouble with fluent answers

AI writes well. That is the first thing most people notice, and it is also where the trouble starts, because we are trained to read fluency as a signal of competence. When a well-organized, confident, grammatically clean paragraph arrives on your screen, some part of your brain has already filed it as reliable before you have evaluated a single claim inside it.

A wrong answer and a right answer look exactly the same. There is no visual difference between the sentence that is true and the sentence that got assembled because it sounded like the kind of thing that would be true.

In a lot of contexts that costs you very little. In our work it lands on grant narratives, donor communications, board reports, and program data, where being wrong is expensive and tends to stay invisible for months. A funder does not write back to tell you which sentence in your application was inaccurate. They just decline you.

What literacy actually means here

When people talk about building AI literacy, most of them mean understanding roughly how the technology works. I have read a fair number of those explanations. They are interesting, and they did very little for the quality of my decisions.

The literacy that changed my work is narrower than that. It is the ability to look at something AI just produced and know which parts you can act on and which parts you have to go check. That skill sits much closer to editing than to engineering, which is good news for nonprofit people, because most of us have spent years editing other people’s work already.

You are mostly applying judgment you already have to a source you are not yet in the habit of doubting.

Four questions to ask of anything AI hands you

1. Can I name the source, and can I go look at it?

Ask where a claim came from, and then actually open the page. If it cannot point you somewhere specific, treat the claim as a hypothesis rather than a fact. This one catches a lot, because the failure is rarely a wild invention. It is usually a plausible detail with no particular origin.

2. Would I have caught this if a person wrote it?

You already have editorial standards, and you apply them to staff work and proposal drafts without thinking about it. The odd thing about AI output is how readily we suspend those standards for it. Read the page as though a colleague handed it to you and your name is going on it, because in every way that matters, that is the situation.

3. What would this look like if it were wrong?

Ask directly for the counter-case. In funder research that means asking for the three reasons this foundation would decline you. In a program report it means asking which conclusion the data does not actually support. It takes one sentence to turn an agreeable assistant into one that will argue with you, and the argument is usually where the useful part is.

4. Is this a fact question or a judgment question?

Facts need verification against a source. Judgment needs your context, and it is not something you can hand off. Whether a funder’s deadline is October 1 is a fact, and you can check it in thirty seconds. Whether your program story is strong enough to spend your one annual application on is a judgment, and it belongs to the people who know the program.

Most bad outcomes I have seen come from treating the second kind of question as though it were the first.

The two-week exercise I would start with

Here is the part I actually want you to do, and it is going to sound like a waste of time.

For the next two weeks, use AI only on tasks where you already know the right answer.

Take a grant you already won and ask it to assess your fit against that funder. Take last year’s year-end appeal and ask it to segment the donor list you already segmented by hand. Take a program report you wrote yourself and ask it to pull out the outcomes.

You are not trying to save any time with this. You are calibrating. You are watching where it is genuinely sharp, where it is confidently vague, and where it quietly manufactures a detail because the shape of the answer seemed to call for one. Two weeks of that gives you a feel for it that is hard to get any other way.

The reason it works is that it drops your checking cost to zero. When you already know the answer, verification is instant, so you can afford to run twenty of them in an afternoon. The moment you point it at ground you cannot check, every verification becomes expensive, and expensive checks are the ones people stop doing. That is usually when a fluent wrong answer gets through.

Most people go about this in the opposite order. They hand AI the hardest thing on their desk on day one, get back an answer they have no way to evaluate, and then either trust it more than they should or decide the whole category is overhyped. Both of those outcomes come from the same missing step.