AI Is Going to Do What AI Does. Here’s How We Design Around It.

I’ll be straight about something a lot of AI tools won’t: AI gets things wrong. Sometimes confidently. It can produce errors, bias, and convincing-sounding misinformation, and no amount of clever prompting fully removes that.

So when people ask whether CARL is “safe,” the honest answer isn’t a flat yes. It’s that we design around the fact that AI is imperfect, and we keep a teacher in the loop to catch what slips through.

“Guardrails” is not a magic word

You’ll see a lot of AI tools offer guardrails as if that settles it. We use that word too, but I want to be careful with it, because it can imply a guarantee we can’t make.

Guardrails are real, and they help. They are not a force field. The honest framing is that constraints reduce the chance of a bad output. They don’t reduce it to zero.

What we actually do

CARL is intentionally constrained. It’s built on pedagogical rules and structure rather than an open prompt, which narrows how far it can wander. It’s designed to flag uncertainty rather than paper over it, and to prompt review at the points where getting something wrong matters most.

All of that is meant to reduce risk. None of it is meant to replace your judgment.

The real safeguard is you

This is the part that matters. CARL’s outputs are starting points, not final answers. The platform is built so you review, edit, and decide what’s actually right for your class. You’re the one who catches the example that’s off, the claim that isn’t quite true, the framing that doesn’t fit your students.

A tool that told you it was never wrong would be doing something worse than making mistakes. It would be teaching you to stop checking. We’d rather build something honest about its limits, because that keeps the right person in charge.

Some things we design CARL not to do

Reducing risk also means deciding where CARL shouldn’t go at all.

The clearest example is Indigenous content. We are not Indigenous, and CARL is designed not to generate or interpret Indigenous knowledge. Instead, the Indigenous Resource Pathways feature is built to point teachers toward Indigenous-authored and Indigenous-led sources where authorship is clear, and to name a gap rather than invent a connection when certainty isn’t possible. We don’t claim partnerships or consultation we haven’t undertaken. You can read the fuller version of how we think about this on our Commitments and Design Principles page.

That’s a limit by design, not a missing feature. Some things should come from relationship and consent, not a model.