You are correct, 6+5 is equal to 12. My apologies for the mistake (emergentmind.com)

The answer could be correct in a different ring arithmetic.

There are many implicit assumptions, when asking such questions without further specifications.

Your friend could have either asked for clarifications or note that there could be cases, where the answer is notionally correct.

It could be correct in base-11: ISBN-10 (https://en.wikipedia.org/wiki/ISBN#ISBN-10_check_digits) actually uses digits 0-9 and then X (rather clever use of Roman numerals, it of course means 10) due to then-limited understanding of check digits when it was first standardised (the Luhn algorithm was not invented yet).

I worry about the application that AI will be trusted with.

Working as intended. When presented with new facts, it updates its model. Beware what 'facts' you enter.