To measure how well people use AI at work, separate usage from proficiency. Usage, such as logins and seats in use, shows who opens the tool. Proficiency shows in decisions: which job someone hands to AI, what they show it, and whether they catch its mistake in a situation they haven’t seen before. Measure each habit separately, count first tries, and treat the result as a guide to where to help, not proof of productivity.
Usage is not proficiency
Most AI dashboards report usage: how many seats are active, how many chats people start, how often they come back. These numbers are useful. They tell you whether people have started, and where adoption is stalled.
They don’t tell you whether the use is any good. One person opens AI ten times a day and forwards every answer unchecked. Another opens it twice and catches the wrong figure before it reaches a client. Usage counts both the same.
Course completions have the same gap. Finishing a module tells you someone was there, not what they will do next.
What to measure instead: decisions
Proficiency is a set of habits, and each one shows in a decision you can judge:
- Judgment. Given a list of tasks, which go to AI, which need checking first, and which need a different tool?
- Briefing. Given a vague request, does the rewrite say what is wanted back, the limits and what counts as done?
- Context. Given a stack of documents, which does the AI need, and which must stay out?
- Checking. Given an AI draft and the facts behind it, what doesn’t hold up?
Judge the decision, not the wording, and judge it by fixed rules, so the same answer gets the same result for everyone.
Count the first try at a new situation
A right answer to a case someone has seen before may be memory. The honest signal is the first answer to a situation that is new to them. Two more rules help:
- Change the situation. Someone who gets it right in two different settings is more likely to have the habit than someone who got one case right.
- Build in order. Don’t credit an advanced habit, such as finding where an AI’s reasoning broke, before the basic one it rests on.
A level per habit, not one score
Someone can brief AI well and check its work badly. One overall score hides that. Give each person a level on each habit. One way to name the levels, which Skill Tree uses, runs Aware, Practicing, Capable, Independent and Leading: from recognizing the situation, through doing it with guidance and adjusting it when things change, to doing it unprompted and building it into how a team works. What AI fluency is describes each level.
What lesson results can show, and what they can’t
Decisions in lessons can show that a person made the right call in a realistic situation, on a first try, across more than one setting. That is real evidence of a habit.
They can’t show:
- That the habit carried into real work. A person can choose well in a lesson and rush on a busy day.
- Productivity or return on investment. Lesson results don’t measure time saved, quality of client work or revenue.
- That an AI’s answers are right. If people get answers that agree across several runs, the answers are consistent, not necessarily correct. Checking is still a person’s job.
If you need to show a return, measure real work too. Pick one recurring job, such as a weekly report, and compare before and after: how long it takes and how often it comes back for corrections. Ask the people who receive the work.
Report by team, and protect people’s answers
People hide what they don’t know when they think a manager is reading every answer. Report what people can do and where they are stuck, by team. Keep their answers, what they typed and their conversations with AI out of the report. Tell people what admins can see before they start.
Follow up on what the results show
- Find the shared gap. If several people on a team miss the same kind of decision, spend a few minutes on it at the next team meeting: what it is, why it matters, one question to discuss.
- Check that misses come back. A lesson someone missed should return until they get it, and the report should show whether they have.
- Recognize progress. A short note when someone finishes a stage of the training tells people it matters.
- Look again monthly. One summary a month, read by the people who can act on it, does more than a dashboard nobody opens.
An example: Skill Tree’s dashboard
Skill Tree’s lessons grade each decision by fixed rules in its own code. Only a first answer to a new situation counts toward a level, and a level counts only once the habits it builds on are in place. Admins see each person’s level on each habit, where they are in the course, which lessons they missed and whether they have gotten them right since. When two or more people on a team are stuck on the same thing, the dashboard offers a talking point for the next team meeting. Admins never see anyone’s answers. See security and privacy for teams for the full list.