Measuring employee trainings - Version 1
Planning, monitoring and reporting are important parts of working structured with cyber security training in smaller as well as larger organisations: Both at the individual level and at teams levels. This moves training from being an “awareness event” to a continuous process, where teams and individuals are maintaining and developing their skill sets.
In this document, we present a methodology for measuring training activities across activities and providers. At the same time, it provides organisations and managers with overviews of skills and skill gaps, and it makes it possible to formulate goals and strategies for training.
Based on a well-known framework
The methodology we describe is based on the NICE framework. However, we have also included “Cyber Security Fundamentals” even though it has not yet been included in the final framework. Therefore, the competence areas are:
· Access controls
· Artificial intelligence (AI) Security
· Asset Management
· Cloud Security
· Communications Security
· Cryptograhy
· Cyber Resiliency
· DevSecOps
· Operating Systems (OS) Security
· Operational Technology (OT) Security
· Supply Chain Security
· Cyber Security Fundamentals
Measuring and monitoring individuals
With the measuring and monitoring, we want to make visible (1) the level, (2) the amount and (3) the continuity of the training. For this, we need to be able to measure both level and amount of training.
Each module, assignment or otherwise piece of training is assigned a level on a scale from 1 to 5 based on the The Dreyfuss Skills Model: 1-Novice, 2-Advanced Beginner, 3-Competent, 4-Proficient, 5-Expert. The levels can be described as follows:
Novices rely heavily on context-free rules and step-by-step instructions. Individuals at this level have minimal experience in the subject matter. They require rules, guidance, and structure to perform tasks.
Advanced beginners recognize situation-specific nuances and can apply experience-based maxims beyond general rules. Individuals at this stage can perform tasks and solve problems with assistance. They have started recognizing patterns and contexts, but still need some guidance.
Competent performers choose specific goals and adopt an overall perspective on what their situation calls for. At this level, individuals are able to independently manage complex situations within the subject. They can make descisions and plan actions using their experience and understanding of the subject matter.
Proficient performers intuitively grasp what a situation calls for but consciously decide responses. Proficient individuals not only manage complex situations but can also anticipate problems and adapt their actions accordingly. They use intuition and experience to guide their decisions.
Experts demonstrate seamless integration of perception and action. Experts are masters in their subject matter. Theu intuitively understand what needs to be done without needing to rely on rules or guidelines. Their knowledge and skills are often seen as a resource by others in the field.
Each module, assignment or otherwise piece of training is also assigned a number of points. The points reflect the effort involved in the training. This will be highly individual, but on a scale from 5-100 points, 5 points correspond to a task that is doable within 15 minutes for a beginner user, whereas 100 points correspond to a task that is doable within 5 hours for an expert user.
For each competence area, the user is assigned a level (1-5). To be on level x, the user needs to have solved at least 5*x activities that include this competence. Moreover, at least two of these should be at level x. These requirements ensure the level and quantity of the training.
To guarantee the continuity, a user is expected to collect at least 200 points during the last 12 months. This must include at least one activity within each of that person’s competence areas, which is at least at the same level as the user.
Each user profile can be presented as a spider diagram (which could also be used for setting targets). Visuals can be added to indicate if sufficient trainings are completed during the last 12 months.
Measuring and monitoring teams
The Dreyfuss Skills Model mapped Competencies
Often, it is desirable to monitor not only the competences of individuals, but also that of teams. For each competence area, 2 values are calculated, which can be plotted also in the spider diagram:
The average score among all team members.
The average score of the three highest scoring team members.
Moreover, the average number of points achieved during the last 12 months is also displayed at a monthly level.
Limitations
The above framework is usual for measuring ongoing trainings in cyber security. It is not a measurement of academic competences, even though the model can be expanded to do this as well.
Download the document here: https://akademi.dit.dk/-/media/Billeder-Filer/Meet-Inspire/Presentations/Measuring%20employee%20trainings%20version%201.ashx