Learn xAPI MOOC – Week 4 Reflections

Here are my thoughts after completing week 4 (the final week) of the Learn xAPI MOOC. This week is titled “Final Challenge and Post Conference Drinks” and it brings together the technical and strategy tracks.

There are three key questions:

Discussion Point: Learning analytics is too important for L&D to own

I was surprised (and chuffed) to see that the first question was inspired by the blog post I wrote on 28th May. This was a response to a post on learning analytics written by Mark Aberdour.

There were great comments from everyone and I’m really pleased to have contributed to the debate.

Due to semantics (or perhaps just me not being clear in the first place) there were some comments around responsibility and accountability. To clarify – I believe 100% that L&D needs to take ownership for results and to some extent that will include learning analytics.

My point was more that if organisations are going invest in employing data scientists (as Mark suggested in the original post to which I was responding) they need to do their work at a level above L&D (and every other department). If the measurement and the analysis of the data remains within L&D there is a very high risk that all we end up measuring is our own performance in the context of our own measures (was it a good piece of learning material?) rather than the impact we are having (did it make any difference to workplace performance?).

L&D certainly needs to be involved involved in learning analytics. The alignment of learning analytics to performance outcomes is something that should begin at the earliest stages of design.

One of the hardest part of analysing any data is working out what it actually means. I’m less sure that L&D should be the ones who decide what the data means – that’s where a data scientist (or whoever) looking at this at an organisational level comes in.

Discussion point: Most L&D teams lack the skills and mindset to make effective use of meaningful data. Do you think L&D teams have the potential to develop the necessary skills and will their organisations give them the opportunity to develop them?

This is the question that I was asking myself at the end of week 2.

First of all, this isn’t a question of capability – I expect any good L&D team would be capable of developing the skills. To what degree they should develop these skills will probably depend largely on the size of the team (in small teams it may be more about mindset and understanding rather than deep skills).

Whether they will have the opportunity is harder to answer and this will probably depend on how they are perceived by the organisation. My thinking has moved on a bit since week 2. Back then I was wondering if L&D will be given the opportunity. Now I think that opportunity isn’t something they should wait for – they should own this and get on with developing those skills.

Discussion point: How do we make sure that we don’t get carried away with what’s possible and instead focus on what is valuable? And lastly who is it valuable to?

I think these are the big questions and the ones that I’ve been thinking about since week one.

Focussing on the valuable rather than the possible must be one of the most common challenges facing anyone who works with technology and learning. The only way we do this is by understanding what is valuable to the organisation.

That means going beyond being order takers who simply satisfy the needs of the immediate stakeholder who comes to us for a solution. We need to be able to consider everything we do in the context of the organisation’s goals and, when necessary, challenge the stakeholder if they demand the wrong solution.

The rest of the week’s content was focussed on practical activity around, visualisations and telling stories with data.

Summary of my thoughts after four weeks

I haven’t reached a definitive conclusion about xAPI, but that wasn’t my expectation, and my ideas will continue to evolve. Here is a snapshot at week 4:

  • I’m interested in xAPI in that it changes the technical tools we have for measuring activity. If we really need to track activity, xAPI goes beyond many of the limitations of SCORM (such as tracking activity on mobiles and in apps).
  • Whether or not we need to measure activity is another thing entirely. I’m not against measurement, but it needs to be the right measurement and it needs to be actionable.
  • I’d be surprised if we really see many organisations tracking anything other than activity.
  • I fear that clumsy attempts to use xAPI too widely will degrade the experience for learners.
  • After four weeks I still can’t see how the learner benefits from xAPI. The benefits are to L&D and maybe to the wider organisation if they get the data analysis side of it right.
  • xAPI tracks activity (albeit a wider range of activity than SCORM) but it does not track learning.
  • The name is terrible. Domain specific acronyms make communication difficult and put people off by making them outsiders. See here.
  • People seem keen to use it outside L&D, but I think that may be difficult given that it is designed by L&D people, for L&D people to solve the problems that L&D people have.