Can an AI create any learning pathway you can think of in under 5 seconds? We are about to find out. A deepdive into how Kinnu produces its learning materials.
Content is king. But is it about to be dethroned?
Bill Gates declared that “Content is King” in his 1996 now famous essay. But is that really true in the age of SEO, where more is more? Today, the wider your net of content, the more users you catch.
And what is the best definition of good quality of learning content? Who judges the quality: the learners or Google’s search algorithm? At Kinnu, we think about quality in terms of giving the learner the fastest route towards mastery: the content needs to be accurate (duh), engaging to make sure you don’t fall asleep while learning and as concise as possible to be respectful of the learners’ time and minimise their cognitive load. We are passionate about efficiently using time and resources, so these are important considerations which also impact what we consider “effective content creation”.
Do humans write better than AI tools like GPT-3? Are humans better at creating diagrams and images? Writing quizzes? Are there specific tasks where humans will continue to excel over AI in the short term? In the long term? These are just some of the many questions which our content and AI team (a big word for 3 frenzied individuals) has to answer in the next year.
Starting with the human
We get a lot of questions about how Kinnu pathways are created. We work with a team of talented writers who are experts in writing but not specialised experts in a topic.
Each pathway takes a week to plan, three weeks to write and a week to edit. Where the topic requires more specialised expertise, we submit it for a subject matter expert review, both in the planning stage and in the final review stage. This process ensures our materials are written balancing both accuracy and approachability.
I know a thing or two about content creation at a MOOC provider and I am quite proud with what we have been able to achieve in our first 3 months of making content at Kinnu. We have produced 35 learning pathways 5 times faster and 10 times cheaper than a typical online education provider would be able to achieve. And we had a team of 2 people instead of a typical team of 20-30.
Where did we find an order of magnitude time saving, you ask? Well, the main time saving comes from no need for institutional approvals. A typical MOOC is stuck for 10 weeks in different parts of an approval process – even things as “no-brainer” as changing a course title to something more SEO friendly can take 2-3 weeks. The second time saver comes from not producing video content, because who has the time to watch talking heads when you can read 2x faster. Of course we also run a tight ship – operations are already running like clockwork, but we also reflect every single month on how to further scale and improve content production.
Our content creation process is mostly human today, relying on work of our external team of writers, with hundreds of in-house editorial hours spent editing drafts, writing questions, selecting images and generating audio . But we want to go faster and get better. We want to understand each part of the process intimately. Why? Because we want to know where it’s appropriate to invite robots to the party. That’s why we’re moving towards AI led learning content creation.
All protein, no carbs and definitely no academese
We have thought long and hard about how to create the best learning content for Kinnu and we feel quite strongly about having non-experts as pathway authors. During my PhD, I read enough academic articles and textbooks to realise that very few experts are also gifted teachers who are able to explain with clarity and simple language core concepts in the field. Of course there are exceptions to this (special mention to Richard Feynman and Roger Penrose).
This is because experts have a tendency to use a foreign language called academese.
We wanted to create learning pathways which are as efficient as possible, providing our learners the shortest pathway towards mastery. No fluff. As few words as possible. Just enough human interest stories to keep you engaged. Enough context and valuable knowledge to build deep expertise, bite by bite. Kinnu is here to save you time and introduce must-know concepts in a topic. The rest of the internet is well suited to help you deepen your knowledge on any topic.
What have we learnt so far?
The verdict so far – our learning content is not epic like CS50 or Steven Pinker’s books, but reads better than most online courses I have done. It’s a lot of reading, it’s hard to find adequate copyright free images, but thank you wholeheartedly to Unsplash and Wikicommons for the ones we have. Is it perfect? No. Despite a talented (if young!) editorial team and a specialty hired proofreading service, the pathways still are ridden with typos (about 5-10 per pathway). The writing is good, but definitely not Pulitzer material. Less dry than Wikipedia can be but more engaging than a typical book written by a deeply passionate author with a strong point of view. We are going fast, but we feel the need to both go slower, to get high polish required for our discerning learners, and faster to get more content onto the app as quickly as possible. And, of course, all content meets Kinnu’s own quality requirement – finding the fastest and most engaging way to learn.
A Voice From A Cloud
From our work so far, we have learned that AI generated voices have gotten much, much better over the last couple of years. Audio generated transcripts can easily be updated, keeping content fresh and accurate in a fast changing world..
Additionally, AI generated transcripts can be arranged via an integrated APi, radically decreasing the operational complexity of content creation. And *drumroll* in our humble opinion AI sounds better than humans do when you slow down or speed up your audio (which most of our audio fans tend to do, based on research)
Down the line we will also experiment with AI-generated videos. Synthesia is an AI video generating tool that is truly impressive. It is super convenient to use, makes it easy to personalise and update the content as well as translate anything into several languages: a powerful technology which is getting both better and cheaper every day.
Where we want to get to
We started Kinnu knowing that seeding the initial learning content was just the beginning. To reach scale and grow the ability to build new content at scale and pace we would need to start using AI, user generated content (UGC) or a combination of both.
The one question that kept coming up in product design meetings was – how many learning pathways are we looking to build, given no constraints around time or resources.
Our content lead thought the maximum number of pathways should be below 1,000 (and who can blame him!). My initial view, having reviewed Wikipedia’s top topics, was closer to 2,000 – 5,000. Then I asked Chris, who answered with the enigmatic “I think it should be either 0 or infinity”. (“Ever heard of shades of grey, Chris?”).
But ultimately, Chris is right – part of the reason people love getting help from human tutors is that they can tailor the response to exactly the learners demand, with full awareness of their context.
State-of-the-art generative AI tools such as GPT-3 (for language) and DALL-E (for images), conversational interfaces allow people to stretch the bounds of their creativity to fully test the strength and weaknesses of algorithms. We decided to have some fun.
The French Revolution experiment
Everyone keeps talking about how great natural language processing (NLP) is, and we saw the incredible progress in text-to-speech processing, so we asked ourselves – “Can AI write one of our learning pathways?”. We picked a topic with lots of documentation across the internet, and also with lots of factual information, which also incidentally we are interested in. Here is how we did it.
First, a benevolent expert (read: my husband) spent 2 hours writing the detailed plan of the learning pathway. It is divided into 6 tiles or about 40 sections, each of which contains about 2 facts per section. The human writes about 3,000 words all in, including headings.
I copy and pasted each of the 80 detailed fact bullets into the OpenAI sandbox ; it took me about 30 minutes. Then, I picked one of the newest algorithms, DaVinci and let the AI write another 50-100 words per section based on each prompt. The writing is actually pretty decent (if this was a university essay, I’d give it a C+).
But the AI is trained on the web and sometimes it makes hilarious (terrifying?) mistakes. No, Emmanuel Macron is not the king of France (as some satirists would have you believe). No, Donald Trump has nothing to do with the French Revolution (though he might have put some of its principles at stake). No, we do not need a full list of religious orders abandoned by the French Revolution.
What other mistakes and inconsistencies were there? Just to be on the safe side, the benevolent expert reviewed the AI written pathway, made edits and added a few interesting facts which were missing. All in all, the expert was quite impressed with the quality of the work provided by AI, but did not think that without his review and rewrites the content was ready to put into our app.
It took him another 3 hours to rewrite the rest of the learning pathway content. All in all, it took him 5.5 hours to write a full learning pathway. Now, we need to add one hour to add images, another to generate audio, 3 hours to write questions. We went from nothing to full learning materials in just under 10 hours! Not bad for a first try – the full pathway is now available on Kinnu. Following our experiences we wrote up our best practices on working with generative AI.
The Tree of Knowledge
But is the expert really needed in the initial stage of figuring out what are the necessary important facts to know about the French Revolution. What if we used the Wikipedia knowledge graph (sidenote: you should donate to Wikipedia now if you haven’t already, so many projects stand on the shoulders of this giant!).
We had previously ran some queries to identify top topics in priority markets on Wikipedia and the list reflects surprisingly well on humanity (in addition to sex and gossip people are more interested in philosophy, psychology and religion than I would have thought). Subheadings of the Wikipedia page on French Revolution align surprisingly well with the structure provided by our expert, providing an avenue for future exploration.
The destin-AI-tion
What else can we automate? Well, we’re going to assume nothing is sacred and will push the boundaries as far as we can, starting with human + computer models and moving towards full automation (fun fact: I learnt this approach when automating myself out of all of my finance jobs at Google).
We’ve mentioned text generation as one potential way to help content creation but there is much more we can do, which includes NLP-assisted question-answer creation, key concept detection, text summarization and more. The text generation example is given with the help of the OpenAI-based GPT-3. But, we also want to develop our internal know-how which will further enhance and develop our tasks.
In our work, we want to derive from the breadth of NLP-research, with the focus on the most recent developments which in particular can accelerate pathway creation. We use cutting edge algorithms which elevate deep learning to be used for language-related automation. Those include transformers such as BERT (of which GPT-3 is also an example).
Questions can be written with an NLP algorithm. Questions can be rephrased by another NLP algorithm. Images can be built with DALL-E or Midjourney.
And the most exciting initiative of them all – imagine a world where you can type “Data science in 5h over one week, I know a little bit, mostly about statistics” – what you get is a personalised content pathway recommendation created from atomic pieces of content, just for you – that’s where we are going.
We are also very keen to see how we can engage our community of learners in helping us build the best learning materials on the planet. There are fantastic examples of how communities co-create knowledge (a big passion of mine). How can we aim this creativity and passion towards this project without replicating the scale and breadth of Wikipedia? But more on that in a future post.
