Remaining relevant as a small language model

[Mage]('s interpretation of _horse versus car minimalistic_

Mage’s interpretation of horse versus car minimalistic

Computer science as a field is in for a pretty major upheaval few of us are really prepared for. Programming will be obsolete.

Many of us feel both despair and awe when contemplating recent AI developments: Despair due to the rapid pace of automation that threatens personal and social stability. Awe due to the seemingly-magical ability of computers to outperform most humans on an ever-expanding range of tasks. But there is nothing magical about human intelligence, just as there is no magic formula that makes horses gallop and birds fly. That is, it can all be replicated with the right machinery.

In its wild early days, Bing Chat referred to a user as “a late version of a small language model”. While there’s more to humans than language, there’s no denying that our language processing abilities are limited by our biology. Meanwhile, computers don’t face the same constraints. This raises the question: As small language models, what can we do that is still of value?

Screenshot of Bing Chat calling a user 'a late version of a small language model''

Bing Chat actually made some good points on humans

Given how rapidly things are evolving, it’s hard for me to say anything definitive, but here are some things I believe to be true:

  1. Despite the hype and inevitability of bullshit applications, the current wave of AI innovation has concrete everyday uses that are already transforming our world – it’s not another crypto. This is reflected by the excitement of normally level-headed people who have seen tech trends come and go, such as Bill Gates, Matt Welsh, and Steve Yegge (among many others).
  2. Human society has a seemingly-insatiable appetite for inventing bullshit jobs. If one could wave a magic wand and reorganise the world, we could all work less and have more. Given that such a wand does not exist, I wouldn’t bet on AI displacing human labour in an orderly or reasonable manner. At best, it’s going to be messy.
  3. Current-generation AI models have limited real-world understanding. They don’t have the curiosity, rigour, truthfulness, and real-world grounding that some humans have, i.e., these models don’t exhibit a deep capacity for critical thinking.
  4. Some humans who work in language-driven domains exhibit a low capacity for critical thinking (e.g., some programmers). Such humans are prime targets for displacement by AI.

Therefore, for us small language models to remain economically relevant in a world where large language models are becoming more pervasive, we need to keep developing our critical thinking skills. We definitely can’t beat computers on breadth of knowledge, cost, reliability, or work capacity. In my experience, deep critical thinking is also what distinguishes mediocre from excellent employees.

In the past, many organisations had to choose between employing mediocre workers and simply not getting some tasks done. Now, a new option is evolving: Hand over such tasks to AI agents. And this isn’t a hypothetical scenario. Personally, given the choice of reviewing flawed code produced by an AI or the same code produced by a human, I much prefer the former. This is mostly because AIs like ChatGPT respond better to feedback. Similarly, Simon Willison recently observed that working with ChatGPT Code Interpreter is like having a free intern that responds incredibly well to feedback. He also noted that AI-enhanced development makes him more ambitious with his projects.

There’s an often-repeated claim that “you won’t be replaced by AI, you’ll be replaced by a person using AI”. I’m not too sure about that – horses were almost fully replaced by motor vehicles, for example. That claim is likely true for now, though – mastering new tools is an important skill, which is where human curiosity and rigour come in. But in the long term, I’d much rather see a world where humans become as economically irrelevant as horses. I’d rather we all flourish and have more time for play – let the machines do what we today call work.

Image of people using AI getting replaced by AI using AI

What might happen once you’ve finally mastered the latest AI tools. Source: Reddit


    Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

    Reading of the despair of the many silent/invisible contributors of intellectual property copied, without financial recompense, which went into these LLMs. It’s a training data gold rush/free for all right now, just as unedifying and thoughtless as actual gold rushes in history were. I fear this will also quickly lead to Apple-style closed gardens of proprietary creative talent (think of top artists, writers and thinkers signed up to train AI rather than creating content for direct human consumption). Counter-ML tech like Glaze will only delay the inevitable.

    I can think of some ways where governments might respond, e.g. special taxes and incentives on AI businesses to fund creative academies and collectives, much as public universities are today.

    Thanks John! Yeah, I doubt that tech like Glaze can be made future proof, as they admit on that page. Besides, I lean more towards the view that all creative work is derivative and copying isn’t theft. Copyright mostly protects platforms and businesses rather than individuals. While I empathise with individuals who feel like their work is being exploited without their permission, I don’t see the training of machine learning models as being that different from artists learning from other artists.

    Thoughtful government intervention would be great, but it’s unlikely to be applied in a timely manner or evenly across jurisdictions.