Using AI to write history - a practical investigation

By Dr Chris Maloney, PhD Student in History at Queen Mary University of London, and a member of the RCPsych History of Psychiatry Special Interest Group. In previous lives, a Consultant Psychiatrist and GP.

This is the first of a few practical guides to using AI in historical research, and in psychiatry in particular.  The focus is on understanding the tools on offer, and carrying out a few experiments, not prescribing a workflow. So, in this first post I’m not suggesting you routinely get AI to write whole histories! It’s an exercise in discovering what it can and can’t do and developing a critical attitude to the output.

There are many ways you can use AIs, from getting help with your filing to having it write a whole history. Caution might suggest beginning with the filing, but this'll tell you nothing about what else it can do. Be reckless: get it to write a history, then you can start to think hard.

Starting out

Take your exploration seriously. There’s a revolution under way, and you need to know what's going on. Put quality time aside. Give yourself a week or two in which to experiment properly.

Start on a subscription plan. Why pay? Because you want a long conversation: in order to produce a paper or two, ask follow-up questions, challenge an answer and try again. Free accounts  can restrict both the intelligence, and how much of it you can use. You need to see what the best  tools can do. Your time and attention will be the costliest investment; a month’s subscription gives you a chance to decide whether it’s worth continuing.

Start with a question about an area you know well. Try a range of prompts—the questions or instructions you type into the chat. Follow up on the answers. Ask for explanations, challenge something you doubt, or point out what the response has missed. See what happens as the conversation develops.

You’ll begin to get a sense of the system’s ‘personality’. We’ll return to anthropomorphism in the next post. For now, notice its habits. Does it ask useful questions in turn? Does it rush to a conclusion? Does it identify an ambiguity? Does it agree with you a little too readily?

It's important you ground all this in your own expertise. Otherwise, a fluent answer may impress you without giving you much basis for judging it. On familiar ground, you can recognise omissions, dubious claims and interpretations that make the evidence look tidier than it is.

If you’ve written about the subject yourself, all the better. But you’ll  need to ask it not to use your published work in preparing its answer (it may otherwise seek it out).  That isn’t a guarantee of independence: your published work, or discussions of it, may already have been included in the model’s training. But set some ground rules.

Get it to produce a complete historical account: a short essay with an argument, evidence and references. Follow the argument. Where does possibility become certainty? Where has a disagreement between historians disappeared? Does the conclusion actually follow from the evidence? Do the references actually exist? And what has it noticed that you hadn’t?

Take these questions back into the conversation. Point to a particular claim and ask what supports it. If you find a mistake, explain it and see how the system responds. Does the correction improve the argument, or does it produce another confident answer that still misses the point?

Then, if you do have a paper of your own, get it to look at it. Ask it to compare your argument with its own account, identify the differences and criticise your reasoning. Ask for specific passages and reasons. See whether it can do more than summarise your position and congratulate you on having one. See if you can get it to argue.

Once you’ve spent some time with one system, it’s worth trying another if your budget allows. Give it the same historical questions, without access to the first system’s answer. Compare what each notices, overlooks and takes for granted. You may find that one suits you better, or that their differences help you see the question differently. Don’t treat agreement between them as confirmation: they may both be making the same mistake.

Once this is done, get it to write your next article for you. Then you can work out exactly why this is a Bad Idea, rather than trusting solely to instinct. Once you’ve got a sense of what happens if you abdicate, you can make better judgments about how to delegate.   

Next in the blog series – Why you should treat your LLM like a person