Tom James

To clarify, even if you give LLM agents a research question, a methodology section, a pile of relevant references, all your codes and data, and all your current results, and ask it to produce a first draft, it is going to be garbage. Some of my coauthors and I tried it. On one occasion, we even got Anthropic and OpenAI models to referee and propose updated drafts to each other until they couldn't find any objections left. And the output was hot garbage -- in spite of how much we gave them to work with. I scrapped almost everything because it was clear that it was much faster to just write myself from scratch than to try to edit that mess. For one project, we also have a large DFM. It has over 500 variables, about 270 of which are for Canada. Searching through that is hard, so we hoped LLMs would help. It was a huge waste of time. Top of the line coding agents are not as good as an average economics PhD at spotting relevant patterns. We literally asked it to output a huge CSV file and did the selection manually. So, even if the question is along the lines of tell me what would be interesting to an economist -- and you dump a pile of papers to give it examples -- the best models suck at it. These models are excellent on narrowly defined tasks. They are great RAs. But that's it. I don't know how well they'll improve for more demanding tasks. Maybe they'll be great one day. Or maybe that's not happening.

Academic note: I have tried to be serious as a scholar, teacher and colleague since 1980s. In doing so I have constantly searched for productivity hacks — things that enable me to be more effective at being serious. Little things, like finding the right time of day to do research and to do teaching, or seeking ways to absorb needed knowledge like lectures at Great Courses or New York Review of Books or TLS. I moved from Word Perfect to Word, I have tried Palm Pilot, Blackberry, iPhone, I had a luggable portable computer that I traded in for laptop, and I used computer programs to take notes but switched back to notebooks and pen, and now use Kindle Scribe rather than ReMarkable. I try, I experiment, I assess. I read books in physical form, electronic form and audio. I watch YouTube and listen to podcasts all to help with lectures and papers. I used to search physically in libraries now I search online. My point is simple — I have always looked for ways to augment my skills and improve productivity. AI and LLMs are NO different in my experience. They don’t think for me, but they help speed up search and information gathering. I still have to exercise judgment, absorb what I need, and execute in a lecture, in writing, or in meetings with students and colleagues. The best productivity hack in my career is face to face dialogue with PhD students about ideas. These conversations are infinitely more interesting to me than me prompting Chat or Claude about material. Even working with a student on a difficult problem or struggling to understand concepts is more intellectually rewarding. Maybe I am doing it wrong, or maybe everyone else is describing their experience wrong. Modern AI is not the Palm Pilot, nor early versions of Netscape Browser (BTW, Browzine is a good hack for journal article access). But its not like having Kenneth Boulding, James Buchanan or Gordon Tullock as a teacher either! Its not like having been mentored by Don Lavoie and Karen Vaughn, or first inspired by Hans Sennholz, or arguing about economics 24/7 with Dave Prychitko and Steve Horwitz, let alone working closely with Israel Kirzner and Mario Rizzo for almost a decade, or teaching and learning from scores of PhD advisees at GMU. It just isn’t. Don’t buy the false narrative. It is not a substitute, it is a tool. You need to learn how to use the tool effectively to increase YOUR productivity as a scholar, teacher, and colleague. So invest in it, but at the end of the day you will have to be the one who uses the tool effectively or poorly. YOU, not the machine make the final judgement call.