What To Do With AI-Generated Legal Scholarship?: Part 2

As I explained in my previous post, I recently tasked AI with comparing two transcripts of the 1807 treason trial of Aaron Burr.  My ultimate question is, what do I do with the document that resulted?  And that breaks down into two sub-parts.  First, do I publish this, either just online informally or with some kind of journal?  And second, how do I describe what my relationship is to it?  Am I a co-author?  The author?  Just a prompter?

Let me start by explaining how the memo was created, and then turn to the questions I have.

I. How the AI Memo Was Created

I need to start with what I did to help create the memo, as that might be relevant to my questions.  The transcripts that needed to be compared were .pdfs of two-volume books from 1807 and 1808 that go for hundreds of pages, although the only parts I cared about were the parts on the privilege against self-incrimination. I used Claude (Opus 4.6 extended),  and I tasked it with comparing the discussions of the legal arguments about the privilege against self-incriminagtion to get a better sense of whether my 2021 article on those arguments based on the Robertson transcript was accurate in light of the Carpenter transcript.

To say that "I tasked Claude" covers up a lot of detail, though.  I went through around 30 rounds of prompting with Claude, over the course of a few hours.  As I went along, I learned about what Claude could and couldn't do and pushed it to do a better job when it was resistant to do more.  For example, when I first asked Claude to compare the documents, it declined, saying it was just too big a task to take the two long pdfs, to make them readable, and then to compare them.  So I started with an easier task: Take my 2021 article, read it, and understand what it claims about the Burr trial, and then read the Carpenter transcript and write an article presenting a comparison.   The first draft reply was a start, and made me think that the enterprise might be ultimately useful.  But it left a lot to be desired.

Over time, I came to realize that there was an art to getting Claude to make the comparisons I needed.  Ultimately it agreed to do a direct comparison of the two transcripts based on the claims I had made in my 2021 article.  And I quickly realized that the comparison really needed lots of direct quotes and page references for everything, so that took a lot of extra time: Among other things, Claude had surprising difficulty with page numbers, in part because one of the two .pdfs had two volumes back to back and Claude could not figure out the pagination.  I had to spot check, and I kept finding errors, which Claude had to keep correcting.

A big breakthrough came when I realized through trial and error that Claude could do comparison screenshots.  That is, instead of just telling me what the two transcripts said, Claude could take screenshots of the relevant discussions and show them side-by-side.  That way I could be more confident that I was getting a real comparison.  Even then, the screenshots needed a lot of correction: Claude started off giving me only about 1/3 of the comparisons correctly, and I had to keep telling it to go back and make sure it was screenshotting the exactly equivalent sections.

As I neared the end, I also added more tasks.  For example, I asked Claude to read the Carpenter transcript and tell me if my 2021 article had accurately summarized the arguments, as well as whether there were any parts of the argument from the Robertson transcript I had missed.  I also asked Claude to say if there was any legal source Robertson had reported than Carpenter hadn't and vice versa, and any legal argument that one had reported that the other habn't.

After about 30 rounds of prompting, I ended up with a 22-page comparison memo.   A typical page of the memo looks like this:

The good news, at least accordiing to Claude, is that all of the substantive points matched really nicely.  The variations between the Carpenter transcript are apparently very minor, the kinds of things you might expect with two independent human beings trying to write down hours of court proceedings and hearing some small things differently. The parts I cared about were a match.

II.  Should I Publish This, At Least in Some Way?

This brings me finally to the big queston I have, what do I do with this 22-page AI-generated memo?   On one hand, if you're interested in my 2021 article, I think the AI comparison is of scholarly interest.  The comparison is pretty noteworthy, at least for the small number of nerds who care about the substantive topic. On the other hand, the AI-generated memo doesn't slot into any traditional understandings I have of either scholarship or non-scholarship.   So I don't know what to do next.

My uncertainty breaks down into two questions.  First, do I publish this?  And second, if I publish it in some sort of way, what should I state as my relationship to it?  Let me explain my thinking as to both questions.

First, on the question of whether to publish this, I am conflicted.

Part of me thinks this memo should be only for my internal use.  At some point, I should just do the work of reading the Robertson transcript again and reading the Carpenter transcript for the first time, and I should write up a comparison myself.  Perhaps the memo should just be a helpful guide as to what I might expect to find, a sort of map that points the way and makes the task a bit quicker. This is akin to how I tend to use research assistants: Write me a background memo for me, which then gives me an helpful idea of what I should look for when I do the work myself.

Alternatively, I could post the AI-generated memo on SSRN.com, where a lot of draft articles are posted, but not try to publish it with any actual journal.  Once it is up at SSRN.com, anyone can find it with a Google search. From a scholarly perspective, that is the important part; that way, people interested in the topic can see the comparison.  But there's no need to actually publish the essay in a journal.  It's not even my work product, but Claude's work product, so there's not much point in getting it formally published.

Or maybe I could (at least try to) publish it?  It seems to me it's an academically interesting document.  Maybe there's a journal that would be interested in publishing it, and if so, why not? That would give the memo a more permanent home, and it might draw more attention to my 2021 article and the issues I was trying to address.

III. What is My Relationship to the Memo? Am I The Author? An Author? Neither?

Finally, there's the set of questions about authorship.  If I just keep this as an internal memo, granted, I don't have to worry about that.  But if I post it on SSRN, or (certainly) if I try to publish it, I do.  Am I an author?  A co-author?  A prompter?  What am I?

One thing that seems clear to me is that I should not publish it the memo as an article single-authored by me.  Perhaps I have an overly romantic notion of authorship, but I feel like authorship implies the moment of sitting in front of a blank page and putting my words on it. There has to be an authenticity behind that, and prompting Claude to write something (even many times) doesn't feel like it makes me the author.  Even if I checked it, I didn't write it.

Another possibility is that maybe I am a co-author.  Maybe my direction of the project, and my repeated prompting, made me a co-author along with Mr. Claude Opus, the actual writer. That seems better than saying I am the author, as at least I am trying to reveal how the memo came to be.  Although a co-authorship approach is a little weird: It's not like Claude and I are two scholars who worked on the article together.  I don't even know if SSRN would allow me to state "Claude Opus" as a co-author. So I'm not sure that fits.

A third possibility is that roles like mine  are something new, and we need to come up with a new vocabulary for it.  Maybe I didn't author the article, but rather I am the prompter of the article.  Maybe I didn't write the article, but rather directed it.   Perhaps, in my role as prompter/director, I shoould write an introduction that explains my goals and how the AI-generated memo came to be.  Basically, I should summarize what I have written in these blog posts so far.  And then I attach the AI-generated memo, for which I take no authorship credit.  That way, the reader knows who did what and where the memo came from, as well as its limits.  There isn't a role of prompter-director now, but maybe there should be?

Right now, at least, my instinct is that I first need to assess how much time it would take to do this myself.  If it won't take too much time, and if I have the time, I should just use the AI-generated memo for my own internal use as a guide for when I do the project the old-fashioned way.  What sees the light of day will be my own human-reasoned and human-written article instead.  Alternatively, if I think the time commitment is too much given other obligations, I think I'll try to take the prompter-director role:  I will write the intro and attach the memo, posting them together on SSRN, with the front page saying "introduction and prompting by" me but the article clearly labeled as written by AI.

Those are my instincts, at least. But I don't know.  What are your thoughts?

The post What To Do With AI-Generated Legal Scholarship?: Part 2 appeared first on Reason.com.