Delightful experience with Anthropic Claude LLM

📅 Published: • Thomas Queste

Lately, I wanted to convert my list of books from a text file to JSON to give it a better structure. Instead of writing a bunch of regexes and string tricks, I went for Anthropic Claude LLM. The experience and results were impressive!

TL;DR

  • Claude was taught to recognize tags, e.g. <book></book>
  • The workbench handles variables, e.g. {{ BOOKS }}
  • You can evaluate your prompt with test cases
  • Flawless, I got exactly the output specified, no hallucination, no wrong JSON, using claude-3-5-sonnet-20240620
  • Exhausted my credit in a few runs, learning that I should not have tried to process my whole book list at once

Problem: I was not consistent

Today when I read a book, I simply add an entry in my books.txt with the title, author, read date, rating, sometimes a comment…

And I am not really consistent!

Sometimes I change the date format, or the rating, or forget to write the author’s name, or put the volume number in the title, or (long list of inconsistencies…).

Solution: an LLM

My first approach was simple code, parsing regexes and… I quickly failed.

So, why not use a smart LLM to extract structured data from my hazardous-non-strict grammar?

I went to Anthropic Claude, because it got impressive results in some tests I read. The API is not free, but you can start with a free 5$ credit, enough for a start.

How it went: The Workbench

The workbench is a web interface where you can test your prompt and see the output.

A pitch then a prompt

I started with a simple sentence (basically the idea in my head) and asked Claude to generate a prompt from that “pitch.”

Prompt: Extract structured data from this list of books in french

Anthropic Workbench: generating a prompt from a short task description

Claude generated a bigger and hopefully more precise prompt, with an example structure and some instructions for itself.

The temperature was already set to 0, which makes the model more deterministic and less creative.

After adapting the fields to my needs (like rating, date read…), the prompt became this one:

You will be given a list of books in French. Your task is to extract structured information from this list and format it as JSON. Here is the list of books:

<book_list>
{{BOOK_LIST}}
</book_list>

For each book in the list, extract the following information:
1. Title (titre)
2. Author (auteur)
3. Description (description)
4. Rating (note)
5. Comment (commentaire)
6. Date Read (date de lecture)
7. Urls (liens)

Keep the JSON keys in English, but extract the values from the French text. If any information is missing for a particular book, use null for that field.

Format the extracted information as a JSON array of objects, where each object represents a book. Use the following structure:

```json
[
  {
    "title": "Book Title",
    "author": "Author Name",
    "description": "Book description",
    "rating": 5,
    "comment": "Reader's comment",
    "dateRead": "2023-05-15",
    "urls": ["https://somewebsite.com/book"]
  },
  // More books...
]
`` `

Note that the "rating" should be a number if available, and the "dateRead" should be in ISO 8601 format (YYYY-MM-DD) if possible.

Here's an example of how to structure a single book entry:

```json
{
  "title": "Le Petit Prince",
  "author": "Antoine de Saint-Exupéry",
  "description": "Un conte philosophique sur l'amitié et la vie",
  "rating": 5,
  "comment": "Un classique intemporel, toujours aussi touchant",
  "dateRead": "2023-04-10",
  "urls": ["https://amazon.com/12345"]
}
`` `

Analyze the provided book list, extract the required information, and format it as a JSON array. If you encounter any ambiguities or difficulties in extracting certain information, use your best judgment to interpret the data.

Provide your final output enclosed in <json_output> tags.

Variables

If you have sharp eyes, you may have noticed the special {{ BOOKS_LIST }} in red.

This is a variable that you can fill in a separate panel in the workbench:

Anthropic Workbench: the BOOK_LIST variable in the prompt and its value panel

Evaluation

You can evaluate your prompt directly in the workbench, providing test cases:

Anthropic Workbench: evaluating the prompt on a test case with a French book

The code

Like Google Vertex AI, Claude directly gives you the code to execute in the API:

Anthropic Workbench: Python code to call the Claude API with the prompt

What failed, lessons learned

Input can be huge, but Output is limited and more expensive.

I have a big list of books (a few hundred), and I tried to process them all at once.
For that, I copied my full list into the variable, and… only a few books were returned.

Yes, indeed, the output is limited to a few thousand tokens depending on the model, so you cannot process a big list at once.

Solutions:

  • Chunk your list in multiple smaller lists
  • Or, change the prompt and process books one by one, calling the LLM for each, which is actually simpler and can even reduce the prompt size, so the tokens, so the speed and cost

Conclusion

I had a good experience with Anthropic Claude LLM. 👍
The newest models are impressive, pretty consistent, and easy to use.

But there are real differences between LLMs and their capabilities.
For instance, I also used Gemini Flash on the same task, and the results were not as good as Claude. It failed to output JSON and only that (no Markdown wrapper…). Same when I tried to get JSONL (one JSON per line). Could be my prompt, could be the model…

Improve this post

Previous post: Maximizing Efficiency: A Guide to Caching in Jest, Prettier, ESLint, and TypeScript
Next post: Get structured output from a Language Model using BAML