Amanda Askell's Prompt Engineering Secrets

Amanda Askell is a philosopher turned prompt engineer who is one of the key people behind Claude's human-like LLM.

She has profound insights into how to effectively communicate with language models. Her approach is a blend of clarity, relentless iteration, and a deep understanding of how these models actually process information.

Whether you're a beginner or an experienced practitioner, these principles will help you write better prompts and unlock the true potential of language models.

1. Building Blocks: The Foundations of Prompt Engineering

The Art of Clarity: Speaking the Model's Language

• Insight: Prompt engineering is fundamentally about clear communication. It’s not just about writing well but ensuring the model understands the task.

• Example: If a prompt is unclear, the model might misinterpret it. For instance, asking the model to identify "rude" or "polite" responses requires defining those terms explicitly. For example, you must define "rude" if you ask a model to label a statement as "rude".

• Philosophical Approach: Like a philosopher dissecting a complex argument, you must define every term and address every potential ambiguity. Amanda's experience in philosophy has made her a better communicator with Language Models.

• Beyond Eloquence: Good writing is not as strongly correlated with prompt engineering success as people might think, but precision and a willingness to iterate are essential.

The Iteration Imperative: The Secret to Prompt Mastery

• Insight: Prompt engineering is not a one-shot deal; it’s an iterative process. Refining prompts often requires hundreds of attempts to achieve the desired output.

• Example: Amanda explains her process: "In a 15-minute span, I'll be sending hundreds of prompts to the model. It's just back and forth, back and forth." This constant testing is vital for success.

• Learning from Mistakes: When the model gets something wrong, ask, "Why did you do that?" to uncover weak points in the instructions and improve them. It’s a chance to refine and learn.

2. Navigating the Unknown: Anticipating Edge Cases

Thinking Like the Model: Understanding Potential Pitfalls

• Insight: A good prompt engineer anticipates how a model might misinterpret instructions, especially in edge cases that go beyond the norm.

• Example: If a prompt asks the model to extract names that begin with "G," what happens if there are no such names? What if the input is an empty string? What if the input is not even a dataset? Testing these unusual scenarios is vital.

• Exploring the Unfamiliar: It's not enough to assume that the model will simply perform your task. You need to anticipate where the model might fail and account for that in your instructions.

Giving the Model an "Out": Preparing for Uncertainty

• Insight: When edge cases arise, give the model clear instructions on how to handle them. It is vital to plan for the unexpected.

• Example: If you ask a model to classify charts as "good" or "bad" and it sees a picture of a goat, what should it do? You should tell it that "If something weird happens and you're really not sure what to do, just output 'unsure.'"

• Handling Ambiguity: Provide specific instructions about how the model should respond when facing an unexpected input.

3. Building Trust: Validating Model Reliability

Trust is Earned: Rigorous Testing is Essential

• Insight: Amanda doesn't trust models by default. She rigorously tests prompts to ensure that the model can be relied upon.

• Example: Amanda’s process involves intense testing. She says, "I don't trust the model ever. I just hammer on it to see if it can perform the task consistently."

• Probing the Model: Don't assume the model will get it right the first time. Test multiple times to make sure it is reliable.

Quality Over Quantity: The Power of Precision

• Insight: A small set of well-crafted prompts is more effective than thousands of poorly designed ones. It's about quality, not volume.

• Example: "A few hundred well-constructed prompts can provide more signal than thousands of loosely crafted ones." Focus on making each prompt count.

• Focus on Precision: Instead of writing a large number of generic prompts, write a smaller number of very well-structured prompts.

4. Effective Prompt Design: Getting the Message Across

Honesty is the Best Policy: Being Direct with Models

• Insight: Avoid role-playing or lying to the model. Instead, be direct and honest about your task, focusing on the specific outcomes you need.

• Example: Instead of saying, "I am a teacher trying to write a quiz," say, "I want you to construct questions that evaluate a language model." You must be direct about your goals.

• Focus on the Task: Be clear about the goal and the instructions, without adding extra filler that is not necessary.

Metaphors: Use Sparingly and with Purpose

• Insight: Metaphors can help explain complex ideas, but they should not distract from the actual task. It is about clear communication, not creative language.

• Example: Using metaphors, such as the "teacher" example above, might confuse the model if the task is unrelated to teaching. Do not make the prompt ambiguous.

• Focus on the Task: Your main focus should always be on the task, and the metaphors should only play a secondary role. Do not make the prompt ambiguous.

5. Taking it Further: Advanced Prompting Techniques

Illustrative Examples: Emphasizing Understanding Over Memorization

• Insight: Use examples that illustrate the task without overfitting to specific data. The model must be able to understand, not simply memorize.

• Example: For a task involving technical documents, give examples from a children’s story to help the model generalize. This will encourage it to extract information rather than memorize the words and patterns.

• Avoid Direct Copies: Avoid examples that are very similar to the actual data the model will see, in order to encourage flexibility.

Chain of Thought Reasoning: Unpacking the Model's Logic

• Insight: Asking the model to explain its reasoning (chain of thought) can improve performance, but it's not foolproof. It gives the model a space to compute the solution.

• Example: "Sometimes the model lays out steps, one of which is wrong, but still reaches the right answer." This shows that the chain of thought is not always a perfect explanation of the solution.

• A Tool for Debugging: Use the chain of thought to debug the model’s reasoning, but do not rely on it completely.

6. The Philosophical Angle: Prompting as a Mental Exercise

Philosophy as a Framework: The Pursuit of Clarity

• Insight: Writing prompts is similar to writing philosophy—both require clarity, precision, and the ability to break down complex ideas into digestible instructions.

• Example: Like philosophy, you should define every term and address every objection. Prompting feels very similar in that you must be clear about the parameters you are using.

• A Methodical Approach: You must be very methodical in your approach, and consider how every term that you use will be interpreted by the model.

Defining Nuanced Concepts: Pinpointing the Exact Task

• Insight: When tasks involve nuanced concepts, define them explicitly in the prompt, making sure the model grasps the specifics.

• Example: "I'll define new concepts in the prompt because sometimes what I want is fairly nuanced." If you want a model to make a "good" chart, you must define what a "good" chart is.

• Putting Concepts into Words: The idea is that you must take what you want and put it into words. Sometimes the thing that you want is highly nuanced, and defining these things is a task of its own.

7. Looking Ahead: The Future of Prompt Engineering

Models Will Prompt Us: A New Paradigm

• Insight: In the future, models may prompt users for clarification instead of the other way around, inverting the current dynamic.

• Example: "The model might ask, 'Do you mean this concept or that one?' to clarify ambiguous instructions." The model might take the lead in trying to understand your specific needs.

• Anticipating Edge Cases: The models might ask you what you want it to do when it encounters a type of input that it doesn't know how to deal with.

Externalizing Your Brain: The Essence of Prompting

• Insight: Prompt engineering is like externalizing your thoughts, breaking down complex ideas into clear instructions for the model.

• Example: "Prompting feels like taking things from your brain, analyzing them, and then externalizing them into the model." You must take what's in your head and transform that into words.

• Making the Invisible Visible: The goal of prompt engineering is to make complex and hard-to-articulate ideas into clear and precise instructions.

8. Practical Tips: Sharpening Your Prompting Skills

Read Your Prompts as a Human: Seeing Through New Eyes

• Insight: Always review your prompts from the perspective of someone encountering them for the first time, allowing you to identify ambiguities that you may have overlooked.

• Example: "Try to read your prompts as if you are a human encountering them for the first time." This helps you identify parts of your prompt that may not make sense.

• Empathy for the Reader: You must try and see the prompt from the point of view of someone else.

Use the Model to Improve Prompts: Leveraging AI for Feedback

• Insight: Ask the model for feedback on your prompts to refine them further, creating a feedback loop of improvement.

• Example: "What could I have said that would make you not make that error? Write that out as an instruction." The model can often tell you what you need to do to improve your prompt.

• An Iterative Process: Testing your prompts and getting feedback from the model is an ongoing process and a cycle that can lead to great improvement.

• Using Papers Directly: Amanda uses models to process academic papers directly. She explains, "I give it the paper and then I'm like, 'Here's a paper about prompting technique. I just want you to write down 17 examples of this.' And then the model just does it 'cause the model read the paper." This demonstrates how she leverages the model's ability to read and understand complex documents, allowing it to generate examples or summaries based on the content of the paper.

9. Common Pitfalls: What Not to Do

Avoiding Anthropomorphism: Remember, It's Not Human

• Insight: Avoid treating models as if they think like humans. They process information differently and do not have feelings or emotions.

• Example: "People over-anthropomorphize models, which can lead to misunderstandings." They operate on logic and information, rather than feelings and intuitions.

• The Differences Between AI and Humans: AI models process information differently than humans do, and this must be considered when writing a prompt.

Overcoming Laziness: The Importance of Detailed Prompts

• Insight: While it’s tempting to rely on the model to "figure it out," clear and detailed prompts yield better results. Avoid the temptation of laziness and make sure to put in the time.

• Example: "There's a laziness that overtakes me if I'm talking to Claude where I hope Claude just figures it out." You must not fall into the trap of expecting the models to simply do what you want without detailed instructions.

• Invest the Time: Make sure you put in the time and effort into your prompts.

10. The Power of Questions: Interacting with the Model

Debugging with Questions: Eliciting the Model's Perspective

• Insight: Use questions to understand why the model made an error and how to improve the prompt.

• Essential Questions:

  • "Why did you do that?"

  • "What could I have said that would make you not make that error?"

  • "What other details can I provide to help you answer better?"

• Learning from the Model: By asking the model these questions, you treat it as a collaborator in refining your prompt and help it to produce the desired results.

Amanda Askell’s insights on prompt engineering highlight the importance of clarity, iteration, and a deep understanding of how models process information. By treating prompt writing as a form of programming with natural language and engaging in a feedback loop with the model, you can achieve more accurate and meaningful results. As she aptly puts it, "Clear prompting is often just me understanding what I want." With practice and patience, anyone can master the art of prompt engineering.

Highly recommend watching all 3 of these videos by Amanda Askell to get better at prompt engineering. She regularly writes prompts for the Enterprise Customers at Claude and her cross-pollination of philosophy and prompt engineering makes her one of the best people in the world to learn from.

Top Tweets of the day

1/

This guy gaslighted Claude to give him a decent answer without giving a moral lecture.

There are many guardrails among the simplest of prompts and sometimes you just want an answer. Gaslighting the LLM is one way to extract info. Social Engineers would do well in this area since they know how to extract info without being suspicious.

2/

Voice-to-text is a breakthrough that should be massively used with o1 since it needs lots of context upfront.

I saw an analogy about o1 recently:

Think of o1 like going to a lab and getting your blood test results back in 3-6 hours. You need to give the lab doctors all your medical history to get the perfect solution for your test.

o1 solves the extra hard problems that no other LLM can solve. It doesn't do well with simple problems for some reason and needs an entirely different chain-of-thought to prompt.

I'm only squeezing very little juice with LLMs and I get a lot out of it when I put a bit of an effort in prompting it.

3/

The faster your users can finish your onboarding, the better your conversion rate will be. This is true for B2B SaaS or B2C Mobile App.

You can see it right now by how many inputs there are on the Facebook page. They used to have 7 inputs but now they have reduced to only 2.

Ideally, you want to fetch data from ClearBit to get company info like how Segment does to build a hyper-personalized landing page depending on the company URL or IP. Extremely advanced stuff for B2B companies that increase revenue fast.

Rabbit Holes

1. Everything I Learned From Doing 1000 Videos by Nas Daily

2. o1 isn’t a chat model (and that’s the point) by Ben Hylak

3. Not So Lazy & Entitled Millennials : David Perell by David Perell

What’d ya think of today’s newsletter? Hit ‘reply’ and let me know.

First time? Subscribe.

Follow me on X.

More Startup Spells 🪄

1. Bypass Facebook Ad Guidelines using ChatGPT (LINK)

2. AI Ads: 300x Conversion Rate (LINK)

3. Pity Marketing on TikTok skyrocketed this app on App Store (LINK)

4. Reverse-Engineering Business Model of Newsletters (LINK)