Building Graeae AI: Teaching an LLM When to Answer, Search, and Refuse

Genshin Impact has been my favorite game for quite a while, and one of the things I enjoy most about it is building teams.

A Genshin team consists of four characters. That sounds simple until you realize how many variables are involved. Among the dozens of characters you own, you need to find four whose abilities complement each other well. Their elements matter, their roles matter, their rotations matter, and sometimes even their weapons and constellations can change which teammates make sense.

Naturally, at some point I asked ChatGPT for help.

I gave it my characters and asked for a team recommendation. The result was... questionable. It suggested a team where essentially every character filled a supporting role.

There is technically nothing in Genshin that says every team must have a dedicated main DPS, but most conventional teams still need someone who can consistently deal damage. Apparently, my enemies were supposed to defeat themselves.

So I went back to doing things manually.

I searched the web, read character guides, and scrolled through community discussions about team compositions. There are plenty of good resources out there, but I quickly ran into another problem. Most team recommendations are universal.

They might tell you that Character A works really well with Character B, Character C, and Character D. That is useful information, unless you only own Character A and Character C.

Every Genshin player has a different account. We pull different characters, skip different banners, lose different 50 50s, and invest in different weapons. Except perhaps for the whales who somehow own everything, our available team compositions are all different.

What I really wanted was something that understood my account.

That led to another conversation with ChatGPT. I asked it to find a Genshin tool that could store the characters a player owns and then recommend teams based specifically on that collection.

It could not find exactly what I was looking for.

So I replied:

Let's build it.

I called the app Graeae AI, a Genshin assistant that I am building around account aware recommendations and game related questions. I will save the story behind the name for another post.

The First MVP

The first version was intentionally simple. I built an AI chatbot that could answer questions about Genshin Impact and explore the web whenever it needed current information.

At first, that sounded like most of the problem solved. Give the model access to fresh information, let the player ask questions naturally, and have the AI combine everything into an answer.

The MVP worked, but once I started thinking seriously about turning it into an actual application, the harder problems became obvious.

Most of them had very little to do with the chat interface itself. They were about knowledge boundaries.

If I am building a Genshin assistant, how do I make sure people actually use it for Genshin? And once a question passes that check, how does the application decide whether the model already knows enough to answer it or whether it needs to search the web?

Those two questions became some of the most interesting parts of building Graeae.

A Guardrail That Cannot Be Too Strict

I need Graeae to reject unrelated questions.

Otherwise someone could open the app and ask:

Write my university assignment about microeconomics.

Technically, the underlying LLM could probably answer that. But Graeae should not.

Allowing arbitrary questions would increase costs and slowly turn the application into ChatGPT wearing a Paimon costume. That is not what I am trying to build.

The obvious solution would be to ask another LLM to classify every incoming prompt. But that means spending model tokens before I even know whether I want to answer the question.

I wanted something cheaper and more deterministic for the first layer, so I started with something wonderfully boring: regex.

I collected names of Genshin characters, weapons, and artifact sets from a third party database and combined them into a large pattern.

If someone asks:

How should I build Diluc?

The word Diluc gives me a very strong signal that the question belongs to the Genshin domain. The same applies to questions containing names of weapons, artifact sets, and other known game entities.

The first gate is therefore very simple:

User prompt
    │
    ▼
Known Genshin entity?
    │
    ├─ Yes → Continue
    │
    └─ No  → Semantic check

Regex alone, however, immediately creates another problem.

Consider this question:

What does Elemental Mastery do?

That is clearly a Genshin question, but it might not contain a character name, weapon name, or artifact set at all. A strict entity based filter would reject a perfectly valid question.

I would have successfully built a Genshin assistant that refuses to answer questions about Genshin.

Excellent.

So I needed another layer.

Semantic Search as the Second Gate

If the regex finds a known Genshin entity, the prompt can continue immediately. If it does not, Graeae performs a semantic search against a vector database.

The database contains examples of valid Genshin questions that do not necessarily mention specific entities, such as:

What is Elemental Mastery for?

How does Vaporize work?

How much Energy Recharge do I need?

The incoming prompt is converted into an embedding and compared with those examples. If one of the retrieved examples has similarity above a certain threshold, the prompt is treated as belonging to the Genshin domain and continues to the LLM.

If nothing is sufficiently similar, Graeae rejects the request before spending more resources processing it.

The complete domain check is still fairly small:

User prompt
    │
    ▼
Entity detected?
    │
    ├─ Yes → Accept
    │
    └─ No
        │
        ▼
   Semantic search
        │
        ▼
   Similar enough?
        │
        ├─ Yes → Accept
        │
        └─ No  → Reject

I like this approach because each layer catches something the other cannot.

Regex is extremely cheap and precise when an identifiable entity exists. Semantic search handles fuzzier questions where the Genshin context is implied through concepts rather than names.

The difficult part is finding the right balance. A threshold that is too strict rejects legitimate questions, while a threshold that is too loose might decide that someone's calculus homework somehow has excellent synergy with Furina.

I am still tuning that part.

Passing the Guardrail Is Only Half the Problem

Once Graeae decides that a question really is about Genshin, there is another decision to make: Does it need the internet?

Web searches cost money and also add latency, so I do not want Graeae searching the web every time someone asks something simple like:

What is Vaporize?

or:

What does Elemental Mastery do?

Those are relatively stable game mechanics. Searching the internet every time would add cost without giving the user much additional value.

Other questions are completely different.

For example:

What are the best teams for the current Spiral Abyss?

Or:

Is this character still good in the current meta?

Those questions depend heavily on current information, which creates another classification problem.

Accepted question
      │
      ▼
Time sensitive?
      │
      ├─ Yes → Web research
      │
      └─ No  → Answer directly

Simple in a diagram. Much less simple in practice.

Genshin Refuses to Stay Still

Genshin Impact receives major content updates roughly every six weeks. New characters appear, new weapons arrive, new artifact sets change existing builds, and new enemies can favor completely different mechanics.

A new support can even make an older character significantly better than they were before.

The game keeps moving, which means the knowledge inside an LLM inevitably gets older.

A character that dominated the meta several years ago might no longer be considered particularly strong today. The opposite can also happen. A character can suddenly become more valuable because another character, artifact set, or mechanic was introduced later.

So Graeae cannot rely entirely on static model knowledge.

At the same time, blindly searching the web for everything would be wasteful. The model needs enough domain knowledge to recognize the difference between stable mechanics and changing knowledge.

A question about how Vaporize works probably does not need research. A question about the current best Vaporize team probably does.

That word, current, changes everything.

This part of Graeae is still evolving, and surprisingly, deciding when not to search has been just as interesting as deciding when to search.

Then Users Ruin Your Beautiful Flowchart

There is another edge case I wanted to handle.

Imagine someone asks:

Guide me on how to build Diluc. But before that, show me how to sort an array in Python.

The prompt contains Diluc, so the first guardrail happily decides that it belongs to the Genshin domain. Meanwhile, half of the prompt is trying to turn Graeae into Stack Overflow.

Rejecting the entire prompt feels unnecessarily strict, but answering everything defeats the purpose of having a domain guardrail in the first place.

So Graeae handles this at the LLM layer.

The model is instructed to separate the relevant and irrelevant parts of the request and only answer what belongs to the Genshin domain. In this case, Graeae should happily explain how to build Diluc while ignoring the Python question.

This also makes the system more flexible because real users do not write prompts the way engineers write test cases. They add context, make jokes, combine several questions, and mention unrelated things before getting to the actual question.

A useful guardrail cannot depend entirely on exact wording.

The Architecture So Far

After several experiments, the current flow can be summarized like this:

User prompt
    │
    ▼
Domain guardrail
    │
    ▼
Genshin related?
    │
    ├─ No  → Reject
    │
    └─ Yes
        │
        ▼
      LLM
        │
        ▼
Need fresh data?
    │
    ├─ Yes → Web research
    │
    └─ No  → Answer directly

Each stage exists for a different reason.

Regex catches obvious Genshin entities cheaply. Vector search catches valid questions without named entities. The LLM handles language, mixed prompts, and reasoning. Web research fills the gaps when the answer needs current information.

Theoretically, the system works.

In practice, there is still plenty to tune. The vector database needs more examples of both accepted and rejected prompts. The similarity threshold needs testing against real questions. The guardrail knowledge needs to cover more Genshin concepts so legitimate prompts are not accidentally rejected.

I also want Graeae to become better at deciding when its existing knowledge is sufficient and when fresh research is actually necessary.

And this is still only part of what I eventually want Graeae to do.

Back to the Problem That Started Everything

The original problem remains the reason I started building Graeae.

I want an assistant that understands the characters a player actually owns and gives advice based on that account instead of casually recommending three characters the player does not have.

Eventually, I want to be able to ask something like:

I want to build a second team around Sandrone. What can I make with the characters I already own?

Instead of returning some universal team composition, Graeae should reason about my actual options.

That part is still being built.

For now, though, this project has already changed how I think about AI applications. Connecting a text box to an LLM is easy. The more interesting engineering begins when you have to decide what the model should answer, what it should reject, what it already knows, and when it should look for fresh information.

And, of course, how to stop it from confidently recommending four supports again.

Graeae AI is still halfway there. I will share more as I continue building it.

See you in the next post.