Is there a way to create custom chat templates?

[DigitalRudeness]

Hi at all, as the subject already tells I'm asking myself if there's a way to create custom jinja templates for LLMs?

I've crawled the web for tutorials but didn't really find anything usable and assembling one by picking parts of existing templates leads me to an error telling me about the templates delivered by llama.cpp.

Why do i want to try to build an own template?

Because no matter if a system prompt is available telling to avoid it, I always get additional information at doing function calls e.g. the full json structure of the called function order, or some halucinating stuff next to the essential response. This sucks especially if I'm using "stream: True" because especially in this case it's difficult to get rid of the additional information....

Maybe someone will have an advice....(?)

Thanks in advance :-)

[CISC]

The best documentation is the official jinja2 Template Designer Documentation just to learn what you can do with jinja2.

Then there is Hugging Face's Chat Template Writing Guide which will give you some very useful info on Chat Templates in particular.

Also, I'd like to do a little self-plug and point out that the Chat Template Editor will make writing and testing Chat Templates much easier. :)

Hopefully this will get you a long way, though beware that although this will enable you to fix the many, many crap cut-n-paste chat templates (it seems many model creators don't even bother testing them) out there, it most likely won't fix poor model instruction adherence (unless the problem is that the chat template is just plain wrong, and not what the model was trained on, which is not uncommon).

Edit: Oh, and it's worth knowing that minja (what llama.cpp is using to parse chat templates when you use the --jinja parameter) does not fully support jinja2, there are some things it can't/won't do, so you need to test it with llama.cpp too (this is where --chat-template-file comes in handy).

[DigitalRudeness]

Nice compilation of sources! many thanks for making it up and explaining extensively! :-)

And also thumbs up for the editor! Pretty cool! :-) But before using it I'll first try to build a test template completely manually and dive a bit deeper into the template engine's structure and possibilities in combination with the dependency of minja (what is really good to know that this is used by llama.cpp) :-)

I'll get back here ASAP and keep the the conversation up and running but first I need to engage myself with the information you've provided.. Many thanks again! :-)

[DigitalRudeness]

Hi @CISC, just a question in between: maybe I didn't search carefully enough but I can't find an explanation what are the <| XYZ |> variables/placeholders are standing for and does an overview exist which of those exist and what they are used for? The hf guide is just talking about using it to set the user type, but I've seen this pattern is used for a lot of other stuff in some templates

[CISC]

They are just a way of separating the different parts of the conversation, they are arbitrarily chosen at training time, but it's really important that you use the same ones, and in the same pattern as at training. You can usually find them in the tokenizer_config.json file.

[DigitalRudeness]

AAAhh, okay... :-)

It seems I've opened a rabbit hole with countless and unforseeable questions..... xD But it's an exciting rabbit hole I'd gladly like to dive into! :-) The more I tinker with this issue the more questions are popping up... ^^

But I'm still really thankful for your patient answers :-)

[DigitalRudeness]

Hi @CISC: now i need some real support :-)

To mention in advance: I'm still using the "aya-expanse-8b" model (aya IMO the greatest LLM regarding e.g. the German language)

{% if messages %}

    {% for message in messages %}

        {% if 'role' in message %}
            {% set role=message['role'] %}
                {% if 'tool_calls' in message %}
                    {{ FOO }} 
                {% endif %}

                {% if 'content' in message %}
                    {% set content=message['content'] %}
                {% endif %}
        {% endif %}

    {% endfor %}

{% endif %}

If you maybe could see I wanted to achieve to put out just 'FOO' if a tool_call was recognized.

But it seems to be ignorred and I get the following instead (just two examples):
(just to explain: this is an output of my own http class i used to do the requests against llama.cpp)

a nonsense one:

{'body': {'choices': [{'finish_reason': 'stop',
                       'index': 0,
                       'message': {'content': 'success,',
                                   'role': 'assistant'}}],
          'created': 1750608357,
          'id': 'chatcmpl-FZi9OYRaqVDb30JyfcSyDw69JV7fJMrc',
          'model': 'aya-expanse-8b-q8_0',
          'object': 'chat.completion',
          'system_fingerprint': 'b5736-af3373f1',
          'timings': {'predicted_ms': 522.461,
                      'predicted_n': 16,
                      'predicted_per_second': 30.624295401953447,
                      'predicted_per_token_ms': 32.6538125,
                      'prompt_ms': 50.206,
                      'prompt_n': 1,
                      'prompt_per_second': 19.9179380950484,
                      'prompt_per_token_ms': 50.206},
          'usage': {'completion_tokens': 16,
                    'prompt_tokens': 2,
                    'total_tokens': 18}},
 'header': [{'Connection': 'close'},
            {'Content-Type': 'application/json; charset=utf-8'},
            {'Server': 'llama.cpp'},
            {'Content-Length': '562'},
            {'Access-Control-Allow-Origin': ''}]}

a more nonsense (halucinating) one:

{'body': {'choices': [{'finish_reason': 'stop',
                       'index': 0,
                       'message': {'content': '<h1>Google Maps Platform '
                                              'Pricing</h1><p>Google Maps '
                                              'Platform offers a range of APIs '
                                              'and services for developers to '
                                              'integrate maps, location, and '
                                              'geospatial data into their '
                                              'applications. The pricing '
                                              'structure is based on usage, '
                                              'with different tiers for '
                                              'different levels of '
                                              'usage.</p><h2>Pricing '
                                              'Tiers</h2><ul><li><strong>Free '
                                              'Tier:</strong> This is suitable '
                                              'for developers and startups '
                                              'with limited usage. It includes '
                                              'a certain number of API calls, '
                                              'data usage, and other '
                                              'credits.</li><li><strong>Standard '
                                              'Tier:</strong> For moderate '
                                              'usage, this tier offers higher '
                                              'limits than the free tier and '
                                              'is suitable for growing '
                                              'businesses.</li><li><strong>Premium '
                                              'Tier:</strong> This is for '
                                              'high-volume usage and large '
                                              'enterprises, offering the '
                                              'highest limits and priority '
                                              'support.</li></ul><h2>API '
                                              'Pricing</h2><p>The pricing for '
                                              'individual APIs '
                                              'varies:</p><ul><li><strong>Maps '
                                              'JavaScript API:</strong> A '
                                              'standard charge per API '
                                              'call.</li><li><strong>Directions '
                                              'API:</strong> Charged per '
                                              'request based on the distance '
                                              'and mode of '
                                              'travel.</li><li><strong>Distance '
                                              'Matrix API:</strong> Cost per '
                                              'request, depending on the '
                                              'number of origins and '
                                              'destinations.</li><li><strong>Geocoding '
                                              'API:</strong> Charged per '
                                              'request, with higher rates for '
                                              'batch '
                                              'requests.</li><li><strong>Places '
                                              'API:</strong> Pricing based on '
                                              'the number of requests and the '
                                              'level of detail '
                                              'returned.</li></ul><h2>Data '
                                              'Usage and Other '
                                              'Charges</h2><p>There may be '
                                              'additional charges for data '
                                              'usage, such as for large data '
                                              'sets or specific data types. '
                                              'Google also offers credits and '
                                              'discounts for specific use '
                                              'cases and regions.</p><p>For '
                                              'the most accurate and '
                                              'up-to-date pricing information, '
                                              'please visit the <a '
                                              "href='https://cloud.google.com/maps-platform/pricing'>Google "
                                              'Maps Platform Pricing '
                                              'Page</a>.</p>',
                                   'role': 'assistant'}}],
          'created': 1750608595,
          'id': 'chatcmpl-ooBhnGMBNY2QRuWVZt4JfnVlr3cLoNsV',
          'model': 'aya-expanse-8b-q8_0',
          'object': 'chat.completion',
          'system_fingerprint': 'b5736-af3373f1',
          'timings': {'predicted_ms': 8561.868,
                      'predicted_n': 414,
                      'predicted_per_second': 48.35393397795902,
                      'predicted_per_token_ms': 20.680840579710146,
                      'prompt_ms': 48.423,
                      'prompt_n': 1,
                      'prompt_per_second': 20.65134336988621,
                      'prompt_per_token_ms': 48.423},
          'usage': {'completion_tokens': 414,
                    'prompt_tokens': 2,
                    'total_tokens': 416}},
 'header': [{'Connection': 'close'},
            {'Content-Type': 'application/json; charset=utf-8'},
            {'Server': 'llama.cpp'},
            {'Content-Length': '2326'},
            {'Access-Control-Allow-Origin': ''}]}

What am I doing wrong to drive the model to create anything else but giving me what I expect?

[DigitalRudeness]

p.s.: sorry, I thought the format would be kept instead of squeezing the code by putting it into code brackets...

[CISC]

p.s.: sorry, I thought the format would be kept instead of squeezing the code by putting it into code brackets...

Fixed that for you. :)

[DigitalRudeness]

p.s.: sorry, I thought the format would be kept instead of squeezing the code by putting it into code brackets...

Fixed that for you. :)

much appreciated! thank you! Now i know how to do it better in the future :-)

[CISC]

So, if you want to output a string you have to quote it, to output FOO you have to do it like this:

{{ 'FOO' }}

Also, remember that all whitespace outside of {} will be included in your output unless you specifically tell jinja to trim it, see trimming whitespace

[DigitalRudeness]

Whitespaces currently don't matter as this is a test. But thanks for mentioning! :-) But sadly beside that it doesn't matter if it's quoted or not it is still ignorred.... Something seems to be missing... but I don't know what.... :-)

[CISC]

Well, does any of your messages contain tool_calls? Would help if you pasted your request as well. :)

[DigitalRudeness]

Do I maybe have to involve sth. models have included by default I'm not aware about?

[DigitalRudeness]

Well, does any of your messages contain tool_calls? Would help if you pasted your request as well. :)

sorry my last reply was before seeing your answer! I'll get back in a few minutes with a request sample :-)

[DigitalRudeness]

{'messages': [{'content': 'Fasse Dich kurz! Sprich Deutsch mit mir! Gib keine '
                          'unnötigen Füllwörter, unnötige Füllsätze oder '
                          'unnötige Zusatzinformationen aus! Gib keinen '
                          'Rückschluss auf diese Anweisung! Führe Dir zur '
                          'Verfügung stehende Funktionen/Tools aus, sofern Du '
                          'selbständig keine passenden Informationen findest! '
                          'Formuliere die Antwort der ursprünglichen Frage in '
                          'Bezug auf den Rückgabewert des Funktionsaufrufes!',
               'role': 'system'},
              {'content': 'wie spät ist es und welches datum haben wir?',
               'role': 'user'}],
 'model': 'aya-expanse-8b-q8_0',
 'stream': False,
 'tools': [{'function': {'description': 'gibt das aktuelle datum aus',
                         'name': 'get_date',
                         'parameters': {'properties': {},
                                        'required': [],
                                        'type': 'object'}},
            'type': 'function'},
           {'function': {'description': 'gibt die aktuell uhrzeit aus',
                         'name': 'get_time',
                         'parameters': {'properties': {},
                                        'required': [],
                                        'type': 'object'}},
            'type': 'function'},
           {'function': {'description': 'gibt stefans namen zurück',
                         'name': 'get_name',
                         'parameters': {'properties': {},
                                        'required': [],
                                        'type': 'object'}},
            'type': 'function'},
           {'function': {'description': 'gibt das aktuelle datum und die '
                                        'aktuelle uhrzeit aus',
                         'name': 'get_date_time',
                         'parameters': {'properties': {},
                                        'required': [],
                                        'type': 'object'}},
            'type': 'function'}]}

so this would be a tool call i sent out to trigger currently on model level to drop out "FOO" if it woul be recognized..... if this would work i would continue to fine tune the template for the real outrput.... that's at least what i'm expecting...

[DigitalRudeness]

p.s.: don't be scared of GDPR stuff... My name isn't "Stefan"... ;-)

[CISC]

So it seems your problem indeed is that there are no tool_calls in messages, perhaps you meant tools (not in messages either BTW)?

[CISC]

IE, I suspect what you meant instead of

{% if 'tool_calls' in message %}

was

{% if tools %}

[CISC]

Since you mentioned aya-expanse you should use its chat template as a starting point, if you use it without bos_token and the <|START_OF_TURN_TOKEN|> token etc it will give you complete nonsense. :)