Playbooks - Cognigy Documentation

Documentation Index

Fetch the complete documentation index at: /llms.txt

Use this file to discover all available pages before exploring further.

Playbooks are conversation scripts that you can run automatically to test your Flows. When running a Playbook, you can set the Flow, Locale, and Snapshot you want to test. Playbooks include:

Steps — simulate a user input and can include an input text and input data.
Assertions — check the AI Agent’s responses and different aspects of the Flow for each Step against a defined value. You can also invert the Assertion check, which means the Assertion checks if the defined value is missing.

Assertion Types

Type	Description	API Value
Message text assertion	Checks the text content of the AI Agent’s response.	`assertText`
Message data assertion	Checks the data the AI Agent sent.	`assertData`
State assertion	Checks the current State of the Flow.	`assertState`
Context assertion	Checks the current data in the Context object of the Flow.	`assertContext`
Slots assertion	Checks if specific Slots have been recognized.	`assertSlots`
Intent assertion	Checks if specific Intents have been recognized.	`assertIntent`

Limitations

You can run up to 10 Playbooks simultaneously. You can change this limit for on-premises installations by specifying the MAX_CONCURRENT_PLAYBOOK_EXECUTIONS variable in the values.yaml file.
A Playbook can contain up to 50 Steps.

Restrictions

Playbooks don’t allow dots (.) in JSON keys. Referencing subkeys like user.name can cause runs to be hidden or Snapshots to fail. Instead, validate the full JSON object:

Valid Example

{
     "user": {
         "name": "John"
     }
}

Invalid Example

{
      "user.name": "John"
}

Working with Playbooks

In Test > Playbooks, you can create, delete, and run Playbooks as well as export them as a Package. To define Steps and Assertions in the Playbook editor, select a Playbook from the list. Playbook runs are Tasks and take place in the background. The results are available in the Playbook editor. Alternatively, you can also run a Playbook and view the results from the Interaction Panel, as well as create Playbooks from a conversation with the AI Agent.

Playbook Editor In the Playbook editor, configure the following settings for all Steps:

Setting	Description
Set timeout per step	The timeout for each step in the Playbook in milliseconds. If a step takes longer than the set timeout, an error is displayed in the Playbook player. The default timeout is 20000 ms.
Abort on error	Select whether to continue or abort the Playbook after a failed step.

For each Step, you can enter input text and data to send to the AI Agent, and add Assertions to check the AI Agent’s response to the input.

Runs Tab On the Runs tab in the Playbook editor, you can run Playbooks and view the results of the Playbook runs. You can also filter the runs by status and creator. After the Playbook run is completed, click it in the list to see the results. In the pane that opens on the right, you can hover over the Assertion checks for more details.

Interaction Panel To run a Playbook in the Interaction Panel, select and run a Playbook from the tab. You can hover over the Assertion checks for more details. To create Playbooks from a conversation, follow these steps:

Chat with the AI Agent on the tab.
Click above the chat window and select Create Playbook.

You are redirected to the Playbook editor. You can then add Assertions and additional Steps.

You can run Playbooks using the Cognigy.AI CLI. Schedule multiple Playbook runs simultaneously using the run command.

You can create, edit, and delete Playbooks using the Cognigy.AI API as well as run Playbooks.

Examples

The following video provides examples of how to create and run a Playbook using the Interaction Panel.

Cognigy.AI Developer Training #7: Testing Flows with Playbooks - YouTube