brunohjs/rasa-model-report

Simple open-source Rasa command-line add-on that generates training model health reports for your projects.

🔍 About

Rasa is the most popular open-source framework for building chat and voice-based AI assistants. The rasa-model-report is a unofficial Rasa add-on to facilitate the work of developers and curators of Rasa chatbots. Rasa provides a lot of valuable data that can be "faceted" and extract different information about the training model. This information makes it possible to discover problems in the training model. The rasa-model-report does just that, it extracts this information to be displayed more clearly in a report. You can see this example.

rasa-model-report is an open-source project.

☕ Donate

You can collaborate with this project donating any amount. Thanks! ☕

📜 Changelog

Changelog can be found here. You can also follow the releases on Github and find planned enhancements for project in the Project Board.

📦 Installation

This module is distributed via Pypi and is required to use Python v3.8 or higher. To install the package, use the command:

pip install rasa-model-report

🚀 Execution

Before anything, is necessary to have the reports generated by the rasa test command. To run the program, use the command:

rasa-model-report

This command must be used in the root of your Rasa project. Otherwise, you can use --path parameter to pass the project path.

Step-by-step

This is the step-by-step guide for using rasa-model-report in your project.

Go to the root folder of your Rasa project.
Train model on your Rasa project using rasa train command.
Run Rasa end-to-end tests using rasa test command.
- This command will generate some data in json, markdown and image files in result/ directory.
- This data is needed for rasa-model-report to generate the report.
(Optional) If you want to know model NLU rating for each sentence in your project, run your project's Rasa API through the command rasa run --enable-api.
- When you run rasa-model-report, automatically it will request NLU rating for each sentence. The result will be in the NLU section of the report.
- If you don't want to use this option, just pass the parameter --disable-nlu or don't run Rasa API (if you don't run Rasa API, rasa-model-report will try to connect, after two tries it will skip this step).
Run rasa-model-report in root project.
- If you haven't installed it, see how to install.
The result will be in the model_report.md file generated in the project root folder.

Below, I created this video to show how to use the rasa-model-report v1.0.0. I used the Rasa sample project (from rasa-init command). In this link is the generated report.

example.mp4

Furthermore, I used the tool in Sara - the Rasa Demo Bot. The result you can check here.

🦾 Rasa Version Support

Not every version of Rasa is supported. Check the table below:

Rasa version	Supported
3.X	✅
2.X	✅
1.X	❌
0.X	❌

⚙️ Options

There are parameters that can be used. Available options are below:

--actions-path TEXT     Actions path. (default: actions/ inside Rasa project
                        path)
--disable-nlu           Disable processing NLU sentences. NLU section will
                        not be generated in the report. Required Rasa API.
-e, --exclude LIST      List of utter and actions that will be exclude in
                        the E2E test coverage. Use commas to separate items.
                        Example: utter_greet,utter_goodbye,action_listen
-h, --help              Show this help message.
--model-link TEXT       Model download link. It's only displayed in the
                        report to model download.
--no-images             Generate model report without images.
--output-path TEXT      Report output path. (default: ./)
-p, --path TEXT         Rasa project path. (default: ./)
--precision INTEGER     Score precision. Used to change precision of the
                        model report overview scores. Can vary between 0 and
                        5 (default: 2)
--project-name TEXT     Rasa project name. It's only displayed in the
                        report. (default: My project)
--project-version TEXT  Project version. It's only displayed in the report
                        for project documentation.
--rasa-api TEXT         Rasa API URL. Is needed to create NLU section of
                        report. (default: http://localhost:5005)
--rasa-version TEXT     Rasa version. It's only displayed in the report for
                        project documentation.
-v, --version           Show installed rasa-model-report version.

Usage examples

Some usage examples with parameters:

Without parameters is usually used at the root of the Rasa project.
```
rasa-model-report
```
When you aren't at the root of the project, use --path parameter.
```
rasa-model-report --path path/to/rasa/project
```

Aren't at the root of the project and without NLU report.

rasa-model-report --path path/to/rasa/project --disable-nlu

Aren't at the root of the project and change the report output directory.

rasa-model-report --path path/to/rasa/project --output-path path/to/place/report

Aren't at the root of the project, the actions path isn't at the root project and change the report output directory.

rasa-model-report --path path/to/rasa/project --output-path path/to/place/report --actions-path path/to/actions/path

If you want exclude some utters and actions from the E2E test coverage.
```
rasa-model-report --exclude utter_greet,action_help
```

💻 Development

The instructions for development and contributing are in the CONTRIBUTING.md file.

🐞 Bugs

Please file an issue for bugs, missing documentation, or unexpected behavior.

See open issues

💬 Discussions

Please file an issue to suggest new features. Vote on feature requests. This helps maintainers prioritize what to work on.

See new ideas discussion

❓ Questions

For questions related to using the add-on, please ask the community on Q&A.

See Q&A