Syntax Highlighter — Core/Plugin (Phase 1–3: Haskell, Rust, Dart)

A modular syntax highlighter built for a university project using a Core/Plugin architecture.

Phase 1 focuses on a Haskell subset and supports:

• ANSI-colored output (terminal)
• HTML/CSS output (browser)

Project structure

src/highlighter/
  cli/                  # Minimal CLI package (python -m highlighter)
  core/                 # Token, Lexer/Parser base, AST base, registry, renderers
    renderers/          # ANSI/HTML renderers
  languages/
    haskell/            # Haskell lexer/parser/highlighter + grammar + rules
tests/                  # Unit + integration + E2E snapshot tests
samples/                # Sample inputs for demos/tests
docs/                   # Scope, examples, theme, usage guide

Phase 1 (Haskell) — scope summary

Implemented subset (for highlighting needs):

• Function declarations with simple patterns
• Type signatures (name :: Type)
• if / then / else
• Binary expressions and application
• List literals and basic list patterns ([], (x:xs))
• Strings/chars/numbers, comments

References:

• Scope document: docs/haskell_scope.md
• Grammar (EBNF): src/highlighter/languages/haskell/grammar.ebnf

Known limitations (Phase 1)

• Haskell layout/offside rule is not implemented (indentation-based parsing is simplified).
• Unary minus may not be supported in the parser (use 0 - x in samples).
• Parser is MVP for subset (goal: correct AST shape for supported constructs).

Phase 2 (Rust) — scope summary

Implemented subset (for highlighting needs):

• fn declarations with blocks
• let / let mut bindings (optional type annotations)
• if / else, while, for ... in ...
• struct declarations and impl blocks with simple methods
• Literals (int/string/char/bool) and comments (//, /* */)

References:

• Scope document: docs/rust_scope.md
• Grammar (EBNF): src/highlighter/languages/rust/grammar.ebnf

Phase 3 (Dart + Web demo) — scope summary

Implemented subset (for highlighting needs):

• class declarations with members (field / constructor / method)
• Top-level functions
• Control flow: if / else, for, while, return
• Basic types and literals (int/string/bool), comments
• Token-based + parser-based highlighting toggle (for speed)

Interactive web demo (Phase 3):

• Language selector (Haskell/Rust/Dart)
• Theme selector (light/dark)
• Live highlighting (debounced) + Parse toggle
• Backend API: /api/meta, /api/highlight

References:

• Dart scope: docs/dart_scope.md
• Final checklist + demo steps: docs/final_checklist.md

Installation

1) Create a virtual environment

Windows (PowerShell):

python -m venv .venv
.venv\Scripts\Activate.ps1

macOS / Linux (bash):

python -m venv .venv
source .venv/bin/activate

2) Install dependencies

pip install -U pip
pip install -e .

Usage (CLI)

The CLI is runnable as a Python module.

Phase 2 (multi-language)

The CLI supports language selection via --lang.

• Phase 1 ships with Haskell enabled by default.
• Phase 2 will add Rust (and later languages) via plugins.

Tip: run python -m highlighter --help to see the currently available --lang choices.

ANSI output (terminal)

python -m highlighter samples/haskell/e2e_01.hs --lang haskell --format ansi

HTML fragment output

python -m highlighter samples/haskell/e2e_01.hs --lang haskell --format html --out out_fragment.html

HTML document output (with optional CSS)

If docs/theme.css exists, it will be embedded automatically:

python -m highlighter samples/haskell/e2e_01.hs --lang haskell --format html-doc --theme light --out out.html

You can also wrap the document with a dark theme:

python -m highlighter samples/haskell/e2e_01.hs --lang haskell --format html-doc --theme dark --out out_dart_dark.html

Skip parsing if needed

If you want token-based highlighting only:

python -m highlighter samples/haskell/e2e_01.hs --lang haskell --format ansi --no-parse

Rust (Phase 2)

Rust is available via --lang rust (Phase 2):

python -m highlighter samples/rust/demo.rs --lang rust --format html-doc --out rust.html

More details:

• Haskell usage: docs/phase1_usage.md
• Rust subset: docs/rust_scope.md

Examples for the report

• HTML/CSS theme: docs/theme.css
• Example snippets: docs/examples.md

Web demo (Phase 3)

Run backend:

python -m uvicorn web_demo.backend.app:app --reload --port 8000

Serve frontend:

cd web_demo/frontend
python -m http.server 5500

Open http://127.0.0.1:5500/.

More detailed steps: docs/final_checklist.md.

Testing (important)

Install dev dependencies (includes pytest):

pip install -e .[dev]

Run all tests:

pytest -q

E2E snapshot tests compare final outputs (ANSI/HTML):

• Test file: tests/test_e2e_haskell.py
• Snapshots: tests/snapshots/haskell/*

To generate/update snapshots:

UPDATE_SNAPSHOTS=1 pytest -q

Rust integration tests validate that the HTML output contains expected token classes:

• tests/integration/test_rust_highlight_html.py

Team workflow (Phase 1)

• Core + renderers: Mahdiyeh
• Haskell implementation + CLI glue: Saghar
• Tests + documentation: Mobina