LSP, Tree-Sitter, and Editor Support

You open a file, everything is monochrome, you can’t jump to a definition, and the experience says “this isn’t ready” before you’ve written a single line. Syntax highlighting was table stakes back when the TextMate grammar landed in Part 10. What I actually needed was an LSP server.

hew-lsp

The LSP server lives in hew-lsp/, a crate in the main compiler workspace. It reuses the same frontend passes as the compiler — lexer, parser, type checker — so diagnostics in the editor match what hew build produces. (This sounds obvious but I’ve used languages where the editor and compiler disagreed about what was an error, and it’s maddening.)

The capabilities right now:

• Diagnostics — real-time error and warning squiggles as you type, powered by the same type checker that runs at compile time. Unused variables, type mismatches, “did you mean?” suggestions for misspelled identifiers — all the same messages you’d see on the command line, but inline.
• Completions — keyword completions, type names, variables in scope, struct fields after ., actor message names. The type checker provides the scope information, so completions know about imported modules and generic instantiations.
• Go-to-definition — jump to where a function, type, struct field, or actor is defined. Works across files when using import "other.hew" or module imports like import app::router.
• Semantic tokens — the LSP sends token-level type information back to the editor, so actors get highlighted differently from structs, receive fn looks different from a regular fn, and type parameters in generics are visually distinct. This layer sits on top of the TextMate grammar and overrides it where the LSP has better information.

Building the LSP on top of the existing compiler passes was the right call. When I added machine/state/event keywords in Part 26, the LSP picked them up immediately — same parser, same type checker, no separate update. The downside is startup time: the LSP loads the full compiler frontend, which takes a noticeable beat on first open. Not slow enough to fix yet, slow enough to notice.

The authority grammar

tree-sitter-hew is the authority grammar for the language. When the syntax changes, grammar.js gets updated first, and everything else syncs from it. The grammar covers the full language surface — function declarations, actor definitions, match expressions with guards, generic type parameters, string interpolation with f"hello {name}", duration literals like 5s and 200ms, the state machine keywords, all of it.

Tree-sitter grammars are interesting to write because they’re declarative but not — you define rules as JavaScript objects, and tree-sitter generates a C parser from them. The tricky parts are precedence and conflict resolution. Hew has a few ambiguities that required explicit precedence annotations: the < in Vec<int> versus the < comparison operator, the | in closure parameters versus bitwise or, => in match arms versus fat arrow in other contexts.

// From grammar.js — generic type arguments need explicit precedence
// to disambiguate Vec<int> from a comparison expression
generic_type: $ => prec(1, seq(
  $.identifier,
  '<',
  commaSep1($.type),
  '>'
)),

The corpus tests in test/corpus/ are essential. Every language construct has at least one test case verifying the parse tree shape. When I add a keyword, the test fails before the grammar is updated — which is the point.

Publishing happens to npm as @hew-lang/tree-sitter-hew. The tree-sitter CLI uses this for tree-sitter highlight in any editor that supports tree-sitter natively — Neovim, Helix, Zed. The highlight queries in queries/highlights.scm map tree-sitter node types to highlight groups:

; highlights.scm
(function_declaration name: (identifier) @function)
(actor_declaration name: (identifier) @type)
(receive_function "receive" @keyword "fn" @keyword)
(type_identifier) @type

The TextMate grammar

VS Code doesn’t use tree-sitter natively (yet — there’s been talk for years), so it needs a TextMate grammar. hew.tmLanguage.json in vscode-hew is a 900-line regex-based tokenizer that covers keywords, string interpolation, comments, number literals, operators, and multi-keyword constructs like receive fn and wire type.

TextMate grammars are regex soup. There’s no getting around it. The grammar works well enough for static highlighting, but it can’t handle things like matching generic brackets across nested types — HashMap<string, Vec<int>> will occasionally mis-highlight the closing >>. The semantic tokens from the LSP clean this up at runtime, but when the LSP hasn’t started yet, you see the TextMate layer, warts and all.

Testing uses vscode-tmgrammar-test with .test.hew files:

receive fn handle(msg: string) -> bool {
// <- keyword.control.hew
//      ^^ keyword.control.hew
//                            ^^^^^^ storage.type.hew
//                                      ^^^^ storage.type.hew

The caret positions map to character ranges and assert specific TextMate scopes.

The syntax pipeline

This is the part that took the longest to get right, mostly because I didn’t design it — it emerged. The canonical source of truth for keywords, operators, and types is syntax-data.json, exported from the compiler’s lexer. When I add a keyword to the Rust lexer, I regenerate that JSON file, and then the pipeline fans out:

1. tree-sitter-hew — update grammar.js manually (no generator, the grammar structure is too semantic for templating)
2. tree-sitter-hew — update queries/highlights.scm
3. vscode-hew — update hew.tmLanguage.json
4. vim-hew — update syntax/hew.vim
5. hew.sh — run node tools/generate-monarch.mjs /path/to/syntax-data.json to regenerate Monaco tokenizer keyword lists
6. hew.run — manually copy the Monaco keyword/type arrays from hew.sh

Step 5 is the only automated one. The generator script reads syntax-data.json, extracts every keyword, type, and operator, and replaces the arrays in src/lib/monaco/hew-language.ts. Step 6 is a manual copy because the playground doesn’t have its own generator yet. (I keep meaning to fix this. I keep not fixing this.)

When I added machine, state, event, on, and when as keywords, I touched six repos. Missed vim-hew the first time and only noticed when someone opened a .hew file in Neovim and state machine blocks were uncolored. The pipeline works, but it’s a pipeline you have to remember to run.

vim-hew

The Vim plugin is simpler than everything else — a syntax/hew.vim file with keyword groups, match patterns for operators and numbers, and region definitions for strings and comments. Vim syntax highlighting is line-oriented and stateless between lines, which means multi-line string interpolation doesn’t highlight correctly. (Vim people know this and accept it. The Vim way is to not have multi-line strings.)

Installation is via any Vim plugin manager — Plug 'hew-lang/vim-hew' or the Neovim equivalent with lazy.nvim. For Neovim users who want tree-sitter-based highlighting instead, nvim-treesitter can use the @hew-lang/tree-sitter-hew parser directly, which gives better results than the regex-based syntax file.

Interactive debugging

The VS Code extension bundles the LSP binary and launches it automatically when you open a .hew file. It publishes to the VS Code Marketplace under hew-lang.hew-lang. Version 1.1.0 added interactive debugging via DAP (the Debug Adapter Protocol) — breakpoints, stepping, call stack, variables, the whole thing.

The debug adapter translates DAP into GDB/LLDB MI (Machine Interface) commands. It auto-detects which backend is available, auto-compiles the .hew file with hew build --debug, and spawns the debugger. Runtime frame filtering hides the internals — hew_runtime_*, __pthread, __libc frames are stripped from the call stack so you see your actor code, not the scheduler plumbing.

There’s also an Actors tree view panel that shows live actor state during debugging — which actors are running, their mailbox depth, their current state. (Whether that’s actually useful for debugging or just satisfying to watch, I honestly can’t tell yet.)

Hover documentation is still missing. The LSP knows the type of everything under your cursor but doesn’t yet surface doc comments. I have the doc comment syntax in the grammar — /// triple-slash comments — but the LSP doesn’t collect them and send them as hover content. It’s on the list, right after the eight other things on the list.