A lot of things described in this article applicable not only for Rust, but for almost any language. Like it's obvious that requests should be handled asynchronously and that conversions from/to UTF-16 are needed. But it's actually not so hard.
I have written a language server for my language too. The hardest thing was to find a way allowing providing useful autocompletion for a document in edited state, when it's not syntactically-correct. This is the trickiest part how to deal with such incorrectness without missing all the context necessary.
I've not written a language server but have written a language plugin for IntelliJ.
I started with writing a correct recursive descent parser. I then extended it to detect, report, and recover from common syntax errors as I encountered them so that the parser is robust. And adding a parser test case for each of these (e.g. one test for each branch through an EBNF construction).
Some examples are:
1. missing keywords when the keyword can be detected from the current context (e.g. missing semicolon at the end of a statement);
2. using the wrong token (e.g. `:` instead of `::` in a C++ namespace qualified name);
3. detecting and ignoring whitespace in a whitespace-sensitive qualification (e.g. in XML QNames);
4. keeping in the prolog state (where functions are defined) when there are errors so that functions after the error don't get lost;
5. lexing incomplete literals like `10e` so they can be handled as integers in the parser and emitting an error for them.
It's a dead-end. Sure, it can work in simple cases, but there will be always a case where such syntax recovery isn't possible. That's why relying only on syntax recovery isn't an option.
Because of that I use a different approach. I do parse on each document editing, but such parsing is guaranteed to produce valid results only up to the point with broken syntax, where editing usually takes place. Such parsing is enough to reconstruct location of the point where editing takes place (namespace/class/function) and to reconstruct local context (local variables declared prior to editing place). This allows to perform almost perfect autocompletion by suggesting global and local names available at the editing point. In order to provide proper suggestion of non-local names declared after the editing point, I do keep a structure for the most recent document state with valid syntax.
With features like "go to definition" I do the same. I store a hash-table with location to definition point mapping, but it's updated only from time to time and only if document syntax is valid. In order to be usable for cases with edits made after building such hash-table I just perform text-based position mapping using accumulated edit events.
Using a JSON TCP connection for what on Windows would be direct function calls (COM) or in Eclipse would be direct function calls between Java modules always felt a bit gross.
In order to have direct function calls, you need to load the plugin's code in your own memory. Which means you expose your own memory to the plugin. Any malicious/buggy plugin could wreak havoc on your program - even managed code doesn't solve that problem in general.
IPC is the natural solution to that - run a process in its own address space and communicate with it via I/O. TCP is not most optimal, but it is uniquous. Same thing with JSON.
In LSP, most of the processing happens in the LSP server anyway - communication overhead between client and server is negligible in comparison to it.
TCP isn't really necessary. Usually a language server just uses stdin/stdout, which are just pipes.
And overall overhead for running a language server in a separate process isn't that big. LSP is designed in such a way that only minimal amount of information is needed to be passed, like edits or short responses. Packing/unpacking JSONs isn't a bottleneck, the heaviest job like program analysis is done in the language server itself without interprocess communication overhead involved.
I agree, the only reason LSP exists as it does is for a world running on electron based applications. Plug-ins and application extensions are not new technology and they are nearly universally more efficient in the forms designed and used prior to 2005-ish. I understand why VSCode exists and why it is used so often by developers, but it is certainly a downgrade from more language specific options that could exist.
There are some arguments that do carry water in favor of using a client/server protocol transmitting JSON, in particular, the ability for a nearly complete decoupling of analysis of code and the displaying and editing of that code. Also, LSP was (to my knowledge) the first language/platform/usecase agnostic protocol intended for use in code editors.
I get why this is where a big chunk of developers have ended up, but I do bemoan the lost potential for a language to grab mindshare and popularity on the usability and performance of its tooling and developer experience via-a-vis a custom designed and hyper specific code editor. I mean, Rust and Elm received endless praise for their error messages as a massive boon to developer experience, so it is a facet of language design and implementation that can act as great advertisement. I just hate that the prevalence of LSP at this point precludes custom editors as a first choice in the current zeitgeist.
It only makes sense in the context of host security and stability, as proven by plugin issues in those IDEs.
However, to come back to your point, there are much better high performance OS IPC mechanisms for inter process communication than sending JSON down through a TCP wire.
The particulars of Rust make this a little more difficult, I think. There’s a certain tension between making your language more concise and adding useful redundancies, and Rust has generally gone to the “concise” side, with some redundancies that can make the tooling a little more painful. Like with imports.
impl std::fmt::Display for Blah {
}
If your language makes you qualify your imports (like above) then your LSP can, delightfully, still reliably do certain ops like renaming, even when chunks of your project aren’t parsing. But if you glob import std::fmt, and glob import something else, you are fucked. Display could come from anywhere (maybe from a module that has a parse error at the moment). I really appreciate languages where glob imports (or their equivalent) are either disallowed entirely or where typical code doesn’t use it.
Meanwhile, if you add a new file, there’s this little dance where you say:
mod mycoolmod;
And then you create mycoolmod.rs. Or you do it the other way around. A little redundancy (the file exists and it is declared), that seems to just create a little friction in the LSP because mycoolmod doesn’t get a working LSP until it’s declared in the parent (you have to create both, and then you get a transient diagnostic that your module is unused for a while yet). A small issue, just another little bit of friction in the tooling of Rust that has nothing to do with the type system.
> Stop letting Microsoft design protocols and APIs.
Unfortunately nobody else stepped up to do it.
I'm glad that the code editors out there didn't wait for your theoretical better designed protocol and decided to adopt LSP. Otherwise we'd still have editor that only support one language properly, and the rest is treated like text. If the price to pay is that it sucks for the handful of people who have to work with it, so be it. For every LSP developer that suffers there are tens of thousands of downstream users who benefit from better language support in their favorite editor!
As someone that got introduced to it in 1996, and was a fan of XEmacs variant, still remembers enough keybindings and Elisp, supported beyond plain syntax highlighting was very much hit and miss, even nowadays.
One of the largest issues of LSP is that each language implementation needs to do everything separately, and that means each language will work differently.
Imagine if all the editing tools in microsoft word were specific to the language you used, and if you mixed German and English, each had different tooling.
Now if you mix Rust, HTML, JS, and CSS, they'll all have separate tooling, seperate "go to definition", and separate refactoring. Worst of all, none of them can see the definitions of the other ones.
So what you'd actually want to do is parse each language into your AST, with proper annotations as to what is what, and have all the "go to definition", the UI rendering, highlighting, refactoring, etc all done generically by the IDE ontop.
Which is much much closer to how Jetbrains IntelliJ does it, and why their tooling can handle "find usages" on an HTML element to find matching querySelector in .js files and matching selectors in .css
And if you subscribe to the AI stuff, you'd also want your AI to operate on this AST so it can learn skills that generalize across all languages.
Start with synchronous function calls instead of JSON. Microsoft knows how to do that - they invented COM and OLE. Function calls enable whatever data sharing is necessary to maintain a coherent view. Imagine trying to do OLE with JSON - just wouldn't work. (Does OLE still exist?)
That doesn't really fix anything. The filesystem is fundamentally a racy shared data structure. If you made the entrypoint API synchronous, any half-way decent editor would shove the LSP queries to the synchronous API into a different thread, because "Do Not Block the UI Thread" is a fundamental principle of good UI programming.
Once you get past that, JSON-over-TCP is just another kind of asynchronous RPC mechanism, one that has the advantage that you can build it in just about any language with out-of-the-box tools. Trying to make a plugin system or a COM or CORBA or OLE based system really cuts out the ability to build language servers in most languages, because you have to be able to build the code in just the right way.
I feel like MS actually learned their lesson with synchronously integrating the language intelligence into the IDE. Old versions of VS would hang or crash based on bugs in the language tooling trying to provide intellisense. You'd restart and it'd work fine till you hit some other weird edge case. Generally this settled to a level of rare-bugginess where you were happy enough with the advantages not to go back to Emacs/VIM, but still annoyed at the occasional restart needed.
In no way does this mean LSP is a perfect solution, but anything synchronous would be a step backwards.
Actually, COM is inspired by DCE/RPC and the initial versions had some similarities.
Parallel to that, IBM had SOM on OS/2, which was even better allowing for metaclasses and proper class inheritance, it was the key mechanism between Smalltalk and C++ on OS/2, where Smalltalk enjoyed a role similar to .NET on Windows nowadays.
OLE naturally still exists when using Office natively on Windows, other vendors seem to have forgotten about it.
COM's role on Windows has grown since Vista, and the Windows team redid many of the Longhorn ideas originally implemented in .NET into COM/C++, with WinRT being an evolution of COM.
I've never heard of anyone calling SOM 'better' at anything this century. I came across it when it was the foundation for OpenDoc at Apple, your comment brought back many bad memories
A lot of things described in this article applicable not only for Rust, but for almost any language. Like it's obvious that requests should be handled asynchronously and that conversions from/to UTF-16 are needed. But it's actually not so hard.
I have written a language server for my language too. The hardest thing was to find a way allowing providing useful autocompletion for a document in edited state, when it's not syntactically-correct. This is the trickiest part how to deal with such incorrectness without missing all the context necessary.
I've not written a language server but have written a language plugin for IntelliJ.
I started with writing a correct recursive descent parser. I then extended it to detect, report, and recover from common syntax errors as I encountered them so that the parser is robust. And adding a parser test case for each of these (e.g. one test for each branch through an EBNF construction).
Some examples are:
1. missing keywords when the keyword can be detected from the current context (e.g. missing semicolon at the end of a statement);
2. using the wrong token (e.g. `:` instead of `::` in a C++ namespace qualified name);
3. detecting and ignoring whitespace in a whitespace-sensitive qualification (e.g. in XML QNames);
4. keeping in the prolog state (where functions are defined) when there are errors so that functions after the error don't get lost;
5. lexing incomplete literals like `10e` so they can be handled as integers in the parser and emitting an error for them.
> recover from common syntax errors
It's a dead-end. Sure, it can work in simple cases, but there will be always a case where such syntax recovery isn't possible. That's why relying only on syntax recovery isn't an option.
Because of that I use a different approach. I do parse on each document editing, but such parsing is guaranteed to produce valid results only up to the point with broken syntax, where editing usually takes place. Such parsing is enough to reconstruct location of the point where editing takes place (namespace/class/function) and to reconstruct local context (local variables declared prior to editing place). This allows to perform almost perfect autocompletion by suggesting global and local names available at the editing point. In order to provide proper suggestion of non-local names declared after the editing point, I do keep a structure for the most recent document state with valid syntax.
With features like "go to definition" I do the same. I store a hash-table with location to definition point mapping, but it's updated only from time to time and only if document syntax is valid. In order to be usable for cases with edits made after building such hash-table I just perform text-based position mapping using accumulated edit events.
Using a JSON TCP connection for what on Windows would be direct function calls (COM) or in Eclipse would be direct function calls between Java modules always felt a bit gross.
In order to have direct function calls, you need to load the plugin's code in your own memory. Which means you expose your own memory to the plugin. Any malicious/buggy plugin could wreak havoc on your program - even managed code doesn't solve that problem in general.
IPC is the natural solution to that - run a process in its own address space and communicate with it via I/O. TCP is not most optimal, but it is uniquous. Same thing with JSON.
In LSP, most of the processing happens in the LSP server anyway - communication overhead between client and server is negligible in comparison to it.
TCP isn't really necessary. Usually a language server just uses stdin/stdout, which are just pipes.
And overall overhead for running a language server in a separate process isn't that big. LSP is designed in such a way that only minimal amount of information is needed to be passed, like edits or short responses. Packing/unpacking JSONs isn't a bottleneck, the heaviest job like program analysis is done in the language server itself without interprocess communication overhead involved.
I agree, the only reason LSP exists as it does is for a world running on electron based applications. Plug-ins and application extensions are not new technology and they are nearly universally more efficient in the forms designed and used prior to 2005-ish. I understand why VSCode exists and why it is used so often by developers, but it is certainly a downgrade from more language specific options that could exist.
There are some arguments that do carry water in favor of using a client/server protocol transmitting JSON, in particular, the ability for a nearly complete decoupling of analysis of code and the displaying and editing of that code. Also, LSP was (to my knowledge) the first language/platform/usecase agnostic protocol intended for use in code editors.
I get why this is where a big chunk of developers have ended up, but I do bemoan the lost potential for a language to grab mindshare and popularity on the usability and performance of its tooling and developer experience via-a-vis a custom designed and hyper specific code editor. I mean, Rust and Elm received endless praise for their error messages as a massive boon to developer experience, so it is a facet of language design and implementation that can act as great advertisement. I just hate that the prevalence of LSP at this point precludes custom editors as a first choice in the current zeitgeist.
LSP helps with not having to develop the same tooling in each editor for each language.
This isn’t for Electron app only, but is also helpful for Emacs, vim or helix (especially when they lack said language plugins).
Now, some IDEs provide capabilities for a language far exceeding what can even be implemented in a LSP.
It only makes sense in the context of host security and stability, as proven by plugin issues in those IDEs.
However, to come back to your point, there are much better high performance OS IPC mechanisms for inter process communication than sending JSON down through a TCP wire.
As others point out, Electron.
> Why building a Rust LSP is hard
Not as hard as understanding what LSP means, apparently.
I was interested until I saw this project is simply an LSP server.
The particulars of Rust make this a little more difficult, I think. There’s a certain tension between making your language more concise and adding useful redundancies, and Rust has generally gone to the “concise” side, with some redundancies that can make the tooling a little more painful. Like with imports.
If your language makes you qualify your imports (like above) then your LSP can, delightfully, still reliably do certain ops like renaming, even when chunks of your project aren’t parsing. But if you glob import std::fmt, and glob import something else, you are fucked. Display could come from anywhere (maybe from a module that has a parse error at the moment). I really appreciate languages where glob imports (or their equivalent) are either disallowed entirely or where typical code doesn’t use it.Meanwhile, if you add a new file, there’s this little dance where you say:
And then you create mycoolmod.rs. Or you do it the other way around. A little redundancy (the file exists and it is declared), that seems to just create a little friction in the LSP because mycoolmod doesn’t get a working LSP until it’s declared in the parent (you have to create both, and then you get a transient diagnostic that your module is unused for a while yet). A small issue, just another little bit of friction in the tooling of Rust that has nothing to do with the type system.Enter hell: LSP assumes that it's the source of truth, but you still need to access the filesystem yourself, and do it in a synchronized way
LSP is an example of utterly horrid technical design.
Stop letting Microsoft design protocols and APIs. They are so. bad. at. it.
> Stop letting Microsoft design protocols and APIs.
Unfortunately nobody else stepped up to do it.
I'm glad that the code editors out there didn't wait for your theoretical better designed protocol and decided to adopt LSP. Otherwise we'd still have editor that only support one language properly, and the rest is treated like text. If the price to pay is that it sucks for the handful of people who have to work with it, so be it. For every LSP developer that suffers there are tens of thousands of downstream users who benefit from better language support in their favorite editor!
we'd still have editor that only support one language properly
Emacs supported zillions of languages before LSP.
Zoom out.
If the old emacs approach is so much better, why is emacs switching to using LSP?
As someone that got introduced to it in 1996, and was a fan of XEmacs variant, still remembers enough keybindings and Elisp, supported beyond plain syntax highlighting was very much hit and miss, even nowadays.
I have been an emacs user for over a decade now, and if you think that language “support” was as good before LSP, I have a bridge to sell you.
LSP is not that bad. They tried to decouple IDE features from language features.
Resolving file paths is not trivial because it's language specific.
Syncing source changes on the LSP server side is necessary to keep packet side small and patching is trivial (span, new text).
(I made an LSP-powered text editor and implemented LSP client from scratch)
I've never looked into LSP, how would you design it?
Well for one I wouldn't design it so both the LSP and the editor need a synchronized view of the underlying file.
One of the largest issues of LSP is that each language implementation needs to do everything separately, and that means each language will work differently.
Imagine if all the editing tools in microsoft word were specific to the language you used, and if you mixed German and English, each had different tooling.
Now if you mix Rust, HTML, JS, and CSS, they'll all have separate tooling, seperate "go to definition", and separate refactoring. Worst of all, none of them can see the definitions of the other ones.
So what you'd actually want to do is parse each language into your AST, with proper annotations as to what is what, and have all the "go to definition", the UI rendering, highlighting, refactoring, etc all done generically by the IDE ontop.
Which is much much closer to how Jetbrains IntelliJ does it, and why their tooling can handle "find usages" on an HTML element to find matching querySelector in .js files and matching selectors in .css
And if you subscribe to the AI stuff, you'd also want your AI to operate on this AST so it can learn skills that generalize across all languages.
Wouldn't a normalized protocol based on AST vocabulary and tree operations make more sense ?
Start with synchronous function calls instead of JSON. Microsoft knows how to do that - they invented COM and OLE. Function calls enable whatever data sharing is necessary to maintain a coherent view. Imagine trying to do OLE with JSON - just wouldn't work. (Does OLE still exist?)
That doesn't really fix anything. The filesystem is fundamentally a racy shared data structure. If you made the entrypoint API synchronous, any half-way decent editor would shove the LSP queries to the synchronous API into a different thread, because "Do Not Block the UI Thread" is a fundamental principle of good UI programming.
Once you get past that, JSON-over-TCP is just another kind of asynchronous RPC mechanism, one that has the advantage that you can build it in just about any language with out-of-the-box tools. Trying to make a plugin system or a COM or CORBA or OLE based system really cuts out the ability to build language servers in most languages, because you have to be able to build the code in just the right way.
I feel like MS actually learned their lesson with synchronously integrating the language intelligence into the IDE. Old versions of VS would hang or crash based on bugs in the language tooling trying to provide intellisense. You'd restart and it'd work fine till you hit some other weird edge case. Generally this settled to a level of rare-bugginess where you were happy enough with the advantages not to go back to Emacs/VIM, but still annoyed at the occasional restart needed.
In no way does this mean LSP is a perfect solution, but anything synchronous would be a step backwards.
Actually, COM is inspired by DCE/RPC and the initial versions had some similarities.
Parallel to that, IBM had SOM on OS/2, which was even better allowing for metaclasses and proper class inheritance, it was the key mechanism between Smalltalk and C++ on OS/2, where Smalltalk enjoyed a role similar to .NET on Windows nowadays.
OLE naturally still exists when using Office natively on Windows, other vendors seem to have forgotten about it.
COM's role on Windows has grown since Vista, and the Windows team redid many of the Longhorn ideas originally implemented in .NET into COM/C++, with WinRT being an evolution of COM.
I've never heard of anyone calling SOM 'better' at anything this century. I came across it when it was the foundation for OpenDoc at Apple, your comment brought back many bad memories
Compared with COM's design, it was much better.
> Start with synchronous function calls instead of JSON.
LSP is a function call protocol. So that is already done?
Moreover, it's not just stupid, it also does not actually solve _anything_.
A synchronous function can also have obsolete indexing information if it races with the code updates.
Stupid idea.
So the LSP crashes and/or goes into a runaway memory consumption loop. And your main application dies with it.
Or what if you want, you know, to be able to use the same LSP from TWO different applications at the same time?
Never mind issues with other managed runtimes not expecting to deal with something else in their address space.