Language-tools: Heavy memory usage of the Language Server

The language server use typescript's language service behind the scenes. I'll guess the high memory is because typescript tries to parse all files configured, in the jsconfig.json, to be included. Because the typescript language service is in the same process as the svelte language server, I don't know how much memory is used by typescript language service.

I am curious about how big is your __sapper__ folder. My project has a jsconfig.json and set to include about 200 js source files and only use 150MB. Can't imagine how does your __sapper__ folder cause the language server to use more than 2GB of memory.

For reference, here's the the repo of my project: b339c2a17e @ Innopoints/frontend

I've freshly cloned it and ran the server and got the following:

• 490 MB when the __sapper__ folder contains the dev build
• 730 MB when the __sapper__ folder contains the dev and production builds

You can try to compile it yourselves to try to reproduce this (note that the dev server won't run without environment variables, but that's okay, since the compilation will succeed anyway). Install deps with Yarn, compile with yarn dev and then open any .svelte file in VS Code.

Also, if it makes any difference whatsoever, I'm using VSCodium, as opposed to regular VS Code

Is there any way to point SLS to only scan .svelte files? I don't think .js/.ts files are really relevant to Svelte code analysis. Especially the ones that are not directly imported, such as the __sapper__ folder.

If not, I'm curious as to how SLS behaves in absence of jsconfig.json. As I mentioned, removing that file returns the memory consumption to adequate 150 MB. And it's definitely worth mentioning in the README, because to me, the person that's not intimately familiar with the TS ways, it was a very big surprise that a seemingly completely unrelated jsconfig.json file is causing such a dramatic difference.

Yeah we should add something to the documentation about that.

The scan of .ts / .js files is needed to provide you with intellisense from svelte to these files. You would not get autocomplete/go to defintion/hover info of these, if they were not included.

It just kinda feels like the __sapper__ folder isn't of much use for IntelliSense, so perhaps this ignore can even be set by default and overriden if necessary

I still get memory usages of up to 2 GB, but it's very hard to trace. Feels like it just blows up at random times while I am editing Svelte source in that project. What could you recommend to cut memory consumption?

The language server detects svelte files and will walk the file tree from that. It will also honor stuff in the jsconfig/tsconfig. From your experience ("blows up", not "steadily increases"), my guess right now is that somehow the server gets to the point where he loads a bunch of unrelated files which he shoulnd't.
You could look into VSCode's Output->Svelte and see if there is anything suspicious (or just copypaste it here)

It's behaving for me today, but yesterday I had to manually kill off svelte language tools processes after closing VS Code (wasn't responding to SIGTERM).

I don't have much more to add at this time.

How big is your project? Do you have a tsconfig.json or jsconfig.json?

@dummdidumm Not much larger than sapper template. I have a basic tsconfig.json which has not changed since I encountered the problem.

{
  "include": [
    "src/**/*"
  ],
  "exclude": [
    "node_modules/*"
  ],
  "compilerOptions": {
    "target": "es2015",
    "module": "es2015",
    "types": [
      "svelte"
    ]
  }
}

So both you and @illright use sapper and have heave memory usage. Maybe a pattern.. will have to investigate this.

@dummdidumm Let me know if I can aid with investigation in any way. Perhaps there is a way to run the SLS through the command line or anything else more reproducible than just fiddling around in the editor?

Svelte (Svelte Language Server) stderr FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
Svelte (Svelte Language Server) stderr  1: 0x55d7276c33b6 node::Abort() [/usr/lib/electron5/electron]
Svelte (Svelte Language Server) stderr  2: 0x55d7276c3985  [/usr/lib/electron5/electron]
Svelte (Svelte Language Server) stderr  3: 0x55d723b01817  [/usr/lib/electron5/electron]
Svelte (Svelte Language Server) stderr  4: 0x55d723b017b4  [/usr/lib/electron5/electron]
Svelte (Svelte Language Server) stderr  5: 0x55d723b6f716  [/usr/lib/electron5/electron]
Svelte (Svelte Language Server) stderr  6: 0x55d723b6e538  [/usr/lib/electron5/electron]
Svelte (Svelte Language Server) stderr  7: 0x55d723b6b626 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/lib/electron5/electron]
Svelte (Svelte Language Server) stderr  8: 0x55d723b7678e  [/usr/lib/electron5/electron]
Svelte (Svelte Language Server) stderr  9: 0x55d723f210b7 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/usr/lib/electron5/electron]
Svelte (Svelte Language Server) stderr 10: 0x55d72410e1be  [/usr/lib/electron5/electron]
Svelte (Svelte Language Server) stderr 11: 0x55d72440b62b  [/usr/lib/electron5/electron]
Svelte (Svelte Language Server) rpc.onClose The RPC connection closed unexpectedly

Just now I again hit 2.1GB and SIGKILLed the SLS. The above could be seen in DevTools of Atom

Oh wait, you are using Atom? Which extension are you using?

@dummdidumm I have both Atom (with ide-svelte, manually bumping the language server version) and VSCodium (with Svelte Beta).

Here's the output of VSCodium's Svelte tab when the blowup happened. It seems to have killed it off and recovered, but the freeze was still there.

Ok I understand, thanks.

Seems like the blowup happened very soon after startup.

If you would investigate a little yourself with the extension locally, that would be great! To setup the extension locally for VSCode(ium, should be the same hopefully), first uninstall it and then set it up locally like explained here. What you wanna focus on is the service. My guess is that at some point it gets too many files which it should deal with, which can be seen here. If you add a console log above (line 72) which logs the files (maybe in an interval of 10 seconds, like setInterval(() => console.log(JSON.stringify(Array.from(new Set([...files, ...snapshotManager.getFileNames(), ...svelteTsxFiles]), null , 3)), 10000)) that could help getting some insights.

So it seems like snapshotManager.getFileNames() contains a bunch of files that are not supposed to be watched according to jsconfig.json. It also responds to file change, in that it doesn't load anything from Sapper up until a build happens, but from then on, the __sapper__/**/*.js files fill up the memory

Ok, I guess that's the source of the problem. It explains the sudden blowup because all the new files got added to the tracking list.

Yeah, that probably is it. I think we need to adjust how TypescriptPlugin handles the ts/js-onWatchedFilesChange. Maybe do it like vetur does is and only delete old stuff, not add new. Or add some best guess paths like __sapper__ / node_modules / dist that should be ignored.

I'll try to do this.

A terrible workaround for until this issue is fixed: locate your Svelte extension directory, open node_modules/svelte-language-server/dist/src/plugins/typescript/service.js and comment out the snapshotManager.getFileNames():

Line 69:

// before:
return Array.from(new Set(__spreadArrays(files, snapshotManager.getFileNames(), svelteTsxFiles)));

// after:
return Array.from(new Set(__spreadArrays(files/*, snapshotManager.getFileNames() */, svelteTsxFiles)));

You lose some IntelliSense, but retain syntax highlighting and at least the performance doesn't drop randomly. And if you're anything like me, that should be good enough to get by :)

Supplementing what @illright said. it is usually located in ~/.vscode/extensions or %userprofile%\.vscode \extensions\ in windows.

Another workaround is to go to node_modules/svelte-language-server/dist/src/plugins/typescript/TypeScriptPlugin.js
Add the following lines to Line 237 where onWatchFileChanges start

        if (/node_modules|__sapper__|dist/.test(fileName)) {
            return;
        }

this method is the source of the problem

I created a PR #165. Could you guys try it in debug and see if it improved?

@jasonlyu123 Yes, I notice improvement. For a while of fiddling around with my project, the memory consumption was kept under control. Sapper rebuilds also seem to not be triggering memory overflows.

Note: kept under control = ~480 MB at peak. This is low enough not to affect my system, but you might still deem this high memory consumption. I've got 8 GB of RAM on my machine.

The pain is real 😭

Even if we default to exclude__sapper__, I'll still recommend put __sapper__ in the exclude of your tsconfig.json/jsconfig.json. Because we are using our own bundle of typescript. the tsserver used by vscode might still include it.

@illright could you check with the latest plugin version? Should be better now.

VS Code seems smooth as usual, about 400 MB of memory used. Any chance the update to SLS can be pushed to the Atom extension? That's the place where I've been noticing performance drops the other day

@orta is in the middle of transferring the Atom plugin into this repo. After this is done, you should get the updates of the language server.

@rob-balfre did the memory usage go down for you aswell? If not, could you specify your setup?

I was experiencing the same problems, adding transpileOnly to the typescript preprocessor seems to greatly improve the situation.
In your svelte.config.js
```
const sveltePreprocess = require("svelte-preprocess");

module.exports = {
preprocess: sveltePreprocess({
typescript: {
transpileOnly: true,
},
}),
// ...other svelte options (optional)
};

I noticed this as well soon after updating VSCode recently. Small projects don't seem to be as big an issue (though saving is still taking a little longer than usual). Saving will hang, then the file will save and format. After that it seems fine. And this is for small projects only. It's like it is trying to index the project folder or something and getting hung up? Bigger projects with more files, etc wind up hanging and erroring out.

The memory usage/error for me happens when saving a Svelte file. It just hangs and hangs with this message in the bottom right:

Here's the Svelte Output window:

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x1143fdbe5 node::Abort() [/Applications/Visual Studio Code.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Electron Framework]
 2: 0x1143fdc54 node::Abort() [/Applications/Visual Studio Code.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Electron Framework]
 3: 0x11010b237 v8::internal::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*) [/Applications/Visual Studio Code.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Electron Framework]
 4: 0x11010b1d7 v8::internal::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*) [/Applications/Visual Studio Code.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Electron Framework]
 5: 0x1101500a5 v8::internal::Heap::StartIdleIncrementalMarking(v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Applications/Visual Studio Code.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Electron Framework]
 6: 0x110151719 v8::internal::Heap::StartIdleIncrementalMarking(v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Applications/Visual Studio Code.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Electron Framework]
 7: 0x11014e3ec v8::internal::Heap::CreateFillerObjectAt(unsigned long, int, v8::internal::ClearRecordedSlots, v8::internal::ClearFreedMemoryMode) [/Applications/Visual Studio Code.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Electron Framework]
 8: 0x11014c002 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Applications/Visual Studio Code.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Electron Framework]
 9: 0x11015746a v8::internal::Heap::PromotedExternalMemorySize() [/Applications/Visual Studio Code.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Electron Framework]
10: 0x110157851 v8::internal::Heap::PromotedExternalMemorySize() [/Applications/Visual Studio Code.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Electron Framework]
11: 0x110358a5a v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/Applications/Visual Studio Code.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Electron Framework]
12: 0x1105e19bf v8::internal::RegExp::CompileForTesting(v8::internal::Isolate*, v8::internal::Zone*, v8::internal::RegExpCompileData*, v8::base::Flags<v8::internal::JSRegExp::Flag, int>, v8::internal::Handle<v8::internal::String>, v8::internal::Handle<v8::internal::String>, bool) [/Applications/Visual Studio Code.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Electron Framework]
13: 0x110c23139 v8::internal::compiler::ZoneStats::ReturnZone(v8::internal::Zone*) [/Applications/Visual Studio Code.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Electron Framework]
14: 0x110bf293d v8::internal::compiler::ZoneStats::ReturnZone(v8::internal::Zone*) [/Applications/Visual Studio Code.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Electron Framework]

<--- Last few GCs --->
al[45813:0x7fe18e004200]    47201 ms: Mark-sweep 4095.0 (4102.8) -> 4094.4 (4103.3) MB, 2004.1 / 0.0 ms  (+ 6.4 ms in 18 steps since start of marking, biggest step 5.3 ms, walltime since start of marking 2020 ms) (average mu = 0.146, current mu = 0.005) all[45813:0x7fe18e004200]    50534 ms: Mark-sweep 4095.7 (4103.3) -> 4095.3 (4104.5) MB, 2648.3 / 0.0 ms  (+ 668.5 ms in 21 steps since start of marking, biggest step 50.7 ms, walltime since start of marking 3332 ms) (average mu = 0.063, current mu = 0.005) 

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x110c23139]
Security context: 0x287bc74e0dd1 <JSObject>
    1: keys [0x287bc74c15b1](this=0x287bcb9bfaa1 <Object map = 0x287be02c4cb9>,0x287bc68e8e29 <Object map = 0x287bcfd3aa99>)
    2: uvException(aka uvException) [0x287b27f974e1] [internal/errors.js:374] [bytecode=0x287b18cb32e9 offset=424](this=0x287bb6f004b1 <undefined>,0x287bc68e8e29 <Object map = 0x287bcfd3aa99>)
    3: handleErrorFromBinding(aka handleError...

[Info  - 9:59:17 PM] Connection to server got closed. Server will restart.
[Error - 9:59:17 PM] Request textDocument/formatting failed.
Error: Connection got disposed.
    at Object.dispose (/Users/babycourageous/.vscode/extensions/svelte.svelte-vscode-99.0.44/node_modules/vscode-jsonrpc/lib/main.js:904:25)
    at Object.dispose (/Users/babycourageous/.vscode/extensions/svelte.svelte-vscode-99.0.44/node_modules/vscode-languageclient/lib/client.js:74:35)
    at LanguageClient.handleConnectionClosed (/Users/babycourageous/.vscode/extensions/svelte.svelte-vscode-99.0.44/node_modules/vscode-languageclient/lib/client.js:2309:42)
    at LanguageClient.handleConnectionClosed (/Users/babycourageous/.vscode/extensions/svelte.svelte-vscode-99.0.44/node_modules/vscode-languageclient/lib/main.js:155:15)
    at closeHandler (/Users/babycourageous/.vscode/extensions/svelte.svelte-vscode-99.0.44/node_modules/vscode-languageclient/lib/client.js:2296:18)
    at CallbackList.invoke (/Users/babycourageous/.vscode/extensions/svelte.svelte-vscode-99.0.44/node_modules/vscode-jsonrpc/lib/events.js:62:39)
    at Emitter.fire (/Users/babycourageous/.vscode/extensions/svelte.svelte-vscode-99.0.44/node_modules/vscode-jsonrpc/lib/events.js:121:36)
    at closeHandler (/Users/babycourageous/.vscode/extensions/svelte.svelte-vscode-99.0.44/node_modules/vscode-jsonrpc/lib/main.js:240:26)
    at CallbackList.invoke (/Users/babycourageous/.vscode/extensions/svelte.svelte-vscode-99.0.44/node_modules/vscode-jsonrpc/lib/events.js:62:39)
    at Emitter.fire (/Users/babycourageous/.vscode/extensions/svelte.svelte-vscode-99.0.44/node_modules/vscode-jsonrpc/lib/events.js:121:36)
    at IPCMessageReader.fireClose (/Users/babycourageous/.vscode/extensions/svelte.svelte-vscode-99.0.44/node_modules/vscode-jsonrpc/lib/messageReader.js:111:27)
    at ChildProcess.<anonymous> (/Users/babycourageous/.vscode/extensions/svelte.svelte-vscode-99.0.44/node_modules/vscode-jsonrpc/lib/messageReader.js:213:45)
    at ChildProcess.emit (events.js:208:15)
    at ChildProcess.EventEmitter.emit (domain.js:476:20)
    at maybeClose (internal/child_process.js:1021:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:283:5)
[Error - 9:59:17 PM] Request textDocument/hover failed.
Error: Connection got disposed.
    at Object.dispose (/Users/babycourageous/.vscode/extensions/svelte.svelte-vscode-99.0.44/node_modules/vscode-jsonrpc/lib/main.js:904:25)
    at Object.dispose (/Users/babycourageous/.vscode/extensions/svelte.svelte-vscode-99.0.44/node_modules/vscode-languageclient/lib/client.js:74:35)
    at LanguageClient.handleConnectionClosed (/Users/babycourageous/.vscode/extensions/svelte.svelte-vscode-99.0.44/node_modules/vscode-languageclient/lib/client.js:2309:42)
    at LanguageClient.handleConnectionClosed (/Users/babycourageous/.vscode/extensions/svelte.svelte-vscode-99.0.44/node_modules/vscode-languageclient/lib/main.js:155:15)
    at closeHandler (/Users/babycourageous/.vscode/extensions/svelte.svelte-vscode-99.0.44/node_modules/vscode-languageclient/lib/client.js:2296:18)
    at CallbackList.invoke (/Users/babycourageous/.vscode/extensions/svelte.svelte-vscode-99.0.44/node_modules/vscode-jsonrpc/lib/events.js:62:39)
    at Emitter.fire (/Users/babycourageous/.vscode/extensions/svelte.svelte-vscode-99.0.44/node_modules/vscode-jsonrpc/lib/events.js:121:36)
    at closeHandler (/Users/babycourageous/.vscode/extensions/svelte.svelte-vscode-99.0.44/node_modules/vscode-jsonrpc/lib/main.js:240:26)
    at CallbackList.invoke (/Users/babycourageous/.vscode/extensions/svelte.svelte-vscode-99.0.44/node_modules/vscode-jsonrpc/lib/events.js:62:39)
    at Emitter.fire (/Users/babycourageous/.vscode/extensions/svelte.svelte-vscode-99.0.44/node_modules/vscode-jsonrpc/lib/events.js:121:36)
    at IPCMessageReader.fireClose (/Users/babycourageous/.vscode/extensions/svelte.svelte-vscode-99.0.44/node_modules/vscode-jsonrpc/lib/messageReader.js:111:27)
    at ChildProcess.<anonymous> (/Users/babycourageous/.vscode/extensions/svelte.svelte-vscode-99.0.44/node_modules/vscode-jsonrpc/lib/messageReader.js:213:45)
    at ChildProcess.emit (events.js:208:15)
    at ChildProcess.EventEmitter.emit (domain.js:476:20)
    at maybeClose (internal/child_process.js:1021:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:283:5)
Initialize language server at  Code/Project
Trying to load config for Code/Project/src/App.svelte
Initialize new ts service at  

This also happens in this larger Svelte project saving other files (.js, etc). If I disable Svelte Beta, everything is fine (except now Svelte files aren't recognized.

I didn't want to start a new issue since it seems related to this but absolutely can if this is unrelated.

Thanks!

Thanks for the information. I have some more questions to narrow this down:

• How many svelte/js files are we talking about?
• Does it happen right away after a few seconds/minutes working with the project or only after some time?
• If you set svelte.plugin.svelte.format.enable to false (https://github.com/sveltejs/language-tools/tree/master/packages/svelte-vscode#sveltepluginsvelteformatenable), does the error still occur? Or is the output littered with memory errors regardless?
• Do you have a tsconfig.json or jsconfig.json? If you don't, what happens if you add one (can be just a simple one)?

Hi @dummdidumm Thanks for the quick reply!

First - as of this morning I opened some stuff to run tests for your questions... and it seems to have vanished into thin air. Classic.

Here are some answers for ya for posterity:

How many svelte/js files are we talking about?

• Small projects (<10 files): The little "Running Svelte Beta" window comes up for <10 seconds, file saves and formats
• Larger projects (>10 files): This is where I was hitting that problem. That window pops up in lower left and tries and tries for about 20-30 seconds. Then kaput.

Does it happen right away after a few seconds/minutes working with the project or only after some time?
So far it's immediately. Open a project, alter a file, save and above situations occur. And I only noticed it after the VSCode update yesterday (or day before). Though Svelte Beta was slower to save in general than the previous extension (about 2-4 seconds before the format and save happen), if that's of any use.

If you set svelte.plugin.svelte.format.enable to false (https://github.com/sveltejs/language-tools/tree/master/packages/svelte-vscode#sveltepluginsvelteformatenable), does the error still occur? Or is the output littered with memory errors regardless?
I can try this next time I notice this happening. Unfortunately since it randomly stopped just now, I don't think it will do any good, heheh.

Do you have a tsconfig.json or jsconfig.json? If you don't, what happens if you add one (can be just a simple one)?
No .tsconfig or .jsconfig in any of my projects. If you have any simple setups to share I could try that as well next time.

So, all in all hooray that it vanished, but damn if we don't know the issue now, hehe.

Classic 😄

I asked the jsconfig.json / tsconfig.json question because we had a similar error previously where the language service would load way too many files at startup because it was finding another config further up the file tree (outside of the project folder) and using that - although that should be fixed by now.

I will add some logging to get the loaded number of files on startup as well as every minute afterwards.

Sounds good - I'll let ya know if anything changes! I'm playing around in one of those >10 files projects now and everything is still smooth sailing...

We added additional measures to prevent initial file bloat. If anyone still experiences high memory usage, please report in here. Else I will close this in a few weeks.

@dummdidumm any updates on the Atom extension?

See #70 for info and progress on this.

Excluding __sapper__ from VSCode watcher stopped the extension to go out of memory for me.