VSCode and system congestion
Over the past few months we've seen a large surge in the number of users using VSCode (via the Remote SSH extension). The extension causes a persistent service, called the VSCode Server, to be launched on the remote system, in order to fulfill and extend VSCode's features to the remote system. Orindarily, this is fine, but since the start of the year, we've see a growing number of incidents affecting the reliability and performance of the login nodes.
What's the problem?
Wether intentionally or not, many users' VSCode sessions launch on the login nodes
(i.e. login001 and login002) and are left to idle long after the user has
"disconnected" from the cluster. This can cause several problems which eventually
lead to performance degradation incident on the login node(s), marked by slow responsiveness
and laggy behaviour:
- linger processes (at least 20 per user session) are left behind, consuming resources
- the Lustre filesystem struggles to handle the increasing file system operations (e.g.
statfs,getattr, etc.) - other processes stall waiting for the filesystem to respond; either they continue to hang or eventually fail
- all interactive sessions (i.e. other logged in user sessions) become laggy and eventually totally unresponsive
Note that these issues happen even when the system is not under load!
The causes for these issues from the VSCode Server service are:
- timer based behaviours, like file indexing and connection checks, keep triggering regardless of user activity
- the process often maintains state within memory, meaning resources are consumed regardless of user actions (e.g. doing something interactive or logged out)
How do we prevent this situation from happening?
The VSCode developers have invested hardly any effort in providing mechanisms that SysAdmins can use to regulate VSCode Server on the remote machines. Intervention by our users is needed.
Within out documentation for VSCode we provide a guide to use SLURM interactive sessions, using compute nodes to host the VSCode Server service, and also Tips on improving the "performance" of VSCode by limiting filesystem operations. We ask all users to read through the documentation and adapt their workflows to help improve the usability of the cluster.
If these system congestion issues keep happening, we might be forced to take more drastic actions, including fully banning the use of VSCode Server on Merlin7.