Reproducible nvim completion framework benchmarks.

Overview

Nvim.Bench

Reproducible nvim completion framework benchmarks.

Runs inside Docker. Fair and balanced


Methodology

Note: for all "randomness", they are generated from the same seed for each run, and therefore "fair".

Input

tmux is used to send keys to simulate ideal human typing.

The words typed are naive tokens from parsing current document into (alphanum + "_") delimited by whitespaces and symbols.

This tokenization should work fairly well for c family of languages, which are the industry standard.

A uniform distribution of whitespaces is also generated from the same buffer.

Measurement

n keystrokes of --samples is performed.

Speed

Using --avg-word-len, --wpm and --variance, a Normal Distribution is constructed of the desired delay between keystrokes.

Data

See ./fs/data/

Modularity

Some frameworks will have by default, very little sources enabled, if any.

Other ones will come with more out of the box.

For a fair comparison: All frameworks tested will have to following enabled, on top of whatever else they come enabled by default:

  • buffer

  • lsp

  • path

The reasoning is that: 1) Almost all authors will have written these sources firsthand, and 2) they seem to be the most useful sources.

No default sources will be disabled, because users don't tend to do that.


Cool, pictures

The plots are kernel density estimations, have no idea why they fitted more than 1 curve for some plots.

I usually use R, not used to python ploting. Anyways, they are an estimate of the true probability density function.

Q0, 50, 95, 100?

Mean min, median, 1 in 20, max, respectively.

Without assuming any statistical distribution:

Q50 is a more robust measure than avg, and Q95 is a decent measure of a common bad value.


Analysis

Please keep in mind that this is purely a synthetic benchmark, which definitely is one of those need context to interpret type of things.

There is no good way to measure real speed across frameworks, raw numbers here come with big caveats.

Study design limitations

Streaming completion

Streaming completion is very good for time to first result (TTFR), but it presents us with an issue:

While the fast sources will return right away, the slower ones might never make it before the next keystroke.

This has the funny effect of removing the influence of slower sources entirely, which is disastrous for study integrity.

The mitigation is actually to set typing speed unrealistically slow, enough so that we have confidence that the LSP servers can catch up.

This is obviously not ideal.

Fast on paper != fast IRL

The most responsive frameworks are not necessarily the fastest ones, because humans still have to choose the results.

For example the streaming completion approach actually has severe trade offs infavor of faster TTFR:

Ranking

Having suboptimal ranking is BAD, it pushes work from fast machines onto slow humans.

The streaming approach has to be additive, because its too disruptive to shift existing menu items around.

Therefore it is limited to sorting only within stream batches, and to make things worse, slower batches typically contain higher quality results.

That means better results will often end up at the bottom, necessitating more work for humans.

Limiting

This is a direct consequence of limited ranking optimizations.

Because the framework have no idea how much each source will send, it has the dilemma of either sending too many results or too little.

Sending too many results in early batches from likely inferior sources will waste the users time, and sending too little will obscure potentially useful completions.

Clarity on when / if results will come in

This is a HCI thing:

Having higher quality results come in slower is likely to inadvertently train users to wait for them. This is evidently bad for input speed.

Conclusion

There is never going to be a closed form solution to "what is the fastest framework", because of the trade offs detailed above.

A toy example of a degenerate framework that returns a single fixed 👌 emoji will probably beat anything out there in terms of raw speed, but it is utterly useless.

Before you reach your own conclusion, the results of this repo must be considered alongside inextricably human measure.

Owner
i love my dog
dogs are love dogs are life
i love my dog
a pull switch (or BYO button) that gets you out of video calls, quick

zoomout a pull switch (or BYO button) that gets you out of video calls, quick. As seen on Twitter System compatibility Tested on macOS Catalina (10.15

Brian Moore 422 Dec 30, 2022
⚙️ Compile, Read and update your .conf file in python

⚙️ Compile, Read and update your .conf file in python

Reece Harris 2 Aug 15, 2022
World's best free and open source ERP.

World's best free and open source ERP.

Frappe 12.5k Jan 07, 2023
Hopefully the the next-generation backend server of bgm.tv

Hopefully the the next-generation backend server of bgm.tv

Bangumi 475 Jan 01, 2023
Absolute solvation free energy calculations with OpenFF and OpenMM

ABsolute SOLVantion Free Energy Calculations The absolv framework aims to offer a simple API for computing the change in free energy when transferring

7 Dec 07, 2022
User management system (UMS), has the primary purpose of connecting to an Active Directory (AD)

💿 Sistema de Gerenciamento de Usuário (SGU) 📚 Sobre o projeto Sistema de gerenciamento de usuários (SGU), tem o objetivo primário de se conectar a u

Patrick Viegas 2 Feb 25, 2022
Medical appointments No-Show classifier

Medical Appointments No-shows Why do 20% of patients miss their scheduled appointments? A person makes a doctor appointment, receives all the instruct

4 Apr 20, 2022
It's a repo for Cramer's rule, which is some math crap or something idk

It's a repo for Cramer's rule, which is some math crap or something idk (just a joke, it's not crap; don't take that seriously, math teachers)

Module64 0 Aug 31, 2022
firefox session recovery

firefox session recovery

Ahmad Sadraei 5 Nov 29, 2022
Application to list countries in order of travel from the United States.

Application to list countries in order of travel from the United States.

Broden Wanner 1 Nov 03, 2021
Fisherman is a free open source fishing bot written in python.

Fisherman is a free open source fishing bot written in python.

Pure | Cody 33 Jan 29, 2022
An Airflow operator to call the main function from the dbt-core Python package

airflow-dbt-python An Airflow operator to call the main function from the dbt-core Python package Motivation Airflow running in a managed environment

Tomás Farías Santana 93 Jan 08, 2023
OLDBot (Online Lessons Discord Bot)

This program is designed to facilitate online lessons. With this you don't need to get up early. Just config and watch the program resolve itself. It automatically enters to the lesson at the specifi

Da4ndo 1 Nov 21, 2021
An AddOn storing wireguard configuration

Wireguard Database Connector Overview Development Status: 0.1.7 (alpha) First of all, I'd like to thank Jared McKnight for wireguard who inspired me t

Markus Neubauer 3 Dec 30, 2021
flake8 plugin which forbids match statements (PEP 634)

flake8-match flake8 plugin which forbids match statements (PEP 634)

Anthony Sottile 25 Nov 01, 2022
万能通用对象池,可以池化任意自定义类型的对象。

pip install universal_object_pool 此包能够将一切任意类型的python对象池化,是万能池,适用范围远大于单一用途的mysql连接池 http连接池等。 框架使用对象池包,自带实现了4个对象池。可以直接开箱用这四个对象池,也可以作为例子学习对象池用法。

12 Dec 15, 2022
synchronize projects via yaml/json manifest. built on libvcs

vcspull - synchronize your repos. built on libvcs Manage your commonly used repos from YAML / JSON manifest(s). Compare to myrepos. Great if you use t

python utilities for version control 200 Dec 20, 2022
IOP Support for Python (Experimental)

TAGS Experimental IOP Framework for Python WARNING: Currently, this project has NO EXCEPTION HANDLING. USE AT YOUR OWN RISK! I. Introduction to Interf

1 Oct 22, 2021
Build Xmas cards with user inputs

Automatically build Xmas cards with user inputs

Anand 9 Jan 25, 2022
In this project, we'll be creating a virtual personal assistant for ourselves using our favorite programming language

In this project, we'll be creating a virtual personal assistant for ourselves using our favorite programming language, Python. We can perform several offline as well as online operations using the bo

Ashutosh Krishna 188 Jan 03, 2023