Reproducible nvim completion framework benchmarks.

Overview

Nvim.Bench

Reproducible nvim completion framework benchmarks.

Runs inside Docker. Fair and balanced


Methodology

Note: for all "randomness", they are generated from the same seed for each run, and therefore "fair".

Input

tmux is used to send keys to simulate ideal human typing.

The words typed are naive tokens from parsing current document into (alphanum + "_") delimited by whitespaces and symbols.

This tokenization should work fairly well for c family of languages, which are the industry standard.

A uniform distribution of whitespaces is also generated from the same buffer.

Measurement

n keystrokes of --samples is performed.

Speed

Using --avg-word-len, --wpm and --variance, a Normal Distribution is constructed of the desired delay between keystrokes.

Data

See ./fs/data/

Modularity

Some frameworks will have by default, very little sources enabled, if any.

Other ones will come with more out of the box.

For a fair comparison: All frameworks tested will have to following enabled, on top of whatever else they come enabled by default:

  • buffer

  • lsp

  • path

The reasoning is that: 1) Almost all authors will have written these sources firsthand, and 2) they seem to be the most useful sources.

No default sources will be disabled, because users don't tend to do that.


Cool, pictures

The plots are kernel density estimations, have no idea why they fitted more than 1 curve for some plots.

I usually use R, not used to python ploting. Anyways, they are an estimate of the true probability density function.

Q0, 50, 95, 100?

Mean min, median, 1 in 20, max, respectively.

Without assuming any statistical distribution:

Q50 is a more robust measure than avg, and Q95 is a decent measure of a common bad value.


Analysis

Please keep in mind that this is purely a synthetic benchmark, which definitely is one of those need context to interpret type of things.

There is no good way to measure real speed across frameworks, raw numbers here come with big caveats.

Study design limitations

Streaming completion

Streaming completion is very good for time to first result (TTFR), but it presents us with an issue:

While the fast sources will return right away, the slower ones might never make it before the next keystroke.

This has the funny effect of removing the influence of slower sources entirely, which is disastrous for study integrity.

The mitigation is actually to set typing speed unrealistically slow, enough so that we have confidence that the LSP servers can catch up.

This is obviously not ideal.

Fast on paper != fast IRL

The most responsive frameworks are not necessarily the fastest ones, because humans still have to choose the results.

For example the streaming completion approach actually has severe trade offs infavor of faster TTFR:

Ranking

Having suboptimal ranking is BAD, it pushes work from fast machines onto slow humans.

The streaming approach has to be additive, because its too disruptive to shift existing menu items around.

Therefore it is limited to sorting only within stream batches, and to make things worse, slower batches typically contain higher quality results.

That means better results will often end up at the bottom, necessitating more work for humans.

Limiting

This is a direct consequence of limited ranking optimizations.

Because the framework have no idea how much each source will send, it has the dilemma of either sending too many results or too little.

Sending too many results in early batches from likely inferior sources will waste the users time, and sending too little will obscure potentially useful completions.

Clarity on when / if results will come in

This is a HCI thing:

Having higher quality results come in slower is likely to inadvertently train users to wait for them. This is evidently bad for input speed.

Conclusion

There is never going to be a closed form solution to "what is the fastest framework", because of the trade offs detailed above.

A toy example of a degenerate framework that returns a single fixed 👌 emoji will probably beat anything out there in terms of raw speed, but it is utterly useless.

Before you reach your own conclusion, the results of this repo must be considered alongside inextricably human measure.

Owner
i love my dog
dogs are love dogs are life
i love my dog
A Python application that helps users determine their calorie intake, and automatically generates customized weekly meal and workout plans based on metrics computed using their physical parameters

A Python application that helps users determine their calorie intake, and automatically generates customized weekly meal and workout plans based on metrics computed using their physical parameters

Anam Iqbal 1 Jan 13, 2022
dotfiles - Cristian Valero Abundio

In this repository you can find various configurations to configure your Linux operating system, preferably ArchLinux and its derivatives.

Cristian Valero Abundio 1 Jan 09, 2022
Catalogue CRUD Application

This Python program creates a relational SQL database hosted on the Snowflake platform, then opens a CRUD GUI to manipulate and view the data. In this application, it is used as a book catalogue. CUR

0 Dec 13, 2022
LanguageCreator - Simple library for easy creation transpilator.

LanguageCreator - Simple library for easy creation transpilator. Create transpilators in one hour! Install. Download code, rename folder to "LanguageC

Ivan Perzhinsky. 2 Dec 31, 2021
This speeds up PyCharm's package index processes and avoids CPU & memory overloading

This speeds up PyCharm's package index processes and avoids CPU & memory overloading

1 Feb 09, 2022
Render your templates using .txt files

PizzaX About Run Run tests To run the tests, open your terminal and type python tests.py (WIN) or python3 tests.py (UNX) Using the function To use the

Marcello Belanda 2 Nov 24, 2021
FollowSpot is a comprehensive audition tracking fullstack web application for entertainment industry professionals.

FollowSpot is a comprehensive audition tracking fullstack web application for entertainment industry professionals. This app allows users to store information/media for all of their auditions while a

Jen Brissman 9 Jul 12, 2022
A topology optimization framework written in Taichi programming language, which is embedded in Python.

Taichi TopOpt (Under Active Development) Intro A topology optimization framework written in Taichi programming language, which is embedded in Python.

Li Zhehao 41 Nov 17, 2022
An kind of operating system portal to a variety of apps with pure python

pyos An kind of operating system portal to a variety of apps. Installation Run this on your terminal: git clone https://github.com/arjunj132/pyos.git

1 Jan 22, 2022
A python package to manage the stored receiver-side Strain Green's Tensor (SGT) database of 3D background models and able to generate Green's function and synthetic waveform

A python package to manage the stored receiver-side Strain Green's Tensor (SGT) database of 3D background models and able to generate Green's function and synthetic waveform

Liang Ding 7 Dec 14, 2022
High-level bindings to the Valhalla framework.

Valhalla for Python This spin-off project simply offers improved Python bindings to the fantastic Valhalla project. Installation pip install valhalla

GIS • OPS 20 Dec 13, 2022
Meaningful and minimalist release notes for developers

Managing manual release notes is hard. Therefore, everyone tends to generate release notes from commit messages. But, you won't get a meaningful release note at the end.

codezri 31 Dec 30, 2022
A powerful and user-friendly binary analysis platform!

angr angr is a platform-agnostic binary analysis framework. It is brought to you by the Computer Security Lab at UC Santa Barbara, SEFCOM at Arizona S

6.3k Jan 02, 2023
A discord group chat creator just made it because i saw people selling this stuff for like up to 40 bucks

gccreator some discord group chat tools just made it because i saw people selling this stuff for like up to 40 bucks (im currently working on a faster

baum1810 6 Oct 03, 2022
Social reading and reviewing, decentralized with ActivityPub

BookWyrm Social reading and reviewing, decentralized with ActivityPub Contents Joining BookWyrm Contributing About BookWyrm What it is and isn't The r

BookWyrm 1.4k Jan 08, 2023
Blender 3.0 Python - Open temporary areas in the Text Editor

PopDrawers When editing text in Blender, it can be handy to have areas like Info, Console, Outliner, etc visible on screen to help with scripting. How

SpectralVectors 7 Nov 16, 2022
Custom Weapons 3 attribute support for Custom Weapons X

CW3toX Allows use of Custom Weapons 3 attributes in Custom Weapons X. Requiremen

2 Mar 01, 2022
Find all social media accounts with a username!

Aliens_eye FIND ALL SOCIAL MEDIA ACCOUNTS WITH A USERNAME! OSINT To install: Open terminal and type: git clone https://github.com/BLINKING-IDIOT/Alien

Aaron Thomas 84 Dec 28, 2022
:art: Diagram as Code for prototyping cloud system architectures

Diagrams Diagram as Code. Diagrams lets you draw the cloud system architecture in Python code. It was born for prototyping a new system architecture d

MinJae Kwon 27.5k Jan 04, 2023
Sheet2export - FreeCAD macro to export spreadsheet

Description This is FreeCAD macro to export spreadsheet to file.

Darek L 3 Jul 09, 2022