Understanding Emacs packages for Python
Revisiting emacs config after a long gap can be challenging. There might be some packages deprecated or no longer actively maintained and/or replaced by newer ones. There are also chances that you don't remember much about your own config, which is not uncommon. Often elisp code gets copy-pasted from some where without complete understanding of the packages.
Revisiting Python related setup can be even more unpleasant, particularly due to a multitude of options with similar, confusing names and overlapping functionalities. Well, that was my recent experience while getting my existing config to work on a personal laptop.
It's worth noting that my existing config, while functional on my office laptop, had fallen behind by a few years. Meanwhile, the emacs ecosystem has evolved and adopted new technologies such as Language server protocol (LSP) etc. Additionally, the Python language itself has evolved in the recent years. As there was no urgency to get the personal laptop up and running, it was the perfect opportunity to commit to some yak-shaving!
The first puzzle was to understand how the config worked on the other laptop in the first place. The second task was to evaluate the newer packages and decide whether it'd be better or worse to switch to them.
Having done this exercise of modernizing the config, I believe I have a decent understanding of the different emacs packages for python, at least the ones I've used or evaluated. In this blog post, I want to give an overview of the emacs ecosystem for python and clarify some of the confusion around packages with similar names. My intention is to keep it informative rather than coming across as a rant.
Let's first talk about the Python major mode variants. There couldn't be a better example of "multiple packages with confusing names".
The default mode for python built into emacs (since version 24.2 and
as of 29.1) is called
python-mode. Not to be confused with (wait for
latter claims to be "the longest continuously maintained Emacs major
mode for editing Python code".
Before 24.2, the python major mode packaged with emacs was called
python.el. After it was replaced, it was extracted out into a
separate package, python-el.
As far as I remember, I have always used whatever mode that emacs came with at any given point in time. I could never tell any difference, so it's OK for me to stick to the default. That's one less thing to manage.
Jedi is python
package/library that provides auto-completion, static analysis and
refactoring utilities. Plugins specific to different editors and IDEs
can be built on top of it. It's
jedi that does the heavy lifting in
most of the emacs+python tooling that I've come across.
jedi.el is a Python
auto-completion package for emacs that uses
jedi (the python
library) under the hood. Bad choice of name considering that the
jedi provides much more than just auto-completion.
Another source of confusion is that the name of the repository on
emacs-jedi but everywhere in the README it's referred to
jedi.el. In other words,
are essentially the same. Henceforth in this post, the emacs package
will be referred to as
jedi.el and the python package as
EPC, an RPC protocol for emacs lisp, to communicate
with a python process for retrieving completion candidates. This is
implemented using two transitive dependencies:
The emacs package emacs-epc which is used by
jedi.elto implement the EPC client in emacs.
To display the auto-completion candidates,
jedi.el uses the
epc are not actively maintained at present which
is one of the reasons I was considering alternatives. That brings us
company-mode is a generic auto/text completion framework for emacs. It has "backends" for retrieving completion candidates and "frontends" for displaying them. It supports extensibility via pluggable backends and frontends.
An important distinction between
that the former is used like a library whereas the latter is more like
a framework. For example,
jedi.el calls functions from the
auto-complete package. Whereas, a company backend can be implemented
as an elisp function which when "registered" in the list of global
backends, will be invoked by
company-mode at the right time and
(add-to-list 'company-backends 'my-company-backend)
Most importantly, company mode is actively maintained, has good support for different languages and seems better in terms of extensibility.
company-jedi is an
emacs package that implements a company backend for Python using
I had briefly installed
company-jedi until I realized2 that
also comes with a company backend to support auto-completion out of
LSP and eglot
In today's date if you're not using LSP, I'd highly recommend you do. I procrastinated for too long.
Just like EPC, LSP also requires client and server components. There are two popular LSP clients for emacs:
I have tried both with python and rust so far3 and decided to settle with eglot. It covers most of my use cases and to me felt faster and actually light weight as advertised. Additionally, the latest version of emacs 29.1 comes with eglot built-in.
eglot's documentation lists 4 server implementations for python (there could be more). Following the confusing-names tradition, they are:
pyls): Supports python 2.7 but not actively maintained since 3 years.
pylsp): Seems like an actively maintained fork of python-language-server but only for Python 3.8+.
- pyright: It's actually a static type checker for python by Microsoft but includes a language server as well.
- jedi-language-server: Actively maintained.
of them use
I am using
jedi-language-server. It's the first one I tried and it
eglot does feel a bit magical as a lot of things will
start working out of the box. It has "soft dependency" on some popular
packages such as
yasnippet i.e. if these
packages happen to be installed and enabled then
automatically upgrade to provide those functionalities. Thankfully
it's possible to opt out of these. For e.g. I prefer
flymake, so I can opt out of
flymake as follows,
(add-to-list 'eglot-stay-out-of 'flymake)
I was a long time user of the python tool virtualenvwrapper but I don't use it anymore since moving to the venv module that comes with Python 3.3+. However I continue to use the emacs package virtualenvwrapper. Although it is feature compatible with the underlying python tool, today my use of it is limited to just activating and deactivating envs.
With minor workarounds, I have managed to get it working with
virtualenvs created using the built-in
venv module in python.
virtualenvwrapper (the emacs package) sets the correct env vars
in the context of the emacs process, it plays well with eglot too5.
Besides this, I just know one other alternative for virtualenv functionality inside emacs, pyvenv which I haven't tried.
The whole package management ecosystem too has evolved in the recent years with advanced tools such poetry. I don't use poetry but there must be an emacs mode for it.
Besides these, I am only using 2 other packages for python development in emacs:
pytest-el which is pretty minimal and works perfectly.
sphinx-doc for generating docstrings and which I'm the author of. I am ashamed that it's not actively maintained. It still works for untyped and unformatted (by black etc.) python code.
So that was my renewed understanding of the different emacs packages for python development. It is specific to my config (which can be found here) and not exhaustive by any means. But I believe I've covered most of the functionality you'd ask from a Python IDE. I hope it helps any one in a similar situation.
One observation I have is that while emacs is known for its stability, the ecosystem of third party packages is quite fast moving. So like me, if you update your config only after a few years, there is a lot to catch up with. Many alternatives providing overlapping functionality and confusing package names don't help.
But in the end, there's not much to complain. It's incredible that all this software is free and open source and it works if you put some efforts from time to time. Also, yak-shaving is extremely satisfying!
python-epc package actually provides
both server and client implementations↩
2. It took me a better part of the day to
figure out that
company-jedi was not actually being used and it was
eglot that was doing the auto-completion. This can be a separate
blog post but I'm adding it here to preserve the context.
company-jedi to my config, I had uninstalled
along with its (now) unused dependencies namely
python-epc. From the name, it's reasonable to believe that
company-jedi must be depending on
jedi. So I was curious about how
it worked in the absence of
Looking at the source code and dependencies lead to more questions!
company-jedi depends on a package named
jedi-core having the
description (as displayed on melpa) "Common code of jedi.el and
company-jedi.el". Interestingly, there's no repo named
github or anywhere else. The link on melpa points to the same repo as
jedi.el and this is the only place where you can find a file named
jedi-core.el with a heading comment same as the above
jedi.el definitely not installed, where was the
jedi-core.el coming from?
The answer is hidden in its melpa recipe
is built using selected files from the
jedi.el repo. That still
doesn't answer the original question - how did
So I uninstalled
company-jedi but while still initializing
company-mode in the
complete continued to work. Turned out that
eglot was providing auto
completion with the help of
jedi-language-server. Even though find this out was painful, overall
it was a #win to avoid a bunch of packages. ↩
3. I haven't tried eglot with Clojure
yet. Earlier I had attempted to use
lsp-mode with Clojure but
discarded that config due to collisions with
cider-mode which is
indispensable to my workflows. Another reason is that I haven't
written any Clojure on my personal laptop yet. ↩
4. Not having used typed python much, I'm not
jedi-language-server compares with
pyright in that
5. With some configuration and the requirement that the virtualenv is activated before the eglot process starts. Perhaps a topic for another blog post ↩