use https://github.com/emacs-jupyter/jupyter for jupyter integration.

use https://github.com/jkitchin/ox-ipynb to export as jupyter notebooks to share with collegues.

All the other packages only makes sense if you are working alone. If you have to share your work with others, and aimed for DS/ML work thats what I would recommend.

I have something like this

;; ----- language server

(require 'eglot)
(add-hook 'python-mode-hook #'eglot-ensure)
(add-to-list 'eglot-server-programs
	     `(python-mode
	       . ,(eglot-alternatives '("pylsp"
					"jedi-language-server"
					("pyright-langserver" "--stdio")))))

;; ----- treesitter

(use-package treesit-auto :ensure t)
;; treesit grammar should be installed here: "~/.emacs.d/tree-sitter/"

;; ----- select interpreter

(setopt python-shell-interpreter ("~/miniconda3/bin/python"))
(setopt python-shell-interpreter-args "-i")

;; ----- conda

(use-package conda
  :ensure t
  :custom
  (conda-anaconda-home "~/miniconda3")
  (conda-env-home-directory "~/miniconda3")
  (conda-env-subdirectory "envs")
  :config
  (unless (getenv "CONDA_DEFAULT_ENV")
    (conda-env-activate "base"))))

;; ----- emacs-jupyter

(use-package jupyter
  :commands
  (jupyter-run-server-repl
   jupyter-run-repl
   jupyter-server-list-kernels))
(org-babel-jupyter-override-src-block "python") ;; so python becomes jupyter-python in org-babel

;; ----- other settings

(setopt python-indent-guess-indent-offset-verbose nil)

I use Emacs 29.1 or Emacs 30, because:

1. they have built in LSP support by Eglot.
2. they have built in tree-sitter support by python-ts-mode

And I use pyright as the LSP backend.

After install pyright, just start Emacs, open python file and M-x eglot will fire up the lsp support.

For the python virtual environment, I use envr(direnv)on macOS and Linux, use pyvenv on Windows platform.

You can see my personal configuration here:

https://github.com/Eason0210/.emacs.d/blob/6428e1a3be904a97ee354c955c879baeed90c099/init.el#L873-L948

It sounds like you’re confused about Python and Emacs as well as their relationship. You’ll need to find a Python data science tutorial and read all the way through it.

This should help with the emacs side: https://github.com/wyuenho/emacs-pet

I like this article: https://robbmann.io/posts/006_emacs_2_python/

It covers Python configuration using mostly built-it modules and features

still using elpy, it werks

jedi-language-server + eglot only.
maybe i’ll extend it to support running unit tests.

Following is my configuration for Python; I am using Emacs 29.1 --with-tree-sitter:

  (use-package treesit
    :config
    (setq treesit-language-source-alist
	  '((python . ("https://github.com/tree-sitter/tree-sitter-python.git"))))
    (add-to-list 'auto-mode-alist
		 `("\\.py[iw]?\\'\\|python[0-9.]*" . python-ts-mode)))

  (use-package eglot
    :hook ((python-mode . eglot-ensure)	; pip install 'python-language-server[all]'
	   (python-ts-mode . eglot-ensure)))