Doom-emacs: [BUG] ob-async with emacs-jupyter

Try adding this to your private config.el:

(add-hook! '+org-babel-load-functions
  (defun +org-babel-load-jupyter-h (lang)
    (and (string-prefix-p "jupyter-" (symbol-name lang))
         (require lang nil t))))

Thanks for the quick reply. Unfortunately it doesn't fix it.

I also tried with a simpler code block:

#+begin_src jupyter-python :session py :results raw drawer
print('test')
#+end_src

#+RESULTS:
: test

#+begin_src jupyter-python :session py :results raw drawer
print('test')
#+end_src

#+RESULTS:
: c172bf02f62c4210de534c8e3147198b

Here is my current config ("new user"):

;;; .doom.d/config.el -*- lexical-binding: t; -*-

;; Place your private configuration here

(def-package! org
  :config
  ;; (setq python-shell-interpreter "/home/pim/anaconda3/bin/python3")
  ;; (setq org-babel-python-command "/home/pim/anaconda3/bin/python3")
  (setq python-shell-interpreter-args "-m IPython --simple-prompt -i")
  (org-babel-do-load-languages
   'org-babel-load-languages
   '((latex . t)
     (python . t)
     (shell . t)
     (sql . t)
     (jupyter . t))))

(add-hook! '+org-babel-load-functions
  (defun +org-babel-load-jupyter-h (lang)
    (and (string-prefix-p "jupyter-" (symbol-name lang))
         (require lang nil t))))

Remove the org-babel-do-load-languages call. It is redundant with Doom's own lazy-loading mechanism for babel plugins. Not sure if it'll fix your issue, but it might.

Otherwise, try switching from the jupyter package to :lang (org +ipython) instead. It's likely the jupyter emacs package is doing something non-standard.

Also, try adding :async yes or :async no to your src blocks and see if it makes a difference.

The problem with ob-ipython, is that I can't seem to be able to use an environment activated with +conda. Also ob-ipython seems to be loosing support.

I was wanting to try emacs-jupyter, as it was able to pickup the conda environment by default and also has potential to use R and remote kernels.

How shall i add the jupyter to org babel if not with org-babel-do-load-languages ?

emacs-jupyter does not use ob-async. You need to have the following setting for jupyter to work properly.

(setq ob-async-no-async-languages-alist
      '("jupyter-python"))

I've added a +jupyter feature to :lang org that sets ob-async-no-async-languages-alist and +org-babel-load-functions. Let me know if that resolves your issue.

Thank you @randomizedthinking and @hlissner for your help.

After adding +jupyter in the org config of .doom.d/init.el, it seems babel does not pick up that language: org-babel-execute-src-block: No org-babel-execute function for jupyter-python!`

Also, emacs-jupyter can be used as a direct repl for python files. It is not just an integration with org-mode like ob-ipython is. To tell you the truth, I am new to doom/emacs and barely used emacs-jupyter, but I could see that the repl provided by emacs-jupyter is vastly superior, providing powerful auto-completion, support for in-buffer images, etc.

All this to ask, does emacs-jupyter deserve its setup as part of the python module?

Although I have very limited knowledge, I am willing to help with this any way i can.

Meanwhile, i added this to my config file:

(use-package! jupyter
  :defer t
  :init
  (map! :after python
        :map python-mode-map
        :localleader
        (:prefix ("j" . "jupyter")
          :desc "Run REPL"      "o" #'jupyter-run-repl
          :desc "Eval funciton"      "f" #'jupyter-eval-defun
          :desc "Eval buffer"      "b" #'jupyter-eval-buffer
          :desc "Eval region"      "r" #'jupyter-eval-region
          :desc "Restart REPL"      "R" #'jupyter-repl-restart-kernel
          :desc "Interrupt REPL"      "i" #'jupyter-repl-interrup-kernel
          :desc "Scratch buffer"      "s" #'jupyter-repl-scratch-buffer
          :desc "Remove overlays"      "O" #'jupyter-eval-remove-overlays
          :desc "Eval string"      "w" #'jupyter-eval-string
          :desc "Inspect at point"      "d" #'jupyter-inspect-at-point
          )))

For the moment, it seems to work well.

Only the integration with orgmode seems to be broken.

@pmercatoris For jupyter to work in org-babel, my current setting is as follows. Per emacs-jupyter document, the language python (or any other languages we plan to use with jupyter) needs to be included before jupyter.

(org-babel-do-load-languages
 'org-babel-load-languages
 '((shell . t)
   (emacs-lisp . t)
   (python . t)
   (jupyter . t)))

I second that it would be great if emacs-jupyter be integrated into the doom emacs framework.

Also thanks for sharing the keybindings. For all the evaluation function, we can use the current doom gr operator -- it would be consistent cross doom repl.

@pmercatoris See if 17d0833 resolves your issue.

If that doesn't work, see if overriding +org-babel-load-jupyter-h like so helps:

(defun +org-babel-load-jupyter-h (lang)
  (and (string-prefix-p "jupyter-" (symbol-name lang))
       (require 'ob-python)
       (require lang nil t)))

@randomizedthinking Please try it without org-babel-do-load-languages. It has been disabled in 3b2a37d9a because it interferes with Doom's lazy loader, in any case.

@hlissner neither 17d0833 nor overriding seem to work.

I get : org-babel-execute-src-block: No org-babel-execute function for jupyter-python! with no highlighting for the source block

Ok, so i added this to the .doom.d/config.el, and now it works! :)

(add-hook! '+org-babel-load-functions
    (defun +org-babel-load-jupyter-h (lang)
      (and (string-prefix-p "jupyter-" (symbol-name lang))
       (require 'ob-jupyter)
           (require lang nil t))))

The only thing is that the source block highlighting does not load when first visiting the file. I need to revisit after executing a source block. Is that intended?

Great, I will add that the +jupyter submodule after we figure out your block highlighting issue (which isn't intended, we have a lazy loader for block highlighting, but jupyter's is a very unusually structured babel plugin).

See if adding this to your config.el resolves the issue:

(after! org
  (defadvice! +org--src-lazy-load-library-a (lang)
    "Lazy load a babel package to ensure syntax highlighting."
    :before #'org-src--get-lang-mode
    (unless (cdr (assoc lang org-src-lang-modes))
      (or (run-hook-with-args-until-success '+org-babel-load-functions lang)
          (require (intern-soft (format "ob-%s" lang)) nil t)
          (require lang nil t)))))

Ok. But it keeps having the same behaviour, as shown below:

How it is at first visit of file

Execute a cell and revisit file:

Did you try the updated snippet? I improved it shortly after posting it.

Yes, with the updated snippet. Here is my config.el:

;;; .doom.d/config.el -*- lexical-binding: t; -*-

;; Place your private configuration here
;;

(setq python-shell-interpreter "ipython"
      python-shell-interpreter-args "--simple-prompt -i")

(after! org
  (defadvice! +org--src-lazy-load-library-a (lang)
    "Lazy load a babel package to ensure syntax highlighting."
    :before #'org-src--get-lang-mode
    (unless (cdr (assoc lang org-src-lang-modes))
      (or (run-hook-with-args-until-success '+org-babel-load-functions lang)
          (require (intern-soft (format "ob-%s" lang)) nil t)
          (require lang nil t)))))

(add-hook! '+org-babel-load-functions
  (defun +org-babel-load-jupyter-h (lang)
    (and (string-prefix-p "jupyter-" (symbol-name lang))
         (require 'ob-jupyter)
         (require lang nil t))))

;; Select window between frames
(after! ace-window
  (setq aw-scope 'global))

;;;
;;; Keybinds

(setq doom-localleader-key ",")

(use-package! jupyter
  :defer t
  :init
  (map! :after python
        :map python-mode-map
        :localleader
        (:prefix ("j" . "jupyter")
          :desc "Run REPL"      "o" #'jupyter-run-repl
          :desc "Eval funciton"      "f" #'jupyter-eval-defun
          :desc "Eval buffer"      "b" #'jupyter-eval-buffer
          :desc "Eval region"      "r" #'jupyter-eval-region
          :desc "Restart REPL"      "R" #'jupyter-repl-restart-kernel
          :desc "Interrupt REPL"      "i" #'jupyter-repl-interrupt-kernel
          :desc "Scratch buffer"      "s" #'jupyter-repl-scratch-buffer
          :desc "Remove overlays"      "O" #'jupyter-eval-remove-overlays
          :desc "Eval string"      "w" #'jupyter-eval-string-command
          :desc "Inspect at point"      "d" #'jupyter-inspect-at-point
          :desc "Associate buffer REPL"      "B" #'jupyter-repl-associate-buffer
          )))

With the above code, try modifying +org-babel-load-jupyter-h to:

(defun +org-babel-load-jupyter-h (lang)
  (and (string-prefix-p "jupyter-" (symbol-name lang))
       (require (intern
                 (format "ob-%s"
                         (string-remove-prefix "jupyter-" (symbol-name lang))))
                nil t)
       (require 'ob-jupyter nil t)
       (require lang nil t)))

Unfortunately, that isn't working either.

In case it might be linked. The source block only executes after 2 x C-c, C-c
The first time it responds:

executing Jupyter-Python code block...
org-babel-jupyter-session-key: Need a valid session and a kernel to form a key

And the second time it works:

executing Jupyter-Python code block...
Starting python3 kernel process...done
Requesting kernel info...done
Code block evaluation complete. [2 times]

Though, it might be the correct behaviour, I thought I would mention it.

I just wrote some improvements for jupyter in af948ca. Please update and let me know if it fixes the problem (and remove the +org-babel-load-functions and +org--src-lazy-load-library-a suggestions I've given you so far).

Perfect, so the execution of the source block now works without any modification to the config.el.

However the issue with the highlighting persists.

Thank you so much for your work.

I know we have to load _one_ of jupyter's libraries, but I don't know which one and when. This would be much faster if I had a jupyter environment to test in. Perhaps each module should include a nix file to set up an minimum viable environment for testing in.

Anyhow, could you try this?

(defadvice! oh-god-please-fix-jupyter-please-oh-please (lang)
  :override #'+org-babel-load-jupyter-h
  (when (string-prefix-p "jupyter-" (symbol-name lang))
    (let ((lang (string-remove-prefix "jupyter-" (symbol-name lang))))
      (unless (assoc lang org-src-lang-modes)
        (require (intern (format "ob-%s" lang))
                 nil t)))
    (with-demoted-errors "Jupyter: %s"
      (require 'jupyter nil t)
      (require lang nil t)
      (require 'ob-jupyter nil t))))

oh-god-please-fix-jupyter-please-oh-please haha doesnt seem to have much compation, as I still get the same behaviour :-(

Don't know anything about Nix, but definitely sounds like a nice feature.

Could it be linked to this comment?

Note, jupyter should be added as the last element when loading languages since it depends on the values of variables such as org-src-lang-modes and org-babel-tangle-lang-exts. After ob-jupyter has been loaded, new source code blocks with names of the form jupyter-LANG will be available. LANG can be any one of the kernel languages found on your system. See jupyter-available-kernelspecs.(https://github.com/dzop/emacs-jupyter#org-mode-source-blocks)

Could it be linked to this comment?

It is, but I've paid attention to that with +org-babel-load-jupyter-h, but it still refuses to work. So let's give it one more try:

(defadvice! oh-god-please-fix-jupyter-please-oh-please (lang)
  :override #'+org-babel-load-jupyter-h
  (when (string-prefix-p "jupyter-" (symbol-name lang))
    (let* ((lang-name (symbol-name lang))
           (lang-tail (string-remove-prefix "jupyter-" lang-name)))
      (and (not (assoc lang-tail org-src-lang-modes))
           (require (intern (format "ob-%s" lang-tail))
                    nil t)
           (add-to-list 'org-src-lang-modes (cons lang-name (intern lang-tail)))))
    (with-demoted-errors "Jupyter: %s"
      (require lang nil t)
      (require 'ob-jupyter nil t))))

If this doesn't work, then we'll just resort to the dumb solution:

(after! ob-jupyter
  (dolist (lang '(python julia R))
    (cl-pushnew (cons (format "jupyter-%s" lang) lang)
                org-src-lang-modes :key #'car)))

I know this will work, but I'll have to depend on people to expand this list as jupyter introduces more languages.

So, apparently not even the dumb solution works for the highlighting. But more importantly, it seems the org-edit-special won't work, as it returns:
org-edit-src-code: No such language mode: jupyter-python-mode

I do hope I have tried everything correctly, as i said I am very new to doom, but made sure to do a doom refresh after any change.

Oh, oops. Could you try my dumb solution again with (after! org-src ...) instead of (after! ob-jupyter ...)?

All working from what i can see!! :D

Highlighting, block execution and special edit. So I'm not sure how you feel about the solution, but pretty glad it's working :)

Thank you!!

Hi! I think something here may have broken R support in org-mode using the Jupyter plugin. Execution of jupyter-python blocks works fine:

But execution of jupyter-R blocks does not:

Any advice on how to proceed?

@mbosley https://github.com/hlissner/doom-emacs/pull/2477 will likely fix your issue.

In the meantime, I will close this thread since the OP's issue was resolved.