With R, and Ananconda installed, we can also use R in Jupyter notebook. So my previous laptop died and now I have to re-install everything again. But this time I ran into some issue that I didn't have with my previous laptop (not too sure why).
So what is necessary to install R kernel in Jupyter is to run the two lines of code in R:
install.packages('IRkernel')
IRkernel::installspec()
But I ran into the following error.

What we need to do is to find where R.exe is located. For me, it's C:\Program Files\R\R-3.6.1\bin\x64. Using Anaconda Prompt, i.e. command line interface for Anaconda, I changed the directory to that. Then I run R.exe and then IRkernel::installspec().

Now we can use R in Juptyer notebook :)

References:
https://github.com/IRkernel/IRkernel
https://github.com/IRkernel/IRkernel/issues/510