====== Apache Zeppelin ======

https://zeppelin.apache.org/

  * https://github.com/apache/zeppelin -> repo on Github 
  
  * [[Apache Zeppelin Documentation]] (my summary of https://zeppelin.apache.org/docs/latest/)
 
  * [[https://zeppelin.apache.org/docs/0.11.1/setup/basics/how_to_build.html|How to build Zeppelin from source]]
  * [[https://zeppelin.apache.org/docs/0.11.1/development/contribution/how_to_contribute_code.html|Contributing to Apache Zeppelin ( Code )]]

  * [[Scala]] -> language used in Zeppelin
===== Local installation =====


==== In Container (Windows at Work) ====

<code>
cd C:\v\IdeaProjects\scalatest\docker
docker-compose up -d
</code>

=== Useful commands in the context of Windows and Docker ===

  * Create an image ('zeppelin-for-dev:latest') from a container ('zeppelinForDev'):
<code>
docker commit zeppelinForDev zeppelin-for-dev:latest
</code>

==== Directly (Windows at Work) ====

**CAUTION:** Zeppelin on Windows is not supported any more. Zeppelin starts, but does not really work. So, it would be better to use a container-based installation.

Location:

''C:\v\pgm\zeppelin-0.11.0''

To start:
<code>
cd C:\v\pgm\zeppelin-0.11.0\bin
zeppelin.cmd
</code> 




==== Linux at Home ====

Location:

''/mnt/sda4/development/zeppelin-0.11.0-bin-netinst/''

To start as daemon:
<code>
cd /mnt/sda4/development/zeppelin-0.11.0-bin-netinst
./bin/zeppelin-daemon.sh
</code> 


To start interactively:
<code>
cd /mnt/sda4/development/zeppelin-0.11.0-bin-netinst/bin
zeppelin.sh
</code> 

=== Problems occured during installation ===

  * ''JAVA_HOME'' should be set to "Java 8 update 151 or newer". (For current user it can be done in ''~/.bashrc''). 
    * For example:
      * ''export JAVA_HOME=/mnt/sda4/development/java/jdk11''


==== Git ====

Origin: ''git@78.47.83.54:/srv/git/heliumpackages.git''
===== Spark interpreter =====

https://zeppelin.apache.org/docs/0.11.0/interpreter/spark.html

  * ...
  * set SPARK_HOME
    * [[https://zeppelin.apache.org/docs/0.11.0/interpreter/spark.html#set-spark_home|Decription for development environment]]
      * Set SPARK_HOME in zeppelin-env.sh
      * Set SPARK_HOME in interpreter setting page
      * Set SPARK_HOME via inline generic configuration
    * For docker container (example):
      * <code>docker run -u $(id -u) -p 8080:8080 -p 4040:4040 --rm -v /mnt/disk1/spark-3.1.2:/opt/spark -e SPARK_HOME=/opt/spark  --name zeppelin apache/zeppelin:0.10.0</code>
  * ...
  * [[https://zeppelin.apache.org/docs/0.11.0/interpreter/spark.html#dependency-management|Spark Dependency management]]
    * set the following variables (e.g. through UI -> "Interpreter"):
      * spark.files
      * spark.jars
      * spark.jars.packages
  * Cache is located under 
    * at home: ''/mnt/sda4/development/workspaces.zeppelin/zeppelin/local-repo/spark''


  * Memory problems
    * if we use data of one year, memory problems can occur. To solve them set the memory parameters of the Spark interpreter higher. 
      * On my home computer I've set:
        * ''spark.driver.memory'' = 2g  
        * ''spark.executor.memory'' = 8g
        * also the heap size of Zeppelin can play a role (but the interpreter parameters seems to be more important)


===== Debugging Zeppelin notebook paragraphs in IntelliJ =====

  * 1. Set ZEPPELIN_INTP_JAVA_OPTS for zeppelin installation
    * This can be done in different ways. Also it depends on your installation (docker or not, Windows or Linux etc). On my docker container (Windows at work) I've set it in  ''${ZEPPELIN_HOME}/conf/zeppelin-env.sh'':
<code bash>
-- not wait for debugger (n=> JVM executes the code immediately)
export ZEPPELIN_INTP_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=6006"
-- If you want Java to wait for you to connect before executing the application, replace suspend=n with suspend=y. (is usually not necessary)
export ZEPPELIN_INTP_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=6006"
</code>
      * Important for debugging if Zeppelin was started not from Docker, but from IntelliJ:
        * in the IntelliJ run configuration set environment variable "ZEPPELIN_INTP_JAVA_OPTS" and DO NOT add JVM parameter
          * I don't really know why, but I think, because the Spark interpreter JVM will be started separately, and then ZEPPELIN_INTP_JAVA_OPTS will be used 
  * 2. open this port in the docker (not necessary if Zeppelin starts not from Docker)
    * e.g. with in the docker compose config: 
<code yaml>
    ports:
      - 127.0.0.1:8080:8080
      - 127.0.0.1:8111:8111
      - 127.0.0.1:5005:5005
      - 127.0.0.1:6006:6006
</code>
  * 3. add a "Remote JVM Debug" configuration in IntelliJ which uses the same port (here 6006)

To start a debugging session:
  - start Zeppelin (from docker container or directly)
  - open browser with the notebook and note which should be debugged
  - start the necessary paragraph (IMPORTANT: do it before to start remote debug. The server should open the JVM debug port at first)
    * Zeppelin will show "Running..."
  - start the remote debug in IntelliJ
    * maybe it should be made twice, because after the first click, the connection will break immediately (maybe it is so, because the debug port should be opened first (see the point above))
  - IntelliJ should stop on the breakpoint

Note: it is not possible to set the value for a variable with F2 key (="Set value"). But it can be done with "Evaluate" functionality. E.g. "var1=123". See [[https://intellij-support.jetbrains.com/hc/en-us/community/posts/360000094924-Can-t-set-the-value-of-a-variable-in-the-debugger|bug description]]

See also: [[https://community.cloudera.com/t5/Community-Articles/How-to-diagnose-zeppelin/ta-p/248268|How to diagnose Zeppelin]]

===== Running Helium Visualization in development mode =====

https://zeppelin.apache.org/docs/0.11.0/development/helium/writing_visualization_basic.html#4-run-in-dev-mode

Prerequisites:
  * Yarn should be installed -> 
<code>
sudo npm install --global yarn
</code>
    * see also : https://classic.yarnpkg.com/en/docs/install
  * ''node'' should have the proper version
    * at the moment ''~16.20.2''
    * the version of Node to use can be changed with: ''nvm use 16.20.2''
      * prerequisite: [[https://github.com/nvm-sh/nvm|nvm]] should be installed  
So in short:
  * start Zeppelin 
    * it is not very important how it is started
      * it works via running main class "ZeppelinServer"
      * ''zeppelin-daemon.sh'' should also work
  * start "Helium dev mode" :
<code>
# Linux at home
cd /mnt/sda4/development/workspaces.zeppelin/zeppelin/zeppelin-web
nvm use 16.20.2
yarn run dev:helium

# all in one
cd /mnt/sda4/development/workspaces.zeppelin/zeppelin/zeppelin-web && nvm use 16.20.2 && yarn run dev:helium


# Linux at work 
cd /v/IdeaProjects/zeppelin/zeppelin-web && yarn run dev:helium
</code>


==== Display a table for tests ====
<code>
%sh echo -e "%table key\tvalue\nsun\t-100\nmoon\t20"
</code>
===== Problems occured =====

These problems are occured in my Docker-based Zeppelin instance. It was not worked properly on some places, so it was necessary to make some configuration changes.

==== Loading JARs ====
The default spark interpreter did not load my JARS, so it is important to set the SPARK_HOME to a spark installation (at least 3.1.2). So firstly:
  * open terminal session in the zeppelin container
    * e.g. ''docker exec -u 0 -it zeppelin2 bash''
  * install Spark 3.1.2 in the container environment (!)
    * see ''https://archive.apache.org/dist/spark/''
  * set the SPARK_HOME in the "Interpreter" settings of Spark 
    * Through the web application: e.g. http://localhost:8080/#/interpreter
  * set path of the Jars in "spark.jars" value 
    * Also on the same page of the web application: http://localhost:8080/#/interpreter
  * after that open your "Notebook" and test to call something from your jar
    * e.g. ''import de.liebib.sparktest.SparkTester''

==== Loading newly compiled JARs ====
To load JARs which was newly compiled it is necessary to restart the Tomcat. Restarting Spark interpreter does  not suffice. So to load new code from JARs there are the following ways: 
  * restart the Tomcat. (I don't know how to do it at the moment.)
  * restart the Spark interpreter.
    * simple restart unfortunately does not work, it is necessary to
      - change confuguration of ''spark.jars'' parameter (e.g. set it to an empty value)
      - save configuration
      - change confuguration of ''spark.jars'' parameter again
      - restart interpreter
  * simply to restart the container completely. Maybe it is the quickest solution.

See also: https://stackoverflow.com/questions/40550011/zeppelin-how-to-restart-sparkcontext-in-zeppelin

==== Dependency helium-dev is necessary to compile zeppelin-examples ====
To compile helium applications (for example those, which are contained in ''zeppelin-examples'') the dependency ''helium-dev'' is necessary, which is **not** saved in the remote apache repository. So to get it, first compile and install it to the local maven repo: 
<code bash>
./mvnw clean install -Phelium-dev -DskipTests
</code>

After that the package should be in your local repo.

==== Enable configurations made in files lying in ''conf'' directory  ====

Configurations made for example in ''conf/zeppelin-site.xml'' (''conf/shiro.ini'' too?) will be not loaded by default. It is necessary to set environment variable ''ZEPPELIN_HOME''. In the case of my development environment at home:
<code>
ZEPPELIN_HOME=/mnt/sda4/development/workspaces.zeppelin/zeppelin
</code>

After that Zeppelin will load configuration files from ''${ZEPPELIN_HOME}/conf''.


===== Misc =====

==== Useful commands ====
  * Start container (with name "zeppelin"):
    * <code>docker run -p 8080:8080 --rm --name zeppelin apache/zeppelin:0.11.0</code>
    * <code>docker run -p 8080:8080 --mount type=bind,source="C:\v\zeppelin\app",target=/app --name zeppelin apache/zeppelin:0.11.0</code>
    * <code>docker run -p 8080:8080 --mount type=bind,source="C:\v\zeppelin\app",target=/app --mount type=bind,source="C:\v\zeppelin\notebook",target=/notebook --name zeppelin apache/zeppelin:0.11.0</code>
  * Start container (with name "stoic_bartik"):
    * <code>docker run -p 8080:8080 --rm --name stoic_bartik apache/zeppelin:0.11.0</code>
  * Login as root in the bash of running container
    * <code>docker exec -u 0 -it zeppelinForDev bash</code>
      * there are also [[https://stackoverflow.com/questions/28721699/root-password-inside-a-docker-container|other possibilities]]

==== Some links ====

  * [[https://bradfordcp.io/posts/running-zeppelin-0.8.2-with-python-and-pyspark-on-windows-10/|Running Zeppelin 0.8.2 on Windows 10 with Python and PySpark support]]
    * can be useful in this (=Windows) context: https://stackoverflow.com/questions/61078714/apache-zeppelin-not-loading-in-a-browser-in-windows-10
  * [[https://stackoverflow.com/questions/19335444/how-do-i-assign-a-port-mapping-to-an-existing-docker-container|How do I assign a port mapping to an existing Docker container?]]
  * [[https://community.cloudera.com/t5/Community-Articles/How-to-diagnose-zeppelin/ta-p/248268|How to diagnose Zeppelin]]
<code>
export ZEPPELIN_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005"
export ZEPPELIN_INTP_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=6006"
</code>

  * [[https://www.jetbrains.com/help/idea/debugging-javascript-in-chrome.html|Debug JavaScript in Chrome]]