Apache Zeppelin

https://zeppelin.apache.org/

• https://github.com/apache/zeppelin → repo on Github

• Apache Zeppelin Documentation (my summary of https://zeppelin.apache.org/docs/latest/)

• How to build Zeppelin from source
• Contributing to Apache Zeppelin ( Code )

• Scala → language used in Zeppelin

cd C:\v\IdeaProjects\scalatest\docker
docker-compose up -d

• Create an image ('zeppelin-for-dev:latest') from a container ('zeppelinForDev'):

docker commit zeppelinForDev zeppelin-for-dev:latest

CAUTION: Zeppelin on Windows is not supported any more. Zeppelin starts, but does not really work. So, it would be better to use a container-based installation.

Location:

C:\v\pgm\zeppelin-0.11.0

To start:

cd C:\v\pgm\zeppelin-0.11.0\bin
zeppelin.cmd

Location:

/mnt/sda4/development/zeppelin-0.11.0-bin-netinst/

To start as daemon:

cd /mnt/sda4/development/zeppelin-0.11.0-bin-netinst
./bin/zeppelin-daemon.sh

To start interactively:

cd /mnt/sda4/development/zeppelin-0.11.0-bin-netinst/bin
zeppelin.sh

• JAVA_HOME should be set to “Java 8 update 151 or newer”. (For current user it can be done in ~/.bashrc).
  • For example:
    • export JAVA_HOME=/mnt/sda4/development/java/jdk11

Origin: git@78.47.83.54:/srv/git/heliumpackages.git

https://zeppelin.apache.org/docs/0.11.0/interpreter/spark.html

• …
• set SPARK_HOME
  • Decription for development environment
    • Set SPARK_HOME in zeppelin-env.sh
    • Set SPARK_HOME in interpreter setting page
    • Set SPARK_HOME via inline generic configuration
  • For docker container (example):

    • docker run -u $(id -u) -p 8080:8080 -p 4040:4040 --rm -v /mnt/disk1/spark-3.1.2:/opt/spark -e SPARK_HOME=/opt/spark  --name zeppelin apache/zeppelin:0.10.0

• …
• Spark Dependency management
  • set the following variables (e.g. through UI → “Interpreter”):
    • spark.files
    • spark.jars
    • spark.jars.packages
• Cache is located under
  • at home: /mnt/sda4/development/workspaces.zeppelin/zeppelin/local-repo/spark

• Memory problems
  • if we use data of one year, memory problems can occur. To solve them set the memory parameters of the Spark interpreter higher.
    • On my home computer I've set:
      • spark.driver.memory = 2g
      • spark.executor.memory = 8g
      • also the heap size of Zeppelin can play a role (but the interpreter parameters seems to be more important)

• 1. Set ZEPPELIN_INTP_JAVA_OPTS for zeppelin installation
  • This can be done in different ways. Also it depends on your installation (docker or not, Windows or Linux etc). On my docker container (Windows at work) I've set it in ${ZEPPELIN_HOME}/conf/zeppelin-env.sh:

-- not wait for debugger (n=> JVM executes the code immediately)
export ZEPPELIN_INTP_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=6006"
-- If you want Java to wait for you to connect before executing the application, replace suspend=n with suspend=y. (is usually not necessary)
export ZEPPELIN_INTP_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=6006"

• Important for debugging if Zeppelin was started not from Docker, but from IntelliJ:
  • in the IntelliJ run configuration set environment variable “ZEPPELIN_INTP_JAVA_OPTS” and DO NOT add JVM parameter
    • I don't really know why, but I think, because the Spark interpreter JVM will be started separately, and then ZEPPELIN_INTP_JAVA_OPTS will be used
• 2. open this port in the docker (not necessary if Zeppelin starts not from Docker)
• e.g. with in the docker compose config:

    ports:
      - 127.0.0.1:8080:8080
      - 127.0.0.1:8111:8111
      - 127.0.0.1:5005:5005
      - 127.0.0.1:6006:6006

• 3. add a “Remote JVM Debug” configuration in IntelliJ which uses the same port (here 6006)

To start a debugging session:

1. start Zeppelin (from docker container or directly)
2. open browser with the notebook and note which should be debugged
3. start the necessary paragraph (IMPORTANT: do it before to start remote debug. The server should open the JVM debug port at first)
   • Zeppelin will show “Running…”
4. start the remote debug in IntelliJ
   • maybe it should be made twice, because after the first click, the connection will break immediately (maybe it is so, because the debug port should be opened first (see the point above))
5. IntelliJ should stop on the breakpoint

Note: it is not possible to set the value for a variable with F2 key (=“Set value”). But it can be done with “Evaluate” functionality. E.g. “var1=123”. See bug description

See also: How to diagnose Zeppelin

https://zeppelin.apache.org/docs/0.11.0/development/helium/writing_visualization_basic.html#4-run-in-dev-mode

Prerequisites:

• Yarn should be installed →

sudo npm install --global yarn

• see also : https://classic.yarnpkg.com/en/docs/install
• node should have the proper version
• at the moment ~16.20.2
• the version of Node to use can be changed with: nvm use 16.20.2
  • prerequisite: nvm should be installed

So in short:

• start Zeppelin
  • it is not very important how it is started
    • it works via running main class “ZeppelinServer”
    • zeppelin-daemon.sh should also work
• start “Helium dev mode” :

# Linux at home
cd /mnt/sda4/development/workspaces.zeppelin/zeppelin/zeppelin-web
nvm use 16.20.2
yarn run dev:helium

# all in one
cd /mnt/sda4/development/workspaces.zeppelin/zeppelin/zeppelin-web && nvm use 16.20.2 && yarn run dev:helium


# Linux at work 
cd /v/IdeaProjects/zeppelin/zeppelin-web && yarn run dev:helium

%sh echo -e "%table key\tvalue\nsun\t-100\nmoon\t20"

These problems are occured in my Docker-based Zeppelin instance. It was not worked properly on some places, so it was necessary to make some configuration changes.

The default spark interpreter did not load my JARS, so it is important to set the SPARK_HOME to a spark installation (at least 3.1.2). So firstly:

• open terminal session in the zeppelin container
  • e.g. docker exec -u 0 -it zeppelin2 bash
• install Spark 3.1.2 in the container environment (!)
  • see https://archive.apache.org/dist/spark/
• set the SPARK_HOME in the “Interpreter” settings of Spark
  • Through the web application: e.g. http://localhost:8080/#/interpreter
• set path of the Jars in “spark.jars” value
  • Also on the same page of the web application: http://localhost:8080/#/interpreter
• after that open your “Notebook” and test to call something from your jar
  • e.g. import de.liebib.sparktest.SparkTester

To load JARs which was newly compiled it is necessary to restart the Tomcat. Restarting Spark interpreter does not suffice. So to load new code from JARs there are the following ways:

• restart the Tomcat. (I don't know how to do it at the moment.)
• restart the Spark interpreter.
  • simple restart unfortunately does not work, it is necessary to
    1. change confuguration of spark.jars parameter (e.g. set it to an empty value)
    2. save configuration
    3. change confuguration of spark.jars parameter again
    4. restart interpreter
• simply to restart the container completely. Maybe it is the quickest solution.

See also: https://stackoverflow.com/questions/40550011/zeppelin-how-to-restart-sparkcontext-in-zeppelin

To compile helium applications (for example those, which are contained in zeppelin-examples) the dependency helium-dev is necessary, which is not saved in the remote apache repository. So to get it, first compile and install it to the local maven repo:

./mvnw clean install -Phelium-dev -DskipTests

After that the package should be in your local repo.

Configurations made for example in conf/zeppelin-site.xml (conf/shiro.ini too?) will be not loaded by default. It is necessary to set environment variable ZEPPELIN_HOME. In the case of my development environment at home:

ZEPPELIN_HOME=/mnt/sda4/development/workspaces.zeppelin/zeppelin

After that Zeppelin will load configuration files from ${ZEPPELIN_HOME}/conf.

• Start container (with name “zeppelin”):

  • docker run -p 8080:8080 --rm --name zeppelin apache/zeppelin:0.11.0

  • docker run -p 8080:8080 --mount type=bind,source="C:\v\zeppelin\app",target=/app --name zeppelin apache/zeppelin:0.11.0

  • docker run -p 8080:8080 --mount type=bind,source="C:\v\zeppelin\app",target=/app --mount type=bind,source="C:\v\zeppelin\notebook",target=/notebook --name zeppelin apache/zeppelin:0.11.0

• Start container (with name “stoic_bartik”):

  • docker run -p 8080:8080 --rm --name stoic_bartik apache/zeppelin:0.11.0

• Login as root in the bash of running container

  • docker exec -u 0 -it zeppelinForDev bash

    • there are also other possibilities

• Running Zeppelin 0.8.2 on Windows 10 with Python and PySpark support
  • can be useful in this (=Windows) context: https://stackoverflow.com/questions/61078714/apache-zeppelin-not-loading-in-a-browser-in-windows-10
• How do I assign a port mapping to an existing Docker container?
• How to diagnose Zeppelin

export ZEPPELIN_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005"
export ZEPPELIN_INTP_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=6006"

• Debug JavaScript in Chrome