Deploying a Ceph Development Environment Cluster

This post explains how we can deploy a Ceph development cluster from Ceph source code.

I tested it in Windows 10 + Docker for Windows with WSL2 engine + WSL2 Ubuntu 20.04.

1. Prerequisites #

Two Github repositores are necessary: Ceph ¹ and Ceph-dev-docker ².

Ceph dev docker is a kind of wrapper that automates all required steps for deloying Ceph development cluster. It users Docker container to deploy the local development of Ceph. Therefore, Docker is also required. Refer to [this] for Docker installation.

Ceph repository is bound into a container and compiled inside the container. It requires substantial amount of time to be fully built. However, after finishing the first build, Ceph stores a compiler cache (ccache) and uses them for further incremental build, which significantly reduce build time. For this, make an empty directory anywhere with any name.

$ git clone https://github.com/ceph/ceph
$ git clone https://github.com/ricardoasmarques/ceph-dev-docker
$ mkdir ceph-ccache

2. Running a Docker Container #

ceph-dev-docker provides a script setup.sh to get a container. All required packages will be installed in this container, Ceph will be compiled here, and run in here as well.

As indicated in README.md in the repository, it takes several variables. If your environment (the location or the name of directories) are different from mine, read it carefully for proper container initialization.

~/ceph-dev-docker $ NAME=ceph-dev CCACHE=/root/ceph-ccache CEPH=/root/ceph ./setup.sh

The shell script builds a Docker container image with docker build, and run it.

Once completed, you are in ther container with zsh. Check whether /ceph is properly mounted. There are several pre-implemented scripts in /shared/bin in the container for Ceph deployment.

3. Building and Running Ceph #

Although Ceph provides [a manual] to build Ceph, ceph-dev-docker provides a more convenient way for this.

/ceph $ setup-ceph.sh

This script installs all the required packages, and build Ceph from source. I prefer this way since I do not have to install packages in the host machine that are not used by others.

It must be run in /ceph directory. The container is implemented assuming that Ceph root directory is /ceph.

Running Ceph is also wrapped in a sciprt: start-ceph.sh. It uses vstart.sh, a Ceph utility that deploys fake local cluster for development purpose ³. Once you are accustomed with the Ceph internal architecture, you can manually run vstart.sh for deployment.

The script runs **3 OSDs, 3 monitors, 3 metadata servers (MDSs), 1 manager, and 1 RADOS gateway (RGW). You can manipulate the number of daemons with OSD,MDS,MON,RGW enviornment variables as specified in [the manual]. The start-ceph.sh overrode the number of RGW from 0 to 1 [src].

4. Ceph Dashboard #

ceph-dev-docker also provides Ceph dashboard by default with npm-start.sh script. By running the script, you can access the Ceph dashboard through https://localhost:4200. For Ceph dashboard features, refer to [this manual].

If you are using WSL, proper network configuration is required. Docker network is isolated from the host. hence you cannot directly access to Docker network namespace even if you are using --net=host container configuration. Refer to [this].

5. Integrating VSCode CMake #

Since Ceph uses Cmake as its build system, it can be integrated into VSCode CMake extension. I personally use Microsoft CMake Tool.

Even ceph-dev-docker uses a script to build Ceph, it is simple to understand. setup-ceph.sh looks like:

ARGS="-DENABLE_GIT_VERSION=OFF -DWITH_TESTS=ON -DWITH_CCACHE=ON $ARGS"
ARGS="-DWITH_PYTHON3=3 -DWITH_RADOSGW_AMQP_ENDPOINT=OFF -DWITH_RADOSGW_KAFKA_ENDPOINT=OFF $ARGS"

...

if [ -d "build" ]; then
    git submodule update --init --recursive
    cd build
    cmake $ARGS ..
else
    ./do_cmake.sh $ARGS
    cd build
fi

ccache make -j

where configuration arguments are given to CMake. Similarly, we can put the arguments into VSCode Cmake tool configuration:

For ccache usage, I add one more argument: -DCMAKE_CXX_COMPILER_LAUNCHER=ccache. Refer to this post for more details.

Now I can directly build Ceph in VSCode!