Deploying a Ceph Development Environment Cluster
Table of Contents
This post explains how we can deploy a Ceph development cluster from Ceph source code.
I tested it in Windows 10 + Docker for Windows with WSL2 engine + WSL2 Ubuntu 20.04.
1. Prerequisites #
Two Github repositores are necessary:
Ceph 1 and
Ceph dev docker is a kind of wrapper that automates all required steps for deloying Ceph development cluster. It users Docker container to deploy the local development of Ceph. Therefore, Docker is also required. Refer to [this] for Docker installation.
Ceph repository is bound into a container and compiled inside the container. It requires substantial amount of time to be fully built. However, after finishing the first build, Ceph stores a compiler cache (ccache) and uses them for further incremental build, which significantly reduce build time. For this, make an empty directory anywhere with any name.
$ git clone https://github.com/ceph/ceph $ git clone https://github.com/ricardoasmarques/ceph-dev-docker $ mkdir ceph-ccache
2. Running a Docker Container #
ceph-dev-docker provides a script
setup.sh to get a container.
All required packages will be installed in this container, Ceph will be compiled here, and run in here as well.
As indicated in
README.md in the repository, it takes several variables. If your environment (the location or the name of directories) are different from mine, read it carefully for proper container initialization.
~/ceph-dev-docker $ NAME=ceph-dev CCACHE=/root/ceph-ccache CEPH=/root/ceph ./setup.sh
The shell script builds a Docker container image with
docker build, and run it.
Once completed, you are in ther container with
zsh. Check whether
/ceph is properly mounted.
There are several pre-implemented scripts in
/shared/bin in the container for Ceph deployment.
3. Building and Running Ceph #
Although Ceph provides [a manual] to build Ceph,
ceph-dev-docker provides a more convenient way for this.
/ceph $ setup-ceph.sh
This script installs all the required packages, and build Ceph from source. I prefer this way since I do not have to install packages in the host machine that are not used by others.
It must be run in
/cephdirectory. The container is implemented assuming that Ceph root directory is
Running Ceph is also wrapped in a sciprt:
start-ceph.sh. It uses
vstart.sh, a Ceph utility that deploys fake local cluster for development purpose 3. Once you are accustomed with the Ceph internal architecture, you can manually run
vstart.sh for deployment.
The script runs **3 OSDs, 3 monitors, 3 metadata servers (MDSs), 1 manager, and 1 RADOS gateway (RGW).
You can manipulate the number of daemons with
OSD,MDS,MON,RGW enviornment variables as specified in [the manual]. The
start-ceph.sh overrode the number of RGW from 0 to 1 [src].
4. Ceph Dashboard #
ceph-dev-docker also provides Ceph dashboard by default with
By running the script, you can access the Ceph dashboard through
For Ceph dashboard features, refer to [this manual].
If you are using WSL, proper network configuration is required. Docker network is isolated from the host. hence you cannot directly access to Docker network namespace even if you are using
--net=hostcontainer configuration. Refer to [this].
5. Integrating VSCode CMake #
Since Ceph uses Cmake as its build system, it can be integrated into VSCode CMake extension. I personally use Microsoft CMake Tool.
ceph-dev-docker uses a script to build Ceph, it is simple to understand.
setup-ceph.sh looks like:
ARGS="-DENABLE_GIT_VERSION=OFF -DWITH_TESTS=ON -DWITH_CCACHE=ON $ARGS" ARGS="-DWITH_PYTHON3=3 -DWITH_RADOSGW_AMQP_ENDPOINT=OFF -DWITH_RADOSGW_KAFKA_ENDPOINT=OFF $ARGS" ... if [ -d "build" ]; then git submodule update --init --recursive cd build cmake $ARGS .. else ./do_cmake.sh $ARGS cd build fi ccache make -j
where configuration arguments are given to CMake. Similarly, we can put the arguments into VSCode Cmake tool configuration:
For ccache usage, I add one more argument:
Refer to this post for more details.
Now I can directly build Ceph in VSCode!