Here is the app if you just want to shorten some links SRTL. It was the shortest domain I could grab it stands for Short Link or at least thats what I’m telling myself.
I recently saw a post on Medium about RocksDB that sparked my curiosity. It’s a high performance key value store using a LSM tree. The DB has very fast read and write performance designed for NVME flash storage. I created this simple project just to test out linxGnu’s grocksdb implementation of a go wrapper, as RocksDB is developed in C++.
Building using Docker
Initially I had some issues with
glibc and other libraries conflicting with what I had on my local machine vs what was installed in my cloud node. To simplify things I used Docker to build the libraries and binaries so that they would be consistent with that environment.
RocksDB can take a while to compile so order is important in your Dockerfile, you don’t want it to have to recompile with each change of your code.
FROM ubuntu:20.04 COPY install.sh /app/install.sh RUN apt update && apt install -y \ build-essential \ git \ curl \ wget \ gcc RUN /app/install.sh RUN wget -P \ /tmp \ https://dl.google.com/go/go1.19.linux-amd64.tar.gz RUN tar -C /usr/local -xzf \ /tmp/go1.19.linux-amd64.tar.gz RUN rm /tmp/go1.19.linux-amd64.tar.gz RUN apt install -y \ libgflags-dev \ libsnappy-dev \ zlib1g-dev \ libbz2-dev \ liblz4-dev \ libzstd-dev ENV GOPATH /go ENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH COPY . /app WORKDIR "/app" RUN go mod download RUN go build -tags builtin_static \ -o srtl ./cmd/http
Testing It Out
RocksDB has sensible defaults if you’re just testing it out. It even has functionality for a TTL, although this is more of a minimum TTL, items only get dropped during the compaction stage so you can’t depend on it as a timer. But in the case of a link shortener with short lived urls, the url functioning until the next compaction is not an issue.
bbto := grocksdb.NewDefaultBlockBasedTableOptions() bbto.SetBlockCache(grocksdb.NewLRUCache(3 << 30)) opts := grocksdb.NewDefaultOptions() opts.SetBlockBasedTableFactory(bbto) opts.SetCreateIfMissing(true) db, err := grocksdb.OpenDbWithTTL(opts, "./db", 86400)
The benefits of using a key value store is once initialized the DB is incredibly easy to use. With simple Put and Get functions. Keys and values are pure byte streams, theres no limit to their size but in my case the keys are only 5ish characters long.
err = db.Put(grocksdb.NewDefaultWriteOptions(), key, value) val, err = db.Get(grocksdb.NewDefaultReadOptions(), key)
Creating a systemd Service
Now that I had a working binary I wanted to use systemd to keep it running. I used the following config:
[Unit] Description=SRTL Go Service ConditionPathExists=/Path/To/Binary/srtl After=network.target [Service] Type=simple User=james Group=james WorkingDirectory=/Path/To/Binary/ ExecStart=/Path/To/Binary/srtl Restart=on-failure RestartSec=10 StandardOutput=syslog StandardError=syslog SyslogIdentifier=srtlgoservice [Install] WantedBy=multi-user.target
Then I ran the below commands to register the service.
sudo systemctl daemon-reload sudo service srtl start sudo service srtl status sudo systemctl enable srtl sudo systemctl start srtl
In my case I had some environment variables that I wanted to be set, in that case you need to create a file in
/etc/systemd/system/srtl.service.d/myenv.conf with the following variables set.
[Service] Environment=HOST=srtl.ie Environment=PORT=8080 Environment=LD_LIBRARY_PATH=/lib:/usr/lib:/usr/local/lib
Rocksdb vs Pebble?
During my research about RocksDb I found a post by CockroachDB who are moving away from RocksDb and replacing it with pebble which was inspired by RocksDb/LevelDb written in go. It has a subset of features but at least from my testing has improved read and write speeds. If you’re comfortable with the different feature set its worth having a look. It is also far easier to manage as you don’t have to deal with compiling libraries.
RocksDb Pebble Set_Key: 4.18µs ± 5% 0.93µs ± 6% -77.77% (p=0.008 n=5+5) Get_Key: 1.86µs ± 4% 1.59µs ± 7% -14.82% (p=0.008 n=5+5)