Background
If you’re aiming to set up OSS RabbitMQ in Kubernetes, you’ve come to the right place. This guide aims to streamline the installation process, integrating best practices and providing a high-level overview of the necessary steps. However, for a more granular breakdown and additional guidance, it’s advisable to consult the official documentation at Installing RabbitMQ Cluster Operator in a Kubernetes Cluster, which also served as a reference for this article.
Note: In this guide, we specifically focus on the Cluster Operator-based RabbitMQ installation.
There’s also a commercial version of RabbitMQ available from VMware, named VMware RabbitMQ for Kubernetes, previously known as VMware Tanzu RabbitMQ for Kubernetes. This version offers some unique features not found in the open-source variant, enhanced by VMware’s engineering team. Notably, it includes features like Warm Standby Replication and Intra-cluster Compression that are designed to enhance its functionality. For more information, you can refer to the official documentation. If you’re interested in exploring this further, I’ve also penned an article, Install VMware RabbitMQ in Kubernetes, detailing the installation process.
Now, let’s dive into the detailed installation steps for OSS RabbitMQ.
Note: The guidelines provided here are tailored to RabbitMQ version 3.13.0, which is the latest version at the time of writing this article.
Prerequisites
- Account at Docker Hub
- Kubernetes cluster with required operator privileges for installation
- Cluster Essentials for VMware Tanzu installed in the Kubernetes cluster.
- Carvel Tools installed
- Kubectl installed
- kapp installed
Getting Started
Below is a summary of the Kubernetes resources required to complete the installation, with a more detailed exploration to follow:
- Namespaces
- Secretgen Controller - A tool for managing secrets dynamically.
- Secrets, SecretImports, and SecretExports - Mechanisms for managing and importing secrets across different namespaces.
- RabbitMQ Cluster Operator - The operator simplifies the deployment and management of RabbitMQ clusters within Kubernetes.
- RabbitMQCluster - Custom resource definitions (CRDs) for deploying RabbitMQ instances.
Detailed Steps
Let’s delve into each of the components one-by-one in detail.
Namespaces: To organize your Kubernetes environment for the OSS RabbitMQ installation, run the following commands in your terminal. This script will create two namespaces: one for secrets management and another for the rabbitmq server cluster. These namespaces help segregate resources and manage access more efficiently within your Kubernetes cluster.
kapp deploy -a oss-rabbitmq-namespaces -y -f - <<EOF
apiVersion: v1
kind: Namespace
metadata:
name: generic-secrets
labels:
name: generic-secrets
---
apiVersion: v1
kind: Namespace
metadata:
name: rabbitmq-clusters
labels:
name: rabbitmq-clusters
EOF
Secretgen Controller: To install the Secretgen Controller in your Kubernetes cluster, which is essential for managing secrets dynamically across namespaces, use the following command in your terminal. This controller simplifies secret creation, update, and management within the cluster. This command fetches and applies the latest Secretgen Controller release directly from its official GitHub repository, ensuring you have the most up-to-date version installed in your cluster.
kubectl apply -f https://github.com/carvel-dev/secretgen-controller/releases/latest/download/release.yml
Secrets, SecretImports, and SecretExports: To facilitate the sharing of secrets across different namespaces within your Kubernetes cluster, follow the steps below. This setup involves creating a secret within a namespace, exporting it, and then importing it into the target namespace(s). Here, we demonstrate how to create and manage a registry-credentials-secret that is exported from the generic-secrets namespace to the rabbitmq-clusters namespace.
Note (1): Replace the
dockerhub usernameanddockerhub passwordwith the Docker Hub credentials.
Note (2): If your container images are hosted in a registry other than Docker Hub, update the registry url to the address of your registry and also ensure the credentials provided are valid for that registry.
kapp deploy -a oss-rabbitmq-secrets -y -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
name: registry-credentials-secret
namespace: generic-secrets
labels:
name: registry-credentials-secret
type: kubernetes.io/dockerconfigjson
stringData:
.dockerconfigjson: |
{
"auths": {
"https://index.docker.io/v1/": {
"username": "dockerhub username",
"password": "dockerhub password",
"auth": ""
}
}
}
---
apiVersion: secretgen.carvel.dev/v1alpha1
kind: SecretExport
metadata:
name: registry-credentials-secret
namespace: generic-secrets
spec:
toNamespaces:
- "*"
---
apiVersion: secretgen.carvel.dev/v1alpha1
kind: SecretImport
metadata:
name: registry-credentials-secret
namespace: rabbitmq-clusters
spec:
fromNamespace: generic-secrets
EOF
RabbitMQ Cluster Operator: To install the RabbitMQ Cluster Operator in your Kubernetes cluster, run the following command in your terminal. This operator simplifies the deployment and management of RabbitMQ clusters, making it easier to scale and maintain your messaging system within Kubernetes.
Note: The operator will be installed in the rabbitmq-system namespace, a dedicated space for managing all resources associated with the RabbitMQ Operator.
kubectl apply -f "https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml"
To verify the availability of the RabbitMQ Cluster Operator and ensure it’s ready to manage RabbitMQ clusters, execute the following command in your terminal. This command lists all the Custom Resource Definitions (CRDs) installed in your Kubernetes cluster, allowing you to confirm the presence of the rabbitmqclusters.rabbitmq.com CRD, which indicates that the operator is successfully installed and operational.
kubectl get customresourcedefinitions.apiextensions.k8s.io
Seeing rabbitmqclusters.rabbitmq.com in the output confirms that the RabbitMQ Cluster Operator is properly set up and ready to manage RabbitMQ clusters in your Kubernetes environment.
Now that the RabbitMQ Cluster Operator is installed in your Kubernetes cluster, you’re ready to define the necessary configurations for your RabbitMQ cluster. By providing these configurations to the operator, you instruct it on how to create and manage the RabbitMQ cluster according to your specifications. This approach greatly simplifies the setup and ongoing maintenance of RabbitMQ within a Kubernetes environment.
To define a RabbitMQ cluster, you’ll need to create a YAML file that specifies the desired state of your RabbitMQ cluster, including the number of nodes, resource limits, storage options, and any other relevant configurations.
RabbitMQCluster: To create the RabbitMQ Server Cluster in your Kubernetes environment, follow the steps outlined below. This process involves defining a RabbitMQCluster resource that instructs the RabbitMQ Cluster Operator to deploy a RabbitMQ server cluster according to your specified configurations.
Note: Update
persistence:storageClassNamewith the correct value that matches the storage class available in your Kubernetes environment. This is crucial for ensuring that your RabbitMQ cluster has the required persistent storage for its data. Additionally, adjustpersistence:storageaccording to your capacity needs. This specifies the amount of persistent storage allocated to each RabbitMQ node in the cluster.
kapp deploy -a oss-rabbitmq-cluster -y -f - <<EOF
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
name: rabbitmq
namespace: rabbitmq-clusters
spec:
replicas: 3
imagePullSecrets:
- name: registry-credentials-secret
service:
type: LoadBalancer
resources:
limits:
cpu: "1.5"
memory: 2Gi
requests:
cpu: "1"
memory: 2Gi
rabbitmq:
additionalPlugins:
- rabbitmq_stream
- rabbitmq_stream_management
additionalConfig: |
cluster_partition_handling = pause_minority
cluster_formation.node_cleanup.interval = 10
vm_memory_high_watermark_paging_ratio = 0.65
vm_memory_high_watermark.relative = 0.7
disk_free_limit.relative = 1.0
collect_statistics_interval = 30000
queue_leader_locator = balanced
log.console.level = info
advancedConfig: ""
persistence:
storageClassName: "storage class name"
storage: 20Gi
override:
statefulSet:
spec:
template:
metadata:
annotations:
prometheus.io/port: "15692"
prometheus.io/scrape: "true"
EOF
Production Recommendation: For production environments, it’s highly recommended to enhance your deployment with the following configurations to ensure optimal operation and reliability.
Pod Anti-Affinity: Implementing pod anti affinity rules ensures that pods are distributed across different nodes. This practice enhances resilience and availability by reducing the likelihood that a single node failure will disrupt the service.
Priority Class: Assigning a priority class indicates the importance of the pods. By marking RabbitMQ pods as system critical, you prioritize their scheduling and resource allocation over lower-priority workloads. This helps maintain RabbitMQ’s performance and availability under resource contention.
Note: Ensure the
system-cluster-criticalpriority class exists in your Kubernetes cluster. If it doesn’t, create it or choose an appropriate alternative and update the configuration accordingly. This setup is pivotal for maintaining robust service levels in a production environment.
Incorporate these settings into your RabbitMQCluster configuration like so:
spec:
override:
statefulSet:
spec:
template:
spec:
priorityClassName: system-cluster-critical
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values:
- rabbitmq
topologyKey: kubernetes.io/hostname
Once your RabbitMQ cluster is successfully deployed, it will be ready to use within a few moments. Following this, you’ll want to obtain the external IP address and the credentials for accessing the RabbitMQ management interface or for connecting your applications to the RabbitMQ service. Here’s how you can retrieve this important information:
Obtaining the IP Address
To find out the external IP address assigned to your RabbitMQ cluster, use the following command. This IP address is crucial for accessing the RabbitMQ management UI or for application connections:
kubectl get svc rabbitmq -n rabbitmq-clusters -o jsonpath="{.items[0].status.loadBalancer.ingress[0].ip}"
This command lists services in the rabbitmq-clusters namespace (or your specified namespace if different), filtering for rabbitmq and then displaying the external IP address.
Obtaining the Default Username and Password
RabbitMQ provides default credentials upon installation, which you can retrieve as follows:
Username: Execute the command below to get the default username:
kubectl -n rabbitmq-clusters get secret rabbitmq-default-user -o jsonpath="{.data.username}" | base64 --decodePassword: Similarly, retrieve the default password using:
kubectl -n rabbitmq-clusters get secret rabbitmq-default-user -o jsonpath="{.data.password}" | base64 --decode
Note: To setup your own default user (here I used
guest), add the below underspec:rabbitmq:additionalConfig.
loopback_users.guest = false
default_user = guest
default_pass = guest
With the IP address and credentials at hand, you can now access the RabbitMQ management UI by navigating to http://:15672 in your web browser, using the obtained username and password for login.
Your applications can connect to the RabbitMQ service using the retrieved IP address on port 5672 for messaging operations. This setup should now be fully operational and ready to integrate with your applications.
To perform a quick connectivity test with your RabbitMQ cluster using the RabbitMQ PerfTest tool, you can use the Docker image provided by the RabbitMQ team. This test helps ensure that your RabbitMQ deployment is functioning correctly and can accept connections. Before running the test, ensure you have Docker installed on your machine and replace , , and with the actual values you obtained from your RabbitMQ setup:
docker run -it --rm pivotalrabbitmq/perf-test:latest --uri amqp://<username>:<password>@<IP Address>:5672 --id "connectivity test 1"
This command runs the PerfTest tool within a Docker container, attempting to connect to your RabbitMQ server using the specified credentials and IP address. The –id “connectivity test 1” part is optional and simply labels the test run for easier identification.
The output from this command will indicate whether the tool was able to successfully connect and interact with your RabbitMQ server. Successful connectivity suggests your RabbitMQ cluster is correctly configured and ready for further use and integration with your applications.
To streamline the management of the Cluster Operator directly from the command line, VMware provides a Kubectl plugin specifically designed for this purpose. This plugin extends kubectl with additional commands for RabbitMQ, making it easier to deploy and manage RabbitMQ clusters within your Kubernetes environment.
Advanced Authentication Configuration for RabbitMQ
To enhance the security and management of your RabbitMQ cluster, you might consider integrating advanced authentication mechanisms such as OAuth 2.0 or LDAP. These methods provide robust frameworks for managing access and permissions, offering a secure and scalable way to handle authentication and authorization.
OAuth 2.0 Integration
OAuth 2.0 offers a powerful standard for secure and flexible authentication. If you’re looking to integrate OAuth 2.0 with your RabbitMQ cluster, it can provide a seamless way to manage access tokens, granting different levels of access based on scopes and roles.
For a step-by-step tutorial on setting up OAuth 2.0 with RabbitMQ, check out Configure RabbitMQ with Oauth2.0. This article covers everything from setting up the OAuth 2.0 server to configuring RabbitMQ for OAuth 2.0 authentication.
LDAP Integration
LDAP is a well-established directory service protocol that allows for a centralized authentication mechanism. Integrating LDAP with RabbitMQ enables you to leverage your existing directory services for user authentication and authorization, streamlining user management and security.
For comprehensive instructions on integrating LDAP with your RabbitMQ cluster, visit Configure RabbitMQ with LDAP. This article provides detailed steps for configuring RabbitMQ to authenticate and authorize users based on LDAP directories.
Using the Kubectl RabbitMQ Plugin
If you are interested to know about using the kubectl plugin to manage the operator to sections Installing the Kubectl RabbitMQ Plugin and Using the Kubectl RabbitMQ Plugin for detailed instructions from article Install VMware RabbitMQ in Kubernetes
Uninstall
To uninstall the RabbitMQ cluster from your Kubernetes environment, you can delete the RabbitMQCluster resource you’ve created. This action will instruct the RabbitMQ Cluster Operator to remove all components associated with the RabbitMQ cluster, including any associated resources and configurations.
Execute the following command in your command shell, ensuring that it’s connected to your Kubernetes cluster:
kapp delete -y -a oss-rabbitmq-cluster
kubectl delete -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml
kapp delete -y -a oss-rabbitmq-secrets
kubectl delete -f https://github.com/carvel-dev/secretgen-controller/releases/latest/download/release.yml
kapp delete -y -a oss-rabbitmq-namespace
Github Repository
For those interested in deploying OSS RabbitMQ with a streamlined approach, leveraging a repository like Github:oss-rabbitmq-install-basic can significantly simplify the process. This repository contains templated code that utilizes ytt, a tool for generating YAML configurations from declared values, enabling customizable and repeatable deployments.
Advantages of Using the Repository
Simplicity: The repository abstracts away the complexity of manually creating inline YAML for RabbitMQ deployment.
Customization: With ytt, you can easily customize the deployment to meet your needs without editing YAML files directly.
Repeatability: Deployments can be repeated across different environments with minimal changes, ensuring consistency and saving time. By following the instructions and utilizing the templated configurations provided in the repository, you can efficiently deploy VMware RabbitMQ for Kubernetes, tailored to your specific requirements.
Additional References
Hope you had fun coding!
comments powered by Disqus