Overview

The Seldon deploy operator allows the creation of Seldon Deploy applications to provide management, monitoring and updating of machine learning models from a range of technologies. It includes the ability to run explanations as well as outlier, drift and adversarial detection against running models. There is a full metrics and logging integration via Prometheus and ELK stacks as well as full GitOps operations to allow full auditing and recovery of updated models.

Integrations to some other tools require those tools to be installed first. See the pre-requisites section for steps to install tools used by Seldon Deploy.

Installation Instructions

Before installing Seldon Deploy, it is recommended to install and configure the dependencies. The tools suggested are all supported by RedHat.

Service Mesh - Required

Ingress functionality is needed to support traffic-splitting on model deployments. The default choice is istio. Install the OpenShift Service Mesh operator by following the RedHat documentation for your OpenShift version. Follow the documentation up to creating a control plane (we’ll then create a project). Here is the documentation for OpenShift 4.7 - linked. Note Jaeger and Kiali are optional and ElasticSearch will be installed later.

Wait until the Mesh Control Plane is created and the pods are up in the istio-system namespace, as per the documentation. Then create a new namespace (project) called Seldon:

oc create namespace seldon

oc label namespace seldon seldon.restricted=false --overwrite=true

And add ‘seldon’ and ‘openshift-user-workload-monitoring’ to the istio Member Roll from the OpenShift UI in ‘Installed Operators’ under ‘istio-system’.

Serverless - Recommended

Seldon Deploy’s request logging features use asynchronous queue processing. This leverages Knative Eventing.

Follow the installation instructions for the Serverless Operator for your version of OpenShift.

When the operator itself is installed then install knative-eventing in the knative-eventing namespace

Create a seldon-logs namespace:

oc create namespace seldon-logs

Then run:

kubectl create namespace seldon-logs
kubectl create -f - <<EOF
apiVersion: eventing.knative.dev/v1
kind: Broker
metadata:
 name: default
 namespace: seldon-logs
EOF

The above installs and configures knative-eventing.

Next install knative-serving in the knative-serving namespace:
https://docs.openshift.com/container-platform/4.7/serverless/admin_guide/installing-knative-serving.html#installing-knative-serving

Elasticsearch - Recommended

Elasticsearch is used for logging requests to machine learning models. Logs can also be indexed for debugging. Elastic Cloud on Kubernetes (ECK) is used for this. You can check the installation instructions for any pre-requisites but likely you don’t need to do much. You can use operatorhub to install the operator.

Install the ECK operator in the openshift-operators namespace with the All Namespaces option. Not the Elasticsearch by Red Hat operator as that is not for use in this tool

Create a seldon-logs namespace if not done so already:
oc create namespace seldon-logs

Go to the Elastic operator under ‘Installed Operators’ in the seldon-logs namespace. Go to ‘Elasticsearch Cluster’ and hit ‘Create Elasticsearch’. We can use a manifest from the elastic docs, changing the name here to elasticsearch-seldon (note this is a single-node elastic - customize if desired):

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-seldon
spec:
  version: 7.12.0
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false

We’ll need to obtain the password in order to connect to elastic (note the secret name depends on the above name):

ELASTIC_PASSWORD=$(kubectl get secret elasticsearch-seldon-es-elastic-user -n seldon-logs -o go-template='{{.data.elastic | base64decode}}')

We’ll need to create secrets in the elastic and seldon namespaces. The username in the original elastic secret is ‘elastic’:

kubectl create secret generic elastic-credentials -n seldon-logs \
  --from-literal=username="elastic" \
  --from-literal=password="${ELASTIC_PASSWORD}" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic elastic-credentials -n seldon \
  --from-literal=username="elastic" \
  --from-literal=password="${ELASTIC_PASSWORD}" \
  --dry-run=client -o yaml | kubectl apply -f -

Cluster Monitoring - Recommended

Seldon deploy has integrated dashboards for monitoring performance of models with Prometheus. For OpenShift 4.7 you first need to configure Cluster Monitoring:

kubectl create -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true 
    prometheusK8s: 
      retention: 30d
EOF

And then User Workfload Monitoring (note this can be customized):

kubectl create -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus: 
      retention: 300d 
      resources:
        requests:
          cpu: 200m 
          memory: 2Gi 
EOF

To confirm you should now see Pods returned by:

kubectl get pod -n openshift-user-workload-monitoring

We’ll need a token in order to connect to prometheus. To generate this, run the following commands as a cluster-admin user:

oc create serviceaccount -n seldon seldon
oc adm policy add-cluster-role-to-user cluster-admin system:serviceaccount:seldon:seldon
JWT=$(oc serviceaccounts get-token -n seldon seldon)
echo -n $JWT > ./jwt-seldon.txt
oc create secret generic jwt-seldon -n seldon --from-file=./jwt-seldon.txt --dry-run -o yaml | kubectl apply -f -

Seldon and the cluster monitoring setup then need access to each other:

kubectl create -f - <<EOF
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
 name: allow-all-namespace
 namespace: seldon
spec:
 podSelector: {}
 ingress:
   - from:
       - namespaceSelector: {}
         podSelector: {}
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
 name: allow-all-namespace
 namespace: openshift-user-workload-monitoring
spec:
 podSelector: {}
 ingress:
   - from:
       - namespaceSelector: {}
         podSelector: {}
EOF

Seldon Deploy

We can now install the Seldon Deploy operator from the Marketplace, using the openshift-operators namespace.

Once the operator is installed, go to Installed Operators in the Seldon project and choose the Seldon Deploy Operator (not the Seldon Core Operator, which will have also been installed automatically). From here you can create an instance of Seldon Deploy. The default configuration for an instance assumes that all of the above steps for required and recommended dependencies have been followed, using the recommended configurations.

If some of the recommended dependencies have been skipped then the Json specifying the instance should be modified:

If the Cluster Monitoring component has not been configured then set openshiftMarketplace.prometheus.monitorSpecs.create to false and prometheus.seldon.jwtSecretName to "".
If either of the Cluster Logging or Serverless components have not been installed then set requestLogger.create to false and elasticsearch.basicAuth to false and if present set elasticsearch.jwtSecretName to "" (if not present then add or skip this one). Also create the seldon-logs namespace.

With Seldon Deploy running, you can find its URL:

ISTIO_INGRESS=$(oc get route -n istio-system istio-ingressgateway -o jsonpath='{.spec.host}'); \ echo http://$ISTIO_INGRESS/seldon-deploy/

You can then run one of the demos (note these require access to google storage buckets). The Batch demo will not work as it requires components not yet available on Openshift Marketplace.

Components Not Covered

Some components used by Seldon Deploy for specific demos (e.g. Batch) are not yet available on Openshift Marketplace. Please contact Seldon if you have questions on these.

See the Seldon Deploy documentation for other configurations such as Gitops and Authentication. Ask further questions in the Seldon community slack.

Seldon Deploy

Seldon Deploy

Software version

Runs on

Delivery method

Rating

Product documentation