How to install Cumin

From ppwiki
Jump to: navigation, search

Cumin is open Source program written in Python that allows you to execute multiple commands on multiple hosts in parallel. For more information on Cumin please see the references section.

Prerequisites

For this tutorial, you need:

- pupetmaster node

- puppetdb node

- cumin node

- some nodes for testing

Environment

Nodes names

All nodes in this tutorial are running Debian Stretch with Puppet 6

- Puppetmaster1002 : 10.192.32.4 Puppet master node

- Puppetdb1002 : IP address 10.192.32.10 PuppetDb node

- cumin1001 : IP address 10.192.48.13 Running cumin

- lab1001: Test node

Installation

One of the required package for Cumin 3.0.2 is python3-clustershell. Debian Stretch doesn't have that package so you will need to download it manually on your puppetmaster and the cumin class will install it. ( See cumin class below)

ppaul@cumin1001:~$ sudo apt-get install python3-clustershell
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package python3-clustershell

You can also install Debian Buster if you want on the cumin node since Debian Buster has the package.

- Test from a node running Debian Buster

ppaul@lab1002:~$ sudo apt-get install python3-clustershell
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Suggested packages:
 vim-addon-manager
The following NEW packages will be installed:
 python3-clustershell
0 upgraded, 1 newly installed, 0 to remove and 1 not upgraded.
Need to get 126 kB of archives.
After this operation, 625 kB of additional disk space will be used.
Get:1 http://apt1001.dfw.labnet/debian buster/main amd64 python3-clustershell all 1.8.1-1 [126 kB]
Fetched 126 kB in 1s (249 kB/s)          
Selecting previously unselected package python3-clustershell.
(Reading database ... 57033 files and directories currently installed.)
Preparing to unpack .../python3-clustershell_1.8.1-1_all.deb ...
Unpacking python3-clustershell (1.8.1-1) ...
Setting up python3-clustershell (1.8.1-1) ...

cumin Class

On your puppetmaster create the cumin class

cd /etc/puppetlabs/code/environments/production/modules
sudo mkdir cumin
cd cumin
sudo mkdir manifests
sudo mkdir templates
cd manifests

Create the master.pp file, copy and paste the contain below

class profile::cumin::master (
   $puppetdb_host  = hiera('puppetdb_host'),
) {
   $cumin_log_path = '/var/log/cumin'
     require_package([
       'python3-dnspython',
       'python3-colorama',
       'python3-pyparsing',
       'python3-tqdm',
       'python3-requests',
       'python3-yaml',
   ])
   file { '/srv/cumin_3.0.2-1_amd64.deb':
     ensure    => present,
     owner      => 'root',
     group      => 'root',
     source     => 'puppet:///modules/cumin/cumin_3.0.2-1_amd64.deb',
   }
   file { '/srv/python3-clustershell_1.8.1-2_all.deb':
     ensure     => present,
     owner      => 'root',
     group      => 'root',
     source     => 'puppet:///modules/cumin/python3-clustershell_1.8.1-2_all.deb',
   }
  package { 'python3-clusterhsell':
     ensure     => installed,
     provider   => dpkg,
     source     => '/srv/python3-clustershell_1.8.1-2_all.deb',
     require    => File['/srv/python3-clustershell_1.8.1-2_all.deb'],
  }
   package { 'cumin':
     ensure     => installed,
     provider   => dpkg,
     source     => '/srv/cumin_3.0.2-1_amd64.deb',
     require    => File['/srv/cumin_3.0.2-1_amd64.deb'],
   } 
   file { $cumin_log_path:
       ensure => directory,
       owner  => 'root',
       group  => 'root',
       mode   => '0750',
   } 
file { '/etc/cumin':
       ensure => directory,
       owner  => 'root',
       group  => 'root',
       mode   => '0755',
   }
  file { '/etc/cumin/config.yaml':
       ensure  => present,
       owner   => 'root',
       group   => 'root',
       mode    => '0640',
       content => template('cumin/config.yaml.erb'),
       require => File['/etc/cumin'],
   }
   file { '/etc/cumin/config-installer.yaml':
       ensure  => present,
       owner   => 'root',
       group   => 'root',
       mode    => '0640',
       content => template('cumin/config-installer.yaml.erb'),
       require => File['/etc/cumin'],
   }
   file { '/etc/cumin/aliases.yaml':
       ensure  => present,
       owner   => 'root',
       group   => 'root',
       mode    => '0644',
       content => template('cumin/aliases.yaml.erb'),
       require => File['/etc/cumin'],
   }
if os_version('debian == stretch') {
       $python_version = '3.5'
   } else {
       $python_version = '3.6'
   }
}

File needed

Download cumin_3.0.2-1_amd64.deb and python3-clustershell_1.8.1-2_all.deb and save those files in /etc/puppetlabs/code/environments/production/modules/cumin/files

http://ftp.us.debian.org/debian/pool/main/c/clustershell/python3-clustershell_1.8.1-2_all.deb
https://github.com/wikimedia/cumin/releases/download/v3.0.2/cumin_3.0.2-1_amd64.deb

Create aliases.yaml.erb, config-installer.yaml.erb and config.yaml.erb in /etc/puppetlabs/code/environments/production/modules/cumin/templates For now we are going just to focus on the config.yaml.erb file. Open the config.yaml.erb and copy and paste the contain below. Leave the other files empty for now.

transport: clustershell
 log_file: <%= @cumin_log_path %>/cumin.log
default_backend: puppetdb
environment:
    SSH_AUTH_SOCK: /run/keyholder/proxy.sock
puppetdb:
    host: <%= @puppetdb_host %>
    port: 443
    api_version: 4
<% if scope.function_os_version(['debian >= stretch']) -%>
    urllib3_disable_warnings:
      - SubjectAltNameWarning
<% end -%>
knownhosts:
    files:
        - /etc/ssh/ssh_known_hosts
clustershell:
    ssh_options:
        - '-o BatchMode=yes'
        - '-o ConnectTimeout=2'
        - '-o StrictHostKeyChecking=yes'

Configure hiera

-Add puppetdb_host to cumin1001.yaml in hiera

puppetdb_host: puppetdb1002.dfw.labnet

-Add cumin IP address to common.yaml in hiera

 cumin_masters:
- 10.192.48.13

Apply the cumin class

Open your site.pp file and apply the cumin class to your cumin node

node 'cumin1001.dfw.labnet' {
       include cumin::master
}

Install Cumin

On the cumin node run

sudo puppet agent -t

verification

navigate to /usr/lib/python3/dist-packages/ and make sure that the cumin folder is there

ppaul@cumin1001:/usr/lib/python3/dist-packages/cumin$ ls -l
total 60
drwxr-xr-x 3 root root  4096 Oct  5 12:56 backends
-rw-r--r-- 1 root root 19656 Jul 30  2018 cli.py
-rw-r--r-- 1 root root  6051 Jul 30  2018 grammar.py
-rw-r--r-- 1 root root  4852 Jul 30  2018 __init__.py
drwxr-xr-x 2 root root  4096 Oct  5 12:56 __pycache__
-rw-r--r-- 1 root root  6877 Jul 30  2018 query.py
-rw-r--r-- 1 root root  2068 Jul 30  2018 transport.py
drwxr-xr-x 3 root root  4096 Oct  5 12:56 transports

We have now Cumin installed on our node. The next setup is to configure our cumin node to talk to the puppetdb node

Configuration

Firewall

On the puppetdb node puppetdb1002 only the puppetmaster node and the icinga node are allowed to talk to the puppetdb node.

8081/tcp                   ALLOW       10.192.16.12      #allow icinga1002.dfw.labnet        
8081/tcp                   ALLOW       10.192.32.4       #allow puppetmaster1002.dfw.labnet

We are going to allow cumin1001.dfw.labnet to talk to the puppetdb node

8081/tcp                   ALLOW       10.192.48.13 

-Testing

ppaul@cumin1001:/srv$ telnet puppetdb1002.dfw.labnet 8081
Trying 10.192.32.10...
Connected to puppetdb1002.dfw.labnet.
Escape character is '^]'.

We see that our cumin is allow to connect on port 8081 on the puppetdb node

Puppetdb 6 and Cumin ssl issue

Right now we have cumin node is allow to connect to the puppetdb node. Let try to see if we can run a cumin command

sudo cumin -b 1 -s 5 'lab1001.dfw.labnet' 'run-puppet-agent'

Result

Caught SSLError exception: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720)

Let us try to curl from our cumin node to the puppetdb node

 ppaul@cumin1001:~$ sudo curl -vG 'https://puppetdb1002.dfw.labnet:8081/pdb/query/v4/facts'
*   Trying 10.192.32.10...
* TCP_NODELAY set
* Connected to puppetdb1002.dfw.labnet (10.192.32.10) port 8081 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: unable to get local issuer certificate
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

We can see that the puppetdb node is not allowing any connection without a certificate.

Let us try to run the same command now by bypassing the certificate. just add -k to the previous command

ppaul@cumin1001:~$ sudo curl -vkG 'https://puppetdb1002.dfw.labnet:8081/pdb/query/v4/facts'
*   Trying 10.192.32.10...
* TCP_NODELAY set
* Connected to puppetdb1002.dfw.labnet (10.192.32.10) port 8081 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS alert, Server hello (2):
* error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0
curl: (35) error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate

Same problem we can not bypass the certificate verification

According to the PuppetDB documentation in the link below, to make secured requests from other hosts, you will need to supply

--cacert
--cert
--key
https://puppet.com/docs/puppetdb/5.2/api/query/curl.html

Let us try to curl again but this time we we follow the documentation. create the bash script below

#!/bin/bash
curl -vG 'https://puppetdb1002.dfw.labnet:8081/pdb/query/v4/facts' \
 --tlsv1 \
 --cacert /etc/puppetlabs/puppet/ssl/certs/ca.pem \
 --cert /etc/puppetlabs/puppet/ssl/certs/cumin1001.dfw.labnet.pem \
 --key /etc/puppetlabs/puppet/ssl/private_keys/cumin1001.dfw.labnet.pem

make it excutable and run it


 ppaul@cumin1001:~$ sudo ./puppetdbquery 
*   Trying 10.192.32.10...
* TCP_NODELAY set
* Connected to puppetdb1002.dfw.labnet (10.192.32.10) port 8081 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/puppetlabs/puppet/ssl/certs/ca.pem
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / DHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=puppetdb1002.dfw.labnet
*  start date: Sep 30 00:52:53 2019 GMT
*  expire date: Sep 29 00:52:53 2024 GMT
*  common name: puppetdb1002.dfw.labnet (matched)
*  issuer: CN=Puppet CA: puppetmaster1002.dfw.labnet
*  SSL certificate verify ok.
> GET /pdb/query/v4/facts HTTP/1.1
> Host: puppetdb1002.dfw.labnet:8081
> User-Agent: curl/7.52.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Fri, 04 Oct 2019 02:33:28 GMT
< Content-Type: application/json;charset=utf-8
< Vary: Accept-Encoding, User-Agent
< Transfer-Encoding: chunked

Now we are able to connect to our puppetdb node with out a problem

What is the fix for now?

The fix for now is to install a SSL proxy using Apache2 or Nginx on the puppetdb node. I decided to use Nginx in my case.

I am not going to discuss about how to install and configure Nginx here but only provide the Nginx config file if that will help

- Nginx config file

server {
       listen 443 ssl http2;
       listen [::]:443 ssl http2;
       server_name <%= @server_name %>;
       root /usr/share/nginx/www;
       ssl_certificate /etc/ssl/certs/wildcard.tx.labnet.crt;
       ssl_certificate_key /etc/ssl/private/wildcard.tx.labnet.key;
       add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload";
       location / {
               proxy_pass http://localhost:8080/;
       }
}

Replace <%= @server_name %>; with your puppetdb node name Note: on the puppetdb node allow the cumin node to connect to port 443

443/tcp                    ALLOW       10.192.48.13 

We are going to run again the same curl command from the cumin node by bypassing the certificate verification

ppaul@cumin1001:~$ curl -vkG 'https://puppetdb1002.dfw.labnet:443/pdb/query/v4/facts'
*    Trying 10.192.32.10...
* TCP_NODELAY set
* Connected to puppetdb1002.dfw.labnet (10.192.32.10) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=TX; L=DAllas; O=IT
*  start date: Feb  6 03:18:24 2019 GMT
*  expire date: Feb  6 03:18:24 2020 GMT
*  issuer: C=US; ST=TX; L=DAllas; O=IT
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5636b9310d00)
> GET /pdb/query/v4/facts HTTP/1.1
> Host: puppetdb1002.dfw.labnet
> User-Agent: curl/7.52.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200 
< server: nginx/1.17.4
< date: Thu, 10 Oct 2019 02:09:41 GMT
< content-type: application/json;charset=utf-8
< vary: Accept-Encoding, User-Agent
< strict-transport-security: max-age=15768000; includeSubDomains; preload

Testing

Run the same cumin command

sudo cumin -b 1 -s 5 'lab1001.dfw.labnet' 'puppet agent -t'

Result

Caught SSLError exception: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720)

Disable certificate verification in cumin

open

/usr/lib/python3/dist-packages/cumin/backends/pupeptdb.py$

change line 503 from

503             resources = requests.get(self.url + self.endpoint, params={'q    uery': query}, verify=True)

to

503             resources = requests.get(self.url + self.endpoint, params={'q    uery': query}, verify=False)

change line 505 from

505             resources = requests.post(self.url + self.endpoint, json={'qu    ery': query}, verify=True)

to

505             resources = requests.post(self.url + self.endpoint, json={'qu    ery': query}, verify=False)

Run the command again

sudo cumin -b 1 -s 5 'lab1001.dfw.labnet' 'puppet agent -t'

Result

ppaul@cumin1001:~$ sudo cumin -b 1 -s 5 'lab1001.dfw.labnet' 'puppet agent -t'
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:845: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See:  https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
1 hosts will be targeted:
lab1001.dfw.labnet
Confirm to continue [y/n]? y
----- OUTPUT of 'run-puppet-agent -q' -----                                  
No ECDSA host key is known for lab1001.dfw.labnet and you have requested strict checking.                                                                 
Host key verification failed.                                                
================                                                             
PASS:  |                                |   0% (0/1) [00:05<?, ?hosts/s]     
FAIL:  |████████████████████████| 100% (1/1) [00:05<00:00,  7.74hosts/s]     
100.0% (1/1) of nodes failed to execute command 'run-puppet-agent -q': lab1001.dfw.labnet
0.0% (0/1) success ratio (< 100.0% threshold) for command: 'run-puppet-agent -q'. Aborting.
0.0% (0/1) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.

How to fix 1?

The ECDSA error is very easy and can be fix quickly by just running

sudo su
ssh lab1001.dfw.labnet 

from the cumin node and answering yes to add the ECDSA key fingerprint

Run the command again

ppaul@cumin1001:~$ sudo cumin -b 1 -s 5 'lab1001.dfw.labnet' 'puppet agent -t'
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:845: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See:    https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
1 hosts will be targeted:
lab1001.dfw.labnet
Confirm to continue [y/n]? y
----- OUTPUT of 'run-puppet-agent -q' -----                                  
Permission denied (publickey).                                               
================                                                             
PASS:  |                                |   0% (0/1) [00:05<?, ?hosts/s]     
FAIL:  |████████████████████████| 100% (1/1) [00:05<00:00,  9.26hosts/s]     
100.0% (1/1) of nodes failed to execute command 'run-puppet-agent -q': lab1001.dfw.labnet
0.0% (0/1) success ratio (< 100.0% threshold) for command: 'run-puppet-agent -q'. Aborting.
0.0% (0/1) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.

How to fix 2?

-Fix 1: The faster way to fix this is to comment or remove line 25 in /etc/cumin/config.yaml

21 clustershell:
22     ssh_options:
23         - '-o BatchMode=yes'
24         - '-o ConnectTimeout=2'
25        # - '-o StrictHostKeyChecking=yes'

- Fix 2: Create a ssk-key for the user root and copy the public key to the remote host (lab1001.dfw.labent) This method really will not work well if you have a lot of nodes.

- Fix 3: The way I fixed this is to use the class target.pp from https://github.com/wikimedia/puppet/blob/production/modules/profile/manifests/cumin/target.pp

As you can see in the target.pp class on line 32 you have

ssh::userkey

so you will need also the ssh userkey.pp class from https://github.com/wikimedia/puppet/blob/production/modules/ssh/manifests/userkey.pp

Run the command again

ppaul@cumin1001:~$ sudo cumin -b 1 -s 5 'lab1001.dfw.labnet' 'puppet agent -t'
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:845: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See:  https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
1 hosts will be targeted:
lab1001.dfw.labnet
Confirm to continue [y/n]? y
----- OUTPUT of 'puppet agent -t' -----                                      
Info: Using configured environment 'production'                              
Info: Retrieving pluginfacts                                                 
Info: Retrieving plugin                                                      
Info: Retrieving locales                                                     
Info: Loading facts                                                          
Info: Caching catalog for lab1001.dfw.labnet                                 
Info: Applying configuration version '1570416225'                            
Notice: Applied catalog in 4.76 seconds                                      
================                                                             
PASS:  |████████████████████████| 100% (1/1) [00:19<00:00, 14.63s/hosts]     
FAIL:  |                                |   0% (0/1) [00:19<?, ?hosts/s]     
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'puppet agent -t'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.

Running commands

  • Run apt-get update on node lab1001.dfw.labnet
ppaul@cumin1001:~$ sudo cumin -b 1 -s 5 'lab1001.dfw.labnet' 'apt-get update'
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:845: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See:  https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
1 hosts will be targeted:
lab1001.dfw.labnet
Confirm to continue [y/n]? y
----- OUTPUT of 'apt-get update' -----                                       
Ign:1 http://apt1001.dfw.labnet/debian stretch InRelease                     
Hit:2 http://apt1001.dfw.labnet/debian stretch Release                       
Hit:3 http://deb.debian.org/debian stretch-backports InRelease               
Hit:4 http://security.debian.org/debian-security stretch/updates InRelease   
Hit:5 http://security.debian.org stretch/updates InRelease                   
Hit:7 http://ams2.mirrors.digitalocean.com/mariadb/repo/10.2/debian stretch InRelease                                                                     
Hit:8 http://apt.puppetlabs.com stretch InRelease                            
Hit:9 https://packages.sury.org/php stretch InRelease                        
Reading package lists...                                                     
================                                                             
PASS:  |████████████████████████| 100% (1/1) [00:07<00:00,  2.93s/hosts]     
FAIL:  |                                |   0% (0/1) [00:07<?, ?hosts/s]     
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'apt-get update'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
  • Run apt-get update on all node with host name starting with db1
 ppaul@cumin1001:~$ sudo cumin -b 1 -s 5 'db1*.dfw.labnet' 'apt-get update'
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:845: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See:  https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
7 hosts will be targeted:
db[1001-1007].dfw.labnet
Confirm to continue [y/n]? y
===== NODE GROUP =====                                                       
(1) db1007.dfw.labnet                                                        
----- OUTPUT of 'apt-get update' -----                                       
Hit:1 http://apt1001.dfw.labnet/debian buster InRelease                      
Hit:2 http://security.debian.org/debian-security buster/updates InRelease    
Hit:3 http://apt.puppetlabs.com stretch InRelease
Hit:4 http://security.debian.org buster/updates InRelease
Hit:5 http://apt.puppetlabs.com buster InRelease
Reading package lists...
===== NODE GROUP =====                                                       
(1) db1004.dfw.labnet                                                        
----- OUTPUT of 'apt-get update' -----                                       
Ign:1 http://apt1001.dfw.labnet/debian stretch InRelease                     
Hit:2 http://apt1001.dfw.labnet/debian stretch Release                       
Hit:3 http://security.debian.org/debian-security stretch/updates InRelease
Hit:4 http://deb.debian.org/debian stretch-backports InRelease
Hit:5 http://security.debian.org stretch/updates InRelease
Hit:6 http://repo.saltstack.com/py3/debian/9/amd64/2018.3 stretch InRelease
Hit:7 http://apt.puppetlabs.com stretch InRelease
Reading package lists...
===== NODE GROUP =====                                                       
(1) db1005.dfw.labnet                                                        
----- OUTPUT of 'apt-get update' -----                                       
Ign:1 http://apt1001.dfw.labnet/debian stretch InRelease                     
Hit:2 http://apt1001.dfw.labnet/debian stretch Release                       
Hit:3 http://security.debian.org/debian-security stretch/updates InRelease
Hit:4 http://deb.debian.org/debian stretch-backports InRelease
Hit:5 http://security.debian.org stretch/updates InRelease
Hit:6 http://apt.puppetlabs.com stretch InRelease
Reading package lists...
===== NODE GROUP =====                                                       
(1) db1001.dfw.labnet                                                        
----- OUTPUT of 'apt-get update' -----                                       
Ign:1 http://apt1001.dfw.labnet/debian stretch InRelease                     
Hit:2 http://apt1001.dfw.labnet/debian stretch Release                       
Hit:3 http://repo.saltstack.com/py3/debian/9/amd64/2018.3 stretch InRelease
Hit:4 http://apt.puppetlabs.com stretch InRelease
Hit:6 http://security.debian.org/debian-security stretch/updates InRelease
Hit:7 http://security.debian.org stretch/updates InRelease
Hit:8 http://deb.debian.org/debian stretch-backports InRelease
Reading package lists...
===== NODE GROUP =====                                                       
(1) db1002.dfw.labnet                                                        
----- OUTPUT of 'apt-get update' -----                                       
Ign:1 http://apt1001.dfw.labnet/debian stretch InRelease                     
Hit:2 http://apt1001.dfw.labnet/debian stretch Release                       
Hit:3 http://deb.debian.org/debian stretch-backports InRelease
Hit:4 http://security.debian.org/debian-security stretch/updates InRelease
Hit:5 http://security.debian.org stretch/updates InRelease
Hit:7 http://apt.puppetlabs.com stretch InRelease
Reading package lists...
===== NODE GROUP =====                                                       
(2) db[1003,1006].dfw.labnet                                                 
----- OUTPUT of 'apt-get update' -----                                       
Ign:1 http://apt1001.dfw.labnet/debian stretch InRelease                     
Hit:2 http://apt1001.dfw.labnet/debian stretch Release                       
Hit:3 http://deb.debian.org/debian stretch-backports InRelease
Hit:4 http://security.debian.org/debian-security stretch/updates InRelease
Hit:5 http://security.debian.org stretch/updates InRelease
Hit:6 http://repo.saltstack.com/py3/debian/9/amd64/2018.3 stretch InRelease
Hit:7 http://apt.puppetlabs.com stretch InRelease
Reading package lists...
================                                                             
PASS:  |████████████████████████| 100% (7/7) [00:53<00:00,  7.06s/hosts]    
FAIL:  |                                |   0% (0/7) [00:53<?, ?hosts/s] 
100.0% (7/7) success ratio (>= 100.0% threshold) for command: 'apt-get update'.
100.0% (7/7) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.

References

https://doc.wikimedia.org/cumin/master/installation.html

https://puppet.com/docs/puppetdb/5.2/api/query/curl.html

https://github.com/wikimedia/puppet/blob/production/modules/ssh/manifests/userkey.pp

http://ftp.us.debian.org/debian/pool/main/c/clustershell/