SaltStack – IT automation infrastructure management tools

Saltstack은 수백, 수천대의 노드에 특정 명령 및 쿼리를 명령어 한번으로 실행할수 있도록 도와주는 자동화 관리 도구 입니다.

 

SaltStack Architecture

SaltStack은 Master, Minion으로 이뤄져 있습니다.

– Master
Salt 명령어 실행과 상태등을 관리하는 중앙서버 입니다.
– Minion
Master에 대한 연결을 유지 및 명령을 기다리는 Agent라고 이해 하시면 됩니다.
– Halite
Saltstack을 쉽게 관리 할수 있는 WEB UI 입니다.

Figure 1. Saltstack Architecture

 

Master와 Minion은 ZeroMQ를 통해 AES 암호화 알고리즘으로 통신을 하며, 관리자가 CUI로 salt 명령어를 통해 직접 명령을 내릴수도 있지만,
Hailte를 가지고 Web UI에서 간단하게 Minion들을 제어 할수도 있습니다.

 

SaltStack 설치

본 설치는 Ubuntu 14.04 (Trusty) LTS에서 테스트 되었으며, 위의 아키텍쳐를 토대로 구성하였습니다.
설치하려는 시스템의 리눅스 배포판 및 버전에 따라 설치 방법이 다소 상이 할수 있으니, 다를 경우 공식 문서를 참고하여 주시기 바랍니다.

공식문서: http://docs.saltstack.com/en/latest/topics/installation/index.html

1. Hostname 서버 구성
각각의 master와 minion들의 서버정보를 /etc/hosts에 등록 합니다.

root@ruo91:~# nano /etc/hosts
172.17.0.18 salt-master
172.17.0.20 salt-minion-1
172.17.0.21 salt-minion-2
172.17.0.22 salt-minion-3
172.17.0.23 salt-minion-4
172.17.0.24 salt-minion-5

 

 

2. SaltStack 저장소 등록 및 설치
공통으로 Master와 Minion에 saltstack의 저장소를 등록 합니다.

root@ruo91:~# apt-get install -y python-software-properties software-properties-common
root@ruo91:~# echo deb http://ppa.launchpad.net/saltstack/salt/ubuntu `lsb_release -sc` main > /etc/apt/sources.list.d/saltstack.list
root@ruo91:~# wget -q -O- "http://keyserver.ubuntu.com:11371/pks/lookup?op=get&search=0x4759FA960E27C0A6" | apt-key add -

 

SaltStack 설치
– Master

root@ruo91:~# apt-get update && apt-get install -y salt-master salt-syndic

 

– Minion

root@ruo91:~# apt-get update && apt-get install -y salt-minion

 

SaltStack 설정 및 실행

1. SaltStack 설정 및 실행
– Master
master를 실행을 하는 방법은 두 가지가 있는데, 첫번째는 데몬 형태로 띄우는 것이고, 두번째는 foreground에서 띄우는 방식이 있습니다.
저는 Saltstack이 어떻게 동작하고 어떤 출력결과를 내뱉는지 확인 하기 위해, foreground 방식으로 실행 해보록 하겠습니다.
(데몬형태로 띄우고자 하시는분은 간단하게 service salt-master start 하시면 됩니다.)

실행하면 TCP 4505 포트가 public(0.0.0.0)으로 시작 되는걸 볼수 있습니다.

root@ruo91:~# salt-master -l info
[INFO ] Found minion id from generate_minion_id(): salt-master
[INFO ] Setting up the Salt Master
[INFO ] Preparing the root key for local communication
[INFO ] salt-master is starting as user 'root'
[INFO ] Current values for max open files soft/hard setting: 524288/1048576
[INFO ] Halite: Not configured, skipping.
[INFO ] Starting the Salt Publisher on tcp://0.0.0.0:4505
[INFO ] Starting the Salt Puller on ipc:///var/run/salt/master/publish_pull.ipc
[INFO ] Setting up the master communication server
[INFO ] Found minion id from generate_minion_id(): salt-master
[INFO ] Found minion id from generate_minion_id(): salt-master
[INFO ] Found minion id from generate_minion_id(): salt-master
[INFO ] Found minion id from generate_minion_id(): salt-master
[INFO ] Found minion id from generate_minion_id(): salt-master
[INFO ] Found minion id from generate_minion_id(): salt-master
[INFO ] Found minion id from generate_minion_id(): salt-master
[INFO ] Found minion id from generate_minion_id(): salt-master
[INFO ] Found minion id from generate_minion_id(): salt-master
[INFO ] Found minion id from generate_minion_id(): salt-master
[INFO ] Found minion id from generate_minion_id(): salt-master
[WARNING ] Although 'dmidecode' was found in path, the current user cannot execute it. Grains output might not be accurate.
[WARNING ] Although 'dmidecode' was found in path, the current user cannot execute it. Grains output might not be accurate.
[WARNING ] Although 'dmidecode' was found in path, the current user cannot execute it. Grains output might not be accurate.
[WARNING ] Although 'dmidecode' was found in path, the current user cannot execute it. Grains output might not be accurate.
[INFO ] Worker binding to socket ipc:///var/run/salt/master/workers.ipc
[WARNING ] Although 'dmidecode' was found in path, the current user cannot execute it. Grains output might not be accurate.
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-1
[INFO ] Authentication failed from host salt-minion-1, the key is in pending and needs to be accepted with salt-key -a salt-minion-1
[INFO ] Worker binding to socket ipc:///var/run/salt/master/workers.ipc
[INFO ] Worker binding to socket ipc:///var/run/salt/master/workers.ipc
[INFO ] Worker binding to socket ipc:///var/run/salt/master/workers.ipc
[INFO ] Worker binding to socket ipc:///var/run/salt/master/workers.ipc

 

 

– Minion
Minion이 Master와 연결이 가능하도록 호스트네임 설정을 해줍니다.

root@ruo91:~# sed -i '/^#master: salt/ s:.*:master\: salt-master:' /etc/salt/minion

 

Master와 실행 방법이 비슷하며, foreground에서 로깅 옵션을 통해 띄우는 방법을 사용하겠습니다.

root@ruo91:~# salt-minion -l info
[INFO ] Setting up the Salt Minion "salt-minion-1"
[WARNING ] Although 'dmidecode' was found in path, the current user cannot execute it. Grains output might not be accurate.
[ERROR ] The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate
[INFO ] Waiting for minion key to be accepted by the master.
[INFO ] Waiting 10 seconds before retry.
[ERROR ] The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate
[INFO ] Waiting for minion key to be accepted by the master.
[INFO ] Waiting 10 seconds before retry.

 

Minion이 실행이 되면 Master로 부터 인증 요청을 하게 되는데, 기본적으로 Master에서는 이 인증을 허용하지 않습니다.
그래서 위와 같이 ERROR 부분에 10초후에 재인증 하겠다고 출력 되는것 입니다.

 

2. SaltStack 키 등록
Master에서 해당 Minion의 키 등록을 통해 허용을 해줘야 합니다.
아래 명령어를 통해 등록되지 않은 Minion을 찾습니다.

root@ruo91:~# salt-key -L
Accepted Keys:
Unaccepted Keys:
salt-minion-1
salt-minion-2
salt-minion-3
salt-minion-4
salt-minion-5
Rejected Keys:

 

해당 Minion들을 일괄 등록 해줍니다.

root@ruo91:~# salt-key -A
The following keys are going to be accepted:
Unaccepted Keys:
salt-minion-1
salt-minion-2
salt-minion-3
salt-minion-4
salt-minion-5
Proceed? [n/Y] y
Key for minion salt-minion-1 accepted.
Key for minion salt-minion-2 accepted.
Key for minion salt-minion-3 accepted.
Key for minion salt-minion-4 accepted.
Key for minion salt-minion-5 accepted.

 

일괄 등록 후 Master/Minion에서는 AES 암호화 알고리즘을 ZeroMQ를 통해 통신을 하게 됩니다.

– Master

[INFO ] Authentication request from salt-minion-2
[INFO ] Authentication accepted from salt-minion-2
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-2
[INFO ] Authentication accepted from salt-minion-2
[INFO ] AES payload received with command _pillar
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-2
[INFO ] Authentication accepted from salt-minion-2
[INFO ] AES payload received with command _minion_event
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-2
[INFO ] Authentication accepted from salt-minion-2
[INFO ] AES payload received with command _minion_event
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-2
[INFO ] Authentication accepted from salt-minion-2
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-2
[INFO ] Authentication accepted from salt-minion-2
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-3
[INFO ] Authentication accepted from salt-minion-3
[INFO ] AES payload received with command _mine
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-3
[INFO ] Authentication accepted from salt-minion-3
[INFO ] AES payload received with command _pillar
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-5
[INFO ] Authentication accepted from salt-minion-5
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-5
[INFO ] Authentication accepted from salt-minion-5
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-3
[INFO ] Authentication accepted from salt-minion-3
[INFO ] AES payload received with command _pillar
[INFO ] AES payload received with command _minion_event
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-3
[INFO ] Authentication accepted from salt-minion-3
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-4
[INFO ] Authentication accepted from salt-minion-4
[INFO ] AES payload received with command _minion_event
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-4
[INFO ] Authentication accepted from salt-minion-4
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-3
[INFO ] Authentication accepted from salt-minion-3
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-3
[INFO ] Authentication accepted from salt-minion-3
[INFO ] AES payload received with command _pillar
[INFO ] AES payload received with command _mine
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-5
[INFO ] Authentication accepted from salt-minion-5
[INFO ] AES payload received with command _minion_event
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-5
[INFO ] Authentication accepted from salt-minion-5
[INFO ] AES payload received with command _minion_event
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-5
[INFO ] Authentication accepted from salt-minion-5
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-5
[INFO ] Authentication accepted from salt-minion-5
[INFO ] AES payload received with command _mine
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-4
[INFO ] Authentication accepted from salt-minion-4
[INFO ] AES payload received with command _minion_event
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-4
[INFO ] Authentication accepted from salt-minion-4
[INFO ] AES payload received with command _minion_event
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-4
[INFO ] Authentication accepted from salt-minion-4
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-4
[INFO ] Authentication accepted from salt-minion-4
[INFO ] AES payload received with command _mine
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-1
[INFO ] Authentication accepted from salt-minion-1
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-1
[INFO ] Authentication accepted from salt-minion-1
[INFO ] AES payload received with command _pillar
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-1
[INFO ] Authentication accepted from salt-minion-1
[INFO ] AES payload received with command _minion_event
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-1
[INFO ] Authentication accepted from salt-minion-1
[INFO ] AES payload received with command _minion_event
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-1
[INFO ] Authentication accepted from salt-minion-1
[INFO ] Clear payload received with command _auth
[INFO ] Authentication request from salt-minion-1
[INFO ] Authentication accepted from salt-minion-1
[INFO ] AES payload received with command _mine

 

 

– Minion

[INFO ] Waiting for minion key to be accepted by the master.
[INFO ] Waiting 10 seconds before retry.
[INFO ] Authentication with master at 172.17.0.18 successful!
[WARNING ] Although 'dmidecode' was found in path, the current user cannot execute it. Grains output might not be accurate.
[INFO ] Added mine.update to schedular
[INFO ] Added new job __mine_interval to scheduler
[INFO ] Minion is starting as user 'root'
[INFO ] Starting pub socket on ipc:///var/run/salt/minion/minion_event_a8653ec967_pub.ipc
[INFO ] Starting pull socket on ipc:///var/run/salt/minion/minion_event_a8653ec967_pull.ipc
[INFO ] Minion is ready to receive requests!
[INFO ] Running scheduled job: __mine_interval

 

SaltStack PING 테스트 및 IP 확인

Salt 명령어를 통해 여러대의 Minion들의 핑 테스트와 IP를 확인 할수 있습니다.

– Ping
사용법은 아래와 같습니다.

salt [minion hostname] [function]

root@ruo91:~# salt '*' test.ping
salt-minion-1:
True
salt-minion-2:
True
salt-minion-3:
True
salt-minion-4:
True
salt-minion-5:
True

 

– IP

root@ruo91:~# salt 'salt-minion-1' network.ip_addrs

or

root@ruo91:~# salt '*' network.ip_addrs
salt-minion-1:
- 172.17.0.20
salt-minion-2:
- 172.17.0.21
salt-minion-3:
- 172.17.0.22
salt-minion-4:
- 172.17.0.23
salt-minion-5:
- 172.17.0.24

 

SaltStack 패키지 설치

여러대의 Minion들에게 Nginx를 설치 하도록 해보겠습니다.
사용법은 아래와 같습니다.

salt '[minion hostname]' pkg.install [package name]

 

모든 Minion에게 Nginx를 설치를 하도록 명령 하면 아래와 같이 Minion의 작업이 실행 됩니다.

root@ruo91:~# salt '*' pkg.install nginx

 

– Minion

[INFO ] Starting a new job with PID 1001
[INFO ] Returning information for job: 20141228051609784102
[INFO ] User root Executing command pkg.install with jid 20141228052009048937
[INFO ] Starting a new job with PID 1012
[INFO ] Executing command "dpkg-query --showformat='${Status} ${Package} ${Version} ${Architecture}\n' -W" in directory '/root'
[INFO ] Executing command ['apt-get', '-q', '-y', '-o', 'DPkg::Options::=--force-confold', '-o', 'DPkg::Options::=--force-confdef', 'install', 'nginx'] in directory '/root'
[INFO ] User root Executing command saltutil.find_job with jid 20141228052014188450
[INFO ] Starting a new job with PID 1020
[INFO ] Returning information for job: 20141228052014188450
[INFO ] User root Executing command saltutil.find_job with jid 20141228052019311997
[INFO ] Starting a new job with PID 1029
[INFO ] Returning information for job: 20141228052019311997
[INFO ] User root Executing command saltutil.find_job with jid 20141228052024443163
[INFO ] Starting a new job with PID 1036
[INFO ] Returning information for job: 20141228052024443163
[INFO ] User root Executing command saltutil.find_job with jid 20141228052029551283
[INFO ] Starting a new job with PID 1043
[INFO ] Returning information for job: 20141228052029551283

 

이후 Nginx에 설치에 필요한 의존성 패키지들의 버전이 출력 되면서 설치가 완료 됩니다.

salt-minion-1:
    ----------
    fontconfig-config:
        ----------
        new:
            2.11.0-0ubuntu4.1
        old:

    fonts-dejavu-core:
        ----------
        new:
            2.34-1ubuntu1
        old:

    geoip-database:
        ----------
        new:
            20140313-1
        old:

    httpd:
        ----------
        new:
            1
        old:

    httpd-cgi:
        ----------
        new:
            1
        old:

    libfontconfig:
        ----------
        new:
            1
        old:

    libfontconfig1:
        ----------
        new:
            2.11.0-0ubuntu4.1
        old:

    libfreetype6:
        ----------
        new:
            2.5.2-1ubuntu2.2
        old:

    libgd3:
        ----------
        new:
            2.1.0-3
        old:

    libgeoip1:
        ----------
        new:
            1.6.0-1
        old:

    libjbig0:
        ----------
        new:
            2.0-2ubuntu4.1
        old:

    libjpeg-turbo8:
        ----------
        new:
            1.3.0-0ubuntu2
        old:

    libjpeg8:
        ----------
        new:
            8c-2ubuntu8
        old:

    libtiff5:
        ----------
        new:
            4.0.3-7ubuntu0.1
        old:

    libvpx1:
        ----------
        new:
            1.3.0-2
        old:

    libxpm4:
        ----------
        new:
            1:3.5.10-1
        old:

    libxslt1.1:
        ----------
        new:
            1.1.28-2build1
        old:

    nginx:
        ----------
        new:
            1.4.6-1ubuntu3.1
        old:

    nginx-common:
        ----------
        new:
            1.4.6-1ubuntu3.1
        old:

    nginx-core:
        ----------
        new:
            1.4.6-1ubuntu3.1
        old:

 

 SaltStack – Halite Web UI 사용

CUI로도 관리를 할수 있지만, WEB UI인 Halite를 통해서도 관리가 가능합니다.

1. Master 설정
Master 서버에서 진행 하며, Halite에 로그인이 가능하도록 해당 구문을 추가 해줘야 합니다.

root@ruo91:~# nano /etc/salt/master

# Halite
external_auth:   # 외부 인증 설정
  pam:
    hailte:           # Halite 시스템 사용자
      - .*
      - '@runner'
      - '@wheel'
    ruo91:          # ruo91 시스템 사용자
        - .*
        - '@runner'
        - '@wheel'
halite:              # halite에 대한 서버 설정
  level: 'debug'
  server: 'gevent'   # cherrypy, paste, gevent 중 택일
  host: '0.0.0.0' 
  port: '8080'
  cors: False

 

Halite는 기본적으로 PAM을 통해 로그인을 합니다. 즉, 시스템 계정을 통해 로그인 한다는 뜻입니다.

따라서, 관리하려는 시스템의 계정을 따로 추가 하여 설정을 하셔야 합니다.

위와 같이 설정을 하고 Master의 서버를 재시작 합니다.
foreground로 로깅 옵션을 활성화 하여 실행하셨다면, Ctrl + C를 눌러 중지 하시고 다시 master 서버를 실행 해주시면 됩니다.

 

– Foreground

root@ruo91:~# salt-master -l info

Ctrl + C

root@ruo91:~# salt-master -l info

 

– Daemon

root@ruo91:~# service salt-master restart

 

2. Halite 시스템 계정 추가

root@ruo91:~# useradd halite
root@ruo91:~# passwd halite
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully

 

3. Halite 설치
Halite를 saltstack github repository에서 받아 옵니다.

root@ruo91:~# git clone https://github.com/saltstack/halite /opt/halite

 

4. Halite 실행
Halite는 halite 디렉토리안에 server_bottle라는 Python 스크립트를 통해 실행 합니다.
즉, /opt/halite 에 받았다면 /opt/halite/halite/server_bottle.py 를 실행 하는 것이지요.

Halite를 실행시 구동할 서버 방식이 총 3가지 cherrypy, paste, gevent가 있는데 저는 gevent 방식을 사용하겠습니다.
실행하기 앞서 Python용 gevent 패키지가 필요하니 패키지를 먼저 설치 후 실행 하도록 하겠습니다.

root@ruo91:~# apt-get install -y python-gevent
root@ruo91:~# cd /opt/halite/halite

root@ruo91:~# ./server_bottle.py -d -C -l debug -s gevent
20141228_055132.610492 Bottle: Running web application server 'gevent' on 0.0.0.0:8080.
20141228_055132.610964 Bottle: CORS is disabled.
20141228_055132.611192 Bottle: TLS/SSL is disabled.
20141228_055132.611424 Bottle: Server options:
{}
Bottle v0.12-dev server starting up (using GeventServer())...
Listening on http://0.0.0.0:8080/
Hit Ctrl-C to quit.

기본 포트는 8080을 제공하므로 이곳으로 접속 합니다.

 

– Console 화면
접속하면 기본적으로 보이는 화면이며 오른쪽 상단에 위치하고 있는 로그인 부분에 위에서 생성한 시스템 계정을 통해 로그인 합니다.

 Figure 2. Halite – Console

– Monitor
Minion들의 상황을 볼수 있는 곳입니다.

 Figure 3. Halite – Console: Monitor

– Monitor -> Job
Monitor의 하위 메뉴 중 작업 내역에 대한 곳입니다. 관리자가 CUI 또는 WEB UI에서 명령어를 실행한 내역을 확인 할수 있는 곳입니다.

 Figure 4. Halite – Console: Monitor – Job

– Monitor -> Event
Minion들의 이벤트 내역을 확인 할수 있는 곳입니다.

 Figure 5. Halite – Console: Monitor – Event

– Monitor -> Minion ->Grain
Grain을 통해 Minion의 시스템 정보를 확인 할수가 있습니다.

 Figure 6. Halite – Console: Monitor – Minion: Grain

– Command
관리자가 CUI를 통해 salt 명령어를 내리는 것처럼 WEB UI에서도 Command 부분을 통해서 Minion들에게 명령을 내릴수가 있습니다.

 Figure 7. Halite – Console: Command

참고

– SaltStack
http://docs.saltstack.com/en/latest/topics/installation/index.html
http://www.slideshare.net/anu-bhaskar/introducing-saltstack

– Halite
https://github.com/saltstack/halite
http://docs.saltstack.com/en/latest/topics/tutorials/halite.html