I haven’t installed an ELK stack since CentOS 7 came out, all of the components which go to make up the stack have been updated quite bit since then so I decided to have a go at installing the stack on a clean CentOS 7 installation.
ELK?# ELK is a term used for a combination of three Open Source products from Elastic ↗
All three products can be used independently, but when they are used together you find yourself with both a powerful and scaleable central logging service.
There is a good introduction to the stack on YouTube;
Elasticsearch# The only pre-requisite for Elasticsearch is a recent version of Java, the quickest way to install this is directly from Oracle;
Installing an ELK Stack on CentOS 7 1/11
c w t c r d g a h m e r o t w o x n o p — v p t f - t n R / o j j - r r r c e o e o - o - o 8 t 8 k * : * i . . e t j t s a r a r e r — . 1 . g . g n z 8 z o * - c h e c k - c e r t i f i c a t e — h e a d e r “ C o o k i e : g p w _ e 2 4 = h t t p % 3 A % 2 F % 2 F w w w . o r a c l e . c o m % 2 F ; o r a c l e l i c e n s e = a c c e p t - s e c u r e b a c k u p - c o o k i e ” “ h t t p : / / d o w n l o a d . o r a c l e . c o m / o t n - p u b / j a v a / j d k / 8 u 4 0 - b 2 5 / j r e - 8 u 4 0 - l i n u x - x 6 4 . t a r . g z " Now we have a java Runtime installed in /opt/jre1.8* lets use alternatives to set it so the system uses it by default;
Installing an ELK Stack on CentOS 7 2/11
a j j J J l a a a a t v v v v e a a a a r ( n v T H a v e M o t e r ) t i r s S v s i S p e i o E o s o n t n R ( — “ u T 1 n M i . t ) n 8 i s . m 6 t 0 e 4 a _ - l 4 E B l 0 n i ” v t / i u r S s o e r n r / m v b e e i n r n t / V j ( M a b v u ( a i b l u j d i a l v 1 d a . 8 2 . 5 o 0 . p _ 4 t 4 0 / 0 - j - b r b 2 e 2 5 1 5 , . ) 8 m * i / x b e i d n / m j o a d v e a ) 1 So thats Java installed and set as the system default, next up is Elasticsearch itself. There is an official Yum repository so lets use it;
Installing an ELK Stack on CentOS 7 3/11
c a t / e t c / y u m . r e p o s . d / e l a s t i c s e a r c h . r e p o / e t c / s y s t e m d / s y s t e m / k i b a n a 4 . s e r v i c e / e t c / n g i n x / c o n f . d / k i b a n a . c o n f and finally start the services;
Installing an ELK Stack on CentOS 7 4/11
s s y y s s t t e e m m c c t t l l s e t n a a r b t l e n g n i g n i x n x
Logstash# Like Elasticsearch there is a Yum repository;
Installing an ELK Stack on CentOS 7 5/11
c a t s s } } I t y s s N p l l P e _ _ U c k T = e e / > r y e t t “ i = c l f > / o i y g c “ u s a / m ” t e . e t r c e = / p > p o k s “ i . / / d e t / t l l c s o / / g p p s k r t i i a / v s t a h l t . s e r / / e c l p e o o r g t s s t / a l s / o h e g - t s f c t o / a r l s w o h a g - r s f d t o e a r r s w . h a k / r e c d y o e ” n r f . . c d r / t 0 ” 1 - l u m b e r j a c k - i n p u t . c o n f < 5 0 0 0 As we will be shipping syslog data to our ELK stack we need to let Logstash how it will look;
Installing an ELK Stack on CentOS 7 6/11
c a a } s d m } } } S a d d y a a Y t d d s t t S _ _ l e c L f f o h O i i g { G e e _ = / l l p > e d d r t i [ c = = / > > { “ l s o [ [ } y g s s “ “ l t r r o a e e g s c c _ h e e t / i i i c v v m o e e e n d d s f _ _ t . a f a d t r m / ” o p 1 , m ” 0 ” , - “ , s % “ y { “ M s @ % M l t { M o i h g m o d . e s c s t H o t } H n a ” : f m m p ] m < } : ” s { s ] ” “ , m e “ s M s M a M g e d ” d = H > H : “ m % m { : S s Y s S ” L O ] G T I M E S T A M P : s y s l o g _ t i m e s t a m p } % { S Y S L O G H O S T : s y s l o g _ h o s t n a m e } % { D A T A : s y s l o g _ p r o g r a m } ( ? : \ [ % { P O S I N T : s y s l o g _ p i d } \ ] ) ? : % { G R E E D Y D A T A : s y s l o g _ m e s s a g e } ” } Finally lets send the data to the Elasticsearch installation on localhost;
Installing an ELK Stack on CentOS 7 7/11
c s } O a t U t d T o P u U t T { / e c t o c d / e l c o g = s > t a r s u h b / y c d o e n b f u . g d / } 3 0 - l u m b e r j a c k - o u t p u t . c o n f < l o c a l h o s t } So thats Logstash configured, lets start the service;
Installing an ELK Stack on CentOS 7 8/11
s c e h r k v c i o c n e f i l g o g l s o t g a s s t h a s r h e s o t n a r t
and thats the main stack installed and configured. You should be able to visit the FQDN you specified in the NGINX configuration and see the Kibana dashboard.
Logstash Forwarder# Now we have our main stack installed, let pump some data into it.
This is all run on the server instances you would to report into your newly configured ELK stack.
First of all, lets import the GPG key for the Forwarder repo;
Installing an ELK Stack on CentOS 7 9/11
r p m — i m p o r t h t t p : / / p a c k a g e s . e l a s t i c s e a r c h . o r g / G P G - K E Y - e l a s t i c s e a r c h
Now add the repo;
Installing an ELK Stack on CentOS 7 10/11
T c { T “ “ “ “ } h “ { “ “ “ “ ] “ } ] } F a h n s t s , e f p / / / , f O t e e e i s i a v i R t r m l l l t a a a e W n w v e i e h r r r l A e o e o c s s s / / / d R t r r u a t ” ” l l l s D / w k s t ” : : o o o ” e o ” ” ” : o g g g : t r : : : f [ [ / / / c k “ m s f { / { [ 1 / f e e a y s 5 e i s c i “ u e “ , t l s u l t m c e c e a r 2 y . t l / s g e b p r i k p e ” a e e o . k c s , n ” p n m i o ” . : o c / n , l s c k t f o “ . o e l i g s d v n s g ” y / e d / u s l r r c r l o s i e a o g c r t g s n k t i ” t e . s o a t i / n } s w o l s h o : o - r 5 g f k 0 s o 0 t r c 0 a w o ” s a n h r f ] - d i , f e g o r u r . r w r a a e t r p i d o o e n r . : c ) r / t e ” t c / l o g s t a s h - f o r w a r d e r . c o n f F O R W A R D as you can see I am shipping /var/log/messages, /var/log/secure and /var/log/fail2ban.log over to my ELK stack.
Finally we need to start the service and configure to start on boot;
Installing an ELK Stack on CentOS 7 11/11
s c e h r k v c i o c n e f i l g o g l s o t g a s s t h a - s f h o - r f w o a r r w d a e r r d e r r e s o t n a r t
and hey presto, we now have data appearing in Kibana;
