ElasticQ – ElasticSearchย realtime correlation analysis

ElasticSearchย Realtime correlation analysis, Detection

 

DOWNLOAD

code sign on “Open Source Developer, JuSeong Han”

YOU LIKE IT, CLICK LIKE BUTTON ๐Ÿ™‚

ElasticQ

 

ElasticQ๋ฅผ ์ด์šฉํ•˜์—ฌ ElasticSearch ์—์„œ ์ƒ๊ด€๋ถ„์„๊ณผ ์‹ค์‹œ๊ฐ„ ์•Œ๋žŒ์„ ์‰ฝ๊ฒŒ ๊ฐœ๋ฐœํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋™์‹œ์— ์—ฌ๋Ÿฌ ๊ฐœ์˜ Query ๊ฒ€์ƒ‰์ด ๊ฐ€๋Šฅํ•˜๊ณ  ํƒ์ง€ ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

With ElasticQย you can easily develop correlation analysis and real-time alarms in Elastic Search.

You can search multiple queries at the same time,ย Detection

Key Features

  • ๋™์‹œ์— ์—ฌ๋Ÿฌ ์งˆ์˜ ๋‚ด์šฉ์„ ์กฐ๊ฑด(์ƒ๊ด€๋ถ„์„)์œผ๋กœ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. 2๊ฐœ์˜ ์ฟผ๋ฆฌ๋ฅผ ํ•˜๋‚˜์˜ ์กฐ๊ฑด์œผ๋กœ ๋งŒ๋“ค์–ด ํƒ์ง€ํ•˜๋Š” ๊ฒƒ์ด ๊ฐ€๋Šฅํ•ฉ๋‹ˆ๋‹ค. ์ด์ œ ํ‚ค๋ฐ”๋‚˜๋กœ ์ฟผ๋ฆฌ๋ฅผ ํ•ด๋ณด๊ณ , ํ•„์š”ํ•œ ์กฐ๊ฑด์„ ์—ฌ๋Ÿฌ๊ฐœ๋ฅผ ์—ฐ๊ฒฐํ•˜์—ฌ ๊ทœ์น™์„ ๋งŒ๋“ค์–ด ๋ณด์„ธ์š”. At the same time, multiple query contents can be used as a condition(Correlation analysis). It is possible to detect two queries as a single condition. Now, you can query the Kibana and create rules by connecting multiple conditions.
  • ํƒ์ง€ ์กฐ๊ฑด์„ ๊ฐฏ์ˆ˜๋ฅผ ํ†ตํ•ด์„œ ์ •๋ฐ€ํ•œ ์กฐ๊ฑด์„ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.ย Precise conditions can be used through the number of detection conditions.
  • ์‹คํ–‰ ์ฃผ๊ธฐ์— ๋งž์ถฐ ์‹ค์‹œ๊ฐ„์œผ๋กœ ํƒ์ง€ํ•ฉ๋‹ˆ๋‹ค. Detect in real time according to the execution cycle.
  • ์‹ค์‹œ๊ฐ„์œผ๋กœ ํƒ์ง€ ์กฐ๊ฑด์„ ์ถ”๊ฐ€ํ•˜๊ฑฐ๋‚˜ ์ˆ˜์ •ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.You can add or modify detection conditions in real time.

HOW to use

๋งŒ์•ฝ ElasticSearch๊ฐ€ ์—†๋‹ค๋ฉด ๋‹ค์Œ ๊ธ€์„ ํ™•์ธํ•˜์„ธ์š”.ย If you do not have ElasticSearch, check out the next post.

PART 1 INSTALL ELASTICSEARCH 6 CLUSTER FOR CENTRALIZED SYSLOG

ElasticQ๋ฅผ ์‹คํ–‰ํ•˜๊ธฐ ์œ„ํ•ด ํ•„์š”ํ•œ ์˜ต์…˜๊ณผ ์‹คํ–‰์‹œ ํƒ์ง€ ์กฐ๊ฑด์„ ์ž…๋ ฅํ•˜๋Š” ํƒ์ง€ ํŒŒ์ผ๋กœ ๊ตฌ๋ถ„๋ฉ๋‹ˆ๋‹ค.

It is divided into an option file for running ElasticQ and a detection file for entering detection conditions at runtime.

์‹คํ–‰ ์˜ต์…˜ (Execution options)

์‹คํ–‰ ์˜ต์…˜์€ ElasticQ์˜ ํƒ์ง€ ์‹œ ์•Œ๋žŒ์„ ๋ฐœ์ƒํ•  ๋•Œ ์‚ฌ์šฉ๋˜๋Š” SMTP ์„œ๋ฒ„๋‚˜ Rule ๋ฆฌ์ŠคํŠธ ํŒŒ์ผ ์œ„์น˜๋“ฑ์„ ์ง€์ •ํ•˜๋Š” ์šฉ๋„๋กœ ์‚ฌ์šฉ๋ฉ๋‹ˆ๋‹ค.

The Execute option is used to specify the location of the SMTP server or Rule list file used when generating an alarm when ElasticQ detects it.

-syslog:<ip>

send to elasticq matched query log on syslog server.

-smtp:<ip>

send to elasticq matched query log on smtp server.

-rulefile:<path>

elasticq rule load from this file.

-sender:<Email>

This address use at email sender

-es:<httpaddress>

Query on elasticsearch server

-install

install services type.

-uninstall

uninstall services type.

 

EX1)ElasticQ -rulefile:rule.ini -sender:<Email> -es:http://192.168.0.1:9200 -syslog:172.16.253.20 -smtp:10.0.0.5

EX2)ElasticQ -rulefile:rule.ini -sender:<Email> -es:http://192.168.0.1:9200 -syslog:172.16.253.20 -smtp:10.0.0.5-install

๋ฃฐ ์ƒ์„ฑ ๊ทœ์น™ (Rule generation rule)

๋ฃฐ์€ ํ•ญ๋ชฉ์€ | ์œผ๋กœ ๊ตฌ๋ถ„ํ•ฉ๋‹ˆ๋‹ค. ํ•ญ๋ชฉ์•ˆ์— ์„ธ๋ถ€ ์˜ต์…˜์€ : ๋กœ ๊ตฌ๋ถ„ํ•ฉ๋‹ˆ๋‹ค.

The rule is | . The detail options within the item are separated by:.

ํ•ญ๋ชฉ ๊ตฌ๋ถ„์ž(Item Separator) |

ํ•ญ๋ชฉ๋‚ด ์˜ต์…˜ ๊ตฌ๋ถ„์ž(Option Separator in Item)ย :ย 

(query:logon AND fail:>:1|time:15:@timestamp|index:logstash|timebase:true|same:procid),address:sysloghost,title:WHORUfile Detect,email:jshan@bluehole.net,webhook:https://~~~~

query

ํ‚ค๋ฐ”๋‚˜์—์„œ ๊ฒ€์ƒ‰๊ณผ ๋™์ผํ•˜๊ฒŒ ์‚ฌ์šฉํ•˜์‹œ๋ฉด ๋ฉ๋‹ˆ๋‹ค.ย : ํƒ์ง€๋˜๋Š” ๊ฐฏ์ˆ˜์˜ ํฌ๊ฑฐ๋‚˜ ์ž‘๊ธฐ๋ฅผ ๊ตฌ๋ถ„ํ•จ > – ๊ฐฏ์ˆ˜๋กœ ์ด์ƒ์„ ์˜๋ฏธ < – ๊ฐฏ์ˆ˜๋กœ ์ดํ•˜๋ฅผ ์˜๋ฏธ : (Number)ย ๊ธฐ์ค€์ด ๋˜๋Š” ํƒ์ง€ ๊ฐฏ์ˆ˜

You can use the same search as Kibana. : Identify the number of detected or unknownย > – Meaning above < –ย Meaning less : (Number) Number of detections as a reference

EX1)query:logon AND fail:>:1ย –>ย  logon AND fail ์ฟผ๋ฆฌ๊ฐ€ 1๊ฐœ ์ด์ƒ์ผ ๊ฒฝ์šฐ, logon AND fail If there is more than one query

EX2)query:ping AND n:>:5 –> ping AND n ์ฟผ๋ฆฌ๊ฐ€ 5๊ฐœ ์ด์ƒ์ผ ๊ฒฝ์šฐ,ย ping AND n If there are more than 5 queries

time

ํƒ์ง€๋ฅผ ์ง„ํ–‰ํ•˜๋Š” ์‹œ๊ฐ„์œผ๋กœ ๋ถ„ ๋‹จ์œ„๋กœ ๊ณ„์‚ฐ๋ฉ๋‹ˆ๋‹ค. : ๋ ˆ์ฝ”๋“œ์˜ย ํƒ€์ž„ ๊ธฐ์ค€ ๊ฐ’์œผ๋กœ ์‹œ๊ฐ„์„ ๊ณ„์‚ฐํ•  ๋•Œ ์ด์šฉ๋˜๋Š” ํ•„๋“œ๋ฅผ ์ง€์ •ํ•ฉ๋‹ˆ๋‹ค.

The time it takes for detection to be counted in minutes. : Specifies the field used when calculating the time by the time base value of the record.

EX1) time:5:createtime –> 5๋ถ„ ๋‹จ์œ„๋กœ query ์กฐ๊ฑด์„ ๊ฒ€์ƒ‰ํ•˜๊ณ  ํƒ€์ž„ ํ•„๋“œ๋Š” createtime,ย Search for a condition in 5-minute increments, and the time field is created

EX2) time:10:timestamp –> 10๋ถ„ ๋‹จ์œ„๋กœ query ์กฐ๊ฑด์„ ๊ฒ€์ƒ‰ํ•˜๊ณ  ํƒ€์ž„ ํ•„๋“œ๋Š” timestamp,ย Search query conditions in 10-minute increments, and the time field is timestamp

index

Elasticsearch ์˜ Query ์กฐ๊ฑด์„ ๊ฒ€์ƒ‰ํ•  ์ธ๋ฑ์Šค๋ช…์„ ์ง€์ •ํ•ฉ๋‹ˆ๋‹ค.

Specifies the index name to search for Elasticsearch query conditions.

EX1) index:logstash –> logstash ์™€ ๋™์ผํ•œ ์ธ๋ฑ์Šค

EX2) index:logstash-2018* –> logstash-2018๋กœ ์‹œ์ž‘ํ•˜๋Š” ๋ชจ๋“  ์ธ๋ฑ์Šค

timebase

Elasticsearch ์˜ ์ธ๋ฑ์Šค๊ฐ€ ํƒ€์ž„์œผ๋กœ ์ƒ์„ฑ๋˜๋Š”์ง€์— ๋Œ€ํ•œ ์กฐ๊ฑด์„ ์„ค์ •ํ•ฉ๋‹ˆ๋‹ค.

Elasticsearch์˜ ์ธ๋ฑ์Šค๊ฐ€ ํƒ€์ž„ ๋ฒ ์ด์Šค์ธ ๊ฒฝ์šฐ ์ธ๋ฑ์Šค ๋ช…์ด ์ž๋™์œผ๋กœย  -YYYY.MM.DD๋กœ ์ƒ์„ฑ๋˜์–ด ์ด๋ถ€๋ถ„์„ ์ง€์›ํ•˜๊ธฐ ์œ„ํ•œ ์˜ต์…˜์ž…๋‹ˆ๋‹ค. ์‹ค์‹œ๊ฐ„ ๊ฐ์‹œ๋ฅผ ์œ„ํ•ด์„œ ์ธ๋ฑ์Šค ๋ช…์ด -YYYY.MM.DD ๊ฐ™์ด ๋ณ€๊ฒฝ๋˜๋Š” ๊ฒฝ์šฐ์—๋งŒ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.

Sets the conditions for whether Elasticsearch’s index is generated in time.

If Elasticsearch’s index is time-based, the index name is automatically generated as -YYYY.MM.DD and is an option to support this part.ย This is useful if the index name changes like -YYYY.MM.DD for real-time monitoring.

EX1)ย index:logstash|timebase:true –> logstash-YYYY.MM.DD ์˜ ์ธ๋ฑ์Šค(ํ˜„์žฌ ์˜ค๋Š˜ ๋‚ ์งœ๋กœ ์ธ์‹),ย Index of logstash-YYYY.MM.DD (currently recognized as today’s date)

EX2) index:logstash-2018*|timebase:false –>ย logstash-2018* ์ธ๋ฑ์Šค, ๋งค ๊ฒ€์ƒ‰์— 2018 ์ „์ฒด๋ฅผ ๊ฒ€์ƒ‰ํ•˜๊ธฐ ๋•Œ๋ฌธ์— ์‹ค์‹œ๊ฐ„ ํƒ์ง€์— ๋ถ€ํ•˜๊ฐ€ ๋ฐœ์ƒ ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.ย logstash-2018* index, every search use 2018 all index, so real-time detection may be overloaded.

same

ํ•„๋“œ์˜ ๋™์ผํ•œ ๊ฐ’์ด ํƒ์ง€ ์กฐ๊ฑด์— ํ•„์š” ํ•  ๋•Œ ์‚ฌ์šฉ๋ฉ๋‹ˆ๋‹ค.ย ๋™์ผํ•œ ๊ณ„์ •๋ช…, IP์˜ ๋กœ๊ทธ์ธ ์‹คํŒจ๋ฅผ ํ™•์ธํ•˜๊ณ ์ž ํ•  ๋•Œ ์œ ์šฉํ•ฉ๋‹ˆ๋‹ค.

The same value in the field is used when a detection condition is required. This is useful when you want to check the login failure of the same account name and IP.

EX1) same:procid –> procid ๊ฐ’์ด ๋™์ผํ•œ ๋ ˆ์ฝ”๋“œ๋กœ ์ˆ˜์ง‘ํ•˜์—ฌ ํƒ์ง€ ํ•ฉ๋‹ˆ๋‹ค. query์˜ ํƒ์ง€ ๊ธฐ์ค€ ๊ฐ’์ด 5 ์ด์ƒ์ด๋ผ๋ฉด ๋™์ผํ•œ procid๊ฐ€ 5๊ฐœ ์ด์ƒ์ผ ๊ฒฝ์šฐ๊ฐ€ ํƒ์ง€ ์กฐ๊ฑด์ด ๋ฉ๋‹ˆ๋‹ค.ย The procid value is collected by the same record and detected. If the detection criteria of the query is 5 or more, the detection condition is 5 or more when the same procid is used.

EX2) same:sysloghost –> sysloghost๊ฐ’์ด ๋™์ผํ•œ ๋ ˆ์ฝ”๋“œ๋กœ ์ˆ˜์ง‘ํ•˜์—ฌ ํƒ์ง€ ํ•ฉ๋‹ˆ๋‹ค.ย The sysloghost value is collected and detected by the same record.

and, not

Query ์กฐ๊ฑด์œผ๋กœ ํ•˜์ง€ ๋ชปํ•œ ๋ณด๋‹ค ์ •๋ฐ€ํ•œ ์กฐ๊ฑด์„ ๋„ฃ์„ ๋•Œ ์‚ฌ์šฉ๋ฉ๋‹ˆ๋‹ค.

It is used when putting more precise condition that can not be done by query condition.

and๋Š” ํ•ด๋‹น ๊ฐ’์ด ํฌํ•จ๋˜์–ด์•ผ ํƒ์ง€ ๋ฉ๋‹ˆ๋‹ค.ย and is detected until the value is included.

not๋Š” ํ•ด๋‹น ๊ฐ’์ด ํฌํ•จ๋˜์ง€ ์•Š์•„์•ผ ํƒ์ง€ ๋ฉ๋‹ˆ๋‹ค.ย ย not is detected unless it is included.

EX1)and:message=network –> messageํ•„๋“œ์—์— network๋ผ๋Š” ๋ฌธ๊ตฌ๊ฐ€ ํฌํ•จ๋˜์–ด์•ผ ํƒ์ง€ ๋ ˆ์ฝ”๋“œ๊ฐ€ ๋ฉ๋‹ˆ๋‹ค.ย ย The message field must contain the phrase network to become a detection record.

EX2)not:programname=mail –> programnameํ•„๋“œ์— mail์ด๋ผ๋Š” ๋ฌธ๊ตฌ๊ฐ€ ํฌํ•จ๋˜์ง€ ์•Š์€ ๋ ˆ์ฝ”๋“œ๋ฅผ ํƒ์ง€ ํ•ฉ๋‹ˆ๋‹ค.ย Detects records that do not contain the phrase mail in the programname field.

ํƒ์ง€ ์กฐ๊ฑด ๊ทœ์น™ (Detection condition rule)

์ „์ฒด์ ์œผ๋กœ ํƒ์ง€ ์กฐ๊ฑด์€ ์ค‘๊ด„ํ˜ธ๋ฅผ ๊ฐ์‹ธ๊ณ  ์ฝค๋งˆ๋ฅผ ํ†ตํ•ด์„œ ๊ตฌ๋ถ„๋ฉ๋‹ˆ๋‹ค.

๋‘๊ฐœ์˜ ํƒ์ง€ ์กฐ๊ฑด์„ ๋„ฃ๊ณ ์ž ํ•œ๋‹ค๋ฉด, ์ฝค๋งˆ์™€ ์ค‘๊ด„ํ˜ธ ๋‘๊ฐœ๋ฅผ ์‚ฌ์šฉํ•˜๋ฉด ๋ฉ๋‹ˆ๋‹ค. ํƒ์ง€ ์กฐ๊ฑด์ด ๋ชจ๋‘ ๋งž์„ ๊ฒฝ์šฐ์— ์•Œ๋žŒ์ด ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค.

Overall detection conditions are enclosed in braces and separated by commas.

If you want to have two detection conditions, you can use a comma and two curly braces. An alarm occurs when all detection conditions are met.

ํƒ์ง€ ์กฐ๊ฑด 1๊ฐœ (1 detection condition)

(query:logon AND fail:>:1|time:15:@timestamp|index:logstash|timebase:true),address:sysloghost,title:WHORUfile Detect,email:jshan@bluehole.net,webhook:https://~~~~

ํƒ์ง€ ์กฐ๊ฑด 2๊ฐœ (2 detection condition)

(query:logon AND fail:>:1|time:15:@timestamp|index:logstash|timebase:true),(query:ping AND n:>:1|time:15:@timestamp|index:logstash|timebase:true),address:sysloghost,title:WHORUfile Detect,email:jshan@bluehole.net,webhook:https://~~~~

address

IP๊ฐ€ ๋™์ผํ•œ ๊ฒƒ๋งŒ ์ทจ๊ธ‰ํ•  ๊ฒฝ์šฐ ์‚ฌ์šฉ๋ฉ๋‹ˆ๋‹ค.ย ํƒ์ง€๊ฐ€ ๋œ ์ด๋ฒคํŠธ์— ๋Œ€ํ•œ IP๋ฅผ ํ™•์ธํ•˜์—ฌ ๋™์ผํ•œ IP๊ฐ€ query ์กฐ๊ฑด์˜ ๊ฐฏ์ˆ˜ ์ด์ƒ์ผ ๊ฒฝ์šฐ์—๋งŒ ํƒ์ง€ ๋ฉ๋‹ˆ๋‹ค. ๋‘๊ฐœ ์ด์ƒ์˜ ํƒ์ง€ ์กฐ๊ฑด์„ ๋„ฃ์—ˆ์„ ๋•Œ ๋™์ผํ•œ ์„œ๋ฒ„์—์„œ ๋ฐœ์ƒํ•œ ๊ฒฝ์šฐ์— ์œ ์šฉํ•ฉ๋‹ˆ๋‹ค.

Used only when the IP is the same. The IP for the detected event is checked and detected only if the same IP is more than the number of query conditions. This is useful if you have two or more detection conditions and you are on the same server.

title

ํƒ์ง€๊ฐ€ ๋˜์—ˆ์„ ๋•Œ ์•Œ๋žŒ์„ ๋ฐœ์ƒํ•˜๋Š” ์ œ๋ชฉ์ž…๋‹ˆ๋‹ค.

The title that triggers an alarm when it is detected.

email

ํƒ์ง€๊ฐ€ ๋˜์—ˆ์„ ๋•Œ ์•Œ๋žŒ์„ ์ „์†กํ•˜๊ณ ์ž ํ•˜๋Š” ๋ฉ”์ผ ์ฃผ์†Œ ์ž…๋‹ˆ๋‹ค.

The e-mail address you want to send an alarm to when it is detected.

webhook

ํƒ์ง€๊ฐ€ ๋˜์—ˆ์„ ๋•Œ ์•Œ๋žŒ์„ ์ „์†กํ•˜๊ณ ์ž ํ•˜๋Š” ์›นํ›… ์ฃผ์†Œ ์ž…๋‹ˆ๋‹ค. Slack, Teams์™€ ์—ฐ๋™ํ•˜์—ฌ ์‚ฌ์šฉ์ด ๊ฐ€๋Šฅํ•ฉ๋‹ˆ๋‹ค.

Web hook address to send alarm when detected. Slack, Teams can be used in conjunction with.

action

(coming soon)

ํƒ์ง€ ๋˜์—ˆ์„ ๋•Œย  batch ํŒŒ์ผ(bat)์ด๋‚˜ exe ํŒŒ์ผ์„ ์‹คํ–‰ํ•˜์—ฌ ์ถ”๊ฐ€ ์กฐ์น˜๊ฐ€ ๊ฐ€๋Šฅํ•˜๋„๋ก ํ•ฉ๋‹ˆ๋‹ค.

address ์กฐ๊ฑด์ด ์žˆ๋‹ค๋ฉด, argument ๊ฐ’์œผ๋กœ ์ „๋‹ฌ ๋ฉ๋‹ˆ๋‹ค.(ํƒ์ง€ ์กฐ๊ฑด์— ๋งž๋Š” IP ๊ฐ’์ด ์—ฌ๋Ÿฌ ๊ฐœ์ธ ํ•œ๋ฒˆ ์‹คํ–‰์— ํ•˜๋‚˜์”ฉ ์ „๋‹ฌํ•˜์—ฌย  ๋ณต์ˆ˜ ์‹คํ–‰ํ•จ)

When detected, execute batch file (bat) or exe file to enable additional action. If an address condition exists, it is passed as an argument value (multiple IP values โ€‹โ€‹matching the detection condition are executed one at a time and executed multiple times)

 

ํƒ์ง€ ๋ฃฐ ํŒŒ์ผย A detect rule file can modify and use

2018-10-26ย rule_20181026.zip

 

ํ•จ๊ป˜ ์“ฐ๋ฉด ์ข‹์€ ๊ฒƒ

Windows Server Log Collector – who is use my system

Facebook Comments