利用grok对Nginx日志进行分解

提前对Nginx日志的格式进行定义，根据定义好的日志进行分解 Nginx日志格式为：

192.168.1.106 192.168.1.106 101.246.116.62 "09/Mar/2017:15:18:59 +0800" "GET" "/xxxx/data/xxxx/hls/20160505/8cd7a8680d795007/FHD/seg-0031.ts" "-" 200 901836 "-" "AppleCoreMedia/1.0.0.9A405 (iPad; U; CPU OS 5_0_1 like Mac OS X; zh_cn)" "-" 0.334 192.168.1.106 192.168.1.106 101.246.116.62 "09/Mar/2017:15:19:00 +0800" "GET" "/xxxx/data/xxxx/hls/20160505/8cd7a8680d795007/FHD/seg-0032.ts" "-" 200 1257532 "-" "AppleCoreMedia/1.0.0.9A405 (iPad; U; CPU OS 5_0_1 like Mac OS X; zh_cn)" "-" 0.415

grok规则如下：

NGUSERNAME [a-zA-Z\.\@\-\+_%]+ NGUSER %{NGUSERNAME} NGINXACCESS %{IP:host} %{IP:server_addr} %{IP:client} \"%{HTTPDATE:timestamp}\" \"%{WORD:method}\" \"%{URIPATHPARAM:request}\" "-" %{NUMBER:response} %{NUMBER:bytes} "-" %{QS:agent} "-" %{NUMBER:duration}

中间增加了Geo的ip定位功能，可以区分客户端是来自哪个地区，最后，日志分解如下：

{ "_index": "logstash-nginx-access-2017.03.09", "_type": "nginx_access", "_id": "AVqx9hOPoUu7TGFKA5Fz", "_score": null, "_source": { "@version": "1", "@timestamp": "2017-03-09T07:26:17.000Z", "path": "/var/log/messages", "host": [ "wh-01", "192.168.1.106" ], "server_addr": "192.168.1.106", "client": "117.75.3.85", "method": "GET", "request": "/xxxx/data/xxxx/hls/20160514/5db03833349436df/FHD/seg-0095.ts", "response": 200, "bytes": 1432936, "agent": "\"-\"", "duration": "0.503", "geoip": { "ip": "117.75.3.85", "country_code2": "CN", "country_code3": "CHN", "country_name": "China", "continent_code": "AS", "region_name": "22", "city_name": "Beijing", "latitude": 39.9289, "longitude": 116.38830000000002, "timezone": "Asia/Harbin", "real_region_name": "Beijing", "location": [ 116.38830000000002, 39.9289 ], "coordinates": [ 116.38830000000002, 39.9289 ] }, "type": "nginx_access" }, "fields": { "@timestamp": [ 1489044377000 ] }, "sort": [ 1489044377000 ] }

grok规则匹配可以去grok debugger进行测试匹配