Compare commits

...

18 Commits
v1.3.0 ... main

Author SHA1 Message Date
Mark Nellemann 3ef961e44b Update README.md 2024-05-17 07:53:25 +00:00
Mark Nellemann bd5e2634d5 Merge pull request 'Depedency updates and cleanup.' (#1) from updates into main
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/tag Build is passing Details
Reviewed-on: #1
2023-11-13 12:48:20 +00:00
Mark Nellemann 1acdd6a93d Cleanup.
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
2023-11-13 13:46:11 +01:00
Mark Nellemann d39837861f Update dependencies and gradle. 2023-10-02 14:25:33 +02:00
Mark Nellemann 422f1fbb71 Read from stdin and forward.
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/tag Build is passing Details
2023-02-05 14:38:54 +01:00
Mark Nellemann d3589faf9e Fix syslog message parsing error resulting in messages being cut at first whitespace.
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/tag Build is passing Details
2023-01-25 08:47:31 +01:00
Mark Nellemann c47f682c34 Improve robustness of GELF forwarding and parsing.
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/tag Build is passing Details
2023-01-22 11:44:03 +01:00
Mark Nellemann b291f87693 Update links and gradle.
continuous-integration/drone/push Build is passing Details
2023-01-06 08:06:52 +01:00
Mark Nellemann 3e7ba2f46e Cleanup and gradle update.
continuous-integration/drone/push Build is passing Details
2023-01-03 12:49:21 +01:00
Mark Nellemann a0ad98bf52 Update 'README.md'
continuous-integration/drone/push Build is passing Details
2023-01-03 08:24:18 +00:00
Mark Nellemann 9fb810c9d2 Drone CI build pipelines
continuous-integration/drone/tag Build is passing Details
2023-01-03 09:04:08 +01:00
Mark Nellemann 8895b2b110 Update README with links to other related projects. 2022-12-17 10:35:55 +01:00
Mark Nellemann 99466b0037 Bump version. 2022-12-14 08:19:11 +01:00
Mark Nellemann 2284b5eecb Merged in gelf-chunked (pull request #9)
Gelf chunked
2022-12-14 07:15:06 +00:00
Mark Nellemann cd6e15584a Correctly parse chuncked GELF messages. 2022-12-14 08:14:00 +01:00
Mark Nellemann 5104bd0750 Work on chunked GELF messages. 2022-12-13 16:59:28 +01:00
Mark Nellemann 4cc11b0587 Work on detecting chunked messages. 2022-12-13 09:14:21 +01:00
Mark Nellemann 4cb04bd687 Correct example with new syntax. 2022-12-08 13:09:21 +00:00
33 changed files with 418 additions and 255 deletions

23
.drone.yml Normal file
View File

@ -0,0 +1,23 @@
---
kind: pipeline
name: default
type: docker
steps:
- name: build
image: eclipse-temurin:8-jdk
commands:
- ./gradlew build
- name: publish
image: eclipse-temurin:8-jdk
environment:
AUTH_TOKEN: # Gitea access token ENV variable
from_secret: auth # Name of DroneCI secret exposed above
commands:
- ./gradlew packages
- for file in build/libs/*-all.jar ; do curl --user "${DRONE_REPO_OWNER}:$${AUTH_TOKEN}" --upload-file "$${file}" "https://git.data.coop/api/packages/${DRONE_REPO_OWNER}/generic/${DRONE_REPO_NAME}/${DRONE_TAG}/$(basename $file)" ; done
- for file in build/distributions/*.deb ; do curl --user "${DRONE_REPO_OWNER}:$${AUTH_TOKEN}" --upload-file "$${file}" "https://git.data.coop/api/packages/${DRONE_REPO_OWNER}/generic/${DRONE_REPO_NAME}/${DRONE_TAG}/$(basename $file)" ; done
- for file in build/distributions/*.rpm ; do curl --user "${DRONE_REPO_OWNER}:$${AUTH_TOKEN}" --upload-file "$${file}" "https://git.data.coop/api/packages/${DRONE_REPO_OWNER}/generic/${DRONE_REPO_NAME}/${DRONE_TAG}/$(basename $file)" ; done
when:
event:
- tag

102
README.md
View File

@ -1,101 +1,3 @@
# Syslog Director
All received messages are written to *stdout* and/or forwarded to remote logging destinations.
![architecture](doc/syslogd.png)
Supported incoming message formats are:
- RFC5424 UDP and TCP
- RFC3164 (BSD) UDP and TCP
- GELF format UDP and TCP (also compressed on UDP)
Supported remote logging destinations are:
- Syslog (RFC5424 over UDP)
- Graylog (GELF over UDP)
- and Grafana Loki (HTTP over TCP).
This software is free to use and is licensed under the [Apache 2.0 License](https://bitbucket.org/mnellemann/syslogd/src/master/LICENSE).
## Usage Instructions
- Install the syslogd package (*.deb* or *.rpm*) from [downloads](https://bitbucket.org/mnellemann/syslogd/downloads/) or build from source.
```text
Usage: syslogd [-dhV] [--[no-]ansi] [--[no-]stdout] [--[no-]tcp] [--[no-]udp]
[-f=<protocol>] [-p=<num>] [--to-gelf=<uri>] [--to-loki=<url>]
[--to-syslog=<uri>]
-d, --debug Enable debugging [default: 'false'].
-f, --format=<protocol> Input format: RFC-5424, RFC-3164 or GELF [default:
RFC-3164].
-h, --help Show this help message and exit.
--[no-]ansi Output in ANSI colors [default: true].
--[no-]stdout Output messages to stdout [default: true].
--[no-]tcp Listen on TCP [default: true].
--[no-]udp Listen on UDP [default: true].
-p, --port=<num> Listening port [default: 1514].
--to-gelf=<uri> Forward to Graylog <udp://host:port>.
--to-loki=<url> Forward to Grafana Loki <http://host:port>.
--to-syslog=<uri> Forward to Syslog <udp://host:port> (RFC-5424).
-V, --version Print version information and exit.
```
The default syslog port (514) requires you to run syslogd as root / administrator.
Any port number above 1024 does not require privileges and can be selected with the *-p* or *--port* option.
### Examples
Listening on the default syslog port:
```
java -jar /path/to/syslogd-x.y.z-all.jar --port 514
```
or, if installed as a *deb* or *rpm* package:
```
/opt/syslogd/bin/syslogd --port 514
```
Forwarding messages on to another log-system on a non-standard port.
```
java -jar /path/to/syslogd-x.y.z-all.jar --syslog udp://remotehost:514
```
Forwarding messages to a Graylog server in GELF format.
```
java -jar /path/to/syslogd-x.y.z-all.jar --gelf udp://remotehost:12201
```
Forwarding to a Grafana Loki server.
```
java -jar /path/to/syslogd-x.y.z-all.jar --loki http://remotehost:3100
```
If you don't want any output locally (only forwarding), you can use the ```--no-stdout``` flag.
## Notes
### IBM AIX and VIO Servers
Syslog messages from AIX (and IBM Power Virtual I/O Servers) can be troublesome with some logging solutions. These can be received with
*syslogd* and then forwarded on to your preferred logging solution.
### Forwarding to Grafana Loki
Forwarding is currently done by making HTTP connections to the Loki API, which works fine for low volume messages, but might cause issues for large volume of messages.
## Development Notes
### Test Grafana Loki
Run Loki and Grafana in local containers to test.
```shell
docker run --rm -d --name=loki -p 3100:3100 grafana/loki
docker run --rm -d --name=grafana --link loki:loki -p 3000:3000 grafana/grafana:7.1.3
```
# Repository moved
Please visit [github.com/mnellemann/syslogd](https://github.com/mnellemann/syslogd)

View File

@ -3,9 +3,9 @@ plugins {
id 'groovy'
id 'application'
id 'jacoco'
id "com.github.johnrengelman.shadow" version "7.1.2"
id "net.nemerosa.versioning" version "2.15.1"
id "nebula.ospackage" version "9.1.1"
id "com.netflix.nebula.ospackage" version "11.5.0"
id "com.github.johnrengelman.shadow" version "7.1.2"
}
repositories {
@ -13,15 +13,15 @@ repositories {
}
dependencies {
annotationProcessor 'info.picocli:picocli-codegen:4.7.0'
implementation 'info.picocli:picocli:4.7.0'
implementation 'org.slf4j:slf4j-api:2.0.5'
implementation 'org.slf4j:slf4j-simple:2.0.5'
implementation 'com.fasterxml.jackson.core:jackson-databind:2.14.1'
implementation 'com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.14.1'
annotationProcessor 'info.picocli:picocli-codegen:4.7.5'
implementation 'info.picocli:picocli:4.7.5'
implementation 'org.slf4j:slf4j-api:2.0.9'
implementation 'org.slf4j:slf4j-simple:2.0.9'
implementation 'com.fasterxml.jackson.core:jackson-databind:2.15.2'
implementation 'com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.15.2'
implementation 'org.apache.commons:commons-collections4:4.4'
testImplementation 'org.spockframework:spock-core:2.3-groovy-3.0'
testImplementation 'org.slf4j:slf4j-api:2.0.5'
}
application {
@ -38,7 +38,7 @@ test {
}
jacoco {
toolVersion = "0.8.8"
toolVersion = "0.8.10"
}
jacocoTestReport {
@ -60,7 +60,7 @@ jacocoTestCoverageVerification {
}
limit {
counter = 'BRANCH'
minimum = 0.2
minimum = 0.3
}
limit {
counter = 'CLASS'
@ -71,8 +71,6 @@ jacocoTestCoverageVerification {
}
check.dependsOn jacocoTestCoverageVerification
apply plugin: 'nebula.ospackage'
ospackage {
packageName = 'syslogd'
release = '1'
@ -122,7 +120,7 @@ jar {
}
}
tasks.create("packages") {
tasks.register("packages") {
group "build"
dependsOn ":build"
dependsOn ":buildDeb"

View File

@ -8,7 +8,7 @@ More information about the AIX errlogger is available on the IBM [knowledge cent
### Prepare the local syslog service
Configure the local syslog service to forward messages to our remote [syslogd](https://bitbucket.org/mnellemann/syslogd/) service.
Configure the local syslog service to forward messages to our remote [syslogd](https://git.data.coop/nellemann/syslogd/) service.
Create an empty local log file:
@ -39,7 +39,7 @@ refresh -s syslogd
### Forward errlogger to the local syslog
We configure the errloger to forward messages to the local syslog service.
We configure the AIX [error logger](https://www.ibm.com/docs/en/aix/7.3?topic=concepts-error-logging-overview) to forward messages to the local syslog service.
Create an odm errnotify logging template file:

View File

@ -1,11 +1,11 @@
# Syslogd as a System Service
# Syslogd as a system service
## Systemd
## For systemd
Edit the **syslogd.service** and configure required options.
To install as a systemd service, copy the [syslogd.service](syslogd.service)
file into */etc/systemd/system/*, edit the file and configure your required options.
To install as a systemd service, copy the **syslogd.service**
file into */etc/systemd/system/* and enable the service:
Enable and start the service:
```shell
systemctl daemon-reload

View File

@ -1 +1 @@
<mxfile host="drawio-plugin" modified="2022-12-07T07:21:50.025Z" agent="5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36" etag="0NRdR40T7b5zyLZSsQHk" version="20.5.3" type="embed"><diagram id="23iRSUPoRavnBvh4doch" name="Page-1"><mxGraphModel dx="1115" dy="620" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1169" pageHeight="827" math="0" shadow="0"><root><mxCell id="0"/><mxCell id="1" parent="0"/><mxCell id="10" style="edgeStyle=orthogonalEdgeStyle;orthogonalLoop=1;jettySize=auto;html=1;curved=1;sketch=1;shadow=1;" parent="1" source="2" target="3" edge="1"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="2" value="RFC--3164" style="rounded=1;whiteSpace=wrap;html=1;sketch=1;shadow=1;fillColor=#ffe6cc;strokeColor=#d79b00;" parent="1" vertex="1"><mxGeometry x="100" y="40" width="120" height="60" as="geometry"/></mxCell><mxCell id="13" style="edgeStyle=orthogonalEdgeStyle;curved=1;sketch=1;orthogonalLoop=1;jettySize=auto;html=1;shadow=1;" parent="1" source="3" target="9" edge="1"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="14" style="edgeStyle=orthogonalEdgeStyle;curved=1;sketch=1;orthogonalLoop=1;jettySize=auto;html=1;shadow=1;" parent="1" source="3" target="6" edge="1"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="15" style="edgeStyle=orthogonalEdgeStyle;curved=1;sketch=1;orthogonalLoop=1;jettySize=auto;html=1;shadow=1;" parent="1" source="3" target="7" edge="1"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="16" style="edgeStyle=orthogonalEdgeStyle;curved=1;sketch=1;orthogonalLoop=1;jettySize=auto;html=1;shadow=1;" parent="1" source="3" target="8" edge="1"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="3" value="syslogd" style="shape=parallelogram;perimeter=parallelogramPerimeter;whiteSpace=wrap;html=1;fixedSize=1;sketch=1;rounded=1;shadow=1;fillColor=#dae8fc;strokeColor=#6c8ebf;" parent="1" vertex="1"><mxGeometry x="280" y="130" width="120" height="60" as="geometry"/></mxCell><mxCell id="11" style="edgeStyle=orthogonalEdgeStyle;curved=1;sketch=1;orthogonalLoop=1;jettySize=auto;html=1;shadow=1;" parent="1" source="4" target="3" edge="1"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="4" value="RFC--5424" style="rounded=1;whiteSpace=wrap;html=1;sketch=1;shadow=1;fillColor=#fff2cc;strokeColor=#d6b656;" parent="1" vertex="1"><mxGeometry x="20" y="130" width="120" height="60" as="geometry"/></mxCell><mxCell id="12" style="edgeStyle=orthogonalEdgeStyle;curved=1;sketch=1;orthogonalLoop=1;jettySize=auto;html=1;shadow=1;" parent="1" source="5" target="3" edge="1"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="5" value="GELF" style="rounded=1;whiteSpace=wrap;html=1;sketch=1;shadow=1;fillColor=#f8cecc;strokeColor=#b85450;" parent="1" vertex="1"><mxGeometry x="130" y="223.5" width="120" height="60" as="geometry"/></mxCell><mxCell id="6" value="Syslog&lt;br&gt;RFC-5424" style="rounded=1;whiteSpace=wrap;html=1;sketch=1;shadow=1;fillColor=#d5e8d4;strokeColor=#82b366;" parent="1" vertex="1"><mxGeometry x="570" y="80" width="120" height="60" as="geometry"/></mxCell><mxCell id="7" value="Grafana Loki" style="rounded=1;whiteSpace=wrap;html=1;sketch=1;shadow=1;fillColor=#e1d5e7;strokeColor=#9673a6;" parent="1" vertex="1"><mxGeometry x="550" y="170" width="120" height="60" as="geometry"/></mxCell><mxCell id="8" value="Graylog" style="rounded=1;whiteSpace=wrap;html=1;sketch=1;shadow=1;fillColor=#fff2cc;strokeColor=#d6b656;" parent="1" vertex="1"><mxGeometry x="420" y="250" width="120" height="60" as="geometry"/></mxCell><mxCell id="9" value="Standard&lt;br&gt;Output" style="rounded=1;whiteSpace=wrap;html=1;sketch=1;shadow=1;fillColor=#f5f5f5;strokeColor=#666666;fontColor=#333333;" parent="1" vertex="1"><mxGeometry x="410" y="30" width="120" height="60" as="geometry"/></mxCell></root></mxGraphModel></diagram></mxfile>
<mxfile host="drawio-plugin" modified="2023-02-05T13:04:08.556Z" agent="5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36" etag="__kq9uG-1g-sjP8t85Xj" version="20.5.3" type="embed"><diagram id="23iRSUPoRavnBvh4doch" name="Page-1"><mxGraphModel dx="809" dy="749" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1169" pageHeight="827" math="0" shadow="0"><root><mxCell id="0"/><mxCell id="1" parent="0"/><mxCell id="10" style="edgeStyle=orthogonalEdgeStyle;orthogonalLoop=1;jettySize=auto;html=1;curved=1;sketch=1;shadow=1;" parent="1" source="2" target="3" edge="1"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="2" value="Syslog&lt;br&gt;RFC--3164" style="rounded=1;whiteSpace=wrap;html=1;sketch=1;shadow=1;fillColor=#ffe6cc;strokeColor=#d79b00;" parent="1" vertex="1"><mxGeometry x="50" y="90" width="120" height="60" as="geometry"/></mxCell><mxCell id="13" style="edgeStyle=orthogonalEdgeStyle;curved=1;sketch=1;orthogonalLoop=1;jettySize=auto;html=1;shadow=1;" parent="1" source="3" target="9" edge="1"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="14" style="edgeStyle=orthogonalEdgeStyle;curved=1;sketch=1;orthogonalLoop=1;jettySize=auto;html=1;shadow=1;" parent="1" source="3" target="6" edge="1"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="15" style="edgeStyle=orthogonalEdgeStyle;curved=1;sketch=1;orthogonalLoop=1;jettySize=auto;html=1;shadow=1;" parent="1" source="3" target="7" edge="1"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="16" style="edgeStyle=orthogonalEdgeStyle;curved=1;sketch=1;orthogonalLoop=1;jettySize=auto;html=1;shadow=1;" parent="1" source="3" target="8" edge="1"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="3" value="syslogd" style="shape=parallelogram;perimeter=parallelogramPerimeter;whiteSpace=wrap;html=1;fixedSize=1;sketch=1;rounded=1;shadow=1;fillColor=#dae8fc;strokeColor=#6c8ebf;" parent="1" vertex="1"><mxGeometry x="280" y="130" width="120" height="60" as="geometry"/></mxCell><mxCell id="11" style="edgeStyle=orthogonalEdgeStyle;curved=1;sketch=1;orthogonalLoop=1;jettySize=auto;html=1;shadow=1;" parent="1" source="4" target="3" edge="1"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="4" value="Syslog&lt;br&gt;RFC--5424" style="rounded=1;whiteSpace=wrap;html=1;sketch=1;shadow=1;fillColor=#fff2cc;strokeColor=#d6b656;" parent="1" vertex="1"><mxGeometry x="40" y="180" width="120" height="60" as="geometry"/></mxCell><mxCell id="12" style="edgeStyle=orthogonalEdgeStyle;curved=1;sketch=1;orthogonalLoop=1;jettySize=auto;html=1;shadow=1;" parent="1" source="5" target="3" edge="1"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="5" value="GELF" style="rounded=1;whiteSpace=wrap;html=1;sketch=1;shadow=1;fillColor=#f8cecc;strokeColor=#b85450;" parent="1" vertex="1"><mxGeometry x="120" y="260" width="120" height="60" as="geometry"/></mxCell><mxCell id="6" value="Syslog&lt;br&gt;RFC-5424" style="rounded=1;whiteSpace=wrap;html=1;sketch=1;shadow=1;fillColor=#d5e8d4;strokeColor=#82b366;" parent="1" vertex="1"><mxGeometry x="570" y="80" width="120" height="60" as="geometry"/></mxCell><mxCell id="7" value="Grafana Loki" style="rounded=1;whiteSpace=wrap;html=1;sketch=1;shadow=1;fillColor=#e1d5e7;strokeColor=#9673a6;" parent="1" vertex="1"><mxGeometry x="550" y="170" width="120" height="60" as="geometry"/></mxCell><mxCell id="8" value="Graylog" style="rounded=1;whiteSpace=wrap;html=1;sketch=1;shadow=1;fillColor=#fff2cc;strokeColor=#d6b656;" parent="1" vertex="1"><mxGeometry x="420" y="250" width="120" height="60" as="geometry"/></mxCell><mxCell id="9" value="Standard&lt;br&gt;Output" style="rounded=1;whiteSpace=wrap;html=1;sketch=1;shadow=1;fillColor=#f5f5f5;strokeColor=#666666;fontColor=#333333;" parent="1" vertex="1"><mxGeometry x="410" y="30" width="120" height="60" as="geometry"/></mxCell><mxCell id="17" value="Standard&lt;br&gt;Input" style="rounded=1;whiteSpace=wrap;html=1;sketch=1;shadow=1;fillColor=#f5f5f5;strokeColor=#666666;fontColor=#333333;" vertex="1" parent="1"><mxGeometry x="180" y="30" width="90" height="60" as="geometry"/></mxCell><mxCell id="19" style="edgeStyle=orthogonalEdgeStyle;orthogonalLoop=1;jettySize=auto;html=1;curved=1;sketch=1;shadow=1;" edge="1" parent="1" source="17" target="3"><mxGeometry relative="1" as="geometry"><mxPoint x="190" y="110" as="sourcePoint"/><mxPoint x="300" y="170" as="targetPoint"/></mxGeometry></mxCell></root></mxGraphModel></diagram></mxfile>

Binary file not shown.

Before

Width:  |  Height:  |  Size: 172 KiB

After

Width:  |  Height:  |  Size: 96 KiB

View File

@ -4,7 +4,7 @@ Description=Syslog Director
[Service]
TimeoutStartSec=0
Restart=always
ExecStart=/opt/syslogd/bin/syslogd --port 514 --no-stdout --syslog=udp://localhost:1514
ExecStart=/opt/syslogd/bin/syslogd --port 514 --no-ansi
[Install]
WantedBy=default.target

View File

@ -1,5 +1,5 @@
id = syslogd
name = syslogd
group = biz.nellemann.syslogd
version = 1.3.0
version = 1.3.5
description = "Syslog Director"

Binary file not shown.

View File

@ -1,5 +1,6 @@
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-7.4-bin.zip
distributionUrl=https\://services.gradle.org/distributions/gradle-7.6.2-bin.zip
networkTimeout=10000
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists

18
gradlew vendored
View File

@ -55,7 +55,7 @@
# Darwin, MinGW, and NonStop.
#
# (3) This script is generated from the Groovy template
# https://github.com/gradle/gradle/blob/master/subprojects/plugins/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
# https://github.com/gradle/gradle/blob/HEAD/subprojects/plugins/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
# within the Gradle project.
#
# You can find Gradle at https://github.com/gradle/gradle/.
@ -80,10 +80,10 @@ do
esac
done
APP_HOME=$( cd "${APP_HOME:-./}" && pwd -P ) || exit
APP_NAME="Gradle"
# This is normally unused
# shellcheck disable=SC2034
APP_BASE_NAME=${0##*/}
APP_HOME=$( cd "${APP_HOME:-./}" && pwd -P ) || exit
# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
@ -143,12 +143,16 @@ fi
if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then
case $MAX_FD in #(
max*)
# In POSIX sh, ulimit -H is undefined. That's why the result is checked to see if it worked.
# shellcheck disable=SC3045
MAX_FD=$( ulimit -H -n ) ||
warn "Could not query maximum file descriptor limit"
esac
case $MAX_FD in #(
'' | soft) :;; #(
*)
# In POSIX sh, ulimit -n is undefined. That's why the result is checked to see if it worked.
# shellcheck disable=SC3045
ulimit -n "$MAX_FD" ||
warn "Could not set maximum file descriptor limit to $MAX_FD"
esac
@ -205,6 +209,12 @@ set -- \
org.gradle.wrapper.GradleWrapperMain \
"$@"
# Stop when "xargs" is not available.
if ! command -v xargs >/dev/null 2>&1
then
die "xargs is not available"
fi
# Use "xargs" to parse quoted args.
#
# With -n1 it outputs one arg per line, with the quotes and backslashes removed.

15
gradlew.bat vendored
View File

@ -14,7 +14,7 @@
@rem limitations under the License.
@rem
@if "%DEBUG%" == "" @echo off
@if "%DEBUG%"=="" @echo off
@rem ##########################################################################
@rem
@rem Gradle startup script for Windows
@ -25,7 +25,8 @@
if "%OS%"=="Windows_NT" setlocal
set DIRNAME=%~dp0
if "%DIRNAME%" == "" set DIRNAME=.
if "%DIRNAME%"=="" set DIRNAME=.
@rem This is normally unused
set APP_BASE_NAME=%~n0
set APP_HOME=%DIRNAME%
@ -40,7 +41,7 @@ if defined JAVA_HOME goto findJavaFromJavaHome
set JAVA_EXE=java.exe
%JAVA_EXE% -version >NUL 2>&1
if "%ERRORLEVEL%" == "0" goto execute
if %ERRORLEVEL% equ 0 goto execute
echo.
echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
@ -75,13 +76,15 @@ set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar
:end
@rem End local scope for the variables with windows NT shell
if "%ERRORLEVEL%"=="0" goto mainEnd
if %ERRORLEVEL% equ 0 goto mainEnd
:fail
rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of
rem the _cmd.exe /c_ return code!
if not "" == "%GRADLE_EXIT_CONSOLE%" exit 1
exit /b 1
set EXIT_CODE=%ERRORLEVEL%
if %EXIT_CODE% equ 0 set EXIT_CODE=1
if not ""=="%GRADLE_EXIT_CONSOLE%" exit %EXIT_CODE%
exit /b %EXIT_CODE%
:mainEnd
if "%OS%"=="Windows_NT" endlocal

View File

@ -1,10 +1 @@
/*
* This file was generated by the Gradle 'init' task.
*
* The settings file is used to specify which projects to include in your build.
*
* Detailed information about configuring a multi-project build in Gradle can be found
* in the user manual at https://docs.gradle.org/6.6.1/userguide/multi_project_builds.html
*/
rootProject.name = 'syslogd'

View File

@ -15,16 +15,6 @@
*/
package biz.nellemann.syslogd;
import biz.nellemann.syslogd.msg.SyslogMessage;
import biz.nellemann.syslogd.net.*;
import biz.nellemann.syslogd.parser.GelfParser;
import biz.nellemann.syslogd.parser.SyslogParser;
import biz.nellemann.syslogd.parser.SyslogParserRfc3164;
import biz.nellemann.syslogd.parser.SyslogParserRfc5424;
import picocli.CommandLine;
import picocli.CommandLine.Command;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.URI;
@ -34,6 +24,19 @@ import java.util.List;
import java.util.Locale;
import java.util.concurrent.Callable;
import biz.nellemann.syslogd.msg.SyslogMessage;
import biz.nellemann.syslogd.net.GelfClient;
import biz.nellemann.syslogd.net.LokiClient;
import biz.nellemann.syslogd.net.TcpServer;
import biz.nellemann.syslogd.net.UdpClient;
import biz.nellemann.syslogd.net.UdpServer;
import biz.nellemann.syslogd.parser.GelfParser;
import biz.nellemann.syslogd.parser.SyslogParser;
import biz.nellemann.syslogd.parser.SyslogParserRfc3164;
import biz.nellemann.syslogd.parser.SyslogParserRfc5424;
import picocli.CommandLine;
import picocli.CommandLine.Command;
@Command(name = "syslogd",
mixinStandardHelpOptions = true,
versionProvider = biz.nellemann.syslogd.VersionProvider.class)
@ -58,6 +61,9 @@ public class Application implements Callable<Integer>, LogReceiveListener {
@CommandLine.Option(names = "--no-stdout", negatable = true, description = "Output messages to stdout [default: true].", defaultValue = "true")
private boolean stdout;
@CommandLine.Option(names = "--no-stdin", negatable = true, description = "Forward messages from stdin [default: true].", defaultValue = "true")
private boolean stdin;
@CommandLine.Option(names = {"-f", "--format"}, description = "Input format: RFC-5424, RFC-3164 or GELF [default: RFC-3164].", defaultValue = "RFC-3164")
private String protocol;
@ -77,7 +83,6 @@ public class Application implements Callable<Integer>, LogReceiveListener {
@Override
public Integer call() throws IOException {
if(enableDebug) {
System.setProperty("org.slf4j.simpleLogger.defaultLogLevel", "DEBUG");
}
@ -115,6 +120,12 @@ public class Application implements Callable<Integer>, LogReceiveListener {
t.start();
}
if(stdin) {
InputReader inputReader = new InputReader(System.in, protocol);
inputReader.addEventListener(this);
inputReader.start();
}
if(udpServer) {
UdpServer udpServer = new UdpServer(port);
udpServer.addEventListener(this);
@ -144,7 +155,7 @@ public class Application implements Callable<Integer>, LogReceiveListener {
if(msg != null) {
if(logForwardListeners.size() > 0) {
if(!logForwardListeners.isEmpty()) {
sendForwardEvent(msg);
}

View File

@ -0,0 +1,70 @@
package biz.nellemann.syslogd;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;
import java.util.Scanner;
import biz.nellemann.syslogd.msg.SyslogMessage;
public class InputReader extends Thread {
private final Scanner input;
private final String protocol;
public InputReader(InputStream inputStream, String protocol) {
input = new Scanner(inputStream);
this.protocol = protocol;
}
@Override
public void run() {
while(input.hasNextLine()) {
SyslogMessage msg = new SyslogMessage(input.nextLine());
msg.hostname = "localhost";
msg.application = "syslogd";
String payload;
if(protocol.equalsIgnoreCase("GELF"))
payload = SyslogPrinter.toGelf(msg);
else if (protocol.equalsIgnoreCase("RFC-5424")) {
payload = SyslogPrinter.toRfc5424(msg);
} else {
payload = SyslogPrinter.toRfc3164(msg);
}
sendEvent(payload);
}
input.close();
}
private synchronized void sendEvent(String text) {
LogReceiveEvent event = new LogReceiveEvent( this, text);
for (LogReceiveListener eventListener : eventListeners) {
eventListener.onLogEvent(event);
}
}
/**
* Event Listener Configuration
*/
protected List<LogReceiveListener> eventListeners = new ArrayList<>();
public synchronized void addEventListener(LogReceiveListener listener ) {
eventListeners.add( listener );
}
public synchronized void addEventListener(List<LogReceiveListener> listeners ) {
eventListeners.addAll(listeners);
}
public synchronized void removeEventListener( LogReceiveListener l ) {
eventListeners.remove( l );
}
}

View File

@ -17,20 +17,19 @@ package biz.nellemann.syslogd;
import java.net.DatagramPacket;
import java.nio.charset.StandardCharsets;
import java.util.Arrays;
import java.util.EventObject;
public class LogReceiveEvent extends EventObject {
private static final long serialVersionUID = 1L;
//private final String message;
private final DatagramPacket packet;
/*
public LogReceiveEvent(final Object source, final String message ) {
super( source );
this.message = message;
byte[] bytes = message.getBytes();
this.packet = new DatagramPacket(bytes, bytes.length);
}
*/
public LogReceiveEvent(final Object source, final DatagramPacket packet) {
super( source );
@ -42,7 +41,7 @@ public class LogReceiveEvent extends EventObject {
}
public byte[] getBytes() {
return packet.getData();
return Arrays.copyOfRange(packet.getData(), packet.getOffset(), packet.getLength());
}
}

View File

@ -103,18 +103,18 @@ public class SyslogPrinter {
* @return
*/
public static String toGelf(SyslogMessage msg) {
StringBuilder sb = new StringBuilder("{ \"version\": \"1.1\",");
sb.append(String.format("\"host\": \"%s\",", msg.hostname));
sb.append(String.format("\"short_message\": \"%s\",", JsonUtil.encode(msg.message)));
sb.append(String.format("\"full_message\": \"%s\",", msg.structuredData));
sb.append(String.format("\"timestamp\": %d,", msg.timestamp.getEpochSecond()));
sb.append(String.format("\"level\": %d,", msg.severity.toNumber()));
sb.append(String.format("\"_facility\": \"%s\",", msg.facility));
sb.append(String.format("\"_severity\": \"%s\",", msg.severity));
sb.append(String.format("\"_application\": \"%s\",", msg.application));
if(msg.processId != null) { sb.append(String.format("\"_process-id\": \"%s\",", msg.processId)); }
if(msg.messageId != null) { sb.append(String.format("\"_message-id\": \"%s\",", msg.messageId)); }
if(msg.structuredData != null) { sb.append(String.format("\"_structured-data\": \"%s\",", msg.structuredData)); }
StringBuilder sb = new StringBuilder("{ \"version\": \"1.1\"");
sb.append(String.format(", \"host\": \"%s\"", msg.hostname));
sb.append(String.format(", \"short_message\": \"%s\"", JsonUtil.encode(msg.message)));
sb.append(String.format(", \"full_message\": \"%s\"", JsonUtil.encode(msg.structuredData)));
sb.append(String.format(", \"timestamp\": %d", msg.timestamp.getEpochSecond()));
sb.append(String.format(", \"level\": %d", msg.severity.toNumber()));
sb.append(String.format(", \"_facility\": \"%s\"", msg.facility));
sb.append(String.format(", \"_severity\": \"%s\"", msg.severity));
sb.append(String.format(", \"_application\": \"%s\"", msg.application));
if(msg.processId != null) { sb.append(String.format(", \"_process-id\": \"%s\"", msg.processId)); }
if(msg.messageId != null) { sb.append(String.format(", \"_message-id\": \"%s\"", msg.messageId)); }
if(msg.structuredData != null) { sb.append(String.format(", \"_structured-data\": \"%s\"", JsonUtil.encode(msg.structuredData))); }
sb.append("}");
return sb.toString();
}

View File

@ -36,7 +36,7 @@ public class SyslogMessage {
// The TIMESTAMP field is a formalized timestamp derived from [RFC3339].
@JsonProperty("timestamp") // 1670357783.694 - in GELF: seconds since UNIX epoch with optional decimal places for milliseconds
public Instant timestamp;
public Instant timestamp = Instant.now();
// The HOSTNAME field identifies the machine that originally sent the syslog message.
@JsonProperty("host")

View File

@ -1,13 +1,14 @@
package biz.nellemann.syslogd.net;
import biz.nellemann.syslogd.LogForwardEvent;
import biz.nellemann.syslogd.SyslogPrinter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.net.InetSocketAddress;
import java.net.SocketException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import biz.nellemann.syslogd.LogForwardEvent;
import biz.nellemann.syslogd.SyslogPrinter;
public class GelfClient extends UdpClient {
private final static Logger log = LoggerFactory.getLogger(GelfClient.class);

View File

@ -15,19 +15,22 @@
*/
package biz.nellemann.syslogd.net;
import biz.nellemann.syslogd.LogForwardEvent;
import biz.nellemann.syslogd.LogForwardListener;
import biz.nellemann.syslogd.SyslogPrinter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.*;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
import java.nio.charset.StandardCharsets;
import java.util.concurrent.ArrayBlockingQueue;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import biz.nellemann.syslogd.LogForwardEvent;
import biz.nellemann.syslogd.LogForwardListener;
import biz.nellemann.syslogd.SyslogPrinter;
public class LokiClient implements LogForwardListener, Runnable {
private final static Logger log = LoggerFactory.getLogger(LokiClient.class);
@ -84,7 +87,7 @@ public class LokiClient implements LogForwardListener, Runnable {
while (true) {
try {
send(blockingQueue.take());
} catch (Exception e) {
} catch (MalformedURLException | InterruptedException e) {
log.warn(e.getMessage());
}
}

View File

@ -26,10 +26,16 @@ import java.net.ServerSocket;
import java.net.Socket;
import java.nio.charset.StandardCharsets;
import java.util.List;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.ArrayList;
public class TcpServer {
private final static Logger log = LoggerFactory.getLogger(TcpServer.class);
private final int port;
private ServerSocket serverSocket;
@ -91,14 +97,14 @@ public class TcpServer {
sendEvent(inputLine);
}
} catch (IOException e) {
e.printStackTrace();
log.warn("run() - read error: {}", e.getMessage());
}
try {
in.close();
clientSocket.close();
} catch (IOException e) {
e.printStackTrace();
log.warn("run() - close error: {}", e.getMessage());
}
}

View File

@ -15,18 +15,22 @@
*/
package biz.nellemann.syslogd.net;
import biz.nellemann.syslogd.LogReceiveEvent;
import biz.nellemann.syslogd.LogReceiveListener;
import java.io.IOException;
import java.net.DatagramPacket;
import java.net.DatagramSocket;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import biz.nellemann.syslogd.LogReceiveEvent;
import biz.nellemann.syslogd.LogReceiveListener;
public class UdpServer extends Thread {
private final static Logger log = LoggerFactory.getLogger(UdpServer.class);
protected DatagramSocket socket;
protected boolean listen = true;
@ -34,6 +38,7 @@ public class UdpServer extends Thread {
socket = new DatagramSocket(port);
}
@Override
public void run() {
byte[] buf = new byte[8192];
@ -44,8 +49,8 @@ public class UdpServer extends Thread {
socket.receive(packet);
//String packetData = new String(packet.getData(), packet.getOffset(), packet.getLength(), StandardCharsets.UTF_8);
sendEvent(packet);
} catch (Exception e) {
e.printStackTrace();
} catch (IOException e) {
log.error("run() - error: {}", e.getMessage());
listen = false;
}
}

View File

@ -1,43 +1,100 @@
package biz.nellemann.syslogd.parser;
import biz.nellemann.syslogd.msg.SyslogMessage;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
import java.time.Instant;
import java.util.Arrays;
import java.util.TreeMap;
import org.apache.commons.collections4.map.PassiveExpiringMap;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.UnsupportedEncodingException;
import java.time.Instant;
import java.util.zip.DataFormatException;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
import biz.nellemann.syslogd.msg.SyslogMessage;
/*
For more information about the GELF format, visit: https://go2docs.graylog.org/5-0/getting_in_log_data/gelf.html
*/
public class GelfParser extends SyslogParser {
private final static Logger log = LoggerFactory.getLogger(GelfParser.class);
private final ObjectMapper objectMapper;
private final int expiryInMills = 10_000;
private final PassiveExpiringMap<Integer, TreeMap<Integer, byte[]>> expiringMap = new PassiveExpiringMap<>(expiryInMills);
public GelfParser() {
objectMapper = new ObjectMapper();
objectMapper.registerModule(new JavaTimeModule());
}
/*
Magic Bytes - 2 bytes: 0x1e 0x0f
Message ID - 8 bytes: Must be the same for every chunk of this message.
Identifies the whole message and is used to reassemble the chunks later.
Generate from millisecond timestamp + hostname, for example.
Sequence number - 1 byte: The sequence number of this chunk starts at 0 and is always less than the sequence count.
Sequence count - 1 byte: Total number of chunks this message has.
All chunks MUST arrive within 5 seconds or the server will discard all chunks that have arrived or are in the process of arriving.
A message MUST NOT consist of more than 128 chunks.
*/
private SyslogMessage parseChunked(byte[] input) {
if(input.length < 12) return null;
byte[] messageId = { input[2], input[3], input[4], input[5], input[6], input[7], input[8], input[9] };
byte seqNumber = input[10];
byte seqTotal = input[11];
byte[] payload = Arrays.copyOfRange(input, 12, input.length);
log.debug("parseChunked() - msgId: {}, seqNo: {}, seqTot: {}, payload: {}", messageId, seqNumber, seqTotal, byteArrayToString(payload));
// messageId byte[] to int
int id = 0;
for (byte b : messageId) {
id = (id << 8) + (b & 0xFF);
}
TreeMap<Integer, byte[]> integerTreeMap;
if(expiringMap.containsKey(id)) {
integerTreeMap = expiringMap.get(id);
} else {
integerTreeMap = new TreeMap<>();
}
integerTreeMap.put((int)seqNumber, payload);
expiringMap.put(id, integerTreeMap);
if(integerTreeMap.size() >= seqTotal) {
StringBuilder sb = new StringBuilder();
integerTreeMap.forEach( (i, p) -> {
sb.append(byteArrayToString(p));
});
return parse(sb.toString());
}
return null;
}
@Override
public SyslogMessage parse(String input) {
if(!input.startsWith("{")) return null; // Avoid trying to parse non-JSON content
SyslogMessage message = null;
try {
message = objectMapper.readValue(input, SyslogMessage.class);
} catch (JsonProcessingException e) {
log.warn("parse() - error: {}", e.getMessage());
log.debug("parse() - error: {}", e.getMessage());
}
return message;
}
/*
https://go2docs.graylog.org/5-0/getting_in_log_data/gelf.html
zlib signatures at offset 0
78 01 : No Compression (no preset dictionary)
78 5E : Best speed (no preset dictionary)
@ -54,18 +111,20 @@ public class GelfParser extends SyslogParser {
@Override
public SyslogMessage parse(byte[] input) {
String text = null;
if(input[0] == (byte)0x78 && input[1] == (byte)0x9c) { // Compressed data - TODO: better detection ?
try {
text = decompress(input);
} catch (DataFormatException | UnsupportedEncodingException e) {
log.error("parse() - error: {}", e.getMessage());
}
} else {
text = byteArrayToString(input);
if(input.length < 8) return null; // TODO: Find proper minimum input length ?
// Compressed data: 0x78 0x9c
if(input[0] == (byte)0x78 && input[1] == (byte)0x9c) {
input = decompress(input);
}
return parse(text);
// Magic Bytes: 0x1e 0x0f
if(input[0] == (byte)0x1e && input[1] == (byte)0x0f) {
return parseChunked(input);
}
return parse(byteArrayToString(input));
}

View File

@ -8,6 +8,10 @@ public class JsonUtil {
public static String encode(String input) {
if(input == null) {
return "";
}
StringBuilder output = new StringBuilder();
for (int i = 0; i < input.length(); i++) {

View File

@ -15,16 +15,20 @@
*/
package biz.nellemann.syslogd.parser;
import biz.nellemann.syslogd.msg.SyslogMessage;
import java.io.UnsupportedEncodingException;
import java.nio.charset.StandardCharsets;
import java.time.Instant;
import java.util.zip.DataFormatException;
import java.util.zip.Inflater;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import biz.nellemann.syslogd.msg.SyslogMessage;
public abstract class SyslogParser {
private final static Logger log = LoggerFactory.getLogger(SyslogParser.class);
public abstract SyslogMessage parse(final String input);
public abstract SyslogMessage parse(final byte[] input);
@ -62,17 +66,24 @@ public abstract class SyslogParser {
}
protected String decompress(byte[] data) throws UnsupportedEncodingException, DataFormatException {
protected byte[] decompress(byte[] data) {
// Decompress the bytes
Inflater decompressor = new Inflater();
decompressor.setInput(data, 0, data.length);
byte[] result = new byte[data.length * 2];
int resultLength = decompressor.inflate(result);
decompressor.end();
try {
// Decompress the bytes
Inflater decompressor = new Inflater();
decompressor.setInput(data, 0, data.length);
//byte[] result = new byte[data.length * 2];
decompressor.inflate(result);
decompressor.end();
// Decode the bytes into a String
return new String(result, 0, resultLength, StandardCharsets.UTF_8);
// Decode the bytes into a String
//uncompressed = new String(result, 0, resultLength, StandardCharsets.UTF_8);
} catch (DataFormatException e) {
log.error("decompress() - error: {}", e.getMessage());
}
return result;
}
}

View File

@ -15,24 +15,26 @@
*/
package biz.nellemann.syslogd.parser;
import biz.nellemann.syslogd.LogReceiveEvent;
import biz.nellemann.syslogd.msg.Facility;
import biz.nellemann.syslogd.msg.Severity;
import biz.nellemann.syslogd.msg.SyslogMessage;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.time.*;
import java.time.Instant;
import java.time.OffsetDateTime;
import java.time.ZoneId;
import java.time.format.DateTimeFormatter;
import java.time.format.DateTimeParseException;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import biz.nellemann.syslogd.msg.Facility;
import biz.nellemann.syslogd.msg.Severity;
import biz.nellemann.syslogd.msg.SyslogMessage;
public class SyslogParserRfc3164 extends SyslogParser {
private final static Logger log = LoggerFactory.getLogger(SyslogParserRfc3164.class);
private final Pattern pattern = Pattern.compile("^<(\\d{1,3})>(\\D{3}\\s+\\d{1,2} \\d{2}:\\d{2}:\\d{2})\\s+(Message forwarded from \\S+:|\\S+)\\s+([^\\s:]+):?\\s+(.*)", Pattern.CASE_INSENSITIVE);
private final Pattern pattern = Pattern.compile("^<(\\d{1,3})>(\\D{3}\\s+\\d{1,2} \\d{2}:\\d{2}:\\d{2})\\s+(Message forwarded from \\S+:|\\S+:?)\\s+([^\\s:]+):?\\s+(.*)", Pattern.CASE_INSENSITIVE);
private final DateTimeFormatter dateTimeFormatter = DateTimeFormatter.ofPattern("yyyy MMM [ ]d HH:mm:ss").withZone(ZoneId.systemDefault());
/**
@ -90,6 +92,7 @@ public class SyslogParserRfc3164 extends SyslogParser {
* @param dateString
* @return
*/
@Override
public Instant parseTimestamp(String dateString) {
// We need to add current year to parse date correctly

View File

@ -15,26 +15,26 @@
*/
package biz.nellemann.syslogd.parser;
import biz.nellemann.syslogd.LogReceiveEvent;
import biz.nellemann.syslogd.msg.Severity;
import biz.nellemann.syslogd.msg.Facility;
import biz.nellemann.syslogd.msg.SyslogMessage;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.time.*;
import java.time.Instant;
import java.util.Arrays;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import biz.nellemann.syslogd.msg.Facility;
import biz.nellemann.syslogd.msg.Severity;
import biz.nellemann.syslogd.msg.SyslogMessage;
public class SyslogParserRfc5424 extends SyslogParser {
private final static Logger log = LoggerFactory.getLogger(SyslogParserRfc5424.class);
private final Pattern pattern = Pattern.compile("^<(\\d{1,3})>(\\d+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\[.*\\]|-)\\s+(\\S+)", Pattern.CASE_INSENSITIVE);
private final Pattern pattern = Pattern.compile("^<(\\d{1,3})>(\\d+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\[.*\\]|-)\\s+(.*)", Pattern.CASE_INSENSITIVE);
/**
* Parses [rfc5424](https://tools.ietf.org/html/rfc5424) syslog messages.
@ -99,6 +99,7 @@ public class SyslogParserRfc5424 extends SyslogParser {
* @param dateString
* @return
*/
@Override
public Instant parseTimestamp(String dateString) {
/*

View File

@ -18,9 +18,6 @@ class GelfParserTest extends Specification {
void "uncompressed GELF message"() {
/*
0x7b 0x220x760x650x720x730x690x6f0x6e0x220x3a0x220x310x2e0x310x220x2c0x220x680x6f0x730x740x220x3a0x220x700x6f0x700x2d0x6f0x730x2e0x6c0x6f0x630x610x6c0x640x6f0x6d0x610x690x6e0x220x2c0x220x730x680x6f0x720x740x5f0x6d0x650x730x730x610x670x650x220x3a0x220x6d0x610x690x6e0x280x290x200x2d0x200x530x740x610x720x740x690x6e0x670x200x560x540x440x2d0x430x610x6d0x650x720x610x220x2c0x220x660x750x6c0x6c0x5f0x6d0x650x730x730x610x670x650x220x3a0x220x6d0x610x690x6e0x280x290x200x2d0x200x530x740x610x720x740x690x6e0x670x200x560x540x440x2d0x430x610x6d0x650x720x610x5c0x5c0x6e0x220x2c0x220x740x690x6d0x650x730x740x610x6d0x700x220x3a0x310x360x370x300x330x350x370x370x380x330x2e0x360x390x340x2c0x220x6c0x650x760x650x6c0x220x3a0x340x2c0x220x5f0x740x680x720x650x610x640x5f0x6e0x610x6d0x650x220x3a0x220x6d0x610x690x6e0x220x2c0x220x5f0x6c0x6f0x670x670x650x720x5f0x6e0x610x6d0x650x220x3a0x220x760x740x640x2e0x630x610x6d0x650x720x610x2e0x410x700x700x6c0x690x630x610x740x690x6f0x6e0x220x7d0x0a0x560x1f0x210x470x710x2c0xeb0x090x740x8e0x060x6f0x7f0xe70x7a0x330x890xb80xe10x6e0xa40x090xe90x9f0x870x1c0x290x900xcd0x880x4b0xea0xce0x9b0xf30x660xbc0x880x7b0x6f0x410x610x8f0x4c0xf00x570xaa0x590x620x180x8b0x820x730x040x350x3e0x120x8b0x930x7b0x130x780x5b0xb50x150x870x4f0xd00x5c0xb60xec0xfb0x070x9a0x690xc70xe8
*/
setup:
def input = '{"version":"1.1","host":"pop-os.localdomain","short_message":"main() - Starting VTD-Camera","full_message":"main() - Starting VTD-Camera\\n","timestamp":1670357783.694,"level":4,"_thread_name":"main","_logger_name":"vtd.camera.Application"}'
@ -39,9 +36,6 @@ class GelfParserTest extends Specification {
void "compressed GELF message"() {
// GELF Magic Bytes: 0x1e, 0x0f
setup:
byte[] input = [ 120, -100, -51, 81, 77, 107, -61, 48, 12, -3, 43, -63, -25, -38, -115, -77, -75, -51, 2, -127, -11, -80, -61, 96, -73, 94, 11, -63, 56, 90, -30, -59, 31, -63, -106, -53, -54, -40, 127, -97, 92, 86, -40, 79, -40, 69, -78, -97, -11, -12, -98, -28, 47, 118, -127, -104, 76, -16, -84, 99, 82, 72, -74, 97, 115, 72, 72, 23, -77, 114, 89, 115, 73, 65, -14, 102, 87, 11, -56, 92, -125, -57, -88, 44, -105, 66, 7, -73, 102, 4, 97, 60, 66, -12, -54, 18, 45, -51, 33, -30, -32, 32, 37, 53, 1, -15, -31, 66, -43, -3, 49, -29, 76, -39, 104, -123, -92, 113, -54, 90, 83, -63, 75, 121, -86, 114, 42, 84, 7, -67, 83, 113, 17, 30, -84, 5, -89, -68, 127, -98, -100, 50, -74, 40, 84, 8, 94, 81, -113, -90, -118, -32, 2, -62, 113, 28, 35, -79, 123, -39, -18, -124, 108, -91, 104, -102, 7, 33, -27, -95, 74, 4, 82, -13, -41, -79, -9, -39, 22, 43, -17, -108, -2, -127, -109, -77, 39, 47, 104, -56, 8, 42, -73, -78, 78, -18, 15, -11, -82, -106, -121, -89, 86, -44, -113, -5, 13, -77, 100, -52, -78, -114, 78, -125, 90, -41, 65, -121, 76, -21, -67, -110, -31, 113, 97, -65, 88, 113, 69, -128, -93, 61, -57, -126, -39, 48, 77, 16, -17, -80, -6, 8, 57, -118, 99, -119, 39, -48, 57, 26, -68, -2, -99, -21, -51, 36, -14, 13, 55, 34, 77, 72, -1, 60, -28, 72, -126, 108, 70, 92, 83, 119, -34, -98, -73, -29, 34, 110, -67, 5, -119, -35, 53, -63, 95, -88, 102, -115, 97, 44, 8, -50, 17, -44, 120, 87, 44, 76, -18, 77, -32, 109, -35, -42, 28, 62, 65, -13, -122, 125, -1, 0, -40, 60, -57, -72 ];
@ -59,4 +53,48 @@ class GelfParserTest extends Specification {
msg.timestamp.toString() == "2022-12-08T12:16:38.046Z"
}
void "chunked GELF message"() {
setup:
byte[] chunk1 = [30, 15, -103, 51, -96, -10, -51, -31, 39, -41, 0, 2, 123, 34, 118, 101, 114, 115, 105, 111, 110, 34, 58, 34, 49, 46, 49, 34, 44, 34, 104, 111, 115, 116, 34, 58, 34, 105, 112, 45, 49, 48, 45, 49, 45, 49, 48, 49, 45, 49, 52, 46, 101, 117, 45, 99, 101, 110, 116, 114, 97, 108, 45, 49, 46, 99, 111, 109, 112, 117, 116, 101, 46, 105, 110, 116, 101, 114, 110, 97, 108, 34, 44, 34, 115, 104, 111, 114, 116, 95, 109, 101, 115, 115, 97, 103, 101, 34, 58, 34, 101, 118, 101, 110, 116, 61, 65, 117, 116, 104, 101, 110, 116, 105, 99, 97, 116, 105, 111, 110, 83, 117, 99, 99, 101, 115, 115, 69, 118, 101, 110, 116, 32, 117, 115, 101, 114, 110, 97, 109, 101, 61, 109, 97, 114, 107, 46, 110, 101, 108, 108, 101, 109, 97, 110, 110, 64, 103, 109, 97, 105, 108, 46, 99, 111, 109, 32, 116, 101, 110, 97, 110, 116, 61, 50, 32, 114, 101, 109, 111, 116, 101, 65, 100, 100, 114, 101, 115, 115, 61, 49, 56, 53, 46, 49, 56, 49, 46, 50, 50, 51, 46, 49, 49, 55, 32, 115, 101, 115, 115, 105, 111, 110, 73, 100, 61, 110, 117, 108, 108, 34, 44, 34, 102, 117, 108, 108, 95, 109, 101, 115, 115, 97, 103, 101, 34, 58, 34, 101, 118, 101, 110, 116, 61, 65, 117, 116, 104, 101, 110, 116, 105, 99, 97, 116, 105, 111, 110, 83, 117, 99, 99, 101, 115, 115, 69, 118, 101, 110, 116, 32, 117, 115, 101, 114, 110, 97, 109, 101, 61, 109, 97, 114, 107, 46, 110, 101, 108, 108, 101, 109, 97, 110, 110, 64, 103, 109, 97, 105, 108, 46, 99, 111, 109, 32, 116, 101, 110, 97, 110, 116, 61, 50, 32, 114, 101, 109, 111, 116, 101, 65, 100, 100, 114, 101, 115, 115, 61, 49, 56, 53, 46, 49, 56, 49, 46, 50, 50, 51, 46, 49, 49, 55, 32, 115, 101, 115, 115, 105, 111, 110, 73, 100, 61, 110, 117, 108, 108, 92, 110, 34, 44, 34, 116, 105, 109, 101, 115, 116, 97, 109, 112, 34, 58, 49, 54, 55, 49, 48, 48, 48, 53, 53, 54, 46, 56, 48, 50, 44, 34, 108, 101, 118, 101, 108, 34, 58, 54, 44, 34, 95, 97, 112, 112, 95, 99, 111, 117, 110, 116, 114, 121, 34, 58, 34, 100, 107, 34, 44, 34, 95, 97, 112, 112, 95, 110, 97, 109, 101, 34, 58, 34, 109, 105, 110, 116, 114, 34, 44, 34, 95, 108, 111, 103, 103, 101, 114, 95, 110, 97, 109, 101, 34, 58, 34, 97, 106, 111, 117, 114, 46, 65, 106, 111, 117, 114, 83, 101, 99, 117, 114, 105, 116, 121, 83, 117, 99, 99, 101, 115, 115, 69, 118, 101, 110, 116, 76, 105, 115, 116, 101, 110, 101, 114, 34, 44, 34, 95, 115, 101, 114, 118, 101, 114, 95, 117, 114, 108]
byte[] chunk2 = [30, 15, -103, 51, -96, -10, -51, -31, 39, -41, 1, 2, 34, 58, 34, 104, 116, 116, 112, 115, 58, 92, 47, 92, 47, 100, 107, 46, 109, 105, 110, 116, 114, 46, 97, 112, 112, 34, 44, 34, 95, 97, 112, 112, 95, 101, 110, 118, 34, 58, 34, 112, 114, 111, 100, 34, 44, 34, 95, 116, 104, 114, 101, 97, 100, 95, 110, 97, 109, 101, 34, 58, 34, 104, 116, 116, 112, 45, 110, 105, 111, 45, 56, 48, 56, 48, 45, 101, 120, 101, 99, 45, 52, 34, 125]
when:
syslogParser.parse(chunk1)
SyslogMessage msg = syslogParser.parse(chunk2)
then:
msg.message == "event=AuthenticationSuccessEvent username=mark.nellemann@gmail.com tenant=2 remoteAddress=185.181.223.117 sessionId=null"
}
void "chunked GELF unordered message"() {
setup:
byte[] chunk1 = [30, 15, -103, 51, -96, -10, -51, -31, 39, -41, 0, 2, 123, 34, 118, 101, 114, 115, 105, 111, 110, 34, 58, 34, 49, 46, 49, 34, 44, 34, 104, 111, 115, 116, 34, 58, 34, 105, 112, 45, 49, 48, 45, 49, 45, 49, 48, 49, 45, 49, 52, 46, 101, 117, 45, 99, 101, 110, 116, 114, 97, 108, 45, 49, 46, 99, 111, 109, 112, 117, 116, 101, 46, 105, 110, 116, 101, 114, 110, 97, 108, 34, 44, 34, 115, 104, 111, 114, 116, 95, 109, 101, 115, 115, 97, 103, 101, 34, 58, 34, 101, 118, 101, 110, 116, 61, 65, 117, 116, 104, 101, 110, 116, 105, 99, 97, 116, 105, 111, 110, 83, 117, 99, 99, 101, 115, 115, 69, 118, 101, 110, 116, 32, 117, 115, 101, 114, 110, 97, 109, 101, 61, 109, 97, 114, 107, 46, 110, 101, 108, 108, 101, 109, 97, 110, 110, 64, 103, 109, 97, 105, 108, 46, 99, 111, 109, 32, 116, 101, 110, 97, 110, 116, 61, 50, 32, 114, 101, 109, 111, 116, 101, 65, 100, 100, 114, 101, 115, 115, 61, 49, 56, 53, 46, 49, 56, 49, 46, 50, 50, 51, 46, 49, 49, 55, 32, 115, 101, 115, 115, 105, 111, 110, 73, 100, 61, 110, 117, 108, 108, 34, 44, 34, 102, 117, 108, 108, 95, 109, 101, 115, 115, 97, 103, 101, 34, 58, 34, 101, 118, 101, 110, 116, 61, 65, 117, 116, 104, 101, 110, 116, 105, 99, 97, 116, 105, 111, 110, 83, 117, 99, 99, 101, 115, 115, 69, 118, 101, 110, 116, 32, 117, 115, 101, 114, 110, 97, 109, 101, 61, 109, 97, 114, 107, 46, 110, 101, 108, 108, 101, 109, 97, 110, 110, 64, 103, 109, 97, 105, 108, 46, 99, 111, 109, 32, 116, 101, 110, 97, 110, 116, 61, 50, 32, 114, 101, 109, 111, 116, 101, 65, 100, 100, 114, 101, 115, 115, 61, 49, 56, 53, 46, 49, 56, 49, 46, 50, 50, 51, 46, 49, 49, 55, 32, 115, 101, 115, 115, 105, 111, 110, 73, 100, 61, 110, 117, 108, 108, 92, 110, 34, 44, 34, 116, 105, 109, 101, 115, 116, 97, 109, 112, 34, 58, 49, 54, 55, 49, 48, 48, 48, 53, 53, 54, 46, 56, 48, 50, 44, 34, 108, 101, 118, 101, 108, 34, 58, 54, 44, 34, 95, 97, 112, 112, 95, 99, 111, 117, 110, 116, 114, 121, 34, 58, 34, 100, 107, 34, 44, 34, 95, 97, 112, 112, 95, 110, 97, 109, 101, 34, 58, 34, 109, 105, 110, 116, 114, 34, 44, 34, 95, 108, 111, 103, 103, 101, 114, 95, 110, 97, 109, 101, 34, 58, 34, 97, 106, 111, 117, 114, 46, 65, 106, 111, 117, 114, 83, 101, 99, 117, 114, 105, 116, 121, 83, 117, 99, 99, 101, 115, 115, 69, 118, 101, 110, 116, 76, 105, 115, 116, 101, 110, 101, 114, 34, 44, 34, 95, 115, 101, 114, 118, 101, 114, 95, 117, 114, 108]
byte[] chunk2 = [30, 15, -103, 51, -96, -10, -51, -31, 39, -41, 1, 2, 34, 58, 34, 104, 116, 116, 112, 115, 58, 92, 47, 92, 47, 100, 107, 46, 109, 105, 110, 116, 114, 46, 97, 112, 112, 34, 44, 34, 95, 97, 112, 112, 95, 101, 110, 118, 34, 58, 34, 112, 114, 111, 100, 34, 44, 34, 95, 116, 104, 114, 101, 97, 100, 95, 110, 97, 109, 101, 34, 58, 34, 104, 116, 116, 112, 45, 110, 105, 111, 45, 56, 48, 56, 48, 45, 101, 120, 101, 99, 45, 52, 34, 125]
when:
syslogParser.parse(chunk2)
SyslogMessage msg = syslogParser.parse(chunk1)
then:
msg.message == "event=AuthenticationSuccessEvent username=mark.nellemann@gmail.com tenant=2 remoteAddress=185.181.223.117 sessionId=null"
}
void "junk GET request"() {
setup:
def input = 'GET /'
when:
SyslogMessage msg = syslogParser.parse(input)
then:
msg == null
}
}

View File

@ -39,4 +39,14 @@ class JsonUtilTest extends Specification {
result == 'here it comes " to wreck the day...'
}
def "test newline decode"() {
setup:
def testQuote = 'here it comes \n to wreck the day...'
when:
def result = JsonUtil.decode(testQuote)
then:
result == 'here it comes \n to wreck the day...'
}
}

View File

@ -50,13 +50,13 @@ class SyslogParserRfc3164Test extends Specification {
void "test rfc3164 normal message"() {
setup:
def input = "<13>Sep 23 08:53:28 xps13 mark: adfdfdf3432434"
def input = "<13>Sep 23 08:53:28 xps13 mark: adfdfdf3432434 abcdefghijklmnopqrstuvwxyz"
when:
SyslogMessage msg = syslogParser.parse(input)
then:
msg.message == "adfdfdf3432434"
msg.message == "adfdfdf3432434 abcdefghijklmnopqrstuvwxyz"
msg.hostname == "xps13"
msg.application == "mark"
}

View File

@ -40,6 +40,7 @@ class SyslogParserRfc5424Test extends Specification {
msg.application == "su"
msg.messageId == "ID47"
msg.processId == null
msg.message == "BOM'su root' failed for lonvick on /dev/pts/8"
}
void "test rfc5424 example2 message"() {

View File

@ -10,10 +10,23 @@ class SyslogPrinterTest extends Specification {
void setup() {
}
void "to plain"() {
setup:
SyslogParser syslogParser = new SyslogParserRfc5424();
String input = '<13>1 2020-09-23T08:57:30.950699+02:00 xps13 mark - - [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] adfdfdf3432434565656 abcdefghijklmnopqrstuvwxyz'
SyslogMessage msg = syslogParser.parse(input)
when:
String output = SyslogPrinter.toString(msg)
then:
output.endsWith("abcdefghijklmnopqrstuvwxyz")
}
void "test toGelf"() {
setup:
SyslogParser syslogParser = new SyslogParserRfc5424();
String input = '<13>1 2020-09-23T08:57:30.950699+02:00 xps13 mark - - [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] adfdfdf3432434565656'
String input = '<13>1 2020-09-23T08:57:30.950699+02:00 xps13 mark - - [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] adfdfdf3432434565656 abcdefghijklmnopqrstuvwxyz'
SyslogMessage msg = syslogParser.parse(input)
when:
@ -26,14 +39,14 @@ class SyslogPrinterTest extends Specification {
void "test toLoki"() {
setup:
SyslogParser syslogParser = new SyslogParserRfc5424();
String input = '<13>1 2020-09-23T08:57:30.950699+02:00 xps13 mark - - [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] adfdfdf3432434565656'
String input = '<13>1 2020-09-23T08:57:30.950699+02:00 xps13 mark - - [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] adfdfdf3432434565656 abcdefghijklmnopqrstuvwxyz'
SyslogMessage msg = syslogParser.parse(input)
when:
String output = SyslogPrinter.toLoki(msg)
then:
output == '{ "streams": [ { "stream": { "hostname": "xps13", "facility": "user", "level": "notice", "application": "mark"}, "values": [ [ "1600845200000000000", "[user.notice] xps13 mark adfdfdf3432434565656" ] ] } ] }'
output == '{ "streams": [ { "stream": { "hostname": "xps13", "facility": "user", "level": "notice", "application": "mark"}, "values": [ [ "1600845200000000000", "[user.notice] xps13 mark adfdfdf3432434565656 abcdefghijklmnopqrstuvwxyz" ] ] } ] }'
}
}