nellemann
/
syslogd
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 3 Wiki Activity

Refactor argument parsing and improve LokiClient send.

This commit is contained in:
Mark Nellemann 2021-03-17 10:45:13 +01:00
parent 883963b033
commit 93874f1f33
4 changed files with 74 additions and 62 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

56
README.md
View File

@ -1,33 +1,35 @@
# Syslog Server # Syslog Server
All received messages are written to *stdout* and/or forwarded to another syslog server. All received messages are written to *stdout* and/or forwarded to a remote logging solution.
The syslog server is able to listen on both UDP and TCP and parses syslog messages in either RFC5424 or RFC3164 (BSD) format. The syslog server is able to listen on both UDP and TCP and parses syslog messages in either RFC5424 or RFC3164 (BSD) format.
The default syslog port (514) requires you to run syslogd as root / administrator. The default syslog port (514) requires you to run syslogd as root / administrator.
If you do not wish to do so, you can choose a port number (with the *-p* or *--port* flag) above 1024. If you do not wish to do so, you can choose a port number (with the *-p* or *--port* flag) above 1024.
Supported remote logging solutions are Syslog (RFC5424 over UDP), Graylog (GELF over UDP) and Grafana Loki.
## Usage Instructions ## Usage Instructions
- Install the syslogd package (*.deb* or *.rpm*) from [downloads](https://bitbucket.org/mnellemann/syslogd/downloads/) or build from source. - Install the syslogd package (*.deb* or *.rpm*) from [downloads](https://bitbucket.org/mnellemann/syslogd/downloads/) or build from source.
- Run *bin/syslogd*, use the *-h* option for help :) - Run *bin/syslogd*, use the *-h* option for help :)
```` ```text
Usage: syslogd [-dghV] [--[no-]ansi] [--[no-]stdout] [--[no-]tcp] [--[no-]udp] Usage: syslogd [-dhV] [--[no-]ansi] [--[no-]stdout] [--[no-]tcp] [--[no-]udp]
[--rfc5424] [-f=<host>] [-p=<port>] [--rfc5424] [-g=<uri>] [-l=<url>] [-p=<num>] [-s=<uri>]
Syslog Server -d, --debug Enable debugging [default: 'false'].
-d, --debug Enable debugging [default: 'false']. -g, --gelf=<uri> Forward to Graylog <udp://host:port>.
-f, --forward=<host> Forward to UDP host[:port] (RFC-5424). -h, --help Show this help message and exit.
-g, --gelf Forward in Graylog (GELF) JSON format. -l, --loki=<url> Forward to Grafana Loki <http://host:port>.
-h, --help Show this help message and exit. --[no-]ansi Output ANSI colors [default: true].
--[no-]ansi Output ANSI colors [default: true]. --[no-]stdout Output messages to stdout [default: true].
--[no-]stdout Output messages to stdout [default: true]. --[no-]tcp Listen on TCP [default: true].
--[no-]tcp Listen on TCP [default: true]. --[no-]udp Listen on UDP [default: true].
--[no-]udp Listen on UDP [default: true]. -p, --port=<num> Listening port [default: 514].
-p, --port=<port> Listening port [default: 514]. --rfc5424 Parse RFC-5424 messages [default: RFC-3164].
--rfc5424 Parse RFC-5424 messages [default: RFC-3164]. -s, --syslog=<uri> Forward to Syslog <udp://host:port> (RFC-5424).
-V, --version Print version information and exit. -V, --version Print version information and exit.
```` ```
### Examples ### Examples
@ -46,30 +48,40 @@ or, if installed as a *deb* or *rpm* package:
Listening on the standard syslog port (requires root privileges) and forwarding messages on to another log-system on a non-standard port. Listening on the standard syslog port (requires root privileges) and forwarding messages on to another log-system on a non-standard port.
``` ```
java -jar /path/to/syslogd-x.y.z-all.jar --forward remotehost:1514 java -jar /path/to/syslogd-x.y.z-all.jar --syslog udp://remotehost:514
``` ```
Forwarding to a Graylog server in GELF format. Forwarding to a Graylog server in GELF format.
``` ```
java -jar /path/to/syslogd-x.y.z-all.jar --forward remotehost:12201 --gelf java -jar /path/to/syslogd-x.y.z-all.jar --gelf udp://remotehost:12201
``` ```
Forwarding to a Grafana Loki server.
```
java -jar /path/to/syslogd-x.y.z-all.jar --loki http://remotehost:3100
```
If you don't want any output locally (only forwarding), you can use the ```--no-stdout``` flag. If you don't want any output locally (only forwarding), you can use the ```--no-stdout``` flag.
## Notes ## Notes
### IBM AIX and VIO Servers
Syslog messages from AIX (and IBM Power Virtual I/O Servers) can be troublesome with some logging solutions. These can be received with Syslog messages from AIX (and IBM Power Virtual I/O Servers) can be troublesome with some logging solutions. These can be received with
syslogd and optionally forwarded on to Graylog, Splunk or other logging solutions. syslogd and then forwarded on to your preferred logging solution.
## Development ## Development Notes
### Test Grafana Loki ### Test Grafana Loki
Run Loki and Grafana in local containers to test.
```shell ```shell
docker run --rm -d --name=loki -p 3100:3100 grafana/loki docker run --rm -d --name=loki -p 3100:3100 grafana/loki
docker run --rm -d --name=grafana --link loki:loki -p 3000:3000 grafana/grafana:7.1.3
``` ```

2
gradle.properties
View File

@ -1,3 +1,3 @@
id = syslogd id = syslogd
group = biz.nellemann.syslogd group = biz.nellemann.syslogd
version = 1.0.13 version = 1.0.14

46
src/main/java/biz/nellemann/syslogd/Application.java
View File

@ -30,6 +30,9 @@ import picocli.CommandLine.Command;
import java.io.IOException; import java.io.IOException;
import java.net.InetSocketAddress; import java.net.InetSocketAddress;
import java.net.URI;
import java.net.URL;
import java.util.Locale;
import java.util.concurrent.Callable; import java.util.concurrent.Callable;
import java.util.regex.Matcher; import java.util.regex.Matcher;
import java.util.regex.Pattern; import java.util.regex.Pattern;
@ -46,7 +49,7 @@ public class Application implements Callable<Integer>, LogListener {
private LokiClient lokiClient; private LokiClient lokiClient;
@CommandLine.Option(names = {"-p", "--port"}, description = "Listening port [default: 514].", defaultValue = "514") @CommandLine.Option(names = {"-p", "--port"}, description = "Listening port [default: 514].", defaultValue = "514", paramLabel = "<num>")
private int port; private int port;
@CommandLine.Option(names = "--no-udp", negatable = true, description = "Listen on UDP [default: true].", defaultValue = "true") @CommandLine.Option(names = "--no-udp", negatable = true, description = "Listen on UDP [default: true].", defaultValue = "true")
@ -64,14 +67,14 @@ public class Application implements Callable<Integer>, LogListener {
@CommandLine.Option(names = "--rfc5424", description = "Parse RFC-5424 messages [default: RFC-3164].", defaultValue = "false") @CommandLine.Option(names = "--rfc5424", description = "Parse RFC-5424 messages [default: RFC-3164].", defaultValue = "false")
private boolean rfc5424; private boolean rfc5424;
@CommandLine.Option(names = { "-s", "--syslog"}, description = "Forward to Syslog UDP host[:port] (RFC-5424).", paramLabel = "<host>") @CommandLine.Option(names = { "-s", "--syslog"}, description = "Forward to Syslog <udp://host:port> (RFC-5424).", paramLabel = "<uri>")
private String syslog; private URI syslog;
@CommandLine.Option(names = { "-g", "--gelf"}, description = "Forward to Graylog host[:port] (GELF).", paramLabel = "<host>") @CommandLine.Option(names = { "-g", "--gelf"}, description = "Forward to Graylog <udp://host:port>.", paramLabel = "<uri>")
private String gelf; private URI gelf;
@CommandLine.Option(names = { "-l", "--loki"}, description = "Forward to Grafana Loki.", paramLabel = "<url>") @CommandLine.Option(names = { "-l", "--loki"}, description = "Forward to Grafana Loki <http://host:port>.", paramLabel = "<url>")
private String loki; private URL loki;
@CommandLine.Option(names = { "-d", "--debug" }, description = "Enable debugging [default: 'false'].") @CommandLine.Option(names = { "-d", "--debug" }, description = "Enable debugging [default: 'false'].")
private boolean enableDebug = false; private boolean enableDebug = false;
@ -90,17 +93,26 @@ public class Application implements Callable<Integer>, LogListener {
syslogParser = new SyslogParserRfc3164(); syslogParser = new SyslogParserRfc3164();
} }
if(syslog != null && !syslog.isEmpty()) { if(syslog != null) {
udpClient = new UdpClient(getInetSocketAddress(syslog)); if(syslog.getScheme().toLowerCase(Locale.ROOT).equals("udp")) {
doForward = true; udpClient = new UdpClient(getInetSocketAddress(syslog));
doForward = true;
} else {
throw new UnsupportedOperationException("Syslog protocol not implemented: " + syslog.getScheme());
}
} }
if(gelf != null && !gelf.isEmpty()) { if(gelf != null) {
gelfClient = new UdpClient(getInetSocketAddress(gelf)); System.err.println(gelf.getScheme());
doForward = true; if(gelf.getScheme().toLowerCase(Locale.ROOT).equals("udp")) {
gelfClient = new UdpClient(getInetSocketAddress(gelf));
doForward = true;
} else {
throw new UnsupportedOperationException("GELF protocol not implemented: " + gelf.getScheme());
}
} }
if(loki != null && !loki.isEmpty()) { if(loki != null) {
lokiClient = new LokiClient(loki); lokiClient = new LokiClient(loki);
doForward = true; doForward = true;
} }
@ -167,6 +179,12 @@ public class Application implements Callable<Integer>, LogListener {
} }
private InetSocketAddress getInetSocketAddress(URI input) {
InetSocketAddress inetSocketAddress = new InetSocketAddress(input.getHost(), input.getPort());
return inetSocketAddress;
}
private InetSocketAddress getInetSocketAddress(String input) { private InetSocketAddress getInetSocketAddress(String input) {
String dstHost; String dstHost;

32
src/main/java/biz/nellemann/syslogd/net/LokiClient.java
View File

@ -3,9 +3,7 @@ package biz.nellemann.syslogd.net;
import org.slf4j.Logger; import org.slf4j.Logger;
import org.slf4j.LoggerFactory; import org.slf4j.LoggerFactory;
import java.io.BufferedReader;
import java.io.IOException; import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStream; import java.io.OutputStream;
import java.net.*; import java.net.*;
import java.nio.charset.StandardCharsets; import java.nio.charset.StandardCharsets;
@ -17,49 +15,33 @@ public class LokiClient {
private final URL url; private final URL url;
public LokiClient(String url) throws MalformedURLException { public LokiClient(URL url) {
this.url = new URL(url); this.url = url;
} }
public void send(String msg) throws MalformedURLException { public void send(String msg) throws MalformedURLException {
URL pushUrl = new URL(url, "/loki/api/v1/push"); URL pushUrl = new URL(url, "/loki/api/v1/push");
log.warn("send() - URL: " + pushUrl.toString());
HttpURLConnection con = null; HttpURLConnection con = null;
try { try {
con = (HttpURLConnection)pushUrl.openConnection(); con = (HttpURLConnection)pushUrl.openConnection();
con.setRequestMethod("POST"); con.setRequestMethod("POST");
con.setRequestProperty("Content-Type", "application/json"); con.setRequestProperty("Content-Type", "application/json");
//con.setRequestProperty("Content-Type", "application/json; utf-8");
//con.setRequestProperty("Accept", "application/json");
con.setDoOutput(true); con.setDoOutput(true);
byte[] input = msg.getBytes(StandardCharsets.US_ASCII); byte[] input = msg.getBytes(StandardCharsets.UTF_8);
try(OutputStream os = con.getOutputStream()) { try(OutputStream os = con.getOutputStream()) {
os.write(input, 0, input.length); os.write(input, 0, input.length);
log.warn("send() - Data: " + msg);
} catch (IOException e) { } catch (IOException e) {
e.printStackTrace(); log.warn(e.getMessage());
} }
StringBuilder content; int responseCode = con.getResponseCode();
if(responseCode != 204) {
try (BufferedReader br = new BufferedReader( log.warn("send() - response: " + responseCode);
new InputStreamReader(con.getInputStream()))) {
String line;
content = new StringBuilder();
while ((line = br.readLine()) != null) {
content.append(line);
content.append(System.lineSeparator());
}
} }
System.out.println(content.toString());
} catch (IOException e) { } catch (IOException e) {
log.warn("send() - " + e.getMessage()); log.warn("send() - " + e.getMessage());
} finally { } finally {