Compare commits

...

4 Commits
v1.3.4 ... main

Author SHA1 Message Date
Mark Nellemann 3ef961e44b Update README.md 2024-05-17 07:53:25 +00:00
Mark Nellemann bd5e2634d5 Merge pull request 'Depedency updates and cleanup.' (#1) from updates into main
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/tag Build is passing Details
Reviewed-on: #1
2023-11-13 12:48:20 +00:00
Mark Nellemann 1acdd6a93d Cleanup.
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
2023-11-13 13:46:11 +01:00
Mark Nellemann d39837861f Update dependencies and gradle. 2023-10-02 14:25:33 +02:00
13 changed files with 86 additions and 173 deletions

109
README.md
View File

@ -1,108 +1,3 @@
# Syslog Director
All received messages are written to *stdout* and/or forwarded to remote logging destinations.
Supported incoming message formats are:
- Syslog RFC5424 - TCP and UDP
- Syslog RFC3164 (BSD) - TCP and UDP
- Graylog GELF - TCP and UDP (compressed & chunked)
Supported remote logging destinations are:
- Syslog (RFC5424 over UDP)
- Graylog (GELF over UDP)
- Grafana Loki (HTTP over TCP).
This software is free to use and is licensed under the [Apache 2.0 License](LICENSE).
![architecture](doc/syslogd.png)
Some of my other related projects are:
- [hmci](https://git.data.coop/nellemann/hmci) for agent-less monitoring of IBM Power servers
- [svci](https://git.data.coop/nellemann/svci) for monitoring IBM Spectrum Virtualize (Flashsystems / Storwize / SVC)
- [sysmon](https://git.data.coop/nellemann/sysmon) for monitoring all types of servers with a small Java agent
## Usage Instructions
- Install the syslogd package (*.deb* or *.rpm*) from [packages](https://git.data.coop/nellemann/-/packages/generic/syslogd/) or build from source.
```text
Usage: syslogd [-dhV] [--[no-]ansi] [--[no-]stdin] [--[no-]stdout] [--[no-]tcp]
[--[no-]udp] [-f=<protocol>] [-p=<num>] [--to-gelf=<uri>]
[--to-loki=<url>] [--to-syslog=<uri>]
-d, --debug Enable debugging [default: 'false'].
-f, --format=<protocol> Input format: RFC-5424, RFC-3164 or GELF [default:
RFC-3164].
-h, --help Show this help message and exit.
--[no-]ansi Output in ANSI colors [default: true].
--[no-]stdin Forward messages from stdin [default: true].
--[no-]stdout Output messages to stdout [default: true].
--[no-]tcp Listen on TCP [default: true].
--[no-]udp Listen on UDP [default: true].
-p, --port=<num> Listening port [default: 1514].
--to-gelf=<uri> Forward to Graylog <udp://host:port>.
--to-loki=<url> Forward to Grafana Loki <http://host:port>.
--to-syslog=<uri> Forward to Syslog <udp://host:port> (RFC-5424).
-V, --version Print version information and exit.
```
The default syslog port (514) requires you to run syslogd as root / administrator.
Any port number above 1024 does not require privileges and can be selected with the *-p* or *--port* option.
### Examples
Listening on the default syslog port:
```
java -jar /path/to/syslogd-x.y.z-all.jar --port 514
```
or, if installed as a *deb* or *rpm* package:
```
/opt/syslogd/bin/syslogd --port 514
```
Forwarding messages on to another log-system on a non-standard port.
```
java -jar /path/to/syslogd-x.y.z-all.jar --to-syslog udp://remotehost:514
```
Forwarding messages to a Graylog server in GELF format.
```
java -jar /path/to/syslogd-x.y.z-all.jar --to-gelf udp://remotehost:12201
```
Forwarding to a Grafana Loki server.
```
java -jar /path/to/syslogd-x.y.z-all.jar --to-loki http://remotehost:3100
```
If you don't want any output locally (only forwarding), you can use the ```--no-stdout``` flag.
## Notes
### IBM AIX and VIO Servers
Syslog messages from AIX (and IBM Power Virtual I/O Servers) can be troublesome with some logging solutions. These can be received with
*syslogd* and then forwarded on to your preferred logging solution.
### Forwarding to Grafana Loki
Forwarding is currently done by making HTTP connections to the Loki API, which works fine for low volume messages, but might cause issues for large volume of messages.
## Development Notes
### Test Grafana Loki
Run Loki and Grafana in local containers to test.
```shell
docker run --rm -d --name=loki -p 3100:3100 grafana/loki
docker run --rm -d --name=grafana --link loki:loki -p 3000:3000 grafana/grafana:7.1.3
```
# Repository moved
Please visit [github.com/mnellemann/syslogd](https://github.com/mnellemann/syslogd)

View File

@ -4,7 +4,7 @@ plugins {
id 'application'
id 'jacoco'
id "net.nemerosa.versioning" version "2.15.1"
id "com.netflix.nebula.ospackage" version "10.0.0"
id "com.netflix.nebula.ospackage" version "11.5.0"
id "com.github.johnrengelman.shadow" version "7.1.2"
}
@ -13,12 +13,12 @@ repositories {
}
dependencies {
annotationProcessor 'info.picocli:picocli-codegen:4.7.1'
implementation 'info.picocli:picocli:4.7.1'
implementation 'org.slf4j:slf4j-api:2.0.6'
implementation 'org.slf4j:slf4j-simple:2.0.6'
implementation 'com.fasterxml.jackson.core:jackson-databind:2.14.2'
implementation 'com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.14.2'
annotationProcessor 'info.picocli:picocli-codegen:4.7.5'
implementation 'info.picocli:picocli:4.7.5'
implementation 'org.slf4j:slf4j-api:2.0.9'
implementation 'org.slf4j:slf4j-simple:2.0.9'
implementation 'com.fasterxml.jackson.core:jackson-databind:2.15.2'
implementation 'com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.15.2'
implementation 'org.apache.commons:commons-collections4:4.4'
testImplementation 'org.spockframework:spock-core:2.3-groovy-3.0'
@ -38,7 +38,7 @@ test {
}
jacoco {
toolVersion = "0.8.8"
toolVersion = "0.8.10"
}
jacocoTestReport {

View File

@ -1,5 +1,5 @@
id = syslogd
name = syslogd
group = biz.nellemann.syslogd
version = 1.3.4
version = 1.3.5
description = "Syslog Director"

View File

@ -1,6 +1,6 @@
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-7.6-bin.zip
distributionUrl=https\://services.gradle.org/distributions/gradle-7.6.2-bin.zip
networkTimeout=10000
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists

View File

@ -15,16 +15,6 @@
*/
package biz.nellemann.syslogd;
import biz.nellemann.syslogd.msg.SyslogMessage;
import biz.nellemann.syslogd.net.*;
import biz.nellemann.syslogd.parser.GelfParser;
import biz.nellemann.syslogd.parser.SyslogParser;
import biz.nellemann.syslogd.parser.SyslogParserRfc3164;
import biz.nellemann.syslogd.parser.SyslogParserRfc5424;
import picocli.CommandLine;
import picocli.CommandLine.Command;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.URI;
@ -34,6 +24,19 @@ import java.util.List;
import java.util.Locale;
import java.util.concurrent.Callable;
import biz.nellemann.syslogd.msg.SyslogMessage;
import biz.nellemann.syslogd.net.GelfClient;
import biz.nellemann.syslogd.net.LokiClient;
import biz.nellemann.syslogd.net.TcpServer;
import biz.nellemann.syslogd.net.UdpClient;
import biz.nellemann.syslogd.net.UdpServer;
import biz.nellemann.syslogd.parser.GelfParser;
import biz.nellemann.syslogd.parser.SyslogParser;
import biz.nellemann.syslogd.parser.SyslogParserRfc3164;
import biz.nellemann.syslogd.parser.SyslogParserRfc5424;
import picocli.CommandLine;
import picocli.CommandLine.Command;
@Command(name = "syslogd",
mixinStandardHelpOptions = true,
versionProvider = biz.nellemann.syslogd.VersionProvider.class)
@ -152,7 +155,7 @@ public class Application implements Callable<Integer>, LogReceiveListener {
if(msg != null) {
if(logForwardListeners.size() > 0) {
if(!logForwardListeners.isEmpty()) {
sendForwardEvent(msg);
}

View File

@ -1,12 +1,12 @@
package biz.nellemann.syslogd;
import biz.nellemann.syslogd.msg.SyslogMessage;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;
import java.util.Scanner;
import biz.nellemann.syslogd.msg.SyslogMessage;
public class InputReader extends Thread {
private final Scanner input;
@ -17,6 +17,7 @@ public class InputReader extends Thread {
this.protocol = protocol;
}
@Override
public void run() {
while(input.hasNextLine()) {

View File

@ -1,13 +1,14 @@
package biz.nellemann.syslogd.net;
import biz.nellemann.syslogd.LogForwardEvent;
import biz.nellemann.syslogd.SyslogPrinter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.net.InetSocketAddress;
import java.net.SocketException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import biz.nellemann.syslogd.LogForwardEvent;
import biz.nellemann.syslogd.SyslogPrinter;
public class GelfClient extends UdpClient {
private final static Logger log = LoggerFactory.getLogger(GelfClient.class);

View File

@ -15,19 +15,22 @@
*/
package biz.nellemann.syslogd.net;
import biz.nellemann.syslogd.LogForwardEvent;
import biz.nellemann.syslogd.LogForwardListener;
import biz.nellemann.syslogd.SyslogPrinter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.*;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
import java.nio.charset.StandardCharsets;
import java.util.concurrent.ArrayBlockingQueue;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import biz.nellemann.syslogd.LogForwardEvent;
import biz.nellemann.syslogd.LogForwardListener;
import biz.nellemann.syslogd.SyslogPrinter;
public class LokiClient implements LogForwardListener, Runnable {
private final static Logger log = LoggerFactory.getLogger(LokiClient.class);
@ -84,7 +87,7 @@ public class LokiClient implements LogForwardListener, Runnable {
while (true) {
try {
send(blockingQueue.take());
} catch (Exception e) {
} catch (MalformedURLException | InterruptedException e) {
log.warn(e.getMessage());
}
}

View File

@ -15,9 +15,6 @@
*/
package biz.nellemann.syslogd.net;
import biz.nellemann.syslogd.LogReceiveEvent;
import biz.nellemann.syslogd.LogReceiveListener;
import java.io.IOException;
import java.net.DatagramPacket;
import java.net.DatagramSocket;
@ -27,6 +24,9 @@ import java.util.List;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import biz.nellemann.syslogd.LogReceiveEvent;
import biz.nellemann.syslogd.LogReceiveListener;
public class UdpServer extends Thread {
private final static Logger log = LoggerFactory.getLogger(UdpServer.class);
@ -38,6 +38,7 @@ public class UdpServer extends Thread {
socket = new DatagramSocket(port);
}
@Override
public void run() {
byte[] buf = new byte[8192];
@ -48,7 +49,7 @@ public class UdpServer extends Thread {
socket.receive(packet);
//String packetData = new String(packet.getData(), packet.getOffset(), packet.getLength(), StandardCharsets.UTF_8);
sendEvent(packet);
} catch (Exception e) {
} catch (IOException e) {
log.error("run() - error: {}", e.getMessage());
listen = false;
}

View File

@ -1,17 +1,19 @@
package biz.nellemann.syslogd.parser;
import biz.nellemann.syslogd.msg.SyslogMessage;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
import org.apache.commons.collections4.map.PassiveExpiringMap;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.time.Instant;
import java.util.Arrays;
import java.util.TreeMap;
import org.apache.commons.collections4.map.PassiveExpiringMap;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
import biz.nellemann.syslogd.msg.SyslogMessage;
/*
For more information about the GELF format, visit: https://go2docs.graylog.org/5-0/getting_in_log_data/gelf.html
*/

View File

@ -15,15 +15,16 @@
*/
package biz.nellemann.syslogd.parser;
import biz.nellemann.syslogd.msg.SyslogMessage;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.nio.charset.StandardCharsets;
import java.time.Instant;
import java.util.zip.DataFormatException;
import java.util.zip.Inflater;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import biz.nellemann.syslogd.msg.SyslogMessage;
public abstract class SyslogParser {
private final static Logger log = LoggerFactory.getLogger(SyslogParser.class);

View File

@ -15,18 +15,21 @@
*/
package biz.nellemann.syslogd.parser;
import biz.nellemann.syslogd.msg.Facility;
import biz.nellemann.syslogd.msg.Severity;
import biz.nellemann.syslogd.msg.SyslogMessage;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.time.*;
import java.time.Instant;
import java.time.OffsetDateTime;
import java.time.ZoneId;
import java.time.format.DateTimeFormatter;
import java.time.format.DateTimeParseException;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import biz.nellemann.syslogd.msg.Facility;
import biz.nellemann.syslogd.msg.Severity;
import biz.nellemann.syslogd.msg.SyslogMessage;
public class SyslogParserRfc3164 extends SyslogParser {
private final static Logger log = LoggerFactory.getLogger(SyslogParserRfc3164.class);
@ -89,6 +92,7 @@ public class SyslogParserRfc3164 extends SyslogParser {
* @param dateString
* @return
*/
@Override
public Instant parseTimestamp(String dateString) {
// We need to add current year to parse date correctly

View File

@ -15,20 +15,21 @@
*/
package biz.nellemann.syslogd.parser;
import biz.nellemann.syslogd.msg.Severity;
import biz.nellemann.syslogd.msg.Facility;
import biz.nellemann.syslogd.msg.SyslogMessage;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.time.*;
import java.time.Instant;
import java.util.Arrays;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import biz.nellemann.syslogd.msg.Facility;
import biz.nellemann.syslogd.msg.Severity;
import biz.nellemann.syslogd.msg.SyslogMessage;
public class SyslogParserRfc5424 extends SyslogParser {
private final static Logger log = LoggerFactory.getLogger(SyslogParserRfc5424.class);
@ -98,6 +99,7 @@ public class SyslogParserRfc5424 extends SyslogParser {
* @param dateString
* @return
*/
@Override
public Instant parseTimestamp(String dateString) {
/*