The stratcon role consists of a C application:
Stratcon is the aggregation point of all data coming from brokers. This service reaches out to each active and configured broker, makes itself a subscriber of the journaled data, and receives it as it is collected.
Like the broker, stratcon has 2 processes: a parent and child. If the child does not heartbeat, the parent is the watchdog and will restart it.
Stratcon keeps a few logs files. Most of these files can be ignored despite their imposing names. Logs are kept in
/var/log/circonus and the list of logs is as follows:
stratcon.log- General purpose log
stratcon-access.log- Logs any requests received over SSL port 43191
stratcon-bad.data- Can be ignored unless otherwise advised by Support.
bad.data, can be ignored
Stratcon also has an interactive console similar to the broker. This console can be reached by telnetting to localhost port 32324. The most commonly used command in this mode is:
This command shows the current status of connected brokers.
Stratcon can be sensitive to IO and CPU starvation. If you discover the child process is continually being restarted, check these resources. Contact Support (firstname.lastname@example.org) if the problem persists.
Stratcon PKI Files
Broker - Stratcon Connectivity Troubleshooting
These procedures should help if a stratcon machine connects to and pulls data from the broker, but is not able to retrieve the data. If this problem manifests, a storage feed would be disconnected on the brokers page, but the alert feed would be unaffected.
First, verify that there is no issue with PKI connectivity. If this is not an issue, then it is possible that the stratcon machine is stuck requesting and waiting for data that is not available on the broker due to a missing file.
Run this command on the Broker to view the stratcon connections in the jlog:
/opt/circonus/bin/jlogctl -s /opt/noit/prod/log/noitd.feed/
This log will tell you about the various stratcon connections to the broker and help you to locate them in the feed. The output should look something like this:
[root@example noitd]# /opt/circonus/bin/jlogctl -s /opt/noit/prod/log/noitd.feed/ /opt/noit/prod/log/noitd.feed/ stratcon-machine105 @ 000027f1:00000094 stratcon-machine010 @ 0000020e:000000dc ~0000003 @ 000027f2:00000060 ~0000004 @ 000027f2:00000060 [root@example noitd]# /opt/circonus/bin/jlogctl -s /opt/noit/prod/log/noitd.feed/ /opt/noit/prod/log/noitd.feed/ stratcon-machine105 @ 000027f2:00000000 stratcon-machine010 @ 0000020e:000000dc ~0000003 @ 000027f2:00000077 ~0000004 @ 000027f2:00000081
The values after the “@” for each stratcon are the filename and the point in the file where the stratcon is to read its next piece of data. For the stratcon that is behind, verify the existence of the file it is trying to read (0000020e in our example), by running this command:
ls -lh /opt/noit/prod/log/noitd.feed/0000020e
If the file is missing, run this command and look for the oldest file:
ls -lat /opt/noit/prod/log/noitd.feed/ | tail
The oldest file name should be “newer” than the file we were looking for in the previous step (using hexidecimal counting). With this filename, we will need to rebuild the checkpoint file that the broker uses to keep track of what data is sent to stratcon.
The checkpoint file to rebuild is the file named cp. to get the hex encoded name run this command:
perl -e 'print unpack("H*", "<stratcon name here>")."\n"'
In our example of
stratcon-machine010, that command yields this result: 7374726174636f6e2d6d616368696e65303130
Lastly, we will rebuild the checkpoint file using the output of the previous few commands, taking the oldest remaining file and the checkpoint file name. The template of the command is this:
sudo perl -e 'print pack("II", 0x<file id>,0);' > /opt/noit/prod/log/noitd.feed/cp.<value>
For example, if the end of the output from “
ls -lat /opt/noit/prod/log/noitd.feed/ | tail” looked like this:
-rw-r----- 1 root root 7648 Jul 7 17:08 00000210.idx -rw-r----- 1 root root 4266881 Jul 7 17:08 00000210 -rw-r----- 1 root root 8 Jul 7 17:00 cp.7374726174636f6e2d6d616368696e65303130 drwxr-xr-x 3 root root 4096 Jul 2 20:30 ..
Then the command to rebuild the checkpoint file should look like this:
sudo perl -e 'print pack("II", 0x00000210,0);' > /opt/noit/prod/log/noitd.feed/cp.7374726174636f6e2d6d616368696e65303130
Once this new file is in place, stratcon will begin pulling down data from the new checkpoint the next time stratcon connects.