Skip to main content

API

The circonus-network-api is a perl web service running under Apache HTTPD on port 8080. In front of this service is circonus-ssl-term, which listens on port 443 to terminate SSL connections and forward them to the API backend.

Logs for the API can be found in /www/logs. Logs generated by HTTPD are piped to rotatelogs and are rotated hourly. These consist of the following logs:

  • circonus_access_log
  • circonus_error_log
  • circonus_ssl_access_log
  • circonus_ssl_error_log

Internal application logs are rotated daily. These logs are useful for debugging and are configurable as needed (see the Additional Web Logging section for details). By default, the API will log the following data:

  • access - The auth token and app requesting access
  • psgi - The request type and endpoint as well as the response
  • exceptions - Stacktraces of errors if any were encountered

Each log line has a timestamp, a tag, and a message. The tag will match up with any tag sent in the API response to the user. When troubleshooting, ask for any tags from the user and look for that string in the logs.

Some common issues, such as slowness or timeouts, can be cause by an overloaded server. Check the httpd processes for saturation, and if needed, restart the circonus-network-api service. Additional API servers may be added by contacting Circonus Sales (sales@circonus.com).

Some events, such as failure of the primary Web DB, or failure of the Message Queue, can lead to server errors from the API. Typically, these events can be corrected by restarting the circonus-network-api service.

If restarting the service does not correct the issues, or you see many requests leading to exceptions with stacktraces that appear to be problems with the service itself, please contact Circonus Support (support@circonus.com).

API PKI Files​

  • /www/etc/ssl/ca.crt
  • /www/etc/ssl/<hostname>.crt
  • /www/etc/ssl/<hostname>.key

Note:

You will also see api.crt and api.key under /www/etc/ssl/. These are the user facing certificates and are not involved in internal SSL communication.

GoAPI​

The Circonus API is being converted from the current Perl web service to a new implementation, called GoAPI, written in the Go language. API functionality is being migrated from the Perl service to GoAPI in a modular way. So, until all functionality is migrated to GoAPI, both systems will be running in your environment.

GoAPI is comprised of a series of back-end microservices, a REST API gateway service, and other proxy services. All GoAPI services are started through the circonus-api service.

The primary REST API gateway service listens on port 8089. Each microservice in the GoAPI background will also listen on its own port.

GoAPI Configuration​

GoAPI loads its configuration from the local environment where it is running. The primary configuration is read from /www/etc/circonus.conf. Local configuration overrides are loaded from /www/etc/circonus.<machine name>.local.conf, /www/etc/local.conf, and from a local.conf file in the directory where the GoAPI binary is started, in that order.

Configuration values can also be overridden by setting environment variables. Any local environment settings used are generally located in the /opt/circonus/etc/goapi.env file. Environment variable settings will override settings from Circonus configuration files.

GoAPI Logs​

All GoAPI services log their output to a stream. The destination of the log output is configurable, but by default, log output is directed to: /var/log/circonus/goapi.log.

The file logger rotates output to a configurable number of log files, based on configurable maximum size. By default, log files are capped at 100MB and the last 5 files are saved, with names like: /var/log/circonus/goapi.log.1, /var/log/circonus/goapi.log.2, etc.

Typical GoAPI log entries will look similar to these:

2019-05-18T16:08:51 [INFO ] account: account server listening on :8095 {"rpc":"server","service":"account"}
2019-05-29T22:52:50 [ERROR] api: invalid authentication cookie {"code":"Forbidden.BadCookie","status":403,"grpc":7,"explanation":"The authentication cookie you supplied is invalid","message":"invalid authentication cookie","tag":"7c0bbdc0-8264-11e9-ad0b-acde48001122","server":"dev.local","time":1559170370,"error":{"message":"rpc error: code = PermissionDenied desc = Forbidden.BadCookie invalid authentication cookie credentials"}} {"app":"","kind":"GET","remote":"[::1]:54913","request":"","uri":"/account/current"}

They will contain a timestamp, a log level indication, the name of the GoAPI service that created the log entry, a log message, and JSON format structured data related to the log entry and/or the error being logged. Error entries will contain a tag value that will be the same for all errors in the same trace.

The stdout/stderr of the GoAPI binary is recorded in the systemd journal, and may have additional information if the service is misbehaving. Use journalctl to view it:

sudo journalctl -xe -u circonus-api

GoAPI Troubleshooting​

Generally, any issues that may occur during GoAPI operation will generate error log entries to the log file. The log entries will list the source of the error and the event causing the error. If the errors are related to connectivity issues, or configuration issues, correcting these issues and restarting the GoAPI service will resolve the errors.

sudo systemctl restart circonus-api

If restarting the service does not resolve the issue, please open a ticket with Circonus Support (support@circonus.com). Please include all recent log entries describing the errors and other surrounding log entries.