API

The circonus-network-api is a perl web service running under Apache HTTPD on port 8080. In front of this service is circonus-ssl-term, which listens on port 443 to terminate SSL connections and forward them to the API backend.

Logs for the API can be found in /www/logs. Logs generated by HTTPD are piped to rotatelogs and are rotated hourly. These consist of the following logs:

  • circonus_access_log
  • circonus_error_log
  • circonus_ssl_access_log
  • circonus_ssl_error_log

Internal application logs are rotated daily. These logs are useful for debugging and are configurable as needed (see the Additional Web Logging section for details). By default, the API will log the following data:

  • access - The auth token and app requesting access
  • psgi - The request type and endpoint as well as the response
  • exceptions - Stacktraces of errors if any were encountered

Each log line has a timestamp, a tag, and a message. The tag will match up with any tag sent in the API response to the user. When troubleshooting, ask for any tags from the user and look for that string in the logs.

Some common issues, such as slowness or timeouts, can be cause by an overloaded server. Check the httpd processes for saturation, and if needed, restart the circonus-network-api service. Additional API servers may be added by contacting Circonus Sales (sales@circonus.com).

Some events, such as failure of the primary Web DB, or failure of the Message Queue, can lead to server errors from the API. Typically, these events can be corrected by restarting the circonus-network-api service.

If restarting the service does not correct the issues, or you see many requests leading to exceptions with stacktraces that appear to be problems with the service itself, please contact Circonus Support (support@circonus.com).

API PKI Files

  • /www/etc/ssl/ca.crt
  • /www/etc/ssl/<hostname>.crt
  • /www/etc/ssl/<hostname>.key

Note:

You will also see api.crt and api.key under /www/etc/ssl/. These are the user facing certificates and are not involved in internal SSL communication.

GoAPI

The Circonus API is being converted from the current Perl web service to a new implementation, called GoAPI, written in the Go language. API functionality is being migrated from the Perl service to GoAPI in a modular way. So, until all functionality is migrated to GoAPI, both systems will be running in your environment.

GoAPI is comprised of a series of back-end microservices, a REST API gateway service, and other proxy services. All GoAPI services are started through the use of a single binary located at /opt/circonus/sbin/goapi.

There are two system services that are used to start GoAPI depending on the server type and functionality needed:

  • circonus-api - This service starts all GoAPI services and proxies and is used on API servers.
  • circonus-api-search - This service starts only a subset of GoAPI services and proxies, and is intended to be run on web servers only.

The circonus-api service includes all of the functionality of the circonus-api-search service, so only one or the other should ever be used on an individual server.

The primary REST API gateway service listens on port 8089. Each microservice in the GoAPI background will also listen on its own port.

GoAPI Configuration

GoAPI loads its configuration from the local environment where it is running. The primary configuration is read from /www/etc/circonus.conf. Local configuration overrides are loaded from /www/etc/circonus.<machine name>.local.conf, /www/etc/local.conf, and from a local.conf file in the directory where the GoAPI binary is started, in that order.

Configuration values can also be overridden by setting environment variables. Any local environment settings used are generally located in the /opt/circonus/etc/goapi.env file. Environment variable settings will override settings from Circonus configuration files.

GoAPI Logs

All GoAPI services log their output to a stream. The destination of the log output is configurable, but by default, log output is directed to: /var/log/circonus/goapi.log.

The file logger rotates output to a configurable number of log files, based on configurable maximum size. By default, log files are capped at 100MB and the last 5 files are saved, with names like: /var/log/circonus/goapi.log.1, /var/log/circonus/goapi.log.2, etc.

Typical GoAPI log entries will look similar to these:

2019-05-18T16:08:51 [INFO ] account: account server listening on :8095 {"rpc":"server","service":"account"}
2019-05-29T22:52:50 [ERROR] api: invalid authentication cookie {"code":"Forbidden.BadCookie","status":403,"grpc":7,"explanation":"The authentication cookie you supplied is invalid","message":"invalid authentication cookie","tag":"7c0bbdc0-8264-11e9-ad0b-acde48001122","server":"dev.local","time":1559170370,"error":{"message":"rpc error: code = PermissionDenied desc = Forbidden.BadCookie invalid authentication cookie credentials"}} {"app":"","kind":"GET","remote":"[::1]:54913","request":"","uri":"/account/current"}

They will contain a timestamp, a log level indication, the name of the GoAPI service that created the log entry, a log message, and JSON format structured data related to the log entry and/or the error being logged. Error entries will contain a tag value that will be the same for all errors in the same trace.

GoAPI Troubleshooting

Generally, any issues that may occur during GoAPI operation will generate error log entries to the log file. The log entries will list the source of the error and the event causing the error. If the errors are related to connectivity issues, or configuration issues, correcting these issues and restarting the GoAPI service will resolve the errors.

The process for doing this will vary depending on the OS used, and whether the process is running on an API or web server. For example, on a CentOS 7 API server:

sudo systemctl restart circonus-api

On a CentOS 7 web server:

sudo systemctl restart circonus-api-search

If restarting the service does not resolve the issue, please open a ticket with Circonus Support (support@circonus.com). Please include all recent log entries describing the errors and other surrounding log entries.