Archived Now bundled in https://github.com/cloudfoundry/log-cache-release

Log Cache persists data in memory from the Loggregator System.

This repository should be imported as:

import logcache "code.cloudfoundry.org/log-cache"

Log Cache indexes data by the source_id field on the Loggregator Envelope. Source IDs should be unique across applications, but not across instances.

In Cloud Foundry terms, the source ID can represent an app guid (e.g. cf app <app-name> --guid), an app name, and/or a component name (e.g. doppler). All results matching one of these three mechanisms and permitted by authorization are returned.

Log Cache's API implements two sets of endpoints - read and meta are source_id oriented, whereas query and query_range are metric oriented, and fulfill the Prometheus API. Either API can be reached via mTLS-authenticated gRPC or CF-token-authenticated HTTPS. An info endpoint is also provided for service version discoverability.

When querying the API via HTTPS, each request must have the Authorization header set with a UAA provided token.

The scopes doppler.firehose and logs.admin are authorized as admin, and return data for all relevant source IDs. This authorization is required to retrieve component logs.

For more limited scopes, Cloud Controller is consulted to establish app permissions.

When querying the API via gRPC, authorization for all app and component data is granted.

Retrieve JSON representation of deployed Log Cache version.

$ curl "https://<log-cache-addr>/api/v1/info"

{
  "version": "X.Y.Z"
}

Retrieve data by source-id. Returns loggregator-domain envelopes, potentially containing logs and/or metrics.

Query Parameters:

• start_time is a UNIX timestamp in nanoseconds. It defaults to the start of the cache (e.g. date +%s). Start time is inclusive. [starttime..endtime)
• end_time is a UNIX timestamp in nanoseconds. It defaults to current time of the cache (e.g. date +%s). End time is exclusive. [starttime..endtime)
• envelope_types is a filter for Envelope Type. The available filters are: LOG, COUNTER, GAUGE, TIMER, and EVENT. If set, then only those types of envelopes will be emitted. This parameter may be specified multiple times to include more types.
• limit is the maximum number of envelopes to request. The max limit size is 1000 and defaults to 100.

$ curl "https://<log-cache-addr>/api/v1/read/<source-id>?start_time=<start-time>&end_time=<end-time>"

{
  "envelopes": {"batch": [...] }
}

Lists the available source IDs that Log Cache has persisted.

{
  "meta":{
    "source-id-0":{"count":"100000","expired":"129452","oldestTimestamp":"1524071322998223702","newestTimestamp":"1524081739994226961"},
    "source-id-1":{"count":"2114","oldestTimestamp":"1524057384976840476","newestTimestamp":"1524081729980342902"},
    ...
  }
}

• count contains the number of envelopes held in Log Cache
• expired, if present, is a count of envelopes that have been pruned
• oldestTimestamp and newestTimestamp are the oldest and newest entries for the source, in nanoseconds since the Unix epoch.

The ultimate goal of these endpoints is to create a fully-compliant, Prometheus-compatible interface. This should allow tools such as Grafana to work directly with Log Cache without any additional translation.

There are still a few metadata endpoints that are unsupported. These should be coming to Log Cache in a future release.

A valid PromQL metric name consists of the character [a-Z][0-9] and underscore. Names can begin with [a-Z] or underscore. Names cannot begin with [0-9]. As a measure to work with existing metrics that do not comply with the above format a conversion process takes place when matching on metric names. Any character that is not in the set of valid characters is converted to an underscore. The metric is not changed in the cache.

e.g., to match on a metric name http.latency use the name http_latency as a search term.

Issues a PromQL instant query against Log Cache data. You can read more detail in the Prometheus documentation here.

$ curl -G "https://<log-cache-addr>/api/v1/query" --data-urlencode 'query=metrics{source_id="source-id-1"}'

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      { "metric": {...}, "value": [ <timestamp>, "<value>" ] },
      ...
    ]
  }
}

Issues a PromQL range query against Log Cache data. You can read more detail in the Prometheus documentation here.

$ curl -G "https://<log-cache-addr>/api/v1/query_range" \
    --data-urlencode 'query=metrics{source_id="source-id-1"}' \
    --data-urlencode 'start=1537290750' \
    --data-urlencode 'end=1537290760' \
    --data-urlencode 'step=1'

{
  "status": "success",
  "data": {
    "resultType": "matrix",
    "result": [
      {
        "metric": {...},
        "values": [
          [ <timestamp>, "<value>" ],
          ...
        ]
      },
      ...
    ]
  }
}

Log Cache provides a plugin for the Cloud Foundry command line tool, which makes interacting with the API simpler.