Prometheus Metrics

The Virtual Signer exposes performance and operational metrics through a built-in Prometheus endpoint. These metrics can be collected, stored, and visualized using tools such as Prometheus and Grafana to monitor the health and performance of your deployment.

Metrics Configuration

The Prometheus metrics endpoint is configured using the following Configuration item:

• EDG_VS_PrometheusPort: Defines the port where the Prometheus metrics server will listen (default: 2112)

When using smart contract based configuration, this can also be specified using:

• PrometheusPort: Sets the Prometheus port in the contract-based configuration

Available Metrics

The Virtual Signer integrates with Prometheus by sending traces, logs, and metrics. Prometheus can serve as a "gateway" to other applications that store, display, or alert on these traces and metrics. By default, Prometheus metrics are exposed through the /metrics endpoint on port 2112:

http://<container-address>:2112/metrics

Exposed Counters

The Virtual Signer tracks the following counters while running:

Prometheus Setup Options

Managed Cloud Options for Prometheus

Rather than self-hosting Prometheus, you can use one of these managed cloud services to collect and store your metrics:

• AWS - Amazon Managed Service for Prometheus (AMP) provides serverless Prometheus-compatible monitoring with automatic scaling.
• GCP - Google Cloud Managed Service for Prometheus offers a fully managed metrics solution integrated with Google Cloud Monitoring.
• Azure - Azure Monitor with Prometheus integration allows collection of Prometheus metrics into Azure's monitoring platform.
• Grafana Cloud - Provides hosted Prometheus metrics collection with generous free tier and simple setup process.
• Chronosphere - Enterprise-grade cloud-native metrics platform designed for large-scale Prometheus deployments.
• Datadog - Offers Prometheus endpoint scraping capabilities as part of its comprehensive monitoring platform.
• New Relic - Provides Prometheus metrics integration with its observability platform.
• Splunk - Supports Prometheus metrics collection as part of its observability cloud offering.
• Dynatrace - Features automatic Prometheus metrics ingestion with its OneAgent technology.

Docker Compose Configuration

To enable metrics collection from your Virtual Signer in a Docker Compose environment, such as the one we set up in Deployment, you should expose the Prometheus port in your docker-compose.yml. Moreover, make sure that your firewall is exposing port 2112 to the outside world if you will connect in from a managed cloud service.

version: "3.9"
services:
  virtual-signer-1:
    image: iofinnet/io-vault-virtual-signer:latest
    ports:
      - "8080:8080"          # API port
      - "2112:2112"          # Prometheus metrics port
    environment:
      - EDG_VS_PrometheusPort=2112
    # other configuration...

Setting Up Local Prometheus to Scrape Metrics

If you prefer to set up Prometheus in your own Docker Compose environment and not rely on a managed cloud option, you should follow these steps. Let's configure a local Prometheus to scrape the metrics endpoint:

1. Create or update your prometheus.yml configuration:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'virtual-signer'
    static_configs:
      - targets: ['virtual-signer-1:2112']

2. Add Prometheus to your Docker Compose file:

services:
  # Virtual Signer service configuration...
  
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
    networks:
      - vsigner-network

volumes:
  prometheus_data:

networks:
  vsigner-network:
    driver: bridge

Visualizing Metrics with Grafana

Grafana will provide some useful visualizations of the metrics data. Once again, you can either use a managed service, like the ones listed above, or run Grafana locally in your own Docker Compose setup.

Let's look at how to set up Grafana locally to visualize the collected metrics:

1. Add Grafana to your Docker Compose configuration:

services:
  # Existing services...
  
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    volumes:
      - grafana_data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=secure-password
    depends_on:
      - prometheus
    networks:
      - vsigner-network

volumes:
  # Existing volumes...
  grafana_data:

2. Access Grafana UI at http://localhost:3000 (default credentials: admin/secure-password)

3. Add Prometheus as a data source:

   • Navigate to Configuration > Data Sources
   • Add Prometheus data source
   • Set URL to http://prometheus:9090
   • Click "Save & Test"

4. Create a dashboard to visualize Virtual Signer metrics:

   • Click "+ Create" > Dashboard
   • Add panels for key metrics
   • Use PromQL queries to display metrics (examples below)

Example Grafana Queries

Here are some example PromQL queries for common monitoring scenarios:

1. Request Rate:

   rate(http_requests_total{job="virtual-signer"}[5m])
   

2. Response Time (95th percentile):

   histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job="virtual-signer"}[5m])) by (le))
   

3. Error Rate:

   sum(rate(http_requests_total{job="virtual-signer", status=~"5.."}[5m])) / sum(rate(http_requests_total{job="virtual-signer"}[5m]))
   

Securing the Prometheus Endpoint

In production environments, consider securing the Prometheus endpoint:

1. Use network isolation to prevent public access to the metrics port
2. Implement authentication for the Prometheus endpoint
3. Use TLS to encrypt metrics traffic

Example Docker Compose configuration with network isolation:

services:
  virtual-signer-1:
    # ... other configuration
    networks:
      - public-network   # For API access
      - metrics-network  # For Prometheus access only
    
  prometheus:
    # ... prometheus configuration
    networks:
      - metrics-network
      
networks:
  public-network:
    # Network exposed to public
  metrics-network:
    # Internal network for metrics only
    internal: true

This setup ensures that the Prometheus metrics are only accessible to other services within the defined network, not to the outside world.

Going into detail on how to do this with Docker Compose is beyond the scope of these pages, but you may find more information on Docker's docs pages.