Alpaca 11.1

Troubleshooting Guide

Overview

The Alpaca stack consists of three different applications: Eureka, Gateway and Server. All three of these components are required to for Alpaca to run. In addition to the three main components, Alpaca is dependent on two third-party pieces of software: MongoDb and RabbitMq. Without these, Alpaca cannot run.

Application Stack

  • Eureka - Eureka is a discovery server. It is used to keep track of the various components in an Alpaca cluster.
  • Gateway - Acts as a load balancer for incoming requests. In a non-HA environment, all requests are routed to the single Alpaca server.
  • Server - The main Alpaca component that serves the Rest API as well as the web interface. The server component is versioned against BroadWorks. Multiple server versions may be in use at once.

Dependencies

  • MongoDb - The NoSQL database used by Alpaca.
  • RabbitMq - The message broker used by Alpaca.

Basic Troubleshooting

Alpaca Components

Checking Status

  • Eureka - alpacactl status eureka
  • Gateway - alpacactl status gateway
  • Server - alapcactl status server-<BW_VERSION>

alpacactl status is an alias for systemctl status. The Active status should be active (running) when Alpaca is running normally.

Example Output 
[root@alpaca ~]# alpacactl status server-24
● alpaca-server-24.service - LSB: Alpaca, a BroadWorks toolkit.
   Loaded: loaded (/etc/rc.d/init.d/alpaca-server-24; generated)
   Active: active (running) since Wed 2025-01-08 11:11:51 EST; 4 weeks 1 days ago
     Docs: man:systemd-sysv-generator(8)
  Process: 2019651 ExecStart=/etc/rc.d/init.d/alpaca-server-24 start (code=exited, status=0/SUCCESS)
    Tasks: 100 (limit: 48825)
   Memory: 1.0G
   CGroup: /system.slice/alpaca-server-24.service
           └─2019669 java -Djava.library.path=/opt/TimesTen/tt1122/lib/ -Dorg.jboss.logging.provider=slf4j -Dalpaca.home=/opt/alpaca-server-24 -Dlogging.config=/etc/alpaca/server/config/logback-spring.xml -Dja>

Jan 08 11:11:51 alpaca.vwave.net systemd[1]: Starting LSB: Alpaca, a BroadWorks toolkit....
Jan 08 11:11:51 alpaca.vwave.net runuser[2019662]: pam_unix(runuser:session): session opened for user alpaca by (uid=0)
Jan 08 11:11:51 alpaca.vwave.net runuser[2019662]: pam_unix(runuser:session): session closed for user alpaca
Jan 08 11:11:51 alpaca.vwave.net alpaca-server-24[2019651]: Starting alpaca-server-24: [  OK  ]
Jan 08 11:11:51 alpaca.vwave.net systemd[1]: Started LSB: Alpaca, a BroadWorks toolkit..

Logs

The default log location for all Alpaca component logs is /var/log/alpaca/. Each component uses its own logging configuration which can be modified at /etc/alpaca/<eureka|gateway|server>/config/logback-spring.xml. More information on the logging configuration can be found here;

  • Eureka - alpaca-eureka.log
  • Gateway - alpaca-gateway.log
  • Server - alpaca-server-<BW_VERSION>.log

If no logs are being produced, check the .out file for each application. The .out file is where all System.out logs go as well as logs that occur before the loggers are initialized. The .out files can be found here:

  • Eureka - /opt/alpaca-eureka/alpaca-eureka.out
  • Gateway - /opt/alpaca-gateway/alpaca-gateway.out
  • Server - /opt/alpaca-server-<BW_VERSION>.out
Other Logs Access Logs

Access logs for the underlying Tomcat container that the Alpaca server is using can be found at /var/log/alpaca/access-logs-<BW_VERSION>.

Other

PID File

Each Alpaca component generates a pid (process id) file on startup. This file contains the process of id of the running application, as assigned by the OS. The pid file can be located here:

  • Eureka - /opt/alpaca-eureka/alpaca-eureka.pid
  • Gateway - /opt/alpaca-gateway/alpaca-gateway.pid
  • Server - /opt/alpaca-server-<BW_VERSION>.pid

Dependencies

Checking Status

  • MongoDB - systemctl status mongod or service mongod status depending on OS.
  • RabbitMq
    • systemctl status rabbitmq-server or service rabbitmq-server status depending on OS.
    • rabbitmqctl status for more detailed information.
    • rabbitmqctl cluster_status for more detailed information when in an HA environment.

Logs

  • MongoDB - The default location is /var/log/mongodb/mongod.log.
  • RabbitMq - The default location is /var/log/rabbitmq/rabbit@<HOST_NAME>.log.

Common Issues

  • Disk Space
    • Verify that there is adequate disk space for the following directories:
      • /var/log/alpaca
      • /var/alpaca/store
      • /opt
      • /var/lib/mongo
      • /var/lib/rabbitmq/
    • Disk space can be checked with df -h on most flavors of Linux.
  • Network Connectivity
    • Ports
      • 8443 - Access to the web interface (by default)
      • All ports should be open between Alpaca instances in an HA environment.
      • BroadWorks Ports:
        • 80 - AS, ADP, XSP, PS - Device files, announcements.
        • 443 - AS, ADP, XSP, PS - Device files, announcements.
        • 11203 - AS - Timesten access.
        • 2208 - ADP, XSP - OCI.
        • 2209 - ADP, XSP - OCI Secure.
  • Invalid configuration
    • The configuration parser expects that all configs will be valid yaml. If the config is invalid, a configuration error will be printed in the .out file. See configuration guide for more details.
  • SeLinux
    • Issues can arise if SeLinux is in enforcing mode.
  • Invalid RabbitMq credentials.
    • Credentials can be tested with rabbitmqctl authenticate_user <username> <password> text [root@alpaca ~]# rabbitmqctl authenticate_user alpaca MyRabbitPass1! Authenticating user "alpaca" ... Success
    • Credentials can be found in the Alpaca configuration. yaml spring: ... rabbitmq: host: alpaca.acme.org username: alpaca password: MyRabbitPass1!

Common Terms

  • Java - The programming language that Alpaca is written in. Alpaca currently uses Java 17.
    • Corretto - Amazon's distribution of Open JDK.
    • JDK - Java Development Kit.
    • Open JDK - Free/open-source implementation/distribution of Java.
  • MinIo - Open source (S3 compatible) object storage software. Can be installed locally.
  • MongoDb - The NoSql database used by Alpaca.
    • NoSql - "Not only SQL". Non-relational database.
  • S3 - Amazon cloud storage.
  • Spring - The primary Java framework used by Alpaca.
    • Spring Boot - An extension of Spring.
  • TimesTen - The in-memory SQL database used by BroadWorks.
  • Yaml - A human-friendly data serialization language for all programming languages.