Reef is shipped as a WSGI application which may be deployed in a standard web server like Apache or Nginx. This provides the most flexible and customisable deployment options. However since this is fairly complex the recommended approach is to run the included ashttpd server to host the Reef application and use a front-end web server like Apache or Nginx as a reverse proxy. This allows to horizontally scale up ashttpd workers while providing a robust front-end server responsible for e.g. static file serving and SSL/TLS termination if desired.
Alternatively it is possible to serve Reef directly from the included ashttpd server, which is convenient for easy testing and demonstrations purposes. This is not as robust, flexible and performant as using the reverse proxy configuration so it not recommended for production usage.
Both the Reef WSGI application and the ashttpd web server have a set of runtime configuration settings. Depending on the deployment it is possible to place these settings in the same or in different configuration files. The default configuration shipped includes a $AS_HOME/etc/ashttpd.conf file which contains the settings for ashttpd as well as the Reef application which will be used in this configuration.
Detailed documentation of these configuration files and the runtime settings is available in the Runtime Settings section of the documentation.
ashttpd comes with a runtime setting called httpd_allowed_hosts which controls what hosts requested by the client it will serve. This settings is a comma-sparated list of hosts in the form hostname:port or [address]:port for IPv6 addresses (* wildcards are supported in both cases). The :port component of this can be omitted and will default to the wildcard, allowing all ports.
The default setting for httpd_allowed_hosts is to allow all (*:*) which could potentially enable malicious users to perform man-in-the-middle-attacks on legitimate users.
Warning
It is strongly recommended to set the httpd_allowed_hosts setting to a more restrictive set of hosts.
Reef uses a database to store events and ships with an SQLite database which is suitable for small deployments and proof of concept activities. However for production deployments it is strongly recommended that a more powerful, external database is deployed.
Important
As mentioned above the Reef WSGI application loads runtime settings from a specific configuration file. When the application is run by the ashttpd server it will setup the environment for the Reef application itself.
However this guide uses some external tools to manipulate the database, in this case it is required to set the AS_DJANGO_CONFIG environment variable so that the Reef WSGI application will correctly load its runtime settings. If AS_DJANGO_CONFIG is left unset the $AS_HOME/etc/reef.conf will be used as default reef configuration file which will not exist in a default installation.
Reefs ships with a default SQLite database which can be used for demos or small deployments. By default it has one superuser configured with the username of “admin” and password “admin”. This allows you to use the admin interface to change the credentials and add other users. It is recommended to change at least the password of the admin user.
If you need would like to create a new superuser using the command line you need to invoke the following command:
$ PYTHONPATH=/opt/abilisoft.com/reef/share/reef.pyz \
AS_DJANGO_CONFIG=/opt/abilisoft.com/reef/etc/ashttpd.conf \
django-admin.py createsuperuser --settings=reef.django_settings \
--username foo --email=foo@example.com
This will prompt you for a password to use.
Abilisoft recommend PostgreSQL, a powerful, open source object-relational database system. It has had more than 15 years of active development and a proven architecture that has earned it a strong reputation for reliability, data integrity, and correctness. It runs on all major operating systems and is fully ACID compliant.
There a number of steps involved in setting up an external database:
The descriptions below assume that Reef is hosted using ashttpd in its default configuration. When hosting Reef on other WSGI servers some details may differ. Before starting, you will need to configure the database itself, setting up a database is not described in detail here (however refer to the section on Example setting up PostgreSQL for for a high level overview).
The Reef server needs connection details for the database which are read the Reef settings configuration file (by default $AS_HOME/etc/ashttpd.conf, see Reef Configuration File) in the [django.db:default] section. For example:
[django.db.:default]
ENGINE = django.db.backends.postgresql_psycopg2
NAME = reef
USER = reef
PASSWORD = secret
HOST = psqlserver
The example above configures a PostgreSQL database running on the host psqlserver that is listening on the default PostgreSQL port (5432). If an alternate port is required then you can set the PORT value accordingly. The ENGINE value specifies the database backend that Reef will use to connect to PostgreSQL. The NAME settings specifies the database name, in this case, reef.
The schema and indexes are created using syncdb subcommand of the django-admin.py tool shipped with reef. There are a few things to look out for when using this: you must ensure the reef python library is loaded and you must point the tool at the reef configuration this is done.
$ PYTHONPATH=/opt/abilisoft.com/reef/share/reef.pyz \
AS_DJANGO_CONFIG=/opt/abilisoft.com/reef/etc/ashttpd.conf \
django-admin.py syncdb --settings=reef.django_settings
This will ask you to create an initial superuser, it is best to create one right away in which case you can skip the last step of creating one explicitly.
If you didn’t create a superuser as part of the previous step you will need to do this now. You need to create an initial superuser from the command line. After this it is possible to create new users using the admin interface of reef. This is done using the createsuperuser subcommand of the django-admin.py tool. As before you need to point it to the reef python library and it’s Django configuration.
$ PYTHONPATH=/opt/abilisoft.com/reef/share/reef.pyz \
AS_DJANGO_CONFIG=/opt/abilisoft.com/reef/etc/ashttpd.conf \
django-admin.py createsuperuser --settings=reef.django_settings \
--username foo --email=foo@example.com
This example demonstrates how to set up the PostgreSQL database on CentOS 5.6. Start by installing the database server itself:
# yum install postgresql84-server
...
# service postgresql initdb
# service postgresql start
The database server should now be installed and running. It is installed under the postgres user and only this user is known to the database for now. Next we need to create a user; ensure the user has a password so the Reef server can connect to the database:
# su - postgres
$ createuser --pwprompt myuser
Enter password for new role:
Enter it again:
Shall the new role be a superuser? (y/n) n
$ logout
#
Once the user is defined we need to give that user a database, we’ll create a database named reef:
# createdb -O myuser reef
Next configure the connection in the Reef runtime settings, ensure the [django.db:default] section in /opt/abilisoft.com/reef/etc/ashttpd.conf configured as follows:
[django.db:default]
ENGINE = django.db.backends.postgresql_psycopg2
NAME = reef
USER = myuser
PASSWORD = secret
Notice how the HOST and PORT directives are not included, since we connect to the localhost we do not need these and will actually break the default PostgreSQL authentication on a localhost.
Now initialise the database for Reef use. This will also prompt you to create a superuser. This is the user with which you will be able to log into the Reef Administration interface to manage other users:
# PYTHONPATH=/opt/abilisoft.com/reef/share/reef.pyz \
> AS_DJANGO_CONFIG=/opt/abilisoft.com/reef/etc/ashttpd.conf \
> django-admin.py syncdb --settings=reef.django_settings
Creating tables ...
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_user_permissions
Creating table auth_user_groups
Creating table auth_user
Creating table auth_message
Creating table django_content_type
Creating table django_session
Creating table django_site
Creating table django_admin_log
Creating table app_eventsvc_servicedef_filterdefs
Creating table app_eventsvc_servicedef
Creating table app_eventsvc_filterdef
Creating table app_eventsvc_filterterm
Creating table app_reef_event
Creating table app_reef_eventattr
Creating table app_reef_eventlog
Creating table app_reef_severity
You just installed Django's auth system, which means you
don't have any superusers defined.
Would you like to create one now? (yes/no): yes
Username (Leave blank to use 'root'): admin
E-mail address: admin@example.com
Password:
Password (again):
Superuser created successfully.
Installing custom SQL ...
Installing indexes ...
Installed 41 object(s) from 2 fixture(s)
#
Start ashttpd as normal:
# ashttpd
ashttpd application initialising (7.1 (donbot.1) r14146 20130510)
#
Navigate your browser to http://localhost:80 and log in to Reef client to ensure your setup is working.
Note
Please heed the ashttpd Security Considerations warning for production uses.
The following shows a couple of simple reverse proxy configurations for ashttpd with Apache and Nginx acting as the proxy. It assumes the ashttpd server is installed and running correctly.
By default Reef serves its own static content (images, stylesheets, etc.,) via ashttpd but it is preferable to offload this task to the reverse proxy server. Other benefits of using a reverse proxy setup is that the proxy server can terminate an optional SSL/TSL connection and it is easier to scale horizontally by adding more ashttpd workers as the load increases.
The following sections demonstrates how to configure the reverse proxy to serve static content directly while forwarding all other requests to an ashttpd instance. The examples will assume the proxy server and ashttpd server are deployed on the same server. Furthermore the URL Reef will be served on is http://reef.example.com/reef/, the extra reef/ path name prefix is used so that it is possible to deploy other Abilisoft Reef web services onto the same server, e.g. one could deploy the Performance web service at http://reef.example.com/perfsvc/ which would be handled by the same proxy server but redirects requests to a different ashttpd instance which serves the performance service.
These examples assume the proxy server and the ashttpd server are running on the same server. For security reasons the ashttpd will only accept connection originating from the local host.
[settings]
httpd_wsgiapp = $AS_HOME/share/reef/reef.wsgi
django_config = $AS_HOME/etc/ashttpd.conf
httpd_addr = 127.0.0.1
httpd_port = 8081
httpd_allowed_hosts = reef.example.com,127.0.0.1
httpd_proxy_prefix = /reef
httpd_proxy_protocol = http
[django]
STATIC_URL = "/static/reef/"
The httpd_proxy_* settings are of particular note here. When Reef generates URLs it uses information made available to it from ahttpd to ensure they point to the right place. If these are not set correctly 404 Not Found errors errors may be encountered when accessing Reef.
This should be set to whatever the URL of the static content is if it is not being served by ashttpd. It is recommended to use the shown value as this allows the proxy server to serve static files for multiple Abilisoft Reef web services as the same time.
Important
Note the quotes around the value for this setting, settings in the [django] section are python literals hence the need for the quotes.
Important
The path here must end in a trailing slash character.
Make sure not to set the FORCE_SCRIPT_NAME Django setting as it will effectively override httpd_proxy_prefix, preventing the Reef client from being able to communicate with the server.
These are the general steps that must be taken in order to ensure Reef behaves correctly behind a reverse proxy. Specific examples for Apache and Nginx are included below.
Warning
HTTP headers can be easily spoofed. Rogue clients may attempt to use the X-Forwarded-Host in an attempt to perform man-in-the-middle attacks.
Avoiding HTTP header spoofing is fairly straight forward, the following measures both address this and it is best to combine both:
Overwrite the X-Forwarded-Host completely. Some servers by default will append their hostname to the end of the header instead of overwriting it. However, ashttpd will only ever use the first item in the list. Therefore a client could set the header on the initial request which would cause their own custom value for the header to be used.
By overwriting the header sent by the client you completely remove the possibility of this.
Use ashttpd‘s httpd_allowed_hosts runtime setting to prevent serving hosts which aren’t trusted. This would result in the rouge client seeing a 403 Forbidden if it requested a host not included in the list.
By default httpd_allowed_hosts is configured to allow all which is insecure. You’re encouraged to set it even if not using a reverse proxy.
Note
The X-Forwarded-For header is intended to be a comma-separated list of proxies which the request has passed through with the origin of the request as the left-most element in the list. E.g.
X-Forwarded-For: client, proxy1, proxy2
Make sure to simply append the reverse proxy’s address to the end of the list and not completely overwrite it. Apache and Nginx provide this functionality.
For Apache the server must have the proxy, proxy_http and headers modules loaded. The configuration should then contain following directives inside the <virtualhost/> directive to configure the reverse proxy:
<VirtualHost *:80>
ServerAdmin webmaster@localhost
# ...
Alias /static/reef/ /opt/abilisoft.com/reef/share/reef/static/
ProxyPass / http://127.0.0.1:8081/
RequestHeader set X-Forwarded-Host reef.example.com:80
</VirtualHost>
Inclusion of the X-Forwarded-For and X-Forwarded-Host headers is done automatically by the ProxyPass directive. However, Apache simply adds its host address to the end of X-Forwarded-Host which would make ashttpd vulnerable to header spoofing (see above). Therefore it is desirable to explicitly overwrite the header.
Important
Be aware that trailing slashes are important in Apache’s configuration files.
To configure a load-balanced solution where Apache distributes the requests between multiple ashttpd workers is also fairly straight forward. Firstly the proxy_balancer module needs to be enabled on top of the proxy and proxy_http modules, then all which is required is a separate <Proxy balancer://...> directive and using it in the ProxyPass directive:
<VirtualHost *:80>
ServerAdmin webmaster@localhost
# ...
<Proxy balancer://reef>
BalancerMember http://127.0.0.1:8081
BalancerMember http://127.0.0.1:8082
</Proxy>
Alias /static/reef/ /opt/abilisoft.com/reef/share/reef/static/
ProxyPass / balancer://reef/
RequestHeader set X-Forwarded-Host reef.example.com:80
</VirtualHost>
Important
Be aware that trailing slashes are important in Apache’s configuration files.
In this configuration it is necessary to start a second ashttpd instance to handle requests. This is fairly simple but the new instance must use a different name in order to have a consistent log file and pid file etc. Additionally it must be told to use the same configuration file but the port to listen on must be changed. So the new instance can be started as such:
/opt/abilisoft.com/reef/bin/ashttpd \
--name=reef2 \
--cfg_inifile=/opt/abilisoft.com/reef/etc/ashttpd.conf \
--httpd_port=8082
If the configuration file is not explicitly overwritten then a configuration file based on the name, i.e. in this case /opt/abilisoft.com/reef/etc/reef2.conf would have been used. However since only the port needs to change it is easier to re-use the same configuration file.
To configure an Nginx web server as a reverse proxy for ashttpd the following configuration should be sufficient:
server {
server_name reef.example.com;
# ...
location /static/reef/ {
alias /opt/abilisoft.com/reef/share/reef/static/;
}
location /reef/ {
proxy_pass http://127.0.0.1:8081/;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
}
}
As with the Apache example, we override the X-Forwarded-Host request header to prevent it being spoofed by the client.
Important
Take particular care to ensure the proxy_pass directive has a trailing / character. If this is missing then Nginx will not re-write the URL for the request when proxying and ashttp will receive requests with the /reef/ prefix still in the ULR which it does not know how to handle. In this case an additional explicit rewrite rule like rewrite /reef/(.*) /$1 break; would be needed.
To configure a load-balanced solution where Nginx distributes the requests between multiple ashttpd workers is also fairly straight forward. For this a upstream directive must be used to list all the ashttpd workers and then the proxy_pass directive modified to use this upstream directive:
upstream reef {
server 127.0.0.1:8081;
server 127.0.0.1:8081;
}
server {
server_name reef.example.com;
# ...
location /static/reef/ {
alias /opt/abilisoft.com/reef/share/reef/static/;
}
location /reef/ {
proxy_pass http://reef/;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
}
}
Important
Be aware that trailing slashes are important in Apache’s configuration files.
In this configuration it is necessary to start a second ashttpd instance to handle requests. This is fairly simple but the new instance must use a different name in order to have a consistent log file and pid file etc. Additionally it must be told to use the same configuration file but the port to listen on must be changed. So the new instance can be started as such:
/opt/abilisoft.com/reef/bin/ashttpd \
--name=reef2 \
--cfg_inifile=/opt/abilisoft.com/reef/etc/ashttpd.conf \
--httpd_port=8082
If the configuration file is not explicitly overwritten then a configuration file based on the name, i.e. in this case /opt/abilisoft.com/reef/etc/reef2.conf would have been used. However since only the port needs to change it is easier to re-use the same configuration file.
The purpose of the Reef site administration pages are to enable an administrator to:
The Reef site administration home page is divided into sections, one for each internal application described above. Each object type name is a link and clicking it (or the associated Change link) will navigate you to a list of items of that type.
Clicking Add will navigate you to a form enabling you to add a new object of a particular type.
Recent Actions shows a lit of recent updates. The links will navigate you to the affected object (if it still exists).
The Event Service Application contains Service and Filter definitions. There are used to specify Services (groups of related events) in the Reef UI.
A service definition groups a set of filter definitions, that in turn group events according to filter terms. See below for description of each service definition attribute.
The Service Def list allows you to edit some fields in view, for more than one service. Click the Save button at the bottom of the page to save any updates made. To modify the Service Def name or any assigned Filter Defs then click on the name link. This will navigate you to an appropriate form.
Navigate to a Service Def update form by clicking on the name link in the Service Def list. Use drop down list to assign an existing Filter Def to the Service Def. Clicking the Add icon will open a pop-up enabling you to create a new Filter Def.
Check the Delete? box against the Filter Def you want to un-assign and click save.
Note
This will not delete the Filter Def, only its relationship to the Service Def. To delete a Filter Def you need to navigate to the Filter Def page.
There are two ways to delete a Service Def:
- In the update form for a Service Def click the Delete button at the bottom left of the page.
- In the Service Def list:
- Check the box to the left of the Service Defs you want to delete
- Open the Action drop down list at the top of the list
- Select Delete selected service defs
- Click Go
A filter definition is assigned to a service definition and groups a set of filter terms. See below for description of each filter definition attribute.
The Filter Def list displays defined filter defs available to be assigned to Service Defs. To modify a Filter Def click on the name link. This will navigate you to an appropriate form.
Navigate to a Filter Def update form by clicking on the name link in the Filter Def list. In the “Filter Terms” section:
Note
Their are lots of filter operators that have various behaviours. Some are binary operators require that you fill in both Cmp val0 and Cmp val1 fields. Refer to the section on Filter Operators for more information.
The exclude check box inverts the filter’s behaviour. When checked, events that match the filter term are excluded from being associated with the overall service state.
Check the Delete? box against the Filter Terms you want to delete and click save.
Note
This will not delete the Filter Def, only its relationship to the Service Def. To delete a Filter Def you need to navigate to the Filter Def page.
There are two ways to delete a Filter Def:
- In the update form for a Filter Def click the Delete button at the bottom left of the page.
- In the Filter Def list:
- Check the box to the left of the Filter Defs you want to delete
- Open the Action drop down list at the top of the list
- Select Delete selected filter defs
- Click Go
The Reef Application pages present data that pertains to captured events and the admin views enable one to fix data issues if required. The update pages behave in a very similar fashion to those already described.
This view presents the event data that can also be seen in the Reef UI Events dashboard. In the event list some attributes can be updated in view. Clicking on an event key navigates you to an event update page where you can also view and edit related event logs and user fields.
As an event is updated over time, as new instances of the event are captured, de-duplicated and correlated the updates are logged. This view duplicates what can be seen in an event update page, but for all events.
An event can have any attributes the source sends to Reef. This view shows such user defined attributes and duplicates what can be seen in an event update page, but for all events.
The Auth Application pages present data that pertains to uses and groups and enable the administrator to manage them. The update pages behave in a very similar fashion to those already described.
This view allows one to filter a user list, add and delete users. Clicking on a username field opens a user update form which has self explanatory fields. Here users can be activated and deactivated, assigned to groups and have permissions updated. Permissions relate to operations within the site admin console and will not affect the way the Reef UI behaves.
This view allows one to add and delete groups. A group is simply a name for a permission set and a way of aggregating users. Clicking on a group field opens a group update form which has self explanatory fields. Here group permissions can be updated. Permissions relate to operations within the site admin console and will not affect the way the Reef UI behaves.
Reef has the ability to connect to a secondary service which makes performance data (e.g. CPU usage) collected by Abilisoft Monitoring Agents available to the client. This performance data service (Perfsvc for short,) like Reef, runs on top of ashttpd and has the same set of runtime settings available to it.
In order to enable the performance data dashboard functionality for Reef some additional configuration must be performed. As well as this a separate Perfsvc instance must be configured and running.
Perfsvc is installed and run in the same way as Reef. The key difference is the default httpd_wsgiapp in the included ashttpd.conf file for Perfsvc is set to perfsvc.wsgi instead of reef.wsgi. As well as this a new Django setting called APP_PERFSVC is needed.
Perfsvc uses a MongoDB database in order to store and access performance metrics generated by monitoring agents.
Below is the default APP_PERFSVC setting.
[django]
APP_PERFSVC = {
'DB_NAME': 'abilisoft',
'DB_HOST': '127.0.0.1',
'DB_PORT': 27017,
'TEST_DB_HOSTS': ['127.0.0.1'],
}
These settings tell Perfsvc where it can find the MongoDB to use for storing metrics.
Note
The MongoDB used by Perfsvc must be installed and configured separately and is not covered in this documentation.
As well as installing and running the Perfsvc instance and it’s corresponding Mongo database, Reef’s ashttpd.conf must have its [django] section updated to include the APP_METRICSVC setting. It is this setting which tells Reef how to communicate with the performance service.
The default Reef ashttpd.conf configuration file comes with a default APP_METRICSVC.
[django]
APP_METRICSVC = {
'PERF_SVC_BASE_URI': 'http://127.0.0.1:8080',
'PERF_SVC_CALL_TIMEOUT': 2.0
}
The performance data metrics managed by Perfsvc identify their origin and what they relate to (e.g. CPU usage, memory usage) via a ‘node path’. A node path is dotted string which represents a hierarchical chain of nodes. For example:
osumi.clockhouse.trivial.cpuMon.total.percentIdle
In this example the first two nodes – osumi.clockhouse – identify the domain name of the host which generated the performance metric. The following three nodes relate to the name of the component, monitor name and sub ID used by the Abilisoft Monitoring Agent used to generate the metric. The final node – percentIdle – indicates what kind of data is held by this metric, which in this example is the ammount of CPU idle time as a percentage.
When a user wants to add a panel to a dashboard they’ll be prompted for a node path. It is this value the Reef client uses in order to graph performance data. So, if the above example node path was entered, then the Reef client would graph the CPU idle time for osumi.clockhouse.
In order to view what node paths are available navigate to Perfsvc in a browser.
Note
Be sure to use the correct port number in the Perfsvc URIs. The examples below use 8081, but by default Perfsvc will attempt to listen on 8080.
This is a navigable interface to the Perfsvc REST API. The entry called metrics contains a link to the metrics list. Clicking on this link will forward the browser to a list of all available metric node paths.
Clicking any of these links will display a page which lists various details about a metric node path such as:
