Getting started with panoptes
This first tutorial will provide a brief tour of a few of the major panoptes components. It does not show off some of the more interesting capabilities of the system, but is a nice simple place to get started.
Let's start by configuring a simple monitor for a single RSS feeds on a single machine.
Installing Panoptes
Before continuing, read and follow the instructions in the 'installing panoptes' document.
Panoptes Configuration
Before continuing, read and follow the instructions in the 'configuring panoptes' document.
Configuring a monitor
Let's configure a couple of simple RSS monitors. The RSS config block should contain a 'sites' key that lists a short name for each feed (e.g. 'slashdot') along with that feed's URL. Here is the RSS plugin configuration embedded in a host.yaml file:
---
myhostname:
plugins:
Panoptes-Monitor-RSS:
sites:
slashdot:
url: http://slashdot.org/slashdot.rss
planetperl:
url: http://feeds.feedburner.com/PlanetPerl?format=xml
privatefeed:
url: http://private/rss
user: myuser
pass: mypass
proxy: http://localhost:8888/
refresh: 3600
The refresh is specified in seconds. Note that the default refresh is every 15 minutes.
Running the monitor
The RSS monitor requires some additional perl modules:
HTML::TableExtract
You can start the monitoring engine by setting into the root of the panoptes project and running:
./panoptes -monitor -start -tail &
This should generate some output similar to:
2009/09/05 21:00:56 Initializing monitoring core...
2009/09/05 21:00:56 Getting host plugins
2009/09/05 21:00:56 Loading plugin: Panoptes-Monitor-RSS
2009/09/05 21:00:57 Scheduling new service check in 187: RSS/planetperl
2009/09/05 21:00:57 Scheduling new service check in 35: RSS/slashdot
2009/09/05 21:00:57 Core and plugins initialized.
2009/09/05 21:00:57 Calling run() method on monitoring core...
2009/09/05 21:00:57 Starting POE kernel: Panoptes::Monitor::Core
Note that the two RSS feeds were new, and so the first checks were announced and were randomized (slashdot will be checked in 35 seconds). Once the feeds have been checked successfully once, they will no longer be displayed when starting the monitoring engine.
Ok, take a short break and let this run for a while. Let it run at least as man seconds as were listed for your feeds to refresh. Come back in a few minutes and we'll see how it's going.
Messaging
When the monitoring component detects the occurrence of a new event, it will generate a message. Events may be external (e.g. a previously unseen article in a monitored RSS feed) or internal (e.g. an error loading the RSS plugin). Event messages may be a simple hash of key-value pairs or may contain values with complex data structures. The monitoring component uses the messaging component to place the event messages in a FIFO queue. These messages are then picked up asynchronously by the rule component. This ensures the monitoring components won't get backed up waiting for the rules engine.
Right now the message queue is a maildir style mailbox. If you have mutt installed, you can take a look in the queue:
mutt -R -f ~/.panoptes/inbox
Note that mutt will prefer to move all the mail out of your inbox on exit unless you have the following line in your ~/.muttrc:
set nomove
You may also see some messages for internally generated events, or some errors (especially if you ran into problems the first couple of times you tried to start up).
If you included the slashdot feed in your config, you should see some messages in your queue from slashdot:
grep '\[slashdot' ~/.panoptes/inbox/new/*
Note that as the rules engine processes messages, it will remove them from the mailbox. Any interesting information should be captured and stored by rules into SQLite/RRD/DBM::Deep/Log files/etc.
Rules
The default rules engine implementation (the only one that currently exists) is based on the concept of email filters. If the event message matches some condition, then perform some action. There are some interesting conditions and actions that are still in development, and you can always write your own custom rule conditions and/or actions in perl, but for now let's focus on some simple ones that are relevant to the task at hand.
Configuring rules
First, let's address the condition. We're looking for messages generated by the Panoptes-Monitor-RSS plugin where the 'site' field matches 'slashdot'. Such a filter condition would look like this:
condition:
plugin: Panoptes-Monitor-RSS
equal:
site: slashdot
At this point you might be asking yourself how to know to know to look for a 'site' field on a message generated by the RSS plugin. The answer is that there is no enforced schema for event messages (although there are some standard metadata fields, e.g. checksum and hostname). Each plugin is free to generate an event message with a schema that corresponds to that monitor's domain (e.g. RSS entries have a title and a description). You'll have to look at the monitor's documentation (or source) to see what fields the plugin generates. Hopefully this will be well documented in the future.
This freedom from schema may be considered a blessing and/or a curse. It certainly makes the system much less complex and much less confusing. My hope and belief here is that the flexibility of the rules system more than compensates for the lack of structure.
In this case we could make the rule a bit more generic. Let's match any message coming from the RSS monitor that has a subject.
condition:
plugin: Panoptes-Monitor-RSS
contains:
subject: 1
Or, why not just display the subject of every single message.
condition:
contains:
subject: 1
Now let's take a look at the pointy end of the stick, the action. Let's start by displaying a console message. The action for a console message is trivial:
action:
console: 1
Similarly, if we wanted to display a growl pop-up on OS X using the growlnotify command, we could use the growl action
action:
growl: 1
Putting it together and integrating it into the host.conf:
---
myhostname:
plugins:
Panoptes-Monitor-RSS:
sites:
slashdot:
url: http://slashdot.org/slashdot.rss
planetperl:
url: http://feeds.feedburner.com/PlanetPerl?format=xml
rules:
50_console_message:
condition:
contains:
subject: 1
action:
console: 1
Note that I gave the rule a name (50_console_slashdot) and that the rule started with a number. The order associated with the rule is a priority that determines where in the chain the rule gets run. This is because one rule may alter a message in a way that may affect another rule. Unlike other rules system, this sort of internal event will not automatically generate a new event. Instead the rules act more like unix filters working in unix pipes. Actions may be as simple as adding a new field (display slashdot posts in green) or might perform more complex tasks like word wrapping or language translation.
Let's make a rule that will colorize slashdot posts. This rule will run before the console rule so that the color will already be set by the time the console rule runs. Here I have extracted only the rules section of the config:
rules:
20_slashdot_color:
condition:
plugin: Panoptes-Monitor-RSS
equal:
site: slashdot
action:
add_field:
color: green
50_notify_message:
condition:
contains:
subject: 1
action:
console: 1
growl: 1
Starting the rules engine
Ok, time to engage the rules engine. From the root of the project, run:
./panoptes -rules -start -tail &
And within a few seconds you should have some pretty green messages showing up on your terminal.
Inserting into a database
Ok, how about storing all the incoming RSS feeds in a sqlite database. For that we can use the sqlite rule. We have to decide which fields we want pulled from the message and inserted into the database. Also we need to specify the schema for those fields. Once that's done, when the rule fires, if the database or table do not exist when the rule is processed, they will be created automatically.
rules:
90_rss_sqlite:
condition:
plugin: Panoptes-Monitor-RSS
contains:
title: {}
action:
sqlite:
table: rss
schema:
site: string
title: string
subject: string
body: string
link: string
timestamp: int
Examples
There are some more config examples in the sample-config directory in the root of the panoptes project.
The end
Well that's all for this time.
For more information on managing the running daemons, see the documentation for the 'panoptes' script. From the root of the panoptes project, run:
./panoptes -help
For more information on the options, try:
./panoptes -help -v
Or, for the complete pod docs:
perldoc ./panoptes
When both the monitoring and rules daemons are running, panoptirules will pick up and deliver messages from panoptirules very quickly.
Next time we'll investigate the database options a bit further and also look at the RRD integration.
At some point I will also discuss the panoptifeed script which can be used to generate an RSS feed of the data from the sqlite tables.