Results processing

As you can see, you can use Puffin to automate some complex tasks. Puffin has a great deal of information that it is maintaining and even more available to share with you as the results of your plan's execution. This section introduces the various results processing features of the Puffin Automation Framework and how they are configured and used.

Logging

I built the Puffin logging system on top of Vinay Sajip's logging module for Python. You can set the logging detail level using either the framework configuration file (engine.xml in the config folder for your workspace) or on the command line. Logging allows you to see -- if you want to -- what the framework is doing step by step during its execution of your plan file. You can set it to log at the lowest, most detailed level (DEBUG) to the highest (SILENT).

By default, Puffin logs information to the standard output stream. However, you can configure Puffin to log to whatever targets are allowed by Vinay's logging module. Listing 12 shows the logging configuration section of the engine.xml file:

<engineConfig>
.
.
.

   <logger defaultDetailLevel="INFO">

      <handler type="StreamHandler">
         <param name="msgFormat"><![CDATA[%(message)s]]></param>
      </handler>
   </logger>
.
.
.

The defaultDetailLevel attribute of the logger element, as its name implies, specifies the default logging level. You can override this setting on the command line like this:

  puffin --logging=DEBUG

This will cause the logging level to be set at the DEBUG level (verbose) for this run of your plan.

The rest of the logger element specifies the output channel to use for the logging messages and the message format for each channel. Puffin comes configured to write to the output stream. However, you can write to a file or other channels. See the PafLogger class in the paf.core.base module and the paf.thirdparty.logging module for more details.

Alerts

At the beginning of this article, I suggested that the target problems that I built Puffin to solve included Web application testing, system monitoring, and data transfer automation, among others. If you are using Puffin to address issues in any of these areas, and the framework's execution of one of your actions fails, then you probably want to know about it.

That's where failed task alerts come in. This feature allows you to set up Puffin to log to a file and initiate an e-mail message for any task that fails. The result is an e-mail that looks something like that shown in Listing 13.

This is an alert notice from the PAF Monitoring system.
An PAF monitoring test has failed:

Task name: makePurchase
Time of Failure: 15 Aug 2002 09:46:21 PM
Action name: generateSelectionCount
Failed Validator Type: ExpressionValidator
Validator Message: 
    [Last evaluation expression used:
(1 <= 0) and (0 > 0)
RESULT: FAILURE]

You can get more information by visiting the following: 
    TODO: http://www.yourdomain.com/reports/20020815214556.html

As Listing 13 shows, failed task alerts tell you what failed and how it failed. These alerts can be logged to a file and/or sent to one or more e-mail recipients. This feature is ready as part of the core distribution but must be configured. The section of the engine.xml configuration file is shown in Listing 14.

   <resultsProcessing reporter="1" alerter="1">

      <alerter email="0" file="1">
         <param name="reportURLBase">
            <![CDATA[TODO: http://www.yourdomain.com/reports]]>

         </param>
         <param name="mailHost" eval="0">
            <![CDATA[TODO: mail.yourdomain.com]]>

         </param>
         <param name="fromAddr" eval="0">
            <![CDATA[TODO: [email protected]]]>

         </param>
         <param name="fromAddrName" eval="0">
             <![CDATA[
                TODO: PAF Monitor [email protected]
             ]]>

         </param>
         <param name="toAddr" eval="0">
            <![CDATA[TODO: [email protected]]]>

         </param>
         <param name="toAddrName" eval="0">
            <![CDATA[
              TODO: Other [email protected]
             ]]>

         </param>
         <param name="subject" eval="0">
            Alert: Site possibly down!
         </param>

      </alerter>

The first item of note above is the first line:

  <resultsProcessing reporter="1" alerter="1">

From this first resultsProcessing element, you can turn on and off both alerts and reports. The rest of the alert details are relatively self-explanatory. Puffin uses the reportURLBase parameter to build the link at the bottom of the task alert message.

Reporting

In the original release of Puffin, I skimped on the reporting of results for the execution of a plan. In the rewrite of Puffin, you now have two options for reporting format with four levels of detail.

As I mentioned earlier, you turn on reporting with the reporter attribute of the resultsProcessing element in your engine.xml file:

  <resultsProcessing reporter="1" alerter="1">

The rest of the features of the reporter configuration are also simple and straightforward as shown in Listing 15:

.
.
.
      </alerter>
   
      <reporter xml="1" html="1" detailLevel="3" ftp="0" makeMenu="1">

         <param name="ftpHostName">
             <![CDATA[TODO: ftp.yourdomain.com]]>
         </param>

         <param name="ftpUserName">
             <![CDATA[TODO: YOUR FTP USERNAME]]>
         </param>

         <param name="ftpPassword">
             <![CDATA[TODO: YOUR FTP PASSWORD]]>
         </param>

         <param name="ftpRemoteTargetDir">
             <![CDATA[TODO: FTP LOCATION FOR FILES]]>
         </param>

      </reporter>
   </resultsProcessing>

You control the main features of Puffin's reporting system through the five attributes of the reporter element above:

• xml -- Whether or not to write an XML report for this plan execution.
  
  
• html -- Whether or not to write an HTML report for this plan execution.
  
  
• detailLevel -- The amount of detail to write to the above two report types. The detail levels range from "0" for task-level success/failure to "3" for everything including the action results for every action in your plan!
  
  
• ftp -- Whether or not you'd like Puffin to transfer (via FTP) your report(s) and their corresponding menu page (see below) to a Web server of your choice. This is a handy feature for easily getting your Puffin plan results to an accessible server. Reports are always written to your current workspace's /reports folder. Setting this attribute to TRUE forces Puffin to copy them to the FTP server upon execution.
  
  
• makeMenu -- Puffin will, if you set this attribute to TRUE, create for you a running menu page with links to the last fifty plan execution reports (HTML and XML). This Web page is purged as it is built so that only the last fifty plan execution reports are listed. This provides a single Web page to which your users can go to easily see any of the last fifty execution reports.

As you can guess, the parameters for the reporting are only for the location (etc) for the FTP site to which you'd like Puffin to transfer your reports (and menu page).

Action token writers

The XML and HTML report formats allow you to maintain a detailed record of the state of each task in your plan as Puffin executes it. From these reports, you can retrieve token values, action results, or parameters for any action in any task in your plan (for any repetition) if you have set the detail level high enough.

Most of the time, these features will serve your needs for data storage and record keeping. Sometimes, however, this level of detail is overkill. Sometimes you just need to maintain a record of the "important data" being used by Puffin in the execution of your plan. Usually this can be accomplished if you can maintain the values of only the action tokens throughout the execution of your plan.

With the latest version of Puffin, you can do exactly that. You simply specify a list of tokens to be stored and how you'd like them stored. For example, the following code (Listing 16) , when added to your plan, will instruct Puffin to store the value of the SESSION_ID action token after each action that in some way affects this token (in other words, has it as an output).

      <token name="SESSION_ID"
             txtStorage="1"
             xmlStorage="1"
             standaloneStorageFileBase="sessionToken.dat"/>
   </tokenPersistence>

You can have as many tokenPersistence elements in your plan (or plans) as you like, but each token will only be maintained according to its last definition (using the token sub-element of tokenPersistence) in the plan.

Upon its initial processing of your plan (before execution begins) Puffin gathers a list of tokens to be persisted. As each action is executed, Puffin checks to see if that action has, as an output, one of the tokens to be stored. If so, it persists the value of that token (after the action has been executed, of course) to one or more target repositories (where the value will be stored), based on your token persistence settings.

You have three choices for how you'd like Puffin to persist the value of an action token (set as attributes in the token element):

1. txtStorage -- Puffin adds the value of each updated token as a line to the token storage text file (/storage/tokenStorage.txt) for the current workspace (see next section for more on workspaces). The value Puffin will add to this file looks like this:

     SESSION_ID...1029546899.88...login...login...repeat1...1...163610295899

   The fields are token name, time stamp, task name, action name, repeat label, iteration number, and token value. Each field is separated from the next by three periods.

2. xmlStorage -- Puffins add an XML element for each updated token value to the XML storage file for your current workspace (/storage/tokenStorage.xml). This file looks like the following (added token in bold):

   <tokenPersistence> <tokenValue name="SESSION_ID" time="1029545227.58" action="login" task="login" repeatLabel="repeat1" iteration="1"> <value><![CDATA[163610295899]]></value> </tokenValue> </tokenPersistence>

   
   

3. standaloneStorageFileBase -- This is a different kind of persistence and is useful for action tokens with large contents. In this example, Puffin generates a separate file for each token value as it is persisted (every time it changes). The attribute specifies the file base name to use. Puffin takes this, adds the current time stamp as a prefix, opens a new file of that combined file name, and writes to it the current value of the token. This feature should be used only when you really need it. It is expensive in terms of CPU and I/O usage.

Workspaces

Throughout this article, I've mentioned a concept called a "workspace." A workspace is a context within which Puffin executes. It defines where Puffin should retrieve its platform configuration (engine.xml), retrieve its action definitions (action.xml), write its reports, store persisted tokens, and look for plan files. Puffin tells you which workspace it is working within in each logging statement (the following indicates that Puffin is working in the context of the default workspace):

  [default] - Plan initialization commencing.

You can set the workspace for Puffin on the command line like this:

  puffin --workspace=puffinhome.org

The above command line initiates puffin and sets the workspace to puffinhome.org. This tells Puffin to look in the /workspaces/puffinhome.org folder for all its configuration information and to write its results into subfolders under this folder. Since we didn't include a plan file on the above command line, Puffin will look for the default plan.xml file in the specified workspace (in the /workspaces/puffinhome.org/plans folder). If it cannot find a plan.xml file in that folder, the above command will result in an error and the abrupt end of Puffin's execution.

Why did I add workspaces into Puffin? There are several reasons.

The first is simply a matter of organization. If you are attempting to test, monitor, or automate data transfer for multiple sites, it helps to separate the action definition and engine configuration (not to mention the resulting report files) into separate areas. Workspaces provide a pre-defined method of doing so.

The next reason is for security. Remember that one of Puffin's target problem spaces is system monitoring. This will require some security because passwords may be involved throughout a complete system test. If you are using Puffin to test multiple systems, you may want to separate workspaces along security-based lines of responsibility.

The third is also a security-related reason. In the future, I plan to offer a Web-based interface to Puffin configuration. I'd like to enable users to offer that as a service over a Web site. The only way to make this acceptable is if person X is only able to see person's X action definitions, etc. Workspaces allow for this.

Finally, in each workspace folder for your installation of Puffin is a workSpaceInfo.xml file. In this file is located the various default information for the contact for the workspace. In the future, it will also contain information concerning scheduled execution of Puffin plans.