Submission Folder

The Submission Folder is central to the operation of the DD-STAT application. Device registration with FCG requires extensive information including:

• Source code for the DD

• Source code for the prior version of this DD (revisions only)

• Log files from the HART Test System (HTS)

• Other documents

The specifics of this are covered in the FieldComm Group guidance document ____________________.

It is necessary for the information to be provided in a uniform way so that automated evaluation of it is possible. The contents of this folder are shown below:

The DD-STAT automation operates from the root of this folder structure. Prior to every operation it examines its current working directory and will terminate if it is not running from a properly structured submission folder.

There are three files in the structure specific to the automation, not covered in the guidance document mentioned above. You must configure these files as described below for the automation to run. These are:

• Test report document

• DeviceOptions.xml

• Filepaths.config

Test Report Document

The automation ships with a test report template:

TR10208-ZZZZ{2.0.12}_HART-EDD-Test-Report.docx

This template contains document sections, summary tables and boilerplate text consistent with the specification.

Note that the automation is sensitive to the format and content of this template. The version number of the document is checked against the current version expected by the automation. A warning is produced if the version of the document is earlier than expected.

The DD-STAT automation works with the test engineer to produce a full text report in the following ways:

• Boilerplate text and document structure in the template follow the requirements of the test specification.

• The automation will insert tokenizer outputs and other relevant automation output directly into the report document in specific sub-sections.

• Test engineers can insert explanations, and screen captures to support the test report.

The automation can be run multiple times, with its output inserted into the report again and again without overwriting the information input directly by the test engineer.

To accomplish this, the test report template includes a custom Word document style called “TFPNotes”. Any text in the document that has this style will be overwritten by the automation when the report function runs. The style is characterized by a black line box around the text.

The automation will also insert Pass/Fail/Skipped notations into the test summary table in each section of the document.

DeviceOptions.XML File

ConfigurationFiles\DeviceOptions.xml

This file contains information important to the test campaign that is not easily or reliably obtained directly from the DD source code.

Each editable field has instructions and comments around it that will guide you to place the correct value there.

General instructions are:

• The fields initially contain the value ‘unknown’ or an invalid value. The automation will flag these as errors until you fill them with valid values.

• True/false parameters start with ‘Is’: ‘IsSupportedInDevice’ for example. Replace ‘unknown’ with a value of ’true’ or ‘false’ with no capitalization.

• The numeric fields are integers.

• If the field contains an enumerated value, the allowable values will be provided in a comment.

The Filepaths.config File

ConfigurationFiles\Filepaths.config

This file provides optional locations for the various parts of the submission. The values provided in the supplied file fit the requirements for the submission folder structure.

Do not edit this file.

The DD Standard Test Automation Tool (DD-STAT) is a comprehensive test automation tool which assists members of FieldComm Group with the process of registering HART DD projects.

Key features

• Collection It assists users to correctly assemble required source code and documentation for the submission.

• Evaluation It examines DD source code and tokenizer outputs, comparing them to the requirements of the DD test specification.

• Reporting It places the results of the examination into a test report template simplifying the reporting process.

• Submission It bundles the information that is required for registration and copies it to FieldComm Group servers.

DD-STAT tests fall into three categories:

• Automated tests evaluate DD source code and other artifacts

• Tokenizer tests inspect specific outputs from the tokenizers

• Manual tests rely on a test engineer to inspect the code base and provide a conclusion

The tool is configured as an extension to the Visual Studio Code ( VSCode ) editor in two pieces:

• A command line interface that could be run from a Windows terminal or in a DevOps pipeline.

• A user interface window that runs inside the VSCode IDE, which eases user interaction with the tool.

Jump right in

DD-STAT requires that the following items be installed:

1. VSCode

2. Required extensions for VSCode (see below)

3. FDI IDE (see installation guide)

4. The DD-STAT application and the DD-STAT VSCode extension

Install and Configure VSCode on Windows

Download and install your VSCode program using the link below. It supplies instructions for getting started using the IDE.

Install Required VSCode Extensions

DD-STAT application uses two third-party extensions for VSCode to enhance user experience. These are:

Download DD-STAT Components

During the beta, the DD-STAT components will be available on a SharePoint server in a ZIP file.

Install the DD-STAT Application

Double-click the installer ddstat.msi file to install the application.

This will install a DDSTAT folder in the C:\HCF folder alongside the HART library and other HART applications.

Install the DD-STAT License

DD-STAT provides a new FieldComm Group license manager, which will provide unified license management for FieldComm Group and FDI products. This license manager is improved to allow the user to install either via a license key or a license file.

Launch the manager from the start menu. Search for FieldComm Group folder and choose the license manager. You will see a pop-up asking which style of license you wish to provide. DD-STAT users are issued a serial number for a license. In the drop-down select ' Enter a serial number' and paste your provided serial number into the UI.

Install the DD-STAT VSCode Extension

Install the ddstat.vsix file as follows:

• In VSCode, go to the Extensions View by clicking on the Extensions icon on the left-hand sidebar (the one that looks like four squares).

• In the Extensions panel, click the More Actions button (three dots) in the top-right corner. From the dropdown, choose Install from VSIX....

• A file dialog will open. Navigate to the location where you downloaded the .vsix file, select it, and click Open.

• VSCode will install the extension, and once it's done, you’ll see it appear in your list of installed extensions. You might be prompted to reload VSCode for the extension to activate, so just click Reload if that happens.

The requirements for submitting an EDD package for registration are complex. Submissions that do not meet the requirements are returned to the developer for resolution. You can shorten the amount of time to have your submission accepted by running this test tool before you submit. These are the same tools used by the FieldComm Group to check your submission package.

Install suggested VSCode extensions

The DD-STAT extension requires these other extensions:

• Microsoft Live Preview

• EDDL

To install them, click on the extensions button at the extreme left of the editor. Then type the names of the extensions into the search bar. When you find each extension, click on it in the extensions of view to install it.

Install the DD-STAT extension

Locate the file ddstat.vsix extension on your computer. Install it using the '...' menu at the top of the extensions view.

After the extension is active, you will see a small pane in the file explorer view to the lower left. Click on the "DD-STAT" bar to show or minimize the user interface.

Operate the Test Automation

Open your Submission folder in VS Code. The extension will not operate from any other folder.

Browse the Submission files as desired using the explorer view at the left.

You operate the extension by pressing the buttons on the user interface. Their operation is described below.

Init

Create a new folder that will contain your DD registration submission files and open it in VSCode.

Then click the Init button. The extension will create the needed folders and documents in the current working directory.

A ConfigurationFiles folder will be created containing 'unknown' configuration settings for the test tool. You will need to provide the correct values for your device in this file before running the test automation.

Audit

The audit command will check to see if your submission folder structure is correct, and also that the required files are present and contain valid data. If the audit shows any errors, the test automation will not operate.

Run

Run the automation on the submission folder in the current working directory.

Results will be available in these files in the TestReports folder:

• audit.log

• summary.html

• detail.log

• console.txt

The run will terminate if VSCode is not open in a proper Submission folder. Check the audit.log file for diagnostics.

Summary

After test completes, click the Summary button to bring up the HTML dashboard display.

Detail

Click Detail button to open the recently completed detail.log file.

Report

Click this button to summarize the results of the automation run in the test report Word document that's found in the root of the current working directory. The document's report name will follow this convention: TR*.docx. The test report must be created from the template "TR10208-ZZZZ{2.0.xxx}_HART-EDD-Test-Report".

Errors and warnings from the tokenizer will be assembled into the correct document sections. Also, errors and warnings from the test automation will be summarized in the correct sections of this document.

A new tab will open in the editor showing the console output from the automation.

Submit

This button will open a new tab in the editor wit instructions how to submit your DD for registration.

Some of the test cases in the test specification are not automated. For some, it is not possible to automate the test case. In others, automation will be added to DD-STAT in future releases.

These test cases must be run manually by the test engineer. This section describes how the automation interoperates with the engineer.

Detail.log file and report summary (HTML)

In the detail.log, manual test cases will be marked with [WARN] logging level. The warning will also show in the graphical report summary. Also, the End: statement will contain the text "manual".

[WARN] - Unknown: TFP711_013 is evaluated manually.
[INFO] - End: TFP711_013  manual  Unknown (Warning)

Test report document

In this Word document, each section of tests contains a table summarizing the pass/fail results. In the automated tests, the disposition column is set by the automation and is overwritten each time the automation runs. In the manual cases, this cell will not be overwritten. This allows the test engineer to manually type the test result into these cells, and the values are preserved from run to run. When you first open the report, these cells will be empty.

In the detailed section for each test case, the result will show as Unknown and the evaluation as Manual. Text which is written directly into the document under the boxed automation results will not be overwritten from run to run.

• Windows 10 or 11

• The current release of FDI Package IDE (installed separately), minimum version 1.7.0.2

• Visual Studio code editor (free)

• Some third party VSCode extensions (free, see below)

The VS Code extension provide a graphic user interface to the DD-STAT application.

When the DD-STAT extension is installed in VSCode, a small user interface panel will be visible whenever the file explorer is showing in the left-hand pane. A small drop-down button (green arrow) enables you to open or collapse this panel.

The Summary button causes the summary.html output from the automation to display in a window with easy access to individual test results.

The Detail button causes the detail.log output from the automation to display in an editor window. See the description of logging outputs in the section describing console operation for ddstat run.

The Submit button will open an editor window with instructions for how to submit your DD for registration to FieldComm Group. Automated submission functionality for this button is under consideration for the first production release of DD-STAT.

You operate the automation using a command line interface (CLI) program named ddstat.exe. To use it, open a Windows terminal in the root of a submission folder.

You may use the Windows terminal that is integrated into VSCode. To do that, from the VSCode menu at the top, select:

Terminal > New Terminal

and a new Windows command terminal will open at the bottom of the VSCode window.

The general form of command lines are:

ddstat <command> <folder>

The first argument is the command, one of:

• init

• audit

• run

• report

The actions for these commands are described in sections below.

If the <folder> argument is not provided, then the automation will expect to find a submission folder configuration in the current working directory.

If the <folder> argument is provided, then the automation will execute from that folder.

If the named folder (or current working directory if there is none) is not a submission folder, the automation will look through all the subfolders and run the automation in each submission folder it finds. You can batch the operation in this manner.

ddstat --version

This prints the version number of the test tool to the console output.

ddstat --help

This prints brief instructions for using the CLI.

ddstat init <folder>

This command creates a new submission in the <folder> argument if supplied or in the current working directory if not. The new folder is populated with the required folders and any initial files that can be supplied by FieldComm Group.

If the <folder> path exists, it will not be modified.

ddstat audit <folder>

This command examines the contents of the <folder> argument if supplied or the current working directory if not. It prints errors and warnings encountered to the console output.

Additionally, it creates a file named ‘audit.log’ in the TestReports folder that contains the result of the audit using a log level of ‘info’. It documents which items in the submission folder are missing or contain invalid data and info lines with contextual information.

Audit.log helps the test engineer to correctly assemble the required configuration, HART Test System log files, and EDD source code required to operate the test.

ddstat run <folder>

This command runs the test automation in the <folder> argument if it is present, or the current working directory if not. It begins by running the audit command and will terminate the execution if any errors are reported during the audit.

This command will make at least two tokenizer runs, one for tokenizer version 8 and one for version 10. For campaigns of type of DDRevision or DeviceRevision, an additional run will be made using the historical DD source code that you provided.

The tokenizer command line will source the header and other Standard Library files appropriately based upon the IDERevision number and the CmnPrac revision number settings that you supplied in your DeviceOptions.xml file. You can observe this in the –I and –d command line options for the tokenizer that show in the logging output.

The Run command creates two folders in the root of the submission folder:

• TestReports

• TokenizerOutPut

The TestReports folder contains output files from the automation run. Some of these contain intermediate data that is created by and used by automation. Other files are of direct interest to the test engineer, these are:

• Audit.log, described above

• Console.txt, which contains the textual output from the automation run

• Detail.log, the detailed logging output from the automation

• Summary.html, the summary of the pass/fail information from the testing

This folder also contains subfolders that will retain historical information from each automation run, including the date and time of the run in the filename. These will be useful if you are making changes and trying to understand the differences that your changes make to your automated test results.

The TokenizerOutPut folder contains the encoded files that are candidates for registration.

The detail.log file will show all the actions required to set up the tokenizer runs, invoke the tokenizers and will also display their console output.

For each test case, the detail.log file will contain information like the following:

[INFO] - Start: TestPointTFP1211Procedure002 :: TFP1211Procedure002 : TFP1211_002 - Testcase Execution has started... 

[INFO] - Description: TFP1211_002 Verify Commands or Data references (LIST, Value Array, Reference Array or other Variable reference which was previously referenced in a command) are not deleted when comparing to the previous source code revision.   

[INFO] - N/A: TestPointTFP1211Procedure002 :: TFP1211Procedure002 : TFP1211_002 does not apply to campaign type NewDevice. 

[INFO] - End: TFP1211_002  automation  Not Applicable (Skipped)

The information begins by listing the TFP number of the test, followed by the text of the description from the test specification.

Next comes information, error and warning statements produced by the test automation or a tokenizer.

Next is an [INFO] line giving the disposition of the test.

Last is an [INFO] End: statement. It lists the short name for the test case, How the test case was evaluated (automation, tokenizer, manual) and the disposition.

When the run command is complete, a tab will open in a VSCode editor named console.txt. This is a capture of the console output from the automation run.

ddstat report <folder>

This command runs the report function in the <folder> argument if it is present, or the current working directory if not.

It begins by locating a single test report document in the root of the submission folder, searching for files that follow the pattern:

TR*.docx

It is an error for more than one such file to exist.

The report command continues by opening the document, inserting results into the summary tables and subsections for each test case. The information placed in the subsections is drawn from the detail.log file produced by the Run command.

Begin your submission process by opening a new folder in Windows and running the init command in it. This will create initial files and a folder structure for you to populate.

Copy your DD source files, HART Test System log files folders and documents into the submission folder structure as required by the FCG registration kit.

Edit the ConfigurationFiles\DeviceOptions.xml file and place appropriate values for your device there. Replace every occurrence of unknown with a valid value.

Rename the test report to include your device’s identification in the file name. Example:

TR10208-F9F4 0101{2.0.12}_HART-EDD-Test-Report

Open the submission folder in VSCode and run the ddstat audit function. It will diagnose the issues that it finds. Repair them, then repeat this process until you have no errors from the audit function.

Execute the ddstat run function and study the output in the detail.log file, especially the console output from the tokenizer runs. Tokenizer runs with no errors output are a requirement for registration, so you will want to ensure that you have repaired all these before you submit for registration.

Click the Summary button to see an overview of the status of each of the TFPs for your DD registration campaign.

Click the Detail button to review the circumstances around any failures.

Click the Report button to see a draft report. After you submit, a FieldComm Group test engineer will produce the same report and add additional commentary to note any additional effort required before registration can proceed.

Click the Submit button and follow the instructions provided to move your DD registration project to FieldComm Group.

Release 1.0.2

The automation now requires CAL073.qa.log. This log reveals whether command 73 is supported by the device or not.

For device revisions and DD revisions, the automation now requires the registered encoded .fma file from the previous revision. The source for the previous DD is still required, but it is not tokenized.

Testcases 1211_002 and 1211_003 are now automated tests.

The automation is updated to allow IDE versions up to 1.7.0.2.

The 'ddstat init' command is enhanced to allow user to add a folder name as an argument. When no argument is provided, the command behaves as before, initializing in the current working directory. User can add a optional path name as an argument. The automation will create a new folder at that path and initialize it.

A false failure is prevented in the case where a device contains a device-specific command that is not listed in the EDD.

A false failure is prevented in the case where commands are defined in the symbols.txt file but not referenced in the EDD.

Improvements in the automation caused the Sample_HART7 EDD that ships with the product to fail. This is repaired.

Resolved an issue where older HART Test System logs failed to parse.

Resolved an issue where the MANUFACTURER line in some DD's was not recognized.

In test case TFP814_014, it is permitted to REDEFINE TYPE INDEX.

Release 1.0.1

• FCG_TT20019_HART-EDD-Test-Specification

• FCG_AG10101_Using-the-HART-Standard-DD

• EDD Submission document (ref...)