Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
The requirements for submitting an DD for registration are described in FieldComm Group support.
The DD Standard Test Automation Tool (DD-STAT) is a comprehensive test automation tool which assists members of FieldComm Group with the process of registering HART DD projects.
Collection It assists users to correctly assemble required source code and documentation for the submission.
Evaluation It examines DD source code and tokenizer outputs, comparing them to the requirements of the DD test specification.
Reporting It places the results of the examination into a test report template simplifying the reporting process.
Submission It bundles the information that is required for registration and copies it to FieldComm Group servers.
Automated tests evaluate DD source code and other artifacts
Tokenizer tests inspect specific outputs from the tokenizers
Manual tests rely on a test engineer to inspect the code base and provide a conclusion
The tool is configured as an extension to the Visual Studio Code ( VSCode ) editor in two pieces:
A command line interface that could be run from a Windows terminal or in a DevOps pipeline.
A user interface window that runs inside the VSCode IDE, which eases user interaction with the tool.
FCG_TT20019_HART-EDD-Test-Specification
FCG_AG10101_Using-the-HART-Standard-DD
EDD Submission document (ref...)
FDI IDE installation guide (http://go.fieldcommgroup.org/fdi-ide-documentation)
ID | Title |
---|---|
1794
TP811_004 - false failure, delete is permitted
1904
TFP711_005 in 11310201 DD needs to be marked “skipped” or “passed”, not inconclusive
1908
TFP715_002 false pass - no AG10101 macros are referenced
1917
new IDE 1.7.0 macro parameter for IMPORT_PV2_3_BASE is not supported
1938
TFP811_004 - Universal is redefined and common practice is not, requires analysis
1940
Fatal while getting command 9 byte count from UAL011a
2043
TFP813_007 False pass - Redefinition macro may only be used when the command 48 bytes do not match
2044
TFP813_002 - wrong macro name is identified
2045
TFP813_008 items shown are not actually VARIABLEs with definitions in the Universal IMPORT
2046
TFP712_001 Remove check for commands 524 and 525 from IMPORT_TABLES_CONDENSED_STATUS and move them to IMPORT_CONDENSED_STATUS_MAP_WRITE
2052
TFP813_007 - move command 48 output from TFP1011_002 to TFP813_007
2053
TFP712_001 -command 48 parameter issue is not detected
2058
Some DD searches should be done on .i not .ddl
2107
TFP813_007 error message is not clear
2111
TFP621_005 device specific command number range issue
2112
TFP911_001 and 002 - not showing passed when errors Tok 859/860 are not present
2120
TFP814_016 review/rework
2126
Using defines as arguments for macro parameters instead of numbers should be a failure
2141
HART Sample DDs
2162
TFP715_011 Commands 72, 73, 74 and 75 having nothing to do with IMPORT_MISC_LOCK
2163
TFP814_023: False Errors
2164
TFP814_013 failure assigned to the wrong TFP
2165
TFP814_014 has no macro
2181
TFP1111_008 - switch from manual to automation
2196
TFP813_008 - false failure - should not fail as long as they are not REDEFINING the CLASS or HANDLING
2198
TFP511_005 - false failure - need to use .i file
2199
TFP811_004 false failure - The reason for the failure is missing for TFP811_004 in the report
2201
TFP621_009 - should be tokenizer not manual
2202
TFP612_004 false pass - report fails to capture tok error 694
2260
TFP715_005 false failure - should be marked “Passed” with a INFO(warning) in the details
2261
TFP1211_001 revise
2264
TFP715_005, 007, 009, 010, 011 and consider adding 001 and 013
2265
TFP715_002, 006, 008, 012 - revise
Getting Started
Quickstart
Documentation
Learn the basics
Help & FAQ
Answer your questions
The VS Code extension provide a graphic user interface to the DD-STAT application.
When the DD-STAT extension is installed in VSCode, a small user interface panel will be visible whenever the file explorer is showing in the left-hand pane. A small drop-down button (green arrow) enables you to open or collapse this panel.
The current plugin does not provide a progress bar when running a test campaign. Most test runs will finish within 60 seconds. A console log will be displayed when the test is completed.
The Init, Audit, Run, and Report buttons on the panel invoke the corresponding command line interface functions. See the command line interface section for each of these for a description of its operation.
The Summary button causes the summary.html output from the automation to display in a window with easy access to individual test results.
The Detail button causes the detail.log output from the automation to display in an editor window. See the description of logging outputs in the section describing console operation for ddstat run.
The Submit button will open an editor window with instructions for how to submit your DD for registration to FieldComm Group. Automated submission functionality for this button is under consideration for the first production release of DD-STAT.
Area | Contents |
---|---|
Outcome | Meaning |
---|---|
Left
Cards representing Test Failure Points (TFP) and the outcome. The exclamation point (!) button above the cards allows you to filter for Pass, Fail, Skip and Warning outcomes.
Top Center
Identity information for the EDD.
Top Right
Time and date of the run.
Right
Detail area. Click on the TFP name to see reasoning behind the outcome.
Pass
Meets the test specification
Fail
Does not meet the test specification
Skip
Test specification does not apply to this EDD
Warning
The automation cannot determine the outcome. This applies to the TFPs that are determined manually. It also applies when insufficient or incorrect data is supplied.
The Submission Folder is central to the operation of the DD-STAT application. Device registration with FCG requires extensive information including:
Source code for the DD
Source code for the prior version of this DD (revisions only)
Log files from the HART Test System (HTS)
Other documents
The specifics of this are covered in the FieldComm Group guidance document ____________________.
It is necessary for the information to be provided in a uniform way so that automated evaluation of it is possible. The contents of this folder are shown below:
The DD-STAT automation operates from the root of this folder structure. Prior to every operation it examines its current working directory and will terminate if it is not running from a properly structured submission folder.
There are three files in the structure specific to the automation, not covered in the guidance document mentioned above. You must configure these files as described below for the automation to run. These are:
Test report document
DeviceOptions.xml
Filepaths.config
The automation ships with a test report template:
TR10208-ZZZZ{2.0.11}_HART-EDD-Test-Report.docx
This template contains document sections, summary tables and boilerplate text consistent with the specification.
Note that the automation is sensitive to the format and content of this template. Especially during the beta period, this template is subject to change. The version number of the document is checked against the current version expected by the automation. A warning is produced if the version of the document is earlier than expected.
The DD-STAT automation interoperates with the test engineer to produce a full text report in the following ways:
Boilerplate text and document structure in the template follow the requirements of the test specification.
The automation will insert tokenizer outputs and other relevant automation output directly into the report document in specific sub-sections.
Test engineers can insert explanations, and screen captures to support the test report.
The automation can be run multiple times, with its output inserted into the report again and again without overwriting the information input directly by the test engineer.
To accomplish this, the test report template includes a custom Word document style called “TFPNotes”. Any text in the document that has this style will be overwritten by the automation when the report function runs. The style is characterized by a black line box around the text.
The automation will also insert Pass/Fail/Skipped notations into the test summary table in each section of the document.
Any text in the second column of these tables may be overwritten by automation. However, the result of manual test cases is not overridden. It is intended that the test engineer fill out these values.
ConfigurationFiles\DeviceOptions.xml
This file contains information important to the test campaign that is not easily or reliably obtained directly from the DD source code.
The beta release does not include User Interface to edit this information, it is edited manually by the user.
Each editable field has instructions and comments around it that will guide you to place the correct value there.
General instructions are:
The fields initially contain the value ‘unknown’ or an invalid value. The automation will flag these as errors until you fill them with valid values.
True/false parameters start with ‘Is’: ‘IsSupportedInDevice’ for example. Replace ‘unknown’ with a value of ’true’ or ‘false’ with no capitalization.
The numeric fields are integers.
If the field contains an enumerated value, the allowable values will be provided in a comment.
ConfigurationFiles\Filepaths.config
This file provides optional locations for the various parts of the submission. The values provided in the supplied file fit the requirements for the submission folder structure.
Do not edit this file.
Begin your submission process by opening a new folder in Windows and running the init command in it. This will create initial files and a folder structure and for you to populate.
Copy your DD source files, HART Test System log files folders and documents into the submission folder structure as required by the FCG registration kit.
Edit the ConfigurationFiles\DeviceOptions.xml file and place appropriate values for your device there.
Rename the test report to include your device’s identification in the file name. Example:
Open the submission folder in VSCode and run the ddstat audit function. It will diagnose the issues that it finds. Repair them, then repeat this process until you have no errors from the audit function.
Execute the ddstat run function and study the output in the detail.log file, especially the console output from the tokenizer runs. Tokenizer runs with no errors output are a requirement for registration, so you will want to ensure that you have repaired all these before you submit for registration.
Click the Summary button to see an overview of the status of each of the TFPs for your DD registration campaign.
Click the Detail button to review the circumstances around any failures that you find.
Click the Report button to see a draft report. After you submit, a FieldComm Group test engineer will produce the same report and add additional commentary to note any needed remedial effort required before registration can proceed.
Click the Submit button and follow the instructions provided to move your DD registration project to FieldComm Group.
You operate the automation using a command line interface (CLI) program named ddstat.exe. To use it, open a Windows terminal in the root of a submission folder.
You may use the Windows terminal that is integrated into VSCode. To do that, from the VSCode menu at the top, select:
Terminal > New Terminal
and a new Windows command terminal will open at the bottom of the VSCode window.
The general form of command lines are:
ddstat <command> <folder>
There is no hyphen in ‘ddstat’.
The first argument is the command, one of:
init
audit
run
report
The actions for these commands are described in sections below.
If the <folder> argument is not provided, then the automation will expect to find a submission folder configuration in the current working directory.
If the <folder> argument is provided, then the automation will execute from that folder.
If the named folder (or current working directory if there is none) is not a submission folder, the automation will look through all the subfolders and run the automation in each submission folder it finds. You can batch the operation in this manner.
This prints the version number of the test tool to the console output.
This prints brief instructions for using the CLI.
To start a new submission project, create a new folder and open a Windows terminal in it. From the command prompt type:
ddstat init
This will create a submission folder, with the required folders and any initial files that can be supplied by FieldComm Group.
Do not run the init command in a submissions folder that has already been populated.
This command examines the contents of the <folder> argument if supplied or the current working directory if not. It prints errors and warnings encountered to the console output.
Additionally, it creates a file named ‘audit.log’ in the TestReports folder that contains the result of the audit using a log level of ‘info’. It documents which items in the submission folder are missing or contain invalid data and info lines with contextual information.
Audit.log helps the test engineer to correctly assemble the required configuration, HART Test System log files, and EDD source code required to operate the test.
The output from the console is errors and warnings only, so sometimes context is missing from that information. If you have questions about the errors or warnings, open the audit.log file which contains [info] output, providing the context needed to understand the diagnostics.
This command runs the test automation in the <folder> argument if it is present, or the current working directory if not. It begins by running the audit command and will terminate the execution if any errors are reported during the audit.
This command will make at least two tokenizer runs, one for tokenizer version 8 and one for version 10. For campaigns of type of DDRevision or DeviceRevision, an additional run will be made using the historical DD source code that you provided.
The tokenizer command line will source the header and other Standard Library files appropriately based upon the IDERevision number and the CmnPrac revision number settings that you supplied in your DeviceOptions.xml file. You can observe this in the –I and –d command line options for the tokenizer that show in the logging output.
The Run command creates two folders in the root of the submission folder:
TestReports
TokenizerOutPut
The TestReports folder contains output files from the automation run. Some of these contain intermediate data that is created by and used by automation. Other files are of direct interest to the test engineer, these are:
Audit.log, described above
Console.txt, which contains the textual output from the automation run
Detail.log, the detailed logging output from the automation
Summary.html, the summary of the pass/fail information from the testing
This folder also contains subfolders that will retain historical information from each automation run, including the date and time of the run in the filename. These will be useful if you are making changes and trying to understand the differences that your changes make to your automated test results.
The TokenizerOutPut folder contains the encoded files that are candidates for registration.
The detail.log file will show all the actions required to set up the tokenizer runs, invoke the tokenizers and will also display their console output.
For each test case, the detail.log file will contain information like the following:
The information begins by listing the TFP number of the test, followed by the text of the description from the test specification.
Next comes information, error and warning statements produced by the test automation or a tokenizer.
Next is an [INFO] line giving the disposition of the test.
Last is an [INFO] End: statement. It lists the short name for the test case, How the test case was evaluated (automation, tokenizer, manual) and the disposition.
When the run command is complete, a tab will open in a VSCode editor named console.txt. This is a capture of the console output from the automation run.
There is no progress indication during the run. Because of the tokenizer runs that are included, it can take a minute or two for this to complete. This is a known deficiency. For now, wait for the console.txt tab to open to indicate completion.
This command runs the report function in the <folder> argument if it is present, or the current working directory if not.
It begins by locating a single test report document in the root of the submission folder, searching for files that follow the pattern:
TR*.docx
It is an error for more than one such file to exist.
The report command continues by opening the document, inserting results into the summary tables and subsections for each test case. The information placed in the subsections is drawn from the detail.log file produced by the Run command.
If you have the test report file open in Word while running the report command, the report command will not be able to update the report document and will terminate with an error.
ID | Title |
---|
Outcome | Meaning |
---|
1852 | TFP714_006 explore automating this case |
1855 | TFP816_001 and _002 - explore automating this case |
1971 | TFP1011_003 - format change - consolidate items into a list |
2135 | Interactive DeviceOptions.xml |
2136 | Use doc properties to compare test report version |
2143 | Show a progress indicator while cli is running |
2222 | Create pipeline installer |
2227 | Make UI buttons interoperate with the terminal shell |
2232 | Sample_HART7 address issues found by Bob W |
2240 | Add ability to bypass the initial Audit check |
2242 | Update ExtentReports nuget |
2247 | Revise dll024b+c processing to handle earlier file format |
2253 | spaces in file names breaks automation |
2262 | TFP1211_002 - revise |
2263 | TFP1211_003 - revise |
2267 | Automation is not finding command 73 responses in test system log files |
2309 | TFP1111_003: Test Report is missing the explanation on the reason for the failure |
2310 | TFP1011_001: Command list in log and report is listed twice. |
2315 | DeviceOptions: IsSupportedInDevice and NumberOfXXX |
2316 | add folder parameter to cli init command |
2301 | Foldername "EDD FDI" usage of space creates issues |
2290 | TFP715_012 Test Report is missing the explanation on the reason for the failure |
2291 | TFP713_001 wrong reason for failure is listed |
2287 | TFP814_021: E5E8 0101: Individual redefinition different than AG10101 allows. |
[INFO] - Pass | Meets the test specification |
[ERROR] - Fail | Does not meet the test specification |
[INFO] - N/A | Does not apply to this EDD |
[WARN] - Inconclusive | The automation cannot determine the outcome. This applies to the TFPs that are determined manually. It also applies when insufficient or incorrect data is supplied. |
Windows 10 or 11
The current release of FDI IDE (installed separately)
Visual Studio code editor (free)
Some third party VSCode extensions (free, see below)
DD-STAT requires that the following items be installed:
VSCode
Required extensions for VSCode (see below)
FDI IDE (see installation guide)
The DD-STAT application and the DD-STAT VSCode extension
Download and install your VSCode program using the link below. It supplies instructions for getting started using the IDE.
DD-STAT application uses two third-party extensions for VSCode to enhance user experience. These are:
Microsoft live preview extension for viewing HTML code.
EDDL extension for syntax highlighting of HART of EDDL source code.
You can install these extensions by following the guidance here.
During the beta, the DD-STAT components will be available on a SharePoint server in a ZIP file.
Double-click the installer ddstat.msi file to install the application.
This will install a DDSTAT folder in the C:\HCF folder alongside the HART library and other HART applications.
The current beta installer may not display a dialog confirming the software is installed. You can verify the software is installed by going to the Start Menu - All App - FieldComm Group where you will find a link to the documentation. You can also verify that C:\HCF\DDSTAT is installed.
DD-STAT provides a new FieldComm Group license manager, which will provide unified license management for FieldComm Group and FDI products. This license manager is improved to allow the user to install either via a license key or a license file.
Internet access is required to activate DD-STAT. This feature allows licenses to be automatically updated as part of a license renewal.
Launch the manager from the start menu. Search for FieldComm Group folder and choose the license manager. You will see a pop-up asking which style of license you wish to provide. DD-STAT users are issued a serial number for a license. In the drop-down select ' Enter a serial number' and paste your provided serial number into the UI.
DD-STAT will be available in the VS Code Marketplace at a later date. Until then, please install manually using the provided file.
Install the ddstat.vsix file as follows:
In VSCode, go to the Extensions View by clicking on the Extensions icon on the left-hand sidebar (the one that looks like four squares).
In the Extensions panel, click the More Actions button (three dots) in the top-right corner. From the dropdown, choose Install from VSIX....
A file dialog will open. Navigate to the location where you downloaded the .vsix file, select it, and click Open.
VSCode will install the extension, and once it's done, you’ll see it appear in your list of installed extensions. You might be prompted to reload VSCode for the extension to activate, so just click Reload if that happens.
After installation, this extension may recommend you install 2 additional extension from the VS Code Marketplace. These extensions provide a means to display html test reports and provide basic highlighting for EDD source code file.
File | Description |
---|---|
ddstat.msi
Installer for command line automation components
ddstat.vsix
Visual Studio code extension component
EULA
End user license agreement
Sample_HART7
The standard HART sample configured in an EDD submission folder.