Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

TestRail and TFS integration


#21

Thanks John. I’m yet to look into using plugins/api but am aiming to start working in some upskilling shortly. Is there some documentation/guidance you could share that might help me get started?

Anything that can help TestRail correctly show defects in VSTS within TestRail would be particularly useful, ta :slight_smile:


#22

Hi John, do you have a template or a txt file for this that you would be willing to share? Would be much appreciated! Many thanks


#23

Here is the php file I’m using. There are custom fields referenced in the code so they will not apply to you. How familiar are you with PHP code and the TFS API?

Look this over and I can answer any question you may have.

<?php if (!defined('ROOTPATH')) exit('No direct script access allowed'); ?>
<?php


class TFS_defect_plugin extends Defect_plugin
{
	private $_api;
	
	private $_server;
	private $_collection;
	private $_project;
	
	private static $_meta_defects = array(
		'author' => 'Company Name',
		'version' => '1.0',
		'description' => 'TFS workitem plugin for TestRail',
		'can_push' => true,
		'can_lookup' => true,
		'default_config' =>
			'; Please configure your TFS connection below.
; Production Server  = http://dav1tfsapp01:8080/tfs/
; Development Server = http://dav1uattfsapp01:8080/tfs/
; Collection = FMCSR-SE/
[connection]
server=
collection=
');

	private static $_meta_references = array(
		'author' => 'Company Name',
		'version' => '1.0',
		'description' => 'TFS reference plugin for TestRail',
		'can_push' => false, // Lookup only
		'can_lookup' => true,
		'default_config' => 
			'; Please configure your TFS connection below.
; Production Server  = http://dav1tfsapp01:8080/tfs/
; Development Server = http://dav1uattfsapp01:8080/tfs/
; Collection = FMCSR-SE/
[connection]
server=
collection=
');

	public function get_meta()
	{
		if ($this->get_type() == GI_INTEGRATION_TYPE_REFERENCES)
		{
			return self::$_meta_references;
		}
		else 
		{
			return self::$_meta_defects;
		}
	}
		
	// *********************************************************
	// * CONFIGURATION
	// *********************************************************

	public function validate_config($config)
	{
		$ini = ini::parse($config);
		
		if (!isset($ini['connection']))
		{
			throw new ValidationException('Missing [connection] group');
		}
		
		$keys = array('server', 'collection');
		
		// Check required values for existance
		foreach ($keys as $key)
		{
			if (!isset($ini['connection'][$key]) ||
				!$ini['connection'][$key])
			{
				throw new ValidationException(
					"Missing configuration for key '$key'"
				);
			}
		}

		$server = $ini['connection']['server'];
		$collection = $ini['connection']['collection'];
				
		// Check whether the server is a valid url (syntax only)
		if (!check::url($server))
		{
			throw new ValidationException('Server is not a valid url');
		}

		if (isset($ini['push.fields']))
		{
			// Rules to verify custom fields
			foreach ($ini['push.fields'] as $field => $option)
			{
				if ($option != 'on')
				{
					continue;
				}

				$this->_validate_field($ini, $field);
			}
		}		
	}
	
	public function configure($config)
	{
		$ini = ini::parse($config);
		$this->_server = str::slash($ini['connection']['server']);
		$this->_collection = $ini['connection']['collection'];
	}

// *********************************************************
// * PUSH
// *********************************************************
		
	public function prepare_push($context)
	{
		// Return a form with the following fields/properties
		return array(
			'fields' => array(
				'title' => array(
					'type' => 'string',
					'label' => 'Title',
					'required' => true,
					'size' => 'full'
				),
				'testurl' => array(
					'type' => 'string',
					'label' => 'URL link for TFS Hyperlink',
					'required' => true,
					'size' => 'full'
				),
				'ref_link' => array(
					'type' => 'string',
					'label' => 'PBI link for TFS',
					'required' => false,
					'size' => 'compact'
				),
				'sqa' => array(
					'type' => 'string',
					'label' => 'SQA',
					'required' => true,
					'remember' => true,
					'size' => 'compact'
				),
				'severity' => array(
					'type' => 'dropdown',
					'label' => 'Severity',
					'required' => true,
					'remember' => true,
					'size' => 'compact'
				),
				'iteration' => array(
					'type' => 'dropdown',
					'label' => 'Iteration',
					'required' => true,
					'remember' => true,
					'size' => 'full'
				),
				'builddefs' => array(
					'type' => 'dropdown',
					'label' => 'Build Definitions',
					'required' => true,
					'remember' => true,
					'cascading' => true,
					'size' => 'full'
				),
				'foundinbuild' => array(
					'type' => 'dropdown',
					'label' => 'Found in Build',
					'required' => true,
					'remember' => true,
					'depends_on' => 'builddefs',
					'size' => 'full'
				),
				'toolversion' => array(
					'type' => 'dropdown',
					'label' => 'SadRobot Version',
					'required' => false,
					'remember' => true,
					'size' => 'compact'
				),
				'simversion' => array(
					'type' => 'string',
					'label' => 'Amumu Version',
					'required' => false,
					'remember' => true,
					'size' => 'compact'
				),
				'vehicle' => array(
					'type' => 'multiselect',
					'label' => 'Vehicle',
					'required' => false,
					'remember' => true,
					'size' => 'compact'
				),
				'description' => array(
					'type' => 'text',
					'label' => 'Steps to Reproduce',
					'required' => true,
					'rows' => 15
				)
			)
		);
	}
	
	public function prepare_field($context, $input, $field)
	{
		$api = $this->_get_api();
		$data = array();
				
		// Take into account the preferences of the user, but only
		// for the initial form rendering (not for dynamic loads).
		if ($context['event'] == 'prepare')
		{
			$prefs = arr::get($context, 'preferences');
		}
		else
		{
			$prefs = null;
		}
		
		switch ($field)
		{
			case 'title':
				$data['default'] = $this->_get_title_default($context);
				break;
		
			case 'testurl':
				$data['default'] = $this->_get_current_test_url($context);
				break;
				
			case 'ref_link':
				$data['default'] = $this->_get_current_ref_link($context);
				break;
        
			case 'sqa':
				list($lastname, $firstname) = explode(", ", $context['user']->name);
				$data['default'] = $firstname." ".$lastname;
				break;
				
			case 'severity':
				$data['default'] = arr::get($prefs, 'severity');
				$data['options'] = $this->_get_severity();
				break;
				
			case 'iteration':
				$data['default'] = arr::get($prefs, 'iteration');
				$data['options'] = $this->_to_id_name_lookup($api->get_iterations());
				break;
			
			case 'builddefs':
				$data['default'] = arr::get($prefs, 'builddefs');
				$data['options'] = $this->_to_id_name_lookup($api->get_builddefinitions());
				break;
				
			case 'foundinbuild':
				$data['default'] = arr::get($prefs, 'foundinbuild');
				$data['options'] = $this->_to_id_name_lookup($api->get_foundinbuild($input['builddefs']));
				break;
				
			case 'toolversion': 
				$data['default'] = arr::get($prefs, 'toolversion');
				$data['options'] = $this->_to_id_name_lookup($api->get_sadrobot());
				break;
				
			case 'simversion':
				$data['default'] = arr::get($prefs, 'simversion');
				break;
				
			case 'vehicle':
				$data['default'] = arr::get($prefs, 'vehicle');
				$data['options'] = $this->_get_vehicles();
				break;
				
			case 'description':
				$data['default'] = "
Description:

Repro Steps:
1.
2.

Expected Behavior:

Actual Behavior:

Correlation:
";
				break;
		}		
		
		return $data;
	}
	
	public function validate_push($context, $input)
	{	
		
	}
	
	public function push($context, $input)
	{
		$api = $this->_get_api($context);
				
		return $api->add_defect($input);
	}
	
// *********************************************************
// * LOOKUP
// *********************************************************
	
	public function lookup($workitem_id)
	{
		$api = $this->_get_api();
		
		$workitem = $api->get_workitem($workitem_id);
		
		if ($workitem == Null)
		{
			throw new ValidationException('workitem = null');
		}
				
		// Build the status_id based on the state property
		if ($workitem->fields->{'System.State'} == 'Closed')
		{
			$status_id = GI_DEFECTS_STATUS_CLOSED;
		}
		elseif ($workitem->fields->{'System.State'} == 'Resolved')
		{
			$status_id = GI_DEFECTS_STATUS_RESOLVED;
		}
		else
		{
			$status_id = GI_DEFECTS_STATUS_OPEN;
		}
		
		if (isset($workitem->fields->{'Microsoft.VSTS.Common.StackRank'}))
		{
			$StackRank = $workitem->fields->{'Microsoft.VSTS.Common.StackRank'};
		}
		else
		{
			$StackRank = '';
		}
		
		if (isset($workitem->fields->{'Microsoft.VSTS.TCM.ReproSteps'}))
		{
			$ReproSteps = $workitem->fields->{'Microsoft.VSTS.TCM.ReproSteps'};
		}
		else
		{
			$ReproSteps = '';
		}
		
		if ($workitem->fields->{'System.WorkItemType'} == 'Bug')
		{
			return array(
				// * required properties
				'id' => $workitem->id,
				'title' => $workitem->fields->{'System.Title'},
				'status_id' => $status_id,
				'status' => $workitem->fields->{'System.State'},
			
				// * optional properties
				//'url' => <the full url to view the workitem>, 
				'description' => $ReproSteps, 
				'attributes' => array(
					'Type' => $workitem->fields->{'System.WorkItemType'},
					'Status' => $workitem->fields->{'System.State'},					
					'Created By' => $workitem->fields->{'System.CreatedBy'},
					'SQA' => $workitem->fields->{'Schilling.VSTS.Bug.SQA'},
					'Stack Rank' => $StackRank, 
					'Severity' => $workitem->fields->{'Microsoft.VSTS.Common.Severity'},
					'Iteration' => $workitem->fields->{'System.IterationPath'}
					)
				);
		}
		else // the worktiem is a Product Backlog Item
		{
			return array(
				// * required properties
				'id' => $workitem->id,
				'title' => $workitem->fields->{'System.Title'},
				'status_id' => $status_id,
				'status' => $workitem->fields->{'System.State'},
			
				// * optional properties
				//'url' => <the full url to view the workitem>, 
				'description' => $workitem->fields->{'Microsoft.VSTS.Common.DescriptionHtml'}, 
				'attributes' => array(
					'Type' => $workitem->fields->{'System.WorkItemType'},
					'Status' => $workitem->fields->{'System.State'},					
					'Created By' => $workitem->fields->{'System.CreatedBy'}
					)
				);
		}
	}
	
	private function _to_id_name_lookup($items)
	{
		$result = array();
		asort($items);
		foreach ($items as $item)
		{
			$result[$item->id] = $item->name;
		}
		return $result;
	}
	
	// *********************************************************
	// * Get title for the workitem
	// *********************************************************
	private function _get_title_default($context)
	{
		$test = current($context['tests']);
		$title = $test->case->title;
		
		if ($context['test_count'] > 1)
		{
			$title .= ' (+others)';
		}
		
		return $title;
	}
  
	// *********************************************************
	// * Get the URL of the current test. 
	// * This is used for the hyperlink in TFS
	// *********************************************************
	private function _get_current_test_url($context)
	{
		$test = current($context['tests']);
		$testurl = $test->url;
		
		return $testurl;
	}
	
	private function _get_current_ref_link($context)
	{
		$test = current($context['tests']);
		$ref = $test->case->refs;
		
		return $ref;
	}
	
	private function _get_severity()
	{
		return array(
			'1 - Critical' => '1 - Critical',
			'2 - High' => '2 - High',
			'3 - Medium' => '3 - Medium',
			'4 - Low' => '4 - Low'
		);
	}	
		
	private function _get_vehicles()
	{
		return array(
			'Bullpen' => 'Bullpen',
			'UHDII' => 'UHDII',
			'UHDIII' => 'UHDIII',
			'HD' => 'HD',
			'GEMINI' => 'GEMINI'
		);
	}
	
	// *********************************************************
	// * API / CONNECTION
	// *********************************************************
	private function _get_api()
	{
		if ($this->_api)
		{
			return $this->_api;
		}
		
		$this->_api = new TFS_api($this->_server, $this->_collection);
		
		return $this->_api;
	}
}

// *******************************************************
// * API connection to TFS
// * If you need data from TFS, this is where you put it
// *******************************************************
class TFS_api
{	
	private $_path;
	private $_curl;

	private $_createmeta = array();


    // ***************************************************
    // * Constructor
    // ***************************************************
    public function __construct($server, $collection)
	{
	    $this->_path = $server.$collection;
    }

	
	// ************************************************************************
	// ** Get the build definition from TFS
	// ************************************************************************
	public function get_builddefinitions()
	{
		$restUrl = $this->_path.'Schilling.Robotics.Framework/_apis/build/definitions';
		
		$response = $this->getRestClient($restUrl);
		
		$builddefs = json_decode(curl_exec($response));
		
		$result = array();
		
		foreach ($builddefs->{'value'} as $def)
		{
			$bd = obj::create();
			
			$bd->id = $def->{'name'};
			$bd->name = $def->{'name'};
			
			$result[] = $bd;
		}
		
		return $result;
	}
	
	// ************************************************************************
	// ** Get the builds associated with the selected definition from TFS
	// ************************************************************************
	public function get_foundinbuild($builddef)
	{
		$restUrl = $this->_path.'Schilling.Robotics.Framework/_apis/build/builds?definition='.$builddef;
		
		$response = $this->getRestClient($restUrl);
		
		$builds = json_decode(curl_exec($response));
		
		$result = array();
		
		foreach ($builds->{'value'} as $build)
		{
			$b = obj::create();
			
			$b->id = $build->{'buildNumber'};
			$b->name = $build->{'buildNumber'};
			
			$result[] = $b;
		}
		
		return $result;
	}
	
	// ************************************************************************
	// ** Get the builds associated with SadRobot from TFS
	// ************************************************************************
	public function get_sadrobot()
	{
		$restUrl = $this->_path.'Schilling.Robotics.Framework/_apis/build/builds?definition=SadRobot';
		
		$response = $this->getRestClient($restUrl);
		
		$builds = json_decode(curl_exec($response));
		
		$result = array();
		
		foreach ($builds->{'value'} as $build)
		{
			$b = obj::create();
			
			$b->id = $build->{'buildNumber'};
			$b->name = $build->{'buildNumber'};
			
			$result[] = $b;
		}
		
		return $result;
	}
	
	// ************************************************************************
	// ** Get the iterations from TFS
	// ************************************************************************
	public function get_iterations()
	{
		$restUrl = $this->_path.'Schilling.Robotics.Framework/_apis/wit/classificationNodes/iterations?$depth=5';
		
		$response = $this->getRestClient($restUrl);
		
		$iterations = json_decode(curl_exec($response));
		
		$result = array();
		
		$p = obj::create();
		$p->id = $iterations->{'name'};
		$p->name = $iterations->{'name'};
		$result[] = $p;
		
		$this->_build_iteration_tree($result, $iterations->{'children'}, $p->name, 0);
		
		return $result;
	}
	
	// ************************************************************************
	// ** Recursively get the iterations and its children
	// ************************************************************************
	private function _build_iteration_tree(&$result, $iteration, $parent, $level)
	{
		foreach ($iteration as $iter)
		{
			$p = obj::create();
			
			$p->id = $parent.'\\'.'\\'.$iter->{'name'};
			$p->name = $parent.'\\'.'\\'.$iter->{'name'};
				
			$result[] = $p;
								
			if ($iter->{'hasChildren'})
			{
				$this->_build_iteration_tree($result, $iter->children, $p->name, $level + 1);
			}
		}
	}
		
	// ***************************************************
    // * Get work item by ID
    // ***************************************************
    public function get_workitem($id)
	{
        $restUrl = $this->_path.'_apis/wit/workitems/'.$id.'?$expand=relations';
				
        $ch = $this->getRestClient($restUrl);
		
        $data = curl_exec($ch);
		
		return $workitem = json_decode($data);
    }
	
	// ***************************************************
	// * Add a new defect
	// ***************************************************
	public function add_defect($options)
	{
		$data = $options;
		
		if (isset($options['title']))
		{
			$data['title'] = $options['title'];
		}
		
		if (isset($options['sqa']))
		{
			$data['sqa'] = $options['sqa'];
		}
    
		if (isset($options['description']))
		{
			$search = array('Description:', 'Repro Steps:', 'Expected Behavior:', 'Actual Behavior:', 'Correlation:');
			$replace = array('<strong><u>Description:</u></strong>',
							 '<strong><u>Repro Steps:</u></strong>',
							 '<strong><u>Expected Behavior:</u></strong>',
							 '<strong><u>Actual Behavior:</u></strong>',
							 '<strong><u>Correlation:</u></strong>');
			$newdesc = str_replace($search,$replace,$options['description']);
			$data['description'] = nl2br($newdesc);
		}
		
		if (isset($options['testurl']))
		{
			$data['testurl'] = $options['testurl'];
		}
		
		if (isset($options['ref_link']))
		{
			$data['ref_link'] = $options['ref_link'];
		}
		
		if (isset($options['severity']))
		{
			$data['severity'] = $options['severity'];
		}
		
		if (isset($options['iteration']))
		{
			$data['iteration'] = $options['iteration'];
		}
		
		if (isset($options['builddefs']))
		{
			$data['builddefs'] = $options['builddefs'];
		}
		
		if (isset($options['foundinbuild']))
		{
			$data['foundinbuild'] = $options['foundinbuild'];
		}
		
		if (isset($options['toolversion']))
		{
			$data['toolversion'] = $options['toolversion'];
		}
		
		if (isset($options['simversion']))
		{
			$data['simversion'] = $options['simversion'];
		}
		
		if (isset($options['vehicle']))
		{
			$data['vehicle'] = $options['vehicle'];
		}
		
		
		$restUrl = $this->_path.'Schilling.Robotics.Framework/_apis/wit/workitems/$bug';
				
		$response = json_decode($this->postRestClient($restUrl, $data));
		
		//throw new ValidationException($response->id);
		
		$id = $response->id;
		
		if (!isset($id))
		{
			$this->_throw_error('No workitem ID received');
		}

		return (string) $id;
	}

	
	
    // ***************************************************
    // * Get an initialized REST request using integrated security.
    // *
    // * @param mixed $url REST URL
    // * @return resource
    // ***************************************************
    private function getRestClient($restUrl)
	{
		//throw new ValidationException($restUrl.' in getRestClient');
        $headers = array('Accept: application/json;api-version=1.0');
        $curl = curl_init();
        curl_setopt($curl, CURLOPT_URL, $restUrl);
        curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);
        curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
        curl_setopt($curl, CURLOPT_HTTPAUTH, CURLAUTH_NTLM);
        curl_setopt($curl, CURLOPT_UNRESTRICTED_AUTH, true);
        curl_setopt($curl, CURLOPT_USERPWD, ":");
				
        return $curl;
    }
	
	private function postRestClient($restUrl, $input)
	{
		// These are the required fields
		// 		Title
		// 		SQA
		// 		ReproSteps (Steps to Reproduce)
		// 		Hyperlink (link back to the test case)
		// 		Severity
		// 		Iteration Path
		// 		Found in Build
		$data =
			'[
				{
					"op": "add",
					"path": "/fields/System.Title",
					"value": "'.$input['title'].'"
				},
				{
					"op": "add",
					"path": "/fields/Schilling.VSTS.Bug.SQA",
					"value": "'.$input['sqa'].'"
				},
				{
					"op": "add",
					"path": "/fields/Microsoft.VSTS.TCM.ReproSteps",
					"value": "'.$input['description'].'"
				},
				{
					"op": "add",
					"path": "/relations/-",
					"value": {
						"rel": "Hyperlink",
						"url": "'.$input['testurl'].'"
					}
				},
				{
					"op": "add",
					"path": "/fields/Microsoft.VSTS.Common.Severity",
					"value": "'.$input['severity'].'"
				},
				{
					"op": "add",
					"path": "/fields/System.IterationPath",
					"value": "'.$input['iteration'].'"
				},
				{
					"op": "add",
					"path": "/fields/Microsoft.VSTS.Build.FoundIn",
					"value": "'.$input['builddefs'].'/'.$input['foundinbuild'].'"
				}
			';
	
		// If the test case has a reference to a TFS PBI it will appear on the plugin dialog
		// automatically. If there is a number on the plugin dialog add it to the push
		if ($input['ref_link'] != null)
		{
			$data .= 
			',
				{
					"op": "add",
					"path": "/relations/-",
					"value": {
						"rel": "System.LinkTypes.Hierarchy-Reverse",
						"url": "'.$this->_path.'_apis/wit/workItems/'.$input['ref_link'].'"
					}
				}
			';
		};
					//$data['description'] = nl2br(html::link_urls(h($options['description'])));

		$build_system_info = "";
		// If the defect has a tool version then add it to the system info
		if ($input['toolversion'] != null)
		{
			$build_system_info .= '<strong>SadRobot Version: </strong>'.$input['toolversion'].'
			';
		};
		
		// If the defect has a simulator version then add it to the system info
		if ($input['simversion'] != null)
		{
			$build_system_info .= '<strong>Amumu Version: </strong>'.$input['simversion'].'
			';
		};
		
		// If the defect has a vehicle then add it to the system info
		if ($input['vehicle'] != null)
		{
			$build_system_info .= '<strong>Vehicle: </strong>';
			foreach ($input['vehicle'] as $vehicle)
			{
				$build_system_info .= $vehicle.' ';
			}
		};
		
		$system_info = nl2br($build_system_info);
				
		$data .=
			',
				{
					"op": "add",
					"path": "/fields/Microsoft.VSTS.TCM.SystemInfo",
					"value": "'.$system_info.'"
				}
			';
		
		$data .= ']';
		
		// Define PATCH content type and JSON result type.
		$headers = array(
			'Content-Type: application/json-patch+json',
			'Accept: application/json;api-version=1.0'
			);

		
		$curl = curl_init();
		curl_setopt($curl, CURLOPT_URL, $restUrl);
		curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);
		// Use HTTP PATCH verb
		curl_setopt($curl, CURLOPT_CUSTOMREQUEST, 'PATCH');
		curl_setopt($curl, CURLOPT_POSTFIELDS, $data);
		// Return a string result to the return variable
		curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
		// Use Windows integrated security
		curl_setopt($curl, CURLOPT_HTTPAUTH, CURLAUTH_NTLM);
		curl_setopt($curl, CURLOPT_UNRESTRICTED_AUTH, true);
		curl_setopt($curl, CURLOPT_USERPWD, ":");
		
		// Get the response and close the resource.
		$response = curl_exec($curl);
		
		curl_close($curl);
		
		return $response;
    }
}

#24

That’s fabulous John, thanks! I’ll look into this with our solution very soon. We’re invested into Azure DevOps (VSTS renamed) so the links will be very different but I have a great dev team to help with that side of things.

This’ll make the TestRail/VSTS link much nicer. Ta


#25

I’m assuming you are using the on prem version not the cloud :frowning:

I’m trying to figure out how to use Azure DevOps with cloud based TRail… there’s quite a bit of manual work required to keep the two in sync


#26

I’m doing the same - the above example doesn’t match up with that setup, so I’m hoping to get some of our developers to look into it.

The (minimal) supplied integration works OK but I’d love to get a Jira-like integration going so that I can create bugs within TestRail and see bug titles in my reports etc