Join 34,000+ subscribers and receive articles from our
blog about software quality, testing, QA and security.

Reporting feature requests


#1

Hey guys,

TestRail, for the most part, is a testing dream come true for me! But, one of the most important parts of testing is letting product managers and other business people know the results of test passes. Of course, this is where reporting comes in.

Unfortunately, the reporting built-in to TestRail, though comprehensive, is nigh unusable. The reason lies within the available methods of delivery. I’ll go over where the problems lie:

  • Email a link to the report.
  1. If we’re going to bother sending an e-mail about a test report, the e-mail should contain the content of the test report, rather than just a link to view the report in Test Rail. This would be a minor problem, but for:
  2. In order to view a TestRail report, one must have a TestRail account. We don’t want to use up TestRail licenses just so a VP can glance at a report once in a while. It would be great if there were a way to either make free, view-report-only accounts, or to make reports generally public.
  • Email the report as an attachment.
    The attachment, as you know, comes in the form of a zip archive of a web page. This presents several problems.
  1. Security. Most people (hopefully!) are fairly guarded when they receive an e-mail with an attachment – there’s always the concern that “Your test result is attached” is social engineering. This could be overcome with some education of the intended audience, but the tool should accommodate the audience – not vice-versa.
  2. Anti-virus. My corporation’s e-mail server strips the javascript files out of the attachments, since .js files in an e-mail could be malicious scripts. When opening the webpage, this makes the pie charts, among other things, disappear. We also get e-mails from the e-mail server that the .js files were stripped from that e-mail. This isn’t a good experience for our intended audience, which could go as high as VPs.
  3. Getting a non-technical business person to unzip a file and open index.html. As above, busy folks don’t want to take the time to pursue a test report. They certainly don’t want to find where the report unzipped to, and look through some obscure-sounding filenames for index.html.

Ultimately, we need reports in the body of e-mail messages. I understand that this is not currently done because the Outlook HTML rendering engine is based on Word, and the reports don’t show up with the active content of the website reports, and also some goofy rendering. At a minimum, I’d like even a very simplified report, using basic tables and some nice fonts, where possible. The active content, while nice, is not necessary for viewing in an e-mail.

I will probably end up writing my own reporter to accomplish this, though I’d greatly prefer a turnkey solution.

Your work is otherwise amazing! This is a fantastic tool, and I pushed hard on execs up to the VP level to get its purchase approved for the company.


#2

Hello Adam,

Thanks for your posting and great to hear that you enjoy working with TestRail and like the product! Your feedback is really appreciated.

Regarding the reports:

Most emails clients only have a very limited HTML engine and it’s very difficult to display (advanced) reports that look the same/similar in all commonly used email clients. It would also be difficult to display charts and graphs. We felt that the ZIP/HTML option would be the best option all things considered, as this format is the only way to provide the same look & feel as the reports directly displayed in TestRail. We recommend adding the configured sender address of TestRail to an email white list so that these email’s pass possible email filters.

The reports, notifications and the ZIP format is an ongoing topic and it’s definitely planned to look into providing additional/other options in the future.

Regards,
Tobias


#3

Hi Tobias,

Thanks for the reply! Like I mentioned, the directly-emailed report doesn’t have to be as pretty as the website. Even a simple table of test results would be fine.


#4

Hi Team,

I strongly agree with Adam’s comments. We really needs more robust reports as management wants to review the reports in smartphone or from computer without unzipping the files. Our mail server blocks zip files with bigger size and zip files are not recommended as per our IT team. As we are getting the reports in zip format we are unable to view the reports in smartphones. Link option is not a good one as the user needs test rail ID. Management team like VP and Director’s don’t need access to Test rail to view reports. They just need reports to view daily in email. Can you guys please take a look into this feature on high priority and provide a feature to view the report from email without clicking the link or unzipping the report. Need a feature to view report in smartphones and email.


#5

My 5 cents here (since I used to work on a reporting solution). Honestly, I haven’t used TestRail reports not even once, so I have no idea how interactive those HTMLs in attachments are, but maybe implementing PDF output would be a sensible approach? and therefore attaching reports to emails as PDF? I’m sure there must be php/c libraries for HTML to PDF conversion — fast googling suggests wkhtmltopdf, for example (free and fast, webkit-based, although LGPL, which you might not favor)


#6

I completely agree with all the points adamtews has raised. I have bought this up via the support channel in the past, but hopefully with a bit more weight from other users, this will be addressed.

I’d be happy with a html table or a .png file of the graph; anything that means my exec team don’t have to deal with .zip attachments or logging into TestRail (which isn’t practical in a multi-national, multi-business environment like ours.)


#7

We’re considering TestRail as our test management tool; haven’t tried the tool out but it looks very promising.

Anyhow, agreed with the points the OP and others made here. It’d be great to have flexible reporting output option, ie. let users choose between a few methods and formats to suite different organizational needs. Portable PDF/PNG formats are good choice, among other delivery methods.

I’m a tech-oriented person, and have to say that I’d not like to unzip attachments to see the actual reports :slight_smile:

Cheers,
Vincent.


#8

Hello all,

Thanks a lot for all the feedback, this is really useful! As you know, most applications like JIRA etc. wouldn’t allow you to view reports without having a login so we added the report forwarding feature to make it easier to share TestRail’s rich reports and statistics with other persons without having access to TestRail. We understand that this also means that this likely reduces the number of licenses/subscriptions teams use (and this is likely the reason many tools don’t support this), but we agree that this is very useful to have so we added the support for this.

From a technical perspective it would be very difficult to render TestRail’s rich reports in a PDF file. Solutions that would convert TestRail’s full HTML reports to a PDF file would mean that we would basically need to embed and ship a full web browser on TestRail’s server, which would be impractical. Another option would be to offer some more basic/feature-reduced PDF versions for sharing, but the issue with this is that most customers would expect the PDF files to include the same rich charts and graphics our regular reports use.

So for the best user experience it’s likely best to invite users directly to TestRail if they need to review reports regularly, as they can just open the reports by clicking on the link.

Another option we might consider for the future is making links to reports accessible without having to login to TestRail. While we would use secure links for this with a unique/random access key in the URL, and we would certainly disable this option by default, we would still have concerns with this approach with teams who might not understand the implications of sharing such links. This is something we need to consider with the large number of teams and customers who use TestRail.

If you are only interested in generating some simple tables you can include in an email directly (which is very limited with the Outlook rendering options as already mentioned by Adam above), you could also automate this via TestRail’s API and a small script to automatically generate such a custom email.

We will continue to review this and will see if we can include some simple PDF format of reports in the future as well.

Thanks,
Dennis


#9

I’d like to vote for this option. When I was evaluating test case management software a couple of years ago, there was one called Zephyr which had something similar. If I remember correctly, the global dashboard and the project dashboards were configurable (by the admin) and could include different status reports for the test runs, user workload, etc. And those dashboard pages were accessible without a license.

On a tangential note, Zephyr had some good features, but the cost and licensing of it made it completely unaffordable for small-to-medium organizations.


#10

+1 for this thread. We are currently evaluating TestRail and this issue is the biggest negative we have come across by far. We just expected reports to be viewable by none technical staff members who did not have TestRail accounts (it says as much in the Reports how-to video).

[Comment by Gurock: please do not share a single TestRail user/login with multiple users, as this would be against the licensing/subscription as every person using TestRail needs a separate login]

In case anyone is interested we came up with a workaround of sorts for this. Create a TestRail user called ‘Reporting’ or something similar and distribute the login details to the heads of departments, make sure you set the account to have Read Only Access. On your local email create an email group called reporting@… and get it to forward to all the heads of departments. Set the Reporting TestRail accounts email to the group address. Now when you produce a report in TestRail click the ‘Email a link to the Report’ option and select Reporting and a link will be emailed out to all the heads of departments and they can log in and check the report if it relates to them (or if they are just being nosey).

This is a poor workaround and a simple PDF or PNG option when creating the report would be much preferable. For us it doesn’t need to be interactive javascript it just need to show the report graph.

Hope this helps!


#11

Hi all,

Thanks again for the feedback! Like most applications (e.g. JIRA, other test management apps, project management tools etc.) you need a login to access data and reports in TestRail. This is of course very important for security reasons so that noone can actually access data who shouldn’t be allowed to do this. Compared to most other applications, TestRail’s reporting feature actually allows you to send reports as attachments to non-TestRail licenses users.

While I’m sure that we would be able to sell more licenses if we didn’t support this (and I guess this is the reason many companies don’t add such a feature), we understand that this is very useful and that’s why we added this. We are also happy to consider additional report formats such as PDF files, but this would be difficult to implement with TestRail’s rich reporting options.

You can alternatively simply invite all your team to TestRail and with TestRail’s large multi-user discounts, most teams take advantage of this and benefit from many other reasons why this is useful (e.g. get feedback on test cases from everyone, allow everyone to contribute to testing).

Please do not share a single login/account with multiple users, as this would be against TestRail’s license/subscription, as you need a separate user login for each person using TestRail (like basically any other application).

We are very reluctant to add an option to TestRail to access reports without login. Because even if we disabled this option by default, added a security key and made it very clear that the report is public, with the large number of customers we have it’s very likely that there will be a situation where customers leak critical data by mistake because of this. As we are designing TestRail for many customers, we need to take such situations into account as well.

We definitely appreciate the feedback and will review other options in the future as well. Most teams simply provide access to TestRail to team members who want to review reports regularly and this is usually the best approach with TestRail or most other applications.


#12

I will keep your reply, but added a short notice for other readers/users to clarify that this wouldn’t be allowed, as it’s easy to miss in my longer reply.


#13

Here’s something I do; it may take a very small amount of extra time, but I download the report myself and then create a ‘.mht’ file from the zipped report files and then attach it to an email. Then all the ‘powers that be’ have to do is double click and they have the report displayed in Internet Explorer without having to mess.

Hope this helps…


#14

\sigh

This thread keeps popping up for me, so I finally took a look at what TestRail reporting is, and what gets packed in those zip files. Turns out there’s just one index.html, a bunch of stylesheets, a few javascripts (so far I’ve seen only jQuery and Highcharts), and a few icons.

This is pretty easy to deal with, @dgurock. No fancy PDF converters or whatsoever needed.

Instead of sending a zip archive, everything can be easily packed into a single HTML file:

  • css — include as internal styles (i.e. in <style> in HTML’s head);
  • js — same, included in <script>;
  • icons and other graphics — inlined via Data URIs. The downside is 33% inflation because of base64, but on a bunch of <1KiB icons does that really matter?

All super-simple to implement, and customers are happy.

P.S. Ah yeah, and unzipped HTML can be one order of magnitude bigger than the zips. However I think there must be something to help deflate those HTMLs in messages, similar to gzip in web. And even if there isn’t, I think it’s a fair trade-off, given that reports size would usually not exceed a few megs.


#15

Unfortunately it’s not as easy as you might expect and we have investigated this option in the past and will evaluate this regularly in the future. If you are designing software for 10s of thousands of users, there are many factors we need to consider, such as browser compatibility, scalability of the implementation and many security relevant aspects especially with customer networks, email systems and security filters etc. (which is one of the reasons an all-in-one HTML page would not work well for many of our customers based on our experience). I think the ability to forward/download/share reports is a unique feature of TestRail as most other products would always require a full license to view such reports. But we also plan to extend the reporting options and review if we can provide additional formats in the future.


#16

Those requiring PDF reports, you can print to PDF using Chrome, no converters needed and it works rather nicely.

When you click the print icon in your report just select printer as “save to pdf” you get a preview just to make sure it all looks good.


#17

+1.

Another vote for printing to PDF.

PDF is generic and all browsers open the file. If you can send a report as PDF that would be excellent.

At the moment the user has to email the report to themselves. Print as PDF and re-distribute. It works but it’s just a bit clunky and negates the excellent auto-email report options you currently have.


#18

Thank you! I added your vote to the list.

Cheers,
Tobias


#19

Add my vote in. I’m glad that your product includes the ability to send reports without a TestRail account, but the .zip attachment only works well if the person on the receiving end is at least competent in tech.

For my purposes the only information these people need to see are what defects occurred, how many test cases are left, and the estimate for testing completion.


#20

Hello Geo,

Thanks for your feedback. We chose the ZIP/HTML version to have the exact same look & feel as the reports in TestRail. You would just need to extract the ZIP and then double-click on the index.html file. This opens your default browser and would show the report, similar to as you would see it in TestRail.

Cheers,
Tobias