Opened 7 years ago

Last modified 7 years ago

#946 new enhancement

Tag coverage per test

Reported by: fare Owned by: kmcorbett
Priority: normal Milestone:
Component: Compiler Version: trunk
Keywords: ITA Cc:

Description

Gail Zacharias 2011-04-19 20:25:45 EDT

Date: Mon, 07 Feb 2011 15:00:39 -0500

From: Daniel Weinreb <dlw@…>

To: Gail Zacharias <gz@…>, David Crawford <dcrawford@…>

[...]

It would be nice if, when one of

our developers changed some lines of code, we could pick a subset of unit tests to run based on the lines they changed vs. which unit tests exercise those lines. The goal is to minmimize the number of unit tests to run, since running them takes time and slows down the development cycle.

I know that finding no line-by-line intersection does not prove that the test will still work, as there are lots of opportunities for dynamic interaction. However, our Hudson buildbots will run all the tests anyway, so it's not like some bug will creep by the unit tests.

We will also have to worry about initialization code that is always called, and probably also code in the unit test framework. That's our problem, not yours, although it's possible that we might need some support in some way; too early to say.

In theory, we could do all this by just running a QRes with unit test 1, then one with unit test2, and so on, getting big report data, and crunching it. That would be rather slow because it takes a relatively long time to start up a QRes.

The proposal is to have "tags", and when a line is covered, it gets marked with the "current tag", which we would set every time we switch from running one test to running a new test.

Comment 1 Gail Zacharias 2011-05-10 17:19:40 EDT

I have the basic blocks for doing this implemented, but I'm not sure how to put them together. Would something like this be what you have in mind:

(1) You run the test suite with code coverage on clean sources from some particular svn revision, and save that image with all the data in it. Perhaps this is done by some buildbot and placed somewhere where people can download it.

(2) At another place and time, you run that saved image in a working directory with modified sources, which invokes a function that returns a list of tests affected by the changes in your working directory compared to the revision used to generate the coverage data.

How does that sound? Or did you have some other approach in mind?

Comment 2 David Crawford 2011-05-10 17:34:18 EDT

The test subset functionality sounds good -- and from your side it would look like just the set of tags which our test env would then map back to a set of tests.

The initial use for tagged coverage is somewhat more straightforward, but with more coverage UI implications. Hackers would review the results of a coverage run and look at the files/classes/packages they are concerned with and see if given tests hit that code, or which tests hit it. Views of which files/classes/packages/directories/etc were hit by a given tag could also be useful.

Haven't looked into cross-revision or revision-to-local coverage-delta comparisons before, though that could be interesting! ... (That came from a misread of your comment, but it still is an interesting thought.)

Comment 3 Gail Zacharias 2011-05-11 15:51:03 EDT

(In reply to comment #2)

The initial use for tagged coverage is somewhat more straightforward, but with more coverage UI implications. Hackers would review the results of a coverage run and look at the files/classes/packages they are concerned with and see if given tests hit that code, or which tests hit it.

So you just want a view that for each file/class/package shows you the list of tests that hit it?

I still want to understand how this information would be used. Aren't there like hundreds and hundreds of tests? So you edit file foo.lisp, and then go and look at some web page that has the coverage info and it tells you that file foo.lisp affects the following 157 tests. Well, that's cute, but now what? Are you going to go in and manually pick out and run the 157 tests?

Haven't looked into cross-revision or revision-to-local coverage-delta comparisons before, though that could be interesting! ... (That came from a misread of your comment, but it still is an interesting thought.)

I am working on a function that does an svn diff, parses the result, and returns a list of tags that were affected by the diff. It needs to run in an image that has the coverage info for the base revision on the diff, which means it can't run the tests directly (because they need to run in an image built from the target of the diff), but I imagine it could be made to construct a test script and run it.

Comment 4 David Crawford 2011-05-11 17:02:07 EDT

the "which tests to rerun" capability is neet, but not the initial primary use. More of a developer looking through their code and seeing that a line/node has no coverage, or coverage but only from foo tests -- when they were expecting tests from bar to hit that. Then they can go and write a test, and a later run will show that not only is line x covered, but the test they wrote is the one that does it.

Comment 5 Gail Zacharias 2011-05-12 11:01:56 EDT

Ok, I get it now, will do.

Xref: ITA bug 98096

Change History (2)

comment:1 Changed 7 years ago by gz

  • Owner set to kmcorbett

comment:2 Changed 7 years ago by gz

  • Keywords ITA added
Note: See TracTickets for help on using tickets.