Coverlay for Scala, or how to produce lcov from cobertura
Too long; didn't read
Visualize your scala code coverage directly in your Scala buffers! Here I show some Elisp to cobertura files to the lcov file, so that Scala coders can leverage coverlay too.
The problem
I always found coverage reports boring to read. Waiting for instrumented code to produce a report is already a bit long. Then you get an html file that you need to open, consult and translate to a place in your code, so that, finally, you can add some test cases to cover those lines.
Apart of the clicking involved, I was looking forward to have my preferred editor show me those reports and immediately displaying me which lines my tests miss to cover. Then just the other day I discovered from Sacha's blog the existence of coverlay. Finally, in 2021 I can see my code coverage!!!
Or not? Argh, it does not support xml reports!
It is a problem indeed
So I think: "there must surely be a converter from lcov to cobertura,
doesn't it?". The answer iiiis... no. Apparently, cobertura is the
format that some continuous integration tools use, while lcov is the
format (quite simple and elegant I must say) that Linux maintainers
came up with. So some people decided to create translators from lcov
to cobertura but NOT the other way around.
It really seems I need to write some Elisp myself to fix this.
And there is a solution
The idea is that coverlay reads an lcov file which looks like this:
SF:/path/to/target.js DA:15,1 DA:16,1 DA:20,1 DA:21,0 DA:22,1 DA:23,1 DA:25,0 DA:28,1 DA:30,1 DA:32,1 DA:33,5 end_of_record
Nice and simple, at first sight you can see that the top is the file location and the lines in the middle represent line numbers and the number of times the tests hit that line.
Now for a Scala project we can create cobertura files with sbt-scoverage, they look like this:
<?xml version="1.0"?>
<!DOCTYPE coverage SYSTEM "http://cobertura.sourceforge.net/xml/coverage-04.dtd">
<coverage
line-rate="0.71" lines-valid="1833" lines-covered="1297" branches-valid="125" branches-covered="86" branch-rate="0.69" complexity="0" version="1.0" timestamp="1609609652641">
<sources>
<source>--source</source>
<source>/home/andrea/workspace/scalafix/scalafix-cli/src/main/scala</source>
</sources>
<packages>
<package name="scalafix.cli" line-rate="0.74" branch-rate="0.83" complexity="0">
<classes>
<class name="scalafix.cli.Cli" filename="scalafix/cli/Cli.scala" line-rate="0.00" branch-rate="0.00" complexity="0">
<methods>
<method name="scalafix.cli/Cli/helpMessage" signature="()V" line-rate="0.00" branch-rate="0.00" complexity="0">
<lines>
<line number="7" hits="0" branch="false"/>
</lines>
</method>
<method name="scalafix.cli/Cli/nailMain" signature="()V" line-rate="0.00" branch-rate="0.00" complexity="0">
<lines>
<line number="12" hits="0" branch="false"/>
</lines>
</method>
<method name="scalafix.cli/Cli/main" signature="()V" line-rate="0.00" branch-rate="0.00" complexity="0">
<lines>
<line number="9" hits="0" branch="false"/>
</lines>
</method>
</methods>
<lines>
<line number="9" hits="0" branch="false"/>
<line number="7" hits="0" branch="false"/>
<line number="12" hits="0" branch="false"/>
</lines>
</class>
<class
name="scalafix.cli.ExitStatus" filename="scalafix/cli/ExitStatus.scala" line-rate="0.78" branch-rate="0.83" complexity="0">
<methods>
<method name="scalafix.cli/ExitStatus/from" signature="()V" line-rate="0.00" branch-rate="0.00" complexity="0">
<lines>
<line number="78" hits="0" branch="false"/>
<line number="76" hits="0" branch="false"/>
<line number="80" hits="0" branch="false"/>
<line number="74" hits="0" branch="false"/>
</lines>
</method>
<method name="scalafix.cli/ExitStatus/<none>" signature="()V" line-rate="1.00" branch-rate="1.00" complexity="0">
<lines>
<line number="42" hits="1" branch="false"/>
<line number="21" hits="1" branch="false"/>
<line number="42" hits="1" branch="false"/>
<line number="42" hits="1" branch="false"/>
<line number="42" hits="1" branch="false"/>
<line number="20" hits="1" branch="false"/>
<line number="42" hits="1" branch="false"/>
<line number="42" hits="1" branch="false"/>
<line number="42" hits="1" branch="false"/>
<line number="23" hits="1" branch="false"/>
<line number="42" hits="1" branch="false"/>
<line number="42" hits="1" branch="false"/>
</lines>
</method>
<method name="scalafix.cli/ExitStatus/is" signature="()V" line-rate="1.00" branch-rate="1.00" complexity="0">
<lines>
<line number="13" hits="1" branch="false"/>
</lines>
</method>
<method name="scalafix.cli/ExitStatus/codeToName" signature="()V" line-rate="1.00" branch-rate="1.00" complexity="0">
<lines>
<line number="49" hits="1" branch="false"/>
<line number="51" hits="1" branch="false"/>
<line number="46" hits="1" branch="false"/>
<line number="49" hits="1" branch="false"/>
<line number="48" hits="1" branch="false"/>
<line number="46" hits="1" branch="false"/>
<line number="47" hits="1" branch="false"/>
<line number="46" hits="1" branch="false"/>
<line number="48" hits="1" branch="false"/>
</lines>
</method>
<method name="scalafix.cli/ExitStatus/merge" signature="()V" line-rate="0.38" branch-rate="1.00" complexity="0">
<lines>
<line number="67" hits="0" branch="false"/>
<line number="64" hits="1" branch="false"/>
<line number="68" hits="0" branch="false"/>
<line number="67" hits="0" branch="false"/>
<line number="68" hits="0" branch="false"/>
<line number="64" hits="1" branch="false"/>
<line number="64" hits="1" branch="false"/>
<line number="68" hits="0" branch="false"/>
</lines>
</method>
<method name="scalafix.cli/ExitStatus/isOk" signature="()V" line-rate="1.00" branch-rate="1.00" complexity="0">
<lines>
<line number="12" hits="1" branch="false"/>
<line number="12" hits="1" branch="false"/>
</lines>
</method>
<method name="scalafix.cli/ExitStatus/apply" signature="()V" line-rate="0.67" branch-rate="0.50" complexity="0">
<lines>
<line number="58" hits="1" branch="false"/>
<line number="55" hits="0" branch="false"/>
<line number="57" hits="1" branch="false"/>
<line number="55" hits="1" branch="false"/>
<line number="55" hits="0" branch="false"/>
<line number="56" hits="1" branch="false"/>
</lines>
</method>
<method
name="scalafix.cli/ExitStatus/generateExitStatus" signature="()V" line-rate="1.00" branch-rate="1.00" complexity="0">
<lines>
<line number="27" hits="1" branch="false"/>
<line number="26" hits="1" branch="false"/>
<line number="26" hits="1" branch="false"/>
<line number="26" hits="1" branch="false"/>
<line number="26" hits="1" branch="false"/>
<line number="25" hits="1" branch="false"/>
<line number="28" hits="1" branch="false"/>
<line number="26" hits="1" branch="false"/>
<line number="26" hits="1" branch="false"/>
</lines>
</method>
</methods>
<lines>
<line number="42" hits="1" branch="false"/>
<line number="67" hits="0" branch="false"/>
<line number="49" hits="1" branch="false"/>
<line number="58" hits="1" branch="false"/>
<line number="78" hits="0" branch="false"/>
<line number="51" hits="1" branch="false"/>
<line number="21" hits="1" branch="false"/>
<line number="64" hits="1" branch="false"/>
<line number="27" hits="1" branch="false"/>
<line number="42" hits="1" branch="false"/>
<line number="26" hits="1" branch="false"/>
<line number="46" hits="1" branch="false"/>
<line number="55" hits="0" branch="false"/>
<line number="68" hits="0" branch="false"/>
<line number="42" hits="1" branch="false"/>
<line number="12" hits="1" branch="false"/>
<line number="26" hits="1" branch="false"/>
<line number="67" hits="0" branch="false"/>
<line number="57" hits="1" branch="false"/>
<line number="68" hits="0" branch="false"/>
<line number="12" hits="1" branch="false"/>
<line number="26" hits="1" branch="false"/>
<line number="42" hits="1" branch="false"/>
<line number="49" hits="1" branch="false"/>
<line number="26" hits="1" branch="false"/>
<line number="20" hits="1" branch="false"/>
<line number="64" hits="1" branch="false"/>
<line number="76" hits="0" branch="false"/>
<line number="42" hits="1" branch="false"/>
<line number="48" hits="1" branch="false"/>
<line number="25" hits="1" branch="false"/>
<line number="55" hits="1" branch="false"/>
<line number="42" hits="1" branch="false"/>
<line number="46" hits="1" branch="false"/>
<line number="64" hits="1" branch="false"/>
<line number="80" hits="0" branch="false"/>
<line number="42" hits="1" branch="false"/>
<line number="47" hits="1" branch="false"/>
<line number="23" hits="1" branch="false"/>
<line number="28" hits="1" branch="false"/>
<line number="68" hits="0" branch="false"/>
<line number="42" hits="1" branch="false"/>
<line number="26" hits="1" branch="false"/>
<line number="46" hits="1" branch="false"/>
<line number="55" hits="0" branch="false"/>
<line number="26" hits="1" branch="false"/>
<line number="56" hits="1" branch="false"/>
<line number="74" hits="0" branch="false"/>
<line number="42" hits="1" branch="false"/>
<line number="48" hits="1" branch="false"/>
<line number="13" hits="1" branch="false"/>
</lines>
</class>
</classes>
Much less readable, but more detailed: here we know package names, line coverage percentage, we have even branch coverage and method coverage!
The little complications:
- we have a different cobertura file for each project module (typically more than one in average sized applications)
- the path of the file is both in the
sourcetag and in theclasstag under thefilenameattribute - same line numbers can be repeated under the
linestag because thehitsattribute is really a boolean value.
So our first challenge is to locate the files. The thing we know is that they always have the same name:
(defun coverlays/find-cobertura-files ()
"Find cobertura.xml files."
(--> (projectile-root-bottom-up ".")
(format "find %s -type f -name \"cobertura.xml\"" it)
(shell-command-to-string it)
(s-split "\n" it)
(-map 's-trim it)
(--remove (or (null it) (s-blank-p it)) it))
The trick is to use the shell's find utility to list the file paths we need. Most of the code above is just to clean the shell output.
Next for each file we want to parse its xml and grab what we need: file paths, line numbers and hits.
(defun coverlays/parse-path-lines-from-cobertura (filepath)
"Parse FILEPATH to an alist with a filepath and lines, which contain number and hits."
(when (f-file-p filepath)
(with-temp-buffer
(insert-file-contents-literally filepath)
(let* ((xml (libxml-parse-html-region (point-min) (point-max)))
(path (nth 2 (nth 3 (esxml-query "sources" xml)))) ;; TODO maybe not universal
(classes (-drop 2 (esxml-query "classes" xml)))
(make-path-lines-from-class
(lambda (class)
`((filepath . ,(s-concat path "/" (alist-get 'filename (nth 1 class))))
(lines . ,(--map (--> it
(nth 1 it)
(--filter
(or (eq (car it) 'number)
(eq (car it) 'hits))
it))
(-drop 2 (nth 3 class))))))))
(-map make-path-lines-from-class classes)))))
The nice esxml library takes the toll of parsing xml away from me. Most of the code in this function goes in grabbing the right information and putting it together to obtain the filepath and the line count (note the TODO in the code: I am not sure the cobertura.xml files always put the module path in that position, but in all my projects has worked fine for now).
This function generates this sort of output:
(((filepath . "some/path")
(lines . (((number. 123) (hits . 1))
((number. 123) (hits . 1))
((number. 124) (hits . 1))))))
Note that I could not find yet the time to merge the duplicate lines: the duplication means that multiple tests have exercised those lines. The lcov format expects the sum of the hits, while the cobertura format expect duplication. I postponed this feature, because it only partially breaks coverlay's functionality: I saw that the statistics buffer fails in error with duplication.
Now we have all the information to produce our lcov file:
(defun coverlays/produce-lcov (filenames-lines)
"Produce lcov file from FILENAMES-LINES."
(--> (--reduce-from
(let ((heading (s-concat "SF:" (alist-get 'filepath it) "\n"))
(line-numbers
(--map
(s-concat "DA:" (alist-get 'number it) "," (alist-get 'hits it) "\n")
(alist-get 'lines it))))
(s-concat acc heading (apply 's-concat line-numbers) "end_of_record\n"))
""
filenames-lines)
))
(defun coverlays/produce-lcov-from-cobertura ()
"Produce lcov file from cobertura"
(interactive)
(let ((file (s-concat (projectile-project-root) "/lcov.lcov")))
(delete-file file)
(--> (coverlays/find-cobertura-files)
(--each
it
(--> (coverlays/parse-path-lines-from-cobertura it)
(coverlays/produce-lcov it)
(write-region it nil file t))))))
The function coverlays/produce-lcov just formats the information
according to the lcov syntax, while
coverlays/produce-lcov-from-cobertura integrates the above functions
to generate all the cobertura files and an lcov from them.
After you run coverlays/produce-lcov-from-cobertura you just have to
call coverlay's coverlay-load-file and you will see for the first
time coverage lines in Scala buffers!!!
Naturally this sounds still a bit boring, doesn't it?
You still have to:
- produce the cobertura files with something like
sbt ;clean;coverage;test;coverageReport - run
coverlays/produce-lcov-from-cobertura= - finally run
coverlay-load-fileon the produced lcov file
We can do better! Let's automate that out :)
I will rely on emacs-async, projectile and the scala-mode-hook:
(defun coverlays/rebuild-cobertura-on-project ()
"Run sbt tests and produce lcov file."
(interactive)
(let ((proj-type (projectile-project-type))
(directory (projectile-project-root)))
(when (or (eq 'bloop proj-type) (eq 'sbt proj-type) (eq 'scala proj-type))
(message "Producing cobertura output...")
(async-start
`(lambda () (let ((default-directory ,directory)) (message "starting %s" ,directory) (call-process "sbt" nil nil nil ";clean;coverage;test;coverageReport;") (message "finishing")))
`(lambda (result)
(cd ,directory)
(coverlays/produce-lcov-from-cobertura)
(message (format "Coverlay mode ready for %s" ,directory)))))))
(add-hook
'scala-mode-hook
(lambda ()
(let ((file (concat (projectile-project-root) "/lcov.lcov")))
(when (or (not (f-file-p file)
(> (time-to-seconds (time-since
(file-attribute-modification-time
(file-attributes (file-truename file)))))
(* 60 60 2)))) ;; maybe just lcov older than 2 hours?
(coverlays/rebuild-cobertura-on-project)))))
(add-hook
'scala-mode-hook
(lambda ()
(when (f-file-p (concat (projectile-project-root) "/lcov.lcov"))
(turn-on-coverlay-mode))))
(add-hook
'coverlay-mode-hook
(lambda ()
(let ((proj-type (projectile-project-type))
(file (concat (projectile-project-root) "/lcov.lcov")))
(when
(and
(or
(eq 'bloop proj-type)
(eq 'sbt proj-type)
(eq 'scala proj-type))
(projectile-project-root)
(f-file-p file))
(coverlay-load-file file)))))
In short what I came up with is to run sbt in the background the first time I visit a project, and when that succeeds to produce the lcov file and load it when I open a Scala buffer.
I created a repository to make this code easy to use: https://github.com/ag91/coverlay-for-scala-example
This is just a first iteration, but better rough than without, no? I will update the repository as soon as I do progress.
Also I still plan to have a fully working coverlay, and maybe who knows even just extend coverlay to support cobertura's branch analysis.
Conclusion
So take the code from my repository and see finally your code coverage in your Scala projects! If you have sbt-scoverage and the Emacs dependencies (look at the README for how to install them), you just need to visit a Scala file in your repository and after a few minutes the coverage should kick in.
Happy testing!