Codecov / Nondeterministic tests

[dreua]

It seems like we have some nondeterministic tests and that messes with codecov: For example I got a red from codecov/project in #994 which didn't touch coverage at all.

In order to mitigate this I tried relaxing the settings a bit to allow one line to change (the if self.quit: return is very likely just nondeterministic and it probably not worth it to add additional tests just to cover it all the time).

In https://app.codecov.io/account/gh/pdfarranger/yaml I set the default settings for the PDF Arranger organization as follows:

coverage:
  status:
    project:
      default:
        target: auto
        threshold: 0.04%

This was meant as just a quick test but I'd like to leave it like this (maybe even increase the threshold) and see how it works.

In a subsequent run I got even more line hits than previously so we might have an even bigger issue with nondeterminism than I originally thought.

[jeromerobert]

(maybe even increase the threshold) and see how it works.

Yes ! It will remove false negative until someone find why they append.

even bigger issue with nondeterminism than I originally thought

Yes. I had the feeling that codecov is sometime selecting a wrong base commit when doing a coverage diff. That's just a feeling, I did not investigated.

[dreua]

What I saw looked more like race conditions in our tests, maybe version changes in our dependencies. But it could be issues in codecov as well, idk.

[jeromerobert]

Hum may be that calling codecov/codecov-action in multiple GHA build is not a great idea. Let met try to call it only one time.

[jeromerobert]

https://about.codecov.io/blog/uploading-code-coverage-in-a-separate-job-on-github-actions/

[jeromerobert]

This should now be fixed (see #998)

[dreua]

Thank you!!

[jeromerobert]

I also added this to the YAML file:

coverage:
  status:
    patch:
      default:
        informational: true

See https://docs.codecov.com/docs/commit-status#informational for details

[dreua]

Added that also to the project:

    project:
      default:
        informational: true

[dreua]

I just merged #1022 and am now wondering what our goals are with codecov: Do we want to enforce full coverage of changes or just have it as a tool to show coverage whenever we feel like it? I didn't feel like enforcing full coverage on this fix (someone would need to craft a PDF to reproduce it which we can publish and that's really not worth the effort). On the other hand this leaves us with red ❌ CI status and someone correctly pointed out that having those on a regular basis makes them less useful. I propose to reconfigure codecov to be green all the time until we decided on something and use it only as an aide in cases where we are interested in the coverage. (Assuming such a configuration is possible.)

[jeromerobert]

I propose to reconfigure codecov to be green all the time

Ok

[dreua]

Looks like you already implemented that, thanks!

[dreua]

Just collecting this here for future reference: Codecov is showing untested lines in a translation only commit, that absolutely seems broken to me
https://github.com/pdfarranger/pdfarranger/pull/1047/files