Mark tests requiring network access

Labels: area/test kind/feature lifecycle/frozen priority/important-longterm

Matěj Cepl (mcepl) opened

Sometimes it's useful to have the tests that use the network marked so they can be skipped easily when we know the network is not available.

This is useful for example on SUSE and openSUSE's build servers (and I guess all other Linux/Unix distributions). When building our packages, the network is disabled so we can assure reproducible builds (among other benefits).

I am not sure how to achieve this with Go, but with Python/pytest it is possible to mark some tests as requiring network access and then skip them.

github-actions (github-actions) commented

This bot triages issues in order to help the maintainers identify what needs attention, according to the following lifecycle rules:

  • After 90 days of inactivity, lifecycle/stale is applied
  • After 90 days of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

This bot will not automatically close stale issues.

To remove the stale status, you can:

  • Remove the stale label from this issue
  • Comment on this issue
  • Close this issue
  • Offer to help out with triaging

To avoid automatic lifecycle management of this issue, add lifecycle/frozen.

github-actions (github-actions) added label lifecycle/stale

sudoforge removed label lifecycle/stale

sudoforge added label area/test

sudoforge added label kind/feature

sudoforge added label lifecycle/frozen

sudoforge added label priority/important-longterm

sudoforge commented

just to add a comment here:

golang doesn't have inbuilt support for test categorization like this, but i do think the tests are a bit intermixed right now and in an ideal world, we shouldn't be running e2e tests non-explicitly.

the "standard" approach for this would probably be to mark every integration/e2e test as "short", e.g.:

func TestFoo(t *testing.T) {
  if testing.Short() {
    t.Skip("skipping TestFoo in short mode")
  }
}

then, these tests could be skipped by passing the -short flag to go test:

go test -short ./...

we don't do this now, so it might make sense to apply this to all of our networked/integration/e2 tests. for more fine-grained test categorization, we could also introduce a custom flag, e.g. -network, making those opt-out by default unless that flag is passed (technically, this could be done by inverting the boolean logic around checking testing.Short() as well, however, that is lexically odd and could cause confusion).

sudoforge removed label lifecycle/pinned