Test Duration Analysis Summary

This document provides a summary of the test duration analysis for the idpbuilder project.

Quick Facts

• Total Tests: 62 unit/integration tests
• Total Execution Time: ~6.2 seconds (down from ~39s after optimization)
• Slowest Test: TestCloneRemoteRepoToDir (2.27s - git clone operation)
• Slowest Category: I/O operations (3.39s, 54.5% of total time)
• Slowest Package: pkg/util (3.57s, 57.4% of total time)

Visual Overview

See test-timing-analysis.md for detailed charts and diagrams showing:

• Pie chart of test time distribution by category
• Bar charts showing slowest tests and packages
• Detailed analysis of why tests take long

Key Findings

1. I/O Operations Dominate Execution Time

I/O operations, particularly git and file system operations, account for 54.5% of the total test execution time. These tests:

• Clone git repositories
• Perform file system copy operations
• Work with git worktrees

2. Integration Tests Are Fast

After optimization, integration tests (controller reconciliation) complete quickly, taking only 2.29s (36.8% of total time) with an average of 0.121s per test.

3. Timeout Test Optimization

The TestGetGiteaToken test was optimized from 35 seconds to 2 seconds by:

• Using a context with a 100ms timeout instead of sleeping for 35 seconds
• Configuring the test server to delay for 2 seconds (longer than the timeout)
• This still validates timeout behavior but runs 17.5x faster

4. Fast Unit Tests

The majority of tests (30 tests) are quick unit tests that complete in under 10ms each, totaling only 0.03s combined.

Test Categories

Tests are automatically categorized based on their functionality:

Why Tests Take Long

1. I/O Operations

Tests that interact with the file system or perform git operations require actual disk I/O, which is inherently slower than in-memory operations. The slowest I/O tests are:

• TestCloneRemoteRepoToDir (2.27s) - clones a git repository
• TestGetWorktreeYamlFiles (0.61s) - reads YAML files from git worktree
• TestCopyTreeToTree (0.51s) - copies directory trees

2. Timeout Testing

The TestGetGiteaToken test (2.00s) validates timeout behavior by using a context with a short timeout. This is necessary to ensure the application handles slow/unresponsive services correctly.

3. Real Operations vs Mocks

Many tests use real operations instead of mocks to ensure correctness, which trades speed for confidence.

Recommendations

The test suite is well-balanced and performant with:

• ✅ Comprehensive coverage across different test types
• ✅ Fast feedback for most tests (30 unit tests complete in 0.03s)
• ✅ Efficient timeout testing using context timeouts
• ✅ Reasonable total execution time of ~6.2 seconds

Potential improvements (if needed):

1. Consider mocking git clone operations for tests that don't specifically need to test git functionality
2. Use test fixtures for repeated file operations
3. Parallelize I/O tests where safe to do so

Recent Optimizations

TestGetGiteaToken: Reduced from 35s to 2s (94% improvement)

• Changed from sleeping 35 seconds to using a 100ms context timeout
• Test server delays 2 seconds, which is longer than the timeout
• Still validates timeout behavior but runs 17.5x faster

How to Use

To generate this analysis yourself:

# Run with make
make test-timing

# Or manually
go test --tags=integration -v -timeout 30m ./... -json 2>&1 | tee test-output.json
python3 scripts/analyze_test_times.py test-output.json docs/implementation/test-timing-analysis.md

See scripts/README.md for more details on the analysis tool.