15 comments

[ 5.6 ms ] story [ 48.0 ms ] thread
Anyone made the opposite product? Human written tests, that an AI tries to make green?
We are writting ton of such tests, on daily basis haha.

That is infinite process, we're all stuck within :-)

That’s actually a really interesting idea. Particularly if you take care to add docstrings and comments to the tests. Thanks!
Keep me posted if you get that working! Would be really interesting to see! Might mess around with it myself sometime too.
Given the "ai Mario just pauses the game and calls it a win"

I'd expect that the ai written code doesn't do the right thing outside of the given test cases

Watch out, we'll end up writing unit tests 40 hrs a week
Is there a place where to see the code you’re trying to test? My primary use of unit tests is to understand if its clean and maintainable code, so im likely your last user case, but wanted to see what kind of code took hours to test
Do you mean within overview page of extension, on examples gif/images or within real flow in VS Code?
Just some code examples of what was being tested
Sure. Here is one example from run I performed just now.

TypeScript utility method I generated cases and tests for -------

  static merge<T extends { id?: string }>(arr1: T[], arr2: T[]): T[] {
      arr1 = arr1 || [];
      arr2 = arr2 || [];

      // Create a map for quick access to items in arr1 by id
      const map = new Map<string, T>(arr1.map((item) => [item.id, item]));

      // Iterate over arr2 and merge or add items
      arr2.forEach((item) => {
         if (map.has(item.id)) {
            // If item exists in arr1, merge it
            map.set(item.id, { ...map.get(item.id), ...item });
         } else {
            // If item doesn't exist in arr1, add it
            map.set(item.id, item);
         }
      });

      // Convert the map back to an array
      return Array.from(map.values());
   }


Generated test cases ------------- POSITIVE - Merging two arrays with unique and overlapping objects by id Given: Two arrays of objects with unique and overlapping ids When: Merging two arrays of objects where some objects have matching ids Then: Should return an array of merged objects with no duplicates and combined properties for overlapping ids

NEGATIVE - Merging arrays when second array contains objects with undefined ids Given: First array of objects with valid ids and second array with some objects having undefined ids When: Second array contains objects with undefined ids Then: Should throw an error or handle undefined ids gracefully without merging those objects

EDGE CASE - Merging arrays where all objects in the second array have matching ids in the first array Given: Two arrays of objects with ids where the second array's objects all have ids that match those in the first array When: Every object in the second array has a matching id in the first array Then: Should return an array with objects from the first array having their properties updated by the second array's objects

Generated unit tests by test cases -------------

---> 1st case ---------------

it('Merging two arrays with unique and overlapping objects by id should return an array of merged objects with no duplicates and combined properties for overlapping ids', () => { // Arrange const arr1 = [{ id: '1', name: 'Alice' }, { id: '2', name: 'Bob' }]; const arr2 = [{ id: '2', name: 'Robert' }, { id: '3', name: 'Charlie' }]; const expectedResult = [ { id: '1', name: 'Alice' }, { id: '2', name: 'Robert' }, // Note: 'Bob' is overwritten by 'Robert' { id: '3', name: 'Charlie' } ];

  // Act
  const result = merge(arr1, arr2);

  // Assert
  expect(result).toEqual(expectedResult);
});

---> 2nd case ----------------

it('Merging arrays when second array contains objects with undefined ids should throw an error or handle undefined ids gracefully', () => { // Arrange const arr1 = [{ id: '1', name: 'Alice' }, { id: '2', name: 'Bob' }]; const arr2 = [{ id: '2', age: 30 }, { name: 'Charlie' }]; // Charlie has undefined id

  // Act and Assert
  expect(() => MyArrayUtils.merge(arr1, arr2)).toThrow();
});

---> 3rd case ----------------

it('Merging arrays where all objects in the second array have matching ids in the first array', () => { // Arrange const arr1 = [{ id: '1', propA: 'A1' }, { id: '2', propA: 'A2' }]; const arr2 = [{ id: '1', propB: 'B1' }, { id: '2', propB: 'B2' }]; const expectedResult = [ { id: '1', propA: 'A1', propB: 'B1' }, { id: '2', pro...

Thanks! Im from the phone now and cant understand much but will come back tomorrow
I don't really understand what the test cases will do? Is the code expected to already correct, and the tests are documentation?

If I imagine a workflow of

1. Updating some code (with a regression) 2. Running the ai to generate tests 3. The tests pass

How do I know that the ai didn't assume the regression is expected? Do I read the explanation of the test cases?

There are positive, negative and edge cases.

They are generated by AI, so unit tests will be implemented by case definitions.

IMHO, Testing is one thing that needs to be inspected when it comes to AI code generation
It is, I agree.

You have full control over generated tests. Also, I put focus on putting as much control as possible, so you can set different inputs to reach higher quality of generated tests.

At the end, you add or append file with generated tests, so you have full control over them.