Solder mask cutouts can be rectangular. To change to rectangular cutout click on the Rectangle tab in the cutouts properties in the properties panel.
Rectangular cutout can have rounded corners.
without and with rounded corners
To edit the parameters of a cutout, first select it. The properties panel will display the properties for the selected cutout.
The x coordinate of the center of the cutout
The y coordinate of the center of the cutout
The width of the cutout.
The height of the cutout.
Drag to from left to right to increase the rounding of the corners.
Select the layer for the cutout