The STK504 has an RS-232 driver operating at target voltages from 1.8V to 5.5V with a maximum speed of 480kbps. There are two transmit and two recieve pins. With the RS-232 port available on the STK500, you have three channels available for your design.
The driver pinsa are available on the connector in the lower right corner of the STK503 (see figure #GUID-2EAB9A0D-9746-4C5F-A4A3-D71C9626B735__STK503.FIGURE.RS232.CONNECTOR and Figure 1).
To use the driver you must connect the logic level pins to the AVR and the RS-232 pins to the external device. The connector is marked with AVR and RS-232, where the pins on the AVR side must be connected to the RxD and TxD on the AVR, and the RS-232 side connects to a connector or your RS-232 enabled device. The arrows show the signal directions.
As the STK503 has no 9 pin or 25 pin connector, you must make your own connector. Wou can however decide if you will use only the TxD and RxD lines, giving you two channels, or if handshake signals like RTS/CTS or DTR/DSR will be used. In the latter case, you must connect the handshake signals to regular PORT pins that are controlled by software.