From the past few years, demand of high bandwidth, high channel capacity increases rapidly. In order to meet these types of requirements we employ wavelength division multiplexing (WDM) over a bidirectional link. WDM performance mainly depends upon various performance metrics such as modulation formats, channel capacity, channel spacing, range etc. This paper shows the comparative analysis of various advanced modulation formats such as non return to zero (NRZ), return to zero (RZ), carrier-suppressed return to zero (CSRZ) and modified duo-binary return to zero (MDRZ). The designed system has achieved the best result at a distance of 120km over 4-channel bidirectional link (without using any amplifier or any dispersion compensating fiber). System performance has been evaluated in terms of BER, Q-factor and eye diagram on optisystem 14.1. From the results, it is found that MDRZ gives the dispersion less transmission and less BER for long distance transmission.