Advanced system design and performance analysis for high speed optical communications