The notion of invexity in vector optimization: smooth and nonsmooth case