hessian matrix of ( (x^2-1)^2+(x^2y-x-1)^2 )