The equation of the cone should be [tex]z=\sqrt{x^2+y^2}[/tex]. Parameterize [tex]S[/tex] by
[tex]\vec s(u,v)=u\cos v\,\vec\imath+u\sin v\,\vec\jmath+u\,\vec k[/tex]
with [tex]1\le u\le2[/tex] and [tex]0\le v\le2\pi[/tex]. Take the normal vector to [tex]S[/tex] to be
[tex]\vec s_v\times\vec s_u=u\cos v\,\vec\imath+u\sin v\,\vec\jmath-u\,\vec k[/tex]
Then the integral of [tex]\vec F[/tex] across [tex]S[/tex] is
[tex]\displaystyle\iint_S\vec F\cdot\mathrm d\vec S=\int_0^{2\pi}\int_1^2(-u\cos v\,\vec\imath-u\sin v\,\vec\jmath+u^3\,\vec k)\cdot(u\cos v\,\vec\imath+u\sin v\,\vec\jmath-u\,\vec k)\,\mathrm du\,\mathrm dv[/tex]
[tex]=\displaystyle\int_0^{2\pi}\int_1^2(-u^2-u^4)\,\mathrm du\,\mathrm dv[/tex]
[tex]=\displaystyle-2\pi\int_1^2(u^2+u^4)\,\mathrm du\,\mathrm dv=\boxed{-\frac{256\pi}{15}}[/tex]
