$6a^2 + 3ab^2 - 4ab - 2b^3 = (6a^2 + 3ab^2) - (4ab + 2b^3) = 3a(2a + b^2) - 2b(2a + b^2) = (2a + b^2)(3a - 2b)$
если $a = -1\frac{1}{8}, b = -1\frac{1}{2}$, то:
$(2 * (-1\frac{1}{8}) + (-1\frac{1}{2})^2)(3 * (-1\frac{1}{8}) - 2 * (-1\frac{1}{2})) = (2 * (-\frac{9}{8}) + (-\frac{3}{2})^2)(3 * (-\frac{9}{8}) - 2 * (-\frac{3}{2})) = (-\frac{9}{4} + \frac{9}{4})(-\frac{27}{8} + \frac{6}{2}) = 0 * (-\frac{27}{8} + \frac{6}{2}) = 0$