Use pdflatex to compile.
To rebuild index you need to prepare your texindy installation:
* find the files tex/inputenc/*.xdy
* convert, say, cp1251.xdy to utf with 'iconv -f cp1251 -t utf8 cp1251.xdy | sudo tee utf8.xdy' or something like that
Now you have the correct utf8.xdy! Use 'texindy -L russian --codepage utf-8 algebra.idx' to rebuild index.