Use pdflatex to compile. To rebuild index you need to prepare your texindy installation:

• find the files tex/inputenc/*.xdy;
• convert, say, cp1251.xdy to utf with iconv -f cp1251 -t utf8 cp1251.xdy | sudo tee utf8.xdy or something like that.

Now you have the correct utf8.xdy! Use texindy -L russian --codepage utf8 algebra.idx to rebuild index.