Question:    How should I rank the hands in lowball game where a straight or a flush can't be played as a low hand?

Answer:    In the lowball game, the ranking of poker hands depends on the kind of game you are playing. For instance, in the classic Deuce-to-Seven lowball, an ace is a high card, and the lowest possible hand is 2-3-4-5-7. So, if you have a hand of 2-3-4-5-8, you can easily beat the player with a A-2-3-6-8. The latter hand can compete against only those hands, which have an ace or a pair. In these games, the ranking of cards starts from the top, i.e., with the highest cards. Therefore, a hand consisting of 2-4-5-6-8 will win over another having 2-3-5-7-8.  
Otherwise, the ace plays as a low card in majority of other types of low or high/low poker. Generally, in home poker, where the players have the freedom to pick-up rules at their will, they use the ace as a low card although straights and flushes do play as high hands. In such circumstances, even if you allow the ace to play for low, the A-2-3-6-8 hand will succumb to 2-3-4-5-8, because an 8-5 low is better than an 8-6.