The first thing you should do when you visit a restaurant for the first time is find out what they do well; although they may have some dishes you will like and some dishes you will dislike (and while the "thing they do best" might be something you would not like at all!), this will give you a good idea of the sorts of things they specialize in.
Next, you should take some time to peruse the menu, looking for items that look "familiar"; even though each restaurant will have their own, signature style with certain dishes, finding a dish you are familiar with is a great way to ensure that you will end up with a meal that you will actually enjoy.
And sometimes when you visit a restaurant for the first time, it is absolutely worth it to order something you have never had before, as the adventure of trying something completely new - at a new restaurant - is sometimes enjoyable enough in and of itself, even if you end up not particularly enjoying the meal (and if you end up loving the meal, it will be that much more worth it!).