For mosts schools, strong reference letters are a must. Reference is the only part of application packets that is almost fully dependent on the referees. It serves as recommendation from people who know you most professionally. This however doesnot mean that there is nothing you can do. You can’t change the past and how you lived your professional life, but here are your parts to improve your chance of admission.
Pick referees you can write you a strong recommendation letter. Make sure the referee is somebody who knows you professionally (your professor, your supervisor, etc). Family and friends are a no no! Ask them specifically if they can (i.e. want) to write you a strong letter. If they cannot, it is better to hear from them than having the school read the letters. Only choose those that agree to write you a strong recommendation letter.
Make sure your referees know you personally. He/she should not be somebody who only knows you on the surface. The more reliable/credible this person, the stronger the letter, however, there is nothing that can replace a personal letter. Having a strong personal letter from your supervisor is much stronger than a he-is-okay statement by a Nobel Laureate.
Share your desire to go for graduate schools with your potential referees. Do it early, keep him updated with your application progress. It helps to do so when he will write your letter of recommendation.