BACKGROUND: Little information is available on characteristics of successful and unsuccessful community health assessment (CHA) reports. METHODS: A consensus process identified criteria for assessing CHA success from a literature review, analysis of CHAs on the Internet, and consultation with experts. Criteria were then turned into questionnaire items. Using these items, a Web-based tool was developed to gather responses from 110 users and potential users about the strengths and weaknesses of CHAs from six New York counties and three from other states. FINDINGS: Respondents tended to rate CHAs positively, with high scores for including important aspects of health, using consistent formats, reproducibility by photocopy, and supporting grant applications. Community health assessments were given low scores because of a lack of focus on positive characteristics and documentation of methods, and failure to indicate relationships among indicators, include narrative and graphics, and be similar to other community planning tools in use. CONCLUSIONS: Community health assessment reports should state their goals and purpose; include the most important aspects of the community's health; allow comparisons with other communities, other benchmarks, and, over time, present data in meaningful subgroups of population; provide sufficient focus on positive characteristics; and document the process and methods that are used to create the CHA.