Email updates

Keep up to date with the latest news and content from BMC Medical Informatics and Decision Making and BioMed Central.

Open Access Research article

Developing a web-based information resource for palliative care: an action-research inspired approach

Annette F Street1, Kathleen Swift1, Merilyn Annells2, Roger Woodruff3, Terry Gliddon4, Anne Oakley5 and Goetz Ottman1*

Author Affiliations

1 Clinical School of Nursing, La Trobe University/Austin Health, Lv 4 Austin Tower, Heidelberg VIC 3084, Australia

2 Nursing & Midwifery, La Trobe University, Bundoora VIC 3083, Australia

3 Director of Palliative Care Austin Health, Suit 9, 210 Burgundy Street, Heidelberg VIC 3084, Australia

4 Research and Development, Royal District Nursing Services, 31 Alma Rd, St. Kilda VIC 3182, Australia

5 CEO, Melbourne City Mission, 471 Nicholson St., North Fitzroy VIC 3068, Australia

For all author emails, please log on.

BMC Medical Informatics and Decision Making 2007, 7:26  doi:10.1186/1472-6947-7-26

The electronic version of this article is the complete one and can be found online at: http://www.biomedcentral.com/1472-6947/7/26


Received:7 December 2006
Accepted:14 September 2007
Published:14 September 2007

© 2007 Street et al; licensee BioMed Central Ltd.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background

General Practitioners and community nurses rely on easily accessible, evidence-based online information to guide practice. To date, the methods that underpin the scoping of user-identified online information needs in palliative care have remained under-explored. This paper describes the benefits and challenges of a collaborative approach involving users and experts that informed the first stage of the development of a palliative care website [1].

Method

The action research-inspired methodology included a panel assessment of an existing palliative care website based in Victoria, Australia; a pre-development survey (n = 197) scoping potential audiences and palliative care information needs; working parties conducting a needs analysis about necessary information content for a redeveloped website targeting health professionals and caregivers/patients; an iterative evaluation process involving users and experts; as well as a final evaluation survey (n = 166).

Results

Involving users in the identification of content and links for a palliative care website is time-consuming and requires initial resources, strong networking skills and commitment. However, user participation provided crucial information that led to the widened the scope of the website audience and guided the development and testing of the website. The needs analysis underpinning the project suggests that palliative care peak bodies need to address three distinct audiences (clinicians, allied health professionals as well as patients and their caregivers).

Conclusion

Web developers should pay close attention to the content, language, and accessibility needs of these groups. Given the substantial cost associated with the maintenance of authoritative health information sites, the paper proposes a more collaborative development in which users can be engaged in the definition of content to ensure relevance and responsiveness, and to eliminate unnecessary detail. Access to volunteer networks forms an integral part of such an approach.

Background

Several recent reports tabled by government as well as non-governmental health agencies foreground a significant disparity between the information needs of health professionals, patients and caregivers givers and the availability of such information. In response, the reports urge governments to invest in the information, education, and training resources available to health-care professionals, patients and caregivers, as well as the wider community, in order to foster awareness and capacity regarding palliative care [2-10].

Whereas the increasing demand for consumer health information on the Internet has been also the topic of much recent medical, social science and information science literature [11-22], there has been a noticeable absence of studies that focus on how to adequately design such online resources. This paper describes the advantages and challenges of a collaborative, action research-inspired approach involving users (doctors, nurses, and consumer and caregiver representatives) that informed the development of a palliative care website [1] providing accessible information for the general public, along with evidence-based pain, symptom and psycho-social information for health professionals.

The aim of the government-funded study was to improve the existing website of Palliative Care Victoria (PCV), a state-level palliative care organisation. In light of the large volume of inquiries from community doctors, nurses and members of the general public, the Director of PCV was of the view that the organisation's response to such enquiries could be improved by enhancing the information for the general public, along with evidence-based pain, symptom and psycho-social information for health professionals, in particular general practitioners and community nurses available through its website. The study also took into account more global policy issues. For instance, it took into account the key recommendations outlined by the Improving Supportive and Palliative Care for Adults with Cancer report tabled by the United Kingdom-based National Institute for Clinical Excellence in 2004. Among other the report underscored the value of high quality information for patients and caregivers, the importance of ensuring that the views of patients and caregivers are taken into account in developing and evaluating cancer and palliative care services, and that patients and caregivers have easy access to a range of easy to read, high quality information materials about cancer and palliative care services that are provided free of charge [27]. Yet, the study's main aim was to tackle a range of information gaps highlighted in recent national government reports [3,4,6,7]. In particular, the study responded to the policies outlined in the Australian Government National Palliative Care Priorities, namely to improve professional awareness and commitment of health professionals to the role of palliative care practices, and to provide quality information to patients and their caregivers [23,24].

Method

As the project was designed to be responsive to feedback from potential user groups an action research approach was undertaken. The actual research design and reporting structure was negotiated with the funding agency at the outset. The action research process enabled the project staff to design an iterative process of continuous improvement through trial, analysis, and feedback from key stakeholders in order to adapt and modify the project to meet emergent needs [25-30]. In this project, the study of effects and outcomes of one stage informed the development process of the next stage and provided guidance for the development of the investigative framework that best addressed the evaluation questions. The following Figure (1) provides an overview of the action research methodology employed.

thumbnailFigure 1. Action Research Process.

Theoretical contributions to the field of educational multi-media design indicate that issues regarding content, language, and 'look and feel' could potentially be resolved through an ongoing consultation with expert user groups [31-33]. In fact, it is widely acknowledged that an ideal design process of educational online resources is based on cybernetic principles of general system theory, which relies on cycles of self-testing, ongoing feedback, and constant adjustments [34]. However, this extremely flexible and responsive approach to web design, although able to accommodate changes in and unanticipated effects of the implementation progresses, is seldom employed due to the high cost of cycling back to stages previously completed [34]. In this project, we employed an iterative, formative development process inspired by these principles.

User feedback was collected in the form of focus group discussions, summary grids, written comments, and 'think aloud protocols', a method where users navigate the prototype site delivering verbal comments throughout the process [34]. An audit trail was designed to keep track of user responses.

The project was constructed in four main stages that were implemented between March 2003 and May 2005. The pre-development stage involved identification of the needs of the target audiences through the literature review, a pre-development online survey, and a gap analysis undertaken by the User Working Groups (UWGs) made up of consumers and health professionals under the oversight of the Project Reference Group (PRG) of key stakeholders. After a lengthy consultation and development process, a test website (beta site) was developed by the Melbourne-based Centre for Online Media and Educational Technology (COMET) for eventual transfer to the Palliative Care Victoria website. The beta (test) site development was undertaken with input from external experts and the User Working Groups. The beta site was then assessed by key stakeholders as well as consumers. Finally, as last modifications were made to the site the sustainability plan was implemented in conjunction with PVC. The following Figure (2) gives an overview of the major project stages:

thumbnailFigure 2. Main Project Stages.

Because our aim was to include, as much as possible, the views and expertise of all potential users of the website, it was attempted to include as many representatives of potential stakeholder groups as possible in the PRG and UWG so that they would be adequately represented during the various project cycles.

Stage 1 – literature review, pre-development survey, website search

The thematic literature review was based on an extensive search of electronic databases (Medline, CINAHL, PubMed) as well as web-based searches (Google, Google Scholar) for additional material, such as reports commissioned by government as well as non-governmental agencies. Due to the limited scope of the literature on palliative care, sources dealing with online cancer information were included in this literature review. Permutations of search terms (Palliative, Cancer, Care, Online, Internet, Information, Needs) were used to collect a total of 388 bibliographical entries in an EndNote file. A researcher scanned the abstracts of these articles for relevance, thus eliminating 314 articles. The remaining 74 articles were sourced for inclusion into the literature review. In addition a separate literature review was conducted focusing on the available clinical evidence in subcategories of palliative care such as pain, medication, and psychosocial needs. For the clinical subject areas systematic reviews submitted to the Cochrane database, clinical guidelines, as well as expert consensus panels were included. The research underpinning the clinical information included on the website was evaluated and ranked by clinical palliative care experts. Due to the dearth of research-based evidence in sub-categories such as psycho-social needs, the PRG decided to include also what is considered 'best practice' in those fields.

A short online survey was designed to provide a cursory overview of users and their palliative care information needs. The 10 question survey was posted on the existing PCV website. Participants were asked:

• why they accessed the website

• whether they were health professionals or patients/caregivers/members of the public,

• what kind of information they were looking for,

• whether they found the desired information,

• to indicate areas that should be considered for inclusion in the website,

• whether it was easy to locate the information on the existing site, and

• to comment information gaps in the current site.

Internet users who accessed the site were asked to complete the form online. The survey was posted on the internet for a total of five and a half months beginning from 8 October 2003. Response to the survey was anonymous. The webmaster collected the responses and coded and pre-compiled the data. Subsequently, the responses were screened, duplicates or void forms were eliminated, and the pre-compiled data was analysed.

Furthermore, an extensive search of existing health websites was undertaken to identify the key features of reputable or 'Health on the Net' (HON) accredited websites. The Economic and Social Council of the United Nations accredited HON Foundation is the leading organization promoting and guiding the deployment of useful and reliable online medical and health information, and its appropriate and efficient use.

Stage 2 – gap analysis

Taking into account the preliminary findings from the literature review, the web search, and the results of the pre-development survey, both UWGs drafted a detailed gap analysis framework addressing the information needs of the stakeholders they represented. Moreover, the Project Reference Group as well as key stakeholders and the experts from the Centre for Online Multimedia and Educational Technologies (COMET) at La Trobe University, evaluated the original website of Palliative Care Victoria for content, relevance, accessibility, links, search potential, design features and language level. The original website catered predominantly for palliative care 'insiders' (mainly general practitioners and physicians) and posted a series of pamphlets and press releases that could be handed on to patients. The literature reviews in conjunction with the website search as well as guidelines provided an overall framework for content and design, whereas the online survey gave an initial indication of the kind of subject areas health professionals, patients, and caregivers might expect the future site to cover. In this sense, the 'gap analysis' provided very rough guidelines for the construction of a beta site. However, it is important to remember that, because of the formative, iterative process that underpinned the design of the website, needs and gap analyses were ongoing as stakeholder representatives co-designed and debated the content of the site through the numerous iteration cycles.

Stages 3 and 4 – beta development and testing

An effective Project Reference Group was formed. The Project Reference Group (PRG) brought together nine professionals active in a range of palliative care-related fields and who held senior positions in councils, associations, and academia. Moreover, the PRG included a Project Officer who had extensive knowledge of the palliative care field. The Project Officer moderated most group discussions. The sizeable networks of these individuals were instrumental during the implementation phase of the project as many of the User Working Group members were recruited through these networks. Moreover, these participants brought to the project expertise in clinical palliative care, project management, team leadership, online educational technology, and research/evaluation techniques that could be drawn upon over the course of the project. Several members of the PRG joined an editorial panel that took responsibility for drafting editorial guidelines and monitoring content. The PRG functioned as the central mediation vehicle and most editorial decisions were discussed within this forum.

Two User Working Groups (UWG) were formed to participate in the sourcing and evaluation of content. One UWG drew on the expertise of focus group participants brought together by the Cancer Information & Support Services (CISS) run by The Cancer Council Victoria, as part of their activities directed at identifying unmet needs for people affected by cancer. The working group included 11 cancer patients, caregivers and volunteers. The other UWG was made up of 12 health professionals and included four general practitioners, two palliative care physicians, and six palliative care nurses representing nine organisations. Again the networks associated with these members were very extensive and greatly facilitated the user-based evaluation of the process. It was envisaged that these groups would meet on a monthly basis. UWG participants as well as members of their professional networks were asked to provide feedback in the form of summary grids (see additional file 1) after each design cycle.

Additional file 1. Summary Grid

Format: DOC Size: 36KB Download file

This file can be viewed with: Microsoft Word ViewerOpen Data

In addition to the PRG and UWGs, a substantial number of experts in fields such as palliative care, nursing, community support, or allied health made major contributions to the project. Also, a consumer focus group was conducted by a market research firm. This focus group dealt primarily with the look and feel (colour schemes, navigability) of the website. Because representatives of all major stakeholders were involved in the reference and working groups, it is reasonable to suggest that these stake holder groups had a direct input into the design of the web site. Moreover, bearing in mind that 197 and 166 participants respectively responded to the pre-development and evaluation online surveys, it is likely that over 300 individuals contributed to the design and evaluation of the PCV website.

Stage 5 – evaluation

An evaluation survey was attached to the new website. Not withstanding minor changes, the evaluation survey mirrored the content of the pre-development survey. Based on the experience with the pre-development survey, it was decided to separate the information for the public and health professionals. The pre-development survey did not make this separation. Moreover, the evaluation survey was more structured providing a list of possible responses that participants could tick in addition to an 'other' category followed by an open-ended prompt to specify this response.

Results

Stage 1 – literature review and online survey

The literature review revealed a growing recognition of the existence of unmet information needs regarding palliative care practice [6,9,35-44]. Recent evidence suggests that doctors, nurses, patients, and caregivers increasingly turn to the internet to meet some of these information needs [12-14,16,18,20,22,45-51]. One of the key concerns that surfaced in the literature focusing on online health is the issue of quality. A large number of studies express concerns with regard to the quality of information provided by web sites that post cancer health information on the Internet [52-58]. Most of these studies alert to the fact that websites commonly display misleading, incomplete, and/or erroneous information and that the confidentiality of data is often not assured [55,59-64]. Surveys of websites and the way users access them suggest that it is difficult and time consuming to identify good online resources and that the capacity of users to find and access valuable information is limited [65,66]. For most researchers the key problem faced by users is how to evaluate the quality of online information and how to establish the trustworthiness of sources.

In particular, researchers [4,6,7,57,64,66-72] identify the following key issues:

• Lack of editorial control,

• misleading or incomplete content,

• inappropriate design,

• inappropriate cultural bias,

• inappropriate commercial content,

• lack of readability (especially for cultural minorities or people with disabilities),

• inappropriate navigation tools,

• overall disorganisation of the site,

• accessibility of audio and visual resources, and

• problems regarding volume management and maintenance.

In response to these quality concerns, authors propose the use of tools to evaluate the quality of health information web sites [73,74], and/or to introduce obligatory standards or evidence-based editorial criteria to improve the quality of health information sites [54,66,75,76]. Yet, recent contributions to the field have been critical of evaluation tools, claiming that they are largely ineffective [77].

Other commentators have recommended the design of designated portal sites or gateway servers that organise and edit web content [78]. To some degree, sites such as OncoLink, the Cancer Net, Cancer Help UK, and others fit into this category. A number of articles give an overview of [79-84] and/or evaluate such sites [85,86]. Designated portal sites or gateway servers generally enjoy impressive approval ratings among clinicians as well as the general public. Nevertheless, recent articles have raised the issue of sustainability and costs, update, and maintenance work faced by online health information providers [87]. More recent evidence questions the proposition that the design of web portals lastingly improves the quality of online health information [88].

In summary, the literature suggests a need for a more comprehensive information and education strategy focusing on palliative care. It makes it clear that an understanding of the audiences to be targeted as well as their online information needs is crucial for the development of reliable and effective health information websites [22]. Moreover, it appears that for online health information resources to be efficient, four basic conditions have to be met: content needs to be clinically authoritative, accessible, sustainable, and has to respond to the information needs of potential user groups. Furthermore, the design process benefits from an ongoing involvement of stakeholder groups. However, there is a dearth of evidence regarding the information that should be included in a site providing educational information on palliative care to health professionals as well as to patients and associated caregivers as well as the methodology might be successfully employed to that end.

Pre-development survey

The survey attached to the existing website generated a total of 197 responses (see Table 1 below). All responses were considered valid, although not all questionnaires were completed in full. The majority of respondents who did not complete all ten questions left out the open ended questions. One hundred and thirty two (67.01%) respondents identified themselves as health professionals and 63 (31.98%) as non-health professionals.

Table 1. Responses to Pre- and Post-Development Surveys

The most important reasons respondents provided for accessing the existing palliative care website were to seek information in support of their role as health professionals (79.26%) and professional practice (13%), to find available resources (66.21%), and to find information that would assist them in their care-giving role (15%). Of the respondents 128 (64.97%) indicated that they had found the information they were looking for and 68 (34.52%) stated that they did not. Respondents to the existing website thought that in-depth information about topics such as pain, grief, volunteering and ethical issues should be included. Also noted was the absence of links for volunteers, a site map, a list of recommended publications, a listing of employment opportunities and short courses, comprehensive information for professionals, course information on professional practice, and contact details.

When asked if it was easy to find the information that they were seeking, 130 (65.99%) of the respondents to the existing site gave an affirmative response, whereas 57 (28.93%) responded negatively. Sixty two (31.47%) participants specified their answer. Of these, 26 (13.20%) gave positive feedback, whereas 20 (10.15%) provided constructive criticism. The issues raised by these respondents were:

• the need for better navigation tools,

• the site's limited content,

• the ineffectiveness of the site's search engine, and

• the fact that the site had not been updated.

Stage 2 – gap analysis of existing site

The expert panel discovered limitations in the organisation, content, language, accessibility, design, navigability, and readability of the existing PCV website. To some degree this finding was expected, given the limited resources available for the ongoing support and development of a site mainly administered by volunteers. Among the most important shortcomings noted by the panel was the very limited scope of the website, the lack of effective search capacity, the complexity of the language used and accessibility issues for people with reading difficulties. It was felt that the website was largely providing information to 'insiders' in the palliative care community but not to general health professionals nor to a substantial segment of caregivers, patients, and the general public.

The two working parties considered material from the gap analysis, the website search, as well as a preliminary analysis of emerging survey responses. Also the working party members' opinions of unmet needs as identified from their own experience or from interaction with their peer networks. Discussion ensued about how much information should be available for a quick scan approach, how much should be printable information sheets or pamphlets, and what links should be included to provide greater depth of knowledge. Each working party developed a framework for the health professional or the consumer web portal respectively. An overview of these frameworks is provided in Tables 2 and 3.

Table 2. Framework for health professional portal

Table 3. Framework for consumer portal

Moreover, working parties in conjunction with the PRG defined the overall scope of the new website delimiting the topics that would be covered by the site. Excluded from the site were topic such as careers and financial entitlements.

Stage 3 and 4 – beta development and testing

Because this phase has been published in detail elsewhere only a short summary of its key elements of this phase is given. The formative user evaluation underpinning the project worked exceptionally well. Perhaps the most surprising element of this phase was the level of support that participants were willing to invest in the project. Whereas the UWGs provided crucial information during the initial stages of the project, the editorial panel in conjunction with key user groups associated with the representatives participating in the UWGs provided valuable input during the evaluation of the beta test site. As anticipated, feedback occurred continuously throughout the project and was circulated through the PRG and UWGs. As a result, the PRG and UWGs were able to respond quickly to user feedback and shape the content and overall design of the website accordingly. Expert evaluators became important mediators of conflicts. Referring to 'good practice' guidelines and by providing their 'expert opinion', they, supported by the PRG, managed to resolve diverging views as well as conflicts of interests that surfaced within UWGs.

Stage 5 – evaluation survey

The online survey posted on the new website resulted in a total of 166 responses (see Table 1 below). A total of 121 (73%) respondents identified themselves as health professionals and 45 (27%) as care givers, patients, or members of the public.

Care givers, patients, and members of the public responding to the new website sought primarily information regarding the health care system (96%), about palliative care services in Victoria (40%), as well as information in order to help others with an illness (33%), help the person they cared for (31%), and help them cope (27%). Health professionals responding to the new website (B) sought information about education (50%), symptom management (37%), advance care planning (31%), loss and grief (25%), talking with clients and families (23%), Palliative Care Victoria (22%), death (21%), services available to their clients (21%), and a resource website for clients (17%).

Of the respondents, 137 (82%) indicated that they had found all or some of the desired information and 18 (11%) stated that they did not. Respondents of the evaluation survey thought that topics such as further education, conferences, scholarships, as well as lists of places to go to as a family, signs of dying/clinical information, volunteering and financial entitlements should be considered.

A total of 122 (74%) respondents claimed that it was easy to access the desired information, whereas 38 (23%) claimed that it was not.

Comments focusing on the new website were positive regarding content and design.

Discussion

Involving users in the identification of content and links for a palliative care website as well as the evaluation of the site's 'look and feel' is time-consuming and requires initial resources, strong networking skills, and ongoing commitment. However, the benefits of engaging key stakeholders in defining and refining the scope and content of such a website outweigh the disadvantages. Over the course of this study, user involvement led to the widening of the scope of the website and alerted the project group to shortcomings regarding content, language, presentation, and usability.

The original intention of the expert re-developed website was to provide information to the consumers and for health professionals in the community who did not have continual access to palliative care expertise. User involvement significantly augmented these aims. In addition to the original intention, the stakeholder-conducted gap analysis in conjunction with the user online survey identified the need for a site that

• provided reputable expert or evidence-based information,

• addressed information requirements in areas such as professional practice, ethical dilemmas or practice standards, as well as topics related to managing palliative pain, grief, advanced care planning, or home care,

• provided clinical content in accessible lay language for the general public,

• guided consumers through the health care system

• provided tools to enhance medical and consumer decision making,

• supplied printable pamphlets for patients and caregivers,

• regularly monitored and updated links and material,

• provided key points that can be easily visually scanned, and

• does not contain distracting graphics that increase download time.

These points would have been missed if the project had proceeded with a common, expert-based web-design.

To some extent, the success effect of the iterative, participatory design of the website is visible in the evaluation survey posted to the new website. For instance, a comparison of the pre-development survey with the evaluation survey shows a decrease of visitors that did not find the information they were looking for from 35% to 11%. Concurrently, the percentage of participants who thought it easy to find the desired information increased from 66 to 74.

Interestingly, since going live, the new website has become an education resource for nursing staff, students, as well as allied health workers. In fact, 46% of the evaluation survey respondents were nurses. Moreover, the evaluation survey suggests that an, albeit small, but nevertheless important number of care givers and patients is accessing the site. Seventeen percent of evaluation survey respondents were care givers or patients. The overall feedback by health professionals as well as patients and caregivers to the new website was, in contrast to the pre-development survey, exclusively positive. What is more, all of the additional topics that health professional thought might be considered for inclusion were outside the scope of the website as defined by UWGs and the PRG.

Yet, the evaluation survey also demonstrates that the portal for patients, families, and their caregivers might require some finetuning. The evaluation survey suggests that whereas the information needs of health professionals are covered by the new site, this is not entirely the case for patients as well as their families and care givers. Given that the information needs of patients and care givers are likely to be more diverse than those of health professionals, this was expected, however. What is more, only two (1%) of respondents to the evaluation survey were General Practitioners (GPs). The reason for this low share in the overall population of respondents has not been analysed. It has to be born in mind that the evaluation survey reflects only on the first eleven months in the life span of the new website. Future surveys will show whether the site will become a central reference for care givers, patients, and GPs.

What is more, the re-development of the website accompanied by the positive feedback the organisation received has led to significant changes in the way PCA manages information. Whereas the old website functioned mainly as a provider of pamphlets and leaflets on palliative care, the new website has assumed a much more central role in the overall operation of the organisation providing education and information. Moreover, PCA plans to web-cast public relations items on the site. Moreover, encouraged by the re-development of the site, the PCV has applied for additional funding in order to integrate web-based information more fully into its communication strategy. Having become aware of the potential of its online strategy, web-support has assumed a more important role in the organisation. This in conjunction with the additional funding PCV was able to attract has impacted positively on the sustainability of the website.

Limitations

Several limitations were recorded. Perhaps the most important, and obvious, limitation is the usefulness of the pre-development survey in scoping possible web content for a future potential audience that is not yet accessing the site. Whereas the pre-development survey indicated that more non-health professionals accessed the site than was expected, the pre-development survey gave only limited indication as to the information needs of patients and their care givers. In fact, the development process relied to large extent on the input of the UWG as well as the iterative evaluations to define the portal for non health professionals.

Another limitation that emerged more clearly in connection with subsequent projects is the use of volunteer participants. It is important to bear in mind that not all projects are suited to a development phase based largely on volunteer participation of busy health professionals. Although the overall cost of the project (A$ 184,872) was comparatively low, the study was fairly resource intense. In particular, the study relied on a large number of participants who, over the course of three years, generously donated their time and energy. Clearly, for participants to make such a sustained contribution, they need to be extremely committed. Hence, it is important that web developers bear in mind that the salience of the project outcome is likely to influence the commitment levels and, by extension, the support of participants. Furthermore, because of the need to mobilise all major stakeholders and to involve consumer representatives, this kind of approach requires strong networking skills and only individuals with good links to key user groups will be likely to gain the support of key user representatives.

Also, because of the large number of individuals involved in the project and because of the iterative evaluation cycles, an enormous amount of information will be generated. The administration and processing of this information requires well-honed project management skills. In part this had to do with the ineffectiveness of one of the research tools: the summary grid. The summary grid asked for an assessment of the content of each of the web pages associated with these questions using the following criteria: clarity of information, accuracy of information, usefulness of links, target audience for the page, and layout and further suggestions. This feedback method was time-consuming and only members of the consumer panel and a few members of the health professional panel provided feedback in this manner. However the written feedback was most thorough and provided excellent guidance to adjust language and content, to provide further information, and/or to include further links. However, the majority of participants provided feed-back in an ad-hoc, unstructured manner that did not correspond with the data-collection time points we had set. This generated an extraordinary amount of information that, at times, threatened to overwhelm our project officer. This prompted us to strengthen the methodology of subsequent web design projects involving user participation.

For reasons beyond our control, we were unable to generate a denominator for the responses to the initial and subsequent surveys, as well as for the subsequent utilisation of the website. As a result we are unable to complete the rhetorical loop demonstrating an improved usage of the website. This clearly limits the explanatory power of the study.

Conclusion

A growing body of literature and government reports highlight the existence of a significant information gap with regard to palliative care. As more and more people use the internet to meet their health information needs, the question of what kind of palliative care information internet users are actually looking for becomes central to any online information strategy. However, research that focuses on the palliative care information needs of internet users is scarce. This study illustrates how a user-based methodology involving key stakeholders and potential users can contribute successfully to the development of content, language, and design of the website as well as to its evaluation. Web developers are encouraged to involve potential users in their website development and pay close attention to the content, language, and accessibility needs of their respective audiences.

Competing interests

The author(s) declare that they have no competing interests.

Authors' contributions

AS and KS conceptualised and designed the project. AS, KS, MA, RW, TG, and AO were involved in the data collection effort and provided critical input through the project. AS and GO conducted the final data analyses and data interpretation and drafted the manuscript. MA, RW, TG, and AO revised the manuscript and contributed to its intellectual content. All authors read and approved the final manuscript.

Acknowledgements

The project was funded by the Department of Health and Ageing, Canberra, Australia. Moreover, to contribute to this project authors were sponsored by their host organisations: Annette F. Street (Prof.) Clinical Nursing La Trobe University, Kathleen Swift Research Assistant Clinical Nursing La Trobe University, Merilyn Annells (Prof.) Community Nursing La Trobe University, Roger Woodruff (Dr) Medical Oncologist and Director of Palliative Care Austin Health, Terry Gliddon (MA AppSci) Research and Development Manger Royal District Nursing Service, Anne Oakley Manager Melbourne Citymission Palliative Care, and Goetz Ottman (Dr) Research Fellow, Clinical Nursing, La Trobe University.

References

  1. Palliative Care Victoria [http://www.pallcarevic.asn.au] webcite

  2. Healthcare Management Advisors: Paediatric Palliative Care Service Model Review. Canberra , The National Palliative Care Program; 2004.

  3. National Palliative Care Strategy: A National Framework for Palliative Care Service Development Canberra , Australian Government: Department of Health and Ageing; 2000.

  4. Sullivan K, Johnston L, Colyer C, Beale J, Willis J, Harrison J, Welsh K: National Indigenous Palliative Care Needs Study. Canberra , The National Palliative Care Program; 2003.

  5. Mount Olivet Community Services: Research Study into the Educational, Training and Support Needs of General Practitioners in Palliative Care. Canberra , The National Palliative Care Program; 2003.

  6. Friedman DB, Hoffman-Goetz L, Arocha JF: Readability of cancer information on the internet.

    Journal of Cancer Education 2004, 19(2):117-122. PubMed Abstract | Publisher Full Text OpenURL

  7. Birru M, Steinman RA: Online health information and low-literacy African Americans.

    Journal of Medical Internet Research 2004, 6(3):e26. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  8. PPR Consulting: Promoting Partnerships in Palliative Care Services. Melbourne , Department of Human Services Victoria; 2004.

  9. Davies E, Higginson IJ: The Solid Facts: Palliative Care. World Health Organization; 2004.

  10. Thain A, Wales A: Information needs of specialist healthcare professionals: a preliminary study based on the West of Scotland Colorectal Cancer Managed Clinical Network.

    Health Information & Libraries Journal 2005, 22(2):133-142. Publisher Full Text OpenURL

  11. Bleakley A, Merzel CR, VanDevanter NL, Messeri P: Computer access and Internet use among urban youths.

    American Journal of Public Health 2004, 94(5):744-746. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  12. Bright MA, Fleisher L, Thomsen C, Morra ME, Marcus A, Gehring W: Exploring e-Health usage and interest among cancer information service users: the need for personalized interactions and multiple channels remains.

    Journal of Health Communication 2005, 10 Suppl 1:35-52. PubMed Abstract | Publisher Full Text OpenURL

  13. Dickerson S, Reinhart AM, Feeley TH, Bidani R, Rich E, Garg VK, Hershey CO: Patient Internet use for health information at three urban primary care clinics.

    Journal of the American Medical Informatics Association 2004, 11(6):499-504. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  14. Doney L, Barlow H, West J: Use of libraries and electronic information resources by primary care staff: outcomes from a survey.

    Health Information & Libraries Journal 2005, 22(3):182-188. Publisher Full Text OpenURL

  15. Escoffery C, Miner KR, Adame DD, Butler S, McCormick L, Mendell E: Internet use for health information among college students.

    Journal of American College Health 2005, 53(4):183-188. PubMed Abstract | Publisher Full Text OpenURL

  16. Rees T: More Americans going online looking for health information.

    Profiles in Healthcare Marketing 2005, 21(4):2. OpenURL

  17. Balmer C: The information requirements of people with cancer: where to go after the "patient information leaflet"?

    Cancer Nursing 2005, 28(1):36-44. PubMed Abstract | Publisher Full Text OpenURL

  18. Basch EM, Thaler HT, Shi W, Yakren S, Schrag D: Use of information resources by patients with cancer and their companions.

    Cancer 2004, 100(11):2476-2483. PubMed Abstract | Publisher Full Text OpenURL

  19. Brotherton JM: Internet is indeed useful source for patients with cancer.

    BMJ 2004, 328(7444):898. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  20. Mick J: Patients used the internet to transform their cancer experiences.

    Evidence Based Nursing 2004., 7(4) OpenURL

  21. Ziebland S, Chapple A, Dumelow C, Evans J, Prinjha S, Rozmovits L: How the internet affects patients' experience of cancer: a qualitative study.

    BMJ 2004, 328(7439):564. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  22. Pereira J, Bruera E, Quan H: Palliative Care on the Net: An online survey of health care professionals.

    Journal of Palliative Care 2001, 17(1):41-45. PubMed Abstract OpenURL

  23. Palliative Care Australia: Standards for providing quality palliative care for all Australians. Yarralumla, ACT ; 2005.

  24. Palliative Care Australia: A Guide to Palliative Care Service Development: A population based approach.

    2005, 52.

  25. Reason P, Bradbury H: Introduction: Inquiry and participation in search of a world worthy of human aspiration. In Handbook of action research: participatory inquiry and practices. Edited by Reason P, Bradbury H. London , SAGE; 2001:1-14. OpenURL

  26. Reason P, Bradbury H: Handbook of action research : participative inquiry and practice. London , SAGE; 2001:468.

  27. Street A: Action research. In Research methods for nursing and health science. Edited by Minichiello V, Sullivan G, Greenwood K, Axford R. Sydney , Pearson Prentice Hall; 2004:278-294. OpenURL

  28. Dick B: Utopia made practical? Action Research comes of age.

    Forum: Qualitative Social Research 2002., 3(1) OpenURL

  29. Kemmis S, McTaggart R: Participatory action research. In Strategies of Qualitative Inquiry. Edited by Denzin NK, Lincoln YS. Thousand Oaks , SAGE; 2003:91-102. OpenURL

  30. Williamson GR, Prosser S: Action research: Politics, ethics and participation.

    J Adv Nurs 2002, 40(5):587-593. PubMed Abstract | Publisher Full Text OpenURL

  31. Gabbard JL, Hix D, Swan JE: User-centered design and evaluation of virtual environments. In Computer Graphics and Applications, IEEE. Volume 19. Virginia Polytech. Inst. & State Univ., Blacksburg, VA; 1999::51-69. Publisher Full Text OpenURL

  32. Weston CMALB T: A Model for Understanding formative evaluation in instructional design.

    Education Technology Research and Development 1995, 43(3):29-48. Publisher Full Text OpenURL

  33. Polson PG, Lewis C, Rieman J, Wharton C: Cognitive walkthroughs: a method for theory-based evaluation of user interfaces. In Int J Man-Mach Stud. Volume 36. Academic Press Ltd.; 1992::741-773. Publisher Full Text OpenURL

  34. Wilson B, Jonassen D, Cole P: Cognitive approaches to instructional design. In The ASTD handbook of instructional technology. Edited by Piskurich GM. New York , McGraw-Hill; 1993. OpenURL

  35. Families need more Information about Postmortem Examinations

    BMJ 2002, 325:1121. Publisher Full Text OpenURL

  36. Contro N, Larson J, Scofiled S, Sourkes B, Cohen H: Family Perspectives on the Quality of Pediatric Palliative Care.

    Archives of Pediatric & Adolescent Medicine 2002, 156(1):14. OpenURL

  37. Coolidge C: Death Wish.

    Forbes 2001, 206. OpenURL

  38. Davies E: What are the Palliative Care Needs of older People and how might they be Met. In Health Evidence Network. World Health Organisation; 2004. OpenURL

  39. Fallowfield L: Palliative Care; Dying cancer patients need more information from doctors.

    Cancerweekly Plus 2002. OpenURL

  40. Foley Pierce S: Improving End-of-Life Care: Gathering Suggestions from Family Members.

    Nursing Forum 1999, 34(2):5. Publisher Full Text OpenURL

  41. Kirchhoff KT, Walker L, Hutton A, Spuhler V, Vaughan Cole B, Clemmer T: The Vortex: Families' Experiences with Death in the Intensive Care Unit.

    American Journal of Critical Care 2002, 11(3):200. PubMed Abstract | Publisher Full Text OpenURL

  42. Murray SA, Boyd K, Kendall M, Worth A, Benton FT, Clausen H: Dying of lung cancer or cardiac failure: prospective qualitative interview study of patients and their carers in the community.

    BMJ 2002, 325:929-932. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  43. Kirk P, Kirk I, Kristjanson LJ: What do patients receiving palliative care for cancer and their families want to be told? A Canadian and Australian qualitative study.

    BMJ 2004, 328(7452):1343. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  44. Wilkes L, White K, O'Riordan L: Empowerment through Information: Supporting rural Families of Oncology Patients in Palliative Care.

    Australian Journal of Rural Health 2000, 8(1):41-46. PubMed Abstract | Publisher Full Text OpenURL

  45. Pereira J, Bruera E, Macmillan K, Kavanagh S: Palliative Cancer Patients and Their Families on the Internet: Motivation and Impact.

    Journal of Palliative Care 2000, 16(4):13. PubMed Abstract OpenURL

  46. Wozar JA, Worona PC: The use of online information resources by nurses.

    Journal of the Medical Library Association 2003, 91(2):216-221. PubMed Abstract | PubMed Central Full Text OpenURL

  47. Balmer C: The information requirements of people with cancer: where to go after the "patient information leaflet"?[see comment].

    Cancer Nursing 2005, 28(1):36-44. PubMed Abstract | Publisher Full Text OpenURL

  48. Brotherton JM: Internet is indeed useful source for patients with cancer.[comment].

    BMJ 2004, 328(7444):898. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  49. Baker L, Wagner TH, Singer S, Bundorf MK: Use of the Internet and E-mail for Health Care Information: Results From a National Survey.

    JAMA 2003, 289(18):2400-2406. PubMed Abstract | Publisher Full Text OpenURL

  50. Pereira J, Bruera E: The Internet as a resource for palliative care and hospice: A review and proposal.

    Journal of Pain and Symptom Management 1998, 16(1):59-68. Publisher Full Text OpenURL

  51. Bath PA, Clark D, Wright M: The International Observatory on End of Life Care (IOELC): An Information Resource for Palliative Care.

    Health Informatics Journal 2004, 10(2):121-126. Publisher Full Text OpenURL

  52. Ekman A, Hall P, Litton JE: Can we trust cancer information on the Internet?--A comparison of interactive cancer risk sites.

    Cancer Causes & Control 2005, 16(6):765-772. PubMed Abstract | Publisher Full Text OpenURL

  53. Felber SH: Re: bladder cancer facts: accuracy of information on the internet.[comment].

    Journal of Urology 2004, 171(6 Pt 1):2383; author reply 2383. PubMed Abstract | Publisher Full Text OpenURL

  54. Ilic D, Risbridger G, Green S: Searching the Internet for information on prostate cancer screening: an assessment of quality.

    Urology 2004, 64(1):112-116. PubMed Abstract | Publisher Full Text OpenURL

  55. Jorgensen KJ, Gotzsche PC: Presentation on websites of possible benefits and harms from screening for breast cancer: cross sectional study.[see comment].

    BMJ 2004, 328(7432):148. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  56. Molassiotis A, Xu M: Quality and safety issues of web-based information about herbal medicines in the treatment of cancer.

    Complementary Therapies in Medicine 2004, 12(4):217-227. PubMed Abstract | Publisher Full Text OpenURL

  57. Viswanath K: Science and society: the communications revolution and cancer control. [Review] [62 refs].

    Nature Reviews Cancer 2005, 5(10):828-835. PubMed Abstract | Publisher Full Text OpenURL

  58. Weissenberger C, Schultze-Seemann W: Re: bladder cancer facts: accuracy of information on the internet.

    Journal of Urology 2004, 172:1544. PubMed Abstract | Publisher Full Text OpenURL

  59. Burkell J, Campbell DG: "What does this mean?" How Web-based consumer health information fails to support information seeking in the pursuit of informed consent for screening test decisions.

    Journal of the Medical Library Association 2005, 93(3):363-373. PubMed Abstract | PubMed Central Full Text OpenURL

  60. Coleman J, Olsen SJ, Sauter PK, Baker D, Hodgin MB, Stanfield C, Emerling A, Hruban RH, Nolan MT: The effect of a Frequently Asked Questions module on a pancreatic cancer Web site patient/family chat room.

    Cancer Nursing 2005, 28(6):460-468. PubMed Abstract | Publisher Full Text OpenURL

  61. Fagerlin A, Rovner D, Stableford S, Jentoft C, Wei JT, Holmes-Rovner M: Patient education materials about the treatment of early-stage prostate cancer: a critical review.

    Annals of Internal Medicine 2004, 140(9):721-728. PubMed Abstract | Publisher Full Text OpenURL

  62. Nilsson-Ihrfelt E, Fjallskog ML, Blomqvist C, Ahlgren J, Edlund P, Hansen J, Malmberg L, Villman K, Andersson G: Breast cancer on the Internet: the quality of Swedish breast cancer websites.

    Breast 2004, 13(5):376-382. PubMed Abstract | Publisher Full Text OpenURL

  63. Schmidt K, Ernst E: Assessing websites on complementary and alternative medicine for cancer.

    Annals of Oncology 2004, 15(5):733-742. PubMed Abstract | Publisher Full Text OpenURL

  64. Simon C, Hegedus S: Exploring websites on cancer clinical trials: an empirical review.

    Contemporary Clinical Trials 2005, 26(5):530-533. PubMed Abstract | Publisher Full Text OpenURL

  65. Jones SD: "Information Rx" project launched in Virginia.

    National Network 2005, 29(3):20. PubMed Abstract OpenURL

  66. Morahan-Martin JM: How internet users find, evaluate, and use online health information: a cross-cultural review. [Review] [61 refs].

    Cyberpsychology & Behavior 2004, 7(5):497-510. OpenURL

  67. Nahm ES, Preece J, Resnick B, Mills ME: Usability of health Web sites for older adults: a preliminary study.

    CIN: Computers, Informatics, Nursing 2004, 22(6):326-34; quiz 335-6. OpenURL

  68. Boulos MN: British internet-derived patient information on diabetes mellitus: is it readable?

    Diabetes Technology & Therapeutics 2005, 7(3):528-535. PubMed Abstract | Publisher Full Text OpenURL

  69. Changrani J, Gany F: Online cancer education and immigrants: effecting culturally appropriate websites.

    Journal of Cancer Education 2005, 20(3):183-186. PubMed Abstract | Publisher Full Text OpenURL

  70. Fikar CR, Keith L: Information needs of gay, lesbian, bisexual, and transgendered health care professionals: results of an Internet survey.

    Journal of the Medical Library Association 2004, 92(1):56-65. PubMed Abstract | PubMed Central Full Text OpenURL

  71. Zeng X, Parmanto B: Web content accessibility of consumer health information web sites for people with disabilities: a cross sectional evaluation.

    Journal of Medical Internet Research 2004, 6(2):e19. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  72. Weissenberger C, Jonassen S, Beranek-Chiu J, Neumann M, Muller D, Bartelt S, Schulz S, Monting JS, Henne K, Gitsch G, Witucki G: Breast cancer: patient information needs reflected in English and German web sites.

    British Journal of Cancer 2004, 91(8):1482-1487. PubMed Abstract | Publisher Full Text OpenURL

  73. Bernstam EV, Sagaram S, Walji M, Johnson CW, Meric-Bernstam F: Usability of quality measures for online health information: Can commonly used technical quality criteria be reliably assessed?

    International Journal of Medical Informatics 2005, 74(7-8):675-683. PubMed Abstract | Publisher Full Text OpenURL

  74. Childs S: Developing health website quality assessment guidelines for the voluntary sector: outcomes from the Judge Project.

    Health Information & Libraries Journal 2004, 21 Suppl 2:14-26. OpenURL

  75. Aarimaa M: Telemedicine--contribution of ICT to health.

    Studies in Health Technology & Informatics 2004, 100:111-116. OpenURL

  76. Gimenez-Perez G, Caixas A, Gimenez-Palop O, Gonzalez-Clemente JM, Mauricio D: Dissemination of 'patient-oriented evidence that matters' on the Internet: the case of Type 2 diabetes treatment.

    Diabetic Medicine 2005, 22(6):688-692. PubMed Abstract | Publisher Full Text OpenURL

  77. Bernstam EV, Shelton DM, Walji M, Meric-Bernstam F: Instruments to assess the quality of health information on the World Wide Web: what can our patients actually use?

    International Journal of Medical Informatics 2005, 74(1):13-19. PubMed Abstract | Publisher Full Text OpenURL

  78. Darmoni SJ, Dahamna B, Roth-Berghofer TR: Seal of transparency heritage in the CISMeF quality-controlled health gateway.

    BMC Medical Informatics & Decision Making 2004, 4:15. BioMed Central Full Text OpenURL

  79. Chambers L, McDowall J, Gelb B: A global children's hospice and palliative care website.

    International Journal of Palliative Nursing 2005, 11(6):292-293. PubMed Abstract OpenURL

  80. Gavrin JR: The web site of the center to advance palliative care.

    Journal of Pain & Palliative Care Pharmacotherapy 2004, 18(4):77-79. PubMed Abstract | Publisher Full Text OpenURL

  81. Junghans TB, Sevin IF, Ionin B, Seifried H: Cancer information resources: digital and online sources.

    Toxicology 2004, 198(1-3):177-193. PubMed Abstract | Publisher Full Text OpenURL

  82. Lewis D, Gundwardena S, El Saadawi G: Caring connection: developing an Internet resource for family caregivers of children with cancer.

    CIN: Computers, Informatics, Nursing 2005, 23(5):265-274. Publisher Full Text OpenURL

  83. Packer BR, Yeager M, Burdett L, Welch R, Beerman M, Qi L, Sicotte H, Staats B, Acharya M, Crenshaw A, Eckert A, Puri V, Gerhard DS, Chanock SJ: SNP500Cancer: a public resource for sequence validation, assay development, and frequency analysis for genetic variation in candidate genes.

    Nucleic Acids Research 2006, 34(Database issue):D617-21. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  84. Williams M: Palliative care web sites.

    Home Healthcare Nurse 2005, 23(11):736-737. PubMed Abstract | Publisher Full Text OpenURL

  85. Kosinska J, Slowikowski P: Technical aspects of portal technology application for e-health systems. [Review] [18 refs].

    Studies in Health Technology & Informatics 2004, 105:12-20. OpenURL

  86. Winefield HR, Coventry BJ, Lambert V: Setting up a health education website: practical advice for health professionals.

    Patient Education & Counseling 2004, 53(2):175-182. Publisher Full Text OpenURL

  87. De Moor GJ, Claerhout B, van Maele G, Dupont D: e-Health standardization in Europe: lessons learned.

    Studies in Health Technology & Informatics 2004, 100:233-237. OpenURL

  88. Glenton C, Paulsen EJ, Oxman AD: Portals to Wonderland: health portals lead to confusing information about the effects of health care.

    BMC Medical Informatics & Decision Making 2005, 5(1):7. BioMed Central Full Text OpenURL

Pre-publication history

The pre-publication history for this paper can be accessed here:

http://www.biomedcentral.com/1472-6947/7/26/prepub