An adoption agency is responsible for the placement of children who are legally eligible to be adopted (whose biological parents' parental rights have been or are in the process of being terminated) with adoptive families.
In order to do this, adoption agencies must be licensed by the state(s) in which they operate, meeting all state requirements regarding staffing, qualifications of staff, services provided, and more.
There are two basic types of adoption agencies: public and private.
Public agencies (usually called Social Services) are state- or county-operated and are responsible for the adoptive placements of children in the foster care system. Public agencies perform home studies, assist pre-adoptive parents with locating a child, provide pre-adoption education, legal services, post-placement visits, and supervise the payment of state and federal funds (subsidies or adoption assistance payments) in the case of children with special needs. They do not operate outside the U.S.
Private agencies work with the adoptive placements of children in the U.S. and, if licensed to operate outside the U.S., with the placements of children living in those countries. Private agencies, either directly through their own staffing or using associated service providers, perform home studies, provide legal services, and more. Some private agencies offer pre- and post-adoption education and services, and in the case of international adoptions, may conduct post-placement visits as required by the child's home country.
Private agencies may be for-profit or non-profit.
Like any "reference" situation, an agency is only going to give out the names and contact information of people who have been happy with their service. The true measure of a good agency is how they handle things if and when there are problems. Very few adoptions are completely problem-free due to constantly changing laws and regulations, both in the USA and in the foreign countries.
It is good to talk to the references given to you by an agency, but it is also recommended that you seek out other adoptive parents who have used the agency and get their assessments. This can often be done through the Internet (yahoo groups, blogs, sites like this one!) and through your own adoption community.
Beware of some adoptive parents who are just out to trash an agency without a valid reason. Some adoptions are unsuccessful due to the actions or inactions of the adoptive parents. Get ALL the facts and use your own judgment to determine if an agency is reputable, ethical and concerned enough about you as an adoptive family to be "your" agency.